Advances in Multidisciplinary Retrieval: First Information Retrieval Facility Conference, IRFC 2010, Vienna, Austria, May 31, 2010, Proceedings (Lecture ... Applications, incl. Internet/Web, and HCI) [1st Edition.] 3642130836, 9783642130830

Citation preview

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

6107

Hamish Cunningham Allan Hanbury Stefan Rüger (Eds.)

Advances in Multidisciplinary Retrieval First Information Retrieval Facility Conference, IRFC 2010 Vienna, Austria, May 31, 2010 Proceedings

13

Volume Editors Hamish Cunningham University of Sheffield Dept. of Computer Science Sheffield S1 4DP, UK E-mail: [email protected] Allan Hanbury Information Retrieval Facility 1040 Vienna, Austria E-mail: [email protected] Stefan Rüger Knowledge Media Institute The Open University Milton Keynes, MK7 6AA, UK E-mail: [email protected]

Library of Congress Control Number: 2010926587 CR Subject Classification (1998): H.3, I.2.4, H.5, H.4, I.2, C.2 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-642-13083-6 Springer Berlin Heidelberg New York 978-3-642-13083-0 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180

Preface

These proceedings contain the refereed papers and posters presented at the first Information Retrieval Facility Conference (IRFC), which was held in Vienna on 31 May 2010. The conference provides a multi-disciplinary, scientific forum that aims to bring young researchers into contact with industry at an early stage. IRFC 2010 received 20 high-quality submissions, of which 11 were accepted and appear here. The decision whether a paper was presented orally or as poster was solely based on what we thought was the most suitable form of communication, considering we had only a single day for the event. In particular, the form of presentation bears no relation to the quality of the accepted papers, all of which were thoroughly peer reviewed and had to be endorsed by at least three independent reviewers. The Information Retrieval Facility (IRF) is an open IR research institution, managed by a scientific board drawn from a panel of international experts in the field whose role is to promote the highest quality in the research supported by the facility. As a non-profit research institution, the IRF provides services to IR science in the form of a reference laboratory, hardware and software infrastructure. Committed to Open Science concepts, the IRF promotes publication of recent scientific results and newly developed methods, both in traditional paper form and as data sets freely available to IRF members. Such transparency ensures objective evaluation and comparability of results and consequently diversity and sustainability of their further development. The IRF is unique in providing a powerful supercomputing infrastructure that is exclusively dedicated to semantic processing of text. It has at its heart a huge collection of patent documents representing the global archive of ideas and inventions. This data is housed in an environment that allows large-scale scientific experiments on ways to manage and retrieve this knowledge. This collection of real data allows IR researchers from all over the world to experiment for the first time on a realistic data corpus. The quality of search results is reviewed and evaluated in many fields but especially those specialising in patent search. IRF conferences wish to resonate in particular with young researchers, who are interested in discussing results obtained using the IRF infrastructure and data resources; learning about complementary technologies; applying their research efforts to real business needs; and joining the international research network of the IRF. The first IRFC aimed to tackle four complementary research areas: – – – –

information retrieval semantic web technologies for IR natural language processing for IR large-scale or distributed computing for the above areas

We believe that this first conference has achieved most of these aims and we look forward to many more instances of the IRFC.

VI

Preface

Acknowledgements. It is never easy to make a conference happen. Our sincere thanks go out to: – the IRF executive board: Francisco Eduardo De Sousa Webber, Daniel Schreiber and Sylvia Thal, for their inspiration, for getting the ball rolling and for exceptional organisational talent – the professional team at the IRF and Matrixware for their help in preparing the conference and this volume: Marie-Pierre Garnier; Katja Mayer; Mihai Lupu; Giovanna Roda; Helmut Berger – the IRF scientific board and John Tait (IRF CSO) for their guidance – Niraj Aswani for his help in preparing the proceedings – the conference programme committee for their hard work reviewing and commenting on the papers: • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •

Yannis Avrithis, CERTH, Greece Leif Azzopardi, University of Glasgow, UK Ricardo Baeza-Yates, Yahoo! Research, Spain Jamie Callan, Carnegie Mellon University, USA Paul Clough, University of Sheffield, UK W. Bruce Croft, University of Massachusetts, USA Norbert Fuhr, University Duisburg-Essen, Germany Wilfried Gansterer, University of Vienna, Austria Charles J. Gillan, Queen’s University Belfast, UK Gregory Grefenstette, Exalead, France Preben Hansen, Swedish Institute of Computer Science, Sweden David Hawking, Funnelback Internet and Enterprise Search, Australia Joemon Jose, University of Glasgow, UK Noriko Kando, National Institute of Informatics (NII), Japan Philipp Koehn, University of Edinburgh, UK Wessel Kraaij, TNO, The Netherlands Udo Kruschwitz, University of Essex, UK Dominique Maret, Matrixware Information Services, Austria Marie-Francine Moens, Catholic University of Leuven, Belgium Henning M¨ uller, University of Applied Sciences Western Switzerland, Switzerland Walid Najjar, University of California Riverside, USA Arcot Desai Narasimhalu, Singapore Management University, Singapore Fredrik Olsson, Swedish Institute of Computer Science, Sweden Miles Osborne, University of Edinburgh, UK Andreas Rauber, Vienna University of Technology, Austria Magnus Sahlgren, Swedish Institute of Computer Science, Sweden Mark Sanderson, University of Sheffield, UK Frank J. Seinstra, Vrije Universiteit, The Netherlands John Tait, IRF, Austria Benjamin T’sou, City University of Hong Kong, China Christa Womser-Hacker, University of Hildesheim, Germany

Preface

VII

– the keynote speakers Mark Sanderson, University of Sheffield, UK, and David Hawking, Funnelback Internet and Enterprise Search, Australia, for providing excellent and inspiring talks – the sponsors: • Matrixware Information Services • ESTeam • The University of Sheffield • STI International • Yahoo! Labs – the Austrian Federal Ministry of Science and Research and Wien Kultur for their support – BCS — The Chartered Institute for IT for endorsing the conference – Matt Petrillo for understanding what Hamish was talking about (at least some of the time) We hope you enjoy the results! May 2010

Hamish Cunningham, University of Sheffield http://www.dcs.shef.ac.uk/˜hamish General Chair Allan Hanbury, Information Retrieval Facility http://www.ir-facility.org/about/people/staff Publications Chair Stefan R¨ uger, The Open University http://people.kmi.open.ac.uk/stefan Programme Chair

VIII

Preface

Supported by

Endorsed by

Table of Contents

Scaling Up High-Value Retrieval to Medium-Volume Data . . . . . . . . . . . . . Hamish Cunningham, Allan Hanbury, and Stefan R¨ uger

1

Sentence-level Attachment Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M-Dyaa Albakour, Udo Kruschwitz, and Simon Lucas

6

Rank by Readability: Document Weighting for Information Retrieval . . . Neil Newbold, Harry McLaughlin, and Lee Gillam

20

Knowledge Modeling in Prior Art Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . Erik Graf, Ingo Frommholz, Mounia Lalmas, and Keith van Rijsbergen

31

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Benjamin Roth and Dietrich Klakow

47

Exploring Contextual Models in Chemical Patent Search . . . . . . . . . . . . . . Jay Urbain and Ophir Frieder

60

Measuring the Variability in Effectiveness of a Retrieval System . . . . . . . . Mehdi Hosseini, Ingemar J. Cox, Natasa Millic-Frayling, and Vishwa Vinay

70

An Information Retrieval Model Based on Discrete Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alberto Costa and Massimo Melucci Logic-Based Retrieval: Technology for Content-Oriented and Analytical Querying of Patent Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Iraklis Angelos Klampanos, Hengzhi Wu, Thomas Roelleke, and Hany Azzam Automatic Extraction and Resolution of Bibliographical References in Patent Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrice Lopez An Investigation of Quantum Interference in Information Retrieval . . . . . Massimo Melucci Abstracts versus Full Texts and Patents: A Quantitative Analysis of Biomedical Entities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bernd M¨ uller, Roman Klinger, Harsha Gurulingappa, Heinz-Theodor Mevissen, Martin Hofmann-Apitius, Juliane Fluck, and Christoph M. Friedrich Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

84

100

120 136

152

167

Scaling Up High-Value Retrieval to Medium-Volume Data Hamish Cunningham1 , Allan Hanbury2 , and Stefan R¨ uger3 1

Department of Computer Science, University of Sheffield Regent Court, 211 Portobello Street, Sheffield S1 4DP, UK [email protected] 2 Information Retrieval Facility Operngasse 20b, 1040 Vienna, Austria [email protected] 3 Knowledge Media Institute, The Open University Walton Hall, Milton Keynes MK7 6AA, UK [email protected]

Abstract. We summarise the scientific work presented at the first Information Retrieval Facility Conference [3] and argue that high-value retrieval with medium-volume data, exemplified by patent search, is a thriving topic in a multidisciplinary area that sits between Information Retrieval, Natural Language Processing and Semantic Web Technologies. We analyse the parameters that condition choices of retrieval technology for different sizes and values of document space, and we present the patent document space and some of its characteristics for retrieval work.

1

How Much Search Can You Afford?

Information retrieval (IR) technology has proliferated in rough proportion to the expansion of knowledge and information as a central factor in economic success. How should end-users choose between the many available options? Three main dimensions condition the choice: –Volume. The GYM big three web search engines (Google, Yahoo, Microsoft) deliver sub-second responses to hundreds of millions of queries daily over hundreds of terabytes of data. At the other end of the scale, desktop search systems can rely on substantial computing resources relative to a small data set. –Value. The retrieval of high-value content (typically within corporate intranets or behind pay-for-use turnstiles) is often mission-critical for the business that owns the content. For example, the BBC allocates a skilled staff member for eight hours per broadcast hour to index their most important content. –Cost. Semantic indexing, conceptual search, linked data and so on share with controlled-vocabulary and meta-data systems a higher cost of H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 1–5, 2010. c Springer-Verlag Berlin Heidelberg 2010 

2

H. Cunningham, A. Hanbury, and S. R¨ uger

implementation and maintenance than systems based on counting keywords and hyperlinks. To process web-scale volumes, GYM use a combination of one of the oldest and simplest retrieval data structures (an inverted file that relates search terms to documents) and a ranking algorithm that utilises the link structure of the web. These techniques work much better than was initially expected, profiting from the vast number of human relevance decisions that are encapsulated in hyperlinks. Problems remain of course: first, there is still much data in which links are not present, and second the familiar problems of ambiguity (index term synonymy and query term polysemy) can lead to retrieval of irrelevant information and/or failure to retrieve relevant information. High-value (or low-volume) content retrieval systems address these problems with a variety of semantics-based approaches that attempt to perform conceptual indexing and logical querying. For example, the BBC system cited above indexes using a thesaurus of 100,000 terms that generalise over anticipated search terms. Similarly in the Life Sciences, publication databases increasingly use rich terminological resources to support conceptual navigation (MeSH, the Gene Ontology, Snomed, the unified UMLS system etc). An important research theme in recent years has been to ask to what degree can we have our cake and eat it? In other words, how far can the lowvolume/high-value methods be extended? One area of new high-value methods is being driven from Natural Language Processing. In the proceedings of this IRFC 2010 conference, Roth and Klakow [12] combine Wikipedia-based concept models for cross-language retrieval. They employ Wikipedia as a low-cost resource that is up-to-date and find that standard Latent Dirichlet Allocation can extract cross-language information that is valuable for IR by simply normalising the training data. Albakour et al [1] look at sentence-level analysis of e-mails with a view to predict whether the e-mail should contain an attachment. This is an example for a problem where a finergrained sentence-level approach is preferable to the common IR approach of a bag-of-words of the entire document. Evaluation measures in IR certainly have to adapt to new models and modus operandi. One example is the Newbold et al [11] novel ranking-by-readability approach, which considers the reading ability (and motivation) of the user. Hosseini et al [6] critique the common approach of comparing retrieval systems by their mean performance. They argue that the averaging process of the typical mean average precision measure does not capture all the important aspects of effectiveness. Instead they explore how the variance of a metric can be used to provide a more informative measure of system effectiveness than mean average precision. Other contributions to potential future high-value methods come from the forefront of theory: Melucci [9] looks at van Rijsbergen’s quantum mechanics framework as a new model for IR. He first discusses a situation in which that framework, and then quantum probability, can be necessary in IR and then describes the experiments designed to this end. Costa and Melucci [2] present a new framework based on the Discrete Fourier Transform (DFT) for IR. Their

Scaling Up High-Value Retrieval to Medium-Volume Data

3

model represents a query term as a sine curve and a query as the sum of sine curves, which is then transformed from the time domain to the frequency domain through DFT. Each document of the collection corresponds to a set of filters in this interpretation, and the retrieval operation corresponds to filtering the result of the query DFT (aka as spectrum). Costa and Melucci’s initial small-scale experiments with spectral methods indicate a competitive retrieval performance. The science and technology reported in this volume also concerns pushing the boundary of scale for the low-volume methods in the general context of patent documentation, to which we now turn.

2

Patent Retrieval

The number and economic significance of patent search professionals (and others who must search patent data) has grown rapidly in recent decades. This growth shows no signs of slowing. There are several schools of thought as to whether this is a good thing, but even in the negative camp many would agree that the load involved in searching prior art is a waste of economic resources. Further, techniques like deliberate obfuscation, defensive patenting and cross-licencing all add to the burden on the most dynamic sectors of industry, and this is especially problematic for small to medium enterprises. Several other factors have increased the value of patents and the criticality of patent searching: 1. Patent applications have been rising at around 7% yearly for more than a decade (plus large increases in the abbreviated Patent Cooperation Treaty process applications) 2. Globalisation, specialisation, the fashion for startups in high-tech, and crosslicencing as a corporate defence mechanism 3. Stronger enforcement In parallel, the publication space of prior art has mushroomed in volume (to take only one of many possible examples, Medline was “growing at the rate of more than 400,000 new entries per year” at the start of the last decade1 . In sum, patents have become “the currency of the knowledge economy” [5]. How do patents differ from Web documents? They are harder in some respects, easier in others. Patent documents are harder because patent searchers require very high recall (web searchers want high precision) and because patents don’t contain hyperlinks (though they do contain references). On the other hand, patent documents are easier because of the lower volume, their more regular structure and because there there are some sub-language effects (turns of phrase and terminology that are particular to the genre and which can aid automated processing). The average value of patent documents is also much higher than the average value of web pages. All in all, patent search is a challenging task for IR researchers, and the strength and diversity of the work reported at IRFC 2010 is testament to the 1

http://www.medscape.com/viewarticle/440693 3

4

H. Cunningham, A. Hanbury, and S. R¨ uger

energy with which the scientific community is addressing this area: Graf et al [4] explore the benefits of integrating knowledge representations in prior art patent retrieval, where they utilise human judgement available in the form of classifications assigned to patent documents. Klampanos et al [7] apply logic-based retrieval to patent searching. They see an initial document search only as the starting point of a chain of searches and decisions that need to be made by patent searchers. They argue that keyword-based retrieval is not suitable for modelling comprehensive retrieval strategies while database-like and logical approaches are better suited to model strategies, reasoning and decision making. Lopez [8] describes experiments with Conditional Random Fields (CRF) for extracting bibliographical references in patent documents. M¨ uller et al [10] contrast abstracts versus full texts and patents in a quantitative analysis of biomedical entities. They research and uncover the density and variety of relevant life science terminologies in Medline abstracts, PubMedCentral journal articles and patents from the TREC Chemistry Track. In yet another patent area Urbain and Frieder [13] explore the development of probabilistic retrieval models for integrating term statistics with entity search using multiple levels of document context to improve the performance of chemical patent search. In the way of conclusion, we believe that the IRFC 2010 conference has demonstrated how cutting-edge-research continues to make inroads in the effort to scale up high-value retrieval to medium-volume repositories.

References 1. Albakour, M.D., Kruschwitz, U., Lucas, S.: Sentence-level attachment prediction. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 6–19. Springer, Heidelberg (2010) 2. Costa, A., Melucci, M.: An information retrieval model based on Discrete Fourier Transform. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 88–103. Springer, Heidelberg (2010) 3. Cunningham, H., Hanbury, A., R¨ uger, S. (eds.): IRFC 2010. LNCS, vol. 6107. Springer, Heidelberg (2010) 4. Graf, E., Frommholz, I., Lalmas, M., van Rijsbergen, K.: Knowledge modeling in prior art search. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 32–47. Springer, Heidelberg (2010) 5. Guellec, D., van Pottelsberghe de la Potterie, B.: The Economics of the European Patent System — IP Policy for Innovation and Competition. Oxford University Press, Oxford (2007) 6. Hosseini, M., Cox, I.J., Millic-Frayling, N., Vinay, V.: Measuring performance volatility of retrieval systems. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 72–87. Springer, Heidelberg (2010) 7. Klampanos, I.A., Wu, H., Roelleke, T., Azzam, H.: Logic-based retrieval: Technology for content-oriented and analytical querying of patent data. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 104–118. Springer, Heidelberg (2010) 8. Lopez, P.: Automatic extraction and resolution of bibliographical references in patent documents. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 119–134. Springer, Heidelberg (2010)

Scaling Up High-Value Retrieval to Medium-Volume Data

5

9. Melucci, M.: An investigation of quantum interference in information retrieval. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 135–150. Springer, Heidelberg (2010) 10. M¨ uller, B., Klinger, R., Gurulingappa, H., Mevissen, H.T., Hofmann-Apitius, M., Fluck, J., Friedrich, C.: Abstracts versus full texts and patents: A quantitative analysis of biomedical entities. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 151–164. Springer, Heidelberg (2010) 11. Newbold, N., McLaughlin, H., Gillam, L.: Rank by readability: Document weighting for information retrieval. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 20–31. Springer, Heidelberg (2010) 12. Roth, B., Klakow, D.: Combining Wikipedia-based concept models for crosslanguage retrieval. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 48–61. Springer, Heidelberg (2010) 13. Urbain, J., Frieder, O.: Exploring contextual models for chemical patent search. In: Cunningham, H., Hanbury, A., R¨ uger, S. (eds.) IRFC 2010. LNCS, vol. 6107, pp. 62–71. Springer, Heidelberg (2010)

Sentence-Level Attachment Prediction M-Dyaa Albakour1,2, Udo Kruschwitz1 , and Simon Lucas1 1

School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe Park, Colchester, CO4 3SQ, UK {malbak,udo,sml}@essex.ac.uk 2 Active Web Solutions Ltd., Broomvale Business Centre, Bramford Road, Ipswich, IP8 4JU, UK [email protected]

Abstract. Attachment prediction is the task of automatically identifying email messages that should contain an attachment. This can be useful to tackle the problem of sending out emails but forgetting to include the relevant attachment (something that happens all too often). A common Information Retrieval (IR) approach in analyzing documents such as emails is to treat the entire document as a bag of words. Here we propose a finer-grained analysis to address the problem. We aim at identifying individual sentences within an email that refer to an attachment. If we detect any such sentence, we predict that the email should have an attachment. Using part of the Enron corpus for evaluation we find that our finer-grained approach outperforms previously reported documentlevel attachment prediction in similar evaluation settings. A second contribution this paper makes is to give another successful example of the ‘wisdom of the crowd’ when collecting annotations needed to train the attachment prediction algorithm. The aggregated non-expert judgements collected on Amazon’s Mechanical Turk can be used as a substitute for much more costly expert judgements.

1

Introduction

Email has certainly become one of the most effective communication tools in today’s organisations. At the same time email has also become one the most challenging digital materials to manage and studies show that current email systems do not fully satisfy users in managing their growing mail boxes, a phenomenon known as email overload [17], [7]. One of the common pitfalls when sending emails is to forget to attach a document resulting in a sequence of emails between the sender and the recipients. We are interested in solving this problem by trying to predict a necessity for an attachment so that the user is notified before sending the message. Previous work using bag-of-word approaches has shown promising results on building such a system by analysing the content of the email message and the user profile [5], [6]. Here we explore performing a finer-grained analysis by looking at the sentences that comprise the email body rather that the whole email. We aim to identify H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 6–19, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Sentence-Level Attachment Prediction

7

sentences that refer implicitly or explicitly to a document attached to the email message. This will enable us to know more about the attachment which might further be useful for suggesting documents to the user. Finding out what sentence in an email relates to an attachment can also be useful in other contexts, e.g. when extracting knowledge from emails and associating extracted facts with the appropriate attachments [1]. Any supervised learning approach requires annotated data. For our purpose we need a realistic set of emails that are annotated to reflect which sentence within each email refers to an attachment. Collecting such annotations can be a costly exercise. Therefore we suggest using Amazon’s Mechanical Turk1 to collect annotations cheaply and quickly from paid non-experts. We will of course have to show that we can get high quality annotations. In summary, this works tries to answer the following main research question: Can sentence-level prediction of attachments beat state-of-the-art methods that are based on a more coarse-grained document-level approach? Beyond this main question we will try to answer two further research questions: – Are non-experts annotations good enough in the task of identifying sentences that refer to attachments in email messages? – Can they substitute expert annotations and therefore be used as a gold standard? The paper is organised as follows, in Section 2 we look at related work. Sections 3 and 4 describe the dataset used for the experiments. Section 5 explains the task of collecting non-expert annotations through Mechanical Turk while Section 6 explains how we developed a sentence classifier and used it to predict emails with attachments. Discussion and future work are presented in Section 7.

2

Related Work

2.1

Attachment Prediction

Attachment prediction in emails is the task of identifying whether an email message contains any files attached to it. It can be useful for notifying users that an email message is missing an attachment before hitting the send button. Commercial applications are available as plugins for common Email Clients such as MS Outlook and Mozilla Thunderbird. For example Forgotten Attachment Detector2 allows users to specify certain phrases that can be looked at to alert them before sending an email that an attachment file is missing. Such systems can handle a limited number of cases. Dredze et al. [5],[6] studied the problem and 1 2

http://mturk.com/ http://www.officelabs.com/projects/forgottenattachmentdetector/ Pages/default.aspx

8

M-D. Albakour, U. Kruschwitz, and S. Lucas

introduced a significantly better system in terms of accuracy. In their work they used a supervised machine learning approach where they trained the classifier on bag-of-words features of the body of the email message and other features such as user profile (their history in sending messages). To our knowledge this is the only published work that studies the problem in sufficient detail and we are therefore using that approach as a baseline. 2.2

Mechanical Turk for NLP and IR

Supervised learning for text categorisation and classification requires annotated examples which can be an expensive and tedious task. Recently Amazon’s Mechanical Turk has proved to be a cheap and a fast platform to obtain document annotation for natural language processing and information retrieval tasks. It has been used recently by different researchers in Information Retrieval and Natural Language Processing as in [16], [3], [13]. Snow et al. show that nonexpert annotations can be used as gold standards for machine learning tasks in various complex natural language processing tasks such as word disambiguation and affect recognition [16]. Callison-Burch [3] explored using Mechanical Turk to evaluate machine translation and found that non-expert judgements are as good as those of experts. In our task we aim to collect finer-grained judgement to mark the specific sentences that refer to an attachment in an email message. We explore using Mechanical Turk for this task. 2.3

Sentence Level Approaches

Sentence-level approaches differ from coarse-grained document-level approaches by breaking the document into sentences and treating each sentence as a learning instance as opposed to considering the whole document. Bennett et al. [2] show the superiority of finer-grained sentence-level methods over coarse-grained document-level approaches in detecting action items in emails. Sentence-level classification in emails has also been successfully applied to text classification [10]. Processing at the sentence level was studied in blogs by Missen et al. [14] to explore the challenges accompanied by sentence-level opinion detection in user generated content. General challenges on the sentence level include the use of informal data which makes sentence detection harder and inaccurate and demonstrates the costly nature of sentence-level annotations.

3

The Corpus

Information Retrieval as a research area has partly been so successful because standard metrics, evaluation regimes and test collections have been developed that make it easy to compare different methodologies. We will use precision and recall (as well as F1) measures. There are only a few publically accessible email corpora that can be considered standard test collections. We will use the Enron

Sentence-Level Attachment Prediction

9

corpus so that we can compare our work with the baseline applied to the same data. The Enron corpus was made public in 2004 during a legal investigation concerning the Enron corporation. It contains 619,446 messages belonging to 158 users [11]. The corpus has triggered a lot of interest in email research. The corpus contains the attachment information but the actual attachments have been excluded from the prepared corpus [11]. Thanks to Dredze et al [5] we obtained a cleaned version of the corpus annotated with attachment information excluding forwarded messages resulting in a corpus of 157,711 messages from 150 users. In this cleaned corpus each email message has an extra header indicating if the message has an attachment or not and an extra header that contains a list of the attachment names if the previous header is positive. A detailed explanation of the cleaning process of the corpus can be found in [5]. We further analysed this cleaned corpus to study the distribution of emails with attachments and their contents in comparison to the rest of the emails in the corpus. We looked at the emails in the ‘sent’ folders of each user as they are more representative of real user-authored emails as emails in other folders might contain promotional content or spam. The ‘sent’ folders contain more than 33,000 emails, out of which around 2500 emails contain attachments. This suggests that usually only 8% of the emails that users send are used to transfer files. Within the emails which contain attachments, we were interested in how many sentences can potentially refer to an attachment. In order to do so we run the dataset through an automatic preprocessing step to clean the body of the emails from noisy text by: – Removing all ‘quoted’ text from messages in the same thread. When a message is a reply to another message in the same thread then email clients would include the body and some header information of previous emails in the thread. We used the heuristics introduced by [19] to eliminate quoted text from the email’s body. – Removing the sender’s signature from the body using simple heuristics. This cleaning step ensures that we only consider text that genuinely represents the email body (the text that the email author has written). After that we split the remaining text into sentences using the OpenNLP3 sentence splitter. Figure 1 illustrates how many sentences are left in an email after the cleaning step. More than 90% of those emails are rather short and contains less than 6 sentences. This observation allows us to estimate the effort and the time needed by annotators to identify sentences referring to an attachment within a single email message. The figure suggests that for a single email message annotators will be often asked to annotate less than six sentences. We also use this ratio when sampling the corpus so that the selected dataset reflects this distribution. 3

http://opennlp.org

10

M-D. Albakour, U. Kruschwitz, and S. Lucas

Fig. 1. Distribution of number of emails with attachments versus number of sentences

4

Data Selection

We will first only look at emails that do contain an attachment. In order to sample a representative data set (representative in terms of users and document length), we performed the following procedure: – Among the 20 users with the largest mail boxes we randomly selected 10 users who sent emails with attachments. – For each user we randomly collected up to 19 email messages so that the sample corpus reflects the distribution of sentences indicated above. As a result we obtained 178 email messages with 765 sentences. The task now is to identify the sentences in those emails which refer to any attached document.

5 5.1

Annotating the Data Amazon Mechanical Turk

Amazon’s Mechanical Turk (AMT) is a place for people to find microwork (a marketplace for crowd sourcing). The work typically pays a few cents, though some tasks take longer and might pay a few dollars. AMT requesters place tasks, which are called HITs (Human Intelligence Tasks), on the site. HITs include things like labelling a product photo with tags or rewriting a sentence in different words. Requesters receive the work done by workers and can choose to reject the work without giving a reason.

Sentence-Level Attachment Prediction

5.2

11

Task Design

The task we are interested in is annotating sentences within emails that refer implicitly or explicitly to an attachment. We used the selected dataset and performed the preprocessing steps explained previously in Section 3. We designed a HIT using the AMT web interface containing the following: – An example of the correct answer. This includes an email message with four sentences in which two are positively referring to an attached document and two negative sentences which are not related to any specific attachment. – The email subject, attachment names and the whole body of the email message to be annotated. – A list of sentences, produced by the OpenNLP sentence splitter on the cleaned body of the email, each of which to be annotated as to whether it refers to an attached document or not. For every task we asked five different unique workers to annotate the sentences as it was shown before that aggregation of multiple independent annotations from non-experts can produce good quality results [16]. We paid $0.05 for each task. We also asked two experts to do the same task for all the emails in the dataset in order to evaluate the quality of the non-expert annotations (experts in this case were researchers in the lab who performed the annotation task independently). Figure 2 shows an example of an email message given to the annotators.

Fig. 2. An example of an email message for the annotation task

12

5.3

M-D. Albakour, U. Kruschwitz, and S. Lucas

Results Processing

We automatically processed the results coming from the AMT workers to eliminate results from workers who did not follow the instructions they were asked when given the task. This is the case when the answer is not in the right format. As a result three workers’ annotations were excluded and their answers were substituted by reproducing the tasks with different workers. 5.4

Quality of Annotations

In order to test whether the non-expert annotations can be used as gold standard in a machine learning context we did some experiments to test: – Whether non-experts agree with each other. This is an indication of how well the non-experts are performing the task. – Whether the combined non-expert annotation agreement with experts is similar to the agreement between experts themselves. Inter-annotator agreement measurement. Fleiss’ Kappa statistical measure [8] was used to assess the reliability of agreement between the multiple annotators. The Kappa measure takes into consideration the probability of agreement by chance and is therefore considered more robust than simple percent agreement calculation. Kappa takes a value in the range of -1 to 1, where -1 indicates perfect disagreement below chance, 0 indicates agreement equal to chance and 1 indicates perfect agreement above chance. Table 1 shows the corresponding Kappa and percentage agreement (PA) values. Agreement among non-experts. As explained earlier we obtained five different judgements from different workers for each sentence in every email in the dataset, resulting in 765 sentences labelled 5 times with a flag indicating if the sentence refers to an attachment or not. The value of κ for these annotations was 0.61 indicating substantial agreement [12]. This is a baseline of good quality work. In the example given in Figure 2, all the non-experts agreed on sentences 1,2,3 to be positive (referring to an attachment), however they disagreed on sentences 4 where only 2 voted for positive and on sentence 5 where only one of the workers voted for positive. Agreement among experts. This measurement was conducted to understand the difficulty of this task in the given dataset. In the case of experts κ is 0.70 as measured between the two experts. As expected, the agreement between the two experts is higher than the agreement between the five non-expert judgements. However, the value is not very close to 1.0 which suggests that the task is not straightforward and in some cases it is difficult to judge if a given sentence refers to an attachment.

Sentence-Level Attachment Prediction

5.5

13

Combining Non-expert Annotations

In this work we followed the approach by Snow et al.[16] to combine the judgements produced by non-experts to produce an aggregation of non-expert annotations. This approach takes the average of the numerical judgement given by workers to produce an aggregated judgement. In our case the binary judgements (Yes/No) for each sentence were aggregated by simply taking the majority of votes, i.e. the sentence is considered positive (in other words, it refers to an attachment) when the number of votes for being positive is larger than those for being negative. We now ask the question of whether these combinations can be used as gold standard for our classification task. In order to do so we treated these aggregated annotations as annotations coming from one annotator and calculated the agreement between this annotator (non-experts combined in Table 1) and each of our experts. As shown in Table 1, the κ value for the agreement between nonexperts and each of the experts (0.73, 0.70) is comparable to the one between the experts themselves which suggest that their judgements are good enough to use in our task. Therefore substituting expert judgements with those of non-expert collected by AMT is justified for our task. Also it is important to realize that such annotations were quicker and cheaper to obtain with AMT. We paid 44 USD to the AMT workers to obtain 765*5=3825 labels in comparison to the time and effort by our two experts and the arrangements needed. Table 1. Inter-Annotator Agreement Inter-annotator agreement Comparison κ Non-expert agreement 0.61 Expert agreement 0.70 Non-experts combined vs. Expert 1 0.73 Non-experts combined vs. Expert 2 0.70

6

PA 0.81 0.85 0.87 0.85

Learning to Predict Attachments

In this section we explain the supervised learning using the gold standard obtained by non-expert annotations to identify sentences that refer to an attachment and therefore identify emails that might contain attachments. We aim to show the superiority of finer-grained sentence level approaches in the attachment prediction problem in comparison with document level approaches used previously. We compare our results for the attachment prediction problem to previous work by Dredze et al.[6] reported on the same corpus using document-level classification.

14

M-D. Albakour, U. Kruschwitz, and S. Lucas

6.1

Experimental Design

The dataset used to collect non-expert judgements contains 178 positive emails (documents) with 765 sentences, out of which 346 where identified by non-experts as positive. The task now is to develop a classifier that can identify positive sentences in an email message. The dataset contains only positive email messages. To make this dataset representative to real world mailboxes we had to add a number negative emails (emails without attachments) that will make the ratio of positive emails realistic, i.e close to 8%. We added 1000 negative emails from the ‘sent’ folders of the same ten users (100 each). Sentences in the additional 1000 negative emails were all treated as negative. We ended up with 1178 email message containing 4440 sentences out of which only 346 are positive. Therefore there is a large number of non-relevant instances and the data is highly skewed. 6.2

N-Gram Features

SVMLight4 , an implementation of Support Vector Machines, was used for classifying sentences. We tried different features to train a linear SVM classifier for the sentence classification. We used the OpenNLP tokenizer to obtain the features and we kept all punctuation marks like ‘.’, ‘!’, ‘?’ etc. We used n-gram features to train the classifier. For n-grams we considered all possible sequences of uni-grams, bi-grams, tri-grams and quad-grams. N-Grams have been shown to be effective in similar classification in emails, namely detecting speech acts [4]. Table 2 shows the performance obtained for sentence classification with different features using the Error Rate measure calculated using standard 10-fold cross validation for each classifier. Based on these results we opted for n-grams as it gives the best performance. Table 2. Performance of the SVM classifier using different features Stems Bi-grams N-grams Error Rate 18.3% 21.3% 16.9%

6.3

Learning with Skewed Data

With the skewed nature of the data, training a sentence classifier is tricky as the non-relevant instances are dominant which means that we need to undersample negative examples [15]. To show the difficulty of learning with skewed data, we experimented with learning on balanced and imbalanced datasets. Table 3 shows the 10-fold cross validation of the sentence classifier on different datasets,

4

http://svmlight.joachims.org/

Sentence-Level Attachment Prediction

15

Table 3. 10-fold cross validation performance results for the SVM sentence classifier on different datasets using N -Grams as features Positive Only All documents Precision 0.835 0.573 Recall 0.777 0.523 F1 0.805 0.547

sentences from only positive emails and sentences from the whole dataset. The performance dropped when using the whole dataset. Actually, recall was very low due to the large number of false negatives as a result of training the classifier on too many negative instances. Another way to handle the situation would be to select the most important features which was argued by Forman [9] to be more important than the actual classification algorithm in highly skewed situations. However the imbalanced nature of the data will affect the feature selection procedure [20]. Therefore we have balanced the data to select features and then used only those features to train the classifier on the whole imbalanced dataset. 6.4

Feature Selection

Which features are the most discriminating ones for our task on a balanced dataset? The motivation behind balancing the data is that feature selection measures can fail to produce good classification accuracy [20]. As argued by Zheng et al. [20], when using one-sided metrics which only take the positive features into account, the negative documents will be very likely to be misclassified and that will dramatically affect the classifier performance due to their dominance. On the other hand, when applying two-sided metrics which take both features into account, it can be difficult for negative features to obtain high scores due to the large number and diversity of negative documents which is the case for negative email sentences. Our approach is to select the features from a balanced dataset of negative and positive sentences (balanced in the sense that we add a small number of emails without attachments to the sample corpus). We created this dataset by taking all the 765 sentences from the 178 positive emails annotated by non-experts through AMT. We then added 70 negative emails taken randomly equally from our ten selected users. We end up with a dataset containing 1023 sentences from positive and negative emails out of which 346 sentences are positive. The resulting sentences contain 41,187 n-gram features. We used Chi Square to rank those features. Chi measures the lack of independence between a feature f and category c [18]. χ2 (f, c) =

¯ c)−P (f,¯ N[P (f,c)P (f,¯ c)P (f¯,c)]2 P (f )P (f¯)P (c)P (¯ c)

16

M-D. Albakour, U. Kruschwitz, and S. Lucas

where N is the total number of documents P (f, c) is the probability of presence of f and membership in c P (f, c¯) is the probability of presence of f and non-membership in c P (f¯, c) is the probability of absence of f and membership in c P (f¯, c¯) is the probability of absence of f and non-membership in c. This results in reducing the number of features to 1045. To give the reader a flavour of the selected features, here is a list of the top bigrams selected: ’here is’, ’is the’, ’please review’, ’, please’, ’the file’, ’attached is’, ’is a’, ’the attached’, ’this is’, ’this spreadsheet’. We then used only these features to evaluate the linear SVM classifier on the whole skewed data. The 10-fold cross validation measuring precision and recall of the classifier gave us 0.809 and 0.549 respectively. Note that these measures reflect how well we can predict sentences that refer to attachments. We will now look at how this can be used to detect emails that expect attachments. 6.5

Experiments on Attachment Prediction on the Document Level

On the document level, an email is considered positive if the classifier identified at least one positive sentence and it is considered negative (i.e. not referring to an attachment) otherwise. We measured the performance on the document level by calculating the confusion matrix of each fold and aggregating the results. We performed a standard 5-fold cross validation on the document level. All sentences in a document are either entirely in the training set or entirely in the testing set for each fold. Each fold contains emails with an equal proportion of positive and negative emails from each user mail box. The first column (‘All’) in table 4 shows the performance on the document level for this setting. These figures shows the improvement of our approach over a bag-of-word approach [6]. Dredze et al. report 67% recall and 83% Precision on a single user setting where the classifier was evaluated on a dataset from the same user. Also in that work additional features such as the user profile were taken into account. In our approach, we obtained a higher recall on a more difficult setting without taking the user profile into account. Furthermore, to compare our method to the cross-user evaluation setting in [6], we performed a 5-fold cross validation on a cross-user setting where we trained the classifier on emails from certain users and tested on emails from other users. The dataset was split into 5 folds on which each contains emails from 2 users out of our ten users.The results are reported in Table 4. This shows the system is also performing much better than previously reported (56% recall, 80% precision, 66% F1) on such settings. F1 is 16% better than previously reported. We also performed a much harder evaluation setting where we took another 600 email messages, not included previously in the initial 1178 emails dataset, from the ‘sent’ folders of our ten users, 100 of which are positive and 500 are negative (equally distributed over all users). We used this dataset as our testing

Sentence-Level Attachment Prediction

17

Table 4. Document-level attachment prediction using sentence level classifiers All Cross-User Precision 0.825 0.835 Recall 0.701 0.704 F1 0.758 0.764

New dataset 0.900 0.630 0.741

set and sentences from our initial dataset as the training set. The column labelled ‘new dataset’ in Table 4 shows the results reported in this setting which suggests that the classifier can scale to a larger dataset.

7

Discussion and Future Work

The main focus of this work was to investigate the usefulness of using sentence-level attachment prediction. We showed that detecting attachment-related sentences in emails can beat state-of-the-art methods in document-level attachment prediction which can be used to inform users that an email is missing an attachment. Although our results consider a difficult cross-user evaluation setting, the emails are specific to the Enron corpus. In a real world setting the system should work for a new user in any environment. Some features identified by the feature selection process might be useful in the Enron corpus but not in other environments. The work also presented yet another validation of the usefulness of Amazon’s Mechanical Turk for non-expert annotation tasks5 . We have shown that the aggregated knowledge collected like that can be used to substitute expert user annotation. This finding has led us to recently collect annotations for the entire cleaned Enron corpus. However due to the complexity arising from larger-scale computation required for the feature selection step we were not able to repeat the experiments required to report the results on the full dataset and we consider this as a future work. Another line of our future work is to link attachments to facts extracted from individual sentences. This is part of a larger project aimed as semantic search over email archives [1].

Acknowledgements This work is funded by a partnership between Active Web Solutions (AWS)6 and the University of Essex and funded by a grant from the Technology Strategy Board under the Knowledge Transfer Partnership scheme, grant number KTP006743, as well as the R&D division of AWS.

5

6

The annotations collected in this study can be obtained by contacting the authors. http://aws.net

18

M-D. Albakour, U. Kruschwitz, and S. Lucas

References 1. Albakour, M.-D., Kruschwitz, U., Blackwell, R., Lucas, S.: Managing collaboration projects with semantic email search. In: Proceedings of the WWW 2009 workshop on Semantic Search (April 2009) 2. Bennett, P.N., Carbonell, J.: Detecting action-items in e-mail. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 585–586. ACM, New York (2005) 3. Callison-Burch, C.: Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 286–295. Association for Computational Linguistics (2009) 4. Carvalho, V.R., Cohen, W.W.: Improving “email speech acts” analysis via n-gram selection. In: ACTS 2009: Proceedings of the HLT-NAACL 2006 Workshop on Analyzing Conversations in Text and Speech, pp. 35–41. Association for Computational Linguistics, Morristown (2006) 5. Dredze, M., Blitzer, J., Pereira, F.: sorry, i forgot the attachment: Email attachment prediction. In: CEAS (2006) 6. Dredze, M., Brooks, T., Carroll, J., Magarick, J., Blitzer, J., Pereira, F.: Intelligent email: reply and attachment prediction. In: IUI 2008: Proceedings of the 13th international conference on Intelligent user interfaces, pp. 321–324. ACM, New York (2008) 7. Fisher, D., Brush, A.J., Gleave, E., Smith, M.A.: Revisiting Whittaker & Sidner’s “email overload” ten years later. In: CSCW 2006: Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, pp. 309–312. ACM, New York (2006) 8. Fleiss, J., Levin, B., Paik, M.C.: Statistical Methods for Rates and Proportions, 3rd edn. John Wiley and Sons Inc., Chichester (2003) 9. Forman, G.: An extensive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res. 3, 1289–1305 (2003) 10. Khoo, A., Marom, Y., Albrecht, D.: Experiments with sentence classification. In: Proceedings of the 2006 Australasian Language Technology Workshop (ALTW 2006), pp. 18–25 (2006) 11. Klimt, B., Yang, Y.: The Enron Corpus: A New Dataset for Email Classification Research. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 217–226. Springer, Heidelberg (2004) 12. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977) 13. Little, G., Chilton, L.B., Goldman, M., Miller, R.C.: TurKit: tools for iterative tasks on Mechanical Turk. In: HCOMP 2009: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 29–30. ACM, New York (2009) 14. Missen, M.M.S., Boughanem, M., Cabanac, G.: Challenges for sentence level opinion detection in blogs. In: ACIS-ICIS, pp. 347–351 (2009) 15. Singhal, A., Mitra, M., Buckley, C.: Learning routing queries in a query zone. SIGIR Forum 31(SI), 25–32 (1997) 16. Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 254–263. Association for Computational Linguistics (2008)

Sentence-Level Attachment Prediction

19

17. Whittaker, S., Sidner, C.: Email overload: exploring personal information management of email. In: CHI 1996: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 276–283. ACM, New York (1996) 18. Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: Fisher, D.H. (ed.) Proceedings of ICML 1997, 14th International Conference on Machine Learning, Nashville, US, pp. 412–420. Morgan Kaufmann Publishers, San Francisco (1997) 19. Yeh, J.-Y.: Email thread reassembly using similarity matching. In: CEAS (2006) 20. Zheng, Z., Wu, X., Srihari, R.: Feature selection for text categorization on imbalanced data. SIGKDD Explor. Newsl. 6(1), 80–89 (2004)

Rank by Readability: Document Weighting for Information Retrieval Neil Newbold, Harry McLaughlin, and Lee Gillam University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom [email protected], [email protected], [email protected]

Abstract. In this paper, we present a new approach to ranking that considers the reading ability (and motivation) of the user. Web pages can be, increasingly, badly written with unfamiliar words, poor use of syntax, ambiguous phrases and so on. Readability research suggests that experts and motivated readers may overcome confusingly written text, but nevertheless find it an irritation. We investigate using readability to re-rank web pages. We take an extended view of readability that considers the reading level of retrieved web pages using techniques that consider both textual and cognitive factors. Readability of a selection of query results is examined, and a re-ranking on readability is compared to the original ranking. Results to date suggest that considering a view of readability for each reader may increase the probability of relevance to a particular user. Keywords: Web IR, Content-based filtering, NLP for IR, Evaluation methods and metrics.

1 Introduction The goal for information retrieval (IR) is retrieving documents relevant to a user’s information needs. The well-known Probability Ranking Principle (PRP) [1] states that descending probability of relevance to a query is the most effective way to present retrieved documents to the user. This generally appears to have been interpreted as generic relevance to a query on the basis of information in and across texts (Robertson [1]) rather than relevance to the user. We consider that the PRP should account for, amongst other things, reading ability of the user, which would provide for a ranking relating to the likelihood that a user would be able to understand content on any given page. Currently, systems will largely produce the same results for experts in scientific fields as for novices, proficient readers, children, people with learning difficulties, and so on. There is likely to be considerable potential benefit in assessing appropriateness for each user. If mechanisms can be produced to learn about the reading abilities of the user, systems can filter and/or re-rank results according to each reader ability. Readability research since Kitson [2] has explored matching text to a particular (type of) reader and a number of readability measures claim to assess text for an appropriate reading level. By themselves, these formulae may be used to H. Cunningham, A. Hanbury, and S. Rüger (Eds.): IRFC 2010, LNCS 6107, pp. 20–30, 2010. © Springer-Verlag Berlin Heidelberg 2010

Rank by Readability: Document Weighting for Information Retrieval

21

match results to readers, though many would be concerned about the limited focus of these measures on sentence and word lengths or syllable counts. Recent considerations of readability by Oakland and Lane [3] account for both text factors and cognitive factors that affect a reader’s ability to understand and enjoy written content. This work provides the inspiration for our present research and has led us towards exploring how measures of text and cognitive factors might be usefully incorporated into filtering. In this paper, we explore the relationship between text and reader by evaluating the rankings according to readability in relation to a number of queries. We consider, further, how to associate text with a specific reader. In Section 2, we discuss readability research and relevance to information retrieval; Section 3 describes our present research in relation to readability; Section 4 outlines results of readability-ranking; Section 5 concludes the paper and makes considerations for future work.

2 Background To associate query results with an individual reader, we need to consider the relationship between text and reader. Historically, readability research has focused primarily on producing a numeric evaluation of style of writing based on Kitson’s [2] work, using sentence length and word length measured in syllables. These readability measures indicate some proportion of the population who could comfortably read a text, and comparisons have been made amongst measures to correlate with specific human performances over largely disjoint sets of texts. Further discussion of these formulae can be found elsewhere [4, 5]. More recent considerations of readability account for reader factors, which consider certain abilities of the reader, and text factors, which consider the formulation of the text (Oakland and Lane [3]). Reader factors include the person’s ability to read fluently, level of prior subject knowledge, lexical knowledge or familiarity with the language, and motivation and engagement. Text factors account to some extent for current readability metrics, but also cover considerations of syntax, lexical selection, idea density, and cognitive load. Oakland and Lane’s view of readability suggest that it may be possible to generically measure the difficulty of text as an artifact, but that “text difficulty” necessitates consideration of each reader. Our work elaborates that of Oakland and Lane in identifying difficulties in the apparently neat separation of the factors. In this section, we propose a new framework for readability that builds on Oakland and Lane by making consideration of the relationship between text, reader and author. We explore, subsequently, how IR systems might use such a framework in associating text to reader. 2.1 Matching Text to Readers In writing a document, or undertaking search engine optimization, an author has to be mindful of the needs of his intended audience, particularly if they are to continue reading. There must be some correlation across three principal aspects of a text: the nature and extent of its subject matter, its use of language, and its logical or narrative structure. The audience can be defined by their degree of interest in the subject, how much they already know about it, their reading ability, and their general intelligence.

22

N. Newbold, H. McLaughlin, and L. Gillam

Two kinds of measure are suited to appraising text structure: logical coherence and propositional density. By logical coherence, we mean the extent to which one statement is ordered according to a chain of reasoning, a sequence or chain of events, a hierarchy or a classificatory system. By propositional density we mean the closeness, measured by intervening words, between one crucial idea and the next. The greater the density of ideas, the larger the cognitive load on the reader. For our current considerations, we are interested in similarities that might occur in re-ranking results by reading level, to assess the effects of such re-rankings, and to make comparisons over extant and proposed readability formulae. Our new framework for readability, describing the factors to be considered is presented in Fig. 1. In the remainder of this section, we elaborate these factors.

Fig. 1. Matches needed for easy reading

2.2 Language When matching text to reader, the author needs to consider the level of language and style of writing. We propose that systems examine the vocabulary familiarity and syntactic complexity. We describe each of these below: Vocabulary Familiarity. Readability metrics generally determine the difficulty of a word by counting characters or syllables. However, Oakland and Lane cite factors such as simplicity or familiarity as more effective means. They suggest that word difficulty can be determined by examining whether the word is challenging, unusual or technical, and describe vocabulary as a text factor contributing to text difficulty.

Rank by Readability: Document Weighting for Information Retrieval

23

The process by which readers develop word familiarity is part of language acquisition which concerns the reader’s development of their language capability. Frequency plays an important role in building knowledge of a language so that it is sufficient to understand its written content. Diessel [6] showed that linguistic expressions stored in a person’s memory are reinforced by frequency so that the language user expects a particular word or word category to appear with a linguistic expression. These linguistic expectations help comprehension. Frequency was also found to be fundamental in reading fluency as words are only analyzed when they cannot be read from memory as sight words. A limited knowledge of words affects reading fluency as readers are likely to dwell over unfamiliar words or grammatical constructions. This impedes the reader’s ability to construct an ongoing interpretation of the text. The reading fluency of the reader is dependent on their familiarity with language. When readers find text populated with unfamiliar words it becomes harder for them to read. This is especially prevalent in scientific or technical documents. The terminological nature of specialized documents means that terms will appear with disproportionate frequently throughout the documents in contrast to what one would expect to encounter in everyday language. Anyone unfamiliar with the terminology would find the document hard to understand, though terms can be identified by exploiting this relationship. We can use word familiarity as an indicator of word difficulty by contrasting frequency within documents with familiarity in general language. Syntactic Complexity. Vocabulary does not tend to exist in isolation. The vocabulary may be well-defined, yet included in overly verbose sentences. Existing readability formulae consider that long and complex sentences can confuse the reader. However, long sentences do not cause a problem because of memory limits but because of comprehension skills. Reciting text verbatim, or summarizing, requires only a superficial understanding. People can reproduce text word by word without understanding it. Pearlmutter and Macdonald [7] demonstrated how unskilled readers can have the necessary knowledge to understand text but fail to use it when decoding sentences: they cannot associate their ongoing interpretation of a sentence to their existing knowledge. This was due to poor comprehension skills in the reader which Daneman and Carpenter [8] described as reading span. A series of sentences devised by Miyake et. al [9] differentiated between high- and low-span readers. These sentences had ambiguous words which could not be resolved until several words later, an example of which is shown below: “Since Ken liked the boxer, he took a bus to the nearest pet store to buy the animal.” High-span readers had no problem disambiguating ‘boxer’ but low-span readers were confused unless the sentence ordering was changed. Here, the low-span readers performed identically to high-span readers showing that they understood the different senses of the ambiguous word. The syntactic structure of a sentence affects comprehension for users with lower reading skills. When matching text to reader, syntactic complexity should be examined to considering whether it is appropriate for the reading level.

24

N. Newbold, H. McLaughlin, and L. Gillam

2.3 Subject To learn from text, a reader needs to associate the new information to their existing knowledge. This task can be helped by the reader’s interest level. Kintsch et al. [10] showed that we find stories easier to remember than technical texts because they are about human goals and actions, something to which we can all generally relate. Scientific and technical texts require specific knowledge that is often uncommon, making the texts impenetrable to those outside the domain. This suggests that readability is not merely an artifact of text with different readers having contrasting views of difficulty on the same piece of text. Familiarity with certain words depends on experience: a difficult word for a novice is not always the same as a difficult word for an expert. Reader characteristics such as motivation and knowledge may amplify or negate problems with difficult text. We propose IR systems measure the assumed knowledge of the reader to determine an individual readability score which we describe in detail below: Assumed Knowledge. Many readability metrics do not make distinctions based on the background knowledge of the reader. As discussed in relation to vocabulary, word familiarity can give a better indication of word difficulty than word length. A longer word may only be difficult for a particular reader if unfamiliar, and certain shorter words may even be more difficult to understand. Consider a general reader confronted in text discussing a ‘muon’: This short term would be rated as simple by current readability formulae. However, a majority of people would be unfamiliar with this term, and only physicists are likely to know more about the term, its definition, and related items. One way to measure background knowledge would be to consider the extent of use of known terms in the text with direct consideration of previous documents within the reader’s experience. Entin and Klare [11] showed that more readable text is beneficial for those with less knowledge and interest. In their study, students were presented with written material below their reading level. When the reader’s interest was high, text below their grade level did not improve comprehension. However, when the reader’s interest was low their comprehension was improved by simpler text. This suggests that more readable text improves comprehension for those less interested in the subject matter. We consider the need to capture and analyze the user’s experience with prior documents as a proxy for reader knowledge and motivation. Given a reading history for a user, we might next build their personalized vocabulary with frequency information, and therefore measure familiarity with words on an individual basis. In the same way that an expert is familiar with the terminology of their subject, we can reflect the background knowledge required by a reader to interpret the text correctly. When matching text to reader, word difficulty should be measured, if the information is available, on an individual basis. 2.4 Structure Well-written text requires a structure that readers can readily use to find the information they need and to understand it correctly. Text can become confusing when ambiguities arise from information inappropriately presented. Most sentences can become multiply ambiguous if taken out of context. When we read text, we build a collection

Rank by Readability: Document Weighting for Information Retrieval

25

of concepts described within the text. These concepts are described by words and phrases which we encode using pragmatic, semantic and syntactic features. When frequently combined linguistic expressions develop into a processing unit, many of the linguistic elements are ignored and the whole chunk is compressed and treated as one semantic unit. These units often develop into terms with multiword units representing singular concepts. This relates back to the assumed knowledge of the reader. However, for readers unfamiliar with the terms, we have identified two methods called ‘Propositional Density’ and ‘Lexical Incoherence’ for processing semantic units. Propositional Density. When a significant amount of information is conveyed in a relatively small amount of text, the reader can become confused. Although long collocations form semantic units that reduce conceptual complexity, problems occur when numerous semantic units are described within a short space of each other causing the reader to make numerous inferences. The number of ideas expressed in the text contributes to the work required of the reader to interpret the text correctly. Propositional density may be measurable by examining the quantity of objects within short distances of each other. These objects can be labeled with single nouns or multiword expressions. By measuring the number of unique semantic units, we can approximate the workload required for processing or interpreting the text correctly. Logical Incoherence. Problems with lexical coherence occur when writers present new information to the reader without making clear its relationship to previous information: the writer assumes that they have provided enough information to allow readers to follow their arguments logically. Semantic units can be referred to by a number of different labels and by identifying these different labels -, synonyms - we can find the prominent ideas in the text. Repetition of terms and their synonyms and other referents provides a structure for the reader to connect with. There is a relationship here to work on lexical cohesion (Hoey [12]). If a large number of new, seemingly unrelated ideas are being introduced, low cohesion would be expected and measurable.

3 Readability Based Web Ranking Readability depends on text factors and readers. IR systems generally appear to make limited efforts in gathering information about the abilities of the user that would assist in tailoring results. Familiarity of words and complexity of the sentences should be contrasted with the knowledge of the reader. For technical or specialized texts, an indication would be that a reader had demonstrated an interest in related prior material and may be more familiar with its terminology. We would assume that a searcher is somehow motivated and engaged, if informative texts are required, that these are presented appropriately and cohesively. We have undertaken experiments to explore the relationship between readability and rank. We retrieve the first 100 results from search queries issued using the Google plug-in for, GATE (General Architecture for Text Engineering) and apply our own implementations of a series of readability formulae to evaluate various considerations over ranking. We use Flesch Easy Reading Formula, Flesch Kincaid Formula, FOG Index, SMOG and ARI, described in [4,5] as measures of vocabulary and syntactic

26

N. Newbold, H. McLaughlin, and L. Gillam

complexity, and a familiarity measure, Eqn. 1, based on contrast with word frequency in the British National Corpus (BNC). Eqn. 1 sentence-based familiarity

1.5 × ln(n SL ∑

f SL nGL ) (1 + f GL ) n SL

(1)

f is word frequency, n is word count at sentence level (SL) or corpus level (GL). A document measure is derived from average sentence value. Information about prior experience of words may be available through analysis of texts with which the user has some familiarity; Eqn. 1. can then be used against such a collection initially, and subsequently revert to contrasts with BNC. We measure propositional density using the number of individual semantic units (concepts) in a sentence. We use POS tagging in GATE to identify singular nouns with considerations of adjacency and dependency from Lauer [13] to determine multiword expressions. Larger values indicate a more complex text. Eqn. 2 sentence-based propositional density

1 + u SL 2 + n SL − c SL

(2)

u is the number of semantic units, n is sentence length, c is the number of collocated words. A document measure is derived using average sentence value. It is possible to obtain a measure of coherence by calculating the average frequency of words in a document: the number of tokens divided by the number of types. Such a measure will be variously skewed by stopwords, use of synonyms and hypernyms/ hyponyms and other kinds of references. To avoid such skew, we measure lexical coherence by counting the total number of nouns and verbs in the document, and resolving headword references where possible using WordNet [14] for both keywords and multiword expressions. Large numbers of infrequently encountered and relatively unrelated ideas indicate poor incoherence, resulting in high scores for our lexical incoherence measure and texts likely to either be stuffed with keywords or fragments or generally poor matches for readers with lower reading age. Consideration is also being made of incorporating measures for semantic distance within our work. Eqn. 3 shows our measure for lexical coherence for a document as log e

(un DL + uv DL ) 2 + (tn DL + tv DL ) 2 + s DL n DL

(3)

where un and uv are the number of unique nouns and verbs in document respectively. Similarly, tn and tv are the total number of nouns and verbs. s refers to the number of sentences and n is the total word count at a document level (DL).

Rank by Readability: Document Weighting for Information Retrieval

27

4 Results An initial set of 10 queries was devised to return a variety of results expected to be suited to different audiences. The results returned were expected to range from general knowledge to technical with some queries covering technical subjects that needed to be understood by people with low reading skills. The first 100 results were analyzed for readability by the formulas, Flesch Kincaid, FOG Index, SMOG, ARI, and familiarity (Eqn. 1) to return the years of education required to understand each text. Table 1 shows the average readability score calculated by each formula, for the first 100 texts returned from each query. The results show that queries such as ‘legal aid’ and ‘mental health advice’ which need to be understood by people with little technical knowledge tended to score lower with the familiarity measure. More complex documents, such as those returned by ‘predict u.s. inflation rate’ and ‘muon neutrino’ were deemed the most difficult by all the measures. Table 1. The 10 search queries, with average readability scores for the first 100 results calculated by each of the readability formulas, and a correlation between the familiarity formula and the other readability formulas Search query legal aid henry viii wives mental health advice stonehenge facts stock market crash pop music news children’s problems harry potter sales predicts u.s. inflation rate muon neutrino Correlation

Familiarity 11.81 11.98 11.99 12.12 12.13 12.32 12.34 12.39 12.77 13.11

Kincaid 10.73 9.19 10.19 9.98 9.56 9.73 11.51 10.05 12.00 12.89 0.98

FOG 12.58 11.01 12.08 12.32 11.49 11.43 13.38 11.43 14.31 16.01 1.00

SMOG 11.72 10.60 11.55 11.71 11.14 10.78 12.41 10.87 13.04 14.34 0.93

ARI 10.05 8.56 10.33 9.96 9.37 9.39 11.78 9.90 11.88 12.46 0.76

Whilst our familiarity formula correlates with the other readability formulas, we decided to use the average reading age from all of the readability formulas as a reliable indicator for readability. The first 100 results are re-ranked according to average reading age with the most readable documents ranking first. We next determined the percentage of results suitable for somebody with 10 years of education or fewer, and how many of the first 10 results would be suitable for the same assumed education level: a large proportion of the returned search results would appear unsuitable for certain queries. We used Spearman's coefficient to evaluate the impact of re-ranking in increasing order of readability (low to high) ranking and found little correlation between readability ranking and search ranking (see Table 2). Least readable results occur for ‘muon neutrino’ and ‘predicts u.s. inflation rate’: both returned texts requiring subject matter expertise and containing difficult terms.

28

N. Newbold, H. McLaughlin, and L. Gillam

Table 2. The 10 search queries, with average readability for the first 100 results for these queries, the number of readable results if 10 years of education is assumed, and a correlation between readability ranked and original ranking of the readability Avg. No. of Top 100 No. of Top 10 Spearman’s for RHO Readability suitable for suitable Reading age 10 Reading age 10 henry viii wives 10.27 64 7 0.309 pop music news 10.73 55 9 0.161 stock market crash 10.74 50 7 0.015 harry potter sales 10.93 49 7 0.095 stonehenge facts 11.22 46 4 0.041 mental health advice 11.23 53 7 -0.006 legal aid 11.38 43 3 0.045 children’s problems 12.28 33 4 0.032 predicts u.s. inflation rate 12.80 17 3 0.014 muon neutrino 13.76 7 0 0.267

Rank Search query

1 2 3 4 5 6 7 8 9 10

Our familiarity formula correlates well other readability measures, but to examine this further, we scaled the results of applying these measures and examined frequencies of documents in six ranges. Results for ‘muon neutrino’ are shown in Fig. 2. and suggest that our familiarity formula considers the results from this query to be less generally readable: typical readability metrics produce low scores in the presence of short words; further work is required to evaluate whether our familiarity measure is a better indicator than these measures.

Kincaid FOG SMOG ARI Familiarity

1

2

3

4

5

6

Fig. 2. Scaled frequencies into six ranges for five readability formula scores on the documents returned for the query ‘muon neutrino’

Propositional density was returning unusually high results. Limited pre-processing was provided by the search plug-in, so we manually cleaned the first 5 results for each query and re-calculated readability (Table 3). The query for ‘mental health advice’ scores high on propositional density due to multiword expressions such as ‘borderline personality disorder’, ‘obsessive compulsive disorder’ and ‘telephone self help groups’. Documents returned by ‘harry potter sales’ also tend to score high due to sales-related descriptions - ‘Los Angeles-based Exhibitor relations’; ‘senior box-office

Rank by Readability: Document Weighting for Information Retrieval

29

analyst’ and titles such as ‘Half-blood prince’. These types of documents would be more readable to an expert in the subject matter. Therefore, high propositional density indicates further need to consider reader familiarity, which would reduce an overall readability score. Documents returned for ‘Stonehenge facts’ get the lowest score for lexical incoherence due to repeating themes of ‘rocks’, ‘construction’ and ‘old’. Further results can be seen in Table 3. Documents returned for ‘henry viii wives’ scored low on all measures except for lexical coherence, where they scored as one of the highest. The fact that these documents discuss each of Henry’s wives shows that a lot of different information is being presented resulting in the high score for incoherence. This indicates that a factor of the text is being addressed here which is ignored by the other measures, we can identify the amount of disparate information in the text. Table 3. The 10 search queries with the cleaned corpus Previous Rank

Search query

1 7 8 5 2 6 3 4 9 10

henry viii wives legal aid children’s problems stonehenge facts pop music news mental health advice stock market crash harry potter sales predicts u.s. inflation rate muon neutrino

Average Readability

9.02 10.16 10.32 10.85 11.33 11.58 11.71 11.79 14.19 14.38

Propositional Lexical Density Incoherence 0.48 5.10 0.37 4.26 0.40 4.89 0.49 3.93 0.44 4.61 0.51 4.94 0.45 5.08 0.50 4.68 0.48 5.70 0.47 4.78

5 Conclusions and Future Work The ease of understanding written information, or the readability of a piece of text is becoming more important with the sprawling digital morass of contributory social media and users complaining about the poor quality of written content on the Web. The demand for filtering out confused or verbose text appears to be significant, but does not yet appear to have been embraced. Our research aims at providing for an advanced readability analysis that can account for extant sentence length and word length approaches, and caters for further measures covering familiarity, complexity, knowledge, density and coherence. In this paper, we have demonstrated how readability metrics by themselves would impact ranking. Indeed, results can be filtered according to ability, or desire, of the user. We present a series of formulas that measure different aspects of the text to indicate their suitability for the reader. The user can select their reading age and interest/knowledge in a subject to return documents that more closely match their requests. Our results indicate that results of certain queries may largely be unsuitable for some users, with technical documents beyond their reading ability. In particular, we suggest that the Probability Ranking Principle should account for the reading ability of the

30

N. Newbold, H. McLaughlin, and L. Gillam

user to provide for a ranking relating to the likelihood that a user would be able to understand content. A range of further research is required for the extensive evaluation of these measures and others which we are devising to address text factors and cognitive factors. This future work may also make strong use of relevant related resources such as the Living Word Vocabulary and Google n-grams data in devising particular measures.

References 1. Robertson, S.E.: The probability ranking principle in IR. Readings in Information Retrieval, 281–286 (1997) 2. Kitson, H.D.: The mind of the buyer. Macmillan, New York (1921) 3. Oakland, T., Lane, H.B.: Language, Reading, and Readability Formulas: Implications for Developing and Adapting Tests. International Journal of Testing 4(3), 239–252 (2004) 4. DuBay, W.H.: The Principles of Readability. Impact Information, Costa Mesa (2004) 5. Gillam, L., Newbold, N.: Quality Assessment. Deliverable 1.3 of EU eContent project LIRICS (2007), http://lirics.loria.fr/doc_pub/T1.3Deliverable.final.2.pdf 6. Diessel, H.: Frequency Effects in Language Acquisition Language Use, and Diachronic Change. New Ideas in Psychology 25(2), 108–127 (2007) 7. Pearlmutter, N.J., MacDonald, M.C.: Individual differences and probabilistic constraints in syntactic ambiguity resolution. Journal of Memory and Language 34, 521–542 (1995) 8. Daneman, M., Carpenter, P.A.: Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior 19, 450–466 (1980) 9. Miyake, A., Just, M.A., Carpenter, P.A.: Working memory constraints on the resolution of lexical ambiguity: Maintaining multiple interactions in neural contexts. Journal of Memory and Language 33, 175–202 (1994) 10. Kintsch, W., Kozminsky, E., Streby, W.J., McKoon, G., Keenan, J.M.: Comprehension and recall as a function of content variables. Journal of Verbal Learning and Verbal Behavior 14, 196–214 (1975) 11. Entin, E.B., Klare, G.R.: Relationships of measures of interest, prior knowledge, and readability comprehension of expository passages. Advances in reading/language research 3, 9–38 (1985) 12. Hoey, M.: Patterns of Lexis in Text. OUP, Oxford (1991) 13. Lauer, M.: Designing Statistical Language Learners: Experiments on Noun Compounds. Ph.D. thesis, Macquarie University, Sydney, Australia (1995) 14. Fellbaum, C.: WordNet: An Electronic Lexical Database. Bradford Books (1998)

Knowledge Modeling in Prior Art Search Erik Graf, Ingo Frommholz, Mounia Lalmas, and Keith van Rijsbergen University of Glasgow, {graf,frommholz,mounia,keith}@dcs.gla.ac.uk http://ir.dcs.gla.ac.uk/

Abstract. This study explores the benefits of integrating knowledge representations in prior art patent retrieval. Key to the introduced approach is the utilization of human judgment available in the form of classifications assigned to patent documents. The paper first outlines in detail how a methodology for the extraction of knowledge from such an hierarchical classification system can be established. Further potential ways of integrating this knowledge with existing Information Retrieval paradigms in a scalable and flexible manner are investigated. Finally based on these integration strategies the effectiveness in terms of recall and precision is evaluated in the context of a prior art search task for European patents. As a result of this evaluation it can be established that in general the proposed knowledge expansion techniques are particularly beneficial to recall and, with respect to optimizing field retrieval settings, further result in significant precision gains.

1

Introduction

Identifying relevant prior art, i.e. trying to retrieve documents in patent and nonpatent literature that are closely related to the matter described in a patent document, is probably the most commonly executed task in the patent domain. These searches form an essential part of the process of determining the patentability of a specific invention [2]. In order for an invention to be viable for patenting, no prior record of a similar or identical product or process may exist (See Section B IV 1/1 in [2] for a more detailed description). A prior art search therefore aims at clarifying whether any such records exist in patent and non-patent literature that have been published prior to the filing of the patent application in question. Since the erroneous granting of a patent can result in later litigation costs of hundreds of million Euros, extensive effort is invested into retrieving every relevant document. In this context the search for prior art constitutes a good example of a recall-focused task. In this study we explore in what ways knowledge modeling and the integration of knowledge representations into the prior art retrieval task can be beneficial in light of these requirements. Modeling and representing knowledge has been widely researched in Cognitive Psychology [20] and Artificial Intelligence (AI) [12] as part of their quests to understand and replicate aspects of human cognition. In the context of Information Retrieval (IR) the integration of knowledge has been explored as part of the Intelligent Information Retrieval (IIR) [10] H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 31–46, 2010. c Springer-Verlag Berlin Heidelberg 2010 

32

E. Graf et al.

and Associative Information Retrieval (AIR) initiatives [27] as a means of building more effective systems. With respect to recall-oriented tasks, mitigating vocabulary mismatch [14], i.e. to allow for the detection of semantic relatedness of documents where it is not reflected through the mutual occurrence of terms, represented a central aim of these approaches. While initial results obtained in these subdomains of IR have been promising, widespread adoption has been limited by a variety of factors. Of these the prohibitively high costs of manual knowledge representation creation, and lack of success concerning the automation of the process, proved to be most severe. As a result knowledge modeling related research in IR relies on the utilization of available knowledge artifacts such as thesauri [19], ontologies, and citations [27]. In the patent domain, a structure that can be interpreted in this sense is given by the International Patent Classification (IPC) system1 . In this system every new patent application is, with respect to technological aspects of the described invention, assigned to one or more classes within a hierarchy consisting of more than 70,000 different elements. These assignments are conducted by patent examiners based on their in depth knowledge of the respective technologies. As a consequence this structure represents a highly precise hierarchical mapping of technological concepts, and provides an excellent basis for the extraction of knowledge. In light of this, the patent domain can be interpreted as an excellent new testbed for revisiting IIR and AIR related concepts. The focus of this study is placed on evaluating the potential benefit of integrating knowledge representations into the prior art patent retrieval task. More specifically, inspired by research from Cognitive Psychology with respect to hierarchical aspects of memory, we propose to model knowledge in the form of a hierarchical conceptual structure extracted from available IPC information. In our chosen representation each element of the IPC hierarchy is comprised of a set of terms reflecting its characteristic vocabulary. To this cause a method aimed at extracting representative vocabularies for the technological aspects covered by specific IPC elements is developed. Based on this representation, strategies concerning the integration of the extracted knowledge into the retrieval task, with respect to the underlying aim of enabling the identification of similarity between a query and a document even in the absence of mutually shared terms, are investigated. Finally the potential benefit of these techniques is evaluated based on the prior art search task of the CLEFIP 09 collection. The remainder of this paper is structured in the following way. Upon giving an overview of relevant previous research from AIR, IIR, and Patent Retrieval in Section 2, we explore the process of knowledge structure extraction based on the IPC in Section 3. In Section 4 an overview of our strategy concerning the integration of knowledge representations into the retrieval process is provided. Section 5 details the experimental setup chosen for the evaluation of our approach with respect to the prior art search task. In Section 6 we report and discuss the obtained results. In the final section we present our conclusions and provide an outlook of future extensions to this work. 1

http://www.wipo.int/classifications/ipc/en/

Knowledge Modeling in Prior Art Search

2

33

Related Work

In the following we will provide an overview of relevant research concerning Patent Retrieval and previous approaches of knowledge integration in IR. The majority of relevant retrieval research in the patent domain has been pioneered by the NTCIR series of evaluation workshops and further diversified and expanded by the CLEF [1] Intellectual Property (IP) Track 2009 [26]. A task related to the prior art search task is presented by the invalidity search run at NTCIR 5 [13], and 6 [18]). Invalidity searches are exercised in order to render specific claims of a patent, or the complete patent itself, invalid by identifying relevant prior art published before the filing date of the patent in question. As such, this kind of search, that can be utilized as a means of defense upon being charged with infringement, is related to prior art search. Likewise to the prior art search task of CLEF IP 09, the starting point of the task is given by a patent document, and a viable corpus consists of a collection of patent documents. The initial challenge with both tasks consists of the automatic formulation of a query w.r.t. a topic document. Commonly applied techniques [26] are based on the analysis of term frequency distributions as a means of identifying effective query term candidates. In addition to these techniques, the usage of bibliographical data associated with a patent document has been applied both for filtering and re-ranking of retrieved documents. Particularly the usage of the hierarchical structure of the IPC classes and applicant identities have been shown to be highly effective [13]. Concerning the integration of bibliographical data such as the IPC into retrieval, our work differs through its utilization of the IPC solely as a source for extracting term relation knowledge that is applied to mitigate the effect of vocabulary mismatch. As outlined in detail in Section 3 its aim lies in improving the query document matching process and it is not envisioned as a potential replacement, or exclusive of application of IPC based filtering or re-ranking methods. In a retrieval context where such information (i.e. IPC classification of the topic) is available these techniques could be applied on result listings returned by a retrieval setup as described in Section 5. As pointed out before, the majority of relevant research in IR stems from the subdomains of Associative IR (AIR) and Intelligent IR (IIR).In AIR the most commonly applied scheme of applying knowledge consists of the construction of conceptual graphs and the application of spreading activation algorithms [9] as a means of expanding user queries. In IIR, which is focusing on the exploration of the ’overlap of AI and IR’ [10], knowledge representations have been utilized in form of semantic networks [7] and hierarchies of retrieval subtopics [22]. A recent study [4] undertaken to evaluate the benefit of query (QE) and document expansion (DE) in the context of ad hoc-retrieval introduced two novel DE methods based on adding terms to documents in a process that is analogous to QE, and on regarding each term in the vocabulary as a query. The study concluded state of the art QE to be generally beneficial and corpus-based DE in its applied form not to be promising.

34

E. Graf et al.

Our work differs from these previously described approaches through its inherent focus on the patent domain, and by aiming at expanding documents with respect to their ’aboutness’ [17] as expressed through their classification with specific IPC codes, and the resulting ’grouping’ with related documents. In view of this, it can be interpreted as a form of meta-data based document expansion.

3

Constructing Knowledge Representations

This section initially recapitulates on the underlying motivation for the integration of knowledge modeling into patent retrieval. This will be followed by an overview of the knowledge representation utilized in this study. Finally key aspects of the IPC based knowledge extraction process are outlined. As noted before, our proposed integration of knowledge representations into patent retrieval is primarily aimed at increasing retrieval effectiveness in terms of recall. A common approach to reach this goal consists of conceiving strategies to mitigate problems associated with vocabulary mismatch. Furnas et al. coined this concept based on ’the fundamental observation’, ’that people use a surprisingly great variety of words to refer to the same thing’ [14]. Attempting to limit potential negative impact can be interpreted as aiming at detecting semantic relatedness of textual artifacts where it is not reflected through the mutual occurrence of terms. In its most trivial form this could be achieved through the expansion of queries or documents with corresponding synonyms. In a more complex form this can be interpreted as the attempt of mimicking the human ability to infer what documents ’are about’, and to base decisions concerning their relatedness on this ’aboutness’ [17]. This notion can best be illustrated through a basic example. Table 1 depicts a sample document collection consisting of the five documents A to E. In this example each document consists of only four terms. In the following, given the hypothetical query q ’matrix collagenase-3 arthritis’, we will illustrate potential rankings with respect to conventional best matching retrieval strategies and our envisioned knowledge expansion strategy. A result list returned by a best match retrieval function such as TF/IDF or BM25 would consist of the documents A,B,D, and E. Document A would be ranked at 1 as it matches two of the query terms. Rank 2 to 4 would fall to documents B,D, and E. As is evident through the chosen sample texts of the documents and the explicit relevancy statements in Table 1 such a ranking does not represent an optimal outcome. An optimal ranking would take the form A,B,C,D,E. The first three documents are related to ’arthritis’, and in this example deemed relevant to the query. A human expert in possession of the knowledge that ’rheumatoid’ and ’arthritis’ are both terms describing medical problems affecting joints and connective tissue, and that ’collagenase-3’ is a ’matrix metalloprotease’, might deduce such an optimal ranking based on the following argumentation. – A should be ranked first as it contains ’collagenase-3’ and ’arthritis’. – B should be ranked second as ’osteoarthritis’ is a subtype of ’arthritis’ and ’collagenase-3’ is a ’matrix metalloprotease’.

Knowledge Modeling in Prior Art Search

35

Table 1. Sample document collection. Matching query terms with respect to query ‘matrix collagenase-3 arthritis’ are highlighted in bold. Doc. A B C D E

arthritis antibody immunohistochemical chondrosarcoma machine

cartilage matrix nonfibrillated matrix matrix

collagenase-3 metalloproteinase rheumatoid metalloproteinase turing

specimen osteoarthritis metalloproteinase vitro zuse

Rel. Yes Yes Yes No No

– C should be ranked third since the term ’rheumatoid’ is closely related to ’arthritis’ and ’collagenase-3’ is a ’matrix metalloprotease’. – D should be ranked fourth since ’collagenase-3’ is a ’matrix metalloprotease’ and D therefore relates to one aspect of the query. While it is, as evidenced through the term ’chondrosarcoma’ relating to another medical condition, it should still be ranked higher than E that is referring to a completely different topic. Research from cognitive psychology indicates that such reasoning with respect to text is enabled through knowledge based term associating during the process of reading [28]. In order to enable an IR system to, in analogy to this, retrieve document C although it does not contain any of the query terms, two things would be necessary: Firstly the extraction of term relation knowledge (e.g. in this case mapping the relation between ’rheumatoid’ and ’arthritis’), and secondly the integration of such a knowledge representation with the documents in the collection (i.e. to allow for consideration of this relationship during the retrieval phase). With regard to the first point, concerning the question of choosing a suitable knowledge representation to benefit patent retrieval, research from cognitive psychology can provide additional guidance. Specifically the described hierarchical aspects [21] of human memory with respect to natural kinds (e.g. collagenase-3 is a matrix metalloprotease, a matrix metalloprotease is a protease, and proteases are enzymes) and artifacts (i.e. artificial concepts such as hard disk drive and tape being magnetic storage devices) seem relevant in consideration of the technical nature of the patent domain. This is further underlined by the reported role of categorical and hierarchical aspects of memory for inductive inference [8] and higher level extraction of meaning [23]. In view of this the choice of a hierarchical structure for storing knowledge seems sensible in regard to inference of relatedness. Such a structure is also well suited considering automatic knowledge extraction based on the available IPC information, as the IPC itself exhibits a hierarchical order. Following this notion, extracted knowledge from the IPC will be modeled in a hierarchical manner that represents specific elements via sets of descriptive terms. By choosing to represent elements in a bag-of-words fashion, as outlined in detail in Section 4, this representation allows to expand documents in a flexible way that enables direct integration with existing retrieval models in a way that is scalable to realistic collection sizes. Subsequently a methodology

36

E. Graf et al.

for the extraction of such an above outlined representation of technological term relatedness based on IPC classification code assignments to patent documents will be introduced. 3.1

IPC Based Knowledge Representation Extraction

This section is focused on providing an overview of the proposed knowledge representation extraction process. As part of this we will also describe in detail the process of generating representative term sets for specific IPC elements. The IPC is a hierarchical classification system comprised of roughly seventy thousand elements. These elements are spread over five main layers that are exemplary depicted in Figure 2 together with a sample IPC classification code. Each patent document is assigned to one or more elements within this hierarchy with respect to its described technical invention. To extract this knowledge, and allow for its representation in form of a hierarchical structure, a methodology comprised of 4 distinct steps is applied. Based on utilization of IPC codes found on patent documents, the methodology aims at representing elements of the IPC hierarchy by the most descriptive terms w.r.t the technological aspects covered by the documents filed to a common classification element. As will be subsequently described in detail the extraction process is based on the statistical analysis of two observed events: A pair of documents belonging to the same IPC element, and a pair of documents sharing a specific term. An overview of this process is depicted in Figure 1, and its four steps are listed below. 1. Document Pair Formation: For a given element E of the IPC hierarchy a representative set of N document pairs is formed out of the set of all documents assigned to this element. 2. Mutual Term Extraction: On completion of this process for each chosen pair of element E the set of all mutual terms is extracted. Requiring a term to occur in at least two documents belonging to E in order to be considered, represents the first selection within the task of extracting a set of terms representative for all documents of E. 3. LL Ratio Computation: To identify the mutual terms that are most representative of E we then apply the Log Likelihood (LL) ratio test on the basis of the extracted mutual terms of the N document pairs. 4. Term Selection: Finally a representative set of terms for the element E is selected by including all terms exhibiting a LL ratio score higher than a chosen threshold t. The above described procedure is then repeated until a representative set of terms has been generated for each element of the IPC. The LL ratio test performed in step (3) of our methodology is described in detail in the following subsection. 3.2

Extracting Representative Term Sets

The basic idea of our extraction approach is based on the identification of significant diversion in the statistical distribution of term occurrence frequencies

Knowledge Modeling in Prior Art Search

37

Fig. 1. Overview of the knowledge extraction process and a depiction of the representative vocabulary extracted for A62B17/04. The listed IPC description represents the complete descriptive text of the specific element.

Fig. 2. Exemplary overview of the IPC system and mapping of IPC code B81B7/04 to hierarchical level denominations

38

E. Graf et al.

within documents belonging to the same IPC element and documents in the rest of the corpus. In the following we will describe the applied process of estimating how much more likely the occurrence of a mutually shared term within a document pair belonging to the same element is in contrast to its occurrence in a pair in the rest of the collection. To this cause we devise a Log Likelihood Ratio test. Such tests have been widely applied to the task of collocation analysis due to good performance with respect to sparse distributions [6]. One advantage using likelihood ratios lies in their clear intuitive interpretation. For example, in our model a likelihood ratio of 900 expresses that the occurrence of a term within a pair of documents belonging to the same IPC element is 900 times more likely than its base rate of occurrence in document pairs in the rest of the collection would suggest. In the following we outline in detail how the Log Likelihood ratio test is applied w.r.t. our aim of extracting representative term sets. In the space W = {d1 d2 , ..., dj dk } we observe two possible events t and e: – Event t: A pair of documents dj dk mutually contains ti . – Event e: A pair of documents dj dk belongs to the same element. Based on this two hypothesis can be formulated: – Hypothesis 1 (Independence): H1: P (t|e) = p = P (t|¬e) – Hypothesis 2 (Dependence): H2: P (t|e) = p1 = p2 = P (t|¬e) With the log-likelihood ratio defined as: log λ =

L(H1) L(H2)

it can be computed as the fraction of two binomials b:

b(cte ,ce ,p)∗b(ct −cte ,n−ce ,p) log λ = log b(c . te ,ce .p1 )∗b(ct −cte ,n−ce ,p2 )

Where c designates the observed counts of the events t, e, and te obtained in the third step (3) of our extraction methodology, and n represents all possible document pairs that can be formed from all N documents contained in a collection: N! . n = (N −2)!∗(2)! With respect to our goal of identifying representative terms, if the mutual occurrence of a term t in a pair of documents belonging to the same element e results in a large LL Ratio, we deem this term to be representative of the class.

4

Integration Strategy

For the integration of the generated knowledge representations described in the previous section we utilize the concept of document fields. In this approach both the text of internal structural elements of a document such as the title, abstract, or passages, as well as external meta-data can be represented in form of a distinct field. This enables the computation of a separate score for each respective field during retrieval. These scores can then be aggregated under consideration of assigned weights with respect to the benefit of distinct fields to a retrieval task.

Knowledge Modeling in Prior Art Search

39

In this form the approach has been successfully applied to a variety of different domains using document internal (e.g. title and body of an e-mail) and external fields (e.g. anchor text for web documents [29]). Basing our integration strategies on this technique allows us to build up on the large amount of research concerning combination techniques of scores [11] for individual fields and allows us to utilize existing retrieval system functionality. In a similar fashion as HTML documents have been expanded with anchor text in previous research, we propose to expand each patent document in the corpus with representative sets of terms according to its classification within the IPC hierarchy. As exemplary outlined in Figure 3 for each document a field representing the text of the document itself, and a field for each hierarchy level can be created with respect to the assigned IPC code. As a consequence, indexing of the collection results in the creation of an index for each level of the IPC. This approach requires additional computational effort during the indexing phase, but enables the application of field-adapted retrieval models such as BM25F. BM25F [25] is a variant of the BM25 Okapi retrieval algorithm, that allows for the combination of scores from several fields in a way, that does not break the non-linear saturation of term frequency in the BM25 function. This form of aggregation has been shown to deliver strong results in field based experimentation in the news, web, and e-mail domain.

Fig. 3. Integration through mapping extracted knowledge structure hierarchy levels via field indices

In the following section we will now describe how this strategy has been implemented in our experimental setup. Further an overview of the test collection and its main task will be provided.

5

Experimental Setup

This section introduces the experimental setup that was applied to evaluate the integration of our knowledge representation. We fill first provide details concerning the corpus and the associated task of the CLEF IP 09 [26] test collection that was used in our experimentation. Following this we will outline the applied indexing process and provide details of the retrieval models that were applied.

40

5.1

E. Graf et al.

Test Collection

For the evaluation of our approach we used the CLEF-IP 09 collection that formed part of the CLEF evaluation workshop. The collection focuses on a patent retrieval task, and features thousands of topics that were created based on a methodology of inferring relevance assessments from the references found on patent documents [15]. The corpus of the collection consists of 1.9 million patent documents published by the European Patent Office (EPO). This corresponds to approximately 1 million individual patents filed between 1985 and 2000. As a consequence of the statutes of the EPO, the documents of the collection are written in English, French and German. The main task of the CLEF-IP 09 track test collection consists of the search for Prior Art. Performed both, by applicants and the examiners at patent offices, it is one of the most common search types in the patent domain. Three sets of topics labeled as S, M, and XL consisting of 500, 1000, and 10000 topics are provided for this task. Each topic consists of an information need in the form of a patent application document and a set of qrels specifying relevant documents for the application. Based on the text of the patent application, participants of the track were required to infer a query in order to retrieve a ranked list of relevant prior art documents. The inference of effective queries formed the main challenge of the task. In this study we will utilize a methodology applied by participants of the 2009 track that is based on identifying effective query terms based on document frequency [16]. 5.2

Retrieval Setup

Indexing of the collection is performed using the MG4J retrieval system [5]. For our document expansion strategy each document is indexed in the form of a set of field indices. While field index ’0’ represents the text of the document itself, knowledge in the form of representative terms is associated with one field index per hierarchy level. No form of stemming was applied. This decision was based on the fact that the corpus contains a large amount of technical terms (e.g. chemical formulas) and tri-lingual documents. In order to increase indexing efficiency, stop-wording based on a minimum stop-word list was applied. Based on this setup we apply the following retrieval models. As baseline for our experiments the BM25 model is applied to a full text index of the collection. BM25 has shown strong performance in the CLEFIP 09 prior art track [24]. For the knowledge expanded field indices the BM25F model [25] is applied. Essentially identical to BM25 it has shown very good performance in field based scenarios due to the ability to set a field specific normalization parameter b in addition to applying weighting of each index [29]. For the initial retrieval runs and optimization the large training topic set consisting of 500 query documents has been applied. The performance with optimized parameter sets was evaluated based on the medium sized (1000) topic set. As stated before the main challenge of prior art search initially consists of the automatic extraction of effective query terms based on a topic posed in form

Knowledge Modeling in Prior Art Search

41

of a several pages long patent application. State of the art automatic query formulation methods rely on choosing query terms with respect to the distribution of term features. One such feature that has been applied to the task of automatic query formulation consists of the global document frequency (df) of terms. It has been found, that effective queries can be formulated by including only those terms of a patent application that occur in a low percentage of the documents in the collection. A query selection parameter called percentage threshold is defined as df N ∗ 100, where N denotes the total number of documents in the collection. A percentage threshold of 0.5% therefore denotes, that all terms from a topic document are included in a query that appear in less than 0.5% of the documents in the collection. In light of this the experimental evaluation of the knowledge integration is divided into two parts: – Query Dependency Analysis: Induced by the lack of one definitive set of queries as for example encountered in Web domain based tracks such as HARD [3], a first step in estimating the benefit of knowledge integration consists of an evaluation of the performance of various knowledge representations with respect to a varied set of generated queries. This step is aimed at generally clarifying if, and in what ways, knowledge based expansion can impact the prior art retrieval task. – Parameter Set Optimization: In order to more precisely estimate the potential benefit of the integration, we propose to conduct parameter set optimization for query generation settings that have shown promising results in the first experimentation phase. The need to optimize the parameter sets of BM25F based retrieval attempts in order to allow for best performance is discussed and outlined in detail in [29]. The results of these investigations are outlined in the subsequent section.

6

Experimental Results and Discussion

In the following the results for both experimentation phases will be outlined. 6.1

Query Dependency Analysis

The results of the query dependent analysis with respect to three generated knowledge representations are depicted in Figure 4 and 5. The graphs outline performance in terms of Recall and MAP w.r.t. the association of knowledge representations extracted based on LL Ratio thresholds of 15, 300, and 900. A baseline is provided by the BM25 model operating on a full text index. In these initial experiments only the lowest level of the IPC (subgroup) is considered for the expansion. Evident from Figure 4 is that the expansion with subgroup level fields benefits recall over all query generation parameters. The observation of a positive effect on recall being retained among the full spectrum of query formulations

42

E. Graf et al.

is a very promising result. Since queries created with a df-threshold of 0.25 exhibit an average query term length of 103.838 in contrast to an average length of 259.548 terms for a 3.25 query-threshold, the applied set of queries represents not only a large spectrum with respect to length, but also in regard of the document frequency of the contained query terms. In light of this it seems reasonable to assume, that the observed positive effect and its robustness might also apply to related query generation methods such as TF/IDF, and potentially also to manually created prior art queries. Further it can be deduced from Figure 4 that the LLRT 15 based representation exhibits the highest amount of variance with regard to the observed performance. Responsible for these results may be application of a comparatively low threshold of 15 in order to select knowledge representation terms. Generally this will lead to the inclusion of more general terms within the representative term set of the modeled subgroup level elements. It seems likely that the inclusion of more general knowledge terms raises the probability of topic drift occurrence. The stricter term selection criteria set by LLRT likelihoods of 300 and 900 in contrast exhibit more robust performances. The negative impact of this observation becomes also evident by studying the MAP related performance shown in Figure 5. While the contained vocabulary of the LLRT 15 based knowledge representation seems still descriptive of the general technological aspects, as expressed in the higher recall with respect to the baseline, substantial noise seems to be introduced, resulting in a clearly visible negative impact on precision. This is not exhibited by the more strict LLRT300 and LLRT 900 knowledge representations, which again show lower variance in their observed performance. Promising with respect to precision is the exhibited strong MAP performance of the LLRT 300 and LLRT 900 based runs for percentage-threshold values of 0.5 and 0.75. In order to estimate their full potential benefit, linear optimization of the BM25 and BM25F parameter sets was performed based on the large training topic set (500) of the Clef-IP 09 collection. 6.2

Parameter Optimization for BM25F

Based on the above reported initial results a complete linear optimization of BM25 and BM25F parameters was performed for a percentage-threshold value of 0.75. Training of the parameters for BM25F followed the strategy of dividing the optimization of k1, and index specific b and w parameters into several smaller optimization tasks as outlined in Zaragoza et al. [29]. Table 2 lists the results. As can be seen from the table the optimization results in a much improved performance for the knowledge expanded BM25F runs. MAP as well as recall are substantially and statistically significantly increased for both the LLRT300 and LLRT900 based knowledge representation. This constitutes an especially promising result in light of the sparse exploration with respect to optimization of the LLRT threshold space.

Knowledge Modeling in Prior Art Search

43

Fig. 4. Recall over various query length and LLRT thresholds for 1 field expansion

Fig. 5. MAP over various query length and LLRT thresholds for 1 field expansion

44

E. Graf et al.

Table 2. Optimized results BM25 baseline versus BM25F knowledge expansion. b1 and w1 constitute the parameter values for the full text index; b2 and w2 represent parameters for the subgroup based knowledge field (** strong statistical significance). Query 0.75 BM25 BM25F,LLRT300 BM25F,LLRT900

7

MAP 0.1036 0.1073 0.1091

% change / 3.57** 5.31**

Recall 0.5377 0.5887 0.5819

% change / 9.48** 8.22**

Bpref 0.5544 0.6083 0.6038

k1 1.2 1.0 0.2

b1 0.4 0.4 0.4

b2 / 0.65 0.7

w1 / 1.0 1.0

w2 / 0.3 1.0

Conclusion and Future Outlook

A first study of modeling knowledge within the task of prior art patent retrieval was presented. Initial results are very promising as it is shown that the proposed knowledge association is beneficial in terms of recall, and very robust with regard to query variation. Given prior optimization of the BM25F parameter space the knowledge association results in significant improvement of both recall and precision in comparison to an in the same manner optimized BM25 baseline. This is specifically encouraging, since in the introduced work only the lowest level of the extracted knowledge representation, corresponding to the subgroup IPC hierarchy level, has been utilized. Integration of higher hierarchy levels constitutes a logical next step and could potentially result in further improvements. Moreover a more fine-grained exploration of the term extraction parameters, and the application of varying methods for representative term selection, merit extensive additional investigation. Further an exploration of n-gram representations, proximity, and document structure exploitation within the extraction and retrieval process should be considered. Finally an evaluation of the potential benefit of the introduced knowledge representations with respect to other tasks such as classification and clustering and the feasibility of applying the introduced techniques to other domains form interesting long term aspects of this research.

Acknowledgments The authors would like to thank Matrixware Information Services2 and the Information Retrieval Facility3 (IRF)for their support of this work. Mounia Lalmas is currently funded by Microsoft Research/Royal Academy of Engineering.

References 1. The Cross-Language Evaluation Forum (CLEF) 2. Guidelines for Examination in the European Patent Office (December 2007) 2 3

http://www.matrixware.com http://www.ir-facility.org/

Knowledge Modeling in Prior Art Search

45

3. Allan, J.: HARD track overview in TREC 2004: High accuracy retrieval from documents. In: Proceedings of the thirteenth Text REtrieval Conference (TREC 2004), no. Ldc, NIST, pp. 1–11 (2004) 4. Billerbeck, B., Zobel, J.: Document expansion versus query expansion for ad-hoc retrieval. In: Proceedings of the 10th Australasian Document Computing Symposium (2005) 5. Boldi, P., Vigna, S.: MG4J at TREC 2005. In: The Fourteenth Text REtrieval Conference (TREC 2005) Proceedings, number SP, Citeseer, vol. 500, p. 266 (2005) 6. Bordag, S.: Elements of Knowledge-free and Unsupervised lexical acquisition. PhD thesis (2007) 7. Cohen, P.: Information retrieval by constrained spreading activation in semantic networks. Information Processing & Management 23(4), 255–268 (1987) 8. Coley, J.: Knowledge, expectations, and inductive reasoning within conceptual hierarchies. Cognition 90(3), 217–253 (2004) 9. Crestani, F.: Application of spreading activation techniques in information retrieval. Artificial Intelligence Review 11(6), 453–482 (1997) 10. Croft, W.B.: Approaches to intelligent information retrieval. Information Processing & Management 23(4), 249–254 (1987) 11. Croft, W.: Combining approaches to information retrieval. Advances in information retrieval 7, 1–36 (2000) 12. Davis, R., Shrobe, H., Szolovits, P.: What is a knowledge representation? AI magazine 14(1), 17 (1993) 13. Fujii, A., Iwayama, M., Kando, N.: Overview of Patent Retrieval Task at NTCIR-5. In: Proceedings of NTCIR-5 Workshop Meeting (2005) 14. Furnas, G., Landauer, T., Gomez, L., Dumais, S.: The vocabulary problem in human-system communication. Communications of the ACM 30(11), 971 (1987) 15. Graf, E., Azzopardi, L.: A methodology for building a patent test collection for prior art search. In: Proceedings of the Second International Workshop on Evaluating Information Access, EVIA (2008) 16. Graf, E., Azzopardi, L., van Rijsbergen, K.: Automatically Generating Queries for Prior Art Search 17. Hutchins, W.: The concept of aboutness in subject indexing. Aslib Proceedings 30(5), 172–181 (1978) 18. Iwayama, M., Fujii, A., Kando, N.: Overview of Classification Subtask at NTCIR-6 Patent Retrieval Task. In: Proceedings of NTCIR-6 Workshop Meeting, pp. 366– 372 (2007) 19. Jing, Y., Croft, W.: An association thesaurus for information retrieval. In: Proceedings of RIAO, vol. 94, Citeseer, pp. 146–160 (1994) 20. Kintsch, W.: The role of knowledge in discourse comprehension: a constructionintegration model. Psychological review 95(2), 163–182 (1988) 21. Markman, E., Callanan, M.: An analysis of hierarchical classification, pp. 325–366. Erlbaum, Hillsdale (1980) 22. McCune, B., Tong, R., Dean, J., Shapiro, D.: RUBRIC: a system for rule-based information retrieval. Readings in information retrieval 9, 445 (1997) 23. Medin, D.L., Rips, L.J.: Concepts and categories: Memory, meaning, and metaphysics. Cambridge Univ. Press, Cambridge (2005) 24. Piroi, F., Roda, G., Zenz, V.: CLEF-IP 2009 Evaluation Summary (2009) 25. Robertson, S., Zaragoza, H., Taylor, M.: Simple BM25 extension to multiple weighted fields. In: CIKM 2004: Proceedings of the thirteenth ACM international conference on Information and knowledge management, pp. 42–49. ACM Press, New York (2004)

46

E. Graf et al.

26. Roda, G., Tait, J., Piroi, F., Zenz, V.: CLEF-IP 2009: retrieval experiments in the Intellectual Property domain. In: CLEF working notes 2009 (2009) 27. Salton, G.: Associative document retrieval techniques using bibliographic information. Journal of the ACM (JACM) 10(4), 440–457 (1963) 28. Wharton, C., Kintsch, W.: An overview of construction-integration model. ACM SIGART Bulletin 2(4), 169–173 (1991) 29. Zaragoza, H., Craswell, N., Taylor, M., Saria, S., Robertson, S.: Microsoft Cambridge at TREC-13: Web and HARD tracks. In: Proceedings of TREC 2004, Citeseer (2004)

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval Benjamin Roth and Dietrich Klakow Spoken Language Systems, Saarland University, Germany [email protected], [email protected]

Abstract. As a low-cost ressource that is up-to-date, Wikipedia recently gains attention as a means to provide cross-language brigding for information retrieval. Contradictory to a previous study, we show that standard Latent Dirichlet Allocation (LDA) can extract cross-language information that is valuable for IR by simply normalizing the training data. Furthermore, we show that LDA and Explicit Semantic Analysis (ESA) complement each other, yielding significant improvements when combined. Such a combination can significantly contribute to retrieval based on machine translation, especially when query translations contain errors. The experiments were perfomed on the Multext JOC corpus und a CLEF dataset. Keywords: Latent dirichlet allocation, explicit semantic analysis, crosslanguage information retrieval, machine translation.

1

Introduction

Dimensionality reduction techniques have traditionally been of interest for information retrieval as a means of mitigating the word mismatch problem. The term concept model is more general than dimensionality reduction and denotes a mapping from the word space to another representation that provides a smoother similarity measure than word statistics and is often induced from cooccurrence counts on paragraph or document level. Such a representation may for example be obtained by matrix approximation [7], by probabilistic inference [20] or techniques making use of the conceptual structure of corpora such as Wikipedia [9]. Cross-language information retrieval can be viewed as an extreme case of word mismatch, since for any two texts the vocabulary is in general disjoint if the languages are not the same. In order to have a cross-lingual similarity measure, it is necessary that concept spaces of different languages are aligned, which is often achieved by extending the notion of co-occurrence to pairs of translated or thematically related texts. While some work has been done on multilingual concept modeling [5, 8, 15–19], often the focus is on one method and a comparison with other methods is missing (but see [4]). One reason for this might be that concept models require adaptations for multilinguality that do not seem to be easily implemented. We will show that the adaptations can in H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 47–59, 2010. c Springer-Verlag Berlin Heidelberg 2010 

48

B. Roth and D. Klakow

fact be minimal and on the data side only. Another question that has not been investigated so far is how different multilingual concept models can contribute to each other and how they can be combined with word models, an approach that is standard for the monolingual case. The rest of the paper is structured as follows: In section 2 we summarize standard and multilingual Latent Dirichlet Allocation, a probabilistic concept model, and Explicit Semantic Analysis, a more recent explicit concept model. In section 3 we outline how we think the forementioned methods can be applied more profitably. Our experiments are described in section 4. In section 4.1 we explore both LDA and ESA on a mate retrieval task and observe much better results for LDA than reported so far and obtain consistent improvement for their combination. In section 4.2 we show how concept models can improve word-based cross-language retrieval. We end with an outlook on future work and conclusion.

2 2.1

Related Work Latent Dirichlet Allocation

Probabilistic Model. Latent Dirichlet Allocation (LDA) [2, 10, 20] is a latent variable model that gives a fully generative account for documents in a training corpus and for unseen documents. Each document is characterized by a topic distribution, words are emitted according to an emission probability dependent on a topic. The main difference to pLSA [11, 12] is that both topic distributions and word emission distributions are assumed to be generated by Dirichlet priors. It is common to parameterize the Dirichlet prior uniformly with parameters α = α1 = · · · = αT and to direct only the “peakiness” of the multinomials drawn from it. The LDA model describes the process of generating text in the following way: 1. For all k topics generate multinomial distributions ψ (zk ) = p(wj |zk ) ∼ Dir(β). 2. For every document d: (a) Generate a multinomial distribution θ(d) = p(zk |d) ∼ Dir(α). (b) Generate a document length, and topics zi ∼ θ(d) for every position i in the document. (c) Generate words wi ∼ ψ (zi ) for every position in the document. Usually, no generative account for the length of the document is given. Practical Issues. The first approach to estimate such a model [2] was to represent and estimate ψ and θ explicitly, resulting in different inference tasks to be solved and combined. Later approaches concentrate on getting a sample of the assignment of words to topics instead by Gibbs sampling [10]. To determine the similarity between two documents, one can compare either their sampled topic vectors or the probability vectors obtained from them [10].

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval

49

When other variational EM estimation techniques are applied, also other parameter vectors might be available and used [2, 5]. The comparison between these vectors can be done either by taking the cosine similarity of their angles or by using probability divergence measures. For language model based information retrieval, one is interested in the probability of a query, given a document. Wei and Croft [22] interpolate a language model based on LDA with a unigram language model directly estimated on the document. Whenever a sampling technique is used, one wants to be sure that the estimates are stable. This could be a problem when only one topic is sampled per position for short documents or queries. The most natural way to overcome this problem is to average the results of several sampling iterations. Multilingual LDA. LDA has been extended for several languages [16] (see also [15] for an investigation of the semantic clustering of this model and [6] for cross-lingual news-linking with it). Most of the components remain the same, the main difference is that for each language lsa different word emission distribution ψls is assumed. Depending on the language of a position in a document, a word is generated conditioned on the corresponding distribution. The model does not use the topic variable to estimate the language, as it could be the case for a monolingual model applied to a multilingual document without any adaptions on either the data or the model side. A theoretically sound model does not mean that it also provides a good bridging between two languages. It is crucial [15] how many “glue documents”, i.e. documents that indeed have counterparts in all compared languages, are available: Although the model does not try to capture the language, the latent variables might tend to structure the monolingual spaces without semantic alignment between the languages (imagine a multilingual text collection with only one glue document as an extreme case). 2.2

Explicit Semantic Analysis

Explicit Semantic Analysis (ESA) [5, 9, 17–19] is another scheme to overcome the word-mismatch problem. In ESA, the association strength of words to the documents in a training collection is computed, and vectors of these asscociations are used to represent the words in the semantic space spanned by the document names. These word representations can be used to compare words, they can also be combined for a comparison of texts. See [1] for the relation to the Generalized Vector Space Model [23]. Formalization. Several formalizations are possible in this setting. The fundamental ingredients that determine an implementation are: – Word vectors: For every word w a vector w, indicating its association strength to the documents is computed. – Text vectors: For a new document (or query) d a vector representation d is computed from the word vectors. – Similarity function: Giving two documents d1 and d2 the similarity is computed using a similarity function on their text vectors.

50

B. Roth and D. Klakow

The word vectors that were used in [9] and found to be optimal in [19] are obtained by taking the respective columns of the tf.idf-weighted document-term matrix A of the training collection. We use this choice of word vectors in our experiments. For the text similarity, several settings have been proposed. In [9] a weighting of the word vectors is used, they are multiplied with scalars equal to, again, an tf.idf weighting of the terms, and then summed. Sorg and Cimiano [19] explore further combination schemes, including the sum of the elements of either the multiset (considering term frequency) or of the set (not considering term frequency) of word vectors. They find that the set combination works best, yielding preliminary text vectors of the form: ˆ= d



w

w∈d

It is beneficial to truncate the resulting vectors at a certain threshold. The thresholding that turned out to be most successful [19] was to retain the 10000 ˆ Again, we use this parametrization in biggest non-zero values of the vectors d. all following experiments that involve ESA. As a similarity function the cosine is suggested [19] and used by us. Multilingual ESA. The application of this model in a multilingual setting is straightforward. For L languages consider document term matrices A(1) · · · A(L) . Construct the matrices in a way that the document rows correspond. For all lan(·) guages each of the rows An contains documents about the same topic across the languages. Therefore only documents can be included that are available in all of the considered languages. For each document the mapping to text vectors is performed using a monolingual matrix corresponding to its language. As the documents are aligned, similarities can be computed across languages. Because the relative frequency is used in the tf.idf-weighting, all documents are normalized and no bias occurs for documents longer in one language than in another.

3

Making Use of Concept Models for CLIR

Making Use of LDA. In our experiments we want to generalize the intuition given for the multilingual LDA model [15]: not only should a large number of glue documents exist, good bridging documents should optimally be of equal length. Experimentation with a small fraction of the training data indicated that the multilingual LDA model and a monolingual LDA model on documents normalized to equal length on both language sides yield about the same performance, while a monolingual model on unnormalized data performs considerably worse. Moreover, highly optimized and parallelized toolkits that allow us to perform training on all Wikipedia articles have only been developed for standard LDA. We believe, therefore, that it is a promising approach to normalize the data suitably to be processed with a standard monolingual LDA model.

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval

51

Wikipedia is used as a parallel training corpus: corresponding articles are concatenated, their length is normalized to match the length of the counterpart in the other language. We propose two methods of length normalization: First, to cut off every document at a certain length. Second, to retain for the longer language side of an article only a random sample of size equal to the smaller language side. A resizing with a scalar is not possible because the sampling process requires integer counts. The vocabulary is uniquely identified for every language by attaching suitable prefixes (en , de ) to the words. Similarity is measured between two texts after inference by taking the cosine between their vectors of sampled topic statistics. Making Use of ESA. ESA is applied as described in section 2.2, which is as closely as possible as reported in [19]. Making Use of Machine Translation. The machine translation retrieval model translates the queries with a standard Moses [14] translation model trained on Europarl [13]. Translated queries and text are then compared by the cosine of their tf.idf-weighted word-vectors. For the document-term matrix D of the target collection with N documents, we use the commonly used weighting function Dn,w log N ′ w ′ ∈W Dn,w ′

tf.idf (w, n) = 

n =1

N 1Dn′ ,w >0

Here, 1Dn′ ,w >0 is an indicator that equals to one if word w has appeared in document dn′ at least once, and that equals to zero otherwise. Model Combination. We use a simple scheme to combine models by concatenating L2 -normalized vectors. Let u be a m-dimensional vector and v be a n-dimensional vector which represent the same document in two different models. Then the model combination with interpolation weight α represents this document by an m + n-dimensional vector w: u α i if 1 ≤ i ≤ m , wi = v|u| (i−m) (1−α) otherwise. |v| This way any models can be combined as long as they are in vector representation. This the case for all models mentioned above, although they rely on very different principles. Similar combinations have been proven effective in the case of pLSA for monolingual retrieval [11].

4

Experiments

We are using two datasets, the Multext JOC corpus for the task of finding translations, and a CLEF 2000 query-based retrieval collection. These datasets and the experiments are described in detail in the next section.

52

B. Roth and D. Klakow

4.1

Mate Retrieval on Multext JOC

There is only one publication [5] known to us that compares LDA for crosslanguage information retrieval with ESA. Interestingly, our experiments on the same dataset will suggest a distinctively different assessment of the potential of LDA for the same task. The basis of the evaluation is the Multext JOC corpus1 which consists of 3500 questions to the European Parliament and of answers to these questions. As in [5] we use the concatenation of a question together with its answer as a query in one language to search the collection of translations in another language for its counterpart. Our experiments were done with English as the query language and German as the target language. Only preprocessing steps that are clear and easy to reproduce were performed. Exactly those questions were retained that to which an answer was assigned and had the same id in English and German. This resulted in a set of 3212 texts in each language, 157 more than were used in [5]2 . Sequences of characters in Unicode letter blocks were considered words. Words with length = 1 or length > 64 and words contained in the Snowball stopword list were ignored. All other words were stemmed with the publicly available Snowball stemmer3 . In contrast to [5], no compound splitting was done. For the training collection all pairs of Wikipedia articles4 were used that have bidirectional direct cross-language references. All markup was stripped off by using the same filter as in a publicly available ESA implementation5 . Wikipedia articles of less than 100 words in either language were ignored and words with a Wikipedia document frequency of 1 were filtered out. The final training corpus consists of 320000 bilingual articles. Performance of retrieval was measured in mean reciprocal rank (mrr). The ESA retrieval experiment was performed using the same parametrization as discribed before and the result of [5] was reproduced to a difference of 1% (in our experiments we obtained a score of mrr = 0.77 compared with mrr = 0.78). As for the LDA experiments, we were interested in the effect of length normalization of the training documents. We compare two methods: First, every document was cut off at a length of 100 words. Second, the method of down sampling the longer language side to the length of the smaller one was applied. We marked each word with a prefix indicating its language and retained a vocabulary size of roughly 770 thousand and 2.1 million for the cut-off method and for the downsampling method respectively. Both training collections were embedded with 125, 250 and 500 dimensions, and additionally with 1000 dimensions for the cut-off corpus (the vocabulary size was the limiting factor with respect to our computing facilities). The Google plda package [21] was used with the sug50 gested parameters (α = and β = 0.01). With the trained model, topics #topics were inferred for the monolingual Multext documents. In order to get a stable 1 2 3 4 5

http://www.lpl.univ-aix.fr/projects/multext The exact document selection criterion of their experiments is unknown to us. http://snowball.tartarus.org/ We used the German snapshot of 2009-07-10 and the English snapshot of 2009-07-13. http://code.google.com/p/research-esa/

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval 0.7

53

lda, cutoff lda, downsampled

0.65

0.6

mrr

0.55

0.5

0.45

0.4

0.35

0.3 100

200

300

400

500

600

700

800

900

1000

dim

Fig. 1. Performance of LDA models estimated with dimensions (=numbers of topics) equal to 125, 250, 500 and 1000. Thick lines indicate combinations of all models up to the dimension on the x− axis. Drops in the perfomance curve on single points may be due to local sampling optima. Table 1. Performance of LDA on Multext LDA method number of topics Cimiano et al. 500 length downsampling 500 length cut-off 500 length downsampling 125 + 250 + 500 length cut-off 125 + 250 + 500 + 1000

mrr .16 .42 .53∗∗ .55∗∗ .68∗∗

estimate, the statistics of 50 sampling iterations were averaged. Similarity in the LDA setup was measured by taking the cosine similarity between the sampling statistics. A drastic improvement over non-normalized LDA can be observed: while [5] report a score of mrr = 0.16 for their 500-dimensional LDA model, we get mrr = 0.53 with the cut-off corpus. We suppose that the reason for this difference is that a non-multilingual LDA model applied to a comparable corpus estimates the predominant language of a document rather than its semantic content. Another improvement can be observed by combining the results of different models, a technique that is usually applied for pLSA [12]. In this case, the samping statistics of runs with different dimensional models were L2 -norm normalized and concatenated without further weighting. This yielded a score of mrr = 0.68 for the cut-off model, showing performance in the same order of magnitude as ESA. Figure 1 and Table 1 give a survey of the results obtained with LDA. Scores significantly better than in the respective line above having

54

B. Roth and D. Klakow 0.9

α lda + (1-α) esa

0.85

mrr

0.8

0.75

0.7

0.65 0

0.2

0.4

0.6

0.8

1

α

Fig. 2. Combining LDA and ESA on the Multext corpus. The improvement over ESA alone is significant with p ≪ 0.005 for .1 ≤ α ≤ .7.

p ≪ 0.005 in the paired t-test are marked with ∗∗. (Of course we could not test against scores reported in [5], for lack of the original numerical data.) In order to determine how different the ESA and the LDA models are and how much can they contribute to each other, we combined the vector representations of both models by different interpolation factors 0 ≤ α ≤ 1. A stable improvement in performance with maximum mrr = 0.89 was achieved for giving the cut-off LDA model a weight of 0.4 and the ESA model a weight of 0.6. See Figure 2. 4.2

Query-Based Retrieval with CLEF2000

Mate retrieval experiments can be criticized as being an unrealistic retrieval scenario. Therefore, a second evaluation was done on the CLEF6 German-English ad-hoc track of the year 2000. The target corpus consists of about 110000 English newspaper articles together with 33 German queries for which relevant articles could be pooled. For our experiments the title and description fields of the queries were used and the narrative was ignored. A common strategy for cross-language retrieval is first to translate the query and then to perform monolingual retrieval. While the translation process would have taken prohibitively long for the Multext corpus, we performed query translation on the CLEF2000 queries with a standard Moses translation model trained on Europarl. Retrieval with the translated queries was done by comparing the cosine of the tf.idf-weighted word-vectors. 6

See http://www.clef-campaign.org/; The evaluation packages are available via the ELRA catalogue http://catalog.elra.info, The CLEF Test Suite for the CLEF 2000-2003 Campaigns, catalogue reference: ELRA-E0008.

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval

55

0.26 0.24 0.22 0.2

map

0.18 0.16 0.14 0.12 0.1 α esa α lda, cutoff α (lda + esa) mt tfidf

0.08 0.06 0

0.2

0.4

0.6

0.8

1

α

Fig. 3. Interpolation of concept models (having interpolation weight α) with machine translation (weighted 1 − α) on CLEF2000. The improvement given by the LDA-ESA is significant with p < 0.005 for .2 ≤ α ≤ .6.

We evaluated both the machine translation model and the concept models trained on Wikipedia. In addition to the most commonly used mean average precision score (map) we also evaluated by geometric mean average precision (gmap), which rewards stable results for hard queries. The ESA and the cut-off LDA models with dim = 500 perform equally well for map, while the combination of LDA dimensions gets a considerably better score. This is in contrast to the findings in the mate-retrieval setup. The reason for that, we suspect, may be that the parameters of ESA have been found in order to optimize such a setting. For gmap, LDA consistently outperforms ESA. For the combined LDA-ESA model a slight improvement could be observed using the combination with α = .5 (henceforth referred to as combined concept model ). The machine translation model (map = .203) performed better than LDA and ESA. When interpolated with the machine translation model all three concept models (LDA, ESA and combined) achieved improvements. The biggest and most stable improvement was achieved by the interpolation of machine translation and combined concept model yielding a score up to map = .245 with equal weight for the concept model and the machine translation model. Figure 3 and Table 2 show an overview of the results. Scores significantly better than in the respective line above, having p < 0.05 and p < 0.005 in the paired t-test, are marked with ∗ and ∗∗ respectively. The evaluation results of the combined system lie well within the range of the participating systems in CLEF 2000 for the same track7 . Particularly, no manually edited lexicon and no compound-splitter is used in our case. 7

For licensing reasons, we are not allowed to make a direct comparison, but see [3] for a survey.

56

B. Roth and D. Klakow Table 2. Query-based retrieval on CLEF2000 method parameters map gmap ESA .071 .003 LDA, cut-off d = 500 .071 .010 LDA, cut-off d = all .155∗ .043∗ LDA+ESA α = .5 .176 .054 MT tf.idf .203 .061 concept+MT α = .5 .245∗∗ .128∗∗

Table 3. Improvement of map through the ESA+LDA concept component compared with error rates. The 4th column indicates the change in comparison to machine translation alone (3rd column). The 5th column contains one + per unknown word, the last column the percentage of junkwords per query. id 18 9 7 4 26 11 12 17 5 15 38 21 14 28 33 16 13 40 31 32 20 36 22 37 19 1 30 24 10 29 39 34 3

query title Unf¨ alle von Brandbek¨ ampfern Methanlagerst¨ atten Doping und Fußball ¨ Uberschwemmungen in Europa Nutzung von Windenergie Neue Verfassung f¨ ur S¨ udafrika Sonnentempel Buschbr¨ ande bei Sydney Mitgliedschaft in der Europ¨ aischen Union Wettbewerbsf¨ ahigkeit der europ¨ aischen... R¨ uckf¨ uhrung von Kriegstoten Europ¨ aischer Wirtschaftsraum USA-Tourismus Lehrmethoden f¨ ur nicht-englischsprachige... Krebsgenetik Die Franz¨ osische Akademie Konferenz u ¨ber Geburtenkontrolle Privatisierung der Deutschen Bundesbahn Verbraucherschutz in der EU Weibliche Priester Einheitliche europ¨ aische W¨ ahrung Produktion von Oliven¨ ol im Mittelmeerraum Flugzeugunf¨ alle auf Start- und Landebahnen Untergang der F¨ ahre Estonia Golfkriegssyndrom Architektur in Berlin Einsturz einer Supermarktdecke in Nizza Welthandelsorganisation Krieg und Radio Erster Nobelpreis f¨ ur Wirtschaft Investitionen in Osteuropa oder Rußland Alkoholkonsum in Europa Drogen in Holland

scores errors mt ap ∆ap unknowns junk(%) 0.000 ap = 0.033 + 0.71 0.000 ap = 0.017 ++ 0.50 0.004 +361.36% 0.33 0.007 +209.09% 0.17 0.071 +167.40% 0.13 0.093 +111.95% 0.09 0.009 +109.57% + 0.50 0.138 +96.09% ++ 0.29 0.060 +89.38% 0.08 0.177 +88.51% 0.10 0.045 +86.84% +++ 0.33 0.165 +73.38% 0.22 0.015 +67.72% + 0.17 0.072 +61.43% + 0.29 0.382 +52.46% + 0.38 0.084 +46.84% 0.33 0.137 +46.68% 0.17 0.015 +46.10% 0.25 0.222 +46.03% 0 0.234 +34.99% 0.22 0.218 +24.13% 0 0.269 +23.29% 0.11 0.094 +19.76% + 0.22 0.926 +4.13% 0.17 0.704 +1.44% 0.25 0.651 +1.27% 0.17 1.000 −.01% + 0.13 0.516 −6.94% 0 0.015 −11.47% 0 0.072 −11.99% 0 0.041 −16.99% 0.15 0.060 −24.18% 0 0.185 −55.66% 0

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval

57

Error Analysis. A querywise error analysis is difficult as the inner workings of quantitative methods are often opaque to human analysis. However, the machine translation output is the most contributing source and it is accessible to examination. We sorted the machine translation output by how much it profited by the concept models in the best performing setting. In Table 3 we report the score that is obtained by machine translation and the increase when combined with the concept models. We analyzed how often a word was obviously unknown by the machine translation system trained on Europarl and therefore wrongly just copied over. It would be possible to recognize this type of error automatically. In addition, for every translated query we counted how many words in it had no semantic meaning related to the purpose of the query and were therefore useless (these words are hence called junk words). Junk words are, for example, function words not filtered by the stopword list, machine translation errors of several kinds and artefacts from the query formulation (e.g. “Gesucht sind Dokumente, die ... liefern” in the description part of query 4). The junk word error type would be more difficult to detect. Although the analyzed data basis is small, we conjecture that the concept model makes such queries more robust which induce one of the two errors, while it might be less useful where a good translation is present and the terms are weighted well: In the cases where the concept models contributed there were, on average, 0.53 unknown words for machine translation and 0.24% junk words, in contrast to 0.14 unknown words and 0.04% junk words in the cases where the concept model decreased the score. For future experiments it might be interesting to test whether a trade-off weighting between translation and concept model conditioned on a reliability score of the machine translation improves performance.

5

Future Work

Our experiments have been done in a vector space retrieval framework, because this made model combination straightforward and allowed direct comparison to other experimental setups reported in the literature. However, it would be interesting to include the cross-lingual LDA-model in a language model setup, similar to [22] in the monolingual case. The inclusion of ESA in such a model would be more complicated. We also leave experiments with more language pairs for future work. While we have indicators to when concept models are beneficial for word-based retrieval, the effects that take place when combining LDA and ESA would be more difficult to uncover. Research in this direction could focus on the influence of term-weighting in ESA and on that of disambiguation by context in the case of LDA.

6

Conclusion

For cross-language retrieval, it is essential to have a cross-language bridge that is immediately available, in the best case for many languages and with up-to-date

58

B. Roth and D. Klakow

vocabulary. To work in practice, a method to extract such bridging information should not rely on specialized algorithms, but on approved techniques for which robust implementations exist that run on large computing facilities. In this work we have shown that Wikipedia is a valuable bridging source and that standard (monolingual) LDA can be applied to multilingual training data when care is taken for suitable length normalization. Thus, we get an improvement of 325% mrr compared to a non-competitive score previously reported for LDA with non-normalized Wikipedia data on the Multext corpus. A second finding is that simple model combinations reliably increase performance. For retrieval of document translations (mate retrieval) the combination of ESA and LDA achieves scores 16% mrr better than reported so far for ESA alone. Concept models based on Wikipedia are also complementary to word based retrieval using machine translation output, here we observe an increase by 21% map compared to a machine translation base-line. While ESA performs better than LDA for mate retrieval, this ranking is reversed for the more relevant task of query-based retrieval. This may be because commonly used ESA-parameters have been tuned for retrieval of document translations.

Acknowledgements We want to thank Matt Lease and the Spoken Language Systems QA-group for helpful discussions.

References [1] Anderka, M., Stein, B.: The ESA retrieval model revisited. In: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pp. 670–671. ACM, New York (2009) [2] Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. The Journal of Machine Learning Research 3, 993–1022 (2003) [3] Braschler, M.: CLEF 2000-Overview of results. In: Peters, C. (ed.) CLEF 2000. LNCS, vol. 2069, p. 89. Springer, Heidelberg (2001) [4] Carbonell, J.G., Yang, Y., Frederking, R.E., Brown, R.D., Geng, Y., Lee, D.: Translingual information retrieval: A comparative evaluation. In: International Joint Conference on Artificial Intelligence, Citeseer, vol. 15, pp. 708–715 (1997) [5] Cimiano, P., Schultz, A., Sizov, S., Sorg, P., Staab, S.: Explicit vs. Latent Concept Models for Cross-Language Information Retrieval. In: Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI 2009 (2009) [6] De Smet, W., Moens, M.F.: Cross-language linking of news stories on the web using interlingual topic modelling. In: The 2nd Workshop on Social Web Search and Mining, SWSM 2009 (2009) [7] Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. Journal of the American society for information science 41(6), 391–407 (1990)

Combining Wikipedia-Based Concept Models for Cross-Language Retrieval

59

[8] Dumais, S.T., Letsche, T.A., Littman, M.L., Landauer, T.K.: Automatic crosslanguage retrieval using latent semantic indexing. In: AAAI Spring Symposuim on Cross-Language Text and Speech Retrieval, pp. 115–132 (1997) [9] Gabrilovich, E., Markovitch, S.: Computing semantic relatedness using wikipediabased explicit semantic analysis. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 6–12 (2007) [10] Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proceedings of the National Academy of Sciences 101(90001), 5228–5235 (2004) [11] Hofmann, T.: Probabilistic latent semantic indexing. In: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 50–57. ACM, New York (1999) [12] Hofmann, T.: Unsupervised Learning by Probabilistic Latent Semantic Analysis. Machine Learning 42(1), 177–196 (2001) [13] Koehn, P.: Europarl: A parallel corpus for statistical machine translation. In: MT summit, vol. 5 (2005) [14] Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al.: Moses: Open source toolkit for statistical machine translation. In: Annual meeting-association for computational linguistics, vol. 45 (2007) [15] Mimno, D., Wallach, H.M., Naradowsky, J., Smith, D.A., McCallum, A.: Polylingual Topic Models. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 880–889 (2009) [16] Ni, X., Sun, J.T., Hu, J., Chen, Z.: Mining multilingual topics from wikipedia. In: Proceedings of the 18th international conference on World wide web, pp. 1155– 1156. ACM, New York (2009) [17] Potthast, M., Stein, B., Anderka, M.: A wikipedia-based multilingual retrieval model. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, p. 522. Springer, Heidelberg (2008) [18] Sorg, P., Cimiano, P.: Cross-lingual information retrieval with explicit semantic analysis. In: Working Notes of the Annual CLEF Meeting (2008) [19] Sorg, P., Cimiano, P.: An Experimental Comparison of Explicit Semantic Analysis Implementations for Cross-Language Retrieval. In: Proceedings of the 14th International Conference on Applications of Natural Language to Information Systems, NLDB 2009 (2009) [20] Steyvers, M., Griffiths, T.: Probabilistic topic models. Handbook of Latent Semantic Analysis, 427 (2007) [21] Wang, Y., Bai, H., Stanton, M., Chen, W.-Y., Chang, E.Y.: Plda: Parallel latent dirichlet allocation for large-scale applications. In: Proc. of 5th International Conference on Algorithmic Aspects in Information and Management (2009), Software available at, http://code.google.com/p/plda [22] Wei, X., Croft, W.B.: LDA-based document models for ad-hoc retrieval. In: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 178–185. ACM, New York (2006) [23] Wong, S.K.M., Ziarko, W., Wong, P.C.N.: Generalized vector spaces model in information retrieval. In: Proceedings of the 8th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 18–25. ACM, New York (1985)

Exploring Contextual Models in Chemical Patent Search Jay Urbain1 and Ophir Frieder2 1

Electrical Engineering & Computer Science Department Milwaukee School of Engineering Milwaukee, WI [email protected] 2 Department of Computer Science Georgetown University Washington, DC [email protected]

Abstract. We explore the development of probabilistic retrieval models for integrating term statistics with entity search using multiple levels of document context to improve the performance of chemical patent search. A distributed indexing model was developed to enable efficient named entity search and aggregation of term statistics at multiple levels of patent structure including individual words, sentences, claims, descriptions, abstracts, and titles. The system can be scaled to an arbitrary number of compute instances in a cloud computing environment to support concurrent indexing and query processing operations on large patent collections. The query processing algorithm for patent prior art search uses information extraction techniques to identify candidate entities and distinctive terms from the query patent’s title, abstract, description, and claim sections. Structured queries integrating terms and entities in context are automatically generated to test the validity of each section of potentially relevant patents. The system was deployed across 15 Amazon Web Services (AWS) Elastic Cloud Compute (EC2) instances to support efficient indexing and query processing of the relatively large 100G+ collection of chemical patent documents. We evaluated several retrieval models for integrating statistics of candidate entities with term statistics at multiple levels of patent structure to identify relevant patents for prior art search. Our top performing retrieval model integrating contextual evidence from multiple levels of patent structure resulted in bpref measurements of 0.8929 for the prior art search task, exceeding the top results reported from the 2009 TREC Chemistry track. Keywords: Information Retrieval, Patent Retrieval, Scaleable Database Systems, Dimensional Modeling, Probabilistic Modeling, Chemical IR.

1 Introduction The TREC Chemistry Track for 2009 was organized to evaluate the statistical significance on the ranking of information retrieval (IR) systems and the scalability of IR H. Cunningham, A. Hanbury, and S. Rüger (Eds.): IRFC 2010, LNCS 6107, pp. 60–69, 2010. © Springer-Verlag Berlin Heidelberg 2010

Exploring Contextual Models in Chemical Patent Search

61

systems when dealing with chemical patents [1]. A test collection was assembled from approximately 1.2M patent files (approximately 100G) of full-text chemical patents and research papers to evaluate two ad-hoc retrieval tasks common to patent investigation: technology survey and prior art search. The goals of technology survey search and prior art search are fundamentally different. Technology survey search is similar to ad-hoc retrieval targeting patent documents using a natural language query to satisfy an information need. Systems are required to return a set of documents that is relevant to this information need. The goal of the technology survey evaluation is to identify how current IR methods adapt to text containing chemical names and formulas. Systems for the technology survey task are evaluated using a pooling, sampling, and expert evaluation methodology. The goal of prior art search is to evaluate the validity of a patent claim. In this task, systems attempt to identify prior art that may invalidate a patent claim. The query set for this evaluation consists of 1000 patent files with prior art references removed. Systems are required to return a set of documents relevant to the prior art of claims stated in the patent. Of special interest in this task was to consider three types of topics: full text patents, description only, and claims only. Systems for this task are evaluated automatically using the known references for each patent. The focus of this research effort is on prior art search. Chemical and patent information retrieval are challenging tasks. Chemical IR requires a chemical named entity identification strategy for dealing with large multiword terms, synonyms, acronyms, and morphological variants used for identifying the same chemical concept. For example, Dipeptidyl peptidase-IV inhibitor can also be referred to as Dipeptidyl peptidase-4, DPP4, DPP-4, or dipeptidylaminopeptidase. Guggulsterone can be identified as Pregna-4,17-diene-3,16-dione, Guggulsterone-E, Guggulsterone-Z, trans-guggulsterone, or as the guggulu steroid extract. Many chemical terms also have hierarchical hyponym-hypernym relationships. For example, silver halides, AgX, could include silver bromide (AgBr), silver iodide (AgI), or silver fluorides. The subject matter of chemical patents can also extend far beyond topics in chemistry and law, and include topics, vocabulary, and related data from disparate fields. For example, many patents include chemical reactions related to metabolic processes and include the biological vocabulary of genes, proteins, and metabolic networks. Pharmaceutical patents typically integrate pharmaceutical nomenclature with chemistry, biology, and medical disease processes and treatment. Engineering patents can include chemical compounds and processes for developing effective adhesives to a broad range of topics in materials science. All of these fields contain large, complex vocabularies that can complicate the process of term normalization and entity identification. Chemical patent retrieval clearly requires not only the identification of entities across disparate fields, but the relationships between entities in the context of how they are applied. Patent documents are often large, relatively complex documents with inconsistent structure and semantics. Due to problems with legal standards for creating titles and abstracts, these fields can provide little additional value for search and can obfuscate

62

J. Urbain and O. Frieder

the true nature of an invention [2]. For example, the European Patent Office (EPO) advises applicants to not disclose inventions in titles to prevent early disclosure. The US Patent and Trade Office (USPTO) provides guidelines limiting abstracts to 150 words. In addition, a significant portion of patent abstracts can be taken up with disclosures. Patent text can also be obfuscated by using “word smithing” to not preclude getting a patent, and at the same time not limit the scope of a patent. In addition, the structure of patent documents can inconsistently include claim text in description fields and vice versa often due to OCR (optical character recognition) processing of documents. The structure of patent documents is important for identifying and validating specific claims as they serve as the legal basis of the patent, however descriptions in other portions of the documents may provide important context and alternative nomenclature [3]. Clearly, information retrieval approaches are needed that integrate multiple forms of evidence from throughout the patent application. To meet the needs of chemical patent IR, and the scalability needs of large patent collections, we have developed a distributed search engine based on a dimensional data model. The model supports entity identification and aggregation of term statistics at multiple levels of patent structure including individual words, sentences, claims, descriptions, abstracts, and titles [4]. The query processing algorithm uses information extraction and locally aggregated term statistics to identify and help disambiguate candidate entities in context. Extracted candidate entities are filtered using discriminative measures of how informative an entity phrase is with respect to the patent document. Structured queries are automatically generated based on the relative distinctiveness of individual terms and candidate entities from the query patent's claims, abstract, title, and description sections. We evaluated several probabilistic retrieval functions for integrating statistics of retrieved entities with term statistics at multiple levels of document structure to identify relevant patents. Our primary objective in this research was to develop a scalable and flexible system and evaluate methods for integrating multiple levels of context as a basis for further research. We first describe our distributed indexing model, followed by the system description, indexing process, query processing algorithms, and our retrieval models.

2 Distributed Dimensional Data Model Paragraphs, sentences, and terms, representing complete topics, thoughts, and units of meaning respectively, provide a logical breakdown of document lexical structure into finer levels of meaning and context [5]. We capture these hierarchical relationships within a search index based on a dimensional data model. As shown in Fig. 1, this dimensional model can be logically represented as an n-dimensional cube. Where each patent document is represented as a series of paragraphs (title, abstract, descriptions, and claims). Each paragraph is represented as a sequence of sentences, and each sentence is represented as a sequence of individual terms. Such a model facilitates

Exploring Contextual Models in Chemical Patent Search

63

Fig. 1. Search index based on dimensional model

search for multi-word terms and efficient aggregation of term statistics within multiple levels of patent structure. The model also supports term variants, however no term variants or synonyms from external data sources were used in this study. As shown in Figure 2, we represent the dimensional index as a star schema [6][7] with a dimension table for each level of document structure (document, paragraph, sentence, term) and three central fact tables or postinglists. The “grain”, i.e., the

Fig. 2. Search index based on dimensional model

64

J. Urbain and O. Frieder

smallest non-divisible element of the database, is the individual word as represented in the term postinglist. Pre-aggregated postinglists for sentences, paragraphs, and documents were generated to improve sentence, paragraph, and document query performance respectively. Sentences aggregate words in sequence by position, paragraphs aggregate sentences, and documents aggregate paragraphs. The index can be extended to include additional dimensions, and allows for efficient formulation of SQL search queries. By indexing each individual word, queries can be developed for searching single- and multi-word terms, and term statistics can be aggregated over different levels of document structure.

3 System Description Indexing, retrieval, and analysis applications were developed in Java. The MySQL 5.1 database with the MyISAM storage engine was used for index storage and retrieval. 15 Amazon Elastic Cloud Compute (EC2) m1.small instances based on the Ubuntu Hardy base (ami-ef48af86) machine image were allocated for processing each of 15 database shards [8]. Each shard was roughly equivalent to the Chemical IR track collection distribution. For example, one shard for EP 000000, one shard for EP 000001, one shard for US 020060, etc. Elastic Block Storage (EBS) volumes of 350G were allocated for each compute instance to accommodate the size of the index and the need to insure persistence of the database if a compute instance was restarted. An m1.small EC2 compute unit consisted of 1.7 GB memory, 1 32-bit virtual core, and 160 GB of storage. Experiments with larger dual-core compute instances improved indexing performance 2-fold per instance, but did not significantly improve query performance. It takes approximately 2 days to construct the entire index. Each m1.small instance cost $0.10 per hour. Additional charges are encountered for data loading and storage. Each compute instance performed roughly equivalent to a standard Pentium 4 laptop with 2G of memory. The total cost of the experiment was approximately $1,000 though this included a fair amount of trial and error to get things running.

4 Indexing Process The indexing process includes the following: 1.

2.

3.

Lexical Partitioning: Documents are parsed into title, abstract, descriptions, and claims. Each is subsequently parsed into paragraphs, and these paragraphs are parsed into sentences. Tokenization: Sentence terms are tokenized, stop words removed, and lexical variants are normalized. Porter stemming [9] is used on each token with the following exceptions: all upper case, mixed case, alpha-numeric terms. Small “s” is also stripped from all upper-case terms. Indexing: Each individual word is stored in the index with positional information and its paragraph type (title, abstract, description, or claim).

Exploring Contextual Models in Chemical Patent Search

65

5 Query Processing 5.1 Paragraph Queries For paragraph search, the query patent title, abstract, up to the first 20 claims, and up to the first 20 descriptions (depending on the experiment) are extracted as paragraphs. Using the following as an illustrative example: “We are a new pharmaceutical company that is interested in entering the area of Dipetidyl peptidase-IV inhibitors…”, each paragraph is processed as follows: 1. 2. 3.

4. 5.

6.

BM25:

Sentences are extracted. Part-of-speed tagging is performed. Candidate entities are identified by locating non-recursive noun phrases: [pharmaceutical_NN company_NN], [Dipetidyl_NN peptidase-IV inhibitors_NNS]. Candidate entities are cached in a collection for subsequent filtering. Stop and function words are removed. The top 7 terms by NIDF (normalized inverse document frequency) above a minimum threshold of 0.10 (0.0 to 1.0) is used to generate a query for each paragraph The top 500 paragraphs are retrieved using the probabilistic BM25 retrieval function [10] shown in equation (1). BM25 is implemented using standard SQL. ⎛ ⎞ ⎜ ⎟ ⎛ N − df + 0.5 ⎞⎜ k1 + 1) * tfd ( ⎟⎛⎜ (k 3 + 1) * tfq ⎞⎟ ∑wq ln⎜⎜ df + 0.5 ⎟⎟⎜ ⎟⎜ k 3 + tfq ⎟⎠ docLen ⎠ k1 * (1 − b) + b * ( ⎝ ) + tfd ⎟⎝ ⎜ avgDocLen ⎝ ⎠

(1)

Note: We used k1=1.4, k2=0, k3=7, and b=0.75

5.2 Entity Queries From the cached list of candidate entities from across title, abstract, claim, and description paragraphs, entities are selected as follows: 1.

2. 3.

Entities must occur in at least 2 paragraphs and have a NIDF > 0.15. We have found that this drastically reduces the number of spurious entity phrases. Entity phrases are ranked by NIDF and their log frequency of occurrence. The top 20 remaining entity phrases are searched within the context of title, abstract, description, and claims paragraphs of target patent documents.

66

J. Urbain and O. Frieder

The following abbreviated query illustrates entity search for “Dipetidyl peptidase-IV inhibitor”. All queries are distributed across all database shards and results are aggregated: select i1.term, p1.docid, p1.parid, p1.sentid, p1.seq, p1.section, d.docnum from invertedindex_qc i1, postinglist_qc p1, paragraphresults_qc d where i1.term=' dipetid’ ' and i1.termid=p1.termid and p1.docid=d.docid and d.parid=p1.parid and exists ( select * from invertedindex_qc i2, postinglist_qc p2 where i2.term=' peptidas' and i2.termid=p2.termid and p1.docid=p2.docid and p1.parid=p2.parid and p1.sentid=p2.sentid and p1.section=p2.section and abs(p2.seq-p1.seq) 0, and then transformed, letting ǫ → 0. The second transformation is the standardized score or z-score whose use in x ¯) IR was motivated by Webber et al. [13]. It is defined as z = (x− , where x σ is a metric’s score, e.g. an AP score. In addition, x ¯ and σ are the average and standard deviation of a set of scores measured across a set of retrieval systems on a fixed topic. Hence, for a particular topic it is defined as z=

(AP − M ean(AP )systems ) SD(AP )systems

(3)

Measuring the Variability in Effectiveness of a Retrieval System

77

In the following, we observe several properties of the logit(AP) and the standardized z-scores.

Fig. 4. The frequency distributions before and after transformation of three runs in Web track. (a) a run with a low MAP, (b) a run with a medium MAP, (c) a run with a high MAP.

Boundary Values and Score Distributions. Figure 4 shows three runs of the Web track collection: (a) with a low MAP value, (b) with a medium MAP value, and (c) with a high MAP value. The distributions of the AP scores before and after both the logit and z-score transformations are presented as frequency histograms. The logit and z-score transformations differ significantly in the way they handle boundary values. The logit transformation transforms the boundaries to extreme values in the transformed space. This is observed by the extreme values at each end of the distributions in the middle column. In contrast, the z-score transformation disperses the boundaries smoothly as illustrated in the right column. In addition, the z-score transformation helps eliminate the source of variance coming from topic difficulty3 before measuring the variability of system effectiveness itself. We now consider the variability in a system’s effectiveness as the standard deviation of the transformed AP values. Let MLAP refer to the mean of the logit-transformed AP values and let MSAP refer to the mean of the standardized z-transformed AP values. Figure 5 shows the scatterplots of the standard 3

A topic is regarded as difficult if the range of effectiveness scores measured across a set of systems is small and near to zero.

78

M. Hosseini et al.

deviations in transformed AP values as a function of their mean values, MLAP and MSAP. As seen in the figure, the logit and z-score transform the scores in different ranges. In addition, there is no longer a monotonic relationship between the values of mean and variability.

Fig. 5. Variability in effectiveness versus mean of transformed AP values: logit (a) and the z-score transformation (b)

3.3

Variability as a Tie Breaker

We consider all pairs of the top 75% (ordered by MAP) of systems participating in either the Robust or Web track of TREC 2004. We compare systems based on the mean of the standardized z-scores (MSAP). We use the paired t-test to measure the significance of MSAP differences. We set the significance level to 0.05. For all the ties we use the F-test and Levene’s test to investigate the proportion of ties for which the variabilities in effectiveness’s scores are significantly different. Table 2. The variability in effectiveness as a tie breaker: number of pairs, ties and broken ties in two tracks of TREC 2004 Collections Pairs Robust Web

Status

before transformation after transformation before transformation 1485 after transformation 3321

Ties 997 857 469 415

(30%) (26%) (31%) (28%)

Broken ties F-test Levene 0 (0%) 106 (11%) 280 (33%) 404 (47%) 1 (0.002%) 21 (0.04%) 140 (34%) 158 (38%)

As seen in Table 2 for the Robust track, 30% of pairs are considered ties, when using AP score, and 26% are considered ties in the transformed space. Interestingly, before transformation, the F-test cannot distinguish any statistical difference in variability, and the Levene’s test can only break 11% of the ties.

Measuring the Variability in Effectiveness of a Retrieval System

79

In contrast, after transformation into the z-space, the F-test can distinguish between 33% and Levene’s test can distinguish between 47% of the tied pairs. A similar effect before and after transformation is observed for the Web track. 3.4

The Effect of Topic Set Size on Measuring Variability in Effectiveness

If we are to use variability to characterize systems, it will be useful to know how many topics are needed to reliably compare two systems in terms of variability in effectiveness. Indeed, we will need to know how likely a decision would change if we compare systems using a different topic set. This performance variation across topic sets has previously been studied in the context of average performance [12]. We perform the same experiment to compare variabilities in systems’ effectiveness. foreach topic set size c from 10, 20, 30, ... , 100 { set the counters to 0; foreach TREC test collection t { foreach pair of systems A and B from track t { foreach trial from 1 to 50 select two disjoint sets of topics X and Y of size c from t; if ( the difference between the variabilities is significant){ d_X=SD(A,X)-SD(B,X); d_Y=SD(A,Y)-SD(B,Y); increment counter; if(d_X * d_Y < 0) { increment swap counter; } } }} error-rate (c) =swap counter /counter;} Fig. 6. Calculating error rates. SD(A, X) is the standard deviation of AP scores of system A measured on the topic Set X.

In our experiment, we first fix the topic set size, and then compute the variabilities in effectiveness of a pair of systems, A and B. Let us assume that System A is less volatile than System B based on this measurement. We then estimate the probability of a changed decision, i.e. finding System B to be less volatile than System A. We estimate this probability by comparing the two systems across several trials that use different topic sets and then counting how many times the preference decision changes. Finally, to estimate the average probability of changing a decision, we repeat the process on different pairs of systems. This average probability (across systems) is called the error rate. The whole process is repeated for different sizes of topic sets. The algorithm for computing the error rate is shown in Figure 6. It is based on the algorithm described in [12]. In our experiment, we compute the error rate for all the pairs regardless of their absolute differences. We run 50 different trials using different combinations of topics in the two disjoint topic sets.

80

M. Hosseini et al.

Furthermore, as Sanderson and Zobel [9] suggested, we only consider pairs with statistically significant differences in variability, as measured by Levene’s test with a significance level of 0.05. Once again we use the runs participating in the Robust track of TREC 2004 using topics 351-450 and 601-700 (199 topics), and the runs participating in the Web track (225 topics). In this experiment, only the top 75% of systems (ranked by MAP) are considered to prevent the poorly performing runs from having an effect on our conclusion [12]. Thus, our data collection consists of 135 runs and 4806 pairs of runs. Note that we transform AP scores using the z-score before measuring variability. The resulting error rate is shown in Figure 7. As expected, the curve shows that the error rate decreases as the topic set size increases. The experiment indicates that 90 topics are required to obtain an error rate less than 0.05. With 80 topics the measured error rate is 0.052 and with 90 topics it is 0.038.

Fig. 7. Error rate versus topic set size using two TREC test collections: web track and robust track of TREC 2004

4

Summary and Discussion

The average of effectiveness, measured across a topic set, does not capture all the important aspects of effectiveness and, used alone, may not be an informative measure of a system’s effectiveness. We defined variability in effectiveness as the standard deviation of effectiveness scores measured across a set of topics. We proposed that a mean-variance graph helps demonstrate effectiveness in a two-dimensional space rather than ranking systems based on their average effectiveness. Our investigation revealed that the bounded values of a metric yield a curious phenomenon where values of average around 0.5 are accompanied with higher variances. We attributed this to the fact that the metric values fall within [0, 1]. This bounds the standard deviation of the scores to a semicircle as proven in Appendix A. Hence, retrieval systems with average effectiveness close to each

Measuring the Variability in Effectiveness of a Retrieval System

81

of the two boundaries have smaller variances than those with average away from the boundaries. However, there might be also other reasons. For example, when the distribution is not symmetric, standard deviation cannot explain the dispersion properly. In Figure 4 it was shown that the distributions of AP scores were skewed toward the upper boundary, 1, and was completely asymmetric. We used two transformation methods to deal with this problem and showed how they differentiate systems effectiveness with the same average score. We finally discussed the minimum sample size required to estimate the variability in effectiveness. In our experiments we observed that 90 topics were required to obtain an error rate less than 0.05. This paper only considered standard deviation as the measure of variability while it would be interesting to consider other measures, e.g. interquartile range and median absolute deviation. In addition, there are several ways to transform scores in a more symmetric space. For example, one might consider both logit and z-score transformation together. That is, the AP scores are first transformed by logit to (-∞ , +∞) and then z-score is used to deal with extreme values. We also note that the minimum sample size reported in Section 3.4 was averaged across different pairs of systems. As truly shown by Lin and Hauptmann [7], the minimum sample size varies across pairs of systems, and it depends on the difference between two systems’ average effectiveness scores and corresponding variances. Mean and variability can be used to evaluate retrieval systems. One may define a new metric as a function of both mean and variability. Such a metric helps rank systems’ effectiveness in a one-dimensional space by considering both mean and variability in effectiveness. In addition, by a hypothetical scenario we showed that how a threshold of user satisfaction helps make preference between volatile and stable systems. However, we need to at least deal with two issues here. Firstly, in order to measure users’ satisfaction we need to evaluate systems from users’ perspective, i.e. directly asking users to express the amount of satisfaction. Such a user-oriented evaluation method provides accurate results but it is extremely expensive and difficult to do correctly. We can also model users’ satisfaction by using implicit feedbacks of users, e.g. click-through data in a search engine query log. This method is less expensive but inaccurate. Secondly, users’ satisfaction threshold may vary across queries. Indeed, the scenario described in Section 1 was simplified by considering the threshold as a constant value. However, in practice, the threshold varies across queries since it is highly depended on users’ information needs and their expectation of the result set. We will consider these issues for future work.

Acknowledgements The authors thank Jun Wang and Jianhan Zhu of UCL and Stephan Robertson of Microsoft Research Cambridge for useful discussion on earlier drafts of this paper.

82

M. Hosseini et al.

References [1] Collins-Thompson, K.: Robust word similarity estimation using perturbation kernels. In: Azzopardi, L., Kazai, G., Robertson, S., R¨ uger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 265–272. Springer, Heidelberg (2009) [2] Collins-Thompson, K., Callan, J.: Estimation and use of uncertainty in pseudorelevance feedback. In: SIGIR 2007: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 303–310. ACM, New York (2007) [3] Cormack, G.V., Lynam, T.R.: Statistical precision of information retrieval evaluation. In: SIGIR 2006: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 533–540. ACM, New York (2006) [4] Hull, D.: Using statistical testing in the evaluation of retrieval experiments. In: SIGIR 1993: Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 329–338. ACM, New York (1993) [5] Lee, C.T., Vinay, V., Mendes Rodrigues, E., Kazai, G., Milic-Frayling, N., Ignjatovic, A.: Measuring system performance and topic discernment using generalized adaptive-weight mean. In: CIKM 2009: Proceeding of the 18th ACM conference on Information and knowledge management, pp. 2033–2036. ACM, New York (2009) [6] Levene, H.: Robust test for equality of variances. Contributions to Probability and Statistics: Essays in Honor of Harold Hotteling, 278–292 (1960) [7] Lin, W.-H., Hauptmann, A.: Revisiting the effect of topic set size on retrieval error. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 637–638. ACM, New York (2005) [8] Sakai, T.: Evaluating evaluation metrics based on the bootstrap. In: SIGIR 2006: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 525–532. ACM, New York (2006) [9] Sanderson, M., Zobel, J.: Information retrieval system evaluation: effort, sensitivity, and reliability. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 162–169. ACM, New York (2005) [10] Smucker, M.D., Allan, J., Carterette, B.: A comparison of statistical significance tests for information retrieval evaluation. In: CIKM 2007: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, pp. 623–632. ACM, New York (2007) [11] Voorhees, E.M.: The trec robust retrieval track. SIGIR Forum 39(1), 11–20 (2005) [12] Voorhees, E.M., Buckley, C.: The effect of topic set size on retrieval experiment error. In: SIGIR 2002: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 316–323. ACM Press, New York (2002) [13] Webber, W., Moffat, A., Zobel, J.: Score standardization for inter-collection comparison of retrieval systems. In: SIGIR 2008: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 51–58. ACM, New York (2008)

Measuring the Variability in Effectiveness of a Retrieval System

83

Appendix A Lemma: For all data sets like X = {x1 , x2 , ..., xN } where 0 ≤ xi ≤ 1, the ¯ Sx ), are confined within a corresponding mean-standard deviation values, (X, semicircle with center (0.5, 0) and radius r=0.5: ¯ − 1 )2 + S 2 ≤ ( 1 )2 ; (X x 2 2 2 2 ¯ +S ≤X ¯ X x

(4)

Proof: with reference to the mean and variance: N  ¯ = 1 X xi N i=1

Sx2 =

(5)

N N N N    1  ¯ 2= 1 ¯ 1 ¯2 = 1 ¯2 (xi − X) x2i − 2 × X( xi ) + X x2 − X N i=1 N i=1 N i=1 N i=1 i

therefore:

N  ¯ 2 + Sx2 = 1 X x2 N i=1 i

(6)

x2i ≤ xi because 0 ≤ xi ≤ 1; therefore: N N 1  2 1  ¯ x ≤ xi = X N i=1 i N i=1

considering (6) and (7) together: N N  1  ¯ 2 + Sx2 = 1 ¯ X x2i ≤ xi = X N i=1 N i=1

then we reach to (4):

¯ 2 + Sx2 ≤ X. ¯ X

Fig. 8. The upper bound of standard deviation for scores bounded in 0 to 1

(7)

An Information Retrieval Model Based on Discrete Fourier Transform Alberto Costa1 and Massimo Melucci2 ´ LIX, Ecole Polytechnique, F-91128 Palaiseau, France [email protected] Department of Information Engineering, University of Padua, I-35131 Padova, Italy [email protected] 1

2

Abstract. Information Retrieval (IR) systems combine a variety of techniques stemming from logical, vector-space and probabilistic models. This variety of combinations has produced a significant increase in retrieval effectiveness since early 1990s. Nevertheless, the quest for new frameworks has not been less intense than the research in the optimization and experimentation of the most common retrieval models. This paper presents a new framework based on Discrete Fourier Transform (DFT) for IR. Basically, this model represents a query term as a sine curve and a query is the sum of sine curves, thus it acquires an elegant and sound mathematical form. The sinusoidal representation of the query is transformed from the time domain to the frequency domain through DFT. The result of the DFT is a spectrum. Each document of the collection corresponds to a set of filters and the retrieval operation corresponds to filtering the spectrum – for each document the spectrum is filtered and the result is a power. Hence, the documents are ranked by the power of the spectrum such that the more the document decreases the power of the spectrum, the higher the rank of the document. This paper is mainly theoretical and the retrieval algorithm is reported to suggest the feasibility of the proposed model. Some small-scale experiments carried out for testing the effectiveness of the algorithm indicate a performance comparable to the state-ofthe-art. Keywords: Discrete Fourier Trasform, Digital Filters, Information Retrieval Models.

1

Introduction

Different views of Information Retrieval (IR) have been proposed and implemented in a series of models in the last four decades, the most famous being the logical models, the vector space models, and the probabilistic models. The variety of combinations of techniques originated from these models produced the significant increase in retrieval effectiveness observed within the Text Retrieval Conference (TREC) initiative [1]. However, the current retrieval technology is far from being optimal within every domain, medium, language or context. For this reason, in addition to the numerous attempts to optimize and experiment the H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 84–99, 2010. c Springer-Verlag Berlin Heidelberg 2010 

An Information Retrieval Model Based on Discrete Fourier Transform

85

(a) Spectrum for the query of the example. (b) Detail of the spectrum near point 200. Fig. 1. Spectrum for a query (1a) and a detail (1b)

current retrieval models for different media, languages and contexts, the quest for new theoretical frameworks has been intense. [2, 3, 4, 5] This paper illustrates a new framework which is based on Discrete Fourier Transform (DFT) and is called Least Spectral Power Ranking (LSPR). The LSPR framework provides the conceptual devices for designing and implementing an innovative class of retrieval systems. An intuitive description of the framework is provided in the remainder of this section while the other sections of the paper explain LSPR in detail. Basically in the LSPR model the query is viewed as a spectrum and each document as a set of filters, with one filter for each document term. Each term of the query is associated with a discrete time sinusoidal signal Ai sin (2πfi nT ), where Ai is the weight of the term i in the collection (e.g. IDF) and fi is the signal frequency (not related with the frequency of a term in a document); an example is depicted in Figure 1a. How the signal frequency is chosen for each term is explained in Section 4.1. Thus, the query becomes a sum of sinusoidal signals for which the DFT can be computed. After the DFT module of the query has been computed, the spectrum looks like a curve which has peaks of amplitude proportional to Ai at the frequency fi and the values of the curve monotonically decrease in the neighborhood of fi as depicted in Figure 1b. The behaviour of the spectrum is crucial because the utilization of a sinusoidal signal combined with the variation in the amplitude of the spectrum causes variations in retrieval effectiveness (see Section 4.1 for more details) – the latter phenomenon has been observed in the experiments carried out during the research reported in this paper. In general, the DFT samples are complex numbers; computing the DFT module means substituting each complex sample with its module. To obtain the ranked document list, the spectrum of the query is filtered by the documents, i.e. the filters. As a matter of fact, each document is represented as a set of filters, each of them is similar to the one in Figure 2: the breadth depends on the weight (e.g. TFIDF or BM25) of the term, while the position of the filter is set so that if a query has a term qi , associated with a peak in the frequency fi in the spectrum, and in the document there is qi , the corresponding

86

A. Costa and M. Melucci

Fig. 2. Example of frequency response for the used filter

(a) A fragment of a spectrum with a filter.

(b) The obtained filtered spectrum.

Fig. 3. Representation of the filtering operation

filter has ZL and ZR near fi (as depicted in Figure 3a). The filtered spectrum is depicted in Figure 3b. The documents are then ranked by the power of the spectrum obtained as the sum of the components of the filtered spectrum. If a document is associated with a low power, it is considered highly relevant by the system, hence, the documents are ordered by increasing power. This is why the model is called Least Spectral Power Ranking (LSPR). The paper is organized as follows. In Section 2 an overview of the main frameworks of the current IR models is presented. Section 3 presents the theory and the methods of DFT and digital filters, which is required to understand the rest of the paper, especially for those unfamilar with digital signal processing. Section 4 introduces LSPR. After that, Section 5 shows how LSPR works with an example (three documents and a query), while Section 6 suggests some future research directions.

2

Related Work

One of the quests in IR is whether a theory exists or not. The question is not without merit because if the answer were affirmative this discipline would have a solid framework in which the methodologies and experiments could be developed with the same efficacy as that of the models and experiments used in Physics to interpret the macro- or microscopic world. A model is the methodological

An Information Retrieval Model Based on Discrete Fourier Transform

87

instrument for formulating theories about IR. Currently, IR systems combine a variety of techniques stemming from logical, vector-space and probabilistic models. It is this variety of combinations investigated over the last four decades that has produced a significant increase in retrieval effectiveness in the last almost twenty years as reported in [1]. The models proposed in the literature aim at capturing some of the variables which affect the retrieval process. Although they are somehow complex, these models are however quite oversimplified because they deliberately ignore many other variables, especially those related to the user and to the context in which the end users operate. What the models proposed in the literature have in common is the use of a “metaphor” for describing them in a simplified manner, thus making their experimentation and dissemination possible. The classical Boolean (or logical) model refers to set theory which views terms as document sets (operands) and queries as set operations [6] – non-classical logical models have been investigated by van Rijsbergen since 1980s [7]. The vectorspace models view terms as basis vectors, documents and queries as vectors of the same space as conceived in the 1960s by Salton [8, 9] before being developed into Latent Semantic Indexing [10] which computes alternative basis vectors for expressing complex terms. The probabilistic models for IR refer to the theories of probability (there are many [11]) which view terms or documents as events (or random variables). There have been many contributions to the development of the probabilistic models for IR, to cite but a few [12, 13, 14, 15, 3, 5]. Specific classes of probabilistic models refer to the Bayesian networks [16] and more recently to statistical language models [17, 3] inspired by speech recognition. The recent book by van Rijsbergen attempts to unify the logical, vector-space and probabilistic models into a single framework – the one used to describe quantum mechanics [4] – which views documents and terms as vectors, retrieval operations as projectors and probability as a trace-based function. The LSPR framework proposed in this paper differs from those proposed in the past because the underlying mathematical basis is different, yet there are some points in common. In our opinion, the main difference is the view through which the terms or in general the information content of the documents and of the queries are represented. In LSPR, the terms are sine curves which are a mathematical representation of signals. As these curves are functions, they can be treated algebraically. This property allows us to express queries and documents as functions which can be transformed for capturing salient features. In this paper, the DFT is the transform investigated. The LSPR framework employs the mathematical constructs proposed in [18, 19] which report a complex framework for document retrieval. In particular, [19] addresses the problem of matching queries and documents in which query terms occur at different positions and then with a variety of patterns. As matching documents by considering all the possible patterns is a difficult task, the authors propose to transform the spatial view of the query term occurrences into another domain. Therefore, each term of each document is represented as a signal and each document is divided into components. The term signals are transformed

88

A. Costa and M. Melucci

into term spectra which are a function of the magnitude and of the phase of the term occurring within each component of each document (magnitude and phase are then indexed by term, document and component). Document scores can be obtained by combining the term spectra components across terms under the assumption that a relevant document would have a high occurrence of query terms (implying a high magnitude of components) and a similar position of each pattern of query terms (implying similar phase). Unfortunately, this framework did not produce significant improvements over the standard TFIDF-based vectorspace model. What makes LSPR different from those models is that the spectrum and filtering are computed on the query and that the documents are the filters. Moreover, LSPR models the query terms as signals independently of the problem addressed in [19], the latter being a specific problem of text retrieval. Indeed, LSPR aims at making the sinusoidal signals a modeling paradigm (as the vector spaces, for example, are the paradigm of the vector-space model) rather than the approach of addressing the problem addressed in [19].

3

Background

3.1

Discrete Fourier Transform

The Fourier transform is essential in digital signal processing, because it allows us to analyze signals in the frequency domain in a more effective and efficient way than the time domain analysis [20, 21]. For continuous time signals the Fourier transform is employed, but if the signals are periodic the Fourier transform can be replaced with the Fourier series. In the discrete time domain, there is something similar and the transform related to discrete time periodic signals is the Discrete Fourier Transform (DFT). DFT is the only transform that can be computed numerically. Our interest in DFT derives from the need to analyze periodic sinusoidal discrete signals in the frequency domain. To this end the Fast Fourier Transform (FFT) is an O (N log N ) algorithm [22, 23]. This approach is a “divideand-conquer” algorithm and was conceived by Cooley and Tuckey in 1965.1 In the following, the DFT is more formally introduced. Let x (nT ) be a discrete time signal sample where T represents the distance between two samples. The DFT of this signal is ˜ (k) = T X

N −1 

x (nT ) e−i2πkF nT

F =

n=0

Fc N

(1)

√ where i = −1, Fc is called sampling frequency, N is the number of samples and k identifies the k-th sample of the DFT. In the remainder of this paper, Fc = 1/T , therefore F = N1T . In this way, Eq. 1 is ˜ (k) = T X

N−1 

x (nT ) e−i

2πn N k

.

n=0

1

Actually, the first who spoke of this algorithm was Gauss, in 1805.

(2)

An Information Retrieval Model Based on Discrete Fourier Transform

89

In order to obtain the power of the spectrum, the module of Equation 2 has to be computed from the complex numbers of the DFT. When x(nT ) is the sampling of a real periodic function, the module of the DFT is symmetric with respect to N/2, hence, we only need to operate with half of the spectrum. 3.2

Spectral Leakage

As stated earlier, the signals used in LSRP are supposed to be periodic, sinusoidal and discrete. The general form for these signals is Ai sin (2πfi nT ) .

(3)

The module of the DFT for the signal (3) has a peak proportional to amplitude Ai at the frequency fi (see Figure 4a), but if the frequency fi of the signal is not a multiple of F , a distorted DFT is obtained; Figure 4b shows this phenomenon, which is called spectral leakage. Usually, in digital signal processing spectral leakage is avoided because it makes the recognition of the correct frequencies of the signals difficult. In contrast in the LSPR model, spectral leakage is kept because the presence of some components with reduced amplitude at the frequencies close to fi makes it possible to better estimate the relevance of a document – the reason will be more clear in Section 4.1 – and introduces some constraints in the choice of F and fi for the signals (3).

(a) DFT of sinusoidal signal with fi (i = 0) multiple of F .

(b) DFT of sinusoidal signal with fi (i = 0) not multiple of F . Fig. 4.

90

A. Costa and M. Melucci

One of the most important theorems in digital signal processing is the sampling theorem (or Whittaker-Nyquist-Kotelnikov-Shannon theorem). There are a lot of equivalent definitions for this theorem. In this paper, the following is used: Theorem 1 (Sampling theorem [20]). Let x(t) be a signal with no frequencies higher than fM Hertz. Then x(t) is uniquely determined by its samples x(nT ) if Fc ≥ 2fM . This theorem is important in the framework proposed in this paper because it gives some conditions for the choice of the frequencies fi of the sinusoidal signals (3), and for the choice of the parameters of DFT. This will be explained better in Section 4. 3.3

Digital Filtering

One of the most important subjects in digital signal processing is digital filter design. A digital filter is a component that transforms an input signal x(nT ) to an output signal y(nT ) through the convolution of the input by the filter. Convolution is quite difficult to compute if the Fourier transform is not used. In contrast, after DFT has been applied to the signals and the filter, the computa˜ ˜ tion is feasible. Let X(k), Y˜ (k) and H(k) be, respectively, the DFT of the input signal, the DFT of the output signal and the DFT of the filter; the relationship among these transforms can be written as: ˜ X(k). ˜ Y˜ (k) = H(k)

(4)

This equation performs filtering in the LSPR model with a simple multiplication between two functions. There are a lot of filters which could be used in this context. In this paper, we used something similar to a notch filter which eliminates the spectrum near

Fig. 5. Module of the transform for a notch filter with cut-off frequency of 10 kHz

An Information Retrieval Model Based on Discrete Fourier Transform

91

a predefined frequency (called cut-off frequency), and keeps the spectrum in corrispondence with the other frequencies unchanged. The design of a filter appropriate for IR is still under investigation. Figure 5 shows an example of the module for a notch filter. Actually, in our tests we used a simplified filter, that is, a filter with a triangular form. This is better explained in Section 4.

4

Least Spectral Power Ranking

In this section Least Spectral Power Ranking (LSPR) which implements the DFT-based framework is introduced. LSPR consists of three main steps: 1. transformation of a query into a spectrum, 2. transformation of a document into a notch filter set, 3. document ranking. Before these three main steps some preprocessing is performed. The preprocessing phase includes all the operations usually done by every IR system, such as stemming and stopwords removal. In this first implementation of the model, we used the well-known TF-IDF weighting scheme for the documents in the collection. Actually, we used the normalized version of TF-IDF: if we call wij the TF-IDF weight for the term i in the document j, the normalized version is wij w ˆ ij =  . 2 i wij

(5)

This expression has been used to implement the strength of the filter. The IDF weighting scheme has been used for implementing the amplitude of the peaks in the spectrum. Other weighting schemes can be used as well. 4.1

Query to Spectrum Transformation

The first step is to set the parameters of the DFT (such as F and N ) and of the sinusoidal signals. We choose to set F = 2 Hz, and to set every frequency of the signals (3) as an odd number: in this way it is not possible for a frequency to be an integer multiple of F , and this guarantees the spectral leakage. This choice allows us to write equation (3) in another way. Recalling that T = N1F :   πfi n Ai sin (2πfi nT ) = Ai sin . (6) N Before explaining how we choose the frequency of each signal, the parameter N has to be set. It is important to remember that the spectrum is associated with the query, and that for each query term there is a sinusoidal signal. In this way the query is a sum of sinusoidal signals. The DFT of this sum is computed using N samples (N is both the number of samples of the sinusoidal signals used as input for the DFT and the number of points of the spectrum). For each term of

92

A. Costa and M. Melucci

the query there are 300 points of the spectrum and the peak of the relative signal is near point 200. These values are chosen because some experiments showed that the amplitude of the spectrum was less than 1% of the peak value 100 points after and before the peak. The use of 300 points assures that each sinusoidal signal is related to a disjoint interval of the spectrum; for example, if the query has 2 terms, there are 2 × 300 = 600 points. The number of points is the same for each term. Methods for choosing the number of points depending on the terms will be studied in the future. The FFT algorithm requires that this value is rounded at the closest power of two, and that this value has to be doubled due to the symmetry of the DFT (this last operation also has the advantage of fulfilling the conditions of Theorem 1); for example, 600 is rounded to 1024 and finally N = 2048 = 1024 × 2. Fc = N F = 2048 × 2 = 4096 Hz, where the maximum frequency fM among the sinusoidal signals is 1001 Hz, as shown below, so the condition of Theorem 1 (Fc ≥ 2fM ) is verified. In order to have the peak of the sinusoidal signal near to point 200 of the relative 300-point interval and to guarantee spectral leakage, its frequency is set to (300 · (i − 1) + 200) F + 1 Hz (7) where i refers to the i-th term of the query. Finally, the amplitude of the sinusoidal signal is the IDF weight of the term. Now we can show with an example how this method works. Suppose there is a query with 2 terms; for the sake of clarity, suppose also that the amplitudes of the two original terms are equal, and we call this value A. When N = 2048, the frequencies of the 2 signals are, respectively, 401 Hz and 1001 Hz using formula (7). The input for the FFT algorithm is a N -points vector S, and it is calculated as     401πn 1001πn S [n] = A sin + A sin , n = 1, 2 . . . , N. (8) 2048 2048 The FFT algorithm returns the DFT of this vector and the module is something similar to Figure 1a. In this figure we can see that the spectrum is symmetric with respect to N/2 (point 1024, which corresponds to 2048 Hz), and the peaks are near the points 200 and 500. Figure 1b shows more clearly the spectrum near point 200. Another important aspect showed by Figure 1b is the side effect of the spectral leakage. The decrease in importance follows the decrease in amplitude of the spectrum: this is an important hypothesis, and the efficacy of the presented model is based on it. Without spectral leakage, the figure would show a single peak at point 200 and the function would be zero at the other points, removing the decrease in the amplitude and making it impossible to estimate the decrease in importance, thus making futile the effect of the filtering operations. 4.2

Document to Filter Transformation

Each document is transformed into a filter set (one for each document term). As stated before, in LSPR, each filter is similar to a “notch” with triangular form.

An Information Retrieval Model Based on Discrete Fourier Transform

93

To be precise, there are two points, ZL (Zero Left ) and ZR (Zero Right ), where the module is 0, and there is a parameter called breadth such that the value of the filter before ZL−breadth and after ZR+breadth is 1. Finally, to connect ZL (ZR) with ZL−breadth (ZR+breadth) points, there is a linear function. An example of this filter is represented in Figure 2, where Figure 3 shows a fragment of a spectrum with a filter, and the obtained filtered spectrum. Two issues needs some explainations: the selection of the breadth (which represents the strength of the filter) and the position of the filter. The former is the weight TF-IDF of the document term normalized by the document length (w ˆ ij ), multiplied by a constant called selectivity. The value of this constant was set to 24, because some experiments showed that for values greater or smaller than 24, the MAP decreased. Probably, this value depends on the collection, but the interesting fact is that 24 is a sort of maximum for a function that relates selectivity and MAP. This reminds us of BM25 thus suggesting some possible future research directions. The latter is computed as follow: If the term was occurring in the query, ZL and ZR were points 200 and 201 of the relative interval; this means that for example if this term is the second of the query, ZL=500 and ZR=501, while ZL=200 and ZR=201 for the first query term, as in Figure 2. Otherwise, the filter is not created. 4.3

Document Ranking

Suppose a query is given as input to the system by the user and consider the set of documents retrieved by the system; they are reranked by a score computed after applying Equation (4). As a matter of fact, the module obtained by the query-tospectrum transformation is filtered by the filter set obtained after the documentto-filter transformation phase. Finally, the score obtained by each document is the sum of the module of the spectrum (the “power”) after filtering, hence, every document is associated with a score. The documents are ordered by increasing power because the more the filters decrease the power of the spectrum, the more the system consideres the document and the query related. Figure 6 summarizes LSPR. In this algorithm, the notation X ← [] represents the initialization of a vector, while X ← [X, i] is used to add a new element i to the vector X. Basically, the rows from 1 to 4 represent the initialization phase; the rows from 5 to 9 set the values of ZL for the filters (as explained in Section 4.2), and the rows from 10 to 22 describe the filtering operation. At the end, the power is computed and the algorithm returns the ranking list. The Filter function used within the algorithm has as inputs the spectrum, the ZL point of the filter, the breadth of the filter and the index i of the term of the query, whereas it returns as output the filtered spectrum. Basically, this function does the multiplications of the module of the spectrum (e.g. that in Figure 1a) by the frequency response of the filter (e.g. that in Figure 2), as shown in Figure 3.

94

A. Costa and M. Melucci

Algorithm: LSPR Input: query Q, collection of documents Output: ranked document list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Compute the vector S, as described in Section 4.1 spectrum ← |DFT(S )| I ← [] (initialize I ) ZL ← [] (initialize ZL) for each term t of the query Q do Retrieve the posting list of t and call it L(t) I ← [I , index of t − 1] ZL ← [ZL, 300 · (index of t − 1) + 200] for each X ∈ L(t) do if it is the first time that document X is selected then f spectrum ← spectrum end if for i = 1 to |I | do breadth ← Round(selectivity · weight of qi in X ) f spectrum ← Filter(f spectrum, ZL[i], breadth, I [i] − 1) end for |f spectrum|

21

power [X ] ←



f spectrum [k]

k=1

22 23 24 25

end for end for Order the documents by increasing power and save the list in result return result Fig. 6. An algorithm to summarize LSPR

4.4

Some Preliminary Results

LSPR was tested by using the CACM test collection and compared with two baselines: – A basic vector-space model. – The Divergence From Randomness (DFR) model implemented by Desktop Terrier v.1.1. After stopword removal and stemming, the retrieved list were measured by trec eval. The Mean Average Precision (MAP) of the vector-space model was 0.242, while the MAP of DFR was 0.329. The MAP of LSPR was 0.348, thus indicating a performance comparable to the state-of-the-art.

An Information Retrieval Model Based on Discrete Fourier Transform

5

95

Example

To give a global idea about how LSPR works, in this section we present an example with a collection of three documents and a query. Suppose we have the following documents in the collection (for the sake of clarity, we consider them after stopword removal, and without stemming): – D1 : (retrieval, data, author, book); – D2 : (information, computer, system, storage, information, data); – D3 : (information, retrieval, system, relevance, MAP, precision, recall, relevance). Suppose also that the query is Q : (information, retrieval, relevance). Table 1 represents the index of the collection, with the normalized TF-IDF weighting schema, and the query with the IDF weighting schema. Table 1. Normalized TF-IDF index matrix and IDF weights for query

author book computer data information MAP precision recall relevance retrieval storage system

D1 0.663 0.663 0 0.245 0 0 0 0 0 0.245 0 0

D2 0 0 0.596 0.220 0.440 0 0 0 0 0 0.596 0.220

D3 Q 0 0 0 0 0 0 0 0 0.136 0.585 0.367 0 0.367 0 0.367 0 0.735 1.585 0.136 0.585 0 0 0.136 0

At this point, we show how LSPR creates the spectrum of the query, as described in section 4.1. First, the frequencies of the terms “information”, “retrieval” and “relevance” are set respectively to 401 Hz, 1001 Hz and 1601 Hz, while N is 2048. Now, the vector S is computed as       401πn 1001πn 1601πn S [n] = 0.585 sin + 0.585 sin + 1.585 sin (9) 2048 2048 2048 where n = 1, 2 . . . , N . The vector S is given as input for the FFT algorithm, and we obtain the spectrum shown in Figure 7 (actually, due to the symmetry, we consider only half of the spectrum). Now, the documents are transformed into filters, as explained in section 4.2. Since D1 contains only the term “retrieval” of the query, with weight 0.245, it is transofmed into a filter with ZL=500 (which corresponds to the peak of the term

96

A. Costa and M. Melucci

Fig. 7. Spectrum for the query

(a) D1

(b) D2

(c) D3

Fig. 8. Spectra after the filtering operations performed on the three documents

“retrieval” in the spectrum) and with breadth=Round(24 · 0.245)= 6. Similarly, D2 corresponds to a filter (“information”) with ZL=200 and breadth=Round(24· 0.440)=11. Finally, D3 is transformed into a set of three filters; the first one (“information”) has ZL=200 and breadth=Round(24 · 0.136)=3, the second one (“retrieval”) has ZL=500 and breadth=Round(24 · 0.136)=3, and the third one has ZL=800 and breadth=Round(24 · 0.735)=18. At the end, after the filtering operations on the spectrum represented in Figure 7, we obtain the filtered spectra of Figure 8. The power (sum of the components of the spectrum) of the spectrum of Figure 7 is 13007.091, while the power of the spectra of Figure 8 are respectively 11836.613, 11649.498 and 6919.414; thus, according to the ranking rule of LSPR, the first document retrieved is D3, the second is D2 and the third is D1.

6

Conclusions and Future Work

This paper describes an IR model, called LSPR, that uses DFT. The paper is mainly theoretical because the focus of the paper was on the model whereas

An Information Retrieval Model Based on Discrete Fourier Transform

97

future work will focus on large scale experimentation. Nevertheless, some experiments were performed on a small scale. Preliminary results show that this model works well, and its efficacy is comparable with the efficacy of the state-of-the-art. LSPR was a first implementation of the framework based on DFT and a variety of implementations can be obtained through the series of parameters illustrated in the paper. It is our opinion that there is large room for improvement by setting appropriate values for sample sizes, thresholds and frequencies. At first glance, such an approach may be difficult, but the sound mathematical framework of DFT may provide useful guidance and significant improvements. Future work has three main objectives. First, efficiency needs to be improved. In this first implementation, since the effectiveness evaluation of the model was our focus, efficiency was not in the priority list. In order to test the model with larger collections these aspects are very important, hence, an optimization of the algorithm is necessary. Second, we have to look for better configuration than those reported in this paper. To obtain this, we will tune the parameters of the model. For example, in this implementation we use the TF-IDF weighting scheme (and IDF for the query), but other schemes such as Okapi BM25 can be tested. Another important issue is the similarity between the terms: using some clustering algorithm [24], we can estimate how two terms are related; these informations can be used to increase the efficacy of the model, for example by associating other different filters with the similar terms. In addition, the internal parameter of LSPR, such as the number of points of DFT and the selectivity will be investigated. Finally, the other feature of LSPR is that the functions are defined over the complex field, the latter being a characteristic of Geometry of IR by van Rijsbergen [4] in which no restrictions were placed on the scalars. The use of the complex field is a question of representation and many operations in that framework are always real (thus permitting ranking). The fact that scalars are complex and the descriptors are represented as sine curves, i.e. signals, helps enlarge the research in IR to other media than text by leveraging the extra representation power given by complex numbers. Moreover, the functions used in the Fourier transform are mutually orthogonal. This property recalls the techniques developed within the vector space models (e.g. Latent Semantic Analysis) and the framework developed by [4]. Further investigation will then be carried out on these connections. If the model leads to good results with larger experimental collections, we can test it in other contexts, such as web information retrieval. Actually, we are working on a modified version of LSPR for the recommender systems.

Acknowledgements We would like to thank Alberto Caccin, Albijon Hoxaj, Marco Lonardi, Enrico Martinelli and Giuseppe Soldo for the tests with Terrier. Moreover, we want to thank Emanuele Di Buccio for his precious suggestions. One of the authors (A. C.) is grateful to Digiteo Project 2009-55D “ARM” for financial support.

98

A. Costa and M. Melucci

References [1] Voorhees, E., Harman, D. (eds.): TREC: Experiment and Evaluation in Information Retrieval. The MIT Press, Cambridge (2005) [2] Robertson, S.: Salton award lecture: On theoretical argument in information retrieval. SIGIR Forum 34(1), 1–10 (2000) [3] Croft, W., Lafferty, J. (eds.): Language Modeling for Information Retrieval. Springer, Heidelberg (2003) [4] van Rijsbergen, C.: The Geometry of Information Retrieval. Cambridge University Press, UK (2004) [5] Fuhr, N.: A probability ranking principle for interactive information retrieval. Journal of Information Retrieval 11(3), 251–265 (2008) [6] Cooper, W.: Getting beyond Boole. Information Processing & Management 24, 243–248 (1988) [7] van Rijsbergen, C.: A non-classical logic for Information Retrieval. The Computer Journal 29(6), 481–485 (1986) [8] Salton, G.: Automatic information organization and retrieval. Mc Graw Hill, New York (1968) [9] Salton, G.: Mathematics and information retrieval. Journal of Documentation 35(1), 1–29 (1979) [10] Deerwester, S., Dumais, S., Furnas, G., Landauer, T., Harshman, R.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990) [11] Fine, T.: Theories of probability. Academic Press, London (1973) [12] Maron, M., Kuhns, J.: On relevance, probabilistic indexing and retrieval. Journal of the ACM 7, 216–244 (1960) [13] Robertson, S., Sparck Jones, K.: Relevance weighting of search terms. Journal of the American Society for Information Science 27, 129–146 (1976) [14] Robertson, S.: The probability ranking principle in information retrieval. Journal of Documentation 33(4), 294–304 (1977) [15] Robertson, S., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), Dublin, Ireland, pp. 232–241 (1994) [16] Turtle, H., Croft, W.: Inference networks for document Retrieval. In: Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), Brussels, Belgium (September 1990) [17] Ponte, J., Croft, W.: A language modeling approach to information retrieval. In: Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR), Melbourne, Australia, pp. 275–281. ACM Press, New York (1998) [18] Park, L.A.F., Ramamohanarao, K., Palaniswami, M.: Fourier domain scoring: A novel document ranking method. IEEE Trans. on Knowl. and Data Eng. 16(5), 529–539 (2004) [19] Park, L.A.F., Ramamohanarao, K., Palaniswami, M.: A novel document retrieval method using the discrete wavelet transform. ACM Trans. Inf. Syst. 23(3), 267– 298 (2005) [20] Oppenheim, A.V., Willsky, A.S., Nawab, S.H.: Signals & systems, 2nd edn. Prentice-Hall, Inc., Upper Saddle River (1996)

An Information Retrieval Model Based on Discrete Fourier Transform

99

[21] Mitra, S.K.: Digital Signal Processing: A Computer-Based Approach, 3rd edn. McGraw-Hill, New York (2006) [22] Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 2nd edn. MIT Press/McGraw-Hill Book Company (2000) [23] Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical recipes in C: The art of scientific computing, 2nd edn. Cambridge University Press, Cambridge (1992) [24] Croft, B., Metzler, D., Strohman, T.: Search Engines: Information Retrieval in Practice, 1st edn. Addison Wesley, Reading (2009)

Logic-Based Retrieval: Technology for Content-Oriented and Analytical Querying of Patent Data Iraklis Angelos Klampanos1, Hengzhi Wu2 , Thomas Roelleke2 , and Hany Azzam2 1

University of Glasgow, UK [email protected] 2 Queen Mary University of London, UK {hzwoo,thor,hany}@dcs.qmul.ac.uk

Abstract. Patent searching is a complex retrieval task. An initial document search is only the starting point of a chain of searches and decisions that need to be made by patent searchers. Keyword-based retrieval is adequate for document searching, but it is not suitable for modelling comprehensive retrieval strategies. DB-like and logical approaches are the state-of-the-art techniques to model strategies, reasoning and decision making. In this paper we present the application of logical retrieval to patent searching. The two grand challenges are expressiveness and scalability, where high degree of expressiveness usually means a loss in scalability. In this paper we report how to maintain scalability while offering the expressiveness of logical retrieval required for solving patent search tasks. We present logical retrieval background, and how to model data-source selection and results’ fusion. Moreover, we demonstrate the modelling of a retrieval strategy, a technique by which patent professionals are able to express, store and exchange their strategies and rationales when searching patents or when making decisions. An overview of the architecture and technical details complement the paper, while the evaluation reports preliminary results on how query processing times can be guaranteed, and how quality is affected by trading off responsiveness.

1 Introduction Patent retrieval has emerged as an important application of information retrieval (IR). Especially, patent searching is likely to be more and more popular as standardised patent data corpora (e.g. MAtrixware REsearch Collection (MAREC)1 ) become available for research communities. In this paper, we discuss a logic-based solution for querying patent data. We maintain that logic-based retrieval can be particularly useful for patent searching due to its abstraction and descriptive power, its capability to be used in a traceable search workflow and the fact that it can be used through query languages similar to the ones used by IP professionals. Additionally, the use of probabilistic logics allows for probabilistic reasoning about entities described by the underlying data, in a dynamic manner. 1

http://www.ir-facility.org/services/data/matrixware-research-collection

H. Cunningham, A. Hanbury, and S. R¨uger (Eds.): IRFC 2010, LNCS 6107, pp. 100–119, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Logic-Based Retrieval

101

In particular, by the term “logical-based retrieval” we refer to the following properties: 1. The ranking of objects is based on the probability that the retrieved objects (for instance documents, inventors, etc.) are based on the probability of the objects being implied by the query. 2. The reasoning about objects, that is the retrieval of data sources, leads to a ranking of sources and this ranking affects the ranking of documents, the ranking of documents leads to a ranking of inventors and/or assignees, and so forth. 3. Logical retrieval allows for open-box, high-level abstraction modelling, in contrast to black-box IR systems where the retrieval function is an integral part of the system (library function). The properties of logical retrieval meet the requirements of patent searching in the sense that patent searching is a complex retrieval task that requires reasoning about objects. Furthermore, it also requires modelling IR in such a way that users (patent searchers) can understand and modify the ranking and reasoning process. From private discussions with patent searchers it is understood that them being in full control of the retrieval process is essential for generating the trust needed to move on from Boolean retrieval. As follows from the above properties, logic-based IR fulfills this requirement. The following example underlines the difference between basic, topical retrieval and complex retrieval. For example, after an initial search for patents about semi-conductors (the initial search typically is a basic topical search), the task is to explore the patents of experienced inventors who filed the most relevant patents about semi-conductors. Logical retrieval supports the reasoning process, that is the modelling of vague predicates such as experienced inventor, the post-processing of retrieval results, and, in general, the combination of evidence from multiple data sources and different dimensions (e.g. content dimension, attribute/field dimensions, semantic annotations). Thus, probabilistic logical retrieval supports the decision-making processes faced by searchers who explore several data sources, compare, combine and re-rank results. However, the flexibility and power of abstraction of logic-based IR comes at the cost of efficiency. Scaling logical retrieval to the data volume of the patent corpus is a challenging task. By combining distributed IR and logical retrieval we can overcome the inherent efficiency problems of logic-based IR and achieve efficient processing of probabilistic logical queries. In distributed information retrieval (DIR) the interface between users and content resourcesis typically provided by a single broker that manages all the transactions between the client (the user) and the servers (the IR engines). This broker is responsible for forwarding queries to the most relevant of the providers as well as fusing and returning result-lists to the user. This requires that the broker has some knowledge over the retrievable content of the participating providers before any query routing can take place. DIR can be described as the set of the following challenges[1]: Resource description is the process during which individual providers or IR engines inform the broker/managing node of their content. Resource selection is the process during which, given a query, the broker makes an informed decision as to which providers may have content that is relevant to the query. These will be the providers that will end-up receiving and responding to the query. Final, results’ merging is the merging of result-lists that

102

I.A. Klampanos et al.

the broker has to perform after having received results from the various, previously selected, providers. The broker has to merge the various result lists into a single one before routing them back to the user. The three problems outlined above occur mainly in a distributed system comprised of autonomous retrieval engines. In a patent-searching scenario we can expect the index distribution to occur in a centrally controlled and computationally efficient environment. In our case we assume the existence of a highly parallel computer infrastructure (for instance IRF’s LDC), however our proposed solution is not restricted to such environments. In distributed logic-based IR, resource selection and fusion are the most critical parts of the solution since they can have a significant impact on retrieval effectiveness and efficiency. In this paper we will show how we model resource selection and fusion in our solution. In the same way as other components of a logical IR system, these two are also pluggable and IP professionals could alter or augment them in order to suit their needs. Again, by supporting the distribution process logically, the IP professionals can here too be in control, being able to drive their work through reasoning about resources and merging strategies in the same manner as they can reason about documents and related patent entities.

2 Motivation and Background 2.1 Logic-Based Retrieval With logic-based retrieval, we refer to the approach of applying a logic-based technology to retrieval tasks. Inherent to logic-based technology is the usage of structured languages to represent knowledge and queries. Therefore, logic-based technology has a higher expressive power than the “bag-of-words” approach of traditional free-textoriented IR. Moreover, logical retrieval support “reasoning”. For example, logical languages allow to express strategic and analytical queries. “Find the patents of Kraft and Cadbury; compare the most important patents; which inventors are the dominant ones in the food provision market.” The application of logical retrieval to large-scale, partly poorly structured data, is challenging for a number of reasons. Because of its expressiveness, logical retrieval demands an algebraic evaluation that is significantly more compex than fetching documnt id’s from the posting list of an inverted list. Also, the modelling of selection, ranking, and fusion strategies in a logical language is challenging. However, the open-box, high-level abstraction is precisely what satisfies the requirements of complex (since advanced, strategic and anlytical) search requirements as they occur, for example, in patent search. The properties of logical retrieval meet the requirements of patent searching in the sense that patent searching is a complex retrieval task that requires reasoning about objects. Furthermore, it also requires modelling IR in such a way that users (patent searchers) can understand and modify the ranking and reasoning process. From discussions with patent searchers this aspect of “I need to understand what is going on” is essential for generating the trust needed to move on from Boolean retrieval, which gives the user a sense of being in “full control” of what is going on.

Logic-Based Retrieval

103

The following example underlines the difference between basic, topical retrieval and complex retrieval. For example, after an initial search for patents about semi-conductors (the initial search is the basic, so-called topical search), the task is to explore the patents of the experienced inventors who filed the most relevant patents about semi-conductors. Logical retrieval supports the reasoning process, that is the modelling of vague predicates such as experienced inventor, the post-processing of retrieval results, and, in general, the combination of evidence from multiple data sources and different dimensions (e.g. content dimension, attribute/field dimensions, semantic annotations). Thus, probabilistic logical retrieval supports the decision-making processes faced by searchers who explore several data sources, compare, combine and re-rank results. Scaling logical retrieval to the data volume size of the patent corpus is challenging. In this paper, we discuss how distributed IR and logical retrieval can be combined to achieve the efficient processing of so-called probabilistic logical programs. 2.2 Probabilistic Datalog Probabilistic Datalog (PD) ([2,3]) is a language for probabilistic reasoning, where deterministic Datalog has its roots in deductive databases. PD was extended in [4,5] to improve its expressiveness and scalability for modelling IR models (ranking functions). In particular, in this paper we use the PD language in order to demonstrate possible implementations for source selection, result fusion, and retrieval strategy composition. Alternative languages, such as probabilistic SQL or probabilistic relational algebra, could have been used instead but PD was preferred due to the clarity of its semantics and compact syntax. To introduce what PD offers, an extract of the syntax of PD is shown in Figure 1. In this syntax, expressions between a pair of curly brackets, i.e. ‘{’ and ‘}’, are optional. A PD rule consists of a head and a body. A head is a goal, and a body is a subgoal list. A PD definition is similar in syntax to a rule, apart from the ‘:=’ indicating the definition; a rule is executed if required (delayed execution), whereas a definition is executed when read by the parser (instant execution). From this point of view, definitions can be views as “materialised views” (as known for DB systems). Specific to the PD shown here (based on [4]) and different from the PD in [6], is the specification of aggregation and estimation assumptions in goals and subgoals. The assumption between predicate name and argument list is the so-called aggregation assumption. For example, for disjoint events, the sum of probabilities is the resulting tuple probability. The assumptions ’DISJOINT’ and ’SUM’ are synonyms, and so are ’INDEPENDENT’ and ’PROD’. The assumption in a conditional is the so-called estimation assumption. For example, for disjoint events, the subgoal “index(Term, Doc) | DISJOINT(Doc)” expresses the conditional probability P(Term|Doc) derived from the statistics in relation “index”. Complex assumptions such as DF (for document frequency) and MAX IDF (max inverse document frequency) can be specified to describe in a convenient way probabilistic parameters commonly used in information retrieval. Expressions with complex assumptions can be decomposed in PD programs with traditional assumptions only. However, for improving the readability and processing (optimisation), complex assumptions can be specified. The decomposition of complex assumptions is shown in [4].

104

I.A. Klampanos et al. pdRule ::= head ’:-’ body pdDef ::= head ’:=’ body head ::= goal body ::= subgoalList subgoalList ::= subgoal {’&’ subgoalList} goal ::= basicGoal | aggGoal | estGoal basicGoal ::= NAME ’(’ argList ’)’ aggGoal ::= NAME aggAssumption ’(’ argList ’)’ | estGoal ::= NAME ’(’ argList ’)’ ’|’ {estAssumption} evidenceKey subgoal ::= goal argList ::= arg {’,’ argList} varList ::= VARIABLE {’,’ varList} var ::= VARIABLE arg ::= VARIABLE | constant constant ::= STRING | NAME | NUMBER evidenceKey ::= ’(’ varList ’)’ tradAssumption ::= ’DISJOINT’ | ’INDEPENDENT’ | ’SUBSUMED’ irAssumption ::= ’DF’ | ’MAX IDF’ | ... aggAssumption ::= tradAssumption | ’SUM’ | ’MAX’ | ... estAssumption ::= tradAssumption | comlexAssumption Fig. 1. Extract of the syntax of probabilistic datalog (PD)

The syntax specification is simplified here to improve readability. For example, for reasons of context-free syntax parsing, constants in goal argument lists need to be quoted, whereas in subgoals, bare constants are sufficient, and quoting of constants is optional. The syntax specification does not make explicit this and other implementational details. A rule is evaluated such that the head is true if and only if the body is true. For example, the following rule demonstrates a common term matching strategy in IR. Queries are represented in relation “qterm(T,Q)” (T is a term, Q is a query ID), and the collection is represented in “term(T,D)” (D is a document id). If T occurs in Q, and T is a term in D, then D is retrieved for Q. 1

coord match retrieve(D, Q) :− qterm(T, Q) & term(T, D);

The main aim of this paper is to demonstrate the usefulness and descriptive power of PD for distributed IR and, in particular, resource selection tasks. We maintain that modelling such tasks at a high level gives expert search users, such as IP professionals, flexibility and tractability of search sessions and predictable retrieval effectiveness. 2.3 Distributed Information Retrieval Distributed information retrieval (DIR), also some times referred to as federated search, deals with the problem of locating and retrieving information from a set of databases

Logic-Based Retrieval

105

or IR engines, as opposed to the classical of IR which relies on a single, centralised store. The interface between the user and the information sources2 is typically provided by a single broker that manages all the transactions between the client (the user) and the servers (the IR engines). This broker is responsible for forwarding queries to the most relevant of the providers as well as fusing and returning result-lists to the user. This requires that the broker has some knowledge over the retrievable content of the participating providers before any query routing can take place[7]. Callan [1] accurately describes the issue of DIR as the set of the following problems: Source description: individual providers or IR engines inform the broker of their content. Source selection: given a query, the broker “retrieves” the sources to “retrieve” from, i.e. the broker ranks the sources, and selects the top-n resources for retrieval. Result fusion: the broker fuses the results received from the previously selected providers. The three phases outlined above occur mainly in a distributed system comprised of autonomous retrieval engines. In a patent-searching scenario we can expect the index distribution to occur in a centrally controlled and computationally efficient environment. In fact, at the centre of our solution there lies a highly parallel computer infrastructure, however our proposed solution is not restricted to such environments. As we discuss in Section 4, the existence of such a tightly controlled environment makes the problem of results’ fusion somewhat redundant. Such an environment, along with the adoption of a DB+IR approach, limits the effects of resource description. However, resource selection is still critical, since different strategies are expected to have a significant impact on retrieval effectiveness and efficiency. For these reasons, resource selection is central to this study, where we attempt to showcase the usefulness, versatility and expressiveness of DB+IR approaches.

3 Data Source Selection In this section we present the two well-known problems of distributed IR of resource description and selection and how these can be expressed in PD and consequently run on a DB+IR system. Resource description is the process by which retrieval engines describe their content to a broker node responsible for conducting the retrieval tasks offered by a distributed or parallel IR system. The type of information shared during resource description, as well as its structure and format, may generally affect the quality of searching. This is especially true in distributed systems linked by slow connections where the transferring of whole documents might be prohibitive. Compact descriptions of documents and incremental description accumulation algorithms (such as query sampling) are therefore generally preferred. As we assume the existence of a multi-CPU supercomputer, the problem of resource description is not central to this study. In our case the broker will 2

In this study the terms sources, referring to data sources and the term resources referring to remote/networked resources coincide and so they are used interchangeably.

106

I.A. Klampanos et al.

Term

term index Doc Source

sailing boats sailing greek islands sailing boats sailing sailing boats sailing sailing boats

s1/doc1 s1/doc1 s1/doc2 s1/doc3 s1/doc4 s2/doc1 s2/doc1 s2/doc2 s2/doc3 s2/doc4 s3/doc1 s3/doc2 s3/doc3

source1 source1 source1 source1 source1 source2 source2 source2 source2 source2 source3 source3 source3

df index Term Source 0.500000 0.250000 0.250000 0.250000 0.750000 0.500000 0.666667 0.333333

sailing boats greek islands sailing boats sailing boats

source1 source1 source1 source1 source2 source2 source3 source3

Fig. 2. Illustraction of data representation

have access to the original parallelised indexes directly and so the resource descriptions will stem from these indexes. In the DB+IR approach we follow, the basic relation representing documents is simply defined by the schema “term(Term, Document)”. In a parallel architecture where many independent engines coexist, we can represent the global document collection by the schema “term src(Term, Document, Source)”, where “Source” uniquely identifies the node sharing the given document. This basic form of resource description provides for a solid basis upon which to distribute collections and searching tasks in a manageable fashion. A small example of such a relation can be seen in Figure 2. This toy collection is also used for demonstrating our modular approach to retrieval strategy modelling in the following sections. Resource selection is the process by which, given a query, a main broker node determines the most appropriate nodes to answer the query. This selection of appropriate resources is based on the descriptions previously made known to the broker by the nodes sharing the searchable content. Therefore, instead of having to reason over the whole collection, an effective resource selection process enables the most relevant set of sources to be probabilistically selected, which decreases the space of evidence needed to be considered upon retrieval 3.1 DF-Based Selection A naive approach to resource selection can be based simply on the document frequencies of query terms inside the document collections to be examined. According to this strategy, the collections with more documents containing the query terms are to be preferred over those with fewer documents. The disadvantages of such a selection strategy can be easily deduced. After all, DF characterises a collection as a whole and does not provide any insight into the retrieval units, the individual documents. However, we decided to include this resource selection strategy in this study because it can serve as a

Logic-Based Retrieval

107

baseline for the comparison of resource selection strategies and also for demonstrating the flexibility of PD when it comes to modularising distributed IR tasks. The DF-based score is defined as follows: nL (t, q) NL (q) nD (t, c) PD (t|c) := ND (c) PL (t|q) :=

Score df(c, q) :=

∑ PL(t|q) · PD(t|c)

(1) (2) (3)

t∈Q

where PL (t|q) denotes the location-based probability of a term t given a query q and is defined by the number of locations (tuples) in which t occurs in q, nL (t, q), over the number of locations present in q, NL (q). Similarly, PD (t|c) denotes the documentbased probability of a term t given a document collection c, defined as the number of documents in which t occurs in c, nD (t, c) over the total number of documents present in c, ND (c). Score df(c, q) can then be defined as the sum of the products of PL (t|q) and PD (t|c), which can also be seen as the sum of documents frequencies, weighted on the query terms. The sum is over the terms in set Q, the set of query terms (for non-query terms, PL (t|q) = 0). A DF-based resource selection strategy expressed in PD can be seen in Figure 3.

1 2 3 4 5

7 8 9 10

12 13 14 15

# DF−based term probability per source # P(t|source): based on document frequency df src(Term, ’source1’) | DF() :− source1(Term, Doc); df src(Term, ’source2’) | DF() :− source2(Term, Doc); df src(Term, ’source3’) | DF() :− source3(Term, Doc); # Query term weighting/normalisation # sum t P(t|q) = 1.0 norm qterm(Term, Query) :− qterm(Term, Query) | DISJOINT(Query); # Retrieval/ranking of sources # sum t P(t|q) ∗ P(t|source) selected SUM(Source, Query) :− norm qterm(Term, Query) & df src(Term, Source);

Fig. 3. DF-based data source selection strategy

The first three rules (lines 3-5) utilise the assumption “df” in order to define the global DF-based probability relation for three data sources. In lines 9, 10, we define the relation “norm qterm” which is the normalised version of the query “qterm” (∑t P(t|q) = 1). The “DISJOINT” assumption is an estimation assumption that defines the evidence

108

I.A. Klampanos et al.

probability as the sum of the probabilities of the tuples with the same evidence key (see Section 2.2). In lines 14, 15 we select the data sources that yield the highest DF scores for our query terms by joining the sources with the “norm qterm” relation. The join operation returns the tuples that match the join condition, and the “SUM” assumption selects the specified columns and aggregates the coinciding tuples in order to sum the probabilities. The next section describes a language modelling-based selection strategy. 3.2 LM-Based Selection In [8] the authors employ a selection rule based on LM. According to this rule, for a query q, a source/collection c, and the global collection g (g comprises all sources), the score is defined as follows: Score lm(c, q) = ∏ [λ · P(t|c) + (1 − λ) · P(t|g)]

(4)

t∈Q

where P(t|c) is the term probability in collection c, P(t|g) is the term probability in the global term probability across all the collections being handled by the system, λ is the mixture parameter, and Q is the set of query terms (here we employ the mathematical convention to use lower case letters for events, and upper case letters for sets

1 2 3

5 6 7

9 10

12 13 14 15

17 18 19

21 22

# P(t|c): source−specific term probability p tc SUM(Term, Source) :− term src(Term, DocId, Source) | DISJOINT(Source); # P(t|g): global term probability (over all sources) p tg SUM(Term) :− term src(Term, DocId, Source) | DISJOINT(); 0.7 model(collModel); 0.3 model(globalModel); w p tc(Term, Source) :− model(collModel) & p tc(Term, Source); w p tg(Term, Source) :− model(globalModel) & p tg(Term) & sources(Source); w p tcg mix(Term, Source) :− w p tc(Term, Source); w p tcg mix(Term, Source) :− w p tg(Term, Source); p tcg mix SUM(Term, Source) :− w p tcg mix(Term, Source); selected PROD(Source, Query) :− qterm(Term, Query) & p tcg mix(Term, Source);

Fig. 4. LM-based data source selection strategy

Logic-Based Retrieval

109

and random variables). Given the resource description introduced in Section 2.3, such a selection strategy can be readily expressed in PD, as shown in Figure 4 3 . In this figure, the rules defined in lines 2 and 6 express the main components of the LM-based selection algorithm, P(t|c) and P(t|g) respectively. It is worth noting that the definition of these two expressions is fairly symmetric, apart from the presence of the attribute “Source” in “p tc”. This is because in “p tc” the term probability is sourcespecific, whereas in “p tg” it is defined across all data sources. This demonstrates the ability of PD to express probabilistic relations on their conceptual and logical characteristics. In lines 9 and 10 we express the mixture parameter λ as a new relation “model” containing exactly two tuples with the corresponding probabilities. Lines 17-19 define the relation “p tcg mix”, which aggregates the probabilities of duplicate tuples as appropriate. Finally, in Line 21, we define the list of the selected data sources, given the query at hand “qterm”. The “PROD” assumption aggregates the probabilities through multiplication, as in Equation 4. 3.3 Other Approaches In addition to the DF-based and LM-based source selection models, there are a range of other approaches to rank sources. [10] discusses a model that is based on the probability ranking principle, saying that there is a cost related to reading relevant and non-relevant documents. Based on this cost-based view, a cost estimate can be defined, and a ranking is optimal if costs are minimal. Other approaches include the application of a “TF-ISF” ranking, where TF is the within-source term frequency, and ISF is the inverse source frequency. In essence, this is a ranking of sources derived from the document-oriented TF-IDF approach. In [11] the so-called CORI model is introduces, with the idea to merge rankings where the origin and semantics of the source-specific retrieval status values might not be known, and therefore, need to be normalised for merging or fusion to take place. Studies comparing resource selection algorithms for various application environments have also been published, such as [12] and [13]. For the purposes of this paper, we focus on the DF-based and LM-based source selection. Future work will include other source selection model, and like for DF-based and LM-based selection, the challenge will be to investigate whether the expressiveness of PD is sufficient, and, if so, whether the respective PD program can be processed efficiently.

4 Retrieval Result Fusion For merging results, we can broadly distinguish between two approaches: Macro fusion. Documents are retrieved from each selected source, and then the results (ranked lists) are merged according to an algorithm that interleaves the sourcespecific retrieval status values. We refer to this fusion as macro fusion. 3

The value of λ = 0.7 is given as an example. Deducing optimal λ values is not within the focus of this study. However, [9] indicate that for keyword queries a λ value between 0.7 and 0.9 is within the optimal range.

110

I.A. Klampanos et al.

Micro fusion. Documents are retrieved from the partial collection that comprises the selected sources. In this approach, the retrieval model views the partial collection as one collection, the ranking is automatically over selected sources, and the fusion is at the level of the sources rather than at the level of retrieval results. We refer to this fusion as micro fusion. Before we model the fusion strategies, we first model basic, TF-IDF-based document retrieval on the full collection. This is shown in Figure 5. 1 2 3

# TF: Term Frequency # P(t occurs | document) tf SUM(Term, Doc) :− term(Term, Doc) | DISJOINT(Doc);

7

# Probabilistic IDF # P(t informs | collection) pidf(Term) | MAX IDF() :− term(Term, Doc);

9

tf idf(Term, Doc) :− tf(Term, Doc) & pidf(Term);

5 6

11 12

tf idf retrieve SUM(Doc, Query) :− qterm(Term, Query) & tf idf(Term, Doc);

Fig. 5. TF-IDF-based ranking strategy

The program contains rules to define a TF-based within-document term probability P(t|d), and an IDF-based probability P(t|c). For P(t|c), the semantics is P(t informs|c) as opposed to the occurrence-based probability P(t occurs|c). The details of the semantics of the PD program to model TF-IDF are not essential for this paper. It is sufficient to view the retrieval/ranking model as a building block that is based on a relation (basic index) modelled in the relation “term(Term, Doc)”. Similar building blocks can be defined for ranking models such as language modelling and BM25. Macro and micro fusion essentially set the probabilistic relation “term”: macro fusion sets it to each selected source and retrieves locally before global fusion; micro fusion sets it to the union of the selected sources. Figure 6 sketches the PD programs describing macro and micro fusion. The macro fusion retrieves documents from each selected data source. The result is collected in the relation “retrieve src(Doc, Query, Source)”. Then, the results are merged, and this merging is described in the rule for “retrieve”. We show here a merging based on ∑c Score(d, q, c)P(r|c), i.e. the score is multiplied with a relevance prior of the respective source/collection. The relevance prior P(r|c) is proportional to the ranking obtained for the sources. To achieve a modular modelling, relations such as “term” and “retrieve” have no source id, whereas relations such as “term src” and “retrieve src” have. The micro fusion merges the selected sources (whereas the macro fusion merges results). The merging is modelled in “partial coll(Term, Doc)”; the relation corresponds

Logic-Based Retrieval

1 2 3 4 5

7 8 9

11 12

111

# INPUT API: # qterm top(Term, QueryId): # The top−selective query terms used for source selection. # select(Source, QuerId): ranking of sources # select top(Source, QueryId): The top−ranked sources. # Each server retrieves from its top−selective data sources. # Retrieve up to 100 docs from each server. retrieve all(DocId, QueryId) :− retrieve:100; # Fuse the results: fuse INDEPENDENT(DocId, QueryId) :− retrieve all(DocId, QueryId);

Fig. 6. “Round Robin” result fusion strategy

to the partial collection (index) comprising the selected sources. Then, the relation “term(Term, Doc)” is set to the partial collection and the retrieval model (e.g. TFIDF, LM, BM25) operates on the sources to retrieve from. Similar to the macro strategy, the micro strategy also considers the score obtained when selecting the sources. In “selected(Source, Query)”, each source has a score (probability), and the index of all sources (in “term src”) is joined with “top2” so that the scores of the sources are reflected in “partial coll”. In the next section, we take a bird’s eye view on strategy modelling, i.e. we view source selection and fusion as just two building blocks of a retrieval strategy that might contain many more building blocks.

5 Retrieval Strategy Modelling This section points out that source selection and result fusion are only two building blocks of a larger picture, namely a retrieval strategy. The benefit of logic-based retrieval becomes evident when viewing a retrieval strategy as a “program” composed of many modules. For example, modules for information extraction, query representation and interpretation and translation, ranking strategies, selection strategies, fusion strategies, feedback strategies, and so forth. It is the facility to re-use pre-defined modules, to exchange and evolve modules, to plug-and-play in if-then-scenarios, which justifies the effort to use logic-based retrieval for information retrieval tasks. To support modular modelling and composability, building blocks have defined in/out interfaces, which correspond to probabilistic relations. For example, “qterm(Term,QueryId)” and “term(Term, DocId)” are input for ranking strategies, output is “retrieve(DocId,QueryId)”. Strategies using a ranking module can rely on the existence and schema of “retrieve”. In a similar way, “qterm(Term,QueryId)” and “df(Term,Source)” are input for a DF-based selection strategy, and output is “select(Source,Queryid)”.

112

1 2 3 4

6 7

9 10 11 12

14 15

17 18 19

I.A. Klampanos et al.

########## DATA SOURCE REPRESENTATION ########## # United source representations full coll(Term, Doc, ’source1’) :− source1(Term, Doc); full coll(Term, Doc, ’source2’) :− source2(Term, Doc); # Query term weighting/normalisation norm qterm(Term, Query) :− qterm(Term, Query) | DISJOINT(Query); ########## SOURCE SELECTION STRATEGY ########## # DF per source df src(Term, ’source1’) | DF() :− source1(Term, Doc); df src(Term, ’source2’) | DF() :− source2(Term, Doc); # DF−based source selection select SUM(Source, Query) :− norm qterm(Term, Query) & df src(Term, Source); ########## RESULT FUSION STRATEGY ######### select top2(Source, Query) :− select(Source, Query):2; # Each back−end server build search space based on select top2.

24

# Fusion servers fuse the results from back−end server. # For example, consider the top−1000 results from each server. fuse(Doc, Query) :− retrieve:1000; ?− fuse:100; # Show the top−100 after fusion

26

########## SEARCH STRATEGIES #########

21 22 23

28 29 30

32 33 34 35 36 37

39 40

# Search strategy 1: Document search retrieve docs SUM(Doc, Query) :− norm qterm(Term, Query) & term(Term, Doc); # Search strategy 2: Inventors of retrieved patents who worked for IBM. retrieve inventors(InventorId, Name, Query) :− retrieve docs(Doc, Query) & workedFor(InventorId, Company) & attribute(name, Company, ”IBM”); # This strategy shows how a content−based search is combined with other # search criteria. # Search strategy 3: Retrieve the patents related to chocolate and # the colour purple (Cadbury has ”its purple” protected).

43

# Query representation: qterm(chocolate,q1); qterm(purple,q1); qterm(colour,q1);

45

choc col patents(Doc) :− retrieve(Doc, Query);

42

51

# Find the patents by Kraft. kraft patents(Doc) :− choc col patents(Docs, Query) & assignee(Doc, Assignee) & attribute(name, Assignee, ”Kraft”); # By Cadbury. cadbury patents(Doc) :− choc col patents(Docs, Query) & ...

53

# How similar are the claims of the top−3 patents?

47 48 49 50

Fig. 7. Modular composition of a retrieval strategy

Logic-Based Retrieval

113

Following such an ADT/API approach, the strategies can be composed in a modular way, and it becomes “easy” to plug-and-play with different strategies. Figure 7 shows an example for a retrieval strategy that consists of: 1. A header to define intentional predicates such as “full coll” and “norm qterm” (query term weighting, ∑t P(t|q) = 1.0 for each query). 2. A source selection/ranking strategy (e.g. document-frequency-based selection or language-modelling-based selection) 3. A document ranking strategy (e.g.TF-IDF, LM) and merging of results 4. A search strategy (e.g. document search, element search, claim search, inventor search) The rules for “df src” define the source-specific DF-based term probabilities. The rule for “select” represents ∑t P(t|q) · P(t|c), which is the score for each source. The result of the selection strategy is in “select(Source, Query)”, which is input to the next module, the fusion strategy. In our example, the top two sources are chosen, which are then joined with the full set of sources. The result of the join is a partial collection comprising of the selected sources. The next step is to define a ranking strategy that performs relevance-based ranking with respect to the partial collection and not the whole collection. For space reasons, the ranking strategy is described as a join of the query terms with the partial collection. An alternative approach would be to utilise ranking strategies such as TF-IDF or LM as it was demonstrated earlier in this paper (see also [4]). Even though the user can define and use their own selection, fusion and ranking strategies, they can be generally thought to be part of the system, the workings of which should not concern the average patent searcher. The next part of the retrieval strategy is the search strategy. It is where a particular task, such as patent or inventor search, is described and would be of great importance to the patent professional. The documents retrieved by the ranking strategy can be used to retrieve more specific objects, such as inventors or assignees. The flexibility to reason about both documents and objects within the same framework is one of the properties of logical retrieval and one of the requirements of patent search. In Figure 7 we show alternative search strategies of increasing complexities. From these strategies it follows that more complicated approaches to searching can be devised, such as to compare previously retrieved claims of rival companies, for instance. Dividing the retrieval strategy into modular parts enables one to plug-in or -out different resource selection strategies, different ranking strategies and/or different search strategies. A patent searcher can, thus, have control over the search task at hand and also understand and modify the ranking and reasoning process.

6 Architecture of the LSLR Matrixware-HySpirit System 6.1 Overview In this section, we introduce the architecture of the Matrixware-HySpirit System, which is a part of the Large-Scale Logic-based Retrieval (LSLR) project for patent searching.

114

I.A. Klampanos et al.

Internet LSLR-Webservice

User

LSLR-HySpirit Merger/Master

User

...

LSLR-HySpirit Worker 1

...

User

LSLR-HySpirit Worker k

... KB 1

KB 2

... KB n

KB 1

KB 2

KB n

Fig. 8. The overall federated search architecture supporting logic-based patent retrieval

Figure 8 shows the architecture of the system. Overall, it consists of two main parts: the first is the patent retrieval core system, which is driven by integrating HySpirit retrieval engines and DIR techniques as discussed in the previous sections, and the second is a dissemination system that provides patent searching functionality to application users, which is implemented based on web-services technology. In Figure 8, the LSLR-HySpirit-Master and LSLR-HySpirit-Worker nodes (machines) together, which are connected by high speed network or data bus, form the retrieval core system. Within each worker node, multiple knowledge-bases (KBs) (i.e. indexes) are built based on different distributed strategies; for example, serial or topical distributions over patent data. Furthermore, LSLR-HySpirit-Workers work independently, while the final results are merged within LSLR-HySpirit-Master. Because the retrieval core system is held within an enterprise’s intranet, which typically allows only for restricted accesses, an LSLR-Webservice is deployed in order to provide the logic-based retrieval functionality to wider public. To make the retrieval system accessible by a webservice, the LSLR-Webservice server is connected to the LSLR-HySpirit-Master server through a safe link, while the two servers use a dedicated protocol to communicate with each other and transfer data. Since the LSLR-Webservice follows a widely used industrial standard, public users are able to access the logic-based retrieval functionality for patent searching conveniently. 6.2 Parallel Processing In our architecture, each LSLR-worker processes each query by first ranking the knowledge bases in terms of how appropriate they are to answer it. It then forwards the query to the top-k KBs. During this phase, and because of our transparent logic-based

Logic-Based Retrieval

115

approach, the set of the selected KB are essentially treated as a single collection, therefore doing away with the need for explicit merging of results. Appropriate data-source selection that may be used here are described in Section 3. It is clear that the more KBs are selected, the more time an LSLR-worker needs in order to reply to a query. On the level of the LSLR-HySpirit-Master, things are setup in a parallel fashion. Upon receiving a query, the LSLR-HySpirit-Master will forward it to all connected LSLR-HySpirit-Workers. Even though the LSLR-Master is in a position to set out retrieval and selection strategies for individual LSLR-HySpirit-Workers to follow, it is effectively unaware of their overall contents. Therefore, given a query, all LSLRHySpirit-Workers will be working in parallel, while they will individually restrict their searching to a subset of their KBs. This hybrid approach represents a tradeoff between the need for physical resources and the responsiveness of the system. If there was a 1-to-1 relationship between the LSLR-HySpirit-Workers and the KBs, we would be needing many more physical resources in order to handle the same amount of content. On the other hand, having a single LSLR-HySpirit-Worker would seriously impede the performance of the system, having to cope with an overwhelmingly large number of KBs. The lack of active selection on the part of the LSLR-manager does not represent a weakness. In the case of a patent collection being distributed in a non-topical manner, selecting LSLR-HySpiritWorkers would not make a difference. In a topical distribution of content, as long as KBs of the same topic are as widely distributed in LSLR-HySpirit-Workers as possible, the retrieval effectiveness would be optimal while the work would be kept as parallelised as possible.

7 Evaluation The main focus of these experiments is on scalability. The reason is that given the expressiveness of PDatalog, any state-of-the-art retrieval model can be implemented, and customisation is thanks to the open-box and modular nature of logical programming an inherent feature. Therefore, as long as the approach scales, good retrieval quality can be achieved. To obtain a measure for the retrieval quality achieved when using DIR concepts, it is reasonable to measure the distance between the DIR result, and the all-collection result, where the latter is viewed to reflect the “optimal” result that can be achieved. On the other hand, one can expect that the more indexes (i.e. knowledge-bases) that are selected to be searched, the DIR results would be more closer to the all-collection one. In this section we present some preliminary results of our parallelisation approach in terms of retrieval and parallelisation efficiency (see Figure 9). For these results we used a base collection of 1,037,785 patent documents, provided to us by Matrixware. The overall collection was divided into indexes of 5,000 documents each. The resource selection algorithms described in Sections 3.1 and 3.2 were measured for efficiency on a single process handling 5, 25 and 200 indexes (or 25,000, 125,000 and 1,000,000 documents respectively). In all cases we selected the first 3 best matching indexes against which the queries were then evaluated simply using TF-IDF. Indicative times for query evaluation are presented in Figure 9(a). These are average times acquired by evaluating

116

I.A. Klampanos et al. #Indexes

5

25

200

Algorithm DF 0.17s 0.69s 9.11s LM 0.18s 0.67s 9.12s (a) Avg Query Processing Time vs Number of Selected Sources. 100 90 80 70 60 3 to 5

50

5 to 7 40

7 to 9

30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

(b) Overlaps between DIR Results (percentages). Fig. 9. Experimental results

100 queries repeatedly, over 10 repetitions. These queries were automatically generated from a larger patent corpus and consist of four terms each. This experiment was executed on IRF’s LDC4 . In addition, we performed iterative DIR runs by selecting 1 to 10 best-matched indexes (amount 208 indexes in total) and executing 25 man-picked queries over the selected indexes. To compare the distance between results, we compared the top 100 respective results of different runs. For instance, comparisons are made between 3 and 5 selected indexes, and 5 versus 7, 7 versus 9, and so on. For the moment, the indexes were built based on serial distribution, thus the overlapping rate can be simply estimated, i.e. 60% for 3 vs. 5 indexes, and 5 vs. 7 sources is about 71%, while 7 vs. 9 sources is about 78%. In principle, a sub-linear overlapping rate is expected to be observed. The results are given in Figure 9(b). From the figure we can see, for most queries, the overlapping rates increase predictably with the increase of considered sources, while the overlapping rates of some queries appear to be saturated with relatively less selected indexes. 4

http://www.ir-facility.org/the irf/infrastructure

Logic-Based Retrieval

117

A noticeable result emerging from this table is that the times observed for both the DF and LM resource selection strategies are virtually the same. This observation might be surprising due their difference in terms of the number of relations they require (Figures 3 and 4). This closeness in performance is due to internal optimisations targeted specifically at language-modelling algorithms. Such optimisations keep the performance of more complicated modelling tasks at a practically viable level. When taking such measurements there are a number of additional factors which affect efficiency. An important first factor has to do with the presence of an effective caching mechanism on the operating system and hardware levels. Such a caching mechanism can cause response times to vary greatly, especially between the first and subsequent calls to the retrieval engine. Indicatively, the first selection and retrieval task to be executed may take a few minutes whereas any subsequent task may take less than a second. This difference in the performance of the system between cold and warm starts is one of the reasons for averaging our time measurements over a number of queries and repetitions. However, this problem of performance when cold-starting the system should not affect our solution directly, since we make the assumption that a number of nodes will be alive on the system at any given time. Another factor that affects the performance of resource selection and query evaluation is the number of sources the system chooses to consult for a given query. In our experiments we would choose the top 3 sources for subsequent query forwarding, however the time complexity of resource selection and query evaluation depending on the number of sources to consult has not been evaluated yet. These as well as additional factors that may affect the system’s performance will be investigated in future work outlined in the next Section. Regardless of these additional factors, the results of this first experiment clearly demonstrate the feasibility of the proposed DB+IR approach for patent retrieval. By achieving sub-second performance, on average, for a single engine handling 25 indexes (or 125,000) documents it becomes clear that a few tens of such nodes would suffice in order to handle millions of patent documents. The internal topology of our solution as well as the exact selection algorithms will affect performance and are all future work we intend to undertake.

8 Summary and Conclusions In this paper, we reported how logical retrieval can assist patent searches to solve search tasks. The initial, keyword-based search for a patent is usually just the starting point for exploring patents, retrieving results for refined or other queries, comparing the results, and making decisions about the (in)validity of the patent application. Logical retrieval helps since search processes can be recorded/stored, can be re-used, can be communicated and composed. Thereby, we view a search process as a “complex retrieval task” involving sub-tasks such as ranking documents, selecting data sources, fusing results, and comparing results. Applying logical retrieval large-scale is challenging. To take advantage of the expressive power of logical retrieval while maintaining scalability, in the LSLR project we investigate how to marry logical retrieval with techniques of distributed IR (DIR). In principle, we can imagine a DIR framework that mediates multiple logical retrieval

118

I.A. Klampanos et al.

engines. A particular contribution of the LSLR project is to implement data-source selection and result fusion within logical retrieval seamlessly. In this paper we presented how these techniques can be designed and implemented as part of a larger infrastructure that supports logic-based patent searching functionality. The main result of the LSLR project is to enable logical retrieval to go large-scale, in order to be applicable for patent search. What source selection and result fusion are for patent searchers, are car engines for drivers: usually, the users are not interested in how it looks under the hood, as long as it is powerful enough to solve the task. It is the under-the-hood engine that empowers solving tasks. We have indicated in this paper how the LSLR system/engine will empower patent users to run initial searches, by selecting from any one of the well-known ranking algorithms, to explore step-wise why a document was retrieved, to save and compare results, and to communicate and evolve retrieval strategies. Such an open-box, highly abstract environment of solving search tasks is standard in DB technology, when using SQL to query data, for instance. Logical retrieval offers the ability to search data in a SQL-like manner, while at the same time it allows users to select and devise ranking models (including other strategy components, such as source selection, result fusion, etc.).

Acknowledgements We would like to thank Matrixware Information Services GmbH for supporting this work.

References 1. Callan, J.: Distributed information retrieval. In: Advances in Information Retrieval, pp. 127– 150. Kluwer Academic Publishers, Dordrecht (2000) 2. Fuhr, N.: Probabilistic datalog - a logic for powerful retrieval methods. In: Fox, E., Ingwersen, P., Fidel, R. (eds.) Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 282–290. ACM, New York (1995) 3. Roelleke, T., Fuhr, N.: Information retrieval with probabilistic datalog. In: Crestani, F., Lalmas, M., Rijsbergen, C.J. (eds.) Uncertainty and Logics - Advanced models for the representation and retrieval of information. Kluwer Academic Publishers, Dordrecht (1998) 4. Roelleke, T., Wu, H., Wang, J., Azzam, H.: Modelling retrieval models in a probabilistic relational algebra with a new operator: The relational Bayes. VLDB Journal 17(1), 5–37 (2008) 5. Wu, H., Kazai, G., Roelleke, T.: Modelling anchor text retrieval in book search based on back-of-book index. In: SIRIG Workshop on Focused Retrieval, pp. 51–58 (2008) 6. Fuhr, N.: Optimum database selection in networked ir. In: Callan, J., Fuhr, N. (eds.) NIR 1996. Proceedings of the SIGIR 1996 Workshop on Networked Information Retrieval (1996), http://SunSite.Informatik.RWTH-Aachen.DE/Publications/CEUR-WS/Vol-7/ 7. EPO: Annual Report 2008. European Patent Office (2008) 8. Si, L., Jin, R., Callan, J., Ogilvie, P.: A language modeling framework for resource selection and results merging. In: CIKM 2002 (2002)

Logic-Based Retrieval

119

9. Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to information retrieval. ACM Trans. Inf. Syst. 22(2), 179–214 (2004) 10. Fuhr, N.: A decision-theoretic approach to database selection in networked ir. ACM Transactions on Information Systems 17(3), 229–249 (1999) 11. Callan, J.P., Lu, Z., Croft, W.B.: Searching distributed collections with inference networks. In: SIGIR 1995: Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 21–28. ACM, New York (1995) 12. Craswell, N., Bailey, P., Hawking, D.: Server selection on the world wide web. In: DL 2000: Proceedings of the fifth ACM conference on Digital libraries, pp. 37–46. ACM, New York (2000) 13. Hawking, D., Thomas, P.: Server selection methods in hybrid portal search. In: SIGIR 2005: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 75–82. ACM, New York (2005)

Automatic Extraction and Resolution of Bibliographical References in Patent Documents Patrice Lopez patrice [email protected]

Abstract. This paper describes experiments with Conditional Random Fields (CRF) for extracting bibliographical references in patent documents. CRF are used for performing extraction and parsing tasks which are expressed as sequence tagging problems. The automatic recognition covers references to other patent documents and to scholarship publications which are both characterized by a strong variability of contexts and patterns. Our work is not limited to the extraction of reference blocks but also includes fine-grained parsing and the resolution of the bibliographical references based on data normalization and the access to different online bibliographical services. For these different tasks, CRF models surpass significantly existing rule-based algorithms and other machine learning techniques, resulting more particularly in a very high performance for patent reference extractions with a reduction of approx. 75% of the error rate compared to previous works.

1

Introduction

Bibliographical citations play a major role in patent information. Citations represent the closest prior art which will be the basis for evaluating the contribution of a patent application and for identifying grantable subject matter. In patent o ffices, the result o f the sea rch pha se is the search report, a co llectio n o f references to pa tents a nd to o ther public do cuments such a s scientific a rticles, technica l ma nua ls o r resea rch disclo sures, so -ca lled No n-Pa tent Litera ture (NPL). In addition to the search report, the text body of the patent document contains usually many bibliographical references introduced in the original application documents or introduced at a further filing stage or at granting stage. A patent document can contain several hundred of such references, while the number of citations in the search report is rarely more than ten. In the context of patent document processing, the exhaustive identification of bibliographical references present in the patent textual description could serve different important purposes: – In patent retrieval, citation relations have proved to be highly valuable. They are used primarily as a common retrieval key, but also for improving the ranking of retrieval results. Previous works in [1] used extensively patent citation information in pre-processing and post-ranking for improving the mean average precision in a prior art search task. H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 120–135, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Automatic Extraction and Resolution of Bibliographical References

121

– For a broad range of applications including information retrieval or automatic summarization, the usage of the citation contexts has contributed to interesting results. A citation context provides additional textual descriptions corresponding to the recognized contribution of the cited document according to the community. [2], for instance, tried to exploit the citation texts for improving the semantic interpretation and the retrieval of articles in the biomedical domain. Applying such techniques for a collection of interrelated documents supposes, however, an accurate identification and resolution of the references. – For prior art search, the identification of references introduced by the applicant in the text of the patent application can help to pre-classify and route the application to the appropriate patent examiners, but also to initiate more efficiently document search and to identify rapidly related applications. – For the purpose of automatic patent classification, [3], for instance, proposed to introduce citation relations for improving automatic patent classification based on a kernel-based approach. – Scientometrics studies are based on document linkages and citations [4]. Identification and accurate resolution of bibliographical references and coreferences are crucial in this domain. – Citations and automatic reference analysis are also used for facilitating the reading of technical documents [5]. – As the understanding of a reference to a patent document can be extremely complicated for a user who is not specialist in patent information, the automatic identification and resolution of references is important for locating patent documents and, more generally, for making the public aware of patent information which is a duty of the patent systems. For all these different reasons, accurate automatic extraction and parsing of references in patent documents appear essential. Relatively to this necessity, the amount of work and research on this topic appears limited. As noted by [6], the applications of information extraction to patent annotation as general are quite scarce. This extraction task is, however, not a solved problem and represents a serious challenge. For [7], a usable method of reference extraction would represent an interesting benefit for patent database producers. [7] points out that a random sample of US granted patents showed an ”alarming” degree of variation in form when citing for instance Japanese unexamined patent applications and that, in absence of a standard for patent citation, the challenge of such extraction in embedded text should not be underestimated. The extraction of non-patent literature appears to the author -by an order of magnitude- more difficult due to the very high variability, idiosyncratic formatting and ambiguities. Early work [8] used template mining for automatic extracting citations in English language patents. Evaluation results for patent references showed a precision of only 64% for a recall of 87%, and for non patent references a precision of 72% for 70% recall, confirming the difficulty of both tasks and the necessity of more sophisticated techniques. Since 2006, the European Patent Office (EPO) proposes in the publication of unexamined patent applications an additional section containing the references

122

P. Lopez

to patents cited in the description by the applicant based on automatic extraction technique. The EPO announced between 70 and 80% of accuracy in automatic citation extraction, which is completed by manual editing. We were not able, however, to find more technical information regarding this service. [6] experimented automatic extraction of various entities of special interest in patent documents based on handcrafted rules. The most difficult entities appeared to be references of patents, with precision/recall in the range of 74/81% and references to non-patent literature, with precision/recall in the range of 70/75%. The experiment was based on a very small amount of evaluation data and was limited to the identification of the citation blocks without subsequent parsing. In previous work [1], we used a basic set of regular expressions for extracting patent citations in patent text bodies. The regular expressions were created based on a set of approx. 50 patterns of patent citations. Although the precision of extraction was relatively good, because the reference results were pruned and limited to a collection of 1 million patent documents, further analysis showed that we were missing at least 40% of the citations and that more advanced techniques were necessary. Parsing of scholar bibliographical references in isolation has been the object of various research works, see [9] and [10]. Statistical and machine learning approaches, in particular based on Conditional Random Fields, clearly surpassed approaches based on handcrafted rules and templates [9]. Relatively to our objectives, the extraction of references embedded in text and, in particular, in patent texts, has been, to our knowledge, addressed recently only by [11], more precisely in the context of Japanese patent applications and covering only references to non-patent literature. [11] used modern machine learning techniques, in particular CRF, and reached an precision/recall of 91.6/86.9% for identifying sentences containing a reference, and 86.2/85.1% for average bibliographical attribute recognition in sentences containing a reference (among five attributes: title, author, source, date and page). The present work addresses the problem of automatic extraction, parsing and linking of patent and non-patent references in multilingual patent texts. This extraction covers first the identification of reference blocks in the textual description of the patent document, illustrated by step 1 in Fig. 1, and second, the parsing of the reference for recognizing the bibliographical attributes of the reference, see step 2 in Fig. 1. The final step is the resolution of the reference, by linking it to the convenient digital representation -in particular the full textvia access to online bibliographical services and OpenURL 1 services (see step 3 in Fig. 1). The next section presents an overview of the bibliographic extraction tool that we have developed for patent documents. We then describe the training and evaluation data created for the present work, containing approx. 2 500 1

OpenURL is a NISO standard for identifying and linking scholarly publications. The standard is maintained by OCLC and is used by all major libraries and scientific organizations.

Automatic Extraction and Resolution of Bibliographical References

type: patent issuing auth. : US number: 5738985

2 Parse

http://v3.espacenet.com/ publicationDetails/biblio? DB=EPODOC&CC=US&NR=5738985

3 Link

123

Compounds can exhibit anti-hepatitis C activity by inhibiting viral and host cell targets required in the replication cycle.A number of assays have been published to assess these activities. A general method that assesses the gross increase of HCV virus in culture is disclosed in U.S. Patent No. 5,738,985 to Miles et al. In vitro assays have been reported in Lohmann et al, J. of Biol. Chem., 274:10807-10815, 1999. A cell line,

1 Extract

Parse & 2 Consolidate

Link 3

Title: Selective Stimulation of Hepatitis C Virus and Pestivirus NS5B RNA Polymerase Activity by GTP Author : Volker Lohmann, Hilary Overton, and Ralf Bartenschlager Journal: Journal of Biological Chemistry DOI: 10.1074/jbc.274.16.10807 Date: 1999 Volume: 274 Pages: 10807-10815 Publisher: The American Society for Biochemistry and Molecular Biology http://sfx.mpg.de/sfx?ctx_ver=Z39.88-2004 &rft.genre=article& rft_id=info%3Adoi%2F10.1074%2Fjbc.274.16.10807

Fig. 1. Example of bibliographical reference extraction, parsing and linking

manually annotated references in 200 patent documents. In section 4 to 6, we present the CRF models used for the identification of patent and NPL references and the subsequent parsing of NPL references. The different tasks are discussed and evaluated, showing in particular a high performance for patent reference extractions. Finally we conclude by discussing our overall findings.

2

System Overview

Figure 2 gives an overview of the processing realized by our bibliographic extraction module for patent documents. The bibliographic extraction module performs the following processing steps: 1. Identification of reference strings: The text body is first extracted from the patent document. The patent document can be a PDF with full text available or an XML representation (currently only the Matrixware’s XML Alexandria format is supported). The patent and non patent reference blocks are first indentified in the text body by two specific Linear-Chain CRF models. The patent and non-patent references follow then two similar but specific processing paths.

124

P. Lopez CRF Model

Patent Ref. blocks Identifier

OPS

FST

Raw patent references

Parsing & Normalization

Normalized patent references

- Full Patent Record - Link to Espacenet

Linker

Patent Documents

NPL Ref. blocks Identifier

CRF Model

Raw nonpatent ref.

Parsing & Normalization

CRF Model

Normalized non-patent references

- OpenURL - BibTeX/ TEI exports

Consolidation & Linker

xISBN

AWS

CrossRef

Fig. 2. Overview of the bibliographic extraction processing

2. Parsing and normalization of the extracted reference strings: The reference text is then parsed and normalized in order to obtain a set of bibliographical attributes. References to patent are parsed and normalized in one step by a Finite State Transducer (FST) which will identify (i) if the patent is referred to as a patent application or a patent publication, (ii) a country code, (iii) a number and (iv) a kind code. Non-patent references are first annotated by a CRF for identifying a set of 12 bibliographical attributes (author, title, journal, date, etc.). Each of the identified attribute is then normalized using regular expressions. 3. Consolidation with online bibliographical services: Different online bibliograhical services are then accessed to validate and enrich the identified reference. For patent references, we use OPS (Open Patent Service2 ), a web service provided by the EPO for accessing the Espacenet patent databases. For non-patent literature, we perform a first look-up via Crossref3 for trying to identify a DOI (Digital Object Identifier) and for consolidating the recognized bibliographical information with the information stored in the CrossRef database. The metadata information provided by Crossref come initially from the publishers and are more reliable for further linking to bibliographical services such as full text access. If no DOI is found, WorldCat xISBN service from OCLC and Amazon Web Services (AWS) are used for trying to identify a printed book. The retrieved metadata are used to correct and complete the list of bibliographical attributes if at least a core of the extracted attributes4 permits to retrieve unambiguously a DOI. 2 3 4

http://ops.espacenet.com http://www.crossref.org This core is either the title and the last name of the first author or the Journal or Book title, the volume and the first page of the cited article, which are normally present in an abbreviated reference. As a consequence, it is possible, for example, to retrieve automatically the title of an article cited with an abbreviated reference, as illustrated by Fig. 1.

Automatic Extraction and Resolution of Bibliographical References

125

4. Linking: Finally a linking is built to the referenced items as an OpenURL address. OpenURL permits the usage of a institutional link resolver for providing dynamic services such as direct access to the full texts according to the subscriptions of the institution or access to the local OPAC. OpenURL and Link Resolvers are well-known and massively used in the digital libraries. For patent references, we created a direct linkage to the cited patent document via EPO Espacenet. As an alternative, exportations of the parsed identified references into BibTeX and TEI formats have also been implemented.

3

Training and Evaluation Corpora

For training and evaluating our extractions, we built a corpus based on 200 patent documents randomly selected from three different patent systems (US, European and PCT) written in three different languages (English, German and French) from 1995 to 2008. We selected proportionally more EP documents for balancing the multilinguality ratio. The constraint for selecting a document was that each patent document must contain at least one reference. All patent and non-patent references present in this corpus were manually annotated. Table 1. Overview of the corpus of references Documents # % EP US WO Total

124 38 38 200

62 19 19 100

Language English French German Total

# % 116 58 33 16.5 51 25.5 200 100

Reference Type # Av. per doc. Patent 1993 9.96 Non-Patent 510 2.55 All 2503 12.51

As there is currently no reference data set for this task, we plan to make this corpus available in the future for allowing the community to compare in the future different approaches. A rapid study of this corpus -which can be viewed as a relatively representative random sample of the patent documents in general- helped us to answer the two following important questions: To which extend the automatic extraction of references in patent text is necessary? The citations introduced by the applicant can also appear as a formatted reference in the search report, marked as D citations. If most of them are reproduced in the search report, an automatic extraction in the text body of the patent would not be necessary. However, the number of the citations introduced in the description of the patent documents is in average much larger than the number of citations in a search report. In the set of 200 patent documents, we examine 43 included search reports. The average number of cited documents

126

P. Lopez

was 4.1. In contrast, the average number of references annotated in the text body was 12.5. In addition, the search report does not systematically cite a reference introduced by the applicant: a citation noted D in the search report appears in the initial text body in less than 39% of the case. This observation shows that the reference extraction task is worthwhile and confirms a similar observation of [7]. Should we limit the extraction of references to the ”Prior art” section? In [11], the extraction was only realized in the specific subsection usually named ”Background of the Invention” or ”Prior Art”, considering that this is the place where the authors cite related bibliographical items. While this assumption might be correct for Japanese patent documents, it is not the case in our sample. We observed that approx. 25% of the references are not introduced in the ”Prior Art section” but in the other patent subsections (”Detailed description of the invention”, ”Embodiments of the inventions”, etc.) and it does not appear appropriate to limit the extraction to only a part of the overall description. References are often used in the description of an embodiment of an invention for legal purposes, more precisely for ensuring that the claims are fully supported by the description and by existing known practices of the technical domain. These references often appear to be the most valuable ones because they are relevant to the claimed invention itself and do not only describe general technical backgrounds.

4

Conditional Random Fields for Bibliographical Information Extraction

CRF have been applied successfully to a variety of domains, including text processing and Named Entity Recognition (NER). A reliable level of accuracy for bibliographical data extraction has been reached since [9] which exploits Linear Chain CRF models for labeling a sequence of data. CRF present the following key advantages: – Their convex likelihood function ensures that the learning converges to the global maximum. – The possibility to use a rich set of features. – They extract features over a sequence of tags using arbitrary amounts of context. Long-distance contextual features are particularly helpful for a task as bibliographical extraction because of the importance of external cues, i.e. features appearing before or after the reference itself. In contrast, HMM feature sets are usually limited to a predicted tag for the current word, the current word itself and the tags of the two previous words. In the case of analysis of bibliographical reference strings taken in isolation, CRF surpassed other machine learning algorithms as HMM and SVM, see [9,11], and appears, therefore, as a good choice for related extraction tasks in patent documents.

Automatic Extraction and Resolution of Bibliographical References

127

When applying a CRF model, different aspects can significantly influence the results: – the selection of features, – the choice of the CRF hyper-parameter, the regularization algorithm and the pruning of unfrequent features, – the token segmentation. As the drawback of CRF is the computationally expensive training, tuning these different aspects is sensitive. In general we use first the features that have been shown to be effective in NER. Table 2 presents first the general features used in all the considered tasks and the task specific features used in combination of the general features. In addition to lexical features, we use more sophisticated pattern matching, general lexicons for English, French and German, and specialized dictionaries depending on the task. Finally in addition to these features, the word forms in a left and right windows of size 5 around the current word are also used (not represented in the table).

Table 2. List of features for CRF models. All

(f1) (f2-5) (f6-9) (f10) (f11) (f12) (f13) (f14) (f15) (f16) (f17) (f18) (f19) (f20) Patent ref. (f21) in context (f22) (f23) NPL ref. (f21) in context (f22’) (f23’) NPL ref. isol. (f21’)

Current token Prefix character 1 to 4 grams of the current token Suffix character 1 to 4 grams of the current token Capitalization information (no, first letter, all) Lower case form Punctuation information of the current token (hyphen, etc.) Number information of the current token (no, one or all digit) Length of the current token Year pattern matching (boolean) Month pattern matching (boolean) Common word (boolean) Common US forname (boolean) Common US family name (boolean) Country name (boolean) Relative position of the current token in the document common country codes (boolean) common kind code pattern (boolean) Relative position of the current token in the document scientific and technical journal names (boolean) abbreviated names for sci-tech journal (boolean) Relative position of the token in the reference string

We used the package CRF++5 for all our different models. This package is particularly fast and able to handle millions of unique feature values. 5

http://crfpp.sourceforge.net/

128

5 5.1

P. Lopez

Extraction of Reference Blocks Patents References

A reference to a patent document contains at least two main informations: (1) the name of the issuing authority usually represented as a country code such as DE, FR, US or a regional code such as EP or WO and (2) a number which depends on each authority. The patent can be expressed as a reference to a patent application or as a reference to a published patent document, depending on the stage of advancement of the patent prosecution, which can be recognized by a different numbering system. The number can include a year, which is often made apparent by a slash (for instance WO 98/45461). Reference to patent documents can also optionally contain (3) a kind code characterizing a particular publication for the patent. Usually, the kind code A refers to documents published prior to the examination of the application, kind code B refers to document corresponding to granted versions and kind code C corresponds to a version corrected after reexamination. The precise meaning of the kind code depends however on the issuing authority. For example, 34 different kind codes are used by the USPTO covering a vast range of legal steps involving a specific patent publication. The main source of variations in patent references are the following one: – The issuing authority can be indicated either by a word (Belgian Patent No. 841,910) or by a code (JP-B-1-25050). It is common to factorize the name of the issuing authority for a group of patents (U.S. Patent Nos. 4,123,282 and 4,510,236). Unfortunately, the reference to the issuing authority can be very flexible (Application No. 98-48096 filed in Korea). – The same issuing authority can be referred by different names (British Patent Publication Nos..., Great Britain Patent Application 2,144,992, etc.), even in the same document. – The numbers can be expressed with or without various punctuation marks, creating a great amount of variations and possible number of tokens. – The kind code can be positioned before or after the number (EP-A1-0 924 946, EP220746A2). – A large amount of additional verbosity appear in many cases: indication about the application stage (U.S. Provisional Patent Application Serial No...), which can be translated or not for Japanese patents (Japanese Application Kokai..., Japanese Unexamined Patent Publication Numbers...), indication for Japanese application dates of the reigns of the Emperors Hirohito, Sho or Showa, and Akihito, Hei or Heisei (Japanese Patent Publication No. Hei 2-18696) In the same patent document and even in the same paragraph, it is not rare to see references to a patent from the same issuing authority using completely different patterns. Adaptation of the CRF Model: In addition to our standard set of features, we consider specialized features in the CRF model indicating if the word is a

Automatic Extraction and Resolution of Bibliographical References

129

common country codes or a common kind codes (see Table 2). After experiments, the hyper-parameter C for the CRF model was set to 2.1. The lexical features characterizing word around a reference are important for preventing false positives, i.e. the incorrect identification of a bibliographical reference in normal text. For patent references, risks of false positives come in particular from names of chemical and biological entities (ANXA2 BE908217 annexin A2), formulas or addresses (Palo Alto, CA 94304). Evaluation: The results for the recognition of patent blocks are given in Table 3. The evaluation is based on the repeated random sub-sampling validation methodology: the corpus is automatically divided randomly ten times using 80% for training and 20% for testing. The reported performance scores are then the micro average for these ten corpus segmentations. Similarly as previous works in bibliographical analysis [9], we present the results following two levels, a word accuracy metric which evaluates the accuracy of identification for each word (predicted label for the word equals the expected label) and an instance accuracy metric which measures the performance for the complete reference (all words of the reference are correctly labeled), i.e. in a way that is sensitive to a single error in any part of the reference. Table 3. Evaluation results for the identification of patent references within the patent text bodies Level

Precision Recall F1

Word 0.9762 Instance 0.9466

0.9774 0.9768 0.9616 0.9540

A precision at instance level close to the one at word level is surprising but is explained at the light of the analysis of errors. Many errors correspond to long sequences of badly tag words in formula or list of numbers, sometimes involving more than ten words. As such an erroneous word sequence forms only one incorrect instance, the impact on the word level is more important than on instance level. In contrast, erroneous label of words at a boundary of a reference, which has a negative impact at instance level, is not frequent. Finally, missed labelings concern mainly very long instances which can involve many words. The result at instance level can be compared with the performance reported in the previous works [8,6], and the claimed accuracy of the EPO automatic reference extraction service, which are all in the range of 70-80% for precision and recall. This performance correspond to a reduction of more than 75% of the error rate as compared to these previous works. This significant improvement both in terms of accuracy and coverage illustrates the benefit of the usage of Machine Learning approaches in general, and CRF in particular for identifying versatile citations in context. The most common errors are false positives in context where a string similar to country code is present near a number, complex coordinations (U.S. Patents

130

P. Lopez

3,933,672, issued January 20, 1976 to Bartoletta et al., and 4,136,045; US Patents 4,899,505 of Williamson et al, 2,905,072 of Oswald, and 938,930 of Wiest) and unfrequent references having a rare country code (DD 237 070, DD corresponding to DDR, the former East Germany). 5.2

Non Patent Literature

Two aspects make the recognition of non-patent literature particularly challenging: – The citation are embedded in natural language. False positive is a sensitive issue because typical bibliographical terms (such as journal, proceedings or conference) can be used in normal text and, similarly, titles and booktitles are lexically identical with normal text. In addition, sequences of references are possible and a segmentation of references within the same sentence is necessary. – Even when compared to usual references found in scientific articles, references to NPL present in patents show a strong versatility. Abbreviated references, in particular, are highly frequent and the usage of ”internal cue phrases” (such as ”Proceedings of” or ”Workshop on”, see [11]) for deciding if a reference occurs in a sentence has a low reliability. Very short references to handbooks, to technical manuals or to books with only a title and an author or a company name are not rare. Variation of citation patterns within the same patent document is very common as there is no guidelines nor standards for applicants with respect to NPL references. There is no template or tool for document composition such as BibTeX, and the verification of references is not a duty of the patent examiners. The low quality of NPL references in published patents in general would not be acceptable for any scientific publishers. Even in the unlikely hypothesis that some of the previous point could be solved in the future with better and stricter patent editing guidelines, the references exising in the accumulated patent collections would need to be processed as they are. Adaptation of the CRF model: In addition to the standard set of features, we introduced in the CRF model as specialized features booleans indicating if the word is part of sequence matching one entry in a list of scientific and technical journal names and abbreviated names. Both lists have been compiled from various online sources and cover more than 11 000 journals, mainly in the biological and medical fields. Evaluation. The evaluation on Table 4 follows the same methodology as for the patent reference blocks and distinguishes word and instance level performances.

Automatic Extraction and Resolution of Bibliographical References

131

Table 4. Evaluation results for the identification of non-patent references embedded in the patent text bodies Level

Precision Recall F1

Word 0.9056 Instance 0.8756

0.9219 0.9137 0.7341 0.7986

In the case of non-patent references, errors at the frontiers of the reference are more frequent. We can note that on instance level, the performance is lower than the step 1 task in [11] which aimed at selecting the sentences where an NPL reference was present among a set of 42 073 sentences taken from the prior art section of Japanese patent documents. [11] achieved a precision of 91.6% and a recall of 86.9%. Our task is, however, more complicated since it also involves the identification of the boundaries of the reference within the sentences. In addition, [11] uses more training data with a set of 1 476 containing a reference, as compared to 510 instances of NPL references in our case. The fact that our data set is multilingual is another difference which can explain the low recall. The performance remains better to the same task performed by [6,8], which both reported F1 score between 71 and 73%. More generally, a lack of training data is apparent in the study of the errors and of the coverage.

6

Resolution of Bibliographical References

6.1

Patent References

Our hypothesis is that the normalization of patent references can be done as a simple regular string rewriting, and, thus, can be implemented with Finite State Transducers6 (FST). We also suppose that the rewriting is deterministic and the usage of weights would not be necessary. Apart a few exceptions, these hypotheses appear valid. The reader not familiar with Finite State Machines can consult the state of the art litterature [12]. FSTs can realize regular string rewriting simply and very efficiently with a small set of operations. The formatting and the identification of bibliographical patent attributes are done in one step by a composition of FSTs. The main reference parser FST takes as input a patent reference text itself compiled as a linear FSA. Transitions of the input language are labeled by a string, a punctuation mark or a single digit. This input FSA is transformed into an identity FST (project operation) that is composed with the main FST encoding the reference parser. The parsed reference encoded as a FSA is obtained 6

Note that regular expressions with string rewriting, which are equivalent to FST, could also be used.

132

P. Lopez

by projecting the output of the FST resulting from the previous composition. For obtaining the final normalized patent reference, the output string is finally reordered in the following sequence: country code, number, kind code. An additional test indicates the type of reference, i.e. patent application or patent publication. In practice a whole set of reference strings can be combined as input, then determinized and minimized in order to factorize computations. The result itself is compacted and all the parsed strings can be obtained by enumerating the path of the resulting automata. The FST has been built manually based on patent information expertise and a validation on a development set of 1 500 patent references. The whole set of references was compiled into a large FSA that we combine with the FST iteratively for extending its coverage. The implementation of the FST processing has been realized with the OpenFST toolkit7 . For evaluating the accuracy of the FST on unseen patent references, we select randomly 250 references in the remaining part of our corpus. The FST fails in 7 cases, corresponding to an accuracy of 97.2%. After additional tests on other patent documents, we observed that the main source of errors are unseen terms and symbols for the issuing authority, i.e. problems of coverage and references to Japanese patent applications prior to 2000. The impact of OCR errors is very limited. We observed only errors for the Japanese term Kokai (for unexamined applications) which is often recognized as Kokal. Concerning the problem of the Japanese calendar conversion8, we use a separate ad hoc post processing in order to retrieve the correct year. An interesting aspect is that the FST can filter out incorrect patent reference blocks produced at the previous step if the spurious reference does not correspond to a valid issuing authority or number pattern. The usage of a FST for realizing this normalization task appears sufficient for covering the observed variations, given that the number of reference attributes is extremely limited. We considered that the identification of the three main components of a patent reference is by nature deterministic and not ambiguous. When an expert in patent information analyses a patent reference, he does not take complex decisions or solve ambiguities, but he applies a large amount of specialized knowledge regarding existing countries, numbers and kind codes. 6.2

Non-patent Literature

In contrast to patent references analysis, NPL references are highly ambiguous and involve a large number of bibliographical attributes. Our CRF model for this task is based on previous work [13] where CRFs are used for parsing scientific references in isolations. It uses a set of features closed to [10] in addition to additional dictionaries, see Table 2. The CRF model annotates the reference based on 7 8

http://www.openfst.org As noted in section 5.1, for Japanese patent applications only, dates are expressed using the Emperor eras. Sho for Hirohito starts in 1926, and Hei for Akihito starts in year 1989. In 2000, the Japanese Patent Office decided to use only the so called Christian era.

Automatic Extraction and Resolution of Bibliographical References

133

12 bibliographical attributes (title, author, date, booktitle, page, volume, issue, location, editor, institution, journal and publisher). An evaluation with the reference CORA dataset [14] showed an accuracy of 95,7% per citation field and 78.9% per citation instance, following the same experimental methodology as [9]. The instance accuracy for citations went up to 83.2% after consolidation of the results using CrossRef bibliographical web services. A relatively large amount of training data9 is available for this task. In total, 3277 training examples were used: – the CORA corpus corresponding to 500 tagged references, – the FLUX-CiM corpus containing 300 tagged references in computer science and 2 000 in the medical domain, – the CiteSeerX dataset containing 200 tagged reference, – a corpus of 177 references in the humanities (English), – an additional set of 100 references in French that we created in the context of [13]. The identified bibliographical attributes were finally normalized with regular expressions. For instance, the author names are formatted following the same sequence: first name, middle name and family name. This last step is also important in case of number and date information for ensuring a reliable final linking and resolution of the reference. Table 5. Evaluation results for the parsing of non-patent references taken in isolation (left) and automatically extracted from the patent text bodies (right) In isolation

In text bodies Prec. Rec. F1

Title Author Journal/Book Date Page Volume Issue Editor Publisher Location All Fields Instance

9

0.87 0.93 0.77 0.92 0.93 0.72 0.67 1.00 0.83 1.00 0.86 0.61

0.95 0.97 0.69 0.85 0.93 0.78 0.53 1.00 0.94 1.00 0.84 0.61

0.91 0.95 0.73 0.88 0.93 0.75 0.59 1.00 0.89 1.00 0.85 0.61

Prec. Rec. F1 Title Author Journal/Book Date Page Volume Issue Editor Publisher Location All Fields Instance

0.92 0.87 0.64 0.77 0.83 0.86 0.56 0.67 1.00 1.00 0.80 0.58

0.86 0.91 0.56 0.72 0.85 0.80 0.48 0.67 0.83 1.00 0.76 0.43

0.89 0.89 0.59 0.74 0.84 0.83 0.51 0.67 0.91 1.00 0.78 0.49

Note that the distinction between issue and volume has been added in the corpora that was not considering it, as, for instance, the CORA data set. The issue attribute is important for accurate automatic reference linking.

134

P. Lopez

We evaluated the parsing of non-patent references from patent documents by selecting randomly 120 references from our patent data set. After a manual annotation of the bibliographical fields of these references, we computed the standard metrics for the references taken in isolation and as identified by the previous extraction step. Table 5 summarizes the results at attribute field level and instance level. Institution was not considered because not enough occurrences were found in the evaluation set, and journal and book titles have been merged. These two evaluations indicate how the CRF model performs on the correct reference strings and what is the impact of the extraction within text. For reference strings taken in isolation, the accuracy of the CRF model is lower than for references coming from the reference CORA data set based on scientific articles. This difference supports our observation about the higher variability of non patent references in patent documents. At field level, the result is similar with the one obtained by [11] with CRF (86,2% precision and 85,4% recall) considering a lower set of five bibliographical attributes.

7

Conclusion

We have presented a bibliographical extraction component covering both patent and non patent references embedded in the text bodies of patent documents in a multilingual context. CRF models have been used first to identify the reference blocks, leading to a very high accuracy for the recognition of patent references and a reduction of the error rate of approx. 75% as compared to existing prior works. The identification of non-patent references appeared more challenging given our relatively small amount of training data but remained at a relatively high precision level. The creation of more training data for NPL references will be one of our future objective. In the next steps of our framework, the reference strings are parsed and normalized. As the patent references have only three bibliographical attributes and do not present ambiguities, a FST was used for this task, which turned out to be very effective. The non-patent reference strings were processed by a CRF specialized in the processing of references in isolation, exploiting a large amount of training data. In addition to these experiments, a new corpus of annotated patents have been created and we hope that, with its free release to the community, it will be easier to compare future works and approaches on similar tasks. We plan to use at a large scale this bibliographical analysis component for the CLEF IP 2010 evaluation Lab. Our goal is to exploit the citation contexts and more complex citation relations for improving the prior art search and the automatic classification tasks. Finally we are also experimenting Linear Chain CRF models for automatically identifying important structures of a patent document and for the automatic extraction of other entities of special interests such as physical quantities, tables and equations.

Automatic Extraction and Resolution of Bibliographical References

135

References 1. Lopez, P., Romary, L.: Multiple retrieval models and regression models for prior art search. In: CLEF 2009 Workshop, Technical Notes, Corfu, Greece (2009) 2. Nakov, P., Schwartz, A., Hearst, M.: Citances: Citation sentences for semantic analysis of bioscience text. In: SIGIR 2004 workshop on Search and Discovery in Bioinformatics, Sheffield, UK (2004) 3. Li, X., Chen, H., Zhang, Z., Li, J.: Automatic patent classification using citation network information: an experimental study in nanotechnology. In: Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, Vancouver, BC, Canada (2007) 4. Meyer, M.: Does science push technology? patent citing scientific literature. Research Policy 29, 409–434 (2000) 5. Wan, S., Paris, C., Dale, R.: Whetting the appetite of scientists: producing summaries tailored to the citation context. In: Proceedings of the 9th ACM/IEEE-CS joint conference on Digital libraries, Austin, USA (2009) 6. Agatonovic, M., Aswani, N., Bontcheva, K., Cunningham, H., Heitz, T., Li, Y., Roberts, I., Tablan, V.: Large-scale, parallel automaic patent annotation. In: Proceeding of the 1st ACM workshop on PAtent Information Retrieval (PAIR), Napa Valley, USA (2008) 7. Adams, S.: The text, the full text and nothing but the text: Part 1 standards for creating textual information in patent documents and general search implications. World Patent Information 32, 22–29 (2010) 8. Lawson, M., Kemp, N., Lynch, M.F., Chowdhury, G.G.: Automatic extraction of citations from the text of english-language patents - an example of template mining. Journal of Information Science 22, 423–436 (1996) 9. Peng, F., McCallum, A.: Accurate information extraction from research papers using conditional random fields. In: Proceedings of HLT-NAACL, Boston, USA (2004) 10. Councill, G., Lee Giles, C., Kan, M.Y.: Parscit: An open-source crf reference string parsing package. In: Proceedings of the Language Resources and Evaluation Conference (LREC 2008), Marrakesh, Morrocco (2008) 11. Nanba, H., Anzen, N., Okumura, M.: Automatic extraction of citation information in japanese patent applications. International Journal of Digital Library 9, 151–161 (2008) 12. Roche, E., Schabes, Y.: Finite-State Language Processing. MIT Press, Cambridge (1997) 13. Lopez, P.: Grobid: Combining automatic bibliographic data recognition and term extraction for scholarship publications. In: Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., Tsakonas, G. (eds.) Research and Advanced Technology for Digital Libraries. LNCS, vol. 5714, pp. 473–474. Springer, Heidelberg (2009) 14. McCallum, A., Nigam, K., Rennie, J., Seymore, K.: Automating the construction of internet portals with machine learning. Information Retrieval Journal 3, 127–163 (2000)

An Investigation of Quantum Interference in Information Retrieval Massimo Melucci University of Padua

Abstract. In the related literature and in particular in the recent book by van Rijsbergen, it was hypothesized that a more general framework used to formalize quantum mechanics, and then quantum probability, would be useful for going beyond the classical retrieval models. This paper first discusses a situation in which that framework, and then quantum probability, can be necessary in Information Retrieval and then describes the experiments designed to this end. The necessity of considering quantum probability stemmed from the experimental observation carried out in this paper that the best terms for query expansion have probability which does not admit classical probability and which instead can be defined within a quantum probability function.

1

Introduction

Information Retrieval (IR) science deals with the prediction that the documents retrieved store information relevant to the user’s needs. Although probabilistic models have largely been employed, they are of course not the only view through which IR has been modeled: logical and vector space models are other important views. Despite the differences between these views, it has been shown that the probabilistic, logical and vector space views are inter-related and can be combined in a single framework [1]. This framework has been used to describe the interference between measurements in Physics, that is, the phenomena in which two situations presumed mutually exclusive are in fact interfering and happening at the same time.1 Hence, van Rijsbergen’s book suggests the intriguing hypothesis that also IR is affected by the interference of measurements.2 The first pillar of van Rijsbergen’s framework is that information objects such as documents, queries or clusters are vectors, planes or in general vector subspaces in a complex Hilbert space. The other pillar of the framework is that the questions asked about these objects, such as “Is this document relevant?” or “Does this term occur in this query?” are projectors which map the subspace representing the object onto the line representing the question. The size of this projection is a probability. The most important implication of this framework is that the probability theory developed within it does not follow all the classical 1

2

In Physics, interference occurs when a photon passes through a slit and the same photon passes through the other slit at the same time. This immense field of knowledge is known as Quantum Mechanics.

H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 136–151, 2010. c Springer-Verlag Berlin Heidelberg 2010 

An Investigation of Quantum Interference in Information Retrieval

137

logical rules: it is this lack of classicity that allows the researchers in this research area to model interference. The hypothesis that interference may occur in IR is important because it would entail the necessity of revisiting the theory of probability in IR, thus paving the way toward a new view of probabilistic retrieval models. Essentially, the distributive law does not hold in quantum probability. For the sake of clarity, suppose the two observables “Is it relevant?” and “Does this term occur?” can be measured on a document, the answer of each being either “yes” or “no”. The distributive law implies the law of total probability (LTP) that says that the probability of an observed value, e.g. term occurrence, can be defined as the sum of the probabilities of two mutually exclusive events, i.e. the probability that the term occurs when the document is relevant and the probability that the term occurs when the document is non-relevant (see Section 6). According to quantum probability, LTP may not hold because the interference may occur (e.g. a document is relevant and the same document is non-relevant at the same time), thus “disrupting” the distributive law. This paper both experimentally and theoretically describes a situation in which quantum probability, which stems from the framework used to describe interference, may be necessary. The necessity of considering quantum probability in IR stemmed from the experimental observation carried out in this paper that the best terms for query expansion have probability of occurrence which does not admit classical probability and can be defined within quantum probability. The figure on the right displays the terms which produce an increase in effectiveness and that do not admit the LTP as the points outside the range [0, 1] on the horizontal axis, whereas the terms displayed within the range [0, 1] on the horizontal axis, which admit the LTP, do not always provide an increase in effectiveness measured on positive side of the vertical axis – this figure will be explained in Section 7.

2

Related Work

The double-slit experiment is perhaps the best known and investigated case of interference and of violation of the LTP in Physics – Feynman’s [2] and Hughes’s [3] works are two excellent descriptions of this phenomenon. There is a source E emitting photons which pass or do not pass through a screen with two slits A, B; those passing are detected by screen S (Fig. 1). When A is open and B is closed, the distribution of the photons detected by S at point y is depicted by curve a. When A is closed and B is open, the distribution of the photons detected by S is depicted by curve b. One would expect that when both slits were open, the distribution of the photons detected by S would be a mix of

138

M. Melucci

Fig. 1. The double-slit experiment; curves show distribution of hits in experiment a, experiment b and experiment c. From [2,3].

curves a and b, that is, a sort of pa + (1 − p)b, where p is the probability that a photon passed through A. On the contrary, the distribution observed when both slits are open is N (far right) in which the proportion of photons may be either less than or greater than pa + (1 − p)b – this was of course observed independently of p. Interference occurs between the photons when both slits are open and it causes the difference between what is expected from classical probability (i.e. the bell-shaped curve) and what is observed (i.e. the wave-shaped curve). In Physics, many interpretations were given to explain the discrepancy between the expected probability and the observed probability, but these are out of the scope of this paper. Due to its relevance, interference has been investigated in disciplines other than Physics. In particular, interference has been investigated in IR, although the studies are still at their very beginning. Melucci pointed out that there is a connection between a formal model for explaining contextual IR and quantum probability [4], whereas Bruza et al. investigated how the co-occurrence of words may be affected by quantum phenomena [5], thus producing meanings that cannot be accounted for by classical probability, and Aerts et al. and Hou and Song investigated how the co-occurrence of words may not reveal non-trivial correlations bu using entanglement (quantum correlations) [6,7]. More recently, Zuccon and Azzopardi claimed that the “both slits open” case resembles the situation of a user looking at two documents retrieved by a search engine and that the quantum-like probability ranking principle should be introduced for ranking these documents [8]. What is common to these works is the hypothesis that quantum theory is a useful framework because it is more general than the classical views of IR. In constrast, this paper aims at verifying experimentally if interference may occur in an IR situation, specifically, in query expansion or suggestion setting, and therefore at suggesting that quantum probability is necessary for modeling interference (at least, in the experimented IR situation). The approach of this paper is similar to that reported in [9] in which an experimental study was designed to demonstrate that interference occurs in human cognition through some small-scaled user studies. In constrast, the experiments of this paper were based on TIPSTER test collections.

An Investigation of Quantum Interference in Information Retrieval

3

139

Problem and Contribution

An argument supporting the use of quantum probability in IR is that, while it does include the classical case as a special case, it may be useful for describing and solving open problems in IR. This argument was given in [1, page 67], that is, if IR models are to be developed within the framework3 unifying the diverse views of IR and including quantum probability, then without further empirical evidence to the contrary it has to be assumed that the vector space logic fails the distributive law. As a consequence, the vector space logic fails the classical probability theory, thus suggesting the need to enlarge the view of probabilistic IR for embracing the quantum case. To our knowledge, the use of quantum probability in IR has only been addressed at the theoretical level. Therefore, the need to enlarge the view of probabilistic IR for embracing the quantum case has still to be demonstrated experimentally. As it is our opinion that the issue is whether quantum probability is really necessary (in IR, at least), this paper aims at providing empirical evidence to the hypothesis that there exists an IR situation in which the view of probabilistic IR has to be enlarged for embracing the quantum case. This paper focusses the quantum case on interference and applies the LTP for investigating quantum interference in an IR situation. Since interference is related to LTP, the violation of LTP will be used for observing interference in IR. As the LTP is implied by the distributive law, when the LTP is violated in an experiment, then the distributive law fails in that experiment, thus suggesting the need to develop IR models within a more general framework and as a consequence the need to accept quantum probability for IR. Therefore, this paper investigates LTP and interference through a series of experiments carried out by using large test collections.

4

A Brief Survey of Probabilistic IR

When a retrieval system is designed using the classical probabilistic models, a document is an element of an urn of elementary events for which one or more observables such as relevance or term occurrence can be considered. If relevance was known for every document, the probabilities of relevance and term occurrence of randomly drawn documents could be exactly calculated and the questions about the relevance of a given document could be answered with certainty. As some observables are unknown for most documents, training sets are built so that the probabilities can be calculated and used as estimations. The probabilities estimated from a training set obey the classical axioms of probability, namely, Kolmogorov’s axioms. This means that the events are represented as subsets of the training set and the probabilities of the events are expressed as measures of these subsets; the usual measure is the frequency relative to the number of elementary events. When the events are represented as subsets of 3

The framework is based on complex Hilbert vector spaces.

140

M. Melucci

(a) A open, B closed

(b) A closed, B open

(c) Either A or B open

Fig. 2. The histograms show the training distribution of term occurrence using the double-slit view

the training set and the probabilities of the events are expressed as measures of these subsets, two facts occur. First, Bayes’ postulate can be used because the intersection of two events (term occurrence and relevance) can be calculated – this intersection can be calculated because both observables have been observed. Second, the LTP holds and the probability that a term occurs in a document is the sum of the probabilities that either a document is relevant and the term occurs or a document is not relevant and the term occurs. Once the system has been trained, it can be tested and thus, for instance, retrieval algorithms or weighting schemes can be evaluated. However, the user, the query, the search task, the location or other contextual properties of the testing phase are different from those of the training phase. Indeed, at the testing phase, the relevance assessments about many documents may be unavailable, the user interface has changed or the user is reluctant to provide explicit relevance ratings. However, even at the testing phase, a user is supposed to provide a term representing his information need and the documents are retrieved or reranked according to that evidence (e.g. query suggestion). As relevance cannot be observed in this context, the probability of relevance cannot be calculated, but only estimated. To this end, the probabilities calculated through the training set can be used, that is, the probabilities of the unobserved relevance are estimated. The estimation of the probabilities of the unobserved relevance passes through Bayes’ theorem which updates the prior probability of relevance to the posterior probability of relevance through the likelihood of relevance. The constraint required by Bayes’ theorem is that the sum of the likelihood of relevance and of the likelihood of non-relevance, respectively multiplied by the prior probability of relevance and of non-relevance, equals the probability of term occurrence. This is equivalent to saying that the LTP must hold. If it did not hold, the sum of the posterior probability of relevance and that of non-relevance would not be one. Although the odds (i.e. the ratio between the posterior probability of relevance and that of non-relevance) may be used and the probability of term occurrence “disappears”, the constraint required by Bayes’ theorem still holds because the ratio is calculated between two probabilities which must sum to one.

An Investigation of Quantum Interference in Information Retrieval

5

141

Trying to Depart from Classical Probabilistic IR

The departure from classical probabilistic IR begins with the violation of the LTP. Fig. 2 gives a pictorial description of how the double-slit experiment may view training in IR. There is a source E emitting a document which may include a term (e.g. “java”). The document passes or does not pass through a slit of a screen with two slits A (i.e. relevance) and B (non-relevance). The term occurrence is detected by the screeen S: if the document “arrives” at the screen, the term occurs in the document. When the document is relevant, A is open, B is closed and the distribution of the relevant documents indexed by the term (i.e., detected by S) is depicted by histogram a of Fig. 2 (a). When the document is not relevant, A is closed, B is open and the distribution of the non-relevant documents indexed by the term (i.e., detected by S) is depicted by histogram 4 . When both slits are b of Fig. 2 (b). In the figure, P (X|A) = 38 , P (X|B) = 12 open, one would expect that the distribution of the terms in the documents de12 4 7 8 3 tected by S would be a mix of histograms a and b, that is, 20 8 + 20 12 = 20 , 8 the fraction 20 being the probability that a document is relevant, that is, the probability that the slits are arranged according to (a) – of course, (a) and (b) are mutually exclusive. The mix is the application of the LTP. This situation is depicted in Fig. 2 (c), albeit it should be noted that the distribution c is produced by assuming that either A or B is open when the document passes through the slit and S detects the document. Suppose that, for example, the user examined a set of documents returned by a search engine and wants to filter the documents including an additional term (e.g. “java”). The systems compute the probability of relevance of the documents including the term and uses Bayes’ theorem to this end. The probability of relevance is P (A|X) = 37 and assumes the LTP such that the probability of 7 the term is 20 . If other candidate terms are possible, the system can compute different probabilities of relevance and rank the terms by this probability. If the training set is considered for estimating the probability of relevance, the LTP is not violated because all the estimated probabilites and the probabilistic space built upon the training set admit Kolmogorov’s axioms. Suppose the system has to select terms for expanding a query or for suggesting them to the user. It can use the probabilities estimated during the training. However, it may well be that the system has or wants to exploit further information about the term and that the probability of term occurrence is no longer 7 20 . Three reasons are possible (other reasons may be imagined). First, the training set may not be or may no longer be a representative sample from which the probabilities are estimated. This inadequacy may occur when the sample used to draw relevance assessments is very small or not drawn at random – the frequency of “java” within twenty top-ranked assessed documents may simply be inappropriate if it is compared with the frequency observed in, say, a list of 1,000 documents or a list of twenty randomly ranked documents. Second, if term occurrence were viewed as term usage as recorded in query log files available to the search engine, the difference between the estimations might be due to the fact that the users would tend to use the term much more (or much less)

142

M. Melucci

frequently than observed in the training phase, thus making the probability of term usage very different from the linear combination of the conditional probabilities implementing the LTP. In other words, the probability that a user decides to use that term to express his information need is different from the probability estimated from the training set. Third, the difference between the estimations of the probability of term occurrence based on data that are different from those used at the training phase may be due to the evolution of the notion of relevance in the user’s mind over time (if he made the initial assessment) or due to the difference from that encoded in the training set (if the assessment was made by a different person). This means that if the user were asked to assess the relevance of the documents not yet assessed, the proportion of relevant documents would be different from those estimated from the training set. Suppose that the probability of occurrence estimated on the basis of these additional information is 12 , thus failing the LTP. This case is depicted in Fig. 3 in which both slits are open. If the Bayes theorem was used anyway, the two posterior probabilities (of relevance and that of non-relevance) would not sum to 1, that is, the LTP is violated. When the LTP Fig. 3. The histogram show the distribution of term occurrence using the is violated, the probability of relevance double-slit view when both slits are cannot be calculated with the Bayes theo- open rem. However, the difference between the probability of relevance computed when the LTP holds and the probability of relevance computed when the LTP is violated, is not really a probability since it may be negative, but it is rather called interference term in quantum probability. When both slits are open, a document is both relevant and non-relevant at the same time. As above mentioned, this is impossible according to the usual logic [1]. Interference may cause the difference between what is expected from the classical probability and what is observed and cannot be modeled using the 7 − 12 classical theory of probability. In the previous example, this difference is 20 and cannot be viewed as an “estimation error”, on the contrary, it can be exploited for investigating the relationship between the violation of the LTP (i.e. the occurrence of an interference term) and the variations in retrieval effectiveness. This is in fact the approach of this paper which is analyzed in Section 6 and implemented in Section 7.

6

A Mathematical Analysis

Let us consider the problem from a mathematical point of view. The problem is, what happens if the probability of term occurrence were estimated using the data not observed during the training phase. The point is that the difference between the probabilities of term occurrence may be so large that the LTP can no longer be applied even when taking account into some estimation errors

An Investigation of Quantum Interference in Information Retrieval

143

which can affect the calculation of these probabilities. If the LTP can no longer be applied then no value of the prior probability of relevance can be used (in the real interval between zero and one, of course) to linearly combine the likelihoods of relevance and obtain the probability of term occurrence. This result was highlighted by Accardi in [10, page 306] and can mathematically be stated as follows. Suppose a training set was built so that A, B means relevance and non-relevance, respectively, and X means term occurrence: P (A) + P (B) = 1 P (X ∧ A) + P (X ∧ B) = P (X)

P (X ∧ A) = P (X|A)P (A) P (X ∧ B) = P (X|B)P (B)

(1)

Thus, the LTP can be expressed as P (X) = P (X|A)P (A) + P (X|B)P (B)

(2)

Suppose the likelihoods P (X|A), P (X|B) are given because they are experimentally observable from the training, the probability P (X) is given because it is experimentally observable from the testing while P (A), P (X ∧ A) and P (X ∧ B) are the unknowns. After a few passages, one can check that a solution of the Algebraic System 1 exists if and only if 0≤

P (X) − P (X|B) ≤1 P (X|A) − P (X|B)

(3)

the latter being called statistical invariant. However, the statistical invariant states the conditions such that any set of documents can be partitioned into relevant and non-relevant document subsets such that the measures of the two subsets are those in which the training set is partitioned. One can also verify that the statistical invariant holds if the probabilities and the likelihoods are estimated from a single training set. This is not the case if the likelihoods P (X|A), P (X|B) have been estimated from the training set (e.g. 3/8 and 4/12, respectively), while P (X) has been estimated with additional information (e.g. 1/2) as depicted by Fig.s 2 and 3. The statistical invariant (Equation 3) is related to the LTP (Equation 2) because if the latter is valid, the former is valid too, and vice versa. From a mathematical point of view, when the LTP does not hold, the most interesting consequence is that the LTP cannot be admitted for any probability of relevance, that is, there may not exist any probability of relevance P (A) such that the Algebraic System has solution. This may happen because the experimental conditions in which the training and testing probabilities may be different to each other. This means that the estimation of the probability of term occurrence during the testing phase has been performed on the basis of the existence of an event space which cannot be partitioned into a subset of events corresponding to the relevant documents and a subset of events corresponding to the non-relevant documents given that the probability of term occurrence in a relevant document is P (X|A) and the probability of term occurrence in a non-relevant document is P (X|B).

144

M. Melucci

The impossibility of partitioning the document space into a subset of events corresponding to the relevant documents and a subset of events corresponding to the non-relevant documents, once P (X|A) and P (X|B) are given, appears quite surprising since it is customary in IR to accept the fact that if a term occurs in a document, the document is either relevant or non-relevant whenever it is assessed. If the LTP cannot be admitted, it has been seen that a probability of relevance P (A) cannot be defined so that the probability of term occurrence can be expressed by the probabilistic model estimated from the training set. From an IR point of view, the inexistence of a prior probability of relevance entails a sort of impossibility situation: If a user were asked to assess the relevance of all the documents indexed by the term used when interacting with the system and P (X) does not admit Equation 3, there would not be any set of relevance assessments given by the user which can lead to a distribution of term occurrences and relevance assessments so that the likelihoods were those measured from the training set. The difference P (X) − (P (X|A)P (A) + P (X|B)P (B)) is called interference term.4 Although the violation of the LTP may seem a negative result, it may in contrast be leveraged to investigate how the violation of the LTP and then the possibility of interference are related to retrieval effectiveness. The next section in fact explains how the violation of the LTP can provide insights on the improvement of retrieval effectiveness. It is this violation that allows us to exploit quantum interference for improving retrieval effectiveness.

7

Experimental Investigation

In this section, the relationship between the violation of the LTP, that is, the violation of the statistical invariant given by Equation 3, and the effectiveness of a retrieval system is investigated. 7.1

Objectives and Research Questions

More specifically, the experimental question was: If the term suggested by the system to the user to expand the original query was so that the probability of occurrence violates the LTP, is the retrieval effectiveness measured on the new list of retrieved documents higher than that measured on the original list of retrieved documents? In order to answer the experimental question, some requirements were defined when designing the experiment. First, a query expansion scenario was hypothesized. Suppose a user was interacting with a retrieval system by issuing some queries in order to be provided with documents relevant to his own information need. The user queries were 4

Without entering into details, in quantum probability, the interference term is the product between the roots of the observed probabilities and the cosine of the angle between the vector representing A and the vector representing B. As this cosine ranges between −1 and +1, the LTP can be violated and interference can be modeled.

An Investigation of Quantum Interference in Information Retrieval

145

8manexT3D1N0, acsys8aln2, anu6alo1, att98atdc, att99atde, bbn1, Brkly21, Brkly22, Brkly26, city6al, CL99SDopt2, Cor6A3cll, Cor7A3rrf, Flab8as, Flab8atd2, Flab8atdn, Flab8ax, fub99td, ibmg97b, ibms99a, INQ502, mds602, mds98td, Mercure1, MITSLStd, ok7ax, ok8amxc, pir9Attd, pirc7Aa, pirc8Aa2, tno7exp1, tno8d3 Fig. 4. The TREC 6, 7, 8 runs used in the experiments as state-of-the-art

mainly informational, that is, there were likely more than one relevant documents and the query could not describe exactly what the user was looking for. After the system retrieved the original list of documents, the user was prompted to add a keyword to his original query and the expanded query was given as input to the system for retrieving a new list of documents. The keyword added by the user had the aim of keeping the documents indexed by it in the original document list, whereas those not indexed were not redisplayed to the user. Second, the experiments was designed to make them reproducible. To this end, the experiments were a simulation of the interaction between the user and a search engine and were performed in a laboratory setting by using test collections, so users and queries were simulated by using test documents, topics and relevance assessments. Discs 4 and 5 of the TIPSTER test collection were used which comprises the Federal Register (FR), the Congressional Records (CR), the Los Angeles Times (LA) and the Foreign Broadcast Information Service (FBIS) document sets, in total, more than 500,000 documents. The TREC 6, 7 and 8 topic sets were used for the experiments, in total, the 150 topics numbered from 301 to 450. Third, it was considered as requirement to adopt the state-of-the-art of IR and in particular the best IR systems tested within TREC with the largest set of topics. To this end, the best runs were selected from each of the set of runs submitted to TREC-6, TREC-7 and TREC-8, in total 32 runs (i.e. original document list). It was therefore assumed that the user was interacting with the “best” search engines available at that time. The adoption of the state-of-the-art allowed us to use quite a wide range of retrieval models and techniques and to observe the variation of effectiveness with respect to a relatively high level of effectiveness. The run tags are listed in Fig. 4. 7.2

Design of the Experiments

When implementing the experiments, it was necessary to implement the scheme depicted by Fig.s 2 and 3. Hence, the emitter E was the source from which the documents are retrieved – it can be thought as the device producing the original list of retrieved documents. The screen corresponds to the method for selecting the term. In this paper, two types of term selection were used. An automatic method and a manual method. For each run and for each topic, the automatic method was implemented as the algorithm which extracted the k most frequent terms occurring in the n top-ranked documents. The manual method was implemented as the algorithm which simply extracted the keywords from the description field of the topic under the hypothesis that if the user were asked

146

M. Melucci

to provide a term to better explain his information need, he would select a term from the description field which is indeed the field implemented to this end. In both cases, i.e. automatic and manual methods, the term was used as a boolean operand to filter all and only the documents retrieved by the run and indexed by the term. This means that a document includes the term if and only if the screen “detects” the term. As for the slits (i.e. A and B), the relevance assessments given to the n topranked documents were used for estimating the training probabilities of relevance, provided N documents were retrieved. All the documents ranked after the n-th position were considered as unassessed. Hence, when A was “open” and B was “closed”, the relevant documents indexed by the term and ranked among the top n were counted, when A was “closed” and B was “open”, the non-relevant documents indexed by the term and ranked among the top n were counted, and when both slits were open, the retrieved documents indexed by the term were counted, thus estimating a probability of term occurrence that may violate the LTP. Hence, the following probabilities could be estimated: Pn (X|A) =

NA (n, X) NA (n)

Pn (X|B) =

NB (n, X) NB (n)

P (X) =

N (X) N

(4)

where, for each run and topic, NA (n, X) is the number of relevant documents indexed by term X and retrieved in the n top-ranked, NB (n, X) is the number of non-relevant documents indexed by term X and retrieved in the n top-ranked, NA (n) is the number of relevant documents retrieved in the n top-ranked, NB (n) is the number of non-relevant documents retrieved in the n top-ranked, N (X) is the number of documents indexed by term X. Once these probabilities were estimated, for each term, for each run and for each topic, the statistical invariant was computed as follows: SIn (X) =

P (X) − Pn (X|B) Pn (X|A) − Pn (X|B)

(5)

The statistical invariant was considered in this paper because it provides a necessary and sufficient condition to claim whether the probability of occurrence of the term selected for filtering and re-ranking the retrieved documents can be compatible within the same sample space defined by the set of relevant and non-relevant documents used for estimating the probability of relevance. Note that P (X) is intentionally estimated from another sample because the experiments aims at studying what happens when the classical probability axioms are violated in terms of retrieval effectiveness and if the variation in effectiveness can be associated to the violation “measured” by the interference term. Note also that the probabilities estimated are not parameters of a family of probability distributions, but they are the actual values which have to admit the classical probability axioms and the LTP. If they do not admit them, they have to be estimates of probabilities admitted by another theory. Furthermore, for each term, for each topic and for each run, the original list of documents was filtered by removing the documents not indexed by the term

An Investigation of Quantum Interference in Information Retrieval

147

Fig. 5. Scatterplots of P (X) (left), P (X|A) (middle), P (X|B) (right). Probabilities are on the horizontal axis, ∆n (X) is on the vertical axis.

and was ranked by the same score provided in the original run, that is, the relative rank of the selected documents was kept. The Average Precision (AP1 ) was then computed for the new, filtered list of documents and compared with the AP0 computed for the original list of retrieved documents (that is, the run). For each term, for each run and for each topic, the comparison between the APs was summarised as:5 A P 1−A P 0 ∆ n (X ) = (6) A P 0 Fig. 5 is an example scatterplot of how the observed probabilities (vertical axis) are distributed with respect to ∆ n (X ) (horizontal axis). The values of P (X ) are concentrated on the [0, 0.5] sub-interval because of, probably, the retrieval algorithms of the runs which include documents represented by low frequency terms. The values of P (X|A) depends on the number of assessed documents – a low number of assessed documents produces a low number of probability values. The same holds for P (X|B). 7.3

Results

For each topic, for each term and for each run, it was possible to relate SIn (X) with ∆n (X) and plot these pairs to produce a scatterplot where a point corresponds to a triple (run, topic, term). From now on, in every scatterplot, the horizontal axis refers to SIn (X) and the vertical axis refers to ∆n (X). Fig. 6(a) is the scatterplot for n = 20 and manual query expansion (i.e. the terms were selected from the topic description and added to the topic title one at a time). The scatterplot has a specific shape. Most of the points lie above the zero, that is, on the positive side of ∆n (X). This result is not surprising since it was already 5

Equation 6 was not normalized between 0 and 1 in order to make the scatterplots more visible (indeed, a normalization would have vertically squeezed the scatterplots and would have displayed them as a confused mass of points).

148

M. Melucci

(a)

(b)

Fig. 6. The scatterplot of SIn (X), ∆n (X) for n = 20 and manual query expansion and for n = 20 and automatic query expansion (k = 10)

known that filtering documents with terms selected from the description field of a TREC topic produces an improvement of the retrieval effectiveness. Another result is that the increase in effectiveness is concentrated around ∆n (X) = 0. This result is also not surprising since it may be due to the large difference between P (X|A) and P (X|B). The surprising outcome displayed by Fig. 6(a) is the set of points placed on the negative side of ∆n (X), that is, the query expansion method caused a decrease in effectiveness when the statistical invariant SIn (X) was between 0 and 1 (when SIn (X) was greater than 1 or less than 0, the AP increased) which happened when the LTP was not violated. This result suggests a preliminary hypothesis of relationship between interference, LTP and variation in retrieval effectiveness. It may be that the specific shape of the scatterplot was due to the selection method used to filter the documents. However, the shape was confirmed after replacing the “manual” method for selecting the terms to expand the query (i.e. the terms are extracted from the description topic field) with an automatic methods as follows: The 20 most frequent terms were selected from each document retrieved by the run and ranked among the n top-ranked. Then, the k most frequent terms were selected from the union of the 20-term sets – this union would be a representation of the main topics covered by the top n ranked documents retrieved by the run. The probabilities needed to compute the statistical invariant were based on the relevance assessments of the n top ranked documents. For n = 20, k = 10, the outcome is depicted in Fig. 6(b) which confirms the shape and the outcome pointed out by Fig. 6(a), which is based on the “manual” term selection method. However, there are less points in Fig. 6(b) than in Fig. 6(a) because there were many automatically selected terms for which SIn (X) was in the form x/0. The other variable was the number n of top ranked documents from which the probabilities of relevance (in fact, those were likelihoods) and then the statistical invariant were computed. Hence, the specific shape can be due to n or k. To test this hypothesis, the experiments were performed with n = 5 and k = 10, thus using the automatic term selection method. The results are displayed in Fig. 7,

An Investigation of Quantum Interference in Information Retrieval

149

Fig. 7. The scatterplot of SIn (X), ∆n (X) for n = 5 and automatic query expansion (k = 10)

Fig. 8. The scatterplot of SIn (X), ∆n (X) for n = 10 and automatic query expansion (k = 20)

thus confirming the shape of the scatterplot albeit with some differences; in particular, the points on the negative side of ∆n (X) are concentrated toward SIn (X) = 1. The shape is more clear in Fig. 8 which displays the case with much more terms n = 10, k = 20 than in the previous figures. In this figure, many terms are effective when SIn (X) < 0. 7.4

Discussion

The experimental question asked in this section was: If the term suggested by the system to the user to expand the original query was such that the probability of occurrence violates the LTP, is the retrieval effectiveness measured on the new list of retrieved documents higher than that measured on the original list of retrieved documents? The experiments performed in this research and reported in this section have suggested a relationship between the variation in AP and the statistical invariant

150

M. Melucci

introduced in [10] for checking whether experimental data can be explained by one classical probabilistic model (that is, the LTP is not violated) or not. The results have suggested that if the statistical invariant cannot be admitted and the LTP is violated, that is, if the selected term cannot be drawn from a sample whose probabilities have been estimated by a set of relevant and non-relevant documents, then the increase in retrieval effectiveness is very likely. In contrast, if the probability of occurrence of the term selected for filtering and re-ranking the retrieved documents does not violate the LTP, the increase in retrieval effectiveness is less likely. The difference between estimations is physiological in Statistics when different sets are used for this purpose – different datasets lead to different estimation. In this paper, this difference is not only an element to be considered when deciding as to whether to retrieve a document or not, but it may related to a quantum-like interference. Furthermore, the results and the analysis of this paper attempt to view the role played by IDF within the unifying framework which also includes quantum probability. To some extent, the experimental results can be explained using the fact that a low value of the probability referred to by situation c of Fig. 2 corresponds to igh IDF – when P (X) is low, the term IDF is high, thus promoting the term to be a good candidate for query expansion. However, the scatterplots show that the increase in retrieval effectiveness is also likely when P (X) is high and P (X|A) − P (X|B) is small (SIn (X) > 1), thus allowing us to think that there are other factors influencing the variations in AP. It is well known that other factors than IDF can be used for predicting query performance – in this paper, it is only noted that these “hidden” variables may be related to what is called interference. Therefore, further analysis is necessary especially in relation to the research carried out in query performance prediction.

8

Concluding Remarks and Future Directions

The previous research work which investigated quantum probability in IR approached the problem by postulating the occurrence of quantum phenomena and in particular interference in IR before applying this hypothesis to some scenario. However, the current literature has not yet shown that quantum phenomena occur in IR – one can rather speak about quantum-like phenomena. The question asked at the beginning of this paper was whether quantum probability, in general, and interference, in particular, is necessary in IR and not only useful for making the probabilistic framework adopted in IR more general and hopefully more effective. This paper addressed the question both theoretically (Section 6) and experimentally (Section 7). Although the experimental results and the discussions in this paper are of course still at an early stage, they support the hypothesis that a more general framework than classical probability is necessary. This more general framework should be based on quantum probability because it may formally incorporate an interference term which explains why the LTP can be violated. In particular, when the interference term is large to violate the statistical invariant, the increase in retrieval effectiveness is higher if measured in terms of

An Investigation of Quantum Interference in Information Retrieval

151

the variation in AP. Our future research work will define the quantum probability retrieval function upon this result. Of course, this conclusion does not imply that other theories of probability are useless or unnecessary. This paper cannot either argue that quantum phenomena occurs in IR as they do in Physics. However, the paper described a situation in which the violation of Accardi’s statistical invariant, which was developed to test the admissibility of quantum probability in Physics, is related to the variation in retrieval effectiveness. It is difficult to say that this is due to quantum-like phenomena, however, it is interesting to note that the “both slits open” situation corresponds to unassessed documents, thus suggesting that relevance is a property of the document only once the document is observed and any relevance values cannot exist before assessment or even before a query. This suggestion was made also in [1, page 20], however, it is a philosophical topic of profound debate in Physics too. Another natural continuation of this work is to analyse this topic in depth in IR. In this paper, the differences between topics or between runs were not discussed. This will be addressed in future work.

References 1. van Rijsbergen, C.: The Geometry of Information Retrieval. Cambridge University Press, UK (2004) 2. Feynman, R., Leighton, R., Sands, M.: The Feynman lectures on Physics. AddisonWesley, Reading (1965) 3. Hughes, R.: The structure and interpretation of Quantum Mechanics. Harvard University Press, Cambridge (1989) 4. Melucci, M.: A basis for information retrieval in context. ACM Transactions on Information Systems 26(3) (2008) 5. Bruza, P., Cole, R.: Quantum logic of semantic space: An exploratory investigation of context effects in practical reasoning. In: We Will Show Them! Essays in Honour of Dov Gabbay, vol. 1, pp. 339–362. College Publications (2005) 6. Aerts, D., Gabora, L.: A theory of concepts and their combinations II: A Hilbert space representation. Kybernetes 34, 176–205 (2005) 7. Hou, Y., Song, D.: Characterizing pure high-order entanglements in lexical semantic spaces via information geometry. In: Bruza, P., Sofge, D., Lawless, W., van Rijsbergen, K., Klusch, M. (eds.) QI 2009. LNCS, vol. 5494, pp. 237–250. Springer, Heidelberg (2009) 8. Zuccon, G., Azzopardi, L., van Rijsbergen, C.: The quantum probability ranking principle for information retrieval. In: Azzopardi, L., Kazai, G., Robertson, S., R¨ uger, S., Shokouhi, M., Song, D., Yilmaz, E. (eds.) ICTIR 2009. LNCS, vol. 5766, pp. 232–240. Springer, Heidelberg (2009) 9. Busemeyer, J.R.: Introduction to quantum probability for social and behavioral scientists. In: Bruza, P., Sofge, D., Lawless, W., van Rijsbergen, K., Klusch, M. (eds.) QI 2009. LNCS, vol. 5494, pp. 1–2. Springer, Heidelberg (2009) 10. Accardi, L.: On the probabilistic roots of the quantum mechanical paradoxes. In: Diner, S., de Broglie, L. (eds.) The wave-particle dualism, pp. 297–330. D. Reidel pub. co., Dordrechtz (1984)

Abstracts versus Full Texts and Patents: A Quantitative Analysis of Biomedical Entities Bernd M¨ uller1,2 , Roman Klinger1 , Harsha Gurulingappa1,2, Heinz-Theodor Mevissen1 , Martin Hofmann-Apitius1,2 , Juliane Fluck1 , and Christoph M. Friedrich1 1

2

Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Schloss Birlinghoven, 53754 Sankt Augustin, Germany Bonn-Aachen International Center for Information Technology (B-IT), Dahlmannstraße 2, 53113 Bonn, Germany

Abstract. In information retrieval, named entity recognition gives the opportunity to apply semantic search in domain specific corpora. Recently, more full text patents and journal articles became freely available. As the information distribution amongst the different sections is unknown, an analysis of the diversity is of interest. This paper discovers the density and variety of relevant life science terminologies in Medline abstracts, PubMedCentral journal articles and patents from the TREC Chemistry Track. For this purpose named entity recognition for various bio, pharmaceutical, and chemical entity classes has been conducted and the frequencies and distributions in the different text zones analyzed. The full texts from PubMedCentral comprise information to a greater extent than their abstracts while containing almost all given content from their abstracts. In the patents from the TREC Chemistry Track, it is even more extrem. Especially the description section includes almost all entities mentioned in a patent and contains in comparison to the claim section at least 79 % of all entities exclusively.

1

Introduction

The bibliome has been considered as the corpus of abstracts in the biomedical domain because of the assumption that they contain the condensed information of a document. The automated extraction of information from full texts and patents has hardly been addressed because of intellectual property restrictions, rare availability and a vast variety of file formats. Technically, abstracts can be processed straightforward and they are broadly available [7] within repositories such as Medline1 . PubMed currently contains about 19 million citations. PubMed Central2 (PMC) is an archive containing about 1.9 million full texts that are freely available under open access. A subset of about 160,000 full texts is available for text 1 2

http://www.pubmed.gov, last accessed December 2009 http://www.pubmedcentral.gov, last accessed December 2009

H. Cunningham, A. Hanbury, and S. R¨ uger (Eds.): IRFC 2010, LNCS 6107, pp. 152–165, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Abstracts versus Full Texts and Patents

153

mining, while the main textual characteristics of this subset are the same as on the full repository [17]. Prior analyzes on the diversity of information in full text documents were done based on the distribution of keywords extracted from the Medical Subject Headings (MeSH)3 . The extraction of single keywords from MeSH reveals that their distribution amongst full texts is quite heterogeneous [16]. The classification of MeSH keywords into 4 different classes (Organism, Diseases, Chemicals/Drugs, and Genes) shows that the highest density of entities is found in the abstracts. Contrastingly, the full texts contain two times more non-unique biomedical entities than the abstracts [15]. The European Patent Office (EPO) has started to provide access to patents by the Open Patent Services4 (OPS), a web service to search and retrieve patents. The search for patents is currently restricted to a maximum of 40 retrieved patent identifiers. Patents retrieved via OPS are only available as scanned images or as images embedded in the Portable Document Format (PDF). Google Patents5 provides a service for search and retrieval from about 7 million patents from the United States Patent and Trademark Office (USPTO). An automated retrieval of patents with Google Patents is not straightforward. Access to 60 million patents in PDF from the EPO, the USPTO and the Japanese patent office is provided by esp@cenet6 [18]. A search with esp@cenet is restricted to a maximal result of 500 patents. The automated processing of patents retrieved from OPS, Google Patents, or esp@cenet demands the application of optical character recognition. Therefore, structural information as e. g. sections such as title, abstract, descriptions or claims is not available. The Text REtrieval Conference (TREC)7 supports research in the field of information retrieval by large-scale evaluations of text retrieval methodologies. The first domain specific track, the TREC Genomics Track, was conducted from 2003 until 2007. Full texts from High Wire Press8 were used as resource for answering biological questions posted by domain experts. In 2009, TREC started a domain specific track in chemistry namely TREC-CHEM providing 1.2 million patents as a working corpus. These patents are provided in an Extensible Markup Language (XML) format. This allows for analyzes of information in different sections. The combination of full text searches with named entities allows entity, semantic and ontological search. Entity search solves the problem of synonyms in search, e. g. “Give me all documents mentioning Aspirin or its synonyms”. Semantic and Ontological search [4], allow to search for semantic concepts like: “Give me all documents mentioning an analgesic”. The application of semantic searches as combination of the full text search with controlled vocabularies has been shown to improve the information retrieval performance [5]. This paper 3 4 5 6 7 8

http://www.nlm.nih.gov/mesh/, last accessed December 2009 http://ops.epo.org, last accessed December 2009 http://www.google.com/patents, last accessed December 2009 http://ep.espacenet.com, last accessed December 2009 http://trec.nist.gov, last accessed December 2009 http://www.highwire.org, last accessed December 2009

154

B. M¨ uller et al.

assesses the diversity of information content in abstracts, full text publications and patents based on the occurrence of different entity classes of biomedical entities such as pharma terms, drug names, genes/proteins, chemical compounds, single nucleotide polymorphisms (SNPs), and diseases. In addition, the distribution of different entity classes amongst various sections of the documents is analyzed. This allows for the comparison of the terminology content of Medline abstracts, PMC full text and a huge chemical patent corpus. Through the analysis of terminology distribution we could obtain an insight of the sheer amount, the variability, and density of terminology we are dealing with in the different corpora. Such numbers are published in this communication for the first time. This quantitative analysis should be followed by a qualitative analysis of the improvements in retrieval performance due to the inclusion of named entities, which is out of the scope of this article.

2 2.1

Methodology Corpus Preparation

Medline, PMC, and TREC-CHEM patents in XML-format were used as resources for the generation of the corpora. The different sections of the Medline citations, the PubMed Central full texts, and the TREC-CHEM patents are clearly distinguished in the structure of the XML document. The Medline citations provide meta-information as the position of the title and abstract in the document. The PubMed Central full texts contain meta-information as title, abstract, body and background. Sections in full texts such as introduction, methodology, results, conclusion and discussion are not always provided in the XML structure. The TREC-CHEM patents have meta-information like the title, abstract, claims and description. The most useful meta-information in Medline documents include annotations from MeSH whereas the documents in PMC and TREC do not have such an annotation. However, the TREC-CHEM patents have IPC classification codes. Medline currently contains about 19 million citations. The citations from Medline that contain a title and an abstract have a count of 10,294,069 as of 2009-11-15. In PubMed Central, about 1.6 million full texts are available for searching whereas only 10 % of the full texts are available for text mining. Partially, PubMed Central full texts contain documents that are not a subset of Medline, i.e., do not have a valid Medline identifier. These documents are excluded from the analyzes resulting in a corpus of 132,867 full texts as of 2009-07-15. The TREC-CHEM working corpus consists of 1,201,231 patents. 2.2

Named Entity Recognition

In order to identify biomedical entities in the text, the ProMiner system [6] and a tagger based on Conditional Random Fields (CRF) [11] are applied for named entity recognition. In the BioCreAtIve I challenge, task 1b [7] as well as in the

Abstracts versus Full Texts and Patents

155

BioCreAtIve II challenge [1], all runs of ProMiner as well as our workflow of CRF for named entity recognition were in the first quartile of the scoring results. The following biomedical entity classes are used in the analysis: Chemical Names (Chem). The dictionary of chemical names is generated by merging the information of compounds from DrugBank9 [19], KEGG [9,10] Compounds10 and KEGG Drugs11 . Similar entries within the different databases were merged based on InChi or CAS similarity. The dictionary for trivial chemical names has 23,575 entries. The dictionary contains generic names, synonyms, systemic names, semi-systemic names, abbreviations, formulae, brand names, and company codes of the chemical compounds of biomedical interest. IUPAC and IUPAC-like Names (IUPAC). The dictionary of chemical names has a limited coverage of IUPAC-like expressions [14]. Therefore, a CRF-based IUPAC tagger is used for the identification of IUPAC-like names [13]. The tagger performs a syntactical normalization of recognized entities based on string similarity. For instance, [2-(carbamoyloxymethyl)-2methylpentyl] carbamate and [2-Carbamoyloxymethyl-2-Methylpentyl] Carbamate are normalized to one entity. Disease Names (Disease). The identification of disease terms is performed with a MeSH disease dictionary. MeSH12 is a biomedical thesaurus with nearly 100,000 concepts that are hierarchically ordered. The MeSH sub-hierarchy corresponding to the class Diseases contains names and synonyms of symptoms, signs, disorders, and syndromes. For example, terms like Parkinson and Alzheimer Disease are found in this subtree of MeSH. The MeSH disease dictionary contains 4,444 entries. Pharmacological Terms (Pharma). The Anatomical Therapeutic Chemical (ATC) classification system is the base for the dictionary of pharmacological terms. ATC is a hierarchy of medical substances ordered by pharmacological, therapeutic and chemical characteristics. H2 receptor antagonist and proton pump inhibitor are examples of entries. The dictionary of pharmacological terms is enriched with synonyms and term variants taken from the Unified Medical Language System (UMLS). UMLS is a metha-thesaurus with 2 million concepts. Drug and compound names are excluded because they are already present in the Chem dictionary. The dictionary of pharmacological terms has 658 entries which are associated with synonyms from the UMLS. Gene/Protein Names (Genes/Proteins). The Genes/Proteins dictionary contains terms and synonyms extracted from SwissProt13 and Entrez Gene14 . Examples of the terms included in this dictionary are gene names, enzymes, receptors, transcriptional, and translational factors such as TATA binding 9 10 11 12 13 14

http://www.drugbank.ca/, last accessed December 2009 http://www.genome.jp/kegg/compound/, last accessed December 2009 http://www.genome.jp/kegg/drug/, last accessed December 2009 http://www.nlm.nih.gov/mesh/, last accessed December 2009 http://www.uniprot.org, last accessed December 2009 http://www.ncbi.nlm.nih.gov/gene/, last accessed December 2009

156

B. M¨ uller et al.

protein, tyrosine kinase, and CCBP2. The Genes/Proteins dictionary is species-specific as it contains only human related information. This dictionary has 35,912 entries. Single Nucleotide Polymorphism Names (SNP). The named entity recognition processed for single nucleotide polymorphisms (SNPs) is a combination of regular expressions and a CRF-based recognition of variation terms [12]. The identified entities are normalized and mapped to the dbSNP15 database. 2.3

Indexing and Retrieval

SCAIView16 [2,8] is a knowledge environment that has a web-interface to apply full text searches in documents combined with semantic searches of named entities provided by NER. Documents with their standoff annotations are indexed with Lucene17 [3]. The web-interface of SCAIView has an aggregated view to visualize the results of the semantic searches ordered according to the selected ranking methods. The documents of the established corpora from Medline, PMC, and TREC-CHEM with the section information as well as the standoff annotations from the named entity recognition are integrated into SCAIView. In Figure 1, an overview of the workflow is shown. PDGene18 is an online database specific to Parkinson Disease. The content is manually curated from full text publications. PDGene provides updated information of genetic associated studies from the literature with a Top Results List for genes that are associated to the prevalence of Parkinson. The gene GBA (Entrez Gene19 Identifier: 2629) is ranked as the first one. Exemplarily, the information need “I want all relevant documents that are associated to the prevalence of the Parkinson Disease and GBA” is conducted. A full text search for “Parkinson AND GBA” in the TREC-CHEM index finds 41 hits. The application of NER for the identification of genes and proteins allows the expansion of the query to GBA with all its synonyms. The semantic search as combination of the full text search for “Parkinson” and the gene GBA with all its synonyms finds 377 hits. The quantitative distributions of the different entity classes in the established corpora from Medline, PMC, and TREC-CHEM are analyzed with SCAIView.

3

Results

The named entity recognition with the dictionaries described in Section 2.2 were performed on the established corpora of Medline with 10,294,069 documents, PubMed Central with 132,867 documents, and TREC-CHEM with 1,201,231 million patents. For each entity class, the following measurements are calculated for the comparison of the corpora and the different document sections: 15 16 17 18 19

http://www.ncbi.nlm.nih.gov/projects/SNP/, last accessed December 2009 http://www.scaiview.com/animal/, public animal version, last accessed March 2010 http://lucene.apache.org, last accessed December 2009 http://www.pdgene.org, last accessed March 2010 http://www.ncbi.nlm.nih.gov/gene/, last accessed March 2010

Abstracts versus Full Texts and Patents


    


    !

157


  

!""#$"%&"

'%(")"

2%

$)01

./ "

*-

2%

$

&'

&*+,*!

Fig. 1. Workflow of preprocessing, named entity recognition, and indexing the corpora

Entities. The total amount of occurring non-unique entities amongst all documents or in all documents amongst the specific section. Unique Entities. The total amount of unique entities amongst all documents or in all document amongst the specific section. Documents. The total amount of documents or document sections where at least one entity of the entity class occurs in. Median. The median of unique entities per document or document section where at least one entity of the entity class occurs in. Mean. The mean of unique entities per document or document section where at least one entity of the entity class occurs in. Entities per Word. Mean of Unique entities per word in each document averaged by the total amount of documents in the corpus. |A\B| G(A, B). The Jaccard-like similarity function: G(A, B) = |A∪B| . This measure describes the ratio of additional unique entities in A in comparison to all unique entities in A and B. In Section 3.1, the results of the Medline corpus are described. In Section 3.2, the PMC full texts are compared to the PMC abstracts. The differences amongst the sections title, abstract, claims, and description in the TREC-CHEM corpus are revealed in Section 3.3.

158

B. M¨ uller et al.

Table 1. Statistics of numbers of entities per document in the Medline corpus

Entity Class Pharma Chem DrugBank Genes/Proteins Disease IUPAC SNP

3.1

Median Mean 1 2 1 1 2 1 2

1.2572 2.1247 1.6596 1.9250 2.3905 1.7911 2.3676

Entities per Words in % 0.1735 0.7627 0.3365 0.4158 1.0797 0.1172 0.0010

Entities

Unique Entities Documents

5,574,060 462 22,098,710 14,824 11,197,301 2,032 16,226,641 16,875 27,559,680 4,340 2,352,876 539,681 17,959 7,071

2,277,130 4,887,273 2,807,696 2,742,201 5,548,966 966,686 5,106

Medline

In the Medline corpus of 10,294,069 documents, 85,027,227 named biomedical entities were recognized. In Table 1, an overview of the measurements for each entity class is shown. The highest amount of entities has the entity class Disease with 27,559,680 followed by Chem with 22,098,710, and Genes/Proteins with 16,226,641. This same tendency holds for the measurements of amount of unique entities and of the entities per word amongst the entity classes. In case of documents that contain at least one entity of an entity class, Disease has the highest amount with 5,548,966 documents followed by Chem with 4,887,273 documents and DrugBank with 2,807,696. The density of non-unique entities is the highest one in the entity class Disease with 1.08 % unique entities per word followed by Chem with 0.76 % and Genes/Proteins with 0.41 %. IUPAC has the second lowest density with 0.12 % unique entities per word. The lowest density is in the SNP class with 0.001 %. From this data it becomes clear that the recognition of Chem entity class in the field of biomedical publications has a high impact. They have even higher overall and mean frequency than the entity class Genes/Proteins. 3.2

PubMed Central

In the title or abstract of the PMC corpus with 132,867 documents, 913,275 named biomedical entities were recognized. The results of the PMC corpus with documents in the title or abstract is shown in Table 2. Since the PMC corpus is a small subcorpus of Medline the amount of entities is in a similar order of 100 fold smaller than in full Medline. The mean unique entities and the entities per word in % are quite similar but overall a little bit lower than in the Medline corpus. Only the class Genes/Proteins has an increase and the Chem class has an significant decrease in mean entities from 2.12 to 1.85. In this corpus the amount of non-unique as well as the mean of unique Chem entities is lower than for the entity class Genes/Proteins. To assess the distribution of unique entities between title and abstract the Jaccard-like similarity is calculated. Table 3 shows the differences between the

Abstracts versus Full Texts and Patents

159

Table 2. Statistics of numbers of entities per document in title and abstract of the PubMed Central corpus

Entity Class Pharma Chem DrugBank Genes/Proteins Disease IUPAC SNP

Median Mean 1 1 1 1 2 1 2

Entities per Words in %

1.2230 1.8491 1.5036 2.0834 2.2511 1.7319 2.0990

0.1573 0.7584 0.2245 0.5802 0.9530 0.0729 0.0033

Unique Entities Entities Documents 54,536 209,497 83,942 242,269 292,687 28,704 1,640

278 3, 769 1, 395 8, 708 3, 238 7, 532 542

23,732 56,578 24,497 38,397 56,428 7,626 424

Table 3. Mean of the number of additional unique entities found in abstract but not in the title. (A: Abstract, T : Title). Genes/ Pharma Chem DrugBank Proteins Disease IUPAC |(A\T )| 0.9976 G(A, T ) 0.8105

1.4278 0.7364

|(T \A)| 0.05081 0.1095 G(T , A) 0.04530 0.0795

SNP

1.173 0.754

1.5670 0.6902

1.6005 0.6489

1.4387 0.8175

2.0589 0.9687

0.0484 0.0407

0.0774 0.0627

0.1184 0.0779

0.0891 0.0489

0 0

title and the abstract. Almost all SNP mentions are found exclusively in the abstract (G (A, T ) is 0.97) and also for the entity classes Pharma and IUPAC the Jaccard-like similarity is above 0.81. For the classes Disease and Genes/Proteins 65 % and 69 % respectively of all entities are unique in the abstract. For the title as expected minor entities (between 0 for SNP and 8 % for Disease) are found only in the title and not in the abstract. This reveals that there is a much higher information content when regarding the title plus abstract. In the 132,867 full texts of the PMC corpus, 14,026,017 non-unique biomedical entities were identified. In Table 4, all measurements for each entity class are shown. Overall the word density decrease compared to the abstracts. Whereas the entity class Disease has clearly the highest word density in abstracts in the full text the the entity class Chem with 0.62 % unique entities per word has the highest density directly followed by Disease with 0.60 %. In contrast the number of documents with at least one entity and the number of unique as well as mean entities strongly increase in full text. The highest increase of retrieved full text documents is fourfold for the classes SNP and IUPAC. The mean entity rates increased most for the classes Chem and Genes/Proteins (around fivefold) and between four and twofold for the other classes. The number of unique entities doubled for almost all classes but the increase is over tenfold for the SNP class. These are 6, 022 more unique entities than found in the title or abstract. Actually,

160

B. M¨ uller et al.

Table 4. Statistics of numbers of entities per document in full texts of the PubMed Central corpus

Entity Class

Median Mean

Pharma Chem DrugBank Genes/Proteins Disease IUPAC SNP

2 5 3 5 6 2 3

2.7747 8.6626 4.7911 9.6903 8.6825 2.8321 5.1731

Entities per Words in % 0.1008 0.6236 0.1748 0.4610 0.6066 0.0542 0.0027

Unique Entities Entities Documents 787,493 3,257,860 1,380,649 4,429,919 3,824,927 316,282 28,887

374 7,898 2,014 17,934 4,108 35,750 6,827

73,163 107,150 75,839 93,662 90,333 28,482 1,796

Table 5. Mean of the number of additional unique entities found in full text but not in title/abstract (A: Abstract, T : Title; BB: Body and Background) Genes/ Pharma Chem DrugBank Proteins

Disease IUPAC

SNP

|(A ∪ T )\BB| 0.1264 G(A ∪ T, BB) 0.0967

0.3959 0.1893

0.1522 0.0972

0.1826 0.1069

0.2899 0.1200

0.2767 0.1324

0.0283 0.0102

|BB\(A ∪ T )| 2.3780 G(BB, A ∪ T ) 0.8089

7.6862 0.7507

4.3054 0.8380

8.8362 0.8372

7.2763 0.7655

2.3683 0.8134

4.6776 0.8566

there are just 244 more SNPs found in 10,294,069 Medline documents than in the 132,867 full texts. In Table 5, the complement of the full text and the title/abstract is shown. The Jaccard-like similarity between these two text parts is comparable to differences between title and abstract. Around 10 % of the named entities are only found in abstracts despite the class SNP. Here only 1 % is exclusively found in the abstracts. Between 75 and 86 % of all entities are exclusively found in the full text. These values clearly show how many information we loose in using only abstracts for information retrieval and extraction. 3.3

TREC-CHEM

An overview of all measurements applied on the patents is shown in Table 6. In the 1,201,231 patents of the TREC-CHEM corpus, 303,906,146 non-unique entities were recognized. In comparison to the PubMed Central full text corpus we have approximate a tenfold increase of documents and an 30fold increase of non-unique entities. Looking for the different entity classes it becomes clear that the increase for all chemical classes like Chem with 153,982,830, DrugBank with 50,044,744, Pharma terms with 31,404,417 and especially IUPAC with 39,779,445 is even much more higher. In contrast the increase of the classes Disease and

Abstracts versus Full Texts and Patents

161

Table 6. Statistics of numbers of entities per document in patents of the TREC-CHEM corpus

Entity Class Pharma Chem DrugBank Genes/Proteins Disease IUPAC SNP

Median Mean 4 22 7 2 3 13 2

5.51 31.82 11.44 5.96 10.34 46.22 31.10

Entities per Words in % 0.0679 0.3684 0.1415 0.0430 0.0467 0.1055 0.0000

Entities

Unique Entities Documents

31,404,417 464 153,982,830 14,563 50,044,744 2,659 25,178,673 18,502 19,318,151 4,244 39,779,445 3,517,668 1,993 1,170

980,714 915,100 1,070,790 823,939 549,819 265,950 38

Genes/Proteins is moderate as would be expected for the IP classes selected for the patent corpus. SNP entities could only be found in 38 documents but with a high mean of 31.10. The mean of the other entity classes have also increased for all chemical classes and is highest for the IUPAC class with 46.22 followed by the Chem class with 31.82. The mean of the classes Disease and Genes/Proteins is decreased in comparison to the PubMed corpus. In contrast the word density in the TREC-CHEM corpus is significant lower compared to PubMedCentral except for the IUPAC class. The highest density with 0. 36 entities per word has the class Chem. DrugBank has a density of 0. 14 and IUPAC a density of 0. 11 and the other entity classes are below 0. 07. In a further analysis the different sections title, abstract, claims, and description were additionally determined. The mean of unique entities per document section and the mean of unique entities per word in the document section is compared in Figure 2. The highest density of unique entities is found in the title. In the other sections the density is nearly the same but the mean of unique entities is highest in the description. The second highest mean is found in the claim section and is lowest in the title. The differences between the document section of the TREC-CHEM patents are calculated between the title and the abstract, claims and description and the title/abstract versus the claims/description (table 7). Compared to the PubMed Central corpus the title in relation to the abstract has a higher Jaccard-like similarity for the chemical entity classes Chem, Pharma and IUPAC. It is highest for the entity class IUPAC with a Jaccard similarity of 0.14. The similarity values between claims and description shows that the description section includes almost all entities mentioned in the claim section. Only up to four percent of all entities are unique to the claim section in comparison to the description section. At least 79 % of all entities are unique to the description section in relation to the claim section. The comparison of the similarity between the title/abstract and the claims/ description shows that between 92 % and 97 % of the entities are unique to the sections claims plus description. Only for Genes/Proteins the amount of unique entities in title plus abstract is around five percent. For the other entities

162

B. M¨ uller et al.

Fig. 2. The entity classes Pharma, DrugBank, Genes/Proteins, IUPAC, Chem, Disease, and SNP with their density of unique entities per word and the mean of unique entities in percent per title, abstract, claims, and description of the patents in the TRECCHEM corpus Table 7. Mean of the numbers of additional unique entities found amongst the different document sections (A: Abstract, T : Title, C: Claims, D: Description)

Pharma

Genes/ Chem DrugBank Proteins

Disease IUPAC

|(T \A)| G(T, A)

0.1395 0.1237

0.1272 0.0914

0.1099 0.1006

0.0905 0.0791

0.1242 0.0972

0.2329 0.1487

0.0000 0.0000

|(A\T )| G(A, T )

0.8577 0.6886

1.6412 0.7514

0.9978 0.7021

0.9286 0.7933

1.5367 0.7055

1.8255 0.7468

4.0000 1.0000

|(C\D)| G(C, D)

0.0626 0.0200

0.2249 0.0109

0.2260 0.0399

0.0966 0.0305

0.1422 0.0191

1.3554 0.0272

0.0000 0.0000

|(D\C)| G(D, C)

4.6915 27.2551 0.8027 0.7914

10.0531 0.7986

5.4661 0.9048

9.0731 0.8817

39.3144 0.8298

28.2105 0.8829

0.0205 0.0013

0.0070 0.0015

0.1075 0.0490

0.0246 0.0053

0.0564 0.0033

0.0000 0.0000

|(C ∪ D)\(A ∪ T )| 5.2417 30.7538 G(C ∪ D, A ∪ T ) 0.9251 0.9323

11.1936 0.9455

5.7404 0.9289

9.9255 0.9373

45.7995 0.9753

31.0000 0.9737

|(A ∪ T )\(C ∪ D)| 0.0127 G(A ∪ T, C ∪ D) 0.0034

SNP

Abstracts versus Full Texts and Patents

163

the Jaccard-like similarity is below one percent for title/abstract. These values clearly indicates that using title/abstract for information retrieval and extraction would not be sufficient.

4

Discussion

In this work, an overview is given about the entity distribution in different information resources available to the text mining community. Named entity recognition was performed on the established corpora of Medline with 10,294,069 documents, PubMed Central with 132,867 documents, and on a large corpus of chemical patents, TRECCHEM with 1,201,231 million patents. We are quite aware that these named entity recognition methods have limitations on precision and recall especially for scientific and patent full text and that we have only partial coverage on chemical names and classes. Nevertheless this analysis gives us an rough estimation we haven’t yet about the coverage and density of the different entity classes in the various corpora and sections. The analysis of named entity distribution in Medline abstracts in comparison to the PMC abstract corpus shows that the mean of unique entities and the entities per word in % are quite similar but lower in the PMC corpus. In addition, the proportion of entities amongst the classes Chem and Genes/Proteins is different. This shows that the information content in the PMC corpus differs from Medline although the textual characteristics of traditional and open Access scientific journals are similar. In the comparison between title, abstract, and the full text in the PMC corpus, the word density decreases from the title over the abstract to the full text. In contrast the number of unique as well as mean entities starkly increase in the full text compared to the abstracts between fivefold for the classes Chem and Genes/Proteins and between four and twofold for the other classes. The number of additional found documents (up to fourfold) containing named entities, clearly indicates the loss of information while exclusively coping with the abstracts. This applies especially for the retrieval of SNP information. In some PubMedCentral full text, almost the same amount of different SNPs is found as in the full Medline corpus. Furthermore, the importance of the full text is underlined with the high Jaccard-like similarity values for body/background versus the low similarity values for title/abstracts. In comparison to the PubMed Central corpus, the analysis of the patent corpus shows a drastic increase of the amount of chemical names in contrast to a decrease of the classes Disease and Gene/Protein. Surprisingly, the word density is lower in patents than in scientific journals. This result has to be treated with caution because it might be a consequence of the broad field of chemistry and the limited coverage of the entity classes in the non-pharmaceutic realm. The Jaccard like similarity in the different sections show unexpected results. Titles have a higher Jaccard like similarity for chemical names compared to the analysis of the PubMed corpus. Probably, abstracts in patents have less coverage than abstracts in scientific publications. The Jaccard like similarity for the sections description and claim revealed that the description section covers almost all entities found in the claims and has overall the highest mean of found entities.

164

B. M¨ uller et al.

Further analysis for information retrieval on full text corpora like scientific full texts and patents are necessary for future work. Methods have to be conducted to cope for the different content and information overload covered by the different sections. First analyses done for the TREC Chemstry Track [5] revealed that weighting the different sections has an impact to the results and that using only the description section is superior than using the claim section only. In addition for the analysis of the chemical space, further chemistry terminology resources are necessary. Here, the terminology knowledge is harbored mainly in proprietary resources.

References 1. Fluck, J., Mevissen, H.T., Dach, H., Oster, M., Hofmann-Apitius, M.: ProMiner: recognition of human gene and protein names using regularly updated dictionaries. In: Proceedings of the Second BioCreative Challenge Evaluation Workshop, pp. 149–151 (2007) (last accessed August 2009) 2. Friedrich, C.M., Dach, H., Gattermayer, T., Engelbrecht, G., Benkner, S., Hofmann-Apitius, M.: @neuLink: A service-oriented application for biomedical knowledge discovery. In: Proceedings of the HealthGrid 2008, pp. 165–172 (2008) (last accessed August 2009) 3. Gospodnetic, O., Hatcher, E.: Lucene In Action. Action Series. Manning Publications Co., Greenwich (2005) 4. Guha, R., McCool, R., Miller, E.: Semantic search. In: WWW 2003: Proceedings of the 12th international conference on World Wide Web, pp. 700–709. ACM, New York (2003) 5. Gurulingappa, H., M¨ uller, B., Klinger, R., Mevissen, H.-T., Hofmann-Apitius, M., Fluck, J., Friedrich, C.M.: Patent retrieval in chemistry based on semantically tagged named entities. In: Voorhees, E.M., Buckland, L.P. (eds.) The Eighteenth Text RETrieval Conference (TREC 2009) Proceedings, Gaithersburg, Maryland, USA (November 2009) 6. Hanisch, D., Fundel, K., Mevissen, H.-T., Zimmer, R., Fluck, J.: Prominer: rulebased protein and gene entity recognition. BMC Bioinformatics 6(suppl. 1), S14 (2005) 7. Hirschman, L., Colosimo, M., Morgan, A., Yeh, A.: Overview of biocreative task 1b: normalized gene lists. BMC Bioinformatics 6(suppl. 1), S11 (2005) 8. Hofmann-Apitius, M., Fluck, J., Furlong, L., Fornes, O., Kol´aˇrik, C., Hanser, S., Boeker, M., Schulz, S., Sanz, F., Klinger, R., Mevissen, T., Gattermayer, T., Oliva, B., Friedrich, C.M.: Knowledge environments representing molecular entities for the virtual physiological human. Philos. Transact. A Math. Phys. Eng. Sci. 366(1878), 3091–3110 (2008) 9. Kanehisa, M., Goto, S.: Kegg: kyoto encyclopedia of genes and genomes. Nucleic Acids Res. 28(1), 27–30 (2000) 10. Kanehisa, M., Goto, S., Hattori, M., Aoki-Kinoshita, K.F., Itoh, M., Kawashima, S., Katayama, T., Araki, M., Hirakawa, M.: From genomics to chemical genomics: new developments in kegg. Nucleic Acids Res. 34(Database issue), D354–D357 (2006) 11. Klinger, R., Friedrich, C.M., Fluck, J., Hofmann-Apitius, M.: Named Entity Recognition with Combinations of Conditional Random Fields. In: Proceedings of the Second BioCreative Challenge Evaluation Workshop, Madrid, Spain, April 2007, pp. 89–91 (2007)

Abstracts versus Full Texts and Patents

165

12. Klinger, R., Friedrich, C.M., Mevissen, H.T., Fluck, J., Hofmann-Apitius, M., Furlong, L.I., Sanz, F.: Identifying gene-specific variations in biomedical text. J. Bioinform. Comput. Biol. 5(6), 1277–1296 (2007) 13. Klinger, R., Kol´ aˇrik, C., Fluck, J., Hofmann-Apitius, M., Friedrich, C.M.: Detection of IUPAC and IUPAC-like Chemical Names. Bioinformatics 24(13), i268–i276 (2008); Proceedings of the International Conference Intelligent Systems for Molecular Biology (ISMB) 14. Kol´ aˇrik, C., Klinger, R., Friedrich, C.M., Hofmann-Apitius, M., Fluck, J.: Chemical names: Terminological resources and corpora annotation. In: Workshop on Building and evaluating resources for biomedical text mining, volume 6th edition of the Language Resources and Evaluation Conference, Marrakech, Morocco (2008) 15. Schuemie, M.J., Weeber, M., Schijvenaars, B.J.A., van Mulligen, E.M., van der Eijk, C.C., Jelier, R., Mons, B., Kors, J.A.: Distribution of information in biomedical abstracts and full-text publications. Bioinformatics 20(16), 2597–2604 (2004) 16. Shah, P.K., Perez-Iratxeta, C., Bork, P., Andrade, M.A.: Information extraction from full text scientific articles: where are the keywords? BMC Bioinformatics 4, 20 (2003) 17. Verspoor, K., Bretonnel Cohen, K., Hunter, L.: The textual characteristics of traditional and open access scientific journals are similar. BMC Bioinformatics 10(1), 183 (2009) 18. White, M.J.: Espacenet, europe’s network of patent databases. Issues in Science & Technology Librarianship 47 (2006) 19. Wishart, D.S., Knox, C., Guo, A.C., Cheng, D., Shrivastava, S., Tzur, D., Gautam, B., Hassanali, M.: Drugbank: a knowledgebase for drugs, drug actions and drug targets. Nucleic Acids Res. 36(Database issue), D901–D906 (2008)

Author Index

Albakour, M-Dyaa 6 Azzam, Hany 100 Costa, Alberto 84 Cox, Ingemar J. 70 Cunningham, Hamish

Lalmas, Mounia 31 Lopez, Patrice 120 Lucas, Simon 6

Fluck, Juliane 152 Frieder, Ophir 60 Friedrich, Christoph M. Frommholz, Ingo 31 Gillam, Lee 20 Graf, Erik 31 Gurulingappa, Harsha

McLaughlin, Harry 20 Melucci, Massimo 84, 136 Mevissen, Heinz-Theodor 152 Millic-Frayling, Natasa 70 M¨ uller, Bernd 152

1

152

Newbold, Neil

Roelleke, Thomas 100 Roth, Benjamin 47 R¨ uger, Stefan 1

152

Hanbury, Allan 1 Hofmann-Apitius, Martin Hosseini, Mehdi 70 Klakow, Dietrich 47 Klampanos, Iraklis Angelos Klinger, Roman 152 Kruschwitz, Udo 6

20

152

100

Urbain, Jay

60

van Rijsbergen, Keith Vinay, Vishwa 70 Wu, Hengzhi

100

31