Proactive Spoken Dialogue Interaction in Multi-Party Environments 9781441959911, 1441959912

Citation preview

Petra-Maria Strauß Wolfgang Minker

Proactive Spoken Dialogue Interaction in Multi-Party Environments 123

Proactive Spoken Dialogue Interaction in Multi-Party Environments

Petra-Maria Strauß • Wolfgang Minker

Proactive Spoken Dialogue Interaction in Multi-Party Environments

Petra-Maria Strauß Ulm University Institute of Information Technology Albert-Einstein-Allee 43 89081 Ulm Germany [email protected]

Wolfgang Minker Ulm University Institute of Information Technology Albert-Einstein-Allee 43 89081 Ulm Germany [email protected]

ISBN 978-1-4419-5991-1 e-ISBN 978-1-4419-5992-8 DOI 10.1007/978-1-4419-5992-8 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2009944071 © Springer Science+Business Media, LLC 2010 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.

Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface

This book describes the development and evaluation of a novel type of spoken language dialogue system that proactively interacts in the conversation with two users. Spoken language dialogue systems are increasingly deployed in more and more application domains and environments. As a consequence, the demands posed on the systems are rising rapidly. In the near future, a dialogue system will be expected, for instance, to be able to perceive its environment and users and adapt accordingly. It should recognise the users’ goals and desires and react in a proactive and flexible way. Flexibility is also required in the number of users that take part in the interaction. An advanced dialogue system that meets these requirements is presented in this work. A specific focus has been placed on the dialogue management of the system on which the multi-party environment poses new challenges. In addition to the human-computer interaction, the human-human interaction has to be considered for dialogue modelling. A prevalent approach to dialogue management has been adapted accordingly. To enable proactive interaction a detailed dialogue history has been implemented. As opposed to common dialogue systems which start from scratch when the interaction begins, the system developed in the scope of this book starts modelling as soon as the conversation enters its specified domain. The knowledge acquired during this early stage of the conversation enables the system to take the initiative for meaningful proactive contributions, already from the first interaction. In order to develop this interaction assistant comprehensive data recordings have been conducted in a multi-modal Wizard-of-Oz setup. A detailed overview and analysis of the resulting corpus of multi-party dialogues is presented. An extensive evaluation of the usability and acceptance of this novel sort of dialogue system constitutes a further significant part of this book.

Contents

Preface................................................................................................ V 1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Introduction on Spoken Language Dialogue Systems . . . . . . . . . 2 1.1.1 System Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.2 Current Trends in Spoken Language Dialogue Systems . 6 1.2 Related Work on Advanced Dialogue Systems . . . . . . . . . . . . . . . 8 1.3 The Computer as a Dialogue Partner . . . . . . . . . . . . . . . . . . . . . . 9 1.4 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.5 Outline of the Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2

Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Corpus Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Evaluation of Spoken Language Dialolgue Systems . . . . . . . . . . . 2.3 Multi-Party Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Speech Acts and other Linguistic Fundamentals . . . . . . . 2.3.2 Conversational Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.3 Human-Human and Human-Computer Interaction . . . . . 2.4 Dialogue Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Dialogue Context and History . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Dialogue Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Information State Update Approach To Dialogue Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.4 Multi-Party Dialogue Modelling . . . . . . . . . . . . . . . . . . . . . 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17 17 20 22 22 25 28 34 35 37

Multi-Party Dialogue Corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Existing Multi-Party Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Wizard-of-Oz Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

51 51 55 55 56

3

39 43 49

VIII

Contents

3.2.3 System Interaction Policies . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 WIT: The Wizard Interaction Tool . . . . . . . . . . . . . . . . . . 3.3 The PIT Corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Data Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Dialogue Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

58 60 65 66 67 68 72

4

Dialogue Management for a Multi-Party Spoken Dialogue System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 4.1 Multi-Party Dialogue Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.1.1 Dialogue Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.1.2 Interaction Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.2 Dialogue Management in the Example Domain of Restaurant Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.2.1 Dialogue Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.2.2 Domain Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.2.3 Task Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.2.4 Information State Updates . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.2.5 Dialogue Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 4.3 Enabling Proactiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.3.1 Optimistic Grounding and Integration Strategy for Multi-Party Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.3.2 System Interaction Strategy . . . . . . . . . . . . . . . . . . . . . . . . 94 4.3.3 Dialogue History for Proactive System Interaction . . . . . 98 4.4 Proactive Dialogue Management Example . . . . . . . . . . . . . . . . . . 102 4.5 Problem Solving Using Discourse Motivated Constraint Prioritisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.5.1 Prioritisation Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.5.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

5

Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.1 Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.1.1 Questionnaire Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.1.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.1.3 Analysing the System Progress . . . . . . . . . . . . . . . . . . . . . . 120 5.1.4 Assessing the Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.2 Evaluating System Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5.2.1 Descriptive Analysis of the PIT Corpus . . . . . . . . . . . . . . 125 5.2.2 Evaluation of Discourse Motivated Constraint Prioritisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.3 Gaze Direction Analysis to Assess User Acceptance . . . . . . . . . . 129 5.4 Assessing Proactiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 5.4.1 Addressing Behaviour During First Interaction Request 133

Contents

IX

5.4.2 Effect of Avatar on Proactiveness . . . . . . . . . . . . . . . . . . . . 134 5.4.3 Subjective Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 6

Conclusions and Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . 141 6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 6.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

A

Wizard Interaction Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

B

Example Dialogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

C

Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

List of Figures

1.1 1.2

SLDS architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Interaction model of the dialogue system. . . . . . . . . . . . . . . . . . . . . 11

2.1 2.2 2.3

Interaction model of the dialogue system. . . . . . . . . . . . . . . . . . . . . 27 IBiS1 information state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Multi-IBiS information state . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.1 3.2 3.3 3.4 3.5 3.6

Data collection setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recording scene from the viewpoint of cameras C3 and C1 . . . . Wizard Interaction Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Phoneme based mouth positions of the avatar . . . . . . . . . . . . . . . . Example database entry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dialogue with crucial points and phases. . . . . . . . . . . . . . . . . . . . .

4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8

Dialogue management component of the system. . . . . . . . . . . . . . 74 Information state structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Example information state. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Ontology for restaurant domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 System life cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Dialogue history as it relates to the dialogue . . . . . . . . . . . . . . . . . 100 Dialogue history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Example information state after getLatestUtterance of utterance 16. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Example information state after integrate of utterance 16. . . . 104 Example information state after consultDB of utterance 16. . . . 104 Example information state after getLatestUtterance of utterance 17. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Example information state after integrate of utterance 17. . . . 105 Example information state after getLatestUtterance of utterance 18. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Example information state after integrate of utterance 18. . . . 106

4.9 4.10 4.11 4.12 4.13 4.14

55 57 62 63 64 66

XII

List of Figures

4.15 Example information state after downdateQUD of utterance 18. . 107 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10

Technical self-assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Usability evaluation over all sessions using AttrakDiff . . . . . . . . . 121 Usability evaluation over all sessions using SASSISV . . . . . . . . . . 121 Usability evaluation using SASSISV . . . . . . . . . . . . . . . . . . . . . . . . 122 Usability evaluation over the different setups using AttrakDiff . . 123 Usability evaluation over the different setups using SASSISV. . . 124 Durations of the dialogues of Session I and II . . . . . . . . . . . . . . . . 126 Comparison of the dialogue phase durations . . . . . . . . . . . . . . . . . 127 Evaluation of the prioritisation algorithm . . . . . . . . . . . . . . . . . . . . 128 Listening behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

A.1 WIT software architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 A.2 Class diagram of the WIT dialogue manager . . . . . . . . . . . . . . . . . 153 C.1 SASSISV questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 C.2 SASSI questionnaire without SASSISV items . . . . . . . . . . . . . . . . 159 C.3 System interaction questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

List of Tables

1.1

Example dialogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.1 2.2 2.3 2.4 2.5 2.6

Dialogue snippet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interaction protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interaction principles by Ginzburg and Fern´ andez . . . . . . . . . . . . Interaction protocol adapted to multi-party situation . . . . . . . . . Interaction principle by Kronlid . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interaction protocol using the AMA principle . . . . . . . . . . . . . . . .

29 42 44 45 45 46

3.1 3.2 3.3 3.4

Example scenario description. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Statistical information of the PIT corpus. . . . . . . . . . . . . . . . . . . . PIT Corpus dialogue act tagset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Annotated example dialogue from the PIT corpus. . . . . . . . . . . . .

58 65 68 70

4.1 4.2 4.3 4.4 4.5 4.6

New interaction principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Interaction protocols using ASPS . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Dialogue system interaction types. . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Contentual motivation for proactive interaction. . . . . . . . . . . . . . . 96 Snippet of example dialogue from the PIT corpus. . . . . . . . . . . . . 102 Prioritisation scheme applied to an extract of a dialogue. . . . . . . 112

5.1

Gaze direction towards dialogue partners according to dialogue phases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 Percentage of U1 addressing U2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Gazing behaviour during addressing . . . . . . . . . . . . . . . . . . . . . . . . 132 Gazing behaviour during listening . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Gazing behaviour during first interaction request. . . . . . . . . . . . . 134 Statistical analysis of proactiveness in Session III dialogues . . . . 135 Subjective ratings of system interaction in of Session III dialogues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

5.2 5.3 5.4 5.5 5.6 5.7

1 Introduction

HAL: ’Excuse me, Frank.’ Frank: ’What is it, HAL?’ HAL: ’You got the transmission from your parents coming in.’ Frank: ’Fine. Put it on here, please. Take me in a bit.’ HAL: ’Certainly.’ Quote from ’2001 – A Space Odyssey’ (1968) by Stanley Kubrick. The HAL 9000 computer is addressing Dave who is resting on his sun bed. app. 1:00 hour into the film

As it was predicted already in 1968 by Stanley Kubrick (1928-1999) and Arthur C. Clark (1917-2008) in the science fiction movie 2001 – A Space Odyssey [Kubrick, 1968] the future has arrived. Computers are by now playing a prominent role in our everyday lives. Over the past decades they have evolved from big, monstrous mainly industrial machines to small mobile and extremely powerful devices that are in one way or another used by presumably every human being in the developed world. The quote by the ’supercomputer’ HAL 9000 from Kubrick’s movie shows that the computer is equipped with human-like qualities. It possesses natural language capabilities for both, understanding and speaking, the ability of logical reasoning and proactive behaviour, just to name a few character traits. The human characters of the movie are quoted in the movie to describe the computer as a sixth member of their space ship crew. A ’HAL-like’ computer has not been developed at present, however, HAL’s characteristics, i.e. his human-like features, are starting to appear in more and more computer systems. Natural language interaction plays an important role due to the fact that speech is for humans still the easiest and most natural way of interaction. Big displays become superfluous which opens the way for ubiquitous computing which lets computers disappear more and more into the background. Security is a further supporting factor for interaction by speech. P.-M. Strauß and W. Minker, Proactive Spoken Dialogue Interaction in Multi-Party Environments, DOI 10.1007/978-1-4419-5992-8_1, © Springer Science + Business Media, LLC 2010

1

2

1 Introduction

This becomes apparent especially in the scope of automotive applications. While operating a vehicle, the driver can interact with the navigation, telephony and media applications by speech without taking the eyes off the road. The automotive environment is also a pioneer domain for proactiveness. State of the art head units inform the driver about traffic hazards coming up on the road. According to the priority of the message, i.e. for instance in terms of the distance to the obstacle which denotes if the driver could be affected immediately, even ongoing phone calls should be interrupted for the driver to receive the message as soon as the system learns about the hazard. As an independent crew member, HAL is further able to communicate with multiple users at the same time while most of today’s computer systems are restricted to one user, i.e. human-computer interaction. If dialogue systems can interact with several users simultaneously many applications would benefit, for instance in the process of achieving a common task. The research presented in this book addresses the presented challenges: A spoken language dialogue system that interacts with two users as an independent dialogue partner. It engages proactively in the interaction when required by the conversational situation and also takes itself back when it is not needed anymore. We thereby focus on the dialogue management functionality of the system (Chapter 4) for which we perform an extensive data collection (Chapter 3) to support the system development. Further, evaluation of the novel sort of dialogue system builds another prominent part of this book (Chapter 5). The envisaged system is introduced in more detail in Section 1.3. First, a short introduction is given on spoken language dialogue systems in general followed by a description of current trends and related work conducted in the area of advanced dialogue systems.

1.1 Introduction on Spoken Language Dialogue Systems 1.1.1 System Architecture The task of a spoken language dialogue system (SLDS) is to enable and support spoken interaction between human users and the service offered by the application. The SLDS deals with two types of information - the one understood by the user (natural language in speech or text) and the one understood by the system (e.g. semantic frames). The system carries out a number of tasks before it can give a response to the user. The tasks are performed by different modules which are usually connected in a sort of pipeline architecture. Figure 1.1 shows a basic architecture of a SLDS. The different modules are described in the following: Automatic Speech Recognition (ASR). The task of the ASR module is the transcription of the speech input (i.e. acoustic signals) of the user into words (e.g. [Jelinek, 1997,Rabiner and Juang, 1993,Jurafsky and Martin,

1.1 Introduction on Spoken Language Dialogue Systems

Acoustic analysis / Speech recognition

Semantic analysis

Dialogue management

Text-to-speech synthesis

3

Application

Natural language generation Dialogue context

Database

Fig. 1.1. SLDS architecture.

2000, Huang et al., 2001]). Using an acoustic model which describes potential signals, a lexicon containing all potential vocabulary and a language model, i.e. a grammar, the acoustic signals are usually mapped to the resulting words or sentences with statistical methodology. Different factors determine the complexity of speech recognition. A system that is to be used by an unknown number of different users possibly speaking in different accents and dialects is said to be speaker-independent. The opposite is a speaker-dependent system which is trained especially for the individual future user. A third intermediate option is a speaker-adaptive system which is developed as a speaker-independent system but can adapt to the actual user through training and usage. The vocabulary of the system further influences the complexity and performance: Small vocabulary is easier to be recognised than large vocabulary. Finally, continuous speech poses a greater challenge than isolated keywords. Natural Language Understanding (NLU). The NLU module tries to extract the semantic information from the word string produced by the speech recogniser (refer to e.g. [Allen, 1995, Jurafsky and Martin, 2000]). It produces a computer readable representation of the information (e.g. as semantic frames) which is then further processed by the dialogue management module. A common approach is to perform rule-based semantic parsing to extract the semantic meaning, e.g. attribute-value pairs, out of the utterances. Another approach include statistical methods for semantic analysis (e.g. [Minker et al., 1999]). Dialogue Management (DM). The dialogue manager is responsible for smooth interaction. It handles the input (in form of a semantic representation)

4

1 Introduction

which is to be integrated into the dialogue context. It organises turn taking and initiative, and performs task or problem solving by interacting with the application. Finally, it induces the output generation to return an appropriate response (e.g. the requested information) or to ask for any information missing in order to be able to fulfil the task. The DM makes use of various knowledge sources which constitute the dialogue context. The main parts are the task model and the dialogue model [McTear, 2002]. The task model contains all task-related parts of the system, such as the task record which holds all user constraints mentioned in the ongoing dialogue so far whereas the dialogue model contains information regarding the dialogue, such as a conversation model which consists of the current speaker, addressee, speech act, etc. The dialogue history can be said to belong to this part of the context as it holds information about the previous utterances of the ongoing dialogue. Further knowledge sources are a domain and world knowledge model which holds the logical structure of the domain and world the dialogue system functions in. A user model can be deployed which holds the information about the users, either to recognise specific users or more general information to be able to make recommendations. All these components are implemented more or less explicitly depending on the type of dialogue management used. Approaches to dialogue management can be classified into three main categories (following the categorisation presented by McTear (2002)): • Finite-state-based approach. The dialogue is always in a certain predefined dialogue state, certain conditions trigger state changes. In this approach, knowledge base and dialogue management strategy are not separated but are represented together in the dialogue states. The approach is rigid and inflexible but very suitable for small, clearly defined applications. • Frame-based approach. The systems implementing this approach deploy a specific task model which determines a set of slots to be filled with values supplied by the user in the course of the dialogue in order for the system to fulfil the task. Conversational aspects of the dialogue are considered only in the scope of task solving. The system is not expected to hold a conversation or know details of the conversation such as regarding the order of the constraints mentioned etc. Thus, no complex models have to be deployed. The approach is suitable for dialogue systems used for information retrieval, such as train departure times etc. • Agent-based approach. This approach is able to model dialogues in a more complex way. With sophisticated models of the conversation and dialogue participant it overcomes the limitations of the aforementioned approaches. Dialogue is no longer limited to certain dialogue stages but rather works towards understanding the dialogue as a whole. It models from the viewpoint of the dialogue system which is modelled as an agent who has goals (e.g. to fulfil the task), intentions, and plans to achieve its goals.

1.1 Introduction on Spoken Language Dialogue Systems

5

The prominent Information State Update approach (e.g. [Ginzburg, 1996, Larsson, 2002]) belongs to the third category. The dialogue which is seen as a state of information that is updated with every utterance is modelled from the viewpoint of the system enabling it to ’understand’ the dialogue as it occurs. This approach is thus very suitable to be adopted for our dialogue system which is to constitute an independent dialogue partner. The approach is introduced in Section 2.4 and later adopted and extended to suit our setup as presented in Chapter 4. A further categorisation differentiates between rule-based and statistical processing of dialogue management. All of the above mentioned categories of dialogue management can be implemented using either approach. The rulebased approach has been state of the art for a long time. Rules, defined by the developer, have to be supplied for all cases that can possibly occur in the dialogue. Accurate processing is thus assured, however, the development of the rule-base is very time-consuming and an increase in the complexity of the application brings about an analogical increase in the complexity of the rule set which can easily reach an unmanageable dimension. Recently, statistical approaches popular in ASR and also in NLU (e.g. [Minker et al., 1999]) are starting to be also deployed to dialogue management e.g. [Levin and Pieraccini, 1997, Singh et al., 2002, Lemon et al., 2006, Williams and Young, 2007]. Statistical techniques are based on statistical modelling of the different processes and learning the parameters of a model from appropriate data. The drawback of this approach is that for development a large amount of training data is needed which is difficult to obtain. Another important task of the dialogue management is problem solving. The dialogue management communicates with the application in order to fulfil the task. The simplest form of an application is a database. The dialogue management would in this case interact by performing database queries based on the current user constraints (contained in the task model) (e.g. [Qu and Beale, 1999]). Problem solving further looks at the outcome of the query and, if necessary, tries to optimise it. For instance, in the case that the query does not yield any results, the constraint set can be modified (for instance by relaxing less important user constraints) until a more satisfying result is achieved (e.g. [Walker et al., 2004, Carberry et al., 1999]). Natural Language Generation (NLG). The response commissioned by the dialogue management module is in this step turned into a natural language utterance. A common practise for NLG is the template based approach. Previously defined templates are filled with the current values. The NLG module is further responsible of structuring the output, i.e. choosing the best or combining the output if various are available or breaking it down into appropriate chunks if the answer is too large. The dialogue history can be consulted to assure responses that are consistent and coherent with the preceding interaction. For a multi-modal system, if e.g. visual output is deployed besides the speech output, the different modalities have to be integrated. The

6

1 Introduction

respective output has to be assigned the appropriate modality always assuring conformity. In general, NLG is from concerned with three tasks [Reiter, 1994, Reiter and Dale, 2000]: • Content determination and text planning to decide on what information, and in what kind of rhetorical structure it should be communicated. • Sentence planning determines the structure of the utterance for instance adapting it in order to fit in well with the current flow of the dialogue. Examples are splitting or conjunction of sentences as well as adding references or discourse markers. • Realisation is responsible for linguistic correctness and adaptation of the content to the actual output modality. A common practise for NLG is the template based approach. Previously defined templates are filled with the current values. The NLG module is further responsible of structuring the output, i.e. chosing the best or combining the output if various are available or breaking it down into appropriate chunks if the answer is too large. The dialogue history can be consulted to assure responses that are consistent and coherent with the preceding interaction. For a multi-modal system, if e.g. visual output is deployed besides the speech output, the different modalities have to be integrated. The respective output has to be assigned the appropriate modality always assuring conformity. Text-to-Speech Synthesis (TSS). Utterances generated in the previous module are converted from textual form into acoustic signals using text-tospeech (TTS) conversion [Dutoit, 2001, Huang et al., 2001]. The text is in a first step converted into a phoneme sequence and prosodic information on a symbolic level. Acoustic synthesis then performs a concatenation of speech units (e.g. for German diphones are common, while syllables are used for Chinese) contained in a database. The generated audio is then played back to the user. A different option yields the most natural sounding speech output that uses pre-recorded audio files. The duty of the NLG module is to simply select the adequate audio file to be played back to the user. A combination of these approaches, popular for commercial dialogue systems, is especially convenient for template-based NLG. The fixed template texts are pre-recorded, all variable parts are generated on the fly (preferably using the same speaker for both recordings). This way, the prompts sound as naturally as possible, however, not losing the flexibility of synthetically produced speech prompts. 1.1.2 Current Trends in Spoken Language Dialogue Systems Today’s commercial dialogue systems are usually deployed for simple tasks. They are predominantly slot-filling small-vocabulary finite-state machines, i.e. systems that match specific incoming keywords to a fixed output, a task that does not demand for elaborate dialogue systems. They are mainly found in

1.1 Introduction on Spoken Language Dialogue Systems

7

telephony applications replacing human operators in call centres. Their main aim is to save cost. A nice side-effect has been achieved by some companies by personifying their dialogue systems to use them as marketing instruments. The systems are given a name and appearance and thus star in commercials and on websites to improve a company’s image and level of awareness. A prominent example for such a system is the award-winning Julie1 (deployed in May 2002) who answers the phone if someone calls for train schedule information to travel within the United States. Insufficient technical performance, however, has been hindering speech based interfaces to obtain large-scale acceptance and application. Broad usage requires good recognition performance of speaker-independent large-vocabulary continuous natural speech which has been posing a great challenge to speech recognition. The last years have been coined by technical advancement and further, user acceptance has been growing due to the fact that people gradually get accustomed to the SLDS. The usefulness and convenience of spoken language interaction has been recognised and thus the range of applications is starting to grow and change. With progressing technology and the quest for smart and apt computer systems the foundation for accelerated progress has been provided. Possibly, scenarios that have for a long time only been found in science fiction might become ordinary scenes of everyday life in the future. A current trend addresses the nature of computer systems. Computers are blending more and more into the background, as described by the term ubiquitous computing. Computers are becoming smaller, almost disappearing, and are deployed more and more in mobile form. Everyday appliances are enriched with computational intelligence trying to ease human life building the basis for intelligent environments. Popular examples are intelligent heating and lighting adjustments and the intelligent refrigerator that keeps track of the contents, recipes, shopping lists and even ingested calories of the users. The overall trend is that computers adapt to the human way of interaction instead of requiring the humans to move towards the system for interaction. All of these facts pose further demands on applications and technology and at the same time show the importance of speech based interaction. It is an intuitive means of communication and does not require any space (e.g. big screens as is the case for haptic interaction) nor visual attention to be deployed and is thus also a suitable way for human-computer interaction in critical situations, such as the car where the driver’s gaze should not be drawn from the road if possible2 . Novel demands are posed on future systems in order to realise the adoption to new applications and environments. The objective of future systems is to actually understand the dialogue they are involved in and to adapt to 1 2

http://www.amtrak.com In practise, as an intermediate step towards speech interaction, current systems adopt speech interaction mostly as an alternative on top of the common forms of interaction and this way trying to gain in user acceptance.

8

1 Introduction

the surroundings and users, to autonomously perceive the user’s needs and desires and to react flexibly and proactively. Future dialogue systems are thus endowed with perceptive skills from different sensory channels (vision, hearing, haptic, etc.) to capture the spacial, temporal, and user specific context of an interaction process. Elaborate conversational skills are required to be able to capture and analyse spoken, gestural, as well as emotional utterings. Integration of perception, emotion processing, and multimodal dialogue skills in interactive systems is expected to not only improve the human-computer communication but also the human-human communication over networked systems. There is further an increasing demand for flexibility in terms of the number of users that are able to take part in the interaction. A system could this way for instance assist a group of users already during the decision process by providing information, immediately reporting problems and thus accelerating the task-solving process. Thus, interaction between various humans and possibly also various computers will be possible that integrates the dialogue system as an independent dialogue partner in the conversation. The research presented in this book focuses on a dialogue system of this kind: The system resembles an independent dialogue partner. It interacts with two users and engages proactively in the conversation when it finds it necessary in respect to advancing the task solving process in the best possible way. A description of the system and objective of this book is presented in detail below after taking a look at related work conducted on advanced dialogue systems.

1.2 Related Work on Advanced Dialogue Systems Various research projects investigate possibilities that open up by enriching multi-party interaction and advanced dialogue systems with the perception of the users’ state, context and needs. Most of the research on multi-party interaction at present is concerned with the meeting scenario as it can benefit greatly from the use of intelligent computer systems which enhance and assist the human communication during (and also after) the meetings. Great effort is put in the design and development of adequate software tools for meeting support and to investigate multi-party interaction phenomena. Meeting assistants could be deployed as meeting browsers or summarisers, i.e. they obtain information about the course and content of a meeting. They can be used for example during the meeting to assist participants who have come late to the meeting, summing up what has been said and who has committed to what. In the same way, easy and fast access of the meeting content is enabled at a later point in time. An example of a tool of this kind is the meeting browser developed in the framework of the Augmented Multi-Party Interaction (AMI) project [Renals, 2005] (and its successor AMIDA). The aim is to develop new multimodal technologies in the context of instrumented meeting rooms and

1.3 The Computer as a Dialogue Partner

9

remote meeting assistants, e.g. a meeting browser which enables browsing through videos and transcripts of the meetings. A second category of meeting assistants denotes tools that directly interact in the meetings. Exemplary tools to support and guide meetings in organisational, informational, and social aspects are developed in the scope of the Neem project [Barthelmess and Ellis, 2005]. Kwaku for instance is a virtual meeting partner which is equipped with emotion and personality and displayed using animation and speech output. It performs organisational tasks, such as watching over the time spent on certain agenda items and reminding the participants proactively to go on to the next point, if necessary. Another example which aims at developing tools that ease the human life is the project Computers in the Human Interaction Loop (CHIL) [Stiefelhagen et al., 2004]. It focuses on developing environments in which computers serve humans giving them more freedom to concentrate on the interaction with other humans by minimising the attention having to spend on operating the computer systems. One of the tools developed in the scope of CHIL is the Connector [Danninger et al., 2005] which perceives the activities, preoccupations, and social relationships of its users in order to determine their disposability and the appropriate device for communication. Another tools is the Memory Jog, a context- and content-aware service providing the user with helpful background information and memory assistance related to an ongoing event or other participants as a sort of personal assistant. While these systems aim at assisting the users with helpful background information none of them have the goal of getting involved in the conversation as an independent interaction partner.

1.3 The Computer as a Dialogue Partner The presented research aims at the development of a spoken language dialogue system which acts as an independent dialogue partner in the conversation with two human users. In the following, the system is presented in detail including with a description of the key characteristics that define this novel system. The system proactively takes the initiative when required by the conversational situation and gets meaningfully involved in the conversation. When it is not needed any more, e.g. the task is solved, the system takes itself back. At the beginning the system silently observes the conversation between the dialogue partners capturing the relevant conversational context and detects whether the current discourse falls within the specified domain. It becomes active in terms of paying close attention and modelling what the users say as soon as the dialogue partners come to speak about its specified domain. Thus, at the point in time of proactive or reactive interaction the system already knows what the users have spoken about and can directly participate in the ongoing conversation instead of starting from scratch. The system eventually reacts to interaction requests of the users who signal intention to communicate by turning their attention towards the system - either explicitly by addressing

10

1 Introduction

the system directly or implicitly by looking at it. To facilitate the development of the system, the setup is currently limited to allowing only one user, the socalled main interaction partner, to pose direct interaction requests towards the system. We adopt the example domain of restaurant selection to show the system’s functionality. In this case, the system becomes active when the users come to speak of wanting to go out to eat. The system notices the topic change and immediately ’listens’ attentively to the users discussing their preferences about choosing an appropriate restaurant. The system stores all relevant data in the dialogue context and history. Starting with the first interaction, the system assists the users in finding a suitable restaurant by performing database queries based on the users’ preferences and dialogue context and providing information about the restaurants. The system is displayed in form of an avatar which moves its lips and talks through synthesised speech output. A restaurant’s menu, city map or local bus schedule can further be presented on the screen when suitable. A short interaction example is shown in Table 1.1. The most significant characteristics of our system are described in detail in the following. U1:

Look at this weather, isn’t it beautiful!

U2:

Absolutely. Why don’t we leave it for today?

U1:

Good idea. I’m getting hungry anyway.

U2:

Do you want to go eat something? I feel like a huge pizza!

U1:

Sounds great! Let’s go somewhere where we can sit outside. [looks at computer] Is there anything over in the park maybe?

S:

Unfortunately not. However, there is Pizzeria Napoli. It’s close to here and has a patio.

U1:

That sounds good. What’s the address?

S:

Pizzeria Napoli is situated on Forest Avenue number fifteen. Would you like to see the menu?

U1:

Oh yes, please. [Menu pops up on the screen.] . . . Table 1.1. Example dialogue [Strauß, 2006].

Multi-Party Interaction Setup The interaction the system encounters is that of a multi-party scenario, i.e. more than two dialogue participants (in short: DPs) are engaged in a conversation, also called a multi-party dialogue. In our setup, the three dialogue partners interact with each other in different ways. Figure 1.2 shows a simple

1.3 The Computer as a Dialogue Partner

11

communication model. The system’s main user (in the following referred to as user U1) interacts directly with both dialogue partners: The human (U2) and the computer (S). The conversation with U2 is natural, the system is addressed either explicitly by calling it by its name or ’computer’, or implicitly by looking at it.

Fig. 1.2. Interaction model of the dialogue system.

The interaction between the second user U2 and the computer is by definition indirect. The system does not react when U2 addresses it with a request, although it hears and (hopefully) understands the utterances3 . Vice versa, it cannot be anticipated that U2 reacts to the system’s actions although equally perceived by both users. The only difference is in the physical access to the computer. U1 is situated directly in front of the screen and thus can be assumed to feel spoken to while U2 could easily feel as a side participant without the direct physical access. The interaction setup is taken up again from the linguistic viewpoint in Section 2.3. It can be argued why we chose this kind of a setup privileging one user and that a different example scenario involving two users which are not equal from the system’s point of view might be better suited, as e.g. in a doctor and patient or expert and customer scenario. For such a scenario it is imaginable to transmit information which is not intended for user U2 via some kind of secret channel that U2 has no access to. However, in our example domain of restaurant selection the information is designed for both users equally. The restriction to one user as the system’s main interaction partner derives from complexity reasons as the system is expected to be aware of the fact when the user looks at the system as a form of interaction request. The office environment with the system running on the main user’s personal computer endorses the scenario further.

3

However, the system’s proactive and cooperative behaviour would still lead to a system interaction if a pause longer than a certain threshold occurs after the request and the system has something meaningfully to contribute.

12

1 Introduction

Proactiveness One of the most important character traits of an independent dialogue partner is its independence. System interaction should not only rely upon interaction requests by the users but also be possible on the system’s own initiative. Proactiveness can be defined in the way that the dialogue system takes the initiative to control a situation instead of waiting to reactively respond to something after it has happened. A system should operate independently and anticipatory, always keeping an eye on its goals. Proactive involvement in the dialogue also implicates that the system takes itself back when it is not needed anymore, e.g. the task is solved or the users have permanently switched to a different topic. Proactive interaction behaviour requires complete understanding of the conversation and its context at any point in time that is relevant for the task solving process. Thus, in contrast to conventional dialogue systems that become active with the system’s first interaction, our envisaged system has to start modelling the dialogue before the system’s first interaction. Only then, the system can make meaningful and proactive contributions. Thus, an extensive dialogue history and context modelling is required to keep track of the complete dialogue (refer to Section 4.3.3). Different Levels of Attentiveness The envisaged dialogue system is an always-on system, i.e. it is not turned on when needed but constantly running in the background. It becomes active and interactive when certain criteria are true (as described below). The different levels of attentiveness the system takes on are the following: • Inactive: While the human users talk about anything but the specified domain the system does not pay full attention. It ’overhears’ the conversation and waits for keywords in order to detect the point in time when the conversation topic changes to its specified domain. • Active: As soon as the system recognises certain keywords, such as ’hungry’ uttered by U1 in the third utterance of the dialogue in the example dialogue in Figure 1.1. The system assumes that the conversation has entered the specified domain and switches over to pay full attention. From that point on, the computer ’listens’ actively to the conversation. It analyses the complete utterances, memorises the users’ preferences and builds a dialogue history. • Interactive: When required by the conversational situation the system gets - reactively upon an interaction request or proactively if it has a solution or problem to report - involved in the conversation by interacting, task and problem solving and presentation of the results. After the task has been solved or the dialogue has permanently moved on to a different topic, the system switches back to inactive mode. At this point, the dialogue history is reset; the system does not deploy a memory over different interactions.

1.4 Challenges

13

1.4 Challenges The dialogue system development is in the scope of this work limited to the dialogue management component as this gives home to the researched qualities of the system described above. Efforts on the development of an approach for semantic analysis for the system have been described in [Strauß and Jahn, 2007]. The system further contains text generation and speech synthesis components which are adopted from the WIT tool presented in Section 3.2.4. In detail, the present book elaborates on the following challenges. • Proactiveness. We introduce a way to enable proactive behaviour as it denotes a main character trait of an independent dialogue partner. Our system encounters an up to now unusual situation for dialogue systems: It steps into an existing conversation held between multiple dialogue participants. The ongoing conversation thus has to be modelled by the system already before its first interaction to be able to interact at an appropriate point with knowledge about the ongoing conversation. We see the potential to handle this challenge in the dialogue history which is used by common dialogue systems mainly for the task of enabling backtracking in case of e.g. misunderstanding. We propose to start modelling the dialogue history as soon as the conversation of the users enters the system’s specified domain in order to build a complete picture of the dialogue and users’ preferences. We further determine points in the dialogue that are suitable for proactive system interaction according to the analysis of our interaction data. • Multi-party dialogue management. The main focus of the development of the presented dialogue system is put on the multi-party dialogue management. It is based on a popular approach for two-party interaction which models dialogue as a state of information that is updated with every new utterance. Existing modifications of this approach that allow for multi-party interaction modelling do not suffice for our setup. We thus introduce an extended and altered version of the approach in order to suit our three-party setup and to allow proactive interaction by sideparticipants. • Discourse-oriented user constraint prioritisation. Within the functionality of problem solving, a novel approach to prioritise user constraints is introduced. It comes into play when an over-constrained situation occurs, i.e. too many or too restrictive preferences were provided by the users restraining the database query so that it yields no result. As far as the authors are aware, up to now there is no such approach for multi-party interaction. We make use of the new challenges and possibilities that come with the multi-party setup and integrate the course of the conversation into the decision process to determine the priority of the constraints.

14

1 Introduction

• Multi-party dialogue corpus. As a further important part of this book, the collection of a rich data corpus is described. 76 multi-party dialogues are recorded within an extensive Wizard-of-Oz environment. The data of the corpus, which consists of audio and video data, contains realistic interactions of the users with the envisaged system and thus forms an ideal basis to assist the development of the dialogue system. As far as the authors are aware, there is no comparable corpus available that consists of dialogue recordings in a human-human-computer setup. • Evaluation. Our presented independent dialogue partner poses a whole new situation for the users. Thus, it is of special interest to us to assess the usability and user acceptance of a novel system of this kind. Usability evaluation is performed on the basis of the questionnaires that the participants of the dialogue recordings filled out. An assessment of user acceptance is performed by analysing the main user’s gaze direction comparing the way it behaves towards the system versus the other user. A further evaluation considers the system’s proactive interaction behaviour.

1.5 Outline of the Book A short introduction on spoken language dialogue systems was provided in this chapter, followed by a short listing of current trends and ongoing related research on advanced dialogue systems. We introduced the dialogue system that builds the centre of this work and listed the scientific challenges that this book aims to achieve. The remainder of the book is structured as follows: Chapter 2 presents fundamentals to provide a background for the subsequent chapters. Section 2.1 provides a description on data collection and corpus development in general and by means of the prominent Wizard-of-Oz method. A short introduction on dialogue system evaluation is given in Section 2.2. Section 2.3 focuses on multi-party interaction presenting theoretical fundamentals on general linguistic theories and properties of human-human and human-computer interaction. Section 2.4 presents basics on dialogue modelling and dialogue management using the Information State Update approach. Existent multi-party extensions to the approach are discussed in Section 2.4.4. Chapter 3 focuses on the development of the multi-modal PIT corpus of German multi-party dialogues. Section 3.1 provides a listing of existent multiparty corpora which have been collected over the past years. As none of these corpora comprises the features required for our research, we build our own corpus. The setup and process of data collection using the Wizard-of-Oz method are outlined in Section 3.2 followed by a description of the corpus itself which consists of a 76 multi-party dialogues (Section 3.3). The section is concluded with a description of the annotation and analysis performed on the collected

1.5 Outline of the Book

15

data. Chapter 4 presents the proactive dialogue management developed for our dialogue system. Section 4.1 describes the multi-party dialogue modelling as well as a new interaction principle introduced in order to allow for proactive interaction. Section 4.2 introduces our example domain of restaurant selection. The remaining task related components of dialogue management are presented within the scope of this section. Section 4.3 concentrates on proactiveness enabling dialogue management strategies and components: Our optimistic grounding and integration strategy, the system’s interaction strategy, as well as the dialogue history keeping. The functioning of the proactive dialogue management is illustrated on an example dialogue extract in Section 4.4. Finally, a look is taken at the constraint based problem solving process deployed by our system in the scope of which a novel algorithm is introduced to prioritise user constraints according to the ongoing dialogue (Section 4.5). Chapter 5 is dedicated to the evaluation of the system. Section 5.1 presents the usability evaluation based on questionnaires the participants filled out prior as well as subsequent to the data recordings. The recordings were conducted with the simulated system (presented in Chapter 3); the evaluation thus rates the system as it is envisaged to perform in its final state. Section 5.2 presents the evaluation of the system performance including the evaluation of the discourse oriented user constraint prioritisation. An assessment of the user acceptance of the dialogue system is presented in Section 5.3. It analyses the behaviour of the main dialogue partner towards the system throughout the interaction in comparison with the behaviour towards the other human dialogue partner. Finally, the system’s proactiveness is assessed as presented in Section 5.4. The book is concluded in Chapter 6 with a summary and description of future work. The appendix contains a more detailed description of the WIT tool (Appendix A). The original German version of the example dialogue listed in Section 3.3.3 is presented in Appendix B. Finally, the part of the questionnaire used for the subjective evaluation as filled out by the participants of Session III recordings (refer to Section 5.1) is displayed in Appendix C.

2 Fundamentals

Dialogue involving two persons has for a long time been a popular topic of research. Dialogue systems involving two participants (the system and the user) have by now been firmly established in everyday life, especially in the field of call centre applications. Multi-party dialogue, i.e. conversation between more than two participants, is on the other side a rather novel field in research. It started most likely with the reclassification of the at that time conventional conversational roles of speaker and hearer (e.g. [Searle, 1969, Austin, 1962]). Clark and Carlson (1982) modified Searle’s speech act theory (1969) to enable multi-party interaction which was the start to multi-faceted research in multiparty dialogue and thus also opening the way for multi-party dialogue systems which have just started to emerge within the last few years. This chapter presents fundamentals to provide a basis for the work presented in the following chapters. Section 2.1 describes the process of data collection that is performed for corpus development and introduces the prominent Wizard-of-Oz technique used in this context. In Section 2.2 fundamentals on dialogue system evaluation are presented. Section 2.3 elaborates on fundamental properties of multi-party interaction from the linguistic point of view regarding speech acts and speaker roles and takes a further look at particular attributes of human-human and human-computer interaction. All of these aspects are presented in general followed by a discussion about how an aspect relates to our setup and envisaged dialogue system. Finally, the Information State Update approach to dialogue management (e.g. [Ginzburg, 1996, Larsson, 2002]) is introduced in Section 2.4 to provide the theoretical basis for the dialogue management presented in the scope of our work.

2.1 Corpus Development Dialogue data can be collected in various ways. What kind of data to be collected and in which way depends mostly on the purpose of the data collection, i.e. what the data is to be used for. The data can be used to analyse dialogue P.-M. Strauß and W. Minker, Proactive Spoken Dialogue Interaction in Multi-Party Environments, DOI 10.1007/978-1-4419-5992-8_2, © Springer Science + Business Media, LLC 2010

17

18

2 Fundamentals

in general or to perform user behaviour studies, i.e. to find out how conversations about a certain topic in a certain setup proceed, as is the case when used in the process of developing a dialogue system. Another important purpose is data material acquisition, i.e. collecting data to be used to train statistical models at all processing stages of the SLDS. Dialogue data can be classified according to different characteristics. It can for instance be distinguished in terms of the dialogue participants. Are only human dialogue partners involved or humans as well as computer systems? Do the users who take part in the recordings resemble the future user group? Will the recorded dialogues be real, realistic or artificial conversations? Recording real dialogues, i.e. the people being recorded engage in a conversation they would hold also if they were not recorded, yield the most authentic data. In a real setting and situation people act and interact out of their own motivation to achieve a certain goal. In artificially set up recordings, participants often receive some sort of compensation, such as e.g. money, cinema tickets or credits at university, which might present a greater motivation than their inner drive to complete an artificial task. A behaviour change is also described by the Hawthorne effect [Adair, 1984] which reports on phenomena that occur when people are aware to be under observation such as e.g. being recorded by camera and microphone. They act differently from how they would normally act in the same circumstances if they were not under observation. Large amounts of data already exist and are freely available. TV shows, news recordings, radio programs and audio books are only a few examples of available speech data. If these data can be used depends greatly on the purpose the data is to be used for. While it features a wide range of interaction amongst humans, it is for instance not suited if the interaction is to include a specific dialogue system. A further dimension is the recording procedure and equipment used. It can range from a simple recording device with integrated microphones to high class equipment, and additionally with webcams or video cameras. What equipment should be used depends generally on the purpose as well as the setup and surroundings. The data can be collected in ’field recordings’, i.e. realistic surroundings where this sort of conversation would normally take place, or in a laboratory. The laboratory could be made to represent a realistic setting, e.g. for an office or intelligent home situation. Finally, what kind of technical realisation of the recordings is needed? Conversations between humans can be recorded without great technical effort. If the recordings are to include a dialogue system, however, a system has to be deployed in order to obtain realistic data. If the data is needed in order to build the system, the famous hen egg problem is encountered. A popular approach to overcome this problem is the Wizard-of-Oz approach which is described in the following.

2.1 Corpus Development

19

Wizard-of-Oz Approach The Wizard-of-Oz (WOZ) technique for data recordings is a popular procedure to obtain realistic interaction data (e.g. [Dahlb¨ ack et al., 1993]) with a system which not yet or only partly exists. The technique is used to collect data in order to help develop the final dialogue system and at the same time enables system evaluation at an early development stage. The name Wizardof-Oz derives from the children’s tale [Baum, 1900] about the ’man behind the curtain’ as the setup is similar to the situation in the story. A human socalled wizard simulates a dialogue system (or essential components thereof) that interacts with the human users like the envisaged system. Ideally, the users do not notice the simulation and behave as if they were interacting with an automatic system rather than a human. The wizard is thereby situated in a different room (or some other hidden place). The behaviour of the simulated system has to be as close as possible to the behaviour of the envisaged system to obtain realistic data. Besides correctness in terms of the simulation another very important point is speed. For the users to believe to be interacting with a real system they are not willing to accept long reaction times as they are not typical for computer systems. The interface the wizard uses for the interaction thus has to ensure easy and quick usage. Equally, deterministic behaviour has to be assured, i.e. the system has to always react the same way under the same circumstances. Thus, in order to assure quick and deterministic system responses, the system is generally already partly functional. For instance, the application backend could already be functional (e.g. using the module at an early stage of development or a prototype with the same functionality) in order to assure deterministic behaviour in the system output. Modules that are more difficult to develop or require the data to be collected for the development (e.g. speech recognition and language understanding modules) are simulated by the wizard. Besides assisting in the development process, WOZ recordings are further a powerful tool for evaluation early in the development process. It can safe a lot of effort and expenses to allow potential problems and need for changes in the development to be identified before the actual implementation and constantly throughout the development process. This is facilitated due to the fact that users do not interact with a prototype with limited functionality but usually with the (partly simulated) envisaged system in its full extent. Evaluation can be performed via observation of the interaction and also by subjective evaluation of the system’s usability in the forms of interviews or questionnaires the users are to fill out after interacting with the system. An introduction to these evaluation methods is giving in the following section. Besides an audio recording functionality the setup can further be equipped with all sorts of logging mechanisms and cameras to record the behaviour of the users in more detail and thus allow further analysis and evaluation of the interaction.

20

2 Fundamentals

Our aim is to obtain realistic dialogue data to analyse the interaction in a novel multi-party setup. No such data is currently available for analysis (refer to Section 3.1). Thus, we perform data collection using the presented WOZ method which perfectly suits our needs. We obtain realistic data as the human users think to be interacting with a fully working system and thus behave in a realistic way. The WOZ setup and recording procedure are presented in Section 3.2. Besides assisting the development by providing interaction models, the WOZ approach further enables us to evaluate the envisaged system already at this early stage of development to assess user acceptance.

2.2 Evaluation of Spoken Language Dialolgue Systems Evaluation for SLDS can be differentiated between subjective and objective evaluation [M¨ oller, 2005]. Subjective evaluation measures factors such as usability and attractiveness of a dialogue system using perceptive judgments obtained from a subjective (i.e. the user’s) point of view. Usability evaluation measures the quality of an interactive system in terms of usefulness and user-friendliness. Interactive systems are used to manipulate the user’s world. Usability evaluation rates the way in which this manipulation takes place, i.e. how useful, usable the system and especially the design of the system is to a user. According to Nielsen (1994), usability consists of five quality attributes: Learnability, efficiency, memorability, errors and satisfaction. These components are said to be precise and measurable. However, in the evaluation of novel interactive systems as the one presented in this book, pragmatic quality which describes accurate functioning and design does not cover all aspects that influence a subjective rating of the interaction, i.e. the user assesses for herself if the product satisfies her needs personally. Thus, further quality characteristics such as stimulation and identity also play an important role. Objective evaluation aims at rating and predicting system performance and quality by analysing interaction parameters collected from the dialogues. Examples of interaction parameters are task success, recognition performance or number of timeouts which can be extracted from the dialogues at runtime or afterwards from the transcribed dialogues. These measures are sometimes further used during the ongoing dialogue e.g. to chose an adequate dialogue strategy. A common method for objective dialogue system evaluation is PARAdigm for DIalogue System Evaluation (PARADISE) [Walker et al., 1997, Kamm et al., 1999, Walker et al., 2000] developed by Walker and colleagues which predicts user satisfaction using task success rates and cost functions. The goal is to maximise user satisfaction by maximising task success while minimising cost. Fully automatic metrics are further extractable in real time in order to facilitate the system to adapt based on its assessments of its current performance. M¨ oller (2005) studied in what ways subjective and objective measures correlate. Different evaluation methods are evaluated and compared to find

2.2 Evaluation of Spoken Language Dialolgue Systems

21

only moderate correlations between subjective quality judgements and interaction parameters. Thus, automatic analysis such as the one performed in the PARADISE framework cannot replace subjective evaluation. M¨ oller concludes noting that ’subjective tests persist as the final reference for describing quality’ [M¨ oller, 2005, p.311]. For our evaluation presented in Chapter 5, we do not use the PARADISE framework for evaluation as our focus is a different one. We aim at appraising the novel features our dialogue system possesses which is primarily achieved through subjective usability evaluation that is able to assess the system as a whole and how it comes across to the users. The process of collecting data for evaluation can be performed in different ways (e.g. [Bortz and D¨ oring, 2006,Diekmann, 2007,Beywl et al., 2007]). Empirical social research identifies three main evaluation methodologies [Beywl et al., 2007]: Content analysis, observation and survey, as shortly described in the following. • Content analysis is a method that analyses the content of data, by breaking it down into components which are then assigned to a system of categories. • Observation can be used to collect verbal or non-verbal actions. An observer observes a certain process and classifies the observed behaviour according certain categories. The observation can be performed in different ways: The observer can be concealed or visible to the observed persons. The observer can be participating in the interaction or not. The interaction can be natural or artificial, etc. • Survey is an evaluation method used to collect opinions and attitude of the respondents. Surveys can be carried out in written or oral form. They can be performed via different communication channels (telephone, direct, online, mail etc.) and in the form of questionnaires or interviews. Many different forms of questions can be used (e.g. open or closed) depending on the aim of the question, i.e. what shall be achieved. The evaluation of our dialogue system uses all three of the presented techniques. Content analysis is used in order to examine the interactions in terms of design decisions on dialogue modelling and dialogue management issues, as described in Section 3.3. The observation technique is used for quantitative analysis of the interactions. The video recordings of the data are analysed in terms of the gaze direction of the main interaction partner, as described in Section 3.3.2. The survey technique is deployed in form of questionnaires that are filled out by the participants before and after the interaction. The design of the questionnaires used for our study is described in Section 5.1.1. The evaluation results are presented in Chapter 5.

22

2 Fundamentals

2.3 Multi-Party Interaction Dialogue can be classified regarding how many participants are involved in the conversation. In monologue, only one person is speaking, all other potential present persons build the audience. In dialogue, two persons are involved in the interaction, taking turns one after the other. If one person is speaking, the other person is addressed and listening and will take the next turn during which the first speaker will then be addressed and take the subsequent turn and so on. Thus, the interaction pattern is clearly defined. In multi-party dialogue more than two speakers are involved in the conversation as active participants. It is not self-evident who will take the next turn as there are usually no strict interaction patterns. Multi-party dialogue is thus more flexible with turn-taking as anyone could possibly take the next turn, even if this person was the addressee in the previous turn. In the following, a closer look is taken at linguistic fundamentals such as the speech act theory and grounding. It is followed by a discourse on dialogical roles which participants adopt in a conversation. The last part of the section considers the different combinations of dialogue participants that can occur in an interaction, namely human-human and human-computer. 2.3.1 Speech Acts and other Linguistic Fundamentals Austin phrased the fundamental linguistic question what people do with words as ’How to do things with words’ [Austin, 1962] and with it brought about the speech act theory (see also, among others, [Searle, 1969,Searle, 1975,Bach and Harnish, 1979, Grice, 1968]). He noted that a speaker says something not just for the cause of speaking itself but rather with the intention of communicating something, i.e. pursuing a certain goal. The act of speaking is therefore divided into three different speech acts: locutionary, illocutionary, and perlocutionary act. The speaking itself, i.e. uttering words or even meaningful sounds, is called the locutionary act. The illocutionary act describes the form in which the speaker intentionally utters the words, e.g. asserting, giving an order, asking a question, or promising something, just to name a few. What the speaker wants to achieve with the uttered words is denoted by the perlocutionary act which can e.g. be an action or change of a state of mind. An illocutionary act is thus successful, if the designated effect is achieved. For an example we look at the utterance ”Shut the window”. It clearly shows the intention of the speaker to be understood by the hearer as an order (illocutionary act) and intends further that the addressee should close the window (perlocutionary act). The illocutionary act (the order) is successful if the addressee closes the window. It would not be if the addressee would ignore the request. Hence, for successful communication it is necessary that the addressee understands the speaker’s intention and behaves cooperatively. Conversation can thus be denominated as a joint action of speaker and addressee [Clark, 1996].

2.3 Multi-Party Interaction

23

Clark and Carlson (1982) were the first recognising a shortcoming in the speech act model. The theory is limited to speaker and hearer1 . It is not differentiated between different types of hearers. If a conversation is held among more than two persons, the conversational roles of speaker and hearer might not be sufficient to describe all participants. In a circle of three dialogue participants, for instance, one speaker gives an order to another DP, addressing her directly by her name: ”Anna, please shut the window”. The third participant (i.e. not Anna and not the speaker) is also part of the audience and is assumed to hear what has been said, however, only the addressed person, Anna, is expected to perform the perlocutionary act of closing the window. Thus, Clark and Carlson replace hearer with addressee and argue that a speaker performs two illocutionary acts with each utterance: The traditional illocutionary act which is directed at the addressee (or addressees) and additionally an informative act which is directed at all the participants of the conversation, addressees and side-participants alike (refer to the section on dialogical roles below for a definition of the dialogical roles). Principle of Responsibility Clark and Carlson (1982) define the Principle of Responsibility which indicates that every participant of a conversation is responsible at all times for keeping track of what is being said, and for enabling everyone else to keep track of what is being said.’ [Clark and Carlson, 1982, p.334]. A speaker therefore has to design his or her contribution according to the audience, according to who is known to be and also might be listening, possibly unintended2 . ’Speakers design their utterances to be understood against the common ground they share with their addressees - their common experience, expertise, dialect, and culture.’ [Clark and Schober, 1989, p.211]. The speaker then ’presupposes the common ground already established; and all the parties, the speaker included, add what is new in that contribution to their common ground.’ [Clark and Carlson, 1982, p.334]. The common ground can differ immensely between different people which ’ordinarily gives addressees an advantage over overhearers in understanding. Addressees have an additional advantage [...] because they can actively collaborate with speakers in reaching the mutual belief that they have understood what was said, whereas overhearers cannot.’ [Clark and Schober, 1989, p.211]. However, the principle of responsibility places not only the responsibility of audience design on the speaker but also a responsibility on the audience. It indicates that all participants have the task to constantly keep track of the conversation to build on their common ground. If they lack to understand, 1 2

In some literature hearer is referred to as audience [Grice, 1968] or addressee [Clark and Carlson, 1982]. Potential unintended listeners might for example cause the speaker to use techniques like concealment [Clark and Schaefer, 1987].

24

2 Fundamentals

they are expected to pose clarification requests to be able to resolve everything fully. Grounding The common ground denotes the common basis of knowledge between dialogue partners. Grounding is the process of adding to this common ground. Due to different previous common knowledge and experience, different common grounds exist between different dialogue partners. In that way, Branigan (2006) claims that in multi-party dialogue there is no common common ground but rather several common grounds between the pairs of participants. The common ground between the speaker and side-participant is a partial common ground as it could contain utterances that are not fully grounded and there is no necessity that full grounding will take place in the further course of the dialogue. In contrast, the real common ground between the speaker and direct addressee(s) deserves the name due to the fact that an addressee has the possibility and responsibility to ground everything fully. However, in a conversation all utterances are witnessed by all DPs and it is the informative act introduced above that informs all participants of the illocutionary act performed towards the addressees. Thus, it is this informative act which is introduced into the common ground, in the same way for all DPs. This enables that in dialogue modelling, one single or common common ground can generally be modelled for all DPs. Clark and Schaefer (1989) claim that an acknowledgement is required for dialogue participants to perform grounding. It cannot be said that being informed about the occurrence of an act denotes that it has been understood, and also not by all DPs in the same way. Depending on the nature of the dialogue, there are different kinds and requirements of acknowledgement or acceptance. In two-party dialogues, acceptance is always of an individual nature, i.e. there is only one addressee and it has to be this addressee who accepts or acknowledges the move. In the multi-party case, however, it can be differentiated between three different kinds of acceptance: individual, communal or distributive acceptance [Kronlid, 2008]. Individual acceptance takes place if there is only one addressee, encountering the same situation as in the twoparty case. Communal acceptance means that if there is various addressees, one of the addressees accepts as a sort of a spokesperson for all addressees3 . Distributive acceptance denotes that each addressee gives feedback individually. The question is thus, which sort of acceptance is expected and required in each case? Can any utterance be considered grounded after communal acceptance? When is distributive acceptance required? There is of course not one valid answer, it depends greatly on the nature of the question. 3

Differing opinions are not considered and would require at least as many responses at there are opinions in the group of the addressees.

2.3 Multi-Party Interaction

25

Further, different grounding strategies can be deployed. An optimistic grounding strategy optimistically assumes acceptance and performs grounding without waiting for explicit feedback of all addressees. However, grounding might in this case be performed hastily. A pessimistic grounding strategy, on the other hand, waits for all feedback until an utterance is considered grounded. Larsson (2002) defines a third strategy, a cautiously optimistic grounding strategy, which follows the optimistic strategy, however, enables rollback in case that grounding has been conducted hastily. In our setup we deploy one single common ground to which all of the participants contribute. We argue that previous common ground which might exist between the users if they are acquainted is not (and cannot be) considered. Only the ongoing conversation is important for the aim of the dialogue to find a solution that fits best for all dialogue participants. Thus, the dialogue participants - even while they are side-participants - are expected to naturally follow the principle of responsibility as finding the best solution is also their goal of the dialogue. As discussed thoroughly in Section 4.3.1 we deploy an optimistic grounding strategy which alleviates us from the need to classify utterances according their individual, communal, or distributive nature. 2.3.2 Conversational Roles A participant takes on certain roles in a dialogue. Dialogical roles denote the part a person plays in the dialogue such as speaker or addressee. Social roles indicate the social position or relationship between the dialogue partners, such as roles within families and professional settings or the level of acquaintance. Further, task roles describe the position a dialogue participant plays in the dialogue in order to achieve a certain task. These roles are often assigned by profession, e.g. a person adopts a role as a moderator or judge in a trial etc. (e.g. [Traum, 2004]). Below, a closer look is taken at dialogical and social roles. Dialogical Roles Clark and colleagues [Clark and Carlson, 1982,Clark and Schober, 1989,Clark, 1996] make the standard speech act theory capable of handling multi-party situations by dividing the hearers into addressee, participant, and overhearer. Participants describe thereby the by the speaker intended audience, divided again into addressees and side-participants (who are currently not addressed). Overhearers are the ones that are not ratified participants. From the sociological viewpoint, Erving Goffman (1981) distinguishes between different types of unintended listeners in eavesdroppers and overhearers: ’Correspondingly, it is evident that when we are not an official participant in the encounter, we might still be following the talk closely, in one of the two socially different ways: either we have purposely engineered this, resulting in ”eavesdropping”,

26

2 Fundamentals

or the opportunity has unintentionally and inadvertently come about, as in ”overhearing”’ [Goffman, 1981, p.131-132]. In the dyadic case, dialogue participants adopt the dialogical roles of speaker and addressee for turn-taking in rotation. In multi-party interaction, however, each participant can take on various roles depending on the circumstances of the conversation. Interaction patterns cannot be clearly defined. With the dialogical role adopted, a participant takes on certain responsibilities. Thus, as an addressee the participant is expected (besides the principle of responsibility described above) to ground and respond. Whereas, as a sideparticipant, the system is expected to ground but not to respond and as an eavesdropper, one is expected not to reveal its presence. However, it cannot be taken for granted that the intended addressee takes the next turn as the side-participant or any other participant takes the next turn instead. Instead of the strict interaction patterns a dialogue system can easily adhere to, for the multi-party situation the system has to be flexible in order to change roles always adapting to the current situation of the conversation. According to which role the system adopts in the dialogue, it is expected to act accordingly. For instance, if it is an addressee it is expected to respond in the following turn. If, however, a different participant steals the turn, the system has to wait for a new chance to speak and during this time, the obligation to respond might even have dissolved. In our setup, the dialogical roles the system can adopt depend first of all on the state of attentiveness the system is currently in. Before the dialogue participants come to speak of the specified domain, they are engaged in dyadic dialogue. The system is in its inactive state and scans the conversation for a familiar trigger. The users are only partially aware that the system is listening. The second participant might not know about the system at all. The main interaction partner might have forgotten at the moment as the system has not been active. It cannot be assumed that the main interaction partner designs its utterances with the system in mind at that stage. Thus, although from the viewpoint of the main interaction partner the system is an intended listener and could thus be assigned the role of a side-participant. However, this does not necessarily include the viewpoint of the second participant. The remaining roles of eavesdropper or overhearer in Goffman’s definition are for unintended listeners only. As it comes closest to the actual situation considering the literal meaning, we still define our system to be adopting the role of an overhearer during inactive state. A similar argumentation can be held during the next phase of the dialogue while the system is in active state. The system is now closely following the conversation and is thus said to adopt the role of a side-participant although it does not get involved in the conversation. The second interaction partner might not be aware of the presence of the system at first. However, the role of the side-participant is justified by the fact that the main interaction partner could anytime address the system with a request which means that she is

2.3 Multi-Party Interaction

27

Fig. 2.1. Interaction model of the dialogue system.

aware of the system’s presence and is thus also expected to include it in her audience design. During the consequent phase of the dialogue, when the system has joined the conversation, both users are aware of the system as an interaction partner. The three interaction partners thus switch randomly between speaker, addressee and side-participant due to the fact that by definition only the main interaction partner is supposed to address the system directly. This, however, has no further effect on the role distribution. The ’indirect’ interaction between the second user (U2) and the system poses an interesting situation. A model of the interaction between the users and the system are depicted in Figure 2.1. As user U2 is not allowed to directly interact with the system, the system (S) is not obligated to respond when U2 is the speaker. However, U2 might quickly realise (so it can be observed in the recorded dialogues) that the system understands him and perform his audience design accordingly, in the way that it addresses the system without the expectation of receiving a direct response or to include the system as a side-participant. Social Roles The situation in which the communication takes place has a substantial impact on the conversation. What are the communication circumstances? Who is talking to whom? What is the relationship between the dialogue participants? The effect on dialogue due to observable differences in status between dialogue partners are confirmed by the theory of status organising processes [Berger et al., 1980]. In Western cultures, people with higher status tend to speak more, equally in dyadic and the multi-party dialogues. Thus, the social roles the participants take on in a conversation have an impact on the conversation. People act differently depending on which roles they possess in a conversation, in general and towards the other DP(s). The dialogue partners can either be at an equal social level or at a different, putting one dialogue partner at a superior level, as found e.g. in a conversation between an adult and a child or between two adults at unequal professional levels as encountered in an employer and employee relationship. The influence of

28

2 Fundamentals

the social factor on the dialogue context has widely been acknowledged, e.g. Bunt (1994) has introduced the social context as part of the complete dialogue context and Traum (2004) talks about specific task roles which relate dialogue participants in certain ways. In order to obtain a wide range of different (such as superior and inferior) behaviour in our corpus, we randomly assigned different roles and scenarios to the dialogue partners during the Wizard-of-Oz recordings. These included e.g. employer and employee, lovers, business colleagues, or friends. 2.3.3 Human-Human and Human-Computer Interaction This section is dedicated to human-human and human-computer interaction. Both types of interaction are first considered separately and discussed from the viewpoint of our own setup which combines both types. Finally, research dealing with the comparison of both types is presented. Human-Human Interaction Face-to-face interaction with other human dialogue partners is for humans the most natural and comfortable way of communicating. Presumably, it is also the fastest and most efficient, e.g. due to the human ability of dissolving ambiguity by interpreting paralinguistic phenomena of communication such as emotions and facial expressions of the other dialogue partner. Semantic content is said to make out only seven percent of a message [Bolton, 1979], the remaining 93 percent denote nonverbal communication. Human dialogue is normally held in order to achieve a certain goal. Each dialogue partner might thereby have her own goal, different from the other’s, however, both dialogue partners work together towards achieving their goals cooperatively which makes out human conversation. Human-human interaction follows certain patterns. Turn-taking, as mentioned above (refer to Chapter 2.3), in dyadic dialogue has the participants switch between the two possible dialogical roles of speaker and addressee. A further characterisation of interaction pattern concerns the purpose of an utterance, i.e. the combination of occurring dialogue acts. An initiating dialogue act induces a reaction from the other dialogue partner in form of another dialogue act. Such initiation - response pairs are called adjacency pair as they are likely to occur adjacent to each other. Prevalent examples for adjacency pairs are question and answer or greet and counter-greet. Depending on the sort of interaction, a dialogue adopts a specific structure, i.e. certain adjacency pairs occur regularly. The knowledge about this structure is necessary in order to understand and model the dialogue and thus facilitates the design of a dialogue system which is to understand conversation between humans. Due to the nature of the task-oriented dialogue we consider, all of our interactions follow an equal pattern. It can be observed that the

2.3 Multi-Party Interaction

29

same phenomena occur regularly and predictably, e.g. the structure of dialogue is very similar and the same adjacency pairs occur. This information is taken into account e.g. in terms of choosing a suitable dialogue act tagset (ref. to Chapter 3.3.2) for the system to be able to model the conversation appropriately. The most frequently occurring adjacency pairs in our scenario consist of proposal and a follow-up act: A proposal from one of the dialogue partners induces a reaction from the other dialogue participant. This response may consist of a simple acknowledgement, an accept or reject, a response with further content, or possibly a counter-proposal. Sometimes, the dialogue partner repeats the proposal which can have the function of acknowledgement, of checking if it was understood correctly or as a way of deferring the dialogue in order to win time to think. Generally, the response follows the proposal directly, however, in some cases it can also occur various turns later in the conversation with possibly even an off-topic discourse happening in the meantime. Table 2.1 shows a short example dialogue snippet labelled with the according dialogue act and the number of the utterance it refers to. User U1 proposes to go to an Italian restaurant. Instead of accepting right away, User U2 repeats U1’s proposal whereupon U1 acknowledges U2’s repetition. In this case, the repetition is to be interpreted as a request for clarification (check act). Utterance

DA, Reference

U1 5: Let’s go to an Italian restaurant.

(sugg,{})

U2 6: An Italian restaurant?

(check,A5)

U1 7: Yes.

(ack,B6)

U2 8: Ok.

(acc,A5) Table 2.1. Dialogue snippet.

The design of our dialogue system is advantageous as it integrates humanhuman interaction with human-computer interaction. The users are able to first come to an initial agreement among themselves before the system gets involved in the conversation. Only when both of the dialogue partners agreed on a choice of preferences, the computer is addressed to start the query process. This seems more efficient and faster than if the interaction would have included the system throughout the entire process. Scaling up the Number of Users If the number of participants of a conversation increases from two to more, certain factors change. Turn-taking becomes less predictive with an increasing number of participants that take part in the dialogue and can possibly take the next turn. The speaker can address more than one person simultaneously.

30

2 Fundamentals

All dialogue participants that are not addressed directly are side-participants. If the set of direct addressees consists of more than one person, an interesting question is who responds and who does not respond? Who is obligated to respond? Are all addressees equally expected to respond to the speaker in form of an answer, accept or reject or even just an acknowledgement to inform the speaker that what was said has been heard? Can an utterance be rated as successfully communicated if it has been acknowledged by all or by some of the addressed participants, or not at all? This question cannot be answered in general. The only generalisation possible to make is in the distinction of the sort of the question or request posed by the speaker. Thus, a question can presuppose either communal (i.e. one speaker takes on the role as a spokesperson) or distributive (every addressee has to respond individually) acceptance. Long distance responses occur more frequently in a setup with more than two interaction partners. If more than one participant responds to an issue raised, only one response can directly follow up the original utterance (without considering overlapping speech), all others have to follow each other. Ginzburg and Fern´ andez (2005) empirically evaluated human-human interaction comparing two-party with multi-party dialogues. Both types of dialogues exhibit adjacency between grounding and acceptance moves and their antecedents. However, while dialogues with two DPs show also adjacency between short answers and their antecedents, in multi-party dialogues long distance short answers are common. Thus, if a long distance short answer is given various turns after the issue was raised in the conversation, different issues might have been raised in the meantime. Thus, the response has to be related to the right issue posing the problem of issue accommodation. Noticeable differences in interaction patterns between large and small groups are shown by Carletta and colleagues (2002). Discussions in small groups involving up to seven participants resemble two-way conversations that occur between all pairs of participants, every participant can initiate conversation. Whereas, discussions in large groups are more like a series of conversations [Jovanovic et al., 2006]. In our setup the first part of the interaction is the only one involving only humans. The multi-party party situation includes a dialogue system and thus belongs to the situation described below. However, if the number of users for the system would be increased as it could be done without great effort, multi-party conversation among humans would be encountered. The qualities of multi-party dialogue described above would then have to be considered for the design of the system. However, as we deploy an optimistic way of grounding the difference would be less significant as the system does not wait for every response before it performs grounding (refer to Section 4.3.1).

2.3 Multi-Party Interaction

31

Human-Computer Interaction The interaction of a human with a computer, such as a task-oriented dialogue system, generally proceeds in a certain way: The user asks questions or poses requests to the system, the system then cooperatively gives information or suggests different options the user can chose from. The user evaluates the information obtained from the system. The interaction is ended when the task has been successfully solved and the user is satisfied with the outcome. The user is obviously aware of the fact that it is interacting with a computer. Nevertheless, conventional social expressions such as thanking and greeting are observed regularly from the user. A system is therefore expected to respond appropriately to not appear rude. Dialogue systems are not necessarily aware of who the user is unless they deploy user modelling in which case a profile for each user is stored that enables the system to retrieve the information about previous interactions or the user’s preferences, for instance. For systems that provide the user with specific or personal information, such as a status request for a booking process for instance, the identification of the user is indispensable. User modelling is further necessary if a system needs to keep track of previous interactions. In the general case, however, a system does not have to be aware of who it is interacting with. From the viewpoint of the human-computer interaction, our system integrates in a way both - single-user and multi-user - setups. The single-user setup is considered in the interaction of the main interaction partner with the system. All remaining factors are multi-party related and will be addressed in the corresponding section below. The main interaction partner addresses the system mainly with requests and is thereby holding the initiative. A typical request would be one of the following: • with propositional content, e.g. ’Can you tell us an Italian restaurant in the city center?’ – repetition of information · complete, i.e. all so far collected information is summed up, · partial, i.e. repeating only parts of the propositions, e.g. what has just been agreed upon – new information • without propositional content, e.g. ’Do you have anything like that?’ Any information that is supplied the system has to integrate, i.e. compare the new information to what has already been collected, and perform a database query (taken the information is conform) to return what the user has requested, e.g. specific information or a list of suggestions. The main other sort of interaction towards the system consists of responses to information from the system in form of e.g. acknowledgements.

32

2 Fundamentals

Scaling up the Number of Users A system can either be aware of the fact that it is interacting with multiple users or ignore it. For systems which serve as pure information seeking applications without a cooperative task solving process behind, the system does not need to be aware of the different users. Being aware of a user means in this context that the different users are modelled in dialogue modelling (refer to Section 2.4). For systems that interact cooperatively with its users, that hold a conversation, should be aware of its counterpart to be able to react to the users appropriately. A further dimension is flexibility in the number of users, i.e. the set of users can be fixed or changing during an interaction. Our system is aware of the two interaction partners. It knows which one is speaking at any moment. However, it does not perform complex user modelling as this is not required for the task at hand and due to the setup of limiting the direct interaction. The two interaction partners build on to the same common ground. Our system deploys an optimistic grounding and integration strategy (refer to Section 4.3.1) that integrates propositions when they are first mentioned without waiting for feedback from the other user. The number of users could be increased with minor alterations in the dialogue modelling as long as one user stays the system’s main interaction partner. The more participants take place in the conversation, the more long distance short answers are expected to occur which complicates the assignment to the right initiating speech act. However, the fact that people tend to subsume and repeat agreements and decisions after long discussions would be fortunate, alleviating the understanding process of the system. This way, it can be assumed that the system will not miss to understand what to include or exclude from the current set of propositions. Impact of System Interaction on the User Behaviour The interaction of the system has a strong bearing on the flow of the conversation in various aspects. To serve and please the user in the best possible way, negative answers should be presented as favourable for the user as possible. Thus, user preferences should be collected and integrated into the system to be able to adapt to the users. If the system understands which constraints are more important to the users than others, it can on its own device alter the query to present results that come as close as possible to what the users request. In our recorded dialogues (refer to Section 3.3), different reactions from the users according the result or information supplied by the system are observed, some of them quite affective. A positive answer and presentation of the query results or retrieved information induces a behaviour ranging from neutral reaction up to the users showing vivid delight, some of them thanking and even praising the system. In a few cases, the users are not pleased with the

2.3 Multi-Party Interaction

33

result and modify the query accordingly. On the other hand, if the answer from the computer is negative (in terms of no results or too many to be read out) this leads to a neutral to negative reaction from the users. While some accept the fact and simply change the request until they are pleased with the outcome, others show frustration, and in the worst case even grumble at the system. However, it has to be noted that during the recordings, the outcome of the database query was not altered. If the query did not yield any result, it was reported to the users. At the same time, if the set of results was too large to name all of them, the number of results was reported together with the suggestion to constrain the query further. The final system takes on a more cooperative way of problem solving. We therefore integrate a way to prioritise user constraints in the problem solving process in order to achieve satisfying answers even if a query with the original set of preferences yields no result (refer to Section 4.5). Comparing Human-Human and Human-Computer Interaction Differences in human-human and human-computer interaction have been noted for a long time. Addressing a computer has been categorised as ’formal’ [Grosz, 1977], or ’computerese’ [Reilly, 1987] showing a telegraphic tendency [Guindon et al., 1987]. Linguistic analysis has revealed that when speaking to a computer users tend to use very few pronouns but a high number of complex nominal phrases [J¨ onsson and Dahlb¨ ack, 1988, Guindon et al., 1987]. In more recent studies, Doran and colleagues (2001) investigated the differences in human-human and human-computer interaction in terms of initiative and dialogue act patterns. They compare the same task as it is performed in two different setups, i.e. in human-human and human-computer interaction. The human-computer data consists of telephony based interaction with different dialogue systems. All dialogues are set in the travel-domain. They are annotated in terms of dialogue acts, initiative and unsolicited information. The empirical evaluation shows that while during the human-human dialogues the initiative is equally shared amongst expert and user, the human-computer dialogues exhibit mostly expert- (i.e. system-) initiative. Users generally talk more in the human-human dialogues and the dialogue act patterns show great differences, e.g. in terms of confirmation. Short confirmations in form of acknowledgments occur very rarely in human-computer interaction. Systems use both long and short confirmations a lot more frequently than the users do and long confirmations about five times more often than short ones. Whereas, in human-human interaction the experts still use more confirmations than the users. However, in this case, shorter confirmations are the norm. Considering the dialogue act distribution it can be said that in human-computer dialogues the users mainly supply task-related information whereas in the human-human setup they tend to also request information. In terms of the exchange of information, the dialogue between the human communication partners is more evenly distributed. Taken together, in human-computer interaction the initia-

34

2 Fundamentals

tive is dominated by the system for most of the time (although all systems deployed for the recordings are said to be mixed-initiative). A difference in the dialogue act patterns shows further domination by the system. Only speculative answers can be given for these phenomena. They could for instance result from the often poor speech recognition ability by automated systems which makes the system fall back to system initiative and forces the users to adapt and interact with the system in a different way, i.e. learned behaviour from the users’ side. Despite the mentioned differences and possibly differing task expectations towards a computer as opposed to another person [J¨ onsson and Dahlb¨ ack, 2000], human-human dialogues are regularly used to imply humancomputer interaction as they are able to demonstrate real interactions and are easier to acquire than using a WOZ setting. Refer to Section 2.1 for a further discussion on characteristics and techniques for data accumulation.

2.4 Dialogue Modelling Dialogue can be described as a sequence of events (i.e. utterances) that act upon the current state of the dialogue, i.e. state of information regarding the dialogue. Each event modifies or updates the current dialogue state according to the event’s contents. The ’world’ in which a dialogue system operates consists of different kinds of information which taken together is referred to as the dialogue context. The context contains all information the dialogue system has access to consisting of static world knowledge and dynamic information about the ongoing dialogue. The dialogue history is an important part of the context holding all relevant information accumulated during the dialogue up to the current point in time. Dialogue modelling provides concepts and tools to organise this information, i.e. integrate incoming information into the context at each dialogue step, perform updates on the data according the input and dialogue state and finally determine an appropriate next move of the system. Dialogue management adopts the dialogue modelling concepts to a real domain and organises turn-taking and grounding (refer to Section 2.4.2 below). This section starts with a closer look at the dialogue context and with the dialogue history a particular part thereof. Subsequently, a prevalent approach to dialogue management, the Information State Update (ISU) approach (e.g. [Larsson and Traum, 2000, Matheson et al., 2000, Larsson, 2002]), is presented. The approach denotes a straight forward and flexible technique which denotes a suitable basis for the dialogue management of our system. Chapter 4 presents in which way it is adopted to our scenario and situation. The approach is in the following introduced in detail using Larsson’s IBiS1 system [Larsson, 2002] as an example implementation. The chapter concludes with a discussion on recent work on adopting the ISU approach to multi-party environments.

2.4 Dialogue Modelling

35

2.4.1 Dialogue Context and History Dialogue context contains all information accessible to the dialogue system. It consists of a static and a dynamic part. The part of context containing the static information consists of the world knowledge relevant to the system which describes information about e.g. the periphery of the system, such as the location, time, date and day of the week, weather conditions or whatever might be relevant for the application. It further includes the concepts of general conversational principles and of the system’s domain and the database the system accesses to solve the task. All this information is said to be static whereas static does not mean that it cannot change (as it would naturally be the case for weather, time and date and possibly also for the database in case of updates). However, the information is not changed by dialogue. The dynamic part of the context comprises all the information that dialogue participants provide during the course of the conversation. The current state of the dialogue changes with every new utterance that bears relevant semantic context. Bunt claims in (2000, p.101): ’There is no room [..] for an “objective” notion of context, since the participants’ communicative behaviour depends solely on how they view the situation, not on what the situation “really” is.’ Clearly, every participant has a different perception of things which could also be a clear reason against the concept of conducting one common ground between all dialogue partners. However, we claim that for dialogue systems as ours with a focus on the task, any possibly occurring differences can be disregarded as they are not expected to have any effect. We thus take the context as being valid for all participants in the same way. According to the Dynamic Interpretation Theory (DIT) introduced by Bunt (1999) which defines a concept of a comprehensive dialogue context. Five different types of context information are distinguished: • Linguistic context comprises the linguistic information of the conversation as a whole, in raw (utterance as plain text) and analysed (representation of utterance after semantic analysis) form. The information about how the dialogue has proceeded so far (the dialogue history) as well as concepts for the future course of the dialogue (dialogue plans) are further contained. Thus, this context contains static and dynamic information. It is further responsible for anaphora and ellipsis resolution, phenomena that occur regularly in natural dialogue. • Semantic context holds the task-related information such as the taskrecord which is often simply referred to as the task model. All propositions that have been uttered during the dialogue (i.e. user preferences or constraints) are stored here and used as a basis for the database queries. Another part of the semantic context is the domain model which describes the domain the system acts in in form of an ontology, for instance. An ontology is used as a formal representation of a domain describing concepts,

36

2 Fundamentals

their properties and relationships among the concepts (refer to Section 4.2 for the description of an ontology of our example domain). • Physical and perceptual context comprises the physical context that is of relevance to the system, e.g. availability of communicative channels, as well as assumptions on the partner’s physical and perceptual presence and attention. • Social context is composed of interactive and reactive pressures on the agent such as communicative rights, obligations and turn management. For instance, if the user posed a request to the system the system then has the obligation to respond. • Cognitive context comprises the agent’s processing status which is relevant e.g. if certain tasks take some time. If the system is aware of this fact it can report it to the user or use the meantime for something else. The dialogue partners’ states are further modelled in this context, e.g. what their attention currently points at, if they are speaking and to whom, where they are looking at and so on. The dialogue history denotes a particular part of the dialogue context. Some systems use the term dialogue history interchangeably to refer to the complete context model, other systems keep the two terms clearly separate which we support in our notion. The dialogue history represents the flow of the dialogue by storing what is and especially has been talked about throughout the dialogue. It depends on the complexity and requirements of a dialogue system what kind of dialogue history is deployed. Simple slot-filling dialogues normally get on with a simple task model during the course of the dialogue without using an explicit dialogue history. It is not necessary to be able to trace back when and in which order certain slots were filled as it matters only what the values are. More dialogically complex dialogue systems that e.g. are able to hold a conversation with the user(s) are expected to be able to recall events which happened previously in the interaction and to know what has been spoken about. A system should for instance know what it has previously suggested in case the users want to go back to this option. The dialogue history is also used for the resolution of elliptical sentences and anaphoric references to entities contained in previous utterances. Further, in the case that understanding errors occur and the system reaches a state of incorrect information it has to be able to fix this by restoring the last correct state, e.g. through backtracking. With the variety of tasks, the way of storing the data differs as much from system to system as does the type of data to be stored. The representations range from complex hierarchical structures containing various pieces of information to simple sequential representations. For instance, the LINLIN system [J¨onsson, 1997] deploys a dialogue history which has the form of a dialogue tree with three levels corresponding to the whole dialogue, to discourse segments and to speech acts. The VERBMOBIL system [Alexan-

2.4 Dialogue Modelling

37

dersson et al., 1995] contains a dialogue memory which contains representations of intentional, thematic and referential information. The intentional layer contains dialogue phases and speech acts which are represented in a tree-like manner, the thematic layer contains domain-related information relevant to the task-solving process, and the referential information consists of the lexical realisation of utterances. The WITAS system [Lemon et al., 2001] is another system which deploys a tree, called the dialogue move tree, which stores dialogue states. The edges are dialogue moves and the branches represent conversational threads. The dialogue history is thus integrated into the dialogue model. The GALAXY system [Seneff et al., 1996] uses semantic frames to represent the data which is stored in the dialogue history in a sequential way. It maintains a history table of objects which can possibly be referred to in the future, making the interpretation process easier. The WAXHOLM system [Carlson et al., 1995] is another representative of systems to use semantic frames to model the dialogue history. The TRAINS system [Traum, 1996] deploys an approach intermediate to the two presented. The dialogue history in this case is modelled as a stack of discourse units which represent the utterances, the corresponding speech acts as well as further information about the utterance such as speaker and initiative. Our system aims at being an intelligent context-aware system. We thus deploy an extensive context model following Bunt’s DIT concept. Our dialogue history has a sequential structure storing the dialogue states in the order they occur, similar to the approach adopted by the TRAINS project. Our dialogue history builds an essential part of the system due to the fact that it enables proactive interaction behaviour. All the systems described above (all of them are single-user systems) start building their dialogue history at the beginning of the conversation which denotes the system’s first interaction. As our system encounters a different situation with the users already speaking before the system interacts, the dialogue history has to model also this interaction between the users (if it is task-relevant) even before the system’s first interaction. Only then, the system is able to achieve full contextual awareness and can interact proactively (refer to Section 4.3.3 for the description of our dialogue history). We further integrate another sort of history within the structure of the task model to memorise already mentioned constraints as described in Section 4.5. 2.4.2 Dialogue Management The dialogue context4 is incorporated in dialogue systems in different ways and more or less detailed. Xu et al. (2002) differentiate dialogue systems in two dimensions regarding if the underlying dialogue model and task model is either explicit or implicit. An explicit dialogue model represents the linguistic context whereas an explicit task model offers means to represent the semantic 4

We treat the dialogue history as part of the dialogue context in this description.

38

2 Fundamentals

knowledge. Systems whose dialogue structure is predefined in form of a finite state machine model the context implicitly. The entire context is represented in the respective state implicitly, the linguistic context as well as task-related information and other context information relevant for the system. In framebased systems the task is explicitly represented and modelled in the frame; often little attention is paid to the dialogue model (e.g. [Chu-Carroll, 1999]). The opposite holds for systems motivated by linguistic theories rooted in semantics and pragmatics. Here, the linguistic and semantic context is represented very detailed, less attention is paid to the task model. Examples for such systems are the GODIS system [Larsson et al., 2000] which uses question under discussion (QUD), originally developed by Ginzburg and colleagues (1996), the IBiS systems [Larsson, 2002] and the EDIS system [Matheson et al., 2000] which applies grounding and obligation, put forward by Traum and colleagues (1994). These systems further deploy the Information State Update (ISU) approach which is presented in detail in the following section. So far, nearly all speech dialogue systems are exclusively concerned only with dyadic dialogue involving two participants. One of the first multiparty dialogue systems is developed by Traum and colleagues [Rickel et al., 2002, Swartout, 2006]. The Mission Rehearsal Project is used for training in the Army. A doctor, a village elder and a Seargent are interacting in a critical situation set in the context of a Middle Eastern country. They use PTT [Matheson et al., 2000, Poesio and Traum, 1998, Traum et al., 1999] for dialogue modelling. Based on the experience with this system, Traum proposes that the multi-party case in dialogue offers the chance for more exact theories of dialogue by showing the short-comings of different dialogue models which perform similar in the two-party case [Traum, 2003]. However, Traum considers only direct communication in the multi-party setup, limiting each contribution to one speaker addressing only one single addressee which does not resemble a fully-fledged multi-party situation. Dialogue management for multi-user dialogue systems is more complex than for single-user systems. The dialogue needs to consider the different dialogue participants. Each interaction partner has to be modelled individually as each one might have different believes and goals. Grounding is a complex process in multi-party interaction as each dialogue partner, and so also the system, has a common ground with every other dialogue partner. In order to achieve one’s goal, all relevant information from any of the witnessed conversations has to be integrated. For instance, if the system’s goal was to find a common solution between all of the users, it would collect the constraints from all of the users and utterances and uses the resulting set to find a common solution. Turn taking makes up another difference as multi-party interaction does not follow clear patterns. The dialogue modelling thus has to provide flexibility in turn-taking including repair mechanisms and ways for alternate proceeding e.g. in case that a different participant steals one’s turn. Depending on the requirements of each system, the modelling can be more or less

2.4 Dialogue Modelling

39

complex. For our system we adopt the Information State Update approach which models and thus understands dialogue step by step which we believe is an important objective for an intelligent dialogue system. The approach is straightforward deploying a sophisticated dialogue model and providing the flexibility to install extensive task and context models. The ISU approach is presented in detail in the following section followed by recent work on extending the approach towards the multi-party situation. 2.4.3 Information State Update Approach To Dialogue Modelling The Information State Update (ISU) approach is a common and suitable approach for extensive dialogue modelling. It models dialogue as a state of information out of the perspective of a dialogue participant. The so-called Information State (IS) is updated according to the content of the incoming utterances. Attempts to modify the ISU approach to make it multi-party capable have been conducted by Ginzburg and Fern´ andez (2005) and Kronlid (2008) and will be discussed in the following section. The remainder of this section is dedicated to a brief introduction of the general ISU approach. The ISU approach is mainly coined by Larsson [Cooper and Larsson, 1998, Larsson and Traum, 2000, Larsson et al., 2000, Larsson, 2002] and Poesio and Traum [Traum et al., 1999, Matheson et al., 2000] and rooted in Ginzburg’s dialogue gameboard theory (1996). Example implementations have emerged especially out of the context of the Trindikit [Larsson and Traum, 2000], a framework for implementing and experimenting with systems using the ISU (such as e.g. EDiS [Matheson et al., 2000], GODIS [Larsson et al., 2000] and it’s successors IBiS1-4 [Larsson, 2002]). The information state theory typically consists of the following components [Larsson, 2002]: • A description of the informational components that should be represented in the IS. • Formal representations of the above components. • A set of dialogue moves that trigger the updates on the IS. • A set of update rules defining the update operations performed on the IS. • An update strategy to decide upon which rules to deploy on the IS at any given point. The information state is the central part of the ISU describing the dialogue at the current point in time. It is able to describe complex parts of a conversation, such as the dialogue participant’s mental state (beliefs), obligations (e.g. to respond) and goals (as well as ways to achieve these). The IS consists of the dynamic information of the dialogue context as described above and models the dialogue at a specific state from the viewpoint of one dialogue

40

2 Fundamentals

participant. It is divided into two parts. The private part contains information known only by the participant that is modelled, such as the participant’s own beliefs, agenda and plans regarding the ongoing conversation. The public part contains information which is believed to be grounded and known by all participants of the dialogue. The public part could either be modelled as shared between all participants [Larsson, 2002] or agent-specific as done by Traum [Traum et al., 1999, Matheson et al., 2000] and Ginzburg (1996) who therefore call it quasi-shared. In the agent-specific case the view on grounded information is believed to be different for each participant. Generally, however, if the interaction is successful, all DPs end up having the same grounded information5 . As an example, the IBiS1 information state is depicted in Figure 2.2. It consists of a private and a shared part and contains the following components: • AGENDA describes actions to perform in the near future. • PLAN contains longer-term actions to achieve a certain goal. • BEL (beliefs) is used to store the results of the database queries, i.e. information only known to the modelled DP. • COM (commitments) contains propositions that the user and system have mutually agreed upon during the dialogue. • QUD (questions under discussion) is a stack of questions raised in the dialogue that have not yet been resolved. The topmost element is the question that is currently being discussed. • LU (latest utterance) contains the information about the latest utterance, i.e. speaker and dialogue move. ⎡

⎡

⎤

⎤

AGEN DA : Stack(Action) ⎥ ⎢ P RIV AT E : ⎣ P LAN : Stack(Action) ⎦ ⎥ ⎢ ⎥ ⎢ BEL : Set(P rop) ⎢ ⎡ ⎤⎥ ⎥ ⎢ COM : Set(P rop) ⎥ ⎢ ⎢ ⎥⎥ ⎢ QU D : Stack(Question) ⎢ SHARED : ⎢ ⎥   ⎥ ⎣ ⎣ SP EAKER : P articipant ⎦ ⎦ LU : M OV E : M ove Fig. 2.2. Example information state as deployed in IBiS1 [Larsson, 2002].

Every utterance of the modelled dialogue induces an update of the IS. The possible updates are determined by update rules which consist of preconditions and effects (both working on the IS), the rule name and the name of the class the rule belongs to. All preconditions have to be true for a rule to fire 5

Taken that no malicious agents are involved in the interaction who e.g. purposely deceive other agents making them believe something which the others don’t believe.

2.4 Dialogue Modelling

41

which induces actions on the IS as defined by the effects part of the rule. The update rules are grouped into certain classes according to their functionality, e.g. rules for the integration of the semantic content of an utterance or rules for general grounding procedures, to load plans, or to select next moves. The order in which the different classes of update rules should fire is regulated by an update strategy. Algorithm 1 shows an example update algorithm, again borrowed from the IBiS1 system [Larsson, 2002]. It consists of conditions and calls to classes of update rules. if NOT latest move == failed then apply clear(/PRIVATE/AGENDA); getLatestMove; integrate; try downdateQUD; try loadPlan; repeat execPlan end

Algorithm 1: Update algorithm of IBiS1 [Larsson, 2002].

The update rule getLatestMove fires as the first rule when a new utterance comes in. It loads the data of the latest utterance into the according parts of the IS (MOVE and SPEAKER field of LU) to make it accessible for further operations. The rule looks as follows6 : RULE: getLatestMove; CLASS: grounding; PRE: $Latest Move==Move $Latest Speaker==Participant EFF: copy(/SHARED/LU/MOVE,Move) copy(/SHARED/LU/SPEAKER,$Latest Speaker)

After loading the latest utterance the data has to be integrated. Thus, as a next step in the algorithm, a rule of the integrate class of update rules is called. Thereby, the rule whose complete set of preconditions are true will fire. Taken for instance, that the latest move was a question posed by the user, the update rule integrateUsrAsk will fire. The rule looks like this: The preconditions of the rule check if the latest speaker was the user and if the latest move was an ask move. The firing of this rule causes an update of QUD, i.e. q is pushed on top of QUD as it denotes the new question that is currently under discussion. Further, an action stipulating a response to the question q is pushed on the AGENDA. Analogously to this rule, 6

The notation used here is slightly modified from Larsson’s notation (2002).

42

2 Fundamentals RULE: integrateUsrAsk ; CLASS: integrate; PRE: $/SHARED/LU/SPEAKER==usr in($/SHARED/LU/MOVE,ask(q)) EFF: push(/SHARED/QUD,q) push(/PRIVATE/AGENDA,respond(q))

integrateSysAsk and integrateAnswer are examples for rules that handle dialogue moves7 . Further update rules are deployed to manage plans and actions. They are defined by the rule classes select move, select action, find plan, exec plan, etc. Example rules are selectAsk, raiseIssues, findPlan, managePlan. Interaction protocols emerge from application of the update rules to a dialogue describing a common flow of a conversation. Table 2.2 shows a simple query and an assertion protocol, slightly modified from their specification in [Ginzburg and Fern´ andez, 2005]. Query

Assertion

LatestMove == ask(A:B,q)

LatestMove == assert(A:B,p)

A: push(QUD,q); release turn

A: push(QUD,p?); release turn

B: push(QUD,q); take turn; make max-qud-specific utterance; release turn

B: push(QUD,p?); take turn; Option 1: Discuss p?; Option 2: Accept p LatestMove = Accept(B:A,p) B: increment(FACTS,p); pop(QUD,p?); release turn A: increment(FACTS,p); pop(QUD,p?)

Table 2.2. Query and assertion interaction protocols of IBiS1.

The presented components of the ISU approach are taken from the IBiS1 system [Larsson, 2002] which assumes perfect communication at all times in terms of that all utterances are understood and accepted and no references are used (all of which occurs frequently in natural dialogue). The successor 7

Which dialogue move is assigned to an utterance is defined by the relation between the content and the activity in which it occurs. The set of dialogue moves deployed by the IBiS1 system are ask, answer, greet and quit.

2.4 Dialogue Modelling

43

system IBiS2 modifies the information state structure to enable grounding which is performed by adding temporary fields in the private part of the information state to store the information before it is grounded (at which point it is integrated into the according fields). We do not go into details here and refer to [Larsson, 2002] for further information. Another extension worth mentioning, however, is a field that holds the previous move (the one before the Latest Move) which besides checking the next move for relevance (what it has been originally designed for by Larsson) also enables backtracking, i.e. that false assumptions can be taken back. This is a first step towards a dialogue history which we claim to be necessary for a complex dialogue system with multiple dialogue participants involved. In the following section we present existing approaches that extend the ISU approach to handle multiparty interaction building on the principles introduced in this section. 2.4.4 Multi-Party Dialogue Modelling Introducing additional participants into the conversation affects the dialogue in various ways (e.g. refer to Section 2.3, [Traum, 2003, Ginzburg and Fern´ andez, 2005, Kronlid, 2008]). It cannot be assumed that all dialogue participants share the same common ground, the one established during the conversation as well as the common ground built during previous interactions of the participants. Thus, the more participants take part in the conversation, the less common common ground is presumable. In two-party conversation turns switch constantly between the two participants, when one speaks the other one is addressed and takes the next turn. In multi-party conversation turn-taking is not that simple; many different situations can occur. Thus, interaction protocols have to be more flexible to be able to consider all the possible situations. Another issue regards the fact that in multi-party dialogue a differing number of participants can be addressed simultaneously. Either one single participant or various participants are addressed, or possibly both situations occur within one single turn. A side-participant could steal the turn and speak instead of the intended addressee. Does this take away the obligation of the addressee to answer? Is a question considered resolved (and can be taken from QUD) after being answered by one participant while a set of participants was addressed? Below, two approaches to multi-party dialogue modelling are introduced which address and try to answer some of these questions. Traum’s approach to multi-party dialogue management [Traum, 2003] regards multi-party dialogue as sets of pairs of two-party dialogue which simplifies the multi-party situation in enabling the use of existing dialogue models. Only direct addressing is considered, i.e. one speaker and one addressee. Different conversations between different pairs of interaction partners may take place at the same time. For natural multi-party interaction, this approach is not considered flexible enough. The first approach which is discussed in the following is introduced by Ginzburg and Fern´ andez (2005). They were the first to extend the ISU ap-

44

2 Fundamentals

proach [Larsson, 2002] to enable multi-party dialogue (or multilogue as they call it). They introduce a way to scale up the interaction protocols to integrate overhearers and side-participants into the protocols and thus enable for distributive and collective addressing and answering behaviour of groups of addressees. The extended protocols regard the different conversational roles presented in Section 2.3.2 and two benchmarks they derive from investigating multi-party dialogues [Ginzburg and Fern´ andez, 2005]: • Multilogue long distance short answers (MLDSA): Querying protocols for multi-party dialogue must license short answers an unbounded number of turns from the original query. • Multilogue adjacency of grounding/acceptance (MAG): Assertion and grounding protocols for multi-party dialogue should license grounding and acceptance moves only adjacently to their antecedent utterance or after an acceptance of a fellow addressee. Ginzburg and Fern´ andez (2005) thus propose three possible modifications of the interaction protocols used for conversational update in multi-party situations according to the principles shown in Table 2.3. Add Overhearers (AOV): Silent overhearers are added as recipients of the informative speech act, i.e. they add what has been said to their common ground without the chance to interact for clarification or feedback. Duplicate Responders (DR): The addressee role is multiplied. Each of the responders updates the information state in the way an addressee does and is expected to respond, one after another, in the form of distributive acceptance. This protocol extension does not enable the reaction of a later responder to an earlier responder’s utterance (which however occurs regularly in conversations). Add Side Participants (ASP): The audience exists of a set of DPs who update their information state without responding. One member of the audience instantiates the addressee role and acts as a sort of spokesperson whose acceptance counts for the whole audience. This protocol extension enforces communal acceptance. Table 2.3. Interaction principles by Ginzburg and Fern´ andez (2005).

In the following, the interaction protocols that have been presented in the foregoing section (refer to Table 2.2) are listed after the modifications performed according the multi-party principles suggested by Ginzburg and Fern´ andez (2005). Table 2.4 shows the querying protocol adopting each of the three proposed principles.

2.4 Dialogue Modelling Query + AO

Query + DR

Query + ASP

LatestMove==

LatestMove==

LatestMove==

ask(A:{B,C1 ,..,Cn },q)

ask(A:B,q) A: push(QUD,q);

A: push(QUD,q);

ask(A:{B,C1 ,..,Cn },q) A: push(QUD,q);

release turn

release turn

release turn

B: push(QUD,q);

B: push(QUD,q);

B: push(QUD,q);

take turn; make max-qud-spec utt; release turn Ci : push(QUD,q)

45

take turn;

take turn;

make max-qud-spec utt;

make max-qud-spec utt;

release turn F or all Ci : push(QUD,q);

release turn Ci : push(QUD,q)

take turn; make max-qud-spec utt; release turn Table 2.4. Interaction protocol query extended to the multi-party situation following the principles of Ginzburg and Fern´ andez (2005).

Kronlid (2008) argues that this approach is insufficient to adequately handle multi-party situations. While the above modifications allow for the distinction of communal and distributive sort of questions, they are not very flexible. Applying DR might be suitable, for instance, for a disagreeing audience or a tutor and student situation, however, in other situations it would seem unnatural to have every addressee of an audience respond one after the other; and only to the principally uttered question due to the fact that it further does not allow for a reaction of one DP to the acceptance of another dialogue participant. ASP might be too restrictive in general. Kronlid defines one of the main challenges of modelling multi-party dialogue the identification of the sort of questions asked. He differentiates between individual, communal and distributive questions. Each type of question specifies a different set of responders. Figure 2.5 shows the principle Kronlid introduces to multiply the addressee role [Kronlid, 2008]. Add Multiple Addressees (AMA): The responder role is duplicated, the responders’ max-qud-specific contributions are optional. Table 2.5. Interaction principle by Kronlid (2008).

The anticipated behaviour lays somewhat in between DR and ASP allowing but not requiring all addressees to respond. One of the addressees

46

2 Fundamentals

responds (by self-selection) after which all other addressees have the chance to do the same. The querying interaction protocol extended with Kronlid’s AMA principle is shown in Table 2.6. Query + AMA LatestMove==ask(A:{B,C1 ,..,Cn },q) A: push(QUD,q); release turn B: push(QUD,q); take turn; make max-qud-spec utt; release turn Ci : push(QUD,q); ( Optional: take turn; make max-qud-specific utterance; release turn ) Table 2.6. Interaction protocol query using the AMA principle.

Applying AMA allows for communal as well as distributive acceptance. However, Kronlid suggests to deploy DR for distributive questions. Commenting on earlier answers or comments of fellow addressees is enabled by introducing acceptance moves and changing the way in which QUD functions. Kronlid extends Larsson’s IBiS1 system to Multi-IBiS (2008) for multi-party interaction. The information state of Multi-IBiS is depicted in Figure 2.3 followed by a description of its components. The main modifications that were performed on the traditional IBiS1 version of the information state are presented in the following: • AGENDA: The agenda is implemented as a queue instead of a stack for better handling of the chronological order. Agenda items are changed in the way that they contain the DP to address as a second argument, if necessary. • PLAN: The plan element is divided into three parts. – ISSUE: The issue which the plan aims to resolve. – THE PLAN: The actual plan. Plans deployed in Multi-IBiS are extended to include the name of a DP who should be addressed to resolve the plan. – OPEN FOR ME: A boolean value denoting if the modelled DP can contribute to the plan or not.

2.4 Dialogue Modelling

⎡

⎡

⎤

47

⎤

AGEN DA : ⎡ Queue(Action) ⎤ ⎢ ⎥ ⎥ ⎢ ISSU E : Question ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎢ P RIV AT E : P LAN : ⎣ T HE P LAN : Stack(Action) ⎦ ⎥ ⎢ ⎥ ⎥ ⎢ ⎣ ⎦ ⎥ ⎢ F OR M E : Boolean OP EN ⎥ ⎢ ⎥ ⎢ : Set(P rop) ⎢ ⎤⎥ ⎡ BEL ⎥ ⎢ COM : Set(P rop) ⎢ ⎛⎡ ⎤⎞ ⎥ ⎢ ⎥⎥ ⎢ ISSU E : Question ⎢ ⎥⎥ ⎢ ⎢ ⎜⎢ SP KR : P articipant ⎥⎟ ⎥ ⎥ ⎢ ⎢ ⎜⎢ ⎥⎟ ⎥ ⎥ ⎢ ⎢ ⎟⎥⎥ ⎢ QU D : StackSet ⎜⎢ ASET : Set(P articipant) ⎥ ⎢ ⎜⎢ ⎥⎟ ⎥ ⎥ ⎢ ⎥ ⎢ ⎢ ⎝⎣ OSET : Set(P articipant) ⎦⎠ ⎥ ⎢ SHARED : ⎢ ⎥⎥ ⎢ ⎥⎥ ⎢ ST AT U S : [OPEN|CLOSED] ⎢ ⎥⎥ ⎢  ⎢ ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢ P U : SP KR : P articipant ⎢ ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢  M OV E : Set(M ove)  ⎣ ⎦⎦ ⎣ SP EAKER : P articipant LU : M OV E : Set(M ove) Fig. 2.3. Multi-IBiS information state [Kronlid, 2008].

• QUD (questions under discussion): is from now on a list allowing addressing of items that are not maximal. – ISSUE: The question. – SPKR: DP who is raising the question. – ASET: Set of DPs who are directly addressed and who also have the right to address the issue. – OSET: Set of DPs with the obligation to address the issue. – STATUS: Addressing status of the issue. It signals if the ASET or OSET could still be further extended (which could be the case while the current utterance is not finished). • PU (previous utterance): The PU field is borrowed from Larsson’s IBiS2 implementation (2002) where it was called PM. While LU holds the latest utterance, PU holds the same information about the utterance before LU. This enables an agent to respond to something uttered up to two turns ago as well as backtracking in case of overhasty grounding due to the cautiously optimistic grounding strategy that is deployed. The Multi-IBiS data model contains further elements besides the information state [Kronlid, 2008]: • Name of the modelled dialogue participant • Names of all dialogue participants: At the beginning of the interaction the list is empty. Participants are added as soon as one starts speaking.8 8

This way, it does not seem possible to consider silent side-participants as addressees. Nothing is said how this case is handled.

48

2 Fundamentals

• Addressing information, i.e. which participant is currently addressing which other DP(s). The information is accessed by the QUD object. When a participant addresses a different DP, the information is updated accordingly. By extending QUD to include the set of addressees (ASET) and obligated responders (OSET) the fulfilment of obligations to answer can be observed. The ASET and OSET fields contain the same participants at the beginning. With every answer, the name of the current DP is removed from OSET. When all obligated responders have answered OSET is empty. For communal acceptance, the set is emptied after one answer. If ASET is empty from the start the addressees are determined by addressing-by-attribution, i.e. participants self-select and make a contribution if they know the answer. Kronlid modifies the dialogue moves and update rules in order to handle multi-party dialogue. The structure of the update rules is changed as the simple assumption of a perfect two-party dialogue with the speaker and responder switching every turn does not apply for the multi-party case. Therefore, every move has to be integrated regardless of the speaker and addressees (getLatestMove is omitted). The update rules need to differentiate between individual, collective and distributive questions. Kronlid’s approach does not suggest a solution to identify each question according to its type. The context is not taken into account; it is rather assumed that a certain question is always of a certain type. Further changes towards the multi-party situation include renaming of some rules, e.g. integrateUsrAsk is changed to integrateOtherAsk, and the extension of all rules to integrate the ASET and OSET to verify if a DP is permitted or obligated to answer. By introducing an acceptance move accept(q) that is added on QUD when a participant answers q, the subsequent speaker is enabled to address this acceptance move or q. QUD is thus not downdated after any resolving answer but q is left on QUD until the latest move does not contain a q-specific move. Kronlid addresses the prevalent multi-party specific questions with his approach in the following way. Whether an obligated addressee is released from the obligation to answer if another participant answers instead, depends on the type of question raised. If the question is distributive the obligation continues and the participant is expected to answer. In the case that the question is of communal character, the participant is released from the obligation. In our case, a distinction of the different question types is not necessary as all questions are of individual nature. Analysing the recordings shows that if the side-participant (who is not addressed) interrupts the conversation it is done out of a significant reason and mostly entails a response from the former speaker outweighing the response from the former addressee and thus taking away the obligation. The question regarding when issues are resolved is also dealt with in the way that the QUD as well as the conditions for downdate

2.5 Summary

49

were loosened in Kronlid’s approach. In our setup, we adopt this and some other modifications that Kronlid conducted on the original information state, however, put more focus on the task model and dialogue history (refer to Section 4.3.3). Regarding the extensions for the interaction protocols none of the suggested principles fully serves our needs. While Ginzburg’s ASP is not flexible enough, Kronlid’s approach considers multiple addressees (a situation we do not encounter). Thus, we suggest a new principle which is presented in Section 4.1.

2.5 Summary In this chapter fundamentals of different techniques and research areas were provided as a basis for the remaining chapters of this book. The Wizard-of-Oz data collection technique introduced in Section 2.1 is deployed for our data recordings presented in Chapter 3. The introduction on dialgoue system evaluation is relevant for Chapter 5 which presents the evaluation conducted for our dialogue system. The linguistic fundamentals and general human-computer interaction discussions (Section 2.3) are considered for the development of the dialogue management component (Chapter 4) which is based on the Information State Update approach that was introduced in Section 2.4 together with existent multi-party extensions. We discussed that these extensions to the ISU approach do not sufficiently cover the novel situation that our system encounters. We thus introduce a new interaction principle to allow for proactive interaction of side-participants and adapt the data structures to the multi-party situation of our system, as presented in Chapter 4.

3 Multi-Party Dialogue Corpus

Adequate dialogue data is needed to investigate the multi-party interaction with a computer system as an independent dialogue partner in the conversation with several humans. A look is taken at existing multi-party corpora in order to see if any of these could be used to investigate our research questions. The presented collection is limited to corpora that make use of multiple modalities. Although a wide variety of corpora is available, as far as the authors are aware, there is no existent collection of data that stresses the designated features important for our research. Thus, we perform the data collection presented in this chapter. We deploy the Wizard-of-Oz technique as introduced in Section 2.1 to simulate the envisaged system in the example domain of restaurant selection (refer to Section 1.3 for a description of the system) to obtain realistic interaction data which is then used to assist the development of the system (Chapter 4) and for evaluation (Chapter 5). The WOZ recording setup and procedure are presented in Section 3.2 followed by a detailed description of the software tool we have developed to support the recordings. The tool is easily adaptable to other domains and requirements and publicly available for other developers. The collected data results in the PIT corpus which is finally presented in Section 3.3.

3.1 Existing Multi-Party Corpora Recent multi-party corpora are presented in the following. Only corpora that deploy multiple modalities are considered and thus, corpora focusing on the audio modality only, such as the ICSI corpus [Janin et al., 2003] and the ISL audio corpus [Burger et al., 2002, Burger and Sloane, 2004], are not listed. AMI(DA) corpus In the framework of the AMI (Augmented Multi-party Interaction) project [Renals, 2005] which aims at developing meeting browsing technology as well as P.-M. Strauß and W. Minker, Proactive Spoken Dialogue Interaction in Multi-Party Environments, DOI 10.1007/978-1-4419-5992-8_3, © Springer Science + Business Media, LLC 2010

51

52

3 Corpus Development

remote meeting assistants, a multimodal data corpus of about 100 hours of meeting recordings was collected [Carletta et al., 2005]. Some of the meetings were naturally occurring meetings, others were scenario-driven meetings put on for the recordings. The meetings had generally four participants and were held in English with a large proportion of the speakers being non-native English speakers. Instrumented meeting rooms were set up on three different sites that were equipped with individual and room microphones, individual and long-shot video cameras, individual electronic pens, presentation slide capture and white-board capture devices. CHIL corpus The CHIL Audiovisual Corpus for Lecture and Meeting Analysis inside Smart Rooms [Mostefa et al., 2007] was created in the framework of the CHIL (Computers in the Human Interaction Loop) project. It consists of synchronised video and audio streams of real lectures and meetings recorded in five smart rooms. Each of the rooms is equipped with a large amount of recording equipment, for instance a minimum of five cameras and 88 microphones per room. The interaction scenarios are meetings (40) and lectures (46). Both consist of one person giving a presentation in front of an audience, differing in the size of the audience (in lectures ten to twenty persons, in meetings three to five) and interaction (a lot of interaction in meetings versus few interaction with the audience in lectures where the focus lies only on the presenter). Interaction language is English but most of the speakers have a non-native English accent. Lectures were between 40 and 60 minutes long, meetings approximately 30 minutes. Manual annotation includes multi-channel verbatim orthographic transcription of the audio modality that includes speaker turns and identities, acoustic condition information and name entities for part of the corpus. The corpus is used within the CHIL project in order to develop audiovisual perception technologies for human activity analysis during lectures and meetings. This includes person localisation and tracking, person and speaker identification, face and gesture recognition, speech recognition, emotion identification, acoustic scene analysis, topic identification, head-pose estimation, focus-of-attention analysis, question answering and summarisation. MSC1 Corpus The MSC1 Corpus [Pianesi et al., 2007] and its successor corpus [Mana et al., 2007] consists of task-oriented meetings of four persons that are to solve a survival task. The data is used to study social behaviour and personality traits from audio-visual cues such as 3D-body tracking and speech activity. The aim is to develop a system automatically predict these personality traits using the audio-visual cues. The meetings of MSC1 were on average around 19 minutes long. The successor corpus features 52 participants in 13 sessions recorded by cameras and microphones and includes also task-success measures.

3.1 Existing Multi-Party Corpora

53

NIST Corpus The NIST audio-visual corpus of the American National Institute of Standards and Technology1 consists of multimodal multi-channel data sets recorded in a smart room using a variety of microphones and video cameras (enabling 2D-tracking). The recorded meetings are partly real, partly scenario-driven. The number of participants per meeting range from three to nine. The first part of the corpus [Garofolo et al., 2004] contains 19 meetings resulting in approximately 15 hours of data. 61 participants were involved, about a third of them are non-native English speakers. Phase two of the corpus [Michel et al., 2007] contains 17 meetings and approximately 19 hours of data, involved a total of 55 subjects and only four of them non-native English speakers. VACE Corpus The VACE Meeting corpus [Chen et al., 2005] was collected in order to support research on understanding meetings by analysing multimodal cues, such as speech gaze, gestures and postures to understand meetings. The meetings were held in the domain of war game scenarios and military exercises, recorded in an instrumented lecture room holding up to eight participants sitting around a meeting table. Each participant is recorded by various microphones and at least two video cameras to enable 3D tracking of the head, torso, shoulders and hands. M4 Corpus Audio-visual corpus of the MultiModal Meeting Manager (M4) project [McCowan et al., 2005] features meetings of four participants, scripted in terms of type and schedule of group actions or note taking. Recordings were conducted with various microphones and video cameras enabling 2D-tracking. A sort of subcorpus of the M4, the multimodal corpus described in [Jovanovic et al., 2006] is put together consisting of twelve meetings recorded at the IDIAP smart meeting room [Moore, 2002], ten dialogues from M4, one from AMI and one other. Mainly four participants took part in each meeting with a total of 23 participants. The total duration of the corpus is approximately 75 minutes. The research aim pursued with this data collection is to study addressing behaviour in face-to-face conversations. The corpus is handannotated with dialogue acts, adjacency pairs, addressees and gaze directions of meeting participants. MRE Corpus A multimodal corpus of task-oriented multi-party dialogues [Robinson et al., 2004] was collected in the scope of the Mission Rehearsal Exercise project 1

http://www.nist.gov

54

3 Corpus Development

(MRE) [Swartout et al., 2005]. The aim of the project is virtual reality training of a decision-maker in a multi-party mission-related setting. One human trainee is interacting with several virtual characters. The corpus comprises approximately ten hours of audio (human simulation radio data) and approximately five hours of video and audio face-to-face interactions between human trainees and virtual agents. Part of the dialogues were recorded using the Wizard-of-Oz technique (ref. to Section 2.1). ATR Corpus The corpus referred to by [Campbell, 2008] was collected at ATR (Advanced Telecommunications Research Institute International2 ) with the purpose of interaction analysis focusing on engagement in dialogue. It contains (besides dyadic dialogues) a set of conversations (not task-oriented meetings) with four participants from different cultural backgrounds. The dialogues were audio and video recorded enabling head and body tracking. Our research aim is to study human-computer interaction in a multi-party scenario. The question is how the computer is integrated in the conversation as an independent dialogue partner. Thus, data suited for the analysis needs to feature the aimed setup of dialogue partners. All (but one) of the described multi-party corpora feature various human dialogue partners, however, none of them deploys a virtual interaction partner. The MRE corpus is, as far as the authors are aware, up to now the only corpus deploying virtual characters. The data, however, is not suitable for our research as it features multiple virtual characters interacting with a single human while we demand multiple humans interacting with a single computer system. Our focus lies further on task-oriented dialogue, i.e. the interaction pursues a certain goal and ends when the task has been solved. The dialogue system is to play the role of the expert and contributes to solving the task. Dialogues that feature multiple humans involved in the same sort of dialogue could possibly be used for analysis in order to learn the designated behaviour the system should possess from the human expert. However, task-oriented interaction data of this kind is not available in the form of multi-party dialogues, most of the presented corpora act in the meeting domain which underlies different principles of interaction than task-oriented dialogue and is thus not suitable. As far as the authors are aware, there is no existent collection of data stressing the designated features important for our research. Therefore, we build our own data corpus which enables us to study the interaction of the humans with the computer system. The data collection procedure and corpus are presented in the remainder of this chapter.

2

http://www.atr.jp

3.2 Wizard-of-Oz Data Collection

55

3.2 Wizard-of-Oz Data Collection An extensive Wizard-of-Oz environment was set up for the collection of multimodal data. The recordings took place in the scope of the research project ’Perception and Interaction in Multi-User Environments - The Computer as a Dialogue Partner’ conducted within the competence centre Perception and Interactive Technologies (PIT) at Ulm University (Germany) whose aim is the development of components and technologies for intelligent and user-friendly human-computer interaction in multi-user environments. Over the time span of 18 months 76 dialogues were recorded in three recording sessions. The setup stayed the same over all three sessions while the system for the wizard interaction was improved from one session to another introducing new features and speeding up the reaction time. The final system is described Section 3.2.4 and in more technical detail in Appendix A. 3.2.1 Experimental Setup The setup of the system is shown in Figure 3.1. The human dialogue partners U1 and U2 interact with the system S which is operated by the human wizard situated in a different room. U1 is the system’s main interaction partner. The

Fig. 3.1. Data collection setup.

56

3 Corpus Development

dialogue system server runs on the computer S. It produces acoustic output in form of synthesised system utterances and visual output consisting of an avatar and further items displayed on the screen (restaurant’s menu, city map or bus schedule). The wizard’s computer is connected to S, i.e. the dialogue system server, via network connection. The wizard controls the system hearing what the users are saying through microphones M1 and M2 whose signals are transmitted via wireless connection and recorded on the wizard’s computer. A webcam is further installed pointing towards the screen for the wizard to check if the system’s display is working correctly. Audio Recordings The speech signals are recorded by three different microphones. One lapel microphone for each human dialogue partner (M1, M2)3 and one room microphone (M3)4 to capture the entire scene including the system output. The signals recorded by the lapel microphones are transmitted via wireless connection to the wizard’s computer where they are recorded. All audio data is recorded at 16 kilohertz with 16 bit resolution – the standard quality for speech recognition. External soundcards5 are used for improved quality and to be independent of the recording computer. The audio signals from the room microphone are recorded on a MiniDisc recorder6 . Video Recordings Three video cameras are installed to record the dialogues (C1-3)7 . For complexity reasons we use one human user as the main interaction partner (U1) for the system and consider only this user’s gaze. Thus, Camera C1 is responsible for the recording of U1’s gaze direction. Camera C2 records user U1’s perspective, C3 captures the entire scene from the long shot. Figure 3.2 shows the scene from the viewpoint of cameras C3 (left) and C1 (right). 3.2.2 Procedure The recordings proceed in the following way: Two participants take part in each recording. Before the interaction with the system they fill out the first part of the questionnaire and are given a random scenario as a guideline for the interaction. After the recording, the second part of the questionnaire is completed. From the system’s perspective, the wizard has to follow certain guidelines in order to achieve homogeneous recordings. In some of the recordings, an emotion eliciting strategy is deployed as described below. 3 4 5 6 7

AKG 97L with AKG WMS40 AKG 1000S CREATIVE Sound Blaster 24-bit S80300 SONY MZ-R700 JVC GR-D270E

3.2 Wizard-of-Oz Data Collection

57

Fig. 3.2. Video recordings from the viewpoint of cameras C3 (left) and C1 (right) during a Session I dialogue [Strauß et al., 2007].

Participants The participants (n=152)8 were students and employees of the university who gave written consent to participate in this study. They were between 19 and 51 years of age (on average 24.4 years); 53 of them were female (4 at session I (10.5%), 18 at session II (45.0%), 31 at session III (41.9%)). Except for seven participants (I: 2, II: 4, III:1) the native language of all participants is German. The main professional backgrounds are computer science (n=41, i.e. 27.0%) and natural sciences (n=27, i.e. 17.8%). Engineering and medicine are equally represented by 24 persons (15.8%). The daily usage time of computers of 149 participants lies between 10 and 720 minutes, with an average of 252 minutes. Computer experience lies between 4 and 21 years with an average of 11.43 years. The participants can be said to be very familiar with technology and adept with computers. For participation they did not receive any compensation other than coffee and cookies. Questionnaires A comprehensive questionnaire was completed by the participants prior as well as subsequent to the interaction with the system for evaluation purposes. The questionnaires collected data about the participants, self-assessment about technical skills, as well as usability and subjective rating of the system. A detailed description of the questionnaires and evaluation results is given in Section 5.1.

8

150 of the participants filled out the questionnaire, the presented data is hence based on this number.

58

3 Corpus Development

Scenarios We randomly assigned different scenarios to the dialogue partners to supply them with a starting point and theme of the dialogue. For that, each dialogue partner received a few statements describing the situation and role to adopt in the dialogue. An example of such a scenario is shown in Figure 3.1. The specified information includes e.g. the relation to the other dialogue partner, a motivation for the restaurant visit or culinary preferences. The combinations of roles include amongst others employer and employee, business colleagues, friends, or a couple in a loving relationship. Occasionally, the roles contained contrary preferences to induce a lively and emotional discussion. Giving the fact that people act differently according to what kind of roles they possess in a conversation (refer to Section 2.3.2) we tried this way to obtain interesting and diverse dialogues in terms of dialogue behaviour, e.g. relating equality or dominance of the dialogue participants. For instance, assuming a conversation between professionally unequal dialogue partners, it is likely for the ’superior person’ to also play a superior role in the dialogue. Less objection and conflict is detected in this case, the ’inferior person’ is more likely to give in. Whereas, in the case of two friends discussing their choice of a restaurant, it can be observed that constraints are changed more often as the dialogue partners articulate their own preferences more freely, trying to convince the counterpart. Although the scenarios and roles are fictitious, these phenomena can be observed in the recorded dialogues. Thus, a wide range of different behaviour is achieved in the corpus of dialogues. For the major part, the participants followed the role play throughout the dialogues. In a few cases, however, the scenario was used only to stimulate the conversation and not referred back to later on in the dialogue. Situation: Two colleagues want to go out to eat after a long day at work. It is about 7 in the evening and the sun is still shining.. Person A - dialogue partner of computer - loves Greek food - had a bad day (is tetchy) - is short in money

Person B - is disgusted by garlic - likes plain German food

Table 3.1. Example scenario description.

3.2.3 System Interaction Policies One of the characteristics of computers is deterministic behaviour. In order to perform an adequate simulation, it is therefore necessary to define the sys-

3.2 Wizard-of-Oz Data Collection

59

tem’s behaviour. Wizard guidelines have to be drawn up to determine how the wizard should control the system to assert uniform system behaviour throughout all dialogues in order to receive unbiased dialogue data. We deploy two different strategies for interaction: One for standard interaction simulating a perfectly working system and another one to elicit emotional behaviour from the users. Both strategies start with the same general principles: The conversation between the two dialogue partners starts without system interaction while the users act out the provided scenarios. The differences emerge once the system has joined in the conversation following the policies as pointed out in the following paragraphs. Standard Policy During a standard recording the system, or wizard respectively, does not interrupt the conversation of the users. Speech recognition and language understanding are simulated to be perfect. Correct and helpful answers are given as frequently and as promptly as necessary and wherever possible. The system’s first and further interactions are triggered by one of the following situations: • Reactively upon user U1 addressing the system directly • Proactively in order to make a significant contribution to the conversation (e.g. to report a problem in the task solving process) • After pauses in the dialogue exceeding a certain threshold (if a meaningful contribution can be made) In the case where U2 addresses the system directly the utterance is recognised, however, no direct response should be given. Yet, two different reactions can be observed during the dialogues. Either, user U1 instantly takes the turn and poses the same or similar request to the system, or U2’s request is followed by a pause which again would mostly lead to an interaction by the wizard. After the users have found a suitable restaurant the wizard generally closes the dialogue saying goodbye and thanking the users. In some cases, however, the participants decide to search for another location, such as a bar for a cocktail after dinner etc. Overall, the recordings that adhere to the standard policy resemble a perfectly working dialogue system. Thus, the data deriving from these recordings are suitable for usability evaluation of the envisaged dialogue system. Emotion Policy An average dialogue that is recorded deploying the scenarios described above attains (besides neutral data) the emotional characteristics of happiness and at times surprise (e.g. when the avatar appears on the screen for the first time). In order to obtain data which exhibits a wider range of emotions, we try to induce emotions by directing the system’s reaction in a certain way.

60

3 Corpus Development

Thus, instead of simulating a perfectly working system the wizard steers the system’s reaction in an unexpected way. The system does not appear perfect but rather rude and faulty in order to provoke anger, boredom, annoyance and surprise. The following types of errors and disorders are deployed throughout the entire conversation: • • • • •

Speech recognition errors Understanding errors Wrong answers Occasional interruptions of the users Arbitrary pauses or omitting

For instance, if a user says “not expensive restaurant” the wizard returns a selection of expensive restaurants ignoring the “not”. As another example, the wizard might purposely suggest a restaurant that does not match the users’ preferences or is even situated in a different city. Needless to say that the emotions expressed by the users are more moderate than artificial emotions played by actors, as e.g. in the Berlin Database of Emotional Speech [Burkhardt et al., 2005], a prevailing corpus for research on emotion recognition. We consider these moderate emotions as more realistic and common in human-computer interaction and therefore very useful for affective computing tasks which, however, are not within the scope of this work. In Section 5.1.4 the usability ratings of the dialogues recorded using the two different interaction policies are compared to each other. 3.2.4 WIT: The Wizard Interaction Tool The dialogue system in use for the Wizard-of-Oz recordings has to be built so as to simulate the envisaged system’s behaviour as closely as possible. We developed the Wizard Interaction Tool (WIT) which allows fast and faultless wizard interaction. It is a client-server tool implemented in Java and freely available for download9 and customisation to individual needs, domain and also languages [Scherer and Strauß, 2008]. The tool enables the wizard to replace the modules that are not yet functional such as speech recognition and semantic analysis. As a further interaction point, the wizard can check the automatically generated prompts before they are sent to the synthesiser. The first recording session was performed while the development of the system was still in its early stages. For the second recording session, the location of the setup was changed and the system had evolved to a more elaborate client server architecture still adding further functionality for the third and final WOZ recording session. The variances in the system setup and functionality between the recording sessions are listed in the following. In the remainder of this section, the interface of the tool and the applied restaurant application are described. Technical details on the implementation and customisation of the tool are listed in Appendix A. 9

http://www.uni-ulm.de/in/pit

3.2 Wizard-of-Oz Data Collection

61

• Session I. The initial system deployed for the wizard interaction is simple and not very elaborated. The wizard is provided with a VoiceXML frontend to type in the keywords. The backend, i.e. the system itself, is a preliminary version of the final system, implemented in Java. Various wizards were involved in the recordings. The system is represented by synthesised speech output. Restaurants’ menus can be displayed on the screen. Various wizards are involved in the recordings. The main drawback of this architecture is the slow response time of the system presumably caused in part by unclear or indirect phrasing of the queries, where it was ambiguous for the wizard how to react. More practise and training for the wizard might have helped against this to some extent. Further, there is a high chance to misspell words in this frontend. Additionally, the participants would start to speak again while waiting for the answer which, of course, the wizard (resp. system) did not interrupt. They would pick up a random topic or alter the request which induced the wizard to change the query once more before responding. On the other hand, if participants ask direct questions and wait for the answer, the response time is considerably shorter. • Session II. The following improvements are realised: – Speeding up the response time: The slow response time of the system was not acceptable, therefore, the system architecture evolved to a client-server architecture entirely implemented in Java. The new frontend permits only specified keywords to be entered deploying an auto-completion functionality. Thus, faster and more correct interaction is enabled for the wizard which improves the response time considerably. – Giving the system a face: An avatar in form of a head is integrated as a form of personification of the system in a visual modality. It moves its mouth according to the synthesised speech output and occasionally blinks with one eye. – Additional functionality: The system’s range of functionality is extended to include the feature to show street maps pointing to the selected restaurant. • Session III. The system is altered in the following two points: – Single wizard: One single wizard conducts all of the recordings to ensure a small variation as possible regarding system behaviour. – Additional functionality: The functionality is expanded in terms of the ability to display bus schedules. The website of the local public transportation service is consulted for the next bus connections from the current location (i.e. university) to the bus station closest to the selected restaurant. – Avatar or no avatar: Recordings can be performed with or without the avatar displayed by simply (un)selecting a check box.

62

3 Corpus Development

– Emotion eliciting interaction: An emotion inducing policy for the wizard behaviour is introduced to obtain emotionally rich dialogue data. Human Computer Interface Figure 3.3 shows screenshots of the computer screens of the wizard (a) and the users (b) when interacting with the system10 . The different interfaces are described in the following.

Fig. 3.3. (a) Screenshot of the Wizard client, (b) Screenshot of the system from the users’ perspective.

Wizard Interface. Figure 3.3(a) shows the system as seen from the wizard’s perspective. Besides the WIT’s graphical user interface (GUI) on the left side, two further objects are displayed on the wizard’s computer screen. On the upper right side the image of the users’ screen is shown. It is forwarded through the Ethernet using a webcam in order to check if the graphical output of the server is correct. An audio recording tool, displayed on the lower right side, records the users’ conversation which is transmitted by the wireless microphones. The design of the tool is very functional focusing on the basic features. It has only a few buttons and text fields that can be controlled by either mouse or keyboard. The wizard enters commands (either preferences picked out in the dialogue or commands to control the system’s functionality) into the field in the top row with auto-completion functionality, i.e. only permitted words can be entered and are automatically completed when typing the beginning of a word. This way, no invalid or non-existent commands can be typed by the wizard reducing errors enormously and enabling very fast interaction. The line below the command box depicts the automatically generated 10

Obviously, the screenshots were taken at different points in the interaction as (a) has no object displayed on the system’s screen while (b) does.

3.2 Wizard-of-Oz Data Collection

63

system utterances that are displayed before prompting in order for the wizard to control and edit the content of the utterances (if necessary). This gives the wizard further the opportunity to determine the prompting time more precisely. Although for our recordings generally not used, the functionality of entering custom utterances is also provided. The current constraints (i.e. up to a point where the wizard starts a new query) as well as the history of the prompts are shown above the result list. From the prompt history entries can be selected to reload the prompt into the prompt field. The result list is displayed in the lower part of the tool and holds the result set of the latest database query. The restaurants are displayed in a table to give the wizard an overview of the matching restaurants and their properties. Entries can be selected, again using the mouse or keyboard, to load a restaurant as the currently selected restaurant which enables the wizard to access the information about this restaurant. User Interface. The users’ perspective of the system is shown in Figure 3.3(b). The interface is operated by the server and consists of synthesised speech as well as graphical output in form of an avatar (seen on the left side of the screenshot) and objects displayed on the screen (e.g. the map on the right side of the screenshot). • Speech Output. The system utterances to be prompted are transmitted by the client to a module which converts text to phonemes. Hadifix11 is used to generate phoneme files interpreted by MBROLA12 to produce WAV-files that are played on the server. • Avatar. The avatar figure consists of various transparent images of the head, eyes, and different mouth positions as seen in Figure 3.4. The phoneme files are further used to produce harmonious avatar output as the avatar moves its mouth according to the phonemes. It further blinks randomly with one eye to give it some kind of a lively touch. • Displayed Objects. The objects displayed on the system’s screen are the restaurant’s menus, street map and bus schedule. The objects are controlled by scripts which are triggered by the wizard to run on the server. A script for instance starts a web browser showing a map that points to the address of the currently selected restaurant. The maps are dynamically created using Google Maps. Directions to or from addresses cannot be displayed. The restaurants’ menus are stored as HTML objects and are displayed in a browser window. Bus schedules are created dynamically consulting a website for a connection from the university’s bus stop to the bus stop closest to the selected restaurant. At the moment, no specific departing time can be enquired; the current date and time are used.

11 12

http://www.ikp.uni-bonn.de/dt/forsch/phonetik/hadifix/HADIFIXforMBROLA http://tcts.fpms.ac.be/synthesis/mbrola

64

3 Corpus Development

Fig. 3.4. Three different examples of phoneme based mouth positions (from left to right: F, U/O, A).

Restaurant Application The example application implemented for the recordings consists of a simple database which the system accesses to perform queries based on the users’ preferences and in order to provide information about certain restaurants. The restaurant database contains information about approximately 100 restaurants in and close to Ulm, Germany. It is implemented in XML. Figure 3.5 shows an example database entry (in the upper part). Each restaurant is repDatabase restaurant entry: 1 2 Asia Wan am Muensterplatz 3 Muensterplatz 14 4 0731 1537371 5 ... 6 exclusive 7 lunch 8 ... 9 Grammar entry of price category: 1 2

3 val=’exclusive’expensive 4 val=’exclusive’exclusive 5 ... 6 val=’moderate’standard 7 val=’moderate’moderate 8 ... 9 val=’inexpensive’cheap 10 val=’inexpensive’inexpensive 11 ... 12

13 Fig. 3.5. Extracts of an example restaurant and grammar entry (both translated from the original version in German into English).

3.3 The PIT Corpus

65

resented by various fields which serve as search criteria and to provide information, such as name, restaurant category, address, phone number, opening hours, public transport access, cuisine, setting (such as terrace, beer garden, children’s corner, non-smoking area), take-away and delivery, specials (such as happy hour or lunch special), web address. The database is complemented by a menu in HTML format for most of the restaurants. Street maps and bus schedules are not part of the database but generated at runtime using the database information such as the address and closest bus station. All permitted values and synonyms for the database entries are defined by a grammar. These are the values the system understands and which can be entered by the wizard. The grammar is also implemented in XML, an extract of the example grammar entry ’price category’ is shown in Figure 3.5 in the lower part.

3.3 The PIT Corpus The corpus consists of 76 dialogues recorded within the setup presented above. Three recording sessions were performed: Session I was recorded in July and August of 2006, Session II in May and June of 2007, and Session III in December 2007. Table 3.2 shows descriptive information of the corpus. Between the recording sessions the system underwent changes which are described in Section 3.2.4. About half of the dialogues were recorded with avatar on the screen, the other half without avatar. One setup was used for all recordings of Session I and Session II, whereas in Session III four different setups were deployed (IIIa-d).

Session

Number of dialogues Duration of session

I

II

19

20

3:47 h

4:18 h

III IIIa

IIIb

14

11

IIIc

IIId

5

7

37 5:40 h

Total

76 13:45 h

Min dialogue duration

3:15 m

4:18 m

2:43 m

2:43 m

Max dialogue duration

26:11 m

33:39 m

18:24 m

33:39 m

Mean dialogue duration

12 m

13 m

9:44 m

11 m

Avatar

-

+

+

-

+

-

51.3%

Emotion-eliciting strategy

-

-

-

-

+

+

15.8%

Table 3.2. Statistical information of the PIT corpus.

66

3 Corpus Development

3.3.1 Data Structure All dialogues follow a certain pattern which arises from the fact that the structure of the dialogue is defined by various crucial points, as depicted in Figure 3.6. The point in time when the conversation between the users enters the specified domain is the first point which induces a change in the system’s state. The second decisive moment is the first interaction of the dialogue system. From now on, three dialogue partners are involved in the conversation. The behaviour of the dialogue partners changes in terms of addressing and gazing. This point thus induces a phase change for the audio and video data, as well as for the system. A further crucial point occurs with the display of an object (other than the avatar) on the screen. Whereas this does not invoke a change in the interaction itself, the object attracts the main dialogue partner’s attention and therefore influences the gazing behaviour. Thus, a phase change in the video data occurs. At the point when the object disappears the former phase is re-entered. When the task is solved and the system retracts from the dialogue, the last decisive moment is reached. This point denotes the end of the dialogue. Audio

phase 1

domain starts

System

Video

phase 2

phase 1

phase 3

first interaction

phase 2

show object

phase 3

hide object

problem solved

phase 2

Fig. 3.6. Dialogue with crucial points and phases.

Audio Data. The audio data can be divided into three phases denoting different characteristics. Each dialogue starts with a domain independent chat between the participants. The second phase is introduced at the point when the conversation switches over to the specified domain and the users start discussing their preferences and aversions in different aspects of the restaurant domain. The third part is characterised by the involvement of the dialogue system in the conversation to achieve the concerted task. The dialogues typically end when the users find a suitable restaurant and thank the system. Some recordings contain various iterations of the restaurant search, i.e. after finding one, instead of ending the dialogue, the users started to look for another restaurant (remaining in the third phase).

3.3 The PIT Corpus

67

Video Data. The video data can also be structured into three interaction phases considering the gaze direction of the main user U1. These phases differ from the dialogue phases described above. The first interaction phase is characterised by the conversation between the human dialogue partners before the first system interaction. During this time, there is almost no gaze directed towards the computer screen. The first interaction of the system initiates phase two and displays the avatar on the screen (where applicable). During this phase, U1’s gaze switches between the computer and user U2, depending on speaker and addressee. The third phase is characterised by an object (other than the avatar) displayed on the screen: Generally, while a restaurant’s menu, a street map, or bus schedule is shown on the screen, most of U1’s gaze points towards the system. When the object disappears, the dialogue returns to the second phase. When the task is solved and the dialogue system retracts from the conversation the avatar is removed from the screen. 3.3.2 Annotation The data is transcribed at the utterance level and annotated with speaker, addressee and dialogue acts. Table 3.3 presents the basic tagset of dialogue acts we used for annotation of the dialogues. It consists of nine one-dimensional dialogue acts. The tagset was generated using a bottom-up approach by empirically analysing the dialogues in order to identify the necessary dialogue acts. The small number of dialogue acts is in our case sufficient to cover all the actually occurring phenomena we aim at identifying in the dialogue in order to solve the task. Figure 1.1 shows the example dialogue from Section 1.3 annotated with the according dialogue acts. Natural dialogue is generally too complex for being coded with a onedimensional tagset [Petukhova and Bunt, 2007]. For computational dialogue modelling, however, the usage and purpose of the dialogue is limited. Thus, sacrifices have to be made and generally simplicity is preferred over the completeness in annotation (e.g. [Traum, 2000]). Complex modelling might contain a lot of information that is redundant for the system as it cannot be used for the limited task. The concepts of our tagset are partly borrowed from Bunt’s DIT++ tagset (e.g. [Petukhova and Bunt, 2007]), a fine grained tagset that considers 11 dimensions for annotation, and partly from the VERBMOBIL tagset [Alexandersson et al., 1998], a one-dimensional tagset which consists of 33 dialogue acts. Although especially the Verbmobil tagset is less complex than the DIT++ and the prominent four-dimensional DAMSL [Core and Allen, 1997] tagset, they are all still too complex to be deployed for our system. For instance, there is no need to differentiate between different forms of suggest; the system does not treat exclamations different from offers or statements, just to name a few. If the system was to be altered in the future to be able to react differently upon these different fine-grained dialogue acts, a different tagset would have to be used.

68

3 Corpus Development

Dialogue Act

Abbr.

Meaning and significance in the domain

suggest

sug

Task-related propsal

request

req

Request for information or database query towards the system; towards the other dialogue partner mainly in order to find out the preferences of the partner

inform

inf

Supplying information (e.g. as done by the system after a query)

accept

acc

Mostly as a positive reaction to suggest

reject

rej

Mostly as a negative reaction to suggest

acknowledge

ack

Signalling of understanding of the previous contribution

check

chk

Repetition of mentioned utterance snippets to elicit approval of what has been understandood. Check acts are occasionally also used as a sort of stall acts in order to win more time to think.

stall

sta

Any fillers uttered in order to defer the dialogue, to keep the floor or win more time to think. Stall acts could also denote uncertainness.

greet

gre

Social act in order to comply with cultural conventions of greeting and introducing oneself, as well as saying goodbye

other

oth

Anything irrelevant for the goal of the dialogue (e.g. discussion of the displayed menu)

Table 3.3. PIT Corpus dialogue act tagset.

Video Data. Anvil [Kipp, 2001] is used for the annotation of the video data. A set of Session I and II dialogues (refer to Section 5.3) was handannotated by one expert with speaker, addressee, gaze direction of user U1 and screen display. Refer to Section 5.3 for the result of the gaze direction analysis of the annotated dialogues. Automatic labelling of the gaze direction of the Session III dialogues is part of future work (refer to Section 6.2). 3.3.3 Dialogue Analysis Table 3.4 depicts one of the dialogues of the corpus13 as an example to show the analysis performed on the data. The dialogue is annotated with dialogue acts, as described above, as well as speaker and addressee information. The dialogue presented here is translated from its original version of German which is listed in Appendix B. 13

The dialogue originates from Session III and has the ID III 09 061207.

3.3 The PIT Corpus

69

ID

Spkr Addr Utterance

DA

1 2 3 4

U1 U2 U1 U2

U2 U1 U2 U1

inf ack sug rej

5

U1

U2

6 7 8 9

U2 U1 U2 U1

U1 U2 U1 U2

10

U2

U1

11 12 13 14 15

U1 U2 U1 U2 S

U2 U1 U2 U1 U1

16

U1

U2

17 18

U2 S

U1 U1

19 20

U1 S

S U1

21 22

U1 U1

S U2

23 U2 24a U1 24b U1

U1 U2 S

Oh, it’s late already. Yeah. So, should we eat something? I just had something a little while ago. I’m not hungry. You are not hungry? Well, what would you.. Where would you go? I’d be up for a cocktail. Yeah, that’s a good idea, but eat a bite at first? Yes, if there’s good cocktails... Yeah, that’s a good idea. But what else could we eat? Well.. I feel like Mexican. What do you think? Hmm, although, then maybe I’ll eat a bit as well. But I’d rather have Italian. Yes, that’s also a good idea. But it shouldn’t be so expensive. Yeah, true. I’m running low on money. Hello, my name is Helmut. I think I could help you. Your query for a locality with Italian cuisine and inexpensive price category yields 6 results. Please confine your request further. So, let’s go to the centre cause we’re gonna meet the others later on anyway.. Right. Your query for a locality with Italian cuisine in the centre with inexpensive price category yields 5 results. Please confine your request further. Well, Italian.. And then cocktails with that? Your query for a locality with Italian cuisine in the centre with inexpensive price category yields Caf´ e Statth¨ ausle. Would you like to see the menu? Oh yes, please %laughter% OK. So, there is Pizza, and Snacks, Soups, Baguettes.. Didn’t you want Mexican? Yes, %laughter% that would be an idea %laughter% and Mexican?

che, req sug acc, sug acc acc, req, sta, sug inf, sug acc sug acc inf gre, inf

sug acc inf

ack, req sug, sug

acc inf sug acc req

70

3 Corpus Development

25

S

U1

26 27 28

U2 U1 S

U1 S U1

29 30 31 32

U1 U2 U1 S

S U1 S U1

33 34 35 36 37 38 39 40

U1 U1 U1 U1 U1 U1 U2 S

S S U2 U2 U2 U2 U1 U1

41 42

U1 U2

S S

Your query for a locality with Mexican cuisine yields 2 results. My suggestions are Restaurant Flamenco and Restaurant Pepper’s. Yeah, but do they serve cocktails? Cocktails? %laughter% Your query for a locality with Mexican cuisine and cocktails yields 2 results. My suggestions are Restaurant Flamenco and Restaurant Pepper’s. Well, that sounds good now. OK. Then it was the same. So, are they in the centre? %laughter% Flamenco is situated in Herdbruckerstrasse 18. Would you like to see a map? Yeah, %laughter% yeah. And what’s the menu like then? %unintelligible% You can’t see it, right? Then let’s go there, %laughter% OK? Yes? Yes. Goodbye. I hope I could help you. Enjoy your meal! %laughter% Thanks.

inf, sug

ack, req req inf, sug

acc ack req inf, sug acc req oth sug sug che acc gre oth oth

Table 3.4: Annotated example dialogue from the PIT corpus.

In the presented dialogue transcript different characteristics can be identified that have an influence on the design of the dialogue management, as follows. Data structure. To demonstrate the data structure described in Section 3.3.1, the transition points that induce phase changes are identified in this dialogue. The domain starts with utterance 3. The system’s first interaction occurs at utterance 15. The system retracts after its closing utterance (40). These three points denote the phases according the dialogue transcripts and audio files. The additional points relevant for the video data are the showing and hiding of the displayed objects other than the avatar which appears at utterance 15. These points can only be determined analysing the video data. By consulting the transcript only, the approximate points in the dialogue of showing the objects can be identified as utterances 22 and 33. The point of

3.3 The PIT Corpus

71

hiding the objects cannot be identified by merely looking at the transcript. In utterance 34, the user asks to see another object. From the transcripts only, it is not clear if the menu is shown or if the conversation goes on about the displayed map, however, the analysis of the video data reveals that the restaurant’s map is displayed upon utterance 34. Proactive system interaction. The system gets proactively involved in the dialogue in three cases. The utterances 15, 18 and 40 are perceived to be proactive interactions. All other system interactions are reactive as they follow a direct interaction request by user U1. The addressee of the utterances can be recognised clearly in the dialogue transcript. Observing the sequences of dialogue acts that precede the proactive interactions a certain pattern becomes apparent: A suggestion is in each case followed by an acceptance move. Investigating the dialogues of the corpus, the sequence of dialogue acts that provides an ideal point in the dialogue for proactive interaction is defined more precisely as presented in the scope of the dialogue management in Section 4.3.2. Another proactive behaviour of the system is the offering of additional helpful resources such as a restaurant’s menu, street map or bus schedule. This can occur in proactive and reactive interactions alike. In the example dialogue, a menu is offered to be shown in the (reactive) utterance 20, a street map in utterances 32. The menu shown upon utterance 34 is displayed on request of the user. Negative feedback. A further interesting point in the dialogue occurs in utterances 9 and 10 where user U2 rejects a suggestion by U1. User U1 suggests Mexican food in utterance 9. In the subsequent utterance 10, U2 implicitly rejects Mexican food stating that she prefers Italian. It is a general question how negative feedback should be treated in the further proceeding of a dialogue. The value could either be taken out of the current valid set of preferences or adopted as a negative constraint. In the present case the user’s intention does not seem to be of the latter case. The reject is of a weak kind, it is assumed the user does not intend to introduce Mexican as a negative constraint. A further possibility would be to add Italian to the current set of valid constraints (i.e. Mexican). The way the system is designed to proceed in such cases is stated in Section 4.3.2. Dialogue acts. The utterances of the dialogues are analysed in order to pick out a set of suitable dialogue acts that cover all possible user (and system) contributions relevant for the task-solving process. The present example contains suggestions proposing new constraints (utt. 6, 9, 10, 12, etc.) and requests, with propositional content (e.g. utt. 24b, 27) or without (e.g. utt. 5). The responses are of an accepting (e.g. utt. 11) or rejecting form (utt. 4), acknowledgements (utt. 2, 30, etc.) or check acts (utt. 5, 38). Further less frequently occurring acts are stall (utt. 9), inform (e.g. utt. 1, 15) and greet

72

3 Corpus Development

(utt. 15 and 40). All utterances that can not be categorised with the defined dialogue acts are marked as other. The tagset is kept as compact as possible as described in Section 3.3.2. Dialogue moves. A further categorisation is performed in terms of dialogue moves which define the function an utterance adopts in the dialogue in order for the system to know how to integrate its content. The finer grained dialogue acts are thus abstracted accordingly. The main moves are also at this level suggest and request. The responses are subsumed as a reply move. Refer to Section 4.2.4 for a description of the dialogue moves. Multiple moves per utterance. Multiple moves can occur in the scope of one utterance. For instance, utterance 26 contains the two dialogue acts reply and request. Both of these acts need to be integrated within the same utterance. This is especially relevant if an utterance contains several propositions which need to be integrated as part of the same utterance for constraint prioritisation reasons as described in Section 4.5. Thus, dialogue modelling should be laid out to handle various moves per utterance.

3.4 Summary This chapter focused on the data collection and development of the PIT corpus of German multi-party dialogues. For the development of a dialogue system suitable dialogue data is needed to obtain interaction models. In Section 3.1 we listed state of the art multi-party multi-modal dialogue corpora. However, none of them possesses the dialogue properties that we intend to investigate. Thus, we recorded our own corpus within an extensive Wizard-of-Oz setup. The resulting PIT corpus consists of audio and video data of 76 dialogues. Post processing includes transcription and annotation of the data as described in Section 3.3. The transcribed dialogues were used for the development of suitable dialogue management (refer to Chapter 4), e.g. to decide about an appropriate range of domain-relevant components, to perform interaction modelling (e.g. to determine appropriate system responses) and the selection of dialogue moves (refer to Section 3.3.3). The questionnaires obtained from the recordings form the basis for the evaluation presented in Chapter 5. The annotated video data was deployed for the evaluation presented in Section 5.3.

4 Dialogue Management for a Multi-Party Spoken Dialogue System

This chapter is dedicated to the multi-party dialogue management functionality of our dialogue system which denotes the most important part of this book. The dialogue management deployed is built on the basis of the Information State Update approach (e.g. [Cooper and Larsson, 1998, Traum et al., 1999, Matheson et al., 2000, Larsson and Traum, 2000, Larsson, 2002]), as introduced in Section 2.4. The ISU approach models dialogue as a state of information that is updated according to the content of the latest incoming utterance. It is very well suited to model an agent-like system as ours which is to stand independently as a conversation partner. The system’s (and also the other users’) state of ’mind’ including believes about itself and the others as well as the goals it aims to achieve can be modelled. The approach provides flexibility for the developer to decide what information should be specified and in what way. It allows modifications and adjustments and thus can be adopted to fit the requirements of our proactive multi-party dialogue system. Modifications of the ISU approach become necessary due to the fact that the existing multi-party extensions to the approach do not fully meet the requirements of our system, as described in Section 2.4.4. Figure 4.1 shows a schematic overview of the dialogue management of the system. Our modified information state builds the central part of the dialogue manager. It is described in Section 4.1 together with a new interaction principle that becomes necessary to extend the functionality of the interaction protocols to enable proactive interaction (refer to the discussion in Section 2.4.4). The task model as part of the information state is described in Section 4.2 in which our example domain is applied and thus, all task-relevant components of the dialogue manager, such as context, domain model and update mechanisms are presented. Section 4.3 describes the strategies the system deploys in dialogue management in order to render the system capable of proactive interaction. For instance, the dialogue history denotes an important part of our setup as it stores the content of the information state at each point of the dialogue, already before the system’s first interaction, and thus enables proactive system interaction (refer to Section 4.3.3). The dialogue manager P.-M. Strauß and W. Minker, Proactive Spoken Dialogue Interaction in Multi-Party Environments, DOI 10.1007/978-1-4419-5992-8_4, © Springer Science + Business Media, LLC 2010

73

74

4 Dialogue Management for a Multi-Party Spoken Dialogue System

Dialogue Management Dialogue history

Information State Input speaker addressee sem.content DA

PRIVATE

...

Problem solving

...

SHARED

Output result DA addressee

Task ...

Update rules & Dialogue plans

Context & Domain

Fig. 4.1. Dialogue management component of the system.

further contains a problem solving functionality that performs the database queries and evaluates the returned results as presented in Section 4.5. In that context, we introduce a new approach to user constraint prioritisation that enables efficient handling of over-constrained situations in the problem solving process. The input to the dialogue manager consists of the latest utterance in an already processed state. The semantic content has been extracted and is entered in the form of semantic representation, attribute value pairs in our case (refer to Section 4.2.3). For a live functioning of the system a natural language input functionality would have to be implemented which has so far only been developed in the form of a proposal of a semantic parser [Strauß and Jahn, 2007] (refer to Section 6.2). Information about the current speaker, addressee and the corresponding dialogue act(s) are further input with the semantic information. The output of the dialogue manager, e.g. if the system is to interact, such as to reply to the previous utterance, consists of the database result (where applicable) and the dialogue act that states in which form the reply should be presented. Further, the addressee of the utterance is listed. The output functionality that has been implemented in the scope of the WIT system (Section 3.2.4) is deployed to generate system prompts. Besides the components of the dialogue management listed explicitly in the diagram, the system’s optimistic grounding and integration strategy (Section 4.3.1) as well as the interaction strategy that (Section 4.3.2) are further presented in this chapter. At last, Section 4.4 presents an example sequence of information states to illustrate how the dialogue management works in practise.

4.1 Multi-Party Dialogue Modelling

75

4.1 Multi-Party Dialogue Modelling We modified the ISU approach in order to suit the requirements of the multiparty setup as well as the task-oriented and proactive nature of our system. In the following, we present the modified dialogue model and interaction protocols. The task model and other contextual parts of the system (e.g. the domain model) as well as the update mechanisms are presented in the scope of our example domain in the subsequent section. 4.1.1 Dialogue Model The information state (IS) we deploy is based on the information state as introduced by Larsson (2002) for his IBiS1 system (refer to Section 2.4). The private part of the IS is basically used in its original form, the structure of the shared part is modified. Some of the alterations proposed by Kronlid (2008) (refer to Section 2.4.4) are adopted for our approach, however, over all we keep it more simple and task-oriented as described below. Our information state is depicted in Figure 4.2, its particular elements are listed in the following: ⎡

⎡

⎤

⎤

AGEN DA : stack(Action) ⎦ ⎥ ⎢ P RIV AT E : ⎣ P LAN : stack(Action) ⎥ ⎢ ⎥ ⎢ BEL : list(P roposition) ⎢ ⎡ ⎤⎥ ⎥ ⎢ T ASK : set(P roposition) ⎥ ⎢ ⎢ ⎢ SO ⎥⎥ : Object ⎢ ⎢ ⎥⎥ ⎢ ⎢ RES : list(Object) ⎥⎥ ⎢ ⎧ ⎢ ⎛⎡ ⎤⎞⎫ ⎥ ⎥ ⎢ ⎢ ⎥⎥ Q : Question ⎨ ⎬ ⎢ ⎢ ⎥⎥ ⎢ ⎢ QU D : list ⎝⎣ SP KR : P articipant ⎦⎠ ⎥ ⎥ ⎥ ⎢ SHARED : ⎢ ⎩ ⎭⎥ ⎢ ⎢ ⎥⎥ ADDS : set(P articipant) ⎢ ⎢ ⎥⎥ ⎧ ⎡ ⎤⎫ ⎢ ⎢ ⎥⎥ SP KR : P articipant ⎨ ⎬ ⎢ ⎢ ⎥⎥ ⎢ ⎢ LU ⎥⎥ : ti ⎣ ADDS : set(P articipant) ⎦ ⎢ ⎢ ⎥⎥ ⎭ ⎩ ⎣ ⎣ ⎦⎦ M V S : list(M ove) DH : {ti−1 [DH Obj] , ti−2 [DH Obj] , . . .} Fig. 4.2. Information state structure at point in time ti of the dialogue.

• AGENDA holds the short-term plan, i.e. the actions the system is to perform next. • PLAN holds the long-term plan the system pursues in order to achieve its goals. The plan changes according to the activity state the system adopts. • BEL (Believes) of the system which are not (yet) public, e.g. the result of the database query before it is presented to the users. We use a list as the data structure in order to enable the user to refer to restaurants in the presented order (e.g. ’the second’).

76

4 Dialogue Management for a Multi-Party Spoken Dialogue System

• TASK constitutes the task model, i.e. contains all currently valid user constraints and is thus used as the basis for the database queries. A more detailed description is given below. • QUD (Question Under Discussion) contains all unresolved questions (Q) that are currently open for discussion. It is used for anaphora resolution and ellipsis interpretation and contains SPKR (Speaker) and ADDS (Addressees) of each question to know who can address whom about this issue. • SO (Selected Object) contains the object which is currently being discussed (usually one of RES). • RES (Result Set) of objects returned by the database query after it has been presented to the users, i.e. it contains the data formerly stored in BEL. Thus, the data structure is the same as for BEL. • LU (Latest Utterance) comprises all relevant information of the most recent utterance: SPKR (Speaker), ADDS (Addressees), and MVS (Moves). The content of LU becomes part of DH after integration. • DH (Dialogue History) contains all states of the IS throughout the dialogue in chronological order. Each element contains all relevant information at that specific point in the dialogue. Refer to Section 4.3.3 for a detailed description of the dialogue history. Our information state differs from Larsson’s IS [Larsson, 2002] (refer to Figure 2.2) as described in the following. Figure 4.3 shows an example information state that depicts the processing of utterance 16 from the dialogue listed in Table 3.4 (Section 3.3.3). A detailed description of the individual elements contained by the IS follows in the remainder of this chapter. The ⎡

⎡

⎤

⎤

AGEN DA : inf orm(res2 ) ⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎦ ⎥ ⎢ ⎥ ⎢ (R , . . . , R ) BEL : res 2 1 5 ⎢ ⎤⎥ ⎡ ⎥ ⎢ T ASK : [f ] f p l 1 2 1 1 ⎥ ⎢ ⎢ ⎥⎥ ⎢ SO : ⎢ ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢ RES : ⎢ ⎧⎛⎡ ⎥⎥ ⎢ ⎤⎞⎫ ⎢ ⎥⎥ ⎢ Q : l ? ⎨ ⎬ 1 ⎢ ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢ ⎝⎣ SP KR : U 1 ⎦⎠ ⎢ SHARED : ⎢ QU D : ⎩ ⎥⎥ ⎭ ⎢ ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢ ⎧ ⎡ADDS : U 2 ⎤⎫ ⎢ ⎥⎥ ⎢ SP KR : U 1 ⎨ ⎬ ⎢ ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢ LU ⎦ : t16 ⎣ ADDS : U 2 ⎢ ⎥⎥ ⎢ ⎩ ⎭ ⎣ ⎦⎦ ⎣ M V S : sug(l1 ) DH : {t15 = [..] , t14 = [..] , . . . t1 = [..]} Fig. 4.3. Example information state.

4.1 Multi-Party Dialogue Modelling

77

private part (AGENDA, PLAN and BEL) of the information state can be deployed in its original version as it serves our needs well as is. It provides the plan and agenda, the system follows, and supplies a field for the temporary storage of the query result (as seen in Table 3.4). The data structure of these fields does not have to be modified from Larsson’s version. The extensions proposed by Kronlid in terms of modifying the structure of the PLAN field are not necessary as we do not aim at deploying complex plans at this moment (refer to Section 4.2). Instead of the commitments field (COM) used in the original IS, we deploy various task-related fields: TASK, RES and SO, all of which belong to the shared part of the IS. TASK can be said to be the pendant of COM as it holds the task model, i.e. all constraints that have been collected from the interaction of all participants (refer to the description of the task model in the following section). The RES field holds the result set obtained from the database query after it has been presented to the participants (it is stored in the BEL field before it is presented). The SO field holds the currently selected object, usually one of the presented objects from RES which the participants selected for discussion. The RES field holds the complete result set during the time that one of the objects is under discussion as the participants could possibly switch to another one of the presented objects after. The case that multiple restaurants are selected at one point is handled as selecting one after the other. The structure of our QUD field is extended following the complex QUD object proposed by Kronlid which enables addressing of QUD elements that are not maximal (by storing the QUD elements in a list as opposed to Larsson’s stack). We further store the speaker and addressee with each issue in order to know who is to address for which question, however, not in Kronlid’s way as there is no need in our case to add a special field for obligated addressees, nor for the status of the element. Note that we add an ADDS field although by definition utterances should be directed at only one addressee per utterance. However, in order to render the system flexible in terms of modelling actually occurring situations, the addressee field can contain more than one participant. Multiple addressees are understood in the way of more than one addressee for one utterance as opposed to different (sets of) addressees for different parts of the utterance, i.e. different moves. In the latter case, a new utterance would commence for dialogue modelling. In the latter case the utterance would be split and a new utterance at the change of the addressee. To adopt the fact that multiple moves can be contained in one utterance we allow a set of moves in the LU field, as also done by Kronlid. A restriction prevails in terms of that the addressees are the same for all moves as described above. For instance, utterance 19 of the example dialogue presented in Table 3.4 in Section 3.3.3 contains two moves: ack and req. These moves are integrated successively within the same integration cycle. In the case that several propositions are uttered within the same utterance (e.g. several suggestions) all propositions are integrated into TASK as part of one utterance

78

4 Dialogue Management for a Multi-Party Spoken Dialogue System

which becomes relevant in terms of the priority value calculation introduced in Section 4.5. Utterance 24 of the indicated example dialogue shows an utterance that has two different addressees for the two dialgoue moves: U2 for acc (utterance 24a) and S for req (24b). The utterance is thus split in two separate utterances for integration. Larsson and Kronlid both deploy a field (PM in IBiS2 and PU in MultiIBiS) to store the utterance mentioned before the latest one (which is stored in LU). While it may suffice a single-user system, we claim that for a multiparty system it is not sufficient to store only one previous utterance to perform backtracking due to the nature of the multi-party dialogue where responses are often not adjacent. Our system as an intelligent dialogue partner further needs a more complex memory of the conversation which leads us to deploy an extensive dialogue history as presented in detail in Section 4.3.3. 4.1.2 Interaction Protocols The discussion in Section 2.4.4 revealed that the existent extensions to the ISU approach do not provide the functionality that is required by the multi-party setup of our system. Kronlid’s Add Multiple Addressees (AMA) principle [Kronlid, 2008] regards multiple addressees with an option to respond. Our setup, however, does not consist of multiple addressees. Every utterance is by definition directed at one addressee; the third DP is a side-participant. Thus, AMA is not the correct principle to use for our case. Duplicate Responders (DR) is (out of the same reason) not suitable. An extension considering one direct addressee and an unspecified number of side-participants is provided with the Add Side-Participants (ASP) principle introduced by Ginzburg and Fern´ andez (2005). ASP includes contextual updates for the side-participants in the same way they are performed by the addressee. This way, the sideparticipants obey the principle of responsibility which requires all participants of a conversation to keep track of the conversation at all times (refer to Section 2.3). They are thus further enabled to make QUD-specific utterances once they are addressed. In perfectly conducted conversation in terms of turn-taking the current speaker selects the next speaker by addressing (either by words or gaze, implicitly or explicitly). The current speaker could at the same time address various participants of which one or more will be the next speaker(s) (see the discussion on distributive and collective questions above). The only case that considers self-selection of the next speaker is the case when no participant was addressed in particular. Kronlid (2008) calls the situation of what happens (in terms of self-selection) when a group of participants is addressed with a question that only some of them can answer addressing-by-attribution1 . Our setup considers natural dialogue in which the 1

The term is actually borrowed from Clark and Carlson (1982) where it has a slightly different meaning in terms of addressing someone about an attribute without knowing which particular hearer relates to this attribute, such as for instance possesses something the speaker wants.

4.1 Multi-Party Dialogue Modelling

79

situation is normally not as ordered. We encounter side-participants who actually make QUD-specific utterances, i.e. they interact without being asked to (as in addressing) - or proactively - to something they are not the addressee and which thus was not primarily designed for them. We thus define proactive side-participants as follows: Definition: Proactive Side-Participants (PSP): Proactive side-participants are side-participants of a conversation who interact, i.e. they are participants who interact without being currently addressed. We introduce a new interaction principle, displayed in Table 4.1, to integrate proactive side-participants. ASP is for this modified in the way that optional answering by the side-participants is allowed, as a step towards AMA. Add Proactive Side-Participants (APSP): The audience consists of a set of dialogue participants (of which at least one is an addressee) who update their information state in the way addressees do. The obligation of the addressees’ QUD-specific response persists until a side-participant makes a relevant (i.e. QUD-specific) contribution. After that, contributions by addressees are optional. Contributions by side-participants are always optional. Table 4.1. New interaction principle.

Due to the modified information state (especially the structure of the QUD field) APSP allows for the addressing of the original question as well as comments of fellow addressees or side-participants. The situation that no specific addressee is assigned is not considered2 . The principles apply for our system in the following way: While the system is still inactive, i.e. while it is domain-spotting, the AO principle is deployed which allows an investigation of the propositional content of the user utterances. As soon as the dialogue enters the specified domain (i.e. active state), the system is modelled as a side-participant in order to allow for proactive interaction and thus, APSP is deployed. Table 4.2 shows the interaction protocols of simple suggest and request moves. The latter move is displayed with and without propositional content. Depending whether an object is currently being discussed (i.e. SO is not empty), the effect of requests with propositional content would differ. If no object is currently selected (as would be the case in our protocol) the effect is the same as a suggest followed by a request without content, otherwise, 2

By definition, this case does not occur in the setup of our system.

80

4 Dialogue Management for a Multi-Party Spoken Dialogue System

Suggest + APSP

Request + APSP

Request + APSP

LatestMove== suggest(U1:U2,p)

LatestMove== request(U1:S,q)

LatestMove== request(U1:S,q(p))

1. U1: release turn; 2. ALL: push(QUD,p?); 3. S: update(TASK,p); queryDB(TASK); 3. U2: take turn; address p

1. U1: release turn; 2. ALL: push(QUD,q); 3. S: take turn; answer

1. U1: release turn; 2. ALL: push(QUD,p?); push(QUD,q); 3. S: update(TASK,p); queryDB(TASK); 3. U2: take turn; address p

LatestMove== accept(U2:U1,p)

LatestMove== inform(S:U1,result)

LatestMove== reject(U2:U1,p)

...

...

...

Table 4.2. Interaction protocols suggest, request and request using APSP.

the propositional content is matched with the selected object. For a more detailed discussion on different possible kinds of requests that can occur refer to Section 4.2.4. The protocols show a normal flow of the conversation. The first two proceed normally; the addressed DP responds to the speaker’s contribution. The third protocol makes use of the by APSP enabled proactive interaction by a side-participant who was not addressed. The system is addressed by U1 with a request containing proposition p. U2 does not agree with p, grabs the turn before S can take it to respond in order to state her disapproval. U2 was only side-participant in this situation, however, due to the urgency of her contribution, U2 interrupts the dialogue before the proper addressee (S) proceeds. After U2’s utterance, the situation is different and S cannot just continue with where left off but has to consider the changed situation. The obligation to answer might in this case even disappear due to the fact that U1 will address U2 in the following turn with a reaction towards U2’s interruption. In this section the dialogue model used for our dialogue management was introduced in detail, stating in what ways the information state approach has to be adapted for our needs. Additionally, we introduced a new interaction principle necessary to enable proactive interaction by side-participants. The new principle was then adopted to our interaction protocols. In the following, our dialogue modelling is adopted to the example domain of restaurant selection. For an example proceeding of our dialogue management refer to Section 4.4 which shows a sequence of information states updates on part of the example dialogue of Section 3.3.3.

4.2 Dialogue Management in the Example Domain of Restaurant Selection

81

4.2 Dialogue Management in the Example Domain of Restaurant Selection The modified approach of dialogue modelling introduced in the foregoing section is now adopted to our domain of restaurant selection resulting in the dialogue management module of our system. The module has been prototypically implemented in Java. For reasons of continuity and understandability we continue the description in easy understandable pseudocode and the notation presented by Larsson. The knowledge base for the application (i.e. restaurant database and menu files), the querying functionality, as well as the NLG and TTS modules are used from the wizard tool described in Section 3.2.4 and Appendix A. ASR and NLU components are at this point not integrated. A suitable statistical semantic parser is proposed in [Strauß and Jahn, 2007] (refer to Section 6.2). However, further work is necessary to achieve a performance sufficient for the scope of this work on dialogue management which aspires perfect parsing functionality. Thus, utterances are currently input in the form of the required semantic representation together with speaker and addressee information and the corresponding DA, as illustrated in Figure 4.1. This section starts with the description of domain-related components and knowledge sources before it adopts the update mechanisms of the system. 4.2.1 Dialogue Context The dialogue context of the system comprises the five types of context according Bunt’s DIT (1999) (refer to Section 2.4.1). The dialogue model and task model are explicit implementations of the linguistic and semantic context, respectively. The domain model, a knowledge source as part of the semantic context, describes the domain in form of a ontology and is presented below. The dialogue history poses an important part of the linguistic context and is described in detail in Section 4.3.3 below. The social context is implicitly contained in the update rules. It is made up of interactive and reactive pressures of the system, i.e. the system’s obligation or intention to address an open issue or to respond to a user request. The cognitive context denotes the system’s state and processing status. For this, a variable is deployed to denote the current activity state of the system: inactive, active, or interactive. Finally, the physical and perceptual context incorporates the physical world the system is interacting in and with. A variable is used to indicate what is currently displayed on the screen as it influences the system’s as well as the users’ actions. Further, another variable is deployed that holds information about if the main interaction partner is currently looking at the system or not. 4.2.2 Domain Model The domain model is an external knowledge base that belongs to the semantic context. It describes the concepts used in the task model and database and

82

4 Dialogue Management for a Multi-Party Spoken Dialogue System

how they related to each other. Our example domain is restaurant selection. It is modelled in form of the (simplified) ontology depicted in Figure 4.4. An ontology is used as a formal representation of the domain describing concepts, their properties and the relationships among these concepts. Concepts (or classes) are abstract objects that can be divided into subclasses and possess superclasses. Individuals are instances of the classes. They denote real objects and represent the knowledge in the ontology. In our case, the five main classes of the ontology are ’cuisine’, ’category’, ’ambiance’, ’price’ and ’location’. Each of these classes is divided into subclasses. For instance, the class ’cuisine’ subsumes all different kinds of cuisines that exist in the modelled world (the subclasses), i.e. ’Hispanic’ of which ’Spanish’ and ’Mexican’ are two instances. Various relations exist between the elements of a ontology in the form of object properties. This way, it can for example be determined that certain districts of the city are adjacent to other districts (on the concept level) and places are adjacent to other places on the individuals level. The ontology could theoretically also be used as a database to hold the actual contents of the modelled domain. It is, however, not a flexible data structure in terms of updates and we further aim at keeping the conceptual and data level separate. The ontology is used for problem solving to consider related objects in the database search if the designated preferences do not yield a result (refer to Section 4.5). It is further used as a dictionary for pragmatic interpretation in order to resolve relations between the actually entered values and concepts the system can map to the entries in the database. For instance, the users want a restaurant ’close to’ where they currently are. With the help of the ontology, the system maps ’close to’ to the neighbouring areas of the current location of the users. The obtained locations can then be used for the database query. 4.2.3 Task Model The task model constitutes the main part of the semantic context as it contains the propositional information necessary to perform the task. Thus, the task model deployed by our system contains all user preferences that have come up in the conversation. These so-called constraints are stored as attribute value pairs, i.e. the category of the constraint together with the value. For instance, the preference of Italian food would be represented as category cuisine, value Italian. The polarisation of the value (whether it is a user preference (positive constraint) or dislike (negative constraint)) is stored in an according priority value which adopts a positive or negative value accordingly. Prioritisation is discussed further in Section 4.5 as part of the problem solving mechanism of the system. The task model is managed through the update mechanisms that integrate the propositions which come up in the conversation into the context. An optimistic integration strategy is deployed to build the task model (refer to Section 4.3.1), i.e. that any proposition that comes up in an utterance is

4.2 Dialogue Management in the Example Domain of Restaurant Selection DANCEFLOOR

DANCEFLOOR

OUTSIDE

AMBIANCE HISPANIC

OUTDOOR

SPANISH

NON SMOKINGAREA

TURKISH SMOKING

MOROCCAN CUISINE

INDIAN

INDIAN

GREEK

GREEK

TERRACE BEERGARDEN

MEXICAN ORIENTAL

SMOKINGAREA

3OEFLINGEN

CHINESE ASIAN

WEST 7ESTSTADT

THAI

&RIEDRICHSAU

SUSHI ITALIAN

EAST

ITALIAN

/STSTADT

PIZZA AMERICAN

83

%SELSBERG

BURGER

NORTH

DISTRICTS

-ICHELSBERG

AMERICAN

THING

UNIVERSITY LOCATION

RIVER

CAFE COFFESHOP

TRAINSTATION

PASTRYSHOP

CENTRE

ICECREAMPARLOUR

OLDTOWN

SNACKBAR

CATEGORY FAST FOOD

THEATRE

FAST FOODPLACE

5LM 5LM

CITY

PUB BAR

CATHEDRAL

.EU 5LM

COCKTAILBAR BEERHALL INEXPENSIVE

RESTAURANT

INEXPENSIVE

RESTAURANT

CHEAP

PIZZERIA

NORMAL PRICECATEGORY

MODERATE MODERATE EXPENSIVE EXCLUSIVCE EXCLUSIVE

Fig. 4.4. Ontolgy deployed for the restaurant domain in a slightly simplified form.

integrated into the task model right away. This happens regardless of whether it is accepted or rejected in the following utterance or if it is even acknowledged and thus explicitly grounded. The aim is to memorise every proposition that has been mentioned during the dialogue, even if it is finally not included in the set of valid user constraints. All participants are aware of the fact that it has been mentioned. In the case that it should come up again later on in the dialogue, the proposition is reintroduced as opposed to being mentioned for the first time. A reject of the introduced proposition in the following utterance (e.g. if the other participant does not agree with a suggested proposition) does

84

4 Dialogue Management for a Multi-Party Spoken Dialogue System

not result in a rollback, i.e. the constraint is not taken out of the task model. The polarisation and value of the constraint are adapted accordingly which will exclude it from the set of currently valid constraints. It will thus not be further considered for database queries, however, it is besides being implicitly included in the dialogue history, still explicitly contained in the task model. 4.2.4 Information State Updates The designated flow of the interaction is described by the interaction protocols (refer to Section 4.1) which are enabled by a composition of information state updates as described in the following. Every utterance induces an update of the information state as described in Section 2.4.3. What kind of update is required is determined by the dialogue move an utterance (or part of an utterance) is assigned. Dialogue moves denote the actions that are performed with an utterance in the dialogue. The type of dialogue move that is assigned to an utterance is defined by the ’relation between the content of the utterance and the activity in which the utterance occurs’ [Larsson, 2002, p.32]. The different dialogue moves have to cover the range of all different sorts of utterances that can occur in the dialogue and need to be differentiated by the system. The task-oriented dialogue we are considering contains mainly the actions of suggestions and requests from the users’ side as well as reactions towards these moves as can be seen in the example dialogue in Section 3.3.3. We therefore provide functionality in terms of the following set of dialogue moves: • • • • • • •

suggest(p) with p:Proposition request(q) with q:Question reply() (*)greet() (*)inform(o) with o:Object *respond(o) with o:Object *consultDB() or *consultDB(o) with o:Object

While suggest, request and also greet moves occur mostly as initiating moves, reply is a so-called backward looking move referring to something already mentioned in the dialogue. The content of the moves is further defined by dialogue acts as introduced in Section 3.3.2. The tagset of dialogue acts used consists of suggest, request, inform, accept, reject, acknowledge, check, stall, greet, other). Some of these acts map directly to a dialogue move, others are mapped various to one act, a reply() move can for instance be an accept, reject or acknowledge. Besides the dialogue moves which denote the actions that occur in relation with certain utterances, further moves are deployed by the system as actions. These moves are marked with a star in the list above. The moves marked with a star in parentheses occur in both domains. The system puts the moves respond(o), inform(o), consultDB() or consultDB(o), and greet() on its

4.2 Dialogue Management in the Example Domain of Restaurant Selection

85

agenda to appoint its immediate actions. The respond(o) with o:Object move is put on the agenda after the system has been addressed with a request for reactive interaction. The object o denotes the piece of information that is requested. The move inform(o) with o:Object is used in the proactive case when the system decides to interact after evaluation of the result set from the latest database query, for instance. The object o is in this case the result set. ConsultDB() or consultDB(o) with o:Object is used to perform a database query either in general (based on the task model) or in order to query about an object o (i.e. a specific restaurant or a proposition) which was specified within the latest utterance. Update rules are used to integrate the content of the incoming utterances into the information state and to decide upon appropriate further actions for the system. The organisation of the point in time as well as what kind of and in what order updates are performed is regulated by the update strategy depicted in Algorithm 2. It is based on Larsson’s update strategy (described in Section 2.4.3) with modifications addressing the fact that in our case the agenda items possibly remain on the agenda for a few turns (whereas the agenda was cleared in the beginning of every cycle in Larsson’s algorithm [Larsson, 2002]). Further, we allow for interruptions (as described below) and change the way QUD is downdated. Newly added QUD elements dissolve existent elements as before, however, stay on QUD in the resolved state for another turn to enable a further reaction (possibly from a different dialogue partner) to the same issue. Only then, it is taken from QUD entirely. while NOT $LatestUtterance == failed do load $LatestUtterance; integrate; try downdateQUD; checkAgenda; try loadPlan; repeat execAgenda end

Algorithm 2: Update algorithm.

The proceeding of the algorithm is presented in the following; an illustrative example is provided in Section 4.4. The algorithm starts with loading the new utterance into the according field (LU) of the IS. The data is in the following step integrated into the information state. Update rules of the class integrate handle all different sorts of input that can occur in the dialogue depending on the dialogue move. Actions to be performed by the system are here put on the AGENDA. In the next step the QUD is checked for possible downdates, i.e. issues that have been resolved (by the utterance before and were not addressed at the current contribution) can be taken from QUD. Sim-

86

4 Dialogue Management for a Multi-Party Spoken Dialogue System

ilarly, the AGENDA is now checked for released or remaining obligations. An intended system action could have become superfluous due to a changed situation caused e.g. by an interruption of another dialogue partner. Existing items on the AGENDA are in the following step aligned with the system’s current plan. Finally, the agenda items are executed. The algorithm proceeds in an atomic manner up to this point. Database queries are performed subsequent to the above block. After the evaluation of the result set and the corresponding agenda update, the system pauses to check the context before it performs a prompt. At this point, another dialogue partner might have already taken the turn; because having been addressed or out of his own motivation. If no other contribution has been uttered the system proceeds as intended, otherwise, the cycle starts again loading the next utterance. Obligations that could not be fulfilled remain on AGENDA and are reconsidered at checkAgenda. The update rules define the operations that can be performed on the information state and are in general based on the rules from the original ISU approach. Continuous modifications have been performed throughout all rules considering the modified interaction protocols and changes implemented on the information state in the process of the adoption to the multi-party situation. We do not list the complete collection but present exemplary rules (adhering to Larsson’s notation) to demonstrate the modifications. Kronlid (2008) argues that for multi-party dialogue modelling the rule getLatestMove is no longer needed due to the fact that, regardless of their origin, all utterances have to be integrated. We claim that the rule is needed to load the latest utterance into the information state and thus keep it. However, we change its name to getLatestUtterance as it loads more information than just the latest move. The rule is listed in Rule 5.1. It is modified from the original in order to allow multiple moves per utterance which occurs frequently in the dialogues (refer to the example dialogue in Section 3.3.3). Further, a set of addressees is deployed in order to know who can speak to whom about what. As input, the values of the latest utterance are assumed to be stored in the variables $Latest Speaker, $Latest Addressees, and $Latest Moves. RULE 5.1: getLatestUtterance; CLASS: grounding; PRE: $Latest Moves==Set(Move) $Latest Speaker==Participant $Latest Addressees==Set(Participant) EFF: copy($/SHARED/LU/MVS,$Latest Moves) copy($/SHARED/LU/SPKR,$Latest Speaker) copy($/SHARED/LU/ADDS,$Latest Addressees)

4.2 Dialogue Management in the Example Domain of Restaurant Selection

87

Another example is provided with two update rules of the integrate class as shown in Rule 5.2 and Rule 5.3. Rule 5.2 integrateSuggest3 is performed after U1 suggested a proposition to U2 (suggest moves towards the system are not common). Rule 5.3 integrateRequest describes the integrating actions performed after U1 requests q from S. Let q be for instance a detail of information about the currently selected restaurant (i.e. SO != 0). The rules integrate the information stored in LU into the appropriate fields of the IS. Thereby, issues are put on QUD (together with speaker and addressee), any propositions are added to TASK, and induced system actions are put on AGENDA (e.g. the action to perform a database query after a constraint is introduced). RULE 5.2: integrateSuggest; CLASS: integrate; PRE: $/SHARED/LU/SPKR==U1 $/SHARED/LU/ADDS==U2 in($/SHARED/LU/MVS,suggest(p)) EFF: put(/SHARED/QUD/SPKR,$/SHARED/LU/SPKR) put(/SHARED/QUD/ADDS,$/SHARED/LU/ADDS) put(/SHARED/QUD/Q,p?) put(/SHARED/TASK,(p)) put(/PRIVATE/AGENDA,reply()) put(/PRIVATE/AGENDA,performDB())

RULE 5.3: integrateRequest; CLASS: integrate; PRE: $/SHARED/LU/SPKR==U1 $/SHARED/LU/ADDS==S in($/SHARED/LU/MVS,request(q)) $/SHARED/SO!=0 EFF: put(/SHARED/QUD/SPKR,$/SHARED/LU/SPKR) put(/SHARED/QUD/ADDS,$/SHARED/LU/ADDS) put(/SHARED/QUD/Q,q) put(/PRIVATE/AGENDA,reply(SO(q)))

Requests occur frequently throughout the dialogues in different forms (as can be seen in the example dialogue in Section 3.3.3). It can occur plain 3

As opposed to Larsson we omit the speaker information in the rule names (e.g. integrateUsrAsk and integrateSysAsk) and implement various rules with the same name. The speaker information is at this point redundant due to the fact that ambiguity is prevented by imposing different, speaker-dependent preconditions which all need to be true for a rule to fire.

88

4 Dialogue Management for a Multi-Party Spoken Dialogue System

without content (e.g. in ’Do you have anything like that?’) or supplying information, such as a proposition (e.g. as in ’Do you have anything Indian?’). These moves belong to the category request for query which aims at queries to obtain a set of restaurants or one specific restaurant, e.g. ’Do you know Bella Napoli?’. The move can further appear in the context of a request for information asking about properties of a specific restaurant, such as a piece of database information (e.g. ’Where is it?’) or requesting information concerning a specific category (e.g. ’Does it have a non-smoking section?’). The difference is that in the latter case the SO field is not empty (or filled during this move). We are in the following taking a closer look at the different possible request moves and system reactions. For this, we abstract from the low-level update rules which operate directly on the fields of the information state. Instead, we list rules in more compact and function-like pseudocode which enable the comprehension of the system’s functionality and line of action. The focus lies on the effect which is to be achieved with the execution of a composition of update rules as it is done in a similar way in the interaction protocols. However, instead of looking at each interaction partner and how they relate, only the viewpoint of the system is described. An example is listed in Rule 5.3’ showing the compact version of Rule 5.3. RULE 5.3’: request; PRE: mt ==request(U1:S,p) SO!=0 EFF: mt+1 =reply(S:U1,SO(cat(p)))

The simple syntax is shortly described: mt == Move(Speaker:Addressee, ’content’) denotes the move at point in time t, thus mt+1 indicates the subsequent move. In the example Rule 5.3’ the target move is a reply of the system addressing U1 with the requested information. The rules differ to the original update rules especially in the relationship between the preconditions and effects parts. The compact rules do not describe one step after the other in terms of operations on the information state but the intended target move leaving out the steps in between. This way, it is focused on what the system does and not how it is performed. When these rules are actually applied, further steps are performed in between and newly incoming moves can possibly influence the flow of the dialogue in the way that they interrupt and render the planned move redundant (as was described in the scope of the update algorithm above). The compact notation does not include a name of a rule class as the rules belong to more than one class of update rules. Rule 5.3’ for instance considers integration as well as generation and also grounding. The rules are listed with the different circumstances in which they can appear whereas only the direct addressing of the system is considered. The moves are again not given distinct names as the preconditions disambiguate the appropriate candidate.

4.2 Dialogue Management in the Example Domain of Restaurant Selection

89

Rule 5.4 shows a request for query without specifying a value. Thus, the result set of the latest database query is returned. Note that the query result is actually stored in BEL first and only at presentation time loaded into RES. However, we skip BEL for presentation and use only RES and SO instead, the latter indicating one single result. An example utterance for this sort of request would be ’Do you have anything like that?’. RULE 5.4: request; PRE: mt ==request(U1:S,{}) SO == 0 EFF: RES = consultDB(TASK) mt+1 =reply(S:U1,RES)

Rule 5.5 contains a request for query with the content of a proposition p. The proposition is integrated into the task model which induces a database query. The result set is in the following returned. An example would be: ’We would like to eat Indian food.’ RULE 5.5: request; PRE: mt ==request(U1:S,p) SO == 0 EFF: try update TASK(p); BEL = consultDB(TASK); mt+1 =reply(S:U1,BEL)

Rule 5.6 depicts a request for information supplying a specific category c. The value of the requested category of the restaurant under discussion (SO) is returned. An example for an utterance of this kind is: ’What’s the address?’ RULE 5.6: request; PRE: mt ==request(U1:S,c) SO != 0 EFF: mt+1 =reply(S:U1,SO(c))

Rule 5.7 shows a request for information specifying a proposition p.4 It is tested if p is part of the currently selected restaurant (SO). In both cases, SO’s value for the category (cat) of p is returned, however, in two distinct prompts. A suitable example is: ’Does it have a non-smoking area?’ Rule 5.8 depicts a request for query supplying a specific restaurant R. General information about the restaurant is returned if the requested restaurant is 4

Note that this is the same rule as displayed in Rule 5.3’.

90

4 Dialogue Management for a Multi-Party Spoken Dialogue System RULE 5.7: request; PRE: mt ==request(U1:S,p) SO != 0 EFF: mt+1 =reply(S:U1,SO(cat(p)))

contained in the database. An example would be: ’Do you know the restaurant Bella Napoli?’ RULE 5.8: request; PRE: mt ==request(U1:S,R) EFF: try SO = consultDB(R) mt+1 =reply(S:U1,SO)

Rule 5.9 contains a request for information supplying a specific restaurant R together with a category c. The value of the requested category c of the specified restaurant (if it is contained in the database) is returned. An example of this move is: ’What kind of food does restaurant Hong-Kong have?’ RULE 5.9: request; PRE: mt ==request(U1:S,({R,c})) EFF: try SO = consultDB(R); mt+1 =reply(S:U1,SO(c))

Rule 5.10 lists a request for information supplying a restaurant R and a proposition p. It is tested if p is part of the specified restaurant R. In both cases, R’s value (if R is contained in the database) for the category cat of p is returned in a prompt according to the result of the test. Example: ’Does restaurant Hong-Kong serve Japanese food?’

RULE 5.10: request; PRE: mt ==request(U1:S,{R,p}) EFF: try SO = consultDB(R); mt+1 =reply(S:U1,SO(cat(p)))

For request for query, the system reply contains information about how many results the query yielded. In the case of one single result, general information about this restaurant is read out. If a menu is available, it is further offered to be shown. If the query yielded 2 to 4 results, the restaurants’ names are listed. In the case of a request for information, the information is supplied.

4.2 Dialogue Management in the Example Domain of Restaurant Selection

91

4.2.5 Dialogue Plans In task-oriented dialogue, the dialogue system has to accomplish a task. The way in which it has to act in order to achieve its goals is specified in the dialogue plans. The overall plan of the system consists of collecting all constraints that come up during the conversation, performing database queries and finally informing the users about the results until one is agreed upon. Different sub-plans are thus deployed which the system adopts according to its current activity state. During inactive state, the system’s plan consists of keyword spotting in order to detect the specified domain. The plan is depicted in Plan 5.1. Once this plan has been successfully achieved, the system’s state changes to active. The plan is now (as well as during interactive state) to find a suitable restaurant. While the system is in active mode it adopts the plan shown in Plan 5.2. The primary aim is to find an interaction point at which the state, and with that also the plan, changes. Plan 5.3 depicts the plan the system follows once it has adopted the interactive state. Plan 5.1: spotDomain { while NOT domainFound do (spot(keywords)) } Plan 5.2: findInteractionPoint { while NOT interactionCriteria do (collect(constraint) queryDB) } Plan 5.3: findRestaurant { while NOT restaurantfound do (collect(constraint) queryDB inform(result)) } In this section, the dialogue modelling of the foregoing section was put into context using the example domain of restaurant selection. Thus, the task and domain model were described, as well as the system’s update mechanisms and dialogue plans. In order to operate, the system further needs a grounding and interaction strategy which are described in the following.

92

4 Dialogue Management for a Multi-Party Spoken Dialogue System

4.3 Enabling Proactiveness The dialogue management strategies for our system are deployed according to enable proactive system interaction. The system interaction strategy is described in Section 4.3.2 in the scope of which we introduce suitable points for proactive interaction in the dialogue identified by empirical analysis. Section 4.3.3 presents the dialogue history deployed in our system which provides the system with the knowledge about the conversation to be able to interact proactively already for its first interaction. In the following, the grounding and integration strategy is presented which operates optimistically, i.e. the system does not wait for explicit grounding of the dialogue partners in order to process the incoming information. This way, the new information is available to the system as soon as possible. 4.3.1 Optimistic Grounding and Integration Strategy for Multi-Party Setup Grounding ensures that all dialogue participants are at the same level of information at each point in the conversation. The common procedure would be to process contributions only when it is ensured that the information has been understood by all dialogue participants. In our case, we deploy an optimistic grounding strategy. That means, the system does not await the reaction of the other dialogue partner(s) that would signal grounding (e.g. in the form of acknowledgement) before processing a contribution. This strategy is enabled by assuming perfect understanding in our system as was the case in the WOZ-recorded dialogues facilitated by the simulation of ASR and NLU. The system does not provide an action to be taken in the case of misunderstanding. If understanding problems were to occur the system as is would have to be changed. A differentiation between negative feedback and understanding problems would be necessary. Wrong values that originate in understanding errors should not be kept in the dialogue history as these values do not denote an intended part of the conversation. The faulty understood value has to be taken out of the task model; and no trace of it should stay in the dialogue history. A cautiously optimistic grounding strategy that enables a rollback of the misunderstood value could be deployed to handle this situation. Larsson (2002) defines a cautiously optimistic grounding strategy as an optimistic strategy with the option for rollback in the case of overhasty integration of a value. In our case, a rollback of that sense is not needed in the case of overhasty integration. A rollback is only needed in the case of understanding problems. A temporary field would have to be integrated into the information state to hold ungrounded information. As soon as the information gets grounded it can be integrated into the task model. The optimistic grounding strategy is chosen in order to be able to immediately integrate the incoming information into the information state. We

4.3 Enabling Proactiveness

93

further deploy an optimistic integration strategy, i.e. the content of every utterance is immediately integrated into the task model without waiting for the other dialogue partner’s response of acceptance or rejection. This is necessary for the system’s proactive behaviour. The aim is for the system to be up to date with the current set of constraints as well as the query result set at each point in the dialogue to be able to proactively interact whenever it becomes necessary. If later on in the dialogue, the constraint is accepted in the response of the dialogue partner, nothing has to be done. Negative feedback effects a change in the task model, i.e. the newly added constraint either has to be taken out or changed in its polarity, as described in the following. Every proposition ever mentioned in the dialogue is thus included in the dialogue history due to the fact that all participants are aware of the fact that this proposition has been mentioned in the dialogue. In the case of reintroduction in the further flow of the conversation, it is not an entirely new constraint and therefore should not be treated as such. There are different ways of dealing with negative feedback. A rejection indicates discontent, i.e. the participant does not agree on a certain point or constraint. The constraint could either be taken out of the set of valid constraints and would thus be not considered for the database queries. Or, the introduced preference could be marked as a negative constraint indicating something the users do not want and would in this way be considered further for the database queries. It is not a simple task to determine which constraint in the dialogue to integrate in which way. A decision is further necessary with every introduction of a new constraint in the case that the task-model already contains another constraint of the same category. Should the constraints be treated conjunctive or disjunctive in the query (i.e. they are each independent query criteria as opposed to combined building one criteria), or should the second one replace the first one? These questions cannot clearly be decided without taking the participant’s intention into account which can rarely be made out by analysing the utterance only. We shortly describe the way these points are handled in our integration strategy as a thorough investigation of this topic goes beyond the scope of our work. In our system, discontent results in a rollback of the conventional kind, i.e. the disagreed value is taken out of the currently valid set of constraints (not entirely, see above). This proceeding is adopted due to subjective analysis of the recorded dialogues where this is thought to have been the users’ intentions in the majority of times. Further, this way is less restrictive and thus less likely to result in an over-constrained situation. If necessary, the users can still adopt the constraint as a negative one later on in the dialogue by mentioning it again. We treat various constraints of the same category as disjunctive values, i.e. each one is individually considered as it is less restrictive. However, for exclusive categories we adopt replacement if the proposed constraints are not related to each other according to the ontology. As an example, if the users propose Chinese food and request Sushi (as in Japanese) a few utterances later, both values stay in the set of constraints. However, if the second proposal

94

4 Dialogue Management for a Multi-Party Spoken Dialogue System

would have been Italian cuisine, the first (’Chinese’) would be replaced as the two values are not related. The same procedure is adopted e.g. on the category ’location’, refer to Section 4.5 for a detailed discussion on the problem solving procedure of the system. 4.3.2 System Interaction Strategy The interaction strategy of the system aims at balancing between being showing presence in the conversation as a third interaction partner and not appearing too intrusive. Thus, the system interacts at appropriate points also proactively if no interaction request has been addressed. At the same time, it tries not to interrupt the interaction by obeying adjacency pair conventions, i.e. awaiting answers or reactions from the counterpart (which would especially be important if grounding was not performed optimistically). The different possible types of system interaction are described in the following. Reactive interaction is the most common type of interaction, i.e. the system is addressed directly by the main interaction partner. In proactive interaction the system interacts without being addressed. It takes place after pauses in the dialogue or if additional information the users could be provided with or should be informed about. In the recorded dialogues of the PIT corpus (refer to Section 3.3) it can be observed that most of the proactive interaction takes place at the first interaction of the system as well as the last (as is the case for instance in the example dialogue presented in Section 3.3.3). In between, the system is mostly addressed and included as a third interaction partner who is endowed with expert knowledge (refer to Section 5.4 for statistical information in this matter). The system responds with the requested data and, whenever appropriate, offers additional information, such as a menu or street map. The system’s interaction takes place as depicted in Table 4.3. The system’s first interaction is a greeting phrase in which the system introduces itself and offers assistance. It proceeds differently depending on the activity state the system is in. Interaction from inactive state is only possible upon a direct interaction request (U1:S) by the main interaction partner. The addressing is recognised by the keyword spotting mechanism. The utterance is then analysed entirely which enables the detection of propositional content. If the contribution was of the form request(p) the system will react with greet directly followed by an inform action5 which presents the result from the database query performed after integrating the proposition. Either way, the system skips in this case the active state and proceeds in the interactive state right away. When the system is in active state the interaction can either be reactive or proactive. Reactive interaction is once again induced by a direct 5

The ’dot’ in the notation is defined to denote that moves being performed successively by one dialogue participant (i.e. the system in this case) whereas the ’comma’-notation used in the following denotes that two moves are performed one after another by different interaction partners.

4.3 Enabling Proactiveness State

Action

Condition

inactive

greet greet.inform

U1:S reactive U1:S && LatestUtterance== reactive request(p)

active

greet

U1:S P IP U1:S && BEL ! = 0 P IP && BEL ! = 0

reactive proactive reactive proactive

interactive respond U1:S inform P IP && BEL ! = 0 suggest(o) with o:Object P IP && SO ! = 0 greet U1:S && SO ! = 0 SP IP && SO ! = 0

reactive proactive proactive reactive proactive

greet.inform

95

Interaction

Table 4.3. Dialogue system interaction types.

interaction request in the same way as described above. Proactive interaction occurs at certain proactive interaction points (PIP) which are specified below. The system’s interaction consists of a greet action which in the case that constraints have already been defined (or are defined within this request) and thus a database query has been (or is) performed is followed by an inform action presenting the result set6 . In the case that a specific object has been selected (i.e. spoken about) by the users, information about this object is given. From interactive mode the system interaction proceeds likewise, however, the actions performed by the system are respond in the reactive, and inform in the proactive case. In the latter case if the condition SO ! = 0 is further true, i.e. an object has already been selected by the users, the system proceeds by offering to show a menu or a city map, depending on the previous utterances. If the users have asked about the location or address a city map is offered whereas if they have been speaking about means of transportation the bus schedule is offered. In the general case, the restaurant’s menu is offered. The closing statement (i.e. the greet move) can be reactive or proactive after a restaurant has been agreed upon. For proactive interaction two criteria have to be considered: The contentual motivation of the system to interact and the appropriate point in time for interaction. The system’s aim of interaction is to assist the users in solving the task. It thus reports facts to the users that help promoting the task solving process such as results and also problems. The question is what kind of information is considered important to be reported proactively? The system can encounter four different situations when performing a database query which 6

The condition BEL ! = 0 implies that BEL is cleared after presentation, i.e. after RES is set (RES = BEL).

96

4 Dialogue Management for a Multi-Party Spoken Dialogue System

are of different significance for the dialogue, as listed in Table 4.4. The contentually relevant situations to be considered occur after a database query, i.e. the result has to be evaluated. Another content to be considered for proactive interaction denotes further the successful ending of the task solving process. If the users have agreed upon a specific restaurant the system can place its closing prompt and the interaction is over. The evaluation of the query result can expose four different situations: An over-constrained situation which yields no results, an under-constrained situation, i.e. the result set is too big to be presented; or it yields a presentable number of results which can either be one object or a small number of objects (2 to 4 in our case) that can be read out individually. The situation that one single result is returned from a query is considered optimal. It is closest to a possible solution denoting a successful end to the conversation and is thus rated most significant. The overconstrained situation constitutes another important situation as it denotes a dead end in the task solving process and thus requires an intervention in the form of a modification of the constraint set which can either be performed directly by the system in terms of compromises or relaxation as described in Section 4.5, or with the help of the users. The presentable number of results is possibly only one step away from the solution as it presents a small number of objects the users can chose from. The under-constrained case is considered least significant as it requires further constraints to be defined before finding a solution. The different levels of importance of the different contentual motivators does at this point not influence the interaction decision to a large extent. Whenever encountering a point in the dialogue suitable for proactive interaction the system takes this chance. As seen from the recorded dialogues, most of the interactions during the interactive phase of the system are reactive. The first and last interactions, however, very frequently occur in a proactive manner. These cases are thus considered separately, as described below. Result Set SO ! = 0 AND FINALLY ACCEPTED BEL == 1 BEL == 0 BEL == 2-4 BEL > 4

Situation

Significance

restaurant found

task solved

optimal over-constrained choice of results under-constrained

task is possibly solved problem, intervention required ok for presentation constraint definition ongoing

Table 4.4. Contentual motivation for proactive interaction.

The point in time for interaction is defined by observing the flow of the dialogue. A pause for instance, i.e. a break in the dialogue that exceeds a certain threshold in duration, denotes an interruption or time which the di-

4.3 Enabling Proactiveness

97

alogue partners use to think. It thus poses an ideal interaction point. The same effect is in natural dialogue often achieved by short utterances, such as acknowledgements or stall and check acts, denoting interruptions or fillers. It can be observed that certain sequences of dialogue acts that occur repeatedly in the recorded dialogues lead to a proactive system interaction (respectively wizard interaction) during the recordings. From observation of the Session III dialogues of the PIT corpus, the most common 7 sequence for proactive interaction consists of a suggest act (uttered by either the system or one of the users) which is followed by at least two acts of agreement by the users, such as one or more accept acts followed by an arbitrary amount of acknowledge acts. At times, a less significant stall or check act is additionally part of the sequence. In pseudo-code, the dialogue act sequence of agreement denoting a strong proactive interaction point (SPIP) is described as follows: [SPIP]

suggest,{[accept]+,[acknowledge]*}[2+]

It can be observed that this sort of sequence occurs at points in the dialogue that conclude something that has been discussed before and thus indicate a suitable point for a proactive system contribution. Linguistically, the sequence displays the termination of an interaction cycle consisting of initiating and concluding act, i.e. several of them in this case. After this, the dialogue continues with a new initiating act by an arbitrary speaker, i.e. either the previous speaker keeps the turn or the other dialogue partner grabs it. This point is thus ideally suited for system interaction of any kind. If this sequence occurs in the process of discussing a certain object it denotes its acceptance. The system offers to show an object or interacts with a closing statement, depending on the previous interaction (as described above). If this sort of agreement occurs at an earlier point in the dialogue, the system provides up-to-date information, i.e. presentation of the database result which can have the form of any of the four kinds listed above. The following sequence of dialogue acts also shows a complete interaction cycle in terms of dialogue acts, however, less assertive. It denotes a suitable proactive interaction point which occurs more frequently throughout the centre part of the dialogue, i.e. between the system’s first and last interaction (as can be seen e.g. in the example dialogue in Section 3.3.3 and Section 4.4 below). It can be described as follows: [PIP]

suggest,accept+

The suggest is uttered by one of the users addressing the other user, which is then responding with an accept act which denotes that both users agree to the proposed item. The system always waits for a reply of any kind by the 7

The most common types of proactive interaction are the system’s first and last utterance which could potentially have an impact on this fact.

98

4 Dialogue Management for a Multi-Party Spoken Dialogue System

other dialogue partner to not appear rude and provide the other participant a chance to react upon the proposition. In the case that the answer to the suggest was a reject no proactive interaction can take place. A rejection induces a change in the task model which denotes a rollback taking out the optimistically inserted constraint. The resulting database query might thus yield the same result that was possibly presented before. The system does not repeat the same result a second time. The system proactively chooses an interaction strategy regarding the presented proactive interaction points and contentual motivators. As seen in Table 4.3, PIP induces proactive interaction in all cases but for the last and final system utterance for which SPIP is required. In which way pauses or silence are to influence and induce proactive behaviour is not considered at this point as the system is not operating in real time. It poses an interesting question, however, which is subject for further research in the domain of non-verbal expressions in interaction which is beyond the scope of this work. 4.3.3 Dialogue History for Proactive System Interaction Proactive behaviour is achieved by full contextual awareness of the system. Constantly observing the situation and context relevant for the system enables it to instantly detect any occurring problems. Conversational and situational awareness should not only prevail of the current point in the interaction but include the past dialogue. Thus, a dialogue history is installed to provide the system with a memory. Besides the dialogical information it is further extended to include all task relevant information in order to be able to refer back to something said or restaurants discussed at an earlier point in the conversation. The aim is to endow our dialogue system with proactive behaviour. To recall, the system goes through the following life cycle: Figure 4.5 shows the different levels of attentiveness the system operates in. While the users talk about a different topic, the system is in inactive mode spotting for its specified domain. As soon as the conversation enters the domain of the system’s expertise the system adopts the active state. It pays close attention in order to model the dialogue between the users and build the task model and dialogue history. With the system’s first interaction it enters interactive mode during which it interacts as an independent dialogue partner assisting the users in every way possible. When the task is solved the system retracts from the conversation and once again adopts inactive mode. The system should not alone rely on the users asking for interaction but be able to decide for itself when it is appropriate and necessary to interact in order to solve a task in the best way possible. Thus, it interacts not only upon explicit interaction requests by the main user (reactive interaction) but also on its own initiative when important occurrences arise that are worth reporting to the users or if pauses occur in the dialogue (proactive interaction). The system’s interaction strategy is described in Section 4.3.2.

4.3 Enabling Proactiveness

99

domain spotting

INACTIVE conversation enters domain

task solved

System activity cycle INTERACTIVE ACTIVE pro- / reactive interaction

interaction required

constructing dialogue history

Fig. 4.5. System life cycle.

A conversation with a dialogue system normally starts with the first interaction of or towards the system which denotes the first utterance of the dialogue in single-party systems. At this point, the interaction and also dialogue history start being modelled. For our system this would be the point when the system steps into interactive mode in order to model the entire human-computer interaction. However, the multi-party situation our system encounters changes the requirements of the dialogue history modelling: The dialogue history has to represent also the duration the system is active in order to enable proactive system interaction, i.e. to be already endowed with knowledge at the point of its first interaction. The dialogue history is thus starting to be build as soon as the dialogue between the users enters the specified domain, i.e. at the point when the system enters active mode. It lasts until the task has been solved and the system takes itself back from the conversation. It thus has knowledge about the complete conversation and can refer back to values mentioned by the users while the system was only silently ’listening’ and does not require the users to repeat their preferences in a request to the system when the system is finally to interact. At the end of the interaction, the dialogue history is discarded. Figure 4.6 pictures the way the dialogue history relates to the dialogue.

100

4 Dialogue Management for a Multi-Party Spoken Dialogue System

$IALOGUE(ISTORY

3YSTEM INACTIVE

DOMAIN STARTS

FIRST INTERACTION

ACTIVE

INTERACTIVE

PROBLEM SOLVED

Fig. 4.6. Dialogue history as it relates to the dialogue.

Besides the common linguistic information necessary for smooth interaction, ellipsis and anaphora resolution, our dialogue history contains also task-related information. The dialogue is tracked in order to be able to reconstruct the conversation. By deploying the dialogue history, the system can resolve references to things mentioned earlier in the dialogue due to the fact that it is able to trace back to former states in the dialogue. It can adopt the former state completely, i.e. to reload the users’ preferences and result set of an earlier point of time, or extract some information from it. Thus, when the users refer back to earlier results as in e.g. ’Let’s just take the one from before’, the system looks in the dialogue history for the last presented result that had been agreed upon, depending on the context, and presents it again. The dialogue history consists of, in sequential representation, snapshots of the dynamic context at the current point in time. As part of the information state, all presented query results are stored (RES) with the information which of these objects is currently under discussion (SO). The users’ reaction to the presented objects can be accessed in the chronologically adjacent item of the dialogue history holding the next contribution. Dialogical information is stored to the extent of being able to fully reproduce the task-related issues. The private part of the information state, as well as the QUD field, is not part of the dialogue history. These components depend on the current point in the dialogue. There is no reason why the PLAN or AGENDA from former states should be accessed. The PLAN could possibly give information about the system’s activity state at the considered former dialogue state, however, this information can e.g. also be retrieved through an utterance’s relative position to the one containing the unambiguous dialogue move of the system’s first interaction. BEL holds the temporary query result and can thus be recalculated with a retrieved task model. The information contained in QUD is also of momentary value and will change to integrate the retrieved information into the current QUD. The structure of the dialogue history is depicted in Figure 4.7. The dialogue history elements are ordered chronologically, identified by a specific utterance number. The dialogue depicted in the excert is currently at number ti . Therefore, the dialogue history starts with ti−1 . The following elements are contained:

4.3 Enabling Proactiveness

⎡

⎛

⎡

SP K ⎢ ⎜ ⎢ ADS ⎢ ⎜ ⎢ ⎢ ⎜ ⎢ ⎢ DH : list ⎜ti−1 ⎢ M V S ⎢ ⎜ ⎢ T ASK ⎢ ⎜ ⎢ ⎣ ⎝ ⎣ RES SO

: : : : : :

⎤

101

⎞⎤

P articipant ⎟⎥ set(P articipant) ⎥ ⎥ ⎟⎥ ⎥ ⎟⎥ list(M ove) ⎥ , ti−2 [...], . . . , t1 [...]⎟ ⎥ ⎟⎥ set(P roposition) ⎥ ⎥ ⎟⎥ ⎦ ⎠⎦ list(Object) Object

Fig. 4.7. Dialogue history at point in time ti .

• • • •

SPK (Speaker): The speaker of this utterance. ADS (Addressees): The addressee or set of addressees of this utterance. MVS: The move(s) contained by the utterance. TASK: The task model in its current state is contained as a snapshot of the specific point in time. For a detailed description of TASK and its elements refer to Section 4.5 below. The current set of user preferences are contained in the dialogue history as they cannot be reconstructed any other way. According to the point in the dialogue when a preference has been introduced, the value’s priority is high or low, however, no immediate conclusion can be drawn about which preferences were valid at a certain point in time due to the fact that a preference can be taken out in the course of the dialogue (i.e. the value is not rising higher) and later be reintroduced (i.e. the value starts rising again from where it had stopped before). The prioritisation values do not have to be remembered and are thus not contained in the dialogue history. Reloading a previous set of user preferences has a different basis of conversational history at a later point in time. The dialogue has proceeded in the meantime and all dialogue participants are aware of this. Reconstructing an earlier situation thus happens on the basis of the current prioritisation values, in terms of a reintroduction of constraints that had been taken out in the meantime while constraints that have been added in the meantime are taken out. • RES (Result Set): Result set of objects returned by the latest database query, i.e. on the basis of the constraints contained in the TASK of this utterance. For this database query, the original prioritisation values had been used. • SO (Selected Object): The currently selected and discussed object (one of RES). The multi-party setup is only explicitly contained in the dialogue history in the speaker and addressee fields. The information about who proposed specific constraints, which could in general be traced using the speaker field, is not necessary for the system. However, if the dialogue was to be modelled more thoroughly in respect to the individual users by deploying user models to keep track of each user’s individual preferences the dialogue history would need to

102

4 Dialogue Management for a Multi-Party Spoken Dialogue System

be extended. It would have to include information of the proposing dialogue participant with each constraint as well as the other participant’s reaction towards it. User modelling would further require speaker identification so the system could load the right user’s profile if more than one main user is to be able to use the system. Situations can be thought of where user modelling is not intended, e.g. if a user does not want the other dialogue partner to know the content of previous interactions with the system, e.g. regarding the restaurant domain about her favourite or frequently visited restaurants. For our setup we do not deploy user modelling and thus, the presented task-related modelling of the dialogue history is sufficient. The dialogue history presented in this section contains all dynamic context information relevant in order to be able to reproduce information from earlier states in the dialogue. While prevalent dialogue systems (refer to Section 2.4.1) deploy a dialogue history mainly with the focus on linguistic and conversational matters, our dialogue history puts an emphasis on the taskrelated components to enable references to previously selected objects and restoring of formerly active constraint sets. Common dialogue histories start modelling with the first interaction of the system unlike ours which starts to model the dialogue as soon as the users come to speak of the specified domain and this way enables proactive system interaction. All task relevant information is already collected already before the system has joined the interaction. The system can thus interact at any appropriate point, proactively or upon request, and directly provide relevant information.

4.4 Proactive Dialogue Management Example A part of the example dialogue presented in Section 3.3.3 is listed in Figure 4.5. In the following, the extract is depicted as a sequence of information states in order to illustrate the presented proactive dialogue management approach. ID

Spkr Addr

Utterance

DA

16

U1

U2

sug

17 18

U2 S

U1 U1

So, let’s go to the centre cause we’re gonna meet the others later on anyway.. Right. Your query for a locality with Italian cuisine in the centre with inexpensive price category yields 5 results. Please confine your request further.

Table 4.5: Snippet of example dialogue from the PIT corpus.

acc inf

4.4 Proactive Dialogue Management Example

103

Figure 4.8 shows utterance 16 ”So, let’s go to the centre cause we’re gonna meet the others later on anyway..” at the point of getLatestUtterance at the beginning of the update cycle. Thus, the LU is filled with the latest details: U1 is the speaker, U2 addressee. The utterance contains one move, suggest. TASK contains all propositions collected in the dialogue up to this point: f2 stands for Italian, p1 denotes inexpensive. The proposition f1 stands for Mexican, however, is not active at the moment which is demonstrated by the parentheses. The RES field contains the set of restaurant presented in the foregoing utterance. The QUD field does not contain an entry for res1 due to the fact that res1 was uttered in the form of an inform as it contains too many entries to be listed in detail. The users were thus asked to alter their query. DH contains the previous utterances. The private part of the IS contains integrate on the AGENDA as it is the next action to be performed regulated by the update strategy. Generally, these actions are handled implicitly without appearing on the agenda. In this case, we place the actions of the update algorithm explicitly on the AGENDA for demonstrative reasons. The plan in this phase of the dialogue is findRestaurant (refer to the description of dialogue plans in Section 4.2.5). ⎡

⎡

⎤

⎤

AGEN DA : integrate ⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎦ ⎥ ⎢ ⎥ ⎢ BEL : ⎢ ⎡ ⎤⎥ ⎥ ⎢ T ASK : [f ] f p 1 2 1 ⎥ ⎢ ⎢ ⎢ SO ⎥⎥ : ⎢ ⎢ ⎥⎥ ⎢ ⎢ RES : res1 = (R1 , . . . , R6 ) ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢ ⎢ QU D : {} ⎥⎥ ⎢ SHARED : ⎢ ⎧ ⎡ ⎤⎫ ⎥ ⎥ ⎥ ⎢ ⎢ SP K : U 1 ⎨ ⎬⎥ ⎢ ⎢ ⎥⎥ ⎥ ⎢ ⎢ LU ⎥ ⎣ ⎦ t : ADS : U 2 16 ⎥ ⎢ ⎢ ⎩ ⎭⎥ ⎣ ⎣ ⎦⎦ M V S : sug(l1 ) DH : {t15 , t14 , . . . t1 } Fig. 4.8. Example information state after getLatestUtterance of utterance 16.

Figure 4.9 depicts the subsequent step in the sequence where the utterance is integrated into the IS. The suggest move initiates that the associated proposition l1 (which stands for towncentre) is put on QUD as the currently discussed topic. The proposition is further included in TASK due to our optimistic handling of the integration (refer to Section 4.3.1). A change in the TASK field induces a database query. Thus, consultDB is put on the AGENDA. The RES field is cleared as res1 is not subject of the current utterance and further outdated after the change of TASK. Before the database query takes place, QUD has to be checked for possible downdates as the next step in the update algorithm.

104

4 Dialogue Management for a Multi-Party Spoken Dialogue System

⎡

⎡

⎤⎤

AGEN DA : downdateQU D, consultDB ⎦⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎥ ⎢ ⎥ ⎢ BEL : ⎥ ⎢ ⎡ ⎤ ⎥ ⎢ T ASK : [f ] f p l 1 2 1 1 ⎥ ⎢ ⎥ ⎢ ⎢ SO ⎥ : ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ RES : ⎥ ⎥ ⎢ ⎧⎡ ⎫ ⎢ ⎥ ⎤ ⎥ ⎢ ⎢ ⎥ : l1 ? ⎬ ⎨ Q ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ QU D : ⎣ SP K : U 1 ⎦ ⎥ ⎥ ⎢ SHARED : ⎢ ⎥ ⎩ ⎭ ⎥ ⎢ ⎢ ⎥ : U 2 ⎥ ⎢ ⎢ ⎥ ⎧ ADS ⎫ ⎡ ⎤ ⎥ ⎢ ⎢ ⎥ SP K : U 1 ⎨ ⎬⎥ ⎥ ⎢ ⎢ ⎥ ⎢ ⎢ LU ⎥ ⎣ ⎦ : t16 ADS : U 2 ⎥ ⎢ ⎢ ⎥ ⎩ ⎭⎦ ⎦ ⎣ ⎣ M V S : sug(l1 ) DH : {t15 , t14 , . . . t1 } Fig. 4.9. Example information state after integrate of utterance 16.

Figure 4.10 shows the information state after the database query has been performed. The query result is stored in BEL. The dialogue is at this point ready for the next utterance. The expected next speaker is U2 with a reply to utterance 16. The system waits for the other party to reply in order not to interrupt. The proactive interaction criteria (as described in Section 4.3.2) are not fulfilled at this point. System interaction would only take place at this point if a ’long’ pause would follow this utterance. Figure 4.11 depicts the information state of the dialogue after loading the next utterance ”Right.”, an accept move by U2. The utterance is integrated in the following step. ⎡

⎡

⎤

⎤

AGEN DA : inf orm(res2 ) ⎦ ⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎥ ⎢ ⎥ ⎢ : res2 = (R1 , . . . , R5 ) ⎢ ⎡ BEL ⎤⎥ ⎥ ⎢ T ASK : [f1 ] f2 p1 l1 ⎥ ⎢ ⎥ ⎢ ⎢ SO ⎥ : ⎢ ⎢ ⎥⎥ ⎢ ⎢ RES : ⎥⎥ ⎢ ⎧⎡ ⎢ ⎥⎥ ⎤⎫ ⎢ ⎢ ⎥⎥ : l1 ? ⎬ ⎨ Q ⎢ ⎢ ⎥⎥ ⎢ ⎢ QU D : ⎣ SP K : U 1 ⎦ ⎥⎥ ⎢ SHARED : ⎢ ⎥⎥ ⎩ ⎭ ⎢ ⎢ ⎥⎥ ⎢ ⎢ ⎧ ADS ⎡ : U2 ⎤⎫ ⎥ ⎥ ⎥ ⎢ ⎢ SP K : U 1 ⎨ ⎬⎥ ⎢ ⎢ ⎥⎥ ⎥ ⎢ ⎢ LU ⎥ ⎣ ⎦ : t16 ADS : U 2 ⎥ ⎢ ⎢ ⎩ ⎭⎥ ⎦ ⎣ ⎣ ⎦ M V S : sug(l1 ) DH : {t15 , t14 , . . . t1 } Fig. 4.10. Example information state after consultDB of utterance 16.

4.4 Proactive Dialogue Management Example

⎡

⎡

105

⎤⎤

AGEN DA : integrate, inf orm(res2 ) ⎦⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎥ ⎢ ⎥ ⎢ = (R , . . . , R ) BEL : res 2 1 5 ⎥ ⎢ ⎡ ⎤ ⎥ ⎢ T ASK : [f ] f p l 1 2 1 1 ⎥ ⎢ ⎥ ⎢ ⎢ SO ⎥ : ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ RES : ⎥ ⎥ ⎢ ⎧⎡ ⎫ ⎢ ⎥ ⎤ ⎥ ⎢ ⎢ ⎥ : l1 ? ⎬ ⎨ Q ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ QU D : ⎣ SP K : U 1 ⎦ ⎥ ⎥ ⎢ SHARED : ⎢ ⎥ ⎩ ⎭ ⎥ ⎢ ⎢ ⎥ : U 2 ⎥ ⎢ ⎢ ⎥ ⎧ ADS ⎫ ⎡ ⎤ ⎥ ⎢ ⎢ ⎥ SP K : U 2 ⎨ ⎬⎥ ⎥ ⎢ ⎢ ⎥ ⎢ ⎢ LU ⎥ ⎣ ⎦ : t17 ADS : U 1 ⎥ ⎢ ⎢ ⎥ ⎩ ⎭⎦ ⎦ ⎣ ⎣ M V S : acc() DH : {t16 , t15 , . . . t1 } Fig. 4.11. Example information state after getLatestUtterance of utterance 17.

Figure 4.12 illustrates the information state after the integration of utterance 17. The accept move is added to QUD with the accepted proposition l1 , the respective speaker and addressee. The content of the previous latest utterance has been put on DH as described in Section 4.3.3. The next step addresses the QUD downdate. The element put on QUD by the accept move dissolves the other element contained in QUD. As described in Section 4.2.4, resolved QUD elements are left on QUD for one extra turn to enable other dialogue partners to address the issue. Thus, no change takes place in the information state. The dialogue modelling is at this point ready to load the next utterance. The system takes the initiative and informs the user about the re⎡

⎡

⎤ ⎤

AGEN DA : downdateQU D, inf orm(res2 ) ⎦ ⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎥ ⎢ ⎥ ⎢ = (R , . . . , R ) BEL : res 2 1 5 ⎢ ⎡ ⎤⎥ ⎥ ⎢ T ASK : [f ] f p l 1 2 1 1 ⎥ ⎢ ⎢ ⎢ SO ⎥⎥ : ⎢ ⎢ ⎥⎥ ⎢ ⎢ RES : ⎥⎥ ⎢ ⎧⎡ ⎢ ⎤ ⎡ ⎤⎫ ⎥ ⎥ ⎢ ⎢ ⎥⎥ Q : l Q : l ? ⎨ ⎬ 1 1 ⎢ ⎢ ⎥⎥ ⎢ ⎢ QU D : ⎣ SP K : U 2 ⎦ , ⎣ SP K : U 1 ⎦ ⎥ ⎥ ⎥ ⎢ SHARED : ⎢ ⎩ ⎭⎥ ⎢ ⎢ ⎥⎥ ADS ⎢ ⎢ ⎥⎥ ⎧ ADS ⎡ : U1 ⎤⎫ : U 2 ⎢ ⎢ ⎥⎥ SP K : U 2 ⎨ ⎬ ⎢ ⎢ ⎥⎥ ⎢ ⎢ LU ⎥⎥ ⎣ ⎦ : t ADS : U 1 17 ⎢ ⎢ ⎥⎥ ⎩ ⎭ ⎣ ⎣ ⎦⎦ M V S : acc() DH : {t16 , t15 , . . . t1 } Fig. 4.12. Example information state after integrate of utterance 17.

106

4 Dialogue Management for a Multi-Party Spoken Dialogue System

⎡

⎡

⎤

⎤

AGEN DA : integrate ⎦ ⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎥ ⎢ ⎥ ⎢ = (R , . . . , R ) BEL : res 2 1 5 ⎢ ⎡ ⎤⎥ ⎥ ⎢ T ASK : [f ] f p l 1 2 1 1 ⎥ ⎢ ⎢ ⎢ SO ⎥⎥ : ⎢ ⎢ ⎥⎥ ⎢ ⎢ RES : ⎥⎥ ⎢ ⎧⎡ ⎢ ⎤ ⎡ ⎤⎫ ⎥ ⎥ ⎥ ⎢ ⎢ : l1 Q : l1 ? ⎬ ⎥ ⎨ Q ⎢ ⎢ ⎥⎥ ⎢ ⎢ QU D : ⎣ SP K : U 2 ⎦ , ⎣ SP K : U 1 ⎦ ⎥ ⎥ ⎥ ⎢ SHARED : ⎢ ⎩ ⎭⎥ ⎢ ⎢ ⎥⎥ ADS : U 2 : U 1 ⎥ ⎢ ⎢ ⎥ ⎧ ADS ⎫ ⎡ ⎤ ⎢ ⎢ ⎥⎥ SP K : S ⎨ ⎬ ⎢ ⎢ ⎥⎥ ⎢ ⎢ LU ⎥⎥ ⎦ : t18 ⎣ ADS : U 1 ⎢ ⎢ ⎥⎥ ⎩ ⎭ ⎣ ⎣ ⎦⎦ M V S : inf (res2 ) DH : {t17 , t16 , . . . t1 } Fig. 4.13. Example information state after getLatestUtterance of utterance 18.

sult of the latest database query (performing the inform move on AGENDA) making use of the detected proactive interaction point: The past two moves constitute a dialogue sequence defined as a convenient interaction point for the system (refer to Section 4.3.2). Figure 4.13 depicts the information state after the details of the new utterance has been loaded into LU and DH has been updated. Figure 4.14 shows the information state after utterance 18 has been integrated. The inform move induces res2 to be cleared from BEL and instead put in RES. The result set contains five items in this case, too many to be presented individually. Thus, the same situation is encountered as described above and no item is added to QUD. ⎡

⎡

⎤

⎤

AGEN DA : downdateQU D ⎥ ⎢ P RIV AT E : ⎣ P LAN : f indRestaurant ⎦ ⎥ ⎢ ⎥ ⎢ : ⎢ ⎡ BEL ⎤⎥ ⎥ ⎢ T ASK : [f1 ] f2 p1 l1 ⎥ ⎢ ⎥ ⎢ ⎢ SO ⎥ : ⎢ ⎢ ⎥⎥ ⎢ ⎢ RES : res2 = (R1 , . . . , R5 ) ⎥⎥ ⎢ ⎧⎡ ⎢ ⎤ ⎡ ⎤⎫ ⎥ ⎥ ⎥ ⎢ ⎢ : l1 Q : l1 ? ⎬ ⎥ ⎨ Q ⎢ ⎢ ⎥⎥ ⎢ ⎢ QU D : ⎣ SP K : U 2 ⎦ , ⎣ SP K : U 1 ⎦ ⎥ ⎥ ⎥ ⎢ SHARED : ⎢ ⎩ ⎭⎥ ⎢ ⎢ ⎥⎥ ADS⎤:⎫ U2 ⎢ ⎢ ⎥⎥ ⎧ ADS ⎡ : U1 ⎢ ⎢ ⎥⎥ SP K : S ⎨ ⎬ ⎢ ⎢ ⎥⎥ ⎢ ⎢ LU ⎥⎥ ⎣ ⎦ : t18 ADS : U 1 ⎢ ⎢ ⎥⎥ ⎩ ⎭ ⎣ ⎣ ⎦⎦ M V S : inf (res2 ) DH : {t17 , t16 , . . . t1 } Fig. 4.14. Example information state after integrate of utterance 18.

4.5 Problem Solving Using Discourse Motivated Constraint Prioritisation

107

In the following, QUD is downdated. This time, the latest utterance did not address the issue resolved in the previous utterance; it is thus taken from QUD. The result is shown in Figure 4.15. The example sequence is ended at this point as the system is waiting for the next utterance. ⎡

⎡

⎤

⎤

AGEN DA : ⎢ P RIV AT E : ⎣ P LAN ⎥ : f indRestaurant ⎦ ⎢ ⎥ ⎢ ⎥ : ⎢ ⎡ BEL ⎤⎥ ⎢ ⎥ T ASK : [f1 ] f2 p1 l1 ⎢ ⎥ ⎢ ⎥ ⎢ SO ⎥ : ⎢ ⎢ ⎥⎥ ⎢ ⎢ RES : res2 = (R1 , . . . , R5 ) ⎥⎥ ⎢ ⎢ ⎥⎥ ⎢ ⎢ ⎥⎥ {} ⎡ ⎢ SHARED : ⎢ QU D : ⎧ ⎤⎫ ⎥ ⎥ ⎢ ⎥ ⎢ SP K : S ⎨ ⎬⎥ ⎢ ⎢ ⎥⎥ ⎢ ⎥ ⎢ LU ⎥ ⎣ ⎦ : t18 ADS : U 1 ⎢ ⎥ ⎢ ⎩ ⎭⎥ ⎣ ⎦ ⎣ ⎦ M V S : inf (res2 ) DH : {t17 , t16 , . . . t1 } Fig. 4.15. Example information state after downdateQUD of utterance 18.

4.5 Problem Solving Using Discourse Motivated Constraint Prioritisation This section is dedicated to the problem solving mechanism of the system. The task at hand is rather simple and deploys constraint-based problem solving (similar to the one presented in [Qu and Beale, 1999]), a common and straightforward way of dealing with problem solving of this kind. During the course of the conversation, the system collects all information relevant for the task which forms the basis for the database queries. Positive (what the users want) and negative (what they don’t want) so-called constraints are combined to narrow down the result set. Depending on its size, the result set of objects is then presented to the users who can choose a suitable object. If the result set is empty or exceeds the number of presentable objects (less or equals four in our case), further actions are necessary. If the database query yields too many results, i.e. a number too high for the individual objects to be read out, an under-constrained situation (in short, UC) occurred. The result set has to be further constrained to obtain a presentable number of results. Alternatively, the results can be grouped in a certain way, these groups or topmost elements (according to some sort of rating) can then be presented to the users. If the query yields no results, an over-constrained situation (in short, OC) occurred, i.e. there are too many or too restrictive constraints. In this case, the system has to find an alternative solution to present to the user. The current constraint set needs to be altered in order to yield a solution. A

108

4 Dialogue Management for a Multi-Party Spoken Dialogue System

common way to handle this problem is to relax one or more constraints until the result is non-empty. The crucial decision hereby is which constraint(s) to relax, preferably the ones that are least important to the users. How can the importance or priority of constraints be determined? This section focuses on a solution taking the course of the dialogue into account without the need to deploy a user model as often deployed in related work of this kind. For instance, Walker and colleagues (2004) deploy weighting of user preferences in the restaurant domain by using the multi-attribute decision theory in combination with user models. They classify the domain in six categories [Whittaker et al., 2002] differentiating between quantitative attributes (food quality, cost, decor, service) and categorical attributes (food type, neighbourhood). While the categories are user-independent the weighting of the attributes is user-dependent and thus, user models are needed to perform a ranking of the categories and hence also the preferences. The ranking is used mainly for generation of adequate system output in order to present the most important information and at the same time not to overload the user but only present an amount of information that can easily be taken in by the user. An approach to prioritise user preferences without deploying a user model is presented by Carberry and colleagues (1999) within the domain of course selection at university. Based on the assumption that it is the obligation of a collaborative dialogue system to come up with the best possible solution, the system recognises preferences that are expressed by the user and also by deducting from patterns in the user’s reactions. A combination of conversational circumstances and semantics is considered to weigh the user preferences. Conversational circumstances regard the situation in which a proposition is uttered, e.g. if it is uttered deliberately in the initial description of the problem (more important) or if it is uttered as a response to a proposal of the system (less important). Semantic aspects include direct expression of the preference or obtained through deduction. A proposal history is further implemented to track all of the presented proposals and the users’ reaction towards them in order to find trends that emerge over time. Our approach is somewhat similar in the way that we also take the time into account and cope without user models. One way to find out about priorities is to simply ask the user about which constraint is more important. However, this method is out of the question for an intelligent system of our kind. Another common and straightforward way to prioritise user preferences in single-user systems utilises the semantics of a constraint-bearing utterance to determine the importance of the constraint to the user. This method is also part of the Carberry approach introduced above. The semantic content of an utterance is analysed for specific words that show a sign of importance, e.g. ’maybe’, ’definitely’, etc. The semantic analysis depends only on the constraints and the single utterances it appears in. In the multi-party case, however, each introduced constraint is further discussed in the course of the dialogue, i.e. rejected or accepted by the other dialogue partner. For instance, a suggestion is introduced with a ’maybe’ (obviously a

4.5 Problem Solving Using Discourse Motivated Constraint Prioritisation

109

weak constraint), this low priority is recognised by the dialogue partner, who can then accept the suggestion (Would this move the constraint from weak to moderate?), or, being aware of the low priority of the proposal, make his or her own counter-suggestion with higher importance. Naturally, conversation partners often have different and differing preferences which complicates the problem solving process immensely. It makes automatic semantic analysis very difficult. We claim that the situation of (at least) two users pursuing a common goal supersede the need for exhaustive semantic analysis. In our case, the conversation serves the purpose of finding a mutually liked object. The dialogue partners are generally not interested in a long discussion but rather in coming to a quick consensus. Besides uttering their own preferences and dislikes, the dialogue partners evaluate each other’s preferences against their own and react accordingly. The second DP analyses the proposal of the first DP and constructs the consequent own utterance according to her liking, e.g. underlining the own and preferred suggestion as the attempt of trying to convince the first dialogue participant that her proposal is the better choice8 . Thus, it can have different reasons why a dialogue participant speaks in a certain way and a rating of preferences over both interaction partners poses a difficult challenge. Instead, we suggest using the ongoing discourse, especially the order of occurrence of the constraints for prioritisation. It thereby does not matter who introduces which preferences as the dialogue aims at a mutual solution. The system does not model the preferences of each user independently but collects all information relevant for the task to form a set of common constraints for the database queries. If the query does not yield any results, an intelligent system is expected to provide an alternative solution. Our solution to this problem is presented in the following. The evaluation of the approach is presented in Section 5.2.2. We compare the actual reaction of the users to over-constrained situations in the recorded dialogues (during which the system only reports the OC situation without proposing a solution) to the way our system performs in the same situation. Further, we compare our approach to manual semantic analysis. The results are very promising. 4.5.1 Prioritisation Scheme For prioritisation, information is extracted out of each utterance according to three categories: Changing Categories, Current Preferences, and Prioritisation Values. • Changing Categories (CC) indicate the topic(s) of the current utterance. For instance, if one of the participants makes the statement of wanting to eat Italian food, the CC field is tagged with category F which stands 8

Clearly, however, it is not just that easy. Different factors influence the utterance design such as for instance social roles dialogue participants take on in the conversation (refer to Section 2.3.2).

110

4 Dialogue Management for a Multi-Party Spoken Dialogue System

for food or cuisine. The other distinguishable categories are location (L), ambiance (A), category (C), price range (P), specials (S) • Current Preferences (CP) lists all currently valid constraints represented by individuals of the respective category and is thus used for a database query. In the example above, ’Italian’ would be categorised belonging to category food (F) and is thus allocated the individual F1 (taken it is the first F-subject in this conversation). A second F-value later on in the dialogue, e.g. Mexican food, would then be tagged F2, etc. This is applied analogously to all other categories (L1, L2, P1 etc.). • Prioritisation Values (PV) manage the priority values of the individuals. Each individual is stored with its according priority value and the actual value it represents. With every recalculation (induced by a change in the CP section) all currently valid values rise by 1 Priority Point (PP). A new individual is introduced with the value ’1 PP’, i.e. it has risen 1 PP from the default value of 0 PP. Negative constraints or dislikes are represented with negative values accordingly (starting at ’-1 PP’). At the beginning of a dialogue, the table contains no entries. As soon as a relevant topic is raised, it is displayed in CC. The corresponding individual is inserted into CP and the PV value is assigned 1 PP (or -1 PP in case of negation). Every time the users modify the constraint set, e.g. by proposing or dismissing one, a change in the CP section occurs and the PV are recalculated: The values of all individuals that are currently represented as valid preferences (in CP) are raised by 1 PP (or lowered by 1 PP for negative values). Thus, the longer a subject stays valid, the higher its priority value becomes, which is obviously the desired effect. That means, as long as a subject is not explicitly abandoned or replaced by a different value due to incompatibility between constraints (and disjunctive values), it is considered valid and part of the current preferences. If a constraint is dismissed it is taken out of CP, its PV stays at the current value. Should it be re-introduced into the dialogue with the same polarity, it is reinserted into CP and the priority calculation starts at the former value. A change in the polarity of a valid constraint is performed by simply adding or removing a minus ’-’ to the PP value. All currently valid individuals are listed in CP which serves as the basis for the system’s database queries. Every change in the constraint set induces a database query so that the system is always up-to-date and ready to interact. Generally, the system interacts for the first time after the users have already come to an initial agreement. As also noted by [Carberry et al., 1999], this first request to the computer deserves special attention as it displays the users’ original preference. Thus, all valid individuals at the time of the first computer request receive a First Request Bonus (FRB) of an extra 1 PP. At present, the prioritisation only comes into play in the case of an overconstrained situation, i.e. if the database query does not yield any results. In order to offer the users a best possible alternative result the system has

4.5 Problem Solving Using Discourse Motivated Constraint Prioritisation

111

to decide which constraint(s) to relax. We deploy the (slightly simplified) Algorithm 3. while overconstrained OR $resultset == $previous resultset do if onto check($relaxcandidate).succeed then present($resultset); break; else if relax($relaxcandidate).succeed then present($resultset); break; end end $relaxcandidate++; end

Algorithm 3: Relaxation algorithm. After execution of the algorithm the obtained result set is further examined. If it is empty again the algorithm proceeds. It also continues in case the result set is not satisfactory. The result set is compared to the result set that was presented to the users in the system’s last turn before the initial OC situation. If the result sets are equal, the same result set as the one that obviously had just been rejected or further constrained by the users would be presented again. Thus, the relaxation algorithm proceeds at this point. The constraint with the lowest priority value is chosen as the initial relaxation candidate (RC). The procedure relaxcandidate++ assigns the next candidate for relaxation. If no result was obtained after the first relaxation, the relaxed constraint (the former RC) is reinserted before the next RC is considered for relaxation. After another unsatisfying result, both constraints are relaxed etc. The presented algorithm is simplified in this matter and also in the way that it assumes that each time there is exactly one constraint with minimal priority value which, however, is not always the case. The implemented algorithm handles this by trying out each of the potential RC and taking the one with the best results. In the process of trying to consider all preferences and avoiding relaxation, the system inspects the ontology for similar or related values of the RC (onto check in Algorithm 3). If for instance no restaurant can be found near the town hall, before relaxing this constraint, it is checked if the query would be successful if the area around the cathedral was considered. The query is thus changed in the way to extend the location by the adjacent neighbourhood around the cathedral. This kind of ontology check can be performed for all exclusive categories (L, F, P, C, and A). Observation of the recorded dialogues showed that there is a need to exclude certain constraints from the relaxation process. Values from the category S (e.g. ’cocktails’) or ’expensive’ of category P, as well as negative constraints

112

4 Dialogue Management for a Multi-Party Spoken Dialogue System

were very important to the users. Regardless at what point these values were introduced in the dialogue, they were repeatedly mentioned and never relaxed. Thus, a protection flag is applied to these constraints to not be considered for relaxation. 4.5.2 Example The prioritisation schem is applied to the dialogue snippet shown in Table 4.6. The dialogue proceeds in the following way: Utterance

CC CP

... U2 1: What do you prefer to eat? U1 2: Let’s go to a Chinese restaurant!

F

{F1,C1}

F1=1PP Chinese C1=3PP restaurant

P

{P1,F1,C1}

P1=1PP exclusive F1=2PP Chinese C1=4PP restaurant

F

{F2,P1,C1}

F2=1PP P1=2PP F1=2PP C1=5PP

U2 3: Oh yeah, Chinese is a good idea. S 4: Your query for a Chinese restaurant returned three hits. My suggestions are fast food restaurant Asia Wok, fast food restaurant Asia Wan and restaurant Panda. U1 5: Ok. But we want one that is a bit more exclusive.

PV

S 6:

I found no result for an exclusive Chinese restaurant. U1 7: OK, then we have to go somewhere else. U2 8: Should we go to an Italian restaurant?

Italian exclusive Chinese restaurant

U1 9: Ok, that’s fine. I love Pizza. ...

Table 4.6. Prioritisation scheme applied to an extract of a dialogue.

4.6 Summary

113

• Utterance 29 introduces ’Chinese’ food. Thus, CC is assigned F for cuisine. CP adds F1 to the list of current preferences. The change in CP induces a recalculation of PV. F1 is added at value 1 PP. Note that C1 is already there due to the fact that earlier in the conversation, a ’restaurant’ was requested. Before recalculation, C1 was at value 2 PP which means that it had received an extra point as FRB. Thus, utterance 4 is not the system’s first interaction. • In utterance 3, Chinese is repeated. No change of the current constraint set, thus no recalculation. • The database query resulted in a presentable result set of three restaurants and is therefore read out. • The query is further constrained introducing ’exclusive’ as P1 with VP 1 PP. The change in CP induces a recalculation. • S 6: OC. The database query did not yield any results. The system did not have any means to resolve the problem to present an alternative result. It simply notified the users about the OC. The users change the result set regarding accordingly. • In utterance 8, ’Italian’ is suggested and thus integrated into the according fields. Italian is disjunctive to Chinese, thus both constraints do not coexist and ’Chinese’ (C1 resp.) is taken out, i.e. it is the constraint the users relax. How would our algorithm have performed in this situation? The algorithm would have tried to relax ’exclusive’ at the first round. However, the new result set would be the exact same as the one presented before. The algorithm thus goes on and chooses ’Chinese’ as the next RC, ’exclusive’ is reinserted into the constraint set used for the query. The check in the ontology for related values yields a strong relation between Chinese and Thai or Japanese food. As there are exclusive Thai as well as Japanese restaurants in the database the system would thus be able to present a solution very close to the initial request. We claim that this solution might be even better (i.e. closer to what the users wanted originally) than what the users actually found proceeding in the dialogue when switching to Italian. Clearly, a statement of this kind can only be assumed using common sense, objective evaluation is hardly possible in fictitious scenarios as no real drive is leading the users’ actions and preferences. A more extensive evaluation is presented in Section 5.2.2.

4.6 Summary This section focused on the dialogue management for our proactive multiparty dialogue system. The prevalent ISU approach (e.g. [Larsson, 2002]) was adopted providing an ideal and flexible basis for an agent-like system as ours. Thus, in order to endow our systems with its designated functionalities of 9

U1 2 stands for DP U1 and the second utterance of this dialogue snippet.

114

4 Dialogue Management for a Multi-Party Spoken Dialogue System

proactive interaction behaviour, intelligent problem-solving and multi-party capability we built on top of the approach (and partly on the existent multiparty extensions). In Section 4.1 we introduced our modified information state, for which we partly adopted the proposed multi-party extensions by Kronlid (2008) (refer to Section 2.4.4) and partly performed additional alterations in order to suit our specific setup. We further introduced a new interaction principle to allow proactiveness in interaction protocols. Our example domain of restaurant selection was applied to dialogue management in Section 4.2. All task-related parts, such as task model, domain and context model, as well as update mechanism, were presented. The components were again adopted to the multi-party situation of our system. Section 4.3 concentrated on the different dialogue management strategies that enable proactive system interaction. Our optimistic grounding and integration strategies are presented as well as the system’s interaction strategy that identifies points in the dialogue that are suitable for proactive interaction. Finally, the extensive dialogue history is described that starts modelling as soon as the dialogue enters the specified domain and thus enables proactive interaction already for the system’s first utterance. We illustrated how the dialogue management performs in practise by listing an example sequence of information states. Finally, we described the constraint based problem solving functionality of the system in Section 4.5. In order to always provide the best possible solutions, we introduced a new discourse-motivated algorithm to prioritise user constraints in multi-party dialogues which allows user-friendly handling of over-constrained situations. The evaluation of this algorithm presented in Section 5.2.2 shows its great performance.

5 Evaluation

In this section, the evaluation of our dialogue system is presented. Established methods for evaluating spoken language dialogue systems differentiate between subjective and objective methods as described in Section 2.2. The aim of the evaluation of our dialogue system is to appraise the user acceptance and rating of this novel sort of interactive system. Thus, the main focus is put on subjective evaluation for which data was obtained through the questionnaires filled out by the participants prior and subsequent to the data recordings. Usability evaluation is performed using two established methods (AttrakDiff [Hassenzahl et al., 2003] and a modified version of SASSI [Hone and Graham, 2000]). Evaluation is performed over the different recording sessions to detect the improvement of the system as well as comparing the different setups with and without avatar using the data of the Session III dialogues. A technical self-assessment of the participants was further conducted in order to validate the comparison of the different recording sessions. The results of the evaluation are presented in Section 5.1. Section 5.2 concentrates on objective measures of the dialogues. First, a data analysis is presented. Further, the performance of our novel algorithm for discourse-oriented user constraint prioritisation is examined. For this, we deploy our system with the implemented prioritisation algorithm to the recorded dialogues that contain at least one over-constrained situation. The relaxation candidate that is proposed by the algorithm is then compared to the users’ actual behaviour during the recordings, i.e. which constraint the users relaxed. A second evaluation compares our algorithm to a manual semantic analysis. A further prominent point of the evaluation is the avatar we deploy as a form of personification of the system to provide an additional, i.e. visual, modality. The aim of this is to augment the presence of the system in the interaction in order to work towards the acceptance of the system as an equivalent interaction partner. We study the effect the avatar has on the users’ interaction behaviour and subjective ratings. Section 5.3 presents an analysis of the main user’s gaze direction. It is investigated in which ways the human-computer interaction differs when deploying an avatar as opposed to only voice output P.-M. Strauß and W. Minker, Proactive Spoken Dialogue Interaction in Multi-Party Environments, DOI 10.1007/978-1-4419-5992-8_5, © Springer Science + Business Media, LLC 2010

115

116

5 Evaluation

and in what ways the interaction differs compared to the human-human interaction with the other user. The subjective ratings are analysed in the scope of the usability evaluation presented in Section 5.1.4. Finally, the proactiveness of the system is assessed as presented in Section 5.4. Objective measures are used to obtain information about the system’s actual interaction behaviour. Further, user ratings provide subjective information about how the system’s interaction is perceived.

5.1 Usability Evaluation Usability evaluation is an important means to measure a dialogue system’s usefulness and user-friendliness (refer to Section 2.2). We deploy two different standardised questionnaires to measure the usability: AttrakDiff [Hassenzahl et al., 2003] and SASSI [Hone and Graham, 2000] (in a modified form, see below), and further apply a technical self-assessment to be able to relate the users’ technical affinity to their rating. The design of our questionnaire is described below. Usability evaluation is performed only of the participants who interact directly with the system (group of users U1). It cannot be determined in what way participants of group U2 can judge the interaction as, by definition, they do not interact directly with the system1 . The presented usability evaluation is performed in two ways. The aim of the first evaluation is to assess the improvement of the system between the different recording sessions (in between which the system was improved in terms of speed and functionality) and what impact it has on the participants’ rating Further, the system is evaluated in terms of its acceptance and appraisal of the participants. How does such a system that acts as an independent dialogue partner come across to the users? The third and final recording session which deploys a system simulated as closely as possible to the envisaged system is thoroughly evaluated. Half of the Session III dialogues include an avatar as visual output of the system, the other half use only speech output. This allows further an assessment of the effect the avatar has on the interaction. 5.1.1 Questionnaire Design The questionnaire in use for the recordings is composed of questions about demographic data of the participant, questions concerning technical self assessment and a subjective rating of the system (including AttrakDiff and SASSISV as described below). It closes with open questions about the recordings, i.e. what the participant liked, not liked and what could be improved. The questionnaire was extended for the third recording session to include additional subjective ratings regarding the system interaction. Appendix C 1

In reality, this is of course not always the case, however, it is not clearly definable which users interacted directly with the system and which not.

5.1 Usability Evaluation

117

displays SASSISV, the remaining SASSI items, and further items for subjective ratings as it was deployed during Session III. The different evaluation techniques used are presented in the following. AttrakDiff AttrakDiff [Hassenzahl et al., 2003]2 is a standardised evaluation method to assess attractiveness of any sort of product in terms of usability and appearance. It consists of 28 word pairs of opposite adjectives that are placed on both ends of a seven-item scale. The system is evaluated in terms of the following scales: • Pragmatic Quality (PQ) describes the human needs for security, control and confidence. Errorless and accurate functioning of the system and userfriendly design are measured in this aspect. Example: ”human - technical” • Hedonic Quality human needs for excitement and pride. It is differentiated between the following to aspects: – Identification (HQI) describes aspects if the users can express and identify themselves with the system in the desired way. Example: ”alienating - integrating” – Stimulation (HQS) is for instance obtained through visual design and novel features. It satisfies the user’s needs for excitement and discoveries. Example: ”dull - captivating” • Attractiveness (ATT) denotes how pleasant and likeable the system appears to the user. Example: ”motivating - discouraging” SASSI(SV) Hone and Graham introduced the 34-item questionnaire for Subjective Assessment of Speech System Interfaces (SASSI) [Hone and Graham, 2000] an approach for a multidimensional analysis especially for dialogue systems. Items such as ”I felt confident using the system.” are to be rated with a 7-point Likert scale. The higher the rating, the more positive it is (except for the Annoyance scale where the opposite holds). We perform evaluation with a modified, i.e. shortened, version of the questionnaire. The ’SASSI Short Version’ (SASSISV) contains 16 items specifying the same six scales as determined by the original SASSI with factors listed in the following. It has to be noted that the constructed SASSISV shows highly significant (p