Learner Corpus Research: New Perspectives and Applications 9781474272889, 9781474272919, 9781474272896

Citation preview

Learner Corpus Research

Corpus and Discourse Series editors: Wolfgang Teubert, University of Birmingham, and Michaela Mahlberg, University of Nottingham Editorial Board: Paul Baker (Lancaster), Frantisek Čermák (Prague), Susan Conrad (Portland), Dominique Maingueneau (Paris XII), Christian Mair (Freiburg), Alan Partington (Bologna), Elena Tognini-Bonelli (Siena and TWC), Ruth Wodak (Lancaster), Feng Zhiwei (Beijing). Corpus linguistics provides the methodology to extract meaning from texts. Taking as its starting point the fact that language is not a mirror of reality but lets us share what we know, believe and think about reality, it focuses on language as a social phenomenon, and makes visible the attitudes and beliefs expressed by the members of a discourse community. Consisting of both spoken and written language, discourse always has historical, social, functional, and regional dimensions. Discourse can be monolingual or multilingual, interconnected by translations. Discourse is where language and social studies meet. The Corpus and Discourse series consists of two strands. The first, Research in Corpus and Discourse, features innovative contributions to various aspects of corpus linguistics and a wide range of applications, from language technology via the teaching of a second language to a history of mentalities. The second strand, Studies in Corpus and Discourse, is comprised of key texts bridging the gap between social studies and linguistics. Although equally academically rigorous, this strand will be aimed at a wider audience of academics and postgraduate students working in both disciplines. Research in Corpus and Discourse Conversation in Context A Corpus-driven Approach

With a preface by Michael McCarthy Christoph Rühlemann Corpus-Based Approaches to English Language Teaching Edited by Mari Carmen Campoy, Begona Bellés-Fortuño and Ma Lluïsa Gea-Valor Corpus Linguistics and World Englishes An Analysis of Xhosa English Vivian de Klerk Evaluation and Stance in War News A Linguistic Analysis of American, British and Italian television news reporting of the 2003 Iraqi war Edited by Louann Haarman and Linda Lombardo Evaluation in Media Discourse Analysis of a Newspaper Corpus Monika Bednarek Historical Corpus Stylistics Media, Technology and Change Patrick Studer Idioms and Collocations Corpus-based Linguistic and Lexicographic Studies Edited by Christiane Fellbaum Investigating Adolescent Health Communication A Corpus Linguistics Approach Kevin Harvey Keywords in the Press: The New Labour Years The New Labour Years Lesley Jeffries and Brian Walker

Meaningful Texts The Extraction of Semantic Information from Monolingual and Multilingual Corpora Edited by Geoff Barnbrook, Pernilla Danielsson and Michaela Mahlberg Multimodality and Active Listenership A Corpus Approach Dawn Knight New Trends in Corpora and Language Learning Edited by Ana Frankenberg-Garcia, Lynne Flowerdew and Guy Aston Representation of the British Suffrage Movement Kat Gupta Rethinking Idiomaticity A Usage-based Approach Stefanie Wulff Sadness Expressions in English and Chinese Corpus Linguistic Contrastive Semantic Analysis Ruihua Zhang Working with Spanish Corpora Edited by Giovanni Parodi Studies in Corpus and Discourse Corpus Linguistics in Literary Analysis Jane Austen and Her Contemporaries Bettina Fischer-Starcke English Collocation Studies The OSTI Report John Sinclair, Susan Jones and Robert Daley

Edited by Ramesh Krishnamurthy With an introduction by Wolfgang Teubert Text, Discourse, and Corpora. Theory and Analysis Michael Hoey, Michaela Mahlberg, Michael Stubbs and Wolfgang Teubert With an introduction by John Sinclair Web As Corpus Theory and Practice Maristella Gatto

Learner Corpus Research New Perspectives and Applications Edited by Vaclav Brezina and Lynne Flowerdew

BLOOMSBURY ACADEMIC Bloomsbury Publishing Plc 50 Bedford Square, London, WC1B 3DP, UK 1385 Broadway, New York, NY 10018, USA BLOOMSBURY, BLOOMSBURY ACADEMIC and the Diana logo are trademarks of Bloomsbury Publishing Plc First published 2018 Paperback edition first published 2019 Copyright © Vaclav Brezina, Lynne Flowerdew and Contributors, 2018 Vaclav Brezina and Lynne Flowerdew have asserted their rights under the Copyright, Designs and Patents Act, 1988, to be identified as Editors of this work. Cover image: a concordance from the Trinity Lancaster Corpus All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage or retrieval system, without prior permission in writing from the publishers. Bloomsbury Publishing Plc does not have any control over, or responsibility for, any third-party websites referred to or in this book. All internet addresses given in this book were correct at the time of going to press. The author and publisher regret any inconvenience caused if addresses have changed or sites have ceased to exist, but can accept no responsibility for any such changes. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress. ISBN: HB: 978-1-4742-7288-9 PB: 978-1-3501-1258-2 ePDF: 978-1-4742-7289-6 ePub: 978-1-4742-7290-2 Series: Corpus and Discourse Typeset by Deanta Global Publishing Services, Chennai, India To find out more about our authors and books visit www.bloomsbury.com and sign up for our newsletters.

Table of Contents List of Contributors Preface Tony McEnery Introduction Vaclav Brezina and Lynne Flowerdew

x xiv 1

Part 1 Task and Learner Variables 1 2 3

The effect of task and topic on opportunity of use in learner corpora Andrew Caines and Paula Buttery

5

Phrasal verbs in spoken L2 English: The effect of L2 proficiency and L1 background Irene Marin Cervantes and Dana Gablasova

28

Investigating the effect of the study abroad variable on learner output: A pseudo-longitudinal study on spoken German learner English Sandra Götz and Joybrato Mukherjee

47

Part 2 Analysis of Learner Language 4

5 6 7

8

Disagreement in L2 spoken English: From learner corpus research to corpus-based teaching materials Dana Gablasova and Vaclav Brezina

69

Self-repetitions in learners’ spoken language: A corpus-based study Marek Molenda, Piotr Pęzik and John Osborne

90

Corpus-driven study of information systems project reports Ryan T. Miller and Silvia Pessoa

112

Beyond frequencies: Investigating the semantic and stylistic features of phrasal verbs in a three-year longitudinal study corpus by Chinese university students Meilin Chen

134

Figurative language in intermediate-level second language writing Justine Paris

155

Index of Names Index of Subjects

173 176

List of Contributors

Vaclav Brezina is a research fellow at the ESRC Centre for Corpus Approaches to Social Science (CASS), Lancaster University. His research interests are in the areas of applied linguistics, corpus design and methodology, and statistics. He is a co-author of the New General Service List (Applied Linguistics, 2015). He also designed a number of different tools for corpus analysis such as BNC64, Lancaster vocabulary tool and Lancaster statistical tool. He is involved as a co-investigator in the development of the Trinity Lancaster Corpus of spoken learner production. Paula Buttery is a senior lecturer in computational linguistics at the Computer Laboratory, University of Cambridge. She is a principal investigator for the Automated Language Teaching and Assessment Institute, and her main fields of research involve language technology for education, automatic processing of non-canonical language, and cognitive computational linguistic questions such as language complexity, learnability and adapting teaching systems to the individual. Relevant publications to date include ‘Linguistic adaptions for resolving ambiguity’ with Ted Briscoe (EVOLANG 2008), ‘Tracking cortical entrainment in neural activity: auditory processes in human temporal cortex’ with William Marslen-Wilson and colleagues (Frontiers in Computational Neuroscience, 2015) and ‘Adaptive communication: Languages with more nonnative speakers tend to have fewer word forms’ with Christian Bentz and colleagues (PLOS ONE, 2015). Andrew Caines is a research associate in the Department of Theoretical and Applied Linguistics, University of Cambridge. His research is funded by Cambridge English Language Assessment through the Automated Language Teaching and Assessment Institute and centres on the automatic processing and assessment of learners’ spoken language. Notable publications to date include ‘Normalising frequency counts to account for “opportunity of use” in learner corpora’ with Paula Buttery (In: Yukio Tono, Yuji Kawaguchi and Makoto

Contributors

xi

Minegishi (eds), Developmental and Crosslinguistic Perspectives in Learner Corpus Research, 2012), and ‘Spoken language corpora and pedagogic applications’ with Michael McCarthy and Anne O’Keeffe (In: Fiona Farr and Liam Murray (eds), The Routledge Handbook of Language Learning and Technology, 2016). Irene Marin Cervantes is currently a PhD researcher in the Department of Linguistics and English Language and a member of the ESRC Centre for Corpus Approaches to Social Science (CASS) at Lancaster University. Her main field of research is the application of corpus linguistics to the study of L2 spoken production, particularly multi-word units and their development in L2 English. Her research interests include vocabulary acquisition, phraseology and corpus-based learning and teaching. She also contributed to the development of the Trinity Lancaster Corpus. Meilin Chen is a research fellow in the Department of English at City University of Hong Kong. She is currently working as the manager and presenter on a government-funded project that aims to disseminate the data-driven approach among PhD students and university English language teachers in the context of writing for publication purposes. Her research interests include learner corpus research, computer-assisted English language learning and teaching, and English for academic purposes. Lynne Flowerdew is a visiting research fellow in the Department of Applied Linguistics and Communication, Birkbeck, University of London. Her main research interests include (applied) corpus linguistics, English for specific purposes and disciplinary postgraduate writing. She has published in these areas in international journals and edited collections and authored/co-edited several books. She has also served as a member of the editorial board of TESOL Quarterly, English for Specific Purposes, Journal of English for Academic Purposes and English Text Construction. Dana Gablasova is a research fellow at the ESRC Centre for Corpus Approaches to Social Science (CASS), Lancaster University. Her research interests include corpus-based studies of language learning and language use, formulaic language, vocabulary and written and spoken production in L1 and L2. Her research appeared in journals such as Applied Linguistics, The Modern Language Journal, Language Learning and English for Specific Purposes. She is the co-author of the New General Service List which was published in Applied Linguistics.

xii

Contributors

Sandra Götz is Senior Lecturer of English linguistics at Justus Liebig University Giessen. Her research areas include applied (learner) corpus linguistics, varieties of English in South Asia, first-, second- and foreign varieties of English, fluency, non-canonical syntactic structures and the teaching and learning of foreign languages with new media. She is currently serving on the board of the Learner Corpus Association as general secretary and is the book review editor of the International Journal of Learner Corpus Research. Tony McEnery is Distinguished Professor of English language and linguistics at Lancaster University and research director of ESRC. He is best known for his work on corpus linguistics. He is a former director of a UK-government-funded research centre called CASS which aims to encourage the uptake of the corpus approach to the study of languages across the social sciences. Ryan T. Miller is an assistant professor in the English department at Kent State University in the United States, where he teaches in the graduate and undergraduate programmes in ‘teaching English as a second language’. His research investigates the development of discipline-specific writing skills and genre knowledge, and cross-linguistic support of reading and reading sub-skills. His publications have appeared in the journals English for Specific Purposes, International Review of Applied Linguistics, Journal of Second Language Writing, Linguistics and Education and TESOL Journal, as well as a number of edited volumes. Marek Molenda is an asystent (assistant) (research-and-teaching fellow) at the Institute of English Studies, University of Łódź, Poland. His research interests include pedagogical applications of corpus linguistics, blended learning and learner-centred lexicography. He has published the results of corpus-based studies of EFL learners’ fluency and confluence. Joybrato Mukherjee is Full Professor of English linguistics at Justus Liebig University Giessen. His research interests are in corpus linguistics, English syntax (with a focus on lexicogrammar) and World Englishes (with a focus on South Asian varieties). He has published widely in all of these areas and has coordinated the compilation of various corpora, for example, the 18-millionword South Asian Varieties of English (SAVE) Corpus. He has served as chair of the Executive Board (president) of the International Computer Archive of Modern and Medieval English (ICAME) from 2011 to 2017.

Contributors

xiii

John Osborne is a professor of English language and linguistics at the University of Savoie, Chambéry, France. His research focuses on the use of learner corpus data in analysing fluency, complexity and propositional content in L2 oral production. As a spin-off from research and teaching interests, he has been involved in several European life-long learning projects (WebCEF, CEFcult, IEREST) aimed at developing resources for the assessment of oral proficiency and for intercultural education. Justine Paris is an associate professor in English linguistics at the University of Lorraine in Nancy, France. She is the ICT coordinator at the LANSAD department (Languages for Other Disciplines department). She specializes in second language acquisition and works on figurative language and metaphor development in L2 speech. She also works on learner autonomy, self-directed learning, as well as self-access learning systems and environments. Silvia Pessoa is an associate teaching professor of English at Carnegie Mellon University in Qatar where she teaches first-year composition and courses in sociolinguistics. Her research areas include second language writing, literacy development, writing in the disciplines and immigration studies. Her work has appeared in the Journal of Second Language Writing and Linguistics and Education.  Piotr Pęzik is an assistant professor of linguistics at the Institute of English Studies, University of Łódź. He has authored publications in the areas of corpus and computational linguistics, information extraction and information retrieval. His areas of interest also include corpus-based phraseology and the wider perspective of prefabrication and compositionality in language.

Preface Tony McEnery In early 1994, three of my colleagues, Steve Fligelstone, Gerry Knowles and Anne Wichmann, and I thought that we would hold a workshop on the use of corpora in the teaching of language and linguistics. We thought that it might be an interesting event, as the use of corpora in teaching in our department, at Lancaster University, had started in earnest. In 1992, when I was appointed to a lectureship in the department, I took over the syntax course and began to teach students to use treebanked corpora in sessions in our new computer laboratory. The idea was to bring various theories of syntax alive by giving students the opportunity to search treebanked corpora for evidence to support, or refute, claims they found in the theories they were being taught. The course was hugely ambitious – the students were trained to use corpus searching software (Longman Miniconcordancer), given an introduction to corpus linguistics and taken on a whirlwind tour of standard theory, extended standard theory, government and binding theory and lexical functional grammar – all in a nineweek course of one lecture and one seminar a week! It was a very demanding course, but no students dropped out and all seemed to really enjoy it. Around the same time, I started to publish work on the possible uses of corpora in translation, which included the training of translators. I also started to use corpora to provide the raw materials for computer aided learning programmes that would train students to undertake morphosyntactic analyses. The idea for doing this had been given to me indirectly by Gerry Knowles – he had also used annotated corpora to provide the examples for a computer aided learning programme, running on the BBC computer, which helped students learn how to undertake phonemic transcription. He had done that in the 1980s and I had used the programme myself as an undergraduate in the early 1980s – my first, unwitting, encounter with corpus linguistics! But while most of us planning the workshop were interested in the teaching of linguistics using corpora, Steve Fligelstone had presented an interesting paper at the ICAME conference in 1992 reflecting on the potential for using corpora in a range of teaching activities, including language teaching. So when we put together the call for the first Teaching and Language Corpora (TaLC) conference, we thought that we should

Preface

xv

include language teaching in the range of activities that people might want to come and talk about. We were quite sure of three things when we put out the call for papers for the first TaLC. First, it would be a small event. Our initial plan was simply to use our offices for the presentations as we could fit up to twelve people at a time in those; so, we thought, if we used Gerry’s office and my office for the event, we could run two parallel streams and comfortably accommodate an event of around twenty-four people. Second, we were quite sure that the focus would be on the teaching of linguistics using corpora. That is what we were primarily interested in and, though we had many good English language teaching colleagues in the department, none of them were overly interested in using corpora; so we thought that this was true generally of the English language teaching community. Finally, we also thought that we would probably know all of the people who would come along. In the early 1990s, the number of corpus linguists was relatively few and they all tended to know one another. So we were fairly confident that we knew our audience. We were decisively wrong about all of these assumptions. We rapidly received a large number of abstracts – around a hundred as I recall. We also received requests from people who did not merely want to give a paper, but who wanted to run a workshop so they could train other people to use software that they were using, or to focus on a topic which they wanted to have a detailed discussion on. Needless to say, we had to very rapidly book a large number of rooms and lecture theatres to run the event. With regard to topics, very few of the abstracts submitted related to the teaching of linguistics – it was clear that the teaching of language using corpora, and the study of the process of language teaching using corpora, was taking off and branching out. The scale of this is something that took us by surprise. The reason it did so was that it was largely being done by people we did not know. Researchers who were primarily focused on language teaching, who did not come to corpus linguistics conferences, had started to use corpora and to experiment with their use in the classroom. So, from our heroic failure to understand what the demand was, who the audience was and where developments were occurring, we stumbled, serendipitously, across the corrective for our ignorance in the response to our call for papers! The resulting conference was, I think, something of a landmark in corpus linguistics and language teaching. The use of corpora in language teaching is now so well established that it is not merely an obvious marriage, it is the touchstone of credibility for many published sets of language teaching materials and, indeed, language tests. Yet the conference also, in my mind at least, represented

xvi

Preface

the moment in my life where I saw corpus linguistics break out of its silo quite decisively. People we had never met before, and who were using corpora in their own way, were suddenly apparent and were enthusiastically embracing the use of corpora. Corpora were becoming mainstream. This brought opportunities and challenges. The opportunity was obvious and I have spent most of my career pursuing it – bringing the benefits of corpus linguistics to an ever wider set of researchers who have some research questions related to language where corpora can help. The challenges are largely about agency and control – in the first TaLC conference I saw people using corpora in ways no established corpus linguist had to do things which, at that point, were not thought to be in the realm of corpus linguistics. That was a challenge I welcomed, though I think such a challenge is always one which academics have to approach with generosity. It is all too easy to label those coming from outside a research community and doing things differently or focusing on questions not usually studied as ‘wrong’ or ‘not linguists’. A non-corpus linguist using corpora in a new way may easily be dealt with in this way. Yet I found it much more interesting to listen to what they were doing and to engage with their ideas and interests critically. In letting in ‘outsiders’ through events such as TaLC, corpus linguistics developed, broadening in terms of topic and scale. It became stronger. Certainly its relevance to real-world research was enhanced. That process of development continues, with similar drivers, to this day. So for me, TaLC was an entirely positive experience – indeed, it is not an exaggeration to say it was ‘eye-opening’. I never quite saw corpus linguistics in the same way again. It was therefore a pleasure to be able to host TaLC 2014 at Lancaster University. It was nice to see the conference ‘come home’ after twenty years. It was still in robust health. It had also taken on its own distinctive shape – the teaching of linguistics had gone and the focus was squarely on language teaching. Nonetheless, the conference programme still paid tribute to the vigour of the interaction of corpus linguistics and language teaching. I would encourage the curious reader of this book to examine this vigour first by reading this book – the papers presented here are testimony to the strength of the field. Yet I would encourage that reader to then go back to the book Teaching and Language Corpora (Wichmann, A., Fligelstone, S., McEnery, T. and Knowles, G.; Routledge, 1997) and scan through that. This represented the cream of the papers from the 1994 event, just as this book represents the best of the 2014 event. It is clear, in looking at the two books as a pair, that the field has moved on – corpora are larger, the research questions are more ambitious and the initial, exploratory phase of

Preface

xvii

work has given way to research programmes heavy with substantive research findings. That is a delight to see. In terms of topic, real changes are apparent too. The teaching of linguistics took up one whole section of the 1997 book; it is absent from this book. Computer aided language learning also has its own section in 1997, predictably perhaps given the experience of the organizers of TaLC 1994, but also the state of the field. Again, this has fallen by the wayside. But as happens in a rich and dynamic field of study, as topics fall away new topics emerge as the field defines itself. Perhaps the single most striking example of this is learner corpus research. Learner corpora are barely mentioned in the 1994 proceedings. Yet in this book the first five chapters use well-established learner corpora, while the remaining three analyse smaller datasets collected for the purposes of a single study, which, however, could be considered DIY learner corpora of a kind. This research area, provided with notable drive and energy in the 1990s by the strong team led by Sylviane Granger at the Université Catholique de Louvain, has moved centre stage in the interaction between language teaching and corpus linguistics. This is rightly so – learner corpora are clearly important in terms of thinking through a more sensitive approach to the impact that learner backgrounds can have on the process of language teaching. By taking these into account people can fine tune language teaching materials, curricula and tests. Learner corpora are of immense significance both in research and practical terms. Reading the book in advance of writing this preface I was struck again by how happy accidents occur in life. While I am quite sure that the interaction between teaching and language corpora would have become apparent eventually, I am very pleased to have been associated with the process of bringing that interaction to light over twenty years ago. While a small part of me would be very happy to say that I, and the others who worked with me, did so because we could see what others could not, the greater part of me is glad that that was not the case. I will certainly never pretend that it was. The truth is altogether more human – a happy accident led to a wonderful discovery. While not perhaps on the scale of Fleming’s discovery of penicillin, it is a useful reminder, nonetheless, that retaining the capacity to be surprised and to accept that while doing one thing you actually ended up doing something quite different is important in research.

Introduction

Corpora play a crucial role in second language (L2) research and pedagogy. In research, corpora offer a unique insight into the production of L2 speakers providing an opportunity to identify features typical of L2 use. For example, with corpus evidence we can address questions such as: Is the use of a word, phrase or a grammatical structure influenced by the first language (L1) background of the speaker? How does L2 develop across proficiency levels? How do learners of a language differ from native speakers? Corpus-based L2 research has also the potential to inform L2 pedagogy by offering recommendations for more efficient language learning and teaching; corpora can also serve as sources for teaching materials. The aim of this book is to bring new perspectives on learner corpus research (LCR) to the corpus linguistics community and highlight the applications of this research in language pedagogy thus bridging the gap between theory and classroom practice. It showcases studies reporting innovative research based on written and spoken learner corpora of different sizes, some well established, others compiled for the purposes of the studies included in this book. The book is divided into two main parts. Part I focuses on the investigation of variables which are related to L2 speakers (proficiency, L1 background and languagelearning experience) and the context of L2 production (topic and task). Part II explores different aspects of the language produced by L2 speakers such as semantics, pragmatics, stylistics and metaphor use. All the studies combined provide a unique insight into the complexity of L2 production and into the methodologies which can be used for its investigation. The common thread to the book is the corpus method, which provides a strong empirical grounding for both quantitative and qualitative explorations. The corpus method offers a principled way of addressing questions connected to L2 speech and writing. The answers are found by examining corpora, which provide systematic empirical evidence of L2 production – in the framework of LCR, all meaningful questions are empirical questions.

2

Learner Corpus Research

The book is intended for both researchers and practitioners – language teachers, material developers and curriculum planners. Each of the chapters makes an explicit link between a corpus-based study and practical pedagogical implications of the research; these implications are devoted a separate section in each chapter. Our hope as editors is that this book will contribute to the debate on how the insights into L2 production that corpora offer can help make the process of language learning and teaching more effective and more enjoyable.

Acknowledgements We wish to thank the authors of all the chapters for their contributions, collegiality and commitment to our book. All chapters were externally reviewed and we are greatly indebted to the external reviewers for their time and valuable, detailed comments on the submissions. We extend special thanks to Karin Aijmer, Alex Boulton, Marcus Callies, Maggie Charles, Peter Crosthwaite, Fiona Farr, Richard Forest, Ana Frankenberg-Garcia, Mélodie Garnier, Gaëtanelle Gilquin, Sandra Götz, Dilin Liu, Susan Nacey, Norbert Schmitt and Wang Lixun. Thanks also go to Agnieszka Leńko-Szymańska and Alex Boulton, editors of the previous TaLC book, for sharing their expertise with us. We would like to thank our editors at Bloomsbury, Gurdeep Mattu and Andrew Wardell, for their guidance and support throughout the whole editing process. We are very grateful to Michaela Mahlberg and Wolfgang Teubert for recommending our book for publication in their ‘Corpus and Discourse’ series with Bloomsbury. Vaclav Brezina Lynne Flowerdew

Part One

Task and Learner Variables

1

The Effect of Task and Topic on Opportunity of Use in Learner Corpora Andrew Caines and Paula Buttery

1 Introduction Only a little attention has been paid in the field of LCR to the effect of situational variables such as document length, task and topic, and yet their true effect needs to be fully understood before strong conclusions can be made about, for example, proficiency-level profiling, learner progress and so on. As Tummers and colleagues (Tummers, Speelman and Geeraerts 2014: 482) state, ‘The phenomenon of confounding variables … is hardly ever explicitly raised in (corpus) linguistics’, even though their importance is acknowledged in other disciplines. However, there is a growing interest in this area (e.g. Ädel 2015; Biber, Gray and Staples 2016; Gries 2003; Hinkel 2009; Khabbazbashi 2017; Kobayashi and Abe 2014), one we pursue here by investigating how various linguistic features are affected by task and by topic – the subject matter and structure of a text, from the high level such as ‘business’ or ‘society’, to the fine-grained (e.g. ‘write marketing strategy’, ‘what public transport is like in my hometown’) – concluding that this is a factor that needs to be fully understood and controlled for in LCR and, by extension, language teaching and assessment. Finding that task and topic are inextricably linked, we refer to both variables in a hybrid way from now on, as the ‘task-topic’ of a given document. In LCR the task and topic of any given text are usually dictated by a ‘prompt’ – a question or statement that prompts a response from the learner. This is typical of learner essay collections, such as the International Corpus of Learner English (ICLE) (Granger et al. 2009), the Longman Learners’ Corpus1 and the Cambridge Learner Corpus (CLC; Nicholls 2003). As a consequence of how corpora such as these tend to be collected in an ongoing process of accumulation, they are not typically balanced in terms of tasks and topics.

6

Learner Corpus Research

For instance, the CLC has been built up over many years from the whole suite of exams set by Cambridge English Language Assessment.2 Due to the exam pathways designed by Cambridge English, their business English certificates (BEC) do not begin until CEFR3 level B1 and go up to level C1, impacting the CLC in the sense that levels A1, A2 and C2 are devoid of business English content and are instead made up of general English essays. An initial topic distinction can be made therefore between ‘business English’ and ‘non-business English’. If topic is not controlled for even at this basic level, apparent linguistic progression from level to level may in fact be confounded by the different constitutions of CEFR level sub-corpora. It is understandable that the CLC has been collected in this way: its rapid and ongoing accumulation has been an invaluable resource for LCR. Nor is it the only such heterogeneous learner corpus. It is the purpose of this chapter to demonstrate that task and topic do affect language use, and that they are variables which need to be controlled for in LCR, along with other external variables such as document length. We refer to a concept we introduced in previous work, ‘opportunity of use’ – the opportunity the learner is afforded to use a linguistic feature, whether a lexical item, particular construction or discourse structure (Buttery and Caines 2012b). For instance, we showed that adverb use does not have a linear relationship with document length. That is, native speakers use disproportionately few adverbs in shorter documents compared to longer documents, and since lower-proficiency learners tend to write shorter documents than higher proficiency learners, they do not have the same opportunity to demonstrate use of adverbs. This needs to be taken into consideration when comparing learners for research or assessment purposes. Hence opportunity of use should be controlled for a fair comparison between proficiency levels, between native speakers and learners, or among individual learners. In this chapter, we discuss the following four areas: (i) a task-topic taxonomy for learner essays, (ii) lexico-syntactic usage differences across tasktopic types, (iii) unsupervised identification of task-topic-type clusters in the scenario where the prompts are unknown to the researcher and (iv) implications of task-topic effect for researchers, assessors and teachers.

2 Previous work Previous relevant work in LCR has mainly focused on the effect of task on language features, with only incidental mention of topic. A representative example of this would be Newton and Kennedy (1996) who report on a 29,000-

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

7

word corpus constructed from four adult learners of English, at what is described as ‘pre-university’ proficiency. The learners were presented with two tasks. The first of these was a spatial shared information map-task regarding the optimal layout of a zoo: each learner was provided with a partial layout and then asked to complete a map of the entire zoo via spoken communication only. The second task involved sharing non-spatial domain-specific information about a medical dilemma. Here learners were required to reach consensus regarding the order in which patients should receive treatment: each learner was again provided with only partial information and via conversation expected to construct a priority list for patients awaiting surgery. The study focused on the effects of the different task types (spatial vs. nonspatial) on the language that was used. For instance, more subordinating conjunctions were used in the non-spatial information task because of the need to argue a case with relationships of cause and effect, condition, result, purpose. The spatial zoo task elicited a greater number of prepositional phrases than the non-spatial medical task because of the need to verify location to create the map. Another area of relevant work is probabilistic topic modelling, which is used for discovering topics in a collection of texts based on lexical features. This tends to proceed via modelling techniques such as ‘latent Dirichlet allocation’ in which a set of documents are considered to be distributed over a fixed vocabulary (Blei 2012). In this approach, identifiable topics emerge through interpretation of the ‘topic’ wordlists generated by the model. It is up to the researcher to generalize over the wordlists and assign each one a label. For example, the most frequently occurring topic in a corpus of 17,000 Science articles features the terms such as ‘human, genome, dna, genetic, genes’ – and we might therefore label this topic ‘genetics’; the second most frequent topic contains words such as ‘evolutionary, species, organisms, life, origin’ – we can label this topic ‘evolution’; and so on. Topic wordlists are not always so easy to label, but the fact that many are as coherent as shown in these examples is a natural consequence of ‘the statistical structure of observed language and how it interacts with the specific probabilistic assumptions of [latent Dirichlet allocation]’ (Blei 2012: 84). But note that in itself, this observation reflects the fact that lexical choice, at least, is statistically skewed by topic, else the linear discriminant analysis (LDA) technique would not work. Initially we present three case studies (sections 4–6) in which we opt not to employ a bottom-up topic modelling approach in this work. Our interest lies not so much in the discovery of topics and their frequency, but rather in the effect of predefined topic classes on linguistic features, thereby adopting a top-down view

8

Learner Corpus Research

of topic and preparing a corpus of labelled texts of even frequency. These studies are based on the assumption that the prompts linked to the essays in a corpus are known or can be obtained. However, there is an alternative scenario in which the prompts are unknown, and so we do discuss how clustering might be used to group prompts in a corpus in a machine learning procedure, demonstrating that essays are readily clustered into something like the task-topic groupings we define.

3 Cambridge Learner Corpus We use the CLC for this study, a collection of essays4 written by students from around the world sitting Cambridge English exams. We were provided with the prompts for a subset of exams taken in 2009.5 Using these, we could identify the expected task-topic for a selection of essays written that year. Having inspected the prompts for the essays in the corpus we settled upon a set of six labels to evenly represent the range of tasks and topics presented by the prompts that year: ‘administrative’, ‘autobiographical’, ‘narrative’, ‘professional’, ‘society’, ‘transactional’. We experimented with higher and lower level label sets, but settled upon this set on the grounds that they formed quite large groups while being at a sufficiently granular level in descriptive terms. For example, we labelled the following prompts ‘autobiographical’ (1), ‘narrative’ (2) and ‘society’ (3): (1) This is part of a letter you receive from your new penfriend, Jenna. I’ve got one close friend who I spend a lot of time with. What about you? Tell me about your friends. How important are they to you? ● ●

Now write a letter to Jenna about your friends. Write your letter in about 100 words on your answer sheet.

(2) Your English teacher has asked you to write a story. Your story must begin with this sentence: ●

It was getting dark and I was completely lost. ●

Write your story in about 100 words on your answer sheet.

(3) Your class is doing a project about education. Your teacher asks you to write about education in a country you know. Write about: what you think is good and bad about education in that country how you think education in that country will change in the future. Write about 100 words. ● ● ●

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

9

From our sample of 84 prompts from the year 2009, we were able to match and assign task-topic labels to 6,953 essays written by 4,784 individual students, giving us a sub-corpus we hereafter refer to as ‘CLC_2009’. B1 is the largest proficiency subset of CLC_2009 and so we limit ourselves to essays at this level only, to control the effect of proficiency as far as practical (a CEFR level still represents a wide range of proficiency), and moreover to constrain document length – a variable we have previously recommended should be controlled (Buttery and Caines 2012b). Since in this project the document is the target entity in question, we needed a corpus with a balanced number of essays for each of our task-topic labels. We considered balancing task-topic sub-corpora by number of tokens, but decided this would have the unsatisfactory consequence of (i) introducing incomplete documents where the requisite token count had been reached before a document break, and (ii) introducing an unwanted label bias in the training data, whereby there would be more occurrences of some labels than others (i.e. the task-topic types with shorter documents would contain more instances than the types with

Table 1.1 Word counts in CLC_2009_B1_balanced for each of six task-topic labels Task-topic

Examples

Essays

Mean document Words length (SD)

administrative

•write to all staff about office equipment

68

2986

43.9 (9.7)

68

7450

110.0 (22.7)

autobiographical •write about your friends •write about last weekend narrative

•write a story which begins with the following sentence

68

8591

126.3 (33.4)

professional

•write a reply to a conference invitation

68

5827

85.7 (21.4)

•write a covering letter for a job application society

•write about the education system in your country

68

9720

142.9 (42.0)

transactional

•invite a friend to a picnic

68

3651

53.7 (17.9)

408

38,225

93.7 9 (45.0)

•write to your teacher to explain your absence from class Total

10

Learner Corpus Research

longer documents). Thus, we opted for a per-document or per-essay perspective in our analysis tasks. We back off to the smallest class in our task-topic taxonomy – namely, ‘society’, for which we have 68 essays. We thus gather 68 essays from each of our six labels to produce a 408-essay (38,225 tokens) section of CLC_2009_B1 (henceforth ‘CLC_2009_B1_balanced’) for all experiments described below. In the case of ‘society’, our 68-essay sub-corpus will include all available essays, whereas in the case of the ‘administrative’, ‘narrative’ and ‘professional’ classes it will include about half of the available essays for each, and in the case of ‘autobiographical’ and ‘transactional’ we sample approximately 1-in-20 essays. The structure of CLC_2009_B1_balanced is given in Table 1.1.

4 Case study 1: Classification based on lexical features We first ask whether task-topic has a noticeable effect on ‘lexical features’ – that is, vocabulary. To address this question, we turned to a machine learning procedure to test whether essays could be correctly assigned their label based only on their lexical contents. We prepared a training set of 90 per cent of the essays in CLC_2009_B1_balanced, each one as an unordered ‘bag of words’ along with its task-topic label, in order to train a naive Bayes classifier.6 The idea is that each word in the bag becomes associated with that label, and the classifier, if presented with a sufficient quantity and distinctiveness of training data, can then use the presence of those features (words) in unlabelled essays to hypothesize which of the set of labels to apply to them. The remaining 10 per cent of the essays from CLC_2009_ B1_balanced were presented as unlabelled bags of words to the classifier, and its accuracy in identifying the labels in this test-set is then calculated by comparison with the true labels which had been held back. This procedure was repeated a further nine times, shifting the windows of the 90 per cent training set and the 10 per cent test-set across CLC_2009_B1_balanced, and taking average accuracy values from all ten iterations – a method known as ‘tenfold cross-validation’. In preparing the essays for classification, not only did we follow the standard practice of removing common function words (e.g. articles, prepositions, pronouns, wh-words, auxiliary verbs), or ‘stopwords’, from the essays, but we also omitted any lexical items given in the prompt itself. Thus, the classifier is trained purely on spontaneously composed lexical features rather than a restricted set of imitated lexical items copied from the given exam materials, and we were not simply identifying the labels of essays according to the inevitable repetition of certain keywords from the prompts.

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

11

The classifier was found to have a mean accuracy of 88.8 per cent (versus a chance baseline of 1 in 6, or 16.7%) averaged across our ten iterations.7 Mean precision and recall are given in Table 1.2, with ‘precision’ for each label l being the proportion of correctly identified documents out of the total number of documents the classifier hypothesizes to have label l, and ‘recall’ being the number of correctly identified documents with label l out of the actual number of documents with label l. As shown in Table 1.2, the balance of precision and recall scores vary by task-topic label. For instance, precision is just 76.8 per cent for ‘autobiographical’ whereas recall is 94.6 per cent. The reverse is true of ‘transactional’, with 96.4 per cent precision and 55.7 per cent recall. The ‘F-measure’, the harmonic mean of precision and recall, is found to be high (greater than 9 in 10) for all labels except ‘autobiographical’ and ‘transactional’. A further aspect of this exercise is that we can query the trained classifiers for the features deemed most informative – that is, words which distinguish strongly between task-topic labels. We obtained the twenty most informative features from each training iteration and list those that occur in at least two of these lists in Table 1.3. Each set of word tokens makes sense when compared with the kinds of prompts associated with each label (Table 1.1). Notably, the number of highly informative words for ‘transactional’ is small – only ‘see’, in fact – symptomatic of the poor F-measure returned for this label in the classification exercise, though its high precision hints at a latent set of mildly informative features (Table 1.2). Evidently, the classifier is over-hypothesizing essays to be ‘autobiographical’, presumably because its most discriminating lexical features occur frequently in other task-topics too. The ‘transactional’ label is underused, however, indicating that its lexical features are relatively indistinguishable from those of other labels. We infer that the ‘transactional’ task-topic is relatively bland in terms of encouraging distinctive lexical choice, a hypothesis confirmed Table 1.2 Mean precision, recall, and F-measure for a naive Bayes classifier trained on CLC_2009_B1_balanced for each of six task-topic labels Task-topic

Precision

Recall

F-measure

administrative

0.977

0.846

0.912

autobiographical

0.768

0.946

0.845

narrative

0.919

0.962

0.944

professional

0.905

0.992

0.948

society

0.869

0.989

0.928

transactional

0.964

0.557

0.694

12

Learner Corpus Research Table 1.3 Word tokens which are highly informative in at least two iterations of the tenfold topic classification exercise Task-topic label

Highly discriminative word tokens

Administrative

Budget, department, computers, soon, old, please

Autobiographical

Cinema, play, time

Narrative

Happy, house, love, person

Professional

Opportunity, talk, company, work, staff

Society

School, learn, teachers, years

Transactional

See

by training a binary classifier on just the ‘administrative’ and ‘transactional’ essays in CLC_2009_B1_balanced and finding that the most informative features for the latter are common verbs such as ‘come’, ‘go’ and ‘think’ – so it is little wonder that these features would be relatively indeterminate against five competitor labels. The other four labels are identified more accurately, especially ‘narrative’ and ‘professional’, in both cases indicative of more restricted lexical domains – because in the former the story is defined by an opening line given in the prompt (even though keywords from the line itself are excluded as stopwords), and in the latter there is a need to discuss work matters. This gives rise to informative features such as ‘opportunity’ and ‘staff ’ in the case of the ‘professional’ type, while the opening line for most of the ‘narrative’ essays in CLC_2009_B1_ balanced initiated a story about ‘the best day of Lisa’s life’, hence ‘happy’ and ‘love’ being discriminative features for this group. In summary, this case study has underlined that not all task-topic types afford equal opportunity of use for learners to demonstrate lexical knowledge in English. That is, the selection of prompt topic is found to have a somewhat deterministic effect on learners’ vocabulary use in various ways, in such a way that we can train a classifier to correctly identify the topic of nine in ten essays based on word features alone. The implication is that the choice of prompt in an exam affords different opportunity to demonstrate knowledge of different vocabulary sets related to the topic. Thus, if the prompt topic is more abstract than concrete, for example, then the learner can be expected to employ more abstract than concrete vocabulary. It remains a matter for future work whether such differences affect assessment of learners – whether a highly abstract vocabulary set correlates with higher grades than a more concrete set.

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

13

5 Case study 2: Word-class frequency At a level away from direct lexical features, one can generalize by examining linguistic use in terms of part-of-speech tags. Such an approach is commonly used in LCR, often as a foundation for analyses of larger syntactic patterns (e.g. Hawkins and Buttery 2010; Hawkins and Filipović 2012). We investigate whether task-topic has an effect on part-of-speech (PoS) frequencies: does tasktopic entail similar distributions of PoS tags? Since they are foundational for much work in LCR, this is an important question to answer, especially having found that document length has a nonlinear effect on adverb use, for example (Buttery and Caines 2012b). We used the RASP System (‘Robust Accurate Statistical Parsing’; Briscoe, Carroll and Watson 2007) to process CLC_2009_B1_balanced, our dataset containing sixty-eight essays for each of our six labels. We gathered PoS tag frequencies for each of four major classes – nouns, adjectives, verbs, adverbs – from each essay. The distribution of these frequencies is presented as density plots in Figure 1.1.8 Density plots serve a similar purpose to histograms, in that they portray distributions, but ‘smooth’ the distribution rather than ‘bin’ it (as in histograms) and therefore offer a way to visualize the underlying distribution. We use the normal (Gaussian) distribution as the ‘kernel’ – the weighting function – for our density plots, and the bandwidth smooths the values by the standard deviation of the kernel (Silverman 1998). Note that the area under each curve sums to one, and thus gives a proportional representation of the underlying PoS distributions. It is apparent from Figure 1.1 that the density distributions of noun, adjective and verb frequencies are broadly similar across the six task-topic labels. There are some differences in height (e.g. nouns, adjectives) and position (e.g. verbs) of the ‘peaks’. Adverb distributions are noticeably more varied, though it should be observed that these are the least frequent word-class represented in Figure 1.1 (see the x-axis) and we might speculate that the differences would in fact smooth out with more data. In any case, we can measure the apparent differences statistically, using the two-sample Kolmogorov-Smirnov test9 (K-S) to determine how the distributions compare in a pairwise fashion: that is, taking the PoS frequencies from each set of essays. The results of these K-S tests, known as the ‘D statistic’, indicate how strongly the two samples differ and are presented in Table 1.6 (nouns and adjectives) and Table 1.7 (verbs and adverbs), with p values smaller than 0.001 being marked with an asterisk.

14

Learner Corpus Research

Figure 1.1 Noun, adjective, verb and adverb frequencies by task-topic in CLC_2009_ B1_balanced.

As shown in Table 1.4, the majority of noun and adjective comparisons are not significantly different and therefore we assume the frequencies are drawn from the same distribution in these cases. For those few tests which do produce significant differences, we note that most involve ‘society’ documents. Table 1.5, however, confirms the visual impression from Figure 1.1, namely, that the verb and adverb distributions are more heterogeneous than the noun and adjective ones. More than half the tests are significantly different, with the ‘transactional’ group the main source of disparity for verbs, and ‘administrative’, ‘autobiographical’ plus ‘narrative’ topics showing significant differences for four of five comparisons each.

Administrative

Transactional Society

Professional

Narrative

Autobiographical Administrative

0.221

0.103

0.235

0.206

–

0.368*

Autobiographical

0.118

0.544*

0.265

0.118

–

0.324

Narrative

0.176

0.515*

0.235

–

0.206

0.338*

Professional

0.279

0.353*

–

0.088

0.206

0.353*

Society

0.515*

–

0.426*

0.397*

0.382*

0.176

Transactional

–

0.279

0.250

0.221

0.176

0.294

Table 1.5 Kolmogorov-Smirnov pairwise task-topic label comparisons of verb (upper left, white background) and adverb (lower right, grey background) frequency distributions in CLC_2009_B1_balanced (D = x, with * indicating p < 0.001) Society

Professional

Narrative

Autobiographical Administrative

Administrative

0.162

0.485*

0.265

0.324

0.485*

–

Autobiographical

0.574*

0.074

0.250

0.206

–

0.647*

Narrative

0.441*

0.176

0.118

–

0.103

0.603*

Professional

0.426*

0.235

–

0.412*

0.471*

0.279

Society

0.574*

–

0.250

0.397*

0.456*

0.515*

Transactional

–

0.132

0.221

0.338*

0.412*

0.441*

15

Transactional

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

Table 1.4 Kolmogorov-Smirnov pairwise task-topic label comparisons of noun (upper left, white background) and adjective (lower right, grey background) frequency distributions in CLC_2009_B1_balanced (D = x, with * indicating p < 0.001)

16

Learner Corpus Research

Considered together with Figure 1.1, these results indicate that ‘society’ essays involve greater use of nouns and adjectives, ‘transactional’ essays involve heavy use of verbs, while adverbs are used with high frequency in ‘autobiographical’ and ‘narrative’ essays and with low frequency in ‘administrative’ essays. The main outcome of this case study is that the tasktopic of an essay, set by the prompt, entails different frequencies of word use. The implication is that for much work in LCR, which as a field tends to treat proficiency sub-corpora holistically, there is a risk of confounding any findings with the effects of task-topic. An increase in x, where x could be the frequency of a noun, for example, could in fact be a consequence of differing proportions of discursive exercises at proficiency level Y compared to proficiency level Z. For example, Cambridge English business certificates are only intended to cover CEFR levels B1, B2 and C1, and thus the CLC A1, A2 and C2 subsections do not include business topic texts. Indeed, we find that between 10 and 20 per cent of the essays in CLC B1, B2 and C1 come from business exams: in CLC_2009_B1 at least, all such essays are exclusively ‘administrative’ or ‘professional’ essays, and if that is true of B2 and C1 business texts also then these three levels each receive a distinctive injection of rather homogeneous data. Such differences in the constitution of proficiency sub-corpora might lead researchers to mis-identify linguistic feature correlates, such as the ones described here, as entirely proficiency-driven, rather than partly or wholly based on task-topic.

6 Case study 3: Subcategorization frames Our third and final investigation of task-topic effect involves ‘subcategorization frames’ (SCFs), a set of 163 frames which describe verbs and their arguments. These range from the single-argument intransitive frame (4), to the twoargument transitive (5), the ditransitive with three arguments (6) and then more complex constructions involving extraposition, clausal complements and so on10 (Briscoe and Carroll 1997; Buttery and Caines 2012a). (4) Stephen surfs. SCF 22: INTRANS (5) Vic bought a juicer. SCF 24: NP (6) Lindsay put Harvey on the floor. SCF 49: NP-PP

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

17

Note that verbs may be associated with more than one SCF depending on argumentation. For instance, ‘surf ’ can also be transitive (7), ‘buy’ can be used as a ditransitive (8) and ‘put’ may take a phrasal particle (9): (7) Stephen surfs the internet. SCF 24: NP (8) Vic bought me a juicer. SCF 37: NP-NP (9) Lindsay put up with his foibles. SCF 76: PART-NP / NP-PART

We extracted SCFs from each of the 408 essays in CLC_2009_B1_balanced, automatically identifying argumentation patterns on the basis of RASP System output. Where there is syntactic ambiguity as to which frame is in use, the possible candidates are concatenated with commas. For example, ‘49,56’ would indicate a syntactic ambiguity between frame 49 (10) and frame 56 (11). (10) He posted a sign to the wall. SCF 49: NP-PP (11) He posted a letter to her. SCF 56: NP-TO-NP

As can be seen in (10) and (11), semantic information actually disambiguates between the two possible syntactic analyses, but since this kind of information is not available to the parser, a frame of concatenated SCF options is posited instead. Through this analysis of SCFs we aim to gain a fuller insight into the variance in constructional use across task-topics. The issue of task-topic has mainly focused on lexical features thus far, and so by moving to the constructional level we test whether this variable affects syntactic use as well as lexical selection. We identified 66 unique SCFs in CLC_2009_B1_balanced, of which 18 are concatenated multiple frames of the type ‘SCF, SCF, (SCF),…’. We obtain frequency counts for these 66 SCFs for each of the 408 essays in CLC_2009_ B1_balanced. This 26,928-cell matrix (408 row, 66 column) may be reduced to a low-dimensional space through LDA using R and the MASS package (R Core Team 2015; Venables and Ripley 2002). In LDA, the vertical dimensions of our data table, the 66 SCFs, are transformed into new axes which combine these variables so that between-group differences are maximized while within-group differences are minimized. Each new dimension ‘explains’ a certain amount of variance found in the data, a process which conventionally allows reduction of the original multiple dimensions to just a few which best account for the dataset (Kuhn and Johnson 2013). In our case, we first establish through ‘principal components analysis’, an unsupervised clustering method, that five dimensions account for approximately

18

Learner Corpus Research

75 per cent of the variance in the data. Our LDA model, also reduced to five dimensions, correctly classifies the task-topic label for 65 per cent of our 408 essays, a highly significant result according to a binomial test (p < 0.001). Figure 1.2 shows a density plot of the 408 essays from CLC_2009_B1_balanced in dimensions 1 and 2 of the LDA, grouped by topic. What is apparent is the relatively high variance in terms of SCF use within the ‘professional’ set of documents, the relatively low variance within ‘administrative’ essays, and the overlap of ‘autobiographical’ and ‘society’ essays on the one hand, versus the four remaining, more distributed labels. The accuracy of the LDA and the clusters apparent in Figure 1.2 indicate that constructional use is somewhat affected by task-topic. To further establish the nature of this effect, we plot a heatmap of frequencies for the twenty most frequent SCFs in CLC_2009_B1_balanced (Figure 1.3). In Figure 1.3 we can see that the ‘autobiographical’, ‘narrative’ and ‘society’ essays contain a greater range of the most frequent SCF types, whereas ‘administrative’, ‘professional’ and ‘transactional’ are more limited in this regard, including some zero (white) values. Moreover, the latter set are relatively low

Figure 1.2 Linear discriminant analysis of subcategorization frames by task-topic in CLC_2009_B1_balanced.

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

19

frequency for SCF 87,96 – prepositional arguments of some kind – while maintaining a high frequency for SCF 18,142 (bare infinitive arguments) and SCF 112,111,110 (to-infinitives), ‘professional’ in particular. In Table 1.6 we present the ten most frequent SCFs by task-topic. Again, we see some notable differences in ranking, with the prepositional SCF 87,96 of lower frequency in ‘professional’ and ‘transactional’, and not even among the ten most frequent SCFs for ‘administrative’. Meanwhile, subordinate that-clauses (SCF 104,109) are ranked more highly for ‘narrative’ and ‘administrative’ than other topics, while ‘society’ uniquely has adjectival arguments (SCF 1,2) in its top ten.

Figure 1.3 Heatmap of frequencies by task-topic for the twenty most frequent subcategorization frames in CLC_2009_B1_balanced, with darker shades indicating higher frequency.

20

Table 1.6 The ten most frequent subcategorization frames for each task-topic sub-corpus in CLC_2009_B1_balanced Administrative

Autobiographical

Narrative

Professional

Society

Transactional

1

24,51: NP(+RS)

24,51: NP(+RS)

87,96: PP

18,142: INF/SC

24,51: NP(+RS)

24,51: NP(+RS)

2

18,142: INF/SC

87,96: PP

24,51: NP(+RS)

24,51: NP(+RS)

22: INTRANS

18,142: INF/SC

3

110,111,112: TO-INF

22: INTRANS

22: INTRANS

110,111,112:TO-INF 87,96: PP

22: INTRANS

4

22: INTRANS

18,142: INF/SC

18,142: INF/SC

22: INTRANS

18,142: INF/SC

110,111,112: TO-INF

5

37,38: NP-NP

49,50,118: NP-PP

110,111,112:TO-INF 49,50,118: NP-PP

110,111,112: TO-INF

87,96: PP

6

32,33: NP-INF

110,111,112: TO-INF

104,109: THAT-S

87,96: PP

49,50,118: NP-PP

37,38: NP-NP

7

23: INTRANSRECIP

37,38: NP-NP

23: INTRANSRECIP

31: NP-FOR-NP

23: INTRANSRECIP

32,33: NP-INF

8

24: NP

23: INTRANSRECIP

95: PP-PP

37,38: NP-NP

32,33: NP-INF

49,50,118: NP-PP

9

104,109: THAT-S

19,20,21: ING

37,38: NP-NP

32,33: NP-INF

37,38: NP-NP

95: PP-PP

10

49,50,118: NP-PP

32,33: NP-INF

19,20,21: ING

24: NP

1,2: ADJP

53,54,55: NP-TOINF

Learner Corpus Research

Rank

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

21

This sketch of SCF frequency differences across task-topic types is presented as evidence for this variable’s constructional effects, which needs to be kept in mind when comparing learners within and between proficiency levels.

7 Differentiation at the prompt level We recognize that the lexical content of the prompts associated with the essays in large learner corpora may not always be available. Thus we attempt to model linguistic use at a per-prompt level, assuming that researchers can at least group the essays by a prompt identifier. As with CLC_2009_B1_balanced, we include only those prompts answered by at least 68 essays. This gives us a new corpus of 612 essays responding to nine prompts, which we again process with the RASP System, and extract PoS frequency counts for nouns, adjectives, verbs and

Figure 1.4 Dendrogram of prompt sub-corpora from CLC_2009_B1_balanced.

22

Learner Corpus Research

adverbs. We then normalize the relative frequencies and type:token ratios and apply a hierarchical clustering algorithm using the core stats package in R (R Core Team 2015). The results of the clustering are presented as a dendrogram in Figure 1.4 (de Vries and Ripley 2015). First, the bottom leaves of Figure 1.4 rather support our topic label taxonomy, in that the ‘autobiographical’ prompt subsets are grouped together, two of the three ‘transactional’ subsets are grouped together, while the singleton ‘administrative’, ‘narrative’ and ‘society’ prompts branch off individually. In the common scenario where the prompts themselves are not available to researchers, but the essays in a learner corpus may still be grouped by a prompt identifier, one could use this method to group the prompts into pseudo-topics whose identity is initially unknown but could be discovered via the classification or LDA techniques employed above. For example, the second branching would give four groups, which we have here known as ‘transactional’, ‘administrative’ + ‘professional’ + ‘transactional’, ‘society’, ‘narrative’ + ‘autobiographical’, but having been rearranged in this bottom-up fashion could be relabelled appropriately.

8 Implications for research, assessment and pedagogy We coined the term ‘opportunity of use’ and emphasized that variables such as document length need to be taken into account when comparing language use across proficiency levels. The point is that learners are not afforded the same opportunity to use adverbs across the learning and examination pathway – there is a bias in higher proficiency-level tasks that allows them more opportunity to demonstrate this know-how, and thus document length introduces a confound to any LCR that treats the corpus as a homogeneous set of documents. Here we consider document task-topic as another variable that needs to be controlled for, in continuing work on opportunity of use. In terms of how this work relates to LCR more generally, we advise researchers, if they do not already, to control for variables such as document length, task, topic and first language as far as possible. We recognize this is not always feasible, given limitations on obtainable data and metadata, and therefore these factors need to be acknowledged and understood where they cannot be fully controlled. With regard to document task-topic we advise the following precautionary steps:

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

23

1: Are the prompts available for the essays in your corpus? Yes. Then review the prompts and design a set of task-topic labels to reflect their diversity; if the distinctiveness of the label set can be confirmed through language modelling and/or statistical tests, all the better. No. Adopt an unsupervised machine learning approach to group the essays into sub-corpora, since document ‘labels’ (i.e. prompts) are unknown. For example, one might employ hierarchical clustering where at least question numbers for the essays are known (as in our corpus), or k-means clustering if no such information is available – where k is set a priori as the number of desired document groupings. 2: Make comparisons across proficiency-level sub-corpora restricted to the same or similar task-topics, using methods such as those described here.

We acknowledge that in reality there are often limitations to what information is readily available, and that despite best efforts the effects of other variables persist – such as the document length confound in our restricted dataset CLC_2009_ B1_balanced (Table 1.1). However, by attempting to control such factors, or at least being aware of them, researchers can avoid making inappropriate inferences over highly heterogeneous data. Examiners and assessors should also be aware that ‘opportunity of use’ is not necessarily equal for certain linguistic features across different task-topic types. In its document on CEFR, the Council of Europe sets out the lexical knowledge expected at each level, as shown in Table 1.7. Milton (2010: 214) raises some valid questions in response to the vocabulary range descriptors set out in Table 1.7, relating to how these broad characterizations of knowledge are to be measured in practice, and furthermore, ‘as to how learners are to demonstrate this knowledge when the tasks presented to them … only allow them to produce a few hundred words, and most of these will be highly frequent and common to most learners’. On the assessment side, Daller and Phelan (2007) found that raters can be inconsistent in applying these vocabulary criteria, while we demonstrate that not all prompts induce the same range of lexical items – indeed we might infer that not all essays encourage or necessitate the same range of lexical items. However, if the topics and tasks are deemed to be the most suitable for the relevant exams, then what’s needed here is awareness of the linguistic consequences on the part of researchers and assessors. From a pedagogical perspective, the effects of prompt topic are to be viewed as a by-product of task and topic variants rather than something to be altered. We suggest that teachers may use the information presented here to their advantage, in order to, for example, focus on a particular lexical set, or a

24

Learner Corpus Research

Table 1.7 CEFR level vocabulary range descriptors (Council of Europe 2001) CEFR level Vocabulary range C2

Has a very good command of a very broad lexical repertoire including idiomatic expressions and colloquialisms, shows awareness of connotative levels of meaning.

C1

Has a good command of a broad lexical repertoire allowing gaps to be readily overcome with circumlocutions; little obvious searching for expressions or avoidance strategies. Good command of idiomatic expressions and colloquialisms.

B2

Has a good range of vocabulary for matters connected to his or her field and most general topics. Can vary formulation to avoid repetition, but lexical gaps can still cause hesitation and circumlocution.

B1

Has a sufficient vocabulary to express him/herself with some circumlocutions on most topics pertinent to his/her everyday life such as family, hobbies and interests, work, travel and current events.

A2

Has sufficient vocabulary to conduct routine, everyday transactions involving familiar situations and topics. Has a sufficient vocabulary for the expression of basic communicative needs. Has a sufficient vocabulary for coping with simple survival needs.

A1

Has a basic vocabulary repertoire of isolated words and phrases related to particular concrete situations.

particular construction type, per the observation that different tasks and topics give learners practice in different aspects of complexity, accuracy and fluency (Yuan and Ellis 2003). As has been shown here, there are certain task-topic types that encourage the use of certain lexico-syntactic constructs. For example, ‘narrative’ essays encourage the use of verb construction with prepositional arguments, ‘autobiographical’ task-topic types encourage use of adverbs, and ‘professional’ prompts lead to a greater use of business vocabulary. It is analyses such as these that may be harnessed in pedagogy to further broaden learners’ linguistic repertoire, as diversity in this respect is known to associate strongly with increasing proficiency (Vercellotti, 2017).

Acknowledgements This chapter reports on research supported by Cambridge English, University of Cambridge. We gratefully acknowledge the help of Dr Nick Saville and Dr Fiona

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

25

Barker of Cambridge English, Professor Ted Briscoe and Dimitrios Alikanoitis of the University of Cambridge and Professor Michael McCarthy of the University of Nottingham. We also thank two anonymous reviewers and the two editors for their detailed and helpful feedback which greatly improved this chapter.

Notes 1 See www.pearsonlongman.com/dictionaries/corpus/learners.html. 2 See www.cambridgeenglish.org. 3 The ‘Common European Framework of Reference for Languages’: see www.coe.int/ lang-CEFR. 4 We use ‘essay’ here in its general sense: ‘a composition of moderate length on any particular subject’ (The Oxford English Dictionary). 5 Our thanks to Dr Fiona Barker of Cambridge English Language Assessment for her help in this. All prompts are © UCLES 2009 (1), and 2014 (2), (3). 6 This type of classifier is Bayesian as it implements Bayes’s theorem, and it is ‘naive’ in the sense that it assumes independence between features; see Bird, Klein and Loper (2009) for further background information. 7 Classifier accuracy without excluding keywords repeated from the prompts was 96.3 per cent, a large improvement on the prompt stopword classifier presented here, indicating that the presence of prompt keywords makes the task of assigning labels much easier, and hinting at a method for unsupervised grouping of unlabelled essays through clustering and the use of prompt keywords as labels. 8 This and all further plots produced using ggplot2 for R unless otherwise stated (Wickham 2009; 25R Core Team 2015). 9 For background information about this test, see Corder and Foreman (2014). 10 For a full list of all 163 frames, download the VALEX package from http://ilexir. co.uk/applications/valex (accessed on 20 June 2016).

References Ädel, A. (2015), ‘Variability in learner corpora’, in S. Granger, G. Gilquin and F. Meunier (eds), The Cambridge Handbook of Learner Corpus Research, 379–400, Cambridge: Cambridge University Press. Biber, D., Gray, B. and Staples, S. (2016), ‘Predicting patterns of grammatical complexity across language exam task types and proficiency levels’, Applied Linguistics, 37(5): 639–68.

26

Learner Corpus Research

Bird, S., Klein, E. and Loper, E. (2009), Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit, Sebastopol, CA: O’Reilly Media. Blei, D. M. (2012), ‘Probabilistic topic models’, Communications of the ACM, 55(4): 77–84. Briscoe, T. and Carroll, J. (1997), ‘Automatic extraction of subcategorization from corpora’, Proceedings of the 5th ACL Conference on Applied Natural Language Processing, Washington, DC, USA, 31 March to 3 April. Association for Computational Linguistics. Available at: https://arxiv.org/pdf/cmp-lg/9702002.pdf (accessed 9 January 2017). Briscoe, T., Carroll, J. and Watson, R. (2007), ‘The second release of the RASP system’, Proceedings of the COLING/ACL Interactive Presentation Sessions, Sydney, Australia, 16–17 July. Association for Computational Linguistics. Available at:https://pdfs. semanticscholar.org/7410/c010a38e7e23f38c3c6e898d5695a4874c61.pdf (accessed on 9 January 2017). Buttery, P. and Caines, A. (2012a), ‘Reclassifying subcategorization frames for experimental analysis and stimulus generation’, Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC 2012), Istanbul, Turkey, 23–25 May. European Language Resources Association. Available at; http://www. lrec-conf.org/proceedings/lrec2012/pdf/1063_Paper.pdf (accessed 9 January 2017). Buttery, P. and Caines, A. (2012b),‘Normalising frequency counts to account for “opportunity of use” in learner corpora’, in Y. Tono, Y. Kawaguchi and M. Minegishi (eds), Developmental and Crosslinguistic Perspectives in Learner Corpus Research, 187–204, Amsterdam: John Benjamins. Corder, G. W. and Foreman, D. I. (2014), Nonparametric Statistics: A Step-by-step Approach, 2nd edn, Hoboken, NJ: John Wiley & Sons. Council of Europe (2001), Common European Framework of Reference for Languages, Cambridge: Cambridge University Press. Daller, H. and Phelan, D. (2007), ‘What is in a teacher’s mind? Teacher ratings of EFL essays and different aspects of lexical richness’, in H. Daller, J. Milton and J. Treffers-Daller (eds), Testing and Modelling Lexical Knowledge, 234–45, Cambridge: Cambridge University Press. de Vries, A. and Ripley, B. D. (2015), ggdendro: Create Dendrograms and Tree Diagrams using ‘ggplot2’, R package version 0,1–17, URL http://CRAN,Rproject,org/package=ggdendro Granger, S., Dagneaux, E., Meunier, F. and Paquot, M. (2009), International Corpus of Learner English v2, Louvain-la-Neuve: Presses universitaires de Louvain. Gries, S. (2003), Multifactorial Analysis in Corpus Linguistics: A study of particle placement, London: Continuum Press. Hawkins, J. and Buttery, P. (2010), ‘Criterial features in learner corpora: theory and illustrations’, English Profile Journal, 1(1): e5.

The Effect of Task and Topic on Opportunity of Use in Learner Corpora

27

Hawkins, J. and Filipović, L. (2012), Criterial Features in L2 English: Specifying the reference levels of the Common European Framework, Cambridge: Cambridge University Press. Hinkel, E. (2009), ‘The effects of essay topic on modal verb uses in L1 and L2 academic writing’, Journal of Pragmatics, 41: 667–83. Khabbazbashi, N. (2017), ‘Topic and background knowledge effects on performance in speaking assessment’, Language Testing, 34(1): 23–48. Kobayashi, Y. and Abe, M. (2014), ‘A machine learning approach to the effects of writing task prompts’, Learner Corpus Studies in Asia and the World, 2: 163–75. Kuhn, M. and Johnson, K. (2013), Applied Predictive Modeling, Berlin: Springer. Milton, J. (2010), ‘The development of vocabulary breadth across the CEFR levels’, in I. Bartning, M. Martin and I. Vedder (eds), Communicative Proficiency and Linguistic Development: Intersections between SLA and Language Testing Research, 211–32, EUROSLA Monograph Series, 1. Newton, J. and Kennedy, G. (1996), ‘Effects of communication tasks on the grammatical relations marked by second language learners’, System, 24: 309–22. Nicholls, Diane (2003), ‘The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT’, Proceedings of the Corpus Linguistics 2003 Conference, Lancaster University, UK, 28–31 March. Available at: http://ucrel.lancs.ac.uk/ publications/cl2003/papers/nicholls.pdf (accessed on 9 January 2017). R Core Team (2015), R: A Language and Environment for Statistical Computing, Vienna: R Foundation for Statistical Computing, URL http://www,R-project,org (accessed 20 June 2016). Silverman, B. (1998), Density Estimation for Statistics and Data Analysis, London: Chapman & Hall. Tummers, J., Speelman, D. and Geeraerts, D. (2014), ‘Spurious effects in variational corpus linguistics’, International Journal of Corpus Linguistics, 19: 478–504. Venables, W. N. and Ripley, B. D. (2002), Modern Applied Statistics with S, 4th edn, Berlin: Springer. Vercellotti, M. L. (2017), ‘The development of complexity, accuracy, and fluency in second language performance: a longitudinal study’, Applied Linguistics, 38 (1): 90–111. Wickham, H. (2009), ggplot2: Elegant Graphics for Data Analysis, Berlin: Springer, Yuan, F. and Ellis, R. (2003), ‘The effects of pre-task planning and on-line planning on fluency, complexity and accuracy in L2 monologic oral production’, Applied Linguistics, 24: 1–27.

2

Phrasal verbs in spoken L2 English: The effect of L2 proficiency and L1 background Irene Marin Cervantes and Dana Gablasova

1 Introduction Much of the English language comes in the form of various multi-word units (Gries 2008; Wray 2002, 2012), with phrasal verbs (PVs) being a prominent formulaic structure (Granger and Paquot 2008; Paquot and Granger 2012). PVs are a very frequent feature of both written and spoken communication across different registers (Biber et al. 1999; Gardner and Davies 2007; Liu 2011; Trebits 2009) which makes them an indispensable component of any successful production and comprehension of the English language. Further, in terms of linguistic processing, PVs, along with other multi-word structures such as collocations, lie at the heart of fluent, native-like communication (Götz 2013; Schmitt and Redwood 2011; Siyanova-Chanturia and Martinez 2015). The importance of formulaic structures in general and PVs in particular for second language learning and use has been reflected in an increase in pedagogically oriented research on PVs. Several corpus-based studies have focused on identification of the most frequent PV forms (Liu 2011) as well as most frequent meaning senses of PVs (Garnier and Schmitt 2015) in order to provide guidance to teaching practitioners and material developers when selecting the PVs to teach. The growing awareness of the role played by PVs in fluent and appropriate communication has been also reflected in the number of pedagogical publications in the last ten years targeting the knowledge of PVs (e.g. English Phrasal Verbs in Use: Advanced (McCarthy and O’Dell 2007); Oxford Phrasal Verbs Dictionary for Learners of English (2006); Oxford Word Skills: Advanced: Idioms & Phrasal Verbs (Gairns and Redman 2011)). Direct attention has also been given to effective pedagogical approaches to teaching

Phrasal verbs in spoken L2 English

29

these structures (e.g. Farsani, Moinzadeh and Tavakoli 2012; Kurtyka 2001; Torres-Martínez 2015; White 2012; Yasuda 2010). However, despite the increased understanding of the PV use in L1 production and the direct focus on PVs in teaching materials and approaches, research brings further evidence that PVs remain difficult to acquire for learners of English (e.g. Garnier and Schmitt 2016; Liao and Fukuya 2004; Schmitt and Redwood 2011). The difficulty is believed to arise from a combination of factors related to semantic and structural properties of PVs as well as to cross-linguistic differences between English and learners’ native languages. In order to better understand the sources of difficulties and what pedagogical interventions may be effective, we need to gain more evidence of learners’ actual use of verb-particle constructions. Learner and L2 user corpora represent an invaluable source of data about patterns of PV use in L2 communication. In order to contribute to the understanding of the patterns of PV use in L2 English and factors that may affect the use, this study focuses on analysing PVs in the communication of L2 speakers of English at three levels of proficiency. As the majority of studies focused predominantly on written language, this study examines PVs in spoken, interactive communication based on the Trinity Lancaster Corpus (TLC; Gablasova et al. 2015). In particular, the study addresses the following two questions: 1. 2.

What is the effect of English proficiency on the frequency of PVs in L2 production? What is the effect of a particular L1 background on the frequency of PVs in L2 production?

2 Review of the literature 2.1 Phrasal verbs in the English language There are various types of multi-word verbs identified in the English language and their classification is a much-debated and complex issue (Biber et al. 1999; Gardner and Davies 2007). Despite the ongoing debates, most of the researchers agree that PVs, one type of the multi-word verbs, consist of a combination of two elements, a (usually) monosyllabic verb (e.g. turn) and an adverbial particle (e.g. in), which do not behave as independent words but rather form a meaning unit that is often not deducible from the meaning of each separate element (e.g. Biber et al. 1999; Gardner and Davies 2007; Liu 2011; Trebits 2009).

30

Learner Corpus Research

As documented by corpus-based studies based on large, general corpora of English, PVs are very frequent in both written and spoken discourse (Biber et al. 1999; Liu, 2011) even though there is strong evidence that their frequency may differ across modalities and registers with some genres showing a higher PV use (e.g. conversation and fiction) (Biber et al. 1999; Liu 2011; Trebits 2009). In terms of their distribution, Gardner and Davies (2007), using the British National Corpus (BNC), reported that the twenty most frequent PVs account for more than half of the PV occurrences in the whole corpus. A similar pattern was found by Liu (2011) who extended the analysis of PVs to the American variety of English and used multiple corpora (the BNC, the Corpus of Contemporary American English (COCA) and the Longman Spoken and Written English). In addition to the frequency of PV forms, Garnier and Schmitt (2015) described the different meaning senses of the most frequent PVs, reporting that in some cases a limited number of meaning senses dominated the use of a particular PV.

2.2 Phrasal verbs in L2 production PVs are considered difficult to acquire for L2 learners for a variety of reasons (e.g. Schmitt and Redwood 2011). First, they represent a more complex structure than single-word verbs which can affect learners’ ability to recognize them as a single semantic unit. The lack of awareness of the structural properties of PVs can be in some cases related to the absence of such forms in learners’ L1 (Garnier and Schmitt 2015; Siyanova and Schmitt 2007; Waibel 2008). In addition, the multi-word format of PVs increases their syntactical flexibility (i.e. many of them may function transitively and intransitively) and movability properties that allow the verb and the particle to be separated by nominal or pronominal forms. Second, the semantics of the PVs present another challenge to learners. PVs can be generally classified in three broad semantic groups: fully compositional or literal (e.g. go out), aspectual (e.g. clean up) and idiomatic (e.g. put off). In the literal and aspectual groups, particles are responsible for determining much of the meaning of the phrasal structure (Dehé 2002) whereas they do not seem to have particular meanings when it comes to idiomatic PVs (Thim 2012). These properties as well as the highly polysemous nature of some of the PVs (Garnier and Schmitt 2015) are likely to contribute to the difficulties that L2 learners have when processing and producing PVs.

Phrasal verbs in spoken L2 English

31

When comparing overall trends in the production of PVs by L1 and L2 speakers, the majority of studies (both on written and spoken discourse) reported that in contrast to L1 users, L2 users produced considerably fewer PVs. For example, Wierszycka (2013) found that the number of PV tokens produced by a group of Polish learners of English in conversation was almost four times smaller than that of L1 speakers. This trend has been observed across different L2 proficiency levels (Hulstijn and Marchena 1989) and discourse types (Siyanova and Schmitt 2007).

2.2.1 The role of L2 proficiency in the use of particle verbs Several areas of PV research identified a connection between L2 proficiency and productive use of PVs in learner language. For example, studies on lexical avoidance have shown that learners’ tendency to avoid PVs decreases as L2 proficiency increases (Dagut and Laufer 1985; Hulstijn and Marchena 1989; Liao and Fukuya 2004). In particular, intermediate learner speech has been reported to contain fewer PVs than that of advanced English speakers, who have often shown a preference for PVs similar to that of native speakers (Chen 2013; Hulstijn and Marchena 1989). These findings are in line with evidence from Liao and Fukuya’s (2004) study on Chinese learners’ PV use, which reported that proficiencylevel differences between intermediate and advanced learners were statistically significant and that advanced learners used PVs more frequently than intermediate learners of English. However, some findings also reported no connection between proficiency, operationalized as the length of stay in an English-speaking country and amount of L2 exposure in a classroom setting and the number of PVs used by learners (Wierszycka 2013). It is possible that this finding could be an artefact of the research design and the approach to evaluating L2 proficiency in the study (i.e. through indirect indicators such as length of stay abroad). Much of the PV research conducted so far has focused on the speech of upper intermediate and advanced learners; however, definitions and assessment of proficiency differed considerably making it difficult to directly compare or interpret the findings from PV studies (Chen and Smakman 2016). For example, proficiency in these studies has been operationalized as a number of years spent learning English and by the length of stay in an English-speaking country (Wierszycka 2013), by national language exams (Chen 2013) or merely based on researchers’ evaluation of the students (Dagut and Laufer 1985). Thus, while previous research has indicated that L2 proficiency plays a role in the frequency

32

Learner Corpus Research

of PV use in L2 production, the sources of the effect appear complex and more research is required to better understand the relationship between (increasing) proficiency and the productive knowledge of PVs.

2.2.2 The role of linguistic background of L2 speakers The L1–L2 correspondence is often considered an important factor which can play a role in the ease or difficulty with which L2 learners acquire a specific linguistic feature. Research indicates that learners in particular avoid L2 forms that do not have an equivalent in their L1 (Laufer 2000; Liao and Fukuya 2004; Siyanova and Schmitt 2007). In PV research, L1 background and specifically the existence of PVs in learners’ L1 have been believed to contribute to the relatively infrequent use of PVs by L2 speakers. For example, Dagut and Laufer (1985: 78) concluded that Hebrew learners avoided using PVs in their L2 communication due to a ‘systemic incongruence between L1 and L2’ given that there are no PVs or similar phrasal structures in Hebrew, a non-Germanic language. In corpus-based research, Gilquin (2015), using the Louvain International Database of Spoken English Interlanguage (LINDSEI), reported that French learners of English used PVs significantly less often than the comparison group of L1 speakers of English. This trend occurred regardless of the verb-particle word order or transitivity pattern and was attributed to the absence of PVs in French. Waibel (2008) found a similar pattern in the written communication of Italian-speaking learners in the Italian component of the ICLE. Despite the fact that Italian does comprise a limited number of phrasalverb-like forms, the lack of an exact L1 equivalent was considered to account for the low frequency of PV tokens. However, the interaction between the L1 and L2 of learners appears to be relatively complex. Hulstijn and Marchena (1989), Laufer and Eliasson (1993) and Liao and Fukuya (2004) analysed PV use by Dutch, Swedish and Chinese students respectively and found that there is a complex interaction between speakers’ L1 and L2. For example, the Dutch learners appeared to avoid using PVs not as a result of PV absence in their L1 (particle verbs exist in Dutch, a language that belongs to the Germanic family) but rather as a result of the inherent complexity and semantic specificity of PVs. Moreover, these learners were afraid of using English PVs that resembled familiar Dutch ones, opting for different verbs instead (Hulstijn and Marchena 1989). However, speakers of Swedish, another language of Germanic origin, were not reluctant to use L2 phrasal forms similar to those in their L1, with Laufer and Eliasson (1993) concluding that semantic similarity does not necessarily lead to avoidance.

Phrasal verbs in spoken L2 English

33

3 Methodology 3.1 Corpus description The TLC (Gablasova et al. 2015) was used in the study. It is a large corpus of semiformal monologic and interactive spoken L2 English production containing more than 3.5 million tokens. The corpus is based on an examination of spoken English (the Graded Examinations in Spoken English (GESE)) developed and administered by Trinity College London, a large international examination board. The TLC samples spoken production of L2 users from a wide age range (eight to seventy-two years) and from a variety of L1 and cultural backgrounds: Argentina, Brazil, China, India, Italy, Mexico, Russia, Sri Lanka and Spain. The corpus contains data produced across different proficiency levels of the CEFR (Council of Europe 2001). The version used in this study contains 595 speakers at B1, 583 speakers at B2 and 271 speakers at C1/C2 level of CEFR. As part of their examinations the candidates engage in different tasks, both monologic and dialogic, such as presentation or conversation (Trinity College London 2016), which increase in number according to the grade at which the candidate is assessed. Data in this study come from two interactive tasks, a discussion and a conversation as these remain the same for candidates at all proficiency levels in the TLC. Each speaking task takes about five to six minutes. In the discussion task the candidate and the examiner discuss a topic of the candidate’s choice; the candidate selected the topic at home and had a chance to explain it to the examiner first. The discussion is based on the examiner asking further questions about various aspects of the topic. In the conversation task the candidate and the examiner talk on a topic of general interest selected from a list that the candidate is familiar with. The topics include, for example, ‘pollution’, ‘recycling’ and ‘early childhood memories’.

3.2 Identifying PVs in the corpus For the purposes of this study, a ‘phrasal verb’ was defined as the combination of any verb proper and an adverbial particle (e.g. carry out). Neither prepositional (e.g. think about) or phrasal prepositional verbs (e.g. come up with) were included in the study. The PVs in the L2 production in the TLC were identified using complex query language (CQL) in Sketch Engine (Kilgarriff et al. 2014). The tag string ‘V + RP’ (verb + particle) was selected so that all verb lemmas with an adjacent or non-contiguous particle were retrieved. The search also allowed to

34

Learner Corpus Research

look for verbs separated from their particles by up to three intervening elements as the following examples show: turn off turn them off turn the lights off turn the blue lights off

That decision was deemed appropriate since PVs with four or more intervening elements are reported to be rare (Gardner and Davies 2007; Liu 2011) and are likely to represent false positives, that is, non-PV tokens. In addition to the automatic PV identification in the corpus, the PV candidates were manually checked in order to ensure a high level of precision in the results (Lüdeling, Evert and Baroni 2007).

3.3 Data analysis Given the high degree of inter-speaker heterogeneity in L2 production, statistical tests that take individual variation into account were used into the aggregate– data approach (Gablasova, Brezina and McEnery, 2017). A one-way analysis of variance (ANOVA) and post hoc Bonferroni tests were used to determine the effect of the independent variables, that is, learners’ proficiency levels and L1 background, on the frequency of PVs. With respect to RQ1, the difference between three proficiency levels (B1, B2 and C1/C2 of CEFR) was tested; in RQ2, speakers from six countries (Argentina, Mexico, Spain, China, Italy and Russia) were divided into four linguistic backgrounds (Spanish, Chinese, Italian and Russian). For RQ2, only data from speakers in the B2 proficiency band were used as it provided the most balanced L1 representation in the TLC.

4 Results 4.1 RQ 1: What is the effect of L2 proficiency on the frequency of PVs produced by L2 users? Table 2.1 shows the descriptive statistics for the frequency of PVs (normalized to 1,000 words) in each proficiency band. As can be seen from the table, even though the overall PV frequency appears relatively low, there was a gradual increase in

Phrasal verbs in spoken L2 English

35

Table 2.1 Frequency of PV occurrence per proficiency band Proficiency band Number of speakers Mean frequency

Standard deviation

B1

595

0.80

1.29

B2

583

0.87

1.19

C1/C2

271

1.14

1.50

the PV use across the three proficiency bands. The one-way ANOVA revealed the significant main effect of proficiency [F (2,1446) = 6.760, p < 0.01, partial eta2 = 0.009] but note that the effect size is very small. The Bonferroni post hoc test showed that there was a significant difference between the advanced group (C1/ C2) and the other two groups (both p < 0.05). However, there was no statistically significant difference between the B1 and B2 groups (p = 1.0). In order to better understand the kinds of PVs that L2 speakers use, Figure 2.1 presents the twenty most frequent PVs in the TLC and their distribution per proficiency level. As shown in the figure, the distribution of these twenty items across the three groups is varied. For example, the two most frequent PVs (go out and grow up) appeared in relatively similar proportions in all three proficiency levels while others were predominantly represented in a particular group of speakers (e.g. get up in B1, wake up and cut down in B2 and bring up and show off in C1/C2).

Figure 2.1 Top twenty PVs in the TLC.

36

Learner Corpus Research

While Figure 2.1 gives an overview of the most frequently used PVs in the TLC, Table 2.2 offers a finer-grained comparison of the most frequent PV lemmas in the speech of B1, B2 and C1/C2 English speakers. The table shows that there is a high degree of overlap in the range of the most frequent PVs across different proficiency levels; however, several PVs occurred with very high frequency only in a particular group of speakers (these are highlighted in bold in the table). In terms of inter-speaker differences, both frequency and range of PVs differed greatly in some cases. With respect to PV frequency, some speakers did not produce any PVs at all while others used high quantities of them (e.g. one speaker produced 15 PVs). In terms of PV range, some of the PVs occurred only in the production of one speaker (e.g. cut down, dress up, take off). Considering the meaning of PVs that appeared very frequently in the Table 2.2 Top twenty PVs per L2 proficiency band B1

B2

C1/C2

go out

go out

go out

2.

grow up

grow up

grow up

3.

get up

wake up

bring up

4.

pick up

find out

find out

5.

give up

pick up

come out

6.

fall down

cut down

give up

7.

come out

get up

come up

8.

find out

come out

show off

9.

wake up

take out

go down

10.

go up

go on

end up

11.

dress up

stand up

get up

12.

go down

give up

pick up

13.

take out

bring up

wake up

14.

come up

go up

go up

15.

turn off

come up

carry out

16.

set off

fall down

help out

17.

go on

take off

go on

18.

take off

take up

take up

19.

put up

go down

put up

20.

make up

end up

hang out

1.

Phrasal verbs in spoken L2 English

37

speech of only one group (or were used only by one speaker), it is possible that these PVs are more topic-dependent and their context of use is more restricted than that of the more general PVs that can be used across a variety of contexts. For instance, ‘cut down’ was more commonly found in environment-related contexts (examples 1–3). (1) people are cutting down trees and ah forests to build buildings (Sri Lanka, C1/C2) (2) they should erm erm they should build a recycle factories and er they shouldn’t erm cut down trees (Russia, B2) (3) pollution is caused by many factor like industrial fumes government activities where noise pollution like bursting of crackers honking we’re cu- cutting down our trees building building more huge buildings malls and et cetera (India, B1)

4.2 RQ 2: What is the effect of L1 background on the frequency of PVs in learner speech? The research question examined PV frequency according to four linguistic backgrounds (Spanish, Chinese, Italian and Russian) of L2 speakers from the B2 proficiency level. The descriptive statistics for the frequency of PVs (normalized to 1,000 words) are presented in Table 2.3. The results of oneway ANOVA showed that the effect of the L1 background on the frequency of PV occurrence was statistically significant [F (3,416) = 3.368, p < 0.05, partial eta2 = 0.024] but the effect size was again very small. The Bonferroni post hoc comparisons revealed that there was a significant difference between the Spanish and Chinese speakers of English (p < 0.05). However, no significant differences were found between speakers from any other L1 background.

Table 2.3 Frequency of phrasal verb occurrence according to L1 background L1 background

No of speakers

Mean frequency

Standard deviation

Spanish Chinese

221 55

0.60 1.09

0.92 1.30

Italian

116

0.69

1.15

Russian

28

0.70

0.80

38

Learner Corpus Research

5 Discussion 5.1 The effect of proficiency on the frequency of PVs in speech First, with respect to an overall frequency, the results showed that PVs represent a relatively small proportion of language used by L2 speakers in the TLC especially at the lower-intermediate (B1) and intermediate (B2) levels of proficiency. This finding contributes to the previous research that found that L2 users tended to avoid multi-word verbs in their production (e.g. Liao and Fukuya 2004). The present study complements and extends the previous findings in two important ways: first, the corpus-based findings in this study are based on data from hundreds of L2 users giving a very robust evidence of the trend. Second, it demonstrated that this trend is typical of L2 speakers from different proficiency levels when engaged in spoken, interactive communication, an area which has not yet been systematically studied at this scale. Second, while the PV use in the lower-intermediate and intermediate speakers was relatively low, the results also demonstrated that this increased for advanced (C1 and C2 levels) speakers. On average, these advanced speakers produced more than one PV per thousand words, which is relatively close to what, for example, Biber et al. (1999) report for native speakers of English in conversation (i.e. around 1.8 per 1,000 words) but falls short of the pattern found by Gardner and Davies (2007) (about 2 words per 150 words). Considering that the communication in TLC can be characterized as semiformal (it takes place in an institutional framework) rather than fully informal, the lower occurrence of PVs could also be to some extent attributed to the registerial constraints. Biber et al. (1999) as well as Liu (2011) found that academic prose was characterized by lower occurrence of PVs than conversation or fiction. The findings that showed the increase in PV use in speakers with higher proficiency are in line with previous research that showed correlation between increasing L2 proficiency and higher use of multi-word verbs (e.g. Dagut and Laufer 1985; Hulstijn and Marchena 1989; Liao and Fukuya 2004; Siyanova and Schmitt 2007). Overall, this study further confirmed that productive use of PVs poses a considerable challenge for L2 learners (Garnier and Schmitt 2015; Gonzalez 2010). Considering factors other than PV frequency, results showed that despite the increase in the PV frequency across proficiency levels, the range of the most frequent PVs remains largely unchanged (see the results in Table 2.2, for example). This, in itself, is not problematic; the majority of the most frequent words used by speakers in the TLC are also among the most frequent words in native speaker English (see,

Phrasal verbs in spoken L2 English

39

e.g., Biber et al. (1999) and Gardner and Davies (2007)). Often the most frequent words in general are those that can be used across a variety of genres and topics (Brezina and Gablasova 2015). The high frequency is also often related to the fact that these words are likely to have numerous meaning senses (Gardner and Davies 2007; Garnier and Schmitt 2015). Looking at the data in the present study, a closer examination revealed that the twenty most frequent PVs in the TLC contain highly polysemous verbs (e.g. put up, come up, come out, take out). However, the data also suggested that the L2 users (especially from the two lower-proficiency levels) do not seem to be fully aware of the polysemous nature of these verbs and tend to associate an average of one or two meanings (often their core meaning) with these verbal forms. The following examples of the verb ‘take out’ illustrate this. (4) S: you directly speak to the customer person you want to take out money you speak to that person you wanna know how to take out money you speak to that person (India, C1/C2) (5) S: er well all the people take out the the trash to the street (Mexico, B2) (6) S: the tradition started when the first Americans came to the territory of Mexico yeah er they for to represent the death of the person they take out the skull of the person and put in some symbols of blood in the in the nose in the eyes or in the mouth (Mexico, B2) According to the Oxford English Dictionary (2006), ten different meaning senses are associated with the verb ‘take out’. However, corpus data show learners’ preference for one meaning only (i.e. remove something from its original place). No examples where this verb was used to express different meaning senses were found. This same pattern was observed in the case of other highly polysemous verbs, for example, come out, come up and get up. In addition, another issue related to the meaning of PVs became apparent in the speech of B1 learners. When PVs contained a less abstract, lexicalized verbal component (e.g. dress in dress up), the intermediate learners appeared unaware of the contribution of particles to the meaning of PVs. Even though both the verb and the particle were produced, learners appeared to rely solely on the meaning of the verb without considering the particle as illustrated by examples (7)–(10) with respect to the verbs dress up and go down. It is possible that speakers from some varieties of English or speakers from a particular L1 background may be more prone to a specific mistake given the cross-linguistic influences or the established usage of English by these speakers. (7) S: I erm think I got privileges like I could have a hairstyle of my choice E: yes

40

Learner Corpus Research

S that I could dress up the way I wanted yes I could go around erm with my friends teenagers (India, B1) (8) E: have you ever lost anything while you were travelling? S: er yeah er in the hotel I lost my er one earring E: ah what were you doing when you lost it? S: er like I was er dressing up and at the base was loose so like er when I was doing it fell down (India, B1) (9) E: what do you need to do to stay healthy? S: I think we should er to go down to play so that we don't become er like very fat … (India, B1) (10) E: do you have any rules in your house about tidying your room? or S: yes after I leave to go down to play I play with my friends I have to keep er tidy my room … (India, B1) While in the above examples, the speakers extended the semantic meaning of the lexical verb to include the meaning sense(s) of a PV that shared the lexical verb, examples (11) and (12) demonstrate that in some cases speakers with lower proficiency extended the meaning of the PV even further. (11) E: what do you need to do if you want to learn new words? how do you record them? S: yes er er how I gr= I grew up my glossary you mean yeah mm erm so I er er I always watch er f= er a lot of er English films er with subtitles and I also read a lot of English books … (Russia, B1) (12) S: … the people erm er learn new foreign language because to grow up er our minds and er to to see other difference cul-cultures in the world and erm to learn different types of er erm of education (Italy, B1) These patterns observed in the speech of L2 users in the TLC suggest that further attention, both in research and teaching practice, needs to be given to appropriate use of PVs and depth of PV knowledge (Garnier and Schmitt 2016) as well as to frequency and range of PV production.

5.2 The impact of the L1–L2 correspondence phenomenon in learner PV use The exploration of L1 background showed that this variable had an effect on PV frequency. In the corpus data analysed, speakers of Chinese seem to use more PVs compared to Spanish, Italian and Russian learners, with the difference

Phrasal verbs in spoken L2 English

41

between Chinese and Spanish speakers being statistically significant. Given that Chinese lacks PVs or similar multi-word verbs, this finding is incongruent with what previous experimental and corpus-based research on cross-linguistic differences in PV use has reported (e.g. Gonzalez 2010; Liao and Fukuya 2004; Waibel 2008), which indicated that learners whose L1 lacks verb-particle constructions are more likely to avoid using PVs. However, as Chen (2013: 433) asserts based on her corpus-based analysis of Chinese PV use, structural L1– L2 differences do not necessarily have ‘a considerable impact on phrasal verb acquisition and production’. It is possible, then, that cross-linguistic differences do not always represent a strong predictor of PV avoidance in the case of nonGermanic languages and that the higher PV frequency in the speech of Chinese learners might be due to external reasons, for example, learners’ L2 exposure or previous language training.

6 Implications for the L2 classroom There are several implications for language pedagogy following the findings of this study. First, this study highlighted the connection between overall proficiency in the language and the productive knowledge of PVs. In particular, the study identified more accurately that an intervention may be needed early on in language-learning process as lower-intermediate and intermediate speakers appeared in particular to avoid the structure. As PVs, along with other multiword expressions, contribute to fluency in spoken production (SiyanovaChanturia and Martinez 2015; Wray 2012), learners should not wait with using these forms until they reach an advanced L2 level. An early introduction and practice of PVs may be of particular importance to learners whose first language does not contain PV-like structures as they may face an additional challenge in understanding how PVs function in language. While there is a growing awareness of the importance of PVs for language learning, it is still difficult to select the PVs that should be given prominence in language classrooms given that there are thousands to choose from in the English language (Garnier and Schmitt 2015; Liu 2011; Schmitt and Redwood 2011). This study highlighted two issues that should be taken into consideration when answering this question. First, the learners across proficiency levels seem to be familiar and able to productively use several of the most frequent PVs; however, they used these PVs repeatedly. Thus, it seems effort should be

42

Learner Corpus Research

made to extend the range to include PVs that are likely to be of relevance to the learners given their specific learning aims. While with lower-level learners these can include more PVs from the lists of the general PVs, with higher-level learners these could include more academic or subject-specific PVs. For example, Trebits (2009) analysed PV patterns in EU documents to assist L2 users with acquiring mastery in this specific subject area. Second, in addition to extending the range of PVs used by L2 speakers, urgent attention needs to be given to the question of appropriate use of PVs. As pointed out in previous literature (Garnier and Schmitt 2015), PVs are highly polysemous and L2 speakers in the TLC seemed to have difficulty with applying the senses appropriately, interpreting the meaning of the particle (e.g. whether it has a literal or an opaque meaning) and distinguishing between lexical verbs and PVs with similar forms (e.g. dress as opposed to dress up). These findings are in line with other research (e.g. Garnier and Schmitt 2016) that pointed to limited knowledge of meaning senses of individual PVs among L2 learners. It is encouraging that this issue has received more attention in recent pedagogically oriented studies, with corpus-based materials (e.g. the PHaVE list, Garnier and Schmitt 2015) produced to assist learners and teachers with the semantic dimension of PVs. In addition to materials that draw attention to different meaning senses of PVs, teaching approaches that focus especially on the semantics of PVs appear important. At lower levels of proficiency these would encourage appropriate, error-free use and at higher levels of proficiency assist the speakers with extending their repertoire of different PV senses. Some effective approaches that stress the meaning-dimension of PVs focused on the semantics of particles in order to show learners that there is an underlying systematicity in PV meaning (Condon 2008; Dirven 2001; Kurtyka 2000; Yasuda 2010). For instance, Side (1990) proposes grouping PVs according to particles and presenting these with sufficient context so that learners can identify core and metaphorical meaning senses of particles. Drawing on conceptual motivation and sociocultural theories, White (2012) suggests implementing a five-step instruction model whereby learners make sense of PVs by associating core particle meanings with metaphorical extensions of those core meanings. In a similar vein, Torres-Martínez (2015) argues that learners can benefit from seeing PVs as hyponyms whose meanings derive from prototypical verbs (e.g. step forward and wander over inherit the semantics of the prototypical verb move).

Phrasal verbs in spoken L2 English

43

7 Conclusion The chapter investigated the use of PVs in L2 spoken, interactive communication with particular attention to speakers’ proficiency in English and their linguistic background. The study is innovative in providing robust evidence about the effect of these variables from a large, new corpus of spoken L2 English. It brought evidence supporting previous findings that indicated the effect of proficiency on the degree to which L2 learners engage with PVs; the findings on the effect of L1 background are also partly in line with previous studies and suggest that factors other than cross-linguistic differences might play a role in the production of PVs in L2 speech. While the study provided further evidence about the trends in the frequency of PV use, it also highlighted the importance of other areas, so far underexplored in corpus-based studies of PVs in spoken communication, namely, the appropriateness and accuracy of PV use. The patterns found in L2 production also emphasized that further attention needs to be paid to the depth of PV knowledge among L2 learners (e.g. Garnier and Schmitt 2016; Liu 2011). Overall, the chapter demonstrated the importance of continuing corpus-based explorations of PV use for both our understanding of the language acquisition and use of multiword constructions in L2 as well as for improvements in language pedagogy.

References Biber, D., Johansson, S., Leech, G., Conrad, S., and Finegan, E. (1999), Longman Grammar of Spoken and Written English, London: Longman. Brezina, V. Gablasova, D. (2015), ‘Is there a core general vocabulary? Introducing the New General Service List’, Applied Linguistics, 36(1): 1–22. Chen, M. (2013), ‘Overuse or Underuse: A Corpus Study of English Phrasal Verb use by Chinese, British and American University Students’, International Journal of Corpus Linguistics, 18(3): 418–42. Chen, X. and Smakman, D. (2016), ‘Avoidance of Phrasal Verbs by Learners of English: Definitional and Methodological Issues’, Paper presented at the Een Colloquium Over Universitair Taalvaardigheidsonderwijs, Leiden University, The Netherlands. Available online: https://openaccess.leidenuniv.nl/bitstream/handle/887/38548/05. pdf?sequence=1 (accessed 30 March 2016). Condon, N. (2008), ‘How Cognitive Linguistic Motivations Influence the Learning of PVs’, in F. Boers and S. Lindstromberg (eds), Cognitive Linguistic Approaches to Teaching Vocabulary and Phraseology, 133–58, Berlin: Mouton de Gruyter.

44

Learner Corpus Research

Council of Europe (2001), Common European Framework of Reference for Languages. Cambridge: Cambridge University Press. Dagut, M. and Laufer, B. (1985), ‘Avoidance of Phrasal verbs—A Case for Contrastive Analysis’, Studies in Second Language Acquisition, 7(1): 73–9. Dehé, N. (2002), Particle Verbs in English: Syntax, Information Structure and Intonation, Amsterdam: John Benjamins. Dirven, R. (2001), ‘English phrasal verbs: Theory and didactic application’, in M. Pütz, S. Niemeier and R. Dirven (eds), Applied Cognitive Linguistics II: Language Pedagogy, 3–27, Berlin: Mouton de Gruyter. Farsani, H. M., Moinzadeh, A. and Tavakoli, M. (2012), ‘Mnemonic effectiveness of CL-motivated picture-elucidation tasks in foreign learners’ acquisition of English phrasal verbs’, Theory and Practice in Language Studies, 2(3): 498–509. Gablasova, D., Brezina, V., McEnery, T. and Boyd, E. (2015), ‘Epistemic stance in spoken L2 English: The effect of task and speaker style’, Applied Linguistics, 1–26. Gablasova, D. Brezina, V. and McEnery, T. (2017), ‘Exploring learner language through corpora: Comparing and interpreting corpus frequency information’, Language Learning, 67(S1): 155–79. Gairns, R. and Redman, S. (2011), Oxford Word Skills: Advanced: Idioms & Phrasal Verbs, Oxford: Oxford University Press. Gardner, D. and Davies, M. (2007), ‘Pointing out Frequent Phrasal Verbs: A CorpusBased Analysis’, TESOL Quarterly, 41(2): 339–59. Garnier, M. and Schmitt, N. (2015), ‘The PHaVE list: A Pedagogical List of Phrasal Verbs and their Most Frequent Meaning Senses’, Language Teaching Research, 19(6): 645–66. Garnier, M. and Schmitt, N. (2016), ‘Picking up Polysemous Phrasal Verbs: How Many do Learners know and what Facilitates this Knowledge?’ System, 59: 29–44. Gilquin, G. (2015), ‘The use of phrasal verbs by French-speaking EFL learners: A constructional and collostructional corpus-based approach’, Corpus Linguistics and Linguistic Theory, 11(1): 51–88. Gonzalez, R. A. (2010), ‘L2 Spanish Acquisition of English Phrasal Verbs: A Cognitive Linguistic Analysis of L1influence’, in M. Campoy, B. Bellés-Fortuño and M. Gea-Valor (eds), Corpus-based Approaches to English Language Teaching, 149–66, London: Continuum. Götz, S. (2013), Fluency in Native and non-native English Speech (Vol. 53). Amsterdam: John Benjamins. Granger, S. and Paquot, M. (2008), ‘Disentangling the phraseological web’, in S. Granger and F. Meunier (eds), Phraseology. An Interdisciplinary Perspective, 27–50, Amsterdam: John Benjamins. Gries, S. T. (2008), ‘Phraseology and linguistic theory: A brief survey’, in S. Granger and F. Meunier (eds), Phraseology. An Interdisciplinary Perspective, 3–25, Amsterdam: John Benjamins.

Phrasal verbs in spoken L2 English

45

Hulstijn, J. H. and Marchena, E. (1989), ‘Avoidance: Grammatical or Semantic Causes?’, Studies in Second Language Acquisition, 11(3): 241–55. Kilgarriff, A., Rychly, P., Smrz, P. and Tugwell, D. (2014), ‘The Sketch Engine: Ten years on’, Lexicography: 1–30. Kurtyka, A. (2001), ‘Teaching English phrasal verbs: A cognitive approach’, in M. Pütz, S. Niemeier and R. Dirven (eds), Applied Cognitive Linguistics II: Language Pedagogy, 29–54, Berlin: Mouton de Gruyter. Laufer, B. (2000), ‘Avoidance of Idioms in a Second Language: The Effect of L1–L2 Degree of Similarity’, Studia Linguistica, 54(2): 186–96. Laufer, B. and Eliasson, S. (1993), ‘What causes avoidance in L2 Learning: L1–L2 difference, L1– L2 similarity, or L2 complexity?’, Studies in Second Language Acquisition, 15(1): 35–48. Liao, Y. and Fukuya, Y. J. (2004), ‘Avoidance of phrasal verbs: The case of Chinese learners of English’, Language Learning, 54(2): 193–226. Liu, D. (2011), ‘The Most Frequently used English Phrasal Verbs in American and British English: A Multicorpus Examination’, TESOL Quarterly, 45: 661–68. Lüdeling, A., Evert, S. and Baroni, M. (2007), ‘Using the web data for linguistic purposes’, in M. Hundt, N. Nesselhauf and C. Biewer (eds), Corpus Linguistics and the Web, 7–24, Amsterdam: Rodopi. McCarthy, M. and O’Dell, F. (2007), English Phrasal Verbs in Use: Advanced, Cambridge: Cambridge University Press. Oxford English Dictionary. (2006), Oxford: Oxford University Press. Oxford Phrasal Verbs Dictionary for Learners of English. (2006), Oxford: Oxford University Press. Paquot, M. and Granger, S. (2012), ‘Formulaic Language in Learner Corpora’, Annual Review of Applied Linguistics, 32: 130–49. Schmitt, N. and Redwood, S. (2011), ‘Learner Knowledge of Phrasal Verbs: A CorpusInformed Study’, in F. Meunier, S. De Cock and G. Gilquin (eds), A Taste for Corpora. In honour of Sylviane Granger, 173–209, Amsterdam: John Benjamins. Side, R. (1990), ‘Phrasal Verbs: Sorting them Out’, ELT Journal, 44(2): 144–52. Siyanova, A. and Schmitt, N. (2007), ‘Native and Nonnative use of Multi-word vs. Oneword Verbs’, International Review of Applied Linguistics in Language Teaching, 45: 119–39. Siyanova-Chanturia, A. and Martinez, R. (2015), ‘The Idiom Principle Revisited’, Applied Linguistics, 36(5): 549–69. Thim, S. (2012), Phrasal Verbs: The English Verb- Particle Construction and its History, Germany : Walter de Gruyter. Torres-Martínez, S. (2015), ‘A constructionist approach to the teaching of phrasal verbs’, English Today, 123, 31(3): 46–58. Trebits, A. (2009), ‘ The Most Frequent Phrasal Verbs in English Language EU Documents–A Corpus-based Analysis and its Implications’, System, 37(3): 470–81.

46

Learner Corpus Research

Trinity College London. (c 2016), Graded Examinations in Spoken English (GESE). Available online: http://www.trinitycollege.com/site/? id=368 (accessed 10 April 2016). Waibel, B. (2008), Phrasal Verbs: German and Italian Learners of English Compared, Saarbrücken: VDM Verlag Dr. Müller. White, B. (2012), ‘A Conceptual Approach to the Instruction of Phrasal Verbs’, The Modern Language Journal, 96(3): 419–38. Wierszycka, J. (2013), ‘Phrasal verbs in learner English: A semantic approach’, Research in Corpus Linguistics, 1: 81–93. Wray, A. (2002), Formulaic Language and the Lexicon. Cambridge: Cambridge University Press. Wray, A. (2012), ‘What do we (think we) know about Formulaic Language? An Evaluation of the Current State of Play’. Annual Review of Applied Linguistics, 32: 231–54. Yasuda, S. (2010), ‘Learning Phrasal Verbs through Conceptual Metaphors: A Case of Japanese EFL Learners’, TESOL Quarterly, 44(2): 250–73.

3

Investigating the Effect of the Study Abroad Variable on Learner Output: A Pseudo-Longitudinal Study on Spoken German Learner English Sandra Götz and Joybrato Mukherjee

1 Introduction It has been noted by various scholars that second language acquisition (SLA) theory needs to take into account learning context as a determining factor in the acquisition/learning process (see Norris and Ortega 2001). One major research focus has been the impact of the ‘study abroad’ (SA) variable on a learner’s language performance. It has been shown, for example, that SA contexts are beneficial in many regards but do not always facilitate or accelerate the learning process (e.g. Barron 2002; Freed 1995). However, empirical research into interrelations between learning contexts and language learners’ performance so far has mainly focused on a small number of learners due to the difficulty involved in comparing a large number of learners’ output before and after their stay abroad. In LCR, however, most corpora include data from a large number of learners who have acquired the foreign language in a variety of different learning contexts and who also represent different proficiency levels. Yet, many learner corpus studies have not taken advantage of the full potential that learner corpora offer with regard to the investigation of the impact of learning contexts (Mukherjee and Götz 2015). Learner corpus studies have therefore often been criticized for defining learning context  – and different proficiency levels arising from different learning contexts – too globally (e.g. Callies 2009; Granger et al. 2009;  Ortega and Byrnes 2008;  Thomas 1994). For example, the

48

Learner Corpus Research

learners’ ‘institutional status’ is often equated with a certain proficiency level of the learners in the corpus, and university students are routinely defined as ‘advanced learners’ although their individual learning contexts have not been taken into consideration. However, recent learner corpora like the LINDSEI (Gilquin, De Cock and Granger 2010) contain detailed learner profiles that provide rich meta-information about each individual learner in the corpus and his/her language-learning experience, including the time spent abroad in up to three English-speaking countries. In this chapter, we would like to suggest one methodological procedure as to how to combine (learner-)corpus-linguistic methodology and SLA theory by testing for correlations between learning context variables (gathered from the information in the learner profiles) on the one hand and learner output variables on the other, and demonstrate how research into learning context can now benefit from the advent of large-scale (learner) corpora. More specifically, we will zoom in on one very central learning context variable and focus on the SA variable in the German component of the LINDSEI corpus (LINDSEI-GE; Brand and Kämmerer 2006) and the effect it has on the learners’ output and performance in the foreign language regarding the linguistic variables of fluency, grammatical accuracy and lexical development. To this end, we will report on the findings obtained from a ‘pseudo-longitudinal’ (as Johnson and Johnson (1999) labelled it) case study. In the remainder of this chapter, we will first summarize previous research on the effect of the SA variable on learner output (see Section 2) and then describe the database and methodology (see Section 3) and the findings (see Section 4) of this chapter. Finally, we will offer some conclusions and sketch out some avenues for future research (see Section 5).

2 Study abroad contexts and learner output: A mixed picture It has always been intuitively appealing among SLA researchers and practitioners as well as foreign-language learners and language-pedagogical laymen to assume that a stay abroad in the target speech community would benefit the languagelearning process. Sanz (2014: 1) describes this assumption as a commonly held belief that ‘while abroad, learners imbibe the language, soak it in, they feel like

Investigating the Effect of the Study Abroad Variable on Learner Output

49

sponges, they are surrounded, covered with language, their brains saturated’. On the one hand, a range of empirical studies have shown that in particular with regard to pragmatic competencies and fluency-related variables, learners may in fact profit from an extended immersion in the target speech community and, thus, from the natural interaction with native speakers (e.g. Lafford 2004; Mora and Valls-Ferrer 2012). On the other hand, other empirical studies have revealed hardly any positive effect of a SA on the learner output, for example, at the morphosyntactic and phonological levels (e.g. Collentine 2004; DiázCampos 2004). As Sanz (2014) points out, such contradictory findings and various empirical observations that clearly run counter to the aforementioned general belief in the beneficial effect of stays abroad have to do with methodological problems. Some of the positive results for SA groups, which are often compared to groups in the classroom context ‘at home’ (AH) as control groups, can be traced back to a range of factors that have remained uncontrolled in many studies, for example, learners’ fundamentally different motivations to communicate and interact (abroad as opposed to at home in the classroom) and their corresponding willingness to take communicative risks. Note also that while SA groups may have been shown to progress better and faster than AH groups, learners taking part in domestic immersion (IM) programmes in the L2 context often outperform the SA groups (e.g. Freed, Segalowitz and Dewey 2004), which makes it clear that comparative studies involving SA groups and AH groups only may lead to somewhat misleading results from a language-pedagogical point of view. This is particularly true if the stay abroad is viewed as the decisive factor per se exerting an influence on the language-learning process: as Dietrich, Klein and Noyau (1995: 277) rightly note, ‘duration of stay is an uninteresting variable. What matters is intensity, not length of interaction’. Of course, there is a correlation between the length of the stay abroad and the potential for interaction: the longer the time spent in the target speech community, the greater the likelihood of opportunities for interaction with the surrounding speech community. But it is certainly important to look at the communicative practice and the degree of interaction that a stay abroad offers, which also entails that it is essential in comparative studies to take into account ‘standardized’ stays abroad that provide learners with comparable opportunities. As we have pointed out elsewhere (Mukherjee and Götz 2015), modern learner corpora provide databases that make it possible to look at much larger language-learner groups than in previous SLA studies. What is more, various

50

Learner Corpus Research

learner corpora – as, for example, the ICLE (Granger et al. 2009) or the LINDSEI (see Gilquin, De Cock and Granger 2010) – include valuable meta-information on all the individual learners, including their stays abroad. This kind of metainformation can be used for a systematic investigation of potential correlations between an individual learner’s stay abroad and the (improvement in the) quality of the learner’s output. Finally, the effect of studies abroad on the language-learning process needs to be analysed from a longitudinal perspective. When the number of learners is high, such longitudinal data may be difficult to obtain. Learner corpora make it possible to circumvent this problem to some extent as they allow for ‘pseudo-longitudinal’ studies, that is, studies based on ‘samples of learner language [that] are collected from groups of different proficiency levels at a single point in time’ allowing the researcher to create a ‘longitudinal picture … by comparing the devices used by the different groups’ (Ellis and Barkhuizen 2005: 97). Such a ‘pseudo-longitudinal’ approach to trace developmental patterns has been increasingly applied over the past few years in LCR; it represents a methodologically sound way of capturing group developments in the learning process of learners that share similar characteristics (e.g. Meunier 2015) and has already led to valuable findings in previous studies (e.g. Abe and Tono 2005; Maden-Weinberger 2015). Against the background of this discussion, our approach in the present study has been to focus on the SA experience of German learners of English and potential correlations between their SA experience on the one hand and their L2 output on the other with regard to a number of fluency-related, accuracyrelated and lexical output variables. We used the LINDSEI-GE corpus, which incorporates standardized meta-information on the SA experience of all the fifty learners included in the corpus. Our approach will be ‘pseudo-longitudinal’ in nature as we will compare groups of learners, categorized according to their different SA experiences in LINDSEI-GE, thus making it possible to construct an overall (pseudo-)longitudinal picture.

3 Database and methodology In order to investigate the effect of the SA variable on the advanced German learners’ performance, we investigated the German component of LINDSEI, which was compiled from fifty interviews (each c. fifteen to twenty minutes in length) with English majors at Justus Liebig University Giessen (see Brand and Kämmerer 2006). These interviews consist of three parts: (i) a monologic part, in

Investigating the Effect of the Study Abroad Variable on Learner Output

51

which the learners had to talk about one of three possible topics they had received shortly before the interview, (ii) free discussion and (iii) relating a picture-story. LINDSEI-GE contains approximately 86,000 words of intermediate to advanced learner speech, approximately ranging from CEFR levels B2–C2 (see Gilquin, De Cock and Granger 2010). For the purposes of this chapter, we use the errortagged version of LINDSEI-GE (see Götz 2015; Kämmerer 2012) with revised tags of the Louvain error-tagging manual (Dagneaux et al. 2005). LINDSEI-GE, like the other LINDSEI components, also includes learner profiles with information on learners’ gender, age, nationality, first language, father’s/mother’s mother tongue, current line of studies, current year of studies, medium of instruction (at school, at university), years of English (at school, at university), other foreign language(s), and detailed and ‘standardized’ information on previous stay-abroad experiences (yes/no, at what age, in which country/countries and duration in months). We used the information on the ‘SA’ variable gathered from the learner profiles to compare learner data at different levels of proficiency in order to test for developmental trends after a short and a long stay abroad. In the present pseudo-longitudinal study based on LINDSEI-GE, we distinguish between three proficiency levels: (i)  learners who have not stayed abroad at all (i.e. 0 months), (ii) learners who had a short stay abroad (i.e. 1–12 months) and (iii) learners who stayed abroad for a longer period (i.e. 13–60 months).1 This distinction is based on the assumption that because of the standardized design of LINDSEI-GE, there is an a priori correlation between these three learner groups with different stay-abroad experiences and their overall proficiency levels. The breakdown of the resulting sub-corpora is shown in Table 3.1. Our overall research question is as follows: Does the SA variable have an effect on the learners’ fluency, accuracy and vocabulary development? Against the background of our discussion of the SA variable in much SA-related research work so far (see Section 2), we assume that our findings will provide a mixed picture, potentially revealing some positive effects of the SA variable with regard to the learners’ fluency. The fluency variables we investigated are temporal fluency variables, or ‘fluencemes’ (Götz 2013), namely unfilled pauses, mean length of runs and speech rate (see Lennon 1990) and ‘fluency-enhancement strategies’, that is strategies of ‘dealing with speech planning phases and easing processing pressure’ (Götz 2013: 42), namely the use of discourse markers and smallwords. The learners’ development in grammatical knowledge was tested by comparing the number of the two most error-prone grammatical errors gathered from the error-tagged version of LINDSEI-GE. These errors are (i) errors related to verb-tense forms

52

Learner Corpus Research

Table 3.1 Learner groups in LINDSEI-GE No stay abroad (0 months), n=14

Short stay abroad (1–12 months), n=26

Long stay abroad (13–60 months), n=10

GE002

GE021

GE001

GE019

GE035

GE009

GE030

GE003

GE034

GE004

GE023

GE037

GE016

GE033

GE006

GE039

GE005

GE024

GE038

GE018

GE036

GE007

GE041

GE008

GE026

GE040

GE022

GE042

GE010

GE043

GE011

GE027

GE044

GE025

GE050

GE012

GE046

GE013

GE028

GE045

GE020

GE049

GE014

GE029

GE047

GE015

GE031

GE048

GE017

GE032

(e.g. (GVT) I’ve been $went$ there. two years ago (GE020)) (with the error tag in brackets in front of the error and the corrected target hypothesis between $ signs), (ii) article-related errors (e.g. I’m . (GA) 0 $a$ football fan (GE004)). The last category we took into consideration is the learners’ lexical development, which we ascertained by assessing the number of words per interview and the number of the two most error-prone lexical error categories in the error-tagged version of LINDSEI-GE, namely (i) lexical single errors, where the learners used the wrong vocabulary item (e.g. But it was erm . erm . a (LS) parody $caricature$. of me (GE027)) and (ii) the ‘lexical phrase/false friend’ category (e.g. We pay fifty Euros (LPF) in the semester $per semester$ (GE028)). Each of these variables was set in relation to the time the learners had spent abroad by applying general linear models with one categorical predictor (i.e. the three stay-abroad groups) and continuous outcome variables (i.e. the fluency, accuracy and lexical variables) (see Gries 2013a,b) based on the software package R (see R Development Core Team 2013).

4 Findings 4.1 The effect of the study abroad variable on fluency 4.1.1 Temporal fluencemes As unfilled pauses (UPs) are the most central temporal fluencemes, we tested the effect of the SA variable on their number and position per one hundred words

Investigating the Effect of the Study Abroad Variable on Learner Output

53

Figure 3.1 The effect of the number of months spent abroad by a learner on the predicted probability of the total number of UPs per one hundred words (phw) (lefthand panel) and the number of UPWCL phw (right-hand panel).

(phw) (both the total number and the number of UPs within clauses), the mean length of runs and the speech rate in words per minute (wpm) of the learners in the three groups. For the total number of UPs, the final model reveals a highly significant correlation between the time the learners have spent abroad and their number of unfilled pauses (F = 4.64, df = 46, p = 0.006, R2 = 0.23). For the number of unfilled pauses within clauses (UPWCL), the model is also significant, but does not explain the variability in the data equally well (F = 3.29, df = 47, p = 0.046, R2 = 0.12). Figure 3.1 shows graphs of the main effect of the SA predictor on the learners’ number and position of UPs. The left-hand panel shows a strong correlation between the time a learner has spent abroad and the number of UPs: learners are predicted to use significantly fewer pauses after a 1–12 month stay abroad, and, again, significantly fewer pauses when they have spent 13–60 months abroad. When we consider the number of UPWCL in the right-hand panel, we find a difference between the group of learners who haven’t spent any time abroad and the group that has stayed 1–12 months, but, interestingly, the result for the latter group is not significantly lower (p = 0.22). However, the model reveals that the predicted decrease for the third group who stayed abroad for 13–60 months is again significant (p = 0.0136). The findings are similar for the learners’ mean lengths of runs (MLR) and their speech rate in words per minute (WPM). For the effect of the SA variable on the learners’ MLR, the model again predicts a highly significant correlation between the time the learners have spent abroad and their MLR (F = 5.21, df = 47, p = 0.009, R2 = 0.18). The model can explain 18 per cent of the variance in

54

Learner Corpus Research

Figure 3.2 The effect of the number of months spent abroad by a learner in LINDSEI-GE on their predicted mean length of runs (left panel) and their predicted speech rate in wpm (right panel).

the data only by considering the SA variable, so it might serve as quite a good explanation of the data, given that many other factors potentially influence a learner’s mean length of runs apart from a stay abroad. The same applies to the effect of the SA variable on the learners’ WPM, as the model turns out to be highly significant (F = 5.77, df = 47, p = 0.006). Again, considering the many factors that can be responsible for a learner’s performance, the model indicates a fairly strong relationship between the two variables and can explain almost 20 per cent of the variance in the data (R2 = 0.19). Figure 3.2 shows the main effect of the SA variable on the learners’ MLR (left-hand panel) and on the learners’ WPM (right-hand panel) with findings similar to the ones for UPs: the learners’ predicted mean length of runs, again, does not significantly increase after a 1–12-month stay abroad (p = 0.183), but increases significantly for learners who have spent 13–60 months abroad (p = 0.002). For their WPM, too, the predicted increase is only marginally significant after a short stay abroad (p = 0.059), but then becomes significant after a long stay abroad (p = 0.001). In essence, very much in line with previous research on the effect of the SA variable on learners’ temporal fluency, we see a significant positive effect of the SA variable on the German learners’ temporal fluency for all the investigated temporal fluencemes.

4.1.2 Fluency-enhancement strategies The second group of features we investigated was the effect of a stay abroad on fluency-enhancement strategies (FES). In particular, we looked at the three

Investigating the Effect of the Study Abroad Variable on Learner Output

55

Figure 3.3 The effect of the number of months spent abroad by learners in LINDSEI-GE on their predicted use of discourse markers per one hundred words (phw) (like in the left-hand panel, you know in the middle and well in the right-hand panel).

Figure 3.4 The effect of the number of months spent abroad by learners in LINDSEI-GE on their predicted use of smallwords per one hundred words (phw) (sort of/sorta in the left panel, kind of/kinda in the middle and quite in the right panel).

most frequently occurring discourse markers (like, you know and well) and most frequent smallwords (sort of/sorta, kind of/kinda and quite). For all the investigated FES, there is great variation in the learners’ use of strategies for each of the forms under scrutiny (which is clearly visible through the wide confidence intervals in Figure 3.3 and Figure 3.4). Also, many learners only make use of one or two of the strategies, so that often only a very small number of learners are responsible for many of the occurrences, making the modelling of the data rather difficult. For the discourse marker like, however, the effect of the SA variable proves to be significant (F = 3.537, df = 47, p = 0.037), but the overall effect size is somewhat smaller than for the temporal fluencemes (R2 = 0.13). The left-hand panel of Figure 3.3 reveals similar patterns as before with a non-significant increase in the use of like after a short stay of 1–12 months abroad (p = 0.4865), which contrasts with the significant increase (p = 0.014) after a long stay of 13–60 months. For the use of the discourse marker you know the variance in the learner data seems to be too high for the model to be able to make any significant predictions

56

Learner Corpus Research

(F = 0.5766, df = 47, p = 0.566). Accordingly, the overall predicted effect of the SA variable on the use of you know is very low (R2 = 0.024) and can only explain about 2 per cent of the variance in the data. However, the middle panel in Figure 3.3 shows an increase in the predicted use of the discourse marker you know after a short stay abroad and another increase after a long stay abroad. This increase is, however, only significant between the group that has not stayed abroad at all and the group that stayed abroad for 13 or more months (p < 0.01). Learners have been shown to highly overuse the discourse marker well (e.g. Müller 2005). Here again, the variance in the learner data is quite wide, as represented in the large confidence intervals in the right-hand panel of Figure 3.3. The model reveals similar patterns as for you know and does not indicate a significant effect of the SA variable on the learners’ use of well (F = 0.5591, df = 47, p = 0.5755, R2 = 0.023). The effect plot in the right-hand panel in Figure 3.3 illustrates a predicted decrease in the use of well after a stay abroad; however, this is only significant between the first and the third group (p < 0.0001). The variation in the learners’ use of smallwords is equally wide, which is again visualized by the confidence intervals around the datapoints in Figure 3.4, so that, again, none of the models predicts a significant effect of the SA variable on the learners’ use of smallwords. The effect size for sort of (F = 1.19, df = 47, p = 0.31) is very small (R2 = 0.05) and also for kind of (F = 2.2, df = 47, p = 0.12, R2 = 0.09); it is even smaller for quite (F = 0.53, df = 47, p = 0.593, R2 = 0.02). However, Figure 3.4 clearly illustrates a significant increase in the learners’ use of sort of and kind of from group 1 to group 3 (for both p < 0.05) in the left-hand and middle panel, as well as a slight increase in the learners’ use of quite after a short stay abroad and then again a significant decrease in the long-stay group in the right-hand panel (both, however, p > 0.05).

4.2 The effect of the study abroad variable on grammatical accuracy In order to assess the impact of a stay abroad on grammatical accuracy in the learner data, we used the error-tagged version of LINDSEI-GE and tested the effect of the SA variable on the two most error-prone subcategories in the category of grammatical errors, that is, the number of (i) verb-tense-related errors (error tag: GVT) and (ii) the number of article-related errors (error tag: GA).

Investigating the Effect of the Study Abroad Variable on Learner Output

57

Figure 3.5 The effect of the number of months spent abroad by learners in LINDSEI-GE on their predicted number of grammatical errors (verb-tense errors in the left-hand panel, article errors in the right-hand panel).

For the total number of GVT errors, the model does not predict a significant effect of the SA variable on the learners’ performance very well (F = 0.241, df = 47, p = 0.787), which is also shown in very small effect sizes for GVT (R2 = 0.032). This might again be due to the great variation within the three groups, as is clearly visible in the left-hand panel in Figure 3.5 although we can observe a steady decrease in the number of GVT errors after a short stay and again after a long stay abroad. These findings are, however, not significant (p > 0.05). For GA errors, the model predicts the learners’ performance better (F = 4.179, df = 47, p = 0.021) and, accordingly, the effect of the SA variable shows an overall weak correlation that explains c. 11 per cent of the variance in the data (R2 = 0.1149). The right-hand panel of Figure 3.5 illustrates a significant decrease in the number of article errors after a short stay abroad (p < 0.05). However, there is an unexpected and significant increase in the number of errors after a long stay abroad (p < 0.01).

4.3 The effect of the study abroad variable on lexical development Finally, we tested the effect of the SA variable on the lexical development of the LINDSEI-GE learners by looking at their overall performance and the two most error-prone lexical error categories. In particular, we tested its effect on (i) the length of the interviews, (ii) the errors in the category ‘lexical single’ (error tag: LS) and (iii) the errors in the ‘lexical phrase/false friend’ category (error tag: LPF).

58

Learner Corpus Research

Figure 3.6 The effect of the number of months spent abroad by learners in LINDSEI-GE on their predicted lexical development (total number of words per interview in the left-hand panel, lexical single errors in the middle and lexical phrase/false friend errors in the right-hand panel).

Again, a linear model was constructed with the stay abroad in months (0 vs. 1–12 vs. 13–30) as the independent variable and the length of the interview in words as the dependent variable. The data produced by this model were significant (F = 3.423, df = 47, p = 0.041, R2 = 0.1149) and showed that there is a significant effect of the time spent abroad on the number of words learners uttered in the interviews, which is visualized in the left-hand panel of Figure 3.6. For LS errors, however, the model did not show a significant effect of the SA variable on the number of LS errors (F = 1.79, df = 47, p = 0.178, R2 = 0.031), which is probably due to the great variance within the individual groups (note, once again, the large confidence intervals), although a steady decrease in LS errors is clearly visible in the middle panel of Figure 3.6. A similar trend appears in the right-hand panel for LPF errors, which steadily (but mildly) decrease from group 1 to group 3. However, the model does not show a significant correlation between the SA variable and the LPF errors (F = 0.213, df = 47, p = 0.809, R2 = 0.009).

4.4 Discussion As far as the effect of the SA variable is concerned, the German learners show an improved performance for all the variables we investigated after a stay abroad. The positive effect of the SA variable seems to be most clearly visible for temporal fluency variables, as the groups seem to be more homogeneous and all of the learners show a lower number of UPs, a longer mean length of runs and a higher speech rate. This finding corroborates previous observations with regard to the positive effect of a stay abroad on learners’ overall fluency (see Section 2).

Investigating the Effect of the Study Abroad Variable on Learner Output

59

For FES, we also found an overall positive effect of the SA variable, but it was not as clearly pronounced as for temporal fluency. On the whole, the learners use the discourse markers you know and like and the smallwords sort of and kind of more frequently after a stay abroad, while, at the same time, showing a less frequent use of the discourse marker well. The decrease in the use of the discourse marker well is a welcome development as it has been shown to be generally overused and functionally overgeneralized in the discourse pragmatics of (German) learners of English (e.g. Müller 2005). There is, however, great variation within the three groups, which makes it difficult to make robust predictions of the effect of the SA variable. Rather, this seems to point towards different preferences of the individual learners for certain FES and a low degree of homogeneity with regard to ad-hoc strategies (see also Götz 2013). This observation vindicates the necessity to look carefully at individual learners’ use of the foreign language in a learner corpus and ‘individualize’, as it were, the analysis (see Mukherjee and Rohrbach 2006). As for grammatical accuracy, the effect of the SA variable does not seem to be straightforwardly clear: on the one hand, we observe a decrease in the learners’ most frequent error category of verb-tense related errors after a stay abroad, but it is clearly less pronounced than one might have expected. For article errors, we even found an increase in the number of article errors after a long stay abroad, which calls for a more qualitative error analysis including the proportion of accurate versus inaccurate uses in future research (e.g. Díez-Bedmar 2015). It may well be that in the analysis of grammatical accuracy, there needs to be a focus on the kind and quality of interaction (including, potentially, native speakers’ feedback on errors) rather than the duration of stay as such (see Section 2). As far as the learners’ lexical development is concerned, the SA variable seems to have a positive effect on the learners’ overall productivity and ease of speaking (represented by the increase in the total number of words uttered in the interviews). Yet, although there is a steady decrease in the number of errors in the two most frequent lexical error categories, this effect did not prove to be significant. Our study thus reveals that a stay abroad may also have overall positive effects on the lexical dimension of learner output. The overall effect sizes of the SA variable which we report for the data range somewhere between small and medium (see Cohen 1988), which means that they can explain ‘only’ between 10 and 25 per cent of the variability in the learner output. However, Plonsky and Oswald (2014) stress that each study needs to

60

Learner Corpus Research

consider the context of the data and the investigated variables. Thus, and despite the overall comparatively low values, we interpret our findings fairly positively especially because learner performance is known to be strongly correlated with a multitude of factors including variables that are very difficult to measure in LCR, such as motivation, culture-sensitivity, personality, persistence or even the mood on the day of the interview, to name but a few (e.g. Hernández 2010; Mukherjee and Götz 2015). Hence, we do feel that the monofactorial models on the effect of the SA variable already reveal highly interesting trends and definitely show a generally positive effect of a stay abroad on almost all of the investigated output variables. We are therefore confident that future multifactorial models in LCR which include more details from learner profiles will be able to provide a more differentiated view on learner output and thus become a highly valuable tool for researchers wishing to combine SLA theory with LCR methodology.

4.5 Pedagogical implications of the study A stay abroad has a measurable and predictable effect on advanced learners’ performance; however, it is not always very clear or unidirectional. On the plus side, we can observe a clearly positive increase in core temporal fluency variables (see also, e.g., DeKeyser 1990; Isabelli 2001; Segalowitz and Freed 2004). This increase in fluency has been claimed to be the ‘most powerful advantage that the SA experience provides students’ (Lafford and Collentine 2006: 109). We also see a marked increase in the learners’ mean interview length that possibly reflects the learners’ comfort in speaking in the foreign language. However, a positive effect of a stay abroad on both the lexical and grammatical development of the learners is not very clearly visible. We see similarly mixed pictures in previous studies on the effect of a stay abroad (e.g. Collentine and Freed 2004; DeKeyser 1990; Kinginger 2013), so that – from research on learners’ increase in performance alone – it seems somewhat difficult to promote a stay abroad to advanced language learners as a panacea for a gain in language proficiency. This being said, not all the possible benefits of a stay abroad can be possibly measured in terms of a decrease in errors and we still consider our findings promising enough to encourage students at tertiary and secondary level to spend a period abroad in an English-speaking country. Since especially fluency and communicative abilities are becoming more and more important in a globalized world, both schools and universities should make a stay abroad in the target language community an obligatory element

Investigating the Effect of the Study Abroad Variable on Learner Output

61

of their study programmes or at least provide students with the necessary funding and scholarship opportunities to be able to take advantage of this unique learning opportunity. Since our findings mostly show that an increase in performance is visible for a short stay abroad (1–12 months), this should definitely be envisaged in secondary education. Our findings also show that long stays (12 months+) seem to have an even more beneficial effect. If time and money permit, long stays at the tertiary level, for example, for an entire MA programme, are definitely to be encouraged.

5 Conclusion and outlook In this chapter, we hope to have shown that ‘pseudo-longitudinal’ learner corpus analyses can be very fruitful and yield interesting results; in general, our analyses have revealed a positive effect of the SA variable on almost all the variables we investigated. Methodologically, pseudo-longitudinal studies can thus present a feasible alternative to truly longitudinal studies, despite the obvious caveat that is pointed out by Meunier (2015: 381), namely that ‘individual trajectories can only be assessed indirectly in quasi- or pseudo-longitudinal studies’. This notwithstanding, pseudo-longitudinal corpus-based studies starting off from the meta-information included in learner corpora offer a useful and viable methodological shortcut, as it were. More specifically, our study shows that the inclusion of learning context variables in learner corpus studies offers a new and valuable tool in the toolbox of learner corpus researchers and SLA researchers alike. The present study is, however, clearly limited by its narrow focus on (i) one learning context variable only and (ii) one learner group made up of advanced German learners of English only. The avenues for future research thus seem extremely wide: with large spoken and written learner corpora available for learners with various L1s that offer a multitude of meta-information in learner profiles, such as the International Corpus of Leaner English, the LINDSEI and the International Corpus Network of Asian Learners of English (ICNALE; Ishikawa 2013) in combination with advanced statistical techniques that have been developed over the past few years (e.g. Gries 2013b), LCR can prove to have a huge potential of putting SLA theories on a more empirical footing by testing the effect of a variety of learning context variables on large amounts of data from learners with various L1 backgrounds.

62

Learner Corpus Research

Acknowledgement We would like to thank Christoph Wolk for statistical advice and Rosemary Bock for proofreading an earlier version of this manuscript.

Note 1 As kindly pointed out by one of the reviewers, we acknowledge the fact that there are definitely methodological disadvantages to transforming a continuous variable (months abroad) into a categorical variable with three categories (no stay, 1–12, 13–60) as this will certainly entail a loss of information and, ultimately, also a loss of power of the models. However, theoretically and conceptually, we consider the three chosen categories much more authentic regarding the reality of students who will consider to either not stay abroad at all, or do so for a short period up to a year or longer than that.

References Abe, M. and Tono, Y. (2005), ‘Variations in L2 spoken and written English: Investigating patterns of grammatical errors across proficiency levels’, in Proceedings from the Corpus Linguistics Conference Series, Birmingham, UK, 14–17 July. Available at: http://www.birmingham.ac.uk/research/activity/corpus/publications/conferencearchives/2005-conf-e-journal.aspx (accessed 27 July 2015). Barron, A. (2002), Acquisition in Interlanguage Pragmatics: Learning how to Do Things with Words in a Study Abroad Context, Amsterdam: John Benjamins. Brand, C. and Kämmerer, S. (2006), ‘The Louvain international database of spoken English interlanguage (LINDSEI): Compiling the German component’, in S. Braun, K. Kohn and J. Mukherjee (eds), Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods, 127–40, Frankfurt/Main: Peter Lang. Callies, M. (2009), Information Highlighting in Advanced Learner English. The SyntaxPragmatics Interface in Second Language Acquisition, Amsterdam: John Benjamins. Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd edn, Hillsdale, NJ: Erlbaum. Collentine, J. (2004), ‘The effects of learning contexts on morphosyntactic and lexical development’, Studies in Second Language Acquisition, 26(2): 227–48. Collentine J. and Freed, B. (2004), ‘Learning context and its effects on second language acquisition: Introduction’, Studies in Second Language Acquisition, 26(2): 153–72.

Investigating the Effect of the Study Abroad Variable on Learner Output

63

Dagneaux, E., Denness, S., Granger, S., Meunier, F., Neff, J. and Thewissen, J. (2005), Error Tagging Manual, Version 1.2. Louvain-la-Neuve: Université catholique de Louvain. DeKeyser, R. (1990), ‘From learning to acquisition? Monitoring in the classroom and abroad’, Hispania, 73: 238–47. Diáz-Campos, M. (2004), ‘Context of learning in the acquisition of Spanish second language phonology’, Studies in Second Language Acquisition, 26(2): 249–73. Dietrich, R., Klein, W. and Noyau, C. (1995), The Acquisition of Temporality in a Second Language, Amsterdam: John Benjamins. Díez-Bedmar, M. B. (2015), ‘Article use and criterial features in Spanish EFL writing: A pilot study from CEFR A2 to B2 levels’, in M. Callies and S. Götz (eds), Learner Corpora in Language Testing and Assessment, 163–90, Amsterdam: John Benjamins. Ellis, R. and Barkhuizen, G. (2005), Analysing Learner Language, Oxford: Oxford University Press. Freed, B. (1995), ‘What makes us think that students who study abroad become fluent?’, in B. F. Freed (ed.), Second Language Acquisition in a Study Abroad Context, 123–48, Amsterdam: John Benjamins. Freed, B. F., Segalowitz, N. and Dewey, D. P. (2004), ‘Context of learning and second language fluency in French: Comparing regular classroom, study abroad, and intensive domestic immersion programs’, Studies in Second Language Acquisition, 26(2): 275–301. Gilquin, G., De Cock, S. and Granger, S. (2010), Louvain International Database of Spoken English Interlanguage. Handbook and CD-ROM. Louvain-la-Neuve: Presses universitaires de Louvain. Götz, S. (2013), Fluency in Native and Nonnative English Speech, Amsterdam: John Benjamins. Götz, S. (2015), ‘Tense and aspect errors in spoken learner language: Implications for language testing and assessment’, in M. Callies and S. Götz (eds), Learner Corpora in Language Testing and Assessment, 191–215, Amsterdam: John Benjamins. Granger, S., Dagneaux, E., Meunier, F. and Paquot, M. (2009), The International Corpus of Learner English. Version 2. Handbook and CD-Rom. Louvain-la-Neuve: Presses Universitaires de Louvain. Gries, S. Th. (2013a), Statistics for Linguistics with R: A Practical Introduction, 2nd rev. and ext. edn, Berlin and New York: De Gruyter Mouton. Gries, S. Th. (2013b), ‘Statistical tests for the analysis of learner corpus data’, in A. DíazNegrillo, N. Ballier, and P. Thompson (eds), Automatic Treatment and Analysis of Learner Corpus Data, 287–309, Amsterdam: John Benjamins. Hernández, T. A. (2010), ‘The relationship among motivation, interaction, and the development of second language oral proficiency in a study-abroad context’, The Modern Language Journal, 94(4): 600–17.

64

Learner Corpus Research

Isabelli, C. L. (2001), ‘Motivation and extended interaction in the study abroad context: Factors in the development of Spanish language accuracy and communication skills’. Ph.D. dissertation. University of Texas at Austin. Ishikawa, S. (2013), ‘The ICNALE and sophisticated contrastive interlanguage analysis of Asian Learners of English’, in S. Ishikawa (ed.), Learner Corpus Studies in Asia and the World 1, 91–118, Kobe, Japan: Kobe University. Johnson, K. and Johnson, H. eds (1999), Encyclopedic Dictionary of Applied Linguistics, Blackwell Publishing. Blackwell Reference online. Kämmerer, S. (2012), ‘Interference in advanced English interlanguage: Scope, detectability and dependency’, in J. Thomas and A. Boulton (eds), Input, Process and Product: Developments in Teaching and Language Corpora, 284–97, Brno: Masaryk University Press. Kinginger, C. (2013), ‘Identity and language learning in study abroad’, Foreign Language Annals, 46(3): 339–58. Lafford, B. A. (2004), ‘The effect of the context of learning on the use of communication strategies by learners of Spanish as a second language’, Studies in Second Language Acquisition, 26(2): 201–25. Lafford, B. A. and Collentine, J. (2006), ‘The effects of study abroad and classroom contexts on the acquisition of Spanish as a second language. From research to application’, in R. Salaberry and B. A. Lafford (eds), The Art of Teaching Spanish: Second Language Acquisition from Research to Praxis, 103–26, Georgetown: Georgetown University Press. Lennon, P. (1990), ‘Investigating fluency in EFL: A quantitative approach’, Language Learning, 40(3): 387–417. Maden-Weinberger, U. (2015), ‘A pseudo-longitudinal study of subjunctives in the Corpus of Learner German’, International Journal of Learner Corpus Research, 1(1): 25–57. Meunier, F. (2015), ‘Developmental patterns in learner corpora’, in S. Granger, G. Gilquin and F. Meunier (eds), Cambridge Handbook of Learner Corpus Research, 378–400, Cambridge: Cambridge University Press. Mora, J. C. and Valls-Ferrer, M. (2012), ‘Oral fluency, accuracy, and complexity in formal instruction and study abroad learning contexts’, TESOL Quarterly, 46(4): 610–41. Mukherjee, J. and Götz, S. (2015), ‘Learner corpora and learning context’, in S. Granger, G. Gilquin and F. Meunier (eds), Cambridge Handbook of Learner Corpus Research, 423–42, Cambridge: Cambridge University Press. Mukherjee, J. and Rohrbach, J.-M. (2006), ‘Rethinking applied corpus linguistics from a language-pedagogical perspective: New departures in learner corpus research’, in B. Kettemann and G. Marko (eds), Planing, Gluing and Painting Corpora: Inside the Applied Corpus Linguist’s Workshop, 205–32, Frankfurt am Main: Peter Lang. Müller, S. (2005), Discourse Markers in Native and Non-native English Discourse, Amsterdam: John Benjamins.

Investigating the Effect of the Study Abroad Variable on Learner Output

65

Norris, J. M. and Ortega, L. (2001), ‘Does type of instruction make a difference? Substantive findings from a meta-analytic review’, Language Learning, 51(s1): 157–213. Ortega, L. and Byrnes, H. (2008), ‘The longitudinal study of advanced L2 capacities: An introduction’, in L. Ortega and H. Byrnes (eds), The Longitudinal Study of Advanced L2 Capacities, 3–20, New York: Routledge/Taylor and Francis. Plonsky, L. and Oswald, F. L. (2014), ‘How big is “big”? Interpreting effect sizes in L2 research’, Language Learning, 64(4): 878–912. R Development Core Team (2013), R: A Language and Environment for Statistical Computing. Foundation for Statistical Computing. Vienna, Austria. http://R-project. org (accessed 13 March 2015) Sanz, C. (2014), ‘Contributions of study abroad research to our understanding of SLA processes and outcomes’, in C. Pérez-Dival (ed.), Language Acquisition in Study Abroad and Formal Instruction Contexts, 1–13, Amsterdam: John Benjamins. Segalowitz, N. and Freed, B. (2004), ‘Context, contact and cognition in oral fluency acquisition: Learning Spanish in at home and study abroad contexts’, Studies in Second Language Acquisition, 26(2): 173–99. Thomas, M. (1994), ‘Assessment of L2 proficiency in second language acquisition research’, Language Learning, 44(2): 307–36.

Part Two

Analysis of Learner Language

4

Disagreement in L2 spoken English: From learner corpus research to corpus-based teaching materials Dana Gablasova and Vaclav Brezina

1 Introduction In order to communicate successfully, speakers need to master both linguistic (lexico-grammatical) and social aspects of the interaction. While lexicogrammatical features have received sufficient attention in traditional pedagogy, the more subtle social aspects of communication are often overlooked. It is therefore encouraging that in recent years there has been more focus on developing sociopragmatic skills in language classrooms (e.g. Ishihara and Cohen 2014; Rose and Kasper 2001; Taguchi 2011). Despite these positive developments, relatively little attention to date has been given to the so-called dispreferred speech acts such as complaints, refusals or expressions of disagreement that are potentially face-threatening and socially disruptive (Glaser 2009). This chapter explores disagreement because it represents a recurring component of everyday discourse and it is a speech act that plays an important role in expressing opinions and defending standpoints. In particular, the chapter focuses on disagreeing in interactive oral communication by L2 speakers of English. It explores a specific construction for expressing disagreement, the so-called agreement-plus-disagreement construction (Pomerantz 1984), sometimes also referred to as ‘yes but’ constructions (e.g. Kreutel 2007; Pekarek Doehler and Pochon-Berger 2011). The chapter has two interconnected aims. First, it aims at a better understanding of communicative patterns of L2 speakers of different proficiency levels when expressing disagreement. In this regard, the study seeks to complement

70

Learner Corpus Research

previous (mostly experimental and classroom-based) studies on disagreeing in L2 by analysing data from a large corpus of L2 spoken production, the TLC. The second objective of the study is to illustrate how a learner corpus can be used in designing activities and materials for teaching of specific speech acts.

2 Background 2.1 Disagreement as a speech act Disagreement is a speech act that is common in everyday communication (Angouri and Locher 2012); it is a relational action which operates at both the content and the social level (Leech 1983; Locher 2004; Rees-Miller 2000; Sifianou 2012). Disagreement at the content level is relatively straightforward and can be defined as ‘an oppositional stance (verbal or non-verbal) to an antecedent verbal (or non-verbal) action’ (Kakavá 1993: 36). On the social level, disagreement represents a potentially face-threatening act and requires careful linguistic handling. The social threat comes from the fact that in expressing an opposing view we may challenge the accuracy or credibility of the previous speaker (Kreutel 2007). This makes disagreement a complex act with the need to carefully manage (soften) the perceived threat to the hearer’s face by various linguistic means in order to maintain a positive interlocutor relationship and to prevent communication breakdown. Arguably, more effort is required in realizing ‘dispreferred’ speech acts, that is, acts that include a potential threat to one of the interlocutors, than ‘preferred’ speech acts, that is, acts that do not contain such threat. Contrary to what is often assumed, disagreement can be seen as a highly cooperative act (Schiffrin 1990), a trait shared with conflict talk in general. For example, Goodwin and Harness Goodwin (1990: 85), talking about arguments and oppositional talk, state that although this type of interaction is ‘frequently treated as disruptive behaviour, it is in fact accomplished through a process of very intricate coordination between the parties who are opposing each other’. It is precisely this complex social and linguistic coordination that makes disagreement a real challenge for L2 speakers of English. So how is this coordination achieved? Disagreement in speech can be expressed by a variety of linguistic means. Pomerantz (1984) in her influential overview of the language involved in agreeing and disagreeing identified several types of disagreement ranging in strength from strong (direct) to weak

Disagreement in L2 spoken English

71

(mitigated). Strong disagreement is characterized by inclusion of an explicit disagreement token (e.g. ‘I disagree with you’) which is usually not mitigated further thus making disagreeing a highly face-threatening act in contexts where it is not expected. By contrast, weak disagreement still conveys the oppositional standpoint of the speaker but it simultaneously softens the potential conflict in several ways in an effort to downplay the challenge to the previous speaker. Pomerantz (1984) identifies several techniques for downtoning the intensity of opposition when disagreeing. The primary technique is to include a token of agreement along with the disagreement thus creating an appearance of a partial agreement. This type of weak disagreement has been identified as the prevalent strategy of expressing disagreement by L1 speakers of English (Kreutel 2007; Pomerantz 1984). The strength of the opposition can be mediated by means of further mitigating features. These can include hedging (i.e. expressions indicating uncertainty about the statement such as ‘maybe’ and ‘I don’t know’), explanations (giving reasons for disagreeing) and delays and hesitations which indicate speakers’ reluctance and discomfort at disagreeing. These techniques are important components of politeness strategies helping speakers to express their views without appearing rude or uncooperative. Even though in general disagreement requires careful handling due to its potentially disruptive social nature, not every context will require the same consideration for speaker’s face when expressing an opposing view (e.g. Angouri and Locher 2012; Georgakopoulou 2001; Locher 2004; Pomerantz 1984); there are settings where disagreement is part of expected discourse as in, for example, problem-solving business meetings (Marra 2012). It is important for speakers to be aware of the variety of social norms that can govern different situations in which they are called to express their opinions and engage in communication; it is likewise important for the researchers to consider the expressions of disagreement within the social framework of the interaction (Georgakopoulou 2001).

2.2 Disagreement in L2 speech The social and linguistic complexity involved in expressing disagreement makes it challenging to master for L2 speakers. Studies that investigated disagreement in L2 speech have reported that L2 speakers tend to express their views more directly and with less attention to mitigating the potential negative impact of their views. As a result, these speakers at times risked sounding too harsh or even rude (Beebe and Takahashi 1989; Kreutel 2007).

72

Learner Corpus Research

Research evidence suggests that language proficiency in the target language is an important factor in the politeness strategies employed by L2 speakers when expressing opposing views. Pekarek Doehler and Pochon-Berger (2011) investigated disagreement sequences in speech from learners of French of two proficiency levels (lower-intermediate and advanced). The authors reported a range of systematic differences found between disagreement strategies of the two groups. Disagreement constructions from the lower-level speakers were characterized by a simpler structure (e.g. the turn contained a lower number of delaying features), a narrow choice of lexical mitigators (no or very few hedges) and an absence of elaborations; by contrast, the advanced learners employed a more complex turn architecture (both turn-initial and non-turn-initial) and considerably richer lexical elaboration which included regular use of hedges and a range of devices marking disagreement. The advanced L2 speakers also used further mitigating strategies such as giving explanations for disagreement or giving examples to support their viewpoints. The trend characterized by an increased range and complexity of disagreement mitigating strategies has been identified also in other studies that investigated differences between L2 speakers at lower and higher level of proficiency (e.g. Bardovi-Harlig and Salsbury 2004; Maíz-Arévalo 2014; Walkinshaw 2009). Despite using more complex disagreement constructions than lowerproficiency speakers, advanced L2 users were still found employing strategies that could be considered too direct with regard to the target sociolinguistic practices. Both Kreutel (2007) and Flores-Ferrán and Lovejoy (2015) reported that compared to a group of native speakers of the language, advanced L2 users applied a more limited range of strategies to downplay the potentially negative effect of disagreement. Moreover, Kreutel (2007) noted that the L2 speakers were more likely than L1 speakers to abandon the opposite standpoint rather than disagree with their interlocutors. A similar observation was made by Walkinshaw (2007) about Japanese speakers of English who were found to be reluctant to disagree with people of (perceived) higher status and who avoided disagreeing in the interactions that involved such interlocutors. Kreutel (2007) attributed this behaviour to the linguistic and social demands of formulating disagreement in speakers’ non-native language because the research participants stated that they would express disagreement in these situations if they were speaking in their L1. Politeness strategies used with disagreement by L2 speakers appear to be strongly related to their proficiency in the target language but the relationship is not completely straightforward. While Pekarek Doehler and Pochon-Berger (2011) attributed the difference found between the two groups of participants in the study mainly to the speakers’ proficiency in French, the authors also

Disagreement in L2 spoken English

73

discussed the possibility that age and maturity of speakers could play a role; all of the speakers in their study came from the same educational system with the lower-level speakers being three to four years younger than the advanced speakers. Others found that L2 speakers’ politeness strategies when disagreeing were also affected by the status of the second interlocutor (Hüttner 2014; Walkinshaw 2007). In general, there appears to be a complex relationship between speakers’ lexical and grammatical proficiency in L2 and their pragmatic skills (e.g. Bardovi-Harlig and Dörnyei 1998; Kasper 2001).

2.3 Research questions The majority of studies dealing with disagreeing in L2 have so far focused mainly on describing different ways in which opposing views have been expressed by L2 speakers of different proficiency; little is known, however, about the effect of proficiency on the selection of the conflict-mitigating features that accompany disagreement. This study focuses on one disagreement construction, the ‘yes-but’ construction. This construction already represents a weaker form of disagreement in which the negative social impact is softened by a partial agreement with the previous interlocutor. This construction is common in oppositional talk among L1 speakers of English (Pomerantz 1984) and has been found in speech of both lower level and advanced L2 users of English (Pekarek Doehler and Pochon-Berger, 2011). Focusing on a particular construction will allow us to observe possible proficiency-related development in the mitigating features accompanying expressions of disagreement. The study addresses the following two research questions. RQ1: How often is the explicit ‘yes-but’ construction used by L2 speakers at each proficiency level? RQ2: Is there a difference in the disagreement strategies used by L2 speakers of three different proficiency levels with the ‘yes-but’ construction?

3 Method 3.1 Corpus The data for this study were taken from the TLC of L2 production which at present contains over 4 million running words (Gablasova, Brezina, McEnery and Boyd, 2015). The corpus contains transcripts from the GESE conducted by

74

Learner Corpus Research

Trinity College London, a major international examination board. The corpus contains data from over 1,900 L2 speakers of English recorded in interaction with examiners (L1 speakers of English) in up to four different speaking tasks. The linguistic and cultural backgrounds of the speakers vary with the majority coming from the following countries: Argentina, Brazil, China, India, Italy, Mexico, Russia, Spain and Sri Lanka. The L2 speakers in the corpus represent three broad proficiency levels: lower-intermediate (B1 level of CEFR), intermediate (B2) and advanced (C1 and C2) level of English proficiency. The proficiency level of the speakers was established using the marks which they received in the GESE exam. Only candidates who achieved at least a ‘pass’ in their exam were included in the corpus to ensure that they fulfilled the minimal requirements for the particular proficiency level. This study draws on a dialogic sub-corpus of the TLC which consists of the spoken production from 1,449 speakers engaged in two interactive dialogic tasks: discussion and conversation. In the discussion the candidates and examiners talk about a topic brought to the exam by the candidates; in the conversation they talk about a topic of general interest which had been preselected from a list known to the candidate before the exam. Both tasks last five to six minutes (for a more detailed description of each task, see Trinity College London, 2010). Overall, the sub-corpus used in the study contains over a million words, representing the three proficiency levels described above. A more detailed overview of the subcorpus can be seen in Table 4.1 Corpus-based studies of learner and L2 user language often employ a reference point according to which the L2 user performance can be evaluated and interpreted. It is common to use expert users of the language to establish the baseline performance; native speakers of the language or successful learners often act as the reference point (Gablasova Brezina and McEnery, 2017). This chapter compares the pragmatic strategies across different levels of proficiency Table 4.1 Overview of the corpus used in the study Speaker age L2 proficiency

Sub-corpus size (words)

No of speakers

Mean

SD

Lower-intermediate

460,012

597

20.0

10.9

Intermediate

573,443

581

20.2

10.1

Advanced

308,906

271

25.0

10.3

Total

1,342,361

1449

Disagreement in L2 spoken English

75

with the speakers from the highest proficiency band considered as the most successful users in the context of this chapter.

3.2 Procedure To identify the target constructions, the following steps were taken. First, MonoConc Pro (Barlow 2003) was used to search the dialogic sub-corpus to find all instances of ‘but’ used in the candidates’ discourse. These occurrences were sorted according to the right and left context (1R and 1L) and then manually checked to identify expressions that explicitly signal agreement. The following expressions (search terms) were identified (the @ symbol represents 0–1 word and allows for a partial modification of the construction): yes @ but; yeah @ but; okay but; your point but; you @ right but; true but; agree @ but. The examples below illustrate some of the target constructions found with ‘yeah @ but’ and ‘yes @ but’: (1) yeah but we are not all great like them like … (2) yeah erm but for example here in Spain er the king is … (3) yes but I don’t know be an the person who feed them …

As the next step, each sub-corpus representing speech from one of the three proficiency levels was searched using the seven search terms listed above. Three lists (one for each proficiency level) containing the target construction were produced. Finally, all of the occurrences were coded manually and only the instances in which the ‘yes-but’ construction was used to express disagreement with the view of the previous speaker were retained for further analysis. The ‘yes-but’ constructions excluded from the analysis were usually used in the following three ways: (1) they were a reply to a question and thus they were not deemed to truly represent contrastive views, (2) they were an instance of noninterpersonal use where the speaker contrasted two of his/her own ideas and (3) they consisted of instances where the function of the construction was unclear as the speaker abandoned the formulation of the message or rephrased it. These cases are illustrated in the examples below (E = examiner, C = exam candidate): (1) E: So why would you be worried then? C: Yeah but there are some disadvantages like er … [topic: the use of internet and the dangers when children go online] (2) C: Yeah, I think so erm well here in in Spain is V of V for Vendetta yeah but not got not for Vendetta it’s curious …

76

Learner Corpus Research

[topic: talking about a specific movie] C: Pollution is a big big problem okay but erm we citizens we don’t do anything. [topic: pollution in Mexico City] (3) E: Because don’t you think erm it might be possible to get a good job? Without going without to university? C: Er yes okay but I also f= er would we you can even er make a career or you can go for … [topic: whether education is necessary for getting a good job]

3.3 Data coding Following Pomerantz’s (1984) description of mitigating features used in disagreement constructions as well as the coding scheme in Pekarek Doehler and Pochon-Berger (2011) the following categories were employed for coding the data: (1) The use of a lexical downtoner, for example, a hedge such as mental verb or epistemic adverbial indicating uncertainty (e.g. ‘I think’, ‘I don’t know’ or ‘maybe’). (2) The use of a delay/hesitation marker. These included the hesitation markers such as ‘er’, ‘erm’, ‘mm’, ‘uh’ or unfilled pause (), laughter, repetition of (a part of) previous speaker’s utterance and a delay marker ‘well’.

4 Results 4.1 RQ1: How often is the explicit ‘yes-but’ construction used by L2 speakers at each proficiency level? Table 4.2 shows the frequency of expressions identified at each stage of the procedure, displaying both the absolute frequencies (Freq.) and frequencies normalized to 100,000 words (NF). As can be seen from the table, as speakers’ English proficiency increased, there was also a rising number of constructions involving contrastive stance expressed by ‘but’ as well as an increasing number of ‘yes-but’ constructions overall, and those used to disagree directly with the previous speaker. Table 4.3 shows how many instances of each expression were found in the speech of speakers from each proficiency level. The percentage shows the

Disagreement in L2 spoken English

77

Table 4.2 The frequency of searched expressions ‘Yes-but’ constructions (total occurrence)

Occurrence of ‘but’

‘Yes-but’ constructions (interpersonal opposition)

L2 proficiency

Freq.

NF

Freq.

NF

Freq.

NF

Lower-interm.

3866

840.4

264

57.4

90

19.6

Intermediate

5498

958.8

368

64.2

171

29.8

Advanced

3405

1102.3

273

88.4

181

58.6

Total

12769

951.2

905

67.4

442

32.9

proportion of each expression out of all constructions found for the given proficiency level. As can be seen from the table, with increasing proficiency the range of expressions signalling agreement in the ‘yes-but’ construction increases moderately. While at the lower-intermediate level most of the agreement tokens are realized either by ‘yes’ or ‘yeah’, the advanced speakers use a somewhat wider range, with expressions such as ‘it’s true but’ occurring more prominently. In addition, a closer analysis of the data showed that increased proficiency is also linked to a greater variety of modification of individual expressions. The following examples illustrate the variation in the expression ‘agree @ but’ used by intermediate and advanced speakers. The examples also demonstrate that the

Table 4.3 The range of agreement markers Lower-intermediate Search terms

Freq.

%

Intermediate Freq.

%

Advanced Freq.

%

yes @ but

62

68.9

98

57.3

66

36.5

yeah @ but

25

27.8

63

36.8

84

46.4

okay but

2

2.2

6

3.5

4

2.2

see your point but

–

–

–

–

5

2.8

you @ right but

1

1.1

–

–

–

–

true but

–

–

2

1.2

19

10.5

agree @ but

–

–

2

1.2

4

2.2

Total

90

100.0

171

100.0

181

100.0

78

Learner Corpus Research

disagreements expressed by these speakers show a greater complexity as they combine multiple strategies (e.g. lexical downtoners, delays and hesitations): (1) yeah I agree but I still the problem is that … [topic: AIDS still being a taboo in society] (2) yes erm I do agree but erm okay I would just like to ask you … [topic: the choice of university subjects and why people decide to study a particular subject] (3) yeah I completely agree but I think erm maybe in that cases or … [topic: whether religion is still as influential today as it used to be] (4) I couldn’t agree more but I think that in er Spain there are … [topic: recycling practices in Spain]

4.2 RQ2: Is there a difference in the disagreement strategies used by L2 speakers of different proficiency levels with the ‘yes-but’ construction? First, let us look at the use of downtoners mitigating the ‘yes-but’ constructions in each sub-corpus. Table 4.4 shows how many target constructions were accompanied by at least one downtoner. The table also shows the proportion (percentage) of the constructions accompanied by a downtoner out of all ‘yesbut’ constructions for each proficiency level and the range of downtoners used.

Table 4.4 Use of lexical downtoners L2 proficiency

Constructions with downtoners

Range of downtoners

Freq.

%

Lower-int.

11

12.2

Intermediate

27

15.8

I think (15), maybe (4), I don’t know (3), I mean (2), I’m not sure (1), probably (1), perhaps (1), I find (1), it depends (1)

Advanced

54

29.8

I think (31), maybe (6), I know (3), it depends (3), I mean (3), actually (2), in my opinion (2), I suppose (1), I don’t think (1), that would depend (1), I feel (1), probably (1), you are never sure (1), it seems (1), it may be (1)

I think (11)

Disagreement in L2 spoken English

79

Table 4.5 Use of hesitation and delay markers Constructions with hesitation & delay markers L2 proficiency

Freq.

%

Lower-int.

27

30.0

Intermediate

56

32.7

Advanced

66

36.5

As is apparent from the table, as speakers’ proficiency increases they are more likely to use a lexical downtoner as part of expressing an opposite viewpoint. The difference between speakers of different English proficiency levels was statistically significant (χ2 = 15.59, p to see it really er how it works RM IP IM RR

Henry (2002a) proposes a more complex classification, in which she describes a number of possible combinations. According to Henry, there are two basic constituents of any repetition. The first of them, the repeatable (‘le répétable’), represents any part of an utterance that can potentially be repeated, and the latter, the repeated (‘le répété’), is any repeatable that was actually reproduced. Since these two elements can be combined with other linguistic devices, such as (un)filled pauses, or other words, the structure of a repetition can be quite complex. Table 5.1 shows the combinations described by Henry (6). In this case, a multiple repetition could be described in the following way: what subjects do you need to to to aah to take R0 R1 R2 (+fp) R3

92

Learner Corpus Research Table 5.1 Repetitions classified according to their structure Compound

Simple Multiple

Direct

+ unfilled pause (up)

+ filled pause (fp)

R0 R1

R0 + R1

e.g. R0 er er R1 R0 w R1

R0 R1 R2 R3 … Rn

+ word (w)

R0 (+ or fp or w) R1 (+ or fp or w) R2

Note that R0 stands for ‘the repeatable’, while R1,2, 3 ... stand for ‘the repeated’. Source: Translated from Henry (2002a: 6).

3 Classification of repetitions In the research on the nature of repetition, two major ways of classifying the occurrences of this phenomenon can be distinguished. The first one is based on the structural description of repetitions. One noteworthy example is work by Henry (2002a,b), where the author develops the aforementioned structure-based classification and provides labels for the subcategories of identical sequences and non-identical sequences. The repetitions based on identical sequences can fall into two categories, namely ‘contiguous’ or ‘non-contiguous’ ones. In this case, the difference lies in the time that elapses between the production of ‘the repeatable’ and ‘the repeated’; any non-linguistic interruptions, such as unfilled pauses, occurring between the two constituents make the repetition ‘non-contiguous’ (Henry 2002b: 6–7). In terms of non-identical sequences, three major types identified by Henry (2002b) are: repetitions combined with unfinished words, repairs and embedded repetitions. Some English examples, based on the original French version (7–8), are listed below. The repetitions are marked in italics, while the accompanying structures are underlined: ●

● ●

repetitions combined with unfinished words: ‘the mos- the most important person…’ repairs: ‘like an like a boss’ embedded repetitions – the element inserted between the repeatable and the repeated becomes the new repeated: ‘the the garden the garden in front of the house…’

While this structural approach, with a finite number of definable labels, seems to provide a robust framework for analysing spoken repetitions, the other,

Self-repetitions in learners’ spoken language: A corpus-based study

93

pragmatically oriented way of classifying occurrences of this phenomenon is much more challenging. Since it is based on the overlapping pragmatic functions of these linguistic devices, few precise boundaries can be drawn in this case. This problem was described by Tannen (1989: 55) who stated that ‘it would be hubris (and hopeless) to attempt to illustrate every form and function of a repetition’. Despite the limitations, a number of studies have been designed to reflect the pragmatically oriented approach. One of such classifications was proposed by Denke (2005), who devised four major categories, that is, hesitational repetition, rhetorical repetition, return to the previous construction and repetition due to involvement, illustrated below. This classification seems to be based on the meaning of a repetition in context, as Denke was interested in the difference between the repetitions that carry some kind of meaning and those which are ‘semantically non-significant’ (Blankenship and Kay 1964). While the ‘non-significant’ hesitational repetitions are easy to define, since they constitute a single type of performance effects, for example, ‘so we brea we we calculate ehm’ (Denke 2005: 183), the other types have more complex definitions (186–91), which could potentially overlap. The following extracts serve to exemplify and define the remaining categories (186–90). ●

●

●

rhetorical repetition – its purpose is to stress the importance of a given piece of information, for example, ‘it’s very very hard to predict’ or ‘for years and years and years and years’ return to the previous construction – ‘repetitions of previously uttered lexical items after a sidetrack of some kind has been pursued’, for example, ‘cause if we have when we go through the method if we have DNA…’ repetition due to involvement – ‘brought about by speaker’s excitement or agitation about the topic at hand’, for example, ‘come on come on come on (laughter from the audience) I want to do a () can you (inaud. + laughter)’

Since Denke’s (2005: 33) research focused chiefly on monologic performance, her labels were not designed to embrace the complexity of more interactive communication. However, such an attempt was made by Kjellmer (2008) who used a much more fine-grained analysis. Kjellmer compiled the following list of the functions of spoken repetitions (46–55): ● ● ●

stuttering structuring – marking transitions within the discourse thinking time:

94

Learner Corpus Research − introducing conversational turns − hesitation/uncertainty – notably stalling, that is, planning ahead − reduction of processing effort – used in order to facilitate ‘the hearer’s understanding’

●

●

●

attention marker: − emphasis − attention calling: • to oneself • to something important turn handling: − turn taking − turn holding − turn yielding change marker: − change of tack – the point where the speaker decides to change the direction in which the conversation is heading − self-repair

4 Quantitative research on repetition Because of the multiple ways of classifying repetitions, quantitative studies do not provide directly comparable data on this subject. Oftentimes, the only common denominator are raw counts for specific lexical items or some widely accepted labels, such as word classes. For instance, the raw count for repeated prepositions is frequently reported, while the data on the use of the repetition for turn handling is, to the best of our knowledge, available in only a few studies (see Kjellmer 2008). The most commonly repeated word classes were described by Persson (1974), who discovered that in the case of hesitational repetitions, most frequent were determiners and prepositions, while relatively few subordinators and coordinators were found. Biber et al. (1999) divided the most frequently reiterated items, which turned out to be chiefly function words, according to word classes; the most frequently used group were personal pronouns (as opposed to accusative pronouns) and possessive determiners, which tend to occur as often as the definite article. Unlike in the case of Persson’s study, the subordinating and coordinating conjunctions were found to have ‘a strong

Self-repetitions in learners’ spoken language: A corpus-based study

95

tendency to form repeats’ (Biber et al. 1999: 1058), while prepositions were much less likely to occur in such a context. Out of all the English verbs, only ‘to be’ in its third person singular form was used often enough to be counted among the most common items. Finally, the authors remark that the frequency for a given repetition decreases sharply with the increase in the number of the repeated items produced. More recent data from Denke (2005) seem to corroborate Biber et al.’s findings, with an infinitive marker to added to the list of the most common items. The author also discovered that non-native speakers of English tend to use articles, both definite and indefinite, more frequently than native speakers. Moreover, Kjellmer (2008) listed the lexical bundle it’s among the items most frequently repeated in native speaker’s English, thus shifting the focus from single words to more complex constructions. Another important aspect is the placement of a repetition in a phrase; Persson (1974), Schegloff (1987), Biber et al. (1999) and Kjellmer (2008) agree that the self-repetition tends to be located at the beginning of a conversational turn. Moreover, Persson (1974), Biber et al. (1999) and Kjellmer (2008) provide the same explanation for this regularity, namely, that the less semantically charged words help one hesitate before embarking on ‘ “heavy” classes of lexical words’ (Persson 1974: 109), or ‘major syntactic units’ (Biber et al. 1999: 1059). Denke (2005) contrasts these findings with the language used by non-native speakers, who frequently utilize hesitational repetition in post-modifying prepositional phrases, for example, ‘like a limited aah number of of of of things he can do? Or’ (example from the Polish-English Learner Corpus (PLEC); see Pęzik 2012). This observation seems to be especially interesting, since Biber et al. (1999) explicitly state that the use of prepositions in repetitions produced by native speakers is surprisingly low, given that these parts of speech meet all the aforementioned criteria concerning the context in which a repetition might occur. Biber et al. (1999: 1060) try to explain this phenomenon by hypothesizing that the preposition often forms a part of a lexical bundle, for example, ‘a lot of ’, which is retrieved as a whole – if true, it might be assumed that the learners’ use of prepositions in self-repetitions is a reflection of the unfinished process of automatization of a given structure. Another difference between native and non-native speakers of English was observed by Denke (2005), who focused on the types of self-repetitions used most frequently by the two groups. The results show that native speakers are more likely to use rhetorical repetitions and repetitions due to involvement,

96

Learner Corpus Research

although the number of occurrences in the latter category seems to be highly idiosyncratic. However, non-native speakers use mostly hesitational repetitions and returns to previous constructions, which seem to represent performance effects.

5 Repetitions in foreign English as a Foreign Language/ English as a Second Language (EFL/ESL) pedagogy The general perception of the repetition in spoken language used to be negative, focusing on communicative ineffectiveness and the redundant nature of this phenomenon (Kjellmer 2008). For instance, Tannen (1989: 53) stressed that repetitions can be perceived as ‘undesirable in conversation’, Jucker (1994: 48) suggests that they are considered to be a ‘waste of effort’ and Aitchison (1994: 27) adds that they ‘tend to be filtered out as mistakes, or else treated as iteratives’. In foreign-language pedagogy, the use of the repetition may be considered undesirable, as described in Mukherjee (2009) and Osborne (2011). Certain EFL fluency assessment measures, also known as ‘fluency markers’ (Molenda 2013: 277) might be considerably affected by the use of repetitions. One of such factors is ‘lexical density’, that is, type-token ratio which helps establish how varied one’s vocabulary is (see Fillmore 1979; Tweedy and Bayen 1998). In this case, the more tokens of a single type are produced, the worse the student’s score is. Another aspect, mentioned by Gatbonton and Segalowitz (2005: 237), is automaticity, or the ability to produce formulaic chunks of a language in a ‘smooth and rapid’ manner. This smooth flow of speech is likely to be disrupted by the use of repetitions, especially in the case of returning to previous constructions, as described by Denke (2005). However, some approaches advocate a more functional perspective, where this phenomenon is regarded as a natural element of spoken language. Merritt (1994: 27) remarks that repetition occurs in spoken form in all languages, cultures and ‘almost all situations’, while Bazanella (1996: vii) stresses the importance of this device in the language, by saying that ‘the universality and pervasiveness of [repetition] have been pointed out by many scholars, since ancient times, but, as in other “obvious” aspects of our life, we are not totally aware of using it or of the different functions we perform using it’. Finally, Kjellmer (2008: 38) states that repetition is an appealing research topic, as it encompasses the most interesting characteristics of spoken language. Those characteristics were aptly described by Hill as ‘the momentary, the individual, the performative, the disorderly’ (as quoted in Macaulay 1997: 121).

Self-repetitions in learners’ spoken language: A corpus-based study

97

In his work on the perception of oral fluency, Osborne (2011: 187) claims that certain performance effects, including repeats, cannot be clearly and unambiguously defined as ‘“normal” or undesirable features of speech’ since it is not their occurrence or lack thereof that makes a speaker appear (dys)fluent. Instead, there ought to be a given frequency beyond which such elements might be considered intrusive. The same study showed that the correlation between the assessment of learners’ proficiency and the number of repeats is not confirmed. The explanation proposed in the article is based on Kormos’s (2000: 359) findings that learners’ progress towards proficiency witnesses a decrease in the number of lexical/grammatical repairs, parallel to the increase in the quantity of self-repetitions used to introduce appropriacy repairs. Therefore, the number of repetitions which might accompany those two phenomena does not have to show a linear correlation with the learners’ progress.

6 Research aims and methods While there exists a considerable body of literature on self-repetitions in general, the issue of whether they should be considered a dysfluency marker or regarded as a natural element of learners’ spoken language still needs further investigation. Previous results suggest that the use of self-repetitions in EFL learners evolves in a complex way as they progress towards oral fluency. Therefore, it was decided to conduct research into the relationship between the self-repetition and language proficiency of EFL learners. The major aim of this study was to investigate the factors that play a role in the use of self-repetition and whether these support the interpretation of selfrepetition as a dysfluency marker. This aim was operationalized by means of the following questions: 1. 2.

Do non-native speakers of English differ in their use of self-repetitions (i.e. in choice and frequency of lexical items) from native speakers of English? Does the use of the most commonly employed repeated segments decrease with the increase in language proficiency of EFL learners?

Since a considerable number of samples of spoken language were needed to answer these questions with confidence, it was decided to use 393 transcribed and proficiency-annotated conversations (190,645 words) that comprise the spoken component of the PLEC (PLEC spoken; Pęzik 2012). This database contains recordings of Polish speakers of English with a CEFR proficiency level

98

Learner Corpus Research

ranging from A2 to C2. All the recordings in the corpus were made between 2010 and 2013, with the aim of investigating the lexico-grammatical, phraseological and phonetic competence of Polish learners of English. The data differ from other collections of learners’ performances (e.g. monologues recorded by Denke (2005)) with respect to their conversational nature, as the corpus recordings were made in order to register non-scripted interactions between at least two speakers of EFL. The spoken component of the British National Corpus (BNC spoken) was used as a reference corpus in this study. With 10,334,947 recorded words, it constitutes a large, balanced dataset of the target variety of English; learners of English in Poland are generally trained to use the British variety of English.

7 Research procedure First, both PLEC spoken and BNC spoken were searched by means of an SQL query whose purpose was to retrieve the reiterated elements. We focused on immediate repetitions. The repetitions were then ranked according to their frequency of occurrence and the results were manually searched for false positives. As a result of this procedure, a total of 104 most frequent repeatable segments were identified and subjected to further analysis. The next step in the process was the statistical comparison of the total frequency of occurrence of the items in question in PLEC spoken and BNC spoken. The chi-square test was used in order to perform this task; the analysis was conducted according to the procedure described by Rayson, Leech and Hodges (1997). This stage was followed by qualitative analysis of the selected self-repetitions. The items subjected to this procedure were selected on the basis of the potential pedagogical relevance of the expected results. To answer the second research question a cross-sectional study was designed, whose aim was to compare the distribution of self-repetitions across different levels of learners’ proficiency. Our hypothesis (H1) was that the number of selfrepetitions can be used in order to predict the level of EFL proficiency. Conversely, H0 stated that there is no statistically significant relationship between the level of learners’ proficiency and the number of self-repetitions used. Linear regression was used to establish statistical significance.

Self-repetitions in learners’ spoken language: A corpus-based study

99

Table 5.2 The number of words in the spoken component of PLEC, arranged according to CEFR language proficiency levels CEFR Level

Words

A2

10,992

B1

19,181

B2

14,921

C1

53,817

C2

29,211

The size of the samples for each CEFR level of language proficiency is presented in Table 5.2. We excluded the cases where establishing the level of proficiency was problematic and the label indicated that a learner is in between two CEFR levels. Each sample was analysed for the occurrences of the five most frequent selfrepetitions and the results were subjected to statistical analysis.

8 Results 8.1 The use of self-repetitions The analysis of the most frequently occurring repeated elements, normalized per million words (pmw), indicates that the majority of the most frequently repeated items belong to the category which can be broadly defined as function words. These can be seen in Table 5.3 which lists the five most frequently reiterated items in each corpus. The full list can be found in the appendix.

Table 5.3 The most common repetitions found in BNC and PLEC PLEC spoken component Lexical item

frequency

pmw

to to

450

II

350

the the

BNC spoken component Lexical item

frequency

pmw

2360.41

II

6360

615.39

1835.87

the the

5328

515.53

272

1426.74

that that

3342

323.37

you you

194

1017.6

aa

2627

254.19

yes yes

179

938.92

it it

2481

240.06

100

Learner Corpus Research

The results presented in the table reveal certain similarities: two items, that is, I and the, appear in both components; each section contains two pronouns and, with the exception of yes in the PLEC table, the remaining items are function words. Finally, both the native and non-native speakers of English seem to prefer single repetitions, where the repeated is reiterated only once. Our findings for native and, more interestingly, non-native speakers corroborate earlier research by Biber et al. (1999), which suggests that the choice of lexical items and word classes which are used in the case of self-repetitions is not a differentiating factor for the two groups. The statistical analysis was conducted for samples from BNC and PLEC. For each corpus, a hundred most commonly repeated function words were selected and the total number of occurrences was calculated for both samples. The results of the statistical tests used to compare both collections indicate that the difference is statistically significant, with chi-squared = 6804.97 and p < 0.01. Another step in the process of comparing differences between the spoken components of PLEC and BNC was the analysis of particular lexical items. Knowing that Polish speakers of English use repetitions more frequently than the native speakers, we wanted to check whether this statement holds true for particular words or expressions. In order to attain this goal, the repetitions were ranked according to the pmw difference while the raw values were subjected to the chi-squared. The ten most frequently repeated items, arranged according to the pmw frequency difference, are presented in Figure 5.1. It should be noted that in each case the difference was found to be statistically significant, with p < 0.05. Interestingly, the largest difference was observed in the case of to – unlike yes, which can be naturally used as a form of assuring the understanding of the question/utterance – to is not normally used as a discourse marker. Thus, it was decided to subject this lexical item to more thorough analysis, in order to discover its functions and to verify its hesitational character (see Biber et al. 1999). Since the biggest difference could be observed in the case of the single repetition of to, all the examples of this structure were manually analysed for the items which followed, in order to determine the context in which Polish learners of English use the repetition most frequently. The analysis revealed that there were four major categories associated with the structure in question, namely: ● ●

●

verbal constructions, for example, to to go, nominal constructions, including optional premodifiers, for example, to to England, communication breakdowns including both to to followed by a period of silence, as well as re-starts of the utterance (see Denke 2005), for example,

Self-repetitions in learners’ spoken language: A corpus-based study

●

101

to to ahm to create (problematic cases were assessed on the basis of the placement of the annotated utterance boundaries), pronominal constructions, for example, to to her.

The breakdown of the 313 occurrences of this repetition in question into the aforementioned categories is presented in Figure 5.2 which also shows the percentage reflecting the frequencies. The data collected show that it is clearly the verbal constructions which make the Polish speakers of English hesitate before proceeding further in their utterance. This may happen in a situation when ‘to’ is located at a boundary between two formulaic phrases, as shown in the following examples.

Figure 5.1 The most frequently repeated lexical items. Note: The items were arranged according to the per million words (pmw) frequency difference.

I would like to to see it really er how it works because so er so if they want to to earn money this way do you think we should let them? er well I listen to to every kind of genre just because I maybe

102

Learner Corpus Research

All the four major categories were also represented, predicted by Denke (2005), in the post-modifying position, where the speaker seems to be looking for the right word to complete the utterance. Some examples are provided below. erm it is easily for little parts to to lose and erm do you just go outside er in your home town or to to a park? so many er possibilities to to mmm to do mmm it’s harder in the city

The results of the analysis of the remaining lexical items presented in Table 5.3 also point to the hesitational nature of the repetitions used by learners. With

Figure 5.2 Types of single repetitions of ‘to’ found in PLEC.

Self-repetitions in learners’ spoken language: A corpus-based study

103

the exception of ‘yes yes yes’, which might be perceived as a means of building confluence (Molenda and Pęzik 2015), all the remaining words were found to have been used as hesitators, normally marking the boundaries between two ‘lexically heavy’ phrases, which seems to confirm previous findings on this subject.

8.2 Self-repetitions and language proficiency The last stage of our project, that is, the cross-sectional study of learners’ selfrepetitions across language proficiency levels, revealed a lot of irregularity. The only element which remained constant in all cases was the unexpectedly high number of self-repetitions at the C2 level. Other levels seem to show more predictable values; however, they were not always indicative of a linear change. The details are presented in Figure 5.3. Excluding the unexpectedly high number of self-repetitions used by C2 learners, three major trends can be observed. While the number of occurrences decreases in the case of ‘to’, ‘you’ and ‘yes’, the opposite tendency can be observed in the case ‘I’. Finally, the definite article shows some fluctuation, which makes it difficult to determine the level of a speaker on the basis of the frequency of occurrence of this lexical item. Statistical analysis for the entire model revealed no statistically significant relationship between the normalized number of self-repetitions and the level of linguistic proficiency (p = 0.231). However, upon the elimination of the C2

Figure 5.3 Distribution of the five most frequently used self-repetitions across the language proficiency levels. Count normalized as per thousand words (pkw).

104

Learner Corpus Research

values, we observed a statistically significant relationship with F (1,18) = 7.777, p = 0.012 and R2 of 0.549. The results for the items whose distribution indicated predictable and constant change, that is, ‘to’, ‘you’ and ‘yes’ – exclusive of the C2 values for these levels – showed a stronger goodness of fit, with F (1,10) = 13.958, p = 0.004 and R2 of 0.583. In this case, the normalized frequency of the repeated items is expected to decrease by 0.657 with an increase in language proficiency by one CEFR level. The results of the statistical tests seem to be inconclusive – the elimination of the C2 level revealed a statistically significant value for the goodness of fit, but even the third model only accounts for 58.3 per cent of the variance. Therefore, we would like to postulate that the sheer number of occurrences of self-repetitions in L2 learners is not a reliable indicator of their language proficiency. While this conclusion corroborates the findings of Osborne’s study (2011), the analysis of the last model promises some more coherent results which might help establish the ‘ceiling values’ expressed in normalized frequency of occurrence, beyond which the number of repetitions might be considered intrusive. We hypothesize that more data, preferably from an international corpus of EFL, would be needed to confirm this claim.

9 Discussion The results of our research do not fully confirm the interpretation that selfrepetition should be considered a marker of dysfluency. While EFL learners use repetitions significantly more frequently than native speakers, they seem to rely on a similar repertoire of lexical items when they reiterate linguistic content. Moreover, the existence of a direct relationship between the number of selfrepetitions and the level of language proficiency remains unproven, although there is an indication that this might be the case across some CEFR levels. The difference between Polish EFL learners and native speakers of English, expressed in the number of self-repetitions of most frequently reiterated function words, is noticeable enough to constitute a reliable marker of nonnative performance, if compared to a reference corpus. Such markers might be used in automated, computer-based speaking assessment (see Brown 2013: 84–4). While the fluctuation in the number of self-repetitions used by the native speakers across different language levels seems to corroborate previous

Self-repetitions in learners’ spoken language: A corpus-based study

105

research, it should be noted that this subject requires further studies, preferably based on larger samples and more diverse L1 background of the EFL speakers. Another possible research direction would be to conduct large-scale quasilongitudinal research on the whole subsystem of the hesitation markers in learners of English, in order to obtain qualitative data concerning the interplay of the hesitators. Such an approach would make it possible to establish whether self-repetitions substitute some other devices (e.g. unfilled pauses) and whether, with the development of L2 proficiency, these might be substituted by other hesitation markers. The in-depth analysis of the repetitions of to shows that Polish students of English most frequently struggle with verbal constructions, and reiterating the preposition in question marks their compositional efforts to combine two formulaic chunks. The preposition to tends to end a hesitation-free phrase (e.g. they want), while its repetition marks the attempt at beginning another one by finding the verb that should follow (e.g. earn money). This means that, according to the definition of a formulaic phrase given by McCarthy (2005), they want to and to earn money are automatized chunks of the language which are combined together at the expense of some effort. In extreme cases, this process is ineffective, which leads either to communication breakdowns or to unnatural multiple repetitions; note that the instances of to repeated two and three times occupy very prominent positions in Figure 5.3, being respectively the second and the fourth most frequent items.

10 Pedagogical implications From among the implications of our research, the most general conclusions of this study pertain to the topic of spoken language assessment and testing. The increased use of hesitational repetitions should be treated as a challenge to teachers and examiners who need to remember that this aspect of the language production is typical of learners and that it might be unrealistic to require of them to reproduce the native-like pattern of repetitions. Also, it should be stressed that the higher proportion of the self-repetitions used is not an immediate indicator of a lower level of language proficiency. Thus, we propose that the testing guidelines be revised and, wherever necessary, amended with explicit information on the assessment of repetitions. This assessment should focus on the effect of the use of the aforementioned device, as opposed to its frequency

106

Learner Corpus Research

of occurrence. In other words, while it is important that a student be able to avoid repetition-based communication breakdowns or slowdowns, requiring learners not to use repetitions more frequently than do native speakers might be pedagogically harmful. Moreover, teachers’ efforts should be directed at gradually decreasing the tendency to use hesitational repetitions at the borderlines between lexical bundles. This can be partially achieved by focusing on the more frequently repeated elements and helping students incorporate them into formulaic phrases. For instance, in the case of the most problematic verbal construction with to (Verb + to + Verb), one should attempt at automatizing to + Verb combinations (i.e. to earn money, to go home) in order to bridge the gap between two verbs. This can be achieved by means of a number of exercises and activities devised by various authors and described in detail in Molenda (2013). From among all the methods, we would like to stress the importance of challenging communicative practice, where the students are encouraged to perform at a ‘higher-than-normal level’ (Nation and Newton 2009). In this case, the performance need might help to automatize the use of the previously introduced formulaic phrases (Molenda 2013: 283). From the perspective of educational materials developers, the awareness of the most common hesitation triggers might be beneficial, as it provides a good context for introducing communication strategies. Instructional materials which aim at teaching students to tackle communication breakdowns could be further improved if the activities highlight the use of the most frequent problematic structures. In such a case, students can practise using the strategies that they learnt in the situation where the repetition-based manifestations of dysfluency are most likely to appear.

11 Implications for further research More research is needed to verify the hypothesis that self-repetitions are a reliable indicator of EFL learners’ level of proficiency. We suggest that such research should include a more fine-grained description of language proficiency levels and that the data should include spoken language produced by international students with various L1 backgrounds. Such an analysis might help establish the aforementioned ‘ceiling values’ which describe the frequencies of occurrence of self-repetitions beyond which students are perceived as dysfluent speakers of English.

Self-repetitions in learners’ spoken language: A corpus-based study

107

Appendix: Self-repetitions normalized as a pmw count for PLEC and BNC repetition

PLEC

pmw

repetition

BNC

pmw

to to

450

2360.41 I I

6360

615.39

II

350

1835.87 the the

5328

515.53

the the

272

1426.74 that that

3342

323.37

you you

194

1017.6 a a

2627

254.19

yes yes

179

938.92 it it

2481

240.06

in in

151

792.05 you you

2451

237.16

so so

144

755.33 no no

2138

206.87

to to to

122

639.93 in in

1840

178.04

no no

107

561.25 to to

1691

163.62

it’s it’s

91

477.33 we we

1637

158.39

for for

91

477.33 it’s it’s

1542

149.2

of of

90

472.08 they they

1476

142.82

they they

83

435.36 yes yes

1419

137.3

the the the

69

361.93 he he

1276

123.46

it it

65

340.95 of of

897

86.79

any any

61

319.97 I I I

772

74.7

that that

58

304.23 for for

485

46.93

some some

49

257.02 I’m I’m

460

44.51

yes yes yes

44

230.8

457

44.22

the the the

no no no

40

209.81 so so

454

43.93

III

36

188.83 no no no

435

42.09

we we

36

188.83 she she

390

37.74

he he he

35

183.59 in the in the

375

36.28

he he

30

157.36 he’s he’s

341

32.99

aa

30

157.36 they’re they’re

272

26.32

to to to to

29

152.12 I’ve I’ve

261

25.25

you you you

25

131.13 on the on the

241

23.32

I’m I’m

24

125.89 a a a

227

21.96

in in in

22

115.4

do do

220

21.29

in the in the

22

115.4

at at

200

19.35

do do

17

89.17

in in in

189

18.29

you can you can

16

83.93

you you you

172

16.64

of of of

16

83.93

that that that

171

16.55

108

repetition

Learner Corpus Research

PLEC

pmw

repetition

BNC

pmw

they they they

15

78.68

some some

161

15.58

no no no no

15

78.68

any any

159

15.38

the the the the

15

78.68

of the of the

155

15

at at

15

78.68

she’s she’s

154

14.9

some some some

12

62.94

yes yes yes

147

14.22

for for for

12

62.94

an an

143

13.84

so so so

12

62.94

no no no no

135

13.06

she she

11

57.7

he he he

130

12.58

no * 5

10

52.45

we we we

126

12.19

many many

10

52.45

it it it

126

12.19

for the for the

9

47.21

IIII

124

12

on the on the

9

47.21

can can

121

11.71

we we we

8

41.96

in a in a

119

11.51

I’ve I’ve

7

36.72

it is it is

115

11.13

it’s it’s it’s

7

36.72

it’s it’s it’s

113

10.93

yes yes yes yes

7

36.72

to to to

111

10.74

any any any

6

31.47

many many

110

10.64

we can we can

6

31.47

they they they

97

9.39

in a in a

6

31.47

you can you can

97

9.39

to the to the

6

31.47

of of of

88

8.51

at the at the

6

31.47

for the for the

80

7.74

to * 5

6

31.47

to the to the

78

7.55

IIII

5

26.23

no * 5

65

6.29

it it it

4

20.98

I can I can

57

5.52

for a for a

4

20.98

the the the the

53

5.13

of the of the

4

20.98

at the at the

46

4.45

that that that

4

20.98

no * 6

39

3.77

you you you you

3

15.74

in in in in

38

3.68

you are you are

3

15.74

we can we can

36

3.48

can can

3

15.74

I’m I’m I’m

31

3

the * 5

3

15.74

yes yes yes yes

31

3

so so so so

3

15.74

for for for

31

3

do do do

2

10.49

a lot a lot

30

2.9

I have I have

2

10.49

that that that that

27

2.61

you have you have

2

10.49

for a for a

26

2.52

Self-repetitions in learners’ spoken language: A corpus-based study

repetition

PLEC

pmw

repetition

he’s he’s

2

10.49

she she she

2

10.49

109

BNC

pmw

they are they are

24

2.32

we are we are

23

2.23

she’s she’s

2

10.49

I am I am

20

1.94

we we we we

2

10.49

you are you are

20

1.94

I can I can

2

10.49

you have you have

19

1.84

you can you can you can

2

10.49

she she she

19

1.84

a lot a lot

2

10.49

we we we we

19

1.84

to * 6

2

10.49

I have I have

17

1.64

of of of of

2

10.49

at at at

17

1.64

at at at

2

10.49

so so so

17

1.64

that that that that

2

10.49

you you you you

12

1.16

I’m I’m I’m

1

5.25

in the in the in the

12

1.16

I am I am

1

5.25

of of of of

11

1.06

you you you you you

1

5.25

they can they can

10

0.97

it is it is

1

5.25

he is he is

9

0.87

aaa

1

5.25

the * 5

9

0.87

they’re they’re

1

5.25

yes * 5

8

0.77

no * 6

1

5.25

to to to to

8

0.77

yes * 5

1

5.25

she is she is

7

0.68

yes * 6

1

5.25

you you you you you

6

0.58

she can she can

1

5.25

in a in a in a

6

0.58

in in in in

1

5.25

do do do

5

0.48

in a in a in a

1

5.25

that * 5

5

0.48

the * 6

1

5.25

any any any

4

0.39

in the in the in the

1

5.25

the * 6

4

0.39

an an

1

5.25

of * 5

4

0.39

of * 5

1

5.25

yes * 6

3

0.29

some some some

2

0.19

he can he can

2

0.19

to * 5

2

0.19

that * 6

2

0.19

you can you can you can

1

0.1

she can she can

1

0.1

110

Learner Corpus Research

References Aitchison, J. (1994), ‘ “Say, say it again Sam”: The treatment of repetition in linguistics’, in A. Fischer (ed.), Swiss Papers in English Language and Literature: Repetition (7), 15–34, Tübingen: Gunter Narr Verlag. Bazanella, C. (1996), Repetition in Dialogue, Tübingen: Max Niemeyer. Biber, D., Johansson, S., Leech, G., Conrad, S. and Finegan, E. (1999), Grammar of Spoken and Written English, Harlow : Pearson. Blankenship, J. and Kay, C. (1964), ‘Hesitation phenomena in English speech: A study in distribution’, Word, 20: 360–72. Brown, J. D. (2013), ‘Research on computers in language testing: Past, present and future’, in M. Thomas, H. Reinders, and M. Warschauer (eds), Contemporary Computer-Assisted Language Learning, 73–94, New York: Bloomsbury. Denke, A. (2005), Nativelike Performance: Pragmatic Markers, Repair and Repetition in Native and Non-native English Speech, Saarbrücken: VDM Verlag. Fillmore, L. W. (1979), ‘Individual differences in second language acquisition’, in C. Fillmore, D. Kempler, and W. Y. S. Wang (eds), Individual Differences in Language Ability and Language Behavior, 203–28, New York: Academic Press. Gatbonton, E. and Segalowitz, N. (2005), ‘Rethinking communicative language teaching: A focus on access to fluency’, The Canadian Modern Language Review, 61(3): 325–53. Henry, S. (2002a), ‘Etude des répétitions en français parlé spontané pour les technologies de parole’, Actes de la 4ième Rencontres des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RECITAL), Nancy, France, 24–27 June. Available at: http://www.atala.org/taln_archives/RECITAL/ RECITAL-2002/recital-2002-long-007.pdf (accessed on 16 January 2017). Henry, S. (2002b), ‘Quelles répétitions à l'oral? Esquisse d'une typologie’, Actes des 2èmes Journées de Linguistique de Corpus, Lorient: France, 12–14 September. Available at: http://www.univ-provence.fr/delic/papiers/Henry-2002lorient.pdf (accessed on 28 January 2014). Jucker, A. H. (1994), ‘Irrelevant repetitions, A challenge to relevance theory’, in A. Fischer (ed.), Swiss Papers in English Language and Literature: Repetition (7), 47–60, Tübingen: Gunter Narr Verlag. Kjellmer, G. (2008), ‘Self-repetition in spoken English discourse’, in T. Nevalainen, I. Taavitsainen, P. Pahta and M. Korhonen (eds), The Dynamics of Linguistic Variation: Corpus Evidence on English Past and Present, 37–60, Amsterdam: John Benjamins. Kormos, J. (2000), ‘The role of attention in monitoring second language speech production’, Language Learning, 50(2): 343–84. Macaulay, R. K.S. (1997), Standards and Variation in Urban Speech: Examples from Lowland Scots, Amsterdam: John Benjamins.

Self-repetitions in learners’ spoken language: A corpus-based study

111

McCarthy, M. (2005), Explorations in Corpus Linguistics, New York: Cambridge University Press. Merritt, M. (1994), ‘Repetition in situated discourse—Exploring its forms and functions’, in B. Johnstone (ed.), Repetition in Discourse: Interdisciplinary Perspectives, 23–36, New York: Ablex Publishing. Molenda, M. (2013), ‘Advanced students’ oral fluency: The neglected component in the CLT Classroom?’ in Ł. Salski and W. Szubko-Sitarek (eds), Perspectives on Foreign Language Learning, 275–89, Łódź: Łódź University Press. Molenda, M. and Pęzik, P. (2015), ‘Extending the definition of confluence. A corpus-based study of advanced learners’ spoken language’, in A. Turula and B. Mikołajewska (eds), Insights into Technology Enhanced Language Pedagogy, 105–18, Frankfurt am Main: Peter Lang. Mukherjee, J. (2009), ‘The grammar of conversation in advanced spoken learner English: Learner corpus data and language-pedagogical implications’, in K. Aijmer (ed.), Corpora and Language Teaching, 203–30, Amsterdam: John Benjamins Nation, I. and Newton, J. (2009), Teaching ESL/EFL Listening and Speaking, New York: Routledge. Osborne, J. (2011), ‘Oral learner corpora and the assessment of fluency in the Common European Framework’ in A. Frankenberg-Garcia, L. Flowerdew and G. Aston (eds), New Trends in Corpora and Language Learning, 181–97, London: Continuum. Pęzik P. (2012), ‘Towards the PELCRA learner English corpus’, in P. Pęzik (ed.) Corpus Data across Languages and Disciplines, Łódź Studies in Language (Vol. 28), 33–42, Frankfurt am Main: Peter Lang. Persson, G. (1974), Repetition in English: Part 1 Hesitational Repetition, Department of English, Uppsala: University of Uppsala. Rayson, P., Leech, G. and Hodges, M. (1997), ‘Social differentiation in the use of English vocabulary: Some analyses of the conversational component of the British National Corpus’, International Journal of Corpus Linguistics, 2(1): 133–52. Schegloff, E. A. (1987), ‘Recycled turn beginnings: A precise repair mechanism in conversation’s turn-taking organisation’, in G. Button and J. R. E. Lee (eds), Talk and Social Organisation, 70–85, Clevedon: Multilingual Matters. Shriberg, E.E. (1994), ‘Preliminaries to a theory of speech disfluencies’, Unpublished Ph.D. thesis, University of California, Berkeley. Tannen, D. (1989), Talking Voices. Repetition, Dialogue, and Imagery in Conversational Discourse, Cambridge: Cambridge University Press. Tweedie, F. J. and Baayen, R. H. (1998), ‘How variable may a constant be? Measures of lexical richness in perspective’, Computers and the Humanities 32: 323–52.

6

Corpus-Driven Study of Information Systems Project Reports Ryan T. Miller and Silvia Pessoa

1 Introduction In recent years, corpus methods have increasingly been applied to the study of disciplinary genres (e.g. Cortes 2004; Flowerdew 2015; Hyland 2008; Nesi and Gardner 2012; Swales 2014). These studies are important because university students must learn to understand and produce disciplinary genres in order to become a full-fledged member of their discipline of study (Johns 1997). However, many faculty members in the disciplines lack explicit knowledge of the rhetorical and linguistic features of disciplinary genres, limiting their ability to effectively teach these genres. Through study of disciplinary genres, applied linguists and ESP tutors can help faculty in the disciplines better scaffold student writing. In this chapter, we use DocuScope, a corpus-based tool for analysis of rhetorical functions in writing, to study the features of a key genre in the field of information systems (IS), project reports. Through quantitative and quantitatively informed qualitative analysis of model texts and learner writing, we identify rhetorical features that typify this genre, and the extent to which students include these features in their writing.

1.1 Disciplinary genres In university-level education, students are expected to write a number of genres in a variety of disciplines. Each of these genres reflects the epistemology, culture and discourse that are valued in each discipline (Canagarajah 2002; Duff 2001; Johns 1997). However, disciplinary genres pose a significant challenge to novice

Corpus-Driven Study of Information Systems Project Reports

113

writers because the communication norms and conventions that are unique to each disciplinary genre are often implicit to disciplinary faculty, and opaque to students. Disciplinary literacy practices can be challenging for students as these practices require an understanding of not only the discipline’s content, but also the expectations and demands of the various genres within the discipline. Therefore, there has been a noticeable shift in recent years towards looking at writing within the context of its disciplinary community (Christie and Maton 2012; Hyland 2004). This work emphasizes the fact that it is not enough for students to know about a subject; they also need to be able to use this knowledge in formats that accord with expectations of their discipline. In the present study, we investigate writing in the discipline of IS.

1.2 Writing in information systems As a discipline, IS focuses on using computers and information technology tools to help businesses achieve their objectives effectively and efficiently. Writing is important in professional IS work, and although writing skills are some of the most requested by employers, a gap still exists between employer expectations and IS graduates’ skills (Liu and Murphy 2012). Previous research has recommended that IS faculty take responsibility for helping students improve their written communication skills (Merhout and Etter 2005). Thus, it is vital for IS students to learn to write IS genres, and IS faculty to have explicit genre knowledge in order to more effectively teach these genres (see Miller and Pessoa (2016) for another study of IS genres). One of the main writing tasks of IS professionals is documentation of IS software development, which occurs via the genre of the project report. A project report is written to document work in each phase of the development of an IS solution. Like other types of software development, IS development often occurs in teams, and occurs incrementally through a series of phases (see Larman and Basili 2003). At the end of each phase, the team documents their progress by writing a project report. Investigating project reports in computer science, Kaneko, Rozycki and Orr (2009) found that although this genre is common among professionals, computer science education lacked instruction in how to write this genre. The project report itself is a macro-genre (Martin 1992), or a longer text containing multiple subgenres, with each subgenre reflecting the overarching

114

Learner Corpus Research

purposes of the macro-genre. Overarching purposes of an IS project report are to document and report a team’s progress in developing an IS software solution, and to show how a team addresses problems they have encountered. Reflecting the importance of documentation in professional IS contexts, project reports are also a part of IS education. In this context, project reports are what Nesi and Gardner (2012: 171) call a genre ‘which prepares [students] for professional practice’ by discussing and seeking ‘solutions to practical problems’, and which may differ from more typical academic writing tasks such as argumentative essays or research reports.

1.3 Theory of genre The basis of our investigation is a theory of genre as representational compositional choices developed by David Kaufer and colleagues (e.g. Ishizaki and Kaufer 2012; Kaufer and Butler 1996, 2000; Kaufer et al. 2005; Klebanov et al. 2016). Kaufer’s theory arises from the rhetorical tradition of genre study, which emphasizes pragmatic function over structure of a text. Although rhetorical theories of genre typically rely on ‘impressionist, selective, and non-operationalized construals of genre features’ (Klebanov et al. 2016: 170), Kaufer’s theory seeks to operationalize pragmatic functions of texts by analysing and categorizing language which instantiates these functions. Kaufer’s theory views genre as recurrent combinations of micro-rhetorical elements that occur in certain stable proportions, and which prime the reader to have a certain experience or understanding of the text, such as acknowledgement of alternative viewpoints, confidence or objectivity. In Kaufer’s theory, these micro-rhetorical elements are organized hierarchically under a number of rhetorical categories, and it is the relative distribution of the various categories which sets genres apart (Klebanov et al. 2016). Kaufer’s theory is congruent with that of Biber (1989, cited in Klebanov et al. 2016) in that both identify genres based on multidimensional analysis of covariation among variables. Biber’s system captures functional distinctions through analysis of grammatical categories, such as parts of speech and types of syntactic phrases, and semantic categories, such as communication verbs or certainty adverbs. Rather than linguistic categories, Kaufer’s system directly targets functional categories (i.e. the experience that is created in the reader when a phrase is used) (Klebanov et al. 2016). Similar to Hoey’s (2005) concept of lexical priming, Kaufer argues that ‘words in use prime an audience’s

Corpus-Driven Study of Information Systems Project Reports

115

experience and different words prime different experiences’ (Kaufer et al. 2004: xvii). Take, for example, the word ‘smear’: whereas the expression ‘to smear a politician’ contributes to a negative expression, ‘smearing soap’ contributes to an expression of everyday motion (Ishizaki and Kaufer 2012: 277).

1.4 Corpus-driven study of academic writing The study of academic writing development in a second language (L2) has been greatly benefited in recent years by the development of corpora of learner writing, and the increased use of corpus tools in the study of writing development (e.g. Granger, Gilquin and Meunier 2013; Lee and Chen 2009; Wulff and Gries 2011). However, much of this focus has been on the frequency of lexical and grammatical forms, rather than on the functions of these forms as they instantiate rhetorical meanings in discourse (Flowerdew 2009; Upton and Connor 2001). Furthermore, most corpus-based research on writing has focused on academic essays and research reports (Nesi and Gardner 2012), with less attention paid to the writing of professional disciplinary genres. Corpus-driven research of disciplinary genres can shed light onto not only the linguistic forms used in specialized texts, but also the relationships between lexico-grammatical choices and rhetorical functions. Thus, in this study, we make use of a corpus-based tool (DocuScope) based on Kaufer’s theory of genre to investigate rhetorical functions in a specific disciplinary genre, IS project reports, seeking to answer two questions: (i) What are rhetorical features of IS project reports? (ii) To what extent do students adopt these rhetorical features in their writing?

2 Methods 2.1 Data source The study was conducted in an undergraduate IS programme at a branch campus of an American university in the Middle East. Major components of this IS programme are two hands-on, semester-length team projects, one of which students complete in their third year (termed Junior Project) and one in their fourth year (Senior Project). According to the Junior Project syllabus, the projects are ‘team-based project course[s] in team-based software development’ in which students collaboratively ‘design and build an information or decision

116

Learner Corpus Research

support system, following a disciplined software project lifecycle approach’. As part of this project lifecycle approach, students produce project reports throughout their development process. The major difference between the two projects is that in the Junior Project, the students design their software solution for an imagined client, whereas in the Senior Project the students work with a real-life client. Instructional materials and student writing were collected. In line with the expectations of the IS discipline, enhancing students’ oral and written communication skills for future professional use is a goal of the IS programme at this university. Reflecting this goal, the Senior Project syllabus states that one of the objectives is for students to develop ‘professional communications through a structured, guided, hands-on process’. Because each phase had a different focus, the sections and content varied from one phase to the next. However, each project report began with an executive summary which described the team’s progress in that specific phase. Because this was the only section which appeared in all of the project reports, we focus our analysis on the executive summary. Instructional materials described the executive summary as one to two-pages that ‘tell the reader, in an abbreviated, accurate, and highly readable form, what is in your report’. Also, it ‘should communicate to a busy reader all important information … the reader needs to know about your team’s project and progress during the phase’. Although sample project reports with executive summaries were given to students as models, there was little instruction about how to write a project report or executive summary, and no focus on language used in writing such reports.

2.2 Texts Our analysis included three relatively small sets of texts. The first set was sample project report executive summaries provided by IS professors (n = 12; mean length = 381 words, SD = 170.9), which were used by IS professors as models in the Junior Project and Senior Project courses. These sample project reports were written by advanced graduate students at the university’s main campus in the United States. Because these texts were used as model texts by the IS faculty, we took the samples as representative of professors’ expectations of IS project reports. The second and third sets of texts were Junior Project (n = 17; mean length = 498, SD = 153.7) and Senior Project (n = 18; mean length = 375, SD = 109.4) executive summaries respectively, written by undergraduate students at the international branch campus. All students were non-native speakers of English,

Corpus-Driven Study of Information Systems Project Reports

117

though they were highly proficient (mean TOEFL iBT score = 97, SD = 12.4). Students came from linguistically and culturally diverse backgrounds, mainly from the Gulf region, the greater Middle East, India, Pakistan and Bangladesh. All project reports were written by teams of three to five students. The Junior Project reports and Senior Project reports were collected in sequential academic years, and thus many of the students were the same in both classes. However, none of the groups consisted of the same members.

2.3 Analysis tool As described earlier, our analysis was conducted using DocuScope (Ishizaki and Kaufer 2012), a dictionary-based text tagging and visualization tool for identifying instances of language reflecting rhetorical functions. The DocuScope software is a string-matching tool that contains over 45 million unique words and phrases divided into 112 micro-rhetorical function categories (see Ishizaki and Kaufer (2012) and Kaufer et al. (2004) for more information), based on Kaufer and Butler’s (1996, 2000) representational theory. For example, DocuScope tags the pronoun ‘I’ as a first person function, ‘if ’ as associated with a contingency function and ‘might happen’ as an uncertainty function. The 112 micro-rhetorical functions are grouped into fifteen clusters (see Table 6.1). For example, the narrative cluster contains linguistic elements for past tense verbs, indication of background narrative (e.g. ‘by the way’) and expressions of time shift (e.g. ‘at that moment’) and time duration (e.g. ‘over the last month’). The analysis in the present study focused on these fifteen rhetorical function clusters. The DocuScope dictionaries have been found to successfully differentiate patterns associated with genres found in the Brown and Freiburg-Brown (Frown) corpora in ways that comport with the intuitions of human classifiers (Collins 2003). In previous research, DocuScope has been used to analyse rhetorical functions in non-native speaker writing, including argumentation in academic writing (Pessoa, Miller and Kaufer 2014), pragmatic functions in academic writing (Zhao and Kaufer 2013), and features of higher- and lowergraded placement essays (Ishizaki and Wetzel 2008). Although DocuScope has often been used for genre analysis, it should be distinguished from tools whose goal is to classify texts into genres (e.g. Argamon et al. 2007; Stein and Eissen 2008). DocuScope, in contrast, is designed to deepen the understanding of how genre is enacted on the textual surface through language choices (Ishizaki and Kaufer 2012).

Table 6.1 Rhetorical function categories and representative subcategories in the DocuScope system Character

Description

Elaboration

Emotion

Abstract Concepts: phenomenology Citation: points out that Authoritative Citation: has conclusively reported Contested Citation: maintains that Quotation: said ‘…’ Metadiscourse: As this chapter has demonstrated

Personal Pronouns: he, she Property of Person: employer, client Dialogue Cue: ,’ he said Oral Cue: uh, hey

Sensory Property: bright red Space Relation: the back of the Scene Shift: go back to a place Motions: place it behind

Generalization: all indicate that Example: for example Exceptions: with the exception of Specifiers: and, more specifically Definition: is defined as

Positivity: is beneficial Anger: livid about Fear: is afraid of Sadness: lose all hope Reluctance: have to admit Apology: apologetically

Project Ahead: hope to Predicted Future: there will be

Past Project back: used to be Future in past: was to be

Insisting Immediacy: at this juncture Insisting: it is imperative that Prohibition: you must not

Personal Relations Positive Relations: give much credit to Promise: pledged to Reassure: find solace in Inclusivity: we all Negative Relations: condemn

Institutional In Media: on TV Common Authorities: it is often said that Responsibility: under the supervision of Public Standards: fairness, injustice Reasoning Reason Forward: therefore Reason back: because Support: is evidence for Contingency: on the condition of Deny: not the case that Concessive: it must be acknowledged

Interaction Curiosity: the challenge as we see it is Question: Did he…? Open Query: Is there any way that…? Direct Address: I urge you to Request: We respectfully ask that Reporting Report States: is made of Report Events: arrived at the agreement Report Recurring Events: again Transformation: changed the nature of

Narrative Narrative Verb: saw the Time Shift: next week Time Duration: over the last month Biographical Time: was earlier known as

Subjectivity 1st Person: I feel that Autobio: I have always Private Thinking: believe that Subjective Time: take our time Confidence: is definite Uncertainty: perhaps

Learner Corpus Research

Future

118

Academic

Corpus-Driven Study of Information Systems Project Reports

119

DocuScope offers a number of advantages over other tools. First, the size of the DocuScope dictionaries is larger than those of other dictionary-based tools. While many such tools use fewer than 10,000 entries (Klein 2013), the DocuScope dictionaries contain more than 45 million unique words and phrases. A further advantage of DocuScope is its ability to classify strings of varying lengths. DocuScope is able to recognize single words and longer strings (up to seventeen words in length); for example, while ‘I’ is coded as a first person function, the five-word phrase ‘in this paper, I will’ is coded as a metadiscourse function in the academic rhetorical function cluster to orient readers at the beginning of an essay.

2.4 Analysis procedure Our analysis examined rhetorical functions in each of the sets of texts (samples, Junior Project reports and Senior Project reports). First, we used DocuScope to analyse the rhetorical functions in the sample project report executive summaries. We did this by comparing the samples with a reference corpus, the Freiburg-Brown (Frown) corpus, which contains 500 texts of approximately 2,000 words each (a total of approximately a million words). This corpus contains a wide variety of sources and genres, including news (88 texts), general prose (206 texts), scientific texts (80 texts) and fiction (126 texts), and is designed to be representative of 1990s written American English (Biber 1993). This corpus was chosen because previous research using the DocuScope tool has used it as a reference corpus (e.g. Marcellino 2014; Witmore and Hope 2007) and this corpus was used in the validation study noted earlier (Collins 2003). By examining differences between the samples and the reference corpus, we can identify the rhetorical functions which typify the samples. Previous studies using the DocuScope tool have also used this technique (e.g. Ishizaki and Kaufer 2007; Ishizaki and Wetzel 2008; Kaufer et al. 2005). Following the analysis of the samples, we subsequently analysed the Junior Projects and Senior Projects, and compared these with the samples to determine whether students were using the same rhetorical and micro-rhetorical functions as the samples, and whether they were using these to similar degrees as the samples. Quantitative comparisons were made using multiple techniques. We used the two-sample Kolmogorov-Smirnov test, a non-parametric method for comparing the distributions of two data sets, and Welch’s t-test. We chose these methods because they are robust to unequal sample size and variance, and because they have been used in previous research using DocuScope (e.g. Airoldi et al. 2006;

120

Learner Corpus Research

Kaufer and Hariman 2008; Marcellino 2014). In both cases, we corrected for multiple comparisons using the Benjamini-Hochberg correction set with a false discovery rate of 0.05, and all reported p-values are FDR-adjusted BenjaminiHochberg p values. Because such a quantitative analysis reveals only the presence or absence of the rhetorical functions, we also conducted subsequent qualitative analyses to better understand how the rhetorical functions were used in the project report executive summaries.

3 Results 3.1 Sample project report executive summaries The analysis of the twelve sample project report executive summaries revealed that some rhetorical functions occurred significantly more frequently in the samples than in the reference corpus (higher-occurrence rhetorical functions) while others occurred significantly less frequently (lower-occurrence rhetorical functions). Below, we describe each of these rhetorical functions, with illustrations of usage in the sample executive summaries.

3.1.1 Higher-occurrence rhetorical functions There were three higher-occurrence rhetorical functions: Personal Relations, Reporting and Future (see Table 6.2). Personal Relations includes language indicating relationships among people. The qualitative analysis showed that this rhetorical function was used in the samples to show that the authors were working together as a team, and writing the project reports as a team. This included extensive use of first person plural personal pronouns and possessive determiners, such as ‘We completed 90% of our tasks on time, and for our project we have completed 5 out of 10 use cases’ (Sample 7). It also included use of inclusive noun phrases such as our team, our group and collaboration, as in ‘Our team has altered its approach’ (Sample 3). Although this is likely a result of the team-based nature of the projects, there was no information in the assignment description indicating that reports should be written in this way. In addition, the use of inclusive ‘we’ in the project reports is somewhat different from how ‘we’ has been found to be used in other academic writing, such as to publicize the writer and their work (Harwood 2005).

Corpus-Driven Study of Information Systems Project Reports

121

Table 6.2 Rhetorical functions occurring more frequently in samples than in the reference corpus Samples

Frown Corpus

M

SD

M

SD

t (df)

Personal Relations

3.61

1.44

0.55

0.39

7.49 (11.0)***

Reporting

15.72

3.76

7.26

1.62

7.77 (11.1)***

Future

2.13

0.90

0.83

0.42

4.99 (11.1)***

Note: *** p < 0.001; Means indicate the mean number of patterns per 100 words of running text.

The Reporting function includes words for reporting information, particularly verbs. This aligns with the overarching function of the project report in general, and the executive summary in particular, of reporting the team’s progress in the development of their software solution. This is different from many other types of writing that students do, such as argumentative writing. Within Reporting, the samples used verbs to report on the processes of their project’s development, such as ‘The lifecycle of our project began with the introduction of the hospital discharge data. It was then followed by the creation of the database’ (Sample 8). Students also used verbs to report on actions they had taken, such as ‘We have already introduced the application to our clients’ (Sample 4) and ‘We identified the most important non-functional requirements’ (Sample 5). The Future rhetorical function, which includes a variety of forward-looking language, was also significantly more frequent in the samples. In the qualitative analysis, we found that this was used to connect the current state of development with plans for future development. For example, the samples used will: ‘Our proposed system will be based on PHP language in a MVC framework’ (Sample 1). In addition, the teams used language that indicates actions taken in order to solve a problem, such as in ‘We reduced the complexity of the data model … in order to increase the performance of the system’ (Sample 6) or ‘Integration will be difficult with [the] existing database … . We will need to build a connector between their old and our new database’ (Sample 1). Overall, the higher-occurrence rhetorical functions suggest that the executive summaries are written from the perspective of a team rather than each individual, they are written to report on the process of development of the project and the actions that were taken during that process and that they use forward-looking language to describe future development of the project or how actions the team has taken will accomplish goals or solve problems.

122

Learner Corpus Research

3.1.2 Lower-occurrence rhetorical functions There were six rhetorical functions that occurred significantly less frequently in the samples than in the reference corpus (see Table 6.3). Different from the higher-occurrence rhetorical functions, whose presence typifies the genre, lower-occurrence rhetorical functions are notable because of their absence. The Subjectivity rhetorical function allows authors to express their own involvement in a text (Kaufer and Butler 2000). This function includes use of first person singular pronouns (e.g. I), and because the project reports are written from the perspective of a team (as described earlier), first person singular pronouns were used very little. In addition, this function includes subjective interpretations and evaluations showing confidence (e.g. ‘Task assignments were done easily’, Sample 4) or uncertainty (‘We discovered… our lack of knowledge of the technologies we were using’, Sample 8), suggesting that the executive summaries are typically written in a more objective, factual style, with less subjective evaluation. The Character rhetorical function includes language for naming and describing individuals or entities, and is typical of narrative writing (Kaufer and Butler 2000). Within this category, we saw in particular that the samples had significantly fewer personal pronouns (he, she, they) and few instances of naming individuals or entities according to their role (e.g. ‘There is no strong developer in the group’, or ‘The client may be unable to specify their needs’; Sample 1). The sample executive summaries were also significantly lower in Description and Emotion. Here, description refers to language which appeals to the senses,

Table 6.3 Rhetorical functions occurring less in samples than in the reference corpus Samples

Frown Corpus

M

SD

M

SD

t (df)

Subjectivity

2.55

0.57

4.78

1.79

12.17 (16.8)***

Character

1.51

0.75

3.88

1.90

10.16 (14.6)***

Description

3.18

0.94

8.06

4.27

14.70 (24.3)***

Reasoning

1.48

1.11

2.90

0.90

2.73 (11.3)*

Emotion

1.37

0.62

2.31

1.02

5.08 (12.5)**

Interaction

0.34

0.34

1.16

0.87

7.94 (14.8)***

Note: *** p < 0.001; ** p < 0.01; * p < 0.05; Means indicate the mean number of patterns per 100 words of running text.

Corpus-Driven Study of Information Systems Project Reports

123

such as colours, sounds, tastes and so on (Kaufer and Butler 2000). Language appealing to positive and negative emotions was almost completely absent, with most of the matched language being descriptions of defects in the system or problems encountered during system development (e.g. ‘We underestimated the time we needed to complete the project’, Sample 8), which were not especially emotional in this context. The lack of descriptive and emotional language again reflects the purpose of the project reports as a whole, and the executive summary in particular, as reporting information in an objective, factual manner. The Reasoning rhetorical function also occurred significantly less in the sample executive summaries. This includes language indicating forward reasoning (e.g. thus, therefore), backward reasoning (e.g. because, owing to the fact) and oppositional reasoning (e.g. it is not the case that) and indicative of logical reasoning in constructing an argument (Pessoa, Miller and Kaufer 2014). There were some instances of these in the samples (e.g. forward: ‘Therefore, the team has altered its approach’, Sample 3; backward: ‘We chose to keep our list … because we do not want to create too large a scope …’ Sample 5; oppositional: ‘Despite the time, we were able to deliver …’ Sample 8), however they were relatively rare. Reflecting the function of the executive summaries as a report, an argumentative rhetorical mode is unnecessary. Almost completely absent was Interaction, which includes language indicating interaction between author and reader, such as direct requests or questions directed to the reader (Kaufer and Butler 2000). Overall, the lower-occurrence rhetorical functions suggest that the sample executive summaries present information in a factual, objective manner and without extensive description, evaluation or indication of emotion.

3.2 Junior Projects After we identified the rhetorical functions in the samples, we examined the student writing, beginning with the seventeen Junior Project reports. First, we compared the Junior Project executive summaries with the reference corpus to determine whether the rhetorical functions that were salient in the samples were also salient in the Junior Projects. Next, we compared the Junior Project reports with the sample reports to determine whether the rhetorical functions occurred to similar degrees. Table 6.4 shows the means for each dataset and the results of the comparisons. All three of the higher-occurrence rhetorical functions (Personal Relations, Reporting and Future) occurred more frequently in the Junior Project executive

Junior Project Rhetorical function

Frown Corpus

124

Table 6.4 Frequency of rhetorical functions in the Junior Project executive summaries and comparisons with the Frown corpus and samples Samples

M

SD

M

SD

t (df)

M

SD

t (df)

Personal Relations

2.84

1.04

0.55

0.39

9.08 (16.1)***

3.61

1.44

1.72 (18.8)

Reporting

13.46

1.91

7.26

1.62

13.21 (16.8)***

15.72

3.76

1.92 (15.0)

Future

1.76

0.67

0.83

0.42

5.66 (16.4)***

2.13

0.90

1.21 (19.3)

Subjective

3.18

1.15

4.78

1.79

5.52 (18.8)***

2.55

0.57

1.96 (24.8)

Character

1.28

0.79

3.88

1.90

12.35 (22.8)***

1.51

0.75

0.78 (24.6)

Descriptive

3.26

1.50

8.06

4.27

11.69 (25.9)***

3.18

0.94

0.17 (26.7)

Reasoning

1.75

0.83

2.90

0.90

5.60 (17.3)***

1.48

1.11

0.68 (19.2)

Emotional

0.90

0.43

2.31

1.02

12.41 (22.7)***

1.37

0.62

2.25 (18.2)

Interactive

0.53

0.36

1.16

0.87

6.59 (22.7)***

0.34

0.34

1.45 (25.0)

Higher occurrence

Note: *** p < 0.001; Means indicate the mean number of patterns per 100 words of running text.

Learner Corpus Research

Lower occurrence

Corpus-Driven Study of Information Systems Project Reports

125

summaries than in the reference corpus, suggesting that these were also higheroccurrence rhetorical functions in the Junior Projects. In addition, none differed significantly from the samples, suggesting that students used these rhetorical functions at rates appropriate for the genre. Similarly, all six lower-occurrence rhetorical functions (Subjectivity, Character, Description, Reasoning, Emotion and Interaction) also appeared significantly less frequently in the Junior Project reports than in the reference corpus, and there were no significant differences between the Junior Projects and the samples, further suggesting that the students incorporated these rhetorical functions to an appropriate degree. However, among the other six rhetorical functions, the Elaboration rhetorical function was found to be higher in the Junior Projects (M = 4.47, SD = 0.95) than either the reference corpus (M = 3.54, SD = 1.23; t(17.9) = 3.94, p < 0.001) or the samples (M = 3.48, SD = 1.08; t(21.8) = 2.53, p = 0.019), suggesting an overuse of this function. The Junior Projects tended to include more elaborated explanation, such as in, ‘We paid a great attention to this section, as it plays a vital role in fixing our system and meeting users’ needs’ (Junior Project 13), or ‘The technical manual is targeted towards the developers of the system, as it will explain [to] the developers the technical requirements, coding techniques, and design components of the system’ (Junior Project 9). This could be a function of the document’s status as a classroom assignment, as such explanations tell the professor the students’ reasoning for their actions or display their understanding of concepts, which would likely not be necessary in a professional context. The lower amount of elaboration in the samples also reflects the professor’s description of the executive summary as ‘abbreviated’ and presenting only ‘the highlights’ (Junior Project syllabus).

3.3 Senior Projects Our analysis of the eighteen Senior Project executive summaries proceeded similarly to our analysis of the Junior Project executive summaries. Table 6.5 summarizes the means for each dataset, and the results of the comparisons. Among the three higher-occurrence rhetorical functions, two (Personal Relations and Future) were both significantly higher than the reference corpus and not significantly different from the samples, suggesting that students used these functions to an appropriate degree. However, the Reporting function occurred significantly less in the Senior Projects than in the samples, though it was still significantly greater than the reference corpus, indicating that the Senior Projects were in the right direction compared to general English, but were not

Senior Projects Rhetorical function

Frown Corpus

126

Table 6.5 Frequency of rhetorical functions in the Senior Project executive summaries and comparisons with the Frown corpus and samples Samples

M

SD

M

SD

t (df)

M

SD

t (df)

Personal Relations

2.30

1.63

0.55

0.39

4.55 (17.1)***

3.61

1.44

2.42 (25.7)

Reporting

11.66

2.29

7.26

1.62

8.09 (17.6)***

15.72

3.76

3.35 (16.4)*

Future

2.03

0.75

0.83

0.42

6.79 (17.4)***

2.13

0.90

0.30 (20.7)

Subjective

2.81

0.90

4.78

1.79

8.71 (22.2)***

2.55

0.57

0.97 (28.0)

Character

1.72

1.02

3.88

1.90

8.45 (21.5)***

1.51

0.75

0.64 (27.6)

Descriptive

3.58

1.32

8.06

4.27

12.32 (32.2)***

3.18

0.94

0.98 (27.8)

Reasoning

1.65

1.12

2.90

0.90

4.69 (17.8)***

1.48

1.11

0.85 (23.6)

Emotional

0.88

0.43

2.31

1.02

12.90 (24.5)***

1.37

0.62

2.36 (17.9)

Interactive

0.64

0.49

1.16

0.87

4.27 (18.9)**

0.34

0.34

2.67 (25.9)

Higher occurrence

Note: *** p < 0.001; ** p < 0.01; * p < 0.05; Means indicate the mean number of patterns per 100 words of running text.

Learner Corpus Research

Lower occurrence

Corpus-Driven Study of Information Systems Project Reports

127

at the level that may be expected in this genre. In particular, the Senior Projects had significantly less reporting of processes than the samples. Closer inspection of the Senior Projects found that they tended to include more description of the product that was being built (e.g. ‘The app also has an admin view’, ‘It is designed to be simple’, Senior Project 8), rather than description of the team’s process in constructing the product (e.g. ‘This development phase started with the successful implementation of several basic CRUD operations’, Senior Project 2). Similar to the Junior Projects, the Senior Projects were significantly lower than the reference corpus and not significantly different from the samples in all six lower-occurrence rhetorical functions. This suggests that the students were able to include these rhetorical functions in their writing in amounts that were appropriate for the genre. Of the other variables, the Institutional rhetorical function (M = 6.03, SD = 1.88) occurred more frequently than either the samples (M = 3.25, SD = 1.97; t(22.6) = 3.79, p = 0.014) or the reference corpus (M = 3.71, SD = 2.25; t(18.8) = 5.09, p < 0.001). The Institutional rhetorical function includes writing that relies on authorized external sources of information rather than the author. From the qualitative analysis, we saw that the Senior Projects tended to refer more to consultations with the client organization than the samples did, for example, ‘After carrying out different meetings with our client to gather and record as much information about the project requirements, the team crafted and developed good understanding of the client’s expectation of the system’ (Senior Project 4). This is in contrast to the sample executive summaries, which also mentioned client requirements, but focused more on the development team’s identification of the requirements rather than consulting with the client: ‘We identified that the most important non-functional requirements for the client are performance and security’ (Sample 5). This may have been a function of the authors of the sample reports being advanced graduate students, who may be more confident in their independent assessment of client needs, without needing as much guidance by the client.

4 Discussion Using corpus-assisted methods, the present study examined rhetorical functions in a specific disciplinary genre in the field of IS, project report executive summaries. The examination of sample executive summaries revealed a significantly greater presence of three rhetorical functions as compared to a reference corpus. These

128

Learner Corpus Research

suggest that traits of project report executive summaries include increased use of language that indicates the team-based nature of the system development. Following the overarching function of reporting, executive summaries report on the processes of the project development and actions that had been completed. We also saw a significantly greater amount of future-looking language to indicate actions that the team will take, especially in response to problems. In addition, the analysis of the samples found six low-occurrence rhetorical functions. The samples showed a lack of first person personal pronouns, which supports that the executive summaries are written from the perspective of a team. In addition, the executive summary genre seems to include writing which is direct, factual and objective, corresponding to the overarching goal of reporting, rather than descriptive, evaluative or subjective. The present study found that even without explicit instruction of rhetorical features of the executive summary genre, students nonetheless included most of the same rhetorical functions as the samples, and did so at similar rates as the samples. However, there were some rhetorical functions in which the student writing differed from the samples, including overuse of elaboration and reliance on external sources of information, and underuse of reporting. These findings may reflect differences between academic essay genres, in which students have received explicit language instruction in English courses, and professional genres such as project reports, in which students received no instruction on language use. Nesi and Gardner (2012) point out that when students are asked to write professional genres, there is often a tension as the reader of the text is not a client or colleague, but rather a teacher who evaluates the text. This may cause student writers to feel a need to explain their actions and choices, provide evidence of their understanding or rely more on information from external sources rather than their own interpretations. The result of this is what Wardle (2009: 774) calls a ‘mutt genre’ with conflicting audiences and purposes.

5 Pedagogical applications The present study has a number of pedagogical implications. First, the findings highlight the importance of using model texts in teaching writing. Although there was no explicit instruction of rhetorical features for writing project report executive summaries, students were nonetheless able to include most of the rhetorical functions in their writing with only the sample texts as input. Although some researchers have argued that using model texts represents

Corpus-Driven Study of Information Systems Project Reports

129

genres too narrowly and inhibits students’ creativity as writers (e.g. Elbow 1999; Smagorinski 1998), the model texts here seem to have allowed students to (perhaps implicitly) pick up micro-rhetorical elements of the sample texts and integrate these into their own writing. We would, however, suggest that instructors go beyond simply supplying model texts by helping students develop explicit knowledge of rhetorical features by making these features (such as those found in the present study) explicit for students. After understanding rhetorical functions through an analysis such as in the present study, instructors can then show students the language through which these functions are implemented in the model texts. In addition to model texts, instructors can create exercises using extracts from project reports. For example, the present study found that an important feature of IS project reports was reporting, specifically about processes that the team undertakes in developing their product, how steps in the process built on each other and how current problems link with future solutions. This was also an area that students showed difficulty in. The Senior Projects showed significantly less reporting of processes than the samples, and, while many project reports included language that connected the current status of the project (or current problems with the project) with future development plans, some focused entirely on the past, such as one that described how the ‘project suffered from major deviation from the original proposal’ (Junior Project 10) without describing how the team planned to address this as they moved forward. Thus, this aspect of project report writing might be a suitable target for explicit classroom instruction of genre features. For example, in the classroom, a teacher could provide students with a list of events that may happen during development of a project, such as those reported in Senior Project 16 (which almost entirely lacked descriptions of processes and linkages between events): ‘A needs analysis was conducted at (popular tourist destination)’, ‘We identified different types of visitors’, ‘The main disadvantages were…’, ‘We aligned the problems and the needs of the visitors’, ‘We saw an opportunity for an innovative approach’ and ‘This web application maps together the two most important things in (this tourist destination).’ The teacher could then ask students to, using these events, collaboratively write a report that puts the events into a series or process with linkages between steps. Last, we would suggest that assignment descriptions and rubrics take into account rhetorical functions and the language through which they are realized. For example, in the courses from which our data were drawn, the assignment descriptions included instructions such as that the reports should ‘communicate to a busy reader all important information’ and should ‘highlight important

130

Learner Corpus Research

conclusions, recommendations, and problems encountered or anticipated’; however, it could be helpful to have more explicit instructions stating that, for example, there should be little elaboration or extended description, and instruction could include examples of language that enacts these functions. Assessment rubrics could also reflect linguistic choices as they construe rhetorical functions.

6 Conclusion Using a combination of quantitative and quantitatively informed qualitative analyses, the present study identified rhetorical features of IS project report executive summaries, finding that this genre is written from a group perspective, connects current problems with future plans and is direct, factual and objective. Although there was no explicit instruction of these rhetorical functions, students were nonetheless able to include most of them in their writing through exposure to model texts. However, deviances from the model texts were observed, which may have resulted from conflicting exigencies due to the genre’s status as both a professional genre and a classroom assignment.

Acknowledgement This publication was made possible by NPRP grant #5-1320-6-040 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors.

References Airoldi, E.M., Anderson, A.G., Fienberg, S.E. and Skinner, K. K. (2006), ‘Who wrote Ronald Reagan’s radio addresses?’, Bayesian Analysis, 2: 289–320. Argamon, S., Whitelaw, C., Chase, P. and Hota, S. R. (2007), ‘Stylistic text classification using functional lexical features’, Journal of the American Society for Information Science and Technology, 58: 802–22. Biber, D. (1989), ‘A typology of English texts’, Linguistics, 27: 3–43. Biber, D. (1993), ‘Representativeness in corpus design’, Literary and Linguistic Computing, 8: 243–57.

Corpus-Driven Study of Information Systems Project Reports

131

Canagarajah, A. S. (2002), A Geopolitics of Academic Writing, Pittsburgh: University of Pittsburgh Press. Christie, F. and Maton, K. (2012), Disciplinarity: Functional Linguistic and Sociological Perspectives, London: Bloomsbury Academic. Collins, J. (2003), ‘Variations in written English: Characterizing the rhetorical language choices in the Brown Corpus of Texts’, Unpublished doctoral dissertation, Carnegie Mellon University. Cortes, V. (2004), ‘Lexical bundles in published and student disciplinary writing: Examples from history and biology’, English for Specific Purposes, 23: 397–423. Duff, P. A. (2001), ‘Learning English for academic and occupational purposes’, TESOL Quarterly, 35: 606–7. Elbow, P. (1999), ‘Individualism and the teaching of writing: A response to Vai Ramanathan and Dwight Atkinson’, Journal of Second Language Writing, 8: 327–38. Flowerdew, L. (2009), ‘Applying corpus linguistics to pedagogy: A critical evaluation’, International Journal of Corpus Linguistics, 14: 393–417. Flowerdew, L. (2015), ‘Using corpus-based research and online academic corpora to inform writing of the discussion section of a thesis’, Journal of English for Academic Purposes, 20: 58–68. Granger, S., Gilquin, G. and Meunier, F. (eds) (2013), Twenty Years of Learner Corpus Research: Looking Back, Moving Ahead, Louvain, Belgium: Presses Universitaires de Louvain. Harwood, N. (2005), ‘‘Nowhere has anyone attempted… In this article I aim to do just that’: A corpus-based study of self-promotional I and we in academic writing across four disciplines’. Journal of Pragmatics, 37: 1207–31. Hoey, M. (2005), Lexical Priming: A New Theory of Words and Language, London: Routledge. Hyland, K. (2004), Disciplinary Discourses: Social Interactions in Academic Writing, Ann Arbor, MI: University of Michigan Press. Hyland, K. (2008), ‘As can be seen: Lexical bundles and disciplinary variation’. English for Specific Purposes, 27: 4–21. Ishizaki, S. and Kaufer, D.S. (2007), ‘A model of rhetorical design strategies’. Paper presented at the conference of the American Association for Applied Linguistics, Costa Mesa, AZ, 21–24 April. Ishizaki, S. and Kaufer, D.S. (2012), ‘Computer-aided rhetorical analysis’, in P. McCarthy and C. Boonithum-Denecke (eds), Applied Natural Language Processing: Identification, Investigation, and Resolution, 276–96, Hershey, PA: Information Science Reference. Ishizaki, S. and Wetzel, D. (2008), ‘Computerized rhetorical analysis of L2 freshman placement essays’. Paper presented at the conference of the American Association for Applied Linguistics, Washington, DC, 29 March to 2 April. Johns, A. M. (1997), Text, Role, and Context, London: Cambridge University Press.

132

Learner Corpus Research

Kaneko, E., Rozycki, W. and Orr, T. (2009), ‘Survey of workplace English needs among computer science graduates’. Paper presented at IEEE International Professional Communication Conference, Waikiki, HI, 19–22 July. Kaufer, D.S. and Butler, B. S. (1996), Rhetoric and The Arts of Design, Mahwah, NJ: Lawrence Erlbaum. Kaufer, D.S. and Butler, B.S. (2000), Designing Interactive Worlds with Words: Principles of Writing as Representational Composition, Mahwah, NJ: Lawrence Erlbaum. Kaufer, D.S., Geisler, C., Ishizaki, S. and Vlachos, P. (2005), ‘Computer-support for genre analysis and discovery’, In Y. Cai (ed.), Ambient Intelligence for Scientific Discovery: Foundations, Theories, and Systems, 129–51. New York: Springer. Kaufer, D. S. and Hariman, R. (2008), ‘A corpus analysis evaluating Hariman's theory of political style’, Text & Talk, 28: 475–500. Kaufer, D. S., Ishizaki, S., Butler, B. S. and Collins, J. (2004), The Power of Words: Unveiling the Speaker and Writer’s Hidden Craft, Mahwah, NJ: Lawrence Erlbaum. Klebanov, B.B., Kaufer, D., Yeoh, P., Ishizaki, S. and Holtzman, S. (2016), ‘Argumentative writing in assessment and instruction: A comparative perspective’, in N. Stukker, W. Spooren and G. Steen (eds), Genre in Language, Discourse and Cognition, 167–92, Boston: de Gruyter. Klein, H. (2013). Text analysis Info: Category Systems. Available at: http://www. textanalysis.info (accessed 17 January 2017). Larman, C. and Basili, V. R. (2003), ‘Iterative and incremental development: A brief history’, Computer, 36: 47–56. Lee, D. Y. W. and Chen, S. X. (2009), ‘Making a bigger deal of the smaller words: Function words and other key items in research writing by Chinese learners’, Journal of Second Language Writing, 18: 149–65. Liu, M. X. and Murphy, D. (2012), ‘Fusing communication and writing skills in the 21st century’s IT/IS curricula’, Information Systems Education Journal, 10: 48–54. Marcellino, W. M. (2014), ‘Talk like a marine: USMC linguistic acculturation and civilmilitary argument’, Discourse Studies, 16: 385–405. Martin, J. R. (1992), English Text: System and Structure, Amsterdam: John Benjamins. Merhout, J. W. and Etter, S. J. (2005), ‘Integrating writing into IT/MIS courses’, International Journal of Information and Communication Technology Education, 1: 74–84. Miller, R. T. and Pessoa, S. (2016), ‘Role and genre expectations in undergraduate case analysis in information systems’, English for Specific Purposes, 44: 43–56. Nesi, H. and Gardner, S. (2012), Genres across the Disciplines: Student Writing in Higher Education, New York: Cambridge University Press. Pessoa, S., Miller, R.T. and Kaufer, D. (2014), ‘Students’ challenges and development in the transition to academic writing at an English-medium university in Qatar’, International Review of Applied Linguistics in Language Teaching, 52: 127–56.

Corpus-Driven Study of Information Systems Project Reports

133

Smagorinski, P. (1998), ‘How reading model essays affects writers’, in J. Irwin and M. A. Doyle (eds), Reading/writing Connections: Learning from Research, 160–76, Newark, DE: International Reading Association. Stein, B. and Eissen, M. (2008), ‘Retrieval models for genre classification’, Scandinavian Journal of Information Systems, 20: 93–119. Swales, J. (2014), ‘Variation in citational practice in a corpus of student biology papers: From parenthetical plonking to intertextual storytelling’, Written Communication, 31: 118–41. Upton, T. A. and Connor, U. (2001), ‘Using computerized corpus analysis to investigate the textlinguistic discourse moves of a genre’, English for Specific Purposes, 20: 313–29. Wardle, E. (2009), ‘“Mutt genres” and the goal of FYC: Can we help students write the genres of the university?’, College Composition and Communication, 60: 765–89. Witmore, M. and Hope, J. (2007), ‘Shakespeare by the numbers: On the linguistic texture of the late plays’, in S. Mukherji and R. Lyne (eds), Early Modern Tragicomedy, 133–53, Cambridge: Brewer. Wulff, S. and Gries, S. (2011), ‘Corpus-driven methods for assessing accuracy in learner production’, in P. Robinson (ed.), Second Language Task Complexity: Researching the Cognition Hypothesis of Language Learning and Performance, 61–88, Philadelphia: John Benjamins. Zhao, H. and Kaufer, D. (2013), ‘DocuScope for genre analysis: Potential for assessing pragmatic functions in second language writing’, in N. Taguchi and J. M. Sykes (eds), Technology in Interlanguage Pragmatics, 235–59, Amsterdam: John Benjamins.

7

Beyond Frequencies: Investigating the Semantic and the Stylistic Features of Phrasal Verbs In A Three-Year Longitudinal Corpus by Chinese University Students Meilin Chen

1 Introduction The essential role of multi-word expressions (MWEs) in successful language production has long been recognized in the field of applied linguistics based on both psycholinguistic (Conklin and Schmitt 2008; Jiang and Nekrasova 2007; Schmitt, Grandage and Adolphs 2004; Wray and Perkins 2000) and corpusbased evidence (Biber, Conrad and Cortes 2004; Biber et al. 1999; Sinclair 1991, 1996, 2004). PVs, like other MWEs, are omnipresent in the English language (Gardner and Davies 2007). Despite their frequency, PVs are perceived as notoriously difficult for ESL/EFL learners because they are often semantically non-compositional, polysemous and syntactically more flexible than other types of MWEs (e.g. variation of particle positions and pronoun or noun insertions are allowed in PVs). They are complex verbs that consist of two lexical items (a verb followed by an adverb particle) and yet function to some extent either lexically or syntactically like a single verb (Quirk et al. 1985: 1150). Give up, for instance, means ‘to abandon or stop doing something’ instead of ‘give’ + ‘up’. It acts as a single word in ‘He gave up smoking recently’, yet it allows pronoun insertion in sentences such as ‘It is hard to give it up’. Empirical SLA studies found that learners tend to avoid using PVs if a single-word verb is available, yet the evidence that accounts for the avoidance has not been conclusive (Dagut and Laufer 1985; Hulstijn and Marchena 1989; Laufer and Eliasson 1993; Liao and Fukuya 2004; Siyanova and Schmitt

Beyond Frequencies

135

2007). Corpus-based studies also reveal a mixed picture of PV use by ESL/ EFL learners, with learners of certain L1s showing little quantitative difference from native speakers (Chen 2013a,b; Hägglund 2001) while learners of other L1s are seen to underuse PVs (Gilquin 2011; Waibel 2007). This chapter aims to shed further light on the complex phenomenon of PV use by learners in the following ways. First, a longitudinal learner corpus was used to track the development of PV use by Chinese EFL learners. Second, semantic analyses were carried out to investigate students’ semantic knowledge of PVs, as they are highly polysemous. Third, a stylistic analysis was conducted to ascertain whether Chinese EFL learners are capable of using PVs appropriately.

2 Literature review 2.1 Cross-sectional learner corpus studies: Phrasal verbs While researchers and English language educators acknowledge the importance of learning PVs, they are less frequently addressed in the literature in comparison with other MWEs due to the syntactic flexibility of this structure. This makes it very challenging and time-consuming to query PVs in a corpus, which involves considerable manual checking. The following are a number of learner corpus studies that have been carried out in spite of the difficulties. The first learner-corpus-based study involving PVs was carried out by De Cock (2005). Working on data from the ICLE, she identified six problems that English language learners have with PVs: avoidance, style deficiency, semantic confusion, lack of collocational awareness, using ‘idiosyncratic’ multi-word verbs and syntactic errors. Also, based on ICLE, Waibel (2007) investigated the correlation between learners’ L1 and their use of PVs and found that, while the frequency of PVs in some sub-corpora of ICLE (e.g. French, Spanish, Italian) is lower than that in native writing in the Louvain Corpus of Native English Essays (LOCNESS), PVs in other sub-corpora (e.g. Dutch and German) are not fewer than those in LOCNESS. Gilquin (2011) carried out a case study of PVs with up in written and spoken interlanguage in comparison with those in native English. The results show that learners tend to use PVs more frequently in writing than native writers, yet their use of PVs in speech is significantly less frequent than that of native speakers. A later study by Gilquin (2015) yielded similar findings. Waibel (2007) attributes the discrepancies between the learners to their L1, that is, learners whose L1 (e.g. a Germanic language) possesses the PV

136

Learner Corpus Research

construction use PVs more frequently than those whose L1 does not (e.g. a Romance language); however, Chen (2013a,b) found otherwise. Chinese learners in her studies did not show considerable numerical differences from their British counterparts in the use of PVs, even though their L1 lacks such a structure. Schmitt and Redwood (2011) investigated the correlation between PV frequency and the acquisition of PVs by learners and found an obvious correlation between the learners’ knowledge of the multi-word verbs and the frequency of these verbs. Such a correlation was also found in Chen (2013b). These studies bring to light the correlation between intralinguistic (Gilquin 2015; Schmitt and Redwood 2011) as well as interlinguistic (Chen 2013b; Gilquin 2015; Waibel 2007) factors and learners’ PV production. However, very few studies investigated diachronically how learners use PVs. An exception is Chen’s (2013a) study, which offers insightful preliminary findings on learners’ PV use from a longitudinal perspective.

2.2 Longitudinal learner corpus studies: An under-researched area It is widely acknowledged that longitudinal research is inevitably a crucial method in SLA research, as learning a language other than L1 is a rather complex process that takes years or even a lifetime (Mellow, Reeder and Foster 1996; Ortega and Iberri-Shea 2005). Notwithstanding its acknowledged importance, longitudinal research is scarce compared with its counterpart, cross-sectional study, due to the difficulties in conducting such research. Similarly, longitudinal research in LCR has long been in a peripheral position. A telling indication is the statistics given by Meunier and Littré (2013: 63) about the number of longitudinal corpora available so far: ‘of the 107 learner corpora listed on the “learner corpora around the world” Web page, only 12 include longitudinal data’. Despite the recent and ambitious effort made by The Modern Language Journal, which managed to ‘gather a collection of robust empirical investigations that would reflect the plethora of previously underexplored possibilities for capturing aspects of L2 development’ for the special issue of 2013 (Hasko 2013: 2), Meunier’s (2015) most recent review of longitudinal learner corpora shows that the number of studies has not increased much. Paquot and Plonsky (2017) did a comprehensive review of quantitative learner corpus studies conducted during the past twenty-five years and found that longitudinal studies account for not more than 5 per cent of the total.

Beyond Frequencies

137

It is without doubt that this 5 per cent has contributed greatly to the understanding of L2 language development. These studies, however, suggest further improvements that can be made in such studies. Myles (2008), for instance, points out that the longitudinal studies conducted so far often focus on a very limited number of subjects. Studies by Li and Schmitt (2009) and Spoelman and Verspoor (2010) both had only one subject. Byrnes (2009) investigated fourteen German L2 learners’ writing development. Although Vyatkina (2012) collected longitudinal data from a group of more than twenty students, due to subject attrition, the group data were only used as cross-sectional data. The longitudinal analysis of her study is only concerned with two individuals. Myles (2008) argues that findings from individuals make it problematic to generalize conclusions; therefore, she advocates the use of longitudinal learner corpora that have data collected from hundreds or even thousands of learners. The discussion points to the necessity of collecting truly longitudinal data from a large population of L2 learners. The Longitudinal Database of Learner English (LONGDALE) built by the research team led by Professor Sylviane Granger is an answer to this call. Inspired by their project, this chapter aims to investigate Chinese learners’ acquisition of PVs based on a three-year longitudinal corpus of essays by a group of over 100 students. Analyses were carried out to explore answers to the following research questions. 1. 2. 3.

Have the Chinese learners made any progress in acquiring PVs, that is, is there an increase in the number of PVs used in their writing? Do the students show any progress in their semantic knowledge of PVs, that is, can they use a PV in more meanings in writing? Can they use PVs more appropriately in style as their study proceeds?

3 Corpora and methodology The corpus was designed to cover three academic years, and the genre of writing was the argumentative essay. While corpus-based findings (Biber et al. 1999) and our intuition suggest that PVs tend to be more colloquial in style, the rationale for using written rather than spoken data is as follows. First, PVs being colloquial in style does not necessarily mean that they are not frequently used in writing. Gardner and Davies (2007: 347) find that PVs occur approximately ‘every 192 words, that is, almost two phrasal verbs per page of written text on average’ in the BNC. Liu (2011) found that among the 150 most frequent PVs

138

Learner Corpus Research

from the BNC and COCA, 65 are equally frequent in five registers (i.e. spoken, fiction, magazine, newspaper and academic), 58 are significantly more frequent in fiction, 2 are significantly more frequent in the academic register, while only 24 are more frequent in the spoken register than in the others.1 Second, argumentative writing is one of the most typical genres produced by the students during their undergraduate studies. It is also the essential writing component of the national proficiency test they were required to take in their second year of study. This longitudinal corpus, therefore, represents the type of writing that students typically produce for undergraduate education in China, rather than writing that is produced for the sake of data collection. All the writing prompts were decided by the course teachers, and no explicit requirement for PV use was given. Around 150 English language majors were chosen randomly for the data collection on the grounds that they were willing to participate in the study and gave permission to the author to use their essays. When the first set of data was collected, these students had just started their study at a university in Mainland China. When they entered university, they had been studying English for at least six years. At the end of their second year of study, they were required to take a national proficiency test for English language majors. This exam requires mastery of a 6,000-word vocabulary including MWEs. A pass score for this exam is roughly equivalent to a 6.5 score for the IELTS exam or 550 for paper-based TOEFL. The learners, therefore, could be considered intermediate to upper intermediate EFL learners. Over the three-year span, altogether six 200-to-300-word essays were collected from each of the participants (two per year). The essay was the writing component (with a thirty-minute time limit) of the final exam of an English course. Altogether 780 essays by 130 students were chosen for the corpus. Essays by those who missed one or more final exams were not included. The data were then put into the computer, edited and PoS-tagged by using the online free CLAWS 7 PoS tagger.2 Table 7.1 details the corpus information. For the data analysis, PVs were extracted from the corpora and determined through the following procedure. First, all co-occurring lexical verbs and adverbial particles were extracted from the corpus using the Concord function of WordSmith Tools 5.0 (Scott 2008). The second step involved a manual check in order to rule out verb + prepositional phrases (e.g. They acted out of concern for anyone who may suffer from the same disease), and the third step was to check all the results in PV dictionaries in order to determine which meanings made them PVs (e.g. These children acted out their anger in the previous game)

Beyond Frequencies

139

Table 7.1 Profiles of the longitudinal Chinese learner corpus Corpus

School level

No. of SS

No. of essays

Corpus size

Mean text Standard length (tokens) Deviation

CH-01

Year 1

130

260

57,280

220

58.58

CH-02

Year 2

130

260

66,331

255

40.57

CH-03

Year 3

130

260

65,017

250

57.50

or verb + particle free combinations (e.g. I don’t want to venture out when it’s pouring), using: Collins COBUILD Dictionary of Phrasal Verbs 2nd Edition (Sinclair et al. 2002), Longman Dictionary of Phrasal Verbs (Longman hereafter) (Courtney 1983) and Oxford Phrasal Verbs Dictionary for Learners of English (Parkinson 2001). PVs obtained through the abovementioned procedures were then analysed quantitatively and qualitatively in order to explore answers to the research questions. The next section presents the results and findings.

4 Results and discussion 4.1 Overall frequencies The overall frequency of PVs in the three sub-corpora does not reveal a correlation between the learners’ use of PVs and their school level. They did not use more PVs in writing as their learning of English proceeded. As can be seen from Table 7.2, the relative frequency of PV tokens in their first-year writing was thirty-four per ten thousand words, but the frequency declined greatly to 27 (chi-square: x2 = 4.61, p = 0.032) in Year 2. In Year 3, the frequency of PV tokens in the learner writing rose back to thirty-one per ten thousand words but was still slightly lower than that in Year 1, although the difference is not statistically significant (chi-square: x2 = 0.51, p = 0.477). To ascertain the impact of the individual idiosyncrasies on the overall frequencies, a statistical test was carried out using Gries’s (2012: 2) measure of dispersion: DP. Ranging from 0 to 1, the DP value indicates the degree of dispersion. Values near 0 indicate that PVs are evenly distributed across writers in proportion to the length of their essays, large values near 1 indicating otherwise. Following Liu’s (2011) categories, sub-corpora with DP values below 0.25 are considered as evenly distributed across writers, those with values between 0.25 and 0.49 would be not evenly distributed and values at 0.5 and above would be very unevenly distributed.

140

Learner Corpus Research

The DP values given in Table 7.2 show that PVs are evenly distributed across writers in CH-01 and CH-02 (0.2469 and 0.2401 respectively) but the distribution is slightly uneven across writers in CH-03 (0.2569). Therefore, individual idiosyncrasies seem not to have an impact on the overall frequencies of PVs in CH-01 and CH-02 but show a visible though not strong impact on those of CH-03. The unevenness is probably caused by the fact that a few Year 3 essays were significantly longer than the mean length (Mean length: 250 words, SD: 57.5), yet no PV was used in these essays. It is too soon to conclude that these few students did not make any progress in Year 3, given that they were able to produce longer texts than the others within the same limit of time. It might be the case that they opted for single-word verbs, which they found more suitable for writing. It is worth noting, though, that the obvious decline in the use of PVs in Year 2 sheds light on the difficulty of English PVs. As explained in Section 3.1, the students are English language majors and required to take a national English proficiency test by the end of the second year, the vocabulary section of which includes questions about PVs. They were expected to study or review PVs during the preparation for the test. The unexpected drop in the use of PVs could only be explained by further investigations through interviewing the learners. However, possible explanations could still be advanced for discussion. First, the decrease of PVs in the learners’ second-year writing may indicate that SLA is a dynamic and complex rather than a simple linear process. The pattern of PV use here is in accordance with the ‘U shape of learning’ witnessed in SLA (Ellis 1994). Second, the learners’ output does not necessarily equal their productive knowledge. Too much explicit input of PVs during the preparation for the national exam might lead to the opposite result, that is, the learners were tired of doing drill exercises of PVs for the exam and in consequence used fewer Table 7.2 Frequencies of PVs in the learner sub-corpora PV types

PV tokens

Corpus

Abs.

Rel.

Abs.

Rel.

DP

CH-01

85

15

193

34

0.2469

CH-02

75

11

179

27

0.2401

CH-03

96

15

204

31

0.2569

Note: Abs. = absolute frequency. Rel. = relative frequency. The relative frequency is calculated by normalizing the absolute frequency to a ten-thousand-token basis.

Beyond Frequencies

141

Table 7.3 Overlapping PVs in the learner sub-corpora CH-01 CH-01 CH-02 CH-03

CH-02

CH-03

No.

31/85

40/85

%

(36.47)

(47.06)

No.

31/75

39/75

%

(41.33)

(52.00)

No.

40/96

39/96

%

(41.67)

(40.63)

PVs. Finally, the less frequent use of PVs could also be an indication of the learners’ growing awareness of the ‘less academic’ nature of PVs, which led them to opt for other more formal alternatives. This possibility will be investigated in Section 4.3. In addition to the tentative explanations given above, a comparison was made between the specific PVs in the sub-corpora in order to ascertain whether the learners used similar PVs at different stages. If this were the case, the assumption that the learners have not made much progress in their first three years of study would be confirmed. Table 7.3 summarizes the results. The comparison yielded 22 PVs that appear in all of the three sub-corpora, which constitutes only 13.1 per cent of the 168 PV types in the whole learner corpus. This indicates that, although the learners did not exhibit a numerical increase in their production of PVs in writing, they did show a varied repertoire of PVs as they used different PVs at different stages. Furthermore, the second row of Table 7.3 shows that even though the learners used fewer PVs in Year 2, this does not necessarily mean that they did not make any progress in PV acquisition. Only 36.47 per cent of the PVs from CH-01 were found in CH-02, so the majority of the PVs in their second year were very different from those in Year 1. No direct support can therefore confirm that the learners made little progress in acquiring PVs. The analyses of semantic and stylistic behaviours of PVs in the rest of Section 4 may reveal more details about this learning process.

4.2 Semantic behaviour of phrasal verbs in the learner sub-corpora Being polysemous is one of the factors that contribute to the difficulty of PVs in SLA. For example, go down (only as a PV) has twenty-three different senses in the dictionaries, put out has twenty-two and pick up has twenty-one. Many of the senses are so different that each sense makes the PV virtually a different lexical

142

Learner Corpus Research

item. Take put out as an example: it means ‘extinguish’ in: He put the fire out; it also means ‘broadcast’ in: The pilot put out a radio message; and if a boat or ship puts out, it leaves its harbour. As each sense represents a different usage of the PVs, the number of senses these PVs have in the sub-corpora may reflect how productive the learners were at different stages. To calculate the number of senses, in which PVs were used across the learner sub-corpora, all the PV occurrences were looked up in the three PV dictionaries, as mentioned in Section 3.2. Although the number of senses of a PV included in these dictionaries may vary, the majority of the meanings overlap. In this study the number of senses a PV type possesses is determined by the dictionary that provides the largest number of senses in order to include as many senses as possible for each PV type. The total number of dictionary definitions of PVs that appear at least twice in each learner sub-corpus was then calculated.3 The average number of senses per PV type across the sub-corpora (see Figure 7.1) reveals a similar pattern to that of the overall results in Section 4.1. The Chinese students used, per PV type, 1.41 different meanings in their first-year writing. This number dropped to 1.21 meanings per PV type in Year 2. The average number of meanings rose to 1.38 in Year 3, but it was still slightly lower than that in Year 1. The results appear not to reveal much semantic progress in the learners’ use of PVs. However, it is worth noting that the result might be affected by the total number of dictionary meanings of the PVs. If the students use a PV in four out of ten dictionary meanings while another PV in three out

Figure 7.1 The average number of senses per PV type across the learner sub-corpora.

Beyond Frequencies

143

Figure 7.2 Proportions of dictionary meanings of PVs across the learner sub-corpora.

of five meanings, they, in fact, use the latter more productively than the former, even though the average number of senses of the latter is smaller than that of the former. Therefore, the proportions of dictionary senses of PVs across the three subcorpora were then calculated, which reveals a more promising picture (see Figure 7.2). The learners used 17.9 per cent of the senses listed in dictionaries in their first-year writing. This dropped slightly to 15.74 per cent in Year 2; however, they showed considerable progress in Year 3 by using 22.37 per cent of the dictionary senses. This indicates a correlation between the learners’ acquisition of PVs and the accumulation of their semantic knowledge of the PVs they already know rather than the number of PVs they use in writing. As for the slight drop in Year 2, in addition to plausible explanations, as given in Section 4.1, the overall low frequency of PVs may also contribute to the smaller semantic variety of PVs in Year 2. To further explore the learners’ semantic knowledge of PVs, we compared the results with the key senses of 150 high-frequency PVs identified by Garnier and Schmitt (2015). They analysed the 150 most frequent PVs in COCA based on a list of senses derived from nine dictionaries and WordNet Search 3.1 and found that, while most PVs are polysemous (more than ten senses), they have a smaller number of key senses and the most frequently used meaning often accounts for at least 50 per cent of the PV’s occurrences in COCA (659). As can be seen from Table 7.4, except for come out and make up, the number of senses of PVs in the learner writing are the same as, or greater than, that of key senses in Garnier and Schmitt (2015). While the number of senses of go out

CH-01

144

Table 7.4 PVs that are used in at least two different senses CH-02

CH-03

PV

No. of senses

No. of PV senses(G & S)

No. of senses

No. of PV senses(G & S)

No. of senses

No. of senses(G & S)

grow up

3

1

go out

3

2

give up

2

1

give up

3

1

go back

2

1

build up

2

1

3

come out

2

4

pour out

2

N/A

2

2

get out

2

1

set up

2

2

go on

2

2

go on

2

2

make up

2

3

keep on

2

1

come out

2

4

go out

2

2

put forward

2

N/A

get up

2

1

break down

2

4

pay back 2

N/A

come back

2

1

stand out

2

get together

2

N/A

go on

2

2

go out

2

2

grow up

2

1

keep away

2

N/A

wipe out

2

N/A

2

Note: No. of senses (G & S) = the number of key senses in Garnier and Schmitt (2015)

Learner Corpus Research

make up 2 set up

Beyond Frequencies

145

in the learner writing is bigger than that in Garnier and Schmitt, due to the fact that the literal (leaving a room, building, etc.) and figurative (leaving a place for social purposes) meaning of go out is combined in their study but split in this study, most of the PVs are used by learners in a variety of senses. For instance, according to Garnier and Schmitt, grow up and give up are 90 per cent and 80.5 per cent of the time used in one key meaning respectively. While most of the time the learners used grow up in its key sense identified by Garnier and Schmitt in their first-year writing, they also used the PV in other senses as in examples 3 and 4. Apart from using give up in its key sense (Example 6), they also used it in two other ways as in examples 7 and 8. Grow up and give up in Garnier and Schmitt (2015): (1) Gradually advance in age and maturity (98 per cent) ‘Seeing my kids growing up is such a lovely thing.’ (2) Stop doing or having something; abandon (activity, belief, possession) (80.5 per cent) ‘She had to give up smoking when she got pregnant.’ Grow up and give up in learner writing: (3) ‘Skyscrapers *is growing up just like bamboo *shots after a delicate spring shower with the development of the whole world. No wonder that skyscrapers are signs showing that we ...’. (Longman: to become bigger in an upward direction) – CH-01 (4) ‘She was always smiling, which made me very happy. I loved her. Now, I grow up. I realize that it is difficult for teachers to educate students.’ (Longman: [of children or young animals] to become older and bigger; develop towards manhood, womanhood, full size, etc.) – CH-01 (5) ‘With the development of our society, our ideas are also growing up. We began to explore the space. Our country’s history is one of a picture of it.’ (Longman: to arise) – CH-01 (6) ‘As we grow up day by day, we have to take the truth into account and give up many impossible dreams.’ (Longman: to stop doing or having [something]; willingly or unwillingly lose [something or someone]; get rid of; do without) – CH-01 (7) ‘I failed in the entrance exam in 2007. But I never *gived up and made my *determine to study hard. Finally, I become a university student.’ (Longman: to stop attempting [something]) – CH-01

146

Learner Corpus Research

(8) ‘… including a large amount of good teachers. They gave up their lives in order to save their students without thinking when the critical moment came.’ (Longman: to offer to do without [something such as time or money], usu. for a good purpose) – CH-01 Although the learners did not use come out in all the four key senses identified by Garnier and Schmitt (2015), they displayed a good command of figurative meanings of the PV. Garnier and Schmitt find that come out is used in its literal meaning 38 per cent of the time, but such literal use is not found in either CH-02 or CH-03. In Year 2, students mainly used the PV in senses as given in examples 13 and 15, and in Year 3 mainly in senses as in examples 14 and 15. The results appear to be rather encouraging given that the texts in COCA are much longer than the student essays in this study. Come out in Garnier and Schmitt (2015):

(9) Leave a place (room, building, container) or appear from it (38 per cent) She went into the bank and came out with some money. (10) Become known or revealed after being kept secret (13.5 per cent) The news came out that he was leaving the team. (11) (Come out and do STH) Make public knowledge a privately held position (11.5 per cent) People need to come out and say what they think about it. (12) Become available or released to the public (film, record, book) (10 per cent) Their new album is coming out next month. Come out in learner writing:

(13) ‘… mobile surely give us a lot of convenient, but a lot of problems come out after the spread of mobile.’ (Longman: to be clear or known) – CH-02 (14) ‘And then many terms, many people, many opinions about linguistics come out like flood.’ (Longman: to appear; be seen or heard) – CH-03 (15) ‘But, we all know that the issue of environment pollution has come out for a long time. According to some surveys, developed countries …’ (Longman: [of info.] to be discovered) – CH-03 Another important feature of PVs is being non-compositional. While literal meanings of PVs may be easy for learners to comprehend and acquire (e.g. Peter wants to go out and have a walk), figurative meanings (e.g. When money goes out, we will be in trouble) might require more effort to learn. The Longman dictionary provides additional semantic information, that is, an asterisk in front of a definition

Beyond Frequencies

147

Figure 7.3 Proportions of dictionary figurative meanings used across learner subcorpora.

if it is a figurative meaning. Such information was recorded during the manual check for analysis. As shown in Figure 7.3, the Chinese learners’ semantic knowledge of PVs appears to improve after two years of undergraduate study. In Year 1, 16.63 per cent of PV use in their writing was figurative. Although this fell to 13.04 per cent in Year 2, the figurative use of PVs in their writing increased to 19.34 per cent in Year 3. This finding, combined with the aforementioned results, reveals the learners’ progress in their semantic knowledge of PVs. They did use PVs not only in a greater variety of meanings as their learning proceeded but also more often in figurative senses.

4.3 Stylistic features of the phrasal verbs in the learner sub-corpora To explore the stylistic features of PVs used in the learner sub-corpora, the fifteen most frequent PVs were selected for analysis. The top fifteen PVs were searched in the four sub-corpora (news, fiction, spoken and academic) of COCA and the BNC. The distribution of the PVs was then calculated by using Gries’s (2008, 2009, 2012: 2) measure for dispersion: DP (for ‘deviation of proportions’). Based on the frequency and distribution in both corpora, a stylistic label was given to the PV. Table 7.5 gives three examples of the stylistic profile of PVs. Bring about is labelled an ‘academic’ PV, as it is not evenly distributed across the sub-corpora of both the BNC and COCA (DP code: 2) and appears much more frequently in the academic sub-corpus than in the other sub-corpora. Find out is considered a fiction or spoken PV because it

148

Learner Corpus Research

Table 7.5 Examples of the stylistic profile of PVs

PV

Stylistic feature

bring about Academic find out make up

Rel. freq. in the sub-corpora Corpus News Fiction Spoken Academic DP code BNC

11.73 5.70

8.26

37.47

2

COCA

6.51

10.72

27.44

2

BNC

53.37 119.67 118.54

30.07

2

COCA

50.84 99.41

162.08

22.35

2

Evenly BNC distributed COCA

56.94 62.69

60.71

57.17

1

56.85 61.65

54.35

46.91

1

Fiction / Spoken

2.80

Note: 1 = fairly evenly distributed (DP value