Computational Semiotics 9781350166615, 9781350166646, 9781350166622

Citation preview

Computational Semiotics

Bloomsbury Advances in Semiotics Semiotics has complemented linguistics by expanding its scope beyond the phoneme and the sentence to include texts and discourse, and their rhetorical, performative, and ideological functions. It has brought into focus the multimodality of human communication. Bloomsbury Advances in Semiotics publishes original works in the field demonstrating robust scholarship, intellectual creativity, and clarity of exposition. These works apply semiotic approaches to linguistics and non-verbal productions, social institutions and discourses, embodied cognition and communication, and the new virtual realities that have been ushered in by the Internet. It also is inclusive of publications in relevant domains such as sociosemiotics, evolutionary semiotics, game theory, cultural and literary studies, humancomputer interactions, and the challenging new dimensions of human networking afforded by social websites. Series Editor: Paul Bouissac is Professor Emeritus at the University of Toronto (Victoria College), Canada. He is a world renowned figure in semiotics and a pioneer of circus studies. He runs the SemiotiX Bulletin [www​.semioticon​.com​/semiotix] which has a global readership.

Titles in the series include: Cognitive Semiotics, Per Aage Brandt Music as Multimodal Discourse, edited by Lyndon C. S. Way and Simon McKerrell Peirce’s Twenty-Eight Classes of Signs and the Philosophy of Representation, Tony Jappy Semiotics of the Christian Imagination, Domenico Pietropaolo The Languages of Humor, edited by Arie Sover The Semiotics of Caesar Augustus, Elina Pyy The Semiotics of Clowns and Clowning, Paul Bouissac The Semiotics of Emoji, Marcel Danesi The Semiotics of Light and Shadows, Piotr Sadowski The Semiotics of X, Jamin Pelkey The Social Semiotics of Tattoos, Chris William Martin

Computational Semiotics Jean-Guy Meunier

BLOOMSBURY ACADEMIC Bloomsbury Publishing Plc 50 Bedford Square, London, WC1B 3DP, UK 1385 Broadway, New York, NY 10018, USA 29 Earlsfort Terrace, Dublin 2, Ireland BLOOMSBURY, BLOOMSBURY ACADEMIC and the Diana logo are trademarks of Bloomsbury Publishing Plc First published in Great Britain 2022 Copyright © Jean-Guy Meunier, 2022 Jean-Guy Meunier has asserted his right under the Copyright, Designs and Patents Act, 1988, to be identified as Author of this work. Cover image © monsitj / Ryzhi / agsandrew / koto_feja / iStock All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage or retrieval system, without prior permission in writing from the publishers. Bloomsbury Publishing Plc does not have any control over, or responsibility for, any third-party websites referred to or in this book. All internet addresses given in this book were correct at the time of going to press. The author and publisher regret any inconvenience caused if addresses have changed or sites have ceased to exist, but can accept no responsibility for any such changes. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress. ISBN: HB: 978-1-3501-6661-5 ePDF: 978-1-3501-6662-2 eBook: 978-1-3501-6663-9 Series: Bloomsbury Advances in Semiotics Typeset by Deanta Global Publishing Services, Chennai, India To find out more about our authors and books visit www​.bloomsbury​.com and sign up for our newsletters.



To Suzanne and Pierre-André: the dynamic signs of my life.



Contents List of illustrations Preface 1 2 3 4 5 6 7 8 9 10 11 12

viii x

The complexity of semiotics 1 Semiotics in computing 17 Computing in semiotics 35 Models in science and semiotics 45 Conceptual models in science 61 Conceptual models in semiotics 73 Formal models in science 91 Formal models in semiotics 109 Computational models in science 133 Computational models in semiotics 155 The workflow of computational semiotics 183 Computer models in science and semiotics 201

Notes References Name Index Subject Index

211 223 252 256

Illustrations Figures 1.1

A variety of ‘texts’ in a computer environment, distinguished by their physical medium 2.1 Llull’s paper wheel machine 2.2 Watt’s fly-ball governor 2.3 Cybernetic loop 2.4 Shannon’s communication graph 3.1 Jan Vermeer, Woman Holding a Balance 4.1 A battery symbol as used in electrical circuit diagrams 4.2 Chaotic attractor of the Rabinovich–Fabrikant system 4.3 Relations between the four models and their sub-models 4.4 Models for a hurricane 4.5 Models for the Kanizsa triangle 4.6 Models for the semiotic analysis of Magritte’s paintings 6.1 Julien Viaud, alias Pierre Loti, Easter Island statues 6.2a–c National Communist Party flags 6.3 Visualization of an audio waveform 7.1 Mapping from an equation to an object: The pendulum 7.2 Mapping of a function between two sets 7.3 Formal reasoning as a two-way process 8.1 Eight world-famous statues 8.2 The semiotic square 8.3 Traffic panels 8.4 Diagrammatic tree of the features of the statues 8.5 Generated Koch curves 8.6 Graph of a game narrative 8.7 Four motherland types 8.8 Vector representation of Pipe and Amants 9.1 Turing Machines A & B 9.2 Pseudocode and flow chart 9.3 Penrose’s Polyomino Tilings 9.4 Functional mapping of a pendulum 9.5 Examples of Minsky type frames 9.6 Visualization of three equations describing three dimensions of Covid-19 9.7 How Coronavirus is Devastating the Restaurant Business 10.1 Schank’s dependency graph

10 17 19 21 22 37 50 51 56 56 57 58 75 81 87 97 98 105 114 117 118 119 119 120 120 124 134 137 139 141 145 151 152 161

 Illustrations ix 10.2 10.3 10.4 10.5 10.6 10.7 10.8 10.9 10.10 10.11 10.12 10.13 10.14 10.15 10.16 11.1 11.2 11.3 11.4 11.5 11.6 11.7 11.8 11.9 11.10 12.1 12.2 12.3 12.4 12.5

Conceptual graph of a sentence Features of the statues in knowledge base form (a list) Ontology of samples of the world largest statues Google’s output or ‘knowledge panel’ on ROME Word co-occurrence network in Jn 1.1-2 (Czachesz 2017) Matrix to a vector Cartesian space and a dot representation Class formation of dot vectors Probabilistic decision for data class splitting K-nearest neighbours classifier Neural net classes Hierarchical clustering Principal component classes K-means classifier Associative classification The abductive learning spiral A typical computational semiotic enquiry pipeline Transcoding of the Mona Lisa into binary form Graphs of the descriptors of nudity in Magritte’s paintings Waveform signature of a musical piece Clusters of the concept of mind in Peirce’s Collected Papers Word cloud for the ‘mind’ and ‘sign’ cluster Word clouds of the ‘mind’ and ‘laws of ideas’ cluster Annotations of segments of the class ‘sign’ Automatic summary of segments Researcher’s final synthesis of the cluster ‘sign’ for the conceptual analysis of ‘mind’ Chinese abacus Schickard’s calculator and the Pascaline: Arithmetic machines Charles Babbage’s Difference Engine Hand crank mechanical calculator, c. 1930 Basic von Neumann computer architecture

161 163 164 166 168 173 173 174 175 175 176 177 177 178 180 185 186 191 193 194 195 195 197 197 198 201 202 203 204

205

Tables 6.1 8.1 8.2 8.3 8.4 9.1 10.1 10.2

Descriptors for Three Magritte Paintings List presentation of statues’ features Data sheet representation of the statues’ features Vector/Matrix representation of the features of the statues Vector représentation of two Magritte’s paintings Extensional Definition of a Function in a Frame-like Presentation A Minsky frame format with slots A filled Minsky frame

83 121 122 123 130 143 161 161

Preface We humans build proxies, surrogates or artefacts by which we situate ourselves in the world, communicate with others and even come to know ourselves. These processes and artefacts are often called representations, signs, signals or icons. Such notions are very general and open to all kinds of definitions. Still, they are omnipresent in all human sciences, including philosophy, anthropology, linguistics, mathematics, literary studies, aesthetics, economics and studies in architecture, religion, communication and media. And now, we increasingly see them making their way into natural sciences. They are found in biology, artificial intelligence and even in neuroscience. One particular feature of these ‘representational artefacts’ is that they are carriers of meaning. It is this feature which confers them their intriguing complexity. And whoever wishes to study them immediately becomes entangled in the relational nature of meaning, which is one that makes these artefacts ‘stand for’ something else to somebody. Any expert who embarks upon this type of study is immediately thrown out of his or her comfort zone. How indeed does one explain the meaning of the NotreDame de Paris cathedral, of the Stonehenge monument or of the pyramids of Egypt, of the Easter Island Moai statues, of the presidential faces on Mount Rushmore? These artefacts are not just huge piles of rocks or bricks that have atomic structures and which hold together by gravity or cement. They are there because they are first and foremost carriers of meaning built by humans. The meaning of certain simple artefacts may be evident to everyone. For instance, the meaning of a wedding ring on one finger may be quite simple to explain. But things become less so when contemplating some of the complex artefacts listed above. Some are even more complex: What is it about Beethoven’s Symphony No. 9, Shakespeare’s Romeo and Juliet, the pilgrimage to Mecca, a benediction by the pope or a visit from the Dalai Lama? What makes an artefact an important carrier of meaning? Can there be some scientific enquiry about such semiotic artefacts? It so happens today that there are two opposite academic disciplines which have overtly put these artefacts at the core of their enquiries: artificial intelligence and semiotics. For the founding fathers of artificial intelligence, signals are what computers process, but what these machines ‘manipulate’ are symbols (Newell 1983). And both signals and symbols are artefacts that ‘stand for’ something else and ultimately do so for somebody. This is why many semioticians have seen the computer as a semiotic machine (Nadin 2007, 2011). And for the founding fathers of semiotics, from Plato to Augustine and from Peirce to Saussure and for so many other semioticians today, it is also this ‘stand for’ relation of signs that is so intriguing and complex. And recently, some semioticians have come to believe that because it manipulates symbols, a semiotic machine such as a computer

 Preface xi may participate in the study of these semiotic artefacts and processes. So, maybe an encounter between computation and semiotics is possible – that is, there may indeed be a place for computational semiotics. Some semioticians explored this through the study of the semiotics of computation. Others took the opposite path: they explored the possible usage of computers in the realm of semiotics. Both are heuristic. It is the second direction that was taken in this book. How may computer technology and the formal structure of mathematical computation encounter semiotics as an epistemic endeavour for analysing and explaining meaning-carrying artefacts and the processes that build this meaning? Can computation help us understand this meaningmaking process? Such is the encounter which is explored in this book. For me, exploring possible answers to these questions has been the focus of some fifty years of research. It is a matter which rests upon the dynamics of conceptual conflicts and paradoxical explanation. It was an adventure into two independent and divergent territories that were said to never share interests. It forced experts in electronic signal-processing technology to dialogue with experts in the interpretation of meaning-carrying artefacts. It forced semiotic artefacts and semiosis to enter the world of computation. Could such a paradoxical project become computational semiotics? This book has two types of readers in mind. It is mainly written for semioticians who are comfortable with using computers in research but who are very resistant to accept computation within the interpretative practice of semiotics, as it would bring them into the naturalist and formalist world. I just hope to show that, on the contrary, computers can enhance the interpretative endeavour of semioticians, as computation may provide great assistance to their irreplaceable insight. On the other hand, the essay is also written with computer scientists in mind. They will not learn much on the definition of computation in this essay. But they surely will become more sensitive to the complexity of semiotic artefacts and processes. This will be especially relevant when they encounter the barrier of meaning or use oracles in their own computations applied to semiotic artefacts. When I started this slow-paced research in the late 1960s, I had asked computer scientists if one day I could use a computer to manipulate texts or other semiotic artefacts. The answer was: No, you can’t. Later, when AI developed, the answer to the same question increasingly became: Yes, you can. And the research became: How can I? But now my preoccupation is more: Why can I? Today, such encounters of computers with semiotic artefacts and processes are everywhere: from e-texts to e-games and from e-music to e-films, computers are so extensively devoted to semiotic artefacts that they have become processors of e-artefacts. Nevertheless, for me, the same question remains. It is still a quest for a better answer and understanding. How is it that we can use computers in the encounter of semiotic artefacts? Are we at a dead-end or on the path towards rational enrichment? For me, this endeavour has been more than a theoretical question and a technological project. It has also been a wonderful collective experience with colleagues and friends who have shared the darkness and light of research. This path was shared by Ismail Biskri, Jean-Pierre Desclés, François Rastier, Dominic Forest, Francis Rousseau, Serge Robert, Pierre Poirier, Ghyslain Lévesque, Stevan Harnad, Vincent Rialle, Christophe

xii

Preface

Malaterre, David Piotrowski, Julien Longhi, Louis Hébert, Jean-Claude Simard, Elias Rizkallah, Georges Leroux and the many students with whom, over the years, so many discussions were held: Georges Nault, Yassine Gargouri, Jean-François Chartier, Louis Chartrand, Davide Pulizzotto, Louis Rompré, Maxime Ste-Marie, Julien Vallières, Francis Lareau, Alaidine Ben Ayed, Jean Danis, Toufik Mechouma and so many others. Finally, I am grateful for the positive critics the three anonymous evaluators have given to the manuscripts. They allowed fine-tuning and the enrichment of many of the arguments and the thesis of this book. The intuition for this book came from Paul Bouissac who, for many years, has discreetly seen the pertinence of my research for semiotics and who invited me to write this book. Throughout the years, this research has been greatly supported by the Social Sciences and Humanities Research Council of Canada and the Université du Québec à Montréal. The book itself received meticulous, patient and unwavering attention from Lila Roussel and Ludovic Chevalier. But the greatest source of energy has been my wife Suzanne, my son Pierre-André and my grandson Cedric, who are the best carriers of meaning that life can give you.

1

The complexity of semiotics

Man has, as it were, discovered a new method of adapting himself to his environment. Between the receptor system and the effector system, which are to be found in all animal species, we find in man a third link which we may describe as the symbolic system. This new acquisition transforms the whole of human life. As compared with the other animals man lives not merely in a broader reality; he lives, so to speak, in a new dimension of reality. (Cassirer 1944: 92)

Introduction In classical semiotics, one of the main human cognitive abilities is the use of signals and symbols to deal with one’s environment, one’s peers and oneself. And in contemporary semiotics, manipulating signals or symbols is a capability inherent to all living organisms. In humans, this ability manifests through the production of complex meaningcarrying artefacts. The most sophisticated type of such artefacts is natural language. But it is not the only one. Indeed, in their self-cognition, as well as in their interaction with others and with the environment, humans are made to constantly negotiate with a diversity of signals and symbolic artefacts. Some are simple – jewellery, make-up, dress codes and so on. Others are slightly more complex – salutations, codes of seduction, table manners, birth ceremonies, funeral processions, marriage contracts, royal coronations, parliament opening ceremonies and so on. Some are highly sophisticated and take various forms: aesthetic (e.g. the arts), cultural (e.g. ceremonies), religious (e.g. churches), technical (e.g. electrical blueprints), scientific (e.g. chemistry symbols), economic (e.g. money), etc. And all living organisms present similar capabilities, but often these will be less complex. From the use of signals to basic symbols, they allow communication among members of a community. Practically no domain of human activity is devoid of the use of signals and of symbolic artefacts. Most have a specific signature, structure and usage. In fact, they may form one of the most complex systems that humans have to learn, master, share and, above all, interpret.

2

Computational Semiotics

All semiotic theories aim to offer some description, explanation and understanding of the nature and usage of the artefacts whose main feature is to carry meaning of one sort or another, but also to study the active process involved in their semiosis or interpretation. Semiotics . . . [is] . . . concerned with ordinary objects in so far (and only in so far) as they participate in semiosis. (Morris 1938: 3) Semiotics is the study of signs. It thus investigates the structure and function of all events which involve signs: the processing of information in machines, the metabolism in organisms, the stimulus-response processes in plants and animals, the activities of perception and orientation in higher creatures, the interactions of primates, communication between humans, the dealings between social institutions, and the delicate processes of interpretation which take place in the comprehension of the complex sign structures in legal matters, in literature, music, and art. (Posner et al. 1997: 1)

Thus, all semiotic theories share a set of theses regarding signs: (a) they are artefacts of some sort; (b) they carry some meaningful content; and (c) they must be interpreted by some sort of inner capacity. ‘A sign is an object which stands for another to some mind’ (Peirce, MS 214, 1873). Semiotics is hence a sort of epistemic endeavour that constructs theories about or applies them to objects, facts or events that carry meaning for living beings. To put it simply, semiotics explores and defines the conditions of signs being signs. ‘It is general semeiotic, treating not merely of truth, but also of the general conditions of signs being signs’ (Peirce 1958: CP 1.444). Non-semioticians may criticize such a definition as being too general. But one must be very careful, for this definition is not about the meaning of a lexical item. It is just an introductory definition for a research programme. Semiotics is a field which studies ‘signs’ in the same general manner as psychology is said to study ‘cognition’ and ‘emotions’, or as neuroscience studies the ‘brain’, for instance. The important point of this definition is that semiotic phenomena are not taken as objects of study because of their physical properties or features, but because of the intriguing property by which they have meaning.

The semiotic paradigms Naturally, this general definition of semiotics has been refined and deepened in various ways. Both the nature of semiotic artefacts and the process of semiosis have been the object of many theories which have become the main paradigms of semiotics. In our research in view of linking semiotics with computation, we may distinguish three main types of such paradigms: philosophical, linguistic and naturalist. These paradigms or types of semiotic theories are not inherently contradictory to one another. They form different points of view for explaining, describing and understanding semiotic artefacts and the process of semiosis. And as they are well known, we shall briefly present them here.

 The Complexity of Semiotics 3 The first and probably the best-known set of semiotic theories are the philosophical ones. They come in three different shades and tonalities: mentalist, pragmatic and hermeneutic theories. Each one has its own concepts, arguments, discourses and methodology. And they are not theoretically independent from one another. They have strong interwoven relations. The first type of philosophical semiotic theories is the mentalist ones, which are also the most famous. They are grounded in the long-standing tradition of Greek and Medieval philosophy (Abelard, Aquinas, Occam). Their formulation was coined by Augustine as ‘aliquid stat pro aliquo’ (Augustine) and reformulated by Poinsot, who insisted that this ‘stat pro’ or, in English, to ‘stand for’ was to be conceptually understood as to ‘represent’. This approach was integrated into most of the modern and contemporary semiotic philosophical trends (from Port-Royal to Kant, Frege and Fodor). This relation has become one of the prototypical features of semiotic artefacts: a symbol or a signal (natural sign) is something that stands for or represents something else. The explanation of the process by which something effectively stands for something else is mainly explored through a philosophy of mind where perception, conceptualization and language will be understood as the dominant components of the processes of semiosis. In other words, according to this mentalist paradigm, semiotic artefacts have the main relational feature of standing for something else. Meaning is the result of a semiotic process where a mind creates a representation that stands for something else. The second set of philosophical theories are the pragmatist ones. They find their original synthetic expression in Peirce. His theory sees a semiotic phenomenon not mainly as a state, but as a complex epistemic activity called ‘semiosis’. This activity is a process by which certain artefacts become carriers of meaning or, more simply, signs. It is a dynamic process that rests upon complex epistemic categorizing operations realized by human agents such as sensing, perceiving and conceptualization. Peirce also complexifies the stand for relation. It links together: (1) something that is the carrier of meaning (i.e. the representamen); (2) something that the sign conveys for somebody (i.e. the interpretant); and (3) that what it stands for (i.e. the object). And semiotic phenomena can come in many forms such as symbols, icons, signals or indices, which all inhabit a dynamic process, semiosis, itself carried out by interpreters. Variants and nuances of this Peircean pragmatist semiotic theory are numerous: in Frege, the stand for relation becomes ‘term’, ‘sense’ (Sinn) and ‘referent’ (Bedeutung); in Morris (1938), it is ‘signal/symbol’, ‘interpretant’ and ‘designatum’. And other variants may add epistemic modalities (Hintikka 1997), contexts and situations (Morris 1938; Barwise and Perry 1983), culture (Luhmann 1995), embodiment (Brier 2008) or cognition (Brandt 2013). A well-known variant of this triadic model has been the behaviourist formulation by Ogden and Richard (1923: 140). Here, the triadicity of the sign is translated in terms where a ‘stimulus’ is the representamen, the ‘reaction’ is the reference or object, while the ‘engram’ is sometimes identified with the ‘thought’. According to this perspective, the study of a semiotic artefact must reveal what, in it, makes it a stimulus, what it stands for or represents and what is the engram it produces in the interpreter.

4

Computational Semiotics

Compared to the mentalist paradigm, this pragmatist view complexifies the stand for relation with respect to three components: representamen, interpretant and object. Meaning emerges in a process of semiosis where (a) some agent, (b) creates interpretation, on (c) an artefact so that it signifies something else. An interesting point here is that these ‘artefacts’ may themselves be other signs. The third philosophical semiotic paradigm is a hermeneutic one. It has its source in the theory of language and textual interpretation initiated by Herder and Hermann and developed by Schleiermacher, Humboldt and Dilthey. And more recently, Gadamer, Habermas, Adorno and Ricoeur enhanced it and applied it to different types of semiotic artefacts. Although not always expressed in classical semiotic terms, these theories emphasize that meaning is given to semiotic artefacts through the epistemic process of understanding the world. In revealing the meaning of semiotic artefacts, hermeneutic semiosis may call upon not only the inner experience of the interpreter but also the culture in which he or she lives. Hence, interpretation is a dynamic process that is circular or, in better terms, spirally enhanced. Some researchers such as von Uexküll explicitly integrated this hermeneutic point of view into semiotics. Semiotic artefacts are mediators in building our own world (Umwelt). And for some others (Jakobson, Martinet, Vygotsky, Halliday, Eco, Apel, etc.), this mediation also had to take into account the important role of language or, more generally, of various means of communication such as dialogue (Ricoeur) or intentionality (Austin, Searle, Grice). For their part, Lotman, Luhmann, Brier, Brandt and many others complexified these relations pertaining to meaning by including contexts, situations, and social and cultural environments. Hence, in this hermeneutic paradigm, both the semiotic artefacts and the process of semiosis have become more complex. They comprise an interpreter whose process of interpretation creates different states or an inner world that stands for the word itself. And various complex intervening factors such as the context, situation, culture or intentions may come into play. Some contemporary semiotic theories associate hermeneutic positions through modern cognitive theories. They may call upon such things as phenomenological experience, perception, memory, conceptualization, categorization, reasoning and information processing. Here, meaning is intrinsically rooted in the human cognitive tendency to explain and understand. For Brandt, a theory of meaning so understood is the project of cognitive semiotics. Thom and Petitot have proposed that semiotics should focus on the phenomenological experience of the world. Finally, some other semioticians will add a communicational dimension to this cognitive framework. In so doing, they build a bridge towards the Morrisian semiotic theory. Signs are not built just for individual cognitive purposes; they also are a means of sharing meaning. Signs participate in the interaction and social negotiation we have with others in our relation to the world (Habermas 1984; Apel 2000). Thus, meaning is the result of an interpretation that emerges through cognition and communication. Communication is a form of interactive meaning creation, for it relies on coordination and cooperation, among other things. The second very important paradigm comes from structuralist and formal linguistic theories. In these theories, attention is directed towards the structure of semiotic

 The Complexity of Semiotics 5 artefacts and the effect it has on the process of semiosis itself. Three main points of view regarding this structure are proposed. One sees it as being similar to a linguistic structure, the second one sees it as a logical form, while the third views it more as a dynamic mathematical form. The first one has its origin in Saussure but developed within the structuralist movement often called the ‘semiological school’ – its leaders include Hjelmslev, Trubetzkoy, Jakobson, Greimas, Lévi-Strauss, Barthes, Rastier, Todorov and Fontanille. But there are important variants in terms of understanding the nature of what this semiotic structure is. The dominant understanding of the structuralist thesis is purely Saussurean. Semiotic artefacts are rarely atomic or isolated entities. They are most often embedded within complex structures of meaning. In other words, they live and operate within some context. The prototypical example of such meaningful systems will be a language (la langue). For Saussure, a language is essentially a set of relational structures. It is a ‘system of interdependent terms in which the value of each term results solely from the simultaneous presence of the others’ (Saussure 1959: 114). It is governed by a multiplicity of structural relationships (syntagmatic, associative, paradigmatic, oppositional, differential, etc.) and can even be seen as a formal system: ‘language, in a manner of speaking, is a type of algebra consisting solely of complex terms’ (Saussure 1959: 122). According to Saussurean structuralism, meaning is a relation between a ‘signifier’ (signifiant) and a ‘signified’ (signifié). And this relation is created in the act of speaking (la parole). Saussure extended the structural thesis to all types of sign structures, and these form the research object of semiology. He defined the latter as ‘a science that studies the life of signs within society’ (Saussure 1959: 16, emphasis removed). This new science, he said, would show us ‘what constitutes signs, what laws govern them’ (ibid.). Hjelmslev (1968) added some complexities to this concept of linguistic structure. Both the signifier and the signified form a structure: a level of expression that is functionally related to a level of meaningful content. For this structuralist school, meaning is deeply embedded within the various structures of semiotic artefacts, for the structure of the signified is functionally dependent upon the structure of the signifiers. An important variant of this structuralist view of language will be offered by Harris (1982) and Chomsky (1956) and their followers. Here, the structure is described by some in algebraic or ‘generative’ terms. For Harrisians, meaning will be distributed in the context of the linguistic structure, while for Chomskyans, it lies in the deep structures underlying the surface and will even take a logical form. Chomsky will refuse to extend his generative view to all other meaningful artefacts. Semiological theories, however, will not retain this view except maybe for the Harrisian contextual distributionality thesis. The second paradigm for understanding the organizational structure of signs, though it is not always seen as a semiotic theory, takes its source in Frege and his followers: Russell, Church, Carnap, Montague, Shaumyan and many others. It also takes language as a prototype for other semiotic artefacts. But here, it is not a matter of natural language but of formal language. Such language is defined as a set of atomic or discrete symbols in which the structure is given by a set of rules called ‘formation’ or ‘transformation rules’. These rules define the syntax of the language and underlie

6

Computational Semiotics

the formation and transformation of sentences or formula. The prototypical examples of this paradigm will often be called logical symbolic languages, such as found for instance in the propositional, predicate, modal, categorial or logical languages. These languages can be seen as complex structural variants for describing other types of semiotic systems. In a sub-paradigm of this logicist view, meaning is not embedded in the logical structure itself. It is given to these structures by a functional association to the syntactic rules and the lexicon of the language. Carnap and Tarski, among others, have offered a formal definition of the relation between meaning and the structure of a language. It will be seen as a function of mapping from its syntactic structure to its semantic structure. And this semantic structure will be mainly defined in referential terms. Carnap and Morris will generalize this formal view of the formation of a logical symbolic system to all types of other signs. In this view, semiotics will become a general formal theory of symbolic systems. The third structuralist trend deepens the concept of structure itself by exploring its mathematical properties. It was mainly developed by Thom (1988), Petitot (1985a, 2009, 2018), Wildgen (1982), Wildgen and Brandt (2010), Piotrowski (2017), Piotrowski and Visetti (2014). According to this paradigm, structure is not mainly understood as being differential or combinatorial; it is a mereological and gestaltic structure and also a dynamical system. In this perspective, a structure is a dynamic process, one with stable and unstable phases in which there are fusions, ruptures and catastrophes, if not chaotic attractors. And the meaning of artefacts is the result of the complex process of integrating multiple interactive (internal or external) formants of a morphodynamic semiosis. In short, semiotic theories influenced by structuralism – be it linguistic, logicist or dynamical – claim that an adequate and rich theory of semiotic artefacts and semiosis must focus on the inner and outer structures of these artefacts. And it is within these complex structures that meaning is embedded or to which it is associated, with semiosis being the process by which meaningful structures are interpreted. Thus, according to these three structuralist semiotic paradigms, there is always (a) an agent that is an interpreter; and (b) a semiosis that is an interpretive process, though this interpretative process is applied to highly complex structured artefacts (sentences, texts, formulas, proofs, etc.). Context is recognized by Saussure as participating in the semiosis, but it is not given the focus. According to more logicist approaches, the context of the interpretation may include the world if not some possible worlds. For dynamicists, the context is an external constituent of the overall structures of a dynamic semiosis. The third type of paradigm is a naturalist one. This naturalism is to be understood at two epistemic levels: that of the object under study and that of the methodology. The first understanding of naturalism focuses on semiotic artefacts or objects: these are entities in nature which humans, animals or some kinds of living beings construct or manipulate and which are seen as carriers of meaning. Such objects can be symbols (language, icons, etc.) or signals. The second understanding sees semiotics as following or as being inspired, at least in part, by the methodologies of the natural sciences: observation, experimentation, formalization and so on. Both approaches will be highly

 The Complexity of Semiotics 7 entangled. Despite strong variances from a theoretical standpoint, they have opened new and important semiotic territories. The first of such variants may be found in the cognitive and informational view of semiotics. Its origin may be traced back to Peirce for whom signs are, before anything else, means to acquire knowledge: ‘a sign is something by knowing which we know something more’ (Peirce 1958: 332). But it was also heavily marked by the functionalist theories of language of Jakobson, Martinet and Halliday, by the pragmatic and intentional philosophy of language of Austin, Searle and Grice, by the social psychology of Vygotsky and by the pragmatic philosophy of communicative action of Habermas and Apel. But in semiotics, it seems to have been Eco (1965) who has been among the first to open the door to the cognitive dimension of semiotics. Through his reflection on symbols and icons, he understood the stand for relation to be grounded in an important cognitive activity: perception. This supposes that semiotics has to take account of some cognitive operations in making a semiosis meaningful. Sebeok1 was more radical. By exploring various types of semiotic artefacts, he insisted on seeing semiosis as linking meaning with communication among cognitive agents. Deely and his colleagues (1986) were among the first semioticians to explicitly ask for a cognitive revolution in semiotics. Faithful to their medieval theoretical background, semiosis was to be seen as an act of ‘cognitio’. And this entailed the need for a semiotic revolution: one that would ground semiotics in our human cognitive experience. ‘The semiotic revolution concerns, first of all, our understanding of human experience itself, and therewith all of human knowledge and belief. What semiotics at this point has shown is that the whole of human experience – the whole of it – is mediated by signs’ (Deely et al. 1986: xii).2 Serson saw that Peirce’s semiotic model ‘goes immeasurably further (and deeper) than the semiotic currently employed implicitly or explicitly by the semiotic scientists’ (Serson 1994). The naturalist twist really came with researchers such as Bouissac (2010, 2012), Rieger (1999a, b), Varela, Petitot et al. (1999), Sonesson (2012), Zlatev (2012), Brandt (2004, 2013), Brier (2008) and many others. For them, semiosis is a natural process. And a cognitive semiotics focuses mainly on the process of semiosis itself: that is how humans deal with symbols, icons and signals. But it also retains the cognitively processed semiotic artefacts such as languages but also gestures, music, film, texts, dance, painting, theatre, electronic media, clown performances, child development, sexuality. In this view, humans are meaning-making agents, meaning which they communicate. As Brandt summarizes well: ‘meaning is based on two distinct capacities of the human mind, cognition and communication’ (Brandt 2019: 2). From a methodological point of view, this cognitive semiosis will be studied through various types of theoretical and formal models, highly inspired by the cognitive sciences and by sciences such as psychology, ethnology, anthropology, cultural studies and to a quite lesser degree linguistics and logic. But this ‘naturalism’ will not be a reductive one. ‘Natural’ does not mean mechanical or merely informational. As Brier says, in semiotics, ‘information is not enough’. Interpretation is a highly complex natural semiosis. So, an adequate cognitive semiotic theory must also take account of

8

Computational Semiotics

the situation, of bodily embedment, ecology, culture and, most of all, of language and communication. Two other sub-disciplines will be distinguished, but they have increasingly been seen as overlapping. Zoosemiotics (Sebeok 1968; Martinelli 2010) will be interested in how animals deal with signals. Biosemiotics, for its part, will focus on signal processing in dynamic biological structures pertaining to life, survival, emotions and even social communication. According to the naturalist paradigm, the process of semiosis will include (a) an interpreter; and (b) a complex set of interpretive cognitive operations such as sensing, perceiving, categorizing, reasoning, memorizing and feeling. Many determinants and factors will intervene: the situation, context, culture, the body, evolution, communication, etc. And it will be applied to many types of artefacts – from symbols (language, music, art, etc.) to signals (behaviours, stimuli, feedback, etc.). Our brief presentation of these three main paradigms did not delve into the details of each conceptual framework – we rather took a bird’s-eye view. This allows us to distinguish two main components of semiotic phenomena: ( 1) Artefacts of which one of the main features is to be a carrier of meaning; and (2) Processes of semiosis by which living beings identify in these semiotic artefacts the meaning they carry. Each of the paradigms presents some specific conceptual framework containing concepts and propositions for describing, explaining and understanding the semiotic artefacts and the processes of semiosis. The philosophical paradigms focus mainly on the process of semiosis. The mentalist view claims that semiosis builds a mental representation through which semiotic artefacts are interpreted. The pragmatist paradigm insists on distinguishing in a semiotic artefact the representation itself from various means (interpreter, producer, situation, etc.) that determine the representation. Finally, the hermeneutic paradigm insists on subjective and external factors that determine the representation, such as culture and environment. The linguistic structuralist paradigm focuses on the inner and outer combinational or dynamic structures of semiotic artefacts and on the embedment of meaning in these structures. Natural language is taken as the prototype of semiotic artefacts. The logicist structuralists are also sensitive to the structure of semiotic artefacts. But, here, structure is seen through their chosen prototype: logical language. Meaning is seen as being somewhat external to the language – it is associated (by an external decision, convention, etc.). Finally, in the naturalist paradigms, living beings are seen as manipulating semiotic artefacts which are symbols or signals. In humans and in some animals, these manipulations will be understood mainly in terms of cognitive information processing. And it takes the form of sensing, perceiving, categorizing, memorizing, feeling and ultimately communicating with other members of a social and cultural community. Through the presentation of these three paradigms, we see that semiotics, like any other science, builds theories about its own specific type of reality: (1) there are

 The Complexity of Semiotics 9 artefacts in the world that have a particular property – they are carriers of something that is called meaning; and (2) there is a specific type of cognitive process that identifies such meaning in these artefacts. But what then appears from this bird’s-eye view upon the paradigms is that an adequate semiotic theory cannot be limited to just one viewpoint. A serious, if not scientific semiotic theory cannot describe, explain or understand semiotic phenomena by taking just a single point of view upon a semiotic phenomenon. Each of the paradigms’ points of view are not necessarily false or true in themselves – just incomplete in regard to the nature of semiotic artefacts and of the processes of semiosis. To reformulate this in general Peircean terms: the aim of semiotics is to find ‘the general conditions of signs being signs’.

Semiotics and complexity If one point of view is not enough for explaining, describing and understanding semiotic phenomena, then how can an all-inclusive integrated semiotics be built? Even if such all-inclusive theory were available, we would eventually discover that the theory would have to deal with a myriad of properties and structures for each semiotic artefact and semiotic process. Indeed, we may imagine that it could be easy to somehow identify a few properties for simple semiotic artefacts such as an American dollar, a candle, a wedding ring or a handshake. But the number of properties grows exponentially for less simple semiotic artefacts such as an Hindu funeral ceremony, clowns in a circus, animal drawings in Neolithic caverns, the Beethoven symphonies, the Easter Island Moai statues, the Stonehenge circle or the Notre Dame Cathedral. There have been many critiques of these various semiotic paradigms. A typical and general one is the critique by Harman: after showing that semiotics uses the word means in a variety of ways, he concludes that ‘there is no reason to think that these theories must contain common principles’ (Harman 1977: 23). Many of these critiques do not always rest upon strong epistemological principles. They are often made by illustrating the semiotic research methods and objects using simple examples of the ‘x means y’ form, such as a smile means this, smoke means that, and so on. Such critiques may indirectly claim that there cannot exist a theory, and even less a scientific theory, that can offer an adequate explanation and understanding of semiotic artefacts or of semiosis precisely because of the multiplicity and complexity of the properties, structures and processes these involve. In their view, a semiotic theory can only offer subjective, heuristic descriptions – nothing more than a wonderful and brilliant epistemic spectacle or representation. The underlying postulate of such criticisms rests upon an implicit epistemological thesis that the objects being characterized by a multiplicity and variety of properties and structures renders their scientific investigation practically impossible. This is a radical claim against semiotics. And it justifies interpretative semiotics. Let’s reformulate the core of this critique. The underlying implicit postulate seems to be, at least in part, the following: because a semiotic phenomenon is complex and contains a multiplicity of features, it cannot be studied scientifically. If this thesis is valid, it should also be applied to most scientific research objects and methods. Would it

10

Computational Semiotics

not also challenge the possibility of scientific enquiry on physical, biological or social artefacts and processes? Aren’t such research objects also plagued with a variety of complex properties, structures and processes and aren’t they also entrenched in some point of view or perspective that is, at least partly, highly subjective? For our part, we prefer to argue that it is precisely because semiotics must be seen as dealing with a complex type of properties and processes that it is interesting and that it may be subject to scientific enquiry. For instance, if semiotic phenomena were very rare, unique and irregular, it would probably be impossible to find out what they are and how they may be explained by some principles or rules. But because they are numerous, it becomes possible to discover in them some type of regularity or another. Take, for example, the semiotic artefact that is a ‘text’.3 Linguists (Weinrich 1976; Halliday and Webster 2014), philosophers (Ricoeur 1970), literary theorists (Genette 1979; Rastier 2009) and editors (Sahle 2013) have distinguished among this class of semiotic artefacts a galaxy of texts: material texts, narrative texts, intentional texts, etc. In the world of computers, the various types of text components and manipulation have to be well distinguished because they all carry many different shades of meaning and interpretations. In Figure 1.1, we illustrate a variety of texts, including those distinguished on the basis of their physical medium.

Figure 1.1  A variety of ‘texts’ in a computer environment, distinguished by their physical medium.

 The Complexity of Semiotics 11 So even if semiotic phenomena are complex, the question remains whether semiotic artefacts can be studied scientifically. To answer this question, we must clarify what it is to claim that a scientific inquiry can deal with objects or artefacts that present complexity in their properties, structure and processes. For our purpose here, we will recall that the notion of complexity4 in science has many meanings. Two of these meanings are relevant for our purpose. The first one is basic epistemic complexity; the second is epistemological complexity. The first meaning of complexity is easy to understand. It pertains to the mode of observation one has about some reality. Here, complexity is often synonymous with complication. It is an epistemic position: something may appear complicated because one may not proceed with the correct means for observing its features. In such a case, the ensuing understanding will be limited. And things look complicated. For instance, for some person, a thermostat may be a complicated gadget because he or she cannot explain how it works. For the same reason, a computer scientist may find that connecting his or her cellular telephone to a new Wi-Fi router may be a complicated procedure. The same situation may occur when explaining what is going on in a funeral ceremony. For a celebrant who masters all of its rituals, the ceremony is not complicated either in its organization or in its meaningful content. But it may be complicated for the family or for a guest who is taking part in such a ceremony for the first time and who does not understand what is going on or what the ceremony means. And in our daily personal and social life, we encounter many such semiotic artefacts. Consider such examples as a fine table setting, a birthday party, a veil worn by a Muslim woman, and so on. For some people, these artefacts may be simple, but for others, they may be complicated and their meaning is not always easy to discover or to understand. Such types of objects, events or processes may appear complex in the sense of complicated because there are many different features and properties to be observed and many explanations to be given, while they are not easily accessible, observable or mastered by the observer. As Ashby formulates in a slightly Shakespearian manner: organization is partly in the eye of the observer (Ashby 1962: 258–9). It is not because such semiotic artefacts are complicated that they are to be excluded from serious and legitimate semiotic enquiries. Some of them may have a simple organization, but in most cases they are not just complicated: they are complex artefacts. It may require much work to unwrap their structure and meaning. The second meaning of complexity is more sophisticated. It is the one that has become important in contemporary science. One main reason for this importance has been the slow obsolescence of the deterministic, positivist and logicist epistemological models of scientific practices. These models appeared inadequate for describing and explaining many types of phenomena that could not integrate easlily and clearly certain types of properties such as continuity, dynamicity, evolution, adaptiveness, emergence, multi-factoriality, parallelism or modality. Although these concepts may have an intuitive, vague and ambiguous meaning, they came to characterize the notion of complexity. They allowed us to study some natural phenomena by focusing on them in a holistic and dynamic manner – that is, to see them as a whole rather than as a collection of local, isolated and static components. In other words, complexity is a particular epistemic point of view by virtue of which properties or attributes are not

12

Computational Semiotics

strictly explained solely by their local, atomic and static components, but also by their global, holistic and dynamic structure. Still, this difference between the local and the global is not sufficient to characterize complexity. The deep problem is in explaining the inner relations between the local and global components through some integrative formal model. A nice synthetic definition of this complexity problem is the following: We call complexity the fact that higher levels of organization display features not predictable from the lower ones. (Arecchi 2002: 132) By complex [we mean that] . . .  knowledge stored in well established models is not sufficient to predict reliably the emergence, and must integrate the deductive chains with extra information which introduces a historical flavour into the scientific procedure. (Arecchi 2002: 113)

In other words, complexity is both an epistemic and an epistemological problem. It is epistemic because one must observe and describe that certain phenomena present a multiplicity of local, low-level properties, structures and processes from which a distinct set of global properties will emerge, supervene and become observable. It is also an epistemological problem because one must explain by means of formal models how such a dynamic phenomenon is realized. Take, for example, a typhoon. A localist and static explanation might emphasize the presence of molecules of air and of water, the amount of pressure and the local temperature. At the global level, one may stress the occurrence of a big, strong and directional movement of wind and low air pressure forming a vortex involving various forces (wind, pressure) over the ocean. Many formal and computational models have been created by climatology and by meteorological science in order to explain not just the constituents of a typhoon, but the various dimensions of the dynamics, formation, development and emergence of a typhoon. These models are applied to the interactions between components such as the temperature on the land and in the ocean, the changes in air pressure, the circular movement, the energy potential and so forth. Such formal models of typhoons aim at explaining their genesis, emergence, development, periodicity and so on. Semiotics encounters similar complex phenomena. Take the following example: the religious procession for Easter in Spain. Every year, on the week before Easter, most Spanish towns and villages have a highly organized religious procession which is very important for the community. A localist explanation will rest upon presenting the basic elements or components of such a social semiotic artefact. It could present, for instance, the local actors, objects and their local meaning (persons, costumes, procession cars, music, religious representations, chants, candles, etc.). A localist view that focuses only on the features of individual participants or artefacts will never approach the meaning of this ritual. Conversely, a holistic explanation would aim at identifying the interrelations between the multiplicity of the participants and components so as to discover integrated structures that traverse the whole of the procession itself and that embed the meaning it may carry. If such an explanation is to be offered, the complexity must be dealt with by offering a formal model, not just by adding new components.

 The Complexity of Semiotics 13 Complexity is one of the most important scientific challenges semiotics must deal with. As we can see, the real epistemological problem of complexity is not mainly one of increasing the number of components or of the features of a phenomenon, but of building formal models of the complexity itself. And this has been done in many scientific domains. Some will use simple formal models, while others will require more sophisticated ones. They call upon various mathematical structures such as classical linear, non-linear, homological, geometric, vector and matrix algebra, dynamic analytic systems, descriptive and inferential statistics, Bayesian probabilities and topology. And not all of these mathematical models are about ‘quantities’. Many are about purely geometrical, topological, generative, graphical, combinatorial and even chaotic structures. In this view, the classical logical or linguistic model is no longer seen as the prototype of formal models for complex phenomena. Thus, the natural language expression complexity is used for naming, in a synthetic but often opaque manner, formal models that, in the study of phenomena, take into account both the multiplicity of their components and their global structural forms, as well as eventual dynamic changes in their states or phases. The existence of these formal models, however, still does not exhaust the problem of complexity. That is, it is not because a sophisticated and refined formal equation is produced for modelling a particular dimension of a complex phenomenon that it can be effectively calculated. Take the following example. It is quite easy to calculate the equation: y= x If x is not too big, most mathematically educated persons will effectively be able to calculate this in an acceptable time lapse. But how about the equation: y = 13 x This equation is conceptually quite similar to the simple square root equation. But the effective calculus necessary for finding the value of y is quite a burden for a human, even one heavily educated in mathematics.5 What this example shows is that complexity does not only apply to the multiplicity of static or dynamic relations, but also to the calculability of the formal models used in studying them. If a simple formal model such as the second equation presented above was already a burden to calculate, one can imagine how sophisticated formal models for complex phenomena may be even more burdensome. In fact, there existed many formal models which were created to explain and to describe complex objects and processes but which could not be calculated because of a lack of effective calculating power or, to put it more simply, because of the lack of ‘manual computing power’. The power of computing machines changed this situation. They do not change the calculability of mathematical functions. They just allow faster effective calculation. Such computing power is quite recent in the development of scientific practices. It allows complex calculable formal models to become effectively calculable with the assistance of a computer. But by the same token, this introduced a lot of change in the practice of science. As sciences integrated such computational power into their

14

Computational Semiotics

enquiries, an increasing number of known sophisticated formal models that could deal efficiently with complex problems could thenceforth be effectively calculated. And this physical computing power was not without effect upon the methodology of science itself. These computing machines could then calculate complex functions containing a myriad of variables and factors. In other words, effectively computable complex formal models could be built. Hence, the term ‘complexity’ is applicable to phenomena that may contain a myriad of sets of complex relations which can, today, not only be modelled formally, but which can also be effectively calculated by computer. And today, there are a multitude of algorithms that can effectively compute the formal models that would not have been effectively computable some time ago. Such algorithms often bear rather metaphoric names: ‘experts’, ‘problem-solvers’, ‘natural language processors’, ‘inferential machines’, ‘artificial intelligence’, ‘autonomous intelligent agents’, ‘neural nets’, ‘deep learning’, ‘distributed memories’ and even sometimes ‘robots’ and ‘androids’. But what is really going on in these computers is effective computing power implementing a variety of algorithms that translate formal computable mathematical functions which map onto complex problems. So, if computational semiotics is to exist and has to deal with complex semiotic phenomena, it must justify how it can enhance its enquiry of such phenomena by calling upon computers which, by nature, themselves call upon algorithms and which, in turn, rest upon calculable functional relations. Computational semiotics cannot be simply defined as computer-assisted classical semiotics – this just hides the real innovation underlying its research programmes: the integration of computation into semiotic theories.

Semiotic complexity and computational semiotics The research object of semiotics is typically presented as being about artefacts that carry meaning and about the processes that must be activated to build or access this meaning. Such a definition is very general. The real problem of semiotics arises when one opens up this definition: it turns into a Pandora’s Box. As we have seen, there are different points of view from which to approach semiotic phenomena – there appears to be a variety of them. But we believe that, in their core theses, they do not contradict one another. Rather, they may be complementary. But, in accepting this complementarity of points of view, the object of semiotics suddenly becomes highly complex. As long as the complexity problem is pushed under the carpet by understanding it as nothing but a small epistemic complication in the observation process, semiotics will dwell in its own paradigm. Each semiotic project will be satisfied with the set of entities and operations that the framework offers. And in most cases, the use of a computer or of a computational approach or point of view will not appear to be heuristic. According to such a perspective, a computational semiotics would be useless and would be no more than the resurrection of an old positivist dream.

 The Complexity of Semiotics 15 But if one accepts or at least recognizes the complexity of semiotic artefacts and processes, then one has to find the means to deal with what underlies such complexity: the multiplicity of types of artefacts, the varieties of their structures, the intricacies of their interpretation, the divergences among meaning-makers and interpreters, the host of interpretations, the myriad of situations, the wealth of social and cultural contexts, the sophistication of communication means and so forth. And in many sciences, as we have shown, when such complexity emerges and is accepted as part of the problem, then formal and computational models become an essential partner for scientific enquiry. This book is an essay on the types of complexities borne by semiotic artefacts. It endeavours to respect the multiplicity of points of view that the various paradigms of semiotics have unveiled. We hope to understand how a semiotic enquiry can deal with the barrier of meaning and of how semiosis manages to pass through this barrier so as to create and identify the meaning. It will not claim that the classical procedures of semiotic enquiry must be replaced by computation or that semioticians should be replaced by computers. What it will do is rather explore this complexity against the horizon of computation.

16

2

Semiotics in computing

The idea of associating computation with semiotics is not new. The roots of a possible encounter between these two epistemic paradigms go far back in history. Many early Indian, Chinese and Greek philosophers discussed logical demonstration and reasoning in terms of forms, irrespective of meaning. For instance, Aristotle, in his Prior Analytics, often used letter symbols as variables or placeholders for representing certain logical forms. The meaning of the symbols was not formally defined, but was given through illustrative cases or examples. Still, this made it possible to represent certain types of logical reasoning solely as the result of the manipulation of such symbols. Thus, logical reasoning was seen as ‘symbol manipulation’. Naturally, such ‘manipulation’ had to be either mentally represented or done by hand. In more contemporary terms, this manipulation would be seen as a calculating operation applied to logical forms represented by symbols. But more thorough research on physical symbol manipulation may be found with the logicians of the Middle Ages.1 During the thirteenth century, John de Salisbury (1115–1180), in his writings, records that one of his students, William de Soissons, had built an engine he called a ‘machina’, which was capable of proving logical argument. It was also said to construct conclusions from logical principles by simply manipulating symbols to show the validity of a process of reasoning. A century later, Ramon Llull (1232/1315) proposed an artefact made of interlocking paper wheels Figure 2.1  Llull’s paper wheel machine containing discrete alphabetical symbols (Uckelman 2010). representing concepts (Figure 2.1). By manipulating the wheels, the artefact could be used to demonstrate that certain symbolic configurations were true or false. Historically, it was interpreted as being a ‘machine’. According to Kneale and Kneale (1988), this would be the first instance of a machine that could compute. The interest of this ‘machine’ rests upon two important properties that will reappear in later theories of computation. First, an effective computation could be carried out on

18

Computational Semiotics

a physical technology (a wheel made of paper), by manipulating concretely the discrete symbols written on it. This manipulation allowed to produce new configurations of symbols which could then be seen as ‘evidence’ of a truth or falsity. Second, the meaning of the symbols was not a property taken into account by the ‘machine’ itself. However, these symbols would be interpreted by humans. The interpretation was therefore external to the machine. Later, Hobbes ([1651] 2010) proposed a famous formula identifying thinking with computation: ‘cogitare est computare’. Computation was seen here as a sort of reckoning that rested upon the manipulation of some general names that were used for signifying thoughts: ‘reasoning is nothing but reckoning (that is, adding and subtracting) of the consequences of general names agreed upon, for the marking and signifying of our thoughts’ (Hobbes [1651] 2010). Leibniz (1588–1679) reformulated this idea in the more synthetic terms of ratiocinator calculus. It specified a specific property of such calculation: it must be a combination of symbols – that is, a set of controlled operations of composition on symbols. This calculation was the core of a lingua caracteristica – an idea that Condillac (1714–1780) will take up in saying that if this language is perfect, it is an algebra: ‘algebra is a well-made language and it is the only one’ (Condillac 1798: 6, translation from French). Boole (1847) will add the most important fibre to the concept of calculating with symbols: the truth of the laws of manipulating symbols must not depend on the meaning of these symbols. In his words: ‘their validity does not depend on the significance of the symbols which they involve, but only on the truth of the laws of their combination’ (Boole 1847: 7). According to Kneale (1948), this would be Boole’s important contribution: the idea that there exists an algebra of entities that need not be arithmetic. In other words, there could exist means of pure manipulation of symbols that are not necessarily about numbers: ‘there could be an algebra of entities which were not numbers in any ordinary sense’ and the ‘numeric’ laws governing these entities need not be arithmetic (Kneale 1948: 160). Thus, in this intellectual tradition, there is a close link between reasoning, calculating, combining and computing, which are operations for manipulating symbols or signs, as such. A valid calculation is a kind of combination of signs that can be performed regardless of the meaning of the signs. Such a thesis is theoretically important. Indeed, it is not enough to have computation in the sense of Hobbes or Boole – that is, computation is not merely a configuration and reconfiguration of symbols. Such configurations require a specific type of governing rule by which the symbols are combined without taking into consideration their meaning. It will take refined mathematical and logical research to define more technically the notion of the rule-governed combination of symbols. It will mature through Turing and Church, Post, Gandy, von Neumann,2 will become an increasingly formal mathematical notion and is to be distinguished from the physical computing electronic machine called a ‘computer’. Today, this thesis is expressed in linguistic and semiotic terms: a calculation or computation is a strict ruled-governed operation that is applied to sets of signs or symbols and produces other sets of signs or symbols regardless of their meaning. As per this formulation, the notion of ‘computation’ is highly entangled in a specific but limited semiotic thesis: there exists a kind of semiotic artefact (symbols) said to

 Semiotics in Computing 19 be computed independently of their meaning. According to this interpretation of computation, there is a theoretical gap between computational theories and classical semiotics: in semiotics, meaning cannot be eliminated from the explanation of semiotic artefacts. This is an important theoretical thesis. It implicitly suggests that building a bridge between computational theory and semiotics will not be an easy task. It raises the core question of this essay: How can there exist a link between a computational theory where, by definition, the rules governing the manipulation of symbols are applied regardless of meaning, and a semiotic theory where by definition symbols are carriers of meaning? Or to put it more straightforwardly: Is a computational semiotics possible? Three interwoven scientific contemporary paradigms implicitly or explicitly explore these questions about the relation between computation and semiotics. The first one comes from autonomous signal systems, cybernetics and information processing in computer engineering. The second one comes from artificial intelligence and the third from intelligent computer systems engineering. Some will even call this research programme ‘computational semiotics’. Indeed, computation can be seen as a semiotic endeavour because it is a process that uses semiotic artefacts that are signals or symbols. But we will see that the bridge extends in one specific direction. Implicitly or explicitly, the direction taken is one by which semiotics enters the computational world.

Autonomous signal systems, information and semiotics The first exploration of a semiotic perspective in the world of computation came from engineers working in the paradigm of autonomous dynamical and controlled systems. They did not explicitly see the relation to semiotics, but their work paved the first path to build a bridge between computation and semiotics. In this scientific field, the classical prototype of such a control system is Watt’s steam engine governor (Figure 2.2). This technology is a kind of physical artefact which autonomously controls the speed of a steam engine. It is made of a main spindle (C) to which two balls (A and B) are attached, each being connected to a set of two rods. The balls spin around with the spindle as it is rotated, and a sleeve around the spindle is attached to a lever (N, P, Q) controlling the steam valve (S). As the speed increases, the kinetic energy of the balls increase and the balls centrifugally spin outwards while their Figure 2.2  Watt’s fly-ball governor link rods pull up the sleeve connected to (Wikimedia Commons). Credit to Id the lever that is attached to an operating AYK92D on Alamy​.co​m.

20

Computational Semiotics

rod, which gradually closes the engine’s valve as the sleeve is pulled upwards so as to decrease the input of steam and slow down the engine. Thus, the intensity of the output is said to control the intensity of the engine’s input. As such, this description has nothing to do with semiotics. But there is a different description which would lay the first bricks towards semiotics. Indeed, it may be described in slightly more abstract and general terms, for instance: a governor is a physical machine where some inputs coming from some external source (steam engine) produces (by some internal physical structure) an output that in turn produces some modification in the external or original source. This modification in the original source produces a modified output that becomes a new input for the machine. This goes on until the input and outputs are stabilized. This description has important epistemic effects. A first one is that the mechanical description is translated into an abstract and more general description: it states that the governor is a relation between inputs and outputs. As an epistemic generalization, this translation allows many instances of similar processes to be brought under a common conceptual framework independent of the physicality of the artefacts and the nature of the processes themselves. For instance, one could apply this abstract description to an air or water pressure governor. Second, because it is expressed in a non-formal language, the observation of the dynamic stabilization operation can be conceptually understood as a special form of dynamic functional dependency relation between an input and output. The description can even be expressed metaphorically as a loop optimization process relating inputs and outputs. In other words, the conceptual framework allows speaking in a generalized manner about a complex dynamical process without necessarily explaining all the details of the internal operations, the types of dependencies and the nature of the inputs and outputs and, all of this, independently of their physical forms. Ultimately, this translation will allow even more abstract translations. For instance, it will be described as a functional equilibrium dependency relation, which can be expressed by means of a mathematical equation, such as the one Gibbs and Duhem proposed for one particular autonomous control system: the thermodynamic system. They presented it by means of an algebraic differential equation of energy in thermodynamic systems called the ‘law of equilibrium’. (G  H   T S) This equation expresses that in a thermodynamic system the difference in energy is a function of a difference in heat and work in a standard state. The Watt governor is therefore just a special application of this law. And it can be generalized so as to be applied to many other similar systems. In fact, there are many types of systems that are ‘auto-adaptive’. They are capable of configuring and reconfiguring their internal structure so as to adapt themselves to their output and adjust their input or, in more global terms, to adapt to their ‘environment’. This idea of auto-adaptive autonomous systems, because it was so clearly conceptualized and formalized, made its way into the minds of many European and American scientists. They transferred this view of dynamical systems interacting with their environment to their own scientific fields. And many related ideas and

 Semiotics in Computing 21 concepts proliferated and criss-crossed over various scientific domains that suddenly saw conceptual similarities between their researches. For instance, biologists von Uexküll (1864–1944) and von Bertalanffy (1901–1972) applied it to living beings and modelled them as evolving dynamic systems interacting with their environments or situated world (Umwelt). Later on, neurophysiologists and cognitive psychologists W. S. McCulloch (1898–1969) and W. Pitts (1923–1969) proposed a formal calculus model of the neural activity of the brain. Von Neumann (1903–1953), for his part, reformulated this adaptive system as an instance of dynamical models (cellular automata) that could simulate the interaction of cells with their environment, their evolution and their reproduction. But one important applied engineer (Wiener, 1894–1964) offered another formal model of this interaction: he conceptually illustrated it through graphs where it took the form of a recursive loop retroacting upon the system itself (Figure 2.3).

Figure 2.3  Cybernetic loop.

But most importantly for our purpose here, this conceptual graph was translated into a formal mathematical function where the output values modified the input values.

This mathematical schema became the prototype of formal models for many more technical developments of what has become the cybernetic paradigm. Still, this model was not yet related to any type of semiotic framework. That was initiated separately and indirectly by Wiener himself, by mathematician Shannon, as well as by linguist Jakobson. For Wiener, cybernetics was understood as a meta-model that could be applied not only to control systems, but also to communication between humans, animals and machines. Thus, communication between beings was understood as an analogue of control systems such as the Watt governor. Both were seen to be in a similar dynamical relation. However, Wiener’s conceptual model, applied to communication, described it mainly by means of graphical and mathematical concepts – he did not call upon semiotic concepts.

22

Computational Semiotics

Shannon (1948) also built a model of communication but, probably unconsciously, introduced in his presentation three concepts that opened the door to semiotics. First, he gave a technical definition of communication between machines as ‘reproducing at one point either exactly or approximately a message selected at another point’ (Shannon and Weaver 1964: 31; emphasis added). By the same token, he expressed the notions of inputs and outputs of a communication process by means of the semiotic term ‘signal’ and added that they were carriers of a message: in other words, inputs and outputs of adaptive systems were now generalized as signals and their important feature was that they were carriers of a meaningful content called a ‘message’. This definition of ‘signal’ and ‘message’ in a communication process was clearly illustrated in the classical graph model of communication (Figure 2.4).

Figure 2.4  Shannon’s communication graph.

This semiotic generalization of the communication processes received a specific formal model where a set of probabilistic equations constructed an information theory. And later on, many physical variants were conceptually understood as complex systems called optimization, dynamic, control or information systems (Wiener [1948] 1961; Bennett 1993). Many other formal mathematical models were used for modelling the complexity of these processes, such as differential, integral, linear, non-linear and matrix algebras. Finally, these implicit semiotic conceptual frameworks and their formal models were applied to many industrial and military domains such as electronic communications, aircraft and manufacturing, and, later, they allowed the creation of some first autonomous artificial agents (robots). Ultimately, they found their way into the neurosciences. Despite the use of the semio-semantic terms of signal, message and information, the relation of cybernetics and communication theory to overt classical semiotic concepts was still not recognized. This happened through the linguist Roman Jakobson.3 He introduced, within the well-established Saussurean and Hjelmslevian structuralist models of linguistics, concepts belonging to the communication and information paradigms. Thus, in line with the work of Buhler, linguistic communication required, in both the production and interpretation of the communication, an emitter, a context, a message, a channel, a code and a receiver. And he added a variety of roles or functions for linguistic communication. In so doing, he changed the dominant structuralist paradigm and, in his later works, he carried the prevalent theory of language over onto a Peircean semiotic background. For instance, he saw Peirce’s concept of legisign, which

 Semiotics in Computing 23 is a conventional sign, as an equivalent to the concept of code in his communication model. But one important influence of Peirce on Jakobson seems to have concerned the role of linguistics for understanding the nature of all other types of signs. It followed that communication could not be restricted to linguistic signs. Through these variants developed by Shannon and by Jakobson, cybernetic theory became entangled with information theories and with linguistic theories. By the same token, cybernetics was put into relation with semiotics. Using a range of conceptual frameworks or of formal models, various types of semiotic systems could be conceptualized as communication/information-processing systems or as symbol/ signal manipulation systems, that is as systems which remained in close dynamic interaction with the environment over time. Still, this entanglement of cybernetics, information and communication theories with semiotics did not really integrate the framework of these cybernetic and control systems as such. This came when control system theories entered the world of computation, as the mathematical models of these control systems and of information theory became increasingly sophisticated in view of calculating complex equations and matrices with the assistance of computers. By the same token, the question arose as to what exactly it is that computers do in control systems, mainly if they are said to be intelligent. For artificial intelligence, computers process symbols, and for autonomous intelligent agents, they process signals. It raises the question: What makes a signal controlling processing system intelligent? This question opened up a more definitive but unclear relation between computation and semiotics.

Artificial intelligence and semiotics The relation of semiotics with autonomous systems and cybernetics was implicit. The explicit encounter of semiotics with the realm of computers actually came from the early theoretical foundations of artificial intelligence. Many semiotic concepts were discreetly called into its foundations (Meunier 1989). Around the 1960s, some computer scientists strode away from the cybernetic view of control systems. In their study of the historical development of artificial intelligence, Newell and Simon (1972) argued that a computational model of intelligence must be distinguished from cybernetics, systems theory, pattern recognition, numerical computing, programming theory, electrical engineering and even from formal logic. The AI school of computing found these paradigms inadequate for building artificial intelligence systems. They needed more than just sophisticated but pure mathematical computation. Some worldly knowledge had to be embedded within them. As Feigenbaum later stated, these systems needed ‘knowledge power’ to not just react to their environment, but mainly to reason and solve problems in an intelligent manner. In this perspective, intelligence would itself be defined simply as an information-processing system of symbols: With a model of an information-processing system, it becomes meaningful to try to represent in some detail a particular man at work on a particular task. Such a

24

Computational Semiotics representation is no metaphor, but a precise symbolic model on the basis of which pertinent specific aspects of the man’s problem solving behavior can be calculated. (Newell and Simon 1972: 5)

Newell often reasserted this thesis: The idea is that there is a class of systems which manipulate symbols and the definition of these systems which manipulate symbols is what is behind the programs in AI. (Newell 1986: 33)

And an AI project is not primarily aimed at processing numbers, but physical symbols. A physical symbol system consists of a set of entities, called symbols, which are physical patterns that can occur as components of another type of entity called an expression (or symbol structure). (Newell and Simon 1976: 116)

Thus, computers manipulate symbols that are said to be physical. Still, it is not this physicality that is the most important feature of symbols, but rather their meaning, that is, ‘what they stand for’: The most fundamental concept for a symbol . . . is that which gives symbols their symbolic character, i.e., which lets them stand for some entity. (Newell 1980: 156)

And what they stand for is their representational function: Representation is simply another term to refer to a structure that designates. (Newell 1980: 176)

This manipulation will become a necessary condition for any intelligent behaviour. There is no intelligent behaviour without this symbol manipulation: At the root of intelligence are symbols, with their denotative power and their susceptibility to manipulation. And symbols can be manufactured of almost anything that can be arranged and patterned and combined. Intelligence is mind implemented by any patternable kind of matter. (Simon 1980: 35)

And later on, Pylyshyn reminded that computation is not a physical process per se – that is the function of computers. Computation is mainly a ‘rule-governed transformation of formal expressions viewed as interpreted symbolic codes’ (Pylyshyn 1984: 50). As we see, even though Newell’s vocabulary may come from classical symbolic logic, the conceptual framework underlying his words is semiotics: a symbol is something that is instantiated in a token physical vehicle. But what makes a symbol a symbol as such is that it must ‘stand for something else’, ‘have a meaning’, ‘refer to something’ or ‘represent something’. Such a theory of symbols relies on an implicit semiotic theory in

 Semiotics in Computing 25 the classical sense: ‘aliquid stat pro aliquo’ (St. Augustine) or ‘they stand for something else’ (Peirce). Even for Newell, this thesis stands out clearly. Hence, AI projects seem so strongly attached to computer technology that we tend to forget that its real originality lies in the complex semiotic system that it mobilizes. AI is a theory on how to manipulate a special type of semiotic artefacts. It studies the functioning of a type of sign called ‘symbol’ in a computable system in such a way that it can be is said to be intelligent. Still, the AI hypothesis has a deep semiotic problem indeed. If symbols are to have a denotative meaning or are to be interpreted, the theory must explain how these symbols receive this meaning or interpretation. Unfortunately, the answer is not evident in this theory. In fact, its position is faithful to the mathematical model that underlies computational theory. The manipulation of symbols is strictly without meaning. As Haugeland put it: ‘a computer is an interpreted automatic formal system – that is to say, a symbol-manipulating machine’ (Haugeland 1986: 106; emphasis added). Here, the past tense of the verb to interpret is important. AI systems manipulate symbols that have already been interpreted. But such a formulation does not say how this interpretation is made or where it comes from. In other words, a computation theory is a strictly syntactic theory of symbol manipulation. It is without semantics. The limits of the syntactic view of computation became a central issue in computer engineering.4 An important solution that was devised was of a Tarskian logicalsemantic type. In this solution, a referential type of meaning is externally given to each type of symbol included in a formal object language by associating them to another specific set of symbols – metalinguistic symbols. However, if these metalanguage symbols are to ground the interpretation of the object language’s symbols, they need to be understood themselves. Interpretation cannot arise from a mere association of non-understandable symbols. If the interpreter knows the meaning of the interpreting symbols, then he or she knows the meaning of the symbols of the object language. If this is the solution, then by this formal technique, the meaning of the symbols in a computer program are hence given by some external intervention upon the original semiotic artefact such as by convention, declaration or some type or another of causal interaction with the world (captors, sensors, etc.). It is by such means that, in this Tarskian solution, a language is said to be ‘interpreted’. So, if this is the case, the meaning is not embedded in the original semiotic artefact of the object language, and hence the meaning is external to the system. This externality problem for interpreting these symbols was not easy to solve. But some solutions were proposed. The most famous one came from the connectionist paradigm. Originally, it saw itself as an alternative to the syntactic symbolic paradigm: ‘the connectionist dynamical system hypothesis provides a connectionist alternative to the syntactic hypothesis of the symbolic paradigm’ (Smolensky 1987: 194). Similar theses were claimed in the neurosciences. The intuition in this solution was to abandon the underlying semiotic thesis that computer processes symbols that ‘stand for something else’ – that is, that they ‘represent something’. The prototypical semiotic concept of representation was hence to be banned: ‘representation is the wrong unit of abstraction in building the bulkiest parts of intelligent systems’ (Brooks 1997: 139).

26

Computational Semiotics

Despite these radical claims, the semiotic concept of representation did not really disappear from this connectionist, neural and robotic paradigm. It was changed into apparently non-semiotic ones, such as the concept of subsymbolic characterization. ‘The subsymbolic characterization of a cognitive system intrinsically binds cognitive systems both to states of the environment and to goal conditions’ (Smolensky 1988: 15). But as many critics made it clear, this subsymbolic thesis is still a semiotic thesis. ‘If “sub”-symbols are still representors manipulated according to precise rules, there are still symbols, expressing concepts in any traditional sense of those terms’ (Lycan 1988: 43). So, a more nuanced version of this semiotic concept of representation was offered: it became a distributed representation. ‘When we speak of a distributed representation, we mean one in which the units represent small, feature-like entities. In this case it is the pattern as a whole that is the meaningful level of analysis. This should be contrasted to a one-unit-one-concept representational system in which single units represent entire concepts or other large meaningful entities’ (Rumelhart, Hinton and McClelland 1986: 47; emphasis removed). As we may guess, a distributed representation is still a representation. And it does not eliminate its ‘stand for’ role. It still leaves unanswered how meaning is given to this different distributed representation. As it is often said, the meaning is just ‘correlated’ to it, ‘associated’ to it. And this problem still lingers in contemporary forms of AI: machine learning, as deep as it may go, always remains a manipulation of symbols whose meaning is still distributed over many complex and interwoven levels of manipulation. And whatever their output is, it still has to be interpreted. This external view of meaning became the core of the many early criticisms of most AI projects.5 According to such views, artificial intelligence would present only a syntactic theory for processing symbols. Nöth (1997) has noted this conceptual crisis: AI is said to be a representational system for referring to something. But formally, it is only a syntactic system with no semantics. So ‘the concept of representation has been in some crisis with those who have used it without a solid semiotic foundation’ (Nöth 1997: 213). From a semiotic point of view, Fetzer even declared that Newell and Simon’s systems do not qualify as semiotic systems (Fetzer 2001: 55). Many debates and counter-solutions have been offered regarding this externality problem of AI. But just like ‘good old-fashioned’ AI, most of them do not really solve it. For instance, in neural AI systems, the meaning of symbols is still external to the system. Sometimes, the meaning is given in a declarative manner (dictionaries, ontologies, programmers, etc.), which is often hidden among ‘big data’. And in robotics, this meaning is sometimes given by some complex causal relation with the environment (captors or various reception sites). A nice metaphoric but problematic formulation of how the ‘distributed representation’ of subsymbols ‘come to have meaning’ is that the system learns them! Unfortunately, from a semiotic point of view, what these systems really learn are new semiotic artefacts or new semiotic artefact structures. The meaning is not grounded in the fact that they can be manipulated in some simple or complex computable way, but rather in the fact that they are embedded in some process of

 Semiotics in Computing 27 semiosis external to the system itself, be it by the interpretative semiosis accomplished by the programmers, by the users or in some causal relations to the environment. A computer, as such, is a physical technology. What is really manipulated are physical events: the electric polarity of transistors in digital computers or qubit states in quantum computers. These physical artefacts are without meaning. When processed, the outputs produced are still physical outputs. But often, these outputs are given, in conceptual frameworks, a semiotically loaded name. They are often called signals. Shannon’s understanding of such physical signals allows the inference that they carry a message or some information. Unfortunately, with Shannon, the concept of message is only defined by the probability of its occurrence. And for the receiver, this probability itself only changes a degree of incertitude. But it does not say anything about the content of the message or about the information content. On the other hand, in Peircean semiotics, a signal is sort of a sign: it is one that stands for something else in some type or another of direct or immediate way: a signal, for Peirce, is a sort of ‘index’ or ‘indice’ type of sign: ‘Indices are signs which stand for their objects in consequence of a real relation to them. An index is a sign which stands for its object in consequence of having a real relation to it’ (Peirce 1866: 8). But what Peirce also adds is that indices carry information. But here, information has a specific meaning. It is related to the acquisition of knowledge: ‘I call any acquisition of knowledge “information”, which has logically required any other experience than experience of the meanings of words’ (Peirce 1910: 11). Thus, there is quite a difference between information content as conceived of by Shannon and as conceived of by Peirce. Dretske proposed a very apt formulation: ‘We convey information by using signs that have a meaning corresponding to the information we wish to convey. But this practice should not lead one to confuse the meaning of a symbol with the information or amount of information carried by the symbol’ (Dretske 1982: 44). So, artificial intelligence, both in its classical form and in its neural form, embeds a host of implicit semiotic concepts. We find elements such as signals, code, symbols, representations, interpretations and information. Computation itself is seen as a complex sort of manipulation – or ‘processing’ – of such semiotic artefacts. Some complex structures of the semiotic artefacts are called ‘knowledge representations’, some types of processing are called ‘inferences’ or even ‘reasoning’, and insertions of new semiotic artefacts are seen as being learned. But if artificial intelligence is to respect the computational theory that is at its foundations, it must also maintain its deep nature: computation is always a syntactic manipulation of such semiotic artefacts. This does not imply that these artefacts do not have meaning. But if they do, it is not the syntactic structure itself that confers it to the semiotic artefacts. In this way, a semiotic theory of computation calls into its model not just the structures of semiotic artefacts but the internal structure of signs and, most of all, the interpreter, the context of the interpretation and the environment. So, if artificial intelligence has opened up a relation between computation and symbols, it has mainly been a nominative one. In reality, it has washed away their core content: the origins and nature of the meaning that symbols manipulate.

28

Computational Semiotics

Computer systems and semiotics The last attempt to explicitly link cybernetic control systems with semiotics came from rather solitary computer scientists. To the best of our knowledge, Zemanek (1966)6 was the first to suggest grounding the emerging technologies of programming languages upon a larger theoretical semiotic background. Symbols in programs were to be interpreted in the context of their use. And semiotic theory was the best way to go. He proposed to see this through Peircean semiotic theory where indexical signs are related to their context of use. But his proposition was not widely adopted. So, it faded away and it took some years for it to be resurrected in the 1990s. But this time, it became very open and explicit. One first claim for this semiotics-computation relation came from Andersen (1986, 1991). As he wrote in a decisive book called A theory of Computer Semiotics: ‘AI lacks theoretical and empirical justification, and cannot be taken seriously as a scientific paradigm’ (Andersen 1991: 1). Computer conceptual frameworks therefore had to be understood through more complex semiotic theories. Computers could not just be seen as autonomous symbol manipulating machines. As such, they must be conceived, above all, as instruments used by humans in a social and communicative working context. ‘The book is a part of an ongoing paradigm change in computer science itself: from seeing the computer system as a self-sufficient mathematical object, the focus is gradually being shifted to the relations between system and work context’ (Andersen 1997: 310; emphasis removed). This introduction was justified by the attention given to the interaction of the computer with its environment, be it just other computers, objects or persons. In parallel to this reaction, some other computer scientists introduced in the autonomous control system the property of being intelligent. These systems, often called autonomous intelligent systems, were not mainly defined by a strict and rigorous internal structure (closed systems), but by having many types of relations to the environment (open systems) with which they must deal. This meant that such systems had to embed feedback from their environment but, above all, that they had to perform under conditions of faults, deadlock, reconfiguration, optimization or fuzziness.7 Finally, these self-controlled systems were to be called intelligent self-controlled systems (Pendergraft 1993; Antsaklis 1994). Opening such research programmes revealed the complexity of intelligent operations. For example, it was no longer easy to distinguish between the environment and the system itself, between the operations of the system and the object upon which it operated, or between the controller and the controlled. Moreover, integrating intelligence within the system also required learning, memory, adaption and evolution. This vision of intelligence came with the emerging new trends in artificial intelligence for which other types of formal models had to be explored (connectionism, neural nets, etc.) so that artificial intelligent systems could deal with environment. In this vein and at the same time, through some important conferences and publications, computer scientists proposed to explicitly integrate semiotics into the modelling of these intelligent systems. This opened up a new stream of semiotic

 Semiotics in Computing 29 research, of which Rieger (1999a) and Perlovsky (1998, 2000), as well as Gudwin and Queiroz (2007), were the leaders. For instance, for Meystel and Albus (2001), an intelligent system must possess complex internal states and operations that allow it to receive information from its environment and from itself. It must learn, adapt, remember, intend and finally engage in action upon itself and its environment. Such operations require different types and levels of information processing. In other words, for these computer semioticians, in an artificially intelligent agent, there exists a multilevel architecture of operations that Meystel, being highly influenced by Peirce, called multiresolution semiosis, in which various types of signs and different modules containing operations should be distinguished, such as perception, world modelling, value judgement and actuation. One important development of the semiotic grounding of these intelligent systems came from Gudwin and his colleagues. Through continuous writings from 1995 to 2005, they developed a specific understanding of computational semiotics, which Gudwin defined as follows: Computational Semiotics refers to the attempt of emulating the semiosis cycle within a digital computer. (Gudwin and Gomide 1997a: 39) Computational semiotics corresponds to the proposition of a set of methodologies that in some way try to use the concepts and terminology of semiotics, but composing a framework suitable to be used in the construction of artificial systems. (Gudwin 1999)8

This type of computational semiotics proposes a new kind of approach to intelligent control and intelligent systems, where an explicit account for the notion of sign is prominent. This focus on the notion of sign allows the inheritance of a large body of theory developed under the scope of semiotic studies in order to help artificial intelligence walk through new frontiers and bridge theoretical gaps which have been disturbing artificial intelligence studies for a long time (e.g. the symbol grounding problem) (Gudwin and Queiroz 2007: 2). We emphasize here two of Gudwin’s main contributions. First, he developed an architecture of semiosis where the different levels are understood in a simplified manner through Peirce’s category theory of firstness, secondness and thirdness. Briefly formulated: firstness is where information enters a system and is what it is. Secondness is a set of second-order operations where the information is organized, classified and memorized by the agent. These operations are applied to the first-level objects and states and even to other second-level ones where they form some sort of controlling operations. And thirdness is the third degree set of operations where the whole system is controlled as a self. Second, he explored various mathematical models corresponding to these multilevel information processes. Thus, computational semiotics uses the concepts brought from semiotics to propose a hierarchy of elementary types of knowledge and base them on a mathematical model (Gudwin and Gomide 1997a: 5). In parallel to these projects, some important research emerged and explored the semiotic paradigms for describing and explaining specific computer operations or applications such as

30

Computational Semiotics

computer human interrelations, information systems, programming languages and library systems. Nadin (1988, 2007, 2012) explicitly saw a computer as a prototype of a semiotic machine. ‘In order to be meaningful, computers ought to be semiotic machines’ (Nadin 2011a: 23). ‘Computers are semiotic machines driven by semiotic engines’ (Nadin M. 2007: 64). Liu (2000) broadened this thesis and applied it to whole information systems in organizations. Brier (2011) extended it to communication activity, context, culture and language. Later on, the role of semiotics in computer science was explored by other computer scientists. For Tanaka-Ishii, semiotics was not called upon only for building intelligent systems. Her research aim was ‘to consider programming languages from the viewpoint of signs and sign systems in general’ (Tanaka-Ishii 2010: 2). And in this perspective the reflexivity embedded in programming language revealed itself as an essential semiotic property. For this researcher, it is this reflexivity feature that allows a computer to really become a semiotic machine. For it is by this property that the stand for relation of the symbols can relate to the hardware and to the external world, be it the user, the culture, etc. Ultimately this research program was called ‘semiotics in programming’. Finally, she believes ‘that understanding the semiotic problems in programming languages leads us to formally reconsider the essential problems of signs’ (Tanaka-Ishii 2010: 2). Afterwards, such a perspective was reused to define many types of algorithms, if not computers themselves. This vision will cross several projects where computers are called upon: for example, in robots,9 human-machine communication10 and information systems.11 Finally, not only computers were analysed through semiotic conceptual frameworks, but also the many new digital semiotic artefacts produced through our human interaction with computers: ‘When we interact with the computer, we can do this only through the manipulation of symbols, which imperfectly represent (a) the real-life object that they stand for; (b) the way in which it is stored and manipulated within the computer; and (c) the way in which it is represented in the code of the application that we are using’ (Clarke 2004: 167). Indeed, this interaction of humans with computers gave rise to a host of new semiotic artefacts that offered important digital features – for example, e-media,12 electronic literature,13 media art,14 multimodal electronic artefacts15 and even a whole digital culture,16 all of which became the object of a specific trend of computational semiotics which later was named digital semiotics.17 For the organizers of the important series of Cosign conferences: Computational semiotics is understood here to be the application of semiotic theories to computer systems and interactive digital media. Three possible aspects of this are: • The way in which meaning can be created by, encoded in or understood by the computer (using systems or techniques based upon semiotics). • The way in which meaning in interactive digital media is understood by the viewer or user (again using systems or techniques based upon semiotics).

 Semiotics in Computing 31 • The way in which semiotics can be used as the starting point for a system for looking critically at the content of interactive digital media. (COSIGN 2002)

As we can now see, although it took quite a long time for semiotics to encounter the computers and their digital artefacts, the former appeared as highly relevant for the understanding of the latter. This conceptual framework of computation has found its way into many fields of computer engineering and knowledge-representation technologies: ontologies,18 conceptual graphs,19 visualization,20 the semantic web21 and so on. Still, even if there is a well-justified relation between computation theories and semiotics and though it is well recognized by semioticians, we must admit that except for a few rare computer scientists (Sowa, Gudwin, Meystel, Albus), the overall conceptual framework of semiotics did not really integrate classical computational theories. One main difficulty in the introduction of semiotics for understanding the computation of symbols or signals (in programming languages or in autonomous intelligent agents) seems to rest on the difficulty of explaining how the meaning of the computable semiotic artefacts comes to be. As Nadin (2012) will say later, maybe the definition of sign had been too limited to be effective in the realm of complex computer programs. As we have shown, this is evident in the case of artificial intelligence. The AI computational model was criticized precisely regarding this externality problem. But in the specific type of computational semiotics for intelligent systems, most of the Peircean concepts which were called upon focused on the triad of sign, interpretant and object. And the process of semiosis was mainly described through Peircean categories. But overall, despite the richness of these concepts, this semiotic reading of computation focused more on the understanding of the complexity of the manipulations of symbols than on the origin of the meaning of the signs that were computed. And AI never became interested in explaining through some semiotic framework how computer symbols came to have their interpretant and how they can come to designate a specific object or referent. Still, in this definition of computational semiotics, the computational understanding of the semiotic relation showed the complexity of the semiotic artefacts (signals and symbols) and processes, but it does not thoroughly explain how the meaning is associated to the symbols themselves. There are many explanations of this theoretical problem of the origin of the meaning of computing’s semiotic artefacts. The first one is that the theoretical problem is due to the definition of computation itself. The classical Turing definition of computation is a process where the symbols have no meaning. If they do have one, then it is external to them. This is the barrier of meaning problem. And how this meaning emerges in computable symbols has not been the main focus of computational semiotics as such. The second explanation is that the problem of the meaning of symbols is itself a easy one. It is in fact a core problem of contemporary philosophy, psychology and of natural computation theories. For Searle (1980) and Harnad (1990), this is the grounding of symbols problem. For Floridi (2004), it is the data grounding problem: how do data acquire meaning? And it has become a central problem of some contemporary naturalist philosophies,22 where it is expressed in terms of the semantics of data or information in interactive computation.

32

Computational Semiotics

Some researchers such as Rieger (1998, 2003), Bundgaard and Stjernfelt (2009), Brandt (2004), and Brier (2008) have explored a similar type of question, but they expressed it in semiotic terms. In 1998, Rieger had already formulated this externality problem in ecological terms, stating that ‘[the] semiotic character consists in [a] multi-level representational system of (working) structures emerging from and being modified by [its] processing’ (Rieger 1998: 840). This process is characterized by Rieger as being ‘multi-resolutional’ and ‘ecological’. Brier (2008) goes slightly deeper into how symbol manipulation processes allow meaning to effectively emerge. Even if computation is seen as information processing, this is not enough to explain the emergence of meaning. Information is not meaning. As he formulates it, meaning is outside: ‘meaning must be interpreted by the receiver outside the transmission’ (Brier 2011: 20). Meaning emerges in communication, in cognition, and in and between interpreting agents who have some common phenomenal and embodied knowledge experience. The basis of meaning is generated by the embodied minds of individuals: ‘I would say that both information concepts presuppose the human mind and its social, embodied and existential concept of meaning’ (Brier 2011: 19). Finally, Brandt (2004) and Sonesson (2012) reformulated this main thesis: meaning is the result of complex cognitive process. And they anchored or grounded it in perception. In other words, for these semiotic theories, meaning rests upon specific capacities of living beings. It is the result of a complex process of semiosis where there are many explaining factors – representation, phenomenological and bodily experience, information processing, embodiment, autopoiesis – and it is realized in some situations, contexts, environments, where cognitive agents interact. By itself, the manipulation of symbols does not produce their meaning!

Conclusion: The semiotics of computing Computational semiotics, as we have seen, has built a fragile bridge between semiotics and computation. And it is important to keep in mind that this research program has taken a specific direction. The path goes from semiotics to computation. What we have is a semiotics of computation or, as Andersen (1997) formulated it, a semiotics of computers. And although this bridge is theoretically heuristic, it was not accepted throughout the computer science community. For some computer scientists, semiotics was often understood as knowledge enquiry restricted to specific semiotic applications or practices which are often very limited and empirically difficult to study – for example, semiotic cultural artefacts that carry secondary symbols. ‘Hence to study these secondary symbols is to study the body of cultural conventions, intentions, aspirations and so on, of individuals and groups’ (Pylyshyn 1983: 118). It is evident that the vocabulary employed by computer science and by contemporary artificial intelligence is not that of semiotics. But if we unwrap the conceptual frameworks underlying this language, we find an implicit semiotic theory. And this

 Semiotics in Computing 33 theory is very specific. According to one of the main postulates of computational theories, an effective computation is one that processes symbols regardless of their meaning even though these symbols may happen to possess meaning for some agent (machine, organism, animal, human). In more linguistic terms, a computational processing of symbols is strictly syntactic, not semantic. There is meaning attached to these symbols, but it is external to it. This is the core of the classical Turing thesis. Naturally, how these symbols acquire meaning has been a matter for a whole research program. It is in this respect that semiotics appears so important for research in computational semiotics. For a semiotic theory of computation, one cannot just declare that computation is the processing of interpreted symbols. One must explain how it acquires and sustains meaning. In other words, the computation processing these symbols must be sensitive to a myriad of constraints, such as the rules that govern the processing, the conditions or context in which the process is carried out and the physical structures that implement it. But if all of these constraints are accepted, the problem plunges into the realm of complexity. The processing of symbols can be seen as a complex process of semiosis if it is determined by more than just a few formal computable functional relations. In other words, a semiotic theory of computation adds complex pragmatic conditions to a syntactic view of computation: a computation is a sort of process of semiosis applied to symbols. It must exist in an embodied agent, be acting in an environment and communicating with other agents. In this semiotic perspective, computational semiotics sees a computer as a mechanical embodied agent that manipulates many types of symbols related to an environment and which can communicate with other computational agents. ‘Computational semiotics can be characterized as aiming at the dynamics of emergent meaning constituted by processes which may be simulated as multi-resolutional representations within the frame of an ecological information processing paradigm’ (Rieger 1998: 840; emphasis removed).

34

3

Computing in semiotics

The barrier of meaning One of the dominant theoretical orientations of computational semiotics has been to explore the application of semiotic frameworks to computation technologies. This aimed to explicate the notion of symbol manipulation realized in a computer. Though this vision of computational semiotics has been heuristic, there exists another vision of computational semiotics that goes in a completely different direction, one where it is rather the computational framework which dives into the realm of semiotics and, more specifically, into the world of semiotic artefacts and semiosis. Just as in the theory of computation, we find in semiotics a thesis about the manipulation of some semiotic artefacts. Depending on their function, these semiotic artefacts go under the different names for signs. And according to the Peircean theory of signs, which is the best known one, they are divided into three main types: symbols, icons and indices, which are also sometimes called ‘signals’. All of these types of semiotic artefacts exist in some physical material. As physical entities, signs are carriers or vehicles of the sign function. And when these physical entities are explicitly recognized as carrying meaning, they receive a specific linguistic name that connotes this signifying function. For instance, a huge piece of marble sculpted in a particular way may be named a statue. A well-organized sequence of sounds of various heights of pitch and of various durations becomes a musical melody. The movement of part of a body or of a whole body is called a gesture or a dance. Well-structured sequences of ink spots on paper or of oral sounds become words, sentences and texts. A flashing red light on a street corner is said to be a traffic control signal. Many of these physical meaning-carrying entities are human-made artefacts. But it may even happen that natural phenomena such as storms will become carriers of meaning. They are seen in certain contexts as signals, if not as symbols. But what is so special about these physical entities or artefacts is their particular feature of meaningfulness. And it has been the role of semiotic theories to offer explanations and models of what form this meaning may take, how meaning comes about and, most of all, what is the actual meaning of specific artefacts. Many of these theories have different points of view regarding the nature of this meaning-carrying function. But it does not mean that they are contradictory to one another. In fact, they are often complementary. But when we integrate their various

36

Computational Semiotics

points of view on meaning, producing a semiotic theory entails a complex enquiry. And one core problem of this meaning feature, as we mentioned earlier, is that it is not an intrinsic property of the physical entity that is its carrier. Rather, the meaning is a property which is extrinsic to the physical entity itself. This externality of meaning has often been called, metaphorically, the wall or the barrier problem of meaning.1 For example, the meaning of a ring on a person’s finger or of an Islamic veil is not in the physicality of the ring or of the veil themselves. Rather, the property of meaning is external to it. Computational semiotics, understood as a semiotics of computation, cannot avoid this problem. It is a core one for a computational approach to semiotic artefacts: How is meaning identified? What it is nature? Why does it exist?

Semiotic enquiries: Research into the realm of meaning Semiotics is a specific type of epistemic enquiry. It investigates certain objects and behaviours, and events in their capacity as semiotic artefacts, that is, as carriers of meaning. Like any other science, it aims at categorizing its objects and at reasoning on them. Theoretically oriented semiotics aims mainly at categorizing its domain through some general conceptual framework. To this end, it proposes concepts and principles for describing, explaining, understanding and interpreting semiotic artefacts and semiosis. These are usually expressed in natural language: statements, propositions and discourses. Rarely are they wrapped up in logical formulas or equations. Applied semiotics, on the other hand and as the name states, is concerned with applying some of these concepts and principles to specific semiotic artefacts or to semiosis. Naturally, each semiotic conceptual framework developed by the theoretical branch of semiotics determines a methodology for each concrete semiotic enquiry. For instance, a philosophical framework will aim at discovering, in semiotic artefacts or in semiosis, features that are indices of elements of mental representations, of the situation or of the interpretation, and hope that this will reveal part of the meaning content of the artefact under study. The structuralist paradigm will pay attention to the structure of the various components that make up a semiotic artefact, both in its static states and in its dynamic process, hoping thereby to reveal the meaning underlying them. A naturalist cognitivist paradigm would not deny the importance of these features, but would rather concentrate on what is at stake when cognitive agents make or identify meanings or when they communicate such meaning to members of a community. Whatever the conceptual framework chosen for investigating semiotic artefacts, these concepts and principles must be applied to reasoning in some manner. For instance, in a deductive approach, the reasoning uses semiotic principles and parameters for identifying and classifying the main semiotic features of the artefacts under study. And the results of the enquiry will be valid until a counterexample appears. It is a theory-driven enquiry. But often, because semiotic artefacts are complex, only a few occurrences – if not only just one – will suffice to be deemed an instantiation of the principles and parameters ensuing from a conceptual framework. A typical example is Greimas’s semiotic square. This formal model was used to identify a particular semiotic

 Computing in Semiotics 37 structure in various types of texts. And it stood as an acceptable proposition for as long as there were no counterexamples. Still, in order to succeed, this deductive approach requires a high-quality method for identifying the various features of a particular semiotic artefact that is seen and used as an instance of the semiotic principles guiding the research. But when the artefact is complex, the features may be numerous and diverse: context, points of view, users, situations, structures, time period and so on. In such a case, the adequation of the object studied with the principles selected may not be as obvious as anticipated. I have ventured myself into such a type of deductive semiotic analysis (Meunier 1994). The conceptual framework chosen was the dominant symbolic mental representational thesis from cognitive science. It was deconstructed into five main cognitive semiotic operations: perceptual, praxiological, normative, epistemic, ipseic and didactic ones. These were applied to a painting by Vermeer: Woman Holding a Balance (1659–61) (Figure 3.1).

Figure 3.1  Jan Vermeer, Woman Holding a Balance. Public domain.

The analysis showed in detail how this painting instantiated the five cognitive operations retained, and that the meaning of this semiotic artefact reveals itself through many underlying complex cognitive and communicative operations. It does

38

Computational Semiotics

not form a homogeneous and uniform structure, but moulds itself to the diversity of cognitive operations to which it is submitted. In this sense, the painting calls upon a complex bundle of cognitive acts of diverse types. Such a deductive approach to semiotic artefacts is omnipresent in practices of semiotic analysis. It is a dominant method in the academic linguistic tradition. For example, in the generative linguistic paradigm, the conceptual model takes the form of a set of rules making up a grammar. The analysis then consists mainly in discovering linguistic segments that are instantiations of the principles and rules of the grammar. Often, this method is seen as a structural and qualitative type of analysis. But surprisingly, it could also be seen as forming a deductive empirical approach, for it has a quantitative feature. First, there may be a certain ‘quantity’ of principles or parameters that have to be defined in the conceptual framework. But second of all, it also has to be applied to a specific ‘quantity’ of instances. Often, the number of such instances is very small, if not close to 1. But just 1 instance is still a quantity. The tag ‘qualitative’ hides this inherent quantitative dimension that is necessarily present in a process of deductive reasoning. It is rather the creativity and intuition at work in (a) explicating the conceptual framework itself, (b) applying it concretely to instances and identifying the features of a particular artefact instantiating the principles and rules put forward in the conceptual framework, and (c) interpreting the result as revealing the artefact’s meaning. But by the same token, it obscures the empirical quantitative explanation underlying it. The second type of reasoning is an inductive one. The well-known characteristic of this method consists mainly in observing a great number of either different or similar semiotic artefact samples with the aim of arriving at some generalizations regarding their features or of finding some statistical or structural regularities or patterns in them. The validity of the results depends on the quantity of the samples. A typical example of such analysis would be the study of thousands of birthday parties in order to find dominant patterns or regularities. It is hoped that through the interpretation of these patterns grounded on observation and on data, meaning would be revealed. A complex example of this type of work may be found in the semiotic analysis conducted by Scardovelli (2018). In this project, 2,264 paintings and drawings of animals and humans on the walls of twenty-six different Palaeolithic caves in southern France were studied. Thousands of photographs were taken, and for each painting and drawing, some 300 feature types were identified, such as living entities (animals, humans, plants), objects (tools, weapons), structures (face or lateral view, framing, composition, size, colour, texture, realism), referentiality (realistic, cosmological, etc.), the position in the caves, etc. One first highly interesting feature of these drawings was the lateralization of the drawing movement – the direction in which the lines were drawn for certain animals (i.e. from right to left or from left to right). This was an indication of the state of development of the right or left side of the brain. Another important one was the distance of the paintings from the cave entrance: towards the entrance, in the middle or deep within the cave. Basic quantitative analyses on these thousands of paintings discovered some oppositions between some of the figures, while some others discovered regularities. The most interesting results were, among many other ones, the following generalizations or regularities: (a) some drawings were clearly drawn from left to right, whereas others

 Computing in Semiotics 39 were drawn from right to left. This indicates an emerging state of specialization of the brain during the Palaeolithic age; (b) the more dangerous animals were mostly situated deep within difficult-to-access spots of the cave, ones where there is less light. This indicates reasoning and decision-making by these humans on some non-explainable facts about danger, life and death – that is, mysteries as such. The result was interpreted within the framework of the semiotic theses of Peirce, Leroi-Gourhan, Greimas, Eco and Luhmann. It focused on the highly abstract but bodily embedded representational capacities of the Palaeolithic human and the power of epistemic, cultural and social communication. But one important point of this analysis is that most of the data were collected and organized using a classical ‘manual’ process. The use of computers was limited to the collection of data, to calculating basic statistics on this data and, ultimately, to writing the thesis itself. Such an approach is clearly inductive. But in more contemporary language of scientific marketing, it appears chic to call it a data-driven methodology. But this seems to be a misnomer. These features are not given (from the Latin verb to give dare: data), but are chosen by the researcher himself or herself. Most often, they are taken from the set of standardized features accepted by the epistemic community specialized in such types of research.2 It is only afterwards that they become the variables of some formal models on which some inductive types of mathematical analysis are applied. This inductive approach is not that widespread in actual semiotic practices. One of the main reasons for this is the time-consuming nature (if not also the economic cost) of the methodology, notably in cases where the artefacts are numerous, are very different from one another and highly complex. This explains, in part, why a deductive approach would be preferred. Even if it appeals to theoretical concepts and principles, an inductive reasoning process is quite time- and energy-consuming. But there is also another reason. Some semioticians see this approach as quantitative and not as qualitative and therefore as too naturalist, if not experimental. But if they were to conduct a deeper analysis of the methodology, they would find that the main problem is not the quantity of artefacts, but rather the conceptual framework that is used to build the corpus, identify the features (variables) and interpret the results. They may even find that after a multitude of inductive analyses of semiotic artefacts, the interpretation to be given will be a rule or a structure – not just a quantity. In other words, determining or identifying the data necessary for interpreting the results of an inquiry requires much creative intuition. These two methods – the deductive and the inductive ones – are both accepted in practices of semiotic enquiry. The deductive method, though, is more widespread than the inductive one. Still, both are entrenched in the externality problem of meaning. And both confront the barrier of meaning, as they seek to unveil the multiple determinants and components of the meaning carried by the semiotic artefacts and associated processes of semiosis. Depending on the specific conceptual framework chosen, the questions underlying the enquiry may include some of the following: What is the structure of the artefact? Who is using it? For what purpose? How is it interpreted? By whom? What is the situation? How has it evolved? How is it communicated? In other words, these methods may involve all the concepts and processes linked to the various semiotic frameworks we have seen.

40

Computational Semiotics

In actual semiotic practices, the use of computers is rather absent, as they are believed to be of little use except for menial tasks such as archiving, counting occurrences or printing. Most projects rely on manual or mental power. And it may be difficult to imagine how most of the procedures embedded in the two reasoning and demonstration methodologies presented in the preceding paragraphs could become computable. In semiotic analyses, it is difficult to think and to accept that the discovery and justification of the type of reasoning activated – often called the interpretation – are computable processes. They indeed look like instances of slow-cooking ‘interpretative’ processes.

Computation in semiotic enquiries: Two solitudes Still, the means by which semiotic artefacts are analysed are changing. Computation is slowly penetrating the semiotic world. The landscape is such that, on the one side, there is a community where the leaders are experts on certain types of semiotic artefacts and, on the other side, one where the leaders are computer experts. Among the first community, the computer is involved either because of its digital semiotic products or to be used as a tool for semiotic analysis itself. In the former case, projects often go under the name of digital semiotics (O’Halloran et al. 2009; Danesi 2016; Cannizzaro 2012, 2016; Liu 2015; etc.). They analyse various multimodal digital products such as e-games, e-music, e-videos, e-films, tweets, digital art, social media, archives, emojis and internet sites. Such digital artefacts carry meaning just as classical, non-digital semiotic ones do. But they are often more complex because of their digital multimedia formats, which necessarily raise new semiotic questions. In the latter case, the computer is used as a tool in various tasks related to classical semiotic methodologies such as archiving documents, building corpora, analysing data and disseminating results. Many of these projects go under the umbrella name of digital humanities.3 But despite their use of a name that refers to digitality, most of them study classical semiotic artefacts such as language, philosophy, literature, arts, popular culture and media, and many cultural themes such as feminism, futurism and gender. They often call upon traditional semiotic conceptual frameworks. For instance, some projects in text analysis study stylistic, narrative, discursive, topical or argumentative structures. Projects in music study influences, styles, composition and patterns, and they may do so synchronously or diachronically. Even very technical digital humanities projects that pertain to e-editing, e-archiving or e-classifying encounter most of the classical semiotic questions about meaning. For instance, the Text Encoding Initiative (TEI), which, by means of computer-assisted text annotations, classifies and categorizes various types of textual semiotic artefacts, can be seen as a highly specialized technical semiotic endeavour. Although these digital humanities projects do not necessarily recognize themselves explicitly as semiotic or call upon traditional semiotic frameworks, the artefacts they study are interesting precisely because they are meaning carriers and have to be interpreted as such. Usually, in these projects, semioticians or experts sensitive to the semiotic content are the leaders and computer experts are consultants, while the computer technology is mostly seen as a tool.

 Computing in Semiotics 41 Despite the great number of such projects, many semioticians resist the introduction of computers into their discipline. In fact, one finds very few explicit semiotic projects where the computer is deeply involved in the methodology itself. It is more often used for basic operations such as taking notes, writing, building corpora, editing and publication. Rarely do we see the use of even basic computer analysis tools. For many, it seems that a semiotic methodology is essentially an interpretative epistemic enquiry and the computer is mainly seen as producing only quantitative or formal results: a computational approach appears too reductive for the explanation and the understanding of semiotic artefacts and systems. There is the idea that these artefacts cannot be studied by algorithms, automata or, even worse, by robots! Hence, the very term ‘computational semiotics’ is a paradoxical expression. This resistance of semioticians to computational semiotics is justified by their attention to the complexity of semiotic artefacts and of semiosis. And for them, meaning is something that is ultimately in the eyes of the interpreter, not in the object itself. It is not easy for them to accept that there could be an automatic computational semiotic analysis of Stonehenge or of a Hindu funerary ritual. The second community that uses the computer in a growing amount of research on semiotic artefacts is made up of computer science experts.4 They are discreetly doing semiotics under various disciplinary names such as ‘artificial intelligence’ or even ‘robotics’. And one such discipline explicitly calls itself ‘computational culture’: ‘In many ways, we’ll be forced to enter the age of computational culture because survivability and sustainability might otherwise be at risk, owing to the unprecedented speed and scale of social changes caused by new scientific and technological developments’ (Hardman et al. 2009). ‘Although the answer to the fundamental computability of culture isn’t clear, we must forge ahead because we simply can’t afford the consequences of avoiding cultural computing now’ (Wang 2009: 2). A convincing and concrete example of this is the exploration of semiotic artefacts by the digital technologist which is the great Google Book project. Its aim is simple: ‘make it easier for people to find relevant books . . . become the largest online body of human knowledge’ (Google Books Project, 20185). This project essentially processes artefacts that are identified through their meaning, that is, their textual content. One of the many difficulties6 that computer experts encounter in these projects is understanding the complexity of the semiotic artefacts they are dealing with. And because of the technical constraints brought about by computation itself, they often tend to break down this complexity into discrete properties and impose upon them some specific set of computable relations. This may be useful for basic tasks such as digital archiving, printing or dissemination. Unfortunately, if the semiotic artefact is complex and the meaning is rich, the processing may rapidly entail reductiveness or unforeseen computational complexity. So, as spectacular as these projects may be, they will have to simplify the semiotic problem by using only a limited number of classical semiotic concepts in order to control their manipulation in a computational manner, and hence lose their complexity. Still, with the increase in computer power, many of these AI projects are exploring with a surprising degree of success increasingly sophisticated semiotic artefacts and processes that only yesterday seemed beyond the reach of computers, such as translation, summarizing, extraction of semantic relations

42

Computational Semiotics

in natural language, discourse, dialogue, sentiment, topic, concept analysis, image recognition and musical composition. Although semiotic artefacts are omnipresent in these projects, their leaders are computer experts. In the best cases, semioticians will be consulted, and they will provide all the possible expertise on the project. But in the long-term, computer experts will keep the lead. And computer technology will continue to invade semiotic territories, but the projects will be enhanced by summary contributions by semioticians. Because of the epistemic solitude separating semiotics from computer science, building an epistemic collaboration between these two academic disciplines will not be an easy task. Here and there, there are individual semioticians and computer experts who collaborate, but in most practices, the mutual encounter is quite limited. Each one continues to ignore the other. And the pertinence of the conceptual frameworks of the other discipline for one’s own research programmes is not seen with acuity. Hence, their practices are isolated from one another. They are two solitudes.

An epistemological paradox We can now see that the great difference between these two scientific practices entails a paradoxical encounter. Both are grounded in two different ways of understanding and explaining knowledge of the world and, in particular, that of semiotic objects. They both belong to different epistemological paradigms. Semiotics and computer science then appear as two outright solitudes that cannot establish a dialogue or collaborate in their methodology for approaching semiotic artefacts and systems. On the one hand, a computer scientist such as Pylyshyn has even declared that ‘The symbols of the semiotician have no meanings and exhibit no behavior unless there is an intelligent, knowing agent to interpret them’ (Pylyshyn 1983: 118). In other words, their ‘interpretative’ process is only an intuitive method for revealing the meaning of semiotic artefacts. In this view, semiotics would just express a higher level of vague and non-provable statements based on socially shared beliefs. Overall, it lacks scientific rigour. Still, they will accept that semiotic artefacts can be seen from various perspectives: cognitive, cultural, informational linguistics, etc. But they will claim that if a semiotic artefact is to be the object of a scientific enquiry, computer processing and some formal computational model are required. These are the sine qua non constraints for a computer processing of semiotic artefacts. In other words, the results of computable operations on semiotic objects are not intuitive interpretations but a set of identifications and transformations of features of semiotic artefacts, so that the whole enquiry may also itself become a new source for understanding their meaning. For semioticians, the study of semiotic artefacts is not a vague and ad hoc interpretation. It has its own constraints. Bachimont (1996) and Rastier (2011) for instance tend to see the introduction of computation in semiotics as reductive. It looks like a variant of the empiricist and naturalist epistemological view of scientific enquiry even if it is applied to semiotic artefacts. Such a perspective would only touch the tip of the iceberg of semiotic phenomena. Formal explanations do not allow adequate description and explanation of the complex nature of semiotic artefacts and semiosis.

 Computing in Semiotics 43 Semioticians cannot see how a formal and computational model can reveal their meaning. For example, it is quite difficult to see how algorithms could reveal the deep meaning of Shakespeare’s Hamlet, of the Basílica de la Sagrada Famiglia, of Indian marriage rituals or of Beethoven’s musical style. Semioticians seek to know what properties make this meaning emerge, for instance, in what cognitive, dialogical, social or cultural context it appears. For semioticians, interpretation does not consist in the explicit revelation of what would be the ordinary popular meaning of a semiotic artefact. A serious semiotic enquiry is a complex epistemic endeavour. First, it must change the focus of the enquiry: the meaning of semiotic artefacts is not an intrinsic property of an artefact itself. Adapting Strawson’s (1960) famous formula: semiotic artefacts do not wear meaning on their sleeves. It is rather the result of a process by which some agent (an interpreter, usually a living being) ‘interprets’ some object or event as meaning something. And semiosis is the name given to the complex dynamic process where meaning is the result of numerous cognitive or mental operations, incarnated in a body and embedded in a social, cultural and physical environment. But there is a second, especially important explanation for this paradox. It pertains to the mode of presentation by which a semiotic enquiry expresses the meaning discovered. Imagine that a semiotic enquiry reveals some components of the meaning of funeral rituals. The semiotician must express these meanings by some semiotic means so that his or her own epistemic community understands what the result of the research is. There are many types of such means. It can be done for example through iconic or gestural language. Sometimes, but rarely, it could even be having the member of the community re-enact the event so as to provide a direct access to the meaning of the original artefact. For example, explaining in a conference the meaning of a firm handshake may be achieved by actually having everybody firmly shake hands with one another. But for most semioticians, this will not be sufficient. Most of the time, the description and the explanation of the meaning discovered in a semiotic enquiry will use natural language or some other type of language (graphs, films, videos, etc.). Natural language may produce propositions, statements, speech, discourses, comments, theories, critiques, explanations or proofs about the meaning discovered. These take various overt forms such as spoken words, sentences, texts, images and so on. And some of these linguistic and semiotics forms are often ritualized and highly structured: a conference is not a seminar or a journal article, for example. Such linguistic production must therefore be interpreted itself and understood, and, in turn, this process forms second-order semiosis. As we can see, semiotics is a complex epistemic enquiry. It has its own requirements. And there are sine qua non sets of epistemological constraints that must be respected. Enquiries by computer experts as well as those by semioticians form two types of epistemic practices which each proceed from their own epistemological posture. One gives precedence to interpretative understanding and the other to computable and empirically verifiable formalizations. The difference between the two epistemological perspectives is often misunderstood by players on each side of this epistemic game. Critics from one side often describe the other’s position through radically opposed simplifications: natural versus human

44

Computational Semiotics

sciences, induction versus deduction, hard versus soft sciences, quantity versus quality, intuition versus experimentation and, why not, laboratory versus armchair sciences. To put it in more traditional terms, it is an opposition between the Geisteswissenschaften and the Naturwissenschaften. Thus, the paradox appears both in practices and in their related epistemological explanations. Perhaps the two practices and visions will really intersect, but this seems to happen only in conferences, journal articles and the like. In other words, the encounter between the disciplines of semiotics and of computer science seems to be nothing more than an empty formula of interdisciplinary discourse. The two solitudes are well entrenched. The two intellectual traditions and epistemological positions do not even see each other as possible collaborators. Is there a way out of this paradoxical encounter, one that respects the signature of both positions? Our answer to this question will not be to give indications on how to use computers in semiotic projects or on what kinds of tools to use in a particular one. Our aim is rather to see if these two opposite epistemological positions regarding semiotic objects share some common denominator or some common ground and how and in what respect they may collaborate in their approach to semiotic phenomena. In more simple terms: Can semiotics live in the digital age? Can computation integrate the practices of semiotics? Whether semiotics like it or not, it will be confronted by the impact of computation within its field. And this will challenge its comfort zone. Hence, semiotics must better understand the real nature of the computational paradigm so that it may adjust its object and methods but also identify its limits. A positive answer to these questions will say that both players will have to take a step towards the other. On the one hand, as computer science finds itself more and more confronted with semiotic phenomena, it will necessarily have to be more sensitive to the inherent complexity of semiotic artefacts. It will have to better understand the semiotic paradigm in order to adjust its methods for processing the various features and dynamics of semiotic complexity. On the other hand, semiotics will have to explore possible avenues concerning how and where it can indeed integrate computation into its methods without betraying itself. In respecting its own signature, it will also have to determine how computation can be sensitive to the features and dynamics of semiotic artefacts and systems. In other words, our research questions are the following: Under what conditions can computers enhance semiotic enquiries? Ultimately, could computational means be employed in semiotics? This will require revising the epistemological understanding of scientific practices, as contemporary pragmatic philosophy of science proposes, and seeing how this can be applied in concrete semiotic projects where computation is called upon.

4

Models in science and semiotics

Theories in science While some computer experts believe that they can use semiotics to theoretically ground the construction of autonomous intelligent agents, we believe that it is possible for semioticians to integrate computation into their own practices. In fact, there are already many projects where computation is involved in the analysis of semiotic artefacts and of semiosis. But because the players on both sides are entrenched in two opposite epistemic postures – one which favours attention to meaning and its interpretation, the other which prefers formalization and computation – their encounter remains promising, but very fragile. Unfortunately, this manner of describing the encounter between semiotics and computation is not very heuristic. It has only fuelled the traditional opposition between explanation and understanding, facts and beliefs, discovery and demonstration, perception and conceptualization, interpretation and experimentation and, why not, intuition and reason. It has been lingering within many debates in the philosophy of science and in epistemology. And any hope to understand the relation of computing in semiotics while remaining in this stance will only solidify the paradox. One way of dissolving – at least in part – this paradox has been more recently explored in the philosophy of science. Rather than trying to solve the theoretical paradox itself, some philosophers have undertaken the task of studying the concrete epistemic operations that are present in actual practices of knowledge acquisition and, more specifically, in the building of scientific theories. In doing this, they have changed the view regarding the nature of a scientific theory. The main issue is no longer a matter of analysing the opposition between the two epistemological traditions, but of understanding what it is to build a scientific theory. One of the important theses of a classical philosophy of science is that, in its pursuit of knowledge, science is an epistemic endeavour that constructs theories about the objects it studies. And recent epistemological research has distinguished at least three main types of visions of what a scientific theory is. These distinctions will be very important in our own understanding of the integration of computers within the practice of semiotics. A first view on the nature of scientific theories is called ‘syntactic’. Originally proposed by the logical positivists (Carnap 1947; Reichenbach 1938; Hempel 1965; Nagel 1961 and many others), a scientific theory was defined as a collection of axiomatic

46

Computational Semiotics

statements and theorems. A scientific theory was thus taken to be a syntactically formulated set of primitive sentences (axioms, postulates, etc.) to which are added a set of inference rules that allow the derivation or deduction of theorems. A second view on theory is the ‘semantic’ one. Here, a theory takes a mathematical form (Suppes 1967). Or, to be more precise, as formulated by van Fraassen (1970: 327), a theory is ‘always a mathematical structure’, but to which is now added some semantic interpretation, such as a Tarskian ‘model’ for example. These two classical visions, rapidly summarized here, have been at the core of the epistemological conception of what a scientific theory is. They define science as a sort of formal semiotic system built out of a strict non-ambiguous vocabulary and a set of rules for their manipulation. Some of the rules control the production of formula. Other rules control the inferential transformations of formula (inductive or deductive) that build demonstrations and proofs. To this formal syntactic structure is associated a formal semantic interpretation that defines the truth conditions of the theory. And these truth conditions are often associated to observational and experimental conditions that explicate them in concrete and complex situations. In this way, a theory is formally related to some actual or possible world. Hence, a scientific theory practically becomes synonymous with a rule-governed epistemic explanatory enquiry. Though such views on the nature of theories have been widespread, they have been highly criticized (Winther 2016) for their limitations and constraints. And although such types of theories are technically acceptable, in practice they are rare and not easily applicable to many types of sciences (Papineau 2010; Leplin 1997). For instance, they have encountered many problems when applied to certain cognitive sciences and more so when applied to social and human sciences. The many difficulties of this classical view of theories will lead to a revision of the classical understanding of the very nature of science. Thus, a third view has been proposed: the pragmatic view. This third, ‘pragmatic’ view takes a different direction. It is grounded in the pragmatism of Peirce, James and Dewey, and in the conventionalism of Poincaré and Adjukiewics, but was developed by European critical philosophy1 and sociology of science, and reformulated and enhanced by the American pragmatist school of epistemology.2 The main thesis of this pragmatic tradition is that the classical formal view of science is an idealistic one. Theories of the latter type are much too constraining and do not correspond to any actual scientific practices. Most scientific practices are not totally formal and are not purely rule-controlled demonstrations and proofs. These criticisms have led to various epistemological reflections on the nature of scientific theories, proposing a view that aims to be closer to the real practices of science. And more recently, this pragmatic view has been renewed and reformulated by Stachowiak (1983), Cartwright (1983), Rheinberger (1997), Giere (1999), Morgan and Morrison (1999), as well as by many others. For these philosophers of science, a theory is not just a set of well-structured and rule-governed sentences whose semantics are given by a set of truth conditions, mainly related to observation and experimentation. Their view is rather that a scientific theory is a complex cognitive endeavour that aims not only at explaining a phenomenon but ultimately at understanding it. A theory is a discourse used by scientists to express ‘principles that govern [a group] of phenomena’ (Morgan and Morrison 1999: 12).

 Models in Science and Semiotics 47 And Winther adds, ‘effective scientific theories magnify understanding, help supply legitimate explanations, and assist in formulating predictions’ (Winther 2016). There are different visions regarding how these principles and explanations are devised. The aim of a scientific theory is to create knowledge of the world and, like any other type of knowledge (ordinary, situated, aesthetic, ethical, etc.), it builds epistemic artefacts that are replacements3 or surrogates of the world and which are often called ‘representations’. In this vision, a theory is seen as a representational state of stabilization of a complex evolving cognitive process process whose main purpose is acquisition of knowledge. It is often iterative, and it expresses the knowledge acquired not only through various non-formal semiotic and discursive forms such as natural languages, schemas and visualizations, but also and mainly through some formal semiotic forms such as mathematical and logical languages and graphs. Jeff Rothenberg (1989) proposed an elegant concise formulation of this idea: ‘Modeling in its broadest sense is the cost-effective use of something in place of something else for some cognitive purpose’ (p. 75). Because there are many types of interactions that science has with the world, these ‘representations’ can hence be seen as various cognitive perspectives and strategies that science must build for dealing epistemically with phenomena. Theories are then the results of these cognitive activities. They express points of view, perspectives and enquiry strategies adopted for understanding a phenomenon (Frigg and Hartmann 2012). The logical positivist definition of theories is but one way of understanding this ‘surrogate view’ of knowledge. And because the concept of representation is theory-laden and highly ambiguous, the concept of model has been proposed to replace it. It is more accurate than the concept of representation. It always involves a representative artefact, but it adds an epistemic function: it expresses a point of view, a perspective, a form of epistemic intervention on the phenomenon studied. Finally, the concept of model has become a core one in the pragmatic view of theory. ‘To explain a phenomenon is to find a model that fits it into the basic framework of the theory and that thus allows us to derive analogues for the messy and complicated phenomenological laws which are true of it’ (Cartwright 1983: 52). In other words, a theory is constituted by a plurality of different types of discursive components. Some are formal, others are non-formal. For Cartwright (1983), and for Morgan and Morrison (1999), a theory of ‘theory’ must be more inclusive than what is proposed by the syntactic and semantic views, but without rejecting them. One way to achieve this has been to approach the concept of theory through the concept of model, and models have become a dominant concept in the pragmatic view on theories. Thus, according to the pragmatic view, a theory is not just a set of interpreted formal statements. It is a complex epistemic process that builds and rests upon models and more so on a set of different types of models. And as such, these models have specific functions, roles and tasks in scientific practices. And these roles are numerous. Constructing models is not a practice restricted to classical natural and human sciences. It is also to be found in semiotics. In fact, semiotics is just a specific type of scientific cognitive activity which must, in its own way, also construct theories of phenomena or realities of a particular type: artefacts carrying meaning. An

48

Computational Semiotics

epistemological analysis of the various semiotic theories that exist, be they structuralist, philosophical or naturalist, quickly reveals that they constantly construct models in their actual practices. Their discourse is surely not one that can be reduced to wellformed axiomatic syntactic statements semantically associated to the meaning of some artefacts in an actual or possible world. It is much more heuristic to envision semiotic theories from a pragmatic perspective where they are seen as different points of view on the objects that are studied by semiotics. From such a perspective, semiotic theories also construct models. And as we shall see later, accepting the possibility of many points of view and therefore of many models will be the means by which they can envision the integration of a computational approach into their methodology. Such a vision invites us to discard the romantic vision of semiotics in which, for example, it is a solitary and patient researcher who studies some type or another of semiotic artefact, offering a personal and non-verifiable interpretation and understanding. Such a romantic image masks the complexity of semiotic practice. Semiotics is not just a meditation nurtured by spontaneous intuitions. In fact, semiotic enquiry is a kind of low-profile research. It is often a long-term epistemic endeavour that calls upon a multitude of discreet but rich cognitive activities, such as attentive observation, controlled conceptualization, rigorous reasoning, subtle linguistic discourse, original verification, numerous social exchanges and often creative technologies. These activities are applied to complex semiotic artefacts whose main feature – meaning – is not easily observable, recognizable and identifiable. In one sense, the concept of model, when applied to semiotics, can be seen as a technical name for the representational or surrogative process semiotics activates in the study of semiotic objects – objects which are themselves representational phenomena. As Coletta summerizes it: ‘semiotic modelling is a category and a process that encompasses ‘representational phenomena’ of almost every type, from how living things create the very conditions for their acts of perception and to how human beings represent in scientific terms the fabric of the space-time continuum and to how social space and time are created by the literal fabric of a new Fedora’ (Coletta 2015: 951). Modelling in semiotics is an epistemic construction about the components and dynamics of semiotic artefacts and processes of semiosis. And this activity of building models is itself a specific type of semiosis. Semiotics is hence a cognitive activity that constructs a set of models for describing, explaining and understanding phenomena or realities of a particular type: artefacts carrying meaning. In this chapter, we explore the role and nature of these models in semiotics,4 but with a view towards the integration of computation into its practices. This will constrain our choices of types of models.

Roles of models in science Whatever the type of models that scientific or semiotic theories build, models will play a variety of roles in knowledge enquiries. Given that in the present research, we aim at determining how computation may integrate semiotics, we shall distinguish three important such roles: epistemic, communicative and expressive.

 Models in Science and Semiotics 49 The first and most important role of a model is an epistemic one. For Cartwright (1983), models constitute the appropriate level of scientific investigation. Models express a point of view, a perspective, an angle on the object under study. They play an important role in building knowledge. They are not just representations of observational data or repositories of sense data. In fact, models transform data into ‘capta’, capta that are created, manipulated, categorized, memorized, ordered, evaluated, expressed in some semiotic form and communicated. For instance, in climatology, one type of model will have to categorize the myriad of data that pertains to temperature, pressure, humidity, wind speed and so on. These data are the result of complex cognitive operations that are, in the words of Kant, epistemic operations that pertain to perception, conceptualization and judgement. As Morgan and Morrison formulate it intuitively, the role of models, ultimately, is to ‘[fit] together . . . bits [of the world] which come from disparate sources’ (Morgan and Morrison 1999: 15). Some models will aim at applying structures formally representing certain specific relations among the data retained for experimentation and validation. In other words, each type of model will build its own specific surrogate means to deal with the features, properties and relations embedded in the phenomena under study. For example, a mathematical equation expresses a model that is a formal surrogate or representation of some relation among the data coming from the reality studied. Thus, models are different from theories in that they focus more specifically on the epistemic view on some phenomenon than on the ontological commitment they may have regarding it. In other words, models are not mini-theories in themselves. They are tools for the building of a theory (Rheinberger  1997). Unfortunately, as Giere (1999) reminds, the relation between a theory and a model is not one that is well understood. There are no necessary or sufficient conditions for identifying which model satisfies a theory. Still, I have the intuition that they are part of the narrative – not of science itself, but of the epistemology of science – which we must remember is a discourse. There are also models built in semiotics, and they do have such an epistemic role. Just like any other scientific enquiry, semiotics is an endeavour of knowledge. It offers different points of view upon semiotic artefacts and therefore has to identify some of their particular features, to categorize them and to reason on them. For example, a religious procession can be studied from different points of view for which there will be various specific semiotic models. For each model, features have to be identified, categories built and reasoning processes applied. But one particular problem semiotics will encounter is that such feature identification, category construction and reasoning processes are much more difficult to carry out than in classical science. For instance, discovering in a religious ceremony the multiple semiotic features, categories and regularities is quite a challenge. And many types of models will be called upon. They may be philosophical, structuralist or naturalist, for example. Still, there are many types of models that are typically used by scientists that are not significantly explored in semiotics. But things are changing; some semioticians believe that semiotic research can introduce, among many other things, some formal, computational and even experimental models. These models would make semiotics more similar to conventional scientific research.

50

Computational Semiotics

The second role of a model in science is an expressive one. Whatever content a model possesses, it must be made accessible to all scientists through some common shared language or through other semiotic forms. The most classical or well-known mode of expression will be formal languages whose goal is to express some mathematical or logical structure. For instance, it could be an algebraic, geometrical or logical language. Each type of language will be specialized in expressing a particular type of mathematical or logical structure. In the scientific practices of today, one particularly important formal language will be computational and algorithmic. Other semiotic forms that are to be found in science are iconic or visual languages5 such as images, graphs, schemas, pictures, figures, photos, maps, diagrams, layouts, portraits, mock-ups, simulations, visualizations and physical prototypes. As with other types of semiotic forms, these support knowledge categorization and reasoning, but do so in their own way. Used in models, iconic languages allow a holistic and parallel representation of the structure of the object under study. In many domains, the term model often means these last types of representation: model planes, model houses, replicas and so on (Campbell 1920). As indicated by Hesse ([1966] 1970), this reminds us that a physical model is not the reality itself but a physical representation of a reality. In this sense, it is a semiotic representation – just as a physical maquette of a house is a representation of a house, an electrical circuit diagram symbol can ‘stand for’ a real battery (Figure 4.1).

Figure 4.1  A battery symbol as used in electrical circuit diagrams.

Another type of such semiotic form may offer an isomorphism not directly with the object under study but with the structure that the formal equation or formula itself represents through some visualization graph – for example, the chaotic attractor’s graph (Figure 4.2). These many semiotic forms occupy an important place in scientific models. Think of their use in fields such as chemistry, electricity, medicine or physics. But as many researchers (Danca and Chen 2004; Haslanger 1995; Bachimont  1996; etc.) have shown, numerous other types of complex semiotic forms are also used in science. Some are gestures, rituals, cultural technè, artefacture and so on. These types of semiotic forms may not immediately represent the object under study, but they often serve for scientific justification, approbation or recognition. Think of a Nobel prize. Or a publication in the journal Science, for instance. There is one particular type of semiotic form that, despite its absence in many models, is regularly used in all types of scientific theories. It includes words, sentences and discourses in natural language. As Craik (1943) says for natural language: ‘We do not try to prove the existence of the external world – we discover it, because the

 Models in Science and Semiotics 51

Figure 4.2  Chaotic attractor of the Rabinovich–Fabrikant system (Danca and Chen 2004).

fundamental power of words or other symbols to represent events . . . permits us to put forward hypotheses and test their truth by reference to experience’ (Craik’ 1943: 29; emphasis mine). Other semiotic forms are surely different from the ones of formal language. Still, they are important in scientific research. Their high flexibility and rich semiotic structures allow them to express the many and various features and dimensions of a complex phenomenon. Unfortunately, natural language forms are so transparent that most scientists do not see them as important in the construction of models. Still, they play an essential role in theories. This expressive role of models is, as one may guess, also quite present in semiotic enquiries. Though classical semiotic enquiries may sparingly use some logical formula and basic mathematical equations (e.g. statistical graphs and even mathematical visualizations to express some of their conceptualizations), they still do not mainly use formal languages, but rather natural languages. One must be more careful regarding this argument. Despite the fact that many academic disciplines do not define themselves as doing semiotics, they in fact often create theories that are about a semiotic artefact. And many even build meta-semiotic theories about such types of research. In these enquiries, they use languages that are increasingly formal, such as statistics, probabilities, topology or logic. And even if these formal models are not always directly related to the meaning components of the artefacts under study, they often reveal deeply important features of the meaning. Take for example the following disciplinary enquiries. Contemporary cognitive sciences (neuroscience, psychology, artificial intelligence, etc.) are not identified as semiotic disciplines. Nevertheless, they constantly use the semiotic ‘stands for’ concept, which is omnipresent for describing a specific type of relation between some entities. It is expressed in natural language in terms of representation, mental surrogates, proxies, substitutes, and so on. And these terms

52

Computational Semiotics

are again used for explaining some specific cognitive operations that are related to the meaning components of semiotic artefacts or of semiosis, such as categorization, emotion, decision, memory, language and communication. Finally, contrary to classical semiotic practices, these semiotic cognitive relations have received a wide variety of mathematical and logical formalizations. In fact, such a practice is present in many human and social sciences which study aesthetic, cultural, religious or educational artefacts and semiosis. Linguistics is probably the discipline whose research object is the most overtly semiotic, as it studies artefacts which are words, sentences and discourses, while intensively using formal and categorial, logical, generative and computational types of formalization. Whether or not these disciplines explicitly recognize that they study semiotic artefacts and processes, the models they construct, even though they may not use semiotic vocabularies and sentences, still contain typical semiotic concepts and statements. But by the same token, they show that these concepts and statements are not immune to formalization. A third role of models pertains to communication. In science, a model may be created, explored, developed and expressed in some language by some individual researcher, but sooner or later it will be shared by a whole epistemic community. Today, we no longer have scientific hermits doing science in isolated castles or inaccessible caverns. Theories are ultimately socially shared knowledge. And it is the content of a theory, that is, the statements, beliefs, desires, intentions, decisions, goals, hypotheses, proofs, conclusions, techniques and strategies that are shared, exchanged, accepted and agreed upon (Kusch 2002), aggregated (List and Pettit 2002) and socially justified (Rorty 1979). They form paradigms (Kuhn 1993) and institutional research programmes (Kitcher  1993). Collaboration and competition are inherent to scientific practices. One could not imagine that contemporary science would exist and be as vivid as it is without its scientific journals, conferences and so on. Today, the internet and various types of multimedia just enhance this communicative role in science. And models are among the most important means of such social interaction and communication. This communicative role is without a doubt also present in semiotics just as in other scientific practices; the semiotic community must exchange beliefs, research and knowledge on semiotic artefacts and semiotic processes. And this community is not without all the classical scientific institutions and organizations: journals, conferences, media and so on. But there is still a special dimension of semiotics that we may highlight: its communication role in various sciences. Not only does semiotics as such involve communication in its own practice, it finds its way into the means of communication used in other sciences. Indeed, many scientists call upon an increasing variety of semiotic means to communicate among themselves. But they are not always aware of the complexity in terms of meaning of the semiotic tools they use. Not every scientist masters the complexity of the syntax, semantics and pragmatics of the semiotic forms they choose for communicating their research projects and their results. For instance, they do not always understand the complexity of the lexical, narrative, stylistic and argumentative dimensions of their non-formal languages. Neither do they foresee all the pragmatic consequences of their scientific discourse. Hence, because of the complexity of the phenomena, scientific and semiotic theories will usually build a multiplicity of models. Every model, whatever its type,

 Models in Science and Semiotics 53 participates in its own way to the explanation and understanding of the object under study. But there is a moment when the model has to be expressed in a semiotic form, be it a formal, a natural or an iconic language. Finally, it has to be communicated among researchers and, ultimately, to the general public. According to this pragmatic view, science builds models that intersect, complement each other, transform, stabilize, weaken and eventually die. Our working hypothesis throughout this research is therefore to unwrap the encounter of semiotics with computation through a scientific endeavour that aims at implementing complex epistemic models that contribute individually and collectively to semiotic knowledge.

Types of models in science and semiotics Scientific theories are built out of a complex interplay of models. Each model contributes, individually and collectively, to the emergence of knowledge. Many philosophers of science6 have shown that, in various scientific practices, just one model, however important it may be, is not enough. There is a need for several others. To paraphrase Minsky’s7 theory of mind, we would say that a theory is a society of models. Not only are models numerous, but, because they offer a variety of points of view, they are also multiple in types. As Baetu aptly put it: ‘Models anchor the diverse pieces of the mosaic of knowledge to a description of a phenomenon, on the one side, and to the methods and tools, experimental or theoretical, used to obtain each piece of the mosaic, on the other’ (Baetu 2014: 2). Although we see the relevance of models for constructing scientific theories, there still does not exist a recognized taxonomy or classification of types of models. For the moment, the rare classifications vary across disciplines, projects and paradigms. In their article on the history of the concept of model, Armatte and Dahan-Dalmédico made the following list: ‘Typical ideals include structural models, conceptual models, operational models, fiction models, black box models, and simulation models, without this list claiming completeness. We could also add forecast, decision and learning models’ (Armatte and Dahan-Dalmédico 2004: 267, our translation). This list is both very limited and quite diverse. And if we look at the technical literature on models, we discover a variety of classifications based on criteria related to research objects (e.g. weather, life, cells, cognition), functions (e.g. illustration, visualization, demonstration, communication), formalisms (e.g. mathematical, logical, statistical models) or to the method used (e.g. experimentation, projection, simulation). But despite this range of possible models, there are some macro-classifications8 of models or some sets of general meta-models that are helpful in organizing some structures among the myriad of available models. One dominant classification among these gives priority to some specific type of models. For example, in philosophy of science, the most important types of models are the formal ones. And various formal languages are used: algebra, topology, geometry and so on. And only a rare few will respond to the epistemological canons of logiconomological types. For classical philosophy of science, these formal types of models are used not only for making predictions but also for reasoning in a rule-governed

54

Computational Semiotics

way – that is, for making inferences or even for providing explanations and ultimately for promoting understanding. More recently, some macro-classifications were proposed for bringing together some models that share some common features. This allows a better understanding of their interactions. For instance, Giere (1999) proposes the following three main classes of models: physical, visual and theoretical models. Weisberg (2013) has proposed a diagram containing three general types: concrete models that use physical structures, mathematical models that call for mathematical structures and computational models that describe effective computational procedures. One of the original points of Weisberg’s proposition is to focus on criteria of similarity or correspondence between models, criteria that relate the procedures to the relevance and importance of the properties of the phenomenon studied. Of interest to our present research, there exists a highly pertinent macroclassification for understanding the relation of computers with semiotics. It is found in cognitive science and artificial intelligence where computer science is indeed put into relation with symbols. It was proposed by Marr (1982), with variants by Pylyshyn (1984) and Dennett (1978), and it aimed at defining a computational theory of cognition in terms of three levels, stances or epistemic postures. In a cognitive type of enquiry, there should be explanations given at (a) a representational or intentional level, (b) at a functional level and (c) at a material or physical level. Recently, Weisberg (2013), in his taxonomy, has added a class of computational models. In this particular set of meta-classifications, the representational, epistemic or intentional stances contain the models that identify various dimensions of the cognitive activities under study, and the functional stances are seen as various types of formal mathematical or algorithmic models applied to the problem. The physical stance is understood as the physical implementation of the formal models – for example, a computer or some other physical simulation. However, these macro-classifications do not always have very clear criteria for identifying or distinguishing these models. When these stances are to be applied to human sciences, we may expect that a few more stances will surely be required. In our present research and inspired by our own scientific practices in the digital humanities and in computer science, we will modify this triplet (representational, computational and physical) and transform it into a quadruplet that will contain four types of classes of models. We see these classes as more inclusive and applicable to many other sciences. We believe them to be more accurate, more related to one another and more heuristic for our research domain. We believe that any research project that calls upon computer technology will allow the building of a relevant theory if it contains at least the four following different types of models, namely conceptual, formal, computational and physical models. (1) Conceptual models have three main functions: (a) they identify the properties (features) and relationships of the semiotic artefacts and processes under study according to the various specific points of view taken in the research, (b) they ultimately serve as an epistemic horizon inasmuch as reasoning on them produces new knowledge and connects them to the other types of models used in the enquiry, and finally (c) they are called upon for embedding the understanding and interpretation of the meaning of the whole endeavour of enquiry.

 Models in Science and Semiotics 55 (2) Formal models identify in the research object the means of encoding in some unambiguous symbolic system some of the features, properties and relationships defined in the conceptual models and upon which highly controlled operations can be applied. In other words, a formal model is embedded in a formal language for which things such as mathematics, logic, grammar and graphs are prototypes. And it has its own epistemic role for categorizing the element it ‘stands for’ and on which it reasons. Some of the relationships explored in these languages will be more interesting than others, such as, for example, functional dependencies. (3) Computational models retain the entities and features of the entities identified by the formal model but translate the functional dependency relations that are recursive into computable functions, that is, into algorithmic processes. (4) Physical models build a hardware architecture (e.g. electronic, mechanical) that can compute the computational models constructed. Although some of these models are more important or more valued than others, none are hegemonic. Often, only a few will respond to the epistemological canons of logico-nomological types. In other words, a scientific project using computers will implement a complex epistemic and dynamic set of models where each one contributes individually and collectively to the knowledge of the object under study. In such a perspective, each particular organization and structuring of these models determines the signature of a particular science. We can unfold each of these models into several sub-models, some being finer and more precise than others. They would be complementary – that is, none would be in opposition to the others while not being substitutable with them. None, therefore, would be the only model in the process since all express different points of view on the research objects. If we accept this perspective, the challenge for our research here is then to better describe, explain and understand the various types of models built by semiotics for the analysis of semiotic artefacts and processes of semiosis. More specifically: where, when and how semiotics can encounter formal and computational models that are enriching and not destructive of the signature of each of these models. We can briefly illustrate the interrelations of these four models and their possible sub-models with the help of Figure 4.3. Afterwards are a few brief prototypical and pedagogical examples that illustrate the nature and structural relations of our proposed four main models before exploring their descriptions and explanations in more detail in the following chapters. A first example comes from climatology. Here, the phenomena to be studied are often complex systems. The use of these four models is essential in their study. Take for example the scientific analysis of a hurricane over the ocean. Such a study requires defining a conceptual model that expresses in natural language some of the elements, operations and dynamics of a hurricane over an ocean. A formal model will retain only a few dimensions presented in the conceptual model and apply mathematical statements to them. If the mathematical structures retained are recursive, a computational model will transform them into algorithms. And a computer model will be designed so that its architecture and power may allow effective computation to be carried out. Finally, the results of the computation will be interpreted in regard to the original conceptual model (Figure 4.4).

56

Computational Semiotics

Figure 4.3  Relations between the four models and their sub-models.

Figure 4.4  Models for a hurricane.

 Models in Science and Semiotics 57 A second example comes from  cognitive neuroscience. Here, the phenomena to be studied are more complex. Often,  what is to be observed is highly dependent on one’s own personal and even cultural perceptions. Take for example the case of a phenomenological illusion such as the Kanizsa illusion. Here, what has to be explained is how the brain processes not the perceived physical object (the three angles), but the illusion of a triangle (Figure 4.5).

Figure 4.5  Models for the Kanizsa triangle.

In studying this Kanizsa illusion, Petitot (2009) had to build a conceptual model where the specific phenomenological data were identified and retained. This was followed by a complex formal model that uses topology, geometry and algebraic dynamic systems. These formal descriptions, in turn, informed the design of algorithms built into a computational model and processed using a computer. But in this research, the computer was often seen as a simulation of the processes of the brain, which is also a physical, non-electronic ‘natural technology’. The results were ultimately interpreted and understood  through a return to the initial conceptual model, that is, what is phenomenologically observed: the illusion of a triangle. A similar set of models can be applied to semiotic enquiries. Here is an example from the computer-assisted semiotic analysis of Magritte’s paintings (Chartrand et al. 2013; Chartier et al. 2019) (Figure 4.6). In the conceptual model, many pertinent features of each painting xi were categorized and reasoned upon through some semiotic framework. A formal model defined a possible functional dependency among these features; a computational model translated the formal model into a computational model (algorithms), which in turn was implemented in a computer to be effectively computed. The result of the hypothesis process allowed a refinement of each model until optimization was attained.

58

Computational Semiotics

Figure 4.6  Models for the semiotic analysis of Magritte’s paintings.

In this model, the traditional contribution of semiotics was found in the conceptual model. But the other models were all dependent on such work.

Conclusion Science is a complex collection of cognitive processes, and these pertain namely to perception, conceptualization, categorization, discovery, reasoning, explanation, justification and action. And many models must be called upon in order to address the processes. Such models are mediators in the emergence of a theory whose aim is to express the knowledge acquired from the reality under study. We have argued that a scientific enquiry that calls upon computer processing must include at least four different types of models that interact constantly: a conceptual model conceptualizes the features or properties of the object under study; a formal model applies some mathematical structures to some aspects of the object; a computational model translates some of these mathematical structures into algorithms; and finally, a physical model proposes a material architecture for implementing the computation in an effective way. Our intuition in the present research is that semiotics is also a pluralistic modelling practice. Faced with the complexity of certain artefacts and semiotic processes, an adequate computational approach to semiotics must also build a multiplicity of models. Depending on its pace of evolution and development, some models will be preferred while others will be cast aside. But no computational semiotic research can think of having only one type of model and of restricting itself to it. Serious semiotic research calling upon computers requires, like any other science, different points of view, angles of approach, expertise, postures and so on. Each type of semiotic project will contain its own sets of models.

 Models in Science and Semiotics 59 In accordance with such a perspective, through a custom and specific configuration of its models, each science builds a clear individual signature. Semiotic projects interested  in a digital approach to their own objects cannot escape these modelling activities. This view, we believe, is a heuristic approach to the problem of the encounter of semiotics and computation. If this meeting hopes to build practices that present a scientific status specific to complex semiotic artefacts, it will be forced to construct theories that will also be based on models that are unique to it. It will then not be the disciplinary status, even less the professional status,  that dictates the value of knowledge, but the quality and types of models that are built into the epistemic process. In the following chapters, each of the four models chosen will have their role in science and in semiotic projects analysed.

60

5

Conceptual models in science

If a scientific project is to be applied to complex phenomena and uses computer technologies in its investigation, then one important type of model that must be built is a conceptual model. Science is a complex cognitive endeavour that aims to explain and understand reality in order to help us act and live within its boundaries. On a heavily rainy day, a person may report that he or she feels a slight drop in atmospheric pressure, smells humidity, hears strong winds and feels resistance. Reporting such experiences may be part of our daily lives. But it is not the main purpose of scientific knowledge to report such simple and basic encounters with reality. It aims at more complex knowledge. It emerges when a certain phenomenon must be described, but above all when it must be explained by some theoretical discourse. And in building theories, science builds models. Although the prototypical scientific models are formal ones, not all of them are of this type. ‘One must recognize the internal pluralism of theories as including nonformal components. Some of these are used to represent organizational and compositional relations of complex systems’ (Winther 2016). One such non-formal descriptive and explanatory model has a specific dynamic: long before a formal model can be used, a scientific enquiry will have to build a comprehensive conceptualization of the reality or the problem to be studied and it has to determine how it will be investigated. In other words, a scientific theory must conceptualize many of the various dimensions both of the object under study and of the methods of studying it. In doing so, it builds a conceptual model. Take, for example, the emergence of a new phenomenon, such as the development of a new type of infectious disease within a social group. Research on such an occurrence cannot begin with the immediate production or even application of a formal model. Researchers will first aim at identifying and conceptualizing as many features or properties of the disease as they can.1 Failing this, they will be unable to truly specify what the object under study is. And it is only after this first preliminary enquiry has been accomplished that they may start offering some possible means to study the object. A first presentation of these features and properties will be made through a myriad of statements. Most of these will be assertions – some basic, others more technical. Some will also be interrogations, orders and so on. For instance, describing the symptoms of Covid-19 may be done using some of the following affirmative statements: some symptoms are coughing, having a fever, fatigue and lack of appetite. The symptoms disappear after some fifteen days. Most deaths occur among the elderly. Younger people appear less likely to develop severe symptoms. The disease is contagious. Interrogative

62

Computational Semiotics

statements could be: Is it a virus or bacteria? What are the causes of the disease? Can it be treated by medication? Is there a vaccine? Imperative statements could be: Everybody must be careful! Wear a face mask! Self-isolate and quarantine! After these basic observation statements have been made, some technical ones will be produced. They may present more specific, precise and characteristic features or properties which may involve a recourse to some instrument. Such statements may pertain to the person’s degree of fever, blood type, age, medical history and so on. Another set of more general statements pertaining to the ways by which to study the phenomenon will normally follow. They may be about the hypotheses, the observation’s instrument, the experimental schema, the parameters, the formal and computational models to be chosen or built, the evaluation strategies and so on. Following this, choosing the means by which to study the phenomenon will be required. Finally, after having done the research and obtained some results, a new set of statements will be produced. Some may be very technical, such as the disease is not caused by bacteria, it is caused by a virus, and they can develop into a full-fledged biological and medical theory – for example, regarding the biochemical processes involved in the type of disease under study. Still, a simple statement such as the one above may not be enough for understanding the research results, so more statements interpreting them will be produced, such as the disease is caused by a new strain of coronavirus. As one can see, few of these statements are formal. They are assertive, interrogative or imperative statements expressed in natural language and produced at the beginning and throughout the research itself, forming a discourse about various dimensions of the phenomenon under study. They pertain to the research object, method and results. They have various epistemic statuses. Some are technical, but others belong to common sense. They show that a scientific enquiry not only includes formal statements, but also many non-formal ones. They are essential in the conceptualization of various aspects of the research object, its enquiry method and, ultimately, in the interpretation of the research results. These statements constitute the core of a conceptual model. And the semantics of the sentences they use belong to some implicit convention shared among the members of an epistemic community.

Conceptual models: Definitions A conceptual model is thus a set of statements, usually presented in a natural language, expressing an internal mental modelling2 process that scientists construct and by which they understand the various components of their research topics. In the words of Johnson-Laird: ‘if you understand inflation, a mathematical proof, the way a computer works, DNA, or a divorce, then you have a mental representation that serves as a model of an entity in much the same way as, say, a clock functions as a model of the earth’s rotation’ (1983: 2). An effective conceptual model is one through which the mental conceptualization of a theory, research object, methodology, evaluation criterion and, ultimately, an interpretation of the results will be expressed.

 Conceptual Models in Science 63 Nersessian (1992) adds that these mental models are intermediate levels of analysis between the phenomenon and some final mathematical model. She explicitly mentions that when scientists communicate their findings, they present them not only by means of mathematical formula but also by way of the conceptual models they have created for understanding the formal model about the reality to be explained. Conceptual models are found in most scientific practices, both in soft and hard sciences. They sometimes go by different but related names. For instance, in ethnology and sociology (Glaser and Strauss 1967; Guizzardi 2005; Jabareen 2009), the various types of conceptual models that are built are called conceptual frameworks: ‘I define conceptual framework as a network, or “a plane”, of interlinked concepts that together provide a comprehensive understanding of a phenomenon or phenomena’ (Jabareen 2009: 51). In philosophy, despite some important semantic differences, conceptual models are designated by means of a variety of terms: belief systems,3 beliefs and expectations,4 implicit knowledge,5 implicit theoretical knowledge or theory-laden knowledge,6 conceptual systems,7 naïve psychology,8 or life world (lebenswelt) or even naïve sciences.9 But surprisingly, information science10 and computer science11 are scientific fields that have explicitly recognized the necessity of conceptual models for building computer applications. In computer science, this has opened a core subdomain called computer engineering. AI rapidly saw that no formal or computational model was possible without a well-defined conceptual model. And it was given the main task of specifying the conceptualization of the problem domain: ‘A conceptual model [is hence] defined as the result of the processes leading from the task to the specification of the conceptualization of the ontological structure of the problem domain’ (Robinson et al. 2015: 2821). Such a definition invited the transformation of a natural language conceptual model into a more controlled and formal conceptual model. Such conceptual modelling became a full sub-speciality of classical artificial intelligence: ‘Conceptual modeling is the activity of formally describing some aspects of the physical and social world around us for the purposes of understanding and communication’ (Mylopoulos et al. 1992: 51, emphasis removed). Depending on its domain of application in computer science, a conceptual model may abandon its natural language expression and be given a computational formalization and some important theoretical foundations. It may go by a variety of names: knowledge level,12 conceptual frame,13 conceptual schema,14 knowledge representation,15 conceptual space,16 ontology and so on. ‘Knowledge should be expressed in an internal form that facilitates its use, corresponding to the requirement of the task to be handled and mirroring the external world (in some way)’ (Sun 2001: 784). This has also opened up specific computer knowledge-based systems and expert systems. Today, these knowledge bases have become an important research field of machine learning17 (Michalski 1981; Mitchell 1997; Goodfellow, Bengio and Courville 2016). An increasing number of these conceptual models have taken a surprising form. They tend to mainly include written statements that have been deposited in the digital world of elibraries, ebooks, emedia, collaborative online encyclopedias, if not the whole internet and whatever form social media takes. And all of these were hoped to be processed by some natural-language understanding computer processes. It is

64

Computational Semiotics

the vision that underlay the Bert project.18 One does not need an autonomous formal conceptual representation such as an ontology: it could be learned on demand. Still, these formal computational knowledge representations do not erase the role of natural language conceptual models in scientific enquiries. The knowledgerepresentation (KR) format is now a component of computational models whose role is to formalize some of the knowledge that a conceptual model generates: A body of formally represented knowledge is based on a conceptualization – an abstract view of the world that we wish to represent. (Genesereth and Nilsson 1987, cited in Staab and Studer 2010: 3) The conceptual model is a concise and precise consolidation of all goal-relevant structural and behavioral features of the SUI [subject under investigation] presented in a predefined format. (Arbez et al. 2015: 2814)

But because of its formalization, it now becomes more of a component of a computational model than of a conceptual natural language model. Despite differences in views, these definitions of conceptual models, non-formal or formal, all support the thesis that in a scientific enquiry, there exists an explicit or implicit conceptual model with a specific signature. And it is even more necessary if there is a formal and a computational model to be built somewhere along the research trajectory. Conceptual models are the epistemic means by which scientists build structured knowledge about reality: ‘we know about nature through the models we build of it, constructing them by abstraction from our experience, manipulating them physically or conceptually, and testing their implications back against nature’ (Mahoney 2010). Nature does not live inside of scientific models. Somewhere or somehow, in and along the process, scientists cannot just sense, perceive or observe reality; they have to conceptualize it in some way: ‘Science is not about nature, but about how we represent nature to ourselves’ (Mahoney  2010) A conceptual model is hence an important epistemic component of science.

Roles of conceptual models Science cannot contain only a set of formal models. It must also include conceptual models. And as with all other scientific models, these models have important roles in the acquisition of knowledge. They construct their own type of conceptualization for describing, explaining and ultimately understanding the research problem, the methods and the research results. For example, if one is presented Newton’s first law of mechanics only through the equation F=ma, the understanding of the problem the equation is meant to explain will be weak and shallow. One needs much more than this equation to understand it. In a scientific enquiry, explicit or implicit conceptual models are created precisely to enable a type of human intelligibility of reality that is most suited to high-level cognitive capacities: conceptualization. In this sense, conceptual models cannot be eliminated

 Conceptual Models in Science 65 from scientific enquiry. And because conceptual models are not formal models, they have important specific epistemic, expressive and communicative roles.

The epistemic roles Just as with other types of models, the epistemic roles of conceptual models are quite specific. For our current purpose, we focus here on the surrogate or representational role of conceptual models. This implies that to overtly represent something, two main epistemic operations are required: a categorizing operation and a reasoning one.

Categorization In a scientific conceptual model, categorization is mainly an operation of cognitive conceptualization. It presents some characteristic sub-operations that give the model its own signature. Three of them are important: classification, ordering and selection. It is mainly through these operations that it interacts with the formal and computational model. In mathematics and logic, classification is a formal operation that determines the conditions under which a number of elements are related to one another so as to form a set or a class.19 In cognitive science, classification is the means by which categorization is realized in cognitive agents. As Harnad formulates it: ‘to cognize is to categorize: cognition is categorization’ (Harnad 2017: 21–54). Classification hence determines the features and properties that entities have in order to form a class or a category. One must not confuse the category as a class with a category stemming from the use of a label. Two identical classes can be formed by different cognitive means. For example, in biology there are specific criteria for determining the class of entities that tomatoes belong to. Still given these criteria, cognitive agents may have different means to classify them. Children may not proceed the same ways as adults. And in different languages there may exist other words by which to label tomatoes. In English, they are labelled ‘tomatoes’, and in French, they are labelled ‘tomates’. And in a same language there may exist some synonyms, such as ‘wolf ’s peach’ in English, or alternate terms in derivative languages such as ‘pomme d’amour’ in Mauritian creole. In some database language, tomatoes may be labelled by a number: T3401. These distinct notions are important in a conceptual model. And although they are related to one another themselves, they must not be confused. Each one initiates different types of enquiries. For instance, finding the right biological criteria for tomatoes is a different enquiry than that of understanding how humans and animals identify tomatoes, or why grocery stores put tomatoes in the vegetable section even though they are, in the terms of the biological sciences, fruits. Hence, in a conceptual model, the classifying operations of categorization abstract properties and features from the entities under study: ‘The conceptual model describes how we have abstracted the model away from our understanding of the real world’ (Robinson et al. 2015: 2817). In simpler terms, they offer the conceptual model a way to describe the problem: ‘A conceptual model therefore captures the results of the

66

Computational Semiotics

modeling process regarding the concepts and their relations that describe the problem, which have to include assumptions and constraints that shape the process and the results’ (Robinson et al. 2015: 2820). In a conceptual model, categorization may often require more than classification, as some ordering among the classes may be useful. In some cases, if classes become too numerous or do not all have the same degree of precision or abstraction, some structuring will be necessary. For instance, a taxonomy and hierarchy may be imposed upon them so to as simplify their usage. Such orderings often play an important role in a scientific discipline. For example, in botany, after having categorized many entities and having classified them as pine trees, the resulting class may be integrated into the more general category of Pinus, which itself will be classified into Pinaceae and afterwards into Coniferae, Pteropsida, Tracheophyta and so on. It may also happen that some sets of features, although they may share some commonalities, will be unable to find an inclusive classifying category. In such a case, it will not be surprising to call upon other means which idealize or even fictionalize the categories: they are seen as ‘tinker toys’ (Cushing 1987) or as a ‘simplified and . . . fictional or idealized representation’ (McCarty 2010: 255). Finally, after the operations of classification and ordering, a selection operation will be applied to the whole set of categories and classes. A conceptual model may generate many of them, most of them referring to many types of features and properties, but, if research must eventually build a formal model (mathematical, computational, etc.), not all the identified features will be relevant for each formal model to be built. A selection will be necessary, because not all features can become the argument of a mathematical function or the input of an algorithm: ‘One model never addresses all relevant features of the target phenomena’ (Green 2013: 172). This means that among all of the classes contained within a conceptual model, only a few will become associated to symbols and to the formula of a formal or computational model. Such a selection has a direct epistemological effect. Each formal and computational model produces an epistemic reduction of the problem under study. Usually, a particular formal statement in a formal language or in algorithms manipulates only the specific set of symbols that are allowed by the mathematical or logical structure retained by the formal models. For instance, in Newton’s formal law of mechanics, the equation F=ma can only manipulate the symbols F, m and a, which are about the categories of force, mass and acceleration. It cannot be applied to other symbols. In other words, each formal and computational model will always require a reductive selection of the categories that a conceptual model may have included. This reductive selection is a constraint that is inherent to a specific scientific enquiry – more so if it is to be linked to a computational model and ultimately to computer processing. Such reductive selection is a sort of filtering process imposed by the use of formal models. From an opposite point of view, this also means that if a formal model, even very rigorous, were to include symbols that have no link with the conceptual model, then the explanation provided would be external to the conceptual model. This, in turn, would probably diminish the understanding of the problem being studied. Let us conclude briefly with one important epistemological consequence of this reductive selection. Because of the preceding arguments, a conceptual model is always

 Conceptual Models in Science 67 embedded in assumptions and beliefs: a conceptual model does not rest solely upon objective conceptualizations. It is therefore laden with implicit assumptions, principles, beliefs, uncertainties – if not biases. Despite their limits, these categories are important in our explanation and understanding of the world. ‘Assumptions are made either when there are uncertainties or beliefs about the real world being modeled’ (Robinson et al. 2015: 2817). And as Feyerabend has often stressed, it is not surprising when a theory which is constituted of many models itself becomes theory-laden. As we may see, in a conceptual model, the categorization operation, within whatever sub-operations it may incorporate, plays an important epistemic role in science. It determines the properties and features of the entities under study, involves their classification and their organization, and ultimately helps select the ones that a particular formal and computational model will be attentive to. In other words, it determines types of interactions a scientific research may have with formal and computational models. The sharper the categorization processes, the more precise these formal and computational models will be.

Reasoning The second epistemic role of a conceptual model is that of reasoning. Just as the notion of categorization is defined differently in many sciences, so is the reasoning process. To put it briefly, two main theoretical points of view regarding conceptual models may be retained. In classical logic theory,20 reasoning is a set of formal operations by which the symbols and formula of a symbolic system are manipulated through the use of some transformation or inferential rules. In psychology,21 reasoning allows the cognitive agent to manipulate some knowledge statements (i.e. that what has been observed, perceived, felt, conceptualized or mentally represented) in such a way that either some knowledge statements may be discarded or that some new knowledge may be discovered. In both cases, the reasoning may be carried out by deductive, inductive and abductive means. In conceptual models, these reasoning processes are not as rule-governed as they are in formal and computational models. In the latter, reasoning is not only rulegoverned, but also in some cases axiomatic. And in computational models, reasoning must also be a computable process. This is quite a constraint. In fact in the conceptual model the reasoning process will be mainly presented in a discursive manner. But, often due to a problem’s complexity and the communication requirements of various epistemic communities, flawlessly rigorous reasoning becomes impossible in a conceptual model. So, when this is the case, as many model theorists have shown, scientific reasoning may call upon a host of subsidiary processes such as analogies,22 metonymies and metaphors,23 fictions24 and narrations.25 And more and more research is emerging on the epistemic conceptual role of iconic and graphical types of conceptualization.26 Furthermore, scientific conceptual models are laden with such reasoning processes. Our point here is that these types of reasoning processes specifically belong to conceptual models. They are not acceptable in formal or computational models. Metaphors are properties of a natural language discourse – not of formal language. And this is precisely

68

Computational Semiotics

because their role is not simply of imparting truth, but understanding. It is therefore not surprising that reasoning may take various non-orthodox forms in conceptual models. These two types of epistemic operations, categorizing and reasoning, show not only the complexity but also the importance of conceptual models. A scientific theory cannot be made up solely of formal models since it exists only by being related to some of the properties and relationships that the conceptual model has categorized and selected. In other words, in scientific research, a conceptual model is a forum where the various semantic components of a formal model are expressed (explicitly or implicitly). Conceptual models do not just ground the immediate meaning of the symbols of formal and computational models but also provide the general context for this meaning (sense and reference), thus opening the door to an integrative understanding of what a scientific endeavour is about. As Duhem (1902, quoted in Ariew 2007) puts it: ‘If a physicist is given only an equation, he is not taught anything. . . . In physics, an equation, detached from the theory that leads to it, has no meaning.’  To complete this quote from Duhem, we would add that the conceptual model is the instance where the formal models ‘make sense’.

Expressive role of conceptual models in science The second main role of a conceptual model is an expressive one. A conceptual model may exist as a mental model within the individual mind of a scientist. It may be implicit and even not be expressed in some explicit way. But in actual scientific practices, a conceptual model will not endure very long if it lingers in the individual mind of a sole researcher. It must rapidly be expressed in one available mode of semiotic expression or another. And there are many modes by which conceptual models can be expressed. But the most widely used and preferred one is through natural language. ‘The term modeling can be used to describe simple prose accounts of the essence of something or situation, or even for metaphors like the desktop metaphor of many graphical user interfaces’ (Sperberg-McQueen 2018: 287). Due to the complexity of the conceptualization involved in scientific practices, a scientific theory is not restricted to building strict formal models. ‘On a broader understanding, the restriction to formal languages may be dropped, so as to include scientific languages (which are often closer to natural language than to logic), or even natural languages’ (Gelfert 2017: 12). In other words, science is not just a set of equations and computer program statements. It also expresses models in words and sentences. Natural language is indeed the dominant mode of expression for conceptual models because of its rich syntactic, semantic and pragmatic structures. The specific semiotic structure of natural language allows the fine-grained expression of complex categories. Through words, sentences, whole texts and discourses, it can create and manipulate a variety of types of statements (affirmative and predicative interrogative). And it can also use many linguistic rhetorical strategies through which conceptual models are constructed and used to talk in a more comprehensive way

 Conceptual Models in Science 69 about the object under study and the whole context in which it exists. This confers the conceptual model its unique signature. And it is through these strategies that the model allows the description, explanation and understanding of the research’s object, methodology and results. We will illustrate this expressive role of conceptual models using a few examples. Our first one comes from a scientific article in biology (Rebecchi 2006). It presents research on Echiniscus trisetosus. Most of the scientific results are presented in a natural language text whose topic is about a ‘long-term anhydrobiotic survival of lichendwelling tardigrades’ (Rebecchi 2006: 23). The research reveals that ‘anhydrobiotic tardigrades can survive for more than a century’ (ibid.). And ‘the longer time they spend in anhydrobiosis, the more the recovery time to reach active life is increased’ (ibid., 27). There are naturally many more such sentences in this article. All of them are part of the conceptual model of this research. But in it, there is one important category that is used in the model: it is the categorial word ‘life’. It designates one important category among many others. It is presupposed that the meaning underlying it is known and understood by the readers and the researchers. But as the reader may guess, this categorial word raises immediate questions: What are the meanings of the word ‘life’? And which one is used here? The authors do not give us much to ground our understanding of the meaning of the ‘life’ category when seen as a possible feature of Echiniscus trisetosus. It does not seem to have the classical Aristotelian meaning: ‘by life we mean self-nutrition and growth (with its correlative decay)’ (Aristotle . . . De Anima II, 1, 384-322). It rather seems to belong to either a popular, tacit concept, or to some shared cultural doxa about life or even to a possibly unknown scientific concept. Hence, we have in this conceptual model a very vague concept. Still, it is important, for it plays an epistemic role. It is part of the overall discourse that ultimately gives a context for understanding the meaning of the technical statement made in the conceptual model for describing, explaining and understanding the whole research. A second example comes from cognitive science, and it is to be found in M. Graziano’s introduction to his neuropsychology book Consciousness and the Social Brain. Here is an excerpt from the introduction: ‘The brain is composed of neurons that pass information among each other. Information is more efficiently linked from one neuron to another, and more efficiently maintained over short periods of time, if the electrical signals of neurons oscillate in synchrony. Therefore, consciousness might be causes by electrical activity of many neurons oscillating together’ (Graziano 2013: 6). Here, we are not reading equations or seeing some physical instance or replica of neurons. We are reading sentences that express conceptual categories specific to a conceptual framework built out of past and contemporary cognitive science and stemming from other theoretical influences, such as concepts from neuroscience, information theory, control theories, philosophical theories of causation, consciousness and attention. They participate in the conceptual models of Graziano’s theory of consciousness and attention. All these sentences were used for determining Graziano’s research object, the methodology, and ultimately the interpretation and understanding of the research results. In fact, they take up some 80 per cent of the book. They are

70

Computational Semiotics

also essential to the understanding of formal models and programmes presented in his theory of ‘consciousness’. And it is through such sentences that the author describes and explains it to us and bestows intelligibility by linking it to our general knowledge. Such a type of conceptual modelling is omnipresent in all sciences. This expressive role is hence essential for conceptual models. And choosing a natural language for expressing oneself may be very useful in scientific practices. But it is not without consequences for scientific theories. Indeed, natural language does not have the structural and semiotic rigour of formal languages. This is often invoked as an argument against its use in science. For instance, at the level of vocabulary and syntax, neologisms will proliferate, while many words may carry an abyss of meanings (e.g. consciousness) and obscure several hidden problems. And not all sentences will be strictly controlled in their grammatical structure. At the level of semantics, ambiguity will be omnipresent and formal rules of inference will not always be rigorously followed. Generalizations may be fuzzy. Modal attitudes will not always be overt and explicit. Finally, the discourse may, for instance, be unstable and unyielding to the evolution of the meaning of the main concepts. And at the level of pragmatics, the use conditions for conventions and presuppositions, as well as for rhetorical, narrative and argumentative structures, may not always be clear and explicit. An identical conceptual content may not be understood in the same manner depending on whether the model is presented in a scientific conference, a scientific disciplinary journal, or in a vulgarization report.

The communicative role of conceptual models in science The third role of conceptual models is a communicative one. Science is not just an individual cognitive endeavour. It is embedded in a social context where many actors and scientific institutions intervene. These take the form of laboratories, universities, grant organizations, government agencies, media and so on. Scientific books and articles do not just publish equations and programmes. We may guess how publication institutions feed off the communicative roles of scientific conceptual models. Social epistemology (Latour and Woolgar [1979] 1986; Haslanger 2006) increasingly shows that the whole social environment participates in various ways in the construction of science. This deeply affects the epistemic roles of conceptual models. Not only are they actors with respect to the knowledge built in actual scientific practices, but they also take on the role of belief systems for future science.

Critiques of conceptual models Despite their omnipresence in scientific practices, conceptual models are not widely accepted. One radical criticism of non-formal models such as conceptual models has lately been well summarized by Sperberg-McQueen (2018: 287): ‘Non-formality makes it disastrously difficult to achieve clear disagreement. For this reason, non-formal models often merit the dismissive response often attributed to the [scientists].’ Notwithstanding

 Conceptual Models in Science 71 all its difficulties and limitations, a natural language conceptual model is still a model in the technical sense of the word, and it will not disappear from scientific practices. Whatever the science one may use as an example, it will not escape this conceptual modelling in a natural language. One can hope to constrain this model by reducing it to a formal metalanguage (Tarski 1983), but no science, not even mathematics, has succeeded in completely eliminating one form or another of conceptual modelling expressed in natural language. Even a computer program must have its ‘documentation’; otherwise it will be incomprehensible to humans. Einstein himself ultimately translated some of his formal models into the theory of ‘relativity’. Gödel presented his theorem in terms of ‘incompleteness’. Skinner translated his ‘behavioural laws’ into ‘operant conditioning’. Lorenz spoke of his chaotic model in terms of ‘attractors’. And Newell and Simon explained their computer programs as making ‘artificial intelligence’. Rumelhart and McClelland translated their vector optimization models into ‘neural networks’ and even ‘minds’. Climate scientists, based on the stabilization states of multiple climatic factors in their formal model, always end up telling us things such as ‘we can anticipate that tomorrow there will be rain showers!’ And natural language remains, in the end, the main  semiotic form that can best express the complexity of the concepts and conceptual frameworks by which humans define and understand their research objects and ultimately their formalization. It is through natural language that a conceptual model will express, in a fine, nuanced and precise but also creative way the multiple dimensions of the research objects: entities and relationships, but also intentions, objectives, methods, controls, ethical norms, epistemic and social scope, etc. Ultimately, this is what makes it so rich. The natural language of the conceptual model is the ultimate anchor for understanding and communication. As Gilles-Gaston Granger says, formal modelling is only one horizon in the whole of a scientific approach: As a thought in practice, it [science] can only present itself as an attempt at shaping, commented on by means of non-formal language. A completely formal language never appears solely as a horizon of scientific thought. And it can be said that the collaboration of the two languages is a transcendental character, that is, one which is dependent on the very conditions of apprehension of an object. (Gaston Granger 1967: 44; our translation)27

Conclusion In most scientific practices, conceptual models present the various conceptualizations for studying a phenomenon. And it is through a shared natural language that such conceptualization, the various points of view on the phenomenon, and the research aim, method and results are expressed and communicated. Some of these conceptual models will be divided into multiple sub-models that will focus on one sub-territory or another among the space of the problem studied. Some critics, especially from naturalistic epistemological positions, will argue that this type of model expresses only superficial observations and that it boils down to

72

Computational Semiotics

superficial descriptions dependent on popular culture and subjective beliefs. These types of models are seen as ambiguous and intuitive, charged with intentions and affects, based on naïve observations and immediate phenomenological analyses. Their ‘empirical’ value is low and questionable. They could be better structured and controlled in their expression and should call for better observational tools. For these reasons, many scientists will forget them and most of the time they will not use them, if they do not outright reject them. Of course, one can dream of reducing whole scientific explanations to a formal model confirmed by experiments or simulations, but this would eliminate the essential epistemic mediation of understanding that is intrinsic to the conceptual model. A scientific theory that contains only equations and artefacts implementing them will be difficult to understand – for humans. Because of these problems, conceptual modelling should be used with caution, as it remains cognitively fragile. It is often only in relation to other models and to many other conceptual spaces that its statements end up participating constructively in a theory. Still, building such types of models will not always be an easy task. The world does not spontaneously jump into models! On the one hand, conceptual models depend on human cognitive abilities and on categorizing, conceptualizing and reasoning, but on the other hand, they must also be sensitive to the negotiations among members of an epistemic community. Models will contain beliefs whose origins are subjective and cultural. They will be theory-laden and have some social functions, among other things. This is precisely where hermeneutics and pragmatics have their hold. They also see science, as a theory, as a model-building process. But they highlight the interpretation task inherent to these models, and they situate them with respect to the researcher’s culture and epistemic community. So, for all their limitations in terms of objectivity and truthfulness, scientific theories cannot do without a conceptual model. Finally, as we shall see in the next chapter, we believe what semiotics does best concerns these conceptual models. Semiotics is a research programme whose specialty is explaining phenomena where some artefacts carry meaning. And if computation is to enter semiotics, this will necessarily call upon a consolidated form of conceptual semiotic model.

6

Conceptual models in semiotics

I propose to describe the characters of a sign. In the first place like any other thing it must have qualities which belong to it whether it be regarded as a sign or not. . . . Such characters of a sign I call its material quality. Peirce, C. S. (1873), On the nature of Signs, Ms 214.

Introduction Semiotics is not considered by most critics to be a strict scientific practice. It is mostly seen as a hermeneutic type of enquiry. At best, it is a soft science. Posner summarizes this position well: When asked what it is they study, semioticians of all cultures tend to respond by formulating dichotomies such as ‘signals and symbols’ or ‘signification and communication’. Their critics tend to reply by claiming that semiotics evidently has no unified subject matter and they draw the conclusion that semiotics may certainly be an interesting hermeneutic practice, but is not entitled to conceive of itself as a scientific discipline. (Posner 2014: 1)

It is true that the meaning content of semiotic artefacts and processes are not always immediately observable and that their discovery is less empirically controlled and rather difficult to reproduce. But this does not mean that a semiotic enquiry is a merely subjective and haphazard epistemic undertaking. We deeply share Sonnesson’s epistemological position regarding the scientific nature of semiotics: Semiotics itself comprehends many models, methods, and philosophical perspectives, and it is just one of the many enterprises which may be seen as occupying a space intermediate to the traditional sciences. Semiotics must be considered a science in its own right, defined by a particular point of view, rather than a domain of reality. (2008: 227)

To make a contrast with reductive visions of semiotics, the pragmatic vision of science is very heuristic. It shows that all scientific research builds theories that are themselves constructed through complex sets of interrelated models, with none of them being hegemonic or sufficient. Among these models, we always find conceptual models that are stepping stones towards classical formal models which in turn are a

74

Computational Semiotics

sine qua non-conditions for other more technical models such as computational and computer models. But sometimes, the reality to be studied may be so complex that the construction of a conceptual exploration of the problem may not itself immediately emerge as a model. Still, this does not mean that it is not present and implicitly at work. And if, eventually, computational semiotics is to be possible, it is important to see if semiotics can indeed build conceptual models and to see what kinds and what relations the conceptual models may have with formal, computational and computer models.

The role of conceptual models in semiotics Conceptual models are not unknown in semiotics. In fact, and probably because of the complexity of semiotic phenomena, they are a core type of model in semiotics. Additionally, their linguistic form and epistemic status do not help them to be recognized as models. From a hermeneutic point of view, the introduction of the notion of ‘model’ in semiotic analysis, even if it is conceptual, does seem the proper way to go. Indeed, a ‘model’ is a theoretical notion that usually belongs to empirical views of science. If it is to have some role in semiotics, it should only be a secondary one. We dare think the opposite. Even a superficial analysis of semiotic theories will show that conceptual models pervade their practices. They are omnipresent and play an essential role, one that is quite similar to the ones built in other sciences. Recall here the Peirce epigraph at the beginning of the chapter: I propose to describe the characters of a sign. In the first place like any other thing, it must have qualities which belong to it whether it be regarded as a sign or not. . . . Such characters of a sign I call its material quality. Peirce Ms 381.

If this is the case, then, technically speaking, in a semiotic inquiry conceptual models must contain statements that conceptualize various ‘characters’ qualities’, that is, properties and features of semiotic artefacts and processes. For semiotic conceptual models, there are, however, some important differences with respect to their role in classical sciences. Indeed what is most important are the features that are external to them and which participate in the emergence of their meaning – that is properties that reveal the ‘standing for’ relation. The discovery of such features becomes the core challenge of semiotic enquiries. This is even more true if the enquiry aims to integrate some computation. Take, for example, the semiotic content of a simple object such as a wedding ring. Its meaning is never in the box in which it is wrapped, and most of it is not even in the ring itself. It is the role of a semiotic conceptual model to conceptualize the various components of artefacts that are the means by which meaning is carried. And because of the complexity involved, the task will not be easy. It is precisely this problem that renders semiotics so difficult, but also so fascinating. Identifying what constitutes the semiotic content

 Conceptual Models in Semiotics 75 of artefacts cannot be done in the blink of an eye. Constructing semiotic conceptual models is often a long-haul project – and often a rich and profound adventure into the unknown. Here are two examples which nicely illustrate the comp­lexity of meaning. A first one is the many res­ earch projects concerning the ‘meaning’ of the famous mega­ Figure 6.1  Julien Viaud, alias Pierre Loti, Easter lithic Moai statues of Easter Island statues. From the drawings of Pierre Loti, alias Island (Figure 6.1).1 Julien Viaud: Julien Viaud, Moai, Rano Raraku, black For practically 200 years, pencil, 21 × 31.5 cm. Musée national de la Marine, these statues have been the Paris (Inv. 31 OA 138). object of numerous and variega­ ted types of enquiry: archaeological, sociological, cultural, geographical, physical, chemical, ethnological and so on. And despite the absence of a semiotic language, one of the ultimate main purposes of these projects has been to reveal ‘the mystery of the meaning of Easter Island’s Moai’ (Wolf 2019). For these giant sculptured stones are not just huge rocks, they are ‘statues’ precisely because they are taken as ‘symbols’ that have meaning. And because there is a myriad of understandings of what their salient features are (epistemic, linguistic, logical, functional, social, cultural, religious, etc.), it is not surprising to discover that there are many conceptual models aiming at describing them. We are just beginning to identify and categorize the various features and structures that these complex artefacts embody. Some of them are intrinsic to the artefacts themselves – for example, their forms, their iconicity, their physicality, etc. But the most important ones are external to them, such as the locus of their extraction from a faraway quarry and the fertility of the soil surrounding them. All of these features are components that contribute to generating the meaning of these artefacts. As a second example, we may recall the semiotic enquiry on the meaning of the Palaeolithic animal drawings and paintings found in the caverns of Southern France (Scardovelli 2018). Most of the important meaning components are not in the animal drawings themselves, but in their form, their location, their immediate environment or, more difficult to apprehend, in the traces of the artist’s work. In such types of complex semiotic projects, one cannot rapidly scan these artefacts and then throw an equation or a graph onto them in the hopes of giving an explanation and offering an understanding of their semiotic content. Before arriving at this ultimate result, there is a long conceptual road to travel so that an adequate but often fragile conceptual model may emerge and be expressed. And it is only then that the development of formal and computational models becomes possible. A similar process exists in the ‘hard’ sciences. No hard science is without conceptual models. Sometimes, they are embedded or hidden in some anterior conceptual

76

Computational Semiotics

models that have been constructed and passed down through the generations or among researchers. The conceptual models that are refined and actually retained may feed off of them and build them anew. These refurbished models may be so widely shared in some epistemic communities or be so familiar to the researchers that they become transparent and seem to not exist. A nice example is found in the debates surrounding the nature of climate change. Climatology is not a new science. And it has built through the years a huge number of conceptual models. Many of them are still in use but they have become transparent. We constantly explain storms, cyclones and hurricanes by means of conceptual models that are not explicit but are unconsciously integrated and which call upon pressure, temperature, humidity and so on. But now, an increasing number of formal and computational models emerge. And we now include in climatology the notions of climate change, environment, human intervention, dynamic systems, aerodynamics, oceanic currents and so on. Still, even with up-todate conceptual models, the new ones manage to touch upon only a small portion of the complex natural phenomenon studied. We do not even agree yet on what these notions are in the conceptual model of the climate. Hence, science does not start and finish in the form of equations and programs, nor as formal, computational or computer models. In many scientific practices, formal and computational models can only arise if the objectives, hypotheses, methodologies and, ultimately, the results of the research are expressed and understood through the conceptual models. For these reasons, just as in rigorous sciences, conceptual models have important but very specific roles in semiotics: epistemic, expressive and communicative roles.

Categorization In semiotics, conceptual models have their own epistemic role in categorizing the object under study as well as for reasoning on it. Because of the complexity of semiotic artefacts and of the process of semiosis, the operation of categorizing breaks down into sub-operations of classification, ordering and selection. However, in the semiotic domain, building classes is not evident or easily given to observation and analysis: much conceptual clarification is required. For example, in research regarding the semiotic content of wedding rings worn on the finger, a conceptual model will expend quite an amount of energy in identifying what is believed to be the classes of the most pertinent semiotic features. These could include its shape, measurements, material, form and so on. But often, for some semiotic artefacts, the most important features and properties are not immediately observable. In fact, in our wedding ring example, the following features or properties may be observed in some or all cases: there is a ritual; it is led by a civil minister; wishes and promises are pronounced; the couple is of same or different sex; the ring is first to be given by one of the engaged and the other one gives a ring in return; the close family is present; the ceremony takes place in an official building, in a church, in the family’s house or in a garden and so on. Identifying these classes of properties and features is not an easy task. Still, such processes of categorization and classification are among the core tasks of a semiotic conceptual model. It is often in such categorization

 Conceptual Models in Semiotics 77 and classification processes that research is creative. They determine the first variables and functions to be used in the formal and computational models. Hypotheses of pertinent features of the artefact under study emerge and their relations become apparent. Ultimately, it is this conceptual model that serves in the interpretation of the research results. The second epistemic sub-categorization task of a semiotic conceptual model is that of ordering. All the features retained are organized, if not structured, by some sort of meta-classification. Consider again our wedding ring example: if the ring presents features discovered to match those of bracelets, necklaces or earrings, then these items could be categorized together into the meta-category of jewels. The various persons present during the ceremony could be categorized as actors or participants. The price of the ring and the clothes would pertain to the economic cost, and so on. Many of these features are meta-classes often belonging to some specific conceptual frameworks, which may be anthropological, sociological, psychological or philosophical, to name but a few. In some cases, these structured meta-classes may even form a hierarchy or a taxonomy. And in the context of some possible computation integration, they could be embedded into more formal conceptual frameworks such as frames, ontologies, folksonomies or conceptual graphs. Because of the complexity of some semiotic artefacts, such an ordering may not be that easy to achieve in the first moments of the conceptualization of the problem. Putting order among the artefacts often comes after completing a good part of the research itself. Finally, we also have a selection of categorizing tasks. In a specific semiotic enquiry, only a particular subset of features will be retained by the conceptual model regarding its relation to formal and computational models. In other words, not all features that have been retained in the conceptual model will be used as the reference of arguments (i.e. constants and variables) of a mathematical function or as an input of an algorithm. This therefore imposes a reduction of the analysis in regard to all the features and processes identified in the conceptual model. Not all are retained. Take our wedding ring example again: imagine a computer-assisted semiotic project that aims at finding the social meaning of wedding rings among certain American ethnic groups. A formal and computational model for this research would not use as its ‘data’ all the features and relations that some rich and profound conceptual model may have identified. On the contrary, and depending on the objective of a particular enquiry, only a few specific features and relations could be retained for a formal and computational semiotic enquiry. A culturally oriented semiotician may want to select the features gold or silver, location, costumes, time of the year and so on. A semiotician more influenced by structuralism may retain specific types of relations such as the rituals and speech act associated with the wedding ring, for instance. This reduction seems highly problematic for many semioticians, for it does not seem to respect the complexity of semiotic phenomena. Still, this does not mean that the non-selected features and relations identified in a conceptual model are not important. They are not just pertinent for the objectives of a particular enquiry, and they may be retained and serve for another project (e.g. type of ring relative to socioeconomic class, etc.). Therefore, the same wedding ring can reveal different meanings

78

Computational Semiotics

depending on the point of view taken. But not all of them may be retained to be inputs of formal and computational models.

Reasoning The second epistemic role of a conceptual model is reasoning. And because the semiotic problems are often wrapped up in a web of dimensions and factors, both descriptive and explicative reasoning become complex. Also, because the semiotic features of the artefacts under study appear more often during the process of discovery than during that of justification, the reasoning does not always follow a nice and clear deductive or inductive path. So, abductive reasoning will dominate and hypotheses will abound. Metaphors, fictions and narrations will often be used and substituted to sharp and precise logical means. In some more complex cases, some iconic languages such as schemas, graphs or static and moving images will be called upon to assist the reasoning process. Even if, for instance, it may seem simple to find the semiotic features of a wedding ring worn on the finger such as ‘it is gold’, ‘it costs 2000$’ or ‘it is worn by the spouses’ – these features being easily observable – there may come a moment where the conceptual model may have to use metaphors to describe the deep psychological conditions related to a wedding ring. It may come to be stated that a ring is a bridge of love between two persons, a path to heaven on earth, and a comedian may even declare it to be the first clip of a handcuff. And in conceptualizing some complex phenomena, such as social media, one may have to call upon graphical representation to get an idea of the intricate relations embedded in them. And sometimes, the semiotic object cannot even offer a handle by which to describe it, or the discourse may even use intentional or anthropomorphic descriptions such as saying about a computer that ‘it does not want to turn on’. Such comments may not become the reference of symbols and formula in formal models, but they still blend into the overall conceptual horizon that weaves the web of understanding. A conceptual model may feed a rich categorization process, but somewhere and somehow its reasoning must become more constrained so that irrationality does not pervade it. Rigorous reasoning in natural language discourse does not necessarily need a formal model to succeed. But a formal and computational model nourished by a rich conceptual model may become an important partner in a semiotic enquiry.

The expressive and communicative roles This brings us to the expressive and communicative roles of semiotic conceptual models, which are highly important. In fact, because of the complexity of their research, semioticians may spend quite a lot of time and energy in building their private conceptual models about the semiotic artefacts they are studying. They will often build, modify and transform them to arrive at an acceptable version. But eventually and often in parallel ways, they must come to an explicit and public expression of their

 Conceptual Models in Semiotics 79 private conceptual models. And as in any other science, they will choose some semiotic forms to express it. The most important expressive form will be oral or written natural language. And here, the concepts and conceptual statements expressed through sentences build a typical semiotic discourse about the object or processes under study. The mental models and their expression become closely, if not intimately, interwoven into some natural language expressive forms. This discourse does not just express the features and relations that characterize the semiotic artefacts under study. It expresses not only something about the objectives of the inquiry itself, but also something about the intuitions and beliefs of the researchers. It formulates hypotheses and parameters for the demonstration and research results. For instance, in our ethnic study of rings, the research may come to express a result by the following sentence: for most American ethnic groups, an engagement ring is a precious metal object that couples put on one of their fingers for overtly declaring their engagement to be married to one another. Other types of sentences would be produced for the hypothesis, for the deployment of the methodology and so on. Unfortunately, because semiotics rarely calls upon formal and computational models for its analyses, its discourse does not always make evident the embedded conceptual model that it contains. But when computation is to be integrated into semiotic research, the important epistemic role of semiotic conceptual models reveals itself. An explicit conceptual model becomes a necessary condition for constructing formal and computational models. The sentences used in a formal or computational model are formula, equations or program lines. But their ultimate meaning and understanding live only in the natural language discourse that constructs the conceptual models. And all and each of the sentences may ultimately be understood because they are included in a specific context given by all the other concepts and frameworks used in the overall conceptual models. There is, however, an important point that must be emphasized here. A semiotic conceptual model must not be conflated with a semantic model. A nice example is the word marriage and the concept of marriage. In certain contexts, the concept underlying the word marriage is much more complex than its lexical meaning. The debates governments and churches have about gay marriage is not about its lexical meaning. The same type of difference between the lexical meaning and the concept is to be found in the word life: the core of social debates about abortion is surely not about the lexical meaning of the word. Thus, because of the complexity of semiotics, most semiotic projects that deal with semiotic artefacts and processes will produce a natural language discourse which often becomes a conceptual model that allows the description, explanation and understanding of the object under study and of the research results. The project may be so complex that simply building the conceptual model will be in itself a rich and heuristic endeavour. A formal model and a computational model must therefore be seen as different types of enquiry. But if this is the case, the computational semiotic research project cannot escape the construction of a conceptual model. Finally, a conceptual model has an important communicative role in semiotic research. It is through this type of model that a large part of the exchanges are conducted among the semiotic community and ultimately with the general public. The whole

80

Computational Semiotics

research purpose of semiotics is not just the production of exhaustive descriptions of the features and structures of semiotic artefacts and processes. Its ultimate aim is to communicate the results of the research and allow a better understanding of the meaning of the semiotic artefacts and processes. This implies that the conceptual models are ‘naturally’ integrated into the various conceptual and cultural frameworks shared by a community. In other words, a conceptual model is a socially shared model. These three roles are thus essential for a semiotic conceptual model. The roles of semiotics is not just to offer theoretical grounding for meaningful artefacts and processes. It is also a theory of how we can describe, explain and above all understand artefacts and processes that carry meaning. But understanding is a dynamic cognitive state: it emerges when the knowledge acquired evolves and is embedded in and through its multi-levelled context or situation. It does not belong solely to the individual cognitive agent, and it can be socially or culturally shared. And it is one of the main roles of conceptual models to offer the clearest means of this understanding. But in a context where computation is to be part of the research endeavour, it is a necessary path to its effectiveness. The conceptual model, however, has its limits: its preferred mode of expression – natural language – is burdened with ambiguities, biases and impressions. Nevertheless, it is essential in the research process, as it is the main language that is immediately available to humans. It is the one by which we mostly express our intuitions, creativity and beliefs about the world, and it is the means by which we integrate semiotic content of artefacts into our lifeworld (Lebenswelt). It is the first language by which we communicate with others. In short, it is ultimately only through natural language that we understand and interpret the objectives and results of semiotic research. Still, some may be dissatisfied with being limited to a conceptual model of semiotic artefacts and processes. They may require more models such as formal and computational models.

Semiotic conceptual frameworks Semiotic artefacts, systems and processes are not simple phenomena. What is usually given to immediate perception and even to conceptualization is an individual ‘carrier’ or physical ‘vehicle’ of some sort. Some of these sign vehicles are simple – for example, a ring, a red light on the corner of a street, a burning candle on a cake, but some are much more complex, such as words in a sentence, a text, a funeral ceremony, a concert, a ballet, a cathedral or the paintings of a particular artist. All these artefacts and processes have been studied via various semiotic paradigms, for they are said to bear meaning, messages and information. If a formal or computational model is to be integrated into their study, semioticians must build a conceptual model. And for each semiotic paradigm, we can see that the conceptual models organize their epistemic expressive and communicative roles into an overall conceptual framework. Each of these conceptual frameworks is not about a  specific semiotic artefact or process. It is rather a meta-conceptual model which, in a sense, offers different theoretical parameters for building specific conceptual models into the study of semiotic artefacts and processes. It is important to briefly recall here the three main paradigms of

 Conceptual Models in Semiotics 81 semiotics: (a) philosophical, (b) structuralist, and (c) naturalist–cognitive. Each one has a specific point of view on what a semiotic artefact or process is and how they should be studied. Each one defines a specific framework for conceptual models. The philosophical frameworks will have the conceptual model focus on the function of the semiotic artefact. The mentalist variant will be attentive to the agent – that is, to the mind that creates the representation. The pragmatist one will insist on the contexts that embed it. And the hermeneutic one emphasizes the complex interpretive circle woven into it. The linguistic and logical structuralist semiotic frameworks do not necessarily negate the preceding point of view. They rather complement and complexify it: the semiotic relation is always a ‘stand for’ relation, but this relation is complex and structured. Still, the three variants of this paradigm understand this structure differently. Some insist either on the inner or outer form of the structure. For instance, the Saussurean and Hjelmslevian paradigms see it as embedded in a complex syntagmatic and paradigmatic structure. The logicists define it using some formal compositional grammar to which meaning is externally attached. And the dynamicists insist on the holist, gestaltist and dynamical dimensions of this structure. Finally, the naturalist semiotic frameworks focus on the methodological points of view: semiotic artefacts should be approached from a cognitive science perspective, be they a matter of human, animal or even of artificial cognition. Semiotic artefacts are objects of perception, categorization and communication. They should be studied through natural science methods that take into account complexity, adaptation and evolution. We can briefly illustrate how our different frameworks determine some parameters in a conceptual model for a simple but rich set of semiotic artefacts: flags. Flags have provided a classical and heuristic case study in semiotics (Knowlton 2012) and in many social and human sciences in particular.2 They even became the object of a specific subdiscipline: vexillology3 (Pasch 1983) – the scholarly study of flags, which has its own journals, conferences and associations. Figure 6.2 shows a small set of flags of political organizations:

Figure 6.2  (a, b and c). National Communist Party flags.

A philosophical conceptual framework would explore these flags from the perspective of their representational function. The mentalist would explain them with regard to what they mean for a mind. The pragmatist would relate their meaning to the situation, the context or the feeling. The hermeneuticist would see their interwoven cultural and political functions. The Saussurean would notice that the meaning of

82

Computational Semiotics

these flags rests upon complex interrelations between their components (colours, shapes, icons, etc.). The logicist–structuralist would see in them a categorial symbolic structure. And the naturalist could focus on the importance of cognitive, social, cultural or political information carried by the various components. Each framework opens up specific and rich enquiries. And each conceptual model would have to categorize and express all of the various characterizing features differently. But most interesting for our research is that we see that each framework imposes constraints upon the integration of formal and computational models. For instance, each one determines a specific type of variable and function or data and computations that are pertinent for the study of an artefact. For example, a philosophical framework will not deliver the same values, inputs, functions and operations as would a structuralist and naturalist framework.

Case studies for conceptual frameworks in a computational environment In this chapter and in the following ones, we will use three main case studies to briefly illustrate how each type of model is used in computational semiotic enquiries. In the present chapter, we focus on the conceptual model. We aim to show what our different types of models require when computation is to be integrated into semiotic projects containing three prototypical types of Peircean semiotic artefacts: iconic, indexical and symbolic signs. In other words, we will present computer-assisted semiotic analyses of three types of semiotic artefacts: paintings, music and texts. Our first example of a conceptual model is a computer-assisted analysis of iconic semiotic artefacts: the whole set of Magritte’s paintings. One can imagine the difficulty of analysing paintings that present such unusual and surrealist topics as may be seen in Magritte’s paintings. Recall here two well-known ones: (a) ‘La trahison des Images/The treachery of Images’ where there is a pipe with an underlying painted French sentence ‘Ceci n’est pas une pipe’. Even though artificial intelligence claims that we can process images using computers in a sophisticated manner, the reality is that AI is mainly used for information recalls, recognitions and recommendations – not in image analysis (Hare et al. 2006;4 Liu et al. 2015). And if some AI deep learning strategies could succeed in recognizing some particular figures in these paintings, semioticians would probably be reluctant to rely on the results obtained. In the current state of the art, machine learning would probably find in Magritte the presence of a woman, horse, tree trunks, hats and smoking pipes. ‘How nice’, the semiotician would say. And he or she would probably add that we did not need a computer to discover this. In other words, image recognition is not image analysis. In semiotics, the analysis of iconic signs (images, photos, films, paintings, etc.) is a common task. One of our projects5 realized a computer-assisted semiotic analysis of the paintings by Magritte. In this project, our four models played an important role. We present here only the general design of the conceptual model. The formal,

 Conceptual Models in Semiotics 83 computational and computer models will be explored in the chapters that follow on semiotics. In this project, the conceptual model contained two interrelated epistemic operations: it expressed a categorization and supported reasoning. The first role was to categorize some of the salient features present in each of the 1,487 paintings. Each feature was categorized in the conceptual model by some forty to fifty semiotic descriptors and expressed by some natural language predicates or names. This description task was hand-produced by another related independent Magritte project (Hébert and Trudel 2003–2013; Hébert, Michelucci and Trudel 2018) whose objective was to put online the digitized scanned copies of the Magritte paintings published in Sylvester, Whitfield and Raeburn (2018) and in the catalogue of Magritte’s paintings and to annotate each of them using a set of descriptors (Trudel and Hébert 20116). Many types of descriptors were possible. But the one chosen was guided by a protocol greatly inspired by the structuralist semiotic paradigm (Hébert and DumontMorin  2012)7 for analysing iconic signs. The focus of the description was on the salient perceptual features of each painting. For instance, descriptors for a painting could categorize them according to their colours, their forms, the objects, persons, animals, buildings, skies, atmosphere, etc. Here is a sample of such tagging for only three paintings: Nu, La Trahison des images (Ceci n’est pas une pipe) and Les Amants. Table 6.1 has a sample of the set of descriptors for three paintings.8  Table 6.1  Descriptors for Three Magritte Paintings Title

Descriptors

Nu (Nude) 1919

, , , , , , , , , , , ,

La Trahison des imagesb (The treachery of Images) (Ceci n’est pas une pipe) 1928

, , , < green background>

Les Amantsc (The Lovers) 1928

, , , , , ,

a

Reproduction of ‘NU’: https://www​.wikiart​.org​/en​/rene​-magritte​/nude​-1919, accessed 2 November 2021. Reproduction of ‘The Treachery of Images’: https://www​.wikiart​.org​/en​/rene​-magritte​/the​-treachery​-of​ -images​-this​-is​-not​-a​-pipe​-1948, accessed 2 November 2021. c Reproduction of ‘The Lovers’ : https://www​.wikiart​.org​/en​/rene​-magritte​/the​-lovers​-1928​-1, accessed 2 November 2021. a

b

Thus, each painting was tagged using semiotic descriptors resulting from a categorization of salient features. The descriptors were a natural language categorization of these features. And in this project, it was these descriptors which became the inputs for a formal computable model. But as each descriptor was a natural language predicate or proper name, it possessed a linguistic semantic content that was quite rich in content and inferences. And this semantic content was used in the result ultimately produced by

84

Computational Semiotics

the formal, computational and computer models. Still, the conceptual model contained many concepts that were not always explicit. They belonged to background knowledge (art history, iconography, social history, cultural beliefs, structuralist semiotic theories, etc.) necessary for analysing the paintings, and more so for interpreting the results of the computational processing that was applied to them. The second task of the conceptual model focused mainly on determining the conceptual ground of the method that would identify patterns in the paintings. The dominant intuition here is that each set of descriptors in a painting can be understood by reference to the Saussurean and Harrisian structuralist and distributional semantic theories. This means that the set of descriptors in all the paintings bear internal contextual relations in each individual painting (for instance, horse + woman + tree). But they also bear external similarity relations with the descriptors of the other paintings. For instance, the painting where the descriptors women + horse are co-present can be seen as having a relation of similarity to the painting where the descriptors woman + columns + man are co-present. In other words, the method would explore syntagmatic and paradigmatic relations. In Firth’s famous dictum: ‘You shall know [the meaning of] a word by the company it keeps’ (Firth 1957: 11). This allows discovering the salient or dominant similarities and regularities in the set of descriptors in each annotation and between the annotations. It is this conceptual model which explains the choice of formal mathematical models, algorithms and computer implementations. And it is through this explicit or implicit conceptual model that the results obtained were interpreted. The modelling strategy chosen by Trudel and Hébert could have been more sophisticated. For instance, it could have been presented using some of the slightly more formal knowledge representations such as frames, graphs or ontologies. In fact, they are close to informal folksonomies. All these types of annotations are acceptable forms for a conceptual model. But the real problem is not in the form, but in the content and in the concrete application to each painting itself. For the Magritte project, the form was a list of approximately 1,400 descriptors, but it still took a few years to attach these descriptors to each painting. More importantly, however, the decision to attach a particular descriptor to a specific painting rested upon the expert’s subjective and intuitive semiotic analysis. To our knowledge, in the current state of computer technology, no computational model could have realized such a task. Maybe in the future some of these descriptors could be discovered by some algorithms if the background conceptual frameworks are available for assisting the tagging. The important thing to point out here is that, in a sense, a computer-assisted analysis of semiotic artefacts requires a conceptual model, whatever the form it may take or whatever its source. Our second example of a conceptual model is a computer-assisted analysis of indexical types of signs or signals that are causally produced: an index is a sign fit to be used as such because it is in real reaction with the object denoted (Peirce, MS 517, 1904). Signals are such sorts of signs. They indicate their source directly because they are directly related to it. Therefore, their meaning (denotation, object) is their very source. At first sight, if the meaning is nothing but the source, then this type of semiotic artefact might not seem very interesting. But if indexical signs are seen from the perspective

 Conceptual Models in Semiotics 85 of their interpretant component, then they become much more interesting. In fact, there are important semiotic fields whose research objects are precisely these types of semiotic artefacts. A classical field of this sort has translated the semiotics of indexical signs in terms of signals and information. In semiotics, these types of signs are studied in various semiotic subdisciplines such as information semiotics (Peirce 1904; de Tienne 2006; Prieto 1972; Nadin 2011a), cybersemiotics (Nöth 2012;9 Brier 2008) and biosemiotics (Hoffmeyer 2008;10 Sebeok 196811). And outside of strict semiotics, many researches (Dretske 1982; Floridi 2004; Rieger 2003; Brier 2008) have explored this informational paradigm to study semiotic artefacts and their technology. It has also found its way into computer semiotics (Tanaka-Ishii 2010; Liu 2005) and even into robotics (Meystel and Albus 2001). Still, as Brier (2008) often pointed it out, the information component may not be enough, for the meaning of a signal is often not limited to ‘denotation’ or the ‘object’ components. It also includes the interpretant component, which is the most interesting one. One important type of such semiotic artefacts is music. Technically speaking, music, physically, is just a sort of sound. But it is a signal when there is an agent receiving this sound. And it becomes music if the agent receiving it possesses a special type of cognitive capacity. The meaning of a musical signal is both in the source and in the interpretant: the inner musical experience of the listener. This musical experience of the listener is indeed indexically related to a physical sound source. A typical example would be the semiotically structured sounds produced by a drum, by a violin and even by vocal cords. But it also has an interpretant whose content is much more than what caused and structured these sounds. Music is an interesting semiotic artefact because of the complexity of the interpretant it creates in the human mind and in emotions (Langer 1953). According to Short (2007), this complexity of the meaning relation in music should in fact lead us to see it not as a signal, but as being similar to the pure icon. In the context of our present project, we illustrate how and where computation can be involved in the analysis of such types of semiotic artefacts. One important fact is that music today is offered increasingly in digital format. Many types of technology can transform an audio signal into a digital electronic signal that a computer can manipulate and output towards some type of speakers. And this music can be stored in databases on which recognition, recall and recommendation tasks can be carried out by some music browser. In such cases, a specific type of semiotic research arises: how can digitally encoded music be recalled in regard to queries that express features of the musical experience rather than features of the physical sound wave or of the metadata – that is, the various and meaningful types of features that music experts attribute to such artefacts? In other words, music experts (creators, musicians, music lovers, etc.) often wish to find musical segments that correspond to their personal inner representation of music (melodies, themes, rhythms, harmonies, atmosphere, styles, etc.). The features of such inner visions of musical segments are neither given in the digital encoding of the audio signals nor found in the canonical metadata (author, artist, dates, producer) attached to audio signal databases. This is the main research question for projects exploring digital music in the very specific semiotic task of musical browsing.

86

Computational Semiotics

In the field of digital music, there exist many computer applications that can browse digital files directly by using music-related keywords such as ‘songwriter’, ‘artist’, ‘producer’, ‘genre’, ‘title’ and so on. Digital applications such as Spotify, Qwant Music, Titan and iMusic contain such music browsers, while the applications Shazam and Audio Tag are said to ‘recognize’ music! These browsers recall both the physical audio segments and the relevant peripheric information attached to them: biography, identification, creators, musicians, etc. They are useful and precious for most users. But they are less useful for creators, musicians or experts whose browsing has a semiotic preoccupation. For instance, in a serendipitous inspiration, a musician may search for some new style, theme, melody or rhythm that corresponds to his or her intuition. This type of query has a semiotic dimension. This search is related to the interpretant the users attached to musical forms. And for these types of users, the classical keywords do not correspond to the lexicon or words that express their idiosyncratic musical competence, knowledge or intuition. For instance, a jazz band will often have a particular idiosyncratic vocabulary to express coordination between members. The same goes for film sound editors. They have their own lexicon. To cope with this situation, Adjiman (2018) has built a music database for searching background music tracks of films or videos. The solution was to use some semiotic language expressions that refer to the film’s style of narrative structure, to the sound editor’s preoccupations, to intuitions and creative processes, to the target public’s demographics, to the type of feeling to be elicited and so on. A classical keyword approach cannot meet such needs in terms of queries. One research project (Rompré 2015; Rompré, Biskri and Meunier 2017) has built a computer application whose browser offers some features and components that do respond to these needs. The project contained our four types of models. Here, we present only the conceptual model. Its main purpose was to conceptualize the many dimensions that participate in the explanation and understanding of how one can apply a computer browser to a music database by making queries that take into account the way music experts or amateurs express their musical experience. The core of the conceptual model underlying this application was to offer an architecture of the various types of categories of the many features of a significant musical artefact – features that are expressed through descriptors. The architecture distinguished three important categories of features. The first type of descriptors conceptualized various features of the acoustic nature of the sound wave (frequency, signal to noise ratio, compression, sample rate, bit depth, encoding format, etc.) for the variety of types of musical sources. Some of these descriptors of musical acoustic signals were expressed by means of natural language and mathematical expressions. But many were expressed directly in formal digital code descriptors whose meaning is more pertinent for computer processing than it is for humans (Figure 6.3). The second type of features is related to some perceptual musical characteristics, such as melody, rhythm, instrumental timbre, pitch, key, texture and harmonization. These features can be expressed in natural language through descriptors. And some generalization reasoning and conventional cultural background will serve to produce higher-level descriptors such as those pertaining to genre (jazz, rock, classical,

 Conceptual Models in Semiotics 87

Figure 6.3  Visualization of an audio waveform.

electronic), artist (Madonna, Billie Eilish) or instrument (guitar, orchestra), for example. These descriptors have an important functional role in organizing a musical database. They are the descriptors often associated with many other peripheric metadata descriptors, such as the name of the composer, the artist, the producer, the time, the date, the studio or the format. There is now an international effort for standardizing this type of functional annotation: the Music Encoding Initiative (MEI)12 (Crawford and Lewis 2016). The third and most important and original type of descriptors and descriptions are extracted from the many commentaries or texts posted online and associated with musical pieces. They express the way these users conceptualize their relation to music. They are semantically richer than sound waves and functional data, as they describe emotions and a variety of relevant musical characteristics which are often idiosyncratic to a musical expert or to a musical epistemic community. Some features express in natural language the intuitions, creativity, emotions, critiques or tastes of expert listeners and producers. They are expressed in natural language terms and are found in the multiple textual comments made by thousands of music users in their conversations and discussions around various types of music. In the project, all three types of descriptors were integrated and then used in the browser to answer queries. Thus, the main components of the conceptual model can be seen as a conceptual frame for ‘representing’ the concepts of digital music where each segment of music is annotated by three types of descriptive categories or metadata. A first one offers descriptors of the digitally encoded music. The second contains functional meta-descriptors for manipulating music files by means of a browser. The third one produces a set of culturally shared lexicons built by the music producers and by the musical community itself. This integrated set of descriptors served as input for formal and computational models of a digital music browser that can recall and recommend musical segments corresponding to queries expressed in part in the technical terms of music industry sound engineers, but mainly in those of the epistemic community of musicians, artists and so on.

88

Computational Semiotics

When used, such descriptors can then allow semiotic paths that go beyond what is provided by classical types of browsers. In fact, we see such browsers emerging in the semantic web. These browsers take into account not only descriptions of digital audio signals and the functional annotations of the digital files, but also many relevant semiotic information features extracted from commentaries, criticisms, recommendations or musical ontologies and found on various computer sites, but mostly on social networks dedicated to music (Passant et al. 2008; d’Aquin  et al. 2011).

Symbolic semiotic artefacts: Concepts and narration in texts  Our third example of a conceptual model is a computer-assisted analysis of symbolic artefacts. Here, we understand symbolic semiotic artefacts in the general Peircean sense – that is, as a type of semiotic artefact that has an object that is conventionally determined by an interpreter determining its interpretant. ‘A  symbol . . . represents its object solely by virtue of being represented to represent it by the interpretant which it determines’ (Peirce 1902: MS: 599). ‘It [a symbol] is a conventional or quasiconventional sign, which represents its object as conforming to some general rule of representation’ (Peirce 1898: MS 484). In the case studies we present here, the symbolic semiotic artefacts are natural language symbols as they appear in two different types of texts. The first one is a collection of philosophical texts where the analysis is conceptual analysis, while the second one is a collection of newspaper texts where the analysis is narrative. In digital humanities communities, this type of computer-assisted analysis is the bread and butter of research projects. In academic literature, it often takes the name of distant reading (Moretti  2013) or computer-assisted reading and analysis of text (CARAT) (Meunier  2009). In computer science, it is associated to text mining (Manning and Schütze 1999; Hotho, Nürnberger and Paaß 2005). Many more specific or specialized applications are embedded in it, such as topic modelling or lexicometry, and semantic, sentiment, argument or conceptual analysis. Still, distinguishing strictly semiotic text analysis from the ones that are, for instance, purely literary, philosophical or historical is not an easy task. In this chapter, we will keep our text analysis as close as possible to a semiotic perspective and focus mainly on the conceptual models that the computational projects require. Our first illustration is a computer-assisted conceptual analysis of text (CACAT) project13 applied to the concept of mind in The Collected Papers of Charles Sanders Peirce (Peirce 1931–58) (Meunier and Forest 2009; Pulizzotto et al. 2016; Chartrand et al. 2018). In this project, the conceptual model’s role is to express in natural language the various components of the computer-assisted conceptual analysis. It requires modelling in natural language discourse (a) the concept of concept;14 (b) the theoretical perspectives of conceptual analysis (logical, structuralist, linguistic, rhetorical, hermeneutical); (c) the methodologies of such conceptual analysis; and (d) the type of possible users and contexts.

 Conceptual Models in Semiotics 89 The conceptual model for a computer-assisted analysis often constitutes an important theoretical discourse which becomes in itself an essential part of the research. And because it is expressed in natural language, it serves as a means of communication between researchers and is the occasion for lively philosophical and linguistic debates. It is hoped that from these debates operational hypotheses will emerge, such as: (1) Conceptual analysis can be realized by the exploration of textual contexts of the canonical forms of a concept’s linguistic expression (a predicate: e.g. the word ‘mind’). (2) The exploration of the textual contexts of a concept is itself realized through some mathematical classification strategy. (3) Classes of similar conceptual contexts can be annotated so as to categorize their semantic content. Some of the operational details inferred from these hypotheses become the input for the formal models. Our second example of a computer-assisted semiotic analysis of symbolic signs is an exploration of narrative structures in newspaper articles about a social crisis. In the spring of 2012, university students protested vehemently in the streets of the cities of Quebec, Canada, requesting, among other things, the abolition of university tuition fees. The street demonstrations of the many thousands of students were numerous and noisy. Pots and pans were banged more in sympathy than in harmony. This went on for months. It became the daily food of journalists. And the thousands of papers written on these series of protests would call it the ‘Maple Spring’ revolution (Printemps érable).15 For Pulizzotto (2019), the newspaper articles were a gem for the exploration of computer-assisted semiotic analysis of narrative structures, one inspired by the structuralist semiotic theories of Saussure, Hjelmslev, Greimas, Rastier and others. We can see how our four models were heuristic in this project. The conceptual model not only defined classical concepts for computer text analysis, but also had to (a) explicate the concept of narrative structure, (b) extend the Saussurean and Harrisian distinction of syntagmatic and paradigmatic linguistic relations to a set of different texts, (c) adapt the Greimasian actantial model, (d) determine various levels of textual structures, and (e) explore a surface and a deep semantic structure description.16 All these were in addition to all the concepts pertaining to the various methodologies involved. Two main hypotheses emerged from this conceptual model: (1) that it is possible to build a formal and computational model of the narrative regularities in a text, and (2) that the Greimasian actantial model (1970)17 and Fillmore’s model of case frames (1968, 1976)18 could be used for this purpose, one operating at the level of the surface structure, the other at the level of the deep structure.

Computation and semiotic conceptual models As we have said at the beginning of this chapter, conceptual models are omnipresent in semiotic projects. They are not always explicit or even recognized as such, but they are

90

Computational Semiotics

still models. In fact, conceptualizing the various components of a semiotic analysis and expressing them in a natural language discourse is probably one of the most important tasks of a semiotic project. And because of the complexity involved, it would probably take most of the research energy. But once semiotic projects introduce computation into their analyses, conceptual models become conditions of the success of the project. In such a context, a conceptual model has to become explicit, for it has to provide the most relevant descriptors and descriptions for constructing both a formal and a computational model. And some of them become the reference of the ‘arguments’ (variables and constants) for the functions of formal models and the inputs of algorithms (e.g. classification, similarity identification, trend description). And it will be the whole of the conceptual model, implicit or explicit, that will be called upon for the ultimate explanation and understanding of the semiotic artefact. Computation in semiotic projects does not eliminate the rich and profound conceptual modelling semioticians build in the understanding of semiotic objects. It just brings them a new challenge whose signature opens up a new door for semiotic research.

7

Formal models in science

Formal models as prototypes of scientific models The means by which conceptual models describe, explain and understand a phenomenon and by which they are themselves expressed make them essential partners in science. But they are not the only ones. As the history of science has shown and as most of contemporary epistemology claims, scientific research cannot be only a set of natural language statements aggregated into informal conceptual models – as precise and refined as they may be. As von Neumann formulates it so well: Sciences do not try to explain, they hardly even try to interpret, they mainly make models. By a model is meant a mathematical construct which, with the addition of certain verbal interpretations, describes observed phenomena. The justification of such a mathematical construct is solely and precisely that it is expected to work. (von Neumann 1995: 628)

Building a scientific theory necessarily requires the inclusion of formal models. In fact, for scientists, formal models are the prototypes, if not the norm, for scientific enquiry: no formal model, no serious science. ‘Scholarly disciplines embody the formalization aspect. . . . It reflects, on the one hand, the need of the human mind for rationalizing and for orientation in an immensely complex reality, to be achieved by building up systematic and coherent general images and symbolic schemes’ (Shanin [1972] 2013: 1). Indeed, this is even more true for any science that requires the integration of computer technology in its enquiries. In today’s scientific practices, even if we admit that formal models form an essential part of scientific enquiries, they are not to be conflated with computational models. Rather, a formal model is a requisite for a computational approach in science, and it has its own signature.

Syntactic definition of formal models In this chapter, we will briefly examine the definition and nature of formal models as they are used in classical science. It will help us to better understand how semiotic

92

Computational Semiotics

research projects have to include them if they are to enter the digital paradigm, and we will present these formal models by keeping in mind their possible usage in a semiotic perspective. The questions which will guide our reflections are the following: What makes a model ‘formal’ and how can it be used to study artefacts whose main characteristic is to be a carrier of meaning? This question may be formulated in more theoretical terms: What is the syntactic and semantic nature of formal models and how are they pragmatically used in science? The answer to this question reaches far back in history and, surprisingly, lies in the humanities, in philosophy and in linguistic theories.1 Today, there are many definitions of formal systems.2 Some general and classical ones are the following: A formalism presents itself as a system of symbols  submitted to manipulation rules. (Ladrière 1957: 12, our translation) To say that something is a formal scheme means only that it is a set of symbols with rules for putting them together – no more and no less. (Marr 1982: 21) [A formalism therefore] is a system of rules and conventions that deploys (and often adds to) the symbolic language of mathematics; it typically encompasses locally applicable rules for the manipulation of its notation, where these rules are derived from, or otherwise systematically connected to, certain theoretical or methodological commitments. (Gelfert 2011: 272–3)

Formally defined, a formal system is a set of formulas created out of symbols, rules and operations. We recall here that in set-theoretical language, a formal system defines a grammar that includes two set elements: FS = . The first set S contains symbols; F contains systematic functions on S.3 Some disciplines call this highly regulated symbolic system a formal language. In such a case, a set of symbols is called the vocabulary or the lexicon, while the set of rules is called the syntax of the language. Expressed differently, a formal language is a set of atomic or discrete symbols (expressions, terms, etc.) and a set of constructed symbols (formulas, sentences, etc.) following rules (composition, generation, inference, automaton, production). Technically speaking, then, a formal model is not the set of rules and symbols as such. It is rather the set of sentences, formulas or equations that can be produced or generated by the syntax (or grammar) of a particular symbolic system. This means that formal languages are not only used in arithmetic or logic models; they can also be used in linguistic models. For example, in linguistics, it can be said that formal grammar allows ‘the generation’ of its sentences. And in mathematics, a formal model will allow the systematic production of its equations. Here are a few examples of formal expressions in various types of formal symbolic systems. Depending on the symbols included in the vocabulary, different types of sentences or formulas may be produced. ( 1) in logical language: Ǝx (Sx & Qx) (2) in arithmetical language: 10+20=30 (3) in algebraic language: z = 4x + 2y, f(x, y)= dx/dy

 Formal Models in Science 93 (4) in set-theoretical language: AUB=BUA (5) in geometrical analytical language: vector AB: A—>B:(x1…, xn→a1, x1+… an, xn) (6) in lambda calculus language: λx[λy[λz[xz(yz)]]] (7) in topological algebraic language: {U = ∪{U{i}}; i ∈ I} (8) (9) (10) (11) (12)

in directed graphs language: in generative grammar language: ((Mary)N (loves)V (((the)ART (cat)N) NP))S in chemistry language: C12H22O11 in genetics language: GGGAAACCC, GGG·AAA·CCC, GGA·AAC, GAA·ACC in a programming language:

Def add5 (x) Return x+5 End Many of these formal symbolic systems can be generalized in a more inclusive set of languages called categorial languages.4 Although these languages are quite abstract, they have the property of sharply distinguishing the two main ingredients of all formal languages: the symbols and the operations or rules applied to them. Although they appear different in their surface structure, it has been shown that there are equivalence relations between many of these languages. Chomsky, for instance, has proven it for some generative grammar languages and automata languages. There are a few important distinctions to be made regarding a possible usage of formal models in a computational environment. The first one distinguishes a formal system or language from an axiomatic one.5 An axiomatic language system is a formal system in which some formulas or equations have been given a specific epistemic status, such as that of being axioms, postulates, theorems or corollaries. And it is a system where there exists a set of specific rules for transforming formulas or sentences, for example in logic (inference rules). These two properties allow the rigorous construction of proofs and demonstrations. Although such axiomatic systems are defined formally, they are highly constraining systems. In certain cases, they may contain so many variables, constants, operations and rules that their manipulation and traceability create a heavy cognitive burden for the users. They become cumbersome during the manipulation of symbols and formulas. It is one of the reasons why they are not the most used ones in actual scientific practices. Thus, an axiomatic system should not be confused with a particular formal system or language. A formal model is an epistemic artefact that is used in scientific theory for explaining phenomena. And to do this, it may call upon many formal languages, including upon an axiomatic language. A second distinction is to be made between a formal language as such and its notation. A notation, sometimes also called a formalism, is a conventional set of symbols and rules by which a formal language is expressed. For example, the formal language that expresses propositional language could be written using Russellian notation or Polish notation.6 Take, for example, the logical formula expressing an implication relation between propositions p and q or m. This can be written using Whitehead-

94

Computational Semiotics

Russell notation by inserting the operators between the arguments themselves, as follows: p ⇒ (q v m). On the other hand, in Łukasiewicz notation, the operators or functors are placed before their argument, as follows: CpAqm. Both notations express the same logical structure, and they could build equivalence relations, meaning that the same formal structure can be expressed using different notations. This distinction will be important for computational models. As we shall later see, programming languages are one sort of formal symbolic system. For instance, the mathematical operation of addition can be expressed in an algebraic language or in a programming language such as C++ or Python. All of these are formal languages, but they each have their own notation and they do not all include the same types of statements. (Tanaka-Ishii 2008). It is not surprizing then that all formal languages do not necessarily offer the same cognitive ease for the manipulation of symbols, whether by humans or by machines. Yet, under certain conditions, they can deliver equivalent results. A third distinction is to be made between formal models and quantitative models. A too hasty understanding of formal languages often conflates them with quantitative languages. Languages are formal by their symbols and by their construction rules. If some are identified with quantities (often meaning ‘numerical’ quantities), it will be mostly because they are applied to a specific domain: that of numbers. But it does not follow that numbers are the unique application domain of formal models. This is an important distinction for semiotics and for the humanities. Formal models are not quantitative by definition. The best examples are generative grammars in linguistics and computer programs in computer science. There are also mixed formal languages such as in chemistry graphs to which numerical symbols are added. Because our interest here pertains to the encounter between semiotics and computers, we shall concentrate on the most frequently used types of formal models: mathematical and grammatical models. The presentation given above of a formal model through its syntactic definition is quite classic. But it aims here to show its main characteristics: a formal language is a symbolic structure that is independent of any meaning its symbols may have. From a syntactic point of view, the examples of formulas and sentences of various formal systems are then only sequences of syntactically well-formed symbolic structures. They are strictly without semantics.

Semantic definition of formal models The second approach to formal models is given through the semantic definition of the formal languages they contain. Indeed, even if these languages can be presented solely through their syntactic structure, this does not mean that in practice they cannot be given a formal semantic definition. If this were not possible, then the languages would be nothing but an empty play on symbols. A semantic interpretation is an operation that allows the symbols and sentences of a language to be meaningful. A symbol or a formula, as Peirce formulated it, is ‘something which stands to somebody for something in some respect or capacity’ (Peirce: CP 2.28). In most philosophies of science, this ‘stand for’ relation has mainly been understood through the following conceptual translation: symbols (simple or constructed) must

 Formal Models in Science 95 represent something (van Fraassen 1980: 64; Giere 1999; Suárez 2004). Or in more technical terms: ‘in a scientific representation some source A – typically a model, a graph, an equation – is used to represent some target B – typically a system, entity or phenomenon’ (Suárez and Solé 2006: 39). This notion of a relation of representation has raised complex questions, including: What is it for a formal symbolic system to ‘represent’ something? How does this representation come to mean something in such a system? Where does this meaning come from? How can a representation be formal? A most classical answer comes from the Fregean logical tradition. It defines and restricts representation as being the sense and reference of symbols and formulas. The formal interpretation of this thesis (Russell, Carnap, Tarski, van Fraassen, among others) has proposed that meaning be mainly defined in terms of reference rather than in terms of sense. The matter of sense is not denied, but determining its nature in a formal system is not an easy problem. For instance, should the symbols be taken as proper names and determine possible worlds (Kripke 1972), or should the sense be identified with the function determining the reference (Tarski 1944)? Because of the many problems with the notion of sense, reference has been given priority. In this view, each symbol receives a specific reference and each formula is given necessary and sufficient conditions under which it is said to be ‘satisfied’, and hence true. Such a formal semantic interpretation is highly rigorous and has been the prototypical approach to the semantics of formal languages. But it has also brought about its own challenges: What does it mean to give a reference to a formal language and, more specifically, to mathematical languages? A first technical set of answers has been to define the reference of symbols and formulas by associating them to entities and relations in a domain or worlds using various sorts of correspondence rules. This allowed formalizing the referential relation itself, either in set-theoretical terms7 or in terms of algebraic topological statespaces.8 A first type of semantics imposed a discrete and static ontology submitted to a combinatorial logic while a second type opened up to a continuous dynamical ontology. Still, both semantic propositions are formal. And their ultimate purpose is to ensure a rigorous reference to each symbol in the system and to guarantee the truth conditions of the sentences or formulas. For instance, the various types of symbols (constants, variables, operators, etc.) will refer to some entities or local, static or dynamic relations projected in state-spaces or worlds. If the reference is satisfied, then the formal language is said to be interpreted. Such formal semantic propositions are intuitively clear if the symbols refer to a common domain or universe such as one where there are objects, persons, events and simple or complex relations among them. For example, one can intuitively interpret the formal logical formula S(x) & P(x) in the following manner: the formula contains predicates, operators and variables. By a substitution rule, the x variables can be changed into a letter, such as j. This delivers the formula S(j) & P(j). And in turn, the semantics of this formula is given by the following correspondence rules: (1) ‘j’ for ‘John’ (2) ‘S’ for the property ‘being a STUDENT’

96

Computational Semiotics

(3) ‘P’ for the property ‘being a POET’ (4) ‘&’ for the conjunction of two states So, the formula is true if John is in the set of all persons who are both students and poets in some actual or possible world. But this type of semantics is not as intuitively clear when the formulas are complex mathematical algebraic equations, mainly when they are used in dynamical non-linear systems such as in Lorenz’s chaos theories where initial conditions and attractors are present, or as in Thom’s catastrophe theory with its specific bifurcation and morphogenesis. The formulas of these types of formal languages raise deep philosophical questions about the reference of mathematical languages. One common intuitive proposition about the reference of mathematical equations has been the claim that mathematical languages refer to numbers. This answer is often, in turn, quickly interpreted as claiming that the semantics of mathematical languages is essentially about quantities. In this intuitive view, a mathematical model is therefore a quantitative model, as opposed to a qualitative one. But, as we know, the majority of mathematicians would not accept such a mathematical thesis. The semantics of many mathematical languages are not just about quantities. For example, take the simple mathematical equation x > y. There may be cases where the symbols can represent numbers when the variables are substituted by some digit symbols such as 1, 2, . . . n. But it is not always the case. If this formula was used in a lattice graph and represented nodes, it would not necessarily be the case that the variables represent numbers. For instance, in a lattice, the equation x > y may represent a hierarchy between nodes that would be about social networks among persons. In categorical linguistics, generative grammars are not about numbers, but words, phrases and sentences. Hence, it is not because a model is formal that it necessarily refers to structures of numbers or, as it is often said, that it is ‘quantitative’, because saying that symbols are about quantities is an ambiguous proposition: numbers are not identical to quantities. And not all formal symbols are numbers. It will be one of the great contributions of the Bourbaki mathematical school (1939) to show that the semantics of mathematical languages is not to be reduced to the world of numbers. On the contrary, for Bourbaki, the essence of mathematics is primarily about ‘structures’, that is, the diversity of relationships that can exist between elements and in what world they can be instantiated, no matter if they are discrete or continuous or if the relations are static or dynamic. The world of numbers is only one of many possibilities. In this Bourbakian view, the referential semantics of mathematical languages is about structures, that is, about relations and entities of whatever the types are. It is this thesis that van Fraassen reactivated in his semantic view of theories: ‘to present a theory is to specify a family of structures, its models; and secondly, to specify certain parts of those models (the empirical substructures) as candidates for the direct representation of observable phenomena’ (van Fraassen 1980: 64). And we could read the Tarskian formal semantic models as an instantiation of this thesis. Hence, the mathematical languages that are used in formal models and theories, be they arithmetic, algebraic, topological, geometrical or statistical, are about static

 Formal Models in Science 97 or dynamic structures. All these formal models, each one having its own signature, ultimately aim at identifying and ‘representing’ the structural relations in the entities they have chosen to study. Some scientific practices have aimed at building strong axiomatic models. But as said earlier, in scientific practices, such an endeavour is rare and quite limited because of the complexity involved in certain types of research. So, often a variety of other mathematical models will be preferred. And they are becoming increasingly grammatical or even graphical as is the case in some contemporary artificial intelligence models. ‘A model of a theory is simply a (typically extra-linguistic) structure that provides an interpretation for, and makes true, the set of axioms associated with the theory’ (Gelfert 2016: 13). It follows, then, that a formal model cannot be understood on the sole basis of its syntactic definition. This would mean that a formal model is only a rule-governed manipulation of symbols without meaning. And if a semantic interpretation is to be added to formal language, its role is to ‘represent’ something. And one hopes that ultimately, the various complex formulas or equations of a formal model, in whatever mathematical language they are expressed, represent at least indirectly something of some world – possible or actual. And if this is the case, then they are sorts of cognitive mediators for these worlds. And they have their own specific epistemic, expressive and communicative roles for describing, explaining and ultimately understanding the research problem, its methods and its results. This dimension of knowledge acquisition can be seen as belonging to a pragmatic approach to formal models (Suárez 2004). In the philosophy of mathematics, various theoretical propositions have been offered for understanding the type of knowledge acquired through such formal models. One traditional position has been to cast this knowledge problem more along the lines of its justification or truthfulness than of its cognitive role. In this perspective, and as we said earlier, one of the main functions of a formal language is to represent something for somebody. ‘The form of a set of equations represents a target system indirectly, via an intermediate “matching model” that is posited as isomorphic to the target system’ (Gelfert 2011: 273). But one must be prudent with the meaning of the term ‘representation’ when used for explaining the semantics of a formal language. Symbols and formulas do not ‘represent’ some object in the way that a mirror9 simply reflects something. Rather, they just ‘make present anew’ some specific dimensions of the object under investigation. To speak metaphorically, models are a sort of filter, idealization or abstraction (Morrison  2009) regarding what is represented. Mathematicians translated this rep­ resentation as specifying the direction of the representation. A mathematical equation is a projection or mapping. It goes from a particular mathematical structure (MS) to some specific object Figure 7.1  Mapping from an equation to an object: The pendulum. structure (OS) (Figure 7.1).

98

Computational Semiotics

Epistemic roles of formal models Inspired by classical and recent research on the pragmatic epistemic role of formal mathematical models in science (Morgan and Morrison  1999; van Fraassen 2010), we shall deepen this ‘representational referential’ role by exploring two main types of cognitive operations that we have focused on in our examination of conceptual models: categorizing and reasoning. Formal models have also their specific way of determining how a particular formal language allows scientists to categorize their objects of investigation and thereafter to reason on them. We will explore such epistemic roles regarding their application to semiotic enquiries aspiring to integrate computation.

Categorization In a formal model, classification, ordering and selection are highly interwoven in the formal language chosen. In fact, and according to a Bourbakian10 point of view, a categorization11 in a formal model can be seen as projecting or mapping some type of structure onto the entities it is applied to. And it is this structure in which the classification, ordering and selection tasks are embedded. For this task, a variety of languages have been developed for expressing as best as possible such a structure. Each formal language is hence specialized in one type or another of categorization of structural elements and relations. And depending on the complexity of the task, many types of symbols may be required. For instance, constants and variables may refer to entities or elements in the domain, while predicates, modalities or operators may be used to refer to properties or relations in which these elements are involved. For instance, the formula F=ma categorizes a state of affairs where there exist dependency relations between what is represented or referred to by the symbols F, m and a. In fact, in the study of a phenomenon, there is a myriad of such relational structures to which science can be attentive. Graphically represented, such functional dependency relation is a mapping from a set A to a set B, where the elements of set A are projected onto set B, but where each element of set A determines only one element in B (Figure 7.2). The categorization role of a computationally oriented for­ mal model is therefore to offer formulas (equations, formulas, graphs, etc.) that unambiguously Figure 7.2  Mapping of a function between two sets. refer to such functional relations between elements designated by the symbols of the formula. This is more so in the context of a possible translation of a formal model into a computational model, where the only functional dependency relations that can be retained are the one that are computable.

 Formal Models in Science 99 To illustrate how a formal model expresses these relations, we will take two examples, one being arithmetic and the other being non-arithmetic. Our first example is a simple arithmetic formula that expresses the functional relation: the ‘square of a number’. This relation determines that for any number in the set of positive numbers, there can only be another number in a range set. In algebraic language, this particular functional relation is typically expressed using the following equation:12 (1) y=x2 or y=x·x But in algebra, there is also another way to express the same functional relation. For instance, it can also be written in set theory notation, which takes the form of a list of doublets, such as: (2) F (x, y): (1, 1), (2, 4), (3, 9) ... (n, n) In fact, in most formal models, there exist two important types of expressions by which functional relations can be expressed: intensional and extensional expressions. Let us again take the function of the square of a number. We have presented it in the previous paragraphs using two different types of expressions. The first equation form does not express a particular token functional relation, but a type of functional relation. And it is a general expression that can be instantiated if values are assigned to the variables. This implies conducting many hidden substitution operations. Such an expression form is called the ‘intensional’ representation of the function. It is what is usually understood as the classical form for expressing a functional relation. But (2) also expresses the same functional relation. Here, it is expressed by listing all the instances of the values of the functional relation. This mode of presentation of a function is called ‘extensional’. It expresses individual token relations. The generality is implicitly expressed by presenting the whole list or by a conventional three dots (‘. . .’) left for the reader to fill. Our second example illustrates another functional relation, but it is applied to the non-numerical domain. It has the same dependency relations and modes of expression. Thus, semanticists will consider that one lexical meaning of the English linguistic expression ‘SON of ’ expresses a functional biological relationship between two persons. It determines that, for any x who is the son of y, there is only one y for each x that is the father. In English, this y is a ‘father’.13 As in the first example, this functional relation SON can be expressed intensionally or extensionally. Intensionally, it will be formulated using a quantified predicate logical sentence such as ƎxƎy [SON of (xi, yi)]. In natural language, such logical expression will be translated into the sentence: a person is the son of another person. Extensionally, the formal expression can also be formulated by a list of doublets [John Smith, Peter Smith], [Robert Manning, Louis Manning], . . . [N1, N2]. It follows then that the language used in formal models can categorize a functional relation in two different ways – it can be expressed by both means (intensional and extensional). In an intensional expression, the procedure to find the value of the function

100

Computational Semiotics

is expressed symbolically by the operator sign ‘x’, while in an extensional expression, it is not expressed overtly. The values are just ‘given’ without explaining why they are so. This ultimately signifies that there are two epistemically equivalent means by which a structural–functional relation can be categorized. This type is of the utmost importance for formal models, for it ultimately says that a formal model may express functional relational structures by two radically different means: by some intensional expression (equation, formula, etc.), or by an extensional list of the relational data themselves. But among all the functional relations that exist in a phenomenon, there is an important but also very problematic one to which a formal model must be attentive to. It pertains to the model’s ability to express with more or less ease either the states of functional relations or the transitions between these states. Let us briefly explain in more details this important type of categorization in formal models. Some formal languages (logic, arithmetics, set theory etc.) will mainly focus on functional relations between discrete components. A typical example is a set-theoretical formula that expresses, among many other relations, the one of intersection or union of two sets of elements: B = B ∩ A. These types of languages are more apt for describing static structures. Some other languages (differential algebra, topology, etc) are better at focusing on functional relations between non discrete components or continuous states. These languages are more adapted for describing dynamical structures. They not only express states, but also the dynamics of the transition between states and even the emergence of states. A typical example is a differential equation that maps onto some moving object or process: a

v . t

But one must therefore be attentive to what is called a dynamical system. Indeed, there are dynamical languages that can be applied to structures that may be atomic or nonatomic components. For instance, for Lindenmayer (1968), languages easily express discontinuous transitions between structural states where the components are atomic or discontinuous. For example, a having the following rule: V = {a,b} P1: a->ab P2: b->ab where the symbols a and b represent lines a and b

 Formal Models in Science 101 will produce the following structural states and ultimately generate a figure of some sort:

Differential calculus, for its part, goes the opposite way. It expresses continuous transitions between structural states containing continuous, non-atomic components. A typical example is the highly complex topological algebraic equation for producing the Möbius strip. More complex examples are formal models that have used for instance geometrical or topological analysis. And they are to be found in practically all of the ‘natural sciences’. The models may be chaotic, catastrophic and so on. As one may see, epistemic categorization operations are important in formal models. They have a very complex role in knowledge acquisition. Categorization does not just identify isolated entities but integrates them into a web of relations that forms structures and into dynamic evolving and emergent states. It ultimately expresses these structures in languages allowing rigorous and complex manipulations that enhance knowledge. But as we know, categorization is not the sole epistemic role of formal models.

Reasoning Reasoning is the second main epistemic role of formal models. In the conceptual model, a reasoning process is seen as a cognitive process where some specific type of rules ensures or at least optimizes transitions between statements. In a formal model, it is seen also as a systematic rule-governed manipulation of symbols, sentences, formulas or equations by whatever cognitive or physical means it is effectively conducted.14 Take, for instance, the modus ponens rule in logical language. The form of this classical rule is: if P, and P ⇒ Q, therefore Q. This allows the transition from a statement P towards a statement Q given the inference that P implies Q. In algebra, transformation rules allow passing from the equation x+a=b to x=b-a, or from ax=b to x= b/a. In linguistics15 generative grammar rules allow the transition from the sentence NP + VP to ART + N + V + ADV and ultimately be given using vocabulary so as to form ‘the dog runs fast’.

102

Computational Semiotics

There are many such types of transition or transformation rules for formulas and equations.16 And each formal language will have its own set of rules. The important point we emphasize here is that all these ‘formal’ operations allow the creation or the generation of new symbols and formulas in a rule-constrained manner. When used in logic for building controlled arguments, they ensure, for example, demonstrations and proofs. In algebra, they generate solutions. Three main classes of formal reasoning processes are usually distinguished: deductive, inductive and abductive processes. We focus on these types of reasoning for they play an important role in computational models, mainly the one that is found in artificial intelligence models. These three main types of reasoning processes are known to the readers, but we will mainly expose here some not-so-evident epistemic effects that certain transition rules have in formal models. They create different relations between two main types of formula: formulas that have a generalization status and formulas that have more individual or existential status. Or to put it otherwise, they create a complex interplay between formula that express laws, general principles and the ones that express individual data or instances of a general formula. In a formal model, the first reasoning process – the deductive one, classically called hypothetico-deductive reasoning – is one where general or universal formulas are transformed into an existential or individuation formula This means that, by following some specific inferential rules17 (existential instantiation), a formula that expresses a general (universal) mathematical statement on a functional relation that contains variables can be transformed into other another statement specifying constants used to refer to some specific value of these variables. For example, take the statement: ∀x (P (x )) It is possible by an existential instantiation rule to pass to the statement (P (a)). In other words, the conclusion (P (a)) is a valid conclusion that is formally expressed by ∀x (P (x))→∀ (P (a)), where x is a variable and a is a constant. And the whole formula means that if one asserts the universal statement that all entities x in the domain chosen have the property P, then one can assert that there exists an individual x that has the property P. When this rule is specifically applied to functional relations, then we have something similar for a functional relation between two entities (x and y): ∀x∀y (F (x, y))→∀ (F (a, b)) where x and y are variables and a and b are constants designating entities within a domain, and where F refers to a functional relation. This reasoning is also at work in the following simple algebraic example: the general formula y = x2 is ultimately transformed into y = 32 by a substitution rule where x is substituted by a constant: 3. Our last example better shows the epistemic operation embedded in the application of the same rule. Take the formula: T  2 L / g This formula expresses a general function. Each symbol can be substituted by a specific constant which is taken from a set of values that are the results of observations concerning

 Formal Models in Science 103 (T), (g) and (L) of a certain object under observation. But as a physical science expert may have seen, the letters that are used surely have a reference to some number, but also they are abbreviations of three components of a well-known artefact: the rod of a pendulum in movement, where the letters L, g and T abbreviate the set of numbers that are the values of length, gravity and time. And the role of abbreviation is not the same as the referential role. The former relates to the conceptual model, while the latter pertains to the formal model. And the two must not be conflated in the explanation of the pendulum. In other words, the formal representation of a functional expression with variables can be replaced by a specific instance of its variables. The formula then expresses an individual instance of a universal or general formula. Still, both the general and individual expression are purely syntactic. They ultimately have to be interpreted with regard to the conceptual model. A similar example could be given for linguistics. Take the general formula S ⇒ NP + VP. By recursive application of substitution inferential rules18 and using other grammatical rules, some of which refer to the vocabulary of the language, it ultimately becomes A dog runs fast or to take Chomsky’s (1957) classic example: Colorless green ideas sleep furiously. The important points to retain here is that original general formulas can be transformed into other formulas expressing an individual instance. And in a formal model, it is these latter formulas which we relate to observations of individual cases. We can translate this deductive reasoning procedure into epistemic terms. It established a relation between, on the one side, formulas or equations expressing some general judgements or assertion and, on the other side, formula or equations expressing some individual, fact, event, state of affairs or just a unique observation. The first ones are often calls laws or regularities, the second ones are called the data. In other words, a deductive reasoning process in a formal model embeds many implicit relations between general statements and the data that are possible instances of these general statements. In formal models, general formulas mainly express functions in an intensional manner, whereas data formulas may express some functions in an extensional manner (e.g. as a list). But because data are mainly given within an experimental schema, they are not always seen as possible expressions of functional relations. Therefore, data are not seen as part of the formal models. They seem external to them. The second type  of reasoning, inductive reasoning or, as it is increasingly being called in computer science, data-driven reasoning, is a process where the inference rules go in the opposite direction of the deduction process. Here, the inference rule called the universal generalization formula allows the transition from formulas expressing individual instances or states of affairs of entities in a domain towards formula expressing general or universal states of affairs. In logical formulation, this universal generalization states, under certain conditions, that: ├ If P(a) & P(b). . . P(xi) then ├∀ x (P(xi)) where xi is a variable and a, b, c. . . z are constants designating all the elements or entities within a closed domain. In the case where a formula expresses a specific functional dependency relation between two entities expressed by constants, the universal generalization we have is: ├ If F(a, m) & F(b, n) . . . F(xi, yj) then ├∀xi ∀xj (F(xi, yj)) where xi and yi are variables and a, b, c. . . n, m are constants designating entities within a domain.

104

Computational Semiotics

In analytic geometry, we can also express these types of functional relations. Take for instance the following vectors that express these specific values: V1: < 2, 4, 4.47> V2: V3: These vectors can be transformed into a general functional relation H: H (b, s)  2 b2  s 2 This formula is just the generalization of the functional dependency existing between H and the relation between b and s. Each vector is the instanciation of this functional relation. More concretely, it expresses the diagonal length H of a rectangular sheet of paper having a base b and a side s! In linguistics, the distributional approach (Harris  1982) contains many such generalizations. After having observed n contextual cases and classified them, a general ‘functional’ rule will be generated. In other words, an inductive reasoning process uses formulas that express an individual state of affairs and value for the variable resulting from observation data. And if these observations are exhaustive, we can, by this inferential rule, generate a new formula that generalizes all such individual states of affairs. As in deductive reasoning, inductive reasoning establishes an intimate relation between two different types of formulas: a relation between a formula that expresses individual states of affairs and a formula that expresses a general state of affairs. Hence, the reasoning process embeds an implicit relation between data statements and laws or pattern-like statements. Many epistemologists have criticized the use of such reasoning processes because of the difficulty of applying them concretely in the context of science. Except in small and limited domains, it is impossible to exhaustively find all cases that warrant a universal statement. Despite this problem, this inductive reasoning process has become highly important in computer science which uses huge databases for which a universal generalization becomes acceptable. The third type of reasoning process is the abduction one. It mitigates the limits of deductive and inductive reasoning. It is a very pertinent reasoning process when induction and deduction cannot be used because it is impossible to access all instances ensuring the generation of a universal statement about the state of affairs. For example, it may be technically impossible to generate a general and universal statement such as all cookies in the world are chocolate cookies! But it may happen that, in some context, a general statement of this kind would be useful. For instance, imagine making a statement about cookies that come from some particular place. The argumentation may go as follows: 1 All cookies in the box are chocolate cookies. Still, a statement like: 2 These are chocolate cookies.

 Formal Models in Science 105 may not be formally inferred from general statement 1, for these cookies may come from elsewhere than the box. So, the correct conclusion could only be: 3 Possibly these cookies are from the box. Such a conclusion introduces possibility. In a more epistemic way, it is seen as a hypothesis or better as a ‘best explanation’. Peirce called this epistemic type of reasoning: ‘abduction’. Logicians see it as a sort of non-monotonic logic expressing defeasible reasoning.19 In this sense, abduction is not pure deductive reasoning. Conclusion 3 is not inferred from universal statement 1. Nor is it pure inductive reasoning, for there are not enough instances to infer a warranted universal statement. In epistemic terms, this reasoning is a means of justifying our beliefs more than a means of validating them (Adler 1994). Although such abductive or defeasible reasoning cannot produce logically valid statements and does not offer the epistemic certitude of induction or deduction, it is still quite used in science. In fact, it is seen as the basis of ‘retro-engineering’ or ‘retroreasoning’. And it is formally interpreted in the mathematical terms of probabilities and optimization. From an epistemic point of view, it allows an enquiry to create hypotheses, contingent explanations or beliefs which afterwards must be justified and if possible proven true. Today this type of reasoning is indeed quite alive in many scientific practices. Abduction is highly pertinent when, in a particular research situation, there are results and possible generalizations but no means to verify them. It turns these results into possible best explanations. Such a type of reasoning is employed in qualitative researches where it is seen as an important process for generating ideas (Muchielli 2007; Hallé and Garneau 2019). In contemporary sciences, it regularly assists the process of discovery and explanation (Magnani 2001). Here, in our semiotic research context, we can see abduction as a process navigating between some formula expressing individual data where it allows to produce some possible or implicit statement expressing a generality. Abductive reasoning is seen here as a means for navigating between general statements and individual statements – for example, between individual data and laws, patterns and principles. For us, the important point to remember is similar to the one we stressed regarding the deductive and inductive reasoning processes: a formal model that calls upon formulas for reasoning rests upon an implicit relation between some general statements and individual statements that are between data, laws, patterns and principles. From a pragmatic point of view, we may conclude that each type of reasoning process is a specific form of negotiation between these two particular types of formulas: data formulas and pattern formulas (Figure 7.3). This negotiation between these two types of formulas induces what is Figure 7.3  Formal reasoning as a two-way typically called an experimental model process.

106

Computational Semiotics

or experimental schema. Each experiment is a means of controlling the relations of individual statements with general statements. Hence, to be effective in a scientific enquiry, a formal model cannot be limited to just one type of formula. It must explicitly or implicitly activate some negotiation among many types of formulas. Depending on the complexity of the object under study, it may happen that some sciences will have to invest more time on the fine-tuning of the general statements than on the data that justifies them. This type of enquiry is often called ‘theory-driven research’. Other sciences will take more time on data collection then on finding the rules and patterns that govern them. These types of enquiries are often called ‘datadriven research’. Despite appearances, no science can be only theory-driven or data-driven. No model can present only one type of statement or another. The term experiment designates a knowledge framework where general and individual types of statements negotiate their interaction. A laboratory is a physical framework in which to accomplish this interaction. And it is often a place that collects particular data. But it is also where formal statements are tested in regard to data. Hence, formal models, on the one hand, show the general formula with overt or hidden variables that can be filled by data. And, on the other hand, they show individual data that are interesting only because they may be possibly generalized into some general formula. Confirmation and exploration, as well as justification and discovery, are entangled in this dynamic negotiation among formulas. Even though models are essential in a scientific enquiry, not all of the formulas they contain are necessarily true. And as many philosophers of science note, in one respect or another, they will be theory-laden and loaded with beliefs and biases. ‘In the case of theoretical [formal] models, denotation will more often than not be guided by background assumptions and theoretical frameworks, as model-building is typically part of a larger project of inquiry’ (Gelfert 2011: 275). But this does not impair the epistemic role of formal models. They are the most rigorous type of models a science can build. Formal models do not guarantee truth. Had this been the case, science would never have erred. But as scientific theories are seen here as a conglomerate of interwoven models, one of the main roles of formal models is to guarantee rigour in a process of enquiry. In this perspective, the epistemic role of a formal model is to enable a highly constrained dynamic manipulation of the formulas so as to diminish these external influences as much as possible.

The expressive and communicative roles of formal models Though it may not be as apparent as it is for conceptual models, formal models also have expressive and communicative roles. It is common to see the expressive role of models as being mainly embedded in natural language. But formal models are always expressed in languages that have their own specific syntactic, semantic and pragmatic parameters. At the syntax level, they contain symbols and compositions of symbols governed by rules.

 Formal Models in Science 107 One particular characteristic of their semantics is that no symbol can be semantically ambiguous. This would affect their consistency. Proofs would become useless. Symbols of formulas must refer to specific entities, relations or operations in a domain. And their pragmatics rest on intentions, objectives and even beliefs all embedded within conventions, standards and norms. For its part, the communicative role of a formal model is quite similar to that of conceptual models. It is used by members of a community for sharing all the information about what the formal language expresses, be it about the hypotheses, experiments, results or methodologies. Naturally, as mastering some formal languages often requires time and energy, they will not always be readily understood by the general public. This will give these languages their technical appearance, if not their esoteric one. They will seem like a foreign language to be translated. A nice example of this difficulty is found in linguistic theories. Even if these theories are about ‘language’, there is quite a cognitive barrier between the formal language of the generative Chomskyan grammar and the formal language of the categorial combinatorial Montague grammar. Mastering one of these formal languages does not entail mastering the other one. We could also mention the social institutions within which scientific communications are embedded, such as laboratories, international conferences, academic programmes, grant institutions, government agencies, scientific media and journals. All are social entities that impose particular norms and build specific scientific cultures, technè or artifactures (Latour and Woolgar [1979] 1986; Haslanger and Saul 2006; Bachimont 1996) by which formal models are communicated. One cannot deny that when these formal languages become so technical, they induce a power play among both the scientific community and the general public.

Critiques of formal models Despite the high epistemic stakes of formal models in science, they are not without their limits. And they have been the object of many critiques. Formal models have been at the foundation of the positivist and empiricist epistemological points of view on science. They appeared as the best parameters for controlling scientific methodologies mainly in their categorization and reasoning processes. But critics have constantly repeated that if formal systems are limited to mere syntax, they may indeed offer systematicity and rigour in categorization and reasoning, but ultimately, they will be without meaning. And as we may recall, understanding is left out of the process, as stated by Duhem: ‘If a physicist is given only an equation, he is not taught anything. . . . In physics, an equation, detached from the theory that leads to it, has no meaning’ (Duhem 1902, quoted in Ariew 2007). When we open this semantic door, the limits of formal models appear. The semantics used in formal models, even if external to the syntax, must itself be formal. But somewhere and somehow, the symbols and formula of this formal semantics must

108

Computational Semiotics

themselves relate to the world we understand. As Harnad (1990a) puts it, they must ultimately be grounded in our experience of this word, and translatable into non-formal surrogate semiotic forms as we, humans, understand such natural languages. The world is always what we talk about. It is not in the language itself. It has to be categorized and conceptualized first. A formal model is just a mediator for understanding this world. And it cannot be the only one. In other words, there is an intimate relation between formal models and conceptual models. Accepting this dual relation between formal models and conceptual models opens a door in line with pragmatic theories. They are models of which the interaction serves for the building of a scientific theory. If we have gone into some technical details regarding the role of formal models, it is to show that their application is not limited to the natural sciences. They indeed can play a role in other types of sciences used to investigate a given phenomenon. But more importantly, a scientific theory that aims at integrating computation into its enquiry must build formal models formally expressing the structures it endeavours to investigate. More so, it must choose structures where the relations can be expressed employing a computable formula. This is the main challenge of computational semiotics.

8

Formal models in semiotics

Formalizing in semiotics Because of the complexity involved in most semiotic research projects, it may take much energy and time to build the most adequate conceptual model for identifying as clearly as possible the many possible features of the object under study, for setting the research objectives and for designing the investigation methods. In semiotics, integrating a formal model into the research process becomes a project in its own right. Even in the digital humanities, where the computer is omnipresent, formal models are not the foremost concern. Computers are looked upon more often as toolboxes than as implementing formal and computational models. However, in some human sciences, formal models have indeed become an important companion in the conduct of research. Analysis can never be content to consider terms, but must, beyond the terms, grasp the relationships that give them their unity. These relationships alone are the object of analysis. (Lévi-Strauss 1973: 110, our translation)

But many of the formal models chosen are of a statistical nature. In addition, one finds more and more core mathematical models that are, for instance, algebraic, topological or geometric. In sociology,1 they may be used for describing and explaining communication, collective actions, social networks, social mobility or cultural trends. In linguistics, automata models2 and category theories3 have been called upon for describing grammars of languages. In psychology, learning4 has been modelled through linear and non-linear algebra. It is not as easy to say that formal models have often been explored in semiotics. The lack of attention to formal models has been one of the main criticisms of semiotics. In the late 2000s, Peer Bundgaard and Frederik Stjernfelt (2009) investigated the future challenges of semiotics by interviewing different semioticians. At the time, the conclusion was clear: semiotics would be renewed by the power of dominant structuralism. This paradigm was seen as a more rigorous approach than what was offered by classical semiotic theory. Yuri Lotman,5 much earlier, had emphasized the importance of mathematics as a natural tool for formalizing semiotics. And more recently, a survey conducted by Kalevi Kull and Ekaterina Velmezova (2014) of many prominent semioticians from various horizons highlighted once again the great lack of

110

Computational Semiotics

formal modelling in semiotic research. In the 1970s, the structuralist hypothesis had brought hope for some formalization. Unfortunately, the actual practices did not live up to expectations. Instead, semiotics continued to be analytical and conceptual. In Kull and Velmezova’s survey, Mihai Nadin summarized this by saying that it ‘became speculative’. And a similar echo came from many computer scientists. For them, semiotics had not attained a high degree of formalization similar to the one artificial intelligence had developed for the manipulation of symbols. This is harsh criticism. But it is not that true. Indeed, the search for formal rigour has always been present in semiotics. Indeed, many examples illustrate the quest for adequate formalization. Charles S. Peirce, as we know, had identified semiotics with logic. The existential graphs were his creation. Robert Marty has shown that many Peircean categories could be translated into lattice algebra. The structuralist movement itself emerged from the Russian ‘formalist’ linguistic school. Louis Hjelmslev saw the semiotic relationship between substance and form, and between content and expression as a functional relationship. The semiotic school of Charles Morris defined these semiotic relations through Fregean-Carnapian propositional, predicate and modal logic. This invited a specific manner of studying language and some of the most complex forms were offered by the applicative grammar of Sebastian Shaumyan and Jean-Pierre Desclés. Claude Lévi-Strauss used algebraic models while Joachim and Michael Lambek used categorical grammars to describe kinship relationships. A. J. Greimas proposed a logical square for his actantial model. It is even said that Jacques Lacan and Roland Barthes proposed to ‘mathematize’ their objects of study – psychoanalysis and fashion – but without much success. Nowakowska (1986) and H.-H. Lieb (1979) designed a general logical model for the composition of signs. Jaakko Hintikka worked in semiotics within the horizon of modal logic, while Petöfi (1978) and Coseriu (2007) explored formal models for narrative structures. And in the 1990s, the formalizations used more mathematical models. For instance, Rieger used various multivariate algebraic classifiers for discovering semantic structures in texts. Gudwin and Gomide (1997a) and Loula, Gudwin and Queiroz (2004) have used mathematical emergent auto-associative, self-organizing models for explaining symbolic communication in robots. René Thom (1988), Per Aage Brandt (2020) and Wolfgang Wildgen (1982) drew inspiration from topology and catastrophe dynamic systems (at the time) to explain the morphogenesis of meaning. In the same way, David Piotrowski and Y.-M. Visetti (2014) and D. Piotrowski (2017) explored this formal diagrammatic modelling for semiolinguistics. Despite the rigour embedded in the use of these models, many semioticians found them to be too reminiscent of the positivist empiricist scientific endeavour. And their main criticism was that formal models lack the interpretative dynamic inherent to all semiotic analyses. Semiotics is, before anything else, a hermeneutic engagement with meaning. But in our view, this judgement must be nuanced, because it is based on an overly narrow conception of mathematical and computational modelling. One could reverse the position and see that formal modelling is in itself a particular form of interpretation and that with other forms of modelling, it can contribute to the understanding of even the most complex semiotic realities. To do this, we must better understand the nature of the complexity of the interpretive filters which are these formal models. But the

 Formal Models in Semiotics 111 most important reason for exploring formal models in semiotics is that they are a sine qua non-condition for computational semiotics. And this will be its most significant challenge. If formal semiotic models cannot be built or used, no computational models will be possible and computers will be nothing but toolboxes – which would make the computational semiotician into no more than a user or peddler of these toolboxes.

Conceptual frameworks and formal models in semiotics Formal models do not jump by themselves into research enquiries. They are complex epistemic artefacts. And in computational types of semiotic enquiry, they are constrained first by the computational models and secondly by the conceptual model in which they are embedded. First, the choice of a formal model is constrained by the computational model into which it will be translated. We could imagine research that has defined interesting and relevant formal models applicable to a semiotic artefact. Unfortunately, it is not because a model is formal that it can automatically be translated into a computational model. For instance, we saw in the field of artificial intelligence the existence of many formal models for analysing semiotic artefacts or processes such as natural language dialogues, translation or image processing. But it did not follow that algorithms were found for computing these formal models. Some of them may have encountered internal complexity problems.6 Or they may have required a heavy load of hardware technologies directly or indirectly recruited for the execution of computation tasks. In other words, not all formal models guarantee computation, and even though it may be possible, there may not exist an effective technology for achieving such computation. Second, in regard to the conceptual model, what is formalized is inherently related to how the semiotic object under study is conceptualized. In some sense, a conceptual model is the ultimate benchmark of the validity of a formal model. In other words, a formal model must be in tune with the problem identified in the conceptual model, for it will ultimately be evaluated and understood by it. And as we have seen, there are various conceptual frameworks involved in the identification of the semiotic problem. And this heavily constrains a formal model in expressing the various dimensions that a conceptual model put forward. Take for example the philosophical conceptual framework. The main thesis of its three main variants (mentalist-representationalist, pragmaticist and hermeneuticphenomenalist) could be briefly summarized in the following statement: a semiotic process is one where somebody interprets something as standing in some manner for something else. Naturally, each variant has its own description and explanation for these entities and relations. We could venture a first formal expression of this conceptual model as: Semiosis is an epistemic operation O (interpretation, etc.) that is realized by some agent U (mind) where a semiotic artefact X (a type of sign: symbol, icon, index) in some ‘stand for’ relation R (‘sense’, interpretant), under some intention I to some object Y (reference) in a world W (possible or actual) for somebody D (other minds, epistemic community, culture, etc.).

112

Computational Semiotics

For its part, the logical linguistic and semiotic structuralist conceptual framework focuses mainly on the embedment of meaning within the structure of the semiotic artefacts or processes. Its main thesis could be summarized by the following semiformal sentence: Semiosis is an operation O that is realized by some agent U that uses a semiotic artefact X (a type of sign: symbol, icon, index) possessing a multi-type structure S in some ‘stand for’ relation R (signified, reference, etc.) to some object Y (physical, mental, etc.) in a world W (other signs) for somebody D (individual or society). Finally, the naturalist semiotic conceptual framework – the cognitive or biological one – focuses on the methodological points of view. Semiotic artefacts and processes should be approached in regard to the dynamic or their formation and they should be studied by the means of natural science observation and experimentation. For the cognitively oriented framework, attention should be put on perception, categorization, communication and on the social environment. For the more biologically oriented ones, the focus should take into account complexity, adaptation, evolution and so on. Its main thesis could be summarized by the following semi-formal sentence: Semiosis is an informative operation O (cognitive, informational, communicative, etc.) that is realized by some agent U (person, animal, cell, etc.), where a semiotic artefact X (a type of sign: symbol, icon, index) under a set of conditions C (context, situation, environment, instrument, etc.) is in some ‘stand for’ relation R (representation, surrogate, mediation, etc.) to some object Y (physical, semiotic, mental) for somebody D. As we may see, these semi-formal translations are constrained by the various dimensions or components of particular conceptual frameworks. Each one highlights some specific features, properties and relations of semiotic artefacts and processes which a formal model should address. Summarizing by means of an even more general statement, we would say that: Semiosis S is an operation O, by which agents U use or recognize that some complex and structured artefact X is in a ‘stands for’ relation R, under some intention I, contextual conditions C and environment W, for something else Y and for somebody D. These components are just a sample of many others that could be added by either of the variants of existing conceptual frameworks. And some new components could be added after some explorative research result. In this perspective, formal models are not independent of the conceptual frameworks – they are in a sense indebted to them. Formal semiotic models must ultimately be relevant to these conceptual models. Often, in fact, these formal models will pinpoint some particular dimension of these more general semiotic frameworks.

Types of formal models for semiotics The understanding of the role of formal models in semiotics invites us to briefly recall the definition of the surrogate tools that are formal symbolic systems or languages. This definition does not change when applied in semiotics. In syntactic terms, a formal language is a set of rules governing the manipulation of symbols.7 Or expressed more

 Formal Models in Semiotics 113 formally, a formal system is a set of two sets  FS = . The first set S contains symbols and the second F contains systematic operations on S. We must remember that such a system by itself has no semantics. If one is to be given, it must be added externally. And it must define what symbols and operations ‘stand for’, what they ‘refer to’. Ideally, such a reference will itself be defined formally. But often, because of the complexity of both the system and what it refers to, the semantics is often expressed in a semi-formal way or is assumed to be known by all members of the epistemic community that uses it. When it is semi-formal, the reference is made by associating the symbols to some natural language terms whose reference is supposedly known to the users. For instance, in our semi-formal translation given above for philosophical types of models, the symbol U is associated to agents, X to artefacts, C to contexts, I to intentions and so on. In other words, in this case, each symbol in a formal language is associated with some expression of a natural language which in turn must refer to some entity in a world. If we accept the Bourbakian semantic thesis that mathematics is about structure, then we can understand that each mathematical language will express one type of structure or another. A set-theoretical language or a logical language does not express the same structure as a geometric mathematical language. For instance, the set-theoretical language will express structures such as inclusion and intersection while some linear algebraic structures may prefer operations on vectors, matrices, etc. In practice, there is no one formal language that can express all possible types of possible structures. It is precisely here that various enquiry will have to choose the type of formal language that can be mapped onto the structures they aim to explore. And this is one of the core challenges for semiotics. What kind of formal system is the most adequate for exploring the many types of semiotic artefacts and processes it studies? For example, to formalize linguistic structures, some have preferred generative type grammars, while others have preferred categorial ones. On the other hand, for cultural artefacts, some may come to choose specific graph languages which may be useful for navigating between events or states. Topology may even be useful for understanding the overlapping lexical meanings. If semiotics is to integrate computation into its enquiry, it will be necessary that its formal model be constrained by what computer models can include. Still, the formal computational model must be adequate for expressing the core components and relations that are explored by the semiotic conceptual framework. On the other hand, they must choose only the types of relations that can be translated into algorithms and implemented into computers. As space does not allow us to go deep into this matter, we will offer an overview of the most pertinent types of formal languages that appear relevant to the semiotic conceptual frameworks and that appear admissible for some possible computational translation and computer processing. Our objective here is to present these formal languages in the most pedagogical way so that semioticians may see the core formal properties as being pertinent for their own research. Still, we also hope that computer scientists and mathematicians will more clearly see how the formal models they manipulate so well can be used for studying semiotic artefacts and processes. For both of these reasons, our presentation thereafter will be informal. It is an informal presentation of formal languages!

114

Computational Semiotics

The epistemic role of formal models in semiotics Like other types of models in science, formal semiotic models participate in knowledge acquisition and communication. They are surrogates or mediators for explicating and explaining the components and relations of semiotic artefacts and processes. The two main epistemic roles of categorization and reasoning are also at work here. But as we know, in a formal model these epistemic operations are embedded in the formal language by which they are expressed and by which they are realized. And this is eminently true in the case of semiotic artefacts and process. Naturally, this puts a toll on explaining these two main epistemic operations to non-experts. So, presenting them in a formal way does not seem to be the best way to go here. We shall rather proceed by examples from which we can abstract some general understanding of the role of categorization and reasoning in a formal semiotic model.

Categorization Let us start with a simple example of a small set of semiotic artefacts on which we apply different types of categorization tasks through some formal languages. This example is a set of the most famous among the huge statues of the world (Figure 8.1).

Figure 8.1  Eight world-famous statues.

 Formal Models in Semiotics 115 In this example, the categorization8 process may be understood as the result of some classification operation that applies features to some entities which in this case are attributed to each statue of the set. These features are expressed by using sentences such as ‘The Thinker is a statue of a male figure’, ‘Christ the Redeemer is a religious statue’. These formulas will then be translated into some symbolic formula of a formal language. In scientific enquiry, it is the role of the conceptual model to offer a first glimpse of the categorization of these statues. Each conceptual framework could for instance focus on the agents (U and D), the artefact (X), the structure (S) of the representation relation (R), the intention (I), the context (C), etc. But these features must always be chosen in regard to their semiotic role, that is, their ‘stand for’ relation (R). In this example, we will limit ourselves to just a few of the features of these statues that are pertinent to their structure S. And for this, we will only retain as pertinent features pertaining to their creator, materiality, size, gender, iconicity and texture. Many more semiotically pertinent features could be called upon, for instance on the basis of their context, situation, entailed emotion, political value, etc. The ones we have chosen for illustration purposes can be, for each statue, expressed using the following natural language sentences: (1) Michelangelo’s David is a huge-size marble sculpture of a nude male. (2) Rodin’s Thinker is a medium-size bronze sculpture of a sitting nude male who is in a thinking position. (3) The Motherland Calls is a huge-size bronze statue of a woman holding a sword in a defiant way. And so forth for the other statues. Once this categorization is done, the features can be translated into symbols of a particular formal language’s vocabulary and be embed into some relational structure. For instance, the features of each statue could be mapped onto basic relations of size, similarity or opposition, but if more features have here been given, more complex relational structures could also be mapped on them.

Types of formal languages for semiotic categorization For semiotic enquiries, there are different types of formal languages that can and have been used for achieving this categorization and structural mapping. What distinguishes them is not mainly the set of symbols they use but the types of mapping they perform on the features, properties and relations of whatever it is they model. Some focus more on the properties and features, while others focus more on different structuring relations that can be mapped onto them. Despite the wide differences in the symbols and operators they use, each formal language has its own signature. When presenting some formal modelling languages here, we will mainly aim at showing their similarities and differences in terms of the mapping they all perform. The objective is to reveal as clearly as possible the role formal models may have for useful categorization in computational semiotics.

116

Computational Semiotics

Grammar-ruled languages for formal semiotic models Our first types of languages are set-theoretical ones. Defined in intuitive terms, these languages are defined as two sets . The syntax builds formula out of symbols S, while the operators O are for operations on symbols and they ultimately produce a formula such as the following: A = {a, b, c}, or m = {(A∪B), (A∩B)}. For their semantics, some symbols map onto collections or sets, whereas some others map onto elements that are usually interpreted as discrete entities of various sorts, such as people, numbers, places or animals and so forth. The operators map onto discrete relations of inclusion, non-inclusion, intersection, disjunction and so on. In classical set theories, these relations are usually not continuous or fuzzy relations. For example, a formal set-theoretical sentence such as European countries = {France, Germany, Italy} is interpreted as a discrete set that includes discrete entities. Our statues example easily illustrates the use of this type of set-theoretical language. Here, each symbol (i.e. natural language term) represents a specific statue or some specific features. And the operators {, } express the inclusion of features in a set. Here are a few formal set-theoretical expressions for our statues example: ●● ●●

male = {David, Christ the Redeemer, Buddha, Moai, Thinker} female = {The Motherland Calls, Liberty}

This can also be expressed using a different notation: ●●

●●

David ∈ male, Christ the Redeemer ∈ male, Buddha ∈ male, Moai ∈ male, Thinker ∈ male The Motherland Calls ∈ female, Liberty ∈ female

We can invert the description and describe each statue by its proper features: ●● ●● ●●

David = {male, human, young, HH>4m} Sphinx = {male, human, huge size, mythical, animal} Thinker = {male, human, medium size}

and so forth. And on these sets, all classical set-theoretical operations can be applied. For example: ●● ●●

Motherland ∪ Liberty = {female, human, HH>4m} human = male ∪ female = {David, Christ the Redeemer9, Moai, Buddha, Thinker, Liberty, Motherland}

One important characteristic of this set-theoretical language is that because each member of a set is a discrete element, it can be mapped onto a finite set of positive integers which means that it can be counted. And this facilitates the use of the model built with this language and its translation into a computational model. To deal with continuous elements, another more sophisticated type of formal language will be necessary, for instance, topology.

 Formal Models in Semiotics 117 A second type of formal model is the logical one. Logical languages are also defined by sets of symbols S and operators O. The symbols are of a different type: constants, variables, predicates, relations, quantifiers, etc. And the operators define the formula as functions applied to arguments. The syntax allows the formation of the following: (P (x), (Q (x)), ((P (x) v Q (x)) ⇒ Q (x)) The semantics of the predicate symbols map onto properties while constants and variables map onto individual entities that possess these properties. Other types of symbols will map some specific relations such as conjunctions or disjunctions or will map the quantifiers. Applied to our statues example, this language delivers the following formulas: ●●

●●

∃x (male [x] & human [x] & religious [x] & HH>[x]) Where x = David10 ∃x (male [x] & human [x] & mythic [x] & animal [x]) Where x = Sphinx

and so forth for the other statues. In semiotics, there is a famous example of the use of this type of formal logical language. But it usually takes the propositional logical form. Here is its expression: (¬P & Q) v (P & ¬Q) v (Q & ¬Q) v (P & ¬P ) This formula expresses a semiotic phenomenon that can be in either two states described by the two discrete and atomic symbols P and Q. Some readers may recognize that this formula, often called the logical-square was used by Jakobson (1971) to express the opposition of phonological components. Greimas (1979) applied it to narrative structure, while Lévi-Strauss (1973) applied it to myths. Petitot (1977, 1988) gave a topological interpretation of it and applied it to actantial linguistic structure. But often, in these researches, this formula, rather than being presented in a logical propositional language, was translated into a diagrammaticiconic form called the semiotic square (Figure 8.2).

Figure 8.2  The semiotic square.

118

Computational Semiotics

Important variants of these logical formal languages are the many combinatorial and applicative type grammars. These languages present a similar functional grammar, but where formulas are built with functional operations applied to their arguments or operands. Classical semantics for these languages map symbols onto some set-theoretical structures where the elements are discrete entities and are included in discrete sets. But more and more of these combinatorial languages are interpreted according to continuous topology (Björner 1995; Desclés 1989a, 2003). And combinatorial approaches seem heuristic for dealing with multi-level semiotic artefacts and communicative processes. In semiotics, Desclés (1990) applied a topological and cognitive interpretation to the semantics of natural language, while Lambek and Lambek (1981) applied it to parental relations. Marty (1990) explored some combinatorial analyses of Peirce’s definition of sign. We have shown (Meunier 1998) that this formal language can be applied to traffic circulation signals where shapes, icons and colours are mainly superimposed onto one another. In this language, we could produce the formal following formulas: S1: S2: S3: or S4: (Figure 8.3)

Figure 8.3  Traffic panels.

Our third type of formal models comes in the form of Chomskyan generative grammars. Their syntax is equivalent to the preceding ones and is in continuity with the preceding logical languages. These languages are defined by a set . They contain terminal and non-terminal symbols and the operators come in the forms of a grammar or of sets of rules for the construction of sentences that are sequences of symbols. Their semantics is complex. It has been given by some ‘deep’ logical form where the symbols and sentences map onto entities, relations and states in some actual or possible world. Applied to a natural language, such grammars may generate the following well-formed English sentence: ((Theart cat N )NP (isV (onadv (theart mat N )NP )VP )). We can apply this language for formally describing the statues of our example, through a set of rules expressed in Figure 8.4 using a tree graph.

 Formal Models in Semiotics 119

Figure 8.4  Diagrammatic tree of the features of the statues.

We can also generate descriptions for our statues: The David statue may be described formally by: ●●

●●

(S (GENDER (MALE)) (BIO-STAT (HUMAN)) (REL-STAT (RELIGIOUS)) (SIZE (MH))) While the Thinker’s description may be described as: (S (GENDER (MALE)) (BIO-STAT (HUMAN)) (REL-STAT (RELIGIOUS)) (SIZE (MH))) and so forth for the other statues.

Some researchers close to semiotics (van Dijk 1972; Pavel 1976) have explored this type of rule modelling for describing the semantic structure of discourse, narration, argumentation and rhetoric in texts. Our fourth type of formal models comes from dynamic systems formal modelling, such as Lindenmayer’s (1968) symbolic L-systems. These languages are also defined by a set . But here, the symbols may be letters, digits or graphical signs such as dots, lines or others. The operators represent a variety of operations (concatenation, superposition, etc.) that can be applied recursively on these symbols. And in each case, a new configuration can be applied. This recursive deployment makes it possible to introduce dynamicity and allows the description of processes. Their semantics maps their symbols onto discrete and continuous entities and relations. Here are samples of the line structure (the Koch curves) generated by such an L-system grammar. This type of language can build grammar that generates complex structures such as the following:  V = {F,+, −} Ω = F->F+F P1: F+F—F+F The following ones produce a figure called a Koch curve. Some see these configurations as models for certain bacteria or even snowflakes.

Figure 8.5  Generated Koch curves.

120

Computational Semiotics

Other more complex grammars generate a set of configurations called fractals. The Barnsley fern is one of the classical prototypes of a dynamic fractal. But although not working directly in the field of semiotics but in one close to it, Dormans and Leijnen (2013: 8) applied these L-systems to the narrative structures of games. They represent mission narratives in graphs like the ones in Figure 8.6.

Figure 8.6  Graph of a game narrative.

It is difficult to apply these languages to our statues example where each one is very different. Each statue cannot be seen as a variation of another one. These four statues do not seem to instantiate a grammar. And applying to them some grammar or L-system rules would be strange. But it would be heuristic to apply an L-System to the Motherland Calls statue and to Delacroix’s Liberté, to the Statue of Liberty and to the Motherland monument in Kiev (Figure 8.7).

Figure 8.7  Four motherland types.

 Formal Models in Semiotics 121 We could imagine some 3D printer that could produce statuettes that are generated by such a type of L-system grammar. One of the main reasons for presenting these L-systems is their aptness for rewriting rules used in computer programming. This makes their translation into algorithms much easier. More importantly, under some conditions, all the other preceding ones can be translated into L-systems. In fact, the set-theoretical, logical, generative and combinatorial languages can all be translated into L-systems. One last variant of these L-system formal languages which is quite popular in artificial intelligence is the frame or list type of formal modelling. Though at first sight, this tabular type of list does not seem to be a formal language, it is still a rule-governed language. What is confusing is that the formalization contains natural language expressions and graphic structures. And the rules are not always transparent. But a small attention to the format shows that these are not natural language sentences but real formal tabular presentations of symbols. And the operators are hidden in the graphic forms (line, square, indentation, etc.). In information management, they are formalized as a table called a ‘data frame’, which is a graphic instantiation of basic predicate expressions with variables. In artificial intelligence, they form one of many variants of knowledge representations using formal models. Many others could be presented here, such as, for instance, the Sowa (1976, 2019) conceptual graph. A list is an accepted visual representation of a relation; it is similar to the concept of a row (Table 8.1). Table 8.1  List presentation of statues’ features

Statue ()

General categories

David

Sphinx

Gender Male (-) Female (-)

Gender Male (-)

Gender Male (+)

Social role Warrior (-)

Social role nil

Social role nil

Bio status Human (-) Animal (-)

Bio status Human

Bio status Animal

Spirit status Mythical (-) Religious (-)

Spirit status Religious

Spirit status Mythical

Size HH > 4m (-) MH < 4m (-)

Size HH > 4m

Size HH > 4m

etc.

Computational Semiotics

122

Before we move on to other types of formal models, let’s summarize some of the important characteristics of these formal language types for semiotics. One of our aims in presenting these formal systems or languages was to informally show that semiotic modelling can be formal without being quantitative. This is important for a computational semiotics research program. It shows that using a computer in semiotic formal modelling does imply manipulating quantities. A second dimension is that an identical description of pertinent contents of artefacts and processes can be expressed in various formal languages. Still, not all of them will be cognitively equivalent. The symbols and operators are not identical and therefore their manipulation by researchers is not always cognitively easy and transparent. For instance, formalizing traffic panels in a set-theoretical language or in a logical combinatorial language is not easy to master and understanding in a list frame or table formalism. But even if they are different on the surface, many of these formal languages can be translated into one another (under certain conditions) because, as it is known, many of these formal languages are equivalent to some other formal computational language such as automata languages, recursive languages, etc. And one important last point is that each of these languages has specific constraints on their semantics. They require their ontology to be about either discrete or continuous entities and relations. Choosing one or the other of these languages may not be adequate for all semiotic artefacts and processes.

Mathematics-like languages for formal semiotic models Let’s now present another set of formal languages whose focus will be more upon dynamic processes than upon components or semiotic artefacts and processes. So, we now delve into a more mathematical type of formal models and into a vector algebra language. Just as for the other formal languages, this one is also defined by a set of symbols S and operators O, that is, . The symbols map onto entities and the operators onto relations. To illustrate this usage, let’s return to our statues. For reasons of simplicity, we once more use the same set of features, but we present them in the form of a list, frame or table very close to the preceding one (Table 8.2). Table 8.2  Data sheet representation of the statues’ features Social Gender

role

Biological status

Spiritual status

Size

Symbol Name

Male

Female Warrior Human Animal Mythic Religious MH4m

d c s b mc

Yes Yes Yes Yes No

No No No No Yes

No No No No Yes

Yes Yes No Yes Yes

No No Yes No No

No No Yes No No

Yes Yes No Yes No

No No No No No

Yes Yes Yes Yes Yes

Yes Yes No

No No Yes

No No No

Yes Yes Yes

No No No

Yes No Yes

No No No

No Yes No

Yes No Yes

ma t l

David Christ-R Sphinx Buddha Motherland Calls Maoi Thinker Liberty

 Formal Models in Semiotics 123 This table contains the same semiotic features and relations expressed through the preceding formal models. Only natural language terms are used as symbols. We can read the lines, such as the Sphinx one, in the following way: Yes, it is the case that the Sphinx has the features of Male, Animal and Mythic and the Size HH>4m.

But we could reverse the reading and produce the following sentence for the feature Male: Yes, it is the case that the feature of Male is attributed to David, Christ the Redeemer, Sphinx, Moai and Thinker.

Let’s transform these tables into one that expresses the same semiotic descriptions but where yes and no are symbolized by the numerical symbols 1 and 0. These symbols do not yet mean a quantity here. They are just a translation of yes/no symbols (Table 8.3). Table 8.3  Vector/Matrix representation of the features of the statues

David Christ-R Sphinx Buddha Motherland Calls Maoi Thinker Liberty

 d  c  s  b ur m  o  t  l

Μ

F

W

Η

A

Μ

R

H4

1

0

0

1

0

0

1

0

1

1

0

0

0

0

0

1

0

1

1

0

0

1

1

1

0

0

1

1

0

0

0

0

0

1

0

1

0

1

1

0

0

0

0

0

1

1

0

0

0

0

1

0

0

1

1

0

0

1

0

0

0

1

0

0

1

0

0

0

1

0

0

1

If all lines are translated into a vector, we then have the whole table or matrix. And each line and column can be written using the following vector algebra expressions: Vi  [V1 , V2 , Vn 1 , Vn ]  (V1 , V2 , Vn 1 , Vn )  V1   V1       V2   V2  Vi              Vn 1   Vn 1   Vn   Vn     

124

Computational Semiotics

Each line and column of the table instantiates the values of this algebraic notation. Let’s just focus on the David and Moai lines and on the Male and Human columns. They can be written in vector algebra form as: Line David d: ; line Moai o: ; Column Male M: ; column Human H: . And the whole matrix can be transformed into an n-dimensional Cartesian vector space and graphically represented in a two dimensional space (each letter is the initial of the name of a statue) (Figure 8.8).

Figure 8.8  Vector representation of Pipe and Amants.

We can apply some basic calculus to our statues example: take, for example, the vectors d (David) and t (Thinker) representing the statues (or inversely, the vector representing the feature vector (column)). We can compare any two vectors u and v, by measuring the cosine of the angle between them: Cos(u, v)  u.v  (u1v1   umvn ) Interpreted in the world of statues, it shows, for instance, that David d and The Thinker t vectors have a small cosine angle. This means that both statues share many features. They are similar in their features. But each of these vectors d and t have a large cosine angle with the s vector. This means that the David and The Thinker statues do not share many features with the statue Sphynx s. And for the columns, the same calculus shows that the Male M vector is quite similar to the Human H vector.

 Formal Models in Semiotics 125 This calculus is simple, and in fact the reader could by just reading the same vectors in the matrix or the Cartesian coordinate plane intuitively discover the closeness and distance between the statue and feature vectors. It is easy to arrive at this conclusion in this example, because there are just a few features and statues. But when the number of vectors and their variables are in the thousands, the intuitive analysis is a different task. Finding just two similar vectors in a ‘big’ matrix is practically impossible. By hand, it is theoretically possible to calculate all of the angles, but that might take a few weeks to complete. Using a computer, it will just take a few seconds. For purposes of illustration, we have chosen for this particular formal language a very basic similarity calculus between vectors. But more and more complex mathematical algebraic ones are available. All can be applied to this matrix. In fact, vectors can be compared, added, multiplied, classified, reduced, differentiated, etc. Angles and surfaces for instance can be calculated. Linear and non-linear algebra can be called upon. Some types of calculus will focus on the discreteness of the entities and relations. Others will focus on continuity and temporality. For semiotic enquiries, there exist many very popular and useful mathematical pattern-recognition strategies that can be used. These types of formal models are very useful for classifying vectors, and they produce classes of either vectors or features. They are based on various statistical, probabilistic and clustering methods such as K-nearest neighbours, factorial analysis, principal component analysis, naive Bayesian clustering, discriminant analysis and many more. And they are easily translatable into algorithms. When applied to our statues example, these formal models reveal classes of similar statues – that of human statues, of animal statues, etc. But often, these models reveal classes, clusters or regularities that humans themselves could not discover even by an attentive search or by using other types of formal languages. More interesting for the various conceptual frameworks we have presented earlier is the possibility to include in the ‘features’ variables of a vector not just phenomenologically observable features such as male or female, but a host of features that belong to the various components that are typically retained in the various conceptual frameworks we have summarized in the preceding paragraphs. Hence, the variables could specify in minute details features of the agent U, the intentions I, the representational form R and, more importantly, the sociocultural context C. For instance, if our sample of the statues were to contain 100 statues and 100 features related to each type of component, maybe we could discover what, in this corpus, are some of the important characteristics of warrior or male statues. And if some features attributed to the statues were semiotically relevant for these components, patterns could be discovered and semiotic spaces would start to emerge. Some hypotheses on meaning would appear. It is also possible to use non-linear algebra and call upon dynamic formal models to dig into such a huge matrix. It may not be very useful for our present sampling of statues, but it may be very heuristic for semiotic artefacts or processes where the features of the artefacts present discontinuities such as is the case of many literary narratives, of cultural events such as ceremonies, of communication environments such as social networks, of semiotic behaviours such as dances, of physical signals such

126

Computational Semiotics

as music, etc. If certain complex non-linear equations can be mapped onto hurricanes which are complex and multidimensional, nothing prevents these calculi from being applied to complex semiotic phenomena presenting significant and relevant features. Still, the main challenge with all these formal models lies in what we have called here the significant and the relevant features. It is indeed possible that some research may require other types of features than the ones that a conceptual model has suggested. These features may even depend on the type of formal model chosen. Semiotic modeling . . . is to find and employ representational formats and processing algorithms which do not prematurely decide and delimit the range of semiotically relevant entities, their representational formats and procedural modes of processing. One of [their] advantages . . . would be that the entities considered relevant would not need to be defined prior to model construction but should emerge from the very processing which the model simulates or is able to enact. (Rieger 1999b: 97)

Finally, let’s recall the main purpose of our presentation of these many formal languages. All can be used in the categorization process we have when, by some cognitive means, we attribute some properties or make a categorial judgement on some reality and ultimately express it through sentences or formulas that associate a category to it. It is the role of a formal language to express the categorization in the control sets of symbols and operators. All the formal languages we have presented managed to express this categorization. For instance, our first logical language used symbols and operators to produce the formula P(x) & Q(x). This formula expresses the categorization that some entity x has a property P and the property Q. But by presenting many of them, we also showed that the same semiotic and semantic content could be expressed through very different formal languages. The real difference is in the types and varieties of symbols and operators that they include. Some are rich and diversified in their symbols and operators. Some can only map onto discrete entities, others onto continuous entities or dynamic changes. The challenge for semiotics is in finding the most adequate formal model for its enquiries. But, in this respect, the vector algebra formal language stands out. It allows a host of geometric analytic operations and a multitude of rigorous inferences and conclusions. As a simple example, in a logical language, it is theoretically possible but practically very difficult to find which two statues are similar in their features, while it is much easier in a vector algebra model. And because this formal language is easily translatable into algorithms, it opens up many heuristic possibilities for computational semiotics.

Reasoning In formal modelling, categorization is highly important, though it is not the only process of importance to participate in scientific knowledge. Reasoning is the other one. And categorizing alone is quite useless if reasoning is not applied to it. And as we

 Formal Models in Semiotics 127 shall briefly see, some formal languages are more or less adequate while others are better equipped for supporting some processes of reasoning upon what has been categorized. As a brief reminder, three main types of reasoning processes are typically used in scientific practices: deductive, inductive and abductive processes. Their general architecture or structure is a play between formulas that have either a universal or an individual epistemic status. And all the various types of formal languages we have presented express these epistemic statuses by some means or another. In deductive reasoning, a universal or general statement is asserted and from it, some individual or existential statement is formally deduced. In inductive reasoning, the reasoning process goes in the opposite direction, that is, from individual existential statements towards the more general or universal ones. And abductive reasoning oscillates between the two types. All the formal languages we have presented above can express the two main types of statements. It does happen that these two types of statements are not always transparent or easy to identify. Some formal languages use overt symbols such as the existential quantifier symbol ∃ or the universal quantifier ∀, while other languages hide these quantifying operators in some accepted convention or in ‘silent’ or non-explicit written symbols such as their position in a table or matrix. Faithful to our pedagogical approach, we will not formally redefine these three reasoning processes here. Instead, we will illustrate them using our sampling of famous statues from which we will then derive some of the relevant characteristics for reasoning in computer-assisted semiotic enquiries. So, let’s look at how some of our formal languages carry out either deductive, inductive or abductive reasoning. As we shall see, the conceptual understanding of the reasoning processes is quite simple. But the effective procedures to realize them is not that transparent. Some language formats better support one type of reasoning process than they do the others. Take for example deductive reasoning realized in a predicate logic model applied to our statues example. Here, the reasoning process is transparent: it usually starts by asserting a universal statement such as: 1) General rule: ∀x (Male (x) v Female (x)) & (Human (x) v Animal (x)) & (Religious (x) v Mythical (x)) & (MH4m)) Afterwards, by inference rules (existential instantiation and disjunction), an existential formula can be generated: 2) ∃x (((Male (x) v Female (x)) & ((Human (x) v Animal (x)) & ((Religious (x) v Mythical (x)) &((MH4m(x))) In turn, by a substitution rule, if the variable x becomes the symbol Sphinx: then the following formula can be generated: 3) ∃x (((Male(Sphynx) v Female (Sphynx)) & ((Human (Sphynx) v Animal (Sphynx)) & ((Religious (Sphynx) v Mythical (Sphynx)) & ((MH4m(Sphynx)))

128

Computational Semiotics

Hence, in our statue ‘database’ a statement such as The Sphinx is a mythical human male that is more than four meters high can be deduced. Unfortunately, not all formal models are as transparent as this predicate logic one is. Indeed, there are certain scientific practices where one starts by asserting a general principle or rule, but where there are no existential statements made. And it is the purpose of the enquiry to identify or discover specific individual statements that it may corroborate its deducible existential instanciation statement and ultimately its general principles or law. In our statues example, all that is required is to find among the given descriptions which one can corroborate the existential instanciation deduced from the general rules expressing the structure of the statues’ features. This type of reasoning has been one of the most popular reasoning procedures in semiotics. In many semiotic analyses, a general principle, law or ‘abstract’ structure is asserted as a working hypothesis. And the purpose of the enquiry is to find some individual semiotic artefact or state of affairs that would be a specific instanciation of the general principle, law or rule corroborating the hypothesis. A typical semiotic example of this deductive reasoning procedure is illustrated in the Greimasian logical square. Take again the general rule: (Q & ¬P ) v (P & ¬Q) v (Q & ¬Q) v (P & ¬P ) The proof of this general semiotic structure formula is usually given by identifying the individual artefact or state of affairs that confirms truthfulness of the general rule. And it will be accepted as truth until some counterexample is discovered. Such a procedure has often served as an explanatory principle for the narrative structures in stories, myths, pictures, etc. This ‘poperian’ reasoning procedure is absolutely acceptable. It is an example of theory-driven research. If the principle is complex and covers many cases, it is a heuristic and valid reasoning process. Still, there is danger in its usage. Often, all that is needed to prove a principle or a rule are just the few cases that are ‘chosen’ to fit it. And there are very often such cases which are then used to prove the general law, rule or principle. There are a few reasons for the popularity of this deductive reasoning process. In its early beginnings, semiotics has in its studies taken natural language and logic to describe and explain the prototypes of semiotic artefacts or processes. And given that in these models, deduction is the usual and preferred reasoning procedure, it became, for semiotics, the most accepted form of a rigorous reasoning process. Another reason for choosing this type of reasoning process has been the empirical epistemological understanding of scientific explanation as a deductive reasoning: a law is confirmed by finding all observable individual states of affairs expressed in each individual counterexample or case. Or in a Poperrian spirit, a law is true until proven false by some individual case. It is not surprising then that early semioticians had not paid much attention to finding a manifold of individual statements as proof of principles. Counterexamples were sufficient to disprove the law. In this perspective, data-driven research would not seem necessary.

 Formal Models in Semiotics 129 Still, for some rare semioticians, it was not the deductive model that was the problem for semiotic enquiries. Rather, it was the particular type of formal symbolic language chosen for reasoning in semiotic inquiries. Indeed, these logical and grammatical types of formal languages were mainly defined by a syntax that is applied to atomic symbols and whose semantics was geared mainly towards discrete entities. Such types of formal languages were not seen as adequate for dealing with semiotic artefacts and processes essentially characterized by their continuity and their dynamicity. Such a deductive process is an essential type of reasoning process. And in a computational semiotic horizon, it will be invaluable. In fact, in artificial intelligence models, it has been and is still an important if not essential reasoning process. Artificial intelligence agents such as expert systems possess many general principles that embed some knowledge database and are used for acting in the world or for solving problems. In such an environment, these principles are often expressed in the form of tables, frames, rules, ontologies, schemas or graphs, to name but a few. A computational semiotics may find it useful to keep this type of reasoning process in its projects. Formal models that express general knowledge may serve as memories of learned or transmitted knowledge. But it is not the only type of reasoning process. The other one is the inductive process. It will be an essential one in acquiring knowledge. Compared to the deductive reasoning process, inductive reasoning goes in the opposite direction. It starts from just a few individual statements and moves towards a general one. The inference rule called ‘universal generalization’ exists, but to be applied rigorously, the reasoning process must have access to all individual statements. Otherwise, if all cases are not available, only a statistical generalization will be possible. In our statues example, a pure generalization is possible, for we exhaustively have access to all eight individual descriptive statements created for each statue. And the process of reasoning leading to discover it was easy, for the constructor of the example knew how to formulate a general statement that would fit all statues. So, our example is pedagogically pertinent, but biased by the specific descriptions built in the example. In ordinary research things are different. Confronted with various types of data, many being expressed through individual statements, it is not easy to discover the general law. And, in fact, there may be many such statistically sound general statements. In other words, in a big dataset, there is usually not just one general pattern that may fit all the data. On the contrary, there are often many such patterns. Discovering them is not possible by strict induction. The process is assisted by various optimizations. One can imagine if our statues example had included many thousands of individual statues, for which there would have existed many different types of individual descriptions, finding some general statements or pattern for all of them would have entailed a tremendous cognitive burden. Not all formal models are technically equipped to support the discovery of a set of general statements that express some pattern, structure or organization. It is here that the vector algebra model becomes appealing. The format allows the easy writing in a vector-matrix format of all the possible statues or features to be

Computational Semiotics

130

studied. Each vector or feature could be expressed in the vector format; we have, for instance, produced an existential statement for the statues David and Moai and for the features Male and Human using a vector format as follows: Line David d: ; line Moai o: , column Male M: ; column Human H: . This vectorial representation could be done for a myriad of other statues . A huge data matrix could be built. The process of arriving at some general patterning statement would call upon a reduction calculus of the matrix and various classificatory calculi. Each pattern so discovered would not necessarily be the only possible one. In fact, there may exist many of them. Each one would then appear as a best hypothesis or as an optimal pattern.

Formal models in our case studies To help us in the concrete understanding and possible usage of formal models, we will hereafter apply them to our three case studies on iconic, signal and symbolic semiotic artefacts.

Iconic semiotic artefacts: Magritte’s painting Our first case study11 example is a computer-assisted analysis of the whole set of paintings by Magritte. The conceptual model has designed a general conceptual schema for systematically tagging with a set of descriptors the relevant semiotic features of all 1,780 paintings. As a reminder, here is on is a sample of the descriptors of the painting: ‘The treachery of images’. Title La Trahison des images (Ceci n’est pas une pipe)

Descriptors

The formal language chosen and associated with the conceptual model is that of vector algebra. The tags and descriptors of the conceptual model were encoded in this formal model as a data vector and matrix. For example, for each painting, each value in the vector encodes the presence or absence (weighted or not) of one of the features (Table 8.4). Table 8.4  Vector representation of two Magritte’s paintings

Pipe Amants Xi

pipe

word

green

head

1 0 x1

1 0 x2

1 0 x3

0 1 x4

man women ceiling 0 1 x5

0 1 x6

0 1 …

wall

tie

…

0 1 …

0 1 …

xn

And the vectors can be expressed using a vector algebraic expression and a matrix: Pipe: ; Amants: < 0, 0, 0, 1, 1, 1, 1, 1, 1>

 Formal Models in Semiotics 131 On these vectors and matrices, some specific mathematical comparisons and classification models were applied, such as clustering analysis,12 K-means analysis, distributional analysis13 and topic analysis.14 And some specific similarities and differences between descriptions of paintings and features were explored. This generates multiple novel interpretation hypotheses on the aesthetics of Magritte’s works.

Indexical semiotic artefacts: Music The second example was the analysis of digital music databases with the aim of indexing them for queries, retrievals and recommendations. The corpus came in three separate types: a set of digitally encoded acoustic musical signals, a set of canonical perceptual descriptors (timbre, pitch, key, texture, harmonization, etc.), a set of cultural categorizations (style, genre, artist, instruments, etc.) and a set of written comments by users of the music databases. Thus, the main components of the conceptual model offer a conceptual framework for ‘representing’ digital music where each segment of music was annotated using these three types of descriptive categories or metadata. The formal models aimed at identifying structures in signal systems. The challenge in this project was to find the best formal model to analyse this corpus. For this task, mathematical models called association rules15 were used for discovering recurring relational patterns in data, the prototypical example being a set of transactions Ti containing tn pieces of information extracted from the three corpora: Ti  t1 , t 2 , t 3 , . . . t n  The association rules express the strength (Ti→Tj) of the co-occurrences of certain features in a transaction. Some rules have a maximum or a minimum strength. This simple model was applied to each of the three corpora, and they were represented as vectors in a matrix. For instance, there existed a vector for describing a particular music piece, a vector for describing the features of the acoustic signal, a vector for the perceptual features and a vector for the natural language words used to comment on the music. It was on these complex matrices that association rules were applied for finding patterns among musical signals, perceptual tags and comments. One advantage of these rules is the traceability of their processing, which is not often the case for neural nets or many classifications. The core of the analysis was to find in this dataset some similarity patterns called item sets that were seen as prototypical transactions and from which were extracted a set of prototypical association rules of the type ‘such and such musical signal is mostly associated with such and such perceptual descriptor and comment’, and so on. In regard to the task, these association rules became the source for dynamic indexing and recommendations of digital music. Translated into semiotic terms, this type of formal model sees each specific semiotic artefact and process as transactions. Each artefact is described as a set of associated semiotic features which possess some degree of strength. Hence, each of Beethoven’s sonatas, of Stephen King’s novels or of Spanish

132

Computational Semiotics

processions will contain a set of associated features that can be explored in order to discover some association rules that best characterize them.

Symbolic semiotic artefacts: The concept and narration in texts Our third case study has been a computer-assisted analysis of symbolic artefacts conducted over two different types of texts. The first16 was an analysis of the concept of ‘Mind’ in Peirce’s collected papers and the second17 was the exploration of the narrative structure of journalistic articles on a social crisis. In both studies, the conceptual model presented the core conceptual hypotheses and methods called upon in the study. For the first, it defined more precisely what is conceptual analysis of a text and for the second, what is the actantial narrative structure of a newspaper article. In both studies, the formal language used was vector algebra language. For the Peirce papers, the vectors were made of all the textual segments forming the context of the word ‘Mind’ (a concordance) or of its lemmatized forms or morphological variants (e.g. ‘mental’). For the journalistic corpus, each article constructed a vector. In both cases, selected words were the ‘features’ of the vectors. All the vectors formed a matrix. On this matrix, various types of classification or equivalence mathematical functions (equations) were applied, such as clustering methods, K-means, Bayesian probabilities, topic modelling and word embeddings. All these formal mathematical models discovered regularities surrounding either the text segments, articles or words. The difference between the studies lay in the type of manual analysis conducted over the classes of segments or articles discovered by these analytics methods. In the Peirce study, each class of segments was analysed to extract the core conceptual predicate structure of a particular thematic dimension of the concept of ‘mind’. For instance: ‘mind as semiotic system’, ‘mind as association of ideas’, ‘the laws of mind’, ‘mind and consciousness’ and so on. Similar analytics were applied to the newspaper articles. In the main class of articles discovered (e.g. on the role of the students, the role of the police, the role of the government), the core classical semiotic actantial patterns were manually extracted. This determined who were the actors, what were the actions done, for what purpose and on whom were they applied to. In these three case studies, formal models were built for only some specific instances or components of the overall dimensions that conceptual models may have explored and made explicit. Expressed negatively, formal models operate a reduction of the complexity of the problem. But reformulated positively, formal models are means of focusing on specific moments or components of a semiotic enquiry. Some of these formal models may be ‘quantitative’, but most often, they are structural: they rest upon mathematical structures, that is, upon relational structures. But for computational semiotics, this is not sufficient: the mathematical structural relations chosen must be amenable to computation.

9

Computational models in science

Introduction In ordinary conversation and even in some scientific discourses, there often seems to be no difference between a formal model and a computational model. Unfortunately, conflating these notions or concepts is not without epistemological consequences. It is true that all computational models rest upon formal models. But the reverse is not to be taken for granted. It is not because one has a formal model that a computational model will be readily available.1 The following explanations may appear technical for semioticians, but they are essential for understanding the specific definitions and roles of computation in scientific enquiries. This is required for semiotics to adequately integrate computation into its practice. Semiotics cannot enter the digital world without understanding more precisely the notion of computation. As we shall see, computational models represent a heavy burden in the encounter of semiotics with the digital. This means that if one applies computer technologies to the analysis of semiotic artefacts that cannot be modelled by formally calculable functions, then the results obtained will not be trustworthy. Therefore, if semiotics is to enter the digital world, it is essential to better understand what is computable per se and what is not computable. Otherwise, the adventure into the digital will rapidly meet a dead end.

Computational models in science: Definitions Computational models are deeply related to formal models. They are a particular sort of formal model. Still, symbols and formulas used in computational models present some specific properties that give them their own signature. Briefly defined, a computational model is one that constructs a special type of formal symbolic system. Its syntax restricts the type of admissible symbols, and it defines a particular type of rules for the manipulation of formulas. Its semantics refers to a particular type of mathematical structure: functional relations that can be ‘calculated’. And its associated pragmatics allows for effective procedures that a computer can implement.

134

Computational Semiotics

A computational model is therefore highly related to a formal model and ultimately to a computer model. With regard to the formal model, a computational model is a set of formal statements or rules that express how the functions of the formal models can be effectively calculated. And with regard to a computer model, the computational rules must be concretely implementable in a physical machine or computer. These characteristics of computational models have an important and decisive epistemic effect on scientific enquiries. They restrict the choice of possible formal models that can be used if the enquiry is to use a computer. And finally, it is only if the formal model itself contains some computable functions that these may, in turn, be included in the computational model – it is only then that a computer model can offer a technology that can effectively compute them. Thus, even if computational models are ontologically deeply entangled with formal and computer models, both models are not identical. And they have specific properties and operations. What mainly characterizes a computational type of model is that its core concept, computation, is about a formal property, not a technology. It appeared as an answer to questions raised by Hilbert: Is there a procedure by which it can be decided, just by manipulating the symbols, whether a specific mathematical formula or equation belongs or not to a formal system? In other words: what is the calculability or computationality of a mathematical formula or equation? Among the many solutions offered, two became very important. One of the first propositions was the Church Thesis (Church 1936). For Church, calculability was to be understood through a type of formal language2 that manipulates recursiveness: the λ-definable calculus. Even though Gödel himself had also worked on recursiveness, he regarded it as ‘thoroughly unsatisfactory’ (Sieg 2006). Practically at the same time, another solution was offered by Turing (1937): calculability would be equivalent (not synonymous) to ‘computation’ if there existed an effective procedure for systematically and productively generating output symbols from some initial input symbols. To demonstrate this thesis, he built two ‘machines’ (Figure 9.1). A first one (called an abstract machine or Machine A) was defined by a list of symbols qi (0 & 1) and a transition state Si. Machine A formally represents a function through a sequence of instructions expressed in formulas containing symbols of type qi (0 & 1) and Si. The second machine (called a physical automaton or Machine B) was a physical machine made of mechanisms containing elements such as a paper roll, some wheels and reading and printing devices, and it Figure 9.1  Turing Machines A & B. was subsequently called a ‘Turing machine’ by Church (1937). Technically, this meant that if a formal function was calculable (Machine A), then it was computable by a physical automaton (Machine B). This claim was later called the

 Computational Models in Science 135 ‘Turing thesis’. It states that ‘every intuitively computable function is computable by an abstract automaton’ (Turing 1936). Turing’s proposition, often called the ‘Strong Turing Thesis’, establishes a relation between two structures: an abstract one and a physical one. The first structure can be called a calculable function if and only if it can be processed or ‘computed’ by an effective procedure, that is, a physical machine. Soare formulated the notion of computation in more contemporary terms: A computation is a process whereby we proceed from initially given objects, called inputs, according to a fixed set of rules, called a program, procedure, or algorithm, through a series of steps and arrive at the end of these steps with a final result, called the output. . . . The concept of computability concerns those objects which may be specified in principle by computations and includes relative computability. (Soare 1996: 286, emphasis removed)

This thesis will be very important for understanding the relationship between the formal expression of a calculable function given in a formal model and its translation into a computable model. It ultimately means that even if formal models can create formulas that represent complex functional relations for some phenomenon under study, nothing guarantees that they are computable and therefore accessible to computer processing. In other words, integrating computer processing in research requires that the formal models called upon must propose only formulas that contain computable functions if they are to be processed. Both theses were later brought together to form an important integrated thesis called the ‘Church-Turing’ thesis by Kleene (1952: 300) who gave it its first formulations. Gandy, a student of Turing, formulated it in the following manner: ‘whatever can be calculated by a machine can be calculated by a Turing machine’ (Gandy 1980). Rogers3 (1987) showed that the same class of partial functions (and of total functions) could be obtained in each case. This Church–Turing thesis4 has been extended to many other types of formal systems and languages. Briefly summarized, it says that calculable functions are ‘equivalent’ if they can be computable by a Turing machine. This has been shown for Post’s (1936) production rule, Curry and Feys’s (1958) combinatorial logic, von Neumann’s (1966) automata, the Gandy (1980) machine, Chomsky’s (1957) automata grammar and many other types of formal symbolic systems. One of the important translations that the notion of computation has received is that of algorithm. This notion is one by which the computer operations are indeed mostly defined. It has become the keyword for naming the set of rules or instructions that a computer follows to accomplish a task. But it was Markov5 ([1954] 1960) who demonstrated the equivalence between algorithmic formalization, Church’s recursive functions and Turing’s machines. And in fact, the instruction part of a Turing machine is algorithmic. Briefly defined, algorithms are effective procedures to achieve the computation of a function. They are presented and expressed in a language through a sequence of instructions applied to input so as to systematically produce some output. Many other formulations of the notion of algorithm exist.

136

Computational Semiotics

For our research purposes here, there are three implicit distinctions in the definition of algorithms that must be emphasized, for they are often either ignored or conflated. They remain important in understanding the epistemic role of algorithms in computational models. The first one is that a procedure is said to be an algorithm only if it is applied to a computable function. Algorithms are therefore finite and effective procedures. This means that if a problem presents high complexity (N.B.: not complication), its formal model may introduce some undecidable functions. This also means that the effective procedures involved may never stop and hence render algorithms useless. The second one is that one must not confuse a computable function with an algorithm. This is because the same computable function can be expressed by several algorithms. For example, there are several different algorithms for calculating the arithmetic mean of a set of numbers, as it is a calculable function. And a third distinction is that there exist many different types of programming languages to express the same computable function and its algorithms. Preferred ones have been von Neumann’s ‘flow chart’ or Miller, Galanter and Pribam’s TOTE. But there are other ones such as McCarthy’s LISP machine, Anderson’s production rules and Chomsky’s automata rules. And today, there is a proliferation of high-level languages that can express algorithmic procedures. This means that even if a programming language is elegant and well-formed, it does not follow that the algorithm and the computable function underlying it is transparent, known, understood and ultimately easily implemented in a computer. To rapidly illustrate the notion of algorithm, let’s take a simple example. Suppose that in our conceptual model, we assert in a natural language the following proposition: it is possible to find the sum of the first one hundred numbers, that is, ‘the sum of 1 + 2 + 3 + 4 . . . 100’. A formal model can be offered as the algebraic equation solution: 100

F(X) 

X  X i

i+1

X n

i=1

One other interesting formal expression was given by mathematician F. Gauss: n

i  i 1

n(n  1) 2

where n = 100. These reformulations show that these mathematical functions can be expressed using different formulas or equations. Each one would give the same results. But each one, in turn, can be translated into an algorithmic programming language such as the following pseudocode for the first equation: Program to calculate the sum of n digits (Figure 9.2). One must notice that each instruction given in a programming language and translated into another one requires creative intuitions and expert conceptualization. And it can sometimes require time and energy.6

 Computational Models in Science 137

Figure 9.2  Pseudocode and flow chart.

This last algorithmic expression could have been presented in other more detailed programming languages. Each language would have to add more symbols and formulas, some of which pertain to instructions referring to the input data, others to many other types of entities such as variables, identifiers, control flows or external devices (printer, discs, memory, etc.). Formally speaking, then: ‘an algorithm is an intensional definition of a special kind of function – namely a computable function. The intensional definition contrasts with the extensional definition of a computable function, which is just the set of the function’s inputs and outputs’ (Dietrich 1999: 11, emphasis modified). Finally, let’s recall that functions are not always defined intensionally by some formula or algorithm; they can also be defined extensionally. This means that a functional relation can be defined through a data list or, more specifically, through its data structures. These are the arguments and values of the function. And, in fact, some see data structures as an extensional definition of computable functions and therefore as an extensional presentation of an algorithm and program.7 This interpretation of data as possible arguments and values of computable functions, as we will see later, will be applied to ‘big data’. It will give them a very important epistemic role in machinelearning algorithms, for they will be seen as implicitly being the expression of possible computable functions. Fodor and Pylyshyn (1988) proposed a reformulation facilitating the comprehension of computability by cognitive science and by the digital humanities. According to this reformulation, a function will be recognized as computable if it has at least one of the following properties: atomicity, systematicity and productivity. For the particular

138

Computational Semiotics

field of cognition to which they applied such a computable function, they added the property of interpretability. The first property involves the admissibility of inputs (called arguments) with respect to a computable function. These inputs must be discrete or ‘atomic’ – that is, non-continuous. Usually, the digital encoding into 0s and 1s symbolically encodes this property. The second and third properties are the most important ones for our discussion. They specify the nature of the procedure or of the operations which will be applied to these inputs. Indeed, these must be systematic and productive, that is, the procedures must enable us to systematically produce increasingly complex sequences of symbols or to reduce them to simpler ones. These two properties of systematicity and productivity are simpler means of talking about recursiveness, algorithms and combinatoriality. If a system solely possesses these three properties, it is de facto a formal system in the sense of Hilbert. And as several researchers will emphasize, and as Searle (1980) and Harnad (1990a, 2017) will regularly reiterate, in the cognitive field, such a system is essentially ‘syntactic’. It is indeed interpretable, but its semantics are external – hence the importance of cognitive science to add the property of interpretability. Still, the reformulation by Fodor and Pylyshyn allows us to briefly present here two classical critiques that have been regularly addressed to computational models. Because it requires atomicity, systematicity and productivity, computationality cannot adequately model phenomena that are continuous and that present complex dynamicity. It was claimed that many dynamic systems may not be approached by classical atomic combinatorial sequential formal systems and Turing machines. The length of this book does not allow us to enter such debates here. Let us simply recall that atomicity, systematicity and productivity are properties of models – and not realities as such. And these models are mediators for describing and explaining. On the basis of this, the question then arises as to whether Turing-type computation models can compute functions that contain Cantorian continuity and parallelism, as is the case for instance with formal dynamic systems. A negative answer to this question would forbid the application of computational models to dynamic and parallel systems. It so happens that many mathematicians, such as Rice (1953), Lacombe (1955) and Grzegorczyk (1957), to name but a few, have proven that analogue machines could also compute functions applied to continuity and that their computation is equivalent to a Turing computation. A similar type of argument concerns the sequentiality of Turing computation. On this topic, Gandy (1988: 33) has shown that computation can be parallel – that is, that there are machines whose computation processes involve parallel changes in many overlapping parts.8 Siegelman (1995) has claimed that the class of Turing’s original physical machine ‘TM’ is just a subclass of the many other computing machines. In other words, formal symbolic systems that model complex dynamic and parallel systems can be computed and therefore presented in algorithms. But it is the problems underlying the notion of productivity that raise the most serious problems for computational models. They raise the question of non-computable functions.

 Computational Models in Science 139

Non-computability In mathematics, the notion of productivity once more encounters the problem of the decidability of theorems for formal systems: are theorems always provable in a formal system? Or, in terms of computation: are all functions calculable? Could it be the case that a formal model contains functions that would not be computable? Here are two examples of such non-computable functions. Our first problem is one that was described by Penrose (1989). It is not presented in formal terms or using an equation. It is more of an illustration of a situation in which we would intuitively expect there to be some functional relations underlying the question asked and where there could exist a calculable equation or algorithm for finding the solution. Here is the problem: imagine that to cover the surface of a rectangle, we have to use atypically shaped tiles such as those shown in Figure 9.3. What is the effective formal procedure for ensuring a perfect fit of the tiles in the rectangle? (Figure 9.3).

Figure 9.3  Penrose’s Polyomino Tilings.

Penrose proved that no recursive function or algorithm exists to show if these shapes could cover an entire surface. In other words, it is not possible to find a calculable function of the type f (x, y) = (x*y) that would enable us to compute the necessary number of tiles for covering such a surface. The second example is a typical mathematical one. It presents itself in the form of a system of several equations all having the following form: Ax + By = C where A, B and C are constants. 3x + 7y = 1 x2 − y2 = z3, and x, y and z are integers This type of system of equations is called ‘diophantine’. Intuitively, just looking superficially at these algebraic equations, it appears to be easy to find a general algorithmic solution for them. But, on the contrary, Matiyasevich ([1970] 1993) and Davis, Matiyasevich and Robinson (1976) demonstrated that no such algorithm exists. In other words, although the values given to the function’s variables may be atomic numerical values, such a type of system of equations is non-computable. In mathematics, the existence of these types of non-computable functions is not a rare occurrence. ‘Everyday mathematics leads us unavoidably to incomputable objects’

140

Computational Semiotics

(Cooper 2004: 1). Many mathematicians and computer scientists have demonstrated that computable functions form only a very narrow subset of all mathematical functions. There is an infinite number of computable functions (ℵ), whereas noncomputable functions are infinitely more numerous (2ℵ). In other words, they are not countable. In short, these two examples chosen among an infinity of other possible ones show that infinity of mathematical functions, though they may be well-formed as functions, are not calculable/computable.

Oracles This problem of non-computability is one which Turing confronted in his doctoral thesis. He proposed an original solution for making non-computable arithmetic functions into computable ones. He appealed to ‘oracles’: ‘With the help of the oracle we could form a new kind of machine (he called them o-machine), having as one of its fundamental processes that of solving a given number-theoretic problem’ (Turing 1939: 161). This specific solution, of the oracles, is highly interesting and important, for it allows the creation of programs that find answers to non-computable problems. Many such oracles or heuristics can be added and appended to form a complete computable program. And because of these oracles, the computation may proceed and eventually end. However, by the same token, the problem of non-computability becomes even more problematic than one would think. By appending oracles, more complexity is brought into the solution. As Chaitin, Doria and da Costa (2012: 36) put it: ‘just add[ing] new axioms . . . increase[s] the complexity H(A) of your theory A!.’ In other words, because oracles may entail redundancy, there is an increased risk of generating greater randomness which, in turn, may steer the systems towards an absolute probability called Omega, which, according to a theorem by Chaitin (1998), renders the system non-computable. Hence, although a complex static or dynamic system with discrete or continuous input may at first appear to be computable, it may paradoxically turn into a non-computable system. Consequently, the greater the complexity of that which needs to be functionally modelled, the higher the probability that it will be non-computable. This non-computability problem is not rare in science. On the contrary, it is omnipresent. It is everywhere. ‘Undecidability and incompleteness are everywhere, from mathematics to computer science, to physics, to mathematically-formulated portions of chemistry, biology, ecology, and economics’ (Chaitin, Doria and da Costa 2012: xiii). And for some contemporary computer scientists, non-computable functions are far more interesting from a theoretical standpoint than computable functions. ‘The subject [of computation] is primarily about incomputable objects, not computable ones’ (Soare 2009: 395).

Epistemic roles of computational models in science Historically speaking, formal models have always been at the heart of scientific inquiries, but accompanying them with computational models is quite recent. Today, however, they

 Computational Models in Science 141 have become an important component of scientific theories. They have radically changed the way science is done. Some see their contributions by the huge number of data they are given and the speed at which they can be processed. But these features belong more to the computer technology that implements them than to the computational models themselves. But as more and more epistemologists have stressed, computational models have their specific role in science, be it epistemic, expressive or communicative. The first role of a computational model is an epistemic one. As computational models are a particular sort of formal model, their epistemic status resembles the latter. Indeed, they have some similar cognitive surrogate role. Like formal models, they represent something in the sense that they are projected or mapped onto the objects to which they are applied. So, if, as computation theories have shown, algorithms are equivalent to effective procedures for calculable functions, then computational models contain algorithms for computing functions projected onto the objects studied by the sciences. For example, in modelling a pendulum, there will be a mapping from a formal model FM that is equivalent to a computable model or algorithm AL, onto some object structure OS itself represented by some specific data. In other words, it is one of the epistemic roles of a computational model to offer a means for algorithmically expressing this functional mapping (Figure 9.4).

Figure 9.4  Functional mapping of a pendulum.

And as computational models use symbols and formula, they also construct their specific type of epistemic categorization and reasoning operations. We will explore these epistemic roles in view of their eventual application in semiotic enquiries that aim at integrating computation in the analysis of some semiotic artefacts.

Categorization role of a computational model The epistemic categorization role of a computational model has a specific signature. It has its own way of constructing and expressing in a computational language (a) the object or entities that are to be submitted to and produced by a computable function, and (b) the calculable functions that underlie the algorithmic statements. That is its main purpose. In more concrete terms, it offers a model of the object and functional

Computational Semiotics

142

relation in terms of algorithmic statements or ‘programs’. A typical example would be that of a computational model which included an algorithm (of the Monte-Carlo type) that uses randomness for computing some function or algorithm to estimate variance or to decide the shortest path between many points, for instance. But what is more interesting is that the categorization of its objects and algorithms can take two main forms. Just as a formal model may express its calculable functions in two ways, a computational model can express them intensionally or extensionally. Take the classical and simple mathematical function of squaring a number. This functional relation in a formal model is expressed intensionally by the following formula:

x  x  x , y  x 2 , or F (x ) : x 2 or extensionally by the list: F (x , y ) : (1, 1), (2, 4), (3, 9)(n, n). Because this function is a computable one, by virtue of the extended Church-Turing thesis, a computational model can express it, under certain conditions, by means of different algorithmic statements or instruction. The first and most usual one is through classical intensional algorithmic formulas or through a program. The second one can express the inputs and outputs of the function through an extensional list in a database. Hence, there are two ways of expressing an identical computable function in a computational model. But they each have a specific epistemic role. The intensional categorizing formulation of an algorithm expresses a function using abstract generalization formulas. It presents the instructions to be followed if the function is to be effectively computed by computers. But for most humans, such formulas often appear only as sets of formal symbolic statements that are cognitively unreadable. More profoundly, it is often not sufficient for explaining or for understanding the object and the processes to which it is applied. In other words, it is not because a computational model correctly expresses computable functions that the expressed procedures are traceable (Gotel et al. 2012) and hence readable and understandable by humans. A simple proof of this is often seen in laboratories or industries, which, over the years, have developed algorithms but without providing the documentation that should have accompanied them. These programs may still be effective, but they may become opaque for later programmers and users. In such cases, the algorithms cannot easily be understood. Not every scientist or programmer is like von Neumann, who was said to have had the capacity to directly read algorithms written in a binary code.9 One reason for this is that intensional algorithmic expressions are expressed in a language that encodes its instructions using a vocabulary and formulas that are either close to machine coding or to some higher-level programming language. This latter language can be simple or complex in its expressive means. For instance, the same computable function can be expressed using various equivalent low-level assembly languages or in higher-level languages (Python, C++, etc.). But this does not guarantee that they will all have the same cognitive content and hence the same

 Computational Models in Science 143 level of understandability for humans. Depending on the language chosen, algorithms can either be understandable or completely opaque. And in effective and concrete processing, the instructions activated may be tractable or non-tractable. In other words, it is not because algorithms are well-formed syntactically or semantically that they are automatically cognitively accessible to humans. It follows then that in a computational model, this type of intensional categorization may become a black box for many users. One may know what the inputs to give to an algorithm and what its expected outputs are; however, because one does not always have access to what it does or to what the computable function underlying it is, the understanding of it will be limited, if not absent. A computational model may also categorize a computable function through some extensional presentation. This is achieved by listing the inputs and their corresponding outputs. But scientists and epistemologists do not tend to see a list as a mode by which a computable function is expressed. But as demonstrated by many computational theories, under certain strict conditions,10 a list can indeed be an acceptable means for presenting a computable function. Take again our squaring function y=x2 or F(x): x2 where the inputs are the value of the argument x, with f(x) or y representing the value of the function. It is the list of these paired values in a domain that can form an extensional presentation of the function (1, 1) (2, 4) (3, 9). . . (n, n). And in computer science, this list of ‘inputs’ and ‘outputs’ can be included in a database and called a ‘data set’. And hence, the list opens up a new way of understanding the categorization role of a computational model. It follows then that computation models can express a calculable functional relation extensionally. The data are its input and outputs. In more concrete terms, it is possible to think of data sets as an extensional definition of an algorithm. In Table 9.1, our preceding example of the squaring function is expressed using a simple data frame. The following data set could be seen as hiding an underlying computable function. Table 9.1  Extensional Definition of a Function in a Frame-like Presentation Squaring function

1

2

3

1 2 3 … 10

1 -

4 -

9 -

100

In science, the notion of data is not often seen through such a perspective. Data receive a variety of definitions. For instance, in computer science, the term ‘data’ may refer to the inputs and outputs given to or produced by a computer and inscribed in a database. In mathematics, the term refers more specifically to a set of numerical values of functions over mathematical entities. In statistics, it may refer to the dependent or independent values of a formal model applied to experiments. In philosophy, data are interpreted as being the result of some knowledge processes such as perception, observation and experimentation. In the theoretical literature, all three meanings are

144

Computational Semiotics

often conflated. The data found in a database are often only seen as a set of ‘numerical values’ collected through some observation or situation. And they do not seem to be functionally related. What is important is that they are well stored, searchable, visualizable and managed through some relational database management system (RDBMS). It was the intuition of Codd (1983),11 creator of the relational database, that data sets could be organized as relational sets. Hence, a data set could be an extensional definition of relations between objects and their attributes or properties. But which relations are to be implemented is determined by the creator of the database. Some other researchers, mainly from the fields of machine learning and data mining, saw these data sets in a very heuristic way. For instance, T. M. Mitchell (1997) and many others saw data sets as possible instantiations of functional relations to be discovered. A data set would therefore be the result of underlying functional regularities that could be submitted to ‘algorithms that can learn regularities in rich, mixed-media data’ (Mitchell 1999: 35). To illustrate this intuition, we may consider a simple experiment: the movement of various sorts of pendulums producing numerous data related to a multiplicity of different features. Some are important, other less so. For instance, the data set could be: the numerical value associated to the length of the rod, the oscillation period, the location, the weight of the ball, its gravity, its price, its level of noise, the temperature, the surrounding atmospheric pressure and so forth. All these data represent features of the pendulum. All may be put into relation, but only a few of them are related by a computable functional dependency. In a computational model, it could be the role of a machine-learning algorithm to discover or approximate which of these features bear computable functional relations. For example, the algorithm could discover that the length L, the gravity g and the time T are data that could be intensionally expressed by means of a classical algebraic equation: T  2

L g

In this example, we can see that in a computational model, data pertaining to some object can be presented either extensionally or intensionally. Extensionally, the data are presented through some ordered list. Intensionally, the data are implicitly embedded in some algorithm which is equivalent to a calculable function. This dual way of presenting data directly regulates the type of reasoning processes that can be applied to them. They could be either deductive or inductive, but abductive reasoning may also be called upon.

Reasoning role of computational models Just as with the conceptual and formal models, reasoning also has an important epistemic role in computational models. But here, the reasoning process is deeply related to the form of the data on which it will be applied – that is, to their intensional form or extensional form. In both cases, reasoning is a rule-governed inferential process that is applied to the data.

 Computational Models in Science 145 From a cognitive standpoint, computational modelling will apply all three of the main types of reasoning processes we have seen in the conceptual and formal model: inductive, deductive and abductive. These three types of reasoning have become the core of reasoning strategies in various computer sciences that came to deal with artefacts that were carriers of meaning.

Top-down deductive reasoning In early artificial intelligence, the preferred type of model was the computational deductive model. It was often said to represent a top-down manner of reasoning. Its general architecture was usually applied to some general knowledge data statements from which, by an inference based on existential and instantiation rules, it will deduce some particular knowledge. In the knowledge-representation (KR) paradigms of AI systems, this top-down approach has been presented through various semi-formal or formal means, schemas and frames, which became knowledge levels, knowledge spaces, knowledge bases, folksonomies, ontologies, knowledge graphs, etc. These KRs are not conceptual models as such, but rather computational expressions of part of the conceptualization of the domain under study. In other words, in a computational knowledge-representation paradigm, knowledge is expressed through an abstract and formal but specific conceptualization of a domain. This is well expressed by Gruber’s classical definition of ontologies: ‘An ontology is an explicit specification of a conceptualization’ (Gruber 1993b: 199, emphasis added). Let’s consider some classic examples of these computational models called ‘frames’. These types of models are usually grounded in some natural language sentences of a conceptual model: for instance, All the cities of England have a mayor. The proposition underlying this English sentence could also have been expressed in another natural language, for instance, French: Toutes les villes d’Angleterre ont un maire. The formal model in turn translates not the sentences itself but the general propositions underlying them. So, it will be expressed directly through a logical formula that dissimulates the general quantification hidden in the English sentence while explicit such as ∀x City (x) ⇒ HAS MAYOR (x). In a Minsky frame type KR, the computational model will embed these statements in a sort of database and it can express extensionally some of the knowledge pertaining to the mayorship of England’s cities (Figure 9.5). Mayored cies of

Mayor name

Mayor employee number

England London

Johnny Walker

WalkJ8594

Oxford

Elizabeth Arden

ArdenW3928

Cambridge

Granny Smith

SmithB3401

Name x

Number x

…. City n

Figure 9.5  Examples of Minsky type frames.

146

Computational Semiotics

The general statement of this frame does not take the form of a propositional logical formula but it implicitly asserts that each city in England has the feature ‘has a mayor’ who, in turn, ‘has a name’ and ‘has an employee number’. Naturally, many other features and sub-features could be added. And onto this knowledge-representation format, some inferential types of algorithms can be applied (e.g. ‘if, then’). So, when a new input is entered, such as London is a city, then the system will ‘deduce’ that if this city is in England then it has a mayor and also that it has a mayor whose name is ‘Johnny Walker’. This type of top-down algorithmic reasoning process is at the core of knowledgebased systems12 or rule-based systems.13 The main architecture of these systems includes (a) a database where general or individual knowledge is stored and retrieved; and (b) a variety of inferential engines applied to the database. In the earlier years of AI, many variants of these frames were wrapped up into computer applications. The prototypes of these systems took the form of expert systems. Their architecture included a knowledge base and an inference engine containing a set of topdown inference rules that implemented a forward-chaining process where the general antecedent allowed the deduction of the instantiated consequent. These systems often also included some other types of backward-reasoning processes that went from the consequent to the antecedent. But these belonged more to the inductive and abductive modes of reasoning. Later on, these frames were transformed into more sophisticated types of knowledge representation, with important types being ontologies and knowledge graphs.14 Today, the semantic web paradigm can be seen as a knowledge base system where the internet serves as a knowledge source. The difference in the processing is that the knowledge data (textual or iconic) must be transformed into a predicative general form called the Resource Description Framework (RDF) so that it may be used by inferential rules.

Bottom-up inductive reasoning processes The top-down reasoning process has always been seen as the essential feature of an artificially intelligent process. The problem with this type of reasoning, as has repeatedly been stressed, lies in the acquisition of this knowledge. Where does it come from? Should it always be given by some external agent or could it not be learned by the system? This is where inductive and abductive reasoning steps in. In computational models, this second type of reasoning process is often called a bottom-up algorithmic process. Its architecture is opposite to that of the top-down approach. It usually contains a set of data which can be extracted or discovered by some algorithmic procedures over some type or another of general knowledge. Strictly speaking, such generalization reasoning is inductive if and only if the data are exhaustively given or asserted. For instance, only if there is an exhaustive data list in which each and every city of England is associated with an individual mayor may we conclude that ‘All the cities of England have a mayor’. This inductive example can be expressed using the following logical formula: (CITY (London) & has MAYOR (London)) & (CITY (Birmingham) & has MAYOR (Birmingham)) . . . (CITY (xi) & MAYOR (xi)) then if ∀x (CITY (x) ⇒ HAS MAYOR (x)).

 Computational Models in Science 147 The general knowledge that ‘All the cities of England have a mayor’ has been produced by a generalization inference rule called universal instantiation. It allows a conclusion to be considered true until a counter example is given. In computer science, these inductive reasoning procedures or bottom-up procedures are now often called data-driven algorithms or data mining analytics. This algorithmic architecture is one where the inputs are a set of structured data to which are applied various types of algorithms that transform, aggregate and classify these data to produce an output that is a sort of general formula, if not the best approximation of a universal formula. In a conceptual model, this reasoning process is seen as identifying regularities, patterns, prototypes and even ‘laws’ governing the data that describe the features and relations of the state of affairs under study. In a formal model, these general formulas are expressed using intensional synthetic formulas. Recall here the list of values of our squaring function. An inductive reasoning process would aim at synthesizing the following extensional list of functional relations: (1, 1), (2, 4), (3, 9), . . . (n, n) by the formula F(x) = x2. This inductive process is usually easy to achieve if the data are finite, not too sparse and do not contain too many different types of variables. But often, the object under study will be complex and present a myriad of micro-features and numerous types of static and dynamic relational structures. Finding the patterns that characterize data and expressing them through some sort of general formula is quite a challenge. It is one of the most important problems that contemporary computational science has to deal with. Still, most actual practices continue to use inductive reasoning processes, but as the object they study is often presented to them through a huge amount of data they are increasingly pressed to explore efficient inductive reasoning procedures supported by data-driven algorithms. Many computational models have proposed algorithms to process these big data in this inductive way. The algorithms they offer are often metaphorically called ‘machine learning’,15 ‘deep learning’16 or, prosaically, ‘pattern recognition’17 or ‘inductive programming logic’.18 These types of algorithms have become the contemporary stars of artificial intelligence research, where ‘intelligence’ is mainly understood in terms of inductive learning procedures. These data are not processed in the form in which they come. They do not jump into algorithms for the simple reason that they are data. Whatever their source or their type, they have to be submitted to many complex operations, sometimes manual, sometimes algorithmic, in order to (a) become inputs for these machine-learning algorithms, (b) be effectively processed by these algorithms, and, finally, (c) be evaluated when applied to concrete cases. In the first moment of this complex procedure, the data must be acquired – a task that is not without complex problems, for their sources are often numerous, anonymous, noisy, non-controlled and of various types. Afterwards, they must be prepared to be made into admissible arguments or inputs for these algorithms. They must hence be cleaned, selected, categorized, annotated, refined, standardized, validated, verified, updated and so on. Afterwards still, they have to be machine-encoded, formatted and normalized. In other words, data are not ‘given’; they are worked on. Despite all these

148

Computational Semiotics

heavy preprocesses and processes, machine-learning type of computational models have the wind in their sails. Some even believe that theory-driven research has been killed! As Anderson puts it, ‘The data deluge makes the scientific method obsolete’ (2008).19 In the second stage of these inductive reasoning procedures, the data have to be processed by some effective algorithms. But these algorithms come in many shades. They are not a ‘one fit for all’. A relevance analysis must be made. Many are rooted in different mathematical models and expressed in different languages. So, we find underlying them various formal mathematical models belonging, for instance, to inductive probabilistic statistics, mathematical classification, statistical optimization, inductive logic, dynamic combinatorics, dissipative dynamics or formal systems. We also see that these use many formal languages such as predicate logic, directed graphs, differential geometry, linear and non-linear algebra, topology and so on. All these formal models and languages possess their particular conditions of use and parameters. They are not easily interchangeable. Choosing one or another is often quite a challenge in itself. Finally, these algorithms have been inspired by various but specific scientific theories: cybernetics,20 vision and behaviour psychology,21 neurobiology,22 insect or swarm intelligence (ants, termites), stigmergy,23 genetics and evolution,24 immunology,25 artificial neural nets, connectionism,26 neurophenomenology,27 situated philosophy of mind,28 catastrophe semiotics,29 radical material epistemology30 and so on. Rapidly summarized, these various inductive reasoning algorithms are understood as types of computable classification and optimization functions that have for arguments some data values and deliver as outputs some patterns in the data. They are often interpreted in cognitive terms and are said to recognize, learn, adapt, discover and evolve. Many sub-procedures have been explored to create flexible but robust learning. Two important ones among these are the ‘supervised’ and ‘non-supervised’ ones. The supervised learning procedure is one that is helped by some training. It uses a subset of well-structured data that serves as a prototype. What is learned in this subset is then applied to the whole set in view of discovering the most similar patterns. The second procedure is called ‘unsupervised’ learning. Without the help of prototypes, the procedure clusters with the help of complex parameters the data that share some common features. These two strategies are often seen as ‘superficial’ learning processes. So, some deeper and more complex learning procedures were proposed. These deep learning procedures usually start by building some interrelated classes of elements that share either some common data features or some interrelated classes of features. Afterwards, these classes are themselves reclassified by means of multi-level strategies so as to build new levels of interrelated classes. And after some iterations and more detailed readjustments, they come to ‘recognize’ more distinguished, solid, optimized or most probable patterns in the data. During the last stage in the application of these inductive algorithmic procedures, some evaluations and interpretations must be performed. Big data are not simple and unproblematic artefacts. They are not without their own epistemological problems. One typical problem is the adequacy of the algorithms with respect to the specific objects

 Computational Models in Science 149 applied. In laboratories, these algorithms will be tested against accepted benchmarks. But in many concrete applications, no such benchmarks exist, so great confidence must be placed in the algorithms themselves. Finally, as theories, data are often biased, noisy, situated and institutionalized, the results may also be oriented in different directions. This entails quite a challenge for their interpretation and validity. As Gould eloquently said: ‘inanimate data can never speak for themselves, and we always bring to bear some conceptual framework, either intuitive and ill-formed, or tightly and formally structured, to the task of investigation, analysis, and interpretation’ (Gould 1981: 166). The third type of reasoning processes is, as expected, the abductive one. In computational models, this reasoning process will be most useful. For example, recalling our cookie example, it may happen that an AI system is given in its knowledge base that All cookies in the box are chocolate cookies. But it may also receive the data: There are chocolate cookies. If the system is to reason correctly, it should not infer from the first general statement that These cookies are from the box. Rather, correct reasoning should only produce the hypothesis that Possibly these cookies are from the box. This is because it could be the case that these cookies come from somewhere else. Because of the heuristic role of this type of reasoning, computational models have been increasingly exploring it as it seems quite natural. It has become an important dimension of many computational models, mainly in artificial intelligence with regard to algorithmic learning. It may be used for problem-solving (Kakas et al.), discourse interpretation (Hobbs et al. 1993), knowledge discovery (Zhou 2019) and programming languages (Console and Saitta 2000). It is currently becoming more and more important in machine learning (Levesque 1989; Crowder, Carbone and Friess 2020; Zhou 2019; Dai et al. 2019). Although abductive reasoning is rich in promise, its transformation into algorithms must still undergo more research. The challenge is in either making probabilistic adjustments, building adaptation strategies or calling upon structures of a priori knowledge bases for validating the hypothetical conclusions. There is an increasing number of such algorithms. Some are ‘one-shot’,31 ‘few shots’,32 ‘prototypical’,33 ‘hybrid neural’ and ‘symbolic’ learning machines.34 We hope to have shown some of the complex epistemic roles of computation models. They are a subtle but sophisticated play between different types of algorithms

150

Computational Semiotics

that categorize the object under study and algorithms that allow reasoning on what has been categorized. The three types of reasoning processes we have presented define the epistemic role of computational models. And they always activate a dialectic relation between general formula, rules or laws and individual data. In certain cases, if the pattern is known and well expressed, then the deductive process will search some data for a proof or disproof. In some circumscribed situations, if all the data are available, then an inductive or at least a best approximation of an inductive process will search namely for a generalization or for rules. Finally, in cases where just a few data are known, an abductive reasoning process will be called upon. It will be the most complex one.

The expressive and communicative role of computational models Expressing and communicating the data, the algorithms and their results is achieved through specific notational forms in computational models. First, as we know, computers do not manipulate the data directly in the form in which they are presented to humans (natural language expressions, scripts, databases, digits, etc.). They process encoded data.35 So, whatever their source, they have to be digitized so that they may serve as inputs to algorithms. Second, the algorithms themselves are expressed in some formal computer language. These languages have specific syntaxes and semantics, and each programming language is specialized in generating instructions for the manipulation of specific types of data and operations in some domain. This gives them their expressive power. Some are very close to a basic, low-level computer language (assembler). Some others are geared towards databases (Cobol, Perl), towards the manipulation of mathematical functions (C++, LISP, MATLAB) and others towards natural language (Python, etc.). All these types of languages and the programmes they express may be so well mastered by the researchers that they become the ‘natural’ way by which the various components and operations of a computational model are communicated. It is common among computer scientists to talk about an algorithm in terms of a ‘formal model’ whose packages of algorithms are named according to the actions they carry out. Here, one ‘googlelizes’. There, one ‘topic modelizes’. Others perform ‘data mining’ or ‘deep learning’. Some even have a proper name: the Monte Carlo algorithm. They become the ‘specialized communication languages’. A new modern lingua franca characteristica is being born! For our research purposes here, one important point to emphasize is the expressive and communicative role that pertains to the presentation of the results of the algorithms. This is quite different from what may be encountered in conceptual and formal models where natural and formal languages dominate. Here, another type of language will often be called upon and is used as a bridge towards the conceptual model. When various complex algorithms are applied to huge data, the presentation of the processing and of the results obtained receive special attention in computational models. Because they are not so transparent and evident and often require a heavy cognitive investment in order to be understood, some sophisticated visualization techniques offer more effective means for expressing and communicating their

 Computational Models in Science 151 processes and, most importantly, their results. These visualization techniques became rapidly welcomed adjuvants and mediators for understanding algorithmic data categorization reasoning processes and their results. Here is a simple example of such a visualization technique. During the Covid-19 epidemic, various experts used many formal equations for describing and analysing several of its dynamics. The equations underlying the algorithm express some functional dependency relations between several variables such as time, infected population, recovered population, death, age, co-morbidity, geographical location and so on. Some of these equations and big data may be cognitively easy to understand for a trained mathematician. But even then, for many expert communities and the general public, due to the complexity of the functions, they are not immediately or intuitively understandable. And as more and more variables and parameters are added, the understanding of what is computed and of what the result is becomes increasingly difficult. So, visualization provides some assistance, leading to the generation of insight. The simple visualization in Figure 9.6 – a classical Cartesian graph – presents the dynamic results which may be produced by the application of computational models to three equations presenting the variance over time of the percentage of recovered population r(t), infected population i(t) and of susceptible individuals who could get Covid-19 s(t) (Figure 9.6).

Figure 9.6  Visualization of three equations describing three dimensions of Covid-19.

Without knowing the exact equations and data that correspond to the visualization of the curves, it is possible to achieve some inductive, deductive and abductive reasoning from this graph. For instance, in this formal model (the SIR model), it can be seen by the r(t) curves compared to the other ones that ‘a low peak level can lead to more than half the population getting sick’ (Smith & Moore 2004). The conclusion is reached by the reader without a knowledge of the mathematical models that underlie the Cartesian graph and its curves.

152

Computational Semiotics

At first sight, this iconic language seems simple. But technically, it is a translation of a specific set of results of computed equations into an iconic form. It is a sort of extensional presentation of the functions expressed through some equations and algorithms. The drawing corresponds to specific symbols of these equations. But one important point must be noted. The drawing is in juxtaposition to the curves representing the dynamics of three equations. So for the visualization to be understood by someone (an expert or a layperson), some parallel information processing is required. And that which is iconic becomes analogical. It is no longer digital. And it is the specific parallelism allowed by this iconic analogical language that gives the visualization its apparent simplicity and epistemic robustness, a rich cognitive value and a heuristic function. This is quite difficult to achieve with the traditional sequential reading of the equations. Depending on the nature of the domain studied, on the formal models used and on the computational format taken, the visualization techniques may be static or dynamic, and they may take various forms: interactive tables, charts, graphs, maps and so on. They can use various metaphoric forms, such as buildings, games, roads, spaces, spots, pies, mountains and textures (see Chen 2013 for a rich set of examples; and Dondero and Fontanille 2014 for a detailed semiotic analysis). The visualization techniques36 enhance the epistemic role of both formal and computational models. They transform underlying formal and algorithmic models into various iconic forms (annotated with natural language). This is evident in Figure 9.7.

Figure 9.7  How Coronavirus is Devastating the Restaurant Business. Table cited in Chart: How coronavirus is devastating the restaurant business, by Rani Molla@ranimolla Mar 16, 2020, 12:20 p.m. EDT https://www​.vox​.com​/recode​/2020​/3​/16​/21181556​/coronavirus​ -chart​-restaurant​-business​-local (open source).

 Computational Models in Science 153 Visualization techniques of the kind used in Figure 9.7 manipulate data not only to provide descriptive knowledge but also to create explanatory knowledge. In science, data visualization has become a full subdomain of computer science in itself. However, the challenge it entails is great, for it requires complex interpretative strategies which are often theory-laden or culturally grounded. And to be valid, it must be related to its underlying formal models; otherwise, they will be used as black boxes about which anything may be said.

Conclusion Because computational models have a specific structure, they play a particular epistemic role in science. Indeed, they allow many sciences to transform the numerous complex calculations required by their formal models into effective procedures. And it is only if these effective procedures are formally formulated in an algorithm and eventually in a programming language that computer technology can effectively and concretely compute them. It then follows from these definitions that computation is per se different from computer technology. A computer is but a physical electronic means by which computation can be realized or implemented, and a computer can only compute electronically if it is given a set of effective procedures, one of which is algorithms or, in a more common manner of speech, programs. The distinctions will become important in the encounter of semiotics with the digital: the real challenge of semiotics is not the digital (which is only a useful encoding procedure), but its embeddedness in computationality. This raises an important question: Can semiotics be computational? Or, can computational semiotics exist?

154

10

Computational models in semiotics

Introduction Classical semiotic artefacts received their conceptual models mainly from the philosophical, structuralist and hermeneutic traditions. Even though these paradigms support formal models, very few semiotic projects will be linked to computational models. Still, a quick survey of these projects reveals that many have indeed integrated the use of computers within the conduct of their research. They indeed regularly use computers throughout their daily work, be it for purposes of text processing, media communication, navigating the web or electronic publication. Some might even use sophisticated types of formal computer applications for conducting linguistic or statistical analysis. Computers are omnipresent in the practice of semiotic research. But such uses do not necessarily entail that they integrate computational models into their conceptual framework or into the methodology itself. Projects that systematically and openly explore semiotic artefacts through computational models are the exceptions. Despite the richness and deepness that some formal semiotic models may have, linking them to computational models is not as evident as it may seem. Compared to the digital humanities1 community, rarely have semioticians2 integrated computational models into their theories or analytical practices. Over the last thirty years, some semiotic researches have made explicit connections with computationality. As we have seen in the previous chapters, one of the main trends originating with Newell’s material representation thesis has understood computers as symbol manipulating machines. And this thesis was popularized by Nadin’s formulation (2007, 2011a), according to which the computer is the prototype of a semiotic machine. On the other hand, many fields of the social sciences and humanities began integrating computational models into their study of semiotic artefacts and processes. We hence saw the birth of computational social sciences,3 history,4 economics,5 anthropology6 and, naturally, linguistics.7 But the discipline which is closest to semiotics has been the digital humanities. Here, fields such as literature, socio-cultural sciences, cultural heritage, theology, the arts and law have studied and analysed their objects assisted by computers. And there is a host of small and large projects which exist under this umbrella term. In some small projects, precise computational models have been used or developed for digitizing, annotating, archiving, searching and analysing specific and unique types of semiotic artefacts. Here are a few examples of original researches

156

Computational Semiotics

among many hundred others. One project found patterns in fifty-seven sermons that were given after Lincoln’s assassination.8 Another has studied 45,000 New York restaurant menus dating from 1840.9 Another has explored the art of early medieval China found in the  Xiangtangshan caves.10 And yet another analysed situated instrument design for musical expression.11 Even though they are not explicitly recognized as computational semiotics projects, they are still about a specific variety of artefacts that are meaning carriers. And their analysis requires conceptual, formal and computational models as well as computer technology. These few cases illustrate that projects need not be huge or be applied to ‘big data’ to integrate computational models into their methodology. But we have also seen impressive projects that developed computational models for digitizing annotating, database-storing and ultimately analyzing a huge number of semiotic artefacts and processes. And most of these projects could not be processed by hand. Good representatives of these types of projects are the collections of important Greek and Latin writings (the Perseus project12), of Medieval French and English writings (the Gallica project13), the UNESCO cultural music heritages14 and the project regarding the complete works of Shakespeare.15 One very important and huge project has been the Google Book Project, which, due to theoretical, cultural and legal problems, is now in a dormant state, although continuing to exist through the Google Book Search and Google Books Partner projects.16 It aimed to give access to a full database of scanned or digitized collection of books. Another important project of this type is the UNESCO and Wikipedia Foundation’s GLAM17 project, where galleries, libraries, archives and museums created a worldwide open-access database of cultural heritage institutions and their collections. And we find the same spirit in some similar associated organizations such as the International Council of Museums (ICOM),18 the International Federation of Library Associations (IFLA)19 and the International Council of Archives (ICA).20 And we see more and more local social, cultural and economic organizations getting involved in some type or another of social, cultural, legal, religious, economic, archeological or digital knowledge repositories. In these projects, data, documents and media are collected digitized, archived annotated, and many are openly distributed. It is unthinkable that such endeavours may not be ‘big semiotic data’ projects or that they do not call upon computational models to explore the data. And these models cannot just include search and retrieval algorithms. Be they small or huge, such projects open up a host of different types of semiotic analyses to semioticians, ones that necessarily embed computational models and algorithms. It is in this perspective that we will study how some dominant and important computational models and best practices can be heuristic in enquiries and in the analysis of meaningful artefacts and processes. They are found mainly outside of the world of semiotic research. In fact, computer science has indulged in these types of enquiries without really acknowledging their semiotic nature. We could even say that what computers process the most today are various types of semiotic artefacts. The most common among these are natural languages in all of their many forms and functions: written, spoken, discursive, conversational, communicational and so

 Computational Models in Semiotics 157 on. Other forms are also explored, many being media or multimedia such as images, music, films or videos, but some others are also various types of digitized semiotic artefacts from the arts and sciences. We could even say that computer science has built its own specific semiotic conceptual frameworks for dealing with meaning-carrying artefacts: it has constructed and applied a host of formal models that could be paired with computational models not only for analysing semiotic artefacts but also for simulating some semiotic processes. The magnitude of these researches should invite semioticians to take a serious look at some of the computational models developed in computer science for studying semiotic artefacts and to try to understand their core concepts and functions. Just as it happened in the classical sciences, computational models can provide semiotics with some other points of view to integrate into semiotic theories and enquiries. For instance, we can recall that seventy years ago, because of their complexity, environmental sciences, life sciences and neuroscience practically did not exist as such in the academic world. Their emergence as scientific disciplines owes quite a lot to the possibility of effectively computing some already known complex formal probabilistic and dynamic models. But, in turn, these effective computational means created new research objects which were supervened by new formal and computational models that the classical sciences could not foresee – precisely because of the lack of computational means. This gives a specific signature to computational semiotics. As we said earlier, the association of the term ‘computational’ to any type of scientific research field cannot just mean that a computer is used in it. For instance, economic science cannot be said to be computational owing to the mere use of computers in the course of its practice. Using a word processor, a database, or even a sophisticated statistical program such as SPSS does not suffice to make economics ‘computational’. The same type of mediation is required if practices such as psychology, biology, physics or linguistics are to be called ‘computational’. Still, things are not that straightforward. It is tempting for researchers to use some complex programs which are often ready-made without really understanding what they do or how they do it and afterwards claim that they are computational models in the research. We must therefore be rigorous about the relation that semiotics should entertain with computation. Nothing is gained if the algorithms are used as black boxes. If computational semiotics is to emerge, it must seriously understand what, where and how computational models intervene in its practices. And this is quite a challenge. But if it succeeds, it can provide a novel explanatory framework for semiotic theories and methodologies. As Rieger (1998) has said so well, it is semiotics itself that, in part, becomes computational. In this perspective, computational semiotics is a specific type of scientific endeavour. Its conceptual and formal models are very attentive to the complexity of meaning components and structures characterizing semiotic artefacts and processes. Its computational models translate these formal models into effective algorithmic procedures that are not without impacting on a large part of the principles, methodologies and results of semiotic inquiries.

158

Computational Semiotics

Epistemic roles of computational models in semiotics In semiotics enquiries, computational models have a specific epistemic signature for their categorizing and reasoning roles. It will have to determine, on the one hand, what type of input categories are admissible for its reasoning algorithms and, on the other hand, what type of reasoning algorithms can process the input categories that semiotic enquiries require. This means then that the categorization and reasoning roles are often deeply knitted together in computational models. For semiotic enquiries, computational models will call on the three main classical types of reasoning algorithms. The first type analyses their inputs in a deductive way – that is, they use general rules or schemas to infer individual instances of these rules. The second ones proceed more inductively – that is, they analyse their inputs to discover the schema, the patterns and the rules that underlie them. In formal terms, the first types of algorithms aim to find the semiotic instances that belong to a certain equivalent class, while the second type aims to discover the equivalent classes to which some individual instances could belong. Finally, some proceed in an abductive way; from a set of data, they generate some possible explanatory schema or hypothesis from which they attempt to infer other instances of the same type. In all three types of processes, the proposed algorithms must be adequate for categorizing their input and reasoning on them. But because of the complexity of the task involved, these algorithms must not be seen as governors but rather as assistants in the exploration of the meaning of the semiotic artefacts and processes under study. But as we know, these algorithms cannot directly categorize semiotics nor reason on semiotic artefacts and processes. They can realize this only through their symbolic surrogates. Therefore, an adequate computational model must proceed indirectly by the manipulation of symbols representing the features and properties of these semiotic artefacts. It so happens that this has also been the core thesis of computational models of artificial intelligence. In one sense, we could say both computational semiotics and artificial intelligence rely on computational models that contain specific and proper sets of algorithms for manipulating symbols that allow various types of categorization and reasoning process. Surely the set of algorithms assisting decision-taking will be different for analysing the meaning of rituals. But there will be many other cases where there exist some similarities between the set of algorithms used by IA and computational semiotics. Take for example: analyzing natural languages sentences, exploring music melodies and finding structures in paintings. All these types of artefacts are carriers of meaning on which adequate algoritms can be applied. Thus, we need to go into a more detailed but still general understanding of the various algorithms that computational projects have explored and developed for these types of reasoning and see it as a heuristic tool for computational semiotics enquiries.

Deductive computational reasoning models usable in semiotics The first type of algorithms that a computational model will offer for reasoning digitized surrogates of semiotic features are the deductive algorithms. These translate the formal deductive reasoning process into what has been called top-down algorithms.

 Computational Models in Semiotics 159 In a semiotic context, this means that the inputs of these algorithms will be (a) general statements about features of semiotic artefacts and (b) statements about individual semiotic artefacts. And the effective procedures of the algorithms will be to infer from the general statements the features that can be attributed to individual artefacts. Technically speaking, these top-down algorithms mainly use inference rules of the logical and existential instantiation. For example, if in a semiotic enquiry, there exist general statements about the features of Renaissance statues, then the algorithm should infer that if Michelangelo’s David statue in Florence is a Renaissance statue, it will have the features that the general statements attribute to the Renaissance statues. Or, conversely, if this statue has the features of this specific schema, it would then be a Renaissance statue. As we know, this type of deductive reasoning algorithms is the same that is at the heart of the earlier computational model of artificial intelligence. It was just formulated in different terms. Indeed, if in an IA project, algorithms are to manipulate symbols deductively, this will require two important components. The first one is that there must exist a set of schema, frames or formula that express general features, properties of individual objects, events and so on. The second one is a set of rules that express the inferential rules inherent to a deductive reasoning process and which are used for deducing systematically and productively the features and properties of individual objects, events and so on. For example, recalling our England knowledge data base on the England cities and mayors, an expert system is given input: the city of London should deduce from a frame declaring All England cities have a mayor that London has a mayor. These principles of intelligent computational systems were used to build and manipulate deductively knowledge representations. At first sight, these components may seem evident and simple, but it took much research to arrive at more and more sophisticated deductive algorithms for various applications. As we have seen in previous chapters, some of the first computational models for such types of algorithms were developed in knowledge engineering section of IA, which was defined as ‘the art of bringing the principles and tools of AI research to bear on difficult applications problems requiring experts’ knowledge for their solution’ (Feigenbaum 1977). Newell (1982) saw this knowledge representation as a necessary and specific cognitive architecture level for all artificial intelligence projects. Theoretically, this KR would be about any type of reality: mundane, physical, spiritual, imaginary, subjective or even about natural language. But in practice, it focused mainly on common-sense knowledge. It was even hoped that it could be a form of encyclopaedic and universal knowledge. The main challenge of this KR thesis has then been to find a language or a formalism that would encode the knowledge that allowed rigorous computational reasoning processes. One of the first formats proposed was originally highly influenced by classical predicate logic. It developed into a whole autonomous formalism called description logic (Levesque and Brachman 1987; Sikos 2017). But soon, a variety of formal structures of KR did emerge and aimed to express the basic subject–predicate relation between the data. To illustrate this type of logical formalism, let’s consider a conceptual model for a certain domain. We first express a sample of its features in a natural language sentence that could be found and retained as important for computer information management systems:

160

Computational Semiotics

All persons in the system must be associated with a name, an age, an education level, someone to which they are married and an occupation. A formal predicative language can be used to express this conceptualization in the following predicate logic language: ∀x ∃y ∃s ∃t (PERSON (x)  HAS NAME (x) & HAS AGE (x, y) & HAS EDUCATION (x, z) & IS MARRIED TO (x, s) HAS OCCUPATION (x, t))

At first sight, such a logical formula is seen as the ideal, formal expressive means to represent the knowledge needed for an intelligent computer system. It has all the rigour and well-formedness that is necessary for categorizing data and reasoning about them. A collection of such logical formulas is also quite isomorphic to a classical database and could come to form a knowledge database. Unfortunately, in sophisticated cases, this type of logical languages and formalisms have revealed themselves to be very opaque for humans to read and use. And eventually, they appeared quite complex for a computer to process, mainly in allowing clear and fluid reasoning. They needed to be translated into a more algorithmically fluid form. A first important proposal to simplify these knowledge statements has been that of ‘semantic memory’ (Quillian 1968). This KR model focused on semantic relations. It concentrated on a subset of relations between entities such as x IS y or x HAS y For example, ‘x IS single’, ‘x HAS education’ and ‘x HAS age 44’. Newell and Simon (1972), for their part, applied these ideas to express knowledge about planning actions goals, sub-goals, strategies, etc. Minsky (1977) proposed some refined format: the  frame models. Schank  and Abelson (1977) as well as Brachman  and  Levesque  (1987) presented  the knowledge as lists, scripts  and  directed  graphs. More and more complex models of this sort were proposed to represent different types of epistemic knowledge (e.g. belief systems). All would be amenable to some sort or other of graph representation. They became ‘knowledge graphs’. If we insisted on the details of these computational models, it was to focus on two important features. The surrogate symbolic systems that express knowledge representations are all de facto a sort of simplified formal language. The syntax is basic and their semantic is implicit: each formula contains some epistemic attitude: assertion, belief and so on. And each one refers to some world. But this is not the only important feature. The second one is an often implicit feature in the predicates used in the language. Many of them do not denote strick and well defined individual entities in a domain but more often they refer to stereotypes, prototypes of objects, persons, situation, events, scenes, processes and so on. In other words, these KR express implicit general statements and not individual cases. For  example, if we take the Minsky frames knowledge format: general or prototypical properties or features are expressed by the predicate words relations, by specific symbols such as ISA or HAS, and variables, by slots marked by ‘. . . ’ (Table 10.1). And when the slots are filled or instantiated by some individual value the frame becomes instantiated for a particular case  (Table 10.2).

 Computational Models in Semiotics 161 Table 10.1  A Minsky frame format with slots Person

Feature/property

Slot to be filled

HAS NAME HAS AGE  HAS EDUCATION IS MARRIED TO HAS OCCUPATION etc.

¨¨ ¨¨ ¨¨ ¨¨ ¨¨

Table 10.2  A filled Minsky frame Person

Feature/property

Slot filled

HAS NAME HAS AGE  HAS EDUCATION IS MARRIED TO HAS OCCUPATION etc.

John Smith 40 B. A. Mary Gold Teacher

Figure 10.1  Schank’s dependency graph.

Figure 10.2  Conceptual graph of a sentence.

Another important variant of these frame languages has been  Schank’s (1975) conceptual dependency graph. Shank’s model could express similar statements but in the form of a graph. This allowed adding features in statements in a more transparent way. For instance, Figure 10.1 is a dependency graph showing more details of the general schema about who marries who and where.

162

Computational Semiotics

What is again important to notice here is that the frame or graph-like knowledge representations aim at expressing in a more cognitively transparent manner the same state of affairs as the original classical predicate logic statements formulate. In other words, even if they use different formal languages, they have a referential equivalence. Slightly later, other variants of these KR models emerged which carried more logical and linguistic information. Sowa (1999, 2019) proposed a more complex graph formulation. For example, the sentence John goes to Boston by bus may be represented as it is in Figure 10.2. For complex sentences containing more and more semantic information, many other logical types of KR were proposed. Still, despite their rigorous structure, except for a toy example, these complex formats were not that adequate for dealing with the emerging tsunami of textual knowledge deeply embedded within the web. Other more adequate and convenient types of KR were required. They took the form of ontologies, which, technically speaking, are just more sophisticated variants of a logic-based language. There have been many definitions given to this notion of ontology (Guarino 1995). Grubers’ (1993a) definition of ontologies as conceptualization is the most cited one. But Guarino and Giaretta’s definition is even more specific and radical: an ontology is defined as ‘a logical theory which gives an explicit, partial account of a conceptualization’ (Guarino and Giaretta 1995, emphasis removed). Sowa (2000, 2010) later added that ‘the subject of ontology is the study of the categories of things that exist or may exist in some domain’ (emphasis removed). In other words, an ontology is a sort of knowledge conceptualization and categorization expressed in some formal computational language. It contains a network of meaningful ‘concepts’ or ‘categories’ that refer to some world. But one must see that this term ‘ontology’ is ambiguous – it is a metonymy. What is given here is a set of words, nodes and vertices that form a graphical discourse from which readers can extract statements. Philosophically speaking, we should say that an ontology is what this graph language refers to or, in Sowa’s terms, ‘what exists’. Let’s illustrate how these notions of knowledge base and ontology could allow a deductive approach in a semiotic enquiry. Let’s recall here our small domain of large statues of the world for which our conceptual model has provided for conceptualization by means of a set of natural language English sentences. (1) Michelangelo’s David is a huge-size marble sculpture of a nude male. (2) Rodin’s Thinker is a medium-size bronze sculpture of a sitting nude male who is in a thinking position. (3) The Motherland Calls is a huge-size bronze statue of a woman defiantly holding a sword. and so forth for the other statues. We have shown that a formal model could translate these sentences into many formal languages: predicate logic, generative type languages, Lindenmayer languages, mathematical vectorial languages and even in data frame form (Figure 10.3).

 Computational Models in Semiotics 163

Figure 10.3  Features of the statues in knowledge base form (a list).

In turn, all these formal knowledge representations can be transformed into an ontology visualized in the graph in Figure 10.4. This graph language contains nodes and arrows by which natural language ‘words’ are added. These ontology-like models have become the dominant formalization for the categorization of knowledge representations. And they have become important models for knowledge bases. And they now underlie the semantic web. The second components of knowledge representations pertain to the inferential rules necessary for rich deductive reasoning. In formal models, the one important logic inferential rule is the universal instantiation rule (∀x P(x)P(a)) to which some modus ponens and modus tollens are added. In a knowledge-based ontology, these rules allow the deduction to take some general statements or formula recognized as true and deduce from them the truth of individual statements. But to deduce such statements, some logical procedures must be activated, such as the equivalent of the substitution of variables by constants, modus ponens, etc. All these operations can be transformed in algorithms because the language of ontologies implement various procedures of formal predicate logic and are known to be computable procedures. Let’s return to our world statues example. Imagine that we had built a more complete knowledge base or ontology for myriads of statues and that we had described all of them by many of their features. It would have then been possible to easily deduce from them who are the sculptors of male marble statues, what are the genders for warrior or

164

Computational Semiotics

Figure 10.4  Ontology of samples of the world largest statues.

mythic statues, what are the features which distinguish religious or mythic statues, etc. If our knowledge base or ontology had included many famous sculptors and described them by numerous features – for example, their life, their work – it would be possible to infer who is the sculptor, for instance, of the Burgher of Calais or to ask which statues have similar features, and so on. Because of their rich deductive power, ontologies have become very popular. They are much more readable for humans21 than their database form. They are often expressed in the graph format that uses very classical mental and logical categories of subject–predicate–object akin to those of natural language. And as we shall later see, this will be the basis of important computable formatting that allows easy digital encoding and interchangeability between knowledge bases. Second, ontologies possess a rich set of standardized algorithmic creation and management tools22 which made them easier to use. They were hence found in many domains: pharmacology, physics, chemistry, medicine, urbanism, architecture, media and so on. They proliferated across industries and public institutions. They helped manage the inventories, databases, intellectual capital, human resources in aeronautics, tourism, commercial firms, health, entertainment, academic institutions and so on. They are already alive in the fields of culture (e.g. CIDOC, ICA, IFLA, EH).23 Surprisingly, they have even entered the field of humanities and art. Indeed, there exist ontologies for Wittgenstein24 and Shakespeare.25 And many of these ontologies are pertinent for semiotic enquiry. We could not imagine today a semiotic inquiry on Shakespearian language without the researcher exploring the Shakespearian ontology. There are also ontologies for

 Computational Models in Semiotics 165 music,26 architecture,27 digital humanities,28 multimedia studies,29 narration studies30 and cultural studies in the GLAM project.31 The ontologies also became even more important in certain specific domains such as in natural language processing. For example, WordNet (Miller 1995) has become the most useful lexical data frame. It includes various definitions of the lexical meaning of words and many semantic relations (synonyms, meronyms, hyponyms, antonyms, etc.) and different entries for the words of English (e.g. bank as a financial institution, as a water stream shore and as a snow heap). FrameNet (Fillmore 1968, 1982) is another type of knowledge base. It adds information about the semantic roles or cases of lexical items, such as agent, action, place, object, instruments, time, aims, paths and source. These knowledge bases or ontologies that a computational model may use for deductive categorizing and reasoning processes are not without deep and very problematic epistemological problems. They are often defined formally as effectively describing some actual domain. This gives them apparent objectivity and realism. But technically, as we have highlighted, they are foremost a conceptualization of a domain. In this sense, they are filters, mediators and points of view upon a domain. They are not surrogates of ‘what exists’, but rather a conceptualization of ‘what exists’. They are not ‘ontological’, – that is, metaphysical ontologies, but as Floridi (2009) formulates it, informational ontologies. And in many domains, this conceptualization is in the minds of experts, who may not overtly express it easily, clearly and objectively. And these experts are not always available for sharing it. Even if they were, experience in computer knowledge acquisition shows that communication and formalization of such knowledge are not granted. On the linguistic level,32 knowledge bases and ontologies are often entangled with all of the natural language idiosyncrasies they are expressed in (e.g. polysemy and taxonomies). Their building, updating, maintenance and updating is very costly in terms of both human energy and money. Finally, they are also heavy consumers of computer power. For these reasons and many others, they  have often become functional tools only for some specific, restricted or limited domain. Despite these harsh criticisms, the knowledge-representation approach has been revamped in more recent AI research. This came through research33 that, with the help of sophisticated emerging natural language tools,34 aims at automatically extracting, partially or completely, knowledge and information from texts – mainly from the web. This invited the representation of knowledge through ontology graphs. The strategy consists of annotating text from all sources with the RDF format (i.e. their subject– predicate–object) in the most automatic35 way possible. And as the OWL web language allows direct translation of RDF texts into ontologies, it is simple to express these in a graph format. This changes the view of the web. It becomes a huge knowledge base expressible in some form of ontology or another. This ultimately meant that a large part of the assets of the web becomes a neverending knowledge base that could be used for all sorts of enquiries. This view of the web invited many specialized knowledge graphs to be created. And they are well alive and are constantly updated. Examples are DBpedia, NELL, Freebase, Springer Nature SciGraph, YAGO and many others. Google took a slightly different approach. Through a natural language processing technique highly sensitive to context (BERT), it generates, practically on demand, a

166

Computational Semiotics

Google knowledge panel or ‘infobox’ (Figure 10.5). This panel translates linguistic data pertaining to ROME into a synthetic readable format where various pieces of information are given about ROME such as IS the capital of Italy, HAS a population of 2.873 million, and so on. From a semiotic point of view, these knowledge bases are often simple and basic types of knowledge. But a more positive way to see them is that if they were extracted from more sophisticated texts, they could, in semiotic inquiries, deliver part of the background or doxastic knowledge – that is, facts, beliefs and culture involved in all interpretations of semiotic artefacts or processes. For example, a semiotic enquiry of Spanish Easter processions, primitive cavern drawings, Stonehenge monuments or just the rituals of American anniversary parties or graduation ceremonies could surely benefit from knowledge bases built through research publications, reports or discussions, among others. In more classical terms, these knowledge bases could become the structured ontological repository of a part of the background knowledge necessary for their interpretation. For computational semiotics, an algorithmic deductive topdown approach may be seen as assisting the extraction of some Figure 10.5  Google’s output or ‘knowledge panel’ possibly pertinent information on ROME. from these refurbished knowledge bases. But as said in the previous paragraphs, semiotic enquiries would need much more refined knowledge content than the prototypical and standardized one offered by these public knowledge bases. For computational semiotics, the important point to notice here is not that these actual knowledge bases can be used as a repository of some standard and doxastic knowledge. It is rather that they can also be constructed for specific and pertinent semiotic projects and ultimately be shared by the semiotic community. In this perspective, a specific knowledge base pertaining to Shakespeare would surely be useful for a computational project on the semiotics of his plays.

 Computational Models in Semiotics 167

Inductive bottom-up reasoning model applicable to semiotic artefacts The second type of algorithmic models to be applied to the data submitted to enquiry is an inductive or bottom-up reasoning process. It postulates that data analysis must rest more upon learned knowledge than upon stipulated or a priori knowledge such as what is given in classical knowledge bases. For computational semiotic enquiries, this does not mean that prior knowledge cannot be used, but rather that this knowledge cannot already exist beforehand in the knowledge base itself. This would be a selfcontradictory enquiry. For example, an enquiry on the Stonehenge site could call upon some the knowledge acquired from the cultural and scientific background, but as such, it cannot contain the results of the ongoing enquiry itself! This has been one important criticism that the new artificial intelligence models have addressed to classical KR models. An intelligent agent is not just a manipulator of knowledge bases, however rich and rigorous they may be. It must also be a learner and creator of knowledge. From a formal model’s point of view, this learning feature rests upon the inductive operation, that is, upon a generalization inference operation. It takes for input a set of statements on individual states of affairs and produces generalized statements about them. A paradigmatic example is: If, among a set of crows, it can be said that this crow is black, this other crow is black and so is the nth one, and there are no other crows around, then it can be inferred that all crows are black in these surroundings. In choosing or building the algorithms for this inductive generalization process, a computational model meets two challenges. A first one is descriptive: the categorization role of a computational model must identify the type of individual data that are adequate for learning algorithms’ input. The second one is procedural: it must identify the effective procedures that deliver the general statement out of the set of individual statements. In other words, given a set of individual data as input, an inductive algorithm must extract or learn the classes, patterns or regularities underlying them.

Descriptive analysis In an inductive reasoning process, this descriptive task is slightly more complex than in a deductive one. Among the multiplicity of data, only certain ones will be admissible to inductive algorithms. For instance, if a grammatical parser is to be applied to a text, it can only accept words as input. This may seem evident for a first intuitive answer: is it not the case that words are indeed the admissible input of a linguistic parser? But this is not a simple question. Indeed, identifying what is a ‘word’ is not an easy task. Linguists and computer experts do not have the same criteria for determining what sort of data is a ‘word’ in a text. For example, should the sequence of letters French fries be considered as one or two words? And what about the German expression Selbstbewusstsein (selfconsciousness); is it one just two words? These examples show that some preliminary decisions have to be made regarding the data to be analysed if they are submitted to

168

Computational Semiotics

some algorithm. This requires that some descriptive analysis be applied to the data for discovering their characteristic properties or features. If the data set is small, it could be easily discovered by some basic human expertise. But if the data are numerous, which is often the case with inductive reasoning, some algorithms are necessary for discovering their pertinent characteristics. The first type of descriptive analysis is mainly structural. It aims at identifying data that are defined by some specific relational structure. For instance, a very classical descriptive analysis may construct a list of the basic types of data submitted to study. For example, a semiotic text analysis could require disambiguating words (river/ snow/financial bank), merging synonyms (Paris/France’s capital), solving anaphoric relations of pronouns (the dog frightened the cat, It . . .) and so on. In our statues example, the local descriptive analysis36 could identify some backgrounds or surroundings that are important features of certain statues. For example, one important feature of many motherland statues is their geographical location (on a hill, a mountain top, etc.). This type of local analysis is often presented by a graph37 of certain co-occurrence relations in the data set. Figure 10.6 provides examples of some co-occurrences of words in the Bible. It reveals that certain words should be retained with their embedding contexts. This means that some data are not just individual words; they are words with their network of associated words. For instance, individual data words such as εν, πρoσ, λογoσ, οντοσ, θεοσ, αρχη. Recent natural language pro­ cessing research38 has explored this contextual type of analysis. The Figure 10.6  Word co-occurrence network in Jn data to which some algorithms are 1.1-2 (Czachesz 2017). applied are not single isolated words anymore. There are now embedded words or phrases, if not whole sentences, texts and discourses. A second descriptive analysis is mainly quantitative and statistical. The descriptions, for instance, may take into account absolute or relative frequencies, probability distributions and information indices of data types or tokens. For instance, in some semiotic analyses, it may be useful to identify music data by the frequencies of the components making up their waveform. In theatre plays, an analysis may choose to exclude the least frequent characters. In texts, it is often necessary to identify words with very low frequencies (hapax) or non-significant frequently repeated words. In paintings, it may be pertinent to know the distribution of the size of some components or the number of iconic figures (persons, animals, etc.). Many of the statistical, categorial or relational descriptors are often used as criteria for ulterior analytical

 Computational Models in Semiotics 169 decisions. For instance, knowing which data features are statistically distinguished may be useful to eliminate them from some analyses. These samples show many types of structural or statistical descriptive operations. They aim to help researchers to identify as best as possible which feature can be pertinent data for inductive algorithms. For instance, having explored the context, certain ambiguous words expressing the features may allow choosing adequate ones for a specific type of analysis. In certain cases, it will eliminate features that do not seem statistically pertinent, while in others they may be kept as pertinent. Still, these descriptive operations are often an end in themselves. They are appreciated for comparing, identifying similarities, discovering similarities, variations or evolutions among the data. In fact, in computational linguistics, these descriptive analyses have led to the development of specialized methodologies; for instance, corpus linguistics,39 computational lexicography40 and textometry41 offer a whole set of conceptual, formal and computational tools (corpora, parser, databases, applications, annotations, dictionaries, ontology data analytics) for exploring natural language, and so on. These tools are not only used for parsing the syntactic, semantic, rhetorical or pragmatic content of texts but also make it possible to explore them in their various digital formats: books, newspapers, manuscripts, transcripts, communications, interviews and so on. Important non-linguistic e-tools are also developed for exploring images, sounds, films, music or multimedia,42 for building electronic archives and databases and for applying various algorithmic analytics. Computational semiotics cannot bypass this descriptive analysis. What is different in a semiotic enquiry is the type of data analysed: the ones that appear to be rich carriers of meaning. Semiotic enquiries can surely use the e-tools developed in these computational linguistics or non-linguistic fields. They can be used for describing and exploring the important features of semiotic artefacts. For example, computer-assisted semiotic analyses of wedding rings will have to identify the features that possibly make such physical objects semiotic artefacts. It is often in these features that a descriptive analysis suggests what the meaning hides. For instance, assigning the feature plastic to a wedding ring would surely reveal something different than the feature 18 karat gold! Assigning to a veil just one or few of the following features may reveal highly different meanings: worn on the head, around the neck, by a woman, by a man, by a Catholic nun, by a Muslim, on the street, in the desert, in the snow, at a cocktail party and so on. Even if these descriptive types of analysis may seem superficial for some semiotic analyses, they can be seen as a first magnifying glass through which to look at the data. They often change the observations made upon the data. But most of all, they raise issues and questions for deepening the research. Their heuristic value is part of the data exploration process. They open up new intuitions, explanations and understanding.

Data analysis Although descriptive algorithms may have revealed some basic structural or statistical regularities for identifying and selecting the data, a serious enquiry cannot

170

Computational Semiotics

limit an inductive algorithm to this type of analysis. It must also and mainly present the algorithmic procedures that support inductive reasoning on the data given or defined. Among the tasks of these data analytics algorithms, an important one is a sort of learning process by which the data are classified and produce new knowledge. They aim to discover regularities, patterns and schemas in the data under study. One customary way of saying this is that they create classes. Classifications are algorithms that organize data into classes or categories (Sebastiani 2002; Harnad 2017). In computer science, constructing these algorithms has been part of the main recent research trends of pattern recognition and artificial intelligence machinelearning objectives. In the pattern recognition field, the classes are seen as regularities: ‘Pattern recognition is concerned with the automatic discovery of regularities in data through the use of computer algorithms and with the use of these regularities to take actions such as classifying the data into different categories’ (Bishop 2006: 1). In artificial intelligence, the classes discovered have been understood as being general representations, aggregations and prototypes, and when they embed some hierarchy, they become schemas, networks and so on. And ultimately, because these classes express mathematical functional dependency relations in the data, their semantic interpretation sees them as underlying data rules or structures. In machine learning, classes become patterns: ‘we define machine learning as a set of methods that can automatically detect patterns in data, and then use the uncovered patterns to predict future data, or to perform other kinds of decision making under uncertainty’ (Murphy 2012: 1). These types of classification algorithms found their way into many fields, such as cognitive science (connectionism), information retrieval (data mining), natural language processing or robotics. They have become the new paradigm of artificial intelligence and are slowly lowering the role of the more classical knowledgerepresentation models. Dynamicist theoreticians ‘have found the notion of representation to be dispensable or even a hindrance’ (van Gelder 1998: 622). ‘Representation is the wrong unit of abstraction in building the bulkiest parts of intelligent systems’ (Brooks 1997: 139). In a conceptual model, classification was seen as a core classical cognitive operation by which the researcher identified some type or another of similarity between the features, properties, attributes or characteristics of the object or data under study. In a formal model formal, this classification operation was seen as a function that mapped individual objects onto classes sharing common features. Technically defined, this classification became a partition function based on equivalence relations. In a computational model, it was translated into a set of effective procedures or algorithms whose input are the features of each object and which deliver at their output a set of objects that share some common features. The various types of classification algorithms are differentiated by the metrics they use to compare the similarity or the difference that exist between the features characterizing each class of objects. Here, we can only present briefly some of the most popular and successful classification algorithms that have been used in artificial intelligence – mainly in its machine-learning branch. We believe that, provided some slight adaptation, certain

 Computational Models in Semiotics 171 concepts, parameters and methodologies can be highly heuristic for computational semiotics. Naturally, we cannot go deeply in the technical content of these inductive algorithms. We will however explore them through their general and main underlying concepts with the pedagogical objective of revealing their relevance for semiotic enquiries.

Machine learning and mathematical models There is a variety of classification computational models that have been explored in computer sciences. One of the most popular ones belongs to the machine learning trend of recent artificial intelligence research. It goes under various names whose semantics express different points of view of the classification algorithms. For instance, pattern recognition and data mining focus on the applications, while discriminant analysis and deep learning are attentive to the formal models underlying them. One must be careful not to conflate these terms or the notions they designate: cognitive learning, machine learning and statistical learning. Learning is a notion that comes from psychology, where it is defined as the process by which cognitive agents acquire and understand new knowledge: skills, behaviours, communications, concepts and so on. It is this notion we use when we say that animals and humans and even society are cognitive agents that learn. But we do not understand what could be meant by the assertion that plants or rocks learn. In artificial intelligence, learning is a notion that is used for describing the behaviour of a computing machine that acquires knowledge from its environment or from its own self. But the notion of machine learning does not have exactly this meaning. It is not cognitive behaviour, but a computational theory about the architecture and algorithms that can discover patterns and regularities in some data set (Mitchell  1999). The notion of statistical learning, for its part, is much more specific. It refers to a specific mathematical model (Mello and Ponti  2018) that calls upon descriptive and inferential statistics with or without probabilities. Naturally, in ordinary discourses, all these notions are tightly knitted together. Still, they are not identical in meaning. For instance, all learning processes are not necessarily computational and surely not identical to the ones that machines use to process data. And machine learning algorithms are not all based on statistical probabilistic learning models. For instance, a classification machine learning algorithm could proceed simply by mapping logical rules or schemas onto some data. These distinctions are important for computational semiotics. Its computational models are not about cognitive learning or statistical learning in semiosis. Rather, they make the hypothesis that machine learning algorithms (some of which use statistics) could be heuristic in the analysis of semiotic artefacts. But if computational semiotics is to use these machine learning algorithms, we must look under their hood to avoid treating them as opaque mechanical black boxes, without understanding exactly what they do when they classify semiotic data. Neglecting this understanding would quickly make computational semiotics into no more than a particular application domain of computer technologies. It would not be any longer a specific epistemic endeavour. It would not deliver an explanation.

172

Computational Semiotics

Machine learning theories propose that learning algorithms are a specific type of mathematical models. Some come from linear43 (and sometimes non-linear44) algebra and others from statistics.45 Still, one finds some mixture of both. Although non-linear models are theoretically adequate for building learning algorithms, they are not the easiest ones to use in machine learning. The difficulty seems to lie in their formal complexity. Modelling learning through topology or analytical geometry is no simple task. Still, because they do use vector calculus, these types of models are appreciated for their decomposition operations, regression strategies, similarity matrices, clustering and classification optimizations. Contrastingly, statistical models have become the most widely used mathematical models for machine learning algorithms. And two types of statistical models are called upon. A first one comes from descriptive statistics. These models are used to explore certain statistical properties of the data, such as their frequency, variance mean, distribution, dispersion, correlation, standard deviation and so on. They are mainly useful during the description phase we have underlined earlier. It is there that the data are identified, categorized and made admissible as inputs to the classification algorithms. The second one comes from inferential statistics. These models are mainly geared towards probabilistic models. They use pure mathematical models for modelling learning as a predictive process. Machine learning is then understood as a set of effective procedures to compute the most accurate predictions or approximations of the regularities, patterns or classes underlying the data.

Algorithmic classifying architectures The originality of machine learning does not only lie in the algorithms. It also presents architectures for organizing these algorithms. Two main types are distinguished: supervised and unsupervised learning. In both architectures, all algorithms contribute to the discovery of best classifications. The choice between the two depends mainly on the nature of the data and the goals of the research. Both types create their own specific algorithmic workflow. A first one belongs to the descriptive phase; they prepare and encode the data. The second one applies classifiers to the encoded data sets. In the context of learning, algorithms prepare the data that represent the object and their features in a very specific format. Once cleaned, selected and annotated, the data are encoded using a vector language. Each object and its features are expressed by a specific vector and all the vectors form a matrix. Vi   v 1 , v 2 ,K v n 1 , v n    v 1 , v 2 ,K v n 1 , v n   v1   v1   v2   v2  Vi   M    M   v n 1   v n 1   vn   vn     

 Computational Models in Semiotics 173

Figure 10.7  Matrix to a vector Cartesian space and a dot representation.

The vectors of this matrix can, in turn, be represented in an n-dimensional Cartesian46 space which itself can be transformed into a Cartesian representation (Figure 10.7). This last type of visualization of a vector space is very useful here for our intuitive understanding of the mathematical models underlying classification algorithms. The second set of algorithms is the core of classifiers. They apply to vectors some specific classifiers so that in the end, they are distributed into different classes. The Cartesian space in Figure 10.8 illustrates the results of applying the classifier to the matrix of vector features. To relate these quite technical Figure 10.8  Class formation of dot vectors. concepts to our semiotic enquiry, we see inductive reasoning as a process whose aim is to discover in the set of individual semiotic artefacts classes of artefacts that share similar features. For instance, in our statues example, this means that each statue is represented by a vector, with the algorithm finding a class of similar statues. In this example, it is easy to predict that the David and Thinker statues form a class. They indeed share more features than they do with the others. The Motherland Calls and Christ the Redeemer form another class. But if the data set contained a huge number of statues (if it was a big data set) and all of them showed many different features, the algorithm would have to use very complex procedures to discover the most adequate classes.

Supervised and unsupervised learning algorithms There exist two main types of classifiers. A first one requires some information about the data to be classified, and its learning is supervised. The other one takes

174

Computational Semiotics

the data as is and proceeds in an unsupervised manner to discover the classes it can best create. Supervised classifiers proceed by training the algorithm on a random sample dataset in which some well-formed classes are pre-defined or recognized: ‘Classification is supervised categorisation when classes are known’ (Hung and Wermter 2004). Their procedures aim at discovering functional relations that underlie the data and use them to classify the other unprocessed data. The learning is often a trial-and-error set of procedures. A typical example is the algorithm for identifying a person’s signature among many other persons’ signatures. In a first step, the algorithm is given a set of labels or recognized samples of a specific person’s signature. Through some training, the algorithm discovers some main patterns underlying the samples. Afterwards, when given the real yet unseen signatures of many persons, it predicts which ones are similar to the training cases of the known person’s signature. Such supervised classifiers mainly use classical inference processes, but they often add a specific metric to ensure a best or correct classification. A first formal mathematical model for this inference process belongs to probabilistic statistics. The function underlying some data is modelled as a predictive functional relation. The algorithm starts by extracting from the data samples some hypothetical pattern, and to each data it assigns a relative probability that it may be an instance of a specific class of data that share this pattern if certain conditions are met. From there on, it uses this hypothesis to classify the rest of the data. Some algorithms (binary decision trees, binomial logistic regression, etc.) split the data into two classes. The first one mirrors as best as possible the samples that have the highest probability of being in the class possessing the pattern. The second class contains the data that was left aside. In turn, the same decision process is applied to this separate class, and so on until all data have been classified (Figure 10.9). For this splitting operation, some classifiers will call upon more statistical criteria than just the probabilistic one. For instance, naive Bayes classifiers will take into account the dependency among the data while support-vector machines47 (SVM) are sensitive to their independency. One very popular supervised classifier called the k-nearest neighbours (k-NN) algorithm proceeds slightly differently. It Figure 10.9  Probabilistic decision for data chooses a k sample of data that are class splitting.

 Computational Models in Semiotics 175 the closest to a new data entry. And it continues in the same manner for the rest of the data (Figure 10.10). One different, special and highly important type of classifiers is the neural nets. These well-known computational models developed in more recent AI are used to simulate many types of cognitive processes: memory, perception, emotions, learning and so on. They have also become very important types of classifiers. Despite their name that likens them to the brain’s organization, they are formal mathematical models. Some components rest upon linear and non-linear algebra models while others Figure 10.10  K-nearest neighbours classifier. call upon probabilistic statistic models. Aggregated serially or in parallel, neural nets contain many different specific layers, units or modules, each one having its own specific parameters. Highly interlinked through multiple feedback mechanisms, they are very effective in classifying data. For this task, most neural classifies are supervised. They need training. This is usually done at the beginning of the process. The more data they are given, the more successful are the results. But there are now more complex neural nets that allow more sophisticated learning often called ‘deep learning’ (LeCun, Bengio and Hinton 2015). Because this architecture introduced discrete internal unsupervised training, tests and parameter changes in many hidden but simple internal units or modules, the quality of the classification was enhanced (Figure 10.11). Ultimately, this mixture of supervised and unsupervised architectures refined the performance of the neural classifiers (Bengio and LeCun 2007). The higher the quality of inputs, the more accurate the output classes. There exist many more types of supervised classifiers. The important point here is to understand that the core of supervised models are the specific mathematical models and architecture they activate. They need training. It is this training that conditions the discovery of the pattern underlying the classes. Afterwards, they use a mathematical model to make a probabilistic statistical Figure 10.11  Neural net classes.

176

Computational Semiotics

hypothesis on what other data could correspond to the template. These architectures and models may be useful for computational semiotics. But we must take into account that the training introduces important epistemological problems: both regarding the criteria used for retaining data (i.e. the variables) of the training phase and the probabilistic metric for predicting classes. And in the semiotic world, this may require important and fine decisions.

Unsupervised classification The unsupervised classification algorithms are quite different from the supervised ones. They are a set of procedures that also aim to discover the underlying functional relations that build a class. The difference is that there is no training on the data. These algorithms are a sort of blind discovery processes. They proceed by self-organization, often step-by-step, so that at the end, all the data are classified into various classes. A typical example is classifying a set of geometric figures of different sizes, forms, diameters, diagonals, perimeter angles or areas into triangles circles, rectangles and squares. The specific challenge for these algorithms is to identify what features can be retained for building the prototypical classes. The core of an unsupervised learning strategy is to find the best or optimal classes, but always in regard to the data given. Some algorithms start by const­ructing an initial cluster of similar indi­vidual vectors which in turn are re-clustered hierarchically into smaller clusters. Some other algorithms measure the simil­arities between all the features of the data. These procedures call upon mathematical models that belong both to linear and non-linear algebra or analytical geo­metry, but also to probabilistic statistics. For instance, they will use metrics such as correlation, distance from a centroid, proper vectors, shared probability distributions, hierarchical clustering or geometric angles. Depending on the academic discipline for which they have been created or used, these types of algorithms come in different shades. Each one offers variants of calculable equivalence function for building optimal classes. Here are just a few examples of these algorithms. One classical unsupervised algorithm is a hierarchical clustering48 classifier. Its core process is to start by creating an initial cluster through a distance metric that identifies the most similar set of features (Figure 10.12). Figure 10.12  Hierarchical clustering.

 Computational Models in Semiotics 177 Then, it leaves aside the others which in turn are re-clustered according to the same procedure. The process is continued until all data have been clusterized. A second rather classical type of classifier called principal component analysis (PCA) finds classes by splitting a set of points through the repeated minimization of the average distance of each point or pair of points to a best-fitting line (Figure 10.13). A third one, called k-means,49 calls upon geometry-like mathematical models. It proceeds Figure 10.13  Principal component classes. by projecting a sort of mean, gravity point or centroid onto a group of data. It tries to group them into ‘k’ number of classes. Each one is built by minimizing the distance from this centre point (Figure 10.14). There are many more types of similar classifiers that explore the distribution of words in natural language documents through these vectorial spaces, such as Latent Semantic Analysis (LSA).50 An important statistical variant uses probabilistic word distributions: Topic Mod­ elling Analysis using LL Dirichlet Allocation (LDA).51 In this latter model, classes of words are interpreted as revealing some Figure 10.14  K-means classifier. regular ‘topics’ present in the documents. Another algorithm uses asso­ciative classification.52 Inspired by the regularities in the transactional behaviours of buyers – for example, clients who buy A also buy B, it translates them into associative inference rules such as IF AB. When the data show frequent associations, some specific metrics allow the revelation of classes of associated data.

178

Computational Semiotics

Figure 10.15 shows that among a large set of geometrical shapes, some shapes, under specific conditions, tend to associate more with others. And, at a higher level, some classes of shapes associate only with some other specific classes and not with other ones. A semiotician reader could interpret these shapes as representing some type of semiotic phenomena: Figure 10.15  Associative classification. colours in paintings, people in ceremonies, partisans in political events and so on. It would seem that certain types of semiotic ‘entities’ associate more with some other types. And some groups associate more with certain other groups. Just like the supervised classifiers, the unsupervised ones may be useful for computational semiotics. But it will be in a different way. They will be highly pertinent when the semiotic data are not homogenous and seem to go in various directions. They help to produce hypothetical classes and generate in this way intuitions about the direction to take. Still, we see that these classifiers present their own specific epistemological problems: the original data may be so noisy that it generates a large set of hypotheses. With each unsupervised classifier being specific, it will deliver different classes. This has the effect of opening up multiple research directions. A more serious issue is that the various parameters that have to be fine-tuned (a priori number of classes, etc.) before the classifier can be applied. And in the semiotic world, this may require important and sensitive decisions which may deeply affect the results.

Abductive reasoning Despite their success in artificial intelligence, the deductive and inductive reasoning processes are not without problems. From a purely logical point of view, algorithmic deductive reasoning requires a priori general knowledge deposited in a knowledge base. But in practice, it has been given by the divine hands of programmers or some other source such as experts or through social collaboration. It therefore comes with all its biases. It may not correspond to a required task. But most importantly, it is not learned. On the other hand, inductive algorithmic reasoning is also problematic. First, the validity of the knowledge produced depends directly on the data given to the classifiers. And as research has shown, its results are not always evident and easy interpretable; many depend on the objectives embedded in the algorithms and the tasks pursued. Second, by logical definition, a generalization produced by induction is valid only if the reasoning had access to all individual instances. If this is not the case, the generalization is not universal. In fact, this is practically never possible except in closed worlds or situations. As we know, the solution proposed to solve this problem has been to introduce probabilistic generalization. In this perspective, classifiers can only ‘predict’ with a certain degree of probability what the

 Computational Models in Semiotics 179 patterns or classes produced will be and they always depend on the quality of the data they are fed. From the very beginning of AI, many researchers highlighted this problem – it was at the heart of most expert systems which were mainly deductive ones. Inspired by Peirce’s semiotics, Charniak and McDermott  (1985) explored means of using computational abductive reasoning in AI. For Levesque (1989), abduction should be understood as an algorithmic reasoning process that produces a best explanation. And influenced by Belnap’s logical entailment models, he formalized abduction in logical-epistemic terms: it produces beliefs in entailment propositions, such as ‘it is A BELIEF that (P=> Q)’. In other terms, the knowledge produced by abduction are hypotheses from data. A few years later, Paul (1993) offered algorithmic procedures for abduction, and Mooney (2000: 181) and his research group proposed to intimately combine abductive reasoning with induction. He saw that ‘these two reasoning processes [induction and abduction] can be integrated’. But this project was decelerated by the new and emerging trends of machine learning, neural nets, connectionism and parallel processing. These trends preferred inductive inferential processing. It became a way to model learning acquisition. As indicated in the preceding paragraphs, it was seen as the best way to discover things such as patterns or rules. So, inductive machine learning architecture took on the lead with the abduction reasoning process being cast to the background. Slowly but surely emerging out of the main trends, decisive attention was once again given to abduction. And abduction emerged again. As Zhou (2019) summarized it: ‘abductive learning provides a new framework, in which machine learning and logical reasoning can be entangled and mutually beneficial’ (Zhou 2019: 76101). The abduction, also known as retro-production, refers to the process of selectively inferring certain facts and hypotheses that explain phenomena and observations based on background knowledge. (Zhou 2019: 76101)

One original contribution of Zhou (2019) is that the hypotheses generated can in turn be added to a repository or a knowledge base that includes already accepted statements and other hypotheses. In turn, if these various types of statements are well formulated (RDF knowledge graphs, etc.), they could be compared to other statements and hence adjusted or corrected. Dai et al. (2019) and his group formally called this reasoning processes abductive learning (ABL). It created an encounter between inductive knowledge learning and logical description. It built a bridge between two classical descriptions of AI: ‘the abductive learning [is] targeted at unifying the two AI paradigms in a mutually beneficial way, where the machine learning model learns to perceive primitive logic facts from data, while logical reasoning can exploit symbolic domain knowledge and correct the wrongly perceived facts for improving the machine learning models’ (Dai et al. 2019: 2815). The integration that abductive learning brings to reasoning processes builds a hybrid learning machine. Induction is applied to data. Hypotheses are generated. Deduction is then used to recognize new incoming data. When the statements of the knowledge repository are stabilized, then it becomes itself a knowledge base. And as this abduction process may be continuous, the knowledge base is contin­uously enhanced. From this

180

Computational Semiotics

refurbished knowledge base, new and original deductions can follow and produce new data that enrich the original database upon which, once again, abduction may be activated. This creates a mov­ ement in learning that can become, as some have called Figure 10.16  The abductive learning spiral. it, a ‘never-ending learning’ process (Mitchell 2018). Hence, by this understanding, abduction becomes part of an important and dynamical spiral learning process (Figure 10.16). Experimental results show that ABL [abductive learning] generalise better than state-of-the-art deep learning models and can leverage learning and reasoning in a mutually beneficial way. (Dai et al. 2019: 2816)

This view of abductive learning has prompted a renewed view of deductive and inductive learning. More and more research is being done on the formal, computational and epistemic nature of these reasoning processes. We cannot delve here into this important research. But we cannot continue to see rule-governed deductive learning and datadriven learning of knowledge as two great solitudes, or worse, as two contradictory computational models. This dichotomic view is slowly breaking down. Both approaches are renewing the understanding of the nature of their content and architecture and stressing their interdependence and calling for collaboration. We see databases and knowledge bases moving towards each other. On one side, the data of classical databases that are the input of these two algorithmic reasons are becoming ‘smart data’. Bare data get closer and closer to annotated data. For instance, data add information about the domain to which they are linked and they express more explicitly their many types of mutual relations. In other words, data cannot be only a bunch of whatever comes to hand. Data must become smart by being annotated. This was achieved by annotating the data using the logical predicative categories of the Resource Description Framework (RDF). Because classical databases frame implicitly embedded entities and relation categories, with the help of the OWL language, it was not a big leap to integrate RDF annotation. Many databases then were transformed into the ontologies and ultimately into graph ontologies. So many classical databases became smart by adding an RDF-based ontology to their data. With regard to traditional knowledge bases, a similar path was followed. From their early frame-like models, knowledge networks and conceptual net lists, they slowly moved into the realm of ontologies by conceptualizing their content through the same basic logical structure of subject–predicate–object as the RDF format. In other words, the various classical knowledge representation formats could all be formalized

 Computational Models in Semiotics 181 into some graph language with an underlying RDF structure which in turn could be expressed in an ontology called a knowledge graph or knowledge base. The impact of this transformation of classical knowledge bases into formal knowledge graphs and ontologies allowed them to integrate or mirror rich knowledge repositories, which were already annotated in RDF and expressed through graphs and ontology formats. This was the leap that the Semantic Web took. In doing so, RDF annotation and the Semantic Web transformed the web’s textual data into a huge knowledge base by using knowledge graphs. In parallel, we saw things such as dictionaries and encyclopaedias transformed into knowledge bases and even into knowledge vaults: DBpedia,53 Freebase,54 NELL,55 YAGO,56 and the ‘defunct’ Google VAULT.57 Many academic and scientific institutions and industries have entered the game. An increasing number of academic projects, including in the fields of mathematics58 and health,59 attempt to have their knowledge, their textual and visual heritage, and their domain-specific wikis changed into knowledge bases that can be manipulated using computers. The cultural domain is not inactive in this field: the GLAM project implicitly sees itself as building specific cultural knowledge bases. New integration research projects have shown that ontologies and web knowledge graphs could be a bridge between databases and knowledge bases. And the learning involved in building this bridge should be abductive.

Algorithmic reasoning in computational semiotics Can these various reasoning computational models penetrate computational semiotics? A first answer is a classical but inescapable one. Indeed, without active deductive, inductive and abductive reasoning models, no semiotic enquiry is possible. A rich semiotic enquiry is necessarily an epistemic type of semiosis. And, somewhere in its deployment, it must induce generalizations that propagate into individual instances. But most of all, it will open up onto creative hypotheses and ultimately onto an ‘act of insight’ which, as Peirce60 resolutely defines it, is the core of abductive reasoning. A second answer, more technological, is that surely computational semiotics cannot completely transform these reasoning processes into algorithmic procedures. Reasoning in not at all computable! But this does not mean that computational models cannot intervene as a sort of oracle during the reasoning process. And as the semiotic world itself becomes more and more wrapped up in digital technology, it will not stop making inroads into the latter’s territories. As these reasoning processes are deeply embedded within semiotic practices, computational semiotics should try harnessing the oracle’s heuristic power to assist the semiotic reasoning processes rather than seeing it as something which threatens it with being overtaken by pure autonomous ‘thinking machines’. And as semiotic knowledge bases proliferate more and more, be it with or without the will and expertise of semioticians, they become unavoidable e-knowledge library ‘vaults’. In such an environment, computational semiotics may indeed become a highly appreciated mediator between the digital and the semiotic world.

182

11

The workflow of computational semiotics

Workflows as pipelines At first glance, a computational model in semiotics may be seen as the application of a Turing machine for the study of semiotic phenomena. For classical semiotics, such an approach may seem horrendous. One may wonder how an electronic mechanical device could be an adequate means for understanding and explaining semiotic artefacts and processes. But such a view rests partly upon a misunderstanding of what a Turing machine is. One must remember that computation is not a concrete mechanical device as such. It is an abstract theory for calculating a specific dependency relation called ‘functional relation’ that can eventually be mapped onto certain components or features of a problem or situation. And as Turing, Gödel and Chaitin have shown, only a subset of these are computable functions. And even if some calculable functions are available, it does not follow that they can be easily translated into algorithms. Complexity is swift to enter the game, be it computational complexity or formal complexity. Hence, not all formal models can become computable ones, though there are an infinite number of such functions. Chaitin, Doria and da Costa (2012) have shown that this non-computability issue pervades all sciences. So, if this is the case, we may expect it to be even worse for semiotics. This was also the intuition of many semioticians: computational semiotic enquiries are not all algorithmic (Eco 1965, 1979). They are not entirely epistemic computational processes (Meunier 2014). This means that what computational semiotic models can best hope for is to pinpoint some algorithms that express specific computable dependency relations or functions. This radically constrains but does not limit the number of algorithms that can be called upon for computational semiotic enquiries. It is therefore the role of a computational model to choose the types of algorithms that can effectively be used in semiotic enquiries. Failing this, the risk will be that a computational semiotic enquiry will choose algorithms off the shelf and use them as black boxes. All algorithms can indeed be fed some data and deliver some data as their outputs. This is what algorithms do – and they do it well. But nothing guarantees that the results will correspond to what was written on the algorithms’ label, and, more importantly, that these labels will correspond to the goals of the semiotic enquiries. In most cases, computational models can only offer algorithms that approximate as best

184

Computational Semiotics

as possible the actual practices of semiotic enquiries in which researchers hypothesize that there exist some functional dependency relations. Therefore, we propose a general computational model that presents a set of algorithms which, at least in what concerns their underlying formal models, can assist the categorization and reasoning processes used in classical semiotic enquiries. But it also must offer algorithms that have proven themselves to be fit for processing semiotic artefacts and which adhere to best practices in the field of computer science, mainly in artificial intelligence. We believe that the best way to present these algorithms is by seeing them as a sequence of algorithms or workflow commonly called a pipeline.1 The whole sequence of algorithms can be seen as meta-algorithm. What is not algorithmic is the choice of and the specific sub-algorithms and the parameters of each one. The pipeline built is illustrative. It presents only a typical and general architecture for possible paths. We focus on the main algorithmic components. In Figure 11.1, we illustrate this pipeline in a diagrammatic form. It contains the most important and typical operations involved in computational semiotic enquiries. We will explain in more detail some of their main sub-operations, keeping in mind that most of our readers are semioticians. This pipeline presents four main modules or phases that a computational semiotic enquiry usually requires. The task of the first module is to gather the semiotic artefacts to be studied. A second one curates the corpus based on the meaningful features of the artefacts. The third module uses various types of algorithmic analytics. This forms the core of a semiotic enquiry. The fourth module interprets the results of the analyses. The fifth evaluates the whole enquiry. We may conceptualize these modules as specialized hubs of algorithms which, alone or together, in sequence or in parallel, assist the core tasks of classical semiotic artefact analysis. The first module is one that aims to gather the semiotic Data Gathering artefacts to be studied. For instance, a semiotic enquiry could be interested in semiotic artefacts such as films, architecture, monuments, social behaviours, music or novels. A collection is a choice among all such artefacts. But strictly speaking, a collection is not the set of the artefacts themselves, but a set of surrogate representants. Indeed, a computer cannot deal directly with the collection of such artefacts. This is the role of libraries, museums, archives, the internet, etc. The real inputs of algorithms are only their digital surrogates which are binary codes. A host of algorithms are called upon for digitizing the original artefacts. Some will be encoded as the result of camera captors. Some will be internal to a scanner of which the output will be transferred into a database. Maybe some optical recognition will be used. Still, some may already be semiotic artefacts digitized by direct insertion (keyboard typing) into some database or taken from the Web. They can also be extracted and collated by crawlers2 and scrapers,3 for example. Artefacts become digitally encoded when they are transformed into a sequence of electronic binary signals (positive/negative) that are afterwards encoded into digital symbols (0/1). In computational languages, these encoded signals are the sole ‘data’ to which a computer has access (Figure 11.2).

 The Workflow of Computational Semiotics 185

Figure 11.1  A typical computational semiotic enquiry pipeline.

Constructing such a digital collection can be a computational project in itself. It has indeed been the endeavour of thousands of digital humanities projects. Digital collections resulting from these projects are available and are open-source. Other digital collections have also been built by numerous large public institutions and cover practically all fields of semiotics. We can think here of the British Museum’s archaeological collection4 or of the Louvre’s collection of paintings,5 for example. They may be of various types such as manuscripts (Newton’s alchemy/chemistry papers6), old books (Diderot’s and d’Alembert’s Encyclopedia7), internet posts (Trump’s tweets8),

186

Computational Semiotics

Figure 11.2  Transcoding of the Mona Lisa into binary form.

audiovisual documents (from the National Film Board of Canada9), sound archives (Duke University’s music databases10) and so on. More often, though, each research may have its specific collections of digital semiotic artefacts, such as cultural archives,11 dance archives12 or national folklore songs, tales and legends.13 There are international networks and research centres specialized in networking a myriad of such digital collections.14 To manipulate these data, many types of peritextual metadata have to be added. For example, they may be descriptive of the artefact’s identification: name, date, source, authors and so on. Other metadata may pertain to their structure (versions, formats), to their identifiers (ISBN, DOI, keywords), to their control (PIN, security, access) and so on. And some statistical information may be added (length, measurements, types, tokens, etc.). Some of these metadata may be included in the data frame itself or may have to be added after having been ingested. In certain disciplinary fields, international standards have been proposed, such as the guidelines of the Text Encoding Initiative (TEI),15 the standards of the World Wide Web Consortium16 or various best practices for annotation. The collections may be managed using various database management systems (DBMS) or tools for managing a particular type of data (text, numbers, photos, etc.), operations (mathematical vs structural) or computer technology (web, cloud). Any semiotic project that would call upon computers must go through this first set of algorithms. And one must not neglect the complexity of the task involved in the construction of such digital collections – it is often in itself a complex computational endeavour. Building such digital collections of semiotic artefacts has become typical of projects which, despite being designated by other names, are indeed computational semiotic endeavours. For example, artefacts that are called ‘books’ are semiotic because they are carriers of meaning. One such collection building project has been the Google

 The Workflow of Computational Semiotics 187 Book project. Unfortunately, as many critics of this project have highlighted,17 the data and metadata seem to be more sensitive to their physical carriers than to their semiotic content. But some other institutions and academic projects are more sensitive to the semiotic content. Today, one finds a great number of collections where the semiotic data are enriched with semiotic information. Prototypical of this are the semiotic artefacts found in the UNESCO’s digital heritage preservation database.18 Finally, there is the last set of algorithms whose task is to construct a corpus. A corpus is different from a collection. The latter is a set of a digital artefacts, while the former is a subset of these but chosen on the basis of their semiotic content and form. Defined by the objectives of the project, a corpus is individuated by the conceptual and formal models involved in the various theories, decisions and criteria underlying the research objectives and analysis methods. For instance, in view of studying the use of light in the collection of J. M. Turner’s paintings, a corpus may be based solely on the Venice watercolours. For the study of movement, another corpus may be built out of his oil paintings. In other words, the same collection can be the source of many corpora. Once a corpus is built, the second type of module is activated. Its Data main task is a curating one. Curaon Indeed, it is not because a database contains digitally encoded music, texts or films that they can automatically become ‘inputs’ to some algorithms. They must be prepared to become admissible inputs to the analytical algorithms they are to be applied to. Here, the categorization and reasoning process defined in the formal model starts playing an important role. For instance, if one aims at determining what is the style of a sequence of digital music, some specific data will have, by some means or another, to be identified as ‘notes’, ‘pitches’, ‘rhythms’ and so on. If a project aims at finding topics, sentiments or arguments in a text, depending on the formal model chosen, it will have to identify what is a ‘word’, a ‘text segment’ and the like. This is a task that requires quite a variety of curating algorithms. We present here only a few samples of the algorithms which may be involved. A first set of algorithms assists in the cleaning of the digital data. Noise, errors and omissions can be corrected by hand or assisted by dictionaries, thesauruses or lists. For instance, in digital text, optical recognition can produce errors and misspellings that have to be corrected. In digital music, signals may be fuzzy, disrupted and so on. Another set of algorithms helps to annotate the data: they add information. These take the form of metadata often called ‘tags’. Depending on the nature of the semiotic artefacts or processes, the added information can be structural, semantic or pragmatic. The structural annotations identify various types of syntagmatic or paradigmatic relations in which the data can be embedded. For instance, for a text, it may be partof-speech tagging. With musical signals, the annotation may refer to tempo, amplitude and so on. An increasing amount of data coming from the semantic web may be tagged using RDF (resource description format). Depending on the semiotic conceptual framework used in the research, semantic annotations may insert intra-type meaning features. For instance, for paintings, the annotations may be about plastic, figurative and cultural features; for texts, they may be about semiotic features or semes; for

188

Computational Semiotics

music, they may be about tonality or style, for instance. For their part, the pragmatic annotations may belong to external meaningful information – for example, the user, the situation, the context or the cultural background. For music, these may indicate the source, the actual uses (e.g. in films), the composer and so on; for texts, they may be the comments, the reviews and so on. For rituals, they may concern the participants or the institutions involved, for example. Some of these tags are manually attributed while others can be ready-made and come from various sources such as ontologies, dictionaries or thesauruses. Digital semiotic artefacts can suddenly find themselves dressed with whole trees of tags of various sorts. When the data are huge and well tagged, their processing may require some reduction. Statistical description algorithms may be called upon and applied to the data to eliminate certain non-significant or non-relevant features either because they are too scarce or too redundant. For instance, in natural language, functional terms may be eliminated or certain linguistic forms may be reduced to their morphological roots, for example, ‘lover’, ‘loving’, ‘loves’, ‘loved’  ‘love’. Finally, before applying the data analytics algorithms, a project will require some exploration tools to navigate the curated corpus and test some of their hypotheses in order to refine them and check for the various problems they may encounter during the enquiry to be carried out. Just as with the preceding module, one must not neglect the human and computing power required in the project. One must be sure that the corpus is an admissible input in consideration of the enquiry’s specific objectives and nature. And one must also confirm that the analytic algorithms are the right ones for a particular corpus. Once the data collection representing semiotic artefacts have Data analysis been archived and the corpus has been pre-processed, the most interesting module may come into action: the procedural analytics module. This module contains various custom or ready-made categorizing and reasoning algorithms related to the formal logical or mathematical models retained for these epistemic tasks. They often profit from the annotations provided in the preceding curating module. Here, they will add some statistical characterizations for determining if the data may contribute significantly to the analytic reasoning algorithms to be applied. For instance, some will identify for the features their information index, their distributional amplitude, their absolute or relative frequency count, their degree of probability and so on. This will often determine the importance of the features relative to the analytics to which they will be submitted. For example, a feature that is evenly distributed across all the artefacts may be very important for deductive analysis but not pertinent for a discriminant inductive classification. Another type of descriptive categorization will be related to the strength of the co-occurrence of some features with other features. It can also take into account the context in which data are embedded. For example, a ‘yes, we can’ sentence might need the context to be specified, for its meaning may be dependent upon the enunciator: a president or a sports coach making a speech. Some structural information may also be necessary for ulterior finer analysis. They may call upon grammar, institutional policies, professional standards, dictionaries, thesauruses, etc.

 The Workflow of Computational Semiotics 189 But most importantly, these various descriptive categorizations are often highly interlinked with the algorithms they will be submitted to. For instance, if deductive reasoning is used, this will often require that the categorization of data make them compatible with some conceptual knowledge base that contains logical formulas, frames, conceptual nets, graph ontologies, etc. On the other hand, if an inductive reasoning approach is chosen, then specific data categorizations will be needed to make them expressible through vectorial or algebraic languages. Once the descriptive categorization has been sufficiently completed, what follows is a choice between the main types of sets of algorithmic reasoning analytics. They form the core of computational models for processing semiotic artefacts: deductive and inductive reasoning. Some recent research even explore abductive reasoning. The first type, that is, the deductive one, presents the data in computable formats such as frames, ontologies or, more generally, knowledge bases. The language chosen allows them to express general and individual knowledge statements. But, more importantly, they support deductive inference to create new knowledge statements. The second set of algorithms transforms the data into a vectorial representation and applies to it some chosen inductive analytic algorithms. The most popular ones are based on statistical and linear or non-linear algebraic classification models (pattern recognition, machine learning, deep learning, etc.) so that schemas, patterns or general classes may be inductively inferred. The third module introduces the interpretation phase into Data computational semiotic enquiries. Interpretaon Interpretation is the epistemic process through which an enquiry goes beyond description and explanation and where the results obtained are integrated with the researchers’ personal, theoretical, pragmatic and cultural background. It is the moment of understanding. It forms an essential part of semiotic enquiries. Typically, it is not an entirely algorithmic process. Depending on the model chosen for the analysis, the interpretation may be distant (Moretti 2013; Underwood 2017), in the sense that the interpreter is confronted with synthetic results from large amounts of data, which widens the scope of the reading. This will be the case when the corpus under study is of the big data type. For example, if one has access to the complete digital collection of Mozart’s music, then computational semiotics may seek to discover dominant musical patterns for certain periods of Mozart’s life and compare them with musical patterns found in music composed by his contemporaries. Here, visualization tools are often appreciated. They present the results through graphical representations which summarize them in an analogical manner. But some interpretations may require a close reading. Indeed, having discovered some general patterns, one may wish to extend the interpretation by returning to the various semiotic artefacts that are instances of these patterns. In such a case, the corpus will be of limited scale. Still, a lot of energy may be required to dig into it. For instance, it may be the case that a particular musical structure associated with Mozart may be present in only a few of his works. A close interpretation would then aim to explain how the various factors (formal, artistic, social, cultural, phenomenological, etc.) gave rise to such a pattern. Here, formal and computational models and tools can assist the fine-grained analysis.

190

Computational Semiotics

It is often at this level of analysis that the descriptive and knowledge-based approaches are heuristic. They are the repository of the expert knowledge to be called upon for rich and profound analysis. But because the interpretation is not all algorithmic, it is the intuition and expertise of the interpreter which will lead while the computer serves as an assistant. Still, in contemporary research where knowledge bases become increasingly accessible and content-rich, the researcher’s personal interpretation may algorithmically be put into relation and compared with a myriad of other interpretations deposited in knowledge bases. This will change the hermeneutics of semiotic interpretation. Finally, the last module comes into action, because in Evaluaon computational semiotics, personal interpretations are not always sufficient. This is part of the epistemological scientific standards of computational paradigms for formally evaluating the course of the enquiry itself, which can be evaluated either internally or externally. Internal contradiction, mistakes and the like have to be identified and if possible corrected and improved upon. Externally, results are shared and submitted to the epistemic community for acceptance and criticism. For these stages of the research, there are a variety of computational models and tools that assist the reading, the interpretation and the evaluation of the results. Some tools produce quantitative measurements and indices regarding recall, precision, coherence, accuracy rates and information loss in the enquiry. And there are external evaluations which are, for their part, grounded on benchmarks, standards and performances. Still, and ultimately, because of the complexity of semiotic artefacts and processes, computational semiotics will call upon some type or another of qualitative evaluation where a subjective understanding will psychologically acknowledge and recognize the objectivity of the research.

Computational models in our case studies To help us in the concrete understanding and possible usage of computational models, and more specifically that of a pipeline, we continue here to present our three case studies on iconic, indexical and symbolic semiotic artefacts. The sampling of experiments we present here illustrate how we have used inductive reasoning algorithms in three computational enquiries on semiotic artefacts: painting, music and texts.

Iconic semiotic artefacts: The Magritte paintings Our first case study is the computer-assisted analysis of Magritte’s paintings. The data collection for this project was made of 1,780 paintings by Magritte. The corpus was identical to this collection. During the curating phase, the conceptual model had built a set of general semiotic descriptors which were used for annotating each of the Magritte paintings. Afterwards, during the analytical phase, computational models expressed the formal models through algorithms and implemented them in a programming language. Often, many such mathematical operations are bundled up

 The Workflow of Computational Semiotics 191 into packaged programs or modules that are easily accessible to researchers. There are hundreds of such program modules available (some are proprietary and some are open-source). In the Magritte project, some of the programs came from open-source libraries but most of the main ones were custom-built using Python or R. Modules were added to the main programs for the graphical representation or ‘visualization’ of the research results. A well-known type of visualization is the word cloud showing the most important words expressing the main features of a class. A few experiments on both types of analysis were made on the corpus of the tagged paintings. Here are some examples of experiments applied to this corpus. The first set of experiments (Chartrand et al. 2013, 2018) applied a classification algorithm to a subset of tags containing the specific descriptor ‘woman’. One experiment produced five classes out of which the most important descriptors were chosen as the most specific representatives (by some criteria) of the content of the class. And a formal graph of relations was applied to show the relations between the words of three classes (Figure 11.3).

Figure 11.3  Graphs of the descriptors of nudity in Magritte’s paintings.

Semioticians will probably see a dominant theme or signature of Magritte in this graph. We notice that the descriptors used in the tags often characterize the female figures on the basis of specific bodily positions and sexual traits. For instance, they are sited, kneeling and often mostly nude. Other experiments (Chartier et al. 2019) applied a similar analysis to tags containing the descriptor ‘man’. Here, contrastingly, the descriptors were veste (jacket), chapeau

192

Computational Semiotics

melon (bowler hat), pantalon (trousers), oreille (ear), col (collar), chaussure (shoe), cravate (tie), visage (face), position de dos (back view), cheveux (hair), moustache (moustache) and nuque (nape). A structuralist semiotician will see here a paradigmatic opposition within the semiotic space formed by Magritte’s paintings in which women and men occupy two opposite areas, as women appear naked, kneeling and wounded, while men are well dressed, standing up and in good shape. We leave it up to readers to draw their own conclusions. Another experiment on a similar subset goes slightly further. It explores the complementary subset of the paintings of which the tags do not contain the descriptor ‘man’, but which still present the signature of ‘man’. In other words, it asks the question whether there are some paintings of Magritte where the man’s syntagmatic signature is present although the figure of a man is not present. This would mean that there is some implicit or not easily detectable paradigmatic signature.19 The algorithms discovered that at least four paintings in the collection presented some similar signature but that was based solely on the some abstract structural iconic components of the paintings. Two of them showed a man viewed from the back,20 a third one was man’s body but with an apple instead of the head.21 And a fourth one contained just a long beard under a man hat.22 The authors of the experiment offer some interpretation of these results – a complex rhetorical phenomenon that opens up an ontological distinction among iconic visual signs: the figurative iconic sign and the abstract iconic sign (with low figurativeness). This distinction is reminiscent of the opposition between the iconization and abstraction of visual signs (St-Martin 1987; Groupe μ 1992; Floch 1981, Sonesson 1988). This is a very interesting and unexpected result. It provides new insights for future works in computational semiotics applied to meaningfull iconic artefacts.

Indexical semiotic artefacts: Music The music analysis is our second case study. Recall that the formal model proposed was related to music recommendations. For example, someone who listened to X also listened to Y, Z. For this task, the computational model included three related corpora. The main formal model was a set of associative rules. And the computational model easily implemented them into specific programs. One of the advantages of this model is the traceability it offers in pattern-recognition tasks to be conducted. In most music recalls and recommendation systems, the criteria used are mainly based on the artists and musical genres (jazz, rock, classical, etc.) previously selected by the individual listener and by other users. In comparison to this classical approach, the research showed that a combination of information extracted from the three sets of descriptors (canonical acoustic terms, sociocultural categories and terms from comments) enhanced the quality of the recommendation. But in regard to computational semiotics, this research illustrates how the formal and computational models can be used for semiotic analysis. First, it shows that like many others, these algorithms can identify structures in purely physical signals. Each piece has a unique signature which looks like the signature in Figure 11.4.

 The Workflow of Computational Semiotics 193

Figure 11.4  Waveform signature of a musical piece.

This signal visualization was generated by taking the output of captors applied to an audio source and by transforming this output, using transducers, into binary electronic signals which in turn were translated into digital code (i.e. 0, 1). This is a classical application of computer technology. It is at the basis of all digital signal converters and is used for digital images, sounds, face detection, robotic vision and so on. From the point of view of computational semiotics, what is interesting here is the recourse to algorithms that can add metadata to these encoded signals with various types of semiotic information. This way an acoustic signal may automatically be given descriptors originating from three distinct types of discourses: the first one pertains to its physical dimension, the second one has a sociocultural basis and the third one pertains to the artistic features. One can easily imagine that more refined descriptors could be added to each of these three sets of descriptors. Much semiotic research in music has developed sophisticated conceptual models of the phenomenological experience of music listening. This could enrich the set of descriptors for musical artefacts. For instance, Adjiman (2018) has looked into semiotic theories to ground a taxonomy of background music. As we can see, there is something similar here to the Magritte project. Future technology will surely develop more and more sophisticated image and music recognition techniques, in addition to associating words or even natural language sentences to images, sounds and films – whether or not computer scientists recognize this to be a typical semiotic operation. Automatically adding the name ‘Trump’ to an image of President Trump following the application by a computer of a facial recognition technology is a computer-assisted semiotic operation. The proper noun ‘Trump’ is a natural language semiotic artefact digitally encoded, and it ‘stands for’ a ‘digitally encoded structure’ we humans call the ‘image’ or ‘picture’ of President Trump. But it is only a human who will ultimately carry out the ‘semantic’ interpretation of both the proper nouns and the digital images. In computational semiotics, however, it is possible to go much further into the semiotic process. Indeed, in dealing with digitally encoded indexical signals, it is possible (albeit not so easy) to algorithmically associate (by formal declarative or leaning algorithms) some sophisticated descriptors and descriptions developed in the conceptual models of semiotic theories. And as future research seems to develop sophisticated knowledge bases constructed algorithmically, this will produce more and more rich descriptors and statements so as to enhance these types of analysis, which will in turn change the hermeneutics of these types of signs.

194

Computational Semiotics

Symbolic semiotic artefacts: Concepts and narration in texts  Our last example is the analysis of concepts and narration in text. The corpus was the subset of Peirce’s writings published in the Collected Papers.23 The aim of the enquiry was a conceptual analysis of ‘mind’ in this corpus. The conceptual model presented the objectives and method of the philosophical practice which is conceptual analysis. The formal model transformed the text into a vector (words and segments of the concordance with ‘mind’ – 744,450 token words and 717 type words) onto which various inductive type classifiers, clusters, K-means, LDA and latent semantic analyses were applied. The computational model transformed these formal models into algorithms using various programs taken from open-source libraries written in Python or ‘R’ and ad hoc analyses were built. Many curating applications were used. In the text mining literature, these formal models receive various names such as lemmatizer, tokenizer or part-of-speech tagger. And the most widely used analytics models are cluster analysis, latent semantic analysis (LSA), topic modelling (LDA), neural nets, genetic algorithms and word embeddings. The results of these classification algorithms are represented through various visualization algorithms. Figure 11.5 has a sample of the results of a classical clustering K-means analysis, shown in a dendrogram. It contains three classes of similar contextual segments of the word ‘mind’. Each terminal leaf is a number representing a segment of the text.

Figure 11.5  Clusters of the concept of mind in Peirce’s Collected Papers.

 The Workflow of Computational Semiotics 195 Here is a sample of these contextual segments for the predicate ‘mind’ in two specific clusters. The main themes or topics of the various clusters are signs, laws of mind, association of ideas, consciousness, reasoning, logic, knowledge, perception, feeling and categories. First, a sample of the segments of the clusters on the theme of ‘sign’.

Segment No. 646 [My definition of a sign is:] A Sign is a Cognizable that, on the one hand, is so determined (i.e., specialized, bestimmt) by something other than itself, called its Object, while, on the other hand, it so determines some actual or potential Mind, the determination whereof I term the Interpretant created by the Sign, that Interpreting Mind is therein determined mediately by the object.  (Peirce, CP 8: 178) (Figure 11.6)

Figure 11.6  Word cloud for the ‘mind’ and ‘sign’ cluster.

Now, a sampling of segments from the cluster on the theme of ‘law of mind’:

Segment No. 5,121 Finally, laws of mind are divided into laws of the universal action of mind and laws of kinds of psychical manifestation.  (Peirce, CP 1: 154) (Figure 11.7)

Segment No. 507 Laws which connect pheno­ mena by a more or less intell­ Figure 11.7  Word clouds of the ‘mind’ and ‘laws of ectual or inward synthesis are ideas’ cluster. divided somewhat broadly into laws of inward relations or resemblance of bodies and laws of mind. (Peirce, CP 1: 511)

196

Computational Semiotics

Segment No. 436 Now, the generalizing tendency is the great law of mind, the law of association, the law of habit taking. (Peirce, CP l 7: 515)

Afterwards, each cluster can be presented by its list of segments and through some form of visualization of the words it contains (word clouds). When presented all together at a distance (Moretti 2013), this type of analysis offers a general and synthetic view of the interrelations between the themes. And often, new unforeseen sub-themes will appear: for instance, in the case of Peirce, a recurrent methodological theme appears regarding the necessity of studying the mind dynamically. But the reader must be prudent. These spectacular visualizations mainly contain a-semantic recombinations of symbols. Although these visualization techniques are the result of a rigorous classificatory process that takes the context into account (the segments), the actual presentation often delivers only types of words. A weak reader may see nothing but a bag of words! A rich interpretation must go further. It requires introducing much more semantic information than what is given in the visualization itself. For instance, interpreting the word clouds shown in Figures 10.6 and 10.7 requires great expertise in Peircean semiotics. To go deeper into the analysis, close reading became necessary. The project explored various methods of such close readings assisted by computers. This type of reading is directly related to many classical expert readings of semiotic artefacts, more specifically of text. The research here called upon many peripheral hand-based methods that could assist the analysis of the segments of a cluster. Here is a sample of the ones used in this project. Prioritizing segments: Through some statistical strategies, the most important segments of a cluster were given priority. For instance, the segments whose words had a high frequency in the distribution of all the words or had a high informational index were the first ones to be read. This means that the noisiest segments will fall down the list. Commenting segments: Each segment is read normally and a comment is formulated in the reader’s own words and manually placed into a category of comment types (in bold): theoretical, critical, definitional, explanatory and so forth. For example, here is the theoretical annotation of a sample of the segments from the ‘sign’ cluster (Figure 11.8). Automatic summary: It often happens that the number of segments is huge. It can become cumbersome to read and annotate all of them. In such an event, an approximate but helpful automatic synthesis can be produced (Figure 11.9). Research in now exploring automatic argumentative annotation of these segments. Final synthesis: Many other interpretative interventions are possible. But the ultimate aim is to arrive at a final handmade interpretive synthesis based on a close reading of all of the classes. The interpreter can choose the size and the content he or she wishes. And for all the sentences of the synthesis and summary, there are specific citations and references to prove their existence in the corpus. Figure 11.10 is a researcher’s final summary of the content of the class ‘sign’.

 The Workflow of Computational Semiotics 197

Figure 11.8  Annotations of segments of the class ‘sign’. Automac summary of segments of class “ Sign” : A sign or representamen is something which stands to somebody for something in some respect or capacity. It addresses somebody that is creates in the mind of that person an equivalent sign or perhaps a more developed sign. A Dicent Symbol or ordinary Proposion is a sign connected with its object by an associaon of general ideas and acng like a Rhemac Symbol except that its intended interpretant represents the Dicent Symbol as being in respect to what it signifies really affected by its Object so that the existence or law which it calls to mind must be actually connected with the indicated Object. The index is physically connected with its object they make an organic pair but the interpreng mind has nothing to do with this connecon except remarking it a er it is established. In the Sign they are so to say welded. That determinaon of which the immediate cause or determinant is the Sign and of which the mediate cause is the Object may be termed the Interpretant. He allows that things without the mind are similar; but this similarity consists merely in the fact that the mind can abstract one noon from the contemplaon of them… they are the results of its own work. No event that occurs to any mind no acon of any mind can constute the truth of that condional proposion

Figure 11.9  Automatic summary of segments.

These processes are applied to all classes. And a full synthesis interpretation is given to the concept of mind in Peirce’s Collected Papers. One limitation of this approach must be noted, however. The methodology does not allow the identification of textual segments that are about the concept of mind but which

198

Computational Semiotics

Figure 11.10  Researcher’s final synthesis of the cluster ‘sign’ for the conceptual analysis of ‘mind’.

do not contain the word ‘mind’. These textual segments are types of paraphrases, ellipses, reformulations or what we call – for lack of a better name – ‘conceptual perisegments’ (Pulizzotto et al. 2016; Chartrand et al. 2018). From a linguistic point of view, these perisegments are sentences that present a sort of complex synonymous content. An adequate conceptual analysis must identify these textual segments as pertinent. They are essential for conceptual analysis. Identifying them increases the number of textual contexts that enrich the conceptual field of a conceptual predicate. New research24 on the extraction of semantic similarity relations should enhance the identification of text segments expressing a concept but which do not use the canonic term of this concept. Summarizing this text analysis experiment, our three models (conceptual, formal and computational) constrained a computer-assisted conceptual textual analysis in identifying similar textual segments whose symbolic signs (words) participate in the building of a common semiotic field, of a ‘conceptual space’ isotopy, thematic or topic – all of which now have to be interpreted through ‘close reading’ accomplished by an interpreter. The second example of a computer-assisted semiotic analysis of symbolic signs was an exploration of narrative structures in newspaper articles about a social crisis. The corpus was constituted by thousands of newspaper articles on the social crisis called the ‘Maple Spring Revolution’ (‘Printemps érable’25). The conceptual model not only identified the core concepts for computer text analysis but also it had to (a) explicate the concept of narrative structure, (b) apply the Saussurean and Harrisian distinctions of syntagmatic and paradigmatic linguistic relations to a set of texts, (c) adapt the Greimasian actantial model, (d) determine various levels of textual structures, and (e) produce a surface26 and a deep semantic structure description,27 in addition to defining all the concepts pertaining to the various methodologies involved. Two main

 The Workflow of Computational Semiotics 199 hypotheses emerged from this conceptual model: (1) it is possible to build a formal and computational model of the narrative regularities in a text, (2) the Greimasian actantial model (1970)28 and the Fillmore action model (1970)29 could be used for this purpose, one operating at the surface structure level, the other at the deep structure level. The formal model, for its part, had to offer exploration strategies for these hypotheses. The text was annotated using a Fillmorean lexical frame (e.g. action, actor, object). Some morphological, syntactic and semantic categories were tagged to token words in the text. The whole text was represented through a vector model. Classical mathematical classification functions (e.g. K-means) were applied to the annotated text. Like in the Peirce project, this classification function produced classes of similar text segments but with greater refinement because of the linguistically annotated text. A modified Greimasian generative actantial narrative schema was explored for the set of segments of each class. These allowed the analysis of the narrative recurrent macrostructure present in the text. The computational model translated these formal models into algorithms. Most of them were constructed ad hoc or taken from various existing libraries.30 The computer architecture chosen was a personal computer. We present here a sample of the results of this computer-assisted semio-narrative project. The analysis was applied to 9,603 articles extracted from seven important newspapers. In various experiments, and depending on the choice of parameters, the K-means clustering technique (Arthur and Vassilvitskii 2007) produced different numbers of classes for similar annotated article. For each experiment, various Greimasian actantial categories were identified in each class. For example, in a particular experiment, one class presented the student associations as the main actant subjects. The actant object was ‘having a dialogue with the government’ and the anti-subjects of the address was the government. The core action identified was the confrontation between the prime minister and the opposition leader. The student associations and the government were thus the two main actors in a social dialogue. In another cluster, the actantial schema had for acting subjects the government, while the object of the action was the governmental law on the rising of the university fees and the anti-subjects were the student associations. Another cluster had the students as subject actants and the police as the anti-subjects. Other similar narrative structures were found in other clusters. To summarize this experiment, it demonstrated how the addition of informational sources such as tagging and of schemas allowed a computer-assisted semiotic analysis to identify some particular narrative structure. The two preceding projects illustrate how a well-modelled computer-assisted textual analysis, though including quite basic computable functions and oracles, can identify interesting thematic and narrative structures in a textual dataset. But they also illustrate that none of the proposed models can do it alone: they all need the others. And the complexity of the analysis prevents it from being realized automatically.

200

12

Computer models in science and semiotics

Computation, computors and computers The last type of model we present here is the physical computer model. Because we often take, in our ordinary discourse, ‘computation’ and ‘computer’ to be synonyms, we tend to conflate a computational model with a computer model. But there is quite a difference between these two concepts. Such parlance confounds a theoretical concept with concrete technology. It is like saying that the design of a building is identical to an actual physical building built accordingly. Computation is a mathematical notion. It pertains to matters of calculability and decidability of mathematical functions and systems. The notion of computation is logically prior to the notion of computer. The notion of computer, for its part, concerns a technology that can perform this computation through some effective physical procedures. Modern computers are ‘electronic’ mechanical devices in which programs supply instructions for effectively computing functions. Here a few examples that clearly illustrate the difference between these two notions. The history of mechanical calculating devices may be traced far back to antiquity. The abacus seems to be the first one we know of (Figure 12.1).

Figure 12.1 Chinese abacus. Encyclopedia Britannica article for ‘abacus’, 9th edition Encyclopedia Britannica, volume 1 (1875) reproduced in Wikipedia: Abacus 6.png. (2 September 2020). Wikimedia Commons, the free media repository. Retrieved 16:54, 19 December 2020 from https://commons​.wikimedia​.org​/w​/index​.php​?title​=File​:Abacus​_6​ .png​&oldid​=447237259.

202

Computational Semiotics

The Greeks used a complex mechanical physical machine called the Antikythera mechanism1 for calculating solar eclipses. Llull (1232–1315) developed a paper wheel for proving logical arguments. Pascal, in 1642, designed a mechanical calculator capable of basic arithmetic operations (Figure 12.2).

Figure 12.2 Schickard’s calculator and the Pascaline: Arithmetic machines. Pascal, B. (1779). Oeuvres complètes de Blaise Pascal, 5 vols. Edited by C. Bossut. La Haye: Detune, Paris: Nyon. Images courtesy of the Library of Congress from: https://www​.computerhistory​ .org​/revolution​/calculators​/1​/47​/196.

Babbage2 presented the principles of his Analytic Engine, which contained the seeds of the architecture of modern computers (Figure 12.3). In addition to these physical calculators, we should add the traditional analogue slide rule calculator and modern hand-manipulated mechanical calculators (Figure 12.4). These examples show that physical effective procedures for calculating a great number of computable mathematical functions existed way before the Turing machine.3 In fact, Turing himself has often repeated that he wanted to build a machine that simulated a ‘computor’ – that is, a ‘human calculating with a pencil and paper’ (Turing 1948: 9). These computors were humans who calculated computable functions. They were mainly working in scientific fields such as astronomy, physics, engineering and warfare. Slowly, the pen and pencil were assisted by the slide rule and by an increasing number of sophisticated analogue gadgets. The Turing paper machine was just

 Computer Models in Science and Semiotics 203

Figure 12.3  Charles Babbage’s Difference Engine. Woodcut after a drawing by Benjamin Herschel Babbage. Source: Google Books. Taken from: Babbage, C. and Campbell-Kelly, M. (1864). Passages from the life of a philosopher. London: Longman, Green, Longman, Roberts & Green. http://books​.google​.co​.uk​/books​?id​=Fa1JAAAAMAAJ​&pg​=PP8​&im.

another type of mechanical effective procedure to compute calculable functions that computors could calculate by hand. But what made it special was, as pointed out by Gödel, that this physical machine had become the proof that a function is calculable. Except for human calculators, all these calculating technologies were built out of interrelated dynamical physical components. And with regard to the notion of computation, they present some interesting features: (1) all these dynamic procedures had to be activated by some initial energy, either by hand (e.g. the abacus, the rule, the crank machine) or by some mechanism based on stored potential energy (e.g. a spring, raised weights, waterfalls and ultimately electricity); (2) some technologies required that external symbols be added so that users could manipulate them (e.g. the Antikythera mechanism, the Babbage machine, the crank machine); and (3) most of

204

Computational Semiotics

Figure 12.4  Hand crank mechanical calculator, c. 1930. In: Mines Technology Museum, Colorado: https://hdl. handle. net/11124/172469.

these machines often required complex training to be used. What these features4 clearly show is that in each, one must distinguish between what is computed (the calculable function) and the physical procedure that concretely computes it (the mechanism). In other words, even though effective computation coexists in a concrete computing machine, the two notions or concepts are different.5 This means that a same calculable mathematical function can be concretely computed by many different machines. For example, both the abacus and the Turing machine can compute the same adding function: ADD(x, y) or ∀x∀y F(x, y)= (x+y). But on the other hand, all these machines cannot all compute the same set of computable functions. For instance, because of its physical limits, an abacus cannot compute the same functions as a slide rule. This shows the importance of the Turing machine, as it is the only ‘mechanical’ machine that can effectively compute any calculable function. This is what makes the physical Turing machine so important. It proved that there are mechanical means by which all calculable functions could be effectively computed. This is the classical Turing thesis. But this thesis does not say that any sort of components and their structuring can compute all calculable functions with the same physical ease.6 And in the case of the original Turing machine, the components and operations were basic: paper wheel, ink, pen; read, print, move and so on. And its architecture was very limited: the speed and the number of operations could be so great that it becomes practically impossible for it to perform the computation of a complex function in a finite and acceptable amount of time. It took many years to develop more efficient physical components to arrive at the modern computer. In the 1940s, von Neumann proposed a more sophisticated

 Computer Models in Science and Semiotics 205 computing machine architecture. It consisted of memory, arithmetic and logical unit, a control unit and input and output devices (Figure 12.5).

Figure 12.5  Basic von Neumann computer architecture.

This von Neumann architecture could still compute any computable function. In technical terms: it is Turing complete. But new, more efficient physical components such as electric switches, vacuum tubes, diodes, transistors and integrated circuits were developed, enhancing the power of the von Neumann architecture and of the original Turing machine. Still, the components concretely used at the time constrained the type of calculable functions it could compute effectively. For instance, due to the processing time required by the physical components, it would be concretely impossible to compute effectively and in an acceptable amount of time complex chaotic mathematical functions such as the gravitational dynamics of a typhoon. Since von Neumann’s original proposals, the physical components of computers have greatly evolved.7 Today, computers’ architecture and components have become more and more sophisticated and complex: miniaturization of chips distributed computers, grid computing, cloud computing, centralized servers, and even quantum and biological computers8 allow previously impossible speed and power in computational processing. And this advancement in computer technology has put forward the importance of computer models in the sciences.

The interrelation of computational and computer models Because of this high-speed development of computer technology, some sciences that already had discovered highly complex formal models and constructed very complex and powerful computational models could now concretely compute them.

206

Computational Semiotics

It is mainly for this reason that computer technologies have deeply changed the shape and conditions of scientific practices. They allowed the exploration of new frontiers in many research domains. Think of genetics, climatology, astrophysics, semantic social networks, distributed management companies, medicine, the environment, cognition, robotics, computer science, biology and so on. An increasing number of epistemologists (Magnani, Nersessian and Thagard 1999; Giere 2010; Latour 1986) see that computer technology is playing a more prominent role in scientific practice and theories. For example, it renews experimental forms, consolidates demonstrations, evidences, validations, simulations, falsifications, assessments, dissemination, publication and so on. Ultimately, it modifies scientific reasoning itself. Because computer technology as such offers more and more effective processing power, it invites the creation of new complex algorithms that embed even more complex formal models. For instance, complex sets of simple algorithms can be chained together and can build some hard to follow and even untraceable mega-algorithms. These new algorithms can map onto complex physical and biological phenomena. So much that we now speak of physical quantum computing, but also of natural,9 biological,10 chemical,11 and even ‘brain’ computing.12 Although there is some metonymic entanglement in the use of the word computation in these denominations, what is computed is always a calculable function. What is important is that some of these means of physical computing can allow the exploration and application of unforeseen computable functions. Even more important is that they are now only accessible through their algorithmic expressions which are known to be physically implementable or concretely computable. Computer models are hence very important in science – they play an important and essential role in contemporary scientific practices. However, one must remember that regardless of how powerful computer technology may be, it cannot compute everything: a computer, whatever its physical form and power, can only ‘compute’ functions which are calculable, with these representing only an infinitely small subset of all possible functions. Some researchers believe, not without some theoretical opposition, that because the traditional computer model based on the physical Turing machine is limited to calculable functions, other types of machines or hypercomputers13 should be explored and built mainly for computing non-calculable functions. In many sciences, physical models (maquettes, simulations, replicas, etc.) are constantly used. Computer models can be seen in the same way. They determine the type of implementation computational models require. They parameterize the physical artefact in which different types of effective computational models used in research can be computed. In fact, though one may have discovered the most complex formal models for one’s research object and even constructed the most efficient algorithm for it, it does not follow that there yet exists a concrete computer that may compute it concretely and in some acceptable time limit. But on the other hand, it may also happen that because of the existence of a powerful computer, new and unforeseen algorithms and formal models may be discovered and experimented with. It is in this respect that computer models are important in science.

 Computer Models in Science and Semiotics 207

Computer models in semiotics Semiotics cannot escape the impact of computer models in its research. Actual computer technologies process electric signals that are ‘encoded’ by symbols which are in turn the basis of computational models. For Newell, these signals are physical symbols: Scientists in AI saw computers as machines that manipulated symbols. The great thing was, they said, that everything could be encoded into symbols, even numbers. (Newell 1983: 196) A physical symbol system consists of a set of entities, called symbols, which are physical patterns that can occur as components of another type of entity called an expression (or symbol structure). (Newell and Simon 1976: 116)

It is this thesis that underlies the semiotic theories of computing (Gudwin 1999; Gudwin and Queiroz 2005, 2007; Nadin 2007, 2011a; Liu 2005; Tanaka Ishii 2008). In this sense, computers are semiotic artefacts of the indexical type. It is not surprising then to say that the signals are carriers of information and that computers are therefore information technologies. Many human and social sciences have already felt this impact on the objects they study, but also upon their methodologies. More specifically and because of the great processing power of computer technology, new domains and methodologies have emerged. They have approached many types of objects, many of which carry meaning. And for analysing them, they call upon concepts and methodologies which, despite a difference in vocabulary, are very close to what semioticians are interested in but which they cannot handle using pen and paper technologies. For example, consider computational linguistics. Here, symbolic linguistic artefacts are explored at various linguistic levels: words, sentences, texts, speech, narration, arguments, whole corpora and so on. They are studied using various specific computational methods: corpus linguistics, lexicometries, textometrics, automatic translation and so on. Another important field is social and cultural computing.14 This field studies sociosemantic representations,15 collective intelligence,16 gaming, social media, etc. Digital history, as it was called, has also been an early user of computers: it found database technologies to be most useful for historical data archiving facilities. We must not forget the use of computation in political science, law, journalism, communication, art and so on. Practically no field of social and human science has escaped the influence of computer technologies. And as we have said earlier, it is fascinating to see the renewed interest in studying the many semiotic artefacts pertaining to these domains, precisely because computer technologies allow the exploration of their complexity. But there is still a particular domain where the impact of computers has been spectacular: the digital humanities. And it is very much alive. According to the Oxford Research Centre for the Humanities, this domain has indeed defined its research field by its relation to computing technology:

208

Computational Semiotics

By digital humanities, we mean research that uses information technology as a central part of its methodology, for creating and/or processing data. . . . The use of the term [Digital Humanities] reflects a growing sense of the importance that digital tools and resources now have for humanities subjects.17

Under this umbrella term, many very ‘soft’ disciplinary fields have found new means for exploring both their traditional research object, but also new ones, including multimodal production (text, images, music, dance, films, e-games, e-media, etc.).

Conclusion We believe that semiotics, both in its object and methodology, will not resist the introduction of the computer into its practices. It will allow the exploration of unforeseen semiotic conceptual, formal and computational models. It will enrich the object and fields it studies, the methodology it uses and the scientific and social impact it may have. The components, structures, patterns and contexts of semiotic artefacts and processes will be seen in a new light. And computational tools such as computer simulations, visualizations, web dissemination, data annotations and data analytics will suddenly appear very heuristic. Many criticisms have been formulated concerning the digital and computational field in the humanities and the human sciences. Some18 underlined that they are too ‘distant’ from their true object. Others have stressed the superficiality19 of the analyses, their lack of critical thinking20 and so on. In one sense, we agree with these criticisms. They underlined real problems and limits in the computational approach to complex semiotic artefacts. And for many other reasons, semiotics will not submit entirely to computational and computer models. But in another sense, these criticisms do not attack computation and computers as such, but rather discourses that oversell and misunderstand their true power and role. McCarthy offers a nice metaphoric criticism of these discourses: ‘computers are essentially modeling machines, not knowledge jukeboxes’ (McCarty 2004: 257, emphasis removed). If computers are indeed modelling machines, then they are just purveyors of a particular point of view upon a phenomenon. As models, they are by definition partial and reductive. They can focus on the phenomenon only if one can map onto them some computable functional dependency relations. Everything else is out of their range. Therefore, they cannot deal with non-computable relations and functions. The danger is to believe that they can. But this does not mean that there is no place for them in semiotics, for there are some interesting computational functions in semiotics. There are in fact countless semiotic structures that can be modelled formally and computationally and hence their algorithms can be implemented in a computer. But most probably, the structures and operations emphasized by critics, such as close reading, criticisms, dialogues and, surely, a host of others, are non-computable ones. It is surely because many semiotic artefacts deal with the complexity of meaning structures emerging, for example, in imagination, emotion, dialogues and culture that the most sophisticated quantitative computation and best deep learning algorithms

 Computer Models in Science and Semiotics 209 cannot go so deeply into them; they rapidly fall into the groundwater of hermeneutics and risk drowning in it! We may recall here the research on Magritte’s paintings, the musical analysis or the conceptual analysis of ‘mind’ in Peirce. These took for objects highly complex iconic, indexical and symbolic semiotic artefacts. A computational model and computer model venturing to delve into their meaning will rapidly hit the wall of non-computationality, which is inherent to the analysis of these semiotics artefacts. But many specific computational models and even simple computer models may help in exploring their complexity heuristically. Throughout such exploration, human interventions are required. Semiotic analyses may hence be assisted by ready-made oracles that are ad hoc algorithms specialized for specific computational tasks. And they can be rapidly integrated into semiotic analysis workflows. Here are but a few examples: new types of repositories (e.g. the cloud, the internet) offer a variety of devices for storing, navigating and recalling semiotic artefacts. Digital annotation tools (e.g. syntactic, semantic, semiotic dictionaries and thesauruses, knowledge bases) can assist pertinent categorizations. New and original types of statistical, lexical, stylistic, thematic, topical or conceptual classifiers, as well as complete narrative and discursive pattern and schema identifiers, allow rich and creative explorations of small or huge sets of semiotic artefacts. New forms of interfaces such as 2D and 3D visualizations open up new avenues for dynamic semiotic investigation. Complex interactive digital communication tools create new ways of collaborating and sharing during semiotic enquiries. New artificial intelligence applications are also becoming important for enquiries into language and iconic artefacts. Also, some tools are interactive either with their environment or with the users. They allow adaptation, adjustment or what some metaphorically call ‘learning’. We may think of the use of all such computer tools in the training of students in the field of semiotics.21 And recent research is also changing the dynamics of knowledge storage and acquisition. There is indeed an important renaissance of knowledge bases and a refinement of deep learning machines but most, of all, an enhancement of their interaction. Knowledge bases that were originally hand-made are now being updated by learning machines that extract knowledge inductively from various sources (scientific publications, literature, the web, history, films, social media, etc.) that dynamically and systematically feed knowledge bases and from which new knowledge can be deductively inferred, which in turn enriches the sources themselves and has us see the world with a transformed point of view. In short, new computer technologies are changing semiotic enquiry in one way or another. They open up a special type of scientific enquiry – not because they are technologies, but because they rest upon computational and formal models which themselves relate in some way or another to conceptual models that are most of the time the mainstay of semiotics. In no way then should computational semiotics be understood as automatic semiotics, but rather as computer-assisted semiotics. And classical interpretation will always remain the core of semiotic analysis. Meanings may be processed by all living beings, but semiotics is a specific type of semiosis: it is essentially a human affair. And for discovering, explaining and understanding this meaning, Hermes may now have efficient oracles as assistants.

210

Notes Chapter 1 1 ‘Semiotics . . . can be informally defined as the science that studies all possible varieties of signs, and the rules governing their generation and production, transmission and exchange, reception and interpretation. Concisely put, semiotics has two complementary and interdependent aspects: communication and signification’ (Sebeok 1976: 272). 2 Dadesio 1995 stresses the cognitive roots of semiotics and the importance of cognitive science for semiotics. 3 Meunier (2018). 4 See Shannon and Weaver (1964); Bertalanffy (1969, 1951a); Agazzi and Montecucco (2002). 5 Still, there is actually one person (Alexis Lemaire) in the world who can mentally calculate the thirteenth root of any number x, even 100-digit-long ones, in 13.55 seconds (Unofficially, Gert Mittring is supposed to have done it later in 11.80). See: https://en​.wikipedia​.org​/wiki​/Alexis​_Lemaire

Chapter 2 1 Formal manipulation of symbols goes farther back in history. Indian grammarian Panini (c. 500 BCE) seems to be one of the first to have developed a formal set of written and recursive rules for explaining Sanskrit sentences (see Kak 1987; Saal 1988; Bod 2015). 2 To be more rigorous, such an abstract definition of computation lacks a relation to an effective material procedure that implements it. We maintain this definition here because we specifically focus on the symbols that lack semantic content. 3 See van de Walle (2008); Eco (1979). 4 See Nielson and Nielson (1992); Scott and Strachey (1971). 5 See Dreyfus (1972); Searle (1980); Winograd and Flores (1986); Hofstadter (1983); Harnad (1990); Fetzer (1990, 2001). 6 See Zemanek (1966). 7 These characteristics required modification of the mathematical models so that continuity, fuzziness, optimization, reconfiguration, default planning and so on could be modeled adequately. Probabilities and topology were hence invoked. 8 See also Gudwin and Queiroz (2007). 9 Döben-Henisch, Erasmus and Hasebrook (2002). 10 De Sousa (2005). 11 Liu (2005); Nadin (2011b). 12 Manovitch L. (2002, 2018); Grigar and O’Sullivan (2021).

212

Notes

13 The Cosign and the Electronic Literature Organization (OLO) conferences have been academic platforms for this domain since 2000. 14 Lovejoy (2009); Miller (2012). 15 Kress (2010); Danesi (2018). 16 Gere (2012). 17 Bastian O. (2015); Leone (2018); O’Halloran et al. (2009). 18 Nowakowska (1986). 19 Sowa (1976, 2000). 20 Dondero and Fontanille (2012, 2014); Chen (2013). 21 Pietarinen (2003); Damjanovic et al. (2004). 22 Dodig-Crnkovic (2006).

Chapter 3 1 Mitchell (2018). 2 Some contemporary artificial intelligence projects aim at discovering these features by ‘extracting’ them automatically from the digitized texts that have been published on the same topic. 3 See the Humanities Commons: https://hcommons​.org/ 4 As Manovich says : ‘Thousands of researchers have already published tens of thousands of papers analyzing patterns in massive cultural datasets’ (Manovich 2016:2). 5 Google Books, Wikipedia (2018): https://en​.wikipedia​.org​/wiki​/Google​_Books (accessed September 2018). 6 Despite the technological qualities and the creative intentions of the Google Book project, it has been met with a host of criticisms. But most of them (even those concerning copyright) closely or remotely pertained to some semiotic nature inherent to the books. See, among many others: Darton, R. (2011).

Chapter 4 1 Duhem, Bachelard, Weber and Manheim. 2 Hanson, Kuhn, Toulmin, Laudan, Hacking and in some sense radicalized by Feyerabend, Lakatos and Thagard. 3 As Stachowiaks (1973: 132–3) formulates it: ‘Modelle sind ihren Originalen nicht per se eindeutig zugeordnet. Sie erfüllen ihre Ersetzungsfunktion für bestimmte – erkennende und/oder handelnde, modellbenutzende – Subjekte, b) innerhalb bestimmter Zeitintervalle und c) unter Einschränkung auf bestimmte gedankliche oder tatsächliche Operationen’. And for Rotenburg (1989: 75) models are also epistemic representational ‘Replacements’ – that is, something that is put in ‘place of ’ as ‘Modeling in its broadest sense is the cost-effective use of something in place of something else for some cognitive purpose’. 4 This perspective could also be applied to many human and social sciences attentive to semiotic artefacts and processes.

 Notes 213 5 Dondero and Fontanille (2012); Bod (2015); Kralemann and Lattmann (2013). 6 Green (2013); Rheinberger (2007, 2009); Morgan and Morrison (1999); Leonelli (2007); Knuuttila (2011); etc. 7 Minsky (1986).  8 Sterrett (2003).

Chapter 5 1 One important process by which to achieve such conceptualization rests upon complex cognitive operations such as perception, observation through instruments, comparisons and so on. The nature of these cognitive operations has been at the core of many epistemological debates on the complex role of observation in science. Ultimately, these observations are expressed linguistically in observation reports. One important role of conceptual models is to conceptualize and express these observations and integrate them into the overall conceptual framework used. Therefore, a conceptual model includes many more concepts than do observation reports. 2 Seeing non-formal model as an important component of scientific reasoning is due to Craik’s intuition (1943). 3 Lewis [1973] 1986. 4 Bunge (1977). 5 Polanyi (1966); Nonaka and Takeuchi (1995). 6 Feyerabend (1975). 7 Brown (2007). 8 Godfrey-Smith (2005); Sellars (1956). 9 Hayes et al. (1979). 10 See Wyssusek (2006). 11 See Mylopoulos, Rose and Woo (1993). 12 Newell (1970). 13 Minsky (1975). 14 Sowa (1976). 15 Gardenfors (2000). 16 Gardenfors (2000). 17 Michalski et al. (1981); Mitchell (1997); Goodfellow, Bengio and Courville (2016). 18 See the Bert project: Mitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., et al. ‘Never-Ending Learning’, Communications of the ACM 61, no. 5 (24 April 2018): 103–115. https://doi​.org​/10​.1145​/3191513. 19 In the more technical mathematics of category theory, a class or a category is understood as a morphism, that is, as a type (e.g. hierarchy) of structure that classes may entertain among themselves. It is the interest of structural (geometrical) and mathematical modelling of classes. 20 Genesereth and Nilsson (1987); Gentner and Smith (2012). 21 Johnson-Laird (2006).Evans 2003, 2008. 22 Gentner and Smith (2012); Niiniluoto (1988); Dretske (1982). 23 Kuhn (1993); Ortony (1993); Lakoff and Johnson (2003). 24 Suárez (2009). 25 Padian (2018); Muindi et al. (2020); Singer (2019).

214

Notes

26 Chen (2013); Dondero and Fontanille (2012, 2014). 27 Granger (1967: 44; our translation).

Chapter 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14

15

16 17 18

Sherwood (2019); Hunt and Lipo (2011). Smith (2000: 28). https://semioticon​.com​/semiotic​/2013​/05​/consider​-vexillology/ Hare et al. (2006). Chartrand et al. (2013); Chartier et al. (2019); Hébert, Michelucci and Trudel (2018). Hébert and Trudel (2003-2013). Hébert and Dumont-Morin (2012). Reproduction of the cited Magritte’s painting can be found on the Wiki art site https:// www​.wikiart​.org​/en​/rene​-magritte​/all​-works​/text​-list. Nöth (2012). Biosemiotics providing a possible bridge between embodiment in cognitive semantics and the motivation concept of animal cognition in ethology (Hoffmeyer 2008). Sebeok (1968: 142–4). Music Encoding Initiative (MEI) Format Family. Web page, 17 May 2019. https://www​ .loc​.gov​/preservation​/digital​/formats​/fdd​/fdd000502​.shtml. Meunier and Forest (2009); Pulizzotto (2019); Chartrand et al. (2018). Conceptual analysis is an expert interpretation methodology for the systematic exploration of semantic and inferential properties of a natural language predicate expressing a concept in a text or in a discourse (Desclés 1997; Fodor 1998; Brandom 1994; Gardenfors 2000). In Quebec, during the spring, the production of maple syrup is an important economic activity. Culturally speaking, it is the occasion for many happy reunions and parties. Because the demonstrations took place during the springtime, newspapers coined the name ‘Maple Spring’ revolution. Kintsch and van Dijk (1975) and Rastier (2009) propose the micro, meso and macro levels of text analysis. These levels are different from Moretti’s distinction between close and distant reading (Moretti 2013). Greimas (1970). Fillmore (1968, 1976).

Chapter 7 1 Bod (2015) has shown that many humanities disciplines had developed formal rule systems for description far before contemporary sciences had. 2 See Kleene (1952); Harrison (1978). 3 One must keep in mind that translating the notions of formal system and of formalism as language is a metaphor. See Nofre, Priestley and Alberts (2014); Bod (2015). In English, this metaphor comes naturally. But in French, its limitations quickly reveal themselves. In a Saussurean paradigm, a formal system cannot be translated as a ‘langue’ nor as a ‘langage’. Still, in French the word ‘langage’ is used for the concept of ‘langage de programmation’, but in this case, the word langage has

 Notes 215

4

5 6 7 8

9 10

11 12 13 14

15 16

17 18

the English metaphoric meaning. In French, linguists would accept the sentence: Le langage de programmation n'est pas une langue. But translating it in English would sound bizarre: A programming language is not a language. It seems also to be the same in German: Programmiersprachen sind keine sprachen. Typical categorial type languages are the Church lambda calculus (1941), the combinatorial calculus of Schönfinkel ([1924] 1967), Curry and Feys (1958), the categorial language of Lambek and Scott (1988), the Montague grammar (1974), the applicative grammar of Shaumyan (1987, 2003), Steedman (1988) and Desclés (2003). See ‘Axiomatic method’, in the Encyclopedia of Mathematics, EMS Press, 2001. Sometimes called ‘normal Polish notation’, ‘Lukasiewicz notation’ or more often ‘prefix notation’. The set-theoretic approach is rooted in Whitehead and Russell (1910), Tarski (1944), Carnap (1947), Montague (1974) and Hintikka (1997). The dynamical state-space approach originates with Leibniz but was really mastered by Poincaré (1892). Afterwards, it was mainly developed in the work of John von Neumann (1966), Hermann Weyl (1949) and René Thom (1974) who applied it to biology through his catastrophe theory. In philosophy of science, it influenced, among many others, Suppe (1967), van Fraassen (1980) and Giere (1999, 2010). See Rorty’s critiques (1979) on the concept of representation in science. This Bourbakian view seems to be the core epistemic role of formal mathematical models. When the objects of study are complex and dynamic, it allows to unify in a common formal language a multitude of structural properties of the object, but also to compare the models themselves. This type of formalization opens up ‘differential’ and ‘distributional’ analyses. It may reveal new ‘observable’ properties and new properties of the object by manipulating the symbols that ‘represent’ these properties in the formal model itself. A categorization is not a truth assertion. It is a sort of cognitive classification act or operation. And the sentence expressing it could be false. One must note here that in this equation, the number ‘2’ is in fact a dual symbol: it is a combination of an elevation symbol (hidden) and a numerical symbol ‘2’ which, by convention, is a complex symbol for a self-multiplication operation applied once. There are other lexical definitions of the word father which, in certain contexts, do not define a functional relation. Theories of formal reasoning are not psychological theories of cognitive reasoning. There has been a lot of research showing the difference between human and logical reasoning. Jonathan Evans (2008), for instance, had defended a dual view process of reasoning: conscious and unconscious automatic reasoning. R1) S ⇒ NP + VP; R2) NP ⇒ ART + N; R3) VP ⇒ V + ADV Classical logical inference rules in logic are non-contradiction, modus ponens, modus tollens and simplification. But one must not confuse a logical inference rule with a material implication. An inference rule is independent of semantics. A material implication has a semantic counterpart. For instance, if it rains, then I take my umbrella is not a logical rule. In natural deduction inspired models, there are many inferential rules: introduction, elimination, substitution, etc. (Gentzen 1955; Kleene 1952). For instance, in English, it is observed by different means that the lexeme ‘There is an ARTicle and that the words ‘dog’, ‘cat’, ‘elephant’ and ‘table’ are NAMES. The formalization of all these observations is expressed in the lexical rules. The substitution (generative rule: NP ⇒ ART+ N) allows the instantiation of each symbol of the first formula in the formula NP ⇒ The dog, etc.

216

Notes

19 The classical logical formalization of abductive inferential rules is the following: If ├∀x(P(x) ⇒ Q(x)), Then if ├ P(xi) is given, then it should the case that Q(xi). But if, in fact, it is Q(xi) that is given, then it cannot be presumed that P(xi). This is because there is no known valid inference rule that validates it. And if the data for Q(xi) & P(xi) are sparse, there is no possible induction to generate ├∀x(P(x) ⇒ Q(x)), so Q(xi) ⇒ P(xi) is invalid.

Chapter 8 1 2 3 4 5 6 7 8

9

10 11 12 13 14 15 16 17

Edling (2002). Chomsky (1957). Montague (1974); Shaumyan ([1977] 2003, 1987). Rumelhart and McClelland (1986). In 1972, Lotman wrote, ‘The ability of various mathematical disciplines to serve as a metalanguage also in the description of the phenomena of art is evident.’ Cited in Kull and Velmezova (2014: 531). P and NP complexity are such cases of computing complexity. They can be solved by polynomial or exponential time. ‘A formalism presents itself as a system of symbols submitted to manipulation rules’ (Ladrière 1957, our translation). And this task presents very complex epistemological and methodological problems. They span from the question of observations, of perception, to theoretical projections. Categorization is not an effortless and innocent cognitive operation. And its results will have a determinant role on the rest of the enquiry. The categorization of Christ the Redeemer in the set of humans shows the importance of the conceptual models. At first site, one would find it adequate to place Christ the Redeemer in the set of humans. But huge theological debates, if not wars, underly the conceptual models that defend or refuse this decision. The rule of variable substitutions is part of metalanguage. And it is not without great theoretical problems regarding the real nature of a variable (Desclés and Cheong 2006). Chartier et al. (2019); Chartrand, Chartier and Meunier (2013). Hartigan (1975). Rieger (1999a); Lenci (2008); Baroni and Lenci (2010); etc. Blei et al. (2003). Agrawal et al. (1993). Meunier and Forest (2009); Pulizzotto et al. (2016) (on Peirce). Pulizzotto (2019) (doctoral thesis).

Chapter 9 1 Some computational models may be so complex that extracting the formal model underlying them may not be an easy epistemic task. In a formal model, some functional dependency relations may be only extensionally defined. Still, if the extensional definition is finite, the function is computable, but this does not mean that

 Notes 217

2 3 4 5 6 7

8 9 10 11 12 13 14 15 16 17 18 19

20 21 22 23 24 25 26 27 28 29

in the formal model, one can formulate this extensional definition of a function using an intensional formula. And this does not mean that there is no algorithm that can be the translation of this computable function. More precisely, Church identified effective calculability with his lambda calculus and afterwards with general recursiveness. The presentation given here of the Church–Turing thesis is just a summary. There would be many more important subtleties to explore in the understanding of this thesis. It has opened up great debates. See Copeland (2002a). This Church–Turing thesis is highly debated in computation theories. It has opened up the Pandora’s Box of hypercomputation (see Copeland 1997, 2002a). Markov’s thesis: any effective computational procedure is equivalent to a suitable normal algorithm. In fact, this transition encounters Quine’s (1960) incommensurability problem. It probably also turns it into some sort of Chaitin’s complexity problem. ‘Programming is essentially about certain “data structures” and functions between them. Algebra is essentially about certain “algebraic structures” and functions between them. Starting with such familiar algebraic structures as groups and rings, algebraists have developed a wider notion of algebraic structure (or “algebra”) which includes these as examples and also includes many of the entities which in the computer context are thought of as data structures’ (Burstall and Landin 1969: 170). See Shagrir (2002) for a presentation of a Gandy machine. See http://w3​.salemstate​.edu/​~tevans​/VonNeuma​.htm This condition pertains to the denumerability and countability of the data. Infinite sets pose certain complex computability problems. Codd (1983: 64–9). Jackson (1998). Green and Abraham (2011). Tiddi et al. (2020); Liu et al. 2020; Chen et al. (2020). Mitchell (1997). LeCun, Bengio and Hinton (2015). Bishop (2006). Muggleton (1999).  ‘There is now a better way. Petabytes allow us to say: “Correlation is enough.” We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters’ (Anderson 2008). ‘Scientists no longer have to make educated guesses, construct hypotheses and models, and test them with data-based experiments and examples. Instead, they can mine the complete set of data for patterns that reveal effects, producing scientific conclusions without further experimentation’ (Prensky 2009). Ashby ([1947] 1962); Wiener ([1948] 1961). Marr (1982); MacKay and Miller (1994). Maturana and Varela (1992). Dorigo and Stützle (2004).  Holland (1975). Farmer, Packard and Perelson (1986); Bersini and Varela (1991). Rumelhart and McClelland (1986). Varela et al. (1999). Clark (1998). Thom (1988); Wildgen (1982); Petitot (1985b); Wildgen and Brandt (2010).

218 30 31 32 33 34 35 36

Notes

Churchland (1989). Li, Fergus and Peronal (2000). Chen et al. (2019). Snell et al. (2017), Prototypical Networks for Few-shot Learning. Garcez et al. (2002). This is usually done through some transducers. Chen (2013).

Chapter 10 1 Despite the fact that most research objects of digital humanities are semiotic artefacts, their conceptual framework is usually not semiotics but linguistics, literature, cultural studies and so on. 2 Out of these main trends, very few ‘semioticians’ began exploring computational modelling for analysing semiotic artefacts, among whom Albus and Meystel (1996), Rieger (1998, 1999b), Bolasco (1998, 2002), Balbi (1998), Meunier (2009, 2014, 2019), Desclés et al. (2009), Rastier (2011, 2018), Rastier et al. (1994), Pincemin (2011), Liu (2005), Nadin (2007), Stockinger (2015), Longhi (2015), Lieto (2016), Lippi and Torroni (2016), Piotrowski and Visetti (2014), Ciula and Eide (2017), Ciula and Marras (2019), Compagno (2018), Volli (2014), Valette (2018), Petitot (2018), Reyes and Sonesson (2019), Pulizzotto (2019), Zarri (1997, 2009) and Caccamo (2019). 3 See Cioffi-Revilla (2010) and the Computational Social Science Society of the Americas. 4 Thomas (2004). See also ‘Digital history’: https://chnm​.gmu​.edu​/digitalhistory/ 5 See the Society for Computational Economics: http://comp​-econ​.org​/index​.htm as well as the Computational Economics journal: https://www​.springer​.com​/journal​ /10614 6 Weber and Bookstein (2011). 7 Association for Computational Linguistics Member Portal (2020). https://www​.aclweb​ .org​/portal/ 8 Lincoln Logarithms: Finding Meaning in Sermons: http://lincolnlogs​ .digitalscholarship​.emory​.edu/ 9 See the project ‘What’s on the menu?’: http://menus​.nypl​.org/ 10 See the Xiangtangshan Caves project: https://digitalhumanities​.uchicago​.edu​/projects​/ xtscaves, accessed May 2020. 11 See the Situated Instrument Design for Musical Expression project: https:// digitalhumanities​.berkeley​.edu​/courses​/situated​-instrument​-design​-musical​ -expression, accessed 1 June 2020. 12 Perseus Project Digital Library project: http://www​.perseus​.tufts​.edu​/hopper/ 13 Gallica project: https://gallica​.bnf​.fr 14 UNESCO Collection of Traditional Music of the World site: https://ich​.unesco​.org​/en​/ collection​-of​-traditional​-music​-00123 15 See the Complete Works of William Shakespeare project: http://shakespeare​.mit​.edu/ accessed June 2020. 16 The Google Books Project has provoked worldwide criticism in regard to intellectual property. See Band (2006). 17 See the GLAM project. Wikipedia, being a partner, offers a fine summary of this project. Many detailed projects are presented in Proffitt (2018).

 Notes 219 18 19 20 21 22 23 24 25 26 27 28 29 30 31

32 33 34 35 36 37 38 39 40 41 42 43 44

45

International Council of Museums (ICOM): http://icom​.museum​/en The International Federation of Library Associations (IFLA). International Council of Archives (ICA). Brickley and Guha (1999). See the following software and frameworks for building ontologies: Protégé, IsaViz, SWOOP and NeOn. See Kapoor and Savita (2010). Ontologies are highly developed in the projects of the International Council of Museums (ICOM), the International Council of Archives (ICA), the International Federation of Library Associations (IFLA), English Heritage (EH). Pithchler and Zöllner-Weber (2012). Chiba (2017). Jugand (2016). Bibliographie sur les classifications, vocabulaires et Billet. Consortium Musica (blog). Accessed 9 November 2019; Rashid, De Roure and McGuinness (2018). Sandaker (2010). Ciula and Eide (2017). Langmead et al. (2016). Jaimes and Smith (2003). Lieto (2016). Lieto and Damiano (2014). Niehaus et al. (2014). Ziku (2020). Some companies are engaged in creating ontologies for the Glam consortium, see: Ontotext. ‘Linked Data Integration for Libraries, Archives and Museums’. Accessed 8 November 2020. https://www​.ontotext​.com​/knowledgehub​/case​-studies​/linked​-data​ -integration​-galleries​-libraries​-archives​-museums/ Tomasi et al. (2015). Rastier (2004). Gerber et al. (2013); Dong and Rekatsinas (2018). SNLP: ‘The Stanford Natural Language Processing Group’. Accessed 30 November 2020. https://nlp​.stanford​.edu​/software​/openie​.html. BERT: Simon (2019); Devlin (2019); Horev (2018). Martinez-Rodriguez et al. (2019). This is a regular type of descriptive analysis found in the medical, biological, geographical and astronomical domains. Easley and Kleinberg (2010). Mikolov et al. (2013a, 2013b). Biber (1998); Stefanowitsch (2020); Fuß et al. (2018). Byrd et al. (1987); Fuertes-Oliviera (2017). Heiden (2010); Lebart and Salem (1994); Lebart et al. (2019); Camargo et al. (2013). Pouyanfar et al. (2018); Guan (2017). Aggarwal (2020). Non-linear algebra (differential, calculus topology, algebraic geometry) is omnipresent in many computational fields, but in machine learning it is more discreetly used for complex optimization problems. Petitot (1985b, 2009, 2018) is probably one of the semioticians who has explored its rich possibilities in cognitive semiotics problems but, to our knowledge, these are not machine learning problems. Mello and Ponti (2018).

220

Notes

46 This will generate n points in a Cartesian space where each coordinate represents a vector whose values are its coordinates. Graphically and by convention, this n dimensionality of the vector will in many cases be represented in a two-dimensional drawing. 47 Chapelle, Haffner and Vapnik (1999). 48 Hartigan (1975). 49 Hartigan (1975). 50 Dumais (2005). 51 Blei (2012). 52 Thabtah (2007); Abdelhamid and Thabtah (2014). 53 Dpedia, in Wikipedia, accessed 24 January 2020 https://en​.wikipedia​.org​/wiki​/ DBpedia 54 Freebase, in Wikipedia accessed 20 December 2020 https://en​.wikipedia​.org​/wiki​/ Freebase_(database) 55 NELL Never-Ending Language Learning. (2020). In Wikipedia accessed: 20 Dec. 202 https://en​.wikipedia​.org​/w​/index​.php​?title​=Never​-Ending​_Language​_Learning​ &oldid​=996010659 56 Yago Project​.​in Wikipedia, accessed 24 January 2020. https://yago​-knowledge​.org/ 57 ‘Google “Knowledge Vault” 20 Dec. 2020 To Power Future Of Search’, 25 August 2014. https://searchengineland​.com​/google​-builds​-next​-gen​-knowledge​-graph​-future​ -201640. 58 Lobo (2020). 59 Aldana et al. (2012). 60 Abudction: ‘It is an act of insight, although of extremely fallible insight’ (5.181). Peirce, Collected Papers, (1931–8, 5, 181).

Chapter 11 1 Wachsmuth (2015); Comer (2017). 2 Crawlers are algorithms that systematically browse the web on the basis of specific parameters. They are brute-force search engines. 3 Scrapers are algorithms that systematically collate web pages. There are a host of available scraper applications, such as Octoparse, Common Crawler, Parsehub and so on. 4 See the British Museum collection: https://www​.britishmuseum​.org​/collection. 5 See the Louvre’s digital painting collection: https://www​.louvre​.fr​/en​/moteur​-de​ -recherche​-oeuvres. 6 See Newton’s chemistry project: https://webapp1​.dlib​.indiana​.edu​/newton/. 7 See the Edition numérique collaborative de l’Encyclopédie de Diderot, de D’Alembert et Jaucourt http://enccre​.academie​-sciences​.fr​/encyclopedie/. 8 See: http://www​.trumptwitterarchive​.com/. 9 https://www​.nfb​.ca/ 10 https://library​.duke​.edu​/music​/resources​/databases 11 https://culture360​.asef​.org​/news​-events​/asean​-digital​-cultural​-heritage​-archive​ -launched/ 12 See the Surrey dance archives: https://www​.surrey​.ac​.uk​/national​-resource​-centre​ -dance​/projects​/digital​-dance​-archives.

 Notes 221 13 See the William A. Wilson Folklore Archive Mormon Collection: https://lib​.byu​ .edu​/collections​/wilson​-folklore​-archive​/forms/; see also the New York University’s Myths, Fairy Tales and Folklore libraries: https://guides​.nyu​.edu​/c​.php​?g​=276897​&p​ =1846442. 14 For example, centerNet, an international network of digital humanities centers: https://dhcenternet​.org/. 15 See the Text Encoding Initiative: https://tei​-c​.org/. 16 See the World Wide Web Consortium: https://www​.w3​.org/. 17 https://www​.newyorker​.com​/business​/currency​/what​-ever​-happened​-to​-google​ -books. 18 See the UNESCO’s Charter on the Preservation of Digital Heritage: https://en​.unesco​ .org​/themes​/information​-preservation​/digital​-heritage. 19 The algorithm used here is based on the orthogonal complement subtraction operator. 20 Le sens de la nuit (https://fr​.most​-famous​-paintings​.com​/MostFamousPaintings​.nsf​/A​ ?Open​&A​=8XYU6E). Le maître d’école: https://www​.magrittegallery​.com​/product​-page​/le​-maitre​-d​-ecole​ -the​-school​-master accessed 20 January 2021. 21 L’idée http://www​.artnet​.com​/artists​/ren​%C3​%A9​-magritte​/lid​%C3​%A9e​-9DES​ -ZIKJnx2hp9TC​-KIgA2: accessed 20 January 2021. 22 Le Pan de nuit: http://www​.artnet​.com​/artists​/ren​%C3​%A9​-magritte​/pan​-de​-la​-nuit​ -7blP1ueA4ig8iBZt​-i60bw2 accessed 20 January 2021. 23 Peirce, Collected Papers (1958). 24 Li, Liu and Liu (2017); Majumder et al. (2016); Nagoudi, Ferrero and Schwab (2017); Panchenko and Morozova (2012); Augenstein et al. (2017). 25 In Quebec, in the springtime, the production of maple syrup is an important economic activity and, culturally, it is the occasion for many happy reunions and parties. Because the manifestations were in spring, the newspapers coined the name ‘Maple Spring Revolution’. 26 Kintsch and van Dijk (1975), Rastier (2009) propose the micro, meso and macro levels of text analysis. 27 These levels are different from Moretti’s (2013) distinction between close and distant reading. 28 Greimas (1970). 29 Fillmore (1976). 30 This shows that a younger generation of semioticians not only masters the methodology of semiotics but also its computational translation and implementation.

Chapter 12 1 2 3 4

Freeth (2014). Halacy (1970). Copeland (2000). The common discourse uses software and hardware. But this distinction does not capture the real difference between computation as a formal theory for calculating a function and computation as a technology. A formal theory is not a ‘soft’ ware.

222

5

6 7

8 9 10 11 12 13 14 15 16 17 18 19 20 21

Notes

And there are many types of formal means by which a calculable function can be expressed. Cleland (2002) is one of the commentators of Turing who has insisted that in the Turing machine model there is an essential difference between the abstract procedures and the effective concrete procedures. ‘Effective procedures are physically possible only in worlds in which (at least some) physical processes have a special kind of causal structure’ (Cleland 2002: 171). This thesis is subject to strong debate and has opened up the theoretical problem of hypercomputation. If Turing machines can compute only computable functions, could there be other types of physical machines that compute non-computable functions? A simple example: an iPhone 5 has 2.7 times the power of a 1985 Crea-2. Also recall Moore’s law stating that the number of transistors engineers could pack on a silicon chip doubles every year. Think also of the transformation in the manner of inputting data in a computer: punch cards, keyboards, floppy discs, compact discs, scanners, USB keys, the cloud and so on. On biological computing, see Marr (2019). Calude and Paun (2000); Rozenberg, Bäck and Kok (eds) (2012). Adleman (1998). Rambidi et al. (1998). Rozenberg, Bäck and Kok (eds) (2012). Copeland, B. J. (2002b), ‘Hypercomputation’, Minds and Machines, 12 (4): 461–502; Loff, B. and Costa, J. (2009), ‘Five Views of Hypercomputation’, IJUC 5 (1 January 2009): 193–207. Sharma et al. (2018). Roth (2005); Chartier and Meunier (2011); Mongeau and Saint-Charles (2014). Levy (1999). An archive of this definition may be viewed using the Internet Archive’s Wayback Machine: https://web​.archive​.org​/web​/20150628161808​/http:/​/digital​.humanities​.ox​ .ac​.uk​/Support​/whatarethedh​.aspx. Ascari (2014). Pylyshyn (1984). Liu (2005). An important factor in the renewal of computational semiotics is the new generation of younger semioticians who manipulate computational models through a variety of computer technologies. This will introduce a ‘digital view’ in the practice of semiotics, maybe not at all levels, but precisely at those where computation is effective.

References Abdelhamid, N. and Thabtah, F. (2014), ‘Associative Classification Approaches: Review and Comparison’, Journal of Information and Knowledge Management, 13 (03): 1450027. Adjiman, R. (2018), ‘Les usages des ambiances sonores dans les films de fiction’, Communications, 102 (1): 137–51. Adleman, L. M. (1998), ‘Computing with DNA’, Scientific American, 279 (2): 54–61. Adler, J. (1994), ‘Testimony, Trust, Knowing’, Journal of Philosophy, 91: 264–75. Agazzi, E. and Montecucco, L. (2002), Complexity and Emergence, Bergamo, Italy: World Scientific Pub. Aggarwal, C. (2020), Linear Algebra and Optimization for Machine Learning: A Textbook, New York: Springer. Aggarwal, C. C. and Zhai, C., eds (2012), Mining Text Data, New York: Springer. Agrawal, R., Imieliński, T. and Swami, A. (1993), ‘Mining Association Rules between Sets of Items in Large Databases’, Proc. Conf. on Management of Data, 207–16, New York: ACM Press. Albus, J. S. and Meystel, A. M. (1996), ‘A Model Architecture for Design and Implementation of Intelligent Control in Large and Complex Systems’, International Journal of Intelligent Control and Systems, 1 (1): 15–30. Aldana, S. G., Gould, D. R., Troy, B., Adams, R., Whitmer, W., Merrill, R. M., George, V. and Noyce, J. (2012), ‘A Review of the Knowledge Base on Healthy Worksite Culture’, Journal of Occupational and Environmental Medicine, 54 (4): 414–19. Andersen, P. B. (1986), ‘Semiotics and Informatics: Computers as Media’, in P. Ingwersen, L. Kajberg and A. Mark Pejtersen (eds), Information Technology and Information Use: Towards a Unified View of Information and Information Technology, 64–97, London: Taylor. Andersen, P. B. (1991), A Theory of Computer Semiotics, Cambridge: Cambridge University Press. Andersen, P. B. (1997), A Theory of Computer Semiotics, Cambridge: Cambridge University Press. Anderson, C. (2008), ‘The End of Theory: The Data Deluge Makes the Scientific Method Obsolete’, Wired, 23 June 2008. Available online: https://www​.wired​.com​/2008​/06​/pb​ -theory/ (accessed 20 June 2020). Antsaklis, P. J. (1994), ‘On Intelligent Control: Report of the IEEE CSS Task Force on Intelligent Control,’ Technical Report of the ISIS (Interdisciplinary Studies of Intelligent Systems) Group, No ISIS-94-001. Apel, K.-O. (2000), ‘Globalization and the Need for Universal Ethics’, European Journal of Social Theory, 3 (2): 137–55. Arecchi, F.T. (2002), ‘Complexity and the Emergence of Meaning: Toward a Semiophysics’, in E. Aggazi and L. Montecucco (eds), Complexity and Emergence, 115–46, Bergamo, Italy: World Scientific Pub. Armatte, M. and Dahan-Dalmédico, A. (2004), ‘Modèles et modélisations, 1950–2000: Nouvelles pratiques, nouveaux enjeux’, Revue d’histoire des sciences, 57 (2): 243–303.

224

References

Arthur, D. and Vassilvitskii, S. (2007), ‘k-means++: The Advantages of Careful Seeding’, Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 1027–35, Society for Industrial and Applied Mathematics. Ascari, M. (2014), ‘The Dangers of Distant Reading: Reassessing Moretti’s Approach to Literary Genres’, Genre, 47 (1): 1–19. Ashby, R. W. ([1947] 1962), ‘Principles of the Self-organizing System’, in H. von Foerster and G. W. Zopf (eds), Principles of Self-organization, 255–78, Oxford: Pergamon Press. Ashby, W. R. (1956), An Introduction to Cybernetics, Hoboken: John Wiley & Sons. Association for Computational Linguistics: What is Computational Linguistics? Available online: https://www​.aclweb​.org​/portal​/what​-is​-cl (accessed 1 October 2020). Augenstein, I., Das, M., Riedel, S., Vikraman, L. and McCallum, A. (2017), ‘Extracting Keyphrases and Relations from Scientific Publications’. ArXiv Preprint ArXiv:1704. 02853. Available online: https://arxiv​.org​/abs​/1704​.02853 (accessed 6 January 2021). Babbage, C. and Campbell-Kelly, M. (1864), Passages From the Life of a Philosopher, London: Longman, Roberts & Green. Bachimont, B. (1996), ‘Herméneutique matérielle et artefacture: des machines qui pensent aux machines qui donnent à penser ’, PhD diss., École Polytechnique, Paris. Bastian, O. (2015), Digital semiotics: Making sense of the world |, https://www​.youtube​.com​ /watch​?v​=8O2ZBnYjcGw (accessed 1 November 2020). Baetu, T. M. (2014), ‘Models and the Mosaic of Scientific Knowledge. The Case of Immunology’, Studies in History and Philosophy of Science, Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 45: 49–56. Balbi, S. (1998), ‘Lo studio dei messaggi pubblicitari con l’analisi dei dati testuali’, Quaderni di Statistica, 1: 155–71. Band, J. (2006), ‘The Google Library Project: Both Sides of the Story’, Plagiary: CrossDisciplinary Studies in Plagiarism, Fabrication and Falsification. Available online: http://hdl​.handle​.net​/2027​/spo​.5240451​.0001​.002 (accessed 12 June 2020). Baroni, M. and Lenci, A. (2010), ‘Distributional Memory: A General Framework for Corpus-Based Semantics’, Computational Linguistics, 36 (4): 673–721. Barwise, J. and Perry, J. (1989), Situations and Attitudes, Cambridge, MA: MIT Press. Bengio, Y. and LeCun, Y. (2007), ‘Scaling Learning Algorithms towards AI’, in L. Bottou, O. Chapelle, D. DeCoste and J. Weston (eds), Large-Scale Kernel Machines, Cambridge: MIT Press. Bennett, S. (1993), A History of Control Engineering: 1930–1955, London: Institution of Engineering and Technology. Bersini, H., and Varela, F.J. (1991), ‘Hints for Adaptive Problem Solving Gleaned From Immune Networks’, in H. P. Schwefel and R. Männer (eds), Parallel Problem Solving from Nature. PPSN 1990. Lecture Notes in Computer Science, vol. 496. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0029775 Biber, D., Conrad, S. and Reppen, R. (1998), Corpus Linguistics: Investigating Language Structure and Use, Cambridge, New York: Cambridge University Press. Bishop, C. M. (2006), Pattern Recognition and Machine Learning, New York: Springer. Björner, A. (1995), ‘Topological Methods’, in R. L. Graham, M. Grötschel and L. Lovász (eds), Handbook of Combinatorics, 2, 1819–72, Cambridge: MIT Press. Blei, D. M., Ng, A. Y. and Jordan, M. I. (2003), ‘Latent Dirichlet Allocation’, Journal of Machine Learning Research, 3: 993–1022. Blei, D. (2012), ‘Probabilistic Topic Models’, Communications of the ACM, 55 (4): 77–84.



References

225

Bod, R. (2015), ‘A Comparative Framework for Studying the Histories of the Humanities and Science’, Isis, 106 (2): 367–77. Bolasco, S. (1998), ‘Meta-data and Strategies of Textual Data Analysis: Problems and Instruments, in Hayashi et al. (eds), Data Science, Classification and Related Methods (Proceedings V IFCS - Kobe, 1996), 468–79, Tokyo: Springer-Verlag. Bolasco, S. (2002), ‘Integrazione statistico-linguistica nell'analisi del contenuto’, in B. Mazzara (ed.), Metodi qualitativi in psicologia sociale, 329–42, Rome: Carocci. Boole, G. (1847), The Mathematical Analysis of Logic, Cambridge: Macmillan, Barclay and Macmillan. Bouissac, P. (1973), La Mesure des gestes: Prolégomènes à la Sémiotique Gestuelle, Berlin: Mouton De Gruyter. Bouissac, P. (2010), Semiotics at the Circus, 1st edition, Berlin, New York: Mouton de Gruyter. Bouissac, P. (2012), Circus as Multimodal Discourse: Performance, Meaning, and Ritual, London, New York: Bloomsbury Academic. Bourbaki, N. (1939), Éléments de mathématique, Paris: Hermann. Brachman, R. and Lévesque, H. J. (1985), Readings in Knowledge Representation, San Francisco: Kaufman. Brandom, R. B. (1994), Making it Explicit, Cambridge: Harvard University Press. Brandt, P. A. (2004), Spaces, Domains and Meanings: Essays in Cognitive Semiotics, Bern: Peter Lang. Brandt, P. A. (2013), The Communicative Mind: A Linguistic Exploration of Conceptual Integration and Meaning Construction, Newcastle: Cambridge Scholars Publishing. Brandt, P. A. (2019), ‘Semiotic Crumbs (Preprint)’. Available online: https://www​ .researchgate​.net​/publication​/335679399​_Semiotic​_Crumbs​_An​_anthology (accessed 6 January 2021). Brandt, P. A. (2020), Cognitive Semiotics, New York: Bloomsbury. Brickley, D. and Guha, R. V. (1999), Resource Description Framework (RDF) Schema Specification. Proposed Recommendation, World Wide Web Consortium: http://www​ .w3​.org​/TR​/PR​-rdf​-schema Brier, S. (2008), Cybersemiotics: Why Information Is Not Enough, Toronto: University of Toronto Press. Brier, S. (2011), ‘Cybersemiotics and the Question of Knowledge’, in G. Dodig-Crnkovic and M. Burgin (eds), Information and Computation, Essays on Scientific and Philosophical Understanding of Foundations of Information and Computation, 2: 1–34, World Scientific Series in information Studies, vol. 3, Singapore. Brooks, R. A. (1997), ‘Intelligence Without Representation’, in J. Haugeland (ed.), Mind Design II, 395–420, Cambridge: MIT Press. Brown, H. L. (2007), Conceptual Systems, London: Routledge. Bundgaard, P. and Stjernfelt, F. (2009), Signs and Meaning: Five Questions, New York: Automatic Press. Bunge, M. (1977), The Furniture of the World, Dordrecht: Reid. Burstall, R. M. and Landin, P. J. (1969), ‘Programs and Their Proofs, an Algebraic Approach’, Machine Intelligence, 4: 17. Byrd, R., Calzolari, N., Chodorow, M., Klavans, J., Neff, M. and Rizk, O. (1987), ‘Tools and Methods for Computational Lexicology’, Computational Linguistics, 13: 219–40. Caccamo, E. (2019), ‘Les procédés algorithmiques au prisme des approches sémiotiques’, Cygne noir, 7: 1–30. http://revuecygnenoir​.org​/numero​/article​/meunier​-modelisation​ -semiotique​-computationnelle

226

References

Calude, C. and Paun, G. (2000), Computing with Cells and Atoms: An Introduction to Quantum, DNA and Membrane Computing, London, New York: Taylor and Francis. Camargo, J. E., Caicedo, J. C. and Gonzalez, F. A. (2013), ‘A Kernel-Based Framework for Image Collection Exploration’, Journal of Visual Languages and Computing, 24 (1): 53–67. Campbell, N. (1920), Physics: The Elements, Cambridge: Cambridge University Press. Cannizzaro, S. (2012), Semiotics / Semiology, Communication Studies, ICT Information and Communications Technologies, Tartu: University of Tartu Press. Cannizzaro, S. (2016), ‘Internet Memes as Internet Signs: A Semiotic View of Digital Culture’, Σημειωτκή - Sign Systems Studies, 44: 562–86. Carnap, R. (1947), Meaning and Necessity: A Study in Semantics and Modal Logic, Chicago: Chicago University Press. Cartwright, N. (1983), How the Laws of Physics Lie, Oxford: Oxford University Press. Cassirer, E. (1944), An Essay on Man, New Haven: Yale University Press. Chaitin, G. J. (1998), The Limits of Mathematics, London: Springer. Chaitin, G. J., Doria, F. A. and da Costa, N. C. A. (2012), Gödel’s Way: Exploits Into an Undecidable World, Boca Raton: CRC Press. Chapelle, O., Haffner, P. and Vapnik, V. N. (1999), ‘Support Vector Machines for Histogram-Based Image Classification’, IEEE Transactions on Neural Networks, 10 (5): 1055–64. Charniak, E. and McDermott, D. (1985), Introduction to Artificial Intelligence, Boston: Addison-Wesley. Chartier, J.-F., Meunier, J.-G., Jendoubi, M. and Danis, J. (2008), ‘Le travail conceptuel collectif: une analyse assistée par ordinateur de la distribution du concept d’accommodement raisonnable dans les journaux québécois’, in S. Heiden and B. Pincemin (eds), Actes des 9es Journées internationales d’analyse statistique des données textuelles (JADT 2008), 297–307, Lyon, France: Presses universitaires de Lyon. Chartier, J.-F. and Meunier, J.-G. (2011), ‘Text Mining Methods for Social Representation Analysis in Large Corpora’, Papers on Social Representations, 20 (2): 37.1–37.4. Available online: http://psr​.iscte​-iul​.pt​/index​.php​/PSR​/article​/view​/452 (accessed 6 January 2021). Chartier, J.-F., Pulizzotto, D., Chartrand, L. and Meunier, J.-G. (2019), ‘A Data-Driven Computational Semiotics: The Semantic Vector Space of Magritte’s Artworks’, Semiotica, 230: 19–69. Chartrand, L., Chartier, J.-F. and Meunier, J.-G. (2013), Des mots pour se retrouver: Recherche d’information dans l’oeuvre de Magritte à l’aide d’un corpus de descripteurs sémiotiques, Montréal: Université du Québec à Montréal: Les cahiers du LANCI, Laboratoire d’ANalyse Cognitive de l’Information, 9. Chartrand, L., Meunier, J.-G., Pulizzotto, D. and Lareau, F. (2018), ‘Un algorithme pour extraire les segments qui expriment un concept – premières expérimentations’, in L. Hébert, Applied Semiotics/Sémiotique appliquée: A Learned Journal of Literary Research on the World Wide Web/Une revue internationale de recherche littéraire sur Internet, 26: 108–124. Chen, C. (2013), Mapping Scientific Frontiers: The Quest for Knowledge Visualization, 2nd ed., London: Springer-Verlag. Chen, W.-Y., Wang, Y.-C. F., Liu, Y.-C., Kira, Z. and Huang, J.-B. (2019), ‘A Closer Look at Few-shot Classification’, arXiv:1904.04232 [cs.CV]. Available online: https://arxiv​.org​/ abs​/1904​.04232 (accessed 6 January 2021). Chen, X., S. Jia and Y. Xiang (2020), ‘A Review: Knowledge Reasoning over Knowledge Graph’. Expert Systems with Applications, 141, no. 112948: 1–21.



References

227

Chiba, J. (2017), ‘Shakespeare’s Ontology’, Thesis, Royal Holloway, University of London. Chomsky, N. (1956), ‘Three Models for the Description of Language’, Institute of Radio Engineers, Transactions on Information Theory, 2 (3): 113–24. Chomsky, N. (1957), Syntactic Structures, The Hague: Mouton De Gruyter. Church, A. (1937), ‘Review of Turing (1936)’, Journal of Symbolic Logic, 2: 42–3. Church, A. (1941), The Calculi of Lambda-Conversion, Princeton: Princeton University Press. Church, A. (1936), ‘A note on the Entscheidungsproblem’, Journal of Symbolic Logic, 1: 40–41. Churchland, P. M. (1989), A Neurocomputational Perspective: The Nature of Mind and the Structure of Science, Cambridge: MIT Press. CIDOC CRM (Conceptual Reference Model). http://www​.cidoc​-crm​.org/ Cioffi-Revilla, C. (2010), ‘Computational Social Science’, Wiley Interdisciplinary Reviews: Computational Statistics, 2: 259–71. Clarke, A. (2004), Computational Semiotics: Past Present and Future. COSIGN 2004: Proceedings. University of Split, Croatia. Ciula, A. and Eide, Ø. (2017), ‘Modelling in Digital Humanities: Signs in Context’, Digital Scholarship in the Humanities, 32 (1): i33–46. Ciula, A. and Marras, C. (2019), ‘Exploring a Semiotic Conceptualisation of Modelling in Digital Humanities Practices’, in A. Olteanu, A. Stables and D. Borun (eds), Meanings & Co.: The Interdisciplinarity of Communication, Semiotics and Multimodality, 33–52, Berlin: Springer Verlag. Clark, A. (1998), Being There: Putting Brain, Body, and World Together Again, Cambridge: MIT Press. Cleland, C. E. (2002), ‘On Effective Procedures’, Minds and Machines, 12: 159–79. Codd, E. F. (1983), ‘A Relational Model of Data for Large Shared Data Banks’, Communications of the ACM, 26: 64–9. Cohen, D. J., and Rosenzweig, R. (2006), Digital History: A Guide to Gathering, Preserving, and Presenting the Past on the Web, vol. 28, Philadelphia: University of Pennsylvania Press. Coletta, W. J. (2015), ‘Semiotic Modeling: A Pragmaticist’s Guide’, in P. P. Trifonas (ed.), International Handbook of Semiotics, 951–80, New York: Springberg. Comer, D. (2017), Essentials of Computer Architecture, 2nd edn, Boca Raton: CRC Press, Taylor & Francis Group. Compagno, D., ed. (2018), Quantitative Semiotic Analysis, New York: Springer. Complete Works of William Shakespeare. Available online: http://shakespeare​.mit​.edu/ (accessed 31 October 2020). Computational Social Science Society of the Americas. (n. d.), The Computational Social Science Society of the Americas. Available online: https://com​puta​tion​also​cial​science​ .org/ (accessed 12 June 2020). Condillac, E. B. (1798), Œuvres Complètes : La langue des calculs, vol. 16, Paris: J. L. G. Brière, Libraire. Console, L. and Saitta, L. (2000), ’On the Relations Between Abductive and Inductive Explanation’. In P. A. Flach and A. C. Kakas (eds), Abduction and Induction: Essays on Their Relation and Integration, Applied Logic Series, 133–51, Dordrecht: Springer. Cooper, S. B. (2004), ‘The Incomputable Alan Turing’, Proceedings of the 2004 International Conference on Alan Mathison Turing: A Celebration of His Life and Achievements, Manchester University: BCS Learning & Development Ltd. Copeland, B. J. (1997), ‘The Broad Conception of Computation’, American Behavioural Scientist, 40: 690–716.

228

References

Copeland, B. J. (2000), ‘The Modern History of Computing’, in E. N. Zalta (ed.), The Stanford Encyclopedia of Philosophy (Winter 2000 first archived. Retrieved 1 March 2017), Metaphysics Research Lab, Stanford University. Copeland, B. J. (2002a), ‘The Church-Turing Thesis’, in E. N. Zalta (ed.), The Stanford Encyclopedia of Philosophy, Standford: Stanford University. Copeland, B. J. (2002b), ‘Hypercomputation’, Minds and Machines, 12 (4): 461–502. Coseriu, E. (2007), Lingüística del texto. Introducción a la hermenéutica del sentido, Ò. Loureda Lamas (ed.), Madrid: Arco Libros/ La Muralla. COSIGN 2002: The 2nd International Conference on Computational Semiotics for Ganes and New Media Accessed 21 January 2020, from https://idw​-online​.de​/en​/event​?print​ =1​&id​=5226 Crawford, T. and Lewis, R. (2016), ‘Review: Music Encoding Initiative’, Journal of the American Musicological Society, 69 (1): 273–85. Costa, M., (2015), Dopo la tecnica. Dal chopper alle similcose, Napoli, Liguori Editore. Craik, K. (1943). The Nature of Explanation, Cambridge: Cambridge University Press. Crowder, J. A., Carbone, J. and Friess, S. (2020), ‘Abductive Artificial Intelligence Learning Models’, in Artificial Psychology, Cham: Springer. http://doi​-org​-443​.webvpn​.fjmu​.edu​ .cn​/10​.1007​/978​-3​-030​-17081​-3_5 Curry, H. B. and Feys, R. (1958), Combinatory Logic (Studies in Logic and the Foundations of Mathematics: Volume I), 1st edn, Amsterdam: North-Holland. Cushing, J. (1987), ‘Models and Methodologies in Current Theoretical High-Energy Physics’, Synthèse, 50: 5–101. Czachesz, I. (2017), ‘Network Analysis of Biblical Texts’, Journal of Cognitive Historiography, 3 (1–2): 43–67. d’Aquin, M., Ding, L. and Motta, E. (2011), ‘Semantic Web Search Engines’, in J. Domingue, D. Fensel and J. A. Hendler (eds), Handbook of Semantic Web Technologies, 659–700, Berlin: Springer-Verlag. Dadesio, T. C. (1995), On Minds and Symbols. The Relevance of Cognitive Science for Semiotics, Berlin: Mouton De Gruyter. Dai, W.-Z., Xu, Q., Yu, Y. and Zhou Z.-H. (2019), ‘Bridging Machine Learning and Logical Reasoning by Abductive Learning’, Advances in Neural Information Processing Systems, 32: 2815–26. Damjanovic, V., Gasevic, D., Devediiic, V. B. and Djuric, D. (2004), ‘The Semiotics Contribution on the Web,’ in 2004 2nd International IEEE Conference on ‘Intelligent Systems’ Proceedings (IEEE Cat. No.04EX791), 1: 244–9. Danca, M.-F. and Chen, G. (2004), ‘Bifurcation and Chaos in a Complex Model of Dissipative Medium’, International Journal of Bifurcation and Chaos in Applied Sciences and Engineering, 14 (10): 3409–47. Danesi, M. (2016), The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet, London: Bloomsbury. Danesi, M. (2018), Understanding Media Semiotics. London: Bloomsbury Publishing. Darton, R, (2011), ‘Six Reasons Google Books Failed’, The New York Review of Books, March 28. Davis, M., Matiyasevich, Y. and Robinson, J. (1976), ‘Hilbert’s Tenth Problem: Diophantine Equations: Positive Aspects of a Negative Solution’, in F. E. Browder (ed.), Mathematical Developments Arising from Hilbert, Part I, New Brunswick, NJ: American Mathematical Society, Rutgers University. de Arbez, G., Birta, L. Hoffmayer, G., Tolk, A. and Wagner, G. (2015), ‘Conceptual Modeling: Definition, Purpose and Benefits’, 2015 Winter Simulation Conference



References

229

(WSC), 2812–2826. Available online: https://www​.informs​-sim​.org​/wsc15papers​/277​ .pdf (accessed 12 June 2020). De Sousa, C. S. (2005), Semiotic Engineering of Human-Computer Interaction, Cambridge: MIT Press. De Tienne, A. (2006), ‘Peirce’s Logic of Information’, Seminario del Grupo de Estudios Peirceanos, Universidad de Navarra. Available online: https://www​.unav​.es​/gep​/ SeminariodeTienne​.html (accessed 11 June 2020). Deely, J., Brook, W. and Kruse, F. E. (1986), Frontiers in Semiotics, Bloomington: Indiana University Press. Dennett, D. (1978) Brains Storms, Cambridge, MA: MIT Press. Desclés, J.-P. (1987), ‘Réseaux sémantiques. La nature logique et linguistique des relateurs’, Langages, 87: 57–78. Desclés, J.-P. (1989a), ‘Intermediate Representations in the Cognitive Sciences’, Semiotica, 77: 121–36. Desclés, J.-P. (1989b), ‘State, Event, Process and Topology’, General Linguistics, 29 (3): 159–200, Philadelphia: The Pennsylvania University Press. Desclés, J.-P. (1990), ‘Archétypes cognitifs et types de procès’, Travaux de linguistique et de philologie, 29: 171–95. Desclés, J.-P. (1997), ‘Schèmes, notions, prédicats et termes’, Logique, discours et pensée. Mélanges offerts à Jean-Blaize Grize, 9–36, Bern: Peter Lang. Desclés, J.-P. (2003), ‘La Grammaire Applicative et Cognitive construit-elle des représentations universelles?’, Linx, 48: 139–60. Desclés, J.-P. and Cheong, K.-S. (2006), ‘Analyse critique de la notion de variable. Points de vue sémiotique et formel’, Mathematics and Social Sciences, 173 (1): 43–102. Desclés, J.-P. and Le Priol, F., eds (2009), Annotations automatiques et recherche d'information, Paris: Hermès / Lavoisier. Devlin, J., Chang, M., Lee, K. and Toutanova, K. (2019), ‘BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding’, ArXiv:1810.04805 [Cs]. Available online: http://arxiv​.org​/abs​/1810​.04805 (accessed 1 July 2020). Dietrich, E. (1999), ‘Algorithm’, in MIT Encyclopedia of the Cognitive Sciences, Cambridge: MIT Press. Cohen, D. J., and Rosenzweig, R. (2006), Digital History: A Guide to Gathering, Preserving, and Presenting the Past on the Web, 28, Philadelphia: University of Pennsylvania Press. Available online: https://chnm​.gmu​.edu​/digitalhistory​/authors​.php (accessed 6 January 2021). Döben-Henisch, G., Erasmus, L. and Hasebrook, J. (2002), ‘Knowledge Robots for Knowledge Workers: Self-Learning Agents connecting Information and Skills’, in L. C. Jain, Z. Chen and N. Ichalkaranje (eds), Intelligent Agents and Their Applications, 59–79, New York: Springer. Dodig-Crnkovic, G. (2006), Investigations into Information Semantics and Ethics of Computing, Västerås: Mälardalen University Press Dissertations, 33. Dondero, M. G. and Fontanille, J. (2012), Des images à problèmes: Le sens du visuel à l’épreuve de l’image scientifique, Limoges: Presses universitaires de Limoges. Dondero, M. G. and Fontanille, J. (2014), The Semiotic Challenge of Scientific Images: A Test Case for Visual Meaning, New York: Legas. Dong, X. L. and Rekatsinas, T. (2018), ‘Data Integration and Machine Learning: A Natural Synergy’, in SIGMOD/PODS '18: International Conference on Management of Data, Houston. Dorigo, M. and Stützle, T. (2004), Ant Colony Optimization, Cambridge, MA: MIT Press.

230

References

Dormans, J. and Leijnen, S. (2013), ‘Combinatorial and Exploratory Creativity in Procedural Content Generation’. Available online: https://repository​.ubn​.ru​.nl​/ bitstream​/handle​/2066​/123037​/123037​.pdf (accessed 12 June 2020). Douven, I. (2017), ‘Inference to the Best Explanation: What Is It? And Why Should We Care?’, in T. Poston and K. McCain (eds), Best Explanations: New Essays on Inference to the Best Explanation, 7–24, Oxford: Oxford University Press. Dpedia Project. https://en​.wikipedia​.org​/wiki​/DBpedia (accessed 24 January 2020). Dretske, F. I. (1982), Knowledge and the Flow of Information, London: MIT. Dreyfus, H. L. (1972), What Computers Can’t Do: A Critique of Artificial Reason, New York: Harper and Row. Duhem, P. (1902), Les théories électriques de J. Clerk Maxwell. Étude historique et critique, Paris: Hermann, 223, cited in Ariew, R. (2007), ‘Pierre Duhem’, E. N. Zalta (ed.), The Stanford Encyclopedia of Philosophy (Fall 2007). Dumais, S. T. (2005), ‘Latent Semantic Analysis’, Annual Review of Information Science and Technology, 38: 188–230. Easley, D. and Kleinberg, J. (2010), Networks, Crowds, and Markets: Reasoning About a Highly Connected World, Cambridge: Cambridge University Press. Eco, U. (1965), L'Œuvre ouverte, Paris: Seuil. Eco, U. (1979), A Theory of Semiotics, Bloomington: Indiana University Press. Edling, C. R. (2002), ‘Mathematics in Sociology’, Annual Review of Sociology, 28: 197–220. Electronic Literature Organization. Retrieved 21 January 2021, from https://eliterature​ .org/. Evans, J. St B. T. (2003), ‘In Two Minds: Dual-process Accounts of Reasoning’, Trends in Cognitive Sciences, 7 (10): 454–9. Evans, J. St. B. T. (2008), ‘Dual-Processing Accounts of Reasoning, Judgment, and Social Cognition’, Annual Review of Psychology, 59: 255–78. Evans, J. St B. T. et K. Frankish. (2009), ‘The duality of Mind: An Historical Perspective’, In Two Minds: Dual Processes and Beyond, Jonathan St B. T.; Keith Evans, Frankish, 1–35, Oxford; New York: Oxford University Press. Farmer, J. D., Packard, N. H. and Perelson, A. S. (1986), ‘The Immune System, Adaptation, and Machine Learning’, Physica D: Nonlinear Phenomena, 22: 187–204. Feigenbaum, E. A. (1977), ‘The Art of Artificial Intelligence: Themes and Case Studies of Knowledge Engineering’, in Proceedings of the 5th International Joint Conference on Artificial Intelligence - Volume 2, 1014–29, IJCAI’77, San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. Fetzer, J. H. (1990), Artificial Intelligence: Its Scope and Limits, Dordrecht: Kluwer. Fetzer, J. H. (2001), Computers and Cognition: Why Minds Are Not Machines, Dordrecht: Kluwer. Feyerabend, P. (1975), Against Method. An Outline of an Anarchistic Theory of Knowledge, London: New Left Book. Fillmore, C. J. (1968), ‘The Case for Case’, in E. Bach and R. T. Harms (eds), Universals in Linguistic Theory, 1–88, New York: Holt, Rinehart and Winston. Fillmore, C. J. (1976), ‘Frame Semantics and the Nature of Language’, Annals of the New York Academy of Sciences: Conference on the Origin and Development of Language and Speech, 280 (1): 20–32. Fillmore, C. J. (1982), In Linguistics in the Morning Calm, Linguistics in the Morning Calm, South Korea edition, Seoul: Hanshin Publishing Co. Firth, J. R. (1957), ‘A Synopsis of Linguistic Theory, 1930–1955’, in Studies in linguistic analysis, Special Volume of the Philological Society, 1–32, Oxford: Oxford University Press.



References

231

Floch, J.-M. (1981), ‘Kandinsky: sémiotique d'un discours plastique non-figuratif ’, Communications, 34: 134–58. Floridi, L. (2004), ‘Outline of a Theory of Strongly Semantic Information’, Minds and Machines, 14: 197–221. Floridi, L. (2009), ‘Against Digital Ontology’, Synthese, 168 (1): 151–78. Fodor, J. (1998), Concepts: Where Cognitive Science Went Wrong, Oxford: Oxford University Press. Fodor, J. A. and Pylyshyn, Z. W. (1988), ‘Connectionism and Cognitive Architecture: A Critical Analysis’, Cognition, 28 (1–2): 3–71. Freebase Project, https://en​.wikipedia​.org​/wiki​/Freebase (accessed 20 December 2020). Freeth, T. (2014), ‘Eclipse Prediction on the Ancient Greek Astronomical Calculating Machine Known as the Antikythera Mechanism’, PLoS One, 9 (Issue 7): 1–14. https:// journals​.plos​.org​/plosone​/article​?id​=10. 1371/journal. pone. 0103275. Frigg, R. and Hartmann, S. (2012), ‘Models in Science’, in E. N. Zalta (ed.), The Stanford Encyclopedia of Philosophy, Fall 2012 Edition. Fuertes-Olivera, P. A. (2017), The Routledge Handbook of Lexicography, London: Routledge. Fuß, E., Konopka, M., Trawiński, B. and Waßner, U. H. (2018), Grammar and Corpora 2018, Heidelberg: Heidelberg University Publishing. Gallica: https://gallica​.bnf​.fr​/accueil​/en​/content​/accueil​-en Gandy, R. (1988), ‘The Confluence of Ideas in 1936’, in R. Herken (ed.), The Universal Turing Machine: A Half-Century Survey, 51–102, Oxford: Oxford University Press. Gandy, R. O. (1980), ‘Church’s Thesis and Principles of Mechanisms’, in J. Barwise, J. J. Keisler and K. Kunen (eds), The Kleene Symposium, 123–48, Amsterdam: North-Holland. Garcez, A., d’Avila, S., Broda, K. and Gabbay, D. M. (2002), Neural-Symbolic Learning Systems: Foundations and Applications, London: Springer-Verlag. Gardenfors, P. (2000), Conceptual Spaces, Cambridge: MIT Press. Gelfert, A. (2011), ‘Mathematical Formalisms in Scientific Practice: From Denotation to Model-Based Representation’, History and Philosophy of Science, 42 (2): 272–86. Gelfert, A. (2017), ‘The Ontology of Models’, in L. Magnani and T. Bertolotti (eds), Springer Handbook of Model-Based Science, 5–23, New York: Springer. Genesereth, M. and Nilsson, N. J. (1987), Logical Foundations of Artificial Intelligence, San Mateo: Morgan Kaufmann. Genette, G. (1979), Introduction à l’architexte, Paris: Seuil. Gentner, D. and Smith, L. (2012), ‘Analogical Reasoning’, in V. S. Ramachandran (ed.), Encyclopedia of Human Behavior, 2nd edn, 130–6, San Diego: Academic Press. Gentzen, G. (1955), Recherches sur la déduction logique [« Untersuchungen über das logische schließen »], trans. R. Feys and J. Ladrière), Presses Universitaires de France. Gere, C (2002), Digital Culture, London: Reaktion Books Gerber, D., Hellmann, S., Bühmann, L., Soru, T., Usbeck, R. and Axel-Cyrille, N. N. (2013), ‘Real-Time RDF Extraction from Unstructured Data Streams’, in H. Alani, L. Kagal, A. Fokoue, P. Groth, C. Biemann, J. Xavier Parreira, L. Aroyo, N. Noy, C. Welty and K. Janowicz (eds), The Semantic Web – ISWC 2013, Lecture Notes in Computer Science, 135–50, Berlin, Heidelberg: Springer. Giere, R. (1999), ‘Using Models to Represent Reality’, in L. Magnani, L. Nersessian and P. Thagard (eds), Model-Based Reasoning in Scientific Discovery, 41–57, New York: Plenum Publisher. Giere, R. N. (2010), Explaining Science: A Cognitive Approach, Chicago: University of Chicago Press.

232

References

GLAM/Projects (2020), in Wikipedia: https://en​.wikipedia​.org​/w​/index​.php​?title​ =Wikipedia​:GLAM​/Projects​&oldid​=978170777. Glaser, B. G. and Strauss, A. L. (1967), The Discovery of Grounded Theory: Strategies for Qualitative Research, Chicago: Adline. Godfrey-Smith, P. (2005), ‘Folk Psychology as a Model’, Philosophers’ Imprint, 5 (6). Available online: https://quod​.lib​.umich​.edu​/p​/phimp​/3521354​.0005​.006/1 (accessed 12 June 2020). Goodfellow, I., Bengio, Y. and Courville, A. (2016), Deep Learning, Cambridge: MIT Press. Google Books (2020), in Wikipedia, The Free Encyclopedia. Available online: https://en​ .wikipedia​.org​/wiki​/Google​_Books (accessed 6 January 2021). Google ‘Knowledge Vault’, To Power Future Of Search, 25 August 2014. https:// searchengineland​.com​/google​-builds​-next​-gen knowledge-graph-future-201640 (accessed 20 December 2020). Gotel, O., Cleland-Huang, J., Huffman Hayes, J., Zisman, A., Egyed, A., Grünbacher, P., Dekhtyar, A., Antoniol, G., Maletic, J. and Mäder, P. (2012), ‘Traceability Fundamentals’, in J. Cleland-Huang, O. Gotel and A. Zisman (eds), Software and Systems Traceability, 3–22, London: Springer. Gould, P. (1981), ‘Letting the Data Speak for Themselves’, Annals of the Association of American Geographers, 71 (2): 166–76. Granger, G. G. (1967), Pensée formelle et sciences de l’homme, Paris: Aubier-Montaigne. Graziano, M. S. A. (2013), Consciousness and the Social Brain, Oxford: Oxford University Press. Green, C. and Abraham, A. (2011), ‘Rule-Based Expert Systems’, Intelligent Systems Reference Library, 17, Berlin: Springer. Green, S. (2013), ‘When One Model Is Not Enough: Combining Epistemic Tools in Systems Biology’, Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 44: 170–80. Greimas, A. J. (1970), Du sens: Essais sémiotiques, Paris: Le Seuil. Greimas, A. J. (1979), Introduction à l’analyse du discours en sciences sociales, Paris: Hachette. Greimas, A. J. and Courtés, J. (1979), Sémiotique: Dictionnaire raisonné de la théorie du langage, Paris: Hachette. Grigar, D. and O’Sullivan, J. (2021), Electronic Literature as Digital Humanities, Bloomsbury Academic. Groupe µ (1992), Traité du signe visuel. Pour une rhétorique de l’image, Paris: Seuil. Gruber, T. R. (1993b), ‘A Translation Approach to Portable Ontology Specifications’, Knowledge Acquisition, 5 (2): 199–220. Grzegorczyk, R. (1957), ‘On the Definitions of Computable Real Continuous Functions’, Fundamentals of Mathematics, 44: 61–71. Guan, L. (2017), Multimedia Image and Video Processing, Boca Raton: Taylor and Francis. Guarino, N. (1995), ‘Formal Ontology Conceptual Analysis and Knowledge Representation’, International Journal of Human-Computer Studies, 43: 625–40. Guarino, N. and Giaretta, P. (1995), ‘Ontologies and Knowledge Bases: Towards a Terminological Clarification’, in J. J. I. Mars (ed.), Towards Very Large Knowledge Bases, 25–32, Amsterdam: IOS Press. Gudwin, R. (1999), ‘From Semiotics to Computational Semiotics’, in Proceedings of the 9th International Congress of the German Society for Semiotic Studies, 7th International



References

233

Congress of the International Association for Semiotic Studies (IASS/AIS), Dresden, University of Technology, October 6–11. Gudwin, R. and Gomide, F. A. C. (1997a), ‘Computational Semiotics: An Approach for the Study of Intelligent Systems, Part One: Foundations’. Available online: ftp://ftp​.dca​.fee​ .unicamp​.br​/pub​/docs​/gudwin​/publications​/r​ep1​_97​.pdf (accessed 12 June 2020). Gudwin, R. and Gomide, F. (1997b), ‘Computational Semiotics : A New Approach for the Study of Intelligent System Part II : Theory and Application’, DCA-FEEC-UNICAMP. Available: https://www.researchgate.net/publication/2299717_Computational_ Semiotics_A_New_Aproach_for_the_Study_of_Intelligent_System_Part_II_Theory_ and_Application (accessed April 2021). Gudwin, R. and Queiroz, J. (2005), ‘Towards an Introduction to Computational semiotics’, KIMAS 2005, Waltham MA, USA. Available online: https://www​.researchgate​.net​ /publication​/4142360​_Towards​_an​_introduction​_to​_computational​_semiotics (accessed 12 June 2020). Gudwin, R. and Queiroz, J. (2007), Semiotics and Intelligent Systems Development, London: Idea Group Publishing. Guizzardi, G. (2005), Ontological Foundations for Structural Conceptual Models, Enschede: Telematica Instituut / CTIT, PhD Thesis Series, No 05-74. Habermas, J. (1984, 1987), Theory of Communicative Action, trans. T. A. McCarty, vol. 1 & 2, Boston: Beacon Press. Halacy, D. S. (1970), Charles Babbage, Father of the Computer, New York: Crowell-Collier Press. Hallée, Y. and Garneau, J. M. É. (2019), ‘L’abduction comme mode d’inférence et méthode de recherche: de l’origine à aujourd’hui’. Recherches qualitatives, 38 (1): 124. Halliday, M. A. K. and Webster, J., eds (2014), Text Linguistics: The How and Why of Meaning, Sheffield, Bristol: Equinox. Hardman, L., van Ossenbruggen, J., Aroyo, L. and Hyvönen, E., eds (2009), ‘Using AI to Access and Experience Cultural Heritage’, Intelligent Systems, 24 (2): 23–5. Hare, J. S., Lewis, P. H., Enser, P. G. B. and Sandom, C. (2006), ‘Mind the Gap: Another Look at the Problem of the Semantic Gap in Image Retrieval’, in E. Y. Chang, A. Hanjalic and N. Sebe (eds), Multimedia Content Analysis, Management and Retrieval 2006, SPIE V, Society of Photo-Optical Instrumentation Engineers, 607309 (1). Available online: https://eprints​.soton​.ac​.uk​/261887​/1​/article​.pdf (accessed 13 June 2020). Harman, G. (1977), ‘Semiotics and the Cinema: Metz and Wollen’, Quarterly Review of Film Studies, 2 (1): 15–25. Harnad, S. (1990a), ‘The Symbol Grounding Problem’, Physica D: Nonlinear Phenomena, 42(1–3):335–46. Harnad, S., ed. (1990b), Categorical Perception: The Groundwork of Cognition, Cambridge: Cambridge University Press. Harnad, S. (2017), ‘To Cognize Is to Categorize: Cognition Is Categorization’, Handbook of Categorization in Cognitive Science, 21–54. New York: Elsevier. Harris, Z. (1982), ‘Mathematical Analysis of Language’, in L. J. Cohen, J. Łoś, H. Pfeiffer and K.-P. Podewski (eds), Studies in Logic and the Foundations of Mathematics, 104: 623–37, Amsterdam: Elsevier. Harrison, M. A. (1978), Introduction to Formal Language Theory, Reading, MA: Addison Wesley. Hartigan, J. A. (1975), Clustering Algorithms, Hoboken: John Wiley & Sons. Haslanger, S. (1995), ‘Ontology and Social Construction’, Philosophical Topics, 23 (2): 95–125.

234

References

Haslanger, S. and Saul, J. (2006), ‘Philosophical Analysis and Social Kinds’, Proceedings of the Aristotelian Society, 80: 89–14. Haugeland, J. (1986), Artificial Intelligence: The Very Idea - John Haugeland, Cambridge: MIT Press. Hayes, P. and Michie, D., eds (1979), ‘The Naive Physics Manifesto’, in Expert Systems in the Micro-electronic Age, Edinburgh: Edinburgh University Press. Hébert, L. and Dumont-Morin, G. (2012), ‘Dictionnaire de sémiotique générale’, in L. Hébert (ed.), Signo, Rimouski: Université du Québec à Rimouski. Available online: http://www​.signosemio​.com​/documents​/dictionnaire​-semiotique​-generale​.pdf (accessed 8 June 2020). Hébert, L. and Trudel, E. (2003–2013), Magritte. Toutes les œuvres, tous les thèmes – Base de données Internet. Available online: http://www​.magrittedb​.com/ (accessed 3 December 2020). Hébert, L., Michelucci, P. and Trudel, E. (2018), Magritte. Perspectives Nouvelles, Nouveaux Regards, Montréal: Nota bene. Heiden, S. (2010), ‘The TXM Platform: Building Open-Source Textual Analysis Software Compatible with the TEI Encoding Scheme’, in R. Otoguro, K. Ishikawa, H. Umemoto, K. Yoshimoto and Y. Harada (eds), 24th Pacific Asia Conference on Language, Information and Computation - PACLIC24, 389–98, Sendai, Japan: Institute for Digital Enhancement of Cognitive Development, Waseda University. Hempel, C. G. (1965), ‘Aspects of Scientific Explanation’, in C. G. Hempel (ed.), Aspects of Scientific Explanation and Other Essays in the Philosophy of Science, 331–496, New York: Macmillan. Hesse, M. ([1966] 1970), Models and Analogies in Science, Notre Dame: Notre Dame University Press. Hintikka, J. (1997), ‘On the Development of the Model-Theoretic Viewpoint in Logical Theory’, in J. Hintikka (ed.), Lingua Universalis vs. Calculus Ratiocinator: An Ultimate Presupposition of Twentieth-Century Philosophy, 104–61, Dordrecht: Springer. Hjelmslev, L. (1968), Prolégomènes à une théorie du langage, suivi de la structure fondamentale du langage, trans. U. Canger and A. Wewer, Paris: Minuit. Hobbes, T. ([1651] 1972), Leviathan, or The Matter, Forme, and Power of a CommonWealth Ecclesiasticall and Civill, I. Shapiro (ed.), New Haven: Yale University Press. Hobbs, J. R., Stickel, M. E., Appelt, D. E. and Martin P. (1993), ‘Interpretation as Abduction’, Artificial Intelligence, 63 (1–2): 69–142. Hoffmeyer, J. (1996), Signs of Meaning in the Universe, Bloomington: Indiana University Press. Hoffmeyer, J. (1998), ‘Surfaces Inside Surfaces: On the Origin of Agency and Life’, Cybernetics and Human Knowing, 5 (1): 33–42. Hoffmeyer, J. (2008), Biosemiotics: An Examination into the Signs of Life and the Life of Signs, Scranton: University of Scranton Press. Hofstadter, D. (1983), ‘Artificial Intelligence: Sub Cognition as Computation’, in F. Machlup and U. Mansfield (eds), The Study of Information: Interdisciplinary Messages, vol. 263, New York: John Wiley & Sons. Holland, J. (1975), Adaptation in Natural, and Artificial Systems, Ann Arbor: University of Michigan Press. Horev, R. (2018), ‘BERT Explained: State of the Art Language Model for NLP’, Towards Data Science. Available online: https://towardsdatascience​.com​/bert​-explained​-state​-of​ -the​-art​-language​-model​-for​-nlp​-f8b21a9b6270 (accessed 1 March 2019).



References

235

Hotho, A., Nürnberger, A. and Paaß, G. (2005), ‘A Brief Survey of Text Mining’, LDV Forum - GLDV Journal for Computational Linguistics and Language Technology, 20 (1): 19–62. Hung, C. and Wermter, S. (2004), ‘Neural Network Based Document Clustering Using WordNet Ontologies’, International Journal of Hybrid Intelligent Systems, 1: 127–42. Hunt, T. L. and Lipo, C. P. (2011), The statues That Walked: Unraveling the Mystery of Easter Island (1st Free Press hardcover ed), New York: Free Press. ICA International Council of Archives. https://www​.ica​.org​/en ICOM International Council of Museums. https://icom​.museum​/en IFLA International Federation of Library Associations and Institutions. http://origin​.ifla​ .org/ Index Thomisticus. Available online: http://www​.corpusthomisticum​.org​/it​/index​.age (accessed 3 October 2020). Jabareen, Y. (2009), ‘Building a Conceptual Framework: Philosophy, Definitions, and Procedure’, International Journal of Qualitative Methods, 8 (4): 49–62. Jackson, P. (1998), Introduction to Expert Systems, 3rd edn, Boston: Addison-Wesley Longman. Jaimes, A. and Smith, J. R. (2003), ‘Semi-automatic, Data-driven Construction of Multimedia Ontologies, 2003 International Conference on Multimedia and Expo. ICME ’03. Proceedings (Cat. No. 03TH8698), 1: 781–4. Jakobson, R. (1971), ‘The Language in Relation to Other Communication Systems’, in R. Jakobson (ed.), Selected Writings II: Word and Language, 697–708, The Hague: Mouton. Johnson-Laird, P. N. (1983), ‘Inference and Consciousness’, in P. J. Johnson-Laird (ed.), Mental Models: Toward a Cognitive Science of Language, 126–45, Cambridge: Harvard University Press. Johnson-Laird, P. N. (2006), How We Reason, Oxford: Oxford University Press. Jugand, J. (2016), Bibliographie sur les classifications, vocabulaires et ontologies. Available online: https://musica​.hypotheses​.org​/409 (accessed 9 November 2020). Kak, S. C. (1987), ‘The Paninian Approach to Natural Language Processing’, International Journal of Approximate Reasoning, 1: 117–30. Kapoor, B. and Savita, S. (2010), ‘A Comparative Study Ontology Building Tools for Semantic Web Applications’, International Journal of Web and Semantic Technology, 1: 1–13. Kintsch, W. and Van Dijk, T. A. (1975), ‘Comment on se rappelle et on résume des histoires’, Langages, 40: 98–116. Kitcher, P. (1993), The Advancement of Science, New York: Oxford University Press. Kleene, S. C. (1952), Introduction to Metamathematics, Amsterdam: North-Holland. Kneale, W. (1948), ‘Boole and the Revival of Logic’, Mind, 57 (226): 149–75. Kneale, W. and Kneale, M. (1988), The Development of Logic, New York: Oxford University. Knowlton, S. A. (2012), ‘Applying Sebeok’s Typology of Signs to the Study of Flags’, Raven, 19: 57–98. Available online: https://scholar​.princeton​.edu​/sites​/default​/files​/steven​.a​ .knowlton​/files​/applying​_sebeok​.pdf (accessed 20 June 2020). Knuuttila, T. (2011), ‘Modelling and Representing: An Artefactual Approach to Modelbased Representation’, Studies in History and Philosophy of Science, 42(2): 262–71. Kralemann, B. and Lattmann, C. (2013), ‘Models as Icons: Modeling Models in the Semiotic Framework of Peirce’s Theory of Signs’, Synthese, 190 (16): 3397–420.

236

References

Kress, G. R. (2010), Multimodality: A Social Semiotic Approach to Contemporary Communication. London and New York: Taylor & Francis. Kripke, S. (1972), Naming and Necessity, Oxford: Blackwell. Kuhn, T. S. (1993), ‘Metaphor in Science’, in A. Ortony (ed.), Metaphor and Thought, 409–19, Cambridge: Cambridge University Press. Kull, K. and Velmezova, E. (2014), ‘What Is the Main Challenge for Contemporary Semiotics’, Sign System Studies, 42 (4): 530–48. Kusch, M. (2002), Knowledge by Agreement: The Programme of Communitarian Epistemology, Oxford: Oxford University Press. Ladrière, J. (1957), Les limitations internes des formalismes, Paris: Gauthier–Villars. Lacombe, D. (1955), ‘Extension de la notion de fonction récursive aux fonctions d’une ou plusieurs variables réelles’, Comptes rendus hebdomadaires des séances de l'Académie des Sciences (Paris), 240: 2478–80, and 241: 13–14, 151–3. Lakoff, G. and Johnson, M. (2003), Metaphors We Live By, Chicago: University of Chicago Press. Lambek, J. (1989), ‘Grammar as Mathematics’, Canadian Mathematical Bulletin, 32 (3): 257–73. Lambek, J. and Lambek, M. (1981), ‘The Kinship Terminology of Malagasy Speakers in Mayotte’, Anthropological Linguistics, 23 (4): 154–82. Lambek, J. and Scott, P. J. (1988), Introduction to Higher Order Categorical Logic, Cambridge: Cambridge University Press. Langer, S. K. (1953), Feeling and Form, New York: Scribner. Langmead, A., Otis, J. M., Warren, C. N., Weingart, S. B. and Zilinksi, L. D. (2016), ‘Towards Interoperable Network Ontologies for the Digital Humanities’, International Journal of Humanities and Arts Computing, 10: 22–35. Latour, B. (1986), ‘Visualization and Cognition: Thinking With Eyes and Hands’, Knowledge and Society, 6: 1–40. Latour, B. and Woolgar, S. ([1979] 1986), Laboratory Life: The Construction of Scientific Facts, Princeton: Princeton University Press. Lebart, L. and Salem, A. (1994), Statistique textuelle, Paris: Dunod. Lebart, L., Pincemin, B. and Poudat, C. (2019), Analyse de données textuelles, Québec: Presses de l’Université du Québec. LeCun, Y., Bengio, Y. and Hinton, G. (2015), ‘Deep Learning’, Nature, 521: 436–44. Lenci, A. (2008), ‘Distributional Approaches in Linguistic and Cognitive Research’, Rivista di Linguistica, 20 (1): 1–31. Available online: http://linguistica​.sns​.it​/RdL​/20​.1​/ALenci​ .pdf (accessed 14 June 2020). Leone, M. (2018), ‘The Semiotics of the Face in the Digital Era’, Perspectives, 17: 27–29. Leonelli, S. (2007), ‘What Is in a Model? Combining Theoretical and Material Models to Develop Intelligible Theories’, in M. D. Laubichler and G. Müller (eds), Modeling Biology: Structures, Behavior, Evolution, 15–36, Cambridge, MA: MIT Press. Leplin, J. (1997), A Novel Defense of Scientific Realism, Oxford: Oxford University Press. Levesque, H. J. (1989), ‘A Knowledge-Level Account of Abduction’, IJCAI'89: Proceedings of the 11th International Joint Conference on Artificial Intelligence, 2: 1061–7. Levesque, H. J. and Brachman, R. J. (1987), ‘Expressiveness and Tractability in Knowledge Representation and Reasoning’, Computational Intelligence, 3 (3(issue 1)): 78–93. Lévi-Strauss, C. (1973), Anthropologie Structurale Deux, Paris: Plon. Levy, P. (1999), Collective Intelligence, Revised edn, Cambridge, MA: Basic Books. Lewis, D. ([1973] 1986), Counterfactuals, Oxford: Blackwell.



References

237

Li, F.-F., Fergus, R. and Peronal, P. (2000), ‘One-Shot Learning of Object Categories’, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28 (4): 594–611. Li, F., Liu, S.-Y. and Liu, Z. (2017), ‘A Review of Methods for Semantic Relation Extraction in Biomedical Field’, Library Tribune, 6: 10. Lieb, H. H. (1979), ‘Formal Semiotics vs Theoretical Semiotics’, in T. Borbé (ed.), Semiotic Unfolding, vol 1, Approaches to Semiotics, 175–80, Berlin: Mouton. Lieto, A. (2016), Computational Models (of Narrative) for Literary Studies. Available online: https://philarchive​.org​/rec​/LIECMO (accessed 12 June 2020). Lieto, A. and Damiano, R. (2014), ‘A Hybrid Representational Proposal for Narrative Concepts: A Case Study on Character Roles’, in M. A. Finlayson, J. C. Meister and E. G. Bruneau (eds), 5th Workshop on Computational Models of Narrative, 41: 106–15. Lincoln Logarithms Project: Finding Meaning in Sermons. Available online: http:// lincolnlogs​.digitalscholarship​.emory​.edu/ (accessed 12 June 2020). Lindenmayer, A. (1968), ‘Mathematical Models for Cellular Interaction in Development, Parts I and II’, Journal of Theoretical Biology, 18: 280–299 and 300–315. Lippi, M. and Torroni, P. (2016), ‘Argumentation Mining: State of the Art and Emerging Trends’, ACM Transactions on Internet Technology (TOIT), article 10, 16 (2). List, C. and Pettit, P. (2002), ‘Aggregating Sets of Judgements: An Impossibility Result’, Economics and Philosophy, 18 (1): 89–110, Cambridge: Cambridge University Press. Available online: https://www​.researchgate​.net​/publication​/30520934​_Aggregating​ _Sets​_of​_Judgments​_An​_Impossibility​_Result (accessed 14 June 2020). Liu, K. (2005), Semiotics in Information Systems Engineering, Cambridge: Cambridge University Press. Liu, K. and Tan, C. (2015), ‘Semiotics in Digital Visualisation’, in J. Cordeiro, S. Hammoudi, L. Maciaszek, O. Camp and J. Filipe (eds), Enterprise Information Systems: 16th International Conference, ICEIS 2014, Lisbonne, Portugal, April 27–30, 2014. Revised Selected Papers, 3–13, New York: Springer. Liu, Weijie, P. Zhou, Z. Zhao, Z.Wang, Q.Ju, H. Deng, and P. Wang. (2020),‘K-BERT: Enabling Language Representation with Knowledge Graph.’ In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) 2901–8. NewYork, AAAI Press, Palo Alto. Lobo, M., Open Mathematics Knowledge Base, Research Gate. Available online: https://osf​ .io​/evq8a/ (accessed 12 November 2020). Loff, B. and Costa, J. (2009), ‘Five Views of Hypercomputation’, International Journal of Unconventional Computing, 5: 193–207. Longhi, J. (2015), ‘La Théorie des objets discursifs: Concepts, méthodes, contributions’, HDR Thesis, Cergy-Pontoise University, Cergy, France. Lotman, Y. M. (1972), Analysis of the Poetic Text, Ann Arbor: University of Michigan Press. Loula, A., Gudwin, R. and Queiroz, J. (2004), ‘Symbolic Communication in Artificial Creatures: An Experiment in Artificial Life’, in A. L. C. Bazzan and S. Labidi (eds), Advances in Artificial Intelligence – SBIA 2004: 17th Brazilian Symposium on Artificial Intelligence, Sao Luis, Maranhao, Brazil, September 29-October 1, 2004. Proceedings, 336–45, Berlin: Springer. Lovejoy, M. (2009), Digital Currents: Art in the Electronic Age. London and NewYork: Routledge. Luhmann, N. (1995), Social Systems, trans. J. Bednarz and D. Baecker, Stanford: Stanford University Press.

238

References

Lycan, W. (1988), ‘Commentaries on Smolensky Papers: On the Proper Treatment of Connectionism’, The Behavioural and Brain Sciences, 2: 1–74. MacKay, D. G. and Miller, M. D. (1994), ‘Semantic Blindness: Repeated Concepts Are Difficult to Encode and Recall under Time Pressure’, Psychological Science, 5 (1): 52–5. Magnani, L., Nersessian, N. J. and Thagard, P., eds (1999), Model-Based Reasoning in Scientific Discovery, 41–57, New York: Plenum Publisher. Mahoney, M. S. (2010) ‘The Structures of Computation and the Mathematical Structure of Nature’, The Rutherford Journal, 3. Available online: http://rutherfordjournal​.org​/ article030107​.html (accessed 6 January 2021). Majumder, G., Pakray, P., Gelbukh, A. and Pinto, D. (2016), ‘Semantic Textual Similarity Methods, Tools, and Applications: A Survey’, Computación y Sistemas, 20 (4): 647–65. Manning, C. D. and Schütze, H. (1999), Foundations of Statistical Natural Language Processing, Cambridge: MIT Press. Manovich, L. (2002), The Language of New Media, Cambridge, MA: MIT Press. Manovich, L. (2016), ‘The Science of Culture? Social Computing, Digital Humanities and Cultural Analytics’, Journal of Cultural Analytics Available on https://culturalanalytics​ .org​/article​/11060 (accessed 08 February 2021). Manovich, L. (2018), Cultural Analytics. Cambridge, MA: MIT Press. Marinelli, R. and Spadoni, G. (2006), ‘Some Considerations in Structuring a Terminological Knowledge Base’, in Proceedings of the Third International WordNet Conference, GWC: Seoul. Markov, A. A. ([1954] 1960), ‘The Theory of Algorithms’, American Mathematical Society Translations, 2 (15): 1–14. Marr, B. (1982), Vision: A Computational Investigation into the Human Representation and Processing of Visual Information, San Francisco: W. H. Freeman. Marr, B. (2019), ‘What Is Biological Computing and How It Will Change Our World’. Available online: https://www​.linkedin​.com​/pulse​/what​-biological​-computing​-how​ -change​-our​-world​-bernard​-marr/ (accessed 21 October 2020). Martinelli, D. (2010), A Critical Companion to Zoosemiotics: People, Paths, Ideas, Berlin: Springer. Martinez-Rodriguez, J. L., Lopez-Arevalo, I., Rios-Alvarado, A. B., Hernandez, J. and Aldana-Bobadilla, E. (2019), ‘Extraction of RDF Statements from Text’, in B. VillazónTerrazas and Y. Hidalgo-Delgado (eds), Knowledge Graphs and Semantic Web, 87–101, Communications in Computer and Information Science, Cham: Springer International Publishing. Marty, R. (1990), L’algèbre des signes: essai de sémiotique scientifique d’après Charles S. Peirce, Amsterdam: John Benjamins Publishing Company. Matiyasevich, Y. V. ([1970] 1993), Hilbert’s 10th Problem, Cambridge: MIT Press. Maturana, H. R. and Varela, F. J. (1992), The Tree of Knowledge: The Biological Roots of Human Understanding, 2nd edn, Boston; New York: Shambhala. McCarty, W. (2004), ‘Modeling: A Study in Words and Meanings’, in S. Schreibman, R. Siemens and J. Unsworth (eds), A Companion to Digital Humanities, 254–70, Oxford: Blackwell. McClelland, J. L. and Rumelhart, D. E. (1989), Explorations in Parallel Distributed Processing: A Handbook of Models, Programs, and Exercises, Cambridge: MIT Press. Mello, R. F. and Ponti, M. A. (2018), Machine Learning: A Practical Approach on the Statistical Learning Theory, London: Springer. Merrell, F. (2004), ‘Abduction Is Never Alone’, Semiotica, 148: 245–75.



References

239

Meunier, J.-G. and Forest, D. (2009), ‘Lecture et analyse conceptuelle assistée par ordinateur: Premières expériences’, in J.-P. Desclés and F. Le Priol (eds), Annotations automatiques et recherche d’information, 211–30, Paris: Hermès. Meunier, J.-G. (1989), ‘Artificial Intelligence and the Theory of Signs’, Semiotica, 77 (1): 43–63. Meunier, J.-G. (1994), ‘Les fonctions cognitives du tableau’, Recherches sémiotiques/ Semiotic Inquiry RSSI, 14 (1–2): 67–93. Meunier, J.-G. (1998), ‘Categorial Structure of Iconic Languages’, Theory and Psychology, 8 (6): 805–25. Meunier, J.-G. (2009), ‘CARAT–Computer-Assisted Reading and Analysis of Texts: The Appropriation of a Technology’, Digital Studies/Le champ numérique, 1 (3). Available online: https://www​.digitalstudies​.org​/articles​/10​.16995​/dscn​.263/ (accessed 12 June 2020). Meunier, J.-G. (2014), ‘Humanités numériques ou computationnelles: enjeux herméneutiques’, Sens public. Available online: https://www​.erudit​.org​/en​/journals​/sp​ /2014​-sp03456​/1043651ar/ (accessed 12 June 2020). Meunier, J.-G. (2018), ‘Le Texte Numérique : Enjeux Herméneutiques’, Digital Humanities Quarterly, 12 (1). Available online: http://www​.digitalhumanities​.org​/dhq​/vol​/12​/1​ /000362​/000362​.html (accessed 12 June 2020). Meunier, J.-G. (2019), ‘Enjeux de la modélisation formelle en sémiotique computationnelle’, Cygne noir, 7. Available online: http://revuecygnenoir​.org​/numero​ /article​/meunier​-modelisation​-semiotique​-computationnelle (accessed 3 November 2020). Meunier, J.-G. and Forest, D. (2008), ‘Computer Assisted Conceptual Analysis (CACAT): The Concept of Mind in The Collected Papers of C. S. Peirce’, Proceedings of Digital Humanities Conference 2008, Oulo, Finland, 74–80. Available online: https://archipel​ .uqam​.ca​/6940​/1​/pdf​%20to​%20upload​.pdf (accessed 12 June 2020). Meystel, A. M. and Albus, J. S. (2001), Intelligent Systems: Architecture, Design, and Control, Hoboken: John Wiley & Sons (Wiley-Interscience Publication). Michalski, R. J., Carbonel, J. and Mitchell, T. M., eds (1981), Machine Learning: An Artificial Intelligence Approach, Berlin: Springer Verlag. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S. and Dean, J. (2013a), ‘Distributed Representations of Words and Phrases and Their Compositionality’, in  C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani and K. Q. Weinberger (eds), Proceedings of the 26th Conference on Advances in Neural Information Processing Systems, 3111–3119. (NIPS), Lake Tahoe: Curran Associates Inc. https://arxiv​.org​/abs​/1310​.4546 Mikolov, T., Yih, W. T. and Zweig, G. (2013b), ‘Linguistic Regularities in Continuous Space Word Representations’, in Proceedings of the 2013 Annual Conference of the NAACLHLT, 746–51, Stroudsburg: Association for Computational Linguistics. Available online: https://www​.aclweb​.org​/anthology​/N13​-1090​.pdf (accessed 12 June 2020). Miller, G. A. (1995), ‘WordNet: A Lexical Database for English’, Communications of the ACM, 38 (11): 39–41. Miller, V. (2012), Understanding Digital Culture. SAGE. Mines Technology Museum, Colorado. https://hdl​.handle​.net​/11124​/172469 Minsky, M. (1975), A Framework for Representing Knowledge, in P. H. Winston (ed.), The Psychology of Computer Vision, 211–77, New York: McGraw-Hill. Minsky, M. (1977), ‘Frame Theory’, in P. N. Johnson-Laird and P. C. Wason (eds), Thinking: Readings in Cognitive Science, 355–76, Cambridge: Cambridge University Press.

240

References

Minsky, M. (1986), The Society of Mind, New York: Simon & Schuster. Mitchell, M. (2018), ‘Artificial Intelligence Hits the Barrier of Meaning’, New York Times, 5 November. Mitchell, T. M. (1997), Machine Learning, New York: McGraw Hill. Mitchell, T. M. (1999), ‘Machine Learning and Data Mining’, Communications of the ACM, 42, 30–36. Mitchell, T., Cohen, W., Hruschka, E., Talukdar, P., Yang, B., Betteridge, J., Carlson, A., Dalvi, B., Gardner, M., Kisiel, B., Krishnamurthy, J., Lao, N., Mazaitis, K., Mohamed, T., Nakashole, N., Platanios, E., Ritter, A., Samadi, M., Settles, B., Wang, R., Wijaya, D., Gupta, A., Chen, X., Saparov, A., Greaves, M., Welling, J. (2018), ‘Never-ending Learning’, Communications of the ACM, 61, 103–15. Available online: https://cacm​.acm​ .org​/magazines​/2018​/5​/227193​-never​-ending​-learning​/fulltext (accessed 12 June 2020). Mongeau, P. and Saint-Charles, J. (2014), ‘Réseaux sociaux et réseaux sociosémantiques et phénomènes de communication’, Communiquer. Revue de communication sociale et publique, 12: 1–7. Montague, R. (1974), Formal Philosophy: Selected Papers of Richard Montague, R. H. Thomason (ed.), New Haven: Yale University Press. Mooney, R. J. (2000), ‘Integrating Abduction and Induction in Machine Learning’, in P. A. Flach and A. C. Kakas (eds), Abduction and Induction. Applied Logic Series, vol. 18: 181–91, Dordrecht: Springer. Moretti, F. (2013), Distant Reading, London: Verso. Morgan, M. S. and Morrison, M., eds (1999), Models as Mediators: Perspectives on Natural and Social Science, Cambridge: Cambridge University Press. Morris, C. W. (1938), Foundations of the Theory of Signs, Chicago: The University of Chicago Press. Morrison, M. (2009), ‘Fictions, Representations, and Reality’, in M. Suárez (ed.), Fictions in Science: Philosophical Essays on Modeling and Idealization, 110–35, London: Routledge. Muchielli, A. (2007), ‘Processus intellectuels fondamentaux sous-jacents aux techniques et méthodes qualitatives’. Recherches qualitatives, Hors-série, 3: 1–27. Muggleton, S. (1999), ‘Inductive Logic Programming: Issues, Results and the Challenge of Learning Language in Logic’, Artificial Intelligence, 114 (1–2): 283–96. Muindi, F. J., Ramachandran, L. and Tsai, J. W. (2020), ‘Human Narratives in Science: The Power of Storytelling’, Trends in Molecular Medicine, 26: 249–51. Murphy, K. P. (2012), Machine Learning: A Probabilistic Perspective, illustrated edition, Cambridge, MA: The MIT Press. Music Encoding Initiative (MEI). https://music​-encoding​.org/ Mylopoulos, J., Rose, T. and Woo, C. (1992), ‘Task Oriented Development of Intelligent Systems’, in P. Loucopoulos and R. Zicari (eds), Conceptual Modeling, Databases, and CASE: An Integrated View of Information Systems Development, New York: John Wiley & Sons. Mylopoulos, J. (1992), ‘Conceptual Modeling and Telos’, in P. Loucopoulos and R. Zicari (eds), Conceptual Modeling, Databases, and CASE: An Integrated View of Information Systems Development, New York: John Wiley & Sons. Nadin, M. (2007), ‘Semiotic Machine’, The Public Journal of Semiotics, 1 (1): 57–75. Nadin, M. (2011a), ‘Information and Semiotic Processes: The Semiotics of Computation’, Cybernetics & Human Knowing, 18 (1–2): 153–75. Nadin, M. (2011b), ‘Computation, Information, Meaning. Anticipation and Games’, International Journal of Applied Research on Information Technology and Computing, 2 (1): 1–27.



References

241

Nagel, E. (1961), The Structure of Science: Problems in the Logic of Scientific Explanation, New York: Harcourt, Brace & World. Nagoudi, M. B., Ferrero, J. and Schwab, D. (2017), ‘Amélioration de la similarité sémantique vectorielle par méthodes non-supervisées’, in 24e Conférence sur le Traitement Automatique des Langues Naturelles (TALN 2017), Orléans, France. https:// hal​.archives​-ouvertes​.fr​/hal​-01531886 (accessed 12 June 2020). NELL Project, Never-Ending Language Learning. (2020). In Wikipedia accessed: 20 December 2020 https://en​.wikipedia​.org​/w​/index​.php​?title​=Never Ending_ Language_Learning&oldid=996010659 Nersessian, N. J. (1992), ‘How Do Scientists Think? Capturing the Dynamics of Conceptual Change in Science’, in R. N. Giere (ed.), Cognitive Models of Science, 5–22, Minneapolis: University of Minnesota Press. Newell, A. (1970), ‘Remarks on the Relationship between Artificial Intelligence and Cognitive Psychology’, in R. Banerji and M. D. Mesarovic (eds), Theoretical Approaches to Non-numerical Problem Solving, 363–400, New York: Springer-Verlag. Newell, A. (1980), ‘Physical Symbol Systems’, Cognitive Science, 4 (2): 135–83. Newell, A. (1982), ‘The Knowledge Level’, Artificial Intelligence, 18 (1): 87–127. Newell, A. (1983), ‘Intellectual Issues in the History of Artificial Intelligence’, in F. Machlup and U. Mansfield (eds), The study of Information: Interdisciplinary Messages, 187–294, New York: John Wiley & Sons. Newell, A. and Simon, H. (1972), Human Problem Solving, Englewood Cliffs: Prentice Hall. Newell, A. and Simon, H. (1976), ‘Symbol Manipulation’, in E. D. Reilly, A. Ralston and D. Hemmendinger (eds), Encyclopedia of Computer Science, New York: Petrocelli/Charter. Newell, A. (1986), ‘The Symbol Level and the Knowledge Level’, in W. Demopoulos and Z. W. Pylyshyn (eds), Meaning and Cognitive Structure. Issues in the Computational Theory of Mind, Norwood: Ablex Publishing Corporation. Newton and Alchemy: The Chymistry of Isaac Newton Project. Available online: https:// webapp1​.dlib​.indiana​.edu​/newton​/project​/about​.do (accessed 3 October 2020). Newton, I. Proposition 1. From the Newton Project Catalogue, Keynes MS. 12. Published June 2010, Indiana University, Bloomington. Available online: https://webapp1​.dlib​ .indiana​.edu​/newton​/search​?field1​=text​&text1​=Proposition​+1​&btnG​.x​=0​&btnG​.y=0 (accessed 5 January 2020). Niehaus, J., Romero, V., Koelle, D., Palmon, N., Bracken, B., Pfautz, J., Weyhrauch, P. (2014), ‘A Flexible Framework for the Creation of Narrative-Centered Tools’, in M. A. Finlayson, J. C. Meister and E. G. Bruneau (eds), 2014 Workshop on Computational Models of Narrative, 41: 130–38. Nielson, H. R. and Nielson, F. (1992), Semantics With Applications: A Formal Introduction, Hoboken: John Wiley & Sons Inc. Niiniluoto, I. (1988), ‘Analogy and Similarity in Scientific Reasoning’, in D. H. Helman (ed.), Analogical Reasoning: Perspectives of Artificial Intelligence, Cognitive Science, and Philosophy, 271–98, Dordrecht: Springer. Nofre, D., Priestley, M. and Alberts, G. (2014), ‘When Technology Became Language: The Origins of the Linguistic Conception of Computer Programming, 1950–1960’, Technology and Culture, 55 (1): 40–75. Nonaka, I. and Takeuchi, H. (1995), The Knowledge Creating Company, Oxford: Oxford University Press. Nöth, W. (1997), ‘Representation in Semiotics and in Computer Science’, Semiotica, 115 (3–4): 203–13.

242

References

Nöth, W. (2012), ‘Charles S. Peirce’s Theory of Information: A Theory of the Growth of Symbols and of Knowledge’, Cybernetics and Human Knowing, 19 (1–2): 137–61. Nowakowska, M. (1981), ‘Formal Semiotics and Multidimensional Semiotic Systems’, Cybernetics and System 12 (1–2): 83–102. O’Halloran, K. L., Tans, S., Smith, B. and Podlasov, A. (2009), ‘Digital Semiotics’, Proceedings 10th IASS-AIS World Congress of Semiotics, A Coruña Spain. 22–26 September 2009. O’Halloran, K., Tan, S., Smith, B. and Podlasov, A. (2009). Digital semiotics. 10th World Congress of Semiotics.Ontotext. ‘Linked Data Integration for Libraries, Archives and Museums’. https://www​.ontotext​.com​/knowledgehub​/case​-studies​/linked​-data​ -integration​-galleries​-libraries​-archives​-museums/ (accessed 8 November 2020). Ogden, C. K. and Richard, I. A. (1923), The Meaning of Meaning, London: Routledge and Kegan Paul. Ortony, A., ed. (1993), Metaphor and Thought, 2nd edn, Cambridge: Cambridge University Press. Padian, K. (2018), ‘Narrative and “Anti-Narrative” in Science: How Scientists Tell Stories, and Don’t’, Integrative and Comparative Biology, 58 (6): 1224–34. Panchenko, A. and Morozova, O. (2012), ‘A Study of Hybrid Similarity Measures for Semantic Relation Extraction’, in Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, 10–18, Avignon, France, April 23 Association for Computational Linguistics, Stroudsburg. Papineau, D. (2010), ‘Realism, Ramsey Sentences and the Pessimistic Meta-Induction’, Studies in History and Philosophy of Science, 41 (4): 375–38. Pascal, B. (1779), Œuvres de Blaise Pascal, C. Bossut (ed.), La Haye: Detune. Pasch, G. (1983), ‘Semiotic Vexillology: The Logical Structure of Flags’, The Flag Bulletin, 22 (3–4): 141–57. Passant, A. and Raimond, Y. (2008), ‘Combining Social Music and Semantic Web for Music-related Recommender Systems’, in Proceedings of the First Social Data on the Web Workshop, Karlsruhe, Germany, CEUR Workshop Proceedings. Available online: CEUR​-WS​.org​/Vol​-405​/pape​r3​.pdf Paul, G. (1993), ‘Approaches to Abductive Reasoning: An Overview’, Artificial Intelligence Review, 7 (2): 109–52. Pavel, T. (1976), ‘“Possible worlds” in Literary Semantics’, The Journal of Aesthetics and Art Criticism, 34 (2): 165–76. Peirce, C. S. (1873), MS 214, On the nature of Sign. In R. Robin, 281: Writings 3: 66–68, Winter-Spring. Available online: https://arisbe​.sitehost​.iu​.edu​/menu​/library​/bycsp​/ logic​/ms214​.htm Peirce, C. S. (1904 [c.]), New Elements (Kaina stoiceia), MS [R] 517, ref: Index¸ (pub. 05. 05. 13-10:34), quoted in M. Bergman and S. Paavola (eds), The Commens Dictionary: Peirce's Terms in His Own Words. New Edition. Available online: http://www​.commens​ .org​/dictionary​/entry​/quote​-new​-elements​-kaina​-stoiceia-5 (accessed 12 April 2020). Peirce, C. S. (1931…), Collected Papers, Cambridge: Harvard University Press. Peirce, C. S. (1906 [c.]), On Signs [R]. MS [R] 793, in Robin, R. (1967), Annotated Catalogue of Papers by Charles S. Peirce. The University of Massachusetts Press. Pendergraft, E. P. (1993), ‘The Future’s Voice – Intelligence Based on Pragmatic Logic’, Technical Report – Creative Intelligence Incorporated, DRAFT (FATE), Jasper, EUA. Penrose, R. (1989), The Emperor’s New Mind, Oxford: Oxford University Press. Perlovsky, L. I. (1998), ‘Semiotics, Mind and Architecture of Target Tracker’, Proceedings of the IEEE ISIC/CIRA/ISAS Joint Conference, Gaithersburg, MD. https://www​



References

243

.researchgate​.net​/publication​/3766852​_Semiotics​_mind​_and​_architecture​_of​_target​ _tracker (accessed 12 June 2020). Perlovsky, L. I. (2000), Neural Networks and Intellect: Using Model-Based Concepts, New York: Oxford University Press. Perseus Digital Library, http://www​.perseus​.tufts​.edu​/hopper/ Petitot, J. (1977), ‘Topologie du carré sémiotique’, Études littéraires, 10 (3): 347–426. Petitot, J. (1985a), Les catastrophes de la parole, Paris: Maloine. Petitot, J. (1985b), Morphogénèse du sens, Paris: Presses universitaires de France. Petitot, J. (1988), ‘Approche morphodynamique de la formule canonique du mythe’, L’Homme, 28: 24–50. Petitot, J. (2009), Morphogéométrie de la vision. Modèles mathématiques et physiques des architectures fonctionnelles, Paris: École Polytechnique. Petitot, J. (2018), ‘The Formalization of Elementary Semiotic Structures’, in D. Compagno (ed.), Quantitative Semiotic Analysis. Lecture Notes in Morphogenesis, 33–54, New York: Springer. Petöfi J. S. (1978), ‘A Formal Semiotic Text Theory as an Integrated Theory of Natural Language’, in W. U. Dressler (ed.), Current Trends in Textlinguistics, 35–46, Berlin: de Gruyter. Pietarinen, A. (2003), ‘The Semantic + Pragmatic Web = The Semiotic Web’, Conference: Proceedings of the IADIS International Conference WWW/Internet 2003, ICWI 2003, Algarve, Portugal, November 5–8. Pincemin, B. (2011), ‘Sémantique Interprétative et Textométrie - Version Abrégée’, Corpus, 10: 259–69. Piotrowski, D. (2017), Morphogenesis of Signs, Berlin: Springer. Piotrowski, D. and Visetti, Y.-M. (2014), ‘Connaissance sémiotique et mathématisation. Sémiogenèse et explicitation’, Versus - Quaderni di Studi Semiotici, 118: 141–70. Pichler, A. and Zöllner-Weber, A. (2012), ‘Towards Wittgenstein on the Semantic Web’, Digital Humanities, Conference Abstracts, 318–21, Hamburg: Hamburg University Press. Poincaré, H. (1892), Les méthodes nouvelles de la mécanique céleste, tome 1, Paris: Gauthier-Villars. Polanyi, M. (1966), The Tacit Dimension, Chicago: University of Chicago Press. Posner, R. (2014), ‘New Semiotics: Between Tradition and Innovation’, 12th World Congress of Semiotics, Sofia 2014, 16–20 September, New Bulgarian University. Available online: http://semio2014​.org​/en​/ways​-of​-understanding​-semiotics​-and​ -hermeneutics (accessed 20 June 2020). Posner, R., Robering, K. and Sebeok, T. A. (1997), Semiotik/Semiotics. Ein Handbuck zu den Zeichen Natur and Kultur/A Handbook on The Sign-Theoretic Foundations of Nature and Culture, Berlin: de Gruyter. Post, E. L. (1936), ‘Finite Combinatory Processes – Formulation’, Journal of Symbolic Logic 1, 103–5; reprinted in Davis, M. ([1965] 2004), The Undecidable: Basic Papers on Undecidable Propositions, Unsolvable Problems and Computable Functions, 288–91, Dover Publications. Pouyanfar, S., Yang, Y., Chen, S.-C., Shyu, M.-L. and Iyengar, S. S. (2018), ‘Multimedia Big Data Analytics: A Survey’, ACM Computing Surveys, 51 (1): 1–34. Prensky, M. (2009), ‘H. Sapiens Digital: From Digital Immigrants and Digital Natives to Digital Wisdom’, Innovate: Journal of Online Education, 5 (3). Available online: https:// www​.learntechlib​.org​/p​/104264/ (accessed 20 June 2020). Prieto, L. (1972), Messages et signaux, Paris: Presses Universitaires de France.

244

References

Proffitt, M., ed. (2018), Leveraging Wikipedia: Connecting Communities of Knowledge, Chicago: ALA Editions. WikiProject Cultural heritage—Wikidata. Available online: https://www​.wikidata​.org​/wiki​/ Wikidata​:WikiProject​_Cultural​_heritage (accessed 5 October 2020). Pulizzotto, D. (2019), ‘La sémiotique computationnelle entre narrativité et apprentissage automatique. Une démonstration de la faisabilité avec un corpus journalistique à propos du printemps érable’, Thèse, Faculté des arts, Université du Québec à Montréal, Montréal. Pulizzotto, D., López, J. A., Chartier, J.-F., Meunier, J.-G., Chartrand, L., Laureau, F. and Le, N. T. (2016), ‘Recherche de “périsegments” dans un contexte d’analyse conceptuelle assistée par ordinateur: le concept “d’esprit” chez Peirce’, Actes de la conférence conjointe JEP-TALN-RECITAL 2016, 2. Pylyshyn, Z. W. (1983), ‘Information Science: Its Roots and Relation as Viewed From the Perspective of Cognitive Science’, in F. Machlup and U. Mansfield (eds), The Study of Information: Interdisciplinary Message, 63–118, New York: John Wiley & Sons. Pylyshyn, Z. W. (1984), Computation and Cognition: Toward a Foundation for Cognitive Science, Cambridge: MIT Press. Quillian, M. R. (1968), ‘Semantic Memory’, in M. Minsky (ed.), Semantic Information Theory, 227–71, Cambridge: MIT Press. Quine, W. O., (1960), Word and Object, Cambridge: MIT Press. Rambidi, N. G, Kuular, T. O. and Makhaeva, E. E. (1998), ‘Information-Processing Capabilities of Chemical Reaction-Diffusion Systems Belousov-Zhabotinsky Media in Hydrogel Matrices and on Solid Supports’, Advanced Materials for Optics and Electronics, 8(4): 163–71. Rashid, S. M., De Roure, D. and McGuinness, D. L. (2018), ‘A Music Theory Ontology’, Proceedings of the 1st International Workshop on Semantic Applications for Audio and Music - SAAM ’18, 6–14, Monterey, CA, USA: ACM Press. Rastier, F. (2004), ‘Ontologie(s)’, Revue des sciences et technologies de l’information, série: Revue d’Intelligence artificielle, 18 (1): 15–40. Rastier, F. (2009), Sémantique interprétative, Paris: Presses Universitaires de France. Rastier, F. (2011), La mesure et le grain: Sémantique de corpus, Paris: Honoré Champion. Rastier, F. (2018), ‘Computer-Assisted Interpretation of Semiotic Corpora’, Quantitative Semiotic Analysis, 123–39, London: Springer. Rastier, F., Cavazza, M. and Abeillé, A. (1994), Sémantique pour l’analyse: de la linguistique à l’informatique, Paris: Elsevier Masson. Rebecchi, L., Guidetti, R., Borsari, S., Altiero, T. and Bertolani, R. (2006), ‘Dynamics of Long-Term Anhydrobiotic Survival of Lichen-Dwelling Tardigrades’, Hydrobiologia, 558 (1): 23–30. Reichenbach, H. (1938), Experience and Prediction: An Analysis of the Foundations and the Structure of Knowledge, Chicago: The University of Chicago Press. Reyes, E. and Sonesson, G. (2019), ‘New Approaches to Plastic Language: Prolegomena to a Computer-Aided Approach to Pictorial Semiotics’, Semiotica, 230: 71–95. Rheinberger, H. (1997), Towards a History of Epistemic Things: Synthesizing Proteins in the Test Tube, Stanford: Stanford University Press. Rheinberger, H. (2007), ‘Experimental Model Systems: An Epistemological Apercu from the Perspective of Molecular Biology’, in M. D. Laubichler and G. B. Müller (eds), Modeling Biology. Structure, Behaviors, Evolution, 37–46, Cambridge: MIT Press. Rheinberger, H. (2009), ‘Recent Science and Its Exploration: The Case of Molecular Biology’, Studies in History and Philosophy of Biological and Biomedical Sciences, 40 (1): 6–12.



References

245

Rice, H. G. (1953), ‘Classes of Recursively Enumerable Sets and Their Decision Problems’, Transactions of the AMS, 74: 358–66. Ricoeur, P. (1970), ‘Qu’est-ce qu'un texte? Expliquer et Comprendre’, in R. Burner, K. Cramer and R. Wieh (eds), Hermeneutik und Dialektik, 181–200, Tübingen: Mohr. Rieger, B. B. (1998), ‘A Systems Theoretical View on Computational Semiotics: Modeling Text Understanding as Meaning Constitution by SCIPS’, Proceedings of the Joint IEEE Conference on the Science and Technology of Intelligent Systems (ISIC/CIRA/ISAS-98), 840–845, Piscataway: IEEE/Omnipress. Rieger, B. B. (1999a), ‘Computing Fuzzy Semantic Granules from Natural Language Texts: A Computational Semiotics Approach to Understanding Word Meanings’, Proceedings of the IASTED International Conference. Artificial Intelligence and Soft Computing, 9–12 August. Available online: https://www​.uni​-trier​.de​/fileadmin​/fb2​/LDV​/Rieger​/ Publikationen​/Aufsaetze​/99​/iasted99​.pdf (accessed 11 June 2020). Rieger, B. B. (1999b), ‘Semiotics and Computational Linguistics: On Semiotic Cognitive Information Processing’, in L. A. Zadeh (ed.), Computing with Words in Information/ Intelligent Systems 1: Foundations, 93–118, Heidelberg: Physica-Verlag. Rieger, B. B. (2003) ‘Semiotic Cognitive Information Processing: Learning to Understand Discourse. A Systemic Model of Meaning Constitution’, in R. Kühn, R. Menzel, W. Menzel, U. Ratsch, M. M. Richter and I.-O. Stamatescu (eds), Adaptivity and Learning: An Interdisciplinary Debate, 347–403, Berlin, Heidelberg: Springer. Robinson, S., Arbez, G., Birta, L. G., Tolk, A. and Wagner, G. (2015), ‘Conceptual Modeling: Definition, Purpose and Benefits’, in 2015 Winter Simulation Conference (WSC), 2812–26, Piscataway: IEEE. Available online: https://ieeexplore​.ieee​.org​/ document​/7408386 (accessed 11 June 2020). Rogers, H. Jr. (1987), Theory of Recursive Functions and Effective Computability, Cambridge: MIT Press. Rompré, L. (2015), Niveau d’abstraction dans l’exploration de données musicales, Thèse Ph.D., Université du Québec à Montréal. Rompré, L., Biskri, I. and Meunier, J.-G. (2017), ‘Using Association Rules Mining for Retrieving Genre-Specific Music Files’, Computer Science FLAIRS Conference 2017, Marco Island. Rorty, R. (1979), Philosophy and the Mirror of Science, Princeton: Princeton University Press. Roth, C. (2005), ‘Generalized Preferential Attachment: Towards Realistic Social Network Models’. Available online: https://arxiv​.org​/abs​/nlin​/0507021 (accessed 12 June 2020). Rothenberg, J. (1989), ‘The Nature of Modeling’, in Artificial Intelligence, Simulation, and Modeling, 75–92, New York: Wiley & Sons. Rozenberg, G., Bäck, T. and Kok, J. N., eds (2012), Handbook of Natural Computing, Berlin: Springer-Verlag. Rumbaugh, J., Blaha, M., Premerlani, W., Eddy, F., and Lorensen, W. E. (1991). Object Oriented Modeling and Design, Englewood Cliffs: Prentice Hall. Rumelhart, D. E., Hinton, G. E. and McClelland, J. L., eds (1986), ‘A General Framework for Parallel Distributed Processing’, in D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing, vol. 1: Explorations in the Microstructure of Cognition: Foundations, 47–76, Cambridge, MA: MIT Press. Rumelhart, D. E., McClelland, J. L. and the PDP Research Group (1986), ‘Parallel Distributed Processing: Explorations in the Microstructure of Cognition’, Computational Models of Cognition and Perception, vols 1–2, Cambridge: MIT Press.

246

References

Saal, F. (1988), Studies in Indian Logic and Linguistics, Chicago: The University of Chicago Press. Sahle, P. (2013), ‘Digitale Editionsformen. Zum Umgang mit der Überlieferung unter den Bedingungen des Medienwandels. Teil 3: Textbegriffe und Recodierung’, PhD Thesis, Universität zu Köln. Sandaker, B. (2010), ‘An Ontology of Structures Space’, in P. J. S. Cruz (ed.), Structures and Architecture, 11–14, Leiden: CRC Press, ISI proceedings. Saussure, F. de ([1916] 1990), Cours de linguistique générale, Paris: Payot. Translation: Saussure, F. (1959), Course in General Linguistics, New York: Philosophical Library. Scardovelli, M. W. (2018), ‘Étude sémiotique sur la latéralisation des figures animales dans l’art pariétal du Paléolitique en France’, PhD diss., Faculté des arts, Université du Québec à Montréal, Montréal. Available online: https://archipel​.uqam​.ca​/11208​/1​/ D3379​.pdf (accessed 12 June 2020). Schank, R. C. (1975), Conceptual Information Processing, Amsterdam: North Holland Pub. Schank, R. and Abelson, R. P. (1977), Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures, Hillsdale: Lawrence Erlbaum Associates. Gallagher, S. and Schmicking, D., eds (2009), Handbook of Phenomenology and Cognitive Sciences, 369–99, Dordrecht and New York: Springer. Schönfinkel, M. ([1924] 1967), ‘On the Building Blocks of Mathematical Logic’, in J. van Heijenoort (ed.), From Frege to Gödel: A Source Book in Mathematical Logic, 1879– 1931, 355–66, Harvard: Harvard University Press. Scott, D. and Strachey, C. (1971), ‘Toward a Mathematical Semantics for Computer Languages’. Available online: https://www​.cs​.ox​.ac​.uk​/files​/3228​/PRG06​.pdf (accessed 1 December 2020). Searle, J. (1980), ‘Minds, Brains, and Programs’, Behavioral and Brain Sciences, 3: 417–57. Sebastiani, F. (2002), ‘Machine Learning in Automated Text Categorization’, ACM Computing Surveys, 34 (1): 1–47. Sebeok, T. A. (1968), ‘Zoosemiotics’, American Speech, 43 (2): 142–4. Sebeok, T. A. (1976), Contributions to the Doctrine of Sign, Bloomington: Indiana University. Sebeok, T. A. (1979), The Sign and Its Masters, Austin: University of Texas Press. Sellars, W. (1956), ‘Empiricism and the Philosophy of Mind’, in H. Feigl and M. Scriven (eds), Minnesota Studies in the Philosophy of Science, 1 (19): 253–329. Serson, B. (1994), ‘Semiotics and Cognitive Science: Peirce’s Theory of Representation and Inference and the Connectionism Versus Classical Cognitivist Debate’, Paper presented at the IASS 5th Congress, Berkeley, CA. Shagrir, O. (2002), ‘Effective Computation by Humans and Machines’, Minds and Machines, 12 (2): 221–40. Shakespeare’s Ontology – Research – Royal Holloway, University of London. Available online: https://pure​.royalholloway​.ac​.uk​/portal​/en​/publications​/shakespeares​ -ontology(0ede1117-8332-4a81-9843-c7200b951bca).html (accessed 8 November 2020). Shanin, T. (ed.) ([1972] 2013), The Rules of the Game: Interdisciplinarity, Transdisciplinarity and Analytical Models in Scholarly Thought, London: Routledge. Shannon, C. E. (1948), ‘A Mathematical Theory of Communication’, The Bell System Technical Journal, 27 (3): 379–423. Shannon, C. E. and Weaver, W. (1964), The Mathematical Theory of Communication, Champaign: University of Illinois Press.



References

247

Sharma, S., Bawa, S. and Lomash, H. (2018), ‘Approaches in Cultural Computing: A Survey and Inference from Social Computing with Dynamics of Mind’, Wireless Personal Communications, 103 (4): 2693–713. Shaumyan, S. K. (1987), A Semiotic Theory of Language, Bloomington Indianapolis: Indiana University Press. Shaumyan, S. K. ([1977] 2003), Applicational Grammar as a Semantic Theory of Natural Languages, Chicago: Chicago University Press. Shen, J. (2009), ‘Stochastic Modeling Western Paintings for Effective Classification’, Pattern Recognition, 42 (2): 293–301. Sherwood, S. C., Van Tilburg, J. A., Barrier, C. R., Horrocks, M., Richard, K. D. and Ramírez-Aliaga, J. M. (2019), ‘New Excavations in Easter Island’s Statue Quarry: Soil Fertility, Site Formation and Chronology’, Journal of Archaeological Science, 111: 104994. Short, T. L. (2007), Peirce’s Theory of Signs, Cambridge: Cambridge University Press. Sieg, W. (2006), ‘Gödel on Computability’, Philosophia Mathematica, 14 (2): 189–207. Siegelmann, H. (1995), ‘Computation Beyond the Turing Limit’, Science, 268 (5210): 545–8. Sikos, L. F. (2017), Description Logics in Multimedia Reasoning, Heidelberg: Springer International Publishing. Simon, H. (1980), ‘Cognitive Science: The Newest Science of the Artificial’, Cognitive Science, 4 (1): 33–46. Simon, K. (2019), ‘Meet BERT, Google’s New AI Tool for Processing Natural Language’, 3Q Digital, 4 November. Singer, A. (2019), Sciencing the Story: Methodological Approaches to Studying the Use of Narrative in Science Communication, Ph. D Dissertation, Michigan University. Situated Instrument Design for Musical Expression, Available online: https:// digitalhumanities​.berkeley​.edu​/courses​/situated​-instrument​-design​-musical​ -expression Smith D. and Moore, L. (2004), ‘The SIR Model for Spread of Disease - The Differential Equation Model’, Convergence, December. Available online: https://www​.maa​.org​/press​ /periodicals​/loci​/joma​/the​-sir​-model​-for​-spread​-of​-disease​-the​-differential​-equation​ -model (accessed 12 November 2020). Smith, W. (2000), ‘Fundamental Theses of Vexillology’, in W. Smith (ed.), Introduction to Vexillology, 28, Winchester: Flag Research Center. Smolensky, P. (1987), ‘Information Processing in Dynamical Systems: Foundations of Harmony Theory’, in D. E. Rumelhart, J. L. McClelland and PDP Research Group (eds), Parallel Distributed Processing, vol. 1: Explorations in the Microstructure of Cognition: Foundations, 194–281, Cambridge, MA: MIT Press. Smolensky, P. (1988), ‘On the Proper Treatment of Connectionism’, Behavioral and Brain Sciences, 11 (1): 1–23. Snell, J., Swersky, K. and Zemel, R. (2017), ‘Prototypical Networks for Few-shot Learning’. Available online: http://papers​.nips​.cc​/paper​/6996​-prototypical​-networks​-for​-few​-shot​ -learning​.pdf (accessed 14 June 2020). Soare, R. I. (1996), ‘Computability and Recursion’, Bulletin of Symbolic Logic, 2: 284–321. Soare, R. I. (2009), ‘Turing Oracle Machines, Online Computing, and Three Displacements in Computability Theory’, Annals of Pure and Applied Logic, 160 (3): 368–99. Society for Computational Economics. http://comp​-econ​.org/ Sonesson, G. (1988), ‘Methods and Models in Pictorial Semiotics’, Report 3 from the Project ‘Pictorial meanings in the society of information’ Lund: Lund University.

248

References

Available online: https://www​.academia​.edu​/download​/50637500​/Methods​_and​ _Models​_in​_Pictorial​_Semiotic20161130​-32737​-4xf9gz​.pdf (accessed 10 October 2021). Sonesson, G. (2008), ’Beyond Methods and Models’, Signs, 2: 277–319. Sonesson, G. (2011), ‘The Mind in the Picture and the Picture in the Mind: Phenomenological Approach to Cognitive Semiotics’, Lexia: Rivista di semiotica, 07/08: 167–82. Sonesson, G. (2012), ‘The Foundation of Cognitive Semiotics in the Phenomenology of Signs and Meanings’, Intellectica: Revue de l’Association pour la Recherche Cognitive, 58 (2): 207–39. Sowa, J. F. (1976), ‘Conceptual Graphs for a Data Base Interface’, IBM Journal of Research and Development, 20 (4): 336–57. Sowa, J. F. (1999), Knowledge Representation: Logical, Philosophical, and Computational Foundations, Pacific Grove: Course Technology Inc. Sowa, J. F. (2000), Knowledge Representation: Logical, Philosophical, and Computational Foundations, Brooks/Cole: New York. Sowa, J. F. (2010), Ontology. Available online: http://www​.jfsowa​.com​/ontology/ (accessed 7 November 2020). Sowa, J. F. (2019), ‘Conceptual Graphs for Representing Conceptual Structures’. Available online: https://www​.researchgate​.net​/publication​/252211806​_Conceptual​_Graphs​_For​ _Representing​_Conceptual​_Structures (accessed 20 June 2020). Sperberg-McQueen, C. M. (2018), ‘Playing for Keeps: The Role of Modelling in the Humanities’, in J. Flander and F. Jannidis (eds), The shape of Data in Digital Humanities: Modeling Texts and Text-based Resources, London: Routledge. Staab, S. and Studer, R. (2010), Handbook on Ontologies, Berlin, Heidelberg: Springer Science & Business Media. Stachowiak, H. (1983), Modelle - Konstruktion der Wirklichkeit (Deutsch) Taschenbuch – 1. München: Fink, Wilhelm. http://archive​.org​/details​/Sta​chow​iak1​973A​llge​mein​eMod​ ellt​heorie (accessed 25 February 2021). Steedman, M. (1988), ‘Combinators and Grammars’, in R. T. Oehrle, E. Bach and D. Wheeler (eds), Categorial Grammars and Natural Languages Structures, 417–42, Dordrecht: Springer. Stefanowitsch, A. (2020), Corpus Linguistics: A Guide to the Methodology Textbooks in Language Science, Berlin: Language Science Press. Sterrett, S. G. (2003), Kinds of Models, Dept. of Philosophy, Duke University (Based on a contribution to a panel discussion: STS Interdisciplinary Roundtable: ‘The Multiple Meanings of Models’, March 20, John Hope Franklin Center, Duke University, Durham). St-Martin, F. (1987), Sémiologie du langage visuel, Québec: Presses de l’Université du Québec. Stockinger, P. (2015), ‘The Semiotic Turn in Digital Archives and Libraries’, Les Cahiers du Numérique, 11: 57–82. Strawson, P. (1960), ‘On Referring’, Mind, 59: 320–44. Suárez, M. (2004), ‘An Inferential Conception of Scientific Representation’, Philosophy of Science, 71 (5): 767–79. Suárez, M., ed. (2009), Fictions in Science: Philosophical Essays on Modeling and Idealization. New York: Routledge. Suárez, M. (2016), ‘Representation in Science’, in P. Humphreys (ed.), Oxford Handbook in Philosophy of Science, 440–59. New York: Oxford University Press.



References

249

Suárez, M. and Solé, A. (2006), ‘On the Analogy Between Cognitive Representation and Truth’, Theoria, 21 (1): 39–48. Sun, R. (2001), ‘Artificial intelligence: Connectionist and symbolic approaches’, in N. J. Smelser and P. B. Baltes (eds), International Encyclopedia of the Social and Behavioral Sciences, vol. 2, Amsterdam: Elsevier. Suppes, P. (1967), ‘What Is a Scientific Theory?’, in S. Morgenbesser (ed.), Philosophy of Science Today, 55–67, New York: Basic Books. Tanaka-Ishii, K. (2010), The semiotics of Programming, New York: Cambridge University Press. Tarski, A. (1944), ‘The Semantic Conception of Truth and the Foundations of Semantics’, Philosophy and Phenomenological Research, 4 (3): 341–76. TEI: Text Encoding Initiative. Available online: https://tei​-c​.org/ (accessed 12 October 2020). Thabtah, F. (2007), ‘A Review of Associative Classification Mining’, The Knowledge Engineering Review, 22: 37–65. Thom, R. (1974), Modèles mathématiques de la morphogénèse, Paris: Union générale d'éditions. Thom, R. (1988), Esquisse d’une sémiophysique, Paris: InterEditions. Thomas II, W. G. (2004), ‘Computing and the Historical Imagination’, in S. Schreibman, R. Siemens and J. Unsworth (eds), Companion to Digital Humanities, 56–68, Oxford: Blackwell. Tiddi, I., Balliet, D. and ten Teije, A. (2020), ‘Fostering Scientific Meta-analyses with Knowledge Graphs: A Case-Study’, in European Semantic Web Conference, 287–303, Cham: Springer. Tomasi, F., Ciotti, F., Daquino, M. and Lana, M. (2015), ‘Using Ontologies as a Faceted Browsing for Heterogeneous Cultural Heritage Collections’, 1st AI*IA Workshop on Intelligent Techniques at Libraries and Archives, 2015: 1–12. Trudel, É. and Hébert, L. (2011), ‘ Protocole d’analyse de la base de données Magritte. Toutes les œuvres, tous les thèmes ’, Montréal: Université du Québec à Montréal. Available online: http://www​.magrittedb​.com​/wp​-content​/uploads​/2019​/06​/Protocole​ _danalyse​.pdf (accessed 20 June 2020). Turing, A. (1939), ‘Systems of Logic Based on Ordinals’, Proceedings of the London Mathematical Society, Series 2, 45: 161–228. Turing, A. M. ([1936] 1937), ‘On Computable Numbers, With an Application to the Entscheidungsproblem’, Proceedings of the London Mathematical Society,  2–42 (1): 230–65. Uckelman, S. L. (2010), ‘Computing with Concepts, Computing with Numbers: Llull, Leibniz, and Boole’, in F. Ferreira, E. Mayordomo and L. M. Gomes (eds), Programs, Proofs, Processes, 427–37, Tilburg: Tilburg University. Underwood, T. (2017), ‘A Genealogy of Distant Reading’, Digital Humanities Quarterly, 11 (Issue 2): 1–12. UNESCO - UNESCO Collection of Traditional Music of the World. (n. d.). Available online: https://ich​.unesco​.org​/en​/collection​-of​-traditional​-music​-00123 (accessed 31 October 2020). Valette, M. (2018), ‘Elements of a Corpus Semantics for Humanities. Application to the Classification of Subjective Texts’, in D. Compagno (ed.), Quantitative Semiotic Analysis, 141–50, Cham: Springer International Publishing. Van de Walle, J. (2008), ‘Roman Jakobson, Cybernetics and Information Theory: A Critical Assessment’, Folia Linguistica Historica, 29 (1): 87–123.

250

References

Van Dijk, T. A. (1972), Some Aspects of Text Grammar: A Study in Theoretical Linguistics, The Hague: Mouton. Van Fraassen, B. C. (1970), ‘On the Extension of Beth’s Semantics of Physical Theories’, Philosophy of Science, 37 (3): 325–39. Van Fraassen, B. C. (1980), The Scientific Image, Oxford: Oxford University Press. Van Fraassen, B. C. (2010), ‘Scientific Representation: Paradoxes of Perspective’, Analysis, 70: 511–14. Van Fraassen, B. C. (2010), Scientific Representation: Paradoxes of Perspective, Oxford: Oxford University Press. Van Gelder, T. (1998), ‘The Dynamical Hypothesis in Cognitive Science’, Behavioral and Brain Sciences, 21: 615–65. Varela, F. J., Petitot, J., Pachoud, B. and Roy, J.-M., eds (1999), Naturalizing Phenomenology: Issues in Contemporary Phenomenology and Cognitive Science, Stanford: Stanford University Press. Volli, U. (2014), ‘L‘analisi semiotica come ricerca empirica sul testo’, COSMO | Comparative Studies in Modernism, No 4. Available online: https://www​.ojs​.unito​.it​/ index​.php​/COSMO​/article​/view​/635 (accessed 12 June 2020). Von Bertalanffy, L. (1951), ‘Problems of General System Theory’, Human Biology’, 23 (4): 302–12. Von Bertalanffy, L. (1969), General System Theory: Foundations, Development, Applications, Revised edn, New York: George Braziller Inc. Von Neumann, J. ([1955] 1995), in M. Muraskin, F. Brody and T. Vamos (eds), The Neumann Compendium, World Scientific 20th Century Mathematics, Singapore. Von Neumann, J. (1966), Theory of Self-Reproducing Automata, Champaign: University of Illinois Press. Wachsmuth, H. (2015), Text Analysis Pipelines: Towards Ad-hoc Large-Scale Text Mining, Singapore, River Edge: Springer International Publishing. Wang, F.-Y. (2009), ‘Is Culture Computable?’, IEEE Intelligent Systems, 24, 2–3. Available online: https://pdfs​.semanticscholar​.org​/7ca9​/545​9ddf​8c69​da7e​ee32​cfc3​5017​ beac40494​.pdf (accessed 24 September 2020). Weber, G. W. and Bookstein, F. L. (2011), Virtual Anthropology: A Guide to a New Interdisciplinary Field, New York: Springer. Weinrich, H. (1976), Sprache in Texten, Stuttgart: Klett. Weisberg, M. (2013), Simulation and Similarity: Using Models to Understand the World, Oxford: Oxford University Press. Weyl, H. (1949), Philosophy of Mathematics and Natural Science, Princeton: Princeton University Press. What’s on the Menu? http://menus​.nypl​.org/ Whitehead, A. and Russell, B. (1910), Principia Mathematica, Cambridge: Cambridge University Press. Wiener, N. ([1948] 1961), Cybernetics or Control and Communication in the Animal and the Machine, New York: MIT Press and John Wiley & Sons. Wildgen, W. (1982), Catastrophe Theoretic Semantics: An Elaboration and Application of René Thom’s Theory, Bern: John Benjamins Publishing. Wildgen, W. and Brandt, P. A., eds (2010), Semiosis and Catastrophes: René Thom’s Semiotic Heritage, Bern, Peter Lang. Winograd, T. and Flores, F. (1986), Understanding Computers and Cognition: New Foundation for Design, Norwood: Ablex Publishing Corporation.

 References 251 Winther, R. G. (2016), ‘The Structure of Scientific Theories’, in E. N. Zalta (ed.), The Stanford Encyclopedia of Philosophy Archive (Winter 2016 Edition), Stanford. Wolf, J. (2019), ‘Environmental News Network – Unearthing the Mystery of The Meaning of Easter Island’s Moai’. Available online: https://www​.enn​.com​/articles​ /61430​-unearthing​-the​-mystery​-of​-the​-meaning​-of​-easter​-island​-s​-moai (accessed 24 September 2020). World Wide Web Consortium (W3C), (n. d.). Available online: https://www​.w3​.org/ (accessed 5 October 2020). Wyssusek, B. (2006), ‘On Ontological Foundations of Conceptual Modelling’, Scandinavian Journal of Information Systems, 18 (1): 63–80. Zarri, G. P. (1997), ‘A Knowledge Representation Tool for Encoding the “Meaning” of Complex Narrative Texts’, Natural Language Engineering, 3 (2): 231–53. Zarri, G. P. (2009), Representation and Management of Narrative Information – Theoretical Principles and Implementation, London: Springer. Zemanek, H. (1966). Semiotics and programming languages. Comunications of the ACM, 9 (3), 139–143. Ziku, M. (2020), ‘Digital Cultural Heritage and Linked Data: Semantically-Informed Conceptualisations and Practices with a Focus on Intangible Cultural Heritage’, Liber Quarterly, 30(1): 1–16. Zlatev, J. (2012), ‘Cognitive Semiotics: An Emerging Field for the Transdisciplinary Study of Meaning’, The Public Journal of Semiotics, 4 (1): 2–24. Zhou, Z. H. (2019), ‘Abductive Learning: Towards Bridging Machine Learning and Logical Reasoning’, Science China: Information Sciences, 62 (7): 076101.

Name Index Abelson, R. P.  161 Adjiman, R.  86, 193 Adler, J.  104 Albus, J.S.  29, 85 Andersen, P. B.  28, 32, 148 Antsaklis, P. J.  1994 Apel, K.O.  4 Arbez, G. et al.  64 Arecchi, F.T.  12 Ariew, R.  68, 107 Armatte, M.  53 Arthur, D.  199 Ashby, R. W.  10, 11 Bachimont, B.  42, 50, 107 Baetu, T. M.  53, 202 Barwise, J.  3 Bengio, Y.  63, 175 Bennett, S.  22 Bishop, C. M.  170 Biskri, I.  86 Björner, A.  118 Boole, G.  18 Bouissac, P.  7 Bourbaki, N.  96 Brachman, R. J.  159, 171 Brandt, P. A.  3, 6, 7, 32, 110 Brier, S.  7, 30, 32, 85 Brooks, R. A.  25, 170 Bundgaard, P.  32, 109 Campbell, N.  50 Cannizzaro, S.  40 Carbone, J.  149 Carnap, R.  45 Cartwright, N.  46, 47, 49 Cassirer, E.  1 Chaitin, G. J.  140, 183 Charniak, E.  179 Chartier, J.-F.  57, 191 Chartrand, L.  57, 88, 191, 196 Chen, C.  152 Chen, G.  50

Chomsky, N.  5, 135 Church, A. H.  134 Clarke, A.  30 Codd, E. F.  144 Coletta, W. J.  48 Condillac, E. B.  18 Console, L.  149 Cooper, S. B.  140 Coseriu, E.  110 Courville, A. I.  63 Craik, K.  50 Crawford, T.  86 Crowder, J.A.  149 Curry, H.B.  135 Cushing, J.  66 Czachesz, I.  168 Da Cost, J.  140, 183 Da Costa, N.C.A.  140, 183 Dahan-Dalmédico, A.  53 Dai, W.-Z. et al.  149, 179 Danca, M.-F.  50 Danesi, M.  40 Davis, M. et al.  139 d’Aquin, M. et al.  88 Dennett, D.  54 de Salisbury, J.  17 Desclés, J.-P.  118 de Soissons, W.  17 De Tienne, A.  85 Dietrich, E.  135 Dondero, M. G.  152 Doria, F.A.  140, 183 Dormans, J.  120 Dretske, F. I.  27, 85 Duhem, P.  68, 107 Dumont-Morin, G.  83 Eco, U.  7, 183 Feigenbaum, E. A.  159 Fetzer, J. H.  2001 Feyerabend, P.  67

 Name Index 253 Feys, R.  135 Fillmore, C. J.  89, 165 Firth, J. R.  84 Floch, J.-M.  192 Floridi, L.  31, 165 Fodor, J.  137, 138 Fontanille, J.  152 Forest, D.  88 Friess, S.  149 Frigg, R.  47 Gandy, R.  135, 138 Garneau, J. M. É.  105 Gelfert, A.  68, 92, 95, 97, 106 Genesereth, M.  64 Genette, J.  10 Giaretta, P.  162 Giere, R. N.  46, 49, 54, 95, 206 Glaser, B. G.  63 Gomide, F.  29, 110 Goodfellow, I.  63 Gotel, O. et al.  142 Gould, P.  149 Granger, G.G.  71 Graziano, M.S.A.  69 Green, S.  66 Greimas, A. J.  89 Groupe μ  192 Gruber, T.R.  145, 162 Grzegorczyk, R.  138 Guarino, N.  29, 110, 162, 207 Gudwin, R.  29, 31, 110, 207 Guizzardi, G.  63 Habermas, J.  4 Hallée, Y.  105 Halliday, M. A. K.  10 Hardman, L. et al.  41 Hare, J. S. et al.  82 Harman, G.  9 Harnad, S.  31, 65, 108, 138, 170 Harris, Z.  4, 104 Hartmann, S.  47 Haslanger, S.  50, 70, 107 Haugeland, J.  25 Hébert, L.  83, 84 Hempel, C. G.  45 Hesse, M.  50 Hintikka, J.  3

Hinton, G.  26, 175 Hjelmslev, L.  5 Hobbes, T.  17, 18 Hobbs, J. R.  149 Hoffmeyer, J.  85 Hotho, A. et al.  88 Hung, C.  174 Jabareen, Y.  63 Jakobson, R.  117 Johnson-Laird, P. N.  62 Kakas, C.  149 Kitcher, P.  50 Kleene, S. C.  135 Kneale, M.  17 Kneale, W.  17, 18 Kripke, S.  95 Kuhn, T. S.  52 Kull, K.  109 Kusch, M.  52 Lacombe, D.  138 Ladrière, J.  92 Lambek, J.  118 Lambek, M.  118 Langer, S. K.  85 Latour, B.  69, 107, 206 Le Cun, Y.  175 Leijnen, S.  120 Leplin, J.  46 Levesque, H. J.  159, 171, 179 Lévi-Strauss, C.  19, 73, 109 Lewis, D.  86 Lieb, H. H.  110 Lindenmayer, A.  100 List, C.  52 Liu, K.  30, 40, 85, 207 Llull, R.  17, 202 Loula, A.  110 Luhmann, N.  3 Lycan, W.  26 McCarty, W.  66, 208 McClelland, J. L.  26, 71 McCulloch, W. S.  21 McDermott, D.  179 Magnani, L.  105 Mahoney, M. S.  64

254

Name Index

Manning, C.D.  88 Markov, A. A.  135 Marr, B.  54, 92 Martinelli, D.  8 Marty, R.  118 Matiyasevich, Y. V.  139 Mello, R.F.  171 Meunier, J.-G.  23, 37, 86, 88, 118, 183 Meystel, A.  29, 85 Michalski, R. J. et al.  63 Miller, G. A.  165 Minsky, M.  160 Mitchell, T. M.  14, 18, 163, 171 Moore, L.  151 Moretti, F.  88, 189, 196 Morgan, M.S.  46, 47, 49 Morris, C.W.  2 Morrison, M.  46, 47, 49, 97 Muchielli, A.  105 Murphy, K.  170 Mylopoulos, J.  63 Nadin, M.  30, 31, 45, 110, 155, 207 Nagel, E.  45 Nersessian, N. J.  63, 206 Newell, A.  23, 24, 26, 37, 207 Nilsson, N.J.  64 Nöth, W.  26, 85 Nowakowska, M.  110 Nürnberger, A.  88 Ogden, C. K.  3 O’Halloran, K, et al.  40 Paaß, G.  88 Papineau, D.  46 Pascal, B.  202 Pasch, G.  81 Paul, G.  179 Pavel, T.  118 Peirce, C.S.  2, 7, 25, 27–9, 35, 46, 73, 84, 85, 88, 94, 179, 191 Pendergraft, E.P.  28 Penrose, R.  139 Perlovsky, L.I.  29 Perry, J.  3 Petitot, J.  6, 7, 57, 117 Petöfi, J. S.  110 Pettit, P.  52

Piotrowski, D.  6, 110 Pitts, W.  21 Ponti, M. A.  171 Posner, R.  2, 73 Prieto, L.  82, 85 Pulizzotto, D.  89, 196 Pylyshyn, Z. W.  24, 32, 42, 44, 137 Queiroz, J.  29, 110, 207 Quillian, M.R.  160 Raimond, Y.  88 Rastier, F.  10, 42 Rebecchi, L. et al. Reichenbach, H.  45 Rheinberger, H.  46, 49 Rice, H. G.  138 Richard, I. A.  3 Ricoeur, P.  10 Rieger, B. B.  7, 29, 32, 33, 85, 126, 157 Robinson, S. et al.  65, 67, 139 Rogers, H.  135 Rompré, L.  86 Rorty, R.  52 Rose, T.  63 Rothenberg, J.  47 Rumelhart, D. E.  26, 71 Sahle, P.  10 St-Martin, F.  192 Saita, L.  149 Saul, J.  50, 70, 107 Saussure, F.  4, 5 Scardovelli, M. W.  38, 76 Schank, R.  161 Schütze, H.  88 Searle, J.  31, 138 Sebastiani, F.  170 Sebeok, T. A.  8, 85 Shanin, T.  91 Shannon, C. E.  22, 27 Short, T. L.  85 Sieg, W.  134 Siegelmann, H.  138 Sikos, L.F.  159 Simon, H.  23, 24, 26, 207 Smith, D.  151 Smolensky, P.  25, 26 Soare, R. I.  135, 140

 Name Index 255 Solé, A.  95 Sonesson, G.  7, 32, 73 Sowa, J. F.  121, 162 Sperberg-McQueen, C. M.  68, 70 Staab, S.  64 Stachowiak, H. H.  46 Stjernfelt, F.  32, 109 Strauss, A.L.  63 Strawson, P.  43 Studer, R.  64 Suárez, M.  95, 97 Sun, R.  63 Suppes, P.  46 Sylvester, D.  83 Tanaka-Ishii, K.  30, 85, 94, 207 Tarski, A.  71, 95 Thagard, P.  206 Thom, R.  6 Trudel, E.  83, 84 Turing, A.  134, 135, 140 Uckelman, S. L.  17 Underwood, T.  189 Van Dijk, T. A.  118 Van Fraassen, B. C.  46, 95–7 Van Gelder, T.  170

Varela, F. J. et al.  7 Vassilvitskii, S.  199 Velmezova, E.  109, 110 Visetti, Y.M.  6, 110 Von Bertalanffy, L.  21 Von Neumann, J.  21, 91, 135 Wang, F.-Y.  41 Weaver, W.  22 Webster, J.  10 Weinrich, H.  10 Weisberg, M.  54 Wermter, S.  174 Whitfield, S.  83 Wiener, N.  21, 22 Wildgen, W.  6, 110 Winther, R. G.  46, 47 Wolf, J.  75 Woo, C.  63 Woolgar, S.  70, 107 Zemanek, H.  28 Zhou, Z. H.  149, 179 Zisman, A.  142 Zlatev, J.  7

Subject Index abduction  67, 102, 105, 126, 127, 144, 146, 149, 150, 158, 178, 179 abduction in visualization  151 abductive (see reasoning abductive) algebra dynamic  13 language  93 linear  13 models  97 non linear  109, 125, 172 vectors and matrix  13 algorithm  41, 135, 136, 147, 150, see computation as computable functions  136 as effective procedures  136 as expression  150 and mega–algorithms  205 Monte Carlo  142, 150 vs. programming language  136 reasoning analytics  189 traceable  142 untraceable  206 American anniversary parties ceremonies  166 American ethnic groups  77 annotation  84, 180 morphological  188 pragmatic  188 semantic  188 archives  40, 43, 120, 140, 156, 169, 184, 186 argumentation  40, 88 artificial intelligence  14, 19, 31, 51, 63, 82, 111, 129, 145, 146, 149, 159, 167, 170, 179 and its environment  32 and neural nets  148 and ontologies  165 and semiotics  23, 27 as symbol manipulation  17 robotics  8, 14, 26, 30 association rule  131

and associations items sets  131 and inference  177 atomicity  138 automata  41, 135 cellular  21 grammars  135 languages  93, 122 models  109 physical  135 autonomous intelligent agents and systems  14 autonomous signal systems  19 autopoiesis  32 Barnsley fern  120 barrier of meaning  35, 39 Bayes classifiers  174 Bayesian probabilities  132 Beethoven’s symphonies  9 behaviour psychology  148 beliefs  7, 42, 45, 52, 63, 67, 79, 80, 106, 160, 166, 170, 179 and assumptions  66, 67, 196 and biases  67, 80, 106, 149, 178 in conceptual models  67 and culture  84 and explanation  105 justification  104 in models  67, 70 systems  63, 70, 160 biology  8, 21, 99, 112 biological computers  205 sciences  21, 62, 65 biosemiotics  8, 85 birthday parties  11, 38 black box  143, 157, 183 black boxes and visualization  153 calculability  18, 133–42 calculable/computable  139 calculable functions  13, 14, 23, 39, 136, 138–44, 183, 203, 204

 Subject Index 257 calculators  203 symbolic calculation  18, 24 calculating machines Abacus  201 Babbage  203 hand crank mechanical calculator  204 Llull paper wheel machine  19 Pascaline  202 Schickard’s calculator  202 case studies Maple spring revolution  89–90, 132, 198–9 Magritte  81–5, 130–1, 190–2 music  8–13, 84–6, 192 Peirce  132, 194–8 catastrophe theory  96, 101, 148 categories categorization  8, 51, 65, 126 computational  158 in computational model  141 in conceptual model  117 in formal models  98 as functional mapping  98 of images  83 as labels  65 in reasoning  8 in semiotic  76 category theory  109 causation  69 Chaitin theorem  140 chaotic models  50, 101 China found in the Xiangtangshan caves  156 Chinese abacus  201 Church thesis  134 Church–Turing thesis  142 classification  6, 66, 76, 170, see also machine learning architecture  172 by association rules  177 vs. categorization  65, 76, 98 classification topics  40, 42 by clustering methods  125, 132 by discriminant analysis  125 by factorial analysis  125 by hierarchical clustering  176 by K–means  130, 132, 177, 195, 199 by K–nearest neighbours  174

by Latent Dirichlet Allocation (LDA)  177 by Latent Semantics Analysis (LSA)  189 meta–classification  77 by naive Bayesian clustering  125 by neural nets  175, 179 and optimization  148, 172 as ordering  65, 66, 76, 77, 97, 98 by Principal component classes  177 by support–vector machines  174 climatology  12, 49, 55, 76, 206 close interpretation  189 close reading  196 clowns  7, 9 code/encoding  27, 184, 187, 193 and communication  23 as data  145, 150, 172 and descriptor  86 and information channel  22 and knowledge  159 and language  142 and music  85, 87, 131 and representation  30 and symbols  26, 27, 138 and vector model  130 cognition  1, 2, 7–37, 47, 48, 50, 52, 53, 72, 85, 107, 138, 143, 145, 170, 175 architecture  159 artificial  81 and categorization  65 cognitive learning  171 and cognitive sciences  4, 9, 21, 26, 32, 37, 51, 54, 58, 61, 65, 69, 81, 101, 126, 137, 138, 170 cognitive semiotics  3, 4, 32, 37 cognitive view of semiotic  7 and concept  64, 65 culture  42, 43 in formal model  98 and information  7 and interpretation  118 and neuro–science  57 paradigm  36 and reasoning  67 robotics  206 social  82

258

Subject Index

surrogates  141 terms  148 and understanding  80 collective actions  109 collective intelligence  207 communication  1, 6, 7, 30, 51, 81, 109, 112, 125, 155, 165, 171, 207 in cybernetics  22 means of… 169 in robots  110 in science  89, 107 in semiotics  52, 114 theory  23 complexity  1, 9, 11, 183 epistemic  11 epistemological  11–13 computation  17, 24 architecture  29 and artificial intelligence  158, 166 and categorization  115 and computer human interrelation  30 and computers  13, 203, 206 as effective procedures  13, 133, 136, 167, 206 as engineering artefacts  63 and function  98, 133, 135, 143 and interpretation  148 and languages  150 as mechanical embodied agents  33 as physical symbol system  24, 27, 111 as physical technology  205 and programming  121 as rule governed systems  18 in scientific models  201 and semiotic artefacts  155, 207 as semiotic machines  207 and semiotics models  201, 207 as technologies  205 as toolboxes  109, 110 computational models in artificial intelligence  27 vs. calculability  13 vs. computers  201 in computer science  44 in computer semiotics  28, 30–45, 85 definition  14, 29, 31, 33, 55, 201

in semiotics  11, 14, 19, 29, 31, 33, 35, 74, 108, 129, 132–83 passim  153, 157, 192, 193, 203 in text mining  88 in linguistics  155, 169, 207 models  23, 42, 58 and non–computationality  139 in sciences  155, 206 in social sciences  155 computer expert/scientists  41 computer science, see sciences computors  201, 202 conceptual analysis  19, 42, 88–9, 101, 132 concept of mind in Peirce  194–9 dependency graph  161 frame  63 graph  21, 31, 77, 121, 122 schema  63 and synonymy  89 conceptual models  62–5, 70 communicative role  70, 79 critiques of… 70 and data  57, 58 definition  38, 54, 62–5, 86 epistemic rôle  65–70 as interpretation  62 in knowledge engineering  159 and natural language  55, 62, 63, 79, 88 in science  53, 55, 61, 73, 75, 89 in semiotics  73–91, 110 models’ names  63 connectionist  25, 148, 170 consciousness  69 context  3, 4, 30, 32, 77, 79–81, 104, 188, see also situation analysis  168 communication  22 conceptual  89, 90, 104 cultural  15, 30, 125, 159, 188 Harrisian  5, 104 interpretation  27 learning  172 linguistic  30, 166, 168, 169, 194–6 pragmatic  81 relational  84

 Subject Index 259 Saussurian  6 in science  69, 104 semiotic  35, 68, 208 situation  4, 8, 37, 112, 115 social  70 use  28, 88, 113 continuity  11, 129 Cantorian continuity  138 corpus  39, 184–7 corpus linguistics  16, 207 Cosign conferences  30 crawlers  184 culture  1, 3, 30, 165 and beliefs  166 community  8, 111 context  15, 30, 125, 159, 188 cultural heritage  165 scientific  107 and technology  107 trends  109 cybernetics  19, 21–3, 148 cybersemiotics  85 data  26, 39, 104, 106, 137, 147, 149–51, 167, 168, 170, 179 annotation  187, 188 as arguments of functions  103 big data  14, 26, 137, 147, 148, 156, 189 big semiotic data  156 and binary signals  184 and biosemiotics  8 vs. capta  49 in Cartesian graph/space  151, 173 cleaning  187 curation  186 data analysis  169, 188 data–driven algorithms  147 data–driven methodology  39 data–driven reasoning  103 data–driven research  106 definition  143 encoding  150 as extensional expressions of functions  143 in frame languages  121 gathering  184 grounding problem  31 interpretation  189

metadata  186, 187 mining analytics  147 preparation  172 as smart data  180 databases  104, 128, 144, see also data; knowledge bases DBpedia  165, 181 decision  51 decidability  139, 149, 201 decision trees  174 deduction  67, 78, 102, 126, 144–6, 179 and computation  158 and critiques of  178 in learning  180 method  37–8 reasoning  38, 103, 105, 127, 128, 145, 158, 159, 163, 178, 189 reasoning algorithms  189 deep learning  14, 82, 147, 150, 171, 175, 180, 189, 208, 209 demonstration  45, 102 description  12 and analysis  167, 168 anthropomorphic  78 and categorization  188 description logic  159 and diagrammatic models  110 and dialogue analysis  42 and knowledge  152 and quantitative analysis  168 Diderot’s and d’Alembert’s Encyclopedia  186 digital art  30, 31, 40 digital coding  184 digital humanities  40, 44, 88, 109, 137, 155, 185, 207 digital semiotics artefacts  14, 31, 40, 187 androïds  14 discourse  3, 33, 42, 43, 46, 48, 62, 69, 70, 171, 201, 208 analysis  42 anthropomorphic  78 graphical  162 interdisciplinary  44 interpretation  149 natural language  50, 52, 67, 78, 79, 88, 90 in science  49, 133 and semantics  119

260

Subject Index

in semiotics  79 and text  68, 168 theoretical  61, 89 distant reading  88 distant interpretation  189 distributed memory  14 distributional analysis  130 distributional linguistics  104 doxa  69 Duke University’s music databases  186 dynamicity in associations  6 catastrophe dynamics  110 cognition  80 combinatorics  148 and complexity  140 continuity  138 a differential calculus  101, 152 dissipative  148 dynamicist  81, 170 dynamicity  11 emergence  33 in extension representations  98, 99 forms  12 fractals  120 in functional structural states  100 functions  20, 98, 137 indexing  131 interactions  23 interpretation  110 knowledge storage  209 languages  100 learning  180 logic  148 mathematical forms  5 models  21, 55, 61, 129, 157 and morphogenesis  96 ontologies  85 parallel systems  138 phenomenon  12 procedures  203 processes  3, 4, 6, 20, 36, 43, 122 relations  9, 13, 21, 96, 100, 101, 140 in semiotics  44, 48, 112, 122, 209 stabilization  20 structure  8, 97, 100 systems  6, 20–23, 57, 76, 96, 119–20, 138 theories  6, 81

typology  13 visualization  152 Easter Island Moai statues  8, 9, 75 Echiniscus trisetosus  69 e–communications  169 e–manuscripts  169 embodiement  3, 32 e–media  40, 208 emergence  11, see also dynamic systems; patterns schema features  159 schemes  78, 170 emojis  40 emotion  51 epistemic functions  12, 20, 41–43, 47, 48, 54, 63, 64, 74, 93, 107, 127, 152, 160, 171 activities  3, 36, 37, 42, 43, 45, 47, 73, 188, 189 artefacts  11, 47, 93 categorization  3, 77, 101, 141 communication  39, 49, 65, 80 community  41, 52, 62, 67, 72, 76, 87, 113, 190 complexity  11, 14 evaluation  190 explanation  46, 103 features  75 knowledge  160 levels  6, 11 modalities  3 models  53, 55 paradigms  17 point of view  49, 54, 105, 179 process & operations  4, 45, 48, 49, 59, 64, 65, 68, 83, 102, 105, 111, 183 reasoning  105, 141, 144, 180 reduction  66 representation  9 role of conceptual model  65 roles  49, 55, 65, 67, 68, 70, 71, 78, 79, 97, 98, 101, 106, 114, 136, 137, 140–2, 149–53, 158 roles of computational models  140 roles of formal models  98 and semiosis  3, 173 solitudes  44

 Subject Index 261 epistemology  11, 42–46, 55, 66, 71, 73, 107, 110, 128, 133, 190 and complexity  11, 13 discovery  40, 48, 105, 176 hypothesis  105, 104, 179 justification  28, 50, 58, 78, 106 paradox  42 questions and problems  12, 43, 44, 54, 148, 165, 176, 178 e–texts formats: e–books evolution  11, 28, 148 experimentation  46, 49, 53, 206 experiments  6 model  105 in the naturalist paradigm  39, 42 schema  62, 105 expert systems  63, 129, 145, 146, 159, 179 explanation  9, 45, 53, 69, 72, 104, 169, 152 holist  12 localist  12 static  12 features  9, 11, 61, 66, 77, 148, 187 and argument and variables of function  39, 66, 77, 144, 172–88 categorization  49, 75, 82, 83 classification  148 and data  39, 77, 144, 148, 169 dynamical  44, 125 expression in language  85, 87, 115, 169 feature representation in vectors  123–5, 132 features as meta–categories  77 graphs  161 information  88, 188 meaningful or semiotic  8, 41, 52, 125, 169, 184, 187 perceptual and observational features  83, 125 and properties or characteristics indices of entities and artefacts  2, 11, 12, 30, 36–38, 42, 49, 51, 55, 58, 61–7, 75–6, 109, 112, 115, 142, 148, 159, 160, 169, 170

prototypical  3 reasoning  78, 158 structures  51, 80 fictions  78 flags and vexillology  81 National Communist Party flags  81 flow chart & pipeline  136, 184 folksonomies  66, 68, 70, 77, 84, 92, 93, 112, 113 formal languages categorial  6, 93 in categorization  115, 126 definition  5, 50, 92, 129 formalism  53, 92–3, 122, 159, 160 in genetics  93 and grammars  113 in graphs  31, 47, 50, 55, 77, 78, 93–6, 110, 113, 118, 120, 121, 129, 145, 148, 151, 152, 160, 191 in knowledge representation  122, 145, 146, 159–68, 179–81, 189–91 Lindenmayer  119, 162 in linguistics  102, 107, 118, 162 in logic  100, 110, 117, 118, 127, 148, 162 in mathematics  93, 95, 98, 112, 113, 116, 130 metalanguage  71 in programming languages  94, 134, 150 in reasoning  127 recursive  122 roles  50, 114, 126 in science  71, 94, 98 in semantic  95, 97 in semiotics  115, 122, 132 in symbolic systems  6, 92, 129 λ–definable calculus  9, 133 formal model  43, 58, 66, 72, 141 as axiomatic systems  93 and conceptual models  108 critiques of…107 definition  55, 92–94 and formalism  92, 93, 159, 160 in science  91 in semiotics  112 as symbolic systems definition  112

262

Subject Index

and truth  107 fractals  1, 20 FrameNet  165 frames  77, 84, 121, 122, 129, 143, 145, 146, 159, 189 annotations  84 conceptual  73, 87 data frame  121, 161, 180, 186 Fillmorian  89, 199 FrameNet  165 frames features  159–60 graphs  84 Minsky  145, 161 and ontologies  84 Shank  161 frameworks  14, 19, 47, 81, 82, 112 abductive  179 computational  35 conceptual  2, 8, 22, 24, 28, 30–2, 36–40, 42, 63, 69, 71, 77, 80–4, 111–15, 125, 131, 149, 155, 157–8, 187 cultural  80 cybernetic  23 explanatory  157 knowledge  106 naturalist  8, 21 philosophical  36, 81, 82 physical  102 semiotic  21, 31, 35, 39, 40, 69, 81, 112 structuralist  81 theoretical  106 Freebase  165, 181 Gallica  projet  156 games  40, 120, 152, 208 gaming  207 Gandy machine  135 Geisteswissenschaften  44 genetics  148, 206 geometry  13 geometrical analytical  93 geometrical models  97 geometric angles  176 GLAM  181 Godel’s incompleteness theorem  71 Google Book project  41, 156 Google VAULT  181

Graphs  31, 47, 50, 55, 77, 78, 93–6, 110, 113, 118, 120, 121, 129, 145, 148, 151, 152, 160, 191 SciGraph  165 hapax  169 hermeneutics  3, 8, 72, 73, 190 Hermes  209 Hindu funerary ritual  41, 43 human–machine communication  30 iconicity  75, 82, 115, 130, 146, 190, 192 analysis  82 artefacts  30, 81 figures  168 forms  152 iconic language  43, 50, 78, 152 iconic signs  82, 83 idiosyncratic musical competence  86 and image  50, 78 image recognition  42, 82, 193 processing  82, 111, 169, 193 visualization  152, 192 immunology  148 incompleteness  140 indexical signs  11, 25, 28, 112, 190, 209, see also signal analysis  84 audio  85, 88 database  88 definition  84 electronic  88 frequency  86 and information  85, 188, 196 interpretant  84 music  192, 193 physical causes  85 induction  14, 44, 67, 78, 102, 104, 117, 126, 129, 146, 178, 179, see also reasoning algorithmic procedures  148, 189 bottom–up reasoning model applicable to semiotic artefacts  167 bottom–up reasoning process  167 critiques  178 generalization process  104–105, 129, 142, 159, 167 learning  147, 180 processes  146–50, 167

 Subject Index 263 programming logic  147 reasoning algorithms  148 visualization  151 infectious disease (Covid–19)  61, 151 inference inferential statistics  172 machines  14 in ontologies  163 process  144, 174 rules  102, 159 information  7, 27, 85, 88, 112, 152, 165, 168, 173, 180 knowledge bases  166 linguistics  162 management  121, 152 and noise  149 in ontologies  165 political  82 processing  19 processing levels  29 retrieval  82, 131, 170 scientific  63, 107 semiotic  85, 187 statistical  186 systems  30 in texts  165, 166 theory  23, 69 view of semiotics  7 insects or swarm intelligence  148 intelligent systems autonomous  28–33 intelligent computer systems engineering  19 intelligent self–controlled systems  28 and representation  170 International Council of Archives (ICA)  156 International Council of Museums (ICOM)  156 International Federation of Library Associations (IFLA)  156 interpretant  3, 4, 11, 27, 85, 86, 88, 195 interpretant in IA  31 interpretation  1, 8, 10, 25, 27, 36, 38, 39, 43, 45, 48, 54, 62, 69, 72, 77, 91, 97, 110, 131, 138, 148, 149, 189, 190, 209 close vs. distant  189 cognitive  118, 166 communication  22

computation  19, 40 data  137, 189 discourse  149 epistemic  41 formal  46, 95, 117 interpreter  3, 7 interpreter in AI  27 interrelation of computational and computer models  205 in machine  18 processes  42¸74, 153 semantic  46, 94, 95, 170, 193 semiosis  7, 27 semiotics  9, 42, 43, 143 in text  10 intuition  38, 45, 79, 169 journalism  207 Kanizsa illusion  57 knowledge bases  63, 145, 146, 166, 180, 189, 190, see also ontologies acquisition  165 and background knowledge  83 and BERT  165 and common–sense knowledge  159 data  145, 149 database  129, 160, 165, 166, 180, 192 frames like  162, 165 Google  165, 166 knowledge graphs  181, 179, 189 knowledge representation  27, 31, 63, 64, 84, 121, 145–6, 159–61, 165, 167, 170, 180 levels  63 as memory  129 nets  189 and ontologies  162, 163, 165, 189 reasoning  146, 178, 179 and representation (KR)  167 vault  181 language natural  43, 50, 51, 69, 79, 80, 87, 128, see also linguistics as an artefact  5, 8, 13, 43, 51 in categorization  83, 86, 113 in communication  80, 89 computer processing  150, 165, 166, 168, 170 in conceptual models  63–79, 88

264

Subject Index

in data frame and frames  123, 145, 150, 159 in deduction  128 in discourse  42, 67, 78, 79, 88, 90 in formal language  99, 121, 136 in knowledge bases  159, 160 and mental models  62 modelling  106 in music description  131 in ontologies  169 predicates  83 processors  14 in reasoning  67 representation role  51, 55 in semiotics  88, 111, 115 statements and propositions  36, 43, 62, 91 structure and grammar  55, 118 in vectorial model  177 visualization  152 lateralization  38 law  28, 109, 171, 207 behavioral  71 and data  147, 150 learning  149 legisign  22 of mind  132, 195 of motion  66 phenomenological  47 as principles  102 regularities  103–5 semiotic  5 symbolic manipulation  18 learning models supervised learning  148, 172, 173–6 unsupervised learning  148, 172–78 lemmatization  132 lexical meanings  113 lexicometries  207 library systems  30 life  69, 79 lingua franca  150 linguistics  3, 51, 52, 109, 179 annotation  199 applicative grammar  110, 118 artefacts  43 categorial grammar  107 communication  22 computer programs  94

computational linguistic  157 conceptual expression  89 corpus linguistics  169 discourse  48 distributional  104 dynamic  117 features  75 formal  92, 110, 113 generative grammar  93, 96, 101 informational  42, 162 lexicography  169 logical  7 relations  89, 138, 198 rhetoric  88 semantics  83 semio–linguistic  110, 112 semiotics  23, 43 statistical  155 structuralist  8, 81 structures  5, 6 syntax  52, 70, 92 theories and models  4, 13, 38, 92, 97, 107 Llull‘ machine  18 logic  7, 51, 65, 66 combinatorial  95, 118, 122, 135, 138 conceptual analysis  88 conceptual framework  111, 112 description logic  159, 179 existential instantiation  102, 127 formal  23 inductive  146 in knowledge representation  162 logic  95 grammar   95, 135 modal  110 non–monotonic  105 Percian  27 predicate  6, 148, 159–61, 162, 163 predicate logic model  127 programming logic  147 propositional  117, 145 and semiotic  110 symbolic  17 Tarskian  25 theories  67, 162 universal generalization  103, 127, 129, 147

 Subject Index 265 logical argument  17, 202 categories  164, 180 demonstration  17, 45, 102 entailment  179 epistemic terms  179 features  75 form  5, 17, 117, 118 formalism  52, 159 formal language  5, 6, 8, 93, 100, 101, 113, 117, 118, 121, 126, 128, 129, 162 formula  36, 103, 145, 146, 160, 189 generalization  178 grammar  55 inference  93, 163 instantiation  159 linguistic  112 models  13, 64, 92, 110, 117, 127, 128, 188 points of view  178 positivism  47 principles  17 procedures  163 reasoning  17, 179 rules  171 square  110, 128 structure  6, 50, 66, 94, 180 tradition  95 units  205 logicist logico–nomological  53 paradigm  8 structuralism  8, 81–2 theories  6, 81 Lorenz’s chaos theories  96 machine learning algorithms  144, 171, 172 classification  170, 171 computation  148 deep learning  189 definition  170–72 induction  179 logical reasoning  179 machine learning  26, 144, 147, 149, 170, 179 mathematical models  171 neural nets  179

theories  172 media  7, 30, 40, 52, 63, 70, 78, 107, 144, 156, 157, 167 memory  28, 51 memory and memorizing  8 mentalist mentalist theory  3 mentalist theory paradigm  3 message  22, 27, 80 metaphors  14, 20, 26, 36, 67, 68, 78, 97, 147, 152, 208, 209 Moai statues of Easter Island  9, 75 modalities  3, 6, 11, 70, 98 logic  110 multi–factoriality  11 multimedia  5, 40, 157, 165, 169 multi–resolution semiosis  29 music analysis  192, 209 annotations and metadata  85, 87 artefacts  2, 7, 8, 35, 40, 85–87, 156–58, 165, 169, 184, 187, 188, 190, 192, 193, 208 background  86, 193 browsers  86, 87 collection  189 commentaries  87 community  87 composition  42, 189 data  168 database  86, 87, 131, 186 denoted object  85 descriptors  86 digital  85, 131, 187 e–music & digital  40, 85, 87, 131 encoding  85, 87 experience  85–87, 193 features  87 in films  86 genres  192 industry  87 interpretant  85 keyword metadata  86 Music Encoding Initiative (MEI)  87 musicians  85–87 patterns in  189 recognition  193 recommendations  192 retrieval  87

266

Subject Index

signals  131, 187 sound wave features  86 style  43 in vector model  131, 143 vocabulary  86 muslim veil  11, 36

and OWL  16, 180 RDF  165 oracle  140, 181, 209 OWL  16, 180 Oxford Research Centre for the Humanities  207

narration analysis  88, 89 cultural studies  125, 165 in discourse and arguments  52, 67, 70, 78, 128, 132, 207, 209 game  120 graphs  120 journalism  89, 101, 132, 198 structure  89, 110, 117, 119, 198, 199 in texts  10, 132, 194 National Film Board of Canada  186 NELL project  165, 181 neural nets  14, 175, 179 and artificial intelligence  148 neuroscience  51, 57, 69 neurobiology  148 neurophenomenology  148 Newton’s alchemy/chemistry papers  186 New York restaurant menus  156 Notre Dame Cathedral  9

Palaeolithic caves drawings  9, 38, 76 parallelism parallel processing  179 parallel visualization  152 parallelism  11 Pascaline  202 patterns  147, 170 categories Peirce  29, 31 pattern recognition  147 patterns in the paintings  84 pendulum  97, 103, 141, 144 Peirce case studies  88–9, 132, 194, 198 perception  2, 4, 32, 39, 45, 48, 58, 81, 112, 143 and conceptualization  3, 45, 80 In Peirce’s theory of Mind  175, 195 Perseus project  156 phenomenological analyses  72 data  57 experience  4, 189, 193 illusion  57 law  47 representation  32 philosophy of science philosophy of  44–6, 53, 94, 106 situated and mind  144 pine trees  66 political science  207 polyomino tilings  139 pragmatic and pragmatist theory in communication  7 and complexity  4 and computation  33 and conceptual frameworks and culture  189 and epistmology  46 and hermeneutics  3, 72, 81 and intention  107

observation analysis  76 background knowledge  179 data  38, 49, 61, 103, 104 experimental  6, 18, 46, 48, 104, 112, 143, 144 instruments  62, 72 language  74 process  14, 181 ontologies  26, 31, 63, 64, 77, 84, 88, 95, 129, 134, 145, 163, 169 algorithms  164 applications  165 definition  145, 162 informational knowledge base & graph  146, 162, 163, 165, 166, 180, 181, 189, 201 language  165, 181, 188 metaphysical & reality  49, 122, 165

 Subject Index 267 and language  7, 55, 68 model  53, 92, 97, 98, 108 Percian  3 and semantic  52 and semiotics  52 in texts  169, 187, 188 and theory  3, 8, 47, 48 uses/context  3, 70, 81, 105 vision of science  8, 44, 46, 56, 73 probabilities  51, 105, 140, 168, 171, 176–8, 188, see also statistics Bayesians  13 generalization  178 in models  174 Omega  140 predictive process  172 shared  176 in topic modelling  132 problems solvers  14, 149 procedural analysis  167 productivity  133, 138 production rule  135 programming language  28, 30, 41, 136, 149 Cobol  150 and computable functions and algorithms  136, 153 C++ as formal languages  94 Lisp  150 Python  94, 141, 150 R  194 reflexivity  30 as semiotic artefacts  30, 42 proofs  6, 43, 46, 52, 62, 92, 102, 150, 203 prototypes  170 psychology  51, 109 RDF (resource description format)  146, 165, 180, 181, 187 reasoning  4, 29, 30, 45, 49, 50, 54, 67–8, 78, 86, 101, 103–26, see also learning abductive  78, 105, 127, 144, 146, 149–51, 178, 179, 181, 184 abstraction  66 actantial model  89, 132 adaptation  11, 28

in algebra  144 algorithmic  181 analogical  67 axiomatic  67 categorization  65, 67, 68, 72, 83, 98 computational models  18, 67, 144–50, 181–3 computing in conceptual models  67–8, 78–9 data analysis  188 data analytics  189 data driven  103 deductive  38, 103, 105, 127, 128, 145, 158, 159, 163, 178, 189 (see also deduction) defeasible  105 discursive forms  67, 78 fictional  67 in formal model  101, 102, 126–30 and formal semiotic model  126 iconic forms  67 inductive  39, 103, 105, 127, 129, 147, 148, 158, 167, 168, 173, 190 (see also induction) inference  27 logic  27, 67 metaphorical  67 in narration  67 natural language poperian’  128 psychology  67 role of computational models  144 rule  53, 67 as rule governed  67 semiotic inquiries  128, 129, 181 as symbol manipulation  17, 18 regularities  125, 132, 147, 170, see also patterns representamen  3 representation  8, 25, 27, 84, 89 as artefact  47, 78 in cognitive science conceptual  65, 111 in connectionism  26 critics of  25 distributed  26 extensional  99

268

Subject Index

formal  50, 95, 97 and hermeneutic  8 in IA  24, 84 intensional  99 and interpretation  27 and knowledge  27, 121, 145, 146, 159–65, 170 mathematical  97, 123, 173, 180–9 and meaning  95 mental and cognitive surrogates, proxies, substitutes etc.  8, 32, 36, 37, 39, 46–8, 50–2, 54, 78, 84, 85 in observation  96 physical  50 referential  98 as rule governed  88 in science  47, 95 semantic  95, 205 and “ stand for relation”  3, 4, 24 systems  31 technologies  31 as a theory laiden concept  47 and visualization  191 retro-engineering  105 rule-based systems  145, 170 manipulation of symbol transformation  24 ruled-governed languages  121 sciences  11, 12, 49 and climatology  76 cognitives  7, 37, 46, 51, 54, 65, 69, 81, 137, 138, 170, 171, 184 communication in  52, 72 communities  32 complexity  11, 14 and computational models  133– 55 passim computer  28, 30, 32, 41, 42, 44, 54, 63, 88, 94, 103, 140, 143, 145, 147, 157, 170, 206 and computer models  201–11 passim and conceptual models  45–60 and culture and data  143 definition  45, 46, 58, 64, 70, 76, 103

and economic science  157 environmental  157 as epistemic endevour and role  61, 140 and epistemology of  47, 49, 74, 107 and formal models  64, 91–109 passim human  4, 5, 44, 45, 47, 81, 109, 207, 208 information  73 and laws  5 and life  157 meteorological  12 methodology  14 and models  14, 45, 48, 50, 52, 98, 106 natural  6, 81, 108 and neurosciences  22, 25, 51, 57, 69, 157 and non computability in  183 and philosophy of  44–6, 53, 94, 106 as point of view  58 political  207 pragmatics of  92, 104 in reasoning  67, 105 and semiotics in  50, 53, 73, 114 social  42, 46, 141, 155, 207 theories  20, 45, 47, 72 and visualization  153 scrapers  184 selection  65, 66 arguments of functions  77 in categorization  77, 98 semantics Bourbakian semantic  113 in computational systems  133, 150, 171 data  31 and formal language and systems  46, 94, 95, 96, 107, 116, 122, 129 formal semantics  107 interpretation  94–6 latent analysis  194 in mathematics  96 model  79 model vs. conceptual model  79

 Subject Index 269 in natural language  62, 118 and pragmatics  5 reference  3, 96 relations  41, 70, 160, 165 representation  25 semantic analysis  88 semantic relations extraction of  41 Tarskian semantics  6, 25, 46, 95, 96 truth conditions  46 view of theories  47, 48 web  31, 88, 146, 181, 188 semiology  5 semiosis  9, 22 analysis of  45 in categoriazation  76 in cognition  7, 8, 48 and complexity  9, 15 in computation  35, 41 in computers  28, 29, 32, 33 dynamic  6 as an epistemic operation  112 externality of  27 features  36 and hermeneutics  4 and interpretation  2, 8 and learning  171 and mathematic  171 and meaning  6, 7, 39 multiresolutional  29 processes  2, 8, 28, 33, 48 programmers’  37 Semiosis definition  43 in semiotic inquiries  182, 209 semiotics applied  36 artefacts passim computation  27 computational paradigm  44 of computing  32 conceptual framework  30, 36, 39, 40, 80 critiqus  4, 9, 109, 110 definition  2, 36, 48 descriptors  83 digital projects  156–8 as an epistemic and scientific enquiry  43, 49, 72

formal forms and system  46, 47 formal models  111–16, 122, 155 hermeneutics paradigm  4, 8, 73 machine  30, 155 mentalist paradigm  3 as model construction  48 naturalist paradigm  8, 36 paradigms  2, 9, 29, 80 philososophical paradigm  4 pragmatist paradigm  8 as research programme roles  48, 115 semiotic square  36 structuralist paradigm  4–5, 112 structuralist view of theory  2, 36 sentiment analysis  42 sentiment analysis  88 sermons on Lincoln’s assassination  156 sexuality  7 Shakespeare  3, 43, 166 signal  1–8, 19, 22, 23, 27, 125, 130, 187, 193, see also indexical signal acoustic  131, 193 binary digital  3, 189, 207 control  35 digital  193 electric  207 encoded  189 music  131 in programing language  31 system  131, 187 visualization  193 signified  5 signifier  5 signs  2–9, 18, 25–30, see also iconic sign convention  23 definition  118 features  73–4 function  35 graphical  119 and mind  195–7 as physical carriers or ‘vehicles’  80 sign characters  74 sign features  3 sign qualities  74 vehicle  80 visual  192

270

Subject Index

similarity calculus between vectors  125 features  170, 173 metonymy  162 patterns  131 relations  84 simulations  206 situation contextual  4, 8, 80 meaningful situations  3, 8, 15, 36, 37, 46, 68, 81, 112, 115, 160, 178, 183 situated philosophy  148 social and cultural computing  207 social context  70 social media  207 social mobility  109 social networks  109 sociology  109 social media  40, 63, 209 Spanish Easter procession  12, 132, 166 statistics  13, 27, 39, 172, 189 analysis  155 Bayesian  13 classification  148 computation  157 descriptive  13, 168, 169, 172, 175, 176, 188, 209 graphs  51 inferential  13, 171, 172 information  186 learning  171 models  53, 93, 97, 109, 143, 172 optimization  148 regularities  38, 129 (see also probabilities) statistical methods  125 statistical regularities for  169 statistics correlation  177 statues of the world  114, 162–8 description  116, 120–1 statues of the world stigmergy  148 stimulus  3, 8 Stonehenge  9, 41 structural analysis  168 annotation  187 e combination/composition  25, 88, 92, 94, 102, 106, 196

features  64 functions  18, 100 information  188 mathematical  132 models  53 regularities  38, 168 relations  28, 55, 97, 98 semantic  187 states  101 sub-symbol  26 systems  1, 6, 55, 67, 92–4 structural/ist-sism Harrisian  94 Hjemslevian  22 linguistic  4–8 logicist  8, 81, 82 paradigm  5, 6, 22, 36, 46, 77, 82, 84, 110, 112, 155 Saussurian  5 semiotic  84, 89 summarizing  41, 196 symbols, symbolic  1–8, 24, 25, 80, 82, 89, 198 analysis  89 artefacts  1, 3, 19, 67, 88, 132, 190 in artificial intelligence  24 categorization  98 communication  110 in computation  25, 27, 28, 30, 31, 54, 66, 68, 94, 158 in computers  31, 155, 203, 207 in conceptual model  66, 68, 73, 75 convention  88 culture  32 definition  19, 88 diagram  50 encoding  24, 138, 184, 185 environment  33 features of  24 formal language  24, 92–7, 107–15, 127, 129, 138, 142 formal model knowledge  179 languages  6, 8, 17, 75, 88, 137, 160, 207 learning system  144 robots  110 manipulation  23–6, 28, 30, 31, 35, 134

 Subject Index 271 meaning and semantic  18, 25–7, 30–3, 75, 78, 94, 95, 107, 110 mental  33 in models  24, 66, 68 physical  24, 207 reasoning  67, 101, 127, 159 representation  158, 160 rules  92, 93, 106 schema  91 semiotics  42, 88, 116–22, 130 synonymy  198

translation automatic  207 Turing machine  133, 138, 150, 183, 204, 206 Turing’s thesis  135 Turner’s paintings  186 tweets  40, 180 Trump’s tweets  186

table settings  11 tabular type of list  121 technè  50, 107 Text Encoding Initiative (TEI)  40, 186 textometrics  169, 207 texts  4, 7, 10, 35, 40, 41, 43, 68, 69, 80, 82, 87–9, 102, 110, 119, 132, 146, 166–8, 177, 187, 188, 190, 207, 208 analysis and mining  40, 89, 168, 194, 198–9 data  181 database  199 interpretation  4 narrative  194, 198 ontologies  177 peritext  186 processing  155 recognition  187 segments  198–9 summary  41, 196, 198 textometrics  164 theatre  7, 168 thermodynamic system  20 topic analysis  130 Latent Dirichlet Allocation (LDA)  177, 194 topic analysis  40, 42, 47, 131, 187 topic modelling  88, 151, 177, 194 words embedding  132 traceability  93 traffic circulation signals  35, 118, 122

vector formal models  13, 71, 93, 104, 111, 126, 130–2, 162, 194 vector language  122, 163, 172, 189 vectorial calculus/algebra  124–6, 172 vectorial representation  124, 130, 162, 189 Vermeer  37 visualization  3, 47, 50, 51, 53, 87, 150–3, 173, 191–6, 206, 209 von Neumann computer architecture  205

understanding  9, 45, 53, 69, 72, 79, 169 UNESCO cultural music heritages  156 UNESCO’  186

Watt’s steam engine governor  19 web/internet  40, 165 web knowledge  181 Web ontology graphs  165 World Wide Web Consortium  186 Web Internet  5, 155, 162 wedding ring  9, 74, 79, 169 word clouds  191, 195–6 WordNet  165 words embedding  7, 132, 168, 174 workflow/pipeline of computational semiotics  183, 185 YAGO  165, 181 zoosemotics  8

272

 273

274

 275

276