TeX by Topic, A TeXnician's Reference [Revision 1.5 ed.] 0201568829

Citation preview

TEX BY TOPIC, A TEXNICIAN’S REFERENCE VICTOR EIJKHOUT DOCUMENT REVISION 1.5, 2019

c 1991-2013 Victor Eijkhout. Copyright Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled ”GNU Free Documentation License”. This document is based on the book TEX by Topic, copyright 1991-2019 Victor Eijkhout. This book was printed in 1991 by Addison-Wesley UK, ISBN 0-201-56882-9, reprinted in 1993, pdf version first made freely available in 2001. Cover design (lulu.com version): Joanna K. Wozniak ([email protected])

Victor Eijkhout – TEX by Topic

1

2

Victor Eijkhout – TEX by Topic

Contents

License Preface 1 1.1 1.2 1.2.1 1.2.2 1.3 1.3.1 1.3.2 1.3.3 1.4 1.5 1.6 1.6.1 1.6.2 2 2.1 2.2 2.3 2.4 2.5 2.5.1 2.5.2 2.5.3 2.6 2.7 2.7.1 2.7.2 2.7.3 2.7.4 2.7.5 2.7.6 2.7.7 2.7.8 2.7.9 2.8

15 21 The Structure of the TEX Processor 23 Four TEX processors 23 The input processor 24 Character input 24 Two-level input processing 24 The expansion processor 25 The process of expansion 25 Special cases: \expandafter, \noexpand, and \the 25 Braces in the expansion processor 26 The execution processor 26 The visual processor 27 Examples 28 Skipped spaces 28 Internal quantities and their representations 28 Category Codes and Internal States 29 Introduction 29 Initial processing 29 Category codes 30 From characters to tokens 32 The input processor as a finite state automaton 32 State N: new line 32 State S: skipping spaces 32 State M: middle of line 33 Accessing the full character set 33 Transitions between internal states 33 0: escape character 34 1–4, 7–8, 11–13: non-blank characters 34 5: end of line 34 6: parameter 34 7: superscript 34 9: ignored character 35 10: space 35 14: comment 35 15: invalid 35 Letters and other characters 35 3

2.9 2.10 2.10.1 2.10.2 2.10.3 2.10.4 2.10.5 2.10.6 2.10.7 2.11 2.11.1 2.11.2 2.11.3 2.12 2.12.1 2.12.2 2.12.3 2.13 3 3.1 3.2 3.2.1 3.2.2 3.3 3.4 3.5 3.5.1 3.5.2 3.5.3 3.5.4 3.6 3.7 3.7.1 3.7.2 4 4.1 4.2 4.2.1 4.2.2 4.2.3 4.3 4.3.1 4.3.2 4.3.3 4.3.4 4.3.5 5 4

The \par token 36 Spaces 37 Skipped spaces 37 Optional spaces 37 Ignored and obeyed spaces 38 More ignored spaces 38 hspace tokeni 39 Control space 39 ‘ ’ 39 More about line ends 40 Obeylines 40 Changing the \endlinechar 40 More remarks about the end-of-line character 41 More about the input processor 41 The input processor as a separate process 41 The input processor not as a separate process 42 Recursive invocation of the input processor 42 The @ convention 43 Characters 45 Character codes 45 Control sequences for characters 46 Denoting characters to be typeset: \char 46 Implicit character tokens: \let 47 Accents 48 Testing characters 49 Uppercase and lowercase 50 Uppercase and lowercase codes 50 Uppercase and lowercase commands 50 Uppercase and lowercase forms of keywords 50 Creative use of \uppercase and \lowercase 51 Codes of a character 51 Converting tokens into character strings 51 Output of control sequences 52 Category codes of a \string 52 Fonts 53 Fonts 53 Font declaration 54 Fonts and tfm files 54 Querying the current font and font names 54 \nullfont 55 Font information 55 Font dimensions 55 Kerning 56 Italic correction 56 Ligatures 57 Boundary ligatures 57 Boxes 59 Victor Eijkhout – TEX by Topic

5.1 5.2 5.2.1 5.2.2 5.2.3 5.2.4 5.3 5.3.1 5.3.2 5.3.3 5.4 5.4.1 5.4.2 5.4.3 5.4.4 5.4.5 5.5 5.6 5.7 5.8 5.9 5.9.1 5.9.2 5.9.3 5.9.4 5.9.5 5.9.6 6 6.1 6.1.1 6.1.2 6.2 6.3 6.3.1 6.3.2 6.4 6.4.1 6.4.2 6.4.3 6.5 6.6 6.6.1 6.7 7 7.1 7.2 7.2.1

Boxes 60 Box registers 60 Allocation: \newbox 60 Usage: \setbox, \box, \copy 61 Testing: \ifvoid, \ifhbox, \ifvbox 61 The \lastbox 61 Natural dimensions of boxes 62 Dimensions of created horizontal boxes 62 Dimensions of created vertical boxes 62 Examples 63 More about box dimensions 64 Predetermined dimensions 64 Changes to box dimensions 65 Moving boxes around 65 Box dimensions and box placement 65 Boxes and negative glue 66 Overfull and underfull boxes 67 Opening and closing boxes 67 Unboxing 68 Text in boxes 69 Assorted remarks 70 Forgetting the \box 70 Special-purpose boxes 70 The height of a vertical box in horizontal mode 70 More subtleties with vertical boxes 70 Hanging the \lastbox back in the list 71 Dissecting paragraphs with \lastbox 72 Horizontal and Vertical Mode 73 Horizontal and vertical mode 73 Horizontal mode 73 Vertical mode 74 Horizontal and vertical commands 74 The internal modes 75 Restricted horizontal mode 75 Internal vertical mode 75 Boxes and modes 76 What box do you use in what mode? 76 What mode holds in what box? 76 Mode-dependent behaviour of boxes 76 Modes and glue 76 Migrating material 77 \vadjust 77 Testing modes 77 Numbers 79 Numbers and hnumberis 79 Integers 79 Denotations: integers 80

Victor Eijkhout – TEX by Topic

5

7.2.2 7.2.3 7.2.4 7.2.5 7.2.6 7.2.7 7.3 7.4 7.5 7.5.1 7.5.2 7.5.3 7.6 7.7 7.7.1 7.7.2 8 8.1 8.1.1 8.1.2 8.1.3 8.1.4 8.1.5 8.1.6 8.2 8.2.1 8.2.2 8.2.3 8.3 8.3.1 8.3.2 8.3.3 8.3.4 8.3.5 8.3.6 8.3.7 8.3.8 8.3.9 9 9.1 9.1.1 9.2 9.2.1 9.2.2 9.2.3 9.3 9.3.1 6

Denotations: characters 80 Internal integers 81 Internal integers: other codes of a character 82 hspecial integeri 82 Other internal quantities: coersion to integer 82 Trailing spaces 82 Numbers 82 Integer registers 83 Arithmetic 83 Arithmetic statements 84 Floating-point arithmetic 84 Fixed-point arithmetic 84 Number testing 84 Remarks 85 Character constants 85 Expanding too far / how far 85 Dimensions and Glue 87 Definition of hgluei and hdimeni 88 Definition of dimensions 88 Definition of glue 89 Conversion of hgluei to hdimeni 90 Registers for \dimen and \skip 90 Arithmetic: addition 90 Arithmetic: multiplication and division 91 More about dimensions 91 Units of measurement 91 Dimension testing 92 Defined dimensions 92 More about glue 92 Stretch and shrink 93 Glue setting 94 Badness 94 Glue and breaking 95 \kern 95 Glue and modes 95 The last glue item in a list: backspacing 96 Examples of backspacing 96 Glue in trace output 97 Rules and Leaders 99 Rules 99 Rule dimensions 100 Leaders 100 Rule leaders 101 Box leaders 102 Evenly spaced leaders 102 Assorted remarks 103 Rules and modes 103 Victor Eijkhout – TEX by Topic

9.3.2 9.3.3 9.3.4 9.3.5 9.3.6 10 10.1 10.2 10.3 10.4 10.4.1 10.4.2 10.4.3 11 11.1 11.2 11.3 11.4 11.5 11.5.1 11.5.2 11.5.3 11.5.4 11.5.5 11.5.6 11.6 11.7 11.7.1 11.7.2 11.8 11.8.1 11.8.2 11.8.3 11.9 11.9.1 11.9.2 11.9.3 11.9.4 11.9.5 12 12.1 12.2 12.3 12.3.1 12.3.2 12.3.3 12.3.4

Ending a paragraph with leaders 103 Leaders and box registers 103 Output in leader boxes 104 Box leaders in trace output 104 Leaders and shifted margins 104 Grouping 105 The grouping mechanism 105 Local and global assignments 106 Group delimiters 106 More about braces 107 Brace counters 107 The brace as a token 108 Open and closing brace control symbols 108 Macros 109 Introduction 109 Layout of a macro definition 110 Prefixes 110 The definition type 111 The parameter text 111 Undelimited parameters 112 Delimited parameters 112 Examples with delimited arguments 113 Empty arguments 114 The macro parameter character 114 Brace delimiting 115 Construction of control sequences 115 Token assignments by \let and \futurelet 116 \let 116 \futurelet 117 Assorted remarks 117 Active characters 117 Macros versus primitives 117 Tail recursion and the \loop macro 118 Macro techniques 119 Unknown number of arguments 119 Examining the argument 119 Optional macro parameters with \futurelet 121 Two-step macros 121 A comment environment 122 Expansion 125 Introduction 125 Ordinary expansion 125 Reversing expansion order 126 One step expansion: \expandafter 126 Total expansion: \edef 127 \afterassignment 127 \aftergroup 128

Victor Eijkhout – TEX by Topic

7

12.4 12.4.1 12.4.2 12.5 12.5.1 12.5.2 12.5.3 12.5.4 12.6 12.6.1 12.6.2 12.6.3 12.6.4 12.6.5 12.6.6 12.6.7 12.6.8 12.6.9 13 13.1 13.2 13.2.1 13.2.2 13.2.3 13.3 13.4 13.5 13.5.1 13.5.2 13.5.3 13.5.4 13.5.5 13.6 13.7 13.8 13.8.1 13.8.2 13.8.3 13.8.4 13.8.5 13.8.6 13.8.7 14 14.1 14.2 14.3 14.4 8

Preventing expansion 129 \noexpand 129 \noexpand and active characters 129 \relax 130 \relax and \csname 130 Preventing expansion with \relax 131 TEX inserts a \relax 131 The value of non-macros; \the 132 Examples 132 Expanding after 132 Defining inside an \edef 133 Expansion and \write 134 Controlled expansion inside an \edef 135 Multiple prevention of expansion 135 More examples with \relax 136 Example: category code saving and restoring 136 Combining \aftergroup and boxes 137 More expansion 138 Conditionals 139 The shape of conditionals 139 Character and control sequence tests 140 \if 140 \ifcat 140 \ifx 141 Mode tests 141 Numerical tests 142 Other tests 142 Dimension testing 142 Box tests 142 I/O tests 142 Case statement 142 Special tests 143 The \newif macro 143 Evaluation of conditionals 144 Assorted remarks 145 The test gobbles up tokens 145 The test wants to gobble up the \else or \fi 145 Macros and conditionals; the use of \expandafter 146 Incorrect matching 147 Conditionals and grouping 147 A trick 148 More examples of expansion in conditionals 148 Token Lists 151 Token lists 151 Use of token lists 151 htoken parameteri 152 Token list registers 152 Victor Eijkhout – TEX by Topic

14.5 14.5.1 14.5.2 15 15.1 15.2 15.3 15.4 16 16.1 16.2 16.3 16.3.1 16.3.2 16.4 16.4.1 16.4.2 16.4.3 16.4.4 17 17.1 17.1.1 17.1.2 17.2 17.2.1 17.2.2 17.2.3 17.2.4 17.2.5 17.2.6 18 18.1 18.2 18.2.1 18.2.2 18.3 18.3.1 18.3.2 18.3.3 18.3.4 18.3.5 18.3.6 19 19.1 19.1.1 19.1.2 19.1.3

Examples 153 Operations on token lists: stack macros 153 Executing token lists 154 Baseline Distances 155 Interline glue 155 The perceived depth of boxes 157 Terminology 158 Additional remarks 158 Paragraph Start 159 When does a paragraph start 159 What happens when a paragraph starts 160 Assorted remarks 160 Starting a paragraph with a box 160 Starting a paragraph with a group 160 Examples 161 Stretchable indentation 161 Suppressing indentation 161 An indentation scheme 161 A paragraph skip scheme 162 Paragraph End 165 The way paragraphs end 165 The \par command and the \par token 165 Paragraph filling: \parfillskip 166 Assorted remarks 166 Ending a paragraph and a group at the same time 166 Ending a paragraph with \hfill\break 167 Ending a paragraph with a rule 167 No page breaks in between paragraphs 167 Finite \parfillskip 167 A precaution for paragraphs that do not indent 168 Paragraph Shape 169 The width of text lines 170 Shape parameters 170 Hanging indentation 170 General paragraph shapes: \parshape 171 Assorted remarks 171 Centred last lines 171 Indenting into the margin 172 Hang a paragraph from an object 172 Another approach to hanging indentation 173 Hanging indentation versus \leftskip shifting 173 More examples 174 Line Breaking 175 Paragraph break cost calculation 176 Badness 176 Penalties and other break locations 177 Demerits 177

Victor Eijkhout – TEX by Topic

9

19.1.4 19.1.5 19.2 19.2.1 19.2.2 19.3 19.3.1 19.3.2 19.4 19.4.1 19.4.2 19.4.3 19.4.4 19.5 20 20.1 20.2 20.3 20.4 20.5 20.5.1 20.5.2 20.5.3 20.5.4 21 21.1 21.2 21.2.1 21.2.2 21.2.3 21.2.4 21.3 21.4 22 22.1 22.2 22.3 22.4 22.4.1 22.4.2 22.4.3 22.5 22.5.1 22.5.2 23 23.1 23.2 10

The number of lines of a paragraph 178 Between the lines 178 The process of breaking 178 Three passes 179 Tolerance values 179 Discretionaries 179 Hyphens and discretionaries 179 Examples of discretionaries 180 Hyphenation 181 Start of a word 181 End of a word 181 TEX2 versus TEX3 182 Patterns and exceptions 182 Switching hyphenation patterns 182 Spacing 185 Introduction 185 Automatic interword space 185 User interword space 186 Control space and tie 187 More on the space factor 188 Space factor assignments 188 Punctuation 188 Other non-letters 189 Other influences on the space factor 189 Characters in Math Mode 191 Mathematical characters 192 Delimiters 192 Delimiter codes 193 Explicit \delimiter commands 193 Finding a delimiter; successors 193 \big, \Big, \bigg, and \Bigg delimiter macros 194 Radicals 194 Math accents 195 Fonts in Formulas 197 Determining the font of a character in math mode 197 Initial family settings 198 Family definition 198 Some specific font changes 198 Change the font of ordinary characters and uppercase Greek 198 Change uppercase Greek independent of text font 199 Change the font of lowercase Greek and mathematical symbols 199 Assorted remarks 199 New fonts in formulas 199 Evaluating the families 200 Mathematics Typesetting 201 Math modes 202 Styles in math mode 202 Victor Eijkhout – TEX by Topic

23.2.1 23.2.2 23.3 23.4 23.5 23.6 23.6.1 23.6.2 23.6.3 23.7 23.8 23.9 23.10 23.10.1 23.10.2 23.10.3 24 24.1 24.2 24.3 24.4 24.5 24.6 24.6.1 24.6.2 24.7 25 25.1 25.2 25.2.1 25.2.2 25.2.3 25.2.4 25.3 25.3.1 25.3.2 25.3.3 25.3.4 25.4 25.4.1 25.4.2 25.4.3 25.4.4 25.4.5 25.5 26 26.1

Superscripts and subscripts 203 Choice of styles 203 Classes of mathematical objects 204 Large operators and their limits 204 Vertical centring: \vcenter 205 Mathematical spacing: mu glue 205 Classification of mu glue 206 Muskip registers 206 Other spaces in math mode 207 Generalized fractions 207 Underlining, overlining 208 Line breaking in math formulas 208 Font dimensions of families 2 and 3 208 Symbol font attributes 208 Extension font attributes 209 Example: subscript lowering 210 Display Math 211 Displays 211 Displays in paragraphs 212 Vertical material around displays 212 Glue setting of the display math list 213 Centring the display formula: displacement 213 Equation numbers 214 Ordinary equation numbers 214 The equation number on a separate line 214 Non-centred displays 214 Alignment 217 Introduction 217 Horizontal and vertical alignment 217 Horizontal alignments: \halign 218 Vertical alignments: \valign 218 Material between the lines: \noalign 218 Size of the alignment 219 The preamble 219 Infinite preambles 219 Brace counting in preambles 220 Expansion in the preamble 220 \tabskip 220 The alignment 221 Reading an entry 221 Alternate specifications: \omit 221 Spanning across multiple columns: \span 222 Rules in alignments 222 End of a line: \cr and \crcr 223 Example: math alignments 224 Page Shape 225 The reference point for global positioning 225

Victor Eijkhout – TEX by Topic

11

26.2 26.3 27 27.1 27.2 27.3 27.4 27.4.1 27.4.2 27.4.3 27.5 27.6 27.6.1 27.6.2 27.6.3 28 28.1 28.2 28.3 28.4 28.4.1 28.4.2 28.4.3 28.4.4 28.4.5 28.4.6 29 29.1 29.2 29.3 29.4 29.5 29.6 30 30.1 30.2 30.2.1 30.2.2 30.2.3 30.3 30.4 30.4.1 30.4.2 30.4.3 30.4.4 30.4.5 30.4.6 12

\topskip 225 Page height and depth 226 Page Breaking 227 The current page and the recent contributions 228 Activating the page builder 228 Page length bookkeeping 228 Breakpoints 229 Possible breakpoints 229 Breakpoint penalties 229 Breakpoint computation 230 \vsplit 231 Examples of page breaking 232 Filling up a page 232 Determining the breakpoint 232 The page builder after a paragraph 233 Output Routines 235 The \output token list 235 Output and \box255 236 Marks 236 Assorted remarks 238 Hazards in non-trivial output routines 238 Page numbering 238 Headlines and footlines in plain TEX 238 Example: no widow lines 238 Example: no indentation top of page 239 More examples of output routines 240 Insertions 241 Insertion items 241 Insertion class declaration 242 Insertion parameters 242 Moving insertion items from the contributions list 243 Insertions in the output routine 244 Plain TEX insertions 244 File Input and Output 245 Including files: \input and \endinput 245 File I/O 246 Opening and closing streams 246 Input with \read 246 Output with \write 247 Whatsits 247 Assorted remarks 248 Inspecting input 248 Testing for existence of files 248 Timing problems 248 \message versus \immediate\write16 248 Write inside a vertical box 249 Expansion and spaces in \write and \message 249 Victor Eijkhout – TEX by Topic

31 31.1 31.1.1 31.1.2 31.2 32 32.1 32.1.1 32.1.2 32.1.3 32.2 33 33.1 33.1.1 33.1.2 33.1.3 33.1.4 33.1.5 33.2 33.2.1 33.2.2 33.2.3 33.2.4 33.2.5 33.2.6 33.3 33.3.1 33.3.2 33.3.3 33.4 33.5 33.6 33.6.1 33.6.2 33.6.3 33.6.4 33.7 33.8 34 34.1 34.2 34.3 34.4 35 35.1 35.2 35.2.1

Allocation 251 Allocation commands 251 \count, \dimen, \skip, \muskip, \toks 252 \box, \fam, \write, \read, \insert 252 Ground rules for macro writers 252 Running TEX 255 Jobs 255 Start of the job 255 End of the job 256 The log file 256 Run modes 256 TEX and the Outside World 259 TEX, IniTEX, VirTEX 259 Formats: loading 259 Formats: dumping 260 Formats: preloading 260 The knowledge of IniTEX 260 Memory sizes of TEX and IniTEX 261 More about formats 261 Compatibility 261 Preloaded fonts 261 The plain format 262 The LATEX format 262 Mathematical formats 262 Other formats 262 The dvi file 263 The dvi file format 263 Page identification 263 Magnification 263 Specials 264 Time 264 Fonts 264 Font metrics 264 Virtual fonts 265 Font files 265 Computer Modern 266 TEX and web 266 The TEX Users Group 267 Tracing 269 Meaning and content: \show, \showthe, \meaning 270 Show boxes: \showbox, \tracingoutput 271 Global statistics 272 Execution tracing 273 Errors, Catastrophes, and Help 275 Error messages 275 Overflow errors 276 Buffer size (500) 276

Victor Eijkhout – TEX by Topic

13

35.2.2

Exception dictionary (307) 276

35.2.3

Font memory (20,000) 276

35.2.4

Grouping levels 277

35.2.5

Hash size (2100) 277

35.2.6

Number of strings (3000) 277

35.2.7

Input stack size (200) 277

35.2.8

Main memory size (30,000) 277

35.2.9

Parameter stack size (60) 277

35.2.10 Pattern memory (8000) 278 35.2.11 Pattern memory ops per language 278 35.2.12 Pool size (32,000) 278 35.2.13 Save size (600) 278 35.2.14 Semantic nest size (40) 278 35.2.15 Text input levels (6) 278 36

The Grammar of TEX 279

36.1

Notations

279

36.2

Keywords

280

36.3

Specific grammatical terms

36.3.1

hequalsi 280

36.3.2

hfilleri, hgeneral texti 280

36.3.3

{} and hleft braceihright bracei 281

36.3.4

hmath fieldi 281

36.4

Differences between TEX versions 2 and 3

37

Glossary of TEX Primitives 283

38

Tables 297

38.1

Character tables

38.2

Computer modern fonts

300

38.3

Plain TEX math symbols

305

38.3.1

Mathcharacter codes 305

38.3.2

Delimiter codes 306

38.3.3

hmathchardef tokensi: ordinary symbols 307

38.3.4

hmathchardef tokensi: large operators 308

38.3.5

hmathchardef tokensi: binary operations 309

38.3.6

hmathchardef tokensi: relations

38.3.7

\delimiter macros 311

39

Index 313

Bibliography 14

280

281

298

310

321 Victor Eijkhout – TEX by Topic

GNU Free Documentation License Version 1.2, November 2002 c 2000,2001,2002 Free Software Foundation, Inc. 51 Franklin St, Fifth Floor, Boston, Copyright MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. 0. PREAMBLE The purpose of this License is to make a manual, textbook, or other functional and useful document ”free” in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others. This License is a kind of ”copyleft”, which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software. We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference. 1. APPLICABILITY AND DEFINITIONS This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The ”Document”, below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as ”you”. You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law. A ”Modified Version” of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language. A ”Secondary Section” is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document’s overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them. The ”Invariant Sections” are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none. The ”Cover Texts” are certain short passages of text that are listed, as Front-Cover Texts or BackCover Texts, in the notice that says that the Document is released under this License. A FrontCover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words. Victor Eijkhout – TEX by Topic

15

A ”Transparent” copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not ”Transparent” is called ”Opaque”. Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standardconforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only. The ”Title Page” means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, ”Title Page” means the text near the most prominent appearance of the work’s title, preceding the beginning of the body of the text. A section ”Entitled XYZ” means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as ”Acknowledgements”, ”Dedications”, ”Endorsements”, or ”History”.) To ”Preserve the Title” of such a section when you modify the Document means that it remains a section ”Entitled XYZ” according to this definition. The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License. 2. VERBATIM COPYING You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3. You may also lend copies, under the same conditions stated above, and you may publicly display copies. 3. COPYING IN QUANTITY If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document’s license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover 16

Victor Eijkhout – TEX by Topic

Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects. If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages. If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public. It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document. 4. MODIFICATIONS You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version: A. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission. B. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement. C. State on the Title page the name of the publisher of the Modified Version, as the publisher. D. Preserve all the copyright notices of the Document. E. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices. F. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below. G. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document’s license notice. H. Include an unaltered copy of this License. I. Preserve the section Entitled ”History”, Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled ”History” in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence. J. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the Victor Eijkhout – TEX by Topic

17

network locations given in the Document for previous versions it was based on. These may be placed in the ”History” section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission. K. For any section Entitled ”Acknowledgements” or ”Dedications”, Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein. L. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles. M. Delete any section Entitled ”Endorsements”. Such a section may not be included in the Modified Version. N. Do not retitle any existing section to be Entitled ”Endorsements” or to conflict in title with any Invariant Section. O. Preserve any Warranty Disclaimers. If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version’s license notice. These titles must be distinct from any other section titles. You may add a section Entitled ”Endorsements”, provided it contains nothing but endorsements of your Modified Version by various parties–for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard. You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one. The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version. 5. COMBINING DOCUMENTS You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers. The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work. In the combination, you must combine any sections Entitled ”History” in the various original documents, forming one section Entitled ”History”; likewise combine any sections Entitled ”Acknowledgements”, and any sections Entitled ”Dedications”. You must delete all sections Entitled ”Endorsements.” 6. COLLECTIONS OF DOCUMENTS 18

Victor Eijkhout – TEX by Topic

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects. You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document. 7. AGGREGATION WITH INDEPENDENT WORKS A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an ”aggregate” if the copyright resulting from the compilation is not used to limit the legal rights of the compilation’s users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document. If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document’s Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate. 8. TRANSLATION Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail. If a section in the Document is Entitled ”Acknowledgements”, ”Dedications”, or ”History”, the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title. 9. TERMINATION You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 10. FUTURE REVISIONS OF THIS LICENSE The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/. Victor Eijkhout – TEX by Topic

19

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License ”or any later version” applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.

20

Victor Eijkhout – TEX by Topic

Preface To the casual observer, TEX is not a state-of-the-art typesetting system. No flashy multilevel menus and interactive manipulation of text and graphics dazzle the onlooker. On a less superficial level, however, TEX is a very sophisticated program, first of all because of the ingeniousness of its built-in algorithms for such things as paragraph breaking and make-up of mathematical formulas, and second because of its almost complete programmability. The combination of these factors makes it possible for TEX to realize almost every imaginable layout in a highly automated fashion. Unfortunately, it also means that TEX has an unusually large number of commands and parameters, and that programming TEX can be far from easy. Anyone wanting to program in TEX, and maybe even the ordinary user, would seem to need two books: a tutorial that gives a first glimpse of the many nuts and bolts of TEX, and after that a systematic, complete reference manual. This book tries to fulfil the latter function. A TEXer who has already made a start (using any of a number of introductory books on the market) should be able to use this book indefinitely thereafter. In this volume the universe of TEX is presented as about forty different subjects, each in a separate chapter. Each chapter starts out with a list of control sequences relevant to the topic of that chapter and proceeds to treat the theory of the topic. Most chapters conclude with remarks and examples. Globally, the chapters are ordered as follows. The chapters on basic mechanisms are first, the chapters on text treatment and mathematics are next, and finally there are some chapters on output and aspects of TEX’s connections to the outside world. The book also contains a glossary of TEX commands, tables, and indexes by example, by control sequence, and by subject. The subject index refers for most concepts to only one page, where most of the information on that topic can be found, as well as references to the locations of related information. This book does not treat any specific TEX macro package. Any parts of the plain format that are treated are those parts that belong to the ‘core’ of plain TEX: they are also present in, for instance, LATEX. Therefore, most remarks about the plain format are true for LATEX, as well as most other formats. Putting it differently, if the text refers to the plain format, this should be taken as a contrast to pure IniTEX, not to LATEX. By way of illustration, occasionally macros from plain TEX are explained that do not belong to the core.

Acknowledgment I am indebted to Barbara Beeton, Karl Berry, and Nico Poppelier, who read previous versions of this book. Their comments helped to improve the presentation. Also I would like to thank the participants of the discussion lists TEXhax, TEX-nl, and comp.text.tex. Their questions and answers gave me much food for thought. Finally, any acknowledgement in a book about TEX ought to include Donald Knuth for inventing TEX in the first place. This book is no exception. Victor Eijkhout Urbana, Illinois, August 1991 Knoxville, Tennessee, May 2001 Austin, Texas, December 2013

Victor Eijkhout – TEX by Topic

21

22

Victor Eijkhout – TEX by Topic

Chapter 1 The Structure of the TEX Processor

This book treats the various aspects of TEX in chapters that are concerned with relatively small, well-delineated, topics. In this chapter, therefore, a global picture of the way TEX operates will be given. Of necessity, many details will be omitted here, but all of these are treated in later chapters. On the other hand, the few examples given in this chapter will be repeated in the appropriate places later on; they are included here to make this chapter self-contained.

1.1

Four TEX processors

The way TEX processes its input can be viewed as happening on four levels. One might say that the TEX processor is split into four separate units, each one accepting the output of the previous stage, and delivering the input for the next stage. The input of the first stage is then the .tex input file; the output of the last stage is a .dvi file. For many purposes it is most convenient, and most insightful, to consider these four levels of processing as happening after one another, each one accepting the completed output of the previous level. In reality this is not true: all levels are simultaneously active, and there is interaction between them. The four levels are (corresponding roughly to the ‘eyes’, ‘mouth’, ‘stomach’, and ‘bowels’ respectively in Knuth’s original terminology) as follows. 1.

2.

3.

The input processor. This is the piece of TEX that accepts input lines from the file system of whatever computer TEX runs on, and turns them into tokens. Tokens are the internal objects of TEX: there are character tokens that constitute the typeset text, and control sequence tokens that are commands to be processed by the next two levels. The expansion processor. Some but not all of the tokens generated in the first level – macros, conditionals, and a number of primitive TEX commands – are subject to expansion. Expansion is the process that replaces some (sequences of) tokens by other (or no) tokens. The execution processor. Control sequences that are not expandable are executable, and this execution takes place on the third level of the TEX processor. One part of the activity here concerns changes to TEX’s internal state: assignments (including macro definitions) are typical activities in this category. The other major thing happening on this level is the construction of horizontal, vertical, and mathematical lists. 23

Chapter 1. The Structure of the TEX Processor

4.

The visual processor. In the final level of processing the visual part of TEX processing is performed. Here horizontal lists are broken into paragraphs, vertical lists are broken into pages, and formulas are built out of math lists. Also the output to the dvi file takes place on this level. The algorithms working here are not accessible to the user, but they can be influenced by a number of parameters.

1.2

The input processor

The input processor of TEX is that part of TEX that translates whatever characters it gets from the input file into tokens. The output of this processor is a stream of tokens: a token list. Most tokens fall into one of two categories: character tokens and control sequence tokens. The remaining category is that of the parameter tokens; these will not be treated in this chapter. 1.2.1

Character input

For simple input text, characters are made into character tokens. However, TEX can ignore input characters: a row of spaces in the input is usually equivalent to just one space. Also, TEX itself can insert tokens that do not correspond to any character in the input, for instance the space token at the end of the line, or the \par token after an empty line. Not all character tokens signify characters to be typeset. Characters fall into sixteen categories – each one specifying a certain function that a character can have – of which only two contain the characters that will be typeset. The other categories contain such characters as {, }, &, and #. A character token can be considered as a pair of numbers: the character code – typically the ASCII code – and the category code. It is possible to change the category code that is associated with a particular character code. When the escape character (by default \ ) appears in the input, TEX’s behaviour in forming tokens is more complicated. Basically, TEX builds a control sequence by taking a number of characters from the input and lumping them together into a single token. The behaviour with which TEX’s input processor reacts to category codes can be described as a machine that switches between three internal states: N , new line; M , middle of line; S, skipping spaces. These states and the transitions between them are treated in Chapter 2. 1.2.2

Two-level input processing

TEX’s input processor is in fact itself a two-level processor. Because of limitations of the terminal, the editor, or the operating system, the user may not be able to input certain desired characters. Therefore, TEX provides a mechanism to access with two superscript characters all of the available character positions. This may be considered a separate stage of TEX processing, taking place prior to the three-state machine mentioned above. For instance, the sequence ^^+ is replaced by k because the ASCII codes of k and + differ by 64. Since this replacement takes place before tokens are formed, writing \vs^^+ip 5cm has the same effect as \vskip 5cm. Examples more useful than this exist. Note that this first stage is a transformation from characters to characters, without considering category codes. These come into play only in the second phase of input processing where characters are converted to character tokens by coupling the category code to the character code. 24

Victor Eijkhout – TEX by Topic

1.3. The expansion processor

1.3

The expansion processor

TEX’s expansion processor accepts a stream of tokens and, if possible, expands the tokens in this stream one by one until only unexpandable tokens remain. Macro expansion is the clearest example of this: if a control sequence is a macro name, it is replaced (together possibly with parameter tokens) by the definition text of the macro. Input for the expansion processor is provided mainly by the input processor. The stream of tokens coming from the first stage of TEX processing is subject to the expansion process, and the result is a stream of unexpandable tokens which is fed to the execution processor. However, the expansion processor comes into play also when (among others) an \edef or \write is processed. The parameter token list of these commands is expanded very much as if the lists had been on the top level, instead of the argument to a command. 1.3.1

The process of expansion

Expanding a token consists of the following steps: 1. See whether the token is expandable. 2. If the token is unexpandable, pass it to the token list currently being built, and take on the next token. 3. If the token is expandable, replace it by its expansion. For macros without parameters, and a few primitive commands such as \jobname, this is indeed a simple replacement. Usually, however, TEX needs to absorb some argument tokens from the stream in order to be able to form the replacement of the current token. For instance, if the token was a macro with parameters, sufficiently many tokens need to be absorbed to form the arguments corresponding to these parameters. 4. Go on expanding, starting with the first token of the expansion. Deciding whether a token is expandable is a simple decision. Macros and active characters, conditionals, and a number of primitive TEX commands (see the list on page 125) are expandable, other tokens are not. Thus the expansion processor replaces macros by their expansion, it evaluates conditionals and eliminates any irrelevant parts of these, but tokens such as \vskip and character tokens, including characters such as dollars and braces, are passed untouched. 1.3.2

Special cases: \expandafter, \noexpand, and \the

As stated above, after a token has been expanded, TEX will start expanding the resulting tokens. At first sight the \expandafter command would seem to be an exception to this rule, because it expands only one step. What actually happens is that the sequence \expandafterhtoken1 ihtoken2 i is replaced by htoken1 ihexpansion of token2 i and this replacement is in fact reexamined by the expansion processor. Real exceptions do exist, however. If the current token is the \noexpand command, the next token is considered for the moment to be unexpandable: it is handled as if it were \relax, and it is passed to the token list being built. For example, in the macro definition Victor Eijkhout – TEX by Topic

25

Chapter 1. The Structure of the TEX Processor

\edef\a{\noexpand\b} the replacement text \noexpand\b is expanded at definition time. The expansion of \noexpand is the next token, with a temporary meaning of \relax. Thus, when the expansion processor tackles the next token, the \b, it will consider that to be unexpandable, and just pass it to the token list being built, which is the replacement text of the macro. Another exception is that the tokens resulting from \thehtoken variablei are not expanded further if this statement occurs inside an \edef macro definition. 1.3.3

Braces in the expansion processor

Above, it was said that braces are passed as unexpandable character tokens. In general this is true. For instance, the \romannumeral command is handled by the expansion processor; when confronted with \romannumeral1\number\count2 3{4 ... TEX will expand until the brace is encountered: if \count2 has the value of zero, the result will be the roman numeral representation of 103. As another example, \iftrue {\else }\fi is handled by the expansion processor completely analogous to \iftrue a\else b\fi The result is a character token, independent of its category. However, in the context of macro expansion the expansion processor will recognize braces. First of all, a balanced pair of braces marks off a group of tokens to be passed as one argument. If a macro has an argument \def\macro#1{ ... } one can call it with a single token, as in \macro 1 \macro \$ or with a group of tokens, surrounded by braces \macro {abc} \macro {d{ef}g} Secondly, when the arguments for a macro with parameters are read, no expressions with unbalanced braces are accepted. In \def\a#1\stop{ ... } the argument consists of all tokens up to the first occurrence of \stop that is not in braces: in \a bc{d\stop}e\stop the argument of \a is bc{d\stop}e. Only balanced expressions are accepted here.

1.4

The execution processor

The execution processor builds lists: horizontal, vertical, and math lists. Corresponding to these lists, it works in horizontal, vertical, or math mode. Of these three modes ‘internal’ and ‘external’ 26

Victor Eijkhout – TEX by Topic

1.5. The visual processor

variants exist. In addition to building lists, this part of the TEX processor also performs modeindependent processing, such as assignments. Coming out of the expansion processor is a stream of unexpandable tokens to be processed by the execution processor. From the point of view of the execution processor, this stream contains two types of tokens: • •

Tokens signalling an assignment (this includes macro definitions), and other tokens signalling actions that are independent of the mode, such as \show and \aftergroup. Tokens that build lists: characters, boxes, and glue. The way they are handled depends on the current mode.

Some objects can be used in any mode; for instance boxes can appear in horizontal, vertical, and math lists. The effect of such an object will of course still depend on the mode. Other objects are specific for one mode. For instance, characters (to be more precise: character tokens of categories 11 and 12), are intimately connected to horizontal mode: if the execution processor is in vertical mode when it encounters a character, it will switch to horizontal mode. Not all character tokens signal characters to be typeset: the execution processor can also encounter math shift characters (by default $) and beginning/end of group characters (by default { and }). Math shift characters let TEX enter or exit math mode, and braces let it enter or exit a new level of grouping. One control sequence handled by the execution processor deserves special mention: \relax. This control sequence is not expandable, but the execution is to do nothing. Compare the effect of \relax in \count0=1\relax 2 with that of \empty defined by \def\empty{} in \count0=1\empty 2 In the first case the expansion process that is forming the number stops at \relax and the number 1 is assigned; in the second case \empty expands to nothing, so 12 is assigned.

1.5

The visual processor

TEX’s output processor encompasses those algorithms that are outside direct user control: paragraph breaking, alignment, page breaking, math typesetting, and dvi file generation. Various parameters control the operation of these parts of TEX. Some of these algorithms return their results in a form that can be handled by the execution processor. For instance, a paragraph that has been broken into lines is added to the main vertical list as a sequence of horizontal boxes with intermediate glue and penalties. Also, the page breaking algorithm stores its result in \box255, so output routines can dissect it. On the other hand, a math formula can not be broken into pieces, and, naturally, shipping a box to the dvi file is irreversible. Victor Eijkhout – TEX by Topic

27

Chapter 1. The Structure of the TEX Processor

1.6

Examples

1.6.1

Skipped spaces

Skipped spaces provide an illustration of the view that TEX’s levels of processing accept the completed input of the previous level. Consider the commands \def\a{\penalty200} \a 0 This is not equivalent to \penalty200 0 which would place a penalty of 200, and typeset the digit 0. Instead it expands to \penalty2000 because the space after \a is skipped in the input processor. Later stages of processing then receive the sequence \a0 1.6.2

Internal quantities and their representations

TEX uses various sorts of internal quantities, such as integers and dimensions. These internal quantities have an external representation, which is a string of characters, such as 4711 or 91.44cm. Conversions between the internal value and the external representation take place on two different levels, depending on what direction the conversion goes. A string of characters is converted to an internal value in assignments such as \pageno=12 \baselineskip=13pt or statements such as \vskip 5.71pt and all of these statements are handled by the execution processor. On the other hand, the conversion of the internal values into a representation as a string of characters is handled by the expansion processor. For instance, \number\pageno \romannumeral\year \the\baselineskip are all processed by expansion. As a final example, suppose \count2=45, and consider the statement \count0=1\number\count2 3 The expansion processor tackles \number\count2 to give the characters 45, and the space after the 2 does not end the number being assigned: it only serves as a delimiter of the number of the \count register. In the next stage of processing, the execution processor will then see the statement \count0=1453 and execute this.

28

Victor Eijkhout – TEX by Topic

Chapter 2 Category Codes and Internal States

When characters are read, TEX assigns them category codes. The reading mechanism has three internal states, and transitions between these states are affected by category codes of characters in the input. This chapter describes how TEX reads its input and how the category codes of characters influence the reading behaviour. Spaces and line ends are discussed. \endlinechar The character code of the end-of-line character appended to input lines. IniTEX default: 13. \par Command to close off a paragraph and go into vertical mode. Is generated by empty lines. \ignorespaces Command that reads and expands until something is encountered that is not a hspace tokeni. \catcode Query or set category codes. \ifcat Test whether two characters have the same category code. \ Control space. Insert the same amount of space that a space token would when \spacefactor = 1000. \obeylines Macro in plain TEX to make line ends significant. \obeyspaces Macro in plain TEX to make (most) spaces significant.

2.1

Introduction

TEX’s input processor scans input lines from a file or terminal, and makes tokens out of the characters. The input processor can be viewed as a simple finite state automaton with three internal states; depending on the state its scanning behaviour may differ. This automaton will be treated here both from the point of view of the internal states and of the category codes governing the transitions.

2.2

Initial processing

Input from a file (or from the user terminal, but this will not be mentioned specifically most of the time) is handled one line at a time. Here follows a discussion of what exactly is an input line for TEX. Computer systems differ with respect to the exact definition of an input line. The carriage return/ line feed sequence terminating a line is most common, but some systems use just a line feed, 29

Chapter 2. Category Codes and Internal States

and some systems with fixed record length (block) storage do not have a line terminator at all. Therefore TEX has its own way of terminating an input line. 1. 2. 3.

An input line is read from an input file (minus the line terminator, if any). Trailing spaces are removed (this is for the systems with block storage, and it prevents confusion because these spaces are hard to see in an editor). The \endlinechar is appended. By default this is character code 13, which is the ASCII code of the hreturni character. However, note that TEX is independent of ASCII. If the value of \endlinechar is negative or more than 255 (this was 127 in versions of TEX older than version 3; see page 281 for more differences), no character is appended. The effect then is the same as if the line were to end with a comment character.

Computers may also differ in the character encoding (the most common schemes are ASCII and EBCDIC ), so TEX converts the characters that are read from the file to its own character codes. These codes are then used exclusively, so that TEX will perform the same on any system. For more on this, see Chapter 3.

2.3

Category codes

Each of the 256 character codes (0–255) has an associated category code, though not necessarily always the same one. There are 16 categories, numbered 0–15. When scanning the input, TEX thus forms character-code–category-code pairs. The input processor sees only these pairs; from them are formed character tokens, control sequence tokens, and parameter tokens. These tokens are then passed to TEX’s expansion and execution processes. A character token is a character-code–category-code pair that is passed unchanged. A control sequence token consists of one or more characters preceded by an escape character; see below. Parameter tokens are also explained below. This is the list of the categories, together with a brief description. More elaborate explanations follow in this and later chapters. 0. 1. 2. 3. 4. 5.

6.

30

Escape character; this signals the start of a control sequence. IniTEX makes the backslash \ (code 92) an escape character. Beginning of group; such a character causes TEX to enter a new level of grouping. The plain format makes the open brace { a beginningof-group character. End of group; TEX closes the current level of grouping. Plain TEX has the closing brace } as end-of-group character. Math shift; this is the opening and closing delimiter for math formulas. Plain TEX uses the dollar sign $ for this. Alignment tab; the column (row) separator in tables made with \halign (\valign). In plain TEX this is the ampersand &. End of line; a character that TEX considers to signal the end of an input line. IniTEX assigns this code to character code 13, which is the ASCII value of the hreturni character. Not coincidentally, 13 is also the value that IniTEX assigns to the \endlinechar parameter; see above. Parameter character; this indicates parameters for macros. In plain TEX this is the hash sign #. Victor Eijkhout – TEX by Topic

2.3. Category codes

7.

Superscript; this precedes superscript expressions in math mode. It is also used to denote character codes that cannot be entered in an input file; see below. In plain TEX this is the circumflex ^. 8. Subscript; this precedes subscript expressions in math mode. In plain TEX the underscore _ is used for this. 9. Ignored; characters of this category are removed from the input, and have therefore no influence on further TEX processing. In plain TEX this is the hnulli character, that is, code 0. 10. Space; space characters receive special treatment. IniTEX assigns this category to the ASCII hspacei character, code 32. 11. Letter; in IniTEX only the characters a..z, A..Z are in this category. Often, macro packages make some ‘secret’ character (for instance @) into a letter. 12. Other; IniTEX puts everything that is not in the other categories into this category. Thus it includes, for instance, digits and punctuation. 13. Active; active characters function as a TEX command, without being preceded by an escape character. In plain TEX this is only the tie character ~, which is defined to produce an unbreakable space; see page 187. 14. Comment character; from a comment character onwards, TEX considers the rest of an input line to be comment and ignores it. In IniTEX the per cent sign % is made a comment character. 15. Invalid character; this category is for characters that should not appear in the input. IniTEX assigns the ASCII hdeletei character, code 127, to this category. The user can change the mapping of character codes to category codes with the \catcode command (see Chapter 36 for the explanation of concepts such as hequalsi): \catcodehnumberihequalsihnumberi. In such a statement, the first number is often given in the form ‘hcharacteri or ‘\hcharacteri both of which denote the character code of the character (see pages 45 and 80). The plain format defines \active \chardef\active=13 so that one can write statements such as \catcode‘\{=\active The \chardef command is treated on pages 46 and 81. The LATEX format has the control sequences \def\makeatletter{\catcode‘@=11 } \def\makeatother{\catcode‘@=12 } in order to switch on and off the ‘secret’ character @ (see below). The \catcode command can also be used to query category codes: in \count255=\catcode‘\{ it yields a number, which can be assigned. Category codes can be tested by \ifcathtoken1 ihtoken2 i Victor Eijkhout – TEX by Topic

31

Chapter 2. Category Codes and Internal States

TEX expands whatever is after \ifcat until two unexpandable tokens are found; these are then compared with respect to their category codes. Control sequence tokens are considered to have category code 16, which makes them all equal to each other, and unequal to all character tokens. Conditionals are treated further in Chapter 13.

2.4

From characters to tokens

The input processor of TEX scans input lines from a file or from the user terminal, and converts the characters in the input to tokens. There are three types of tokens. • •

•

2.5

Character tokens: any character that is passed on its own to TEX’s further levels of processing with an appropriate category code attached. Control sequence tokens, of which there are two kinds: an escape character – that is, a character of category 0 – followed by a string of ‘letters’ is lumped together into a control word, which is a single token. An escape character followed by a single character that is not of category 11, letter, is made into a control symbol . If the distinction between control word and control symbol is irrelevant, both are called control sequence. The control symbol that results from an escape character followed by a space character is called control space. Parameter tokens: a parameter character – that is, a character of category 6, by default # – followed by a digit 1..9 is replaced by a parameter token. Parameter tokens are allowed only in the context of macros (see Chapter 11). A macro parameter character followed by another macro parameter character (not necessarily with the same character code) is replaced by a single character token. This token has category 6 (macro parameter), and the character code of the second parameter character. The most common instance is of this is replacing ## by #6 , where the subscript denotes the category code.

The input processor as a finite state automaton

TEX’s input processor can be considered to be a finite state automaton with three internal states, that is, at any moment in time it is in one of three states, and after transition to another state there is no memory of the previous states. 2.5.1

State N: new line

State N is entered at the beginning of each new input line, and that is the only time TEX is in this state. In state N all space tokens (that is, characters of category 10) are ignored; an end-of-line character is converted into a \par token. All other tokens bring TEX into state M. 2.5.2

State S: skipping spaces

State S is entered in any mode after a control word or control space (but after no other control symbol), or, when in state M, after a space. In this state all subsequent spaces or end-of-line characters in this input line are discarded. 32

Victor Eijkhout – TEX by Topic

2.6. Accessing the full character set

2.5.3

State M: middle of line

By far the most common state is M, ‘middle of line’. It is entered after characters of categories 1–4, 6–8, and 11–13, and after control symbols other than control space. An end-of-line character encountered in this state results in a space token. .

start −−− −−−→

10

.

-

N

−−other −−−−→

??

14 5: insert \par

?? stop

2.6

other

-

10 −−− −−−−−−−−−−−−→ 0+11,11, . . . −−−−−−−−−−−−−−−→ 0+10 −−− −−−−−−−−−−−−→ . other ?? 14 5: insert space M

?? stop

.

10

-

S

??

14 5

?? stop

Accessing the full character set

Strictly speaking, TEX’s input processor is not a finite state automaton. This is because during the scanning of the input line all trios consisting of two equal superscript characters (category code 7) and a subsequent character (with character code < 128) are replaced by a single character with a character code in the range 0–127, differing by 64 from that of the original character. This mechanism can be used, for instance, to access positions in a font corresponding to character codes that cannot be input, for instance because they are ASCII control characters. The most obvious examples are the ASCII hreturni and hdeletei characters; the corresponding positions 13 and 127 in a font are accessible as ^^M and ^^?. However, since the category of ^^? is 15, invalid, that has to be changed before character 127 can be accessed. In TEX3 this mechanism has been modified and extended to access 256 characters: any quadruplet ^^xy where both x and y are lowercase hexadecimal digits 0–9, a–f, is replaced by a character in the range 0–255, namely the character the number of which is represented hexadecimally as xy. This imposes a slight restriction on the applicability of the earlier mechanism: if, for instance, ^^a is typed to produce character 33, then a following 0–9, a–f will be misunderstood. While this process makes TEX’s input processor somewhat more powerful than a true finite state automaton, it does not interfere with the rest of the scanning. Therefore it is conceptually simpler to pretend that such a replacement of triplets or quadruplets of characters, starting with ^^, is performed in advance. In actual practice this is not possible, because an input line may assign category code 7 to some character other than the circumflex, thereby influencing its further processing.

2.7

Transitions between internal states

Let us now discuss the effects on the internal state of TEX’s input processor when certain category codes are encountered in the input. Victor Eijkhout – TEX by Topic

33

Chapter 2. Category Codes and Internal States

2.7.1

0: escape character

When an escape character is encountered, TEX starts forming a control sequence token. Three different types of control sequence can result, depending on the category code of the character that follows the escape character. •

• •

If the character following the escape is of category 11, letter, then TEX combines the escape, that character and all following characters of category 11, into a control word. After that TEX goes into state S, skipping spaces. With a character of category 10, space, a control symbol called control space results, and TEX goes into state S. With a character of any other category code a control symbol results, and TEX goes into state M, middle of line.

The letters of a control sequence name have to be all on one line; a control sequence name is not continued on the next line if the current line ends with a comment sign, or if (by letting \endlinechar be outside the range 0–255) there is no terminating character. 2.7.2

1–4, 7–8, 11–13: non-blank characters

Characters of category codes 1–4, 7–8, and 11–13 are made into tokens, and TEX goes into state M. 2.7.3

5: end of line

Upon encountering an end-of-line character, TEX discards the rest of the line, and starts processing the next line, in state N. If the current state was N, that is, if the line so far contained at most spaces, a \par token is inserted; if the state was M, a space token is inserted, and in state S nothing is inserted. Note that by ‘end-of-line character’ a character with category code 5 is meant. This is not necessarily the \endlinechar, nor need it appear at the end of the line. See below for further remarks on line ends.

2.7.4

6: parameter

A parameter character – usually # – can be followed by either a digit 1..9 in the context of macro definitions or by another parameter character. In the first case a ‘parameter token’ results, in the second case only a single parameter character is passed on as a character token for further processing. In either case TEX goes into state M. A parameter character can also appear on its own in an alignment preamble (see Chapter 25).

2.7.5

7: superscript

A superscript character is handled like most non-blank characters, except in the case where it is followed by a superscript character of the same character code. The process that replaces these two characters plus the following character (possibly two characters in TEX3) by another character was described above. 34

Victor Eijkhout – TEX by Topic

2.8. Letters and other characters

2.7.6

9: ignored character

Characters of category 9 are ignored; TEX remains in the same state. 2.7.7

10: space

A token with category code 10 – this is called a hspace tokeni, irrespective of the character code – is ignored in states N and S (and the state does not change); in state M TEX goes into state S, inserting a token that has category 10 and character code 32 (ASCII space). This implies that the character code of the space token may change from the character that was actually input. 2.7.8

14: comment

A comment character causes TEX to discard the rest of the line, including the comment character. In particular, the end-of-line character is not seen, so even if the comment was encountered in state M, no space token is inserted. 2.7.9

15: invalid

Invalid characters cause an error message. TEX remains in the state it was in. However, in the context of a control symbol an invalid character is acceptable. Thus \^^? does not cause any error messages.

2.8

Letters and other characters

In most programming languages identifiers can consist of both letters and digits (and possibly some other character such as the underscore), but control sequences in TEX are only allowed to be formed out of characters of category 11, letter. Ordinarily, the digits and punctuation symbols have category 12, other character. However, there are contexts where TEX itself generates a string of characters, all of which have category code 12, even if that is not their usual category code. This happens when the operations \string, \number, \romannumeral, \jobname, \fontname, \meaning, and \the are used to generate a stream of character tokens. If any of the characters delivered by such a command is a space character (that is, character code 32), it receives category code 10, space. For the extremely rare case where a hexadecimal digit has been hidden in a control sequence, TEX allows A12 –F12 to be hexadecimal digits, in addition to the ordinary A11 –F11 (here the subscripts denote the category codes). For example, \string\end gives four character tokens \12 e12 n12 d12 Note that the escape character \12 is used in the output only because the value of \escapechar is the character code for the backslash. Another value of \escapechar leads to another character in the output of \string. The \string command is treated further in Chapter 3. Spaces can wind up in control sequences: \csname a b\endcsname Victor Eijkhout – TEX by Topic

35

Chapter 2. Category Codes and Internal States

gives a control sequence token in which one of the three characters is a space. Turning this control sequence token into a string of characters \expandafter\string\csname a b\endcsname gives \12 a12 10 b12 . As a more practical example, suppose there exists a sequence of input files file1.tex, file2.tex, and we want to write a macro that finds the number of the input file that is being processed. One approach would be to write \newcount\filenumber \def\getfilenumber file#1.{\filenumber=#1 } \expandafter\getfilenumber\jobname. where the letters file in the parameter text of the macro (see Section 11.5) absorb that part of the jobname, leaving the number as the sole parameter. However, this is slightly incorrect: the letters file resulting from the \jobname command have category code 12, instead of 11 for the ones in the definition of \getfilenumber. This can be repaired as follows: {\escapechar=-1 \expandafter\gdef\expandafter\getfilenumber \string\file#1.{\filenumber=#1 } } Now the sequence \string\file gives the four letters f12 i12 l12 e12 ; the \expandafter commands let this be executed prior to the macro definition; the backslash is omitted because we put \escapechar=-1. Confining this value to a group makes it necessary to use \gdef.

2.9

The \par token

TEX inserts a \par token into the input after an empty line, that is, when encountering a character with category code 5, end of line, in state N. It is good to realize when exactly this happens: since TEX leaves state N when it encounters any token but a space, a line giving a \par can only contain characters of category 10. In particular, it cannot end with a comment character. Quite often this fact is used the other way around: if an empty line is wanted for the layout of the input one can put a comment sign on that line. Two consecutive empty lines generate two \par tokens. For all practical purposes this is equivalent to one \par, because after the first one TEX enters vertical mode, and in vertical mode a \par only exercises the page builder, and clears the paragraph shape parameters. A \par is also inserted into the input when TEX sees a hvertical commandi in unrestricted horizontal mode. After the \par has been read and expanded, the vertical command is examined anew (see Chapters 6 and 17). The \par token may also be inserted by the \end command that finishes off the run of TEX; see Chapter 28. It is important to realize that TEX does what it normally does when encountering an empty line (which is ending a paragraph) only because of the default definition of the \par token. By redefining \par the behaviour caused by empty lines and vertical commands can be changed completely, and interesting special effects can be achieved. In order to continue to be able to cause the actions 36

Victor Eijkhout – TEX by Topic

2.10. Spaces

normally associated with \par, the synonym \endgraf is available in the plain format. See further Chapter 17. The \par token is not allowed to be part of a macro argument, unless the macro has been declared to be \long. A \par in the argument of a non-\long macro prompts TEX to give a ‘runaway argument’ message. Control sequences that have been \let to \par (such as \endgraf) are allowed, however.

2.10

Spaces

This section treats some of the aspects of the space character and space token in the initial processing stages of TEX. The topic of spacing in text typesetting is treated in Chapter 20. 2.10.1

Skipped spaces

From the discussion of the internal states of TEX’s input processor it is clear that some spaces in the input never reach the output; in fact they never get past the input processor. These are for instance the spaces at the beginning of an input line, and the spaces following the one that lets TEX switch to state S. On the other hand, line ends can generate spaces (which are not in the input) that may wind up in the output. There is a third kind of space: the spaces that get past the input processor, or are even generated there, but still do not wind up in the output. These are the hoptional spacesi that the syntax of TEX allows in various places. 2.10.2

Optional spaces

The syntax of TEX has the concepts of optional spaces and ‘one optional space’: hone optional spacei −→ hspace tokeni | hemptyi hoptional spacesi −→ hemptyi | hspace tokenihoptional spacesi In general, hone optional spacei is allowed after numbers and glue specifications, while hoptional spacesi are allowed whenever a space can occur inside a number (for example, between a minus sign and the digits of the number) or glue specification (for example, between plus and 1fil). Also, the definition of hequalsi allows hoptional spacesi before the = sign. Here are some examples of optional spaces. • A number can be delimited by hone optional spacei. This prevents accidents (see Chapter 7), and it speeds up processing, as TEX can detect more easily where the hnumberi being read ends. Note, however, that not every ‘number’ is a hnumberi: for instance the 2 in \magstep2 is not a number, but the single token that is the parameter of the \magstep macro. Thus a space or line end after this is significant. Another example is a parameter number, for example #1: since at most nine parameters are allowed, scanning one digit after the parameter character suffices. • From the grammar of TEX it follows that the keywords fill and filll consist of fil and separate l s, each of which is a keyword (see page 280 for a more elaborate discussion), and hence can be followed by optional spaces. Therefore forms such as fil L l are also valid. This is a potential source of strange accidents. In most cases, appending a \relax token prevents such mishaps. Victor Eijkhout – TEX by Topic

37

Chapter 2. Category Codes and Internal States

•

The primitive command \ignorespaces may come in handy as the final command in a macro definition. As it gobbles up optional spaces, it can be used to prevent spaces following the closing brace of an argument from winding up in the output inadvertently. For example, in \def\item#1{\par\leavevmode \llap{#1\enspace}\ignorespaces} \item{a/}one line \item{b/} another line \item{c/} yet another the \ignorespaces prevents spurious spaces in the second and third item. An empty line after \ignorespaces will still insert a \par, however.

2.10.3

Ignored and obeyed spaces

After control words spaces are ignored. This is not an instance of optional spaces, but it is due to the fact that TEX goes into state S, skipping spaces, after control words. Similarly an end-of-line character is skipped after a control word. Numbers are delimited by only hone optional spacei, but still a\count0=3 b gives ‘ab’, because TEX goes into state S after the first space token. The second space is therefore skipped in the input processor of TEX; it never becomes a space token. Spaces are skipped furthermore when TEX is in state N, newline. When TEX is processing in vertical mode space tokens (that is, spaces that were not skipped) are ignored. For example, the space inserted (because of the line end) after the first box in \par \hbox{a} \hbox{b} has no effect. Both plain TEX and LATEX define a command \obeyspaces that makes spaces significant: after one space other spaces are no longer ignored. In both cases the basis is \catcode‘\ =13 \def {\space} However, there is a difference between the two cases: in plain TEX \def\space{ } while in LATEX \def\space{\leavevmode{} } although the macros bear other names there. The difference between the two macros becomes apparent in the context of \obeylines: each line end is then a \par command, implying that each next line is started in vertical mode. An active space is expanded by the plain macro to a space token, which is ignored in vertical mode. The active spaces in LATEX will immediately switch to horizontal mode, so that each space is significant. 2.10.4

More ignored spaces

There are three further places where TEX will ignore space tokens. 38

Victor Eijkhout – TEX by Topic

2.10. Spaces

1. 2. 3.

When TEX is looking for an undelimited macro argument it will accept the first token (or group) that is not a space. This is treated in Chapter 11. In math mode space tokens are ignored (see Chapter 23). After an alignment tab character spaces are ignored (see Chapter 25).

2.10.5

hspace tokeni

Spaces are anomalous in TEX. For instance, the \string operation assigns category code 12 to all characters except spaces; they receive category 10. Also, as was said above, TEX’s input processor converts (when in state M) all tokens with category code 10 into real spaces: they get character code 32. Any character token with category 10 is called hspace tokenispace! token. Space tokens with character code not equal to 32 are called funny spaces. After giving the character Q the category code of a space character, and using it in a definition \catcode‘Q=10 \def\q{aQb} we get \show\q macro:-> a b because the input processor changes the character code of the funny space in the definition. Space tokens with character codes other than 32 can be created using, for instance, \uppercase. However, ‘since the various forms of space tokens are almost identical in behaviour, there’s no point dwelling on the details’; see [27] p. 377. 2.10.6

Control space

The ‘control space’ command \ contributes the amount of space that a hspace tokeni would when the \spacefactor is 1000. A control space is not treated like a space token, or like a macro expanding to one (which is how \space is defined in plain TEX). For instance, TEX ignores spaces at the beginning of an input line, but control space is a hhorizontal commandi, so it makes TEX switch from vertical to horizontal mode (and insert an indentation box). See Chapter 20 for the space factor, and chapter 6 for horizontal and vertical modes. 2.10.7

‘ ’

The explicit symbol ‘ ’ for a space is character 32 in the Computer Modern typewriter typeface. However, switching to \tt is not sufficient to get spaces denoted this way, because spaces will still receive special treatment in the input processor. One way to let spaces be typeset by

is to set

\catcode‘\ =12 TEX will then take a space as the instruction to typeset character number 32. Moreover, subsequent spaces are not skipped, but also typeset this way: state S is only entered after a character with category code 10. Similarly, spaces after a control sequence are made visible by changing the category code of the space character. Victor Eijkhout – TEX by Topic

39

Chapter 2. Category Codes and Internal States

2.11

More about line ends

TEX accepts lines from an input file, excluding any line terminator that may be used. Because of this, TEX’s behaviour here is not dependent on the operating system and the line terminator it uses (CR-LF, LF, or none at all for block storage). From the input line any trailing spaces are removed. The reason for this is historic; it has to do with the block storage mode on IBM mainframe computers. For some computer-specific problems with end-of-line characters, see [2]. A terminator character is then appended with a character code of \endlinechar, unless this parameter has a value that is negative or more than 255. Note that this terminator character need not have category code 5, end of line. 2.11.1

Obeylines

Every once in a while it is desirable that the output uses the same line breaking as the input. The following piece of code does the trick: \catcode‘\^^M=13 % \def^^M{\par}% The \endlinechar character is here made active, and its meaning becomes \par. The comment signs prevent TEX from seeing the terminator of the lines of this definition, and expanding it since it is active. However, it takes some care to embed this code in a macro. The definition \def\obeylines{\catcode‘\^^M=13 \def^^M{\par}} will be misunderstood: TEX will discard everything after the second ^^M, because this has category code 5. Effectively, this line is then \def\obeylines{\catcode‘\^^M=13 \def To remedy this, the definition itself has to be performed in a context where ^^M is an active character: {\catcode‘\^^M=13 % \gdef\obeylines{\catcode‘\^^M=13 \def^^M{\par}}% } Empty lines in the input are not taken into account in this definition: these disappear, because two consecutive \par tokens are (in this case) equivalent to one. A slightly modified definition for the line end as \def^^M{\par\leavevmode} remedies this: now every line end forces TEX to start a paragraph. For empty lines this will then be an empty paragraph. 2.11.2

Changing the \endlinechar

Occasionally you may want to change the \endlinechar, or the \catcode of the ordinary line terminator ^^M, for instance to obtain special effects such as macros where the argument is terminated by the line end. See page 121 for a worked-out example. There are a couple of traps. Consider the following: 40

Victor Eijkhout – TEX by Topic

2.12. More about the input processor

{\catcode‘\^^M=12 \endlinechar=‘\^^J \catcode‘\^^J=5 ... ... } This causes unintended output of both character 13 (^^M) and 10 (^^J), caused by the line terminators of the first and last line. Terminating the first and last line with a comment works, but replacing the first line by the two lines {\endlinechar=‘\^^J \catcode‘\^^J=5 \catcode‘\^^M=12 is also a solution. Of course, in many cases it is not necessary to substitute another end-of-line character; a much simpler solution is then to put \endlinechar=-1 which treats all lines as if they end with a comment. 2.11.3

More remarks about the end-of-line character

The character that TEX appends at the end of an input line is treated like any other character. Usually one is not aware of this, as its category code is special, but there are a few ways to let it be processed in an unusual way. Terminating an input line with ^^ will (ordinarily, when \endlinechar is 13) give ‘M’ in the output, which is the ASCII character with code 13+64. If \^^M has been defined, terminating an input line with a backslash will execute this command. The plain format defines \def\^^M{\ } which makes a ‘control return’ equivalent to a control space.

2.12

More about the input processor

2.12.1

The input processor as a separate process

TEX’s levels of processing are all working at the same time and incrementally, but conceptually they can often be considered to be separate processes that each accept the completed output of the previous stage. The juggling with spaces provides a nice illustration for this. Consider the definition \def\DoAssign{\count42=800} and the call \DoAssign 0 The input processor, the part of TEX that builds tokens, in scanning this call skips the space before the zero, so the expansion of this call is \count42=8000 Victor Eijkhout – TEX by Topic

41

Chapter 2. Category Codes and Internal States

It would be incorrect to reason ‘\DoAssign is read, then expanded, the space delimits the number 800, so 800 is assigned and the zero is printed’. Note that the same would happen if the zero appeared on the next line. Another illustration shows that optional spaces appear in a different stage of processing from that for skipped spaces: \def\c.{\relax} a\c. b expands to a\relax b which gives as output ‘a b’ because spaces after the \relax control sequence are only skipped when the line is first read, not when it is expanded. The fragment \def\c.{\ignorespaces} a\c. b on the other hand, expands to a\ignorespaces b Executing the \ignorespaces command removes the subsequent space token, so the output is ‘ab’. In both definitions the period after \c is a delimiting token; it is used here to prevent spaces from being skipped. 2.12.2

The input processor not as a separate process

Considering the tokenizing of TEX to be a separate process is a convenient view, but sometimes it leads to confusion. The line \catcode‘\^^M=13{} makes the line end active, and subsequently gives an ‘undefined control sequence’ error for the line end of this line itself. Execution of the commands on the line thus influences the scanning process of that same line. By contrast, \catcode‘\^^M=13 does not give an error. The reason for this is that TEX reads the line end while it is still scanning the number 13; that is, at a time when the assignment has not been performed yet. The line end is then converted to the optional space character delimiting the number to be assigned. 2.12.3

Recursive invocation of the input processor

Above, the activity of replacing a parameter character plus a digit by a parameter token was described as something similar to the lumping together of letters into a control sequence token. Reality is somewhat more complicated than this. TEX’s token scanning mechanism is invoked both for input from file and for input from lists of tokens such as the macro definition. Only in the first case is the terminology of internal states applicable. 42

Victor Eijkhout – TEX by Topic

2.13. The @ convention

Macro parameter characters are treated the same in both cases, however. If this were not the case it would not be possible to write things such as \def\a{\def\b{\def\c####1{####1}}} See page 114 for an explanation of such nested definitions.

2.13

The @ convention

Anyone who has ever browsed through either the plain format or the LATEX format will have noticed that a lot of control sequences contain an ‘at’ sign: @. These are control sequences that are meant to be inaccessible to the ordinary user. Near the beginning of the format files the instruction \catcode‘@=11 occurs, making the at sign into a letter, meaning that it can be used in control sequences. Somewhere near the end of the format definition the at sign is made ‘other’ again: \catcode‘@=12 Now why is it that users cannot call a control sequence with an at sign directly, although they can call macros that contain lots of those ‘at-definitions’? The reason is that the control sequences containing an @ are internalized by TEX at definition time, after which they are a token, not a string of characters. Macro expansion then just inserts such tokens, and at that time the category codes of the constituent characters do not matter any more.

Victor Eijkhout – TEX by Topic

43

Chapter 2. Category Codes and Internal States

44

Victor Eijkhout – TEX by Topic

Chapter 3 Characters

Internally, TEX represents characters by their (integer) character code. This chapter treats those codes, and the commands that have access to them. \char Explicit denotation of a character to be typeset. \chardef Define a control sequence to be a synonym for a character code. \accent Command to place accent characters. \if Test equality of character codes. \ifx Test equality of both character and category codes. \let Define a control sequence to be a synonym of a token. \uccode Query or set the character code that is the uppercase variant of a given code. \lccode Query or set the character code that is the lowercase variant of a given code. \uppercase Convert the hgeneral texti argument to its uppercase form. \lowercase Convert the hgeneral texti argument to its lowercase form. \string Convert a token to a string of one or more characters. \escapechar Number of the character that is to be used for the escape character when control sequences are being converted into character tokens. IniTEX default: 92 (\).

3.1

Character codes

Conceptually it is easiest to think that TEX works with characters internally, but in fact TEX works with integers: the character codes. The way characters are encoded in a computer may differ from system to system. Therefore TEX uses its own scheme of character codes. Any character that is read from a file (or from the user terminal) is converted to a character code according to the character code table. A category code is then assigned based on this (see Chapter 2). The character code table is based on the 7-bit ASCII table for numbers under 128 (see Section 38.1). There is an explicit conversion between characters (better: character tokens) and character codes using the left quote (grave, back quote) character ‘: at all places where TEX expects a hnumberi you can use the left quote followed by a character token or a single-character control sequence. Thus both \count‘a and \count‘\a are synonyms for \count97. See also Chapter 7. The possibility of a single-character control sequence is necessary in certain cases such as 45

Chapter 3. Characters

\catcode‘\%=11 or \def\CommentSign{\char‘\%} which would be misunderstood if the backslash were left out. For instance \catcode‘%=11 would consider the =11 to be a comment. Single-character control sequences can be formed from characters with any category code. After the conversion to character codes any connection with external representations has disappeared. Of course, for most characters the visible output will ‘equal’ the input (that is, an ‘a’ causes an ‘a’). There are exceptions, however, even among the common symbols. In the Computer Modern roman fonts there are no ‘less than’ and ‘greater than’ signs, so the input ‘’ will give ‘¡¿’ in the output. In order to make TEX machine independent at the output side, the character codes are also used in the dvi file: opcodes n = 0 . . . 127 denote simply the instruction ‘take character n from the current font’. The complete definition of the opcodes in a dvi file can be found in [25].

3.2

Control sequences for characters

There are a number of ways in which a control sequence can denote a character. The \char command specifies a character to be typeset; the \let command introduces a synonym for a character token, that is, the combination of character code and category code. 3.2.1

Denoting characters to be typeset: \char

The \char command can be used to generate characters in the output that are difficult to have in the input. This can be because they are TEX special characters, or because their font position corresponds to an unprintable ascii character. Characters can be denoted numerically by, for example, \char98. This command tells TEX to add character number 98 of the current font to the horizontal list currently under construction. Instead of decimal notation, it is often more convenient to use octal or hexadecimal notation. For octal the single quote is used: \char’142; hexadecimal uses the double quote: \char"62. Note that using two single quotes, as \char’’62, is incorrect; the process that replaces two quotes by a double quote works at a later stage of processing (the visual processor) than number scanning (the execution processor). Because of the explicit conversion to character codes by the back quote character it is also possible to get a ‘b’ – provided that you are using a font organized a bit like the ASCII table – with \char‘b or \char‘\b. The \char command looks superficially a bit like the ^^ substitution mechanism (Chapter 2). Both mechanisms access characters without directly denoting them. However, the ^^ mechanism operates in a very early stage of processing (in the input processor of TEX, but before category code assignment); the \char command, on the other hand, comes in the final stages of processing. In effect it says ‘typeset character number so-and-so’. There is a construction to let a control sequence stand for some character code: the \chardef command. The syntax of this is 46

Victor Eijkhout – TEX by Topic

3.2. Control sequences for characters

\chardefhcontrol sequenceihequalsihnumberi, where the number can be an explicit representation or a counter value, but it can also be a character code obtained using the left quote command (see above; the full definition of hnumberi is given in Chapter 7). In the plain format the latter possibility is used in definitions such as \chardef\%=‘\% which could have been given equivalently as \chardef\%=37 After this command, the control symbol \% used on its own is a synonym for \char37, that is, the command to typeset character 37 (usually the per cent character). A control sequence that has been defined with a \chardef command can also be used as a hnumberi. This fact is used in allocation commands such as \newbox (see Chapters 7 and 31). Tokens defined with \mathchardef can also be used this way. 3.2.2

Implicit character tokens: \let

Another construction defining a control sequence to stand for (among other things) a character is \let: \lethcontrol sequenceihequalsihtokeni with a character token on the right hand side of the (optional) equals sign. The result is called an implicit character token. (See page 116 for a further discussion of \let.) In the plain format there are for instance synonyms for the open and close brace: \let\bgroup={ \let\egroup=} The resulting control sequences are called ‘implicit braces’ (see Chapter 10). Assigning characters by \let is different from defining control sequences by \chardef, in the sense that \let makes the control sequence stand for the combination of a character code and category code. As an example \catcode‘|=2 % make the bar an end of group \let\b=| % make \b a bar character {\def\m{...}\b \m gives an ‘undefined control sequence \m’ because the \b closed the group inside which \m was defined. On the other hand, \let\b=| % make \b a bar character \catcode‘|=2 % make the bar character end of group {\def\m{...}\b \m leaves one group open, and it prints a vertical bar (or whatever is in position 124 of the current font). The first of these examples implies that even when the braces have been redefined (for instance into active characters for macros that format C code) the beginning-of-group and end-ofgroup functionality is available through the control sequences \bgroup and \egroup. Here is another example to show that implicit character tokens are hard to distinguish from real character tokens. After the above sequence Victor Eijkhout – TEX by Topic

47

Chapter 3. Characters

\catcode‘|=2 \let\b=| the tests \if\b| and \ifcat\b} are both true. Yet another example can be found in the plain format: the commands \let\sp=^ \let\sb=_ allow people without an underscore or circumflex on their keyboard to make sub- and superscripts in mathematics. For instance: x\sp2\sb{ij} gives x2ij If a person typing in the format itself does not have these keys, some further tricks are needed: {\lccode‘,=94 \lccode‘.=95 \catcode‘,=7 \catcode‘.=8 \lowercase{\global\let\sp=, \global\let\sb=.}} will do the job; see below for an explanation of lowercase codes. The ^^ method as it was in TEX version 2 (see page 33) cannot be used here, as it would require typing two characters that can ordinarily not be input. With the extension in TEX version 3 it would also be possible to write {\catcode‘\,=7 \global\let\sp=,,5e \global\let\sb=,,5f} denoting the codes 94 and 95 hexadecimally. Finding out just what a control sequence has been defined to be with \let can be done using \meaning: the sequence \let\x=3 \meaning\x gives ‘the character 3’.

3.3

Accents

Accents can be placed by the hhorizontal commandi \accent : \accenth8-bit numberihoptional assignmentsihcharacteri where hcharacteri is a character of category 11 or 12, a \charh8-bit numberi command, or a \chardef token. If none of these four types of hcharacteri follows, the accent is taken to be a \char command itself; this gives an accent ‘suspended in mid-air’. Otherwise the accent is placed on top of the following character. Font changes between the accent and the character can be effected by the hoptional assignmentsi. An unpleasant implication of the fact that an \accent command has to be followed by a hcharacteri is that it is not possible to place an accent on a ligature, or two accents on top of each other. In some languages, such as Hindi or Vietnamese, such double accents do occur. Positioning accents on top of each other is possible, however, in math mode. The width of a character with an accent is the same as that of the unaccented character. TEX assumes that the accent as it appears in the font file is properly positioned for a character that is 48

Victor Eijkhout – TEX by Topic

3.4. Testing characters

as high as the x-height of the font; for characters with other heights it correspondingly lowers or raises the accent. No genuine under-accents exist in TEX. They are implemented as low placed over-accents. A way of handling them more correctly would be to write a macro that measures the following character, and raises or drops the accent accordingly. The cedilla macro \c in plain TEX does something along these lines. However, it does not drop the accent for characters with descenders. The horizontal positioning of an accent is controlled by \fontdimen1, slant per point . Kerns are used for the horizontal movement. Note that, although they are inserted automatically, these kerns are classified as explicit kerns. Therefore they inhibit hyphenation in the parts of the word before and after the kern. As an example of kerning for accents, here follows the dump of a horizontal list. \setbox0=\hbox{\it \‘l} \showbox0 gives \hbox(9.58334+0.0)x2.55554 .\kern -0.61803 (for accent) .\hbox(6.94444+0.0)x5.11108, shifted -2.6389 ..\tenit ^^R .\kern -4.49306 (for accent) .\tenit l Note that the accent is placed first, so afterwards the italic correction of the last character is still available.

3.4

Testing characters

Equality of character codes is tested by \if: \ifhtoken1 ihtoken2 i Tokens following this conditional are expanded until two unexpandable tokens are left. The condition is then true if those tokens are character tokens with the same character code, regardless of category code. An unexpandable control sequence is considered to have character code 256 and category code 16 (so that it is unequal to anything except another control sequence), except in the case where it had been \let to a non-active character token. In that case it is considered to have the character code and category code of that character. This was mentioned above. The test \ifcat for category codes was mentioned in Chapter 2; the test \ifxhtoken1 ihtoken2 i can be used to test for category code and character code simultaneously. The tokens following this test are not expanded. However, if they are macros, TEX tests their expansions for equality. Quantities defined by \chardef can be tested with \ifnum: \chardef\a=‘x \chardef\b=‘y \ifnum\a=\b % is false based on the fact (see Chapter 7) that hchardef tokenis can be used as numbers. See also section 13.2 Victor Eijkhout – TEX by Topic

49

Chapter 3. Characters

3.5

Uppercase and lowercase

3.5.1

Uppercase and lowercase codes

To each of the character codes correspond an uppercase code and a lowercase code (for still more codes see below). These can be assigned by \uccodehnumberihequalsihnumberi and \lccodehnumberihequalsihnumberi. In IniTEX codes ‘a..‘z, ‘A..‘Z have uppercase code ‘A..‘Z and lowercase code ‘a..‘z. All other character codes have both uppercase and lowercase code zero. 3.5.2

Uppercase and lowercase commands

The commands \uppercase{...} and \lowercase{...} go through their argument lists, replacing all character codes of explicit character tokens by their uppercase and lowercase code respectively if these are non-zero, without changing the category codes. The argument of \uppercase and \lowercase is a hgeneral texti, which is defined as hgeneral texti −→ hfilleri{hbalanced textihright bracei (for the definition of hfilleri see Chapter 36) meaning that the left brace can be implicit, but the closing right brace must be an explicit character token with category code 2. TEX performs expansion to find the opening brace. Uppercasing and lowercasing are executed in the execution processor; they are not ‘macro expansion’ activities like \number or \string. The sequence (attempting to produce \A) \expandafter\csname\uppercase{a}\endcsname gives an error (TEX inserts an \endcsname before the \uppercase because \uppercase is unexpandable), but \uppercase{\csname a\endcsname} works. As an example of the correct use of \uppercase, here is a macro that tests if a character is uppercase: \def\ifIsUppercase#1{\uppercase{\if#1}#1} The same test can be performed by \ifnum‘#1=\uccode‘#1. Hyphenation of words starting with an uppercase character, that is, a character not equal to its own \lccode, is subject to the \uchyph parameter: if this is positive, hyphenation of capitalized words is allowed. See also Chapter 19. 3.5.3

Uppercase and lowercase forms of keywords

Each character in TEX keywords, such as pt, can be given in uppercase or lowercase form. For instance, pT, Pt, pt, and PT all have the same meaning. TEX does not use the \uccode and \lccode tables here to determine the lowercase form. Instead it converts uppercase characters to lowercase by adding 32 – the ASCII difference between uppercase and lowercase characters – to their character code. This has some implications for implementations of TEX for non-roman alphabets; see page 370 of the TEX book, [27]. 50

Victor Eijkhout – TEX by Topic

3.6. Codes of a character

3.5.4

Creative use of \uppercase and \lowercase

The fact that \uppercase and \lowercase do not change category codes can sometimes be used to create certain character-code–category-code combinations that would otherwise be difficult to produce. See for instance the explanation of the \newif macro in Chapter 13, and another example on page 48. For a slightly different application, consider the problem (solved by Rainer Sch¨opf) of, given a counter \newcount\mycount, writing character number \mycount to the terminal. Here is a solution: \lccode‘a=\mycount \chardef\terminal=16 \lowercase{\write\terminal{a}} The \lowercase command effectively changes the argument of the \write command from ‘a’ into whatever it should be.

3.6

Codes of a character

Each character code has a number of hcodenameis associatedcodenames with it. These are integers in various ranges that determine how the character is treated in various contexts, or how the occurrence of that character changes the workings of TEX in certain contexts. The code names are as follows: \catcode h4-bit numberi (0–15); the category to which a character belongs. This is treated in Chapter 2. \mathcode h15-bit numberi (0–"7FFF) or "8000; determines how a character is treated in math mode. See Chapter 21. \delcode h27-bit numberi (0–"7 FFF FFF); determines how a character is treated after \left or \right in math mode. See page 193. \sfcode integer; determines how spacing is affected after this character. See Chapter 20. \lccode, \uccode h8-bit numberi (0-255); lowercase and uppercase codes – these were treated above.

3.7

Converting tokens into character strings

The command \string takes the next token and expands it into a string of separate characters. Thus \tt\string\control will give \control in the output, and \tt\string$ will give $, but, noting that the string operation comes after the tokenizing, \tt\string% will not give %, because the comment sign is removed by TEX’s input processor. Therefore, this command will ‘string’ the first token on the next line. The \string command is executed by the expansion processor, thus it is expanded unless explicitly inhibited (see Chapter 12). Victor Eijkhout – TEX by Topic

51

Chapter 3. Characters

3.7.1

Output of control sequences

In the above examples the typewriter font was selected, because the Computer Modern roman font does not have a backslash character. However, TEX need not have used the backslash character to display a control sequence: it uses character number \escapechar. This same value is also used when a control sequence is output with \write, \message, or \errmessage, and it is used in the output of \show, \showthe and \meaning. If \escapechar is negative or more than 255, the escape character is not output; the default value (set in IniTEX) is 92, the number of the backslash character. For use in a \write statement the \string can in some circumstances be replaced by \noexpand (see page 134). 3.7.2

Category codes of a \string

The characters that are the result of a \string command have category code 12, except for any spaces in a stringed control sequence; they have category code 10. Since inside a control sequence there are no category codes, any spaces resulting from \string are of necessity only space characters, that is, characters with code 32. However, TEX’s input processor converts all space tokens that have a character code other than 32 into character tokens with character code 32, so the chances are pretty slim that ‘funny spaces’ wind up in control sequences. Other commands with the same behaviour with respect to category codes as \string, are \number, \romannumeral, \jobname, \fontname, \meaning, and \the.

52

Victor Eijkhout – TEX by Topic

Chapter 4 Fonts

In text mode TEX takes characters from a ‘current font’. This chapter describes how fonts are identified to TEX, and what attributes a font can have. \font Declare the identifying control sequence of a font. \fontname The external name of a font. \nullfont Name of an empty font that TEX uses in emergencies. \hyphenchar Number of the hyphen character of a font. \defaulthyphenchar Value of \hyphenchar when a font is loaded. Plain TEX default: ‘\-. \fontdimen Access various parameters of fonts. \char47 Italic correction. \noboundary Omit implicit boundary character.

4.1

Fonts

In TEX terminology a font is the set of characters that is contained in one external font file. During processing, TEX decides from what font a character should be taken. This decision is taken separately for text mode and math mode. When TEX is processing ordinary text, characters are taken from the ‘current font’. External font file names are coupled to control sequences by statements such as \font\MyFont=myfont10 which makes TEX load the file myfont10.tfm. Switching the current font to the font described in that file is then done by \MyFont The status of the current font can be queried: the sequence \the\font produces the control sequence for the current font. Math mode completely ignores the current font. Instead it looks at the ‘current family’, which can contain three fonts: one for text style, one for script style, and one for scriptscript style. This is treated in Chapter 21. See [44] for a consistent terminology of fonts and typefaces. 53

Chapter 4. Fonts

With ‘virtual fonts’ (see [26]) it is possible that what looks like one font to TEX resides in more than one physical font file. See further page 265.

4.2

Font declaration

Somewhere during a run of TEX or IniTEX the coupling between an internal identifying control sequence and the external file name of a font has to be made. The syntax of the command for this is \fonthcontrol sequenceihequalsihfile nameihat clausei where hat clausei −→ at hdimeni | scaled hnumberi | hoptional spacesi Font declarations are local to a group. By the hat clausei the user specifies that some magnified version of the font is wanted. The hat clausei comes in two forms: if the font is given scaled f , TEX multiplies all its font dimensions for that font by f /1000; if the font has a design size dpt and the hat clausei is at ppt, TEX multiplies all font data by p/d. The presence of an hat clausei makes no difference for the external font file (the .tfm file) that TEX reads for the font; it just multiplies the font dimensions by a constant. After such a font declaration, using the defined control sequence will set the current font to the font of the control sequence. 4.2.1

Fonts and tfm files

The external file needed for the font is a tfm (TEX font metrics) file, which is taken independent of any hat clausei in the \font declaration. If the tfm file has been loaded already (for instance by IniTEX when it constructed the format), an assignment of that font file can be reexecuted without needing recourse to the tfm file. Font design sizes are given in the font metrics files. The cmr10 font, for instance, has a design size of 10 point. However, there is not much in the font that actually has a size of 10 points: the opening and closing parentheses are two examples, but capital letters are considerably smaller. 4.2.2

Querying the current font and font names

It was already mentioned above that the control sequence which set the current font can be retrieved by the command \the\font. This is a special case of \thehfonti where hfonti −→ \font | hfontdef tokeni | hfamily memberi hfamily memberi −→ hfont rangeih4-bit numberi hfont rangei −→ \textfont | \scriptfont | \scriptscriptfont A hfontdef tokeni is a control sequence defined by \font, or the predefined control sequence \nullfont. The concept of hfamily memberi is only relevant in math mode. Also, the external name of fonts can be retrieved: 54

Victor Eijkhout – TEX by Topic

4.3. Font information

\fontnamehfonti gives a sequence of character tokens of category 12 (but space characters get category 10) that spells the font file name, plus an hat clausei if applicable. After \font\tenroman=cmr10 \tenroman the calls \the\font and \the\tenroman both give \tenroman. The call \fontname\tenroman gives cmr10.

4.2.3

\nullfont

TEX always knows a font that has no characters: the \nullfont. If no font has been specified, or if in math mode a family member is needed that has not been specified, TEX will take its characters from the nullfont. This control sequence qualifies as a hfontdef tokeni: it acts like any other control sequence that stands for a font; it just does not have an associated tfm file.

4.3

Font information

During a run of TEX the main information needed about the font consists of the dimensions of the characters. TEX finds these in the font metrics files, which usually have extension .tfm. Such files contain • • •

global information: the \fontdimen parameters, and some other information, dimensions and the italic corrections of characters, and ligature and kerning programs for characters.

Also, the design size of a font is specified in the tfm file; see above. The definition of the tfm format can be found in [25].

4.3.1

Font dimensions

Text fonts need to have at least seven \fontdimen parameters to describe font dimensions (but TEX will take zero for unspecified parameters); math symbol and math extension fonts have more (see page 208). For text fonts the minimal set of seven comprises the following: 1. 2. 3. 4. 5. 6. 7.

the slant per point; this dimension is used for the proper horizontal positioning of accents; the interword space: this is used unless the user specifies an explicit \spaceskip; see Chapter 20; interword stretch: the stretch component of the interword space; interword shrink: the shrink component of the interword space; the x-height: the value of the hinternal uniti ex, which is usually about the height of the lowercase letter ‘x’; the quad width: the value of the hinternal uniti em, which is approximately the width of the capital letter ‘M’; and the extra space: the space added to the interword space at the end of sentences (that is, when \spacefactor ≥ 2000) unless the user specifies an explicit \xspaceskip.

Victor Eijkhout – TEX by Topic

55

Chapter 4. Fonts

Parameters 1 and 5 are purely information about the font and there is no point in varying them. The values of other parameters can be changed in order to adjust spacing; see Chapter 20 for examples of changing parameters 2, 3, 4, and 7. Font dimensions can be altered in a hfont assignmenti, which is a hglobal assignmenti (see page 106): \fontdimenhnumberihfontihequalsihdimeni See above for the definition of hfonti.

4.3.2

Kerning

Some combinations of characters should be moved closer together than would be the case if their bounding boxes were to be just abutted. This fine spacing is called kerning, and a proper kerning is as essential to a font as the design of the letter shapes. Consider as an example ‘Vo’ versus the unkerned variant ‘Vo’ Kerning in TEX is controlled by information in the tfm file, and is therefore outside the influence of the user. The tfm file can be edited, however (see Chapter 33). The \kern command has (almost) nothing to do with the phenomenon of kerning; it is explained in Chapter 8.

4.3.3

Italic correction

The primitive control symbol \/ inserts the italic correction of the previous character or ligature. Such a correction may be necessary owing to the definition of the bounding box of a character. This box always has vertical sides, and the width of the character as TEX perceives it is the distance between these sides. However, in order to achieve proper spacing for slanted or italic typefaces, characters may very well project outside their bounding boxes. The italic correction is then needed if such an overhanging character is followed by a character from a non-slanting typeface. Compare for instance ‘TEX has’ to ‘TEX has’, where the second version was typed as {\italic\TeX\/} has The size of the italic correction of each character is determined by font information in the font metrics file; for the Computer Modern fonts it is approximately half the ‘overhang’ of the characters; see [19]. Italic correction is not the same as \fontdimen1, slant per point. That font dimension is used only for positioning accents on top of characters. An italic correction can only be inserted if the previous item processed by TEX was a character or ligature. Thus the following solution for roman text inside an italic passage does not work: {\italic Some text {\/\roman not} emphasized} The italic correction has no effect here, because the previous item is glue. 56

Victor Eijkhout – TEX by Topic

4.3. Font information

4.3.4

Ligatures

Replacement of character sequences by ligatures is controlled by information in the tfm file of a font. Ligatures are formed from hcharacteri commands: sequences such as fi are replaced by ‘fi’ in some fonts. Other ligatures traditionally in use are between ff, ffi, fl, and ffl; in some older works ft and st can be found, and similarly to the fl ligature fk and fb can also occur. Ligatures in TEX can be formed between explicit character tokens, \char commands, and hchardef tokenis. For example, the sequence \char‘f\char‘i is replaced by the ‘fi’ ligature, if such a ligature is part of the font. Unwanted ligatures can be suppressed in a number of ways: the unwanted ligature ‘halflife’ can for instance be prevented by half{}life, half{l}ife, half\/life, or half\hbox{}life but the solution using italic correction is not equivalent to the others. 4.3.5

Boundary ligatures

Each word is surrounded by a left and a right boundary character (TEX3 only). This makes phenomena possible such as the two different sigmas in Greek: one at the end of a word, and one for every other position. This can be realized through a ligature with the boundary character. A \noboundary command immediately before or after a word suppresses the boundary character at that place. In general, the ligature mechanism has become more complicated with the transition to TEX version 3; see [22].

Victor Eijkhout – TEX by Topic

57

Chapter 4. Fonts

58

Victor Eijkhout – TEX by Topic

Chapter 5 Boxes

The horizontal and vertical boxes of TEX are containers for pieces of horizontal and vertical lists. Boxes can be stored in box registers. This chapter treats box registers and such aspects of boxes as their dimensions, and the way their components are placed relative to each other. \hbox Construct a horizontal box. \vbox Construct a vertical box with reference point of the last item. \vtop Construct a vertical box with reference point of the first item. \vcenter Construct a vertical box vertically centred on the math axis; this command can only be used in math mode. \vsplit Split off the top part of a vertical box. \box Use a box register, emptying it. \setbox Assign a box to a box register. \copy Use a box register, but retain the contents. \ifhbox \ifvbox Test whether a box register contains a horizontal/vertical box. \ifvoid Test whether a box register is empty. \newbox Allocate a new box register. \unhbox \unvbox Unpack a box register containing a horizontal/vertical box, adding the contents to the current horizontal/vertical list, and emptying the register. \unhcopy \unvcopy The same as \unhbox / \unvbox, but do not empty the register. \ht \dp \wd Height/depth/width of the box in a box register. \boxmaxdepth Maximum allowed depth of boxes. Plain TEX default: \maxdimen. \splitmaxdepth Maximum allowed depth of boxes generated by \vsplit. \badness Badness of the most recently constructed box. \hfuzz \vfuzz Excess size that TEX tolerates before it considers a horizontal/vertical box overfull. \hbadness \vbadness Amount of tolerance before TEX reports an underfull or overfull horizontal/vertical box. \overfullrule Width of the rule that is printed to indicate overfull horizontal boxes. \hsize Line width used for text typesetting inside a vertical box. \vsize Height of the page box. \lastbox Register containing the last item added to the current list, if this was a box. \raise \lower Adjust vertical positioning of a box in horizontal mode. \moveleft \moveright Adjust horizontal positioning of a box in vertical mode. \everyhbox \everyvbox Token list inserted at the start of a horizontal/vertical box. 59

Chapter 5. Boxes

5.1

Boxes

In this chapter we shall look at boxes. Boxes are containers for pieces of horizontal or vertical lists. Boxes that are needed more than once can be stored in box registers. When TEX expects a hboxi, any of the following forms is admissible: • \hboxhbox specificationi{hhorizontal materiali} • \vboxhbox specificationi{hvertical materiali} • \vtophbox specificationi{hvertical materiali} • \boxh8-bit numberi • \copyh8-bit numberi • \vsplith8-bit numberitohdimeni • \lastbox A hbox specificationi is defined as hbox specificationi −→ hfilleri | to hdimenihfilleri | spread hdimenihfilleri An h8-bit numberi is a number in the range 0–255. The braces surrounding box material define a group; they can be explicit characters of categories 1 and 2 respectively, or control sequences \let to such characters; see also below. A hboxi can in general be used in horizontal, vertical, and math mode, but see below for the \lastbox. The connection between boxes and modes is explored further in Chapter 6. The box produced by \vcenter – a command that is allowed only in math mode – is not a hboxi. For instance, it can not be assigned with \setbox; see further Chapter 23. The \vsplit operation is treated in Chapter 27.

5.2

Box registers

There are 256 box registers, numbered 0–255. Either a box register is empty (‘void’), or it contains a horizontal or vertical box. This section discusses specifically box registers; the sizes of boxes, and the way material is arranged inside them, is treated below. 5.2.1

Allocation: \newbox

The plain TEX \newbox macro allocates an unused box register: \newbox\MyBox after which one can say \setbox\MyBox=... or \box\MyBox and so on. Subsequent calls to this macro give subsequent box numbers; this way macro collections can allocate their own boxes without fear of collision with other macros. The number of the box is assigned by \chardef (see Chapter 31). This implies that \MyBox is equivalent to, and can be used as, a hnumberi. The control sequence \newbox is an \outer macro. Newly allocated box registers are initially empty. 60

Victor Eijkhout – TEX by Topic

5.2. Box registers

5.2.2

Usage: \setbox, \box, \copy

A register is filled by assigning a hboxi to it: \setboxhnumberihequalsihboxi For example, the hboxi can be explicit \setbox37=\hbox{...} or \setbox37=\vbox{...} or it can be a box register: \setbox37=\box38 Usually, box numbers will have been assigned by a \newbox command. The box in a box register is appended by the commands \box and \copy to whatever list TEX is building: the call \box38 appends box 38. To save memory space, box registers become empty by using them: TEX assumes that after you have inserted a box by calling \boxnn in some mode, you do not need the contents of that register any more and empties it. In case you do need the contents of a box register more than once, you can \copy it. Calling \copynn is equivalent to \boxnn in all respects except that the register is not cleared. It is possible to unwrap the contents of a box register by ‘unboxing’ it using the commands \unhbox and \unvbox, and their copying versions \unhcopy and \unvcopy. Whereas a box can be used in any mode, the unboxing operations can only be used in the appropriate mode, since in effect they contribute a partial horizontal or vertical list (see also Chapter 6). See below for more information on unboxing registers. 5.2.3

Testing: \ifvoid, \ifhbox, \ifvbox

Box registers can be tested for their contents: \ifvoidhnumberi is true if the box register is empty. Note that an empty, or ‘void’, box register is not the same as a register containing an empty box. An empty box is still either a horizontal or a vertical box; a void register can be used as both. The test \ifhboxhnumberi is true if the box register contains a horizontal box; \ifvboxhnumberi is true if the box register contains a vertical box. Both tests are false for void registers. 5.2.4

The \lastbox

When TEX has built a partial list, the last box in this list is accessible as the \lastbox. This behaves like a box register, so you can remove the last box from the list by assigning the \lastbox to some box register. If the last item on the current list is not a box, the \lastbox acts like a void box register. It is not possible to get hold of the last box in the case of the main vertical list. The \lastbox is then always void. As an example, the statement Victor Eijkhout – TEX by Topic

61

Chapter 5. Boxes

{\setbox0=\lastbox} removes the last box from the current list, assigning it to box register 0. Since this assignment occurs inside a group, the register is cleared at the end of the group. At the start of a paragraph this can be used to remove the indentation box (see Chapter 16). Another example of \lastbox can be found on page 72. Because the \lastbox is always empty in external vertical mode, it is not possible to get hold of boxes that have been added to the page. However, it is possible to dissect the page once it is in \box255, for instance doing \vbox{\unvbox255{\setbox0=\lastbox}} inside the output routine. If boxes in vertical mode have been shifted by \moveright or \moveleft, or if boxes in horizontal mode have been raised by \raise or lowered by \lower, any information about this displacement due to such a command is lost when the \lastbox is taken from the list.

5.3

Natural dimensions of boxes

5.3.1

Dimensions of created horizontal boxes

Inside an \hbox all constituents are lined up next to each other, with their reference points on the baseline of the box, unless they are moved explicitly in the vertical direction by \lower or \raise. The resulting width of the box is the sum of the widths of the components. Thus the width of \hbox{\hskip1cm} is positive, and the width of \hbox{\hskip-1cm} is negative. By way of example, a\hbox{\kern-1em b}-gives as output b– a which shows that a horizontal box can have negative width. The height and depth of an \hbox are the maximum amount that constituent boxes project above and below the baseline of the box. They are non-negative when the box is created. The commands \lower and \raise are the only possibilities for vertical movement inside an \hbox (other than including a \vbox inside the \hbox, of course); a hvertical commandi – such as \vskip – is not allowed in a horizontal box, and \par, although allowed, does not do anything inside a horizontal box. 5.3.2

Dimensions of created vertical boxes

Inside a \vbox vertical material is lined up with the reference points on the vertical line through the reference point of the box, unless components are moved explicitly in the horizontal direction by \moveleft or \moveright. 62

Victor Eijkhout – TEX by Topic

5.3. Natural dimensions of boxes

The reference point of a vertical box is always located at the left boundary of the box. The width of a vertical box is then the maximal amount that any material in the box sticks to the right of the reference point. Material to the left of the reference point is not taken into account in the width. Thus the result of a\vbox{\hbox{\kern-1em b}}-is ba– This should be contrasted with the above example. The calculation of height and depth is different for vertical boxes constructed by \vbox and \vtop. The ground rule is that a \vbox has a reference point that lies on the baseline of its last component, and a \vtop has its reference point on the baseline of the first component. In general, the depth (height) of a \vbox (\vtop) can be non-zero if the last (first) item is a box or rule. The height of a \vbox is then the sum of the heights and depths of all components except the last, plus the height of that last component; the depth of the \vbox is the depth of its last component. The depth of a \vtop is the sum of the depth of the first component and the heights and depths of all subsequent material; its height is the height of the first component. However, the actual rules are a bit more complicated when the first component of a \vtop or the last component of a \vbox is not a box or rule. If the last component of a \vbox is a kern or a glue, the depth of that box is zero; a \vtop’s height is zero unless its first component is a box or rule. (Note the asymmetry in these definitions; see below for an example illustrating this.) The depth of a \vtop, then, is equal to the total height plus depth of all enclosed material minus the height of the \vtop. There is a limit on the depth of vertical boxes: if the depth of a \vbox or \vtop calculated by the above rules would exceed \boxdepth, the reference point of the box is moved down by the excess amount. More precisely, the excess depth is added to the natural height of the box. If the box had a to or spread specification, any glue is set anew to take the new height into account. Ordinarily, \boxmaxdepth is set to the maximum dimension possible in TEX. It is for instance reduced during some of the calculations in the plain TEX output routine; see Chapter 28. 5.3.3

Examples

Horizontal boxes are relatively straightforward. Their width is the distance between the ‘beginning’ and the ‘end’ of the box, and consequently the width is not necessarily positive. With \setbox0=\hbox{aa} \setbox1=\hbox{\copy0 \hskip-\wd0} the \box1 has width zero; /\box1/ gives ‘/aa / ’ The height and depth of a horizontal box cannot be negative: in \setbox0=\hbox{\vrule height 5pt depth 5pt} \setbox1=\hbox{\raise 10pt \box0} the \box1 has depth 0pt and height 15pt Vertical boxes are more troublesome than horizontal boxes. Let us first treat their width. After \setbox0=\hbox{\hskip 10pt} Victor Eijkhout – TEX by Topic

63

Chapter 5. Boxes

the box in the \box0 register has a width of 10pt. Defining \setbox1=\vbox{\moveleft 5pt \copy0} the \box1 will have width 5pt; material to the left of the reference point is not accounted for in the width of a vertical box. With \setbox2=\vbox{\moveright 5pt \copy0} the \box2 will have width 15pt. The depth of a \vbox is the depth of the last item if that is a box, so \vbox{\vskip 5pt \hbox{\vrule height 5pt depth 5pt}} has height 10pt and depth 5pt, and \vbox{\vskip -5pt \hbox{\vrule height 5pt depth 5pt}} has height 0pt and depth 5pt. With a glue or kern as the last item in the box, the resulting depth is zero, so \vbox{\hbox{\vrule height 5pt depth 5pt}\vskip 5pt} has height 15pt and depth 0pt; \vbox{\hbox{\vrule height 5pt depth 5pt}\vskip -5pt} has height 5pt and depth 0pt. The height of a \vtop behaves (almost) the same with respect to the first item of the box, as the depth of a \vbox does with respect to the last item. Repeating the above examples with a \vtop gives the following: \vtop{\vskip 5pt \hbox{\vrule height 5pt depth 5pt}} has height 0pt and depth 15pt, and \vtop{\vskip -5pt \hbox{\vrule height 5pt depth 5pt}} has height 0pt and depth 5pt; \vtop{\hbox{\vrule height 5pt depth 5pt} \vskip 5pt} has height 5pt and depth 10pt, and \vtop{\hbox{\vrule height 5pt depth 5pt} \vskip -5pt} has height 5pt and depth 0pt.

5.4

More about box dimensions

5.4.1

Predetermined dimensions

The size of a box can be specified in advance with a hbox specificationi; see above for the syntax. Any glue in the box is then set in order to reach the required size. Prescribing the size of the box is done by \hbox to hdimeni {...}, \vbox to hdimeni {...} If stretchable or shrinkable glue is present in the box, it is stretched or shrunk in order to give the box the specified size. Associated with this glue setting is a badness value (see Chapter 8). If no stretch or shrink – whichever is necessary – is present, the resulting box will be underfull or overfull respectively. Error reporting for over/underfull boxes is treated below. Another command to let a box have a size other than the natural size is 64

Victor Eijkhout – TEX by Topic

5.4. More about box dimensions

\hbox spread hdimeni {...}, \vbox spread hdimeni {...} which tells TEX to set the glue in such a way that the size of the box is a specified amount more than the natural size. Box specifications for \vtop vertical boxes are somewhat difficult to interpret. TEX constructs a \vtop by first making a \vbox, including glue settings induced by a hbox specificationi; then it computes the height and depth by the above rules. Glue setting is described in Chapter 8. 5.4.2

Changes to box dimensions

The dimensions of a box register are accessible by the commands \ht, \dp, and \wd; for instance \dp13 gives the depth of box 13. However, not only can boxes be measured this way; by assigning values to these dimensions TEX can even be fooled into thinking that a box has a size different from its actual. However, changing the dimensions of a box does not change anything about the contents; in particular it does not change the way the glue is set. Various formats use this in ‘smash’ macros: the macro defined by \def\smash#1{{\setbox0=\hbox{#1}\dp0=0pt \ht0=0pt \box0\relax}} places its argument but annihilates its height and depth; that is, the output does show the whole box, but further calculations by TEX act as if the height and depth were zero. Box dimensions can be changed only by setting them. They are hbox dimenis, which can only be set in a hbox size assignmenti, and not, for instance changed with \advance. Note that a hbox size assignmenti is a hglobal assignmenti its effect transcends any groups in which it occurs (see Chapter 10). Thus the output of \setbox0=\hbox{---} {\wd0=0pt} a\box0b is ‘a— b ’. The limits that hold on the dimensions with which a box can be created (see above) do not hold for explicit changes to the size of a box: the assignment \dp0=-2pt for a horizontal box is perfectly admissible. 5.4.3

Moving boxes around

In a horizontal box all constituent elements are lined up with their reference points at the same height as the reference point of the box. Any box inside a horizontal box can be lifted or dropped using the macros \raise and \lower. Similarly, in a vertical box all constituent elements are lined up with their reference points underneath one another, in line with the reference point of the box. Boxes can now be moved sideways by the macros \moveleft and \moveright. Only boxes can be shifted thus; these operations cannot be applied to, for instance, characters or rules. 5.4.4

Box dimensions and box placement

TEX places the components of horizontal and vertical lists by maintaining a reference line and a current position on that line. For horizontal lists the reference line is the baseline of the surrounding \hbox; for vertical lists it is the vertical line through the reference point of the surrounding \vbox. Victor Eijkhout – TEX by Topic

65

Chapter 5. Boxes

In horizontal mode a component is placed as follows. The current position coincides initially with the reference point of the surrounding box. After that, the following actions are carried out. 1. If the component has been shifted by \raise or \lower, shift the current position correspondingly. 2. If the component is a horizontal box, use this algorithm recursively for its contents; if it is a vertical box, go up by the height of this box, putting a new current position for the enclosed vertical list there, and place its components using the algorithm for vertical lists below. 3. Move the current position (on the reference line) to the right by the width of the component. For the list in a vertical box TEX’s current position is initially at the upper left corner of that box, as explained above, and the reference line is the vertical line through that point; it also runs through the reference point of the box. Enclosed components are then placed as follows. 1. If a component has been shifted using \moveleft or \moveright, shift the current position accordingly. 2. Put the component with its upper left corner at the current position. 3. If the component is a vertical box, use this algorithm recursively for its contents; if it is a horizontal box, its reference point can be found below the current position by the height of the box. Put the current position for that box there, and use the above algorithm for horizontal lists. 4. Go down by the height plus depth of the box (that is, starting at the upper left corner of the box) on the reference line, and continue processing vertically. Note that the above processes do not describe the construction of boxes. That would (for instance) involve for vertical boxes the insertion of baselineskip glue. Rather, it describes the way the components of a finished box are arranged in the output. 5.4.5

Boxes and negative glue

Sometimes it is useful to have boxes overlapping instead of line up. An easy way to do this is to use negative glue. In horizontal mode {\dimen0=\wd8 \box8 \kern-\dimen0} places box 8 without moving the current location. More versatile are the macros \llap and \rlap, defined as \def\llap#1{\hbox to 0pt{\hss #1}} and \def\rlap#1{\hbox to 0pt{#1\hss}} that allow material to protrude left or right from the current location. The \hss glue is equivalent to \hskip 0pt plus 1fil minus 1fil, which absorbs any positive or negative width of the argument of \llap or \rlap. The sequence \llap{\hbox to 10pt{a\hfil}} is effectively the same as \hbox{\hskip-10pt \hbox to 10pt{a\hfil}} which has a total width of 0pt. 66

Victor Eijkhout – TEX by Topic

5.5. Overfull and underfull boxes

5.5

Overfull and underfull boxes

If a box has a size specification TEX will stretch or shrink glue in the box. For glue with only finite stretch or shrink components the badness (see Chapter 19) of stretching or shrinking is computed. In TEX version 3 the badness of the box most recently constructed is available for inspection by the user through the \badness parameter. Values for badness range 0–10 000, but if the box is overfull it is 1 000 000. When TEX considers the badness too large, it gives a diagnostic message. Let us first consider error reporting for horizontal boxes. Horizontal boxes of which the glue has to stretch are never reported if \hbadness ≥ 10 000; otherwise TEX reports them as ‘underfull’ if their badness is more than \hbadness. Glue shrinking can lead to ‘overfull’ boxes: a box is called overfull if the available shrink is less than the shrink necessary to meet the box specification. An overfull box is only reported if the difference in shrink is more than \hfuzz, or if \hbadness < 100 (and it turns out that using all available shrinkability has badness 100). Setting \hfuzz=1pt will let TEX ignore boxes that can not shrink enough if they lack less than 1pt. In \hbox to 1pt{\hskip3pt minus .5pt} \hbox to 1pt{\hskip3pt minus 1.5pt} only the first box will give an error message: it is 1.5pt too big, whereas the second lacks .5pt which is less than \hfuzz. Also, boxes that shrink but that are not overfull can be reported: if a box is ‘tight’, that is, if it uses at least half its shrinkability, TEX reports this fact if the computed badness (which is between 13 and 100) is more than \hbadness. For horizontal and vertical boxes this error reporting is almost the same, with parameters \vbadness and \vfuzz. The difference is that for horizontal overfull boxes TEX will draw a rule to the right of the box that has the same height as the box, and width \overfullrule. No overfull rule ensues if the \tabskip glue in an \halign cannot be shrunk enough.

5.6

Opening and closing boxes

The opening and closing braces of a box can be either explicit, that is, character tokens of category 1 and 2, or implicit, a control sequence \let to such a character. After the opening brace the \everyhbox or \everyvbox tokens are inserted. If this box appeared in a \setbox assignment any \afterassignment token is inserted even before the ‘everybox’ tokens. \everyhbox{b} \afterassignment a \setbox0=\hbox{c} \showbox0 gives > \box0= \hbox(6.94444+0.0)x15.27782 .\tenrm a .\tenrm b Victor Eijkhout – TEX by Topic

67

Chapter 5. Boxes

.\kern0.27779 .\tenrm c Implicit braces can be used to let a box be opened or closed by a macro, for example: \def\openbox#1{\setbox#1=\hbox\bgroup} \def\closebox#1{\egroup\DoSomethingWithBox#1} \openbox0 ... \closebox0 This mechanism can be used to scoop up paragraphs: \everypar{\setbox\parbox= \vbox\bgroup \everypar{} \def\par{\egroup\UseBox\parbox}} Here the \everypar opens the box and lets the text be set in the box: starting for instance Begin a text ... gives the equivalent of \setbox\parbox=\vbox{Begin a text ... Inside the box \par has been redefined, so ... a text ends.\par is equivalent to ... a text ends.}\Usebox\parbox In this example, the \UseBox command can only treat the box as a whole; if the elements of the box should somehow be treated separately another approach is necessary. In \everypar{\setbox\parbox= \vbox\bgroup\everypar{}% \def\par{\endgraf\HandleLines \egroup\box\parbox}} \def\HandleLines{ ... \lastbox ... } the macro \HandleLines can have access to successive elements from the vertical list of the paragraph. See also the example on page 72.

5.7

Unboxing

Boxes can be unwrapped by the commands \unhbox and \unvbox, and by their copying versions \unhcopy and \unvcopy. These are horizontal and vertical commands (see Chapter 6), considering that in effect they contribute a partial horizontal or vertical list. It is not possible to \unhbox a register containing a \vbox or vice versa, but a void box register can both be \unhboxed and \unvboxed. Unboxing takes the contents of a box in a box register and appends them to the surrounding list; any glue can then be set anew. Thus \setbox0=\hbox to 1cm{\hfil} \hbox to 2cm{\unhbox0} is completely equivalent to \hbox to 2cm{\hfil} 68

Victor Eijkhout – TEX by Topic

5.8. Text in boxes

and not to \hbox to 2cm{\kern1cm} The intrinsically horizontal nature of \unhbox is used to define \def\leavevmode{\unhbox\voidb@x} This command switches from vertical mode to horizontal without adding anything to the horizontal list. However, the subsequent \indent caused by this transition adds an indentation box. In horizontal mode the \leavevmode command has no effect. Note that here it is not necessary to use \unhcopy, because the register is empty anyhow. Beware of the following subtlety: unboxing in vertical mode does not add interline glue between the box contents and any preceding item. Also, the value of \prevdepth is not changed, so glue between the box contents and any following item will occur only if there was something preceding the box; interline glue will be based on the depth of that preceding item. Similarly, unboxing in horizontal mode does not influence the \spacefactor.

5.8

Text in boxes

Both horizontal and vertical boxes can contain text. However, the way text is treated differs. In horizontal boxes the text is placed in one straight line, and the width of the box is in principle the natural width of the text (and other items) contained in it. No hvertical commandis are allowed inside a horizontal box, and \par does nothing in this case. For vertical boxes the situation is radically different. As soon as a character, or any other hhorizontal commandi (see page 74), is encountered in a vertical box, TEX starts building a paragraph in unrestricted horizontal mode, that is, just as if the paragraph were directly part of the page. At the occurrence of a hvertical commandi (see page 75), or at the end of the box, the paragraph is broken into lines using the current values of parameters such as \hsize. Thus \hbox to 3cm{\vbox{some reasonably long text}} will not give a paragraph of width 3 centimetres (it gives an overfull horizontal box if \hsize > 3cm). However, \vbox{\hsize=3cm some reasonably long text} will be 3 centimetres wide. A paragraph of text inside a vertical box is broken into lines, which are packed in horizontal boxes. These boxes are then stacked in internal vertical mode, possibly with \baselineskip and \lineskip separating them (this is treated in Chapter 15). This process is also used for text on the page; the boxes are then stacked in outer vertical mode. If the internal vertical list is empty, no \parskip glue is added at the start of a paragraph. Because text in a horizontal box is not broken into lines, there is a further difference between text in restricted and unrestricted horizontal mode. In restricted horizontal mode no discretionary nodes and whatsit items changing the value of the current language are inserted. This may give problems if the text is subsequently unboxed to form part of a paragraph. See Chapter 19 for an explanation of these items, and [7] for a way around this problem. Victor Eijkhout – TEX by Topic

69

Chapter 5. Boxes

5.9

Assorted remarks

5.9.1

Forgetting the \box

After \newcount\foo, one can use \foo on its own to get the \foo counter. For boxes, however, one has to use \box\foo to get the \foo box. The reason for this is that there exists no separate \boxdef command, so \chardef is used (see Chapter 31). Suppose \newbox\foo allocates box register 25; then typing \foo is equivalent to typing \char25. 5.9.2

Special-purpose boxes

Some box registers have a special purpose: • • •

5.9.3

\box255 is by used TEX internally to give the page to the output routine. \voidb@x is the number of a box register allocated in plain.tex; it is supposed to be empty always. It is used in the macro \leavevmode and others. when a new \insert is created with the plain TEX \newinsert macro, a \count, \dimen, \skip, and \box all with the same number are reserved for that insert. The numbers for these registers count down from 254. The height of a vertical box in horizontal mode

In horizontal mode a vertical box is placed with its reference point aligned vertically with the reference point of the surrounding box. TEX then traverses its contents starting at the left upper corner; that is, the point that lies above the reference point by a distance of the height of the box. Changing the height of the box implies then that the contents of the box are placed at a different height. Consider as an example \hbox{a\setbox0=\vbox{\hbox{b}}\box0 c} which gives abc and \hbox{a\setbox0=\vbox{\hbox{b}}\ht0=0cm \box0 c} which gives a c b By contrast, changing the width of a box placed in vertical mode has no effect on its placement. 5.9.4

More subtleties with vertical boxes

Since there are two kinds of vertical boxes, the \vbox and the \vtop, using these two kinds nested may lead to confusing results. For instance, \vtop{\vbox{...}} is completely equivalent to just \vbox{...} 70

Victor Eijkhout – TEX by Topic

5.9. Assorted remarks

It was stated above that the depth of a \vbox is zero if the last item is a kern or glue, and the height of a \vtop is zero unless the first item in it is a box. The above examples used a kern for that first or last item, but if, in the case of a \vtop, this item is not a glue or kern, one is apt to overlook the effect that it has on the surrounding box. For instance, \vtop{\write16{...}...} has zero height, because the write instruction is packed into a ‘whatsit’ item that is placed on the current, that is, the vertical, list. The remedy here is \vtop{\leavevmode\write16{...}...} which puts the whatsit in the beginning of the paragraph, instead of above it. Placement of items in a vertical list is sometimes a bit tricky. There is for instance a difference between how vertical and horizontal boxes are treated in a vertical list. Consider the following examples. After \offinterlineskip the first example \vbox{\hbox{a} \setbox0=\vbox{\hbox{(}} \ht0=0pt \dp0=0pt \box0 \hbox{ b}} gives a (b while a slight variant \vbox{\hbox{a} \setbox0=\hbox{(} \ht0=0pt \dp0=0pt \box0 \hbox{ b}} gives a ( b The difference is caused by the fact that horizontal boxes are placed with respect to their reference point, but vertical boxes with respect to their upper left corner. 5.9.5

Hanging the \lastbox back in the list

You can pick the last box off a vertical list that has been compiled in (internal) vertical mode. However, if you try to hang it back in the list the vertical spacing may go haywire. If you just hang it back, \setbox\tmpbox=\lastbox \usethetmpbox \box\tmpbox baselineskip glue is added a second time. If you ‘unskip’ prior to hanging the box back, \setbox\tmpbox=\lastbox \unskip \usethetmpbox \box\tmpbox things go wrong in a more subtle way. The hinternal dimeni \prevdepth (which controls interline glue; see Chapter 15) will have a value based on the last box, but what you need for the proper interline glue is a depth based on one box earlier. The solution is not to unskip, but to specify \nointerlineskip: \setbox\tmpbox=\lastbox \usethetmpbox \nointerlineskip \box\tmpbox Victor Eijkhout – TEX by Topic

71

Chapter 5. Boxes

5.9.6

Dissecting paragraphs with \lastbox

Repeatedly applying \last... and \un... macros can be used to take a paragraph apart. Here is an example of that. In typesetting advertisement copy, a way of justifying paragraphs has become popular in recent years that is somewhere between flushright and raggedright setting. Lines that would stretch beyond certain limits are set with their glue at natural width. This single paragraph is but an example of this procedure; the macros are given next. \newbox\linebox \newbox\snapbox \def\eatlines{ \setbox\linebox\lastbox % check the last line \ifvoid\linebox \else % if it’s not empty \unskip\unpenalty % take whatever is {\eatlines} % above it; % collapse the line \setbox\snapbox\hbox{\unhcopy\linebox} % depending on the difference \ifdim\wd\snapbox1000 and the current space factor is (b) < c > 25.4.3

Spanning across multiple columns: \span

Sometimes it is desirable to have material spanning several columns. The most obvious example is that of a heading above a table. For this TEX provides the \span command. Entries are delimited either by &, by \cr, or by \span. In the last case TEX will omit the tabskip glue that would normally follow the entry thus delimited, and it will typeset the material just read plus the following entry in the joint space available. As an example, \tabskip=1em \halign{&#\cr a&b&c&d\cr a&\hrulefill\span\hrulefill&d\cr} gives a a

b

c

d d

Note that there is no tabskip glue in between the two spanned columns, but there is tabskip glue before the first column and after the last. Using the \omit command this same alignment could have been generated as \halign{&#\cr a&b&c&d\cr a&\hrulefill\span\omit&d\cr} The \span\omit combination is used in the plain TEX macro \multispan: for instance \multispan4 gives \omit\span\omit\span\omit\span\omit which spans across three tabs, and removes the templates of four entries. Repeating the above example once again: \halign{&#\cr a&b&c&d\cr a&\multispan2\hrulefill&d\cr} The argument of \multispan is a single token, not a number, so in order to span more than 9 columns the argument should be enclosed in braces, for instance \multispan{12}. Furthermore, a space after a single-digit argument will wind up in the output. For a ‘low budget’ solution to spanning columns plain TEX has the macro \hidewidth, defined by \newskip\hideskip \hideskip=-1000pt plus 1fill \def\hidewidth{\hskip\hideskip} Putting \hidewidth at the beginning or end of an alignment entry will make its width zero, with the material in the entry sticking out to the left or right respectively. 25.4.4

Rules in alignments

Horizontal rules inside a horizontal alignment will mostly be across the width of the alignment. The easiest way to attain this is to use \noalign{\hrule} 222

Victor Eijkhout – TEX by Topic

25.4. The alignment

lines inside the alignment. If the alignment is contained in a vertical box, lines above and below the alignment can be specified with \vbox{\hrule \halign{...} \hrule} The most general way to get horizontal lines in an alignment is to use \multispan \multispan n\hrulefill which can be used to underline arbitrary adjacent columns. Vertical rules in alignments take some more care. Since a horizontal alignment breaks up into horizontal boxes that will be placed on a vertical list, TEX will insert baselineskip glue in between the rows of the alignment. If vertical rules in adjacent rows are to abut, it is necessary to prevent baselineskip glue, for instance by the \offinterlineskip macro. In order to ensure that rows will still be properly spaced it is then necessary to place a strut somewhere in the preamble. A strut is an invisible object with a certain height and depth. Putting that in the preamble guarantees that every line will have at least that height and depth. In the plain format \strut is defined statically as \vrule height8.5pt depth3.5pt width0pt so this must be changed when other fonts or sizes are used. It is a good idea to use a whole column for a vertical rule, that is, to write \vrule#& in the preamble and to leave the corresponding entry in the alignment empty. Omitting the vertical rule can then be done by specifying \omit, and the size of the rule can be specified explicitly by putting, for instance, height 15pt in the entry instead of leaving it empty. Of course, tabskip glue will now be specified to the left and right of the rule, so some extra tabskip assignments may be needed in the preamble. 25.4.5

End of a line: \cr and \crcr

All lines in an alignment are terminated by the \cr control sequence, including the last line. TEX is not able to infer from a closing brace in the α part that the alignment has ended, because an unmatched closing brace is perfectly valid in an alignment entry; it may match an opening brace in the u part of the corresponding preamble entry. TEX has a primitive command \crcr that is equivalent to \cr, but it has no effect if it immediately follows a \cr. Consider as an example the definition in plain TEX of \cases: \def\cases#1{% \left\{\,\vcenter{\normalbaselines\m@th \ialign{ $##\hfil$& \quad##\hfil \crcr #1\crcr}}% \right.} Because of the \crcr after the user argument #1, the following two applications of this macro \cases{1&2\cr 3&4} and \cases{1&2\cr 3&4\cr} both work. In the first case the \crcr in the macro definition ends the last line; in the second case the user’s \cr ends the line, and the \crcr is redundant. After \cr and after a non-redundant \crcr the htoken parameteri \everycr is inserted. This includes the \cr terminating the template line. Victor Eijkhout – TEX by Topic

223

Chapter 25. Alignment

25.5

Example: math alignments

The plain format has several alignment macros that function in math mode. One example is \matrix, defined by \def\matrix#1{\null\,\vcenter{\normalbaselines\m@th \ialign{\hfil$##$\hfil && \quad\hfil$##$\hfil\crcr \mathstrut\crcr \noalign{\kern-\baselineskip} #1\crcr \mathstrut\crcr \noalign{\kern-\baselineskip}}}\,} This uses a repeating (starting with &&) second preamble entry; each entry is centred by an \hfil before and after it, and there is a \quad of space in between columns. Tabskip glue was not used for this, because there should not be any glue preceding or following the matrix. The combination of a \mathstrut and \kern-\baselineskip above and below the matrix increases the vertical size such that two matrices with the same number of rows will have the same height and depth, which would not otherwise be the case if one of them had subscripts in the last row, but the other not. The \mathstrut causes interline glue to be inserted and, because it has a size equal to \baselineskip, the negative kern will effectively leave only the interline glue, thereby buffering any differences in the first and last line. Only to a certain point, of course: objects bigger than the opening brace will still result in a different height or depth of the matrix. Another, more complicated, example of an alignment for math mode is \eqalignno. \eqalignno\centering \def\eqalignno#1{\begin{disp}l@y \tabskip\centering \halign to\displaywidth{ \hfil$\@lign\displaystyle{##}$% -- first column \tabskip\z@skip &$\@lign\displaystyle{{}##}$\hfil% -- second column \tabskip\centering &\llap{$\@lign##$}% -- third column \tabskip\z@skip\crcr % end of the preamble #1\crcr}} Firstly, the tabskip is set to zero after the equation number, so this number is set flush with the right margin. Since it is placed by \llap, its effective width is zero. Secondly, the tabskip between the first and second columns is also zero, and the tabskip before the first column and after the second is \centering, which is 0pt plus 1000pt minus 1000pt, so the first column and second are jointly centred in the \hsize. Note that, because of the minus 1000pt, these two columns will happily go outside the left and right margins, overwriting any equation numbers.

224

Victor Eijkhout – TEX by Topic

Chapter 26 Page Shape

This chapter treats some of the parameters that determine the size of the page and how it appears on paper. \topskip Minimum distance between the top of the page box and the baseline of the first box on the page. Plain TEX default: 10pt \hoffset \voffset Distance by which the page is shifted right/down with respect to the reference point. \vsize Height of the page box. Plain TEX default: 8.9in \maxdepth Maximum depth of the page box. Plain TEX default: 4pt \splitmaxdepth Maximum depth of a box split off by a \vsplit operation. Plain TEX default: \maxdimen

26.1

The reference point for global positioning

The page positioning on the paper is governed by a TEX convention, to which output device drivers must adhere, that the top left point of the page is one inch from the page edges. Unfortunately this may lead to lots of trouble, for instance if a printer (or the page description language it uses) takes, say, the lower left corner as the reference point, and is factory set to US paper sizes, but is used with European standard A4 paper. The page is shifted on the paper if one assigns non-zero values to \hoffset or \voffset: positive values shift to the right and down respectively.

26.2

\topskip

The \topskip ensures to a certain point that the first baseline of a page will be at the same location from page to page, even if font sizes are switched between pages or if the first line has no ascenders. Before the first box on each page some glue is inserted. This glue has the same stretch and shrink as \topskip, but the natural size is the natural size of \topskip minus the height of the first box, or zero if this would be negative. 225

Chapter 26. Page Shape

Plain TEX sets \topskip to 10pt. Thus the top lines of pages will have their baselines at the same place if the top portion of the characters is ten point or less. For the Computer Modern fonts this condition is satisfied if the font size is less than (about) 13 points; for larger fonts the baseline of the top line will drop. The height of the page box for a page containing only text (and assuming a zero \parskip) will be the \topskip plus a number of times the \baselineskip. Thus one can define a macro to compute the \vsize from the number of lines on a page: \def\HeightInLines#1{\count@=#1\relax \advance\count@ by -1\relax \vsize=\baselineskip \multiply\vsize by \count@ \advance\vsize by \topskip} Calculating the \vsize this way will prevent underfull boxes for text-only pages. In cases where the page does not start with a line of text (for instance a rule), the topskip may give unwanted effects. To prevent these, start the page with \hbox{}\kern-\topskip followed by what you wanted on top. Analogous to the \topskip, there is a \splittopskip for pages generated by a \vsplit operation; see the next chapter.

26.3

Page height and depth

TEX tries to build pages as a \vbox of height \vsize; see also \pagegoal in the next chapter. If the last item on a page has an excessive depth, that page would be noticeably longer than other pages. To prevent this phenomenon TEX uses \maxdepth as the maximum depth of the page box. If adding an item to the page would make the depth exceed this quantity, then the reference point of the page is moved down to make the depth exactly \maxdepth. The ‘raggedbottom’ effect is obtained in plain TEX \raggedbottomby giving the \topskip some finite stretchability: 10pt plus 60pt. Thus the natural height of box 255 can vary when it reaches the output routine. Pages are then shipped out (more or less) as \dimen0=\dp255 \unvbox255 \ifraggedbottom \kern-\dimen0 \vfil \fi The \vfil causes the topskip to be set at natural width, so the effect is one of a fixed top line and a variable bottom line of the page. Before \box255 is unboxed in the plain TEX output routine, \boxmaxdepth is set to \maxdepth so that this box will made under the same assumptions that the page builder used when putting together \box255. The depth of box split off by a \vsplit operation is controlled by the \splitmaxdepth parameter.

226

Victor Eijkhout – TEX by Topic

Chapter 27 Page Breaking

This chapter treats the ‘page builder’: the part of TEX that decides where to break the main vertical list into pages. The page builder operates before the output routine, and it hands its result in \box255 to the output routine. \vsplit Split of a top part of a box. This is comparable with page breaking. \splittopskip Minimum distance between the top of what remains after a \vsplit operation, and the first item in that box. Plain TEX default: 10pt \pagegoal Goal height of the page box. This starts at \vsize, and is diminished by heights of insertion items. \pagetotal Accumulated natural height of the current page. \pagedepth Depth of the current page. \pagestretch Accumulated zeroth-order stretch of the current page. \pagefilstretch Accumulated first-order stretch of the current page. \pagefillstretch Accumulated second-order stretch of the current page. \pagefilllstretch Accumulated third-order stretch of the current page. \pageshrink Accumulated shrink of the current page. \outputpenalty Value of the penalty at the current page break, or 10 000 if the break was not at a penalty. \interlinepenalty Penalty for breaking a page between lines of a paragraph. Plain TEX default: 0 \clubpenalty Additional penalty for breaking a page after the first line of a paragraph. Plain TEX default: 150 \widowpenalty Additional penalty for breaking a page before the last line of a paragraph. Plain TEX default: 150 \displaywidowpenalty Additional penalty for breaking a page before the last line above a display formula. Plain TEX default: 50 \brokenpenalty Additional penalty for breaking a page after a hyphenated line. Plain TEX default: 100 \penalty Place a penalty on the current list. \lastpenalty If the last item on the list was a penalty, the value of this. \unpenalty Remove the last item of the current list if this was a penalty. 227

Chapter 27. Page Breaking

27.1

The current page and the recent contributions

The main vertical list of TEX is divided in two parts: the current page and the list of recent contributions. Any material that is added to the main vertical list is appended to the recent contributions; the act of moving the recent contributions to the current page is known as exercising the page builder. Every time something is moved to the current page, TEX computes the cost of breaking the page at that point. If it decides that it is past the optimal point, the current page up to the best break so far is put in \box255 and the remainder of the current page is moved back on top of the recent contributions. If the page is broken at a penalty, that value is recorded in \outputpenalty, and a penalty of size 10 000 is placed on top of the recent contributions; otherwise, \outputpenalty is set to 10 000. If the current page is empty, discardable items that are moved from the recent contributions are discarded. This is the mechanism that lets glue disappear after a page break and at the top of the first page. When the first non-discardable item is moved to the current page, the \topskip glue is inserted; see the previous chapter. The workings of the page builder can be made visible by setting \tracingpages to some positive value (see Chapter 34).

27.2

Activating the page builder

The page builder comes into play in the following circumstances. • Around paragraphs: after the \everypar tokens have been inserted, and after the paragraph has been added to the vertical list. See the end of this chapter for an example. • Around display formulas: after the \everydisplay tokens have been inserted, and after the display has been added to the list. • After \par commands, boxes, insertions, and explicit penalties in vertical mode. • After an output routine has ended. In these places the page builder moves the recent contributions to the current page. Note that TEX need not be in vertical mode when the page builder is exercised. In horizontal mode, activating the page builder serves to move preceding vertical glue (for example, \parskip, \abovedisplayskip) to the page. The \end command – which is only allowed in external vertical mode – terminates a TEX job, but only if the main vertical list is empty and \deadcycles = 0. If this is not the case the combination \hbox{}\vfill\penalty−230 is appended, which forces the output routine to act.

27.3

Page length bookkeeping

The height and depth of the page box that reaches the output routine are determined by \vsize, \topskip, and \maxdepth as described in the previous chapter. TEX places the \topskip 228

Victor Eijkhout – TEX by Topic

27.4. Breakpoints

glue when the first box is placed on the current page; the \vsize and \maxdepth are read when the first box or insertion occurs on the page. Any subsequent changes to these parameters will not be noticeable until the next page or, more strictly, until after the output routine has been called. After the first box, rule, or insertion on the current page the \vsize is recorded in \pagegoal, and its value is not looked at until \output has been active. Changing \pagegoal does have an effect on the current page. When the page is empty, the pagegoal is \maxdimen, and \pagetotal is zero.

Accumulated dimensions and stretch are available in the parameters \pagetotal, \pagedepth, \pagestretch, \pagefilstretch, \pagefillstretch, \pageshrink, and \pagefilllstretch. \pagetotal\pagedepth\pag are set by the page builder. The stretch and shrink parameters are updated every time glue is added to the page. The depth parameter becomes zero if the last item was kern or glue. These parameters are hspecial dimenis; an assignment to any of them is an hintimate assignmenti, and it is automatically global.

27.4

Breakpoints

27.4.1

Possible breakpoints

Page breaks can occur at the same kind of locations where line breaks can occur. The breakpoints in vertical lists are: • at glue that is preceded by a non-discardable item; • at a kern that is immediately followed by glue; • at a penalty. TEX inserts interline glue and various sorts of interline penalties when the lines of a paragraph are added to the vertical list, so there will usually be sufficient breakpoints on the page. 27.4.2

Breakpoint penalties

If TEX decides to break a page at a penalty item, this penalty will, most of the time, be one that has been inserted automatically between the lines of a paragraph. If the last item on a list (not necessarily a vertical list) is a penalty, the value of this is recorded in the parameter \lastpenalty. If the item is other than a penalty, this parameter has the value zero. The last penalty of a list can be removed with the command \unpenalty. See Section 5.9.6 for an example. Here is a list of the penalties in vertical mode: \interlinepenalty Penalty for breaking a page between lines of a paragraph. In plain TEX this is zero, so no penalty is added in between lines. TEX can then find a valid breakpoint at the \baselineskip glue. \clubpenalty Extra penalty for breaking a page after the first line of a paragraph. In plain TEX this is 150. This amount, and the following penalties, are added to the \interlinepenalty, and a penalty of the resulting size is inserted after the \hbox containing the first line of a paragraph instead of the \interlinepenalty. Victor Eijkhout – TEX by Topic

229

Chapter 27. Page Breaking

\widowpenalty Extra penalty for breaking a page before the last line of a paragraph. In plain TEX this is 150. \displaywidowpenalty Extra penalty for breaking a page before the last line above a display formula. The default value in plain TEX is 50. \brokenpenalty Extra penalty for breaking a page after a hyphenated line. The default value in plain TEX is 100. If the resulting penalty is zero, it is not placed. Penalties can also be inserted by the user. For instance, the plain format has macros to encourage (possibly, force) or prohibit page breaks\penalty: \def\break{\penalty-10000 } % force break \def\nobreak{\penalty10000 } % prohibit break \def\goodbreak{\par\penalty-500 } % encourage page break Also, \vadjust{\penalty ... } is a way of getting penalties in the vertical list. This can be used to discourage or encourage page breaking after a certain line of a paragraph. 27.4.3

Breakpoint computation

Whenever an item is moved to the current page, TEX computes the penalty p and the badness b associated with breaking the page at that place. From the penalty and the badness the cost c of breaking is computed. The place of least cost is remembered, and when the cost is infinite, that is, the page is overfull, or when the penalty is p ≤ −10 000, the current page is broken at the (last remembered) place of least cost. The broken-off piece is then put in \box255 and the output routine token list is inserted. Box 255 is always given a height of \vsize, regardless of how much material it has. The badness calculation is based on the amount of stretching or shrinking that is necessary to fit the page in a box with height \vsize and maximum depth \maxdepth. This calculation is the same as for line breaking (see Chapter 8). Badness is a value 0 ≤ b ≤ 10 000, except when pages are overfull; then b = ∞.

underfull page b = 10 000

feasible breakpoints b < 10 000 overfull page b=∞

. . . Some penalties are implicitly inserted by TEX, for instance the \interlinepenalty which is put in between every pair of lines of a paragraph. Other penalties can be explicitly inserted by the user or a user macro. A penalty value p ≥ 10 000 inhibits breaking; a penalty p ≤ −10 000 (in external vertical mode) forces a page break, and immediately activates the output routine. Cost calculation proceeds as follows: 1. When a penalty is so low that it forces a page break and immediate invocation of the output routine, but the page is not overfull, that is b < ∞ and p ≤ −10 000 the cost is equal to the penalty: c = p. 2. When penalties do not force anything, and the page is not overfull, that is 230

Victor Eijkhout – TEX by Topic

27.5. \vsplit

3.

4.

b < ∞ and |p| < 10 000 the cost is c = b + p. For pages that are very bad, that is b = 10 000 and |p| < 10 000 the cost is c = 10 000. An overfull page, that is b = ∞ and p < 10 000 gives infinite cost: c = ∞. In this case TEX decides that the optimal break point must have occurred earlier, and it invokes the output routine. Values of \insertpenalties (see Chapter 29) that exceed 10 000 also give infinite cost.

The fact that a penalty p ≤ −10 000 activates the output routine is used extensively in the LATEX output routine: the excess |p| − 10 000 is a code indicating the reason for calling the output routine; see also the second example in the next chapter.

27.5

\vsplit

The page-breaking operation is available to the user through the \vsplit operation. \setbox1 = \vsplit2 to \dimen3 assigns to box 1 the top part of size \dimen3 of box 2. This material is actually removed from box 2. Compare this with splitting off a chunk of size \vsize from the current page. The extracted result of \vsplith8-bit numberitohdimeni is a box with the following properties. •

• •

Height equal to the specified hdimeni; TEX will go through the original box register (which must contain a vertical box) to find the best breakpoint. This may result in an underfull box. Depth at most \splitmaxdepth; this is analogous to the \maxdepth for the page box, rather than the \boxmaxdepth that holds for any box. A first and last mark in the \splitfirstmark and \splitbotmark registers.

The remainder of the \vsplit operation is a box where • • •

all discardables have been removed from the top; glue of size \splittopskip has been inserted on top; if the box being split was box 255, it already had \topskip glue on top; its depth has been forced to be at most \splitmaxdepth.

The bottom of the original box is always a valid breakpoint for the \vsplit operation. If this breakpoint is taken, the remainder box register is void. The extracted box can be empty; it is only void if the original box was void, or not a vertical box. Typically, the \vsplit operation is used to split off part of \box255. By setting \splitmaxdepth equal to \boxmaxdepth the result is something that could have been made by TEX’s page builder. After pruning the top of \box255, the mark registers \firstmark and \botmark contain the first and last marks on the remainder of box 255. See the next chapter for more information on marks. Victor Eijkhout – TEX by Topic

231

Chapter 27. Page Breaking

27.6

Examples of page breaking

27.6.1

Filling up a page

Suppose a certain vertical box is too large to fit on the remainder of the page. Then \vfil\vbox{ ... } is the wrong way to fill up the page and push the box to the next. TEX can only break at the start of the glue, and the \vfil is discarded after the break: the result is an underfull, or at least horribly stretched, page. On the other hand, \vfil\penalty0 % or any other value \vbox{ ... } is the correct way: TEX will break at the penalty, and the page will be filled. 27.6.2

Determining the breakpoint

In the following examples the \vsplit operation is used, which has the same mechanism as page breaking. Let the macros and parameter settings \offinterlineskip \showboxdepth=1 \def\High{\hbox{\vrule height5pt}} \def\HighAndDeep{\hbox{\vrule height2.5pt depth2.5pt}} be given. First let us consider an example where a vertical list is simply stretched in order to reach a break point. \splitmaxdepth=4pt \setbox1=\vbox{\High \vfil \HighAndDeep} \setbox2=\vsplit1 to 9pt gives > \box2= \vbox(9.0+2.5)x0.4, glue set 1.5fil .\hbox(5.0+0.0)x0.4 [] .\glue 0.0 plus 1.0fil .\glue(\lineskip) 0.0 .\hbox(2.5+2.5)x0.4 [] The two boxes together have a height of 7.5pt, so the glue has to stretch 1.5pt. Next, we decrease the allowed depth of the resulting list. \splitmaxdepth=2pt \setbox1=\vbox{\High \vfil \HighAndDeep} \setbox2=\vsplit1 to 9pt gives > \box2= \vbox(9.0+2.0)x0.4, glue set 1.0fil .\hbox(5.0+0.0)x0.4 [] .\glue 0.0 plus 1.0fil 232

Victor Eijkhout – TEX by Topic

27.6. Examples of page breaking

.\glue(\lineskip) 0.0 .\hbox(2.5+2.5)x0.4 [] The reference point is moved down half a point, and the stretch is correspondingly diminished, but this motion cannot lead to a larger dimension than was specified. As an example of this, consider the sequence \splitmaxdepth=3pt \setbox1=\vbox{\High \kern1.5pt \HighAndDeep} \setbox2=\vsplit1 to 9pt This gives a box exactly 9 points high and 2.5 points deep. Setting \splitmaxdepth=2pt does not increase the height by half a point; instead, an underfull box results because an earlier break is taken. Sometimes the timing of actions is important. TEX first locates a breakpoint that will lead to the requested height, then checks whether accommodating the \maxdepth or \splitmaxdepth will not violate that height. Consider an example of this timing: in \splitmaxdepth=4pt \setbox1=\vbox{\High \vfil \HighAndDeep} \setbox2=\vsplit1 to 7pt the result is not a box of 7 points high and 3 points deep. Instead, > \box2= \vbox(7.0+0.0)x0.4 .\hbox(5.0+0.0)x0.4 [] which is an underfull box. 27.6.3

The page builder after a paragraph

After a paragraph, the page builder moves material to the current page, but it does not decide whether a breakpoint has been found yet. \output{\interrupt \plainoutput}% show when you’re active \def\nl{\hfil\break}\vsize=22pt % make pages of two lines a\nl b\nl c\par \showlists % make a 3-line paragraph will report ### current page: [...] total height 34.0 goal height 22.0 prevdepth 0.0, prevgraf 3 lines Even though more than enough material has been gathered, \output is only invoked when the next paragraph starts: typing a d gives ! Undefined control sequence. {\interrupt \plainoutput }

d when \output is inserted after \everypar. Victor Eijkhout – TEX by Topic

233

Chapter 27. Page Breaking

234

Victor Eijkhout – TEX by Topic

Chapter 28 Output Routines

The final stages of page processing are performed by the output routine. The page builder cuts off a certain portion of the main vertical list and hands it to the output routine in \box255. This chapter treats the commands and parameters that pertain to the output routine, and it explains how output routines can receive information through marks. \output Token list with instructions for shipping out pages. \shipout Ship a box to the dvi file. \mark Specify a mark text. \topmark The last mark on the previous page. \botmark The last mark on the current page. \firstmark The first mark on the current page. \splitbotmark The last mark on a split-off page. \splitfirstmark The first mark on a split-off page. \deadcycles Counter that keeps track of how many times the output routine has been called without a \shipout taking place. \maxdeadcycles The maximum number of times that the output routine is allowed to be called without a \shipout occurring. \outputpenalty Value of the penalty at the current page break, or 10 000 if the break was not at a penalty.

28.1

The \output token list

Common parlance has it that ‘the output routine is called’ when TEX has found a place to break the main vertical list. Actually, \output is not a macro but a token list that is inserted into TEX’s command stream. Insertion of the \output token list happens \outputinside a group that is implicitly opened. Also, TEX enters internal vertical mode. Because of the group, non-local assignments (to the page number, for instance) have to be prefixed with \global. The vertical mode implies that during the workings of the output routine spaces are mostly harmless. The \output token list belongs to the class of the htoken parameteris. These behave the same as \toksnnn token lists; see Chapter 14. Assigning an output routine can therefore take the following forms: \outputhequalsihgeneral texti or \outputhequalsihfillerihtoken variablei 235

Chapter 28. Output Routines

28.2

Output and \box255

TEX’s page builder breaks the current page at the optimal point, and stores everything above that in \box255; then, the \output tokens are inserted into the input stream. Any remaining material on the main vertical list is pushed back to the recent contributions. If the page is broken at a penalty, that value is recorded in \outputpenalty, and a penalty of size 10 000 is placed on top of the recent contributions; otherwise, \outputpenalty is set to 10 000. When the output routine is finished, \box255 is supposed to be empty. If it is not, TEX gives an error message. Usually, the output routine will take the pagebox, \shipoutappend a headline and/or footline, maybe merge in some insertions such as footnotes, and ship the page to the dvi file: \output={\setbox255=\vbox {\someheadline \vbox to \vsize{\unvbox255 \unvbox\footins} \somefootline} \shipout\box255} When box 255 reaches the output routine, its height has been set to \vsize. However, the material in it can have considerably smaller height. Thus, the above output routine may lead to underfull boxes. This can be remedied with a \vfil. The output routine is under no obligation to \deadcyclesdo anything useful with \box255; it can empty it, or unbox it to let TEX have another go at finding a page break. The number of times that the output routing postpones the \shipout is recorded in \deadcycles: this parameter is set to 0 by \shipout, and increased by 1 just before every \output. When the number of dead cycles reaches \maxdeadcycles, TEX gives an error message, and performs the default output routine \shipout\box255 instead of the routine it was about to start. The LATEX format has a much higher value for \maxdeadcycles than plain TEX, because the output routine in LATEX is often called for intermediate handling of floats and marginal notes. The \shipout command can send any hboxi to the dvi file; this need not be box 255, or even a box containing the current page. It does not have to be called inside the output routine, either. If the output routine produces any material, for instance by calling \unvbox255 this is put on top of the recent contributions. After the output routine finishes, the page builder is activated. In particular, because the current page has been emptied, the \vsize is read again. Changes made to this parameter inside the output routine (using \global) will therefore take effect.

28.3

Marks

Information can be passed to the output routine through the \markmechanism of marks. The user can specify a token list with 236

Victor Eijkhout – TEX by Topic

28.3. Marks

\mark{hmark texti} which is put in a mark item on the current vertical list. The mark text is subject to expansion as in \edef. If the mark is given in horizontal mode it migrates to the surrounding vertical lists like an insertion item (see page 77); however, if this is not the external vertical list, the output routine will not find the mark. Marks are the main mechanism through which the output routine can obtain information about the contents of the currently broken-off page, in particular its top and bottom. TEX sets three variables: \botmark the last mark occurring on the current page; \firstmark the first mark occurring on the current page; \topmark the last mark of the previous page, that is, the value of \botmark on the previous page. If no marks have occurred yet, all three are empty; if no marks occurred on the current page, all three mark variables are equal to the \botmark of the previous page. For boxes generated by a \vsplit command (see previous chapter), the \splitbotmark and \splitfirstmark \splitbotmark\splitfirstmarkcontain the marks of the split-off part; \firstmark and \botmark reflect the state of what remains in the register. Marks can be used to get a section heading into the headline or footline of the page. \def\section#1{ ... \mark{#1} ... } \def\rightheadline{\hbox to \hsize {\headlinefont \botmark\hfil\pagenumber}} \def\leftheadline{\hbox to \hsize {\headlinefont \pagenumber\hfil\firstmark}} This places the title of the first section that starts on a left page in the left headline, and the title of the last section that starts on the right page in the right headline. Placing the headlines on the page is the job of the output routine; see below. It is important that no page breaks can occur in between the mark and the box that places the title: \def\section#1{ ... \penalty\beforesectionpenalty \mark{#1} \hbox{ ... #1 ...} \nobreak \vskip\aftersectionskip \noindent} Let us consider another example with headlines: often a page looks better if the headline is omitted on pages where a chapter starts. This can be implemented as follows: \def\endofchapter \chapter#1{ ... \def\chtitle{#1}\mark{1}\mark{0} ... } \def\theheadline{\expandafter\ifx\firstmark1 \else \chapheadline \fi} Victor Eijkhout – TEX by Topic

237

Chapter 28. Output Routines

Only on the page where a chapter starts will the mark be 1, and on all other pages a headline is placed.

28.4

Assorted remarks

28.4.1

Hazards in non-trivial output routines

If the final call to the output routine does not perform a \shipout, TEX will call the output routine endlessly, since a run will only stop if both the vertical list is empty, and \deadcycles is zero. The output routine can set \deadcycles to zero to prevent this. 28.4.2

Page numbering

The page number is not an intrinsic property of the output routine; in plain TEX it is the value of \count0. The output routine is responsible for increasing the page number when a shipout of a page occurs. Apart from \count0, counter registers 1–9 are also used for page identification: at shipout TEX writes the values of these ten counters to the dvi file (see Chapter 33). Terminal and log file output display only the non-zero counters, and the zero counters for which a non-zero counter with a higher number exists, that is, if \count0 = 1 and \count3 = 5 are the only non-zero counters, the displayed list of counters is [1.0.0.5]. 28.4.3

Headlines and footlines in plain TEX

Plain TEX has token lists \headline and \footline; these are used in the macros \makeheadline and \makefootline. The page is shipped out as (more or less) \vbox{\makeheadline\pagebody\makefootline} Both headline and footline are inserted inside a \line. For non-standard headers and footers it is easier to redefine the macros \makeheadline and \makefootline than to tinker with the token lists. 28.4.4

Example: no widow lines

Suppose that one does not want to allow widow lines, but pages have in general no stretch or shrink, for instance because they only contain plain text. A solution would be to increase the page length by one line if a page turns out to be broken at a widow line. TEX’s output routine can perform this sort of trick: if the \widowpenalty is set to some recognizable value, the output routine can see by the \outputpenalty if a widow line occurred. In that case, the output routine can temporarily increase the \vsize, and let the page builder have another go at finding a break point. Here is the skeleton of such an output routine. No headers or footers are provided for. \newif\ifLargePage \widowpenalty=147 \newdimen\oldvsize \oldvsize=\vsize \output={ 238

Victor Eijkhout – TEX by Topic

28.4. Assorted remarks

\ifLargePage \shipout\box255 \global\LargePagefalse \global\vsize=\oldvsize \else \ifnum \outputpenalty=\widowpenalty \global\LargePagetrue \global\advance\vsize\baselineskip \unvbox255 \penalty\outputpenalty \else \shipout\box255 \fi \fi} The test \ifLargePage is set to true by the output routine if the \outputpenalty equals the \widowpenalty. The page box is then \unvbox ed, so that the page builder will tackle the same material once more. 28.4.5

Example: no indentation top of page

Some output routines can be classified as abuse of the output routine mechanism. The output routine in this section is a good example of this. It is imaginable that one wishes paragraphs not to indent if they start at the top of a page. (There are plenty of objections to this layout, but occasionally it is used.) This problem can be solved using the output routine to investigate whether the page is still empty and, if so, to give a signal that a paragraph should not indent. Note that we cannot use the fact here that the page builder comes into play after the insertion of \everypar: even if we could force the output routine to be activated here, there is no way for it to remove the indentation box. The solution given here lets the \everypar terminate the paragraph immediately with \par\penalty-\specialpenalty which activates the output routine. Seeing whether the pagebox is empty (after removing the empty line and any \parskip glue), the output routine then can set a switch signalling whether the retry of the paragraph should indent. There are some minor matters in the following routines, the sense of which is left for the reader to ponder. \mathchardef\specialpenalty=10001 \newif\ifPreventSwitch \newbox\testbox \topskip=10pt \everypar{\begingroup \par \penalty-\specialpenalty \everypar{\endgroup}\parskip0pt \ifPreventSwitch \noindent \else \indent \fi \global\PreventSwitchfalse } \output{ \ifnum\outputpenalty=-\specialpenalty Victor Eijkhout – TEX by Topic

239

Chapter 28. Output Routines

\setbox\testbox\vbox{\unvbox255 {\setbox0=\lastbox}\unskip} \ifdim\ht\testbox=0pt \global\PreventSwitchtrue \else \topskip=0pt \unvbox\testbox \fi \else \shipout\box255 \global\advance\pageno1 \fi} 28.4.6

More examples of output routines

A large number of examples of output routines can be found in [40] and [41].

240

Victor Eijkhout – TEX by Topic

Chapter 29 Insertions

Insertions are TEX’s way of handling floating information. TEX’s page builder calculates what insertions and how many of them will fit on the page; these insertion items are then placed in insertion boxes which are to be handled by the output routine. \insert Start an insertion item. \newinsert Allocate a new insertion class. \insertpenalties Total of penalties for split insertions. Inside the output routine, the number of held-over insertions. \floatingpenalty Penalty added when an insertion is split. \holdinginserts (TEX3 only) If this is positive, insertions are not placed in their boxes at output time. \footins Number of the footnote insertion class in plain TEX. \topins Number of the top insertion class. \topinsert Plain TEX macro to start a top insert. \pageinsert Plain TEX macro to start an insert that will take up a whole page. \midinsert Plain TEX macro that places its argument if there is space, and converts it into a top insert otherwise. \endinsert Plain TEX macro to wind up an insertion item that started with \topinsert, \midinsert, or \pageinsert.

29.1

Insertion items

Floating information, items that are not bound to a fixed place in the vertical list, such as figures or footnotes, are handled by insertions. The treatment of insertions is a strange interplay between the user, TEX’s internal workings, and the output routine. First the user specifies an insertion, which is a certain amount of vertical material; then TEX’s page builder decides what insertions should go on the current page and puts these insertions in insertion boxes; finally, the output routine has to do something with these boxes. An insertion item looks like \insert \inserth8-bit numberi{hvertical mode materiali} 241

Chapter 29. Insertions

where the 8-bit number should not be 255, because \box255 is used by TEX for passing the page to the output routine. The braces around the vertical mode material in an insertion item can be implicit; they imply a new level of grouping. The vertical mode material is processed in internal vertical mode. Values of \splittopskip, \splitmaxdepth, and \floatingpenalty are relevant for split insertions (see below); the values that are current just before the end of the group are used. Insertion items can appear in vertical mode, horizontal mode, and math mode. For the latter two modes they have to migrate to the surrounding vertical list (see page 77). After an insertion item is put on the vertical list the page builder is exercised.

29.2

Insertion class declaration

In the plain format the number for a new insertion class is allocated by \newinsert: \newinsert\myinsert % new insertion class which uses \chardef to assign a number to the control sequence. Insertion classes are allocated numbering from 254 downward. As box 255 is used for output, this allocation scheme leaves \skip255, \dimen255, and \count255 free for scratch use.

29.3

Insertion parameters

For each insertion class n four registers are allocated: • • •

•

242

\box n When the output routine is active this box contains the insertion items of class n that should be placed on the current page. \dimen n This is the maximum space allotted for insertions of class n per page. If this amount would be exceeded TEX will split insertions. \skip n Glue of this size is added the first time an insertion item of class n is added to the current page. This is useful for such phenomena as a rule separating the footnotes from the text of the page. \count n Each insertion item is a vertical list, so it has a certain height. However, the effective height, the amount of influence it has on the text height of the page, may differ from this real height. The value of \count n is then 1000 times the factor by which the height should be multiplied to obtain the effective height. Consider the following examples: – Marginal notes do not affect the text height, so the factor should be 0. – Footnotes set in double column mode affect the page by half of their height: the count value should by 500. – Conversely, footnotes set at page width underneath a page in double column mode affect both columns, so – provided that the double column mode is implemented by applying \vsplit to a double-height column – the count value should be 2000. Victor Eijkhout – TEX by Topic

29.4. Moving insertion items from the contributions list

29.4

Moving insertion items from the contributions list

The most complicated issue with insertions is the algorithm that adds insertion items to the main vertical list, and calculates breakpoints if necessary. TEX never changes the \vsize, but it diminishes the \pagegoal by the (effective) heights of the insertion items that will appear before a page break. Thus the output routine will receive a \box255 that has height \pagegoal, not necessarily \vsize. 1.

2.

3.

4.

When the first insertion of a certain class n occurs on the current page TEX has to account for the quantity \skip n. This step is executed only if no earlier insertion item of this class occurs on the vertical list – this includes insertions that were split – but \box n need not be empty at this time. If \box n is not empty, its height plus depth is multiplied by \count n/1000 and the result is subtracted from \pagegoal. Then the \pagegoal is diminished by the natural component of \skip n. Any stretch and shrink of \skip n are incorporated in \pagestretch and \pageshrink respectively. If there is a split insertion of class n on the page – this case and the previous step in the algorithm are mutually exclusive – the \floatingpenalty is added to \insertpenalties. A split insertion is an insertion item for which a breakpoint has been calculated as it will not fit on the current page in its entirety. Thus the insertion currently under consideration will certainly not wind up on the current page. After the preliminary action of the two previous points TEX will place the actual insertion item on the main vertical list, at the end of the current contributions. First it will check whether the item will fit without being split. There are two conditions to be checked: • adding the insertion item (plus all previous insertions of that class) to \box n should not let the height plus depth of that box exceed \dimen n, and • either the effective height of the insertion is negative, or \pagetotal plus \pagedepth minus \pageshrink plus the effective size of the insertion should be less than \pagegoal. If these conditions are satisfied, \pagegoal is diminished by the effective size of the insertion item, that is, by the height plus depth, multiplied by \countn/1000. Insertions that fail on one of the two conditions in the previous step of the algorithm will be considered for splitting. TEX will calculate the size of the maximal portion to be split off the insertion item, such that (a) adding this portion together with earlier insertions of this class to \box n will not let the size of the box exceed \dimen n, and (b) the effective size of this portion, added to \pagetotal plus \pagedepth, will not exceed \pagegoal. Note that \pageshrink is not taken into account this time, as it was in the previous step. Once this maximal size to be split off has been determined, TEX locates the least-cost breakpoint in the current insertion item that will result in a box with a height that is equal to this maximal size. The penalty associated with this breakpoint is added to \insertpenalties, and \pagegoal is diminished by the effective height plus depth of the box to be split off the insertion item.

Victor Eijkhout – TEX by Topic

243

Chapter 29. Insertions

29.5

Insertions in the output routine

When the output routine comes into action – more precisely: when TEX starts processing the tokens in the \output token list – all insertions that should be placed on the current page have been put in their boxes, and it is the responsibility of the output routine to put them somewhere in the box that is going to be shipped out. The plain TEX output routine handles top inserts and footnotes by packaging the following sequence: \ifvoid\topins \else \unvbox\topins \fi \pagebody \ifvoid\footins \else \unvbox\footins \fi Unboxing the insertion boxes makes the glue on various parts of the page stretch or shrink in a uniform manner. With TEX3 the insertion mechanism has been extended slightly: \holdinginsertsthe parameter \holdinginserts can be used to specify that insertions should not yet be placed in their boxes. This is very useful if the output routine wants to recalculate the \vsize, or if the output routine is called to do other intermediate calculations instead of ejecting a page. During the output routine the parameter \insertpenalties holds the number of insertion items that are being held over for the next page. In the plain TEX output routine this is used after the last page: \def\dosupereject{\ifnum\insertpenalties>0 % something is being held over \line{}\kern-\topskip\nobreak\vfill\supereject\fi}

29.6

Plain TEX insertions

The plain TEX format has only two insertion classes: the footnotes and the top inserts. The macro \pageinsert generates top inserts that are stretched to be exactly \vsize high. The \midinsert macro tests whether the vertical material specified by the user fits on the page; if so, it is placed there; if not, it is converted to a top insert. Footnotes are allowed to be split, but once one has been split no further footnotes should appear on the current page. This effect is attained by setting \floatingpenalty=20000 The \floatingpenalty is added to \insertpenalties if an insertion follows a split insertion of the same class. However, \floatingpenalty > 10 000 has infinite cost, so TEX will take an earlier breakpoint for splitting off the page from the vertical list. Top inserts essentially contain only a vertical box which holds whatever the user specified. Thus such an insert cannot be split. However, the \endinsert macro puts a \penalty100 on top of the box, so the insertion can be split with an empty part before the split. The effect is that the whole insertion is carried over to the next page. As the \floatingpenalty for top inserts is zero, arbitrarily many of these inserts can be moved forward until there is a page with sufficient space. Further examples of insertion macros can be found in [42].

244

Victor Eijkhout – TEX by Topic

Chapter 30 File Input and Output

This chapter discusses the various ways in which TEX can read from and write to external files. \input Read a specified file as TEX input. \endinput Terminate inputting the current file after the current line. \pausing Specify that TEX should pause after each line that is read from a file. \inputlineno Number of the current input line. \message Write a message to the terminal. \write Write a hgeneral texti to the terminal or to a file. \read Read a line from a stream into a control sequence. \newread \newwrite Macro for allocating a new input/output stream. \openin \closein Open/close an input stream. \openout \closeout Open/close an output stream. \ifeof Test whether a file has been fully read, or does not exist. \immediate Prefix to have output operations executed right away. \escapechar Number of the character that is used when control sequences are being converted into character tokens. IniTEX default: 92. \newlinechar Number of the character that triggers a new line in \write and \message statements.

30.1

Including files: \input and \endinput

Large documents can be segmented in TEX by putting \inputparts in separate input files, and loading these with \input into the master file. The exact syntax for file names is implementation dependent; most of the time a .tex file extension is assumed if no explicit extension is given. File names can be delimited with a space or with \relax. The \input command is expandable. If TEX encounters in an input file the \endinput statement, it acts as if the file ends after the line on which the statement occurs. Any statements on the same line as \endinput are still executed. The \endinput statement is expandable. 245

Chapter 30. File Input and Output

30.2

File I/O

TEX supports input and output streams for reading and writing files one line at a time. 30.2.1

Opening and closing streams

TEX supports up to 16 simultaneous input and 16 output streams. The plain TEX macros \newread and \newwrite give the number of an unused stream. This number is assigned by a \chardef command. Input streams are completely independent of output streams. Input streams are opened by \openin \openinh4-bit numberihequalsihfilenamei and closed by \closein \closeinh4-bit numberi Output streams are opened by \openout \openouth4-bit numberihequalsihfilenamei and closed by \closeout \closeouth4-bit numberi If an output file does not yet exist, it is created by \openout; if it did exist, an \openout will cause it to be overwritten. The output operations \openout, \closeout, and \write can all three be prefixed by \immediate; see below. 30.2.2

Input with \read

In addition to the \input command, which reads a whole file, TEX has the \read operation, which reads one line from a file (or from the user terminal). The syntax of the read command is \readhnumberitohcontrol sequencei The effect of this statement is that one input line is read from the designated stream, and the control sequence is defined as a macro without parameters, having that line as replacement text. If the input line is not balanced with respect to braces, TEX will read more than one line, continuing for as long as is necessary to get a balanced token list. TEX implicitly appends an empty line to each input stream, so the last \read operation on a stream will always yield a single \par token. Read operations from any stream outside the range 0–15 – or streams not associated with an open file, or on which the file end has been reached – read from the terminal. If the stream number is positive the user is prompted with the name of the control sequence being defined by the \read statement. \read16 to \data displays a prompt \data= and typing ‘my name’ in response makes the read statement equivalent to 246

Victor Eijkhout – TEX by Topic

30.3. Whatsits

\def\data{my name } The space at the end of the input derives from the line end; to prevent this one could write {\endlinechar=-1 \global\read16 to \data} 30.2.3

Output with \write

TEX’s \write command \writehnumberihgeneral texti writes a balanced token list to a file which has been opened by \openout, to the log file, or to the terminal. Write operations to a stream outside 0–15 – or to a stream that is not associated with an open file – go to the log file; if the stream number is positive they go to the terminal as well as to the log file. The token list argument of \write, defined as hgeneral texti −→ hfilleri{hbalanced textihright bracei can have an implicit opening brace. This argument is expanded as if it were the replacement text of an \edef, so, for instance, any macros and conditionals appearing are expanded. No commands are executed, however. This expansion occurs at the time of shipping out; see below. Until that time the argument token list is stored in a whatsit item on the current list. See further Chapter 12 for a discussion of expansion during writing. A control sequence output by \write (or \message) is represented with a trailing space, and using character number \escapechar for the escape character. The IniTEX default for this is 92, the code for the backslash. The trailing space can be prevented by prefixing the control sequence with \string.

30.3

Whatsits

There is an essential difference in execution between input and output: operations concerning output (\openout, \closeout, \write) are executed asynchronously. That is, instead of being done immediately they are saved until the box in which they appear is shipped out to the dvi file. Writes and the other two output operations are placed in ‘whatsit’ items on whichever list is currently being built. The actual operation occurs when the part of the page that has the item is shipped out to the dvi file. This delayed output is made necessary by TEX’s asynchronous output routine behaviour. See a worked-out example on page 134. An \immediate\write – or any other \immediate output operation – is executed on the spot, and does not place a whatsit item on the current list. The argument of a \special command (see page 264) is also placed in a whatsit. Whatsit items in leader boxes are ignored. Victor Eijkhout – TEX by Topic

247

Chapter 30. File Input and Output

30.4

Assorted remarks

30.4.1

Inspecting input

TEX records the current line number in the current input file in the hinternal integeri parameter \inputlineno (in TEX3). If the parameter \pausing is positive, TEX shows every line that is input on the terminal screen, and gives the user the opportunity to insert commands. These can for instance be \show commands. Inserted commands are treated as if they were directly in the source file: it is for instance not necessary to prefix them with ‘i’, as would be necessary when TEX pauses for an error. 30.4.2

Testing for existence of files

TEX is not the friendliest of systems when you ask it to input a non-existing file. Therefore the following sequence of commands can be used to prevent trouble: \newread\instream \openin\instream= fname.tex \ifeof\instream \message{File ’fname’ does not exist!} \else \closein\instream \input fname.tex \fi Here an input stream is opened with the given file name. The end-of-file test is also true if an input stream does not correspond to a physical file, so if this conditional is not true, the file exists and an \input command can safely be given. 30.4.3

Timing problems

The synchronization between write operations on the one hand, and opening/closing operations of files on the other hand, can be a crucial point. Auxiliary files, such as are used by various formats to implement cross-references, are a good illustration of this. Suppose that during a run of TEX the auxiliary file is written, and at the end of the run it has to be input again for a variety of purposes (such as seeing whether references have changed). An \input command is executed right away, so the file must have been closed with an \immediate\closeout. However, now it becomes possible that the file is closed before all writes to it have been performed. The following sequence remedies this: \par\vfil\penalty -10000 \immediate\closeout\auxfile The first three commands activate the output routine in order to close off the last page, so all writes will indeed have been performed before the file is closed. 30.4.4

\message versus \immediate\write16

Messages to the user can be given using \messagehgeneral texti, which writes to the terminal. Messages are appended to one another; the line is wrapped when the line length (a TEX compile-time constant) has been reached. In TEX version2, a maximum of 1000 characters is written per message; this is not a compile-time constant, but is hard-wired into the TEX program. 248

Victor Eijkhout – TEX by Topic

30.4. Assorted remarks

Each message given with \immediate\write starts on a new line; the user can force a new line in the message by including the character with number \newlinechar. This parameter also works in \message. The plain TEX default for \newlinechar is -1; the LATEX default of 10 allows you to write \message{two^^Jlines}. 30.4.5

Write inside a vertical box

Since a write operation winds up on the vertical list in a whatsit, issuing one at the start of a \vtop will probably influence the height of that box (see Chapter 5). As an example, have the \vtop{\write\terminal{Hello!}\hbox{more text}} dangling from will have the

dangling from the baseline (and when this book is TEXed the message more text ‘Hello!’ appears on the screen). 30.4.6

Expansion and spaces in \write and \message

Both \write and \message expand their argument as if it were the replacement text of an \edef. Therefore \def\a{b}\message{\a} will write out ‘b’. Unexpandable control sequences are displayed with a trailing space (and prefixed with the \escapechar): \message{\hbox\vbox!} will write out ‘\hbox \vbox !’. Undefined control sequences give an error here. Expandable control sequences can be written out with some care: \message{\noexpand\ifx} \message{\string\ifx} {\let\ifx\relax \message{\ifx}} all write out ‘\ifx’. Note, however, that spaces after expandable control sequences are removed in the input processor, which goes into state S after a control sequence. Therefore \def\a{b}\def\c{d} \message{\a \c} writes out ‘bd’. Inserting a space can be done as follows: \def\space{ } % in plain TeX \message{\a\space\c} displays ‘b d’. Note that \message{\a{ }\c} does not work: it displays ‘b{ }d’ since braces are unexpandable character tokens.

Victor Eijkhout – TEX by Topic

249

Chapter 30. File Input and Output

250

Victor Eijkhout – TEX by Topic

Chapter 31 Allocation

TEX has registers of a number of types. For some of these, explicit commands exist to define a synonym for a certain register; for all of them macros exist in the plain format to allocate an unused register. This chapter treats the synonym and allocation commands, and discusses some guidelines for macro writers regarding allocation. \countdef Define a synonym for a \count register. \dimendef Define a synonym for a \dimen register. \muskipdef Define a synonym for a \muskip register. \skipdef Define a synonym for a \skip register. \toksdef Define a synonym for a \toks register. \newbox Allocate an unused \box register. \newcount Allocate an unused \count register. \newdimen Allocate an unused \dimen register. \newfam Allocate an unused math family. \newinsert Allocate an unused insertion class. \newlanguage (TEX3 only) Allocate a new language number. \newmuskip Allocate an unused \muskip register. \newskip Allocate an unused \skip register. \newtoks Allocate an unused \toks register. \newread Allocate an unused input stream. \newwrite Allocate an unused output stream.

31.1

Allocation commands

In plain TEX, \new... macros are defined for allocation of registers. The registers of TEX fall into two classes that are allocated in different ways. This is treated below. The \newlanguage macro of plain TEX does not allocate any register. Instead it merely assigns a number, starting from 0. TEX (version 3) can have at most 256 different sets of hyphenation patterns. The \new... macros of plain TEX are defined to be \outer (see Chapter 11 for a precise explanation), which precludes use of the allocation macros in other macros. Therefore the LATEX format redefines these macros without the \outer prefix. 251

Chapter 31. Allocation

31.1.1

\count, \dimen, \skip, \muskip, \toks

For these registers there exists a hregisterdefi command, for instance \countdef, to couple a specific register to a control sequence: hregisterdefihcontrol sequenceihequalsih8-bit numberi After the definition \countdef\MyCount=42 the allocated register can be used as \MyCount=314 or \vskip\MyCount\baselineskip The hregisterdefi commands are used in plain TEX macros \newcount et cetera that allocate an unused register; after \newcount\MyCount \MyCount can be used exactly as in the above two examples. 31.1.2

\box, \fam, \write, \read, \insert

For these registers there exists no hregisterdefi command in TEX, so \chardef is used to allocate box registers in the corresponding plain TEX macros \newbox, for instance. The fact that \chardef is used implies that the defined control sequence does not stand for the register itself, but only for its number. Thus after \newbox\MyBox it is necessary to write \box\MyBox Leaving out the \box means that the character in the current font with number \MyBox is typeset. The \chardef command is treated further in Chapter 3.

31.2

Ground rules for macro writers

The \new... macros of plain TEX have been designed to form a foundation for macro packages, such that several of such packages can operate without collisions in the same run of TEX. In appendix B of the TEX book Knuth formulates some ground rules that macro writers should adhere to. 1.

252

The \new... macros do not allocate registers with numbers 0–9. These can therefore be used as ‘scratch’ registers. However, as any macro family can use them, no assumption can be made about the permanency of their contents. Results that are to be passed from one call to another should reside in specifically allocated registers. Note that count registers 0–9 are used for page identification in the dvi file (see Chapter 33), so no global assignments to these should be made. Victor Eijkhout – TEX by Topic

31.2. Ground rules for macro writers

2.

3.

4.

5.

\count255, \dimen255, and \skip255 are also available. This is because inserts are allocated from 254 downward and, together with an insertion box, a count, dimen, and skip register, all with the same number, are allocated. Since \box255 is used by the output routine (see Chapter 28), the count, dimen, and skip with number 255 are freely available. Assignments to scratch registers 0, 2, 4, 6, 8, and 255 should be local; assignments to registers 1, 3, 5, 7, 9 should be \global (with the exception of the \count registers). This guideline prevents ‘save stack build-up’ (see Chapter 35). Any register can be used inside a group, as TEX’s grouping mechanism will restore its value outside the group. There are two conditions on this use of a register: no global assignments should be made to it, and it must not be possible that other macros may be activated in that group that perform global assignments to that register. Registers that are used over longer periods of time, or that have to survive in between calls of different macros, should be allocated by \new....

Victor Eijkhout – TEX by Topic

253

Chapter 31. Allocation

254

Victor Eijkhout – TEX by Topic

Chapter 32 Running TEX

This chapter treats the run modes of TEX, and some other commands associated with the job being processed. \everyjob Token list that is inserted at the start of each new job. \jobname Name of the main TEX file being processed. \end Command to finish off a run of TEX. \bye Plain TEX macro to force the final output. \pausing Specify that TEX should pause after each line that is read from a file. \errorstopmode TEX will ask for user input on the occurrence of an error. \scrollmode TEX fixes errors itself, but will ask the user for missing files. \nonstopmode TEX fixes errors itself, and performs an emergency stop on serious errors such as missing input files. \batchmode TEX fixes errors itself and performs an emergency stop on serious errors such as missing input files, but no terminal output is generated.

32.1

Jobs

TEX associates with each run a name for the file being processed: the \jobname. If TEX is run interactively – meaning that it has been invoked without a file argument, and the user types commands – the jobname is texput. The \jobname can be used to generate the names of auxiliary files to be read or written during the run. For instance, for a file story.tex the \jobname is story, and writing \openout\Auxiliary=\jobname.aux \openout\TableOfContents=\jobname.toc will create the files story.aux and story.toc. 32.1.1

Start of the job

TEX starts each job by inserting the \everyjob token list into the command stream. Setting this variable during a run of TEX has no use, but a format can use it to identify itself to the user. If a format fills the token list, the commands therein are automatically executed when TEX is run using that format. 255

Chapter 32. Running TEX

32.1.2

End of the job

A TEX job is terminated by the \end command. This may involve first forcing the output routine to process any remaining material (see Chapter 27). If the end of job occurs inside a group TEX will give a diagnostic message. The \end command is not allowed in internal vertical mode, because this would be inside a vertical box. Usually some sugar coating of the \end command is necessary. For instance the plain TEX macro \bye is defined as \def\bye{\par\vfill\supereject\end} where the \supereject takes care of any leftover insertions. 32.1.3

The log file

For each run TEX creates a log file. Usually this will be a file with as name the value of \jobname, and the extension .log. Other extensions such as .lis are used by some implementations. This log file contains all information that is displayed on the screen during the run of TEX, but it will display some information more elaborately, and it can contain statistics that are usually not displayed on the screen. If the parameter \tracingonline has a positive value, all the log file information will be shown on the screen. Overfull and underfull boxes are reported on the terminal screen, and they are dumped using the parameters \showboxdepth and \showboxbreadth in the log file (see Chapter 34). These parameters are also used for box dumps caused by the \showbox command, and for the dump of boxes written by \shipout if \tracingoutput is set to a positive value. Statistics generated by commands such as \tracingparagraphs will be written to the log file; if \tracingonline is positive they will also be shown on the screen. Output operations to a stream that is not open, or to a stream with a number that is not in the range 0–15, go to the log file. If the stream number is positive, they also go to the terminal.

32.2

Run modes

By default, TEX goes into \errorstopmode if an error occurs: \errorstopmodeit stops and asks for input from the user. Some implementations have a way of forcing TEX into errorstopmode when the user interrupts TEX, so that the internal state of TEX can be inspected (and altered). See page 275 for ways to switch the run mode when TEX has been interrupted. Often, TEX can fix an error itself if the user asks TEX just to continue (usually by hitting the return key), but sometimes (for instance in alignments) it may take a while before TEX is on the right track again (and sometimes it never is). In such cases the user may want to turn on \scrollmode, which instructs TEX to fix as best it can any occurring error without confirmation from the user. This is usually done by typing ‘s’ when TEX asks for input. In \scrollmode, TEX also does not ask for input after \show... commands. However, some errors, such as a file that could not be found for \input, are not so easily remedied, so the user will still be asked for input. 256

Victor Eijkhout – TEX by Topic

32.2. Run modes

With \nonstopmode TEX will scroll through errors and, in the case of the kind of error that cannot be recovered from, it will make an emergency stop, aborting the run. Also TEX will abort the run if a \read is attempted from the terminal. The \batchmode differs only from nonstopmode in that it gives messages only to the log file, not to the terminal.

Victor Eijkhout – TEX by Topic

257

Chapter 32. Running TEX

258

Victor Eijkhout – TEX by Topic

Chapter 33 TEX and the Outside World

This chapter treats those commands that bear relevance to dvi files and formats. It gives some global information about IniTEX, font and format files, Computer Modern typefaces, and WEB. \dump Dump a format file; possible only in IniTEX, not allowed inside a group. \special Write a hbalanced texti to the dvi file. \mag 1000 times the magnification of the document. \year The year of the current job. \month The month of the current job. \day The day of the current job. \time Number of minutes after midnight that the current job started. \fmtname Macro containing the name of the format dumped. \fmtversion Macro containing the version of the format dumped.

33.1

TEX, IniTEX, VirTEX

In the terminology established in TEX: the Program, [25], TEX programs come in three flavours. IniTEX is a version of TEX that can generate formats; VirTEX is a production version without preloaded format, and TEX is a production version with preloaded (plain) format. Unfortunately, this terminology is not adhered to in general. A lot of systems do not use preloaded formats (the procedure for making them may be impossible on some operating systems), and call the ‘virgin TEX’ simply TEX. This manual also follows that convention. 33.1.1

Formats: loading

A format file (usually with extension .fmt) is a compact dump of TEX’s internal structures. Loading a format file takes a considerably shorter time than would be needed for loading the font information and the macros that constitute the format. Both TEX and IniTEX can load a format; the user specifies this by putting the name on the command line % tex &plain or at the ** prompt 259

Chapter 33. TEX and the Outside World

% tex This is TeX. Version .... ** &plain preceded by an ampersand (for UNIX, this should be \& on the command line). An input file name can follow the format name in both places. IniTEX does not need a format, but if no format is specified for (Vir)TEX, it will try to load the plain format, and halt if that cannot be found. 33.1.2

Formats: dumping

IniTEX is the only version of TEX that can dump a format, since it is the only version of TEX that has the command \dump, which causes the internal structures to be dumped as a format. It is also the only version of TEX that has the command \patterns, which is needed to specify a list of hyphenation patterns. Dumping is not allowed inside a group, that is { ... \dump } is not allowed. This restriction prevents difficulties with TEX’s save stack. After the \dump command TEX gives an elaborate listing of its internal state, and of the font names associated with fonts that have been loaded and ends the job. An interesting possibility arises from the fact that IniTEX can both load and dump a format. Suppose you have written a set of macros that build on top of plain TEX, superplain.tex. You could then call % initex &plain superplain *\dump and get a format file superplain.fmt that has all of plain, and all of your macros. 33.1.3

Formats: preloading

On some systems it is possible to interrupt a running program, and save its ‘core image’ such that this can be started as an independent program. The executable made from the core image of a TEX program interrupted after it has loaded a format is called a TEX program with preloaded format. The idea behind preloaded formats is that interrupting TEX after it has loaded a format, and making this program available to the user, saves in each run the time for loading the format. In the good old days when computers were quite a bit slower this procedure made sense. Nowadays, it does not seem so necessary. Besides, dumping a core image may not always be possible. 33.1.4

The knowledge of IniTEX

If no format has been loaded, IniTEX knows very little. For instance, it has no open/close group characters. However, it can not be completely devoid of knowledge lest there be no way to define anything. Here is the extent of its knowledge. • \catcode‘\\=0, \escapechar=‘\\ (see page 30). 260

Victor Eijkhout – TEX by Topic

33.2. More about formats

• • • • • •

•

\catcode‘\^^M=5, \endlinechar=‘\^^M (see page 30). \catcode‘\ =10 (see page 31). \catcode‘\%=14 (see page 31). \catcode‘\^^?=15 (see page 31). \catcode x =11 for x = ‘a..‘z,‘A..‘Z (see page 31). \catcode x =12 for all other character codes (see page 31). \sfcode x =999 for x = ‘A..‘Z, \sfcode x =1000 for all other characters (see page 188). \lccode‘a..‘z,‘A..‘Z=‘a..‘z, \uccode‘a..‘z,‘A..‘Z=‘A..‘Z, \lccode x =0, \uccode x =0 for all other characters (see page 50). \delcode‘.=0, \delcode x =-1 for all other characters (see page 193). \mathcode x ="!7100 +x for all lowercase and uppercase letters, \mathcode x ="!7000 +x for all digits, \mathcode x = x for all other characters (see page 198). \tolerance=10000, \mag=1000, \maxdeadcycles=25.

33.1.5

Memory sizes of TEX and IniTEX

• • • •

The main memory size of TEX and IniTEX is controlled by four constants in the source code: mem bot, mem top, mem min, and mem max. For IniTEX’s memory mem bot = mem min and mem top = mem max; for TEX mem bot and mem top record the main memory size of the IniTEX used to dump the format. Thus versions of TEX and IniTEX have to be adapted to each other in this respect. TEX’s own main memory can be bigger than that of the corresponding IniTEX: in general mem min ≤ mem bot and mem top ≤ mem max. For IniTEX a smaller main memory can suffice, as this program is typically not meant to do real typesetting. There may even be a real need for the main memory to be smaller, because IniTEX needs a lot of auxiliary storage for initialization and for building the hyphenation table.

33.2

More about formats

33.2.1

Compatibility

TEX has a curious error message: ‘Fatal format error: I’m stymied’, which is given if TEX tries to load a format that was made with an incompatible version of IniTEX. See the point above about memory sizes, and Chapter 35 for the hash size (parameters hash size and hash prime) and the hyphenation exception dictionary (parameter hyph size). 33.2.2

Preloaded fonts

During a run of TEX the only information needed about fonts is the data that is found in the tfm files (see below). Since a run of TEX, especially if the input contains math material, can easily access 30–40 fonts, the disk access for all the tfm files can become significant. Therefore the plain format and LATEX load these metrics files in IniTEX. A TEX version using such a format does not need to load any tfm files. On the other hand, if a format has the possibility of accessing a range of typefaces, it may be advantageous to have metrics files loaded on demand during the actual run of TEX. Victor Eijkhout – TEX by Topic

261

Chapter 33. TEX and the Outside World

33.2.3

The plain format

The first format written for TEX, and the basis for all later ones, is the plain format, described in the TEX book. It is a mixture of • definitions and macros one simply cannot live without such as the initial \catcode assignments, all of the math delimiter definitions, and the \new... macros; • constructs that are useful, but for which LATEX and other packages use a different implementation, such as the tabbing environment; and • some macros that are insufficient for any but the simplest applications: \item and \beginsection are in this category. It is the first category which Knuth meant to serve as a foundation for future macro packages, so that they can live peacefully together (see Chapter 31). This idea is reflected in the fact that the name ‘plain’ is not capitalized: it is the basic set of macros. 33.2.4

The LATEX format

The LATEX format, written by Leslie Lamport of Digital Equipment Corporation and described in [31], was released around 1985. The LATEX format, using its own version of plain.tex (called lplain.tex), is not compatible with plain TEX; a number of plain macros are not available. Still, it contains large parts of the plain format (even when they overlap with its own constructs). LATEX is a powerful format with facilities such as marginal notes, floating objects, cross referencing, and automatic table of contents generation. Its main drawback is that the ‘style files’ which define the actual layout are quite hard to write (although LATEX is in the process of a major revision, in which this problem will be tackled; see [36] and [35]). As a result, people have had at their disposal mostly the styles written by Leslie Lamport, the layout of which is rather idiosyncratic. See [6] for a successful attempt to replace these styles. 33.2.5

Mathematical formats

There are two formats with extensive facilities for mathematics typesetting: AmsTEX [45] (which originated at the American Mathematical Society) and LAMSTEX [46]. The first of these includes more facilities than plain TEX or LATEX for typesetting mathematics, but it lacks features such as automatic numbering and cross-referencing, available in LATEX, for instance. LAMSTEX, then, is the synthesis of AmsTEX and LATEX. Also it includes still more features for mathematics, such as complicated tables and commutative diagrams. 33.2.6

Other formats

Other formats than the above exist: for instance, Phyzzx [53], TeXsis [37], Macro TEX [17], eplain [4], and TEXT1 [15]. Typically, such formats provide the facilities of LATEX, but try to be more easily adaptable by the user. Also, in general they have been written with the intention of being an add-on product to the plain format. This book was, in its incarnation published by Addison-Wesley, also written in an ‘other format’: the Lollipop format. This format does not contain user macros, but the tools with which a style designer can program them; see [14]. The current version of this book is written in LATEX. 262

Victor Eijkhout – TEX by Topic

33.3. The dvi file

33.3

The dvi file

The dvi file (this term stands for ‘device independent’) contains the output of a TEX run: it contains compactly dumped representations of boxes that have been sent there by \shipouthboxi. The act of shipping out usually occurs inside the output routine, but this is not necessarily so. 33.3.1

The dvi file format

A dvi file is a byte-oriented file, consisting of a preamble, a postamble, and a list of pages. Access for subsequent software to a completed dvi file is strictly sequential in nature: the pages are stored as a backwards linked list. This means that only two ways of accessing are possible: • given the start of a page, the next can be found by reading until an end-of-page code is encountered, and • starting at the end of the file pages can be read backwards at higher speed, as each beginning-of-page code contains the byte position of the previous one. The preamble and postamble contain • the magnification of the document (see below), • the unit of measurement used for the document, and • possibly a comment string. The postamble contains in addition a list of the font definitions that appear on the pages of the file. Neither the preamble nor the postamble of the file contains a table of byte positions of pages. The full definition of the dvi file format can be found in [25]. 33.3.2

Page identification

Whenever a \shipout occurs, TEX also writes the values of counters 0–9 to the dvi file and the terminal. Ordinarily, only counter 0, the page number, is used, and the other counters are zero. Those zeros are not output to the terminal. The other counters can be used to indicate further structure in the document. Log output shows the non-zero counters and the zero counters in between. 33.3.3

Magnification

The magnification of a document can be indicated by the hintegeri \mag, which specifies 1000 times the magnification ratio. The dvi file contains the value of \mag for the document in its preamble and postamble. If no true dimensions are used the dvi file will look the same as when no magnification would have been used, except for the \mag value in the preamble and the postamble. Whenever a true dimension is used it is divided by the value of \mag, so that the final output will have the dimension as prescribed by the user. The \mag parameter cannot be changed after a true dimension has been used, or after the first page has been shipped to the dvi file. Plain TEX has the \magnification macro for globally sizing the document, without changing the physical size of the page: Victor Eijkhout – TEX by Topic

263

Chapter 33. TEX and the Outside World

\def\magnification{\afterassignment\m@g\count@} \def\m@g{\mag\count@ \hsize6.5truein\vsize8.9truein\dimen\footins8truein} The explanation for this is as follows: the command \m@g is saved with an \afterassignment command, and the magnification value (which is 1000 times the actual magnification factor) is assigned to \count@. After this assignment, the macro \m@g assigns the magnification value to \mag, and the horizontal and vertical size are reset to their original values 6.5truein and 8.9truein. The \footins is also reset.

33.4

Specials

TEX is to a large degree machineindependent, but it still needs a hook for machine-dependent extensions. This is done through specials. The \special command writes a hbalanced texti to the dvi file which TEX does not interpret like other token lists: it assumes that the printer driver knows what to do with it. The \special command is not supposed to change the x and y position on the page, so that the implementation of TEX remains independent of the actual device driver that handles the \special. The most popular application of specials is probably the inclusion of graphic material, written in some page description language, such as PostScript . The size of the graphics can usually be determined from the file containing it (in the case of encapsulated PostScript through the ‘bounding box’ data), so TEX can leave space for such material.

33.5

Time

TEX has four parameters, \year, \month, \day, and \time, that tell the time and date when the current job started. After this, the parameters are not updated. The user can change them without this having any effect. All four parameters are integers; the \time parameter gives the number of minutes since midnight that the current job started.

33.6

Fonts

Font information is split in the TEX system into the metric information (how high, wide, and deep is a character), and the actual description of the characters in a font. TEX, the formatter, needs only the metric information; printer drivers and screen previewers need the character descriptions. With this approach it is for instance possible for TEX to use with relative ease the resident fonts of a printer. 33.6.1

Font metrics

The metric information of TEX’s fonts is stored in tfm files, which stands for ‘TEX font metrics’ files. Metrics files contain the following information (see [25] for the full definition): 264

Victor Eijkhout – TEX by Topic

33.6. Fonts

• • • • • •

the design size of a font; the values for the \fontdimen parameters (see Chapter 4); the height, depth, width, and italic correction of individual characters; kerning tables; ligature tables; information regarding successors and extensions of math characters (see Chapter 21).

Metrics files use a packed format, but they can be converted to and from a readable format by the auxiliary programs tftopl and pltotf (see [28]). Here pl stands for ‘property list’, a term deriving from the programming language Lisp. Files in pl format are just text, so they can easily be edited; after conversion they can then again be used as tfm files. 33.6.2

Virtual fonts

With virtual fonts (see [26]) it is possible that what looks like one font to TEX resides in more than one physical font file. Also, virtual fonts can be used to change in effect the internal organization of font files. For TEX itself, the presence of virtual fonts makes no difference: everything is still based on tfm files containing metric information. However, the screen or printer driver that displays the resulting dvi file on the screen or on a printer will search for files with extension .vf to determine how characters are to be interpreted. The vf file can, for instance, instruct the driver to interpret a character as a certain position in a certain font file, to interpret a character as more than one position (a way of forming accented characters), or to include \special information (for instance to set gray levels). Readable variants of vf files have extension vpl, analogous to the pl files for the tfm files; see above. Conversion between vf and vpl files can be performed with the vftovp and vptovf programs. However, because virtual fonts are a matter for device drivers, no more details will be given in this book. 33.6.3

Font files

Character descriptions are stored in three types of files. gf Generic Font files. This is the file type that the Metafont program generates. There are not many previewers or printer drivers that use this type of file directly. pxl Pixel files. The pxl format is a pure bitmap format. Thus it is easy to generate pxl files from, for instance, scanner images. This format should be superseded by the pk format. Pixel files can become rather big, as their size grows quadratically in the size of the characters. pk Packed files. Pixel files can be packed by a form of run-length encoding: instead of storing the complete bitmap only the starting positions and lengths of ‘runs’ of black and white pixels are stored. This makes the size of pk files approximately linear in the size of the characters. However, a previewer or printer driver using a packed font file has to unpack it before it is able to use it. The following conversion programs exist: gftopxl, gftopk, pktopxl, pxltopk. Victor Eijkhout – TEX by Topic

265

Chapter 33. TEX and the Outside World

33.6.4

Computer Modern

The only family of typefaces that comes with TEX in the standard distribution is the Computer Modern family of typefaces. This is an adaptation (using the terminology of [44]) by Donald Knuth of the Monotype Modern 8A typeface that was used for the first volume of his Art of Computer Programming series. The ‘modern faces’ all derive from the types that were cut between 1780 and 1800 by Firmin Didot in France, Giambattista Bodoni in Italy, and Justus Erich Walbaum in Germany. After the first two, these types are also called ‘Didone’ types. This name was coined in the Vox classification of types [52]. Ultimately, the inspiration for the Didone types is the ‘Romain du Roi’, the type that was designed by Nicolas Jaugeon around 1692 for the French Imprimerie Royale. Didone types are characterized by a strong vertical orientation, and thin hairlines. The vertical accent is strengthened by the fact that the insides of curves are flattened. The result is a clear and brilliant page, provided that the printing is done carefully and on good quality paper. However, they are quite vulnerable; [50] compares them to the distinguished but fragile furniture from the same period, saying one is afraid to use either, ‘for both seem in danger of breaking in pieces’. With the current proliferation of low resolution (around 300 dot per inch) printers, the Computer Modern is a somewhat unfortunate choice. Recently, Donald Knuth has developed a new typeface (or rather, a subfamily of typefaces) by changing parameters in the Computer Modern family. The result is a so-called ‘Egyptian’ typeface: Computer Concrete [24]. The name derives from the fact that it was intended primarily for the book Concrete Mathematics. Egyptian typefaces (they fall under the ‘M´ecanes’ in the Vox classification, meaning constructed, not derived from written letters) have a very uniform line width and square serifs. They do not have anything to do with Egypt; such types happened to be popular in the first half of the nineteenth century when Egyptology was developing and popular.

33.7

TEX and web

The TEX program is written in WEB, a programming language that can be considered as a subset of Pascal , augmented with a preprocessor. TEX makes no use of some features of Pascal, in order to facilitate porting to Pascal systems other than the one it was originally designed for, and even to enable automatic translation to other programming languages such as C. For instance, it does not use the Pascal With construct. Also, procedures do not have output parameters; apart from writing to global variables, the only way values are returned is through Function values. Actually, WEB is more than a superset of a subset of Pascal (and to be more precise, it can also be used with other programming languages); it is a ‘system of structured documentation’. This means that the WEB programmer writes pieces of program code, interspersed with their documentation, in one file. This idea of ‘literate programming’ was introduced in [21]; for more information, see [43]. Two auxiliary programs, Tangle and Weave, can then be used to strip the documentation and convert WEB into regular Pascal, or to convert the WEB file into a TEX file that will typeset the program and documentation. 266

Victor Eijkhout – TEX by Topic

33.8. The TEX Users Group

Portability of WEB programs is achieved by the ‘change file’ mechanism. A change file is a list of changes to be made to the WEB file; a bit like a stream editor script. These changes can comprise both adaptations of the WEB file to the particular Pascal compiler that will be used, and bug fixes to TEX. Thus the TeX.web file need never be edited.

33.8

The TEX Users Group

TEX users have joined into several users groups over the last decade. Many national or language users groups exist, and a lot of them publish newsletters. The oldest of all TEX users groups is simply called that: the TEX Users Group, or TUG , and its journal is called TUGboat . You can reach them at TEX Users Group P.O. Box 2311 Portland, OR 97208-2311, USA or electronically at [email protected] on the Internet.

Victor Eijkhout – TEX by Topic

267

Chapter 33. TEX and the Outside World

268

Victor Eijkhout – TEX by Topic

Chapter 34 Tracing

TEX’s workings are often quite different from what the programmer expected, so there are ways to discover how TEX arrived at the result it did. The \tracing... commands all write statistics information of a certain kind to the log file (and to the terminal if \tracingonline is positive), and a number of \show... commands can be used to ask the current status or value of various items of TEX. In the following list, only \show and \showthe display their output on the terminal by default, other \show... and \tracing... commands write to the log file. They will write in addition to the terminal if \tracingonline is positive. \meaning Give the meaning of a control sequence as a string of characters. \show Display the meaning of a control sequence. \showthe Display the result of prefixing a token with \the. \showbox Display the contents of a box. \showlists Display the contents of the partial lists currently built in all modes. This is treated on page 77. \tracingcommands If this is 1 TEX displays primitive commands executed; if this is 2 or more the outcome of conditionals is also recorded. \tracingmacros If this is 1, TEX shows expansion of macros that are performed and the actual values of the arguments; if this is 2 or more htoken parameteris such as \output and \everypar are also traced. \tracingoutput If this is positive, the log file shows a dump of boxes that are shipped to the dvi file. \showboxdepth The number of levels of box dump that are shown when boxes are displayed. \showboxbreadth Number of successive elements on each level that are shown when boxes are displayed. \tracingonline If this parameter is positive, TEX will write trace information to the terminal in addition to the log file. \tracingparagraphs If this parameter is positive, TEX generates a trace of the line breaking algorithm. \tracingpages If this parameter is positive, TEX generates a trace of the page breaking algorithm. \tracinglostchars If this parameter is positive, TEX gives diagnostic messages whenever a character is accessed that is not present in a font. Plain default: 1. 269

Chapter 34. Tracing

\tracingrestores If this parameter is positive, TEX will report all values that are restored when a group ends. \tracingstats If this parameter is 1, TEX reports at the end of the job the usage of various internal arrays; if it is 2, the memory demands are given whenever a page is shipped out. \tracingassigns (eTEX extension) If this parameter is positive, all assignments are traced. \tracinggroups (eTEX extension) If this parameter is positive, group openings and closings are traced, including a report of the type of group. \tracingifs (eTEX extension) If this parameter is positive, conditionals are traced, with starting line and nesting level. \tracingscantokens (eTEX extension) This traces invocations of \scantokens. \showifs (eTEX extension) When issued, this command displays all active conditionals.

34.1

Meaning and content: \show, \showthe, \meaning

The meaning of control sequences, and the contents of those that represent internal quantities, can be obtained by the primitive commands \show, \showthe, and \meaning. The control sequences \show and \meaning are similar: the former will give \show\meaningoutput to the log file and the terminal, whereas the latter will produce the same tokens, but they are placed in TEX’s input stream. The meaning of a primitive command of TEX is that command itself: \show\baselineskip gives \baselineskip=\baselineskip The meaning of a defined quantity is its definition: \show\pageno gives \pageno=\count0 The meaning of a macro is its parameter text and replacement text: \def\foo#1?#2\par{\set{#1!}\set{#2?}} \show\foo gives \foo=macro: #1?#2\par ->\set {#1!}\set {#2?} For macros without parameters the part before the arrow (the parameter text) is empty. The \showthe command will display on the log file and terminal the tokens that \the produces. After \show, \showthe, \showbox, and \showlists TEX asks the user for input; this can be prevented by specifying \scrollmode. Characters generated by \meaning and \the have category 12, except for spaces (see page 35); the value of \escapechar is used when control sequences are represented. 270

Victor Eijkhout – TEX by Topic

34.2. Show boxes: \showbox, \tracingoutput

34.2

Show boxes: \showbox, \tracingoutput

If \tracingoutput is positive the log file will \tracingoutput\showboxreceive a dumped representation of all boxes that are written to the dvi file with \shipout. The same representation is used by the command \showboxh8-bit numberi. In the first case TEX will report ‘Completed box being shipped out’; in the second case it will enter \errorstopmode, and tell the user ‘OK. (see the transcript file)’. If \tracingonline is positive, the box is also displayed on the terminal; if \scrollmode has been specified, TEX does not stop for input.

The upper bound on the number of nested boxes that is dumped is \showboxdepth; \showboxdepth\showboxbreadtheac time a level is visited at most \showboxbreadth items are shown, the remainder of the list is summarized with etc. For each box its height, depth, and width are indicated in that order, and for characters it is stated from what font they were taken. After \font\tenroman=cmr10 \tenroman \setbox0=\hbox{g} \showbox0 the log file will show \hbox(4.30554+1.94444)x5.00002 .\tenroman g indicating that the box was 4.30554pt high, 1.94444pt deep, and 5.00002pt wide, and that it contained a character ‘g’ from the font \tenroman. Note that the fifth decimal of all sizes may be rounded because TEX works with multiples of 2−16 pt. The next example has nested boxes, \vbox{\hbox{g}\hbox{o}} and it contains \baselineskip glue between the boxes. After a \showbox command the log file output is: \vbox(16.30554+0.0)x5.00002 .\hbox(4.30554+1.94444)x5.00002 ..\tenroman g .\glue(\baselineskip) 5.75002 .\hbox(4.30554+0.0)x5.00002 ..\tenroman o Each time a new level is entered an extra dot is added to the front of the line. Note that TEX tells explicitly that the glue is \baselineskip glue; it inserts names like this for all automatically inserted glue. The value of the baselineskip glue here is such that the baselines of the boxes are at 12 point distance. Now let us look at explicit (user) glue. TEX indicates the ratio by which it is stretched or shrunk. \hbox to 20pt {\kern10pt \hskip0pt plus 5pt} gives (indicating that the available stretch has been multiplied by 2.0): \hbox(0.0+0.0)x20.0, glue set 2.0 .\kern 10.0 .\glue 0.0 plus 5.0 Victor Eijkhout – TEX by Topic

271

Chapter 34. Tracing

and \hbox to 0pt {\kern10pt \hskip0pt minus 20pt} gives (the shrink has been multiplied by 0.5) \hbox(0.0+0.0)x0.0, glue set - 0.5 .\kern 10.0 .\glue 0.0 minus 20.0 respectively. This is an example with infinitely stretchable or shrinkable glue: \hbox(4.00000+0.14000)x15.0, glue set 9.00000fil This means that the horizontal box contained fil glue, and it was set such that its resulting width was 9pt. Underfull boxes are dumped like all other boxes, but the usual ‘Underfull hbox detected at line...’ is given. Overfull horizontal boxes contain a vertical rule of width \overfullrule: \hbox to 5pt {\kern10pt} gives \hbox(0.0+0.0)x5.0 .\kern 10.0 .\rule(*+*)x5.0 Box leaders are not dumped completely: .\leaders 40.0 ..\hbox(4.77313+0.14581)x15.0, glue set 9.76852fil ...\tenrm a ...\glue 0.0 plus 1.0fil is the dump for \leaders\hbox to 15pt{\tenrm a\hfil}\hskip 40pt Preceding or trailing glue around the leader boxes is also not indicated.

34.3

Global statistics

The parameter \tracingstats can be used to force TEX to report at the end of the job the global use of resources. Some production versions of TEX may not have this option. As an example, here are the statistics for this book: Here is how much of TeX’s memory you used: String memory (bounded by ‘pool size’): 877 strings out of 4649 9928 string characters out of 61781 Main memory, control sequences, font memory: 53071 words of memory out of 262141 2528 multiletter control sequences out of 9500 20137 words of font info for 70 fonts, out of 72000 for 255 272

Victor Eijkhout – TEX by Topic

34.4. Execution tracing

Hyphenation: 14 hyphenation exceptions out of 607 Stacks: input, nest, parameter, buffer, and save stack respectively, 17i,6n,19p,245b,422s stack positions out of 300i,40n,60p,3000b,4000s

34.4

Execution tracing

If your TEX run gives strange results and you want to inspect TEX’s workings in detail, you can use: •

• • • • •

\tracingcommands shows in detail what executable commands are executed; set this to 1 to see primitives that are executed, and 2 to see if conditionals are taking the true or false branch. \tracingmacros shows you the expansion of macros if it is at least 1; with 2 the expansion of token parameters is also traced. \tracingpages lets you see the page breaking algorithm in action. a positive value of \tracingrestores will display values that are restored after a group. \tracinglostchars will cause a trace messages if you’re trying to access a position in a font that is not filled. Most of the time these traces go only to the log file, since they are rather voluminous. Set \tracingonline to a positive value to get them also to the terminal.

Victor Eijkhout – TEX by Topic

273

Chapter 34. Tracing

274

Victor Eijkhout – TEX by Topic

Chapter 35 Errors, Catastrophes, and Help

When TEX is running, various errors can occur. This chapter treats how errors in the input are displayed, and what sort of overflow of internal data structures of TEX can occur. \errorcontextlines (TEX3 only) Number of additional context lines shown in error messages. \errmessage Report an error, giving the parameter of this command as message. \errhelp Tokens that will be displayed if the user asks further help after an \errmessage.

35.1

Error messages

When TEX is running in \errorstopmode (which it usually is; see Chapter 32 for the other running modes), errors occurring are reported on the user terminal, and TEX asks the user for further instructions. Errors can occur either because of some internal condition of TEX, or because a macro has issued an \errmessage command. If an error occurs TEX shows the input line on which the error occurred. If the offending command was not on that line but, for instance, in a macro that was called – possibly indirectly – from that line, the line of that command is also shown. If the offending command was indirectly called, an additional \errorcontextlines number of lines is shown with the preceding macro calls. A value of \errorcontextlines = 0 causes ... to be printed as the sole indication that there is a context. Negative values inhibit even this. For each macro in the sequence that leads to the offending command, TEX attempts to display some preceding and some following tokens. First one line is displayed ending with the – indirectly – offending command; then, one line lower some following tokens are given. This paragraph ends \vship1cm with a skip. gives ! Undefined control sequence. l.1 This paragraph ends \vship 1cm with a skip. If TEX is not running in some non-stop mode, the user is given the chance to do some error patching, or to ask for further information. In general the following options are available: 275

Chapter 35. Errors, Catastrophes, and Help

hreturni TEX will continue processing. If the error was something innocent that TEX could either ignore or patch itself, this is the easy way out. h Give further details about the error. If the error was caused by an \errmessage command, the \errhelp tokens will be displayed here. i Insert. The user can insert some material. For example, if a control sequence is misspelled, the correct command can sometimes be inserted, as i\vskip for the above example. Also, this is an opportunity for inserting \show commands to inspect TEX’s internal state. However, if TEX is in the middle of scanning something complicated, such commands will not be executed, or will even add to the confusion. s (\scrollmode) Scroll further errors, but display the messages. TEX will patch any further errors. This is a handy option, for instance if the error occurs in an alignment, because the number of subsequent errors tends to be rather large. r (\nonstopmode) Run without stopping. TEX will never stop for user interaction. q (\batchmode) Quiet running. TEX will never stop for user interaction, and does not give any more terminal output. x Exit. Abort this run of TEX. e Edit. This option is not available on all TEX system. If it is, the run of TEX is aborted, and an editor is started, opening with the input file, maybe even on the offending line.

35.2

Overflow errors

Harsh reality imposes some restrictions on how elaborate TEX’s workings can get. Some restrictions are imposed by compile-time constants, and are therefore fairly loose, but some depend strongly on the actual computer implementation. Here follows the list of all categories of overflow that prompt TEX to report ‘Capacity exceeded’. Most bounds involved are (determined by) compile-time constants; their values given here in parentheses are those used in the source listing of TEX in [27]. Actual values may differ, and probably will. Remember that TEX was developed in the good old days when even big computers were fairly small. 35.2.1

Buffer size (500)

Current lines of all files that are open are kept in TEX’s input buffer, as are control sequence names that are being built with \csname...\endcsname. 35.2.2

Exception dictionary (307)

The maximum number of hyphenation exceptions specified by \hyphenation must be a prime number. Two arrays with this many halfwords are allocated. Changing this number makes formats incompatible; that is, TEX can only use a format that was made by an IniTEX with the same value for this constant. 35.2.3

Font memory (20,000)

Information about fonts is stored in an array of memory words. This is easily overflowed by preloading too many fonts in IniTEX. 276

Victor Eijkhout – TEX by Topic

35.2. Overflow errors

35.2.4

Grouping levels

The number of open groups should be recordable in a quarter word. There is no compile-time constant corresponding to this. 35.2.5

Hash size (2100)

Maximum number of control sequences. It is suggested that this number should not exceed 10% of the main memory size. The values in TEX and IniTEX should agree; also the hash prime values should agree. This value is rather low; for macro packages that are more elaborate than plain TEX a value of about 3000 is more realistic. 35.2.6

Number of strings (3000)

The maximum number of strings must be recordable in a half word. 35.2.7

Input stack size (200)

For each input source an item is allocated on the input stack. Typical input sources are input files (but their simultaneous number is more limited; see below), and token lists such as token variables, macro replacement texts, and alignment templates. A macro with ‘runaway recursion’ (for example, \def\mac{{\mac}}) will overflow this stack. TEX performs some optimization here: before the last call in a token list all token lists ending with this call are cleared. This process is similar to ‘resolving tail recursion’ (see Chapter 11). 35.2.8

Main memory size (30,000)

Almost all ‘dynamic’ objects of TEX, such as macro definition texts and all material on the current page, are stored in the main memory array. Formats may already take 20 000 words of main memory for macro definitions, and complicated pages containing for instance the LATEX picture environment may easily overflow this array. TEX’s main memory is divided in words, and a half word is supposed to be able to address the whole of the memory. Thus on current 32-bit computers the most common choice is to let the main memory size be at most 64K bytes. A half word address can then be stored in 16 bits, half a machine word. However, so-called ‘Big TEX’ implementations exist that have a main memory larger than 64K words. Most compilers will then allocate 32-bit words for addressing this memory, even if (say) 18 bits would suffice. Big TEXs therefore become immediately a lot bigger when they cross the 64K threshold. Thus they are usually not found on microcomputers, although virtual memory schemes for these are possible; see for instance [47]. TEX can have a bigger main memory than IniTEX; see Chapter 33 for further details. 35.2.9

Parameter stack size (60)

Macro parameters may contain macro calls with further parameters. The number of parameters that may occur nested is bounded by the parameter stack size. Victor Eijkhout – TEX by Topic

277

Chapter 35. Errors, Catastrophes, and Help

35.2.10 Pattern memory (8000) Hyphenation patterns are stored in a trie array. The default size of 8000 hyphenation patterns seems sufficient for English or Italian, for example, but it is not for Dutch or German. 35.2.11 Pattern memory ops per language The number of hyphenation ops (see the literature about hyphenation: [32] and appendix H of [27]) should be recordable in a quarter word. There is no compile-time constant corresponding to this. TEX version 2 had the same upper bound, but gave no error message in case of overflow. Again, for languages such as Dutch and German this bound is too low. There are versions of TEX that have a higher bound here. 35.2.12 Pool size (32,000) Strings are error messages and control sequence names. They are stored using one byte per character. TEX has initially about 23 000 characters worth of strings. The pool will overflow if a user defines a large number of control sequences on top of a substantial macro package. However, even if the user does not define any new commands overflow may occur: crossreferencing schemes also work by defining control sequences. For large documents a pool size of 40 000 or 60 000 is probably sufficient. 35.2.13 Save size (600) Quantities that are assigned to inside a group must be restored after the end of that group. The save stack is where the values to be restored are kept; the size of the save stack limits the number of values that can be restored. Alternating global and local assignments to a value will lead to ‘save stack build-up’: for each local assignment following a global assignment the previous value of the variable is saved. Thus an alternation of such assignments will lead to an unnecessary proliferation of items on the save stack. 35.2.14 Semantic nest size (40) Each time TEX switches to a mode nested inside another mode (for instance when processing an \hbox inside a \vbox) the current state is pushed on the semantic nest stack. The semantic nest size is the maximum number of levels that can be pushed. 35.2.15 Text input levels (6) The number of nested \input files has to be very limited, as the current lines are all kept in the input buffer.

278

Victor Eijkhout – TEX by Topic

Chapter 36 The Grammar of TEX

Many chapters in this book contain pieces of the grammar that defines the formal syntax of TEX. In this chapter the structure of the rewriting rules of the grammar is explained, and some key notions are presented. In the TEX book a grammar appears in Chapters 24–27. An even more rigorous grammar of TEX can be found in [1]. The grammar presented in this book is virtually identical to that of the TEX book.

36.1

Notations

Basic to the grammar are grammatical terms These are enclosed in angle brackets: htermi control sequences These are given in typewriter type with a backslash for the escape character: \command Lastly there are keywords Also given in typewriter type keyword This is a limited collection of words that have a special meaning for TEX in certain contexts; see below. The three elements of the grammar are used in syntax rules: hsnarki −→ boojum | hemptyi This rule says that the grammatical entity hsnarki is either the keyword boojum, or the grammatical entity hemptyi. There are two other notational conventions. The first is that the double quote is used to indicate hexadecimal (base 16) notation. For instance "ab56 stands for 10 × 163 + 11 × 162 + 5 × 161 + 6 × 160 . The second convention is that subscripts are used to denote category codes. Thus a12 denotes an ‘a’ of category 12. 279

Chapter 36. The Grammar of TEX

36.2

Keywords

A keyword is sequence of characters (or character tokens) of any category code but 13 (active). Unlike the situation in control sequences, TEX does not distinguish between lowercase and uppercase characters in keywords. Uppercase characters in keywords are converted to lowercase by adding 32 to them; the \lccode and \uccode are not used here. Furthermore, any keyword can be preceded by optional spaces. Thus both true cm and truecm are legal. By far the strangest example, however, is provided by the grammar rule hfil uniti −→ fil | hfil unitil which implies that fil L l is also a legal hfil dimeni. Strange errors can ensue from this; see page 130 for an example. Here is the full list of all keywords: at, bp, by, cc, cm, dd, depth, em, ex, fil, height, in, l, minus, mm, mu, pc, plus, pt, scaled, sp, spread, to, true, width.

36.3

Specific grammatical terms

Some grammatical terms appear in a lot of rules. One such term is hoptional spacesi. The term optional space is probably clear enough, but here is the formal definition: hoptional spacesi −→ hemptyi | hspace tokenihoptional spacesi which amounts to saying that hoptional spacesi is zero or more space tokens. Other terms may not be so immediately obvious. Below are some of them. 36.3.1

hequalsi

In assignments the equals sign is optional; therefore there is a term hequalsi −→ hoptional spacesi | hoptional spacesi=12 in TEX’s grammar. 36.3.2

hfilleri, hgeneral texti

More obscure than the hoptional spacesi is the combination of spaces and \relax tokens that is allowed in some places, for instance \setbox0= \relax\box1 The quantity involved is hfilleri −→ hoptional spacesi | hfilleri\relaxhoptional spacesi One important occurrence of hfilleri is in hgeneral texti −→ hfilleri{hbalanced textihright bracei A hgeneral texti follows such control sequences as \message, \uppercase, or \mark. The braces around the hbalanced texti are explained in the next point. 280

Victor Eijkhout – TEX by Topic

36.4. Differences between TEX versions 2 and 3

36.3.3

{} and hleft braceihright bracei

The TEX grammar uses a perhaps somewhat unfortunate convention for braces. First of all { and } stand for braces that are either explicit open/close group characters, or control sequences defined by \let, such as \let\bgroup={ \let\egroup=} The grammatical terms hleft bracei and hright bracei stand for explicit open/close group characters, that is, characters of categories 1 and 2 respectively. Various combinations of these two kinds of braces exist. Braces around boxes can be implicit: \hboxhbox specificationi{hhorizontal mode materiali} Around a macro definition there must be explicit braces: hdefinition texti −→ hparameter textihleft braceihbalanced textihright bracei Finally, the hgeneral texti that was mentioned above has to be explicitly closed, but it can be implicitly opened: hgeneral texti −→ hfilleri{hbalanced textihright bracei The closing brace of a hgeneral texti has to be explicit, since a general text is a token list, which may contain \egroup tokens. TEX performs expansion to find the opening brace of a hgeneral texti. 36.3.4

hmath fieldi

In math mode various operations such as subscripting or applying \underline take an argument that is a hmath fieldi: either a single symbol, or a group. Here is the exact definition. hmath fieldi −→ hmath symboli | hfilleri{hmath mode materiali} hmath symboli −→ hcharacteri | hmath characteri See page 48 for hcharacteri, and page 192 for hmath characteri.

36.4

Differences between TEX versions 2 and 3

In 1989 Knuth released TEX version 3.0, which is the first real change in TEX since version 2.0, which was released in 1986 (version 0 of TEX was released in 1982; see [20] for more about the history of TEX). All intermediate versions were merely bug fixes. The main difference between versions 2 and 3 lies in the fact that 8-bit input has become possible. Associated with this, various quantities that used to be 127 or 128 have been raised to 255 or 256 respectively. Here is a short list. The full description is in [22]. All ‘codes’ (\catcode, \sfcode, et cetera; see page 51) now apply to 256 character codes instead of 128. A character with code \endlinechar is appended to the line unless this parameter is negative or more than 255 (this was 127) (see page 30). Victor Eijkhout – TEX by Topic

281

Chapter 36. The Grammar of TEX

No escape character is output by \write and other commands if \escapechar is negative or more than 255 (this was 127) (see page 35). The ^^ replacement mechanism has been extended (see page 33). Parameters \language, \inputlineno, \errorcontextlines, \lefthyphenmin, \righthyphenmin, \badness, \holdinginserts, \emergencystretch, and commands \noboundary, \setlanguage have been added. The value of \outputpenalty is no longer zero if the page break was not at a penalty item; instead it is 10 000 (see page 228). The plain format has also been updated, mostly with default settings for parameters such as \lefthyphenmin, but also a few macros have been added.

282

Victor Eijkhout – TEX by Topic

Chapter 37 Glossary of TEX Primitives

This chapter gives the list of all primitives of TEX. After each control sequence the grammatical category of the command or parameter is given, plus a short description. For some commands the syntax of their use is given. For parameters the class to which they belong is given. Commands that have no grammatical category in the TEX book are denoted either ‘hexpandable commandi’ or ‘hprimitive commandi’ in this list. Grammatical terms such as hequalsi and hoptional spacei are explained in Chapter 36. \- hhorizontal commandi Discretionary hyphen; this is equivalent to \discretionary””{-}{}{}. Can be used to indicate hyphenatable points in a word. Chapter 19. \char32 hhorizontal commandi Control space. Insert the same amount of space as a space token would if \spacefactor = 1000. Chapter 2,20. \char47 hprimitive commandi Italic correction: insert a kern specified by the preceding character. Each character has an italic correction, possibly zero, specified in the tfm file. For slanted fonts this compensates for overhang. Chapter 4. \abovehdimeni hgeneralized fraction commandi Fraction with specified bar width. Chapter 23. \abovedisplayshortskip hglue parameteri Glue above a display if the line preceding the display was short. Chapter 24. \abovedisplayskip hglue parameteri Glue above a display. Chapter 24. \abovewithdelimshdelim1 ihdelim2 ihdimeni hgeneralized fraction commandi Generalized fraction with delimiters. Chapter 23. \accenth8-bit numberihoptional assignmentsihcharacteri hhorizontal commandi Command to place accents on characters.Chapter 3. \adjdemerits hinteger parameteri Penalty for adjacent not visually compatible lines. Default 10 000 in plain TEX. Chapter 19. \advancehnumeric variableihoptional byihnumberi harithmetic assignmenti Arithmetic command to increase or decrease a hnumeric variablei, that is, a hcount variablei, hdimen variablei, hglue variablei, or hmuglue variablei. Chapter 7,8. \afterassignmenthtokeni hprimitive commandi Save the next token for execution after the next assignment. Only one token can be saved this way. Chapter 12. \aftergrouphtokeni hprimitive commandi Save the next token for insertion after the current group. Several tokens can be saved this way. Chapter 10. 283

Chapter 37. Glossary of TEX Primitives

\atophdimeni hgeneralized fraction commandi Place objects over one another. Chapter 23. \atopwithdelimshdelim1 ihdelim2 i hgeneralized fraction commandi Place objects over one another with delimiters. Chapter 23. \badness hinternal integeri (TEX3 only) Badness of the most recently constructed box. Chapter 5. \baselineskip hglue parameteri The ‘ideal’ baseline distance between neighbouring boxes on a vertical list; 12pt in plain TEX. Chapter 15. \batchmode hinteraction mode assignmenti TEX patches errors itself and performs an emergency stop on serious errors such as missing input files, but no terminal output is generated. Chapter 32. \begingroup hprimitive commandi Open a group that must be closed with \endgroup. Chapter 10. \belowdisplayshortskip hglue parameteri Glue below a display if the line preceding the display was short. Chapter 24. \belowdisplayskip hglue parameteri Glue below a display. Chapter 24. \binoppenalty hinteger parameteri Penalty for breaking after a binary operator not enclosed in a subformula. Plain TEX default: 700. Chapter 23. \botmark hexpandable commandi The last mark on the current page. Chapter 28. \boxh8-bit numberi hboxi Use a box register, emptying it. Chapter 5. \boxmaxdepth hdimen parameteri Maximum allowed depth of boxes. Default \maxdimen in plain TEX. Chapter 5. \brokenpenalty hinteger parameteri Additional penalty for breaking a page after a hyphenated line. Default 100 in plain TEX. Chapter 27. \catcodeh8-bit numberi hinternal integeri; the control sequence itself is a hcodenamei. Access category codes. Chapter 2. \charhnumberi hcharacteri Explicit denotation of a character to be typeset. Chapter 3. \chardefhcontrol sequenceihequalsihnumberi hshorthand definitioni Define a control sequence to be a synonym for a character code. Chapter 3. \cleaders hleadersi As \leaders, but with box leaders any excess space is split into equal glue items before and after the leaders. Chapter 9. \closeinh4-bit numberi hprimitive commandi Close an input stream. Chapter 30. \closeouth4-bit numberi hprimitive commandi Close an output stream. Chapter 30. \clubpenalty hinteger parameteri Additional penalty for breaking a page after the first line of a paragraph. Default 150 in plain TEX. Chapter 27. \copyh8-bit numberi hboxi Use a box register and retain the contents. Chapter 5. \counth8-bit numberi hinternal integeri; the control sequence itself is a hregister prefixi. Access count registers. Chapter 7. \countdefhcontrol sequenceihequalsih8-bit numberi hshorthand definitioni; the control sequence itself is a hregisterdefi. Define a control sequence to be a synonym for a \count register. Chapter 7. \cr hprimitive commandi Terminate an alignment line. Chapter 25. \crcr hprimitive commandi Terminate an alignment line if it has not already been terminated by \cr. Chapter 25. \csname hexpandable commandi Start forming the name of a control sequence. Has to be balanced with \endcsname. Chapter 11. \day hinteger parameteri The day of the current job. Chapter 33. 284

Victor Eijkhout – TEX by Topic

\deadcycles hspecial integeri Counter that keeps track of how many times the output routine has been called without a \shipout taking place. If this number reaches \maxdeadcycles TEX gives an error message. Plain TEX default: 25. Chapter 28. \def hdefi Start a macro definition. Chapter 11. \defaulthyphenchar hinteger parameteri Value of \hyphenchar when a font is loaded. Default value in plain TEX is ‘\-. Chapter 4,19. \defaultskewchar hinteger parameteri Value of \skewchar when a font is loaded. Default value in plain TEX is -1. Chapter 21. \delcodeh8-bit numberi hinternal integeri; the control sequence itself is a hcodenamei. Access the code specifying how a character should be used as delimiter after \left or \right. Chapter 21. \delimiterh27-bit numberi hmath characteri Explicit denotation of a delimiter. Chapter 21. \delimiterfactor hinteger parameteri 1000 times the part of a delimited formula that should be covered by a delimiter. Plain TEX default: 901. Chapter 21. \delimitershortfall hinteger parameteri Size of the part of a delimited formula that is allowed to go uncovered by a delimiter. Plain TEX default: 5pt. Chapter 21. \dimenh8-bit numberi hinternal dimeni; the control sequence itself is a hregister prefixi. Access dimen registers. Chapter 8. \dimendefhcontrol sequenceihequalsih8-bit numberi hshorthand definitioni; the control sequence itself is a hregisterdefi. Define a control sequence to be a synonym for a \dimen register. Chapter 8. \discretionary{pre-break}{post-break}{no-break} hhorizontal commandi Specify the way a character sequence is split up at a line break. Chapter 19. \displayindent hdimen parameteri Distance by which the box, in which the display is centred, is indented owing to hanging indentation. This value is set automatically for each display. Chapter 24. \displaylimits hprimitive commandi Restore default placement for limits. Chapter 23. \displaystyle hprimitive commandi Select the display style of math typesetting. Chapter 23. \displaywidowpenalty hinteger parameteri Additional penalty for breaking a page before the last line above a display formula. Default 50 in plain TEX. Chapter 27. \displaywidth hdimen parameteri Width of the box in which the display is centred. This value is set automatically for each display. Chapter 24. \dividehnumeric variableihoptional byihnumberi harithmetic assignmenti Arithmetic command to divide a hnumeric variablei (see \advance). Chapter 7. \doublehyphendemerits hinteger parameteri Penalty for consecutive lines ending with a hyphen. Default 10 000 in plain TEX. Chapter 19. \dph8-bit numberi hinternal dimeni; the control sequence itself is a hbox dimensioni. Depth of the box in a box register. Chapter 5. \dump hvertical commandi Dump a format file; possible only in IniTEX, not allowed inside a group. Chapter 33. \edef hdefi Start a macro definition; the replacement text is expanded at definition time. Chapter 11. \else hexpandable commandi Select hfalse texti of a conditional or default case of \ifcase. Chapter 13. \emergencystretch hdimen parameteri (TEX3 only) Assumed extra stretchability in lines of a paragraph in third pass of the line-breaking algorithm. Chapter 19. Victor Eijkhout – TEX by Topic

285

Chapter 37. Glossary of TEX Primitives

\end hvertical commandi End this run. Chapter 32. \endcsname hexpandable commandi Delimit the name of a control sequence that was begun with \csname. Chapter 11. \endgroup hprimitive commandi End a group that was opened with \begingroup. Chapter 10. \endinput hexpandable commandi Terminate inputting the current file after the current line. Chapter 30. \endlinechar hinteger parameteri The character code of the end-of-line character appended to input lines. IniTEX default: 13. Chapter 2. \eqnohmath mode materiali$$ heqnoi Place a right equation number in a display formula. Chapter 24. \errhelp htoken parameteri Tokens that will be displayed if the user asks for help after an \errmessage. Chapter 35. \errmessagehgeneral texti hprimitive commandi Report an error and give the user opportunity to act. Chapter 35. \errorcontextlines hinteger parameteri (TEX3 only) Number of additional context lines shown in error messages. Chapter 35. \errorstopmode hinteraction mode assignmenti Ask for user input on the occurrence of an error. Chapter 32. \escapechar hinteger parameteri Number of the character that is used when control sequences are being converted into character tokens. IniTEX default: 92. Chapter 3. \everycr htoken parameteri Token list inserted after every \cr or non-redundant \crcr. Chapter 25. \everydisplay htoken parameteri Token list inserted at the start of a display. Chapter 24. \everyhbox htoken parameteri Token list inserted at the start of a horizontal box. Chapter 5. \everyjob htoken parameteri Token list inserted at the start of each job. Chapter 32. \everymath htoken parameteri Token list inserted at the start of non-display math. Chapter 23. \everypar htoken parameteri Token list inserted in front of paragraph text. Chapter 16. \everyvbox htoken parameteri Token list inserted at the start of a vertical box. Chapter 5. \exhyphenpenalty hinteger parameteri Penalty for breaking a horizontal line at a discretionary in the special case where the prebreak text is empty. Default 50 in plain TEX. Chapter 19. \expandafter hexpandable commandi Take the next two tokens and place the expansion of the second after the first. Chapter 12. \fam hinteger parameteri The number of the current font family. Chapter 22. \fi hexpandable commandi Closing delimiter for all conditionals. Chapter 13. \finalhyphendemerits hinteger parameteri Penalty added when the penultimate line of a paragraph ends with a hyphen. Plain TEX default 5000. Chapter 19. \firstmark hexpandable commandi The first mark on the current page. Chapter 28. \floatingpenalty hinteger parameteri Penalty amount added to \insertpenalties when an insertion is split. Chapter 29. \fonthcontrol sequenceihequalsihfile nameihat clausei hsimple assignmenti Associate a control sequence with a tfm file. When used on its own, this control sequence is a hfonti, denoting the current font. Chapter 4. \fontdimenhnumberihfonti hinternal dimeni Access various parameters of fonts. Chapter 4. \fontnamehfonti hprimitive commandi The external name of a font. Chapter 4. 286

Victor Eijkhout – TEX by Topic

\futurelethcontrol sequenceihtoken1 ihtoken2 i hlet assignmenti Assign the meaning of htoken2 i to the hcontrol sequencei. Chapter 11. \gdef hdefi Synonym for \global\def. Chapter 11. \global hprefixi Make the next definition, arithmetic statement, or assignment global. Chapter 10,11. \globaldefs hinteger parameteri Override \global specifications: a positive value of this parameter makes all assignments global, a negative value makes them local. Chapter 10. \halignhbox specificationi{halignment materiali} hvertical commandi Horizontal alignment. Display alignment: $$\halignhbox specificationi{...}hoptional assignmentsi$$ Chapter 25. \hangafter hinteger parameteri If positive, this denotes the number of lines before indenting starts; if negative, its absolute value is the number of indented lines starting with the first line of the paragraph. The default value of 1 is restored after every paragraph. Chapter 18. \hangindent hdimen parameteri If positive, this indicates indentation from the left margin; if negative, this is the negative of the indentation from the right margin. The default value of 0pt is restored after every paragraph. Chapter 18. \hbadness hinteger parameteri Threshold below which TEX does not report an underfull or overfull horizontal box. Plain TEX default: 1000. Chapter 5. \hboxhbox specificationi{hhorizontal materiali} hboxi Construct a horizontal box. Chapter 5. \hfil hhorizontal commandi Horizontal skip equivalent to \hskip 0cm plus 1fil. Chapter 8. \hfill hhorizontal commandi Horizontal skip equivalent to \hskip 0cm plus 1fill. Chapter 8. \hfilneg hhorizontal commandi Horizontal skip equivalent to \hskip 0cm plus -1fil. Chapter 8. \hfuzz hdimen parameteri Excess size that TEX tolerates before it considers a horizontal box overfull. Plain TEX default: 0.1pt. Chapter 5. \hoffset hdimen parameteri Distance by which the page is shifted to the right of the reference point which is at one inch from the left margin. Chapter 26. \holdinginserts hinteger parameteri (only TEX3) If this is positive, insertions are not placed in their boxes when the \output tokens are inserted. Chapter 29. \hrule hvertical commandi Rule that spreads in horizontal direction. Chapter 9. \hsize hdimen parameteri Line width used for text typesetting inside a vertical box. Chapter 5,18. \hskiphgluei hhorizontal commandi Insert in horizontal mode a glue item. Chapter 8. \hss hhorizontal commandi Horizontal skip equivalent to \hskip 0cm plus 1fil minus 1fil. Chapter 8. \hth8-bit numberi hinternal dimeni; the control sequence itself is a hbox dimensioni. Height of the box in a box register. Chapter 5. \hyphenationhgeneral texti hhyphenation assignmenti Define hyphenation exceptions for the current value of \language. Chapter 19. \hyphencharhfonti hinternal integeri Number of the character behind which a \discretionary{}{}{} is inserted. Chapter 4,19. Victor Eijkhout – TEX by Topic

287

Chapter 37. Glossary of TEX Primitives

\hyphenpenalty hinteger parameteri Penalty associated with break at a discretionary in the general case. Default 50 in plain TEX. Chapter 19. \ifhtoken1 ihtoken2 i hexpandable commandi Test equality of character codes. Chapter 13. \ifcasehnumberihcase0 i\or...\orhcasen i\elsehother casesi\fi hexpandable commandi Enumerated case statement. Chapter 13. \ifcathtoken1 ihtoken2 i hexpandable commandi Test whether two characters have the same category code. Chapter 13. \ifdimhdimen1 ihrelationihdimen2 i hexpandable commandi Compare two dimensions. Chapter 13. \ifeofh4-bit numberi hexpandable commandi Test whether a file has been fully read, or does not exist. Chapter 30. \iffalse hexpandable commandi This test is always false. Chapter 13. \ifhboxh8-bit numberi hexpandable commandi Test whether a box register contains a horizontal box. Chapter 5. \ifhmode hexpandable commandi Test whether the current mode is (possibly restricted) horizontal mode. Chapter 13. \ifinner hexpandable commandi Test whether the current mode is an internal mode. Chapter 13. \ifmmode hexpandable commandi Test whether the current mode is (possibly display) math mode. Chapter 13. \ifnumhnumber1 ihrelationihnumber2 i hexpandable commandi Test relations between numbers. Chapter 13. \ifoddhnumberi hexpandable commandi Test whether a number is odd. Chapter 13. \iftrue hexpandable commandi This test is always true. Chapter 13. \ifvboxh8-bit numberi hexpandable commandi Test whether a box register contains a vertical box. Chapter 5. \ifvmode hexpandable commandi Test whether the current mode is (possibly internal) vertical mode. Chapter 13. \ifvoidh8-bit numberi hexpandable commandi Test whether a box register is empty. Chapter 13,5. \ifxhtoken1 ihtoken2 i hexpandable commandi Test equality of macro expansion, or equality of character code and category code. Chapter 13. \ignorespaces hprimitive commandi Expands following tokens until something other than a hspace tokeni is found. Chapter 2. \immediate hprimitive commandi Prefix to have output operations executed right away. Chapter 30. \indent hprimitive commandi Switch to horizontal mode and insert box with width \parindent. This command is automatically inserted before a hhorizontal commandi in vertical mode. Chapter 16. \inputhfile namei hexpandable commandi Read a specified file as TEX input. Chapter 30. \inputlineno hinternal integeri (TEX3 only) Number of the current input line. Chapter 30. \inserth8-bit numberi{hvertical mode materiali} hprimitive commandi Start an insertion item. Chapter 29. \insertpenalties hspecial integeri Total of penalties for split insertions. Inside the output routine the number of held-over insertions. Chapter 29. \interlinepenalty hinteger parameteri Penalty for breaking a page between lines of a paragraph. Default 0 in plain TEX. Chapter 27. 288

Victor Eijkhout – TEX by Topic

\jobname hexpandable commandi Name of the main TEX file being processed. Chapter 32. \kernhdimeni hkerni Add a kern item of the specified hdimeni to the list; this can be used both in horizontal and vertical mode. Chapter 8. \language hinteger parameteri (TEX3 only) Choose a set of hyphenation patterns and exceptions. Chapter 19. \lastbox hboxi Register containing the last element added to the current list, if this was a box. Chapter 5. \lastkern hinternal dimeni If the last item on the list was a kern, the size of this. Chapter 8. \lastpenalty hinternal integeri If the last item on the list was a penalty, the value of this. Chapter 27. \lastskip hinternal gluei or hinternal mugluei. If the last item on the list was a skip, the size of this. Chapter 8. \lccodeh8-bit numberi hinternal integeri; the control sequence itself is a hcodenamei. Access the character code that is the lowercase variant of a given code. Chapter 3. \leadershbox or ruleihvertical/horizontal/mathematical skipi hleadersi Fill a specified amount of space with a rule or copies of box. Chapter 9. \left hprimitive commandi Use the following character as an open delimiter. Chapter 21. \lefthyphenmin hinteger parameteri (TEX3 only) Minimum number of characters before a hyphenation. Chapter 19. \leftskip hglue parameteri Glue that is placed to the left of all lines. Chapter 18. \leqnohmath mode materiali$$ heqnoi Place a left equation number in a display formula. Chapter 24. \lethcontrol sequenceihequalsihtokeni hlet assignmenti Define a control sequence to a token, assign its meaning if the token is a command or macro. Chapter 11. \limits hprimitive commandi Place limits over and under a large operator. This is the default position in display style. Chapter 23. \linepenalty hinteger parameteri Penalty value associated with each line break. Default 10 in plain TEX. Chapter 19. \lineskip hglue parameteri Glue added if distance between bottom and top of neighbouring boxes is less than \lineskiplimit. Default 1pt in plain TEX. Chapter 15. \lineskiplimit hdimen parameteri Distance to be maintained between the bottom and top of neighbouring boxes on a vertical list. Default 0pt in plain TEX. Chapter 15. \long hprefixi Indicate that the arguments of the macro to be defined are allowed to contain \par tokens. Chapter 11. \looseness hinteger parameteri Number of lines by which this paragraph has to be made longer (or, if negative, shorter) than it would be ideally. Chapter 19. \lowerhdimenihboxi hprimitive commandi Adjust vertical positioning of a box in horizontal mode. Chapter 5. \lowercasehgeneral texti hprimitive commandi Convert the argument to its lowercase form. Chapter 3. \mag hinteger parameteri 1000 times the magnification of the document. Default 1000 in IniTEX. Chapter 33. \markhgeneral texti hprimitive commandi Specify a mark text. Chapter 28. \mathaccenth15-bit numberihmath fieldi hprimitive commandi Place an accent in math mode. Chapter 21,23. \mathbinhmath fieldi hmath atomi Let the following hmath fieldi function as a binary operation. Chapter 23. Victor Eijkhout – TEX by Topic

289

Chapter 37. Glossary of TEX Primitives

\mathcharh15-bit numberi hprimitive commandi Explicit denotation of a mathematical character. Chapter 21. \mathchardefhcontrol sequenceihequalsih15-bit numberi hshorthand definitioni Define a control sequence to be a synonym for a math character code. Chapter 21. \mathchoice{D}{T}{S}{SS} hprimitive commandi Give four variants of a formula for the four styles of math typesetting. Chapter 23. \mathclosehmath fieldi hmath atomi Let the following hmath fieldi function as a closing symbol. Chapter 23. \mathcodeh8-bit numberi hinternal integeri; the control sequence itself is a hcodenamei. Code of a character determining its treatment in math mode. Chapter 21. \mathinnerhmath fieldi hmath atomi Let the following hmath fieldi function as an inner formula. Chapter 23. \mathophmath fieldi hmath atomi Let the following hmath fieldi function as a large operator. Chapter 23. \mathopenhmath fieldi hmath atomi Let the following hmath fieldi function as an opening symbol. Chapter 23. \mathordhmath fieldi hmath atomi Let the following hmath fieldi function as an ordinary object. Chapter 23. \mathpuncthmath fieldi hmath atomi Let the following hmath fieldi function as a punctuation symbol. Chapter 23. \mathrelhmath fieldi hmath atomi Let the following hmath fieldi function as a relation. Chapter 23. \mathsurround hdimen parameteri Kern amount placed before and after in-line formulas. Chapter 23. \maxdeadcycles hinteger parameteri The maximum number of times that the output routine is allowed to be called without a \shipout occurring. IniTEX default: 25. Chapter 28. \maxdepth hdimen parameteri Maximum depth of the page box. Default 4pt in plain TEX. Chapter 26. \meaning hexpandable commandi Give the meaning of a control sequence as a string of characters. Chapter 34. \medmuskip hmuglue parameteri Medium amount of mu glue. Default value in plain TEX: 4mu plus 2mu minus 4mu Chapter 23. \messagehgeneral texti hprimitive commandi Write a message to the terminal. Chapter 30. \mkern hprimitive commandi Insert a kern measured in mu units. Chapter 23. \month hinteger parameteri The month of the current job. Chapter 33. \movelefthdimenihboxi hprimitive commandi Adjust horizontal positioning of a box in vertical mode. Chapter 5. \moverighthdimenihboxi hprimitive commandi Adjust horizontal positioning of a box in vertical mode. Chapter 5. \mskip hmathematical skipi Insert glue measured in mu units. Chapter 23. \multiplyhnumeric variableihoptional byihnumberi harithmetic assignmenti Arithmetic command to multiply a hnumeric variablei (see \advance). Chapter 7. \muskiph8-bit numberi hinternal mugluei; the control sequence itself is a hregister prefixi. Access skips measured in mu units. Chapter 23. \muskipdefhcontrol sequenceihequalsih8-bit numberi hshorthand definitioni; the control sequence itself is a hregisterdefi. Define a control sequence to be a synonym for a \muskip register. Chapter 23. 290

Victor Eijkhout – TEX by Topic

\newlinechar hinteger parameteri Number of the character that triggers a new line in \write and \message statements. Plain TEX default −1; LATEX default 10. Chapter 30. \noalignhfilleri{hvertical (horizontal) mode materiali} hprimitive commandi Specify vertical (horizontal) material to be placed in between rows (columns) of an \halign (\valign). Chapter 25. \noboundary hhorizontal commandi (TEX3 only) Omit implicit boundary character. Chapter 4. \noexpandhtokeni hexpandable commandi Do not expand the next token. Chapter 12. \noindent hprimitive commandi Switch to horizontal mode with an empty horizontal list. Chapter 16. \nolimits hprimitive commandi Place limits of a large operator as subscript and superscript expressions. This is the default position in text style. Chapter 23. \nonscript hprimitive commandi Cancel the next glue item if it occurs in scriptstyle or scriptscriptstyle. Chapter 23. \nonstopmode hinteraction mode assignmenti TEX fixes errors as best it can, and performs an emergency stop when user interaction is needed. Chapter 32. \nulldelimiterspace hdimen parameteri Width taken for empty delimiters. Default 1.2pt in plain TEX. Chapter 21. \nullfont hfontdef tokeni Name of an empty font that TEX uses in emergencies. Chapter 4. \numberhnumberi hexpandable commandi Convert a hnumberi to decimal representation. Chapter 7. \omit hprimitive commandi Omit the template for one alignment entry. Chapter 25. \openinh4-bit numberihequalsihfilenamei hprimitive commandi Open a stream for input. Chapter 30. \openouth4-bit numberihequalsihfilenamei hprimitive commandi Open a stream for output. Chapter 30. \or hprimitive commandi Separator for entries of an \ifcase. Chapter 13. \outer hprefixi Indicate that the macro being defined should occur on the outer level only. Chapter 11. \output htoken parameteri Token list with instructions for shipping out pages. Chapter 28. \outputpenalty hinteger parameteri Value of the penalty at the current page break, or 10 000 if the break was not at a penalty. Chapter 27,28. \over hgeneralized fraction commandi Fraction. Chapter 23. \overfullrule hdimen parameteri Width of the rule that is printed to indicate overfull horizontal boxes. Plain TEX default: 5pt. Chapter 5. \overlinehmath fieldi hmath atomi Overline the following hmath fieldi. Chapter 23. \overwithdelimshdelim1 ihdelim2 i hgeneralized fraction commandi Fraction with delimiters. Chapter 23. \pagedepth hspecial dimeni Depth of the current page. Chapter 27. \pagefilllstretch hspecial dimeni Accumulated third-order stretch of the current page. Chapter 27. \pagefillstretch hspecial dimeni Accumulated second-order stretch of the current page. Chapter 27. \pagefilstretch hspecial dimeni Accumulated first-order stretch of the current page. Chapter 27. \pagegoal hspecial dimeni Goal height of the page box. This starts at \vsize, and is diminished by heights of insertion items. Chapter 27. \pageshrink hspecial dimeni Accumulated shrink of the current page. Chapter 27. Victor Eijkhout – TEX by Topic

291

Chapter 37. Glossary of TEX Primitives

\pagestretch hspecial dimeni Accumulated zeroth-order stretch of the current page. Chapter 27. \pagetotal hspecial dimeni Accumulated natural height of the current page. Chapter 27. \par hprimitive commandi Close off a paragraph and go into vertical mode. Chapter 17. \parfillskip hglue parameteri Glue that is placed between the last element of the paragraph and the line end. Plain TEX default: 0pt plus 1fil. Chapter 17. \parindent hdimen parameteri Size of the indentation box added in front of a paragraph. Chapter 16,18. \parshape hinternal integeri Command for general paragraph shapes: \parshapehequalsin i1 `1 . . . in `n specifies a number of lines n, and n pairs of an indentation and line length. Chapter 18. \parskip hglue parameteri Amount of glue added to vertical list when a paragraph starts; default value 0pt plus 1pt in plain TEX. Chapter 16. \patternshgeneral texti hhyphenation assignmenti Define a list of hyphenation patterns for the current value of \language; allowed only in IniTEX. Chapter 19. \pausing hinteger parameteri Specify that TEX should pause after each line that is read from a file. Chapter 32. \penalty hprimitive commandi Specify desirability of not breaking at this point. Chapter 19,27. \postdisplaypenalty hinteger parameteri Penalty placed in the vertical list below a display. Chapter 24. \predisplaypenalty hinteger parameteri Penalty placed in the vertical list above a display. Plain TEX default: 10 000. Chapter 24. \predisplaysize hdimen parameteri Effective width of the line preceding the display. Chapter 24. \pretolerance hinteger parameteri Tolerance value for a paragraph that uses no hyphenation. Default 100 in plain TEX. Chapter 19. \prevdepth hspecial dimeni Depth of the last box added to a vertical list as it is perceived by TEX. Chapter 15. \prevgraf hspecial integeri The number of lines in the paragraph last added to the vertical list. Chapter 19. \radicalh24-bit numberi hprimitive commandi Command for setting things such as root signs. Chapter 21. \raisehdimenihboxi hprimitive commandi Adjust vertical positioning of a box in horizontal mode. Chapter 5. \readhnumberitohcontrol sequencei hsimple assignmenti Read a line from a stream into a control sequence. Chapter 30. \relax hprimitive commandi Do nothing. Chapter 12. \relpenalty hinteger parameteri Penalty for breaking after a binary relation, not enclosed in a subformula. Plain TEX default: 500. Chapter 23. \right hprimitive commandi Use the following character as a closing delimiter. Chapter 21. \righthyphenmin hinteger parameteri (TEX3 only) Minimum number of characters after a hyphenation. Chapter 19. \rightskip hglue parameteri Glue that is placed to the right of all lines. Chapter 18. \romannumeralhnumberi hexpandable commandi Convert a positive hnumberi to lowercase roman representation. Chapter 7. 292

Victor Eijkhout – TEX by Topic

\scriptfonth4-bit numberi hfamily memberi; the control sequence itself is a hfont rangei. Access the scriptfont of a family. Chapter 22. \scriptscriptfonth4-bit numberi hfamily memberi; the control sequence itself is a hfont rangei. Access the scriptscriptfont of a family. Chapter 22. \scriptscriptstyle hprimitive commandi Select the scriptscript style of math typesetting. Chapter 23. \scriptspace hdimen parameteri Extra space after subscripts and superscripts. Default .5pt in plain TEX. Chapter 23. \scriptstyle hprimitive commandi Select the script style of math typesetting. Chapter 23. \scrollmode hinteraction mode assignmenti TEX patches errors itself, but will ask the user for missing files. Chapter 32. \setboxh8-bit numberihequalsihboxi hsimple assignmenti Assign a box to a box register. Chapter 5. \setlanguagehnumberi hprimitive commandi (TEX3 only) Insert a whatsit resetting the current language to the hnumberi specified. Chapter 19. \sfcodeh8-bit numberi hinternal integeri; the control sequence itself is a hcodenamei. Access the value of the \spacefactor associated with a character. Chapter 20. \shipouthboxi hprimitive commandi Ship a box to the dvi file. Chapter 28. \showhtokeni hprimitive commandi Display the meaning of a token on the screen. Chapter 34. \showboxh8-bit numberi hprimitive commandi Write the contents of a box to the log file. Chapter 34. \showboxbreadth hinteger parameteri Number of successive elements that are shown when \tracingoutput is positive, each time a level is visited. Plain TEX default: 5. Chapter 34. \showboxdepth hinteger parameteri The number of levels that are shown when \tracingoutput is positive. Plain TEX default: 3. Chapter 34. \showlists hprimitive commandi Write to the log file the contents of the partial lists currently built in all modes. Chapter 6. \showthehinternal quantityi hprimitive commandi Display on the terminal the result of prefixing a token with \the. Chapter 34. \skewcharhfonti hinternal integeri Font position of an after-placed accent. Chapter 21. \skiph8-bit numberi hinternal gluei; the control sequence itself is a hregister prefixi. Access skip registers Chapter 8. \skipdefhcontrol sequenceihequalsih8-bit numberi hshorthand definitioni; the control sequence itself is a hregisterdefi. Define a control sequence to be a synonym for a \skip register. Chapter 8. \spacefactor hspecial integeri 1000 times the ratio by which the stretch component of the interword glue should be multiplied. Chapter 20. \spaceskip hglue parameteri Interword glue if non-zero. Chapter 20. \span hprimitive commandi Join two adjacent alignment entries, or (in preamble) expand the next token. Chapter 25. \specialhgeneral texti hprimitive commandi Write a token list to the dvi file. Chapter 33. \splitbotmark hexpandable commandi The last mark on a split-off page. Chapter 28. \splitfirstmark hexpandable commandi The first mark on a split-off page. Chapter 28. \splitmaxdepth hdimen parameteri Maximum depth of a box split off by a \vsplit operation. Default 4pt in plain TEX. Chapter 5,26. Victor Eijkhout – TEX by Topic

293

Chapter 37. Glossary of TEX Primitives

\splittopskip hglue parameteri Minimum distance between the top of what remains after a \vsplit operation, and the first item in that box. Default 10pt in plain TEX. Chapter 27. \stringhtokeni hexpandable commandi Convert a token to a string of one or more characters. Chapter 3. \tabskip hglue parameteri Amount of glue in between columns (rows) of an \halign (\valign). Chapter 25. \textfonth4-bit numberi hfamily memberi; the control sequence itself is a hfont rangei. Access the textfont of a family. Chapter 22. \textstyle hprimitive commandi Select the text style of math typesetting. Chapter 23. \thehinternal quantityi hprimitive commandi Expand the value of various quantities in TEX into a string of (character) tokens. Chapter 12. \thickmuskip hmuglue parameteri Large amount of mu glue. Default value in plain TEX: 5mu plus 5mu. Chapter 23. \thinmuskip hmuglue parameteri Small amount of mu glue. Default value in plain TEX: 3mu. Chapter 23. \time hinteger parameteri Number of minutes after midnight that the current job started. Chapter 33. \toksh8-bit numberi hregister prefixi Access a token list register. Chapter 14. \toksdefhcontrol sequenceihequalsih8-bit numberi hshorthand definitioni; the control sequence itself is a hregisterdefi. Assign a control sequence to a \toks register. Chapter 14. \tolerance hinteger parameteri Tolerance value for lines in a paragraph that does use hyphenation. Default 200 in plain TEX, 10 000 in IniTEX. Chapter 19. \topmark hexpandable commandi The last mark of the previous page. Chapter 28. \topskip hglue parameteri Minimum distance between the top of the page box and the baseline of the first box on the page. Default 10pt in plain TEX. Chapter 26. \tracingcommands hinteger parameteri When this is 1, TEX displays primitive commands executed; when this is 2 or more the outcome of conditionals is also recorded. Chapter 34. \tracinglostchars hinteger parameteri If this parameter is positive, TEX gives diagnostic messages whenever a character is accessed that is not present in a font. Plain TEX default: 1. Chapter 34. \tracingmacros hinteger parameteri If this is 1, the log file shows expansion of macros that are performed and the actual values of the arguments; if this is 2 or more htoken parameteris such as \output and \everypar are also traced. Chapter 34. \tracingonline hinteger parameteri If this parameter is positive, TEX will write trace information also to the terminal. Chapter 34. \tracingoutput hinteger parameteri If this parameter is positive, the log file shows a dump of boxes that are shipped to the dvi file. Chapter 34. \tracingpages hinteger parameteri If this parameter is positive, TEX generates a trace of the page-breaking algorithm. Chapter 34. \tracingparagraphs hinteger parameteri If this parameter is positive, TEX generates a trace of the line-breaking algorithm. Chapter 34. \tracingrestores hinteger parameteri If this parameter is positive, TEX will report all values that are restored when a group level ends. Chapter 34. 294

Victor Eijkhout – TEX by Topic

\tracingstats hinteger parameteri If this parameter is positive, TEX reports at the end of the job the usage of various internal arrays. Chapter 34. \uccodeh8-bit numberi hinternal integeri; the control sequence itself is a hcodenamei. Access the character code that is the uppercase variant of a given code. Chapter 3. \uchyph hinteger parameteri Positive if hyphenating words starting with a capital letter is allowed. Plain TEX default 1. Chapter 19. \underlinehmath fieldi hmath atomi Underline the following hmath fieldi. Chapter 23. \unhboxh8-bit numberi hhorizontal commandi Unpack a box register containing a horizontal box, appending the contents to the list, and emptying the register. Chapter 5. \unhcopyh8-bit numberi hhorizontal commandi The same as \unhbox, but do not empty the register. Chapter 5. \unkern hprimitive commandi Remove the last item of the list if this was a kern. Chapter 8. \unpenalty hprimitive commandi Remove the last item of the list if this was a penalty. Chapter 27. \unskip hprimitive commandi Remove the last item of the list if this was a skip. Chapter 8. \unvboxh8-bit numberi hvertical commandi Unpack a box register containing a vertical box, appending the contents to the list, and emptying the register. Chapter 5. \unvcopyh8-bit numberi hvertical commandi The same as \unvbox, but do not empty the register. Chapter 5. \uppercasehgeneral texti hprimitive commandi Convert the argument to its uppercase form. Chapter 3. \vadjusthfilleri{hvertical mode materiali} hprimitive commandi Specify in horizontal mode material for the enclosing vertical list. Chapter 6. \valignhbox specificationi{halignment materiali} hhorizontal commandi Vertical alignment. Chapter 25. \vbadness hinteger parameteri Threshold below which overfull and underfull vertical boxes are not shown. Plain TEX default: 1000. Chapter 5. \vboxhbox specificationi{hvertical materiali} hprimitive commandi Construct a vertical box with reference point on the last item. Chapter 5. \vcenterhbox specificationi{hvertical materiali} hprimitive commandi Construct a vertical box vertically centred on the math axis. Chapter 23. \vfil hvertical commandi Vertical skip equivalent to \vskip 0cm plus 1fil. Chapter 8. \vfill hvertical commandi Vertical skip equivalent to \vskip 0cm plus 1fill. Chapter 8. \vfilneg hvertical commandi Vertical skip equivalent to \vskip 0cm minus 1fil. Chapter 8. \vfuzz hdimen parameteri Excess size that TEX tolerates before it considers a vertical box overfull. Plain TEX default: 0.1pt. Chapter 5. \voffset hdimen parameteri Distance by which the page is shifted down from the reference point, which is one inch from the top of the page. Chapter 26. \vrule hhorizontal commandi Rule that spreads in vertical direction. Chapter 9. \vsize hdimen parameteri Height of the page box. Chapter 5,26. \vskiphgluei hvertical commandi Insert in vertical mode a glue item. Chapter 8. \vsplith8-bit numberitohdimeni hprimitive commandi Split off the top part of a vertical box. Chapter 5,27. \vss hvertical commandi Vertical skip equivalent to \vskip 0cm plus 1fil minus 1fil. Chapter 8. \vtophbox specificationi{hvertical materiali} hprimitive commandi Construct a vertical box with reference point on the first item. Chapter 5. Victor Eijkhout – TEX by Topic

295

Chapter 37. Glossary of TEX Primitives

\wdh8-bit numberi hinternal dimeni; the control sequence itself is a hbox dimensioni. Width of the box in a box register. Chapter 5. \widowpenalty hinteger parameteri Additional penalty for breaking a page before the last line of a paragraph. Default 150 in plain TEX. Chapter 27. \writehnumberihgeneral texti hprimitive commandi Generate a whatsit item containing a token list to be written to the terminal or to a file. Chapter 30. \xdef hdefi Synonym for \global\edef. Chapter 11. \xleaders hleadersi As \leaders, but with box leaders any excess space is spread equally between the boxes. Chapter 9. \xspaceskip hglue parameteri Interword glue if non-zero and \spacefactor ≥ 2000. Chapter 20. \year hinteger parameteri The year of the current job. Chapter 33.

296

Victor Eijkhout – TEX by Topic

Chapter 38 Tables

297

Chapter 38. Tables

38.1

Character tables dec

CHAR

ASCII CONTROL CODES b7 b6 b5

0

0

0

0 0

0

1

BITS 16

NUL

0 1

SOH

1 2

STX

2

ETX

3

22

EOT

4 5

DC3

23

ENQ

0 1 1 0 6

DC4

4 14

24

7

0 1 1 1 7

8

BEL BS

9

A

HT LF

B 12

1 1 0 0 C

D

FF CR

E

CAN

30

EM

11 19

15

31

SUB

32

13 1B

ESC

14 1C

15 1D

17 1F

25

26

27

28

29

2A

33

2B

%

34

2C

&

35

2D

’

36

2E

(

37

2F

43 33

44 34

45 35

46 36

47 37

50 38

)

51 39

3

63

52 3A

4

64

53 3B

5

65

54 3C

6

66

55 3D

7

67

56 3E

8

70

57 3F

D

45

46

47

48

9

71

49

E

72

4A

F

73

4B

G

74

4C

H

75

4D

I

76

4E

J

77

4F

123

104 54

105 55

106 56

107 57

110 58

111 59

112 5A

K

113 5B

T

124

U

125

V

126

W

127

X

130

64

65

66

67

68

Y

131

69

117 5F

c

143

132

6A

d

144

133

6B

e

145

6C

f

146

135

6D

g

147

136

6E

h

150

137

6F

t

74

164

u

75

165

v

76

166

w

77

167

x

78

170

121

i

151

y

79

171

122

j

152

7A

z

172

123

k

{ 153

7B

173

124

l

| 154

7C

174

125

m

} 155

7D

175

126

n

156

111

_

163

120

110

ˆ

s

73

119

109

]

162

118

108 134

r

72

117

107

[

161

116

106

Z

q

71

115

105

95

O

63

\

116 5E

142

104

114 5C

115 5D

b

103

94

N

62

160

114

102

93

M

141

101

92

L

a

100

91

79

?

S

90

78

>

103 53

61

p

70 113

99

89

77

=

122

88

76




103 53

61

70 113

99

89

77

=

122

88

76




R

87

77

=

52

140

98

86

76