Automorphic Representation of Unitary Groups in Three Variables. (AM-123), Volume 123 9781400882441

Citation preview

Annals of Mathematics Studies Number 123

Automorphic Representations of Unitary Groups in Three Variables by

Jonathan D. Rogawski

PRINCETO N UNIVERSITY PRESS

PRINCETON, NEW JERSEY 1990

Copyright ©

1990 by Princeton University Press

A L L R IG H T S RE SER VE D

The Annals of Mathematics Studies are edited by Luis A . Caffarelli, John N . Mather, John Milnor, and Elias M . Stein

Princeton University Press books are printed on acid-free paper, and meet the guidelines for permar nence and durability of the Committee on Produc­ tion Guidelines for Book Longevity of the Council on Library Resources

Printed in the United States of America by Princeton University Press, 41 W illiam Street Princeton, New Jersey

L ib ra ry o f C on gress C atalo gin g-in -P u b licatio n D a t a Rogawski, Jonathan David. Autom orphic representations of unitary groups in three variables / by Jonathan D. Rogawski. p.

cm. - (Annals of mathematics studies ; no. 123)

Includes bibliographical references (p.

) and index.

IS B N 0-691-08586-2 (cloth: acid-free paper) IS B N 0-691-08587-0 (paper: acid-free paper) 1. Unitary groups.

2. Trace formulas. 3. Representations of

groups. 4. Automorphic forms. QA171.R617 512’.2-dc20

I. Title. II. Series.

1990 90-38535

To Julie, with love

Table of Contents Page Introduction 1. Preliminary definitions and notation

ix 3

2. The trace formula

11

3. Stable conjugacy

19

4. Orbital integrals and endoscopic groups

39

5. Stabilization

68

6. Weighted orbital integrals

79

7. Elliptic singular terms

89

8. Germ expansions and limit formulas

112

9. Singularities

134

10. The stable trace formula

153

11. The unitary group in two variables

161

12. Representation theory

171

13. Automorphic representations

198

14. Comparison of inner forms

232

15. Additional results

246

References

251

Subject Index

258

Notation Index

259

Introduction

IX

It is a basic problem in the theory of automorphic representations to compare the trace formulas of reductive groups G 1 and G over a global field F in various situations. For example, G' may be an inner form of G, or the set of fixed points of an automorphism of G. The base change problem for a cyclic extension E / F , in which G = Rese / f ( G ,)-) is an example of the latter situation. Such a comparison was carried out in the representationtheoretic context for the first time in the case of GL(2) ([JL]), resulting in the Jacquet-Langlands correspondence between automorphic represen­ tations of GL(2) and unit groups of quaternion algebras. The base change problem for G L(n) and cyclic extensions was treated in the work of Saito, Shintani, and Langlands for n = 2 and was generalized to n > 2 by ArthurClozel ([£ i], [AC]). To deal with more general reductive groups, it is neces­ sary to “stabilize” the trace formula, in the sense of Langlands ([L 2], [Le])A comparison of trace formulas involves a correspondence between the conjugacy classes of the two groups, and, in general, such a correspondence must be phrased in terms of stable conjugacy. The stable trace formula is also an essential ingredient in the comparison of the trace formula with the Lefschetz trace formula in the theory of Shimura varieties ((L 7], [Mo]). The first part of this book (§1-§10) is devoted to developing a stable trace formula for the group G = U ( 3), the quasi-split unitary group in three variables defined with respect to a quadratic extension E/F. At the same time, the comparison of trace formulas for G and Rese /f ( G ) is made. The stabilization procedure is described in §5 and is carried out in §10 (Theorem 10.3.1). The combination of the stable trace formula and the base change comparison yields a great deal of information about automorphic representations of J7(3). These consequences are worked out in the second part of the book (§13-15). Stabilization involves several concepts and ingredients due to Langlands which are known under the general heading of “endoscopy” . These include stable conjugacy, endoscopic groups, and transfer of orbital integrals. The definitions are reviewed in §3 and §4. At the level of representations, en­ doscopy shows up in the theory of L-packets, i.e., the partition of the set of irreducible admissible representations of a connected reductive group over a local F into finite sets II called L-packets which satisfy certain natural properties. Such a partition has been shown to exist in the archimedean case ([Lg]), but not yet, in general, in the p-adic case. For GL(n), locally and globally, all L-packets are singletons. Once local L-packets are defined

X

Introduction

at all places, global X-packets are defined as restricted tensor products II = nv, where II v is a local X-packet which, for almost all v, contains an unramified representation. A restricted tensor product ® 7rv belongs to II if 7rv G lit, for all v and ttv is unramified for almost all v. The first example of a stable trace formula was obtained for the group SL(2) by Labesse and Langlands. In this case, two representations axe de­ fined to lie in the same X-packet (locally and globally) if they are conjugate under the adjoint action of P G L 2(X'). Although this definition also works for SL(n), using conjugation by P G L n(.F), in general X-packets cannot be defined in terms of the action of the adjoint group or any other larger group. Let II be a discrete X-packet on SL(2), i.e., II contains elements which oc­ cur discretely in X2(S'X2( i r')\5 X 2(A )). Any two members of II have the same local components almost everywhere. However, they may occur in the discrete spectrum with different multiplicities. Let us call n stable if all members of n have the same multiplicity. The stable trace formula for SL( 2 ) leads to a classification of the set of n which axe not stable in terms of automorphic characters 6 of groups T ( F ) \ T ( A ), where T is the torus defined by the norm one group E 1 of a quadratic extension E/F. By the Langlands classification for tori, 0 is associated to a homomorphism 0* : W f —> LT where W f is the Weil group of F. There is a natural homo­ morphism (p : l T —» l SL(2) and the composition l SL(2) defines an X-packet n(0) which is known to be cuspidal ([L 2]). By the results of [LL], n is stable unless it is of the form n (0) and, furthermore, there is a formula for the multiplicities of the representations belonging to the packets n(0). The tori T are endoscopic groups for SL(2), and the re­ sults just quoted illustrate the pattern that is expected to hold in general, that the X-packets which are not stable will occur as functorial transfers of L -packets on endoscopic groups. The group G = U(3) has only one proper elliptic endoscopic group, namely, the group H = U ( 2 ) x ?7(1). By the general theory (§4), there is an embedding of X-groups £h : LH LG. We also have the embedding if>G' LG LG, where G = Res#/f(G ), corresponding to the base change lifting from U (3) to GL(3)/£. In §13, as a consequence of the global theory, we define local L-packets for G and we prove the existence of the transfers of automorphic L-packets associated to £// and rpc by the principle of functoriality. This leads to a classification of automorphic L-packets of G (§3.13). Let n s(G) be the set of discrete X-packets on G whose base change transfer

Introduction

XI

to GL(3)/£ is discrete. The elements of IIS(G ) axe stable in the above sense (Theorem 13.3.3). Furthermore, the base change transfer defines a bijection between I i s( G ) and the set of discrete representations of GL(3)/£ which are invariant under the automorphism g —> where the bar denotes con­ jugation with respect to E / F , and whose central character, viewed as a character of /#, has trivial restriction to Ip- Let IIe(G ) be the set of cus­ pidal L-packets which are transfers of cuspidal L-packets on H . As in the case of SL(2), there is a multiplicity formula for the elements of L-packets belonging to IIe(G ) (Theorem 13.3.7). The union IIa(G ) U IIe(G ) is dis­ joint and accounts for most of the discrete spectrum of G. To describe the remaining discrete representations, we define an enlarged L-packet n(/o), which we call an “A-packet” , for each one-dimensional automorphic rep­ resentation p of H. Locally, if v is a place of F, the L-packet £h(Pv) consists of a single representation 7rn(pv) (§ 12 ). The A-packet II(pv) con­ tains 7Tn(pv) and it contains an additional representation tt3 (pv) precisely when v remains prime in E. Every discrete representation of G which does not belong to an L-packet in 11, (G ) or n e(G ) belongs to a unique A-packet n(p). The existence of these A-packets is predicted by the conjectures of J. Arthur ([A 4], [Rs]), as is the structure of the multiplicity formula for them (Theorem 13.3.7). The representations 7rn(pv) are non-tempered and the philosophy of [A4] suggests that the only cuspidal representations of G with a non-tempered component are those belonging to the A-packets n(p). If E v/Fv is C/R, then G v is the real group L 2,i(R ) and the representa­ tions of the form 7rn(pv) include all those such that H 1 ( S, F , F ® 7r) 7^ 0 for some finite dimensional representation F of G v. The representation 7rs(pv) is either square-integrable or is a component of a certain reducible principal series representation. In particular, we obtain examples where the latter representations occur as local components of cuspidal representations. By Theorem 13.3.1, the multiplicity of an automorphic representation of G which occurs in the discrete spectrum is always equal to one. In addition, two L-packets which are equal locally almost everywhere coincide. In §14, the comparison between 17(3) and its inner forms is carried out. This leads to a generalization (Theorem 15.3.1) of the vanishing theorem of Rapoport-Zink ([RZ]), according to which 7f 1 ( r , F ) = 0 for all congruence subgroups r of 17(2,1) which arise from global inner forms of Z7(3) associ­ ated to division algebras over an imaginary quadratic extension of Q with an involution of the second kind.

xii

Introduction

The oscillator representation and L-function methods have been used to investigate automorphic representations of 1/(3). The existence of cuspidal representations of G with local component of the form 7rn(pv) was first discovered by Howe and Piatetski-Shapiro ([HP]). Results on the transfer from H as well as base change to E for 17(3) were obtained by Gelbart and Piatet ski-Shapiro ([G P]). In most cases, the trace formula approach leads to a more precise and complete formulation of the results. This approach has also been considered in papers of Flicker. The results of the present work have been applied to determine the zetafunctions of the Shimura varieties associated to unitary groups in three variables for the case E quadratic imaginary ([Mo]). As a consequence, ^-adic representations are associated to cohomological automorphic repre­ sentations on 17(3). This theory has been applied in [BR 2] to obtain some cases of the Tate conjectures for these Shimura varieties. Furthermore, the endoscopic transfer from H leads to a construction of ^-adic representations for Hilbert modular forms ([B R 3]). The present work relies on the general theory of endoscopy, as begun by Langlands and developed and expanded in the papers of Kottwitz and Shelstad. Results due to Langlands-Shelstad on the transfer of orbital in­ tegrals for 17(3) ([LS 2]) play an important role. In addition, we rely on the lectures presented in two seminars held at the Institute for Advanced Study during the year 1983-1984. The general twisted form of the Arthur trace formula developed in [M] is used for the base change comparison of 17(3) with GL(3). The approach to the stable trace formula taken here is based on the lectures presented in [A]. My thanks are due to R. Langlands for help and encouragement on numerous occasions. I wish to thank The Institute for Advanced Study, the National Science Foundation and the Sloan Foundation for support at various times during the preparation of this book.

Automorphic Representations of Unitary Groups in Three Variables

C H A PTE R 1 Preliminary definitions and notation

1.1. The symbol F will be used to denote a local field or global field of characteristic zero. The ring of integers of F will be denoted by Of- If F is a number field, let A f , Cf, and C f be the adeles, ideles, and idele classes of P , respectively, and if F is local, let C f = F*. Let W f be the absolute Weil group of F and if L j F is a finite extension, let W l /f be the Weil group of L j F . If L j F is Galois, let T ( L j F ) be the Galois group. The symbol T will denote T ( F / F ) , where F is an algebraic closure of F. The norm and trace maps will be denoted by Np/p and Trp/F, respectively and we write L 1 and L° for their kernels. We write Ip for the norm one elements in Ip If F is global, v will denote a place of F. Let F v and 0 V be the completions of F and O f at u, and let L v = L ® f F v. 1.2. Let G be an algebraic group over F. We will write G for the group G ( F ) of P-points of G- If F is global, bold G will denote the group of AF-points of G, and if v is a place of P , we set G v = G ( F V). Let be center of G. Let X * ( G ) = Hom(G, G m) be the lattice of characters of G and let X * ( G ) f be the sublattice of P-rational characters. Let X * (G ) be the lattice dual to X * ( G ) . If F is global, let G 1 = {g £ G : |x(#)| — 1 f° r all x € X * ( G ) f }1.3. The symbol G will henceforth denote a connected reductive group over F. Fix a minimal F-parabolic subgroup Po of G and let Mo be a Levi factor of P 0- A subgroup M containing Mo is called a Levy subgroup if it is a Levi factor of a parabolic subgroup. Let £ (G ) be the set of Levi subgroups of G. A parabolic subgroup of G will be called standard if it contains Po- Let CP be the set of standard parabolic subgroups of G. If P € JP, the unique Levi factor of P containing M q will be denoted by M p and N p will denote the unipotent radical of P. Set TVo = Np0. When F is global, we fix a maximal compact subgroup K = HVK V of G as follows. We assume that K v is special for all v and that for almost

4

Chapter 1

all finite v, K v = G (0 „), where G ( 0 V) = «'(G?V) fl GL„(Ot,) for some fixed embedding i of G in G Ln/p. The Iwasawa decomposition G = P K holds. 1.4. Let £ be a (possibly trivial) automorphism of G of finite order. Elements x,y (E G are called e-conjugate if g~ 1 xe(g) = y for some g £ G. For 7 G G, the subgroup {) denote the space of smooth functions / on G such that supp(/ ) is compact modulo Z, f ( z g ) = u>~1 ( z ) f ( g ) for z G Z, and if F is archimedean, such that / is K -finite (where A" is a fixed maximal compact subgroup of G ). If F is p-adic and AT is a hyperspecial maximal compact subgroup of G, let IH(G) = TC(G,u;) be the Hecke algebra of bi-Pf-invariant functions in C(G,u>). This Hecke algebra is non-zero only if uj is trivial on Z fl AT. If A is global, we denote 5f(G „) by *KV.

1 .6 . Representations. All representations of reductive groups over local field are assumed to be admissible. By abuse of notation, we will not distinguish between an irreducible representation and its isomorphism class. If 7r is any representation, let J H ( tt) denote the set of irreducible constituents of n. The set of irreducible admissible representations of a reductive group G will be denoted by E{ G). According to the local Langlands correspondance (which is known, in general, if F is archimedean, and in special cases if F

Preliminary definitions and notation

5

is p-adic), E ( G ) is partitioned into finite subsets called P-packets. The set of P-packets will be denoted by 11(G). Let P € 7 and let a be a representation of a Levi factor M p of P , regarded as a representation of P on which N p acts trivially. We denote by the representation of G unitarily induced from (G). ([Bo]). The dual root data ^ (G )A is the quadruple ^ (G )A = (X * (T ), A * ,X * (T ), A *). The Galois group T acts on i/>(G). If r G T, then r defines an isomorphism between the root data of ( S , T ) and ( t ( B ) , t ( T ) ) , and hence induces an automorphism of $ (G). We obtain a homomorphism T —> Out(G). The dual group G of G is the complex, connected reductive group whose root data if?(G) is isomorphic to ^ (G )A ([Bo], [Kts]). Let (J5,T, { X a } ) be a splitting for G, i.e., (B , T ) is a Borel pair and { X Q} is a set of basis elements for the root spaces in Lie(G ) associated to the set of simple roots of T with respect to B. The splitting defines a section s of the map Aut(G ) Out(G). For 6 G Out(G), s(f F v = D w x D w> and a induces an anti­ automorphism of D w with D w>. In this case, projection onto the first or second factors induces an isomorphism of G v onto D^ or respectively (these groups are isomorphic via g —> a (^ )_1). If v remains prime, then a induces an isomorphism of D v with its opposite algebra. This shows that the class of D v in the Brauer group has order two. In particular, D v is the split algebra if diuiE(D) is odd. In this case, G v is isomorphic to U$ for some Hermitian matrix G G L n( E v). By a theorem of Landherr ([L]), if v is finite, then the equivalence class of a Hermitian matrix $ is determined by the class of det() in F * / N e / f ( E * ) ' Observe that U_^ is isomorphic to U_\$ f° r all A G and hence isomorphism class of U_,$ depends only on F * / ( F * ) uN e / f ( E v ) - In particular, if n is odd, there is a

Preliminary definitions and notation

9

unique isomorphism class of unitary groups with respect to E v/ F v. In the global case, the equivalence class of 3> is determined by the class of det() in F * / N e / f ( F * ) and the signatures of 3> at archimedean places v such that E v/ F v is isomorphic to C /R . 1.10. U n itary g ro u p s is three v ariab les. We fix the following notation. If G = U(n) or GL(n), B will denote the Borel subgroup of upper-triangular matrices, N will denote its unipotent radical, and M will denote the diagonal subgroup of B. A diagonal matrix with diagonal entries a i , ... , a n will be denoted by d (a i, ... , a n). Let otj be the simple root of M defined by Oij(a\ , . . . , an)) = a j /flj+i for j = 1,.. . , n — 1. For x ,z £ E such that xx — z + 2 , set

If G = U(3), then N = { u (x ,z) : x,z £ F , xx = z + z} and M = { d ( a ,/ ^ a -1 ) : a £ E * , /3 £ E 1}. We will often let ) be the space of measurable functions

‘ 6 € P \ G

Then J q is integrable over ZG\G and its integral is equal to J q (/ ) ([M ]). 2.3 T h e fine x _exP ansi ° n‘ The space L ( G ) decomposes into an orthogonal direct sum of subspaces L X(G ), indexed by the set { x } of cusp­ idal data. Let ( M i , p) be a representative for x and let Pi be the standard parabolic subgroup containing M i as a Levi subgroup. Let p1 be the re­ striction of p to and let if? be a smooth function on G such that (i) ijj(zmng) = u ( z ) ^ ( g ) for n £ N p n m £ M i and z £ Z. (ii) For all g £ G, the function m —►ip(mg) belongs to the p 1 -isotypic component of the space L q ( M i \ M \ ) of cusp forms on M\. (iii) The set of a £ Apx(R )° such that 'tp(amnk) ^ 0 for some m £ M j, n £ N p 1 , k £ Jf, is compact modulo v4g(R)°By a basic result ([L 9]), the series

v is ) = E «€ P \ G

converges and ip belongs to L (G ). Let L X( G) be the closed span of functions of this form. More generally, let P £ IP, set M = M p, N = N p , and suppose that M i C M . Then a subspace L X(M ) of L(M ,u/) is defined, where uP is the restriction to Z m of the central character of p.

15

The trace formula

Let cr be a unitary discrete representation of M and let abethedirect sum of the irreducible subspaces of L X( M ) isomorphic to cr.Extend cr to a representation of P trivial on N and let be the representation of G induced from a. We realize on the space of functions ^ on G such that (i) {^ng) = (g) for 7 £ P, n £ N . (ii) For all g £ G, the function m —> (mg) belongs to a. (iii) f |(I/F) parametrizes the set of £-conjugacy classes within the stable-e-conjugacy class of 7 . Let I d = 7 fl Gder- Then T>(I/F ) is contained in the image of H 1 ( F ^I d) in H 1 ( F^I ) since g can be chosen inside Gder(F). Depending on the context, T>(I/F) will also be denoted by D g (7 / F ), or fD (^/F). We denote the e-stable conjugacy class of 7 by 0 £_st(7). If 6 G ^D(7 ) F ) is represented by { r ( g ) g ~ 1 } , 7 6 will denote an element in the conjugacy class of g~1j g (thus j 6 is only well-defined up to G- conjugacy). The map ad(g-1 ) induces an E-isomorphism between G_^e and GseIn fact, if x £ G 7C(E ), then r ( g ~ 1 xg) = aTg~ 1 r ( x ) g a ~1, where aT = T( g ) ~ 1 9 ^ Gse(F). Hence ad(g~1) defines an inner twisting. In particular, G7e and G_$e are isomorphic if G_^£ is abelian.

Chapter 3

20

Let 0 £_ st(G ) denote the set of stable e-semisimple conjugacy classes in G. We will drop the subscript e when e is trivial. 3.2. Some results o f K o ttw itz. In this section, we assume that e is trivial. Let 7 £ G be a semisimple element and let I = G7. By a theorem of Steinberg ([St], pg. 57), the assumption that Gder is simply connected implies that I is connected. THEOREM 3.2.1 (Kottwitz-Steinberg, [Kt 2], Theorem 4.1); I f G_ is quasi­ split and G^el is simply connected, then every conjugacy class in G ( F ) which is defined over F contains an element of G. Suppose that G; is an inner form of G and let if : G — ►G; be an inner twisting. By definition, for t £ T there exist elements x T £ G ( F ) such that T('ij)(g)) = x r < i p(T(g))x ~ 1 for all g £ G. The G'(F)-conjugacy class of V>(t) is therefore defined over F. If G; is quasi-split, then the intersection of this class with G' consists of a stable conjugacy class O s t^ ) for some 7 ' £ G by Theorem 3.2.1. In this case, we obtain a map

Oe-st(G) — + Oe-st(G') sending 0st(7) to 0 st(7 /). We will say that a class 0 st(7 ') in 0 £_ st(G ') occurs in G if it is in the image of the above map. As observed in [Kta], the map H — ►Z ( H ) defines an exact functor from the category of connected reductive groups over F with respect to normal Fhomomorphisms (F-homomorphisms whose images are normal subgroups) and the category of diagonalizable groups over C with a T-action. Let V denote the torus G/G^er. Then the sequence

1

►V

►Z ( G ) — > Z (Gder) — ♦ 1

is exact. Hence V = Z ( G ) since Gder is simply connected and Z(Gder) is trivial. Similarly, we have a T-equivariant exact sequence

1 — >Z ( G )

Z ( f ) — ►Z ( I d) — > 1

and the associated long exact cohomology sequence yields: (3.2.1)

M Z (G f) —

M Z (if) —

M Z ( I d) r )

H \ F ,Z(G ))

where, for a complex algebraic group H , wq( H ) is the finite abelian group of connected components of H. Let 01(1/F ) denote the kernel of b if F is local and let 01(1/F ) denote the set of x £ ivo(Z(I d) r ) such that b(x) is

21

Stable conjugacy

locally trivial if F is global. In the global case, 91(1/F) maps to 91(1/F v) for all v. Let A ( G ) be the dual of the abelian group 7To(Z(G)t ). Dual to (3.2.1), we have A ( I d) —

A ( I ) — >A(G ) .

Let 8(1/F) be the image A ( I d) in A ( I ) . If F is local, then 91(1/F ) and 8,(1/F) are dual abelian groups. Assume that F is local for the rest of this section. By [Kt4], Lemma 4.3, there is a canonical morphism H 1^ , I ) — ►A ( I ) such that the right square of the following diagram commutes: H l ( F , I d) ------- > H X( F J )

A ( I d)

------- ► A ( I )

> H \F,G)

► A(G) .

The left square also commutes because the map H 1 ( F yI ) — > A ( I ) is functorial with respect to normal homomorphisms of connected reductive group ([K t4], Theorem 1 .2 ). The bottom arrows are group homomorphisms while the top arrows are maps of pointed sets. Suppose that 7 ' £ 0 st(7 ) and let a be the element of T>(j/F ) correspond­ ing to the conjugacy class of 7 '. Let inv(7 , 7 ') be the image of a in A ( I ) . Then inv( 7 , 7 ') lies in 8(1 /F) and each element of 91(1/F ) can be viewed as a function k (inv( 7 , 7 ')) on 0 st( 7 ) which is constant on conjugacy classes. The /c-orbital integrals described in §4 are built using elements of 91(1/F). In particular, the role of a semisimple conjugacy class 0 st(7) within the endoscopic analysis of G is governed by 91(1/F). If F is p-adic or if F is local and 7 is regular, the map H 1 ( F , I ) — > A ( I ) is an isomorphism ([K t4], Lemma 1.2), and inv( 7 , 7 ') is trivial if and only if 7 and 7 ' are e-conjugate. Let T be an F-torus. Let A denote the adele ring of F, i.e., the direct limit of the rings A k where K ranges over the finite extensions of F in F. It is a consequence of Tate-Nakayama duality ([K tj]) that: (3.2.2)

f H ^F.T) A(T) - |

if F is local if F ig global

To calculate A (T ), it is often convenient to use that A ( T ) is isomorphic to the Tate cohomology group H ~ x(F, X + ( T ) ) . Recall that H ~ 1 (F, X * (T )) is defined as follows ([Se]). Let K / F be a Galois extension over which T splits

22

Chapter 3

and let N ^ / f he the norm map on X * (T ), i.e., N k / f (

=: ]C r (^)> where

the sum is over G a l ( K /F) . Then H - 1 (F, X * ( T ) ) is canonically isomorphic to Ker( N k / f ) ! 3 X * ( T ) where 5 is the ideal in Z [ G s l ( K /F ) ) generated by elements of the form (r — 1 ). The next result is a special case of [Kt3], Lemma 2.2. LEMMA 3.2.2.Let T be an F-torus. Then 7To(Tr ) is canonically isomorphic to jy 1 (F ,X * (T )). 3.3. A n obstruction. Let F be a global field and assume that Gder satisfies the Hasse principle. We also assume, for simplicity, that G is quasi­ split (see [Kt3] for the general case). Let 7 £ G be a semisimple element and set

0,t(7/A) = { 7 ' € G : 7 ^ is stably conj. to 7 in G v for all u}. If 7 ' £ 0 st( 7 /A), then j'v is conjugate to 7 by an element of K v for almost all v. Indeed, by [Kt4], Proposition 7.1, this is the case for all v such that (i) G v is unramified and K v is hyperspecial, (ii) 7 and 7 ^ belong to K v, and (iii) 1 — 0 ( 7 ) is either 0 or a unit for every root a of G. It follows that 91*9~X ~ 7 f° r some g £ G (A ), where A is the union of A ^ over all finite extensions L/F. We can suppose that g 6 Gder(A). Then the cocycle {aT = r ( g ) g ~ 1} takes values in I d( A ). Observe that 7 ' is Gder-conjugate to an element of G if and only if there is a choice of g such that {aT} takes values in I d( F ) . In fact, if aT £ I d{ F ) for all r, then the image of {aT} in H 1 (F^Gder) is locally trivial. By the Hasse principle, there exists x G Gdei(F) such that T{9 ) 9 ~ 1 — t ( x ) x ~ x and 7 ' is Gder-conjugate to x ~l ^x. According to [Kt4], §2 .6, there is an exact sequence (for any connective reductive group) (3.3.1)

H \ F , I d) — ►H \ F , I d( A ) ) M

A ( I d)

Set obs(7 ') = / (a), where a is the class in H 1 ( F ) I d( A ) ) defined by {a r }. We see that obs(7 ;) is trivial if and only if 7 ' is Gder-conjugate to an element of G. The following proposition, due to Kottwitz ([K t4]), gives the condition for 7 ' to be G-conjugate to an element of G in terms of obs(7 '). P R O P O S I T I O N 3.3.1: Assume that Gder is simply connected and satisfies the Hasse principle. Let 7 ' £ 0 st( 7 /A). Then 7 ' is G-conjugate to an element of G if and only if K^obs^*)) = 1 for all k £ 9l(I/F).

Stable conjugacy

23

C o r o l l a r y 3.3.2: Let 7 ' e 0 st( 7 /A ). Then P K I I F )\~1

K(obs(i' )) Kex(i/F)

is equal to 1 or 0 according as 7 ' is or is not G -conjugate to an element of G. 3.4. C artan subgroups o f u n itary groups. Let G be a unitary group defined by a pair (jD, a) consisting of a division algebra D of rank n over E and an involution a of the second kind. Let T be a Cartan Subgroup of G. Then a stabilizes the centralizer V of T in D and ( L ', a ') is a pair of the second kind, i.e., V is a finite-dimensional, semisimple commutative algebra over E and af induces a on E. Let T^l ' ,a’) be the F -torus such that T(L',a- ) { F ) = { x € L'* : xa ' ( x) = 1} . Then T is isomorphic to T ^ ^ a>y For any extension K / L , let ( Gm)K/L denote the L -torus Res^/^(Gm). If (£ ', a') is an irreducible pair of the second kind, i.e., is not isomorphic to a direct sum of more than one non-trivial pair, then either V is a field or V — L n ® L ", where L " is a field extension of E and a\x, y) = (a "(y ), a " ( x ) ) , where a " is an involution of the second kind on L " . In the latter case, ,af) 1S isomorphic to (G m)^///p. If L ' is a field and L is the fixed field of a, then T( l ’,af) isomorphic to k e r w h e r e is the norm map from (G m) l ,/f (£*m)L/F- Clearly, every pair is isomorphic to a direct sum of irreducible pairs and T is isomorphic to a product of tori of the form where (Z/,a') is irreducible. We will say that Cartan subgroups T\ and T 2 of G are stably conjugate if there exist regular elements 7 j G Tj such that 71 is stably conjugate to 72 . This is the case if and only if there exists g G G ( F ) such that gT\g- 1 = T 2 and the isomorphism t — > gtg~x is defined over F. (a ) Two Cartan subgroups of G are stably conjugate if and only if they are isomorphic as F-tori. (b) Assume that F is a global and that n is odd. Let T be an F-torus of dimension n of the form T(L',a')- Then T embeds in G if and only if Tv embeds in G v for all places v of F. PROPOSITION 3.4.1:

Proof: If T\ and T 2 are Cartan subgroups of G which are isomorphic as i^-tori, then T i ( E ) and T 2 ( E ) isomorphic Cartan subgroups of D* and

24

Chapter 3

hence gT\(E)g ~1 = T 2 ( E ) for some g € D*. The map t — > ad(g)t from T\ to T 2 is defined over E and it is defined over F if a(g)g centralizes Ti. Let V be the centralizer of T\ in D, and let a' be the automorphism of V defined by ot\£) — g~ 1 a(g£g~~1 )g. Since Ti and T 2 are isomorphic as F-tori, the pairs ( L ' , a ) and (I/, a ') are isomorphic and there exists an automorphism f3 of V such that ot\£) = /?(a(/J_1(^))). As is well-known (cf. [We], page 301), f3 is induced by ad(n) for some n in the normalizer of V in D. We obtain g~ 1 a(g£g~1)g = n a(n - 1 ^n)n-1 . It follows that a(g)gna( n) and hence a(gn)gn, centralize V and we can replace g by gn. This proves (a). We now prove (b). If Tv embeds in G v for all places v of F , then V Q f F v embeds in D F v for all v. Hence V embeds in T>, and T embeds in G if and only if an embedding i : V — > D exists such that i(a' (£)) = a(£) for £ £ V . Fix any embedding of V into D and regard V as a subalgebra of D. There exists x £ D* such that ol\£) = a ( x ~ 1 £x) for £ E V since the embeddings £ — ►a{£) and £ — ► are conjugate in D * . The problem is to show that there exists g £ D* such that g~ 1 a ,(£)g = a ( g ~ 1 £g) for £ 6 V . This is the case if and only if the equation ga(g) = tx has a solution for some g G D* and t € L 1 . Observe that if ( g , t ) is a solution, then a(tx) = tx. A solution exists locally everywhere by hypothesis. Suppose that gi £ (T> (g) R )* and t\ £ ( V (g) R)* give an archimedean solution. If £ £ V H \ F , G 0) where Go = {g € G : N Ef E(g) = 1 } is the special unitary group associated to G (cf. [Kn]). Since Go is simply connected, H 1 ( F v,Gov) = 0 for all finite v by Kneser’s theorem. The Hasse principle holds for Go ([Kn]), and hence (y,z) £ A is in the image of D* if it is in the image of ( D x _1. Hence Z (G )r = { ± 1 } and (a) follows. If F is global, then H 1 ( E , Z ( G ) ) = H om (r£, C *) and the restriction to E of an element x £ ker 1 ( F , Z ( G ) ) to E is trivial by the Chebotarev density theory. Hence x defines an element of H 1 ( F ( E /F ) , Z ( G ) ) . This implies (b) since H 1 ( T ( E /F ) , Z ( G ) ) coincides with its localization at any place which remains prime in E. Part (c) is immediate from (b) and the definitions. P R O P O S I T I O N 3.5.2: Let T be a Cartan subgroup of G. Suppose that T is isomorphic to T(L',a') ® 7(L",a") where ( L ^ a 1) is isomorphic to a direct sum of r irreducible pairs (Li,oti) such that Li is a field, and ( L n, a ,f) is a direct sum of irreducible pairs ( L ,n, a ,n) such that L ,n is not a field. Let L be the subalgebra of V fixed a1. (a) H 1 ( F , T ) is canonically isomorphic to L/N^t (b) A ( T ) is naturally isomorphic to (Z/2)r . (c) I f T is anisotropic, then £ (T /F ) is isomorphic to the subgroup {(£ j) £ (Z/2)r : Eej = 0}. The order of9l( T/F) is 2r~1. (d) 9l(T/F) is isomorphic to the image of H ^ F , X * ( T ) ) in H X(F, X * ( T d)) (e) X * (T )) is isomorphic to (Z/2)r . The image of in H ~ \ F , X * ( T ) ) has index two.

Proof: To prove (a) and (b), we can assume that T corresponds to an irreducible pair of the second kind. If T = T^l " ,&")'> where L " is not a field, then T is isomorphic to {Gm)K/E f° r some extension K/E. In this case,

26

Chapter 3

H 1 ( F , T ) = H 1 ( E , ( G tn) k/e)^Y Shapiro’s lemma, and H l ( E, (G m)K/E) — 0 by Theorem 90. It is clear that N / i is snrjective in this case, and (a) follows. By (3.2.2), the triviality of A { T ) follows by Theorem 90 in the local case and by the vanishing of H 1 for the idele class group in the global case. If T = T{ l , ^ where V is a field, then, then (a) follows from theorem 90 and the cohomology sequence associated to the exact sequence defined by the norm:

1 — ►T — ►( G m) L*/F — ►(G m) L/F — > 1 .

To prove (b), we use that A ( T ) is isomorphic to H ~ 1 (F^ X * (T )). Let {p-7} be the set of F-embeddings of V into F and let {pj} be the dual basis of X * ( ( G m)L' j F ). As a right T f -module, X * ( T ) is isomorphic to the submodule of ® Z pj spanned by the set { ( a ' — 1 Since T is anisotropic, ker( N l i / f ) coincides with X * (T ) and H ~ X(F, X * (T )) is isomorphic to the quotient of X * ( T ) by the span of {( a ' — 1 )p j(p — 1) : p G I V } . For all j, k there exists p E T f such that pjp = pk or a'pk and hence

( H 1 (F, G) is injective by the cohomology exact sequence. It follows from (3.5.1) for the case 7 = 1 that if is injective. If T is any anisotropic Ftorus, then H ° ( F , X * ( T ) ) = X * ( T ) r = {0 }. If F is local, this implies that H 2 ( F , T ) is also trivial by Tate-Nakayama duality ([Kta], §3). Since Z q is anisotropic, this shows that if) is surjective in the local case. Suppose that F is global. The Hasse principle holds for Gder by Landherr’s theorem ([K n]) and also for Gad ([Ha], Satz 4.3.2). The upper horizontal and right vertical arrows in the following diagram are isomorphisms and the lower

Chapter 3

28

horizontal arrow is injective. H \ F , G dei) ------- ►n J T ^ .G d e r ) VH \ F , G * a) ------- > n ^ ^ . G a d ) Hence ^ is also an isomorphism. 3.6. Classification o f C artan subgroups. We now suppose that G is a unitary group in 3 variables. Let H = Hq x U ( l ) where Ho is a unitary group in 2 variables. It is convenient to classify the Cartan subgroups of G according to four types. If K / F is a quadratic extension distinct from jE7, let V — K E and let ol be the automorphism of V of the second kind which fixes K . Denote T(L',a') by X/c. If L / F is a cubic extension, let V — L E , let a 1 be the automorphism of the second kind of V whose restriction to L is trivial, and denote by T l . By the results of §3.4, every Cartan subgroup of G or H is isomorphic to one of the following types of tori: Type (0): ( G m) B/F x E 1 Type (1): E 1 x E 1 x E 1 Type ( 2 ): TK x E 1, where K is a quadratic extension of F distinct from E Type (3): T l where L / F is a cubic extension. By Proposition 3.4.1(a), the stable conjugacy class of a Cartan subgroup of type ( 2 ) (resp., type (3)) is determined by the extension K / F (resp., L/F), and there is a unique stable conjugacy class of Cartan subgroups of type (0) or ( 1 ). We will say that a Cartan subgroup To of Ho is of type ( j ) if T 0 x Z7(l) is of type (j ). According to Proposition 3.5.2(d), 0l(T/F), is isomorphic to Z/ 2)2 (resp., (Z/2)) if T is of type ( 1 ) (resp., type ( 2 )), and 9tfT/F) is trivial if T is of type (3). 3.6.1: Let T be an anisotropic Cartan subgroup of H . Then natural map &h( T/F) — ►8 g ( T /F ) is injective and its image has index two.

LEM M A

Proof. T is of the form T0 x E 1 where To is a Cartan subgroup of Ho. The lemma follows easily from Proposition 3.5.2. 3.7. W e y l groups. We denote the Weyl group of a Cartan subgroup T of a group G_ will be denoted by ^ g (T ). Let £If ( T , G ) be the subgroup of J2g(T) consisting of elements whose action is defined over F. The Galois

Stable conjugacy

29

group r acts on Q,g ( T ) and l i F(T, G) is the group of fixed points of this action. Let fi(T , G ) be the Weyl group of T in G. The symbol G will be omitted from the notation when this causes no ambiguity. For simplicity, we will not distinghish in the notation between an element of a Weyl group and a representative for that element. Now let G = 17(3) and let H = U ( 2) x 17(1). We identify H with the subgroup G 1 of G, where 7 = d (l, —1 , 1 ). P r o p o s it i o n 3.7.1:

(a) ftF(T, G ) = 53 i f T is of type ( 1 ). (b) f i F(T, G) = Z /2 if T is of type ( 2 ). (c) T C H, then f i F (T, JT) = Z/2. Proof: If T is of type ( 1 ), then T = ( E 1 )3 and ft(T, G) acts by permuting the factors. This action is defined over F and (a) follows. If T is of type ( 2 ), then f i F(T, G) has order at most 2 and hence (c) implies (b). Part of (c) follows from the corresponding fact for Cartan subgroups of the derived group, SL2(F ), of H. Let:

where a E F * is chosen so that G is isomorphic to £/$/. Let T ' be the diagonal subgroup in U$>’ . It is of type ( 1 ) and the elements:

(for any choice of a, b E E *) generate 0 F (T ', G'). Furthermore, w 6 f l(T #,G ') and w' € f l(T ',G ') if and only if iVE/F(a ) - 1 = N E/F(b) = a . It follows that i l ( T /,G ') = S 3 or Z/2 according as a E iVF?* or a £ N E * . If F is p-adic, then all unitary groups in 3 variables are isomorphic and a is arbitrary. Hence there exist conjugacy classes { T i } and {T 2} of Cartan subgroups of type ( 1 ) such that ft(T i,G ) = S 3 and 0 (T 2,G ) = Z/ 2 . If 7 E Ti is regular, then 0 st(7) contains 4 conjugacy classes since £ g(T i /F ) = (Z/2)2. If F is real, then there is a unique conjugacy class { T } of Cartan subgroups of type ( 1 ) and Q(T, G) = Z/ 2 .

30

Chapter 3

3.7.2: Let F be a local field. There exists a Cartan subgroup T of type ( 1 ) contained in H such that 0 (T , G) = Z/ 2 . P R O P O S IT IO N

Proof: Let T C H be a Cartan subgroup of type ( 1 ). If £2(T, G) = 53 , then F is p-adic and fi(T , H ) = Z/ 2 . Let 7 be a regular element in T. Since T>h(T/F) = £>h(T/F) has order two and injects in 8>g(T/F) by Lemma 3.6.1, there exists 7 ' £ H which is stably conjugate but not conjugate to 7 in G. The centralizer T ' of 7 ' is not conjugate to T in G and hence ft(T ,G ) = Z/ 2 . 3.8. Singular sem isim ple elements. Let G = 17(3). For £ £ F *, let be the unitary group in two variables defined by the Hermitian form:

The isomorphism class of H'^ depends only on £ modulo N E * and we obtain a bijection between F* /NE* and the set of isomorphism classes of unitary groups in two variables over F with respect to E. The group H\ is quasi­ split and is isomorphic to 17(2). The subgroup of elements of determinant one in is isomorphic to the norm one subgroup of the unique quaternion algebra over F which is ramified precisely at the set of places v of F at which £ is not a norm from E v. Set x E 1 and let H = H\. 3.8.1: Let 7 £ G be a semisimple element which is singular but not central. (a) I f 7 ' £ 0st(7), then G y is isomorphic to H £ for some £ £ F * / NE * . The map sending 7 ' to £ defines a bijection between the set of conjugacy classes within 0 st(7) and F * / NE * . In particular, two elements in 0 st( j ) are conjugate if and only if their centralizers are isomorphic. Let Gl be an inner twist of G defined by a pair ( D , a ) . (b) 0 st(7) transfers to G f if and only if D — Ms (E ) . (c) Suppose that F = R . I f G 1 is compact, then 0 st(7) transfers to a single conjugacy class in G ' . (d) I f F is global, then the set of conjugacy classes in G ' which transfer to Ost ( j ) is parametrized by the set of £ £ F * / N E * such that £ is negative at all real places of F at which G is compact. P R O P O S IT IO N

Proof: Two semisimple elements in G ( E ) = GL 3(F 7) are conjugate if and only if they the same set of eigenvalues. Thus 0 st(7) is determined

Stable conjugacy

31

by the set of eigenvalues of 7 , which is of the form {a , a, /?} where a ^ /3. Furthermore, T ( F / E ) acts on {a ,a ,/ ?} and the action must be trivial. Hence a,/3 £ E * , and in fact a, /3 £ F 1 since set of eigenvalues is also stable under a; — ►x ~ l . Up to stable conjugacy, we can assume that

7 - ( ‘

*

Then G 7 = H\ and G* = {(g(T/F ) by Lemma 3.6.1. This proves the next proposition. PROPOSITION 3.8.2: Assume that E / F is local. Let

7 be a singular non­ central element in M . Then there exist 7 ', 7 " £ H which are regular as elements of H such that { 7 *, 7 ” } is a set of representatives for the conjugacy

classes within 0 st( 7 )* 3.9. N on-sem isim ple classes. A unitary group in 3 variables contains non-semisimple elements if and only if it is quasi-split. Let G = U (3). Every unipotent element in G is conjugate to an element of TV, and an element u(x, z) is regular if and only if x 0. A unipotent element u £ G will be called singular if it is not regular and u ^ 1 . PROPOSITION 3.9.1: The set of regular unipotent elements in U(3) consists

of a single conjugacy class. Proof: The regular unipotent classes form a single Gad-conjugacy class. The assertion follows from Lemma 3.5.3(a). All singular unipotent elements are conjugate to an element of the form n(£) for some t £ E ° . If u = n(t), then G u = S • N where S = {m £ M : a 3(m ) = 1). The conjugacy class of u is determined by t mod N E * . Suppose that 7 £ G has non-trivial unipotent part and non-central semi­ simple part. Then the unipotent part of 7 is conjugate to n( t) for some

Stable conjugacy

33

t E F* and there exist a,/? E E 1 such that, up to conjugacy,

,

u;::)

a) \0 0 1 /

and G 7 = S • {n( t ) : t E i£°}. The conjugacy class of 7 is determined by t mod N E * and n,/3. 3.10.

T h e n orm map. Let G be a connected, reductive group over F

such that G^er is simply connected and let G = Rese /f (G)> where E / F is a cyclic extension of degree I. Let a be a generator of T ( E / F ) . Over E , G is isomorphic to the product of I copies of G in such a way that a acts by ( x i , x 2,... ,x/) --- ►(cr(x^),cr(xi),... , a ( x t - i ) ) and G = { ( x , a ( x ) , . . . , ae- \ x ) ) : x £ G ( E ) } . We identify G with G(25) by projection on the first factor. Let e be the algebraic automorphism of G consisting of a cyclic shift to the left. Then c induces a on G ( E ) under the identification. The map i : G —> G defined by x —> (x, x , ... , x) is an isomorphism of G with the subgroup of G fixed by e Define the norm map N : G — ►G by N ( S ) = 6e( 6 )e 2 ( 6 ) - - - e e- 1 ( 8 ) . There is a bijection between the set of e-conjugacy classes in G ( F ) and the set of conjugacy classes in G(jF), defined by sending an £-conjugacy class 0 e(£) to z- 1 (0 ), where 0 is the intersection of { N ( 6 f) : S' E 0 f(^ )} with i ( G ( F ) ) . The relation iV (x“ 1 fc (x )) = x ~ 1 N ( 6 )x for x E G ( F ) shows that 0 is a union of conjugacy class in G ( F ) . If wi,u )2 E G ( F ) and *(1^1 ) is conjugate to i(w2 ) in G { F \ then w\ is conjugate to w2 in G ( F ) and hence 0 is a conjugacy class. If 8 = (x i, x 2,... , then N ( 6) =

■•• , ( x i x 2, . . . , x e- 1 ) ~ 1 -y(xix2, . .. ,a:/_i))

where 7 = x ix 2 ... x^, and iV(6) is conjugate in G ( F ) to i ( 7 ). This shows that the image of 0 €(e( 8 /F) is isomorphic to H 1 ( F ^ E 1) and 8 is econjugate to z 8 for z E F * . But then 8 is e-conjugate to z 8 if 8 is e-stably conjugate to a scalar element, i.e., if N ( 6 ) is a scalar. This completes the proof of (c) and (e) also follows from the remarks of §3.11.

38

Chapter 3

3.13. e-classes when n = 3. Let G = U ( 3) and let H = H 0 x 17(1), where Fo = 17(2). Denote the determinant on the 17(2)-factor of H by deto. PROPOSITION 3.13.1: Let T be a Cartan subgroup of H of type (1) or (2).

Let 8 £ T and let v £ T>£( 8 /F). Then deto(t„) £ N E * if and only if v belongs to the image of T>h(T/F). Proof: Let T d = {(£, det(t)-1 ) £ Ho x 17(1)} and let T d = T fl (Tfo)derThen T>e( 8 /F) = H x{ F , T d) by Proposition 3.11.2(b), since i7*(F, T d) in­ jects in H 1 (F, T). As with (3.12.1), deto induces an exact sequence H \ F , T d) — > H x( F , T d) — ►H ^ F . E 1) . Now T>h(T/F) is contained in the image of H 1 (F, T d) in i7*(F, T d). Since the determinant induces an isomorphism of H 1 (F, Ho) with H X(F, F 1), the image of H 1 ( F , T d) in H 1 ( F, Ho) is trivial and T)//(T/F) coincides with the image of H \ F , T d) in H \ F , T d). P R O P O S I T I O N 3.13.2: I f 8 £ G is e-semisimple but not e-regular, then 8 is stably e-conjugate to an element of the form d(a,f3,a). (a) I f N ( 8 ) is not central, D £( 8 / F ) is naturally isomorphic to F* /NE*. (b) I f N ( 8 ) is central, D e( 8 /F) is naturally isomorphic to H 1 ( F , G 8La).

Proof: The first statement is clear since every non-regular diagonal el­ ement of G is of the form N(d( a, ft,a)) and (a) follows from Proposition 3 .1 1 .2(b) and (3.8.1). If N ( 8 ) is central, then 8 is stably conjugate to a scalar element. We can assume that 8 is scalar. Then the assertion follows from ( 3 .5 .2 ), since H 1 ( F , G ) maps onto i7 2(F, Gad) by Lemma 3.5.3. The set H 1 (F, Gad) parametrizes the F-forms of G. If F is local, H 1 (F, Gad) is trivial unless E / F = C/R, and i F ( R , Gad) = Z/ 2 . For F global, F a(F ,G ad) = IL ff 1 (F v,G ad) as observed in §3.5.

C H A PTE R 4

Orbital integrals and endoscopic groups

In §1-6, we define endoscopic groups and the problem of transfer of orbital integrals. The endoscopic groups for unitary groups are determined in §7 and we describe the problems for unitary groups in two and three variables treated in this work from the point of view of functoriality in §8. The remaining sections deal with the transfer of orbital integrals for these cases. 4.1. /c-orbital integrals. For the general discussion, we assume that Gder is simply connected. Suppose that F is local and that e is trivial. If 7 £ G is semisimple, an element n E 3£(G 1 / F ) defines a function 7 ' —> /c(inv(7 , 7 ')) on 0 st(7). If 7 is regular, define the K-orbital integral of / E C { G , u ) by: (4.1.1.)

$ k(7,/ ) = Y

* ( inv(7 ,7 '))$ (7 ',/ ) •

{V }

where { 7 ' } is a set of representatives for conjugacy classes within 0 st( 7 )If k is trivial, st( 7 ,/). The ordinary orbital integrals $ ( 7 ',/ ) determine the /-c-orbital integrals and vice versa by the orthogonality relations for the finite abelian group £(G 7 /F). A distribution T on C(G,lo) is called invariant if T ( f 9) = T ( f ) for all g E G, where f 9( x ) = f ( g ~ 1 xg). If $ st( 7 ,/ ) = 0 for all regular semisimple 7 , we will say that / is stably equivalent to zero. An invariant distribution will be called stable, or stably invariant, if T ( f ) = 0 for all / which are stably equivalent to zero. If 7 is not regular semisimple, then the sum (4.1.1) with k trivial need not define a stable distribution. It is necessary to modify the definition (4.1.1) by inserting appropriate coefficients. The coefficients are determined by the asymptotic expansions of orbital integrals around semisimple elements. For any connected reductive group G, let q(G) denote one-half the (real) dimension of the symmetric space attached to G if F is archimedean and

40

Chapter 4

let q(G) be the F-rank of Gder if F is p-adic. Following [Kt5], let e(G) = ( _ 1 y(G)~q(G')^ where G' is the quasi-split form of G. If 7 is semisimple, then G 7 is reductive and connected (since Gder is simply connected). Define (4.1.2)

$ * (7 ,/ ) = X ] e( 7 ') * ( inv(7 ,7 '))$ (7 ',/)• { 7'}

where e( Y) . If 7 is regular, then G 7 is a torus and the signs e ( j f) axe trivial, so (4.1.2) is compatible with (4.1.1). 4.2. Endoscopic groups. To each pair (G ,e) is associated a set of auxiliary groups called endoscopic groups. These were first defined in [F 3] in the ordinary ( e trivial) case. The introduction of endoscopic groups in the twisted case is due to Shelstad. In this section, we follow [KS]. An element s £ G will be called e-semisimple if the endomorphism ad(s)o e of G fixes a Bor el pair (if, T ) (recall: B is a Borel subgroup of G and T is a maximal torus in B). If s is e-semisimple, then the connected component G(se)° of the e-centralizer G(se) = {g £ G : g~ 1 se(g) = 5} is a connected reductive subgroup of G by [St], Corollary 9.4. An endoscopic triple is a triple (if, 5, 77), consisting of a quasi-split group if , an e-semisimple element s in G and an L -map 77 : LH —* LG, (we take the Weil forms of the F-groups) which satisfies the following two conditions. (I): 77 restricts to an isomorphism of complex groups from i f to G( se)° . Define A(w) = se(rj(w))s ~ 1 7/(tz;)- 1 for w 6 W f(II): A takes values in Z (G ) (in which case A defines a cocycle with values in Z ( G )) and the class of A in H 1 ( W f , Z ( G ) ) is locally trivial (resp. trivial) if F is global (resp. local). The endoscopic group is the quasi-split connected reductive group if. The triple, or just i f itself, is called elliptic if 77( ( Z ( i f ) r )°) C Z ( G ) T . Note that if e is trivial, then (G, l,id.) is an elliptic endoscopic triple for G. Let (if;, 5,-, 77^), i = 1 , 2 , be endoscopic triples for G. An isomorphism between them is a pair (o, /?) of maps a : if , -> i f 2 f3: l H 2 ^

lE x

where a is an F-isomorphism and j3 is an F-homomorphism satisfying: (i) /? induces an isomorphism of H 2 with i f i dual to a. (ii) There exists g G G such that gsis(g ) ~ 1 s^ 1 € Z(G )C ent (772, G)° and 772 = ad(g) o r j i o f3.

Orbital integrals and endoscopic groups

41

In particular, if a is a 1 -cocycle on W f with values in Z ( H ), then (if, syrfa) is isomorphic to (if, 6,?/), where rja(h x w) = rj(a(w)h x tt>). Every element of i f defines an automorphism of (if, 5 , 77) by conjuga­ tion. Denote the automorphism group of ( H,s,r/) by A u t(if, 6, 77) and let A (if, s, 7 7 ) = A u t (if, 5 ,r/)/if. Then A (if, s,rj) is a finite group. 4.3. Tran sfer o f o rb ita l integrals. Recall that Ost(G ) is the set of stable conjugacy classes of semisimple elements in G. Let T be a maximal F-torus in G. If 7 E G (F ) is semisimple, then the G(F)-conjugacy class of 7 intersects T ( F ) in an ftc^^-orbit. The orbit is fixed by T if and only if the F-conjugacy class of 7 is defined over F. Sending a stable conjugacy class to the image of its intersection with T ( F ), we obtain a map 0 st(G ) - » [T(F)/SlG( T ) f . Assume that G is quasi-split and Gder is simply connected. Then every F-rational G(F)-conjugacy class intersects G non-trivially by the KottwitzSteinberg theorem (Theorem 3.2.1) and in this case the map is bijection. Assume further that e is trivial. Let (if, s, 7 7 ) be an endoscopic datum for G, and let ( B h , T h ) and ( B , T ) be Borel pairs which are fixed by T in i f and G, respectively. There exists y £ G such that ad(y) 077 takes ( B h , T h ) into (R , T ). The set of roots of Th in B h maps to the set of roots a of T in B such that a(ysy~x) = 1. In particular, H h ( T h ) is mapped to a subgroup of Q.g(T) under ad(y) o 7 7 . Define t/>a : Th —> T by V>A(t) = yrjfyy- 1 . For r G T, there exists x T £ G such that

for h £ i f and consequently, t/>a(t(/ i)) = nrr ( 7/>A(/i))n“ 1, where nT = yxTr ( y ~ 1) belongs to 0>g ( T) . The actions of T on X * (T h ) and X * (T ) therefore differ by a twisting with values in f I g (T) . Let (T_h ,B_h ) and (T-iB.) be Borel pairs in H and G, respectively. We have canonical identifications of X * ( T h ) and X * ( T ) with X * ( T h ) and X * (T ), respectively, under which Q,g ( T ) and SIh {T h ) are identified with S lc(T) and O h (T # ), respectively. The map t/>a defines an isomorphism X*(Th ) X * ( T ) which carries £Ih ( T h ) into a subgroup of 0 g (^ )- This gives rise to an F-isomorphism xj) : T_H —> T? the inverse of the map natu­ rally associated to X * ( T h ) —* X * ( T ) . Assume that T_H and T axe defined over F. Then the T- actions on X * (T ) and X * ( T ) differ by a twisting with values in Q,g ( T ) . In fact, if B_ is defined

42

Chapter 4

over F , then the T-actions coincide. On the other hand, if T r is another Cartan subgroup of G defined over F , then T ' = g T g _1 for some g G G (F ). The 1 -cocycle {^ - 1 r (^ )} takes values in Q,g(T) and ad(#) intertwines the T-action on X * ( T ') with the twist of the T-action on X * (T ) by {^ _ 1 r (^ )}. The analogous statements hold for X * ( T h ) and X * ( T h ), and hence there is a 1-cocycle {wT : r G T } with values in Q g ( T ) such that ^ ( T( 0 ) = ad(u;r )(r (^ (^ ))) . In particular, xp induces a T-equivariant map T h ( F ) / Q h ( T h ) -> T(F)/SlG( T ) and hence a map:

A g/ h :

- Ost(G) .

The map Ag/h 1S independent of all choices. Note also that xp- 1 restricts to an embedding of Z g in Z h which is defined over F. Observe that T can be chosen so that xp is defined over F. In fact, let 7 G Th be an element such that ^ ( 7 ) is regular in G (F ). The conjugacy class containing ^ ( 7 ) is defined over F and, by the Kottwitz-Steinberg theorem (Theorem 3.2.1), there exists g G G ( F ) such that gxp(y)g~ 1 is F-rational. Let xp'(t) = gxp(t)g~l . Then *A'(t) = (^ r - r (s ')_ 1 ) _ V /(7)(5wJr r (ff)“ 1) and hence gwTr ( g ) ~ 1 G g T ( F ) g ~1 since ^ '( 7 ) is regular. It follows that xp' is defined over F and we can replace xp and T by xp1 and g T g _1. A semisimple element 7 ' G H is called G-regular if Ag///(7 ;) 1S a regular class in G. Suppose that 7 ' G T//. Then 7 ' is called (G, H )-regular if 0 (^ ( 7 ') ) ^ 1 for each root a of T which is not the image of a root of T h in H. Let 7 ' be a (G, F)-regular element of T//. Suppose that xp is defined over F (this entails no loss of generality since the choice of T is arbitrary) and let 7 — 'lP ( l ' ) ' Then xp defines an F-isomorphism between the root data of T_H in F 7, and that of T in Gy. It follows that xp extends to an isomorphism of F y with G 7 which is an inner twisting over F. In particular, we can identify Z ( H ^ ) with Z (G 7) and, if F is local, we can choose compatible measures on i7y and G7. Let T be a maximal F-torus in F y . Then Z ( H ) and Z ( F y ) can both be viewed as subgroups of T and as such, Z ( H ) C Z ( H y ) , where this inclusion

Orbital integrals and endoscopic groups

43

is r-eq u iv arian t and independent of the choice of T. Since = Z (G 7 ), we obtain a canonical T-equivariant inclusion of Z ( H ) in Z (G 7 ). Now s defines an element s' of Z (G 7 ). By Condition (II) of §4 .2, the image of sr in Z ( G 1 ) / Z ( G ) is T-invariant and the image of s in 7r0([Z (G 7 )/ Z (G )]r ) defines an elem ent /c of SR(G7 / F ). We also obtain an element kv of 0^{Gy / F v) for all v. Assume now th a t F is local. According to a conjecture of Langlands, there is a function A g ^ ( 7 f , 7 ) on pairs consisting of a G -regular element 7 h £ H and 7 £ A G/ / / ( 0 st ( 7^ ) ) w ith the following property: for all / £ G (G ,c j), there exists function f H £ C(H,u>) whose orbital integrals m atch w ith those of / in the following sense: ( 4.3 . 1)

$ 8t( 7 b J h ) = A G//f( 7 ir, 7 ) * “( 7 , / ) .

for G -regular 7 ^ , where the orbital integrals are defined using com patible m easures on Hy> and G7 . We w rite / —* f H if / and f H correspond via ( 4 .3 . 1). T he correspondence depends on the choice of m easures on G and iJ , and f H is determ ined only up to the addition of a function on H all of whose stable orbital integrals vanish. As suggested in [K t^ , ( 4 .3. 1) should hold if 7 ^ is only assum ed to be (G , if)-re g u la r (cf. Proposition 8 .1.3). T he function A g / h ( 7 h , 7 ) is called a transfer factor. It depends only on the stable conjugacy class of 7 h in H and the conjugacy class of 7 in G. If 7 ; is stably conjugate to 7 in G, then consistency implies th a t ( 4 .3 . 2)

A g / //(7 h , 7 ') = A G//f( 7 K ,7 )« (in v (7 , 7 ') ) •

A general definition of the transfer factor has been proposed by Langlands and Shelstad ([L S ]). The definition specifies A q j h up to a non-zero con­ stan t th a t depends on certain choices. Globally, there exists a com patible collection {A g v/ h v} ° f local transfer factors such th a t for all G-regular semisimple 7 h € H and all 7 = (77J £ G such th a t 77 £ A Gv/Hv(Ost('yH)) for all v, the product A g /h (7 h >7) = n ^ . ( 7 H , 7 . ) V

exists ( A g v/ h v( 7 h , 7 V) = 1 for alm ost all v ) and the value A g / h ( i h , 7 ) is independent of the choice of com patible collection. F urtherm ore, A q / h satisfies the key property: ( 4 .3 .3 )

A G/ # ( 7 h , 7 ) = /c(obs(7 ))

44

Chapter 4

where o b s (7 ) E A (G ^ ) as in §3.3. Suppose th a t 7 E AG/ / / ( 0 st( 7 //)) and let / = ![,,/„ E C (G ,tj). For alm ost all v, $ Kv( y , f v ) = $ ( 7 , / „ ) , i.e., $ (7 ',/ ,; ) = 0 if 7 ' is stably conju­ gate but not conjugate to 7. This is the case if f v is the unit in *HV and (1 —0 ( 7 ) ) is a u n if or zero f°r roots of G ([K t4], § 7.3). We m ay therefore set $ * ( 7 , / ) = n v$ K”( 7 , / v). By ( 4 .3 .2) and ( 4 .3.3), $ * ( 7 , / ) = Y 1 K( obs(7 {V }

where {7'} is a set of representatives for the G-conjugacy classes in 0 st (7 / A ). The sum is finite by the above rem ark. By ( 4 .3.3), if 7 h is G-regular, then A g / h (7 H j 7 ) = 1 since o b s (7 ) is trivial, and hence $ * ( 7 , / ) = $ st( 7 H , / ^ ) , where f H = U vf ^ . Functions / and related by a relation of the type ( 4 .3 . 1) will be considered in several different situations below. W henever an equality is w ritten between orbital integrals of functions defined on different groups, it will be tacitly assum ed th a t they are related by the appropriate transfer. 4 . 4 . Functoriality. The transfer between /c-orbital integrals on G and stable orbital integrals on H (in the cases where it is known), defines a dual m ap from stably invariant distributions on H to invariant distributions on G. The m ap on distributions should lead to character identities between L-packets on H and L-packets on G which are associated via rj by the principle of functoriality. This program has been carried in various cases, (for example, in the real case [S i]) bu t is not yet known in general. It, as well as its global counterpart, is carried out in §13 for the cases described in §4.8 below.

4.5. The fundamental lemma. Suppose th at F is p-adic. A con­ nected reductive group G is said to be unramified over F if it is quasi-split and splits over an unram ified extension of F . In this case, the action of W f on G factors through the projection of W f onto Y ( F uri/ F ) , where jFun is the m axim al unram ified extension of F . Assume th a t G is unram ified and let T be a m axim al torus contained in a Borel subgroup B of G. Let K be a hyperspecial m axim al com pact subgroup of G. An irreducible rep­ resentation of G is called unram ified if it contains a non-zero if-invariant vector. Let E U( G ) be the set of unram ified representations.

Orbital integrals and endoscopic groups

45

A character x of T is said to be unram ified if it is trivial on th e m axim al com pact subgroup T c. Let IIM(T ) be th e set of unram ified characters of T. The space of If-fixed vectors in the principal series representation 7g (x ) is one-dim ensional, by virtue of the Iwasawa decom position G = B K , and i d x ) contains a unique irreducible unram ified constituent 7rx . T w o rep­ resentations 7rx and 7xx> are equivalent if and only if x x ' lie m the same 12(T , G )-orbit and every element of E U( G ) is of the form 7rx for some X € I P (T ) . Fix an element wp € W p whose projection to T ( F nn/ F ) is the Frobenius elem ent. Recall ([B o ]) th a t there is a canonical bijection between IIu(T )/ f i(T , G ) (a n d hence E U(G)) and the set of semisimple G- conjugacy classes in L G of the form {g x w f }- T he conjugacy class {(w) x w where i/> factors through the projection of W p onto T ( F un/ F ) . Let T h be a m axim al torus contained in a Borel subgroup of H . Assum e th a t 77 is unram ified. We obtain a m ap II u (T h )/Q(T, H ) —► I P ( T g ) / ^ ( T , G) corresponding to the m ap sending {t x w p } to {r](t x w f)} , and hence a m ap ( 4.5 . 1)

E U(H) -> E U(G).

Suppose th a t lo is an unram ified character of a torus Z contained in the center of G and let IIU(T , to) be the subgroup of IIU(T ) consisting of characters whose restriction to Z is w. T hen IIU(T , w) has the stru ctu re of an algebraic variety isom orphic to ( C * ) r where r is the split rank of Z \ G over F . The Satake transform of a function f G TC(G,w) is the function / A(x ) on IIU(T , w) defined by / A(x ) — Tr(7rx ( / ) ) . By the result of Satake ([B o ]), / A(x ) is an ^ ( ^ , G )-invariant polynom ial function on IIU(T , w) (i.e., a L aurent polynom ial in r variables) and the Satake transform defines an isom orphism between TC(G,w) and the algebra of fl(T , G )-invariant poly­ nom ial functions on IIu(T ,w ). T he torus Z is canonically em bedded in Z h - There is a character (i of Z such th a t x\% = x V \% if ^ x ' m aPs to 7rx under ( 4 .5. 1). T he em bedding 77 gives rise to an algebra hom om orphism

46

Chapter 4

which sends / E TC(G,u>) to the elem ent rj(f) such th a t Tr(7rx/ (i7 (/ ))) = Tr(7rx( / ) ) if 7rx/ m aps to 7rx . T he fundam ental lem m a is the assertion th a t there exists a choice of transfer factor A q / h (recall th a t A q / h only defined up to a non-zero m ultiple) such th a t ( 4 .3. 1)) holds w ith f H = r](f). It is not yet known in general. 4 . 6 . Endoscopic groups associated to unitary groups. Let G be a u n itary group in n variables. In this section we determ ine the isom orphism classes of elliptic endoscopic triples (H,s,rj) for G (here e is trivial). P R O P O S IT IO N

4 .6 . 1 . Let (H,s,rj) be an elliptic endoscopic triple for G.

Then H is isomorphic to U ( a ) x U(b ) where a and b are positive integers such that a -f b = n. The triple is determined by {a, 6} up to isomorphism. Furthermore, A( H, s,rj) has order 2 if a — b and has order 1 otherwise. Proof: T he centralizer of a semisimple element s in G L „ (C ) is isom or­ phic to GLn i(C ) x . . . x G Lnm(C ), where ( n i , . . . , n m) is a p artitio n of n. By Condition (II) of §4 .2, srj(w)s ~1 = X(w) where A belongs to ker 1( W f , Z ( G ) ). By Proposition 3 .5. 1(b ), A is a trivial cocycle. Since W e acts trivially on G, A(w) = 1 for w E WeWe identify H with the centralizer of s via rj, keeping in m ind th a t the T-action on H is not the restriction of the T-action on G. If r E T # , then r acts trivially on G and r acts on H via a d (# ) for some g E G which com m utes with s. Hence g E H and since r preserves a splitting, it acts trivially. This shows th a t the action of T on H factors through T ( E / F ) . Replacing s by rs for some r E Z ( G ) if necessary, we can assum e th a t 5

E Z(H)r .

Now a acts by an autom orphism of order 2 such th at ( Z ( H )a)0 C Z ( G ) a = { ± 1}, since (H, s,rj) is elliptic. It is easy to see th a t if ( Z { H Y ) °

is finite, then a m ust leave the blocks of H stable and m ust act on Z ( H ) by x —» x ~ l . The diagonal entries of s are therefore ±1 and, if H ^ G, th en m = 2. Suppose th a t H = GLa(C ) x GL&(C) and th a t rj(w) = xf(w) x w , for w E W f - Let w a be a fixed element of W e / f whose projection to T ( E / F ) is a)& where $ a,b = [ ^ a , ^ ] E H , and [g,h\ denotes the block diagonal element in G w ith g and h along the diagonal (j is defined in

Orbital integrals and endoscopic groups

47

§ 1.9). Hence rj(wa) = x wa for some z G Z { H ) and it is easy to see th a t up to equivalence, we can assum e th a t r j ( w = 4>a &4>-1 x wa. It follows th a t H = U (a ) x U(b). T he action of W e on H is trivial. Hence ip(w) G Z ( H ) for w G W p, and there exist characters /xi,/i2 of We such th a t V(z ) = [vi(z)\a, H2(z)\b] X Z

where \m denotes the m x m identity m atrix. We regard pi and p2 as characters of Ce- If the image of w in C e is z, then the image of waww~l is ~z and since, 4)(waww~l ) = ad(7 ](w). Then

LEMMA 4.9 .2 :

T r i i o i x W ) = T r i z n U ^ X f 11)) i f f —* f H Proof: By the character formula for principal series representations

f d g (l)H l,f)x (l)d j ■

TV (»'g(x)(/)) =

Z\M

The eigenvalues of 7 G M satisfy follows that r ( 7 ) = /i(7 i)- This gives

and

= 72~* > fr°m which it

= p (7)L>g ( t )^ (7 , /) and T r(iG(x )(/ )) =

/ D H{ ' l ) $ { l , f H) x H ~ ' ( l ) d ' l , Z\M

and the lemma follows from the character formula for H . For the case G = U(2) and H = U ( 1 ) x Z7(l), we fix an embedding of H in G and define A g / h ( 7 ) = M_ 1 (7i - 1 2 ) D g ( i ) , for 7 = ( 7 1 , 73) € H. Similarly, for the case H = 17(2) x 17(1) and C = U ( 1 ) x U( 1) x 17(1), define A/f/c(7) =

(7i - l s ) D H{ l )

for 7 = ( 7 1 , 72 , 73 ) € C. 4.9.3: Let H = 17(2) x 17(1). For all f € (7(77, wp_1), there exists f c € C(C,u) that LEMMA

(4.9.2)

A „ /C( 7 )$ K( 7 , /) = #(7, /C)

/or a// H-regular 7 m C, where k G Ol(H^/F) is the element corresponding to C. I f F is p-adic, E / F is unramified, and the characters p and to are unramified, then (4.9.2) holds with f c — £ c ( f ) tf f € 3-C(i7, co/i-1 ). The techniques of [LL] used in the case of SL(2) can be applied in a straightforward way to establish Lemma 4.9.2. We omit the details. A

Orbital integrals and endoscopic groups

57

statement analogous to Proposition 4.9.2 for the pair G = ?7(2), H = U ( l ) x ?7(1) also holds.

4.10 The twisted case. In this section, let G = U (3). We consider the orbital integral transfers needed to compare the ordinary trace formula for G with the twisted trace formula for G. For 8 £ G and v £ rD€( 8 / F ), let 6 V denote a representative of the e-conjugacy class within 0 e_ st(£) associated to v. If 8 is e-semisimple, set e(S) = e(Gse)- Assume that 7 = N ( 8 ) is semisimple and that Gse = G7. Then 3l(G 7 /F ) and T>e( 8 / F ) are paired by Proposition 3.11.2(a). For k £ IR(G7 /.F), define

E

( 4 .10 .1 )

u€V,(6/F)

where

$e(M )=

j &e(g))dg , G(«e)"\G

Recall that G(£( 6, M N(6 ))di

Z M l~‘ \M

= / Dg( t/)$(7, f)x (l)d j Z\M

(cf. [Li], §7). If 8 = d( x, y, z) € M , then t

( S ) = h ( x z ) ~ 1t ( 7 ) =

7

= d(x/z,y/y,z/x),

f i ( x z ) _1 f i(x/~z) =

1 .

and the second equality follows similarly.

4.11 Tw isted transfer for U(2). For the group G = ResE/F(U(2)), define $ £ ( 6, ^), for e-semisimple 8 as in (4.10.1). If N ( 6 ) is elliptic regular or scalar, then T>e( 8 /F) is isomorphic to F * / N E * by Proposition 3.12.1.In thissection,k, will denote the non-trivial character of T>e( 8 /F) if T>€( 8 /F) is non-trivial and the trivial character otherwise. Explicitly,

^ ^

f fi(det(tu))

if N ( 8 )

is elliptic regular

\ k(v) = /i(det(st( 7 ,/ ) coincide for / £ C ( G v, u v). We distin­ guish between them in the notation to make the notation in the global case uniform. In the case n = 3, we identify H v with G\jst(7o, f H) by the main property of the transfer (cf. §4.3). Suppose that T is of type ( 1 ) and that ft(T, G) = Z/2 (Prop. 3.7.2). Let s G f I f { T ) . The { t ( s - 1 )s } defines an element of rD(/yo/F). If {