The Theory of Information and Coding [2 ed.] 0511037147, 0521000955, 9780511037146

Citation preview

This page intentionally left blank



 




    

  
   


          
 !"#  $%& '! & ()

*& *&+ , ,   % &-




 




    

 . ) 11 12 1( 13 21 22 26 27 2) 2: 2( 23 6< 61 62 66 6. 67 6) 6: 6( 63 .< .1 .2 .6 .. .7 .) .: .( .3 7< 71 72 7. 77 7) 7: 7( 73 )< )1 )2 )6 ). )7 )) ): )( )3 :< :1 :2 :) :( (< (1 (2 (6 (. (7

/ & 0 Symmetry and separation of variables  - Permanents /  0&  / 0 * Continued fractions      0 /
% Mathematical theory of entropy    The Cauchy problem   &4 5 0&&    & &-*&& Birkhoff interpolation /  !& Graph theory 0   Field extensions and Galois theory 0   The one-dimensional heat equation

   Computation and automata  /*& 8&9 Theory of matroids   %*    &  0  &!%& Regular variation   &!*&;  '  $; Rational approximation of real functions  /*& 8&9 Combinatorial geometrics  *   =&*! Algorithmic algebraic number theory 0 -4&  0 * >& Functional equations containing several variables  5!-4   *4&"#   & Iterative functional equations  ' >4!  Factorization calculus and geometric probability  $&>&%  &   ,, Volterra integral and functional equations  $&   *  Basic hypergeometric series
 %&& Comparison of statistical experiments

 &! & Intervals methods for systems of equations  5&-*!# Exact constants in approximation theory   !   0 +& Combinatorial matrix theory  /*& 8&9 Matroid applications  # Operator algebras in dynamical systems / %& Model theory  *  ' # General orthogonal polynomials  -*&& Convex bodies     0 =>-4+# Stochastic equations in in®nite dimensions

 ?&   '&%  ! ,&  /*&   =&%& Oriented matroids
 
%   !-*& Stopping times and directed processes    Computation with ®nitely presented groups

  & Banach algebras and the general theory of  -algebras  -&!@ Handbook of categorical algebra I  -&!@ Handbook of categorical algebra II  -&!@ Handbook of categorical algebra III

 5#   &> Introduction to the modern theory of dynamical systems '  -*#; Combinatorial methods in discrete mathematics '  -*#; Probabilistic methods in discrete mathematics   * Skew Fields -* 0 & Geometric tomography &%&  #& 0  && ;&  Pade approximants 0 5?-&# Bounded arithmetic, propositional logic, and complex theory   & Geometric applications of Fourier series and spherical harmonics    In®nite dimensional optimization and control theory

  * $ Minkowski geometry   $  
  %*; Nonnegative matrices and applications 5
%& Sperner theory  ;&#;-  "    - Eigenspaces of graphs  &%&  >&&   &!@ Combinatorial species and tree-like structures      /-* Representations of the classical groups

 &*  0!%-#&   &4 Design Theory volume I 2 ed

 &-*  0 /&4& Orthonormal systems and Banach space geometry &%&
 &" -* #&+  ? + Special Functions  -- Quantum ®eld theory for mathematicians

  ;; Geometry of sporadic groups I

 &*  0!%-#&   &4 Design Theory volume II 2 ed   # Lie's Structural Approach to PDE Systems   !#   A! Orthogonal polynomials of several variables 0 +>&+ The foundations of mathematics in the theory of sets    &    &+     ;  Navier±Stokes equations and turbulence  &   &#& Geometries on Surfaces  5 #     Asymptotics and Mellin±Barnes integrals



 




    



The Theory of Information and Coding Second Edition

 0 -


California Institute of Technology

          The Pitt Building, Trumpington Street, Cambridge, United Kingdom    The Edinburgh Building, Cambridge CB2 2RU, UK 40 West 20th Street, New York, NY 10011-4211, USA 477 Williamstown Road, Port Melbourne, VIC 3207, Australia Ruiz de Alarcón 13, 28014 Madrid, Spain Dock House, The Waterfront, Cape Town 8001, South Africa http://www.cambridge.org © Cambridge University Press 2004 First published in printed format 2002 ISBN 0-511-03714-7 eBook (Adobe Reader) ISBN 0-521-00095-5 hardback

Contents

Editor's statement

page ;

Section editor's foreword

@

Preface to the ®rst edition

@

Preface to the second edition

@

Introduction >&  &

1 12 16

Part one: Information theory 1

Entropy and mutual information 11 -&&  ;>& 12 -&&  ;&- 16 -&&  ;>&  ;&- >&  &

1: 1: 66 6: .. .3

2

Discrete memoryless channels and their capacity±cost functions 21 *& -$-+B- ,!- 22 *& -*& -% *&& >&  &

7< 7< 7( )( :6

3

Discrete memoryless sources and their rate-distortion functions 61 *& & ,!- 62 *& !-& -% *&& >&  &

:7 :7 (. 31 36

;

;

Contents

4

The Gaussian channel and source .1 *& ! -*& .2 *& ! !-& >&  &

37 37 33 1&  &

112 12< 122

6

Survey of advanced topics for part one )1 !- )2 *& -*& -% *&& )6 *& !-& -% *&&

126 126 126 161

Part two: Coding theory 7

Linear codes :1 !-D *& %&&  $+-*&-# -& :2 + & &-%  q+ + &- -*& :6  % %& &+  -& $&, -& :.  % -& :7 + & &-%  %&& q+ -*& :) /&%* &! &  *& -/  && >&  &

163 163 1.6 1.) 1.( 1.3 176 17( 1)7

8

Cyclic codes (1 !- (2 *,&%& &-& , -+-- -& (6 +--  % -& (. !& -&- (7 &-% >!& -&-% -+-- -& >&  &

1): 1): 1(1 137 133 217 22< 22(

9

BCH, Reed±Solomon, and related codes 31 !- 32  -&  -+-- -& 36 &-%  -&  &D *& #&+ &H! 3.
!-I %* , $+  37 &-%  -&  "D *& %*  3) &&B  -& 3: &-% "*& &!& & $&&

26< 26< 26. 26) 2.. 2.3 276 2))

Preface 3(

*& 826129 + -& >&  &

; 2:: 2(2 232

10

Convolutional codes 1& %4&  &- &-* & - $%  &-%4>& >-* , $&&+ *& - ! >& , ;! &  &- " >& &-&&   &  && -*%&   *$& * * &&$& " #& *& - & "&+ !& "*&&   &&&  & --&>&  J&  "*-*  - >& $$& >! "*&&  *  +& $&&& >&-!& , !,J-& ,  ,  *&+   !--& +  -& $+ *& -  ! , ;&+ & &%&&% $>&   * &,   $  !-* ,K!% &&;  *& $$@  , ,!-  *& -&   *&& , $>>+    & "*&  & * - &  *& - - ,,  %& *& >- &!  * J& & >&-!& , *& & ! !$!% , &&-*   ,J-! , +& "*    $&-  %& ! *& >- &!  *& &&; & >& -
&-& * !--&&&  %;%  $&& * -*&;& * >?&-;& $&*$ *& J ,  # ianarlo ota

;

&"

  , ,    *& *& , "* "& - - !-

  & , --&    ;   !-* !$ *& $&--!$ , $*$*&   %;& &   *;% &-*%+ /& "&  *& %&! , !& * *& &-% *  %& - , $>&  &&  &-%  %  &-% ,  - >& $$-*&   +& -  -$& "+D * -- $$& , 13.( # *& >* ,  &" -*$& , *& -  *& $ *+ +& *&& * %"  %%&% &!&  * K&%% J&   & ,  & %+ &;& * >&- & $ , ! + %!%&

*& $&& %$* 8-!+ " %$*  &9   &@-&& !-  *& " $&- , - !-D -%   

*& J 8"*-*  *& !>?&- ,  "9   &&% ! , *& $"&  >&!+ , %&>L *& &- >&%  >>+ *&+ "*-* *& -*$& >&%! >+ * &-*&  ;&  !&@$&-& "+ ark 5ac General Editor &-  >>+

 
 * *& - *&+ ,  !- Bell System Tech. J. 27 813.(9 !-D 6:3B6(2L  &D -&& && +&  6(2B.&  $ . O9 *" * Pe ˆ O1 p9Q ‡ pO1 Q9 "*&&   1 2n Qˆ 2 O2 n‡19 : n 2

0.5 0.6 0.7 0.8 0.9 0.10

0.11

O>9 *" * *& $&, -& , *& %! R ˆ 2n -*& & +& &@-+ *&  & ;!& , Pe 

! % R > 1  R   *" *"  -;& *& ;%!& NN- K$$%II &%+  $ .   On k9 -&  *& && , % >+ &-& *& ,"% ;&- ! % *&+ & NN+II ;& , -& " , *& R ˆ .=:  % -&D 11&%  #& J& ->! , *& " "+  J  *& & $+ *& !>?&- %&" ! " *!& , &&-* $$&  ,  *&+ & $!>*& &-* +&

*! *  !;&+ -#"&%&  *& !H!& ,*& , ,  *&+ &+  *& > , * 13.( $$& !   *&   !;&+ -#"&%&  *&   $ $13.( ->!  *& !>?&-M &+ &;&+ & , * $$& -& NN *& - *&+ , - !-II * $;&  >&  $-&& !-& , &&-* & , &&  O &@ $&   13:6 -&- , *& #&+ $$&  *& &;&$ & , ,  *&+ F27G * " *& !*  -!* , 12 , *& .3 $$& -&  *& !* $$&& & * *&&  &9 2 O$ 19  *&  %&& , & -*& 72 O59  *& >+ + &- -*& 6 O$ 19  & & p   -& *& transition $>>+  *& crossover $>>+ , *&  . O$ 69  *& & *& binomial &; , * , ! , Pe &&;& ,!*& - coef®cient NK ONNN -*& KII9 && *& ! >& , "+ , -*% K >?&- "*! &$& ! ,  & , N >?&- ! &-+   &H!  N ON 19 ON 29 . . . ON K ‡ 19=KOK 19 . . . 2 . 1 O ,!*& $$&& , >  -&,J -& && 5!* F:G ' 19  *&& & 2k n‡1 ,,&& "+ *& -*& -! -!& &@-+ k &  % *& 2n ‡ 1  & >
-* !-* $& --! "* 2 n‡1 $>>+ pk O1 p92 n‡1 k   *& & pk O1 p92 n‡1 k  &H!  *& k $>>+ * &@-+ k , *& 2n ‡ 1 -$& , *& !-& > & &-&;&  & 7 O$ 79  &- $-&  + &   $& , + $  *   *& % ,  && && *& NN&&II  *& & $&   NN+ $ II   $+-*&-# ,!& * ,&-! -%&  !&  %&>%&

Part one ,  *&+


$+ 

1 !! , 

1.1 Discrete random variables !$$& X   -&&  ;>& *  & "*& %& R ˆ fx1  x2  . . .g  J&  -!>& & pi ˆ PfX ˆ xi g O $>> - & %+ -! $$&@ 9 *& entropy , X  &J& >+
1 HOX 9 ˆ pi % : O1:19 p i i>1

* &J && &>  *& >& , *& %*  $!$&+ &, !$&-J& , &-&+ *"&;& "& * && *& >&b &$+ >+ H b OX 9  + * *& &$+ , X  >&% &!&  >&b ! & 2 ! & -& bits O>+ %9  >&e ! & -& nats O! %9 &- , pi ˆ +  & >+L && > 119 + , R  J& *& ! O119 +  -;&%&L  * -& "& & HOX 9 ˆ ‡1 Example 1.1 & X &$&& *& !- & ,  %&  ,  , & *& R ˆ f1 2 6 . 7 )g  pi ˆ 1) , &-* i && HOX 9 ˆ % ) ˆ 2:7( > ˆ 1::3  h Example 1.2 & R ˆ f+ PfX ˆ + &$+ ,!- H 2 O p9 "*-* " &J& 
H O H  *&& %*+ ,,&& "+D HOX 9  *& &$+ , *&  ;>& X L HO p9 ˆ p % p O1 p9 %O1 p9 , < < p < 1L  HO p1  p2  . . .  pr 9 ˆ  pi % pi 1 , p   $>>+ ;&-9 h 1 2 1 Example 1.3 , *& !  && >+ A  , *&  nˆ2 On % n9 ;>& X  &J& >+ PfX ˆ ng ˆ OAn %2 n9 1 , n ˆ 2 6 . . .  *& HOX 9 ˆ ‡1 O&& > 129 h  ! ! * HOX 9 - >& *!%* ,   >! X D

&!& , *& ,"% *%

O9 *&  ! , NN, II $;& >+  >&; , X  O>9 ! NN!-&+II >! X  O-9 *& NN &II , X   *& &@ ,&" $%$* "& " -! *&& $$&& , + >! *& && *! >&   &&+ * HOX 9 &  ,- &!& *&& *%   &&$ *& - &&  "& && *&& & + $>& ,!- ,   ;>& X * *& *& $$&&  >& -!& >&" >! + HOX 9 "  , *& !+ , - !- $>&   &-* x 2 R &J& IOx9 ˆ % PfX ˆ xg *& I   &"  ;>&  HOX 9   ;&%& *& ,!- IOx9 O&& % 129 - >& &$&&  *&  ! , ,  $;& >+ *& &;& fX ˆ xg

--%  * &$& *& & $>>&  &;&  *& & ,  "& &-&;& "*&  --! -& &;& O& * --! "* $>>+ 19 $;&  ,  "*&&  !#&+ &;& $;&  ;&+ %&  ! , ,   &@ $& !$$& +! ;&  -&

1.1 Discrete random variables

13

Figure 1.2 *& ,!- IOx9

"* -! "& + NN+&  II H!& , +! #& NN/ ;&  >& 127PII  %  NNII "& +! "! *;& %& ;&+ & ,  -& !-* &@& & %&;+  &@-&&%+  $>>& ;&&+ , +! %  NN+&II +! "! *;& && !-* , "  , $&$& ;& *& -&  #& *&  & H!&  "! %&  NNII  ,&" "! %&  NN+&II  *& ;&%&  ! , ,  $;& "! >& HO p9 "*&& p ˆ P f%&  &* > 127g &;& ?! >&,& &-&;% *& -&I &$+ +! "! $>>+ >& %*+ @!L * &K&- *& ,- *     ! , !-&+ &@ >! *& "& HO p9  &H!+  &!& , * !-&+1 + ,  $& -&! "#& "&& %&  &- *& -&I "& *& "! >&- & &@& &+ >&  %* >&%  !$&- *& -& , >&%  -*& * "+ + NNII * &K&- *& ,- * *&  ;>& X &$&&% *& -&I &$+   ;&+   && HO p9 &!& *&  & , X 

  & -&& &@ $& &J& X >+ PfX ˆ &; , *& NN>II X $;& & NN>II , ,  ! J *&& --& *& @ ! $>& ;!& , HOX 9  &  , *& 4& , R Theorem 1.1 Let X assume values in R ˆ fx1  x2  . . .  xr g Then < < HOX 9 < % r Furthermore HOX 9 ˆ < iff pi ˆ 1 for some i and HOX 9 ˆ % r iff pi ˆ 1=r for all i Proof -& &-* pi  < 1 &-* & pi % p i 1  O119  > !&  ;>&  *&  NN II # ,  ;>&  +  & * *& @ ! ;!& , *& ,!- HO p1  p2  . . .  pr 9  p ˆ O p1  . . .  pr 9  %& ;& *& r 1  &  $&@ f pi > & X  Y  H!+ HOX jY 9 -& *& conditional entropy 2 , X  %;& Y   &   * &+ "& !-&  & & &   x  *& %& , X  y  *& %& , Y  &J&D pOx9 ˆ PfX ˆ xg pO y9 ˆ PfY ˆ yg pOx y9 ˆ PfX ˆ x Y ˆ yg

O1:29

pOxj y9 ˆ PfX ˆ xjY ˆ yg ˆ pOx y9= pO y9 pO yjx9 ˆ PfY ˆ yjX ˆ xg ˆ pOx y9= pOx9: O *   --+  >%!!  , >!&+ &-&+ $$ $& !>-$ " >& & , &@ $& pX Ox9 pY j X O y x9 * && " & *"&;& + "*& -! ! >& & !>!& , *& && x yL &&
@ $& 1)9 ! &J D   1 HOX jY 9 ˆ E % pOxj y9 ˆ


x y

pOx y9 %

1 : pOxj y9

O1:69

O  O169 "& >&;& *&  & -;&  "&  , ! O119D < % < 1 ˆ &*;  %;&& >+  r 3 s @ , transition probabilities O pO yjx99 *& ! >& pO yjx9 &$&& *& $>>+ * y " >& *& !$! %;& * x  *& $! &+ *& ! >& pO yjx9 ! ,+


pO yjx9 >
& pO yjx9 Example 1.4 O*& >+ + &- -*& &+ -!&  *&  !-9 && r ˆ s ˆ 2  *& %$* # #& *D

22

Entropy and mutual information

Example 1.5 O *& >+ &!& -*&9 && r ˆ 2 s ˆ 6 *& $! & >&& NN&& NN& !-* & #&+ * < ! ?  1 ! ?  *&

1.1 Discrete random variables ! $ PfY ˆ 1jX ˆ & &>& h

!$$& " * *& $!     & &&-& --%   $>>+ >! pOx9  f+ PfX ˆ xg ˆ pOx9

x 2 f& Y "*-* " &$&& *& output , *& -*& *& ? >! , X  Y  %;& >+ pOx y9 ˆ PfX ˆ x Y ˆ yg ˆ PfX ˆ xgPfY ˆ yjX ˆ xg ˆ pOx9 pO yjx9  *&

% >! , Y  pO y9 ˆ PfY ˆ yg
ˆ PfY ˆ yjX ˆ xgPfX ˆ xg x

ˆ




pO yjx9 pOx9:

x

 +

pOxj y9 ˆ pOx y9= pO y9
ˆ pO yjx9 pOx9= pO yjx99 pOx99: x9

&-& -&$%  &;&+    $! >! *&&   $ ,  ;>&D X  *& NN$!II   Y *& NN!$!II ,  *& -*& ;&&+ %;& + $ OX  Y 9 , -&&  ;>&  *&& &@     $! >! !-* * X  *& $!  Y  *& !$!D  $+ &J& *& -*&I  $>>& >+ pO yjx9 ˆ PfY ˆ yjX ˆ xg  *& " %;& + && $ OX  Y 9 ,  ;>&   $>&  *# , Y   NN+II ;& , X  *   *& &! ,  % X *!%*  -&   Example 1.6 & X ! & *& ;!& 1 2 &-* "* $>>+ 1.  & Y ˆ X 2  *& -&$%   # #& *D

2.

Entropy and mutual information

 * &@ $& X  Y & !-&&  +&   -& * Y $;&  -&>&  ! , NN, II >! X O&& > 11&;& * Y ˆ y *& -& *& ! >& pOxj y9 ˆ PfX ˆ xjY ˆ yg , J@& y &$&& *& - >! , X  %;& * Y ˆ y "& &J& *& conditional entropy , X  %;& Y ˆ yD
1 HOX jY ˆ y9 ˆ : pOxj y9 % pOxj y9 x

* H!+  &,   ;>& &J&  *& %& , Y L & ! &J& *& conditional entropy HOX jY 9   &@$&-D
HOX jY 9 ˆ pO y9 HOX jY ˆ y9 y

ˆ


y

ˆ


x y

pO y9




pOxj y9 %

x

pOx y9 %

1 pOxj y9

1  pOxj y9

 %&& & "*
H O169 *! ,  %;& $ X  Y ,  ;>& HOX jY 9 represents the amount of uncertainty remaining about X after Y has been observed Example 1.7 & *& ,"%   "*-*   $-! -& , *& >+ &!& -*& ,
@ $& 17D

1.1 Discrete random variables

27

&& pX O&; , Y &!-& ! !-&+ >! X  h /& " $&&  &-*- &

  HOX jY 9 * " >& !&,! &

Theorem 1.2 Let X  Y  Z be discrete random variables. Using obvious  notation Osee Eqs. O1299 de®ne, for each z AOz9 ˆ x y pO y9 pOzjx y9 Then HOX jY 9 < HO Z9 ‡ EO% A9: Proof

 HOX jY 9 ˆ E % ˆ




1 pOxj y9

pOx y z9 %

x yz

ˆ


z



pOz9

1 : pOxj y9


pOx y z9 1 % : pOz9 pOxj y9 x y

2)

Entropy and mutual information

 J@& z pOx y z9= pOz9 ˆ pOx yjz9   $>>+ >!   "& - $$+ 0&&I &H!+  *& & !  *& &!   
1
pOx y z9 HOX jY 9 < : pOz9 % pOz9 x y pOxj y9 z ˆ


z

pOz9 %



pOx y z9 1 ‡ : pOz9 % pOz9 pOxj y9 z x y

! pOx y z9= pOxj y9 ˆ pOx y z9 pO y9= pOx y9 ˆ pO y9 pOzjx y9:

h

Corollary ONNFano's inequalityII9 Let X and Y be random variables, each taking values in the set fx1  x2  . . .  xr g Let Pe ˆ PfX 6ˆ Y g Then HOX jY 9 < HOPe 9 ‡ Pe %Or

19:

Proof  *&& 12 &J& Z ˆ < , X ˆ Y  Z ˆ 1 , X 6ˆ Y  *& AO& ,&!&L && > 111G I &H!+ *  &&% *&!- &$& *# , HOX jY 9  *&  ! , ,  &&&  && & X -& Y  #" & "+  && & X   J && & "*&*&   X ˆ Y L , X ˆ Y  "& & & , *"&;& X 6ˆ Y  *&& & r 1 & % $>& , X && % "*&*&   X ˆ Y  &H!;&  && % *&  ;>& Z &J&  *& $,L -& HO Z9 ˆ HOPe 9  #& HOPe 9 >   * , X 6ˆ Y O* *$$& "* $>>+ Pe 9 *&  ! , ,  &&&  J ! "*-* , *& & % r 1 ;!& X *  >+ *&& 11   %Or 19 Example 1.8 /& $$+ I &H!+  *& -*& ,
@ $& 1: && rˆ PfX ˆ Y g ˆ 26 Pe ˆ 16 I >!  *! HOX jY 9 < 1 6  1 H 6 ‡ 6 % 2 ˆ 16 % 6 ‡ 26 % 62 ‡ 16 % 2 ˆ % 6 16 % 2 ˆ 1:272< > O &@ $& "*&& I &H!+ & >&& && >1119 h " -& HOX 9 &$&& ! !-&+ >! X >&,& "& #" Y  HOX jY 9 &$&& ! !-&+ ,& *& ,,&&-& HOX 9 HOX jY 9 ! &$&& *&  ! , ,  $;& >! X >+ Y *  $ H!+  -& *& mutual information >&"&& X  Y   && >+ IOX L Y 9D

1.1 Discrete random variables IOX L Y 9 ˆ HOX 9

HOX jY 9:

2: O1:.9

O 
@ $& 1: I 2 OX L Y 9 ˆ & &*& $;&  &%;& O&% 
@ $& 1: IO& & O *& ;&%&9 >+ >&;% *& !$! , *& -*& Theorem 1.3 For any discrete random variables X and Y IOX L Y 9 > && , &@ $& * *& -;-& ;OX L Y 9 , &@ $& &-
@ $& 1) *&&   &+ ;&J& ;OX L Y 9 ˆ < >! I 2 OX L Y 9 ˆ 1 >9 %
H O1.9BO1:9   $>&  $;&  &&+ &;&  $  ,- >! !! , D IOX L Y 9 ˆ IOY L X 9 IOX L Y 9 ˆ HOY 9

O1:(9 HOY jX 9

IOX L Y 9 ˆ HOX 9 ‡ HOY 9

O1:39 HOX  Y 9

O1:1+ HOX  Y 9 ˆ




pOx y9 %

x y

1 : pOx y9

O1:119

*& $, , *&& &*$ & &,  > 11. *&+ - >& &+ & & >&& >+ & , *& '& % *"  % 17    ,!,! &@&-&  %;& ,  &$& , &-* , *& &  $& >+ % 17  &@ $&
H O1(9 &@$&& *& NN !!+II , !! ,  L HOX  Y 9 ˆ HOX 9 ‡ HOY jX 9 >&- & NN! !-&+ >! X  Y  *& ! , ! !-&+ >! X  ! !-&+ >! Y  -& X  #"II    " , "& & %;& *&&  ;>& X  Y  Z "& &J& *& !! ,  IOX  Y L Z9 ONN*&  ! , ,  X  Y $;& >! ZII9 %!+ "*
H O1:9 >+

Figure 1.5

& - '& %

,
H O1.9  O1(9BO11+ *&  $>>& pO yjx9    2 >+ *&  $>>& pOzj y9 ˆ pOzjx y9 /& *;& &+ >&;& * %;& + $ ,  ;>& OX  Y 9   $>&  &;&    "* X  *& $!  Y  *& !$! "&;&    !& * ,

6
& *&& &@  $ ,  I !-* * X  Y  Z *;& *& &*$ , % 1) &&   -& *  &-&+  !,J-& - , *  * OX  Y  Z9 ,   #; -* *  pOzj y9 ˆ pOzjx y9 O& Z &$&  X + *!%* Y 9 " &I ! & * OX  Y  Z9   #; -*   % 1) *& >+

*&& 1. IOX L Z9 < IOX  Y L Z9  -& OX  Y  Z9   #; -* IOX  Y L Z9 ˆ IOY L Z9 &-& IOX L Z9 < IOY L Z9 " , OX  Y  Z9   #; -*   O Z Y  X 9 O&& > 1179  *&-& IOX L Z9 < IOX L Y 9 -& *   &@& &+  $ , *&&- $$ &+ , #; -* "& $+    *&&  Theorem 1.5 If OX  Y  Z9 is a Markov chain, then  IOX L Y 9 IOX L Z9 < IOY L Z9:

h

&,&% %  % 1) "& J *  I &  NN&#II ,  , *&  I & && - O& , Y   &J& ,!- , X  Z  &J& ,!- , Y 9 "& - *# , *& -&  % 1)   # , $-&% -J%! @-+ *&& 17 + *  $-&% - + &+ , M O   $ %&&4 , * &&
H O11799 Example 1.9 & X 1  X 2  X 6 >& &$&&  ;>&L *& OX 1  X 1 ‡ X 2  X 1 ‡ X 2 ‡ X 6 9   #; -*   IOX 1 L X 1 ‡ X 2 ‡ X 6 9 < IOX 1 L X 1 ‡ X 2 9 O&& > 11)  1639 h Example 1.10  % 1) ! & * X  &->& >+ PfX ˆ *  I & >+ + &- -*& "* & $>>+ p *& IOX L Y 9 ˆ 1

H 2 O p9

IOX L Z9 ˆ 1

H 2 F2 pO1

> p9G

>:

*&& " ,!- & $&  ,"D O  &@& , *

Figure 1.6  ,  *&I ;&" ,  #; -*

1.1 Discrete random variables

61

&@ $& && > 11(  12! *& -;&@+ , IOX L Y 9 "*&   ;&"&   ,!- &*& , *& $! $>>& pOx9  , *&  $>>& pO yjx9 Theorem 1.6 IOX L Y 9 is a convex \ function of the input probabilities pOx9 Proof /& *# , *&  $>>& pO yjx9  >&% J@&  -& " $!  ;>& X 1  X 2 "* $>>+ >! p1 Ox9  p2 Ox9 , X I $>>+ >!   -;&@ - > pOx9 ˆ á p1 Ox9 ‡ â p2 Ox9 "& ! *" * áIOX 1 L Y1 9 ‡ âIOX 2 L Y2 9 < IOX L Y 9 "*&& Y1  Y2  Y & *& -*& !$! -&$%  X 1  X 2   X &$&-;&+   * -& *& ,"% $! "*-* !& >;!  **D áIOX 1 L Y1 9 ‡ âIOX 2 L Y2 9 IOX L Y 9
pO yjx9
pO yjx9 ˆ á p1 Ox y9 % â p2 Ox y9 % ‡ p1 O y9 p2 O y9 x y x y

O&&
H:O1::99


pO yjx9 Fá p1 Ox y9 ‡ â p2 Ox y9G % pO y9 x y ˆá


x y

p1 Ox y9 %


pO y9 pO y9 p2 Ox y9 % ‡â : p1 O y9 p2 O y9 x y

O1:129

/& " $$+ 0&&I &H!+  &-* , *& >;& !   &@ $&

pO y9 pO y9 p1 Ox y9 % p1 Ox y9 < % : p p O y9 1 1 O y9 x y x y

62

Entropy and mutual information

!


x y

p1 Ox y9


pO y9
pO y9 p1 Ox y9 ˆ p1 O y9 p1 O y9 x y ˆ


pO y9 . p1 O y9 p1 O y9 y

ˆ 1: &-& *& J !

 O1129  < &   ;>& >!& --%  PfX ˆ ig ˆ pi 

*& IOX L X 9 ˆ HOX 9 ˆ HO p1  p2  . . .  pr 9 *& &! " ," ,

*&& 1) h Theorem 1.7 IOX L Y 9 is convex [ in the transition probabilities pO yjx9 Proof && *& $! $>>& pOx9 & J@& >! "& & %;& " & ,  $>>& p1 O yjx9  p2 O yjx9   -;&@ - > pO yjx9 ˆ á p1 O yjx9 ‡ â p2 O yjx9   &H!&  *" * IOX L Y 9 < áIOX L Y1 9 ‡ âIOX L Y2 9

O1:169

"*&& Y  Y1  Y2 & *& -*& !$! -&$%  *&  $>>& pO yjx9 p1 O yjx9  p2 O yjx9 % !% >;!  *& ,,&&-& >&"&& *& &,  %* & , O1169  O&&
H O1799
pOxj y9 Fá p1 Ox y9 ‡ â p2 Ox y9G % pOx9 x y
x y

ˆá


x y

*& J !

á p1 Ox y9 % p1 Ox y9 %

p1 Oxj y9 pOx9


x y

â p2 Ox y9 %

p2 Oxj y9 pOx9


pOxj y9 pOxj y9 p2 Ox y9 % ‡â : O1:1.9 p1 Oxj y9 p 2 Oxj y9 x y

 O11.9  >+ 0&&I &H!+

1.2 Discrete random vectors  
pOxj y9 < á % p1 Ox y9 p1 Oxj y9 x y ˆ á %


x y

ˆ á %


y

 + *& &- !

66

 pOxj y9 p1 O y9

p1 O y9 ˆ &"&&  $ ,  ;>&   *  ;>&  * &- "& " %&& 4& *& &J  &J& HOX9 HOXjY9  IOXL Y9 "*&& X  Y & >+  ;&- ! $ , ;&"  *   ;&- X ˆ OX 1  X 2  . . .  X n 9  ?!  J&  ,  ;>& X i  *& >! , X O*& ? >! , X 1  X 2  . . .  X n 9  *& ,!- pOx1  x2  . . .  xn 9 ˆ PfX 1 ˆ x1  X 2 ˆ x2  . . .  X n ˆ xn g "*&& &-* xi   *& %& , X i  %-&  *& &J  &- 11 *! -;-& *& && * HOX 9 HOX jY 9 IOX L Y 9 &$& +  *& >! ,!- pOx9 pO yjx9 &-    + "+  *& ,- * *& ;!& ! & >+ X  Y & & ! >& &-& "& -  &&+ &@& *&& &J  >+  ;&-L , &@ $& *& &$+ , X ˆ OX 1  . . .  X n 9  &J& 
1 HOX9 ˆ pOx9 %  pOx9 x "*&& *& !   &@&& ;&  ;&- x  *& %& , X  >;!+ *&&  11B1: &  !&

*& %&&4 , *&& 17  >+  ;&- *  $-!+  $ $$- "*-* "& " -! & *& & ,  - !- +& *"  % 1: O-, % !  - !- +& 9 $$+% *&& 17  *& !> #; -* OU X V9 "& %& IOUL V9 < IOXL V9  + IOXL V9 < IOXL Y9 &-& , *&  ;>& , % 1: IOUL V9 < IOXL Y9:

O1:179

* &!  -& *& data-processing theorem & >!+  + * *& ,  $-&% O*& "# & >+ *& &-&  &-& , % 1:9 - + &+ , M  + , &@ $& * *& + -*& !$! Y  % 1: - & ,  >! *& !-& &H!&-& U * & *& &-&I & & V O/*& *  !& *&&-+ *&  $-&% , *& &-&  &;&*&& &H!&  && * ,  !>&9 /& " - &   $ , &H!& ;;% IOXL Y9  n iˆ1 IOX i L Yi 9 "*&& X ˆ OX 1  X 2  . . .  X n 9  Y ˆ OY1  Y2  . . .  Yn 9 &  $ , n &  ;&-

Theorem 1.8 If the components OX 1  X 2  . . .  X n 9 of X are independent, then IOXL Y9 >

n


IOX i L Yi 9:

iˆ1

Figure 1.7 %&& -

!- +& 

1.2 Discrete random vectors

67

Proof &% E && &@$&-  *& ?  $& $-& , X  Y "& *;&   pOxjy9 IOXL Y9 ˆ E % Osee Eq: O1:799 pOx9   pOxjy9 ˆ E %  pOx1 9 pOx2 9 . . . pOxn 9 -& X 1  X 2  . . .  X n & ! & &$&&  *& *& *   n n

pOxi j yi 9 IOX i  Yi 9 ˆ E % pOxi 9 iˆ1 iˆ1 

 pOx1 j y1 9 . . . pOxn j yn 9 ˆ E % : pOx1 9 . . . pOxn 9

&-& n


IOX i  Yi 9

IOXL Y9

iˆ1



 pOx1 j y1 9 . . . pOxn j yn 9 ˆ E % pOxjy9   pOx1 j y1 9 . . . pOxn j yn 9 < % E ˆ< pOxjy9

>+ 0&&I &H!+ -& *  &@$&- 

pOx y9f


g ˆ pOx1 j y1 9 . . . pOxn j yn 9 pOy9 xy

xy

ˆ 1:

h

Example 1.11 & X 1  X 2  . . .  X n >& &$&& &-+ >!&  ;>& "* -  &$+ H  & ð >&  $& ! , *& & f1 2 . . .  ng  & Yi ˆ X ðOi9  *& IOXL Y9 ˆ nH >!  IOX i L Yi 9 ˆ kH "*&& k  *& ! >& , J@& $ , ð *  *& ! >& , &%& i "* ðOi9 ˆ i  $-! , ð *  J@& $ ,  &@ $& , ðOi9  i ‡ 1 O  n9 *& IOX i L Yi 9 ˆ < O&& > 1269 h , "& *# , OY1  Y2  . . .  Yn 9  *& n !$! ,  + -*& "*& *& $! & X 1  X 2  . . .  X n  *&& 1( & ! * , *& $! & &$&& Y $;& & ,  >! X * *&   ! ,

6)

Entropy and mutual information

,  $;& >! &-* X i >+ *& -&$% Yi  *& &@ *&& " & ! * , "& $ *& ! $ , &$&&-& >! *& X i  ! & & * *& OX Y9 -*&  & +& *  pO y1  . . .  yn jx1  . . .  xn 9 ˆ

n 

pO yi jxi 9

O1:1)9

iˆ1

*& !  H!& ,,&&M Theorem 1.9 If X ˆ OX 1  . . .  X n 9 and Y ˆ OY1  . . .  Yn 9 are random vectors and the channel is memoryless, that is, if O11)9 holds, then IOXL Y9
 1269 h Corollary If X ˆ OX 1  X 2  . . .  X n 9 then HOX9
&  ;&- "*-* - ! &  !-!>& ! >& , ;!& - $&&+ %! & & , * !>?&- $;&  >& ;&+ ,J-!   "& * -& !&;& "*  ,&" && &+ ,-  &,& *& &&& && &&"*&&7 , &

 "& * && *& &$+ HOX 9 - >& &J& ,  -&&  ;>& >!  "+ ! !  >& J&M "&;& *& &J , *& !! ,  >&"&&  $ X  Y ,  ;>& O ;&-9 - >& %&&4&  *& -&& -&   !-* & &&%  !&,! "+ *& #&+  * %&&4  *&  ,  discrete quantization ,   ;>& X  , X    ;>& "* >! ,!- FOx9 ˆ PfX < xg  , fSi  i ˆ 1 2 . . .g ˆ P   $) , *& & & R   J&  -!>& ! >& , &>&%!& &!>& !>& *& quantization of X >+

6(

Entropy and mutual information

P && >+ FX G p  >+

&&+ >+ FX G  *& -&& 

;>& &J&

PfFX G ˆ ig ˆ PfX 2 Si g  ˆ dFOx9: Si

, X  Y &  $ ,  ;>& "& &J& *& IOX L Y 9  IOX L Y 9 ˆ !$ IOFX G P L FY G Q 9 PQ

!! ,  O1:1:9

"*&& *& NN!$II  #& ;&  $ , $ P  Q , *& & &  + , X ˆ OX 1  . . .  X r 9  Y ˆ OY1  . . .  Ys 9 &  $ ,  ;&- IOXL Y9 ˆ !$ IOFXGL FYG9 O1:1:99 "*&& *& !$& !  #& ;&  $ FXG ˆ OFX 1 G . . .  FX r G9 FYG ˆ OFY1  . . .  FYs G9 , *& - $&  ;>&

$ P1    >&  &J& & , P2 , &;&+ &  P1   !>& ,  & &  P2  -& * , P1   &J& & , P2  FX G P 2   && - ,!- , FX G P 1    >+ *&& 17 O && > 1229 IOFX G P 2 L Y 9 < IOFX G P 1 L Y 9 , +  ;>& Y  * >&; *" * *& NN!$II  &J O11:9   ,-  # ,    *& $ P  Q >&- & -&%+ J& &&   &+  *" * , P1  P2  Q1  Q2 & $  FX G1 FX G2  FY G1  FY G2 & *& -&$% -&&  ;>& *&& &@ $ P  Q -&$%   ;>& FX G  FY G !-* * IOFX GL FY G9 > IOFX G i L FY G i 9

i ˆ 1 2:

O1:1(9

" , X  Y & &+ -&& "* %& fx1  x2  . . .g  f y1  y2  . . .g >+ &&-%  X $  *  & - & * & xi    Y $  *  & - & * & yj  -&+ IOFX GL FY G9 ˆ IOX L Y 9
H!+ -&+  &J& & -! -&& IOFX GL FY G9   &J O11:9 &!-&  ! && &J &J O11:9 , IOX  Y 9  ,+  $& >!    -;&& , -!+ - $!% IOX  Y 9 ,  %;& $ ,  ;>& &+ "& "! #&  *;&  , ! , IOX  Y 9 %! 
H O179 O1)9  O1:9 ;;% &% *& * !  >! !,!&+  %&&  !-* , ! &@: "&;& , X  Y & !,J-&+  * !-* , !  &@  "& * " && & ! ! & * X  Y *;&  -!! ? &+ & * *&&

1.3 Nondiscrete random variables and vectors

63

&@  -!! &%;& ,!- pOx y9 &J& , $ Ox y9 , & ! >& !-* * , A  B & &;  *& & &   PfX 2 A Y 2 Bg ˆ pOx y9 dx dy: B A

 * ! *& ;! && , X  Y & %;& >+ 1 1 pOx9 ˆ pOx y9 dy qO y9 ˆ pOx y9 dx 1

1

 *& - && >+ pOx y9 pOxj y9 ˆ  qO y9

pO yjx9 ˆ

& ! " ! & * *& &% 1 1 hOX 9 ˆ dx pOx9 % pOx9 1 1 1 hOX jY 9 ˆ pOx y9 % >* &@

1

1

pOx y9 : pOx9

O1:139 1 dx dy pOxj y9

O1:2+  &


, x 1 , x< , x1 ,


>&  -!>& & , $ !-* * Äxi ˆ xi x 1 J& å1 , Äxi , å2 ,  i  + -*& O yj 9 !-* * å1 , Ä yj , å2  & FX G && *& H!4  , X >& , *& $ , *& & & -% , *& *, $& &; Fxi 1  xi 9  & FY G && *& H!4 , Y -&$ %  *& $ -% , *& &; F y j 1  yj 9 /& " !-& *& -& D  xi pOi9 ˆ PfFX G ˆ ig ˆ pOx9 dx qO j9 ˆ PfFY G ˆ jg ˆ

xi

1

yj

1

 yj

qO y9 dy

pOi j9 ˆ PfFX G ˆ i FY G ˆ jg ˆ

 yj  xi yj

1

xi

pOx y9 dx dy 1

pOij j9 ˆ PfFX G ˆ ijFY G ˆ jg ˆ pOi j9=qO j9:

.
& -!! >+ *& ;!& *&& , &%(

&

pOi9 ˆ Äxi pOsi 9 qO j9 ˆ Ä yj qOtj 9

O1:219

,  & si 2 Fxi 1  xi G tj 2 F y j 1  yj G  + >+ ;#%  " &  &;!& *&& ( "& *;& pOi j9 ˆ

 yj  xi yj

1

xi

pOxj y9qO y9 dx dy 1

ˆ pOsij jtij 9

 yj  xi yj

1

xi

qO y9 dx dy 1

ˆ pOsij jtij 9Äxi Ä yj qOtj 9

O1:229

,  & $ Osij  tij 9  Fxi 1  xi G 3 F y j 1  yj G &-& pOij j9 ˆ Äxi pOsij jtij 9:

O1:269

" >+
H O1.9 IOFX GL FY G9 ˆ H OFX G9 HOFX GjFY G9 ˆ  1 i j pOi j9 % pOij j9  + O1219 "& *;& HOFX G9 ˆ




Äxi pOsi 9 %

i



i pOi9 % pOi9


1 1 Äxi pOsi 9 % ‡ : pOsi 9 ÄOx i9 i

1

O1:2.9

 O12.9 *& J !   $$@   *& &% , hOX 9 O&& O11399   ,   &!%* å2  " -;&%& *& &- ! -;&%& O  ! & * % å1 1 9 >&-!& Äxi . å1   + HOFX GjFY G9 ˆ




Äxi Ä yj pOsij jtij 9qOtj 9 %

i j

‡


i

Äxi %

1 pOsij jtij 9

1
Ä yj pOsij jtij 9qOtj 9: Äxi j

O1:279

*& J !  O1279  *& &% ,  &$,!- $$@   pOx y9 % pOxj y9 1    , å2   &!%*  " $$@ & hOX jY 9

*& &- !  &-  *& &- !  O12.9 -& >+ O1229

1.3 Nondiscrete random variables and vectors  
1
yj xi Ä yj pOsij jtij 9qOtj 9 ˆ pOx y9 dx dy Äxi j y j 1 xi 1 j ˆ

.1

1 . Äxi pOsi 9: Äxi

*! "& J+ > IOFX GL FY G9 ˆ




Äxi pOsi 9 %

i


i j

1 pOsi 9

Äxi Ä yj pOsij jtij 9qOtj 9 %

1 : pOsij jtij 9

O1:2)9

!  "& *;& &+ >&;& *& J !  O12)9   $$@   hOx9  *& &-  hOX jY 9  >+ #% *& X  Y $ -&%+ J& &  å2 !  1669  * & hOX 9  !!+ -& *& differential entropy , X   %!*  , *& +  absolute entropy

  *& -&& -& "& - &+ &@& *& &J , IOX L Y 9  hOX 9   ;&-  &J& IOXL Y9  %&& ?! #& -&& H!4 , &-* - $& &$&+  hOX9 ! & *& &@&-& ,   n & -!! &+ pOx9  &J& hOX9 ˆ pOx9 % pOx9 dx Example 1.13 & X ˆ OX 1  X 2  . . .  X n 9 "*&& *& X i & &$&&

.2

Entropy and mutual information

! O 9  ;>& "* *& &+ , X  *& ,!- gOx9 ˆ

n  iˆ1

O2ðó 2i 9

1=2

& ì i  ;-& ó 2i  *& 

&@$

Oxi

ì i 92

2ó 2i

*& ,,&& &$+ , X  &+ - $!&D  1 hOX9 ˆ gOx9 % dx gOx9   n 
Oxi 2 1=2 ˆ gOx9 % O2ðó i 9 &@$ iˆ1

ˆ

n 
iˆ1

O"*&& gi Oxi 9 ˆ O1= ˆ O-&



 g i Oxi 9

1 2

%O2ðó 2i 9 ‡

 2ðó 2i 9 &@$F Oxi

n
iˆ1

2 1 2F %O2ðó i 9

Oxi

 :

ì i 92

O1:2:9



2ó 2i

ì i 92

dx



2ó 2i

dxi 

ì i 92 =2ó 2i G9

‡ 1G

 gi Oxi 9 dxi ˆ 1  g i Oxi 9Oxi ì i 92 dxi ˆ ó 2i 9 n ˆ % 2ðeOó 21 . . . ó 2n 91= n : 2

 *& $&- -& n ˆ 1 "& *;& hOX 9 ˆ 12 % 2ðeó 2 :

h

   &&%   $ ,- *  %  n &  ;>& "* %;& ;-& &$&& !  ;>& *;& *& %& ,,&& &$+ Theorem 1.11 If X ˆ OX 1  X 2  . . .  X n 9 has density pOx9 and if EFOX i ì i 92 G ˆ ó 2i  i ˆ 1 2 . . .  n then hOX9 < On=29 % 2ðeOó 21 ó 22 . . . ó 2n 91= n  with equality iff pOx9 ˆ gOx9 Osee De®nition O12:99 almost everywhere  *& % &+ pi Ox9 , X i J& pi Ox9 dx ˆ 1 Proof + *+$*& pi Ox9Ox ì i 92 dx ˆ ó 2i  &-&  ," , *& - $! ,
@ $& 116 *

1.3 Nondiscrete random variables and vectors  1 n pOx9 % dx ˆ % 2ðeOó 21 . . . ó 2n 91= n : gOx9 2

.6

*! , Y &&  n &  ;&- >!& --%  *& ! &+ gOx9  gOx9 hOX9 hOY9 ˆ pOx9 % dx: pOx9 + 0&&I &H!+ *"&;& *& & &%   < % gOx9 dx ˆ  16.  1679 h ;% &!& >&K+ *!%* *& &&% $- , ,,&& &$+ "& &!  !  --& !! ,  !    *" "*  & &!& , -&>+ * *&&  16 1. 17 1(  13 &  !&  *& %&& -& *&&  16 1(  13 $&&  &! ,J-!&L *&& &  & & #  *& $, Theorem 1.3. *& ,- * IOXL Y9 > < ,"  &&+ , *& ,- * IOFXGL FYG9 > < , + -&& H!4 FXG FYG &;& , FXG  FYG & &$&&  & FXG  FYG   IOFXGL FYG9 ˆ < ,  H!4L *&-& IOXL Y9 ˆ < >+
H O11:9 + , X  Y & &$&&   $>&  J H!4 FXG  FYG "*-* &  &$&& O&& > 16)9   IOXL Y9 > IOFXGL FYG9 . &; & >;& "& *;& * , X 1  . . .  X n & &$&&  & FX 1 G FX 2 G . . .  FX n G   >+

*&& 1( IOFXGL FYG9 >

n


IOFX i GL FYi G9:

O1:2(9

iˆ1

!  *& H!4 , *& FX i G  FYi G >&- & -&%+ J& *& &, & , O12(9 $$-*& IOXL Y9  *& %* & $$-*& n iˆ1 IOX i L Yi 9 Theorem 1.9. && *&  $>&   $$$&+ %&&4& *& *+$*& pOyjx9 ˆ

n  iˆ1

pO yi jxi 9

O1:239

..

Entropy and mutual information

O&&
H O11)99   * $$&+ "! & ! , J&   !+ , - $>>+ >! O&& &% && F.G '  }'1&  >+ de®ning -*&  >& & +& , O1239 * , &;&+ H!4 , *& $!   !$! , *& -*& * -  !!+ &+  ;&,+ O&& > .2:9  * *& ;%& * *&& 13  &&+  $& IOFXGL FYG9
&  $;% * IOX  Y L Z9 > IOY L Z9  %&& -& , &;&+ H!4 , X  Y  Z "& %& IOFX G FY GL F ZG9 > IOFY GL F ZG9 &-+ , *&& 1. "&;&     &+  $;& *& &- $ , *&& 1. & * IOX  Y L Z9 ˆ IOY L Z9 ,, OX  Y  Z9 ,   #; -* *& $>&  * , OX  Y  Z9   #; -*  , FX G  F ZG & $ , X  Z  +  >& $>&  J  $ FY G , Y !-* * OFX G FY G F ZG9   #; -* O&& > 16:B1639 /& $&,&   >!& *& && "*  && $, >! &,& &  #& F2.G *$& 6 "*&& *& -& &!  $;& "* ,! %3

Problems 1.1 *" * *& -;& < % 1=< ˆ < $&  &J O119 , HOX 9  ,-& , "& #& &*& , *& ,"% " ! $D O9 HOX 9   -!! ,!- , *& $>>+ ;&- p ˆ O p1  p2  . . .9 O>9 , X  Y &  ;>& &J&  *&  &  $& $-&  , X ˆ Y & *& HOX 9 ˆ HOY 9 1.2 , X  &J&  
@ $& 16 *" * HOX 9 ˆ ‡1 1.3 & X >&  -&&  ;>&  & f >&  &;!& ,!- &J&  *& %& , X  *" * HOX 9 > HF f OX 9G "* &H!+ ,, f  &  &  *& & fxD PfX ˆ xg . >+ >! O p1  p2  . . .  pn 9   &%& m < < m < n

Problems

.7

m &J& qm ˆ 1 jˆ1 pj  *" * HO p1  . . .  pn 9 < HO p1  . . .  pm  qm 9 ‡ qm %On m9 /*& & &H!+ *P 1.7 & f Ox9 >& + ,!- &J& ,  x > 1 , X   -&&  ;>& "*  n %& R ˆ fx1  . . .  xn g &J& *& f entropy , X >+ H f OX 9 ˆ iˆ1 pi f O1= pi 9 "*&& pi ˆ PfX ˆ xi g O+ &$+D f Ox9 ˆ % x:9 O9 , f Ox9  -;&@ \ J *& >& $>& !$$& >!  HOX 9 * &$& +  n *" >+ &@ $& * *& $>>+ >! * -*&;& * @ ! +  >& !H!& O>9 , f Ox9 ˆ % x=x *" * H f OX 9 , %Oe9=e F f Ox9   -;&@ \G O-9 /# *&  $ O>9  *" *  ,- H f OX 9 < %O69=6 "* &H!+ ,, &@-+ *&& , *& pi I &H! 16  *& & & &  &H!&-& , &%;& & ! >& !-* * *& ! ZOs9 ˆ n e a n s -;&%& ,  !,J-&+ %& s O &@ $& , *&& & + J&+ +  aI * -  J&9  &-* á . < &J& ÖOá9 ˆ !$ f HOp9D pn an ˆ ág *" * ÖOá9  %;& $ & -+ >+ ÖOá9 ˆ % ZOs9 sZ9Os9= ZOs9 "*&& Z9Os9= ZOs9 ˆ á F HintD ,  an s pn an ˆ á &J&  &" $>>+ = ZOs9   >! >+ qn ˆ e $$+ 0&&I &H!+  *& ! pn %Oqn = pn 9G , a ˆ Oa1  a2  . . .  aN 9 * + J&+ + - $& *" *  %$* , ÖOá9 ;&! á # #& *D

;& *& -& , *& $ A B  C 1.9 & p ˆ O p1  p2  . . .9>&  -!>& $>>+ >! *  pn > < , 1 n ˆ 1 2 . . .   1 nˆ1 pn ˆ 1 *" * , nˆ1 pn % n -;&%& *& HOp9  J& ;&&+, HOp9  J&  p  & O& p1 > p2 >


9 *" * pn % n , 1 *" >+ &@ $& * *& & &   !& , *& ! $ , -+  $$& 1.10 & X  Y >& *&  ;>& &J& 
@ $& 1) *" * ;OX  Y 9 ˆ   *&   , "*-* X   + ;& , Y  1.11  &-&+  !,J-& -  *& ? $>>+ >! pOx y z9 , &H!+  *  *&& 12 % * &! , + r > 2  + ;!& < < Pe < 1 -!-  ;>& X  Y "* %& f1 2 . . .  rg , "*-* I &H!+ O*& -+  *&& 129  *$

.)

Entropy and mutual information

1.12 *" * I &H!+  $& >*  !$$& >!   "& >!  Pe  &  , HOX jY 9 &$& *& !$$& >! *&!-+ 1.13 !$$& X  Y &  ;>& "*-* >* ! & ;!&   J& %!$ &J& Z ˆ Y X  !% *&& 12 *" * HOX jY 9 < HO Z9 , X  Z & &$&& *" * HOX jY 9 ˆ HO Z9 1.14 '&,+
H O179BO11&   ;>& !, + >!&  f& Y ! % *& ;!& f>+ p  &H!;&   %&  "* & $>>+ 12F1 O1 2 p9 n G  *&-& *  n!1 IOX < L X n 9 ˆ &"&& X  Y %;& ZII9  &J& >+ IOX L Y j Z9 ˆ


x yz

pOx y z9 %

pOx yjz9  pOxjz9 pO yjz9

"*&& "& $ *& !! -;& >! *& !  ;&D O9 IOX L Y j Z9 ˆ IOY L X j Z9 O>9 IOX L Y  Z9 ˆ IOX L Z9 ‡ IOX L Y j Z9 O-9 IOX L Y j Z9 > < "* &H!+ ,, OX  Z Y 9   #; -* 1.20 O& #; & A ˆ Oaij 9 >&  r 3 r -*- @ *   r -*9 1 aij > <  a ˆ 1 , i ˆ >& ,    & PfY ˆ jjX ˆ ig ˆ aij  &  & J& --& , -$& , * -*&D

" & O p<  p1  . . .  p r 1 9 ˆ p >&  $>>+ ;&- "*-*  >& "* r 1 pi aij  j ˆ ;& #&-*  >!& --%  p O9 *" * X <  X 1  X 2  . . . & &-+ >!&  ;>& & "* - & *&+ &$&&P O *& &H!&-&

Problems

.:

X <  X 1  X 2  . . .  -&  J& #; -*L && && F.G ' 1 *$& A'  A' 9 O>9 *& O$& + >9 entropy , *& #; -* X <  X 1  . . .  &J&  H ˆ 

n!1

1 HOX <  X 1  . . . X n 1 9: n

*" * Hˆ


i j

pi aij %

1 : aij

O-9  $!& *& >& $>>+ >!  *& &$+ , *& #; -* "*&  $>>+ -& & %;& >+ 

 q  q1  . . .  q r 1 <   < < < 1  q1  q2  . . .  q<  á 1 á  < < < 1      .. : 1 1   â 1 â < <  .  2 2 < < 1 < q r 1  q<  . . .  q r 2 O  *&  &@ $& &-* "   &, -+-- *, , *& $&-&% &9 1.21  IOX L Y 9 -+ -;&@ \  pOx9P   -+ -;&@ [  pO yjx9P 1.22 & X ˆ OX 1  . . .  X n 9  Y ˆ OY1  . . .  Ym 9 >& -&&  ;&-  & f  g >& &;!& ,!- , n  m & ;>& &$&-;&+ *" * IF f OX9L gOY9G < IOXL Y9 1.23 !$$+ *& % & 
@ $& 111  112 n 1.24 *" * &H!+ *  *&& 1( ,, pOxjy9 ˆ iˆ1 pOxi j yi 9 ,  Ox y9 1.25 *" * &H!+ *  *&& 13 ,, Y1  Y2  . . .  Yn & &$&&  *&-& * &H!+ *  *& -+  *&& 13 ,, X 1  . . .  X n & &$&& 1.26 ;&  $;&D , X  Y ˆ OY1  Y2  . . .  Yn 9 & -&&  ;&- n IOXL Y9 < iˆ1 IOXL Yi 9 1.27 & X  Z >& &$&&  ;>& "* -!! &+ ,!-   & Y ˆ X ‡ Z , hOY 9  hO Z9 &@ *" * IOX L Y 9 ˆ hOY 9 hO Z9 *" * *&  & , ! * , X  Z &  ;&- 1.28 O!9 & X  Z >& &$&&  ;>& X -&&  Z "* -!! &+  & Y ˆ X ‡ Z *" * Y *  -!! &+  * , hOY 9  hO Z9 &@ *& IOX L Y 9 ˆ hOY 9 hO Z9 *" * *&  & &! * , X  Z &  ;&- 1.29 , X ˆ OX 1  . . .  X n 9 & &$&& !  ;>& "* & ì i  ;-& ó 2i  X9 ˆ OX 19  . . .  X 9n 9 &  "* & ì9i  ;-& ó 9i 2  *" * X ‡ X9 ˆ OX 1 ‡ X 19  . . .  X n ‡ X 9n 9 &  "* & ì i ‡ ì9i  ;-& ó 2i ‡ ó 9i 2  1.30 , X 1  X 2  . . .  X n & &$&&  ;>& "* ,,&& &$& hOX i 9 ˆ hi  *" *

.(

Entropy and mutual information hOX 1 ‡


‡ X n 9 >

1 2 %

n


!

e

2 hi



iˆ1

"* &H!+ ,, *& X i & ! "* ;-& ó 2i ˆ e 2 hi =2ðe 1.31 , X *  -!! &+ *" * !$ HOFX G9 ˆ ‡1 "*&& *& !$& !  #& ;&  -&& H!4 , X  1.32 !-   ;>& "*& &+ pOx9  -!! ,  & x !-* * hOX 9 ˆ 1 1.33 & X >&   n & ;&- "* -!! n & &+ pOx9  ,,&& &$+ hOX9 & f >&  -!!+ ,,&& >& && $$% ,
!-& n$-& En  &, *" * hF f OX9G ˆ hOX 9 ‡ pOx9 %jJ j dx "*&& J ˆ J Ox1  . . .  xn 9  *& 0-> , *& ,  f  1.34 & X ˆ OX 1  . . .  X n 9 >&   ;>& "* &+  !$$& EFOX i ì i 9OX j ì j 9G ˆ r? ,  i j ;& * hOX9
& "*& -;-& @  O r? 9 O&& && F.G ' 2 &- )9 1.35 & f Ox9 >&  -!! &;!& ,!- &J&   &; I *& >?&- , * $>&   J ! *" %& *& ,,&& &$+ hOX 9 - = I  >& , X *  &+ ,!- * J& pOx9 ˆ < , x 2 I pOx9 f Ox9 dx ˆ A "*&& ,O f 9 , A , !$O f 9 O &@ $& , I ˆ O 1 19  f Ox9 ˆ Ox ì92 "& " ;& >+  &" !&  *& &  & -& , *&& 1119 &J& GOs9 ˆ I e sf Ox9 dx *" * *&& &@ s< "* G9Os< 9=GOs< 9 ˆ A  &J& qOx9 ˆ e s< f Ox9 =GOs< 9 , x 2 I qOx9 ˆ < , x 2 = I *" * hOX 9 < % GOs< 9 ‡ s< A "* &H!+ ,, X I &+ ˆ qOx9   &;&+"*&& F HintD && > 1(G $$+ * %&& &-*H!&  *&& -&D O9 f Ox9 ˆ % x I ˆ O1 19 O>9 f Ox9 ˆ x I ˆ O& *" * *&& &@ -&& H!4 FX G  FY G "*-* &  &$&&

*& &@ *&& $>&  & , && "*  & #"&%& , %&& #; -*L && , &@ $& && F.G ' 2 &- '   1.37 & OX  Y  Z9 >&  #; -* !-* * , + H!4 FX G  F ZG *&& &@  H!4 FY G !-* * OFX G FY G F ZG9   -&& #; -* ;& * IOX L Y  Z9 ˆ IOX L Y 9 1.38 & #; -* , *& , OX  f OX 9 Z9  OX  Y  f OY 99 *" * !-* -* *;& *& $$&+ &->&  > 16:  *&-& * *&& 1. * , *&  F HintD  *& OX  f OX 9 Z9 -& , fSi g  *& X $ & f OSi 9 >& *& Y $G

Notes

.3

1.39 & X 1  X 2  X 6 >& &$&& &-+ >!& !  ; >& "* & <  ;-& 1 *" * *& &H!&-& OX  Y  Z9 "*&& X ˆ X 1  Y ˆ X 1 ‡ X 2   Z ˆ X 1 ‡ X 2 ‡ X 6   #; -* O&&
@ $& 139 ! *"  * , FX G  F ZG & H!4 , X  Z "*-* >* ! & & * " ;!& OFX G FY G F ZG9  &;&  #; -* , + H!4 FY G , Y 

Notes 1 O$ 139 !$$& +! -! # H!& Qn D NN/ ;&  >& nPII *# >! *" +! !-&+ O@&+M9 "! -&&  n &-&& 2 O$ 2! X 9 6 O$ 219  *@ *& - ?% NN-&&II & J&  countable

*&& "! >&  $>&  &% !  I *;& -!>& $!  !$! $*>& >!  &   -K- "* --&$& , *&&- !%& "& * !!+    . O$ 2:9 *&& &J #& && + "*& pOx9 pO y9 6ˆ ?&-  %;&  *& J *&& -*$& , #&I ># F2.G ) O$ 6:9 $ , *& & & R   -&- , !>& OSi 9 "* [ Si ˆ R  Si \ Sj ˆ ö , i 6ˆ j : O$ 6(9 * & &   -+ !&  #& F2.G   *" * IOX L Y 9  &*& J&  &H!  *& &@$&-& ;!&  *& ?  $& $-& , X  Y , *& %* , *& #+ &;;& dì xy =dì x 3 ì y  "*&& ì xy  *& ? $>>+ &!& , OX  Y 9  ì x 3 ì y  *& $!- , *& % X  Y &!& *! + >-+   dì xy IOX L Y 9 ˆ % dì xy : dì x 3 ì y  ( O$ .&% $$& *&&  * , f Ox9  -!! f Ox9 gOx9 dx ˆ f Ox 9 < S gOx9 dx ,  & x< 2 S O&& $ F2G *&& 1.1) , S &@ $&9 3 O$ ..9 &*$  >&, %!&  #&I $,   & *& && &!  * IOX  Y L Z9 ˆ IOY L Z9 ,, OX  Y  Z9   #; -* *! , *& conditional mutual information IOX L ZjY 9  &J& >+ IOX L ZjY 9 ˆ IOX  Y L Z9 IOY L Z9 *& &H!& &!  IOX L ZjY 9 ˆ < ,, OX  Y  Z9   #; -* *$& 6  #&  "*+ &;&  *& $$&& , - !! ,  & J %;&  >- &!&*&&- &J , IOX L ZjY 9  *& & %;& >;& *& $;& ;! && &+ $$&& #& IOX L ZjY 9 > < O&H!;&  *&& 1.9  IOX L ZjY 9 ˆ IO ZL X jY 9 O&& > 1139  J+ $;& * IOX L ZjY 9 ˆ < ,, OX  Y  Z9  #; +  &- 6) *& $;& * IOX L ZjY 9 ˆ IOX  Y L Z9 IOY L Z9  >!& *& &!  5 %;

2 -&& & +& -*&  *& -$-+B- ,!-

2.1 The capacity±cost function

discrete memoryless channel O L && $ 2< ,,9  -*-&4& >+ " J& &D AX  *& input alphabet  AY  *& output alphabet   & , transition probabilities pO yjx9 &J& , &-* x 2 AX  y 2 AY  "*-*  ,+ pO yjx9 > >& & -;&&+ $+&   r 3 s -*- @1 Q ˆ Oqxy 9 "*& " & &@& >+ AX  -!  >+ AY  !*& & -& "* &-* $! x *&&   &%;& ! >& bOx9 *& NN-II2 , x !+ AX " >& #&  f>+ * y1  . . .  yn  *& !$! %;& * x1  . . .  xn n  *& $!  *& $!- iˆ1 pOxi jxi 9 *& - , &% x1  . . .  xn  &J&  bOx9 ˆ

n


bOxi 9:

O2:19

iˆ1

, *& n $! & &->& $>>-+ >+ *&  ;>& X ˆ OX 1  X 2  . . .  X n 9 "* ? >! ,!- pOx9 ˆ pOx1  . . .  xn 9 *& average cost  &J& >+ EFbOX9G ˆ

n


EFbOX i 9G

iˆ1

ˆ




pOx9bOx9:

O2:29

x

 &-* n ˆ 1 2 . . .  "& &J& *& n* capacity±cost function Cn O â9 , *& -*& >+ Cn O â9 ˆ

@fIOXL Y9 D EFbOX9G < nâg

O2:69

"*&& *& @ !  O269  &@&& ;&  $ OX Y9 ˆ OOX 1  . . .  X n 9 OY1  . . .  Y n 99 , n &  ;&- , "*-* O9 *& - $>>& PfYjXg & -& "* *& %;&

72

Discrete memoryless channels and their capacity±cost functions

-*&  $>>& & PfY1 ˆ y1  . . .  Y n ˆ yn jX 1 ˆ x1  . . .  n X n ˆ xn g ˆ iˆ1 pO yi jxi 9  O9 *& $! ;&- X J& EFbOX9G < nâ /& * -  $! ;&- X  test sourceL ,  J& EFbOX9G < nâ "& * -  âadmissible &-& *& @ 4  O269  #& ;&  n dimensional âadmissible test sources &;& & # >! *& ,!- Cn O â9 &  &  & * ,  %;& @ ,  $>>& O pO yjx99 *& ,!- IOXL Y9   -!! ,!- , *& $! >! pOx9 *& & , !-* >!   ,+% pOx9bOx9 ˆ EFbOX9G < nâ   - $- !>& , r n   &
!-& $-&   *& ,!- IOXL Y9 -!+ -*&;&  @ ! ;!&6 *  *& & NN @II *& * NN!$II $$&  O269 &- -& * , "& &J& â  >+ â



ˆ

 bOx9

O2:.9

x2 AX

*& EFbOX9G > n:â     Cn O â9  &J& + , â > â  + >&;& * , â1 . â2  *& & , & !-& ,+% EFbOX9G < nâ2   !>& , *& & , & !-& ,+% EFbOX9G < nâ1    Cn O â1 9 > Cn O â2 9 & Cn O â9   -&% ,!- , â > â  

*& capacity±cost ,!-. , *& -*&  " &J& >+ 1 CO â9 ˆ !$ Cn O â9: n n

O2:79

*& ! >& CO â9 " ! !  &$&& *& @ !  ! , ,  * - >&  & &>+ ;& *& -*& $& ! ,  & , *& -*& ! >& !&  !-*  "+ * *& ;&%& -  < â $& ! ,  & *& $&-& & & , * &! *I channel coding theorem " >& $;&  &- 22 ! >?&-  * &-   &;&$ &-*H!& , - $!% CO â9 ,  %;&    - ,!- /& >&% >+ *"% * *& ,!- Cn O â9 &  convex Theorem 2.1 Cn O â9 is a convex \ function of â > â Proof & á1  á2 > ! p1 Ox9 p2 Ox9 "*-* -*&;& Cn O â1 9 Cn O â2 9 &$&-;&+ *  , Y1  Y2 && *& !$! -&$%  X1  X2  *&

2.1 The capacity±cost function  EFbOX i 9G < nâ i  i ˆ 1 2: IOX i L Yi 9 ˆ Cn O â i 9

76 O2:)9 O2::9

&J&  ,!*& & !-& X "* >! pOx9 ˆ á1 p1 Ox9 ‡ á2 p2 Ox9   & Y >& *& -&$% !$! *& EFbOX9G ˆ x pOx9bOx9 ˆ   á1 x p1 Ox9bOx9 ‡ á2 x p2 Ox9bOx9 ˆ á1 EFbOX1 9G ‡ á2 EFbOX2 9G < nOá1 â1 ‡ á2 â2 9 O&& O2)99   X  Oá1 â1 ‡ á2 â2 9 >& *! IOXL Y9 < Cn Oá1 â1 ‡ á2 â2 9 ! -& IOXL Y9   -;&@ \ ,!- , *& $! $>>+ >! pOx9 O *&& 1)9 IOXL Y9 > á1 IOX1 L Y1 9 ‡ á2 IOX2 L Y2 9 ˆ á1 Cn O â1 9 ‡ á2 Cn O â2 9 O&&
H O2:99 h ! &@ &! *" * *& $$&+ , >& &J O279    > ,& 7 Theorem 2.2 For any   Cn O â9 ˆ nC1 O â9 for all n ˆ 1 2 . . . and all â > â   Proof & X ˆ OX 1  . . .  X n 9 >&  â >& & !-& * -*&;& Cn O â9 *  EFbOX9G < nâ

O2:(9

IOXL Y9 ˆ Cn O â9

O2:39

"*&& Y ˆ OY1  . . .  Y n 9  *& -&$% -*& !$! + *&& 13 IOXL Y9
%% O239 O21! ,!- *&  &  X   & Y1  Y2  . . .  Y n >& *&  -&$% -*& !$! *& EFbOX9G ˆ EFbOX i 9G < nâ >+ O21.9 n  IOXL Y9 ˆ iˆ1 IOX i L Y i 9 O *&&  1(  139 ˆ nC1 O â9 *! h Cn O â9 > nC1 O â9  "& Corollary For a memoryless channel CO â9 ˆ C1 O â9 Osee de®nition O2799 FN.B. This result is not true for channels with memory; see Prob. 21 â    -;&@+  $& *    -!! , â . â  L &&

$$&@  O    -!!  ⠈ â  L && > 279 /& * " %!& * CO â9  -!+ - , !,J-&+ %& â & ! &J& C @ ˆ @fCO â9 D â > â  g *  C

@

ˆ

@fIOX L Y 9g

O2:1)9

"*&& *& @ !  #& ;&  O& &9 & !-& X  "*  >!  EFbOX 9G C @  -& *& capacity , *& -*& , "& &J& â

@

ˆ

fEFbOX 9G D IOX L Y 9 ˆ C

@ g

O2:1:9

*& -&+ CO â9 ˆ C @ ,  â > â @   CO â9 , C @ , â , â @  ," , *& ,- * CO â9  -&%  -;&@ \ , â > â  

2.1 The capacity±cost function

77

 - , â > â @ * CO â9  -!+ -+ -&% , â â < â @ O&& > 2)9 &-&  * &; CO â9 -! >& &J& >+ CO â9 ˆ

@fIOX L Y 9 D EFbOX 9G ˆ âg

â



%  *&& ,- "& && *  +$- CO â9 -!;& %* #  *"  % 21 Example 2.1 O-!&9

 Qˆ

q p

 p  q

bO&  & ;& *& -*& & %&&+ ,  >+ â > â  CO â9 !%*  &$&& *& @ ! &  "*-* ,  - >&  & ;& *& -*& , *& ;&%& $! - ! >& < â   ! >?&-  * &-  #& *&& ;%!& &% $&-& & *& ,"% *!%* &@$& & /& &  ,  !-&   &H!&-& U ˆ OU1  U2  . . .  U k 9 , &$&& &-+ >!&  ;>& "* -  >! ,!- PfU ˆ ?&-     *&& k NN>II , ,  ;& *& -*& !% *& -*& n  & "*  ;&%& - < â & X ˆ OX 1  . . .  X n 9 >& *& -&$% -*& $! Y ˆ OY1  Y2  . . .  Y n 9 ^ ˆ OU ^1 . . .  U ^ k 9 *& &-&;&I & & , U "*-* *& -*& !$!  U "& ! & &$& +  Y O% 279 ^ i 6ˆ Ui g , å ,  i /& ! & *   $&+ % +&  + Pf U ^ > "*&& å   &   ! >& *& >+ *&& 1( IOUL U9 k ^ ^ i 9 ˆ HOU i 9 HOUi j U ^ i 9 ˆ % 2 HOUi j U ^ i9  IOUi L U uˆ1 IOU L U i 9 > % 2 HOå9 >+ I &H!+ O-+  *&& 129 *! ^ > kF1 H 2 Oå9G  >+ *& $-&% *&& "& *;& IOUL U9 O
H O11799 IOUL V9 < IOXL Y9 + >+
H O269 IOXL Y9 < Cn O â9 ˆ nCO â9  >% *&& *&& &H!& "& *;& O $& % & >& 29D k CO â9 < : n 1 H 2 Oå9

O2:139

*&  k=n "*-* "& * - *& rate , *& +&  &$&& *& ! >& , bits per channel use >&%  & >+ !  %+ - ! - +&  *& >! O2139   -&% ,!- , *& > & $>>+ åL *   !$%D  + &&+ * *& & &>+ "& "  - !-& *& "& "& ! - !-& R!;&+
H O2139 + * , *& -*&  >&% !&  * *& ;&%& $! -  < â  , "& "*  &%  +& "* & r . CO â9 *& *& &!% & $>>+ å  >!& >&" >+ å > H 1 F1 CO â9=rG . H 2 1 O1 C @ =r9 . R  PfUi 6ˆ Ui g , å ,  iM * !% &! !&  !& *  -& *& channel coding theorem *& #&+  * &!  *& --&$ ,  -& "*-* "& " &->& ;&  &%& n  O-*&9 code of length n ;& AX   !>& C ˆ fx1  x2  . . .  x M g , A nX ( *& & , *& -&  &J&  r ˆ O1=n9 % MQ, % & >& 2 *& &   ! , bits per O-*& $!9  symbol *& -&  â-admissible , bOx i 9 ˆ njˆ1 bOxij 9 < nâ ,  i "*&& x i ˆ Oxi1  . . .  xin 9  *& &@$ , x i  - $&

decoding rule , *& -& C   $$% f D A Yn ! C [ f?g *& $&- + > NNPII && &-& ,!&L  %J--& " & &%& >&"

-& - >& !&  &%  - !- +&   *& && , % 27  ," & k >&  &%& !-* * k < %2 M *&  " >& $>&  %  distinct -&" x i  &-* , *& 2 k $>& !-& &H!&-&3 && $$% , *& & ,  $>& !-& &H!&-&  *& -& C  -&  encoding rule , *& !-& &H!&-&  >&  &  u ˆ Ou1  . . .  uk 9 *& &   encode u  & , *& -&" x i ; *& &-% !&    x i ;& *& -*&  *& *& & , *& -*& $$&  + ;& , x i L -  y *& &-&;& &-& y   -&" x j O NNPII9 ; *& &-% !& f L *& & & u^ , u  *& *& !H!& !-& &H!&-& u^ O, *&&  &9 * -&$  *& -&" x j  *& & $>>+ , *& +&  %;& * x i   & && >+ POi9 E   *& %;& >+ POi9 E ˆ Pf f Oy9 6ˆ x i g ˆ




f pOyjx i 9 D f Oy9 6ˆ x i g

O2:2?&- ,  "  %;& $ ;& &@ $&M9

2.2 The channel coding theorem Example 2.4 AX ˆ AY ˆ f>+ p , p , 2  !-* & , p  ;&+   O&&
H O 2119 Example 2.6 AX ˆ f! O2139 &- * "& & !  &% , %;& ;!& , â R , CO â9  å . 2d Rne  POi9 E , å ,  å *& ,"% *&& & *   $>&   * Theorem 2.4 Let a   with capacity±cost function CO â9 be given. Then, for any â< > â  and real numbers â . â< , R , CO â< 9, å . 9 M > 2d Rne  O-9 POi9 E , å for all i ˆ 1 2 . . .  M Corollary (the channel coding theorem for DMC's 10 ). For any R , C @ and å . 2d Rne  O>9 POi9 E , å for all i ˆ 1 2 . . .  M Proof of corollary & â< ˆ â

@

 *&&

2.

h

Proof of Theorem 2.4 *!%*! *& $, *# , n   %& &%&  " >& $&-J& & $&-&+ & & *& & Ù ,  $ Ox y9 -% , & -*& $! &H!&-& x ˆ Ox1  . . .  xn 9  & -*& !$! &H!&-& y ˆ O y1  . . .  yn 9 >* , &%* n  + > Ù ˆ A nX 3 A Yn  /& #& Ù    $& $-& >+ &J% pOx y9 ˆ pOx9 pOyjx9L

O2:219

pOx9 ˆ pOx1 9 . . . pOxn 9 "*&& pOx9   $>>+ >!  AX * -*&;& CO â< 9L  pOyjx9 ˆ pO y1 jx1 9 . . . pO yn jxn 9 "*&& pO yjx9 & *& -*&  $>>& & ! " -*& R9  ,+ R , R9 , CO â< 9  &J&  !>& T  Ù  ,"D T ˆ fOx y9 D IOxL y9 > nR9g

O2:229

2.2 The channel coding theorem

)6

"*&& IOxL y9 ˆ %2 F pOyjx9= pOy9G *& & T - >& *!%* ,  *& & , $ "*-* & -& %&*& O&& > 21:9   -& &&  "& &J&  !>& B  A nX D B ˆ fx D bOx9 < âng:

O2:269

&& B  *& & , â >& -&" + "& &J& *& & T   T >+ T  ˆ fOx y9 D Ox y9 2 T  x 2 Bg:

O2:2.9

" & C ˆ fx1  x2  . . .  x M g >& + -& %  > , &%* n /& &J&  &-% !&  ," , y  &-&;& "& &@ & *& & SOy9 ˆ fx D Ox y9 2 T  g  B O"*-* -! >& *!%* ,   NN$*&&II ! y9 , SOy9 - exactly one -&" x i "& & f Oy9 ˆ x i  *&"& &*& >&-!& SOy9 -  -&"  & * & -&" "& & f Oy9 ˆ ? *  "& &>&&+ #&  &11 * &-% !&  &$-&  % 2)

Figure 2.6 *& &-% !& !&  *& $, , *&&

2.

).

Discrete memoryless channels and their capacity±cost functions

, "& !& *& -& C "* *& &-% !& ?! &->& , x i   &  , y  &-&;&  & - --! ,, &*& x i 2 = SOy9  x j 2 SOy9 ,  & j 6ˆ i &-& O&&
H O22& &"& 


POi9 ÄOx i  y9 pOyjx i 9 ‡ ÄOx j  y9 pOyjx i 9 E < y

y

j6ˆ i

ˆ Qi Ox1  . . .  x M 9:

O2:2:9

! %    &*" J  -& fx1  . . .  x M g , "*-* Qi  ;&+   ,  i  !&!+ ,!&+ *"&;& Qi   &@& &+ - $-& ,!-  - >& &;!& &@$-+ O &;& -&+ & &9 &@-&$ , *&  $& $>& -&Q-&+  , >% - $&@ -& "* %& n  *!%& MM  "*  *& $  !-% *& >! POi9 E < Qi  *& J $-&P *& $  * *!%*    $>&  & & Qi ,  $-! -&   $>&  & & *& ;&%& ;!& , Qi  fx1  . . .  x M g %& ;& *& & ,  $>& -&M

*%+12 * ;&%& " ! !  $$-* < , M ˆ 2 Rn  n ! 1M * & #>& $, &-*H!&  -& random coding >&-!& "& -*& *& -& fx1  . . .  x M g NN  II --%   -& $>>+ >! & ! " && *"  "#  &

*& J &$   &->& *& $$$& $>>+ >!  *& & ,  $>& -&   pOx1  . . .  x M 9 ˆ

M 

pOx i 9

iˆ1

"*&& , x i ˆ Oxi1  xi2  . . .  xin 9 pOx i 9 ˆ

n

kˆ1

pOxik 9 * $>>+ 

2.2 The channel coding theorem

)7

>! -&$  *& &@$& & , -*% *& -& NN +II >+ $-#% *& ;! -&" -& &$&&+ --%  *& $>>+ >! pOx9 * -*&;& CO â< 9 /& " ;&" Qi Ox1  . . .  x M 9    ;>&  *&  $& $-& ,  $>& -&L  &@$&-& ;!&  O&&
H O22:99

EOQi 9 ˆ E


y

ˆ E1 ‡





ÄOx i  y9 pOyjx i 9 ‡ E ÄOx j  y9 pOyjx i 9 y

j6ˆ i


j6ˆ i

E2O j9 :

O2:2(9

 "& >! E1 D

E1 ˆ ˆ


x1 :::x M


x i y

ˆ




pOx1 9 . . . pOx M 9


y

ÄOx i  y9 pOyjx i 9

pOx i 9 pOyjx i 9ÄOx i  y9 pOx y9ÄOx y9

O&& O2:2199

xy

ˆ PfOx y9 2 = T g

O&& O2:2)99

ˆ PfOx y9 2 = T  x 2 = Bg

O&& O2:2.99

< PfOx y9 2 = T g ‡ Pfx 2 = Bg:

O2:239

&-&

E1 < PfIOxL y9 , nR9g ‡ PfbOx9 . âng

!

O&& O2:229 O2:269 O2:2399

))

Discrete memoryless channels and their capacity±cost functions IOxL y9 ˆ % ˆ % ˆ

n


pOyjx9 pOy9 n  pO yk jxk 9 pO yk 9 kˆ1

pO yk jxk 9 pO yk 9

%

kˆ1

ˆ

n


IOxk L yk 9:

kˆ1

&-& IOxL y9  *& ! , n &$&& &-+ >!&  ;>& IOxk L yk 9 + &J EFIOxk L yk 9G ˆ IOX L Y 9 ˆ CO â< 9  * &-* IOxk L yk 9 * & CO â< 9 -& R9 , CO â< 9  ," , *& "&# " , %& ! >& O&& $$&@ 9 * 

n!1

PfIOxL y9 , nR9g ˆ & "* & < â<  -& â . â<   ," * 

n!1

PfbOx9 . nâg ˆ % O2239 O26& -!&D

R9 n

O&& O22299 &-& *& >!

2.2 The channel coding theorem
E2O j9 < pOx9 pOy9

):

Oxy92T

% O22(9 O2239  O2669 "& %& EOQi 9 < PfIOxL y9 , nR9g ‡ PfbOX9 . nâg ‡ M :2

R9 n

:

O2:6.9

, M ˆ 2:2d Rne  *&  &  O26.9  < .:2 nO R9 R9  -& R9 . R  ," * , !,J-&+ %& n * & - >& &     && /& *;& &+ >&;& O&& O26& &J&  *&  $& $-& ,  $>& -& *& , O22:9  O2679 "& *;& , M ˆ 2:2d Rne  n %& &!%* EOPE 9 , å=2:

*! *& ;&%& ;!& , PE  , å=2   *&& ! >&  $-! -& Ox1  . . .  x M 9 "* PE Ox1  . . .  x M 9 , å=2 * -& +  ,+ *& --! , *&& 2. >&-!&  + - -&" x i , "*-* bOx i 9 . nâ E POi9 & * *, , *& -&" x i * E . å ! , POi9 > å , O26)9 "& "! *;& PE > å=2  --  , "& E Oi9 &&& *& -&" "* P E > å , *& -& "& >  -& "* 16 > 2d Rne -&" , "*-* POi9

* -& *&&,& J& E , å ,  i --! O>9  O-9 , *&& 2. + -& * , bOx i 9 . nâ *& &-% $*&& SOy9 ˆ fx D Ox y9 2 T  bOx9 < nâg - - x i  *  POi9 E ˆ 1 &-& * &" -& - - + -&" "*-* &  â >&   O9  J&  h

)(

Discrete memoryless channels and their capacity±cost functions Problems 2.1  $!& *& - $&& -$-+B- ,!- , *& ,"% -*&D

 1 < < O9 Q ˆ  < q p  bO - >& &-&;& -&-+  erased *  &-&;&  NNPII9 2.2 & ! -  -*& weakly symmetric , *& -!  ,   @ Q - >& $&  !>& Ci !-* * , &-* i  *& @ Qi , & >+ *& -!   Ci &-* "   $& ! , &;&+ *& "  *&  &  !& , -!  O &@ $& Q9  $%& 7:  "&#+ + &- >!  + &-9 *" * *& -$-+ ,  "&#+ + &- -*&  -*&;& "* &H!$>>& $! 2.3 & ! &J&  strongly symmetric -*& , & r  & "*&  @ Q * *& , !& >&" , r ˆ .D

 q p p p  p q p p    p p q p p p p q "*&& p < 1=r  Or 19 p ‡ q ˆ 1 *& >?&- , * &@&-&  , +!  - $!& CO â9 , * -*& , >+ $! - bOx9 >+ !$$+% *& &  *& ,"% %! & O ! & â  ˆ  2.9 , X   & !-& -*&;% CO â9 & f Ox9 ˆ PfX ˆ xg gOx9 ˆ PfY ˆ xg , x 2 f 1(  > *& $ &- !D

Problems CO â9 ˆ % AOë9 âˆ

2.4

2.5

2.6 2.7 2.8

ëA9Oë9=AOë9 1 q

p

HOq9

)3 O1

q9%Or

19

OA9Oë9=AOë9 ‡ BP 9

 ë %& ;& *& &; ë 2 Fë<  ?&- , * &@&-&   *" * *&&   &&  , %&&+  ! % â  ˆ ?&- , * &@&-&   $;& * CO â9  -!!  ⠈ â     * !$$+ *& &  *& ,"% %! & & Op1  p2  . . .9 >&  &H!&-& , $>>+ ;&- &$&&% *& $! >! , & !-& -*&;% CO â1 9 CO â2 9 . . .  "*&&  n!1 â n ˆ â   *& *&&   !>&H!&-& Op n1  p n2  . . . 9 "*-* -;&%&   $>>+ ;&- p , X   & !-& "* $! >! p *& EFbOX 9G ˆ â   IOX L Y 9 ˆ  â!⇠CO â9 &-& CO â  9 >  â!⇠CO â9 -& CO â9   -&% ,!- , â > â   * $;& *& &H!& -!+ & f Ox9 >&  -&% -;&@ \ ,!- , x > x1  ! & f Ox9  - , x > x2 "*&& x1 , x2  *" * f Ox9  -+ -&% , x1 < x < x2  *-&4&  ,!+  +! - *&  I "* C @ ˆ & AO19 X 3 A X  !$! $*>& A Y O29 O19 O29 : 3 A Y   $>>& pOO y1  y2 9j Ox1  x2 99 ˆ p O y1 jx1 9 p O y2 jx2 9

:
&% !&  $&9 *" * *& -$-+B- ,!- , *& $!- -*&  %;& >+ CO â9 ˆ

@fC1 Ot9 ‡ C2 O â t

t9g:

*! *" * *& %$* , CO â9  >& >+ % >* -& , *& -!;& C1 O â9  C2 O â9  $  *& " -!;& *;% *&  & $& 2.10 O *& -$-+ ,   $& -*& "* & +9 & Z 1  Z 2  . . . >& &-+ >!&  ;>&  #; -* #% ;!&  f 12& !&  &J&  ;& -*& "* AX ˆ AY ˆ f+ *& !& Yi  X i ‡ Z i O  r9 "*&& X i  Yi & *& $!  !$! , *& -*&   & i *& ndimensional capacity  &J&  C O n9@ ˆ @fIOXL Y9g  X %& ;&  n & & !-&  *& capacity  C @ ˆ !$O1=n9C On9@  *" * C O n9@ ˆ n % r HO Z 1  . . .  Z n 9 ˆ n % r HOp9 On 19 H "*&& p   $>>+ ;&- &->% *& -  >! , *& Z i I  H  *& &$+ , *& -* -!& * C @ ˆ % r H $$+ * &!  *& -& "*&& r ˆ 2  *& -*I  $>>& & %;& >+ *& -*- @   q p Qˆ : p q 2.11  *& -*&  - ,!- 
@ $& 27 *" &@$-+ *"  -*&;& "*  $ & >+ *& -% *&&  *  %;&  $&-& &-$ ,  -& C * * $$&& O9 O>9  O-9 , *&& 2. 2.12  $&& *& & ,
@ $& 2)  *" *" *& &-% ,!- 1 -! >&  $;&  -*&;& POi9 E ˆ 6 , i ˆ 1 2 2.13  *& $, , *&& 2. "& %;&  &@$- &-% !& ,  %;& -& C ˆ fx1  . . .  x M g O&& $$ )69 * !& "*& &H!& , ! $!$&   $    &+ !&  $--&  * &@&-& "& * !-&  !+ " *& &-% !& "*-* & >&& * *& &  *&& 2. >! & & ,J-!  +4& *! !$$& *& $>>+ * x i " >&  &  pi  i ˆ 1 2 . . .  M ;& *& &-&;& ;&- y minimum error probability decoding O
9 -*& *& -&" , "*-* *& -  $>>+ pOx i jy9  %& Maximum likelihood decoding O 9 -*& *& & , "*-* pOyjx i 9  %& O9 *" *
 ;& !$    & *  *& $>>+ , &-& & %;& * y  &-&;&    4& >+
  *&-&  * M *& ;&%& & $>>+ PE ˆ iˆ1 pi POi9 E   4& O>9 *" * , pi ˆ 1=M ,  i *&  $&,  &-+ 
 O-9    &J& *& Hamming distance d H Ox y9 >&"&& x  y  *& ! >& , - $&  "*-* *&+ %&& *" *   &-& "+ $-#  -&" x i "*-*  -&  y "* &$&-  * -& O&& > 21:   &- :69 2.14 O!9 & *& ,"% -& , &%* . ;& f& < â "& -! &H!& * *& @ ! ">& - >& < â *  &J& CO â9 ˆ !$fIOX L Y 9 D bOX 9 < â "* $>>+ 1g  $!& CO â9 , *& -*& ,
@ $& 21 22  26 O$$ 77B7)9 & *& -% *&& &  ; , CO â9P    *" * *& & T &J& 
H O2229  , *& , T ˆ fOxL y9 D d H OxL y9 < rg "*&& d H Ox y9  *& Hamming distance >&"&& x  y O& *& ! >& , - $&  "*-* x  y ,,&9  J r  &  , n R9  å *& -;& $>>+ , *& -*& O ! & pOx ˆ &  J& &  & f >&  ,!- $$% Ù  *& & ! >& ;& * *&& &@  && & ù 2 Ù !-* * f Où9 , y ,,   $>&  &J&  $>>+ >!  Ù !-* * EF f Où9G , y & *&   "*& @ ,  $>>& 

1 1  < < < 2 2    < 12 12 < <      Q ˆ  < < 12 12 < :   < < < 1 1   2 2 1 < < < 12 2

O *&&   - - & bOx9 ˆ < ,  x9 O9  $!& C @  O>9   -& , &%* 1 "* & % 2  POi9 E ˆ < ,  i O-9   -& , &%* 2 "* & 12 % 7  POi9 E ˆ < ,  i O9 , fx1  . . .  x M g  + -& , &%* n , * -*& "* POi9 E ˆ < ,  i *" * *& -&I &  , % 721. 2.20 *& >+ &!& -*& * @ ,  $>>&  < ? 1 Qˆ < q p <  1 < p q "*&& < < p < 12 p ‡ q ˆ 1 O ! &  $! - & bOx9 ˆ < ,  x9  -$-+  C @ ˆ 1 p O&& > 21>9 ! & * * -*&  &H!$$& "* noiseless, delayless feedback *  *& &-&;&  >&  - !-& >-#  *& && *& + > *& * &-&;& *& && $ *& ,"% NN-%II &%+ , - !-% *& !$! ,  >+ + &- !-& ;& * -*&D *& &&+ &$& &-* + > !   J+ &-&;& -&-+

:2

Discrete memoryless channels and their capacity±cost functions O9 -!& *& ;&%& ! >& , -*& + > &H!& $& !-& + > , * &%+  !& O>9 % *& &! , $ O9 &% , + R , C @ å . 2d Rne  POi9 E , å , &-* i O&& -+  *&& 2.917  *& &@ @ $>&  "& " #&-*  $, ,  -% *&& ,    "*! - - *    & "+ %&   *& "+ "&#& * *& -*& -% *&& O-+  *&& 2.9  "& ! #&  & &J  &-* $ x1 x2 2 AX  "& &J&
 J Ox1  x2 9 ˆ pO yjx1 9 pO yjx2 9 y2 AY

 J< ˆ

fEOJ OX 1  X 2 99g

"*&& *&  4  #& ;&  &$&& &-+ >!&  ;>& ! % ;!&  AX  + *&  $ H!+ R<  &J& >+ R< ˆ

%2 J < :

*& *&&  >& $;&  *D R< theorem for a    + R , R< *&& &@  -& fx1  x2  . . .  x M g "*  & M ˆ d2 Rn e -&"  M Oi9 , &%* n   $$$& &-% !& !-* * , PE ˆ O iˆ1 P E 9=M && *& ;&%& &-% & $>>+ *& PE , 2 nO R< R9  O * *&&  %& * *& -*& -% *&&  *  %;&  &@$- & & , *"   PE - >& &   ,!- , n   "&#&  * R< , C @   * , & R< , R , C @  &   $+ * PE ! <  $>&  9 2.21 &  -& , &%* n -% + " -&" x1 ˆ Ox11  x12  . . .  x1 n 9  x2 ˆ Ox21  x22  . . .  x2 n 9 ! & * %;&  &-&;& n!$& y *& &-& !$! x1 , pOyjx1 9 . pOyjx2 9  x2 , pOyjx2 9 . pOyjx1 9 & Y1 ˆ fy D pOyjx1 9 . pOyjx2 9g Y2 ˆ fyD pOyjx2 9 . pOyjx1 9g , POi9 E && *& $>>+ , &-& & %;& * x i  & *" * n  POi9 J Ox1 k  x2 k 9 i ˆ 1 2 E < kˆ1

>+ ;&,+% *& ,"% &$D
POi9 pOyjx i 9 E < y2Yi


+ POi9 E *& $>>+ , &-& & %;& * x i   &  *" * POi9 E
&  > 222 ;&  -& "*&& *& ;! -&" & -*& &$&&+ --%  *& $>>+ >! * -*&;& J <  > *& & & : EFPOi9 E G, M 2

R< n

:

2.24 " - $&& *& $, , *& R< -% *&&  2.25 *" * ,   " # R< ˆ 1 %2 1 ‡ 2 pO1 p9  "*&&

p  *& " > & $>>+ !*& & *" * , R< , C @  "*&& C @ ˆ 1 H 2 O p9  *& -$-+ , *&  2.26  $!& R< , *& >+ &!& -*& O&& > 21>9 1 2 C @

Notes 1 O$ 7& "* *& $$&+ * *& ! , *& &&  &-* "  1 O-, > 12>+ @  *;& - " &$+  -!  !  : O$ 7(9 ! % *&  $& %*  & >& 2 ( O$ ) A n && *& & fOa1  a2  . . .  a n 9D a i 2 Ag , && n!$& , A 3 O$ )& !&  - !-& *& !$! , + !-& >+ & +&  *&"& $;& *& ! >& , $>& !-&

:.

Discrete memoryless channels and their capacity±cost functions

!$!  < M * >!& #& *& -% *&& O *&& 2.9 !-* & ;!>& * ! -! -& -& ,&" & !-& & --!&+ && >+  >+ + &- !-& 1< O$ )29 * -+ "*-* & * we can communicate arbitrarily reliably at rates below channel capacity  -& *& channel coding theorem -&  &&  & 2 *& - -  !!+ >&   *& & %&& &! , *&& 2.  &;&+ ! $ 11 O$ )69 * &-% !&   *& >& $>& &L && > 216  21. "&;&   &;&+ &+  +4&   !,J-&+ -&  $   " !  $;& *& -% *&&  12 O$ ).9 ! && > 21( 16 O$ ):9 & * &&% -&" , C - -*%& POi9 E , *& & %  Oi9 -&" -& P E ˆ f pOyjx i 9D y 2 = f 1 Ox i 9g 1. O$ :19 *   , ! &@ $& ,  -*& "*  4& zero-error capacity   #" *  -& &@ , * -*& "* PE ˆ <  & . 12 % 7L && *& $$& >+ * F27G $$ 112B116  ;C4 F71G 17 O$ :29 * &! >&- & & &&%  %* , *& *&&  !&  * * ,&&>-# - -&& *& -$-+ ,   M O&& * F27G $ 12-#L && F27G $$ 6:6B.6)

3 -&& & +& !-&  *& & ,!-

3.1 The rate-distortion function &  ,  !-& * $!-& &;&+ ! ,  &  + > u ,  J& & AU  -& *& source alphabet !$$& * *& &H!&-& , + > $!-& >+ *& !-& - >& && >+  &H!&-& U 1  U 2  . . . , &$&& &-+ >!&  ;>& "* -  >!  ,!- PfU ˆ ug ˆ pOu9 !-*  ,  !-&  -&  discrete memoryless source O 9  *& ! >& pOu9 & -& *& source statistics " !$$& * "& & &H!&    *& !-&I !$! ;&  -*&   -& & /& ! & *   & !-& + > u 2 AU " >& &$!-&  *& &   + > v "*-*   && & , *& J& & AV  -& *& destination alphabet OAV " !!+ >!  "+ - AU   !>&9  !$$& * , &-* $ Ou v9 *&&   &%;& ! >& dOu v9 "*-* &!& *& &  distortion -!& "*& *& !-& + > u  &$!-&  *& & + > v *& ,!- d  -&  distortion measure1 + !$$& * *& ,!- d  &@&&  $ Ou v9 ˆ Ou1  u2  . . .  uk L v1  v2  . . .  v k 9 , A Uk 3 A Vk >+ dOu v9 ˆ

k


dOui  v i 9:

O6:19

iˆ1

  !!+ >!  "+ -;&&  #& AU ˆ f & & >+ *& !-&  & V1  V2  . . .  Vk >& any -&- , k  ;>& #% ;!&  *& & $*>& AV   &J&  *&  &  $& $-&  *& Ui I /& - --!& *& !! ,  IOUL V9 >&"&& *&  ;&- U ˆ OU1  . . .  U k 9  V ˆ OV1  . . .  Vk 9   *& average distortion EOd9 ˆ EFdOU V9G &J& >+
EOd9 ˆ pOu v9dOu v9 uv

ˆ




pOu9 pOvju9dOu v9:

O6:29

uv

O  O629 *& !   &@&& ;& *& r k s k $ Ou v9 ˆ Ou1  . . .  uk  v1  . . .  v k 9 "*&& ui 2 AU  v i 2 AV   pOu v9 ˆ PfU ˆ u V ˆ vg pOvju9 ˆ PfV ˆ vjU ˆ ug9 /& " &J&6 *& ,!- Rk Oä9 "*-*   ,!- , *& !-& - O pOu99 *&  @ D  *& & ! >& ä >+ Rk Oä9 ˆ

fIOUL V9D EOd9 < käg:

O6:69

 O669 *&  4  &@&& ;&  $ OU V9 ˆ OOU1  . . .  Uk 9 OV1  . . .  Vk 99 , k &  ;&- ! % ;!&  A Uk 3 A Vk , "*-* U1  . . .  U k & &$&& "* -  >! ,!- PfU ˆ ug ˆ pOu9 "*&& O pOu99 & *& %;& !-& -  *& ;&%&  EOd9 &J&  O629  < kä -& *& !-& - O pOu99 & J@&  - $!% Rk Oä9 ,  J@& ä "& ! ;+ *& - $>>& pOvju9 * &J& V *&& $>>& - >& *!%* ,   $>>& &J%  -*& "* U  $! V  !$!  *& $&& -&@ * -*&  !!+ -&  k &

3.1 The rate-distortion function

::

test channel  *&  4 O669    #& $-& ;&  & -*& "*& ;&%&   < kä &;& $& + & # >! *& ,!- Rk Oä9 &  &  & * , J@& O pOu99 *& ,!- IOUL V9   -!! ,!- , *& r k s k  $>>& pOvju9 *& !>& , *& & ,  $>>& "*&& EOd9 < kä   - $- &% , r k s k  &
!-& $-&   *& ,!- IOUL V9 -!+ -*&;&   ! ;!&  * &%. *  *& & "& "& NN II *& * NN,II 
H O669 &- -& * *&  ! $>& ;!& , EOd9  %;& >+ k
ä   "*&&
ä  ˆ pOu9
 dOu v9 O6:.9 v

u2 AU

  + >& && , O629 -& EOd9 > Ouv9 pOu v9 v dOu v9 ˆ kä  

*! Rk Oä9  &J& + , ä > ä  + >&;& * , ä1 . ä2  *& & , k & & -*& ,+% EOd9 < ä2   !>& , *& , "*-* EOd9 < ä1    Rk Oä1 9 < Rk Oä2 9 *  Rk Oä9   &-&% ,!- , ä > ä  

*& rate-distortion ,!- , *& !-&  " &J&  ROä9 ˆ , k

1 Rk Oä9: k

O6:79

*& ! >& ROä9 ! !  &$&& *&  ! ! >& , > O! %  %*  & >& 29 &&&  &$&&  !-& + > , "& & "%  &&  ;&%&  , ä *& $&-& & & , * &! *I source coding theorem " >& &  $;&  &- 62 ! >?&-  *& & , * &-   &;&$ &-*H!& , - $!% ROä9 ,  %;&     &!& ! J &!  * *& ,!- Rk Oä9 & -;&@ Theorem 3.1 Rk Oä9 is a convex [ function of ä > ä Proof !$$& á1  á2 > ä @ 

 

! *" * , ä1

Rk Oá1 ä1 ‡ á2 ä2 9 < á1 Rk Oä1 9 ‡ á2 Rk Oä2 9:

O6:)9

  * & pi Oujv9 >& *&  $>>& ,  & -*& -*&;% Rk Oä i 9 i ˆ 1 2 *& , V1  V2 && *& & -*& !$! IOUL Vi 9 ˆ Rk Oä i 9

O6::9

:(

Discrete memoryless sources and their rate-distortion functions EOd i 9 < kä i 

i ˆ 1 2

O6:(9

"*&& d i ˆ dOU Vi 9 && *& ;&%&   *& i* & -*& " &J&  &" & -*& "*  $>>& pOvju9 ˆ á1 p1 Ovju9 ‡ á2 p2 Ovju9 , V && *& !$! , * & -*& *& , O629  O6(9 EFdOUL V9G ˆ á1 EFdOUL V1 9G ‡ á2 EFdOUL V2 9G < Oá1 ä1 ‡ á2 ä2 9 &-& * & -*&   >& , *& --! , Rk Oá1 ä1 ‡ á2 ä2 9   IOUL V9 > Rk Oá1 ä1 ‡ á2 ä2 9  *& *& * -& IOUL V9   -;&@ [ ,!- , *&  $>>& pOvju9 O *&& 1:9 IOUL V9 < á1 IOUL V1 9 ‡ á2 IOUL V2 9 ˆ á1 Rk Oä1 9 ‡ á2 Rk Oä2 9 *&  " &H!& - >&  %;& O6)9   $;& *&& 61 h

*& &@ &! *" * ,    *& - $! , ROä9  -& >+ && * "! $$& , &J O679 Theorem 3.27 For a DMS, Rk Oä9 ˆ kR1 Oä9 for all k and ä > ä

 

Proof & pOvju9 >& *&  $>>& ,  k & & -*& -*&;% Rk Oä9 *& IOUL V9 ˆ Rk Oä9

O6:39

EFdOU V9G < kä:

O6:1+ *&& IOUL V9 >

k


1( "& *;&

IOUi L Vi 9:

O6:119

iˆ1

, "& &J& äi ˆ EFdOUi  Vi 9 "& *;& IOU i L Vi 9 > R1 Oäi 9 EFdOU V9G ˆ

i ˆ 1 2 . . .  k k
iˆ1

äi < kä:

O6:129 O6:169

k  >% O6119  O6129 "& *;& IOUL V9 > iˆ1 R1 Oä i 9 ! -& R1  -;&@ [   k
ä1 ‡


‡ ä k R1 Oäi 9 > k
R1 k iˆ1 > kR1 Oä9

3.1 The rate-distortion function

:3

>+ O6169  *& ,- * R1   &-&% ,!- , ä &-& Rk Oä9 ˆ IOUL V9 > kR1 Oä9  $;& *& $$& &H!+ & pOvju9 >&  &  & & -*& * -*&;& R1 Oä9  &J& pOvju9 ˆ k & +& & -*&  &J& iˆ1 pOv i jui 9   &+  ;&,+ * *& * EOd9 < kä  IOUL V9 ˆ kR1 Oä9 O&& > 669 *! Rk Oä9 < kR1 Oä9  "&  *&& 62  $;& h Corollary ROä9 ˆ R1 Oä9 ˆ Proof * ,"  O679

fIOU L V 9D EOd9 < äg:

&&+ ,

*&&

62  &J O669  h

& ! " $!& ,  %&& &-$ , *& ,!- ROä9 /& &+ #" * ROä9   &-&% -;&@ [ ,!- , ä > ä   *& -;&@+  $&  &&+ O&& $$&@ 9 * ROä9  -!! , ä . ä  ROä9   -!!  ä ˆ ä   >! "& &;& *& $, , * ,-  > 6. !*& & "& * " *" * ROä9 ˆ < ,  !,J-&+ %& ä && , "& &J& ä @ >+
ä @ ˆ  pOu9dOu v9 O6:1.9 v

u

*& ROä9 ˆ < ,, ä > ä @   && * >&;& *  & -*& "*-* $ &;&+ $! u && -+   v , "*-*  pOu9dOu v9 ˆ ä @ " *;& IOU L V 9 ˆ <  EOd9 ˆ ä @  * *" * ROä9 ˆ < , ä > ä @ ;&&+ , ROä9 ˆ


v

pOv9
ä

@

ˆä

u

@

>+ O61.9 *! , ROä9 ˆ ä @  -& ROä9  &-&%  -;&@ [ , ä > ä    - , ä > ä @   ," * ROä9  -+ &-&% , ä  < ä < ä @ O&& > 2)9    * %& ROä9  %;& >+

(
 6( *"&;&   *" * *&&   &&  , %&&+  ! % * ä  ˆ  u *&& &@  & & & + > , "*-* dOu v9 ˆ  6:9 "&;& ,    ! % * &-* " , *& @ D *  & & HO p9 HOä9 * "& >! ! !  >& *& ;!& , ROä9 , < < ä < p  $;& * "& ! $!-&  & -*& "* EOd9 ˆ ä IOU L V 9 ˆ HO p9 HOä9 *& >& "+   *   &J&  NN>-#"II & -*& *   %;& *&  $>> & pOujv9 >+ pOujv9 ˆ ä , u 6ˆ v ˆ 1 ä , u ˆ v *& >-#" & -*&  *"  % 6. &+ !-*  & -*& " *;& EOd9 ˆ ä HOU jV 9 ˆ HOä9 "&;& "& ! #& !& *   $>&  -*& á ˆ PfV ˆ & 1 1 1 P V ˆ ‡2jU ˆ <  P V ˆ 2jU ˆ <   d & & &-D AU ˆ AV ˆ f+ I &H! + O-+  *&& 129 HOU jV 9 < ä %Or 19 ‡ HOä9 &-& ROä9 > % r ä %Or 19 HOä9  *" *& $$& &H!+ ! & < < ä < 1 1=r  &J&  & -*& >+  1 ä , v ˆ u ä pOvju9 ˆ , v 6ˆ u: r 1

*&   $& --! %;& EOd9 ˆ ä IOU L V 9 ˆ HOV 9 % r HF1 ä ä=Or 19 . . .  ä=Or 19G ˆ % r ä %Or 19 HOä9

HOV jU 9 ˆ h

(.

Discrete memoryless sources and their rate-distortion functions 3.2 The source coding theorem

*& & ,!- "*-* " !& ,  $!&+ *& - $ , ;&"  &- 61 *  >&!,! - !-*&&%J--& "*-*  *D ROä9 is the number of bits needed to represent a source symbol, if a distortion ä is allowable *!  !-& + > - >& NN- $&&II  ROä9 >L -& ROä9 &-&&  ä -&& & - $&  $>&  ä -&&  * & NN& *&+II   & & -& NN- $& *&+II

 && "*+ ROä9 %* *;& * %J--& -& *& ,"% ! & OU1  U2  . . .  U k 9 ˆ U &$&& *& J k + > & & >+  -&   & ! !$$& * *&& k + > & NN- $&&II  n > OX 1  X 2  . . .  X n 9 ˆ X  *    &*" $>&  & -;& , X k & + > OV1  . . .  Vk 9 ˆ V !-* * k EFdOU  V i i 9G < kä & *&& --! -&   &>&  & iˆ1 * *& n > X 1  . . .  X n &$&& *& k !-& + > U 1  . . .  Uk "*  ;&%&  < ä *& &*$ >&"&& U X V - >& #&-*&  *"  % 6) " "& #" ,
H O669  O679 * IOUL V9 > Rk Oä9 > kROä9L , *&& 17 * IOUL V9 < IOXL V9L , *& &J , IOXL V9 O&&
H O1.99 * IOXL V9 < HOX9L  , *&& 11 * HOX9 < n >  >% *&& &! "& *;& kROä9 < IOUL V9 < IOXL V9 < HOX9 < n *  O $& % & >& 29 n > ROä9: O6:1)9 k

*&  n=k  O61)9 &$&& *& ! >& , > $& !-& + >  *& >;&  - $& -*& & *! "& &&  &&+ *  & ROä9 > & &&&  &$&&  !-& + > , *& ;&%&  ! >& < ä *& source coding theorem  >& $;& >&" & *   -& &&  & * ROä9 > & &&&  "& ! &J&  !-& -& !-& -& , &%* k   !>& , A Vk  *   & C ˆ fv1  v2  . . .  v M g , destination &H!&-& , &%* k  rate  &J&  >& R ˆ k 1 %2 M  &-* !-& &H!&-& u ˆ Ou1  . . .  uk 9 , &%* k & f Ou9 >&  -&" v i "*-*  NN-&II  u  *& && * dOu f Ou99 < dOu v j 9

j ˆ 1 2 . . .  M:

Figure 3.6 %&& - $& -*& &

O6:1:9

3.2 The source coding theorem

*& average distortion , *& -& C  &J&  >& 1
dOC9 ˆ pOu9dOu f Ou99 k k

(7

O6:1(9

u2 A U

"*&&  O61(9 pOu9 ˆ pOu1 9 pOu2 9 . . . pOuk 9  *& $>>+ * *& J k + > & & >+ *& !-& " >& u1  . . .  uk  , M < 2 n  !-*  !-& -& - >& !&  &;&   - $& -*& & , *&  &$-&  % 6)  ,"  &-* , *& M !-& -&" v i %  - >+ n!$& xOv i 9 ˆ Ox1 Ov i 9 . . .  xn Ov i 99 -& M < 2 n  * " >& $>& *& !-& &H!&-& u ˆ Ou1  . . .  uk 9  *& &$&&& >+ *& n > x ˆ xF f Ou9G  *& & &H!&-& v  #&  >& *& -&" f Ou9 O-& *& $$% v ! xOv9  &  & f Ou9 - >& !H!&+ &-;&& , xF f Ou9G9   -& * *& ;&%&  , *& -*& &  &@-+ dOC9  &J&  O61(9  *& - $&  n=k  ?! d%2 M e=k Example 3.1 O-!&9
;&+*%  *&  &   *& $&;! $$& -& ,
@ $& 61 O$$ :7 (19 &@-&$ * " "& $&-4& *& !-& -  p ˆ q ˆ 12 /& -&  !-& -& , &%* : "* 1) -&"  &+ *& 1) " , *& O: .9  % -& &->&  *& !- O&& $ . ,,9  " *" *&& * &-* , *& 12( >+ ;&- , &%* : ,,&&    & $ ,  & -&" &-&   1 12( 1) 1 dOC9 ˆ ˆ : : 12( (

*& - $&   n=k ˆ .=: ˆ  61.9 %;& 1+ CL T  *& & $+ &$&&& *& >+ &J O61(9 , dOC9 1
dOC9 ˆ pOu9dOu f Ou99 k u ˆ

1
1
pOu9dOu f Ou99 ‡ pOu9dOu f Ou99: k u2S k u2T

O6:2+ * *& !-& &H!&-& " >& $+ &$&&& >+ C *  PfdOu f Ou99 . kä0g " dOu f Ou99 " >& %&& * kä0 ,, dOu v i 9 . kä0 , &-* i ˆ 1 2 . . .  M O&& O61:99 &-& , "& &J&  *&* ,!-  1 , dOu v9 < kä0 ÄOu v9 ˆ O6:229 < , dOu v9 . kä0  *& &-& !  O6219 >&- & *& !&-& ! u pOu9 F1 ÄOu v1 9G . . . F1 ÄOu v M 9G   , "& &J& KOC9 ˆ


u

pOu9

M 

F1

ÄOu v i 9G

O6:269

iˆ1

*& & & O6219 >&- & dOC9 < ä0 ‡ B
KOC9:

O6:2.9

 ;&" , O62.9 ! $, " >& - $&& , "& - J  !-& -& C , &%* k "*   2bkR9c -&" , "*-* KOC9 , Oä9 ä0 9=B /& "  >& >&  J !-*  -& &-+ >! "& " >& >&  &!-& *& &@&-& , !-*  -& &-+ >+ & , random coding  *& " "& " ;&%& KOC9 "* &$&-   -& $>>+ >! ;& *& & ,  $>& !-& -& , &%* k "* 2b kR9c -&"L * ;&%& " >& *"  $$-* <  k ! 1 *! , !,J-&+ %& k *& ;&%& " >& ,Oä ä0 9=B , "*-*  ," *  & & $-! !-& -&  * KOC9 , Oä ä0 9=BL * -& " ,+ *& --! , *&& 6.  " ! #  *& ;&%& KOC9 ;&  !-& -& , &%* k "* M ˆ 2b kR9c -&"  , -!& "& ! $&-,+ *& $>>+ >! "& & ;&%% "* &$&-  *& %* -*-& ! !  >& -&+ &&  *& >! , *&  ;>& V   & -*& * -*&;& ROä9 *! , *& & , *& $, & pOu v9 &&  $>>+ >!  AU 3 AV -*&;% ROä9 *  IOU L V 9 ˆ ROä9 EFdOU L V 9G < ä:

*&

% >!  AU  AV & %;& >+

O6:279 O6:27>9

((

Discrete memoryless sources and their rate-distortion functions
pOu9 ˆ pOu v9 O!-& -9 v

pOv9 ˆ




pOu v9:

u

/& &@& * $>>+ >!  $ Ou v9 ˆ Ou1  . . .  uk  v1  . . .  v k 9 , A Uk 3 A Vk >+ ! % * *& !-&  & -*& & & +& *  >+ &J% pOu9 ˆ

k 

pOui 9

iˆ1

pOvju9 ˆ

k 

pOv i jui 9

O6:2)9

iˆ1

,

"*-* ," pOu v9 ˆ

k 

pOui  v i 9

iˆ1

pOv9 ˆ

k 

pOv i 9:

O6:2)>9

iˆ1

*& $>>+ % &  *& & ,  !-& -& , &%* k "* M -&" * "& "  *& & * %  *& -& C ˆ fv1  . . .  v M g *& $>>+ pOC9 ˆ

M 

pOv i 9

iˆ1

"*&& pOv i 9  %;& >+ O62)>9 O * $>>+ % &   & & &->& >+ +% *& !-& -&  -*&  NN II --%  *& $>>+ >! pOv9:9 &-% *& &J , KOC9 O&& O62699 "& >&%  - $!&  ;&%& EOK9D

3.2 The source coding theorem EOK9 ˆ


v1 :::v M

ˆ




pOv1 9 . . . pOv M 9

u

ˆ

M


pOu9

v1 :::v M iˆ1




pOu9

u


pOu9 pOv9F1

u

v2 A Vk

F1

ÄOu v i 9G

iˆ1

pOv i 9F1




M 

(3

ÄOu v i 9G

ÄOu v9G

M

:

O6:2:9

O *&  &$ >;& &&  *& ,- "*-*  >;! ,  >-!& >+  * , f Ox9   ,!- &J&   J& & A *&  M


f Ox9 ˆ ... f Ox1 9 . . . f OxM 9:9 x2 A

*& & !

x1 2 A

 O62:9 
pOv9F1

xM 2 A




ÄOu v9G ˆ 1

v

pOv9ÄOu v9

v

  O62:9 >&- & EOK9 ˆ


u

 pOu9 1




M pOv9ÄOu v9

:

O6:2(9

v

O  * $ *& && *! >& >&  && &-+ * O62(9 &$&& *& $>>+ *  !-& &H!&-& " >& NN$+ &$&&&II >+  !-& -& v1  . . .  v M -*&  NN II9 *& &@ &$  *& $,  *& &  , *& & !  O62(9  * & &J&  1 , dOu v9 < kä 0  IOuL v9 < kR 0 Ä< Ou v9 ˆ < *&"& "*&& IOu v9 ˆ %2 F pOvju9= pOv9G *& , O6229 Ä< Ou v9 < ÄOu v9  

pOv9Ä< Ou v9 < pOv9ÄOu v9: O6:239 v

v

, Ä< Ou v9 ˆ 1 *& IOu v9 ˆ %2 F pOvju9= pOv9G < kR0   pOv9 > 2 kR 0 pOvju9 &-&

pOv9Ä< Ou v9 > 2 kR 0 pOvju9Ä< Ou v9: O6:6% O6239  O66* &   O66.9 $$-* <  k $$-*& J+  & * -& M ˆ 2b kR9c  R9 . R0 O&& O61399 &@$O 2 kR 0
M9 , &@$F 2 kO R9 R 0 9 1 G $$-*& < ;&+ $+ &- -& * 1 Ä< Ou v9  1 ,, &*& dOu v9 . kä 0  IOuL v9 . kR0  
pOu v9F1 Ä< Ou v9G uv

< PfdOU V9 . kä0g ‡ POIOUL V9 . kR 0g

O6:679

*& $>>&  O6679 >&% #& ;& *& OU V9 $-& "*& $>>+ >!  &->&  O62)9 ! dOU V9 ˆ

k


dOUi  Vi 9

iˆ1

 *& ! , &$&& &-+ >!&  ;>& &-* , "*-* * & EFdOU  V 9G < ä , ä0 >+ O627>9  O6139   >+ *&

Problems

31

"&# " , %& ! >& O&& $$&@ 9 *& J $>>+  O6679 $$-*& <  k -&&  + IOUL V9 ˆ

k


IOUi L Vi 9

iˆ1

O&& &  $ 6:9   ! , &$&& &-+ >!&  ;>& &-* "* & IOU V9 ˆ ROä9 , R0 O&& O6279  O61399   >+ *& $$- , *& "&# " *& &- $>>+  O6679 $$-*& <  k -&&  >%  *&& ,- "& && * *& !$$& >!  EOK9 %;& >+ O66.9 $$-*& & !& *& ,- * ROä9 &&  >& -+ -;&@9 *" * *& k & & -*& &J&  *& & , *& $, ,

*&& 62 * EFdOU9G < kä  IOUL V9 ˆ kR1 Oä9

*& >?&- , * $>&  , +!  $;& * ROä9  -!!  ä ˆ ä    * >+ !$$+% *& &  *& ,"% %! & & Q1  Q2  . . . >&  &H!&-& , -*- -& &->% *&   $>>& , & -*& -*&;% ROä1 9 ROä2 9 . . .  "*&&  n!1 ä n ˆ ä   *& *&&   !>&H!&-& Q n1  Q n2  . . . "*-* -;&%&   -*@ Q *& & -*& -&$%  Q * EFdOU  V 9G ˆ ä   IOU L V 9 ˆ  k!1 ROä n k 9 *! ROä  9 <  ä!ä‡ ROä9   R  -!!  ä ˆ ä    $ (< "& *"& *  !,J-& - , RO ä  *&& &@  & -*& pOvju9 -*&;% ROä9 !-* * *& -*- @ QOu v9 ˆ pOvju9 * *&  & + &+  D & QOðOu9 rOv99 ˆ QOu v9 ,  u 2 U  v 2 V  F HintD , Q< Ou v9 &->&  & -*& -*&;% ROä9 &J& Qi Ou v9 ˆ Q< Oð i Ou9 r i Ov99  *" * Qi &->& -*& -*&;% ROä9 , i ˆ 1 2 6 . . . *& &J&  n & 1 QOu v9 ˆ n 1 iˆ< Qi Ou v9 "*&& n  *& & -  !$& , *& & , ð  rG 3.7  *&  %&& ! *& - $! , ROä  9     $& &

*& ,"% &!  ,& *&$,! *"&;& '&,+ *   -&-  &-* u 2 AU  & BOu9 && *& & , NN>& & &$&& ;& , uII *  BOu9 ˆ fv 2 AV D dOu v9 < dOu v99 ,  v9 2 AV g *& *&& &@  >-#" & -*& pOujv9 -*&;% ROä  9 !-* * pOujv1 9 ˆ pOujv2 9 "*&&;& v1  v2 2 BOu9

$$+ * &!  *& $>& &;&  *&  @

< D ˆ 1 1

, J% ROä < 1 1

 9

, *& !-&

1 1 1 6 6 6

 1 < : 1

3.8 &  J@& !-& p ˆ O p1  p2  . . .  pr 9   @ D "* ^ "*-*  & ,!- ROä9 &  &"  @ D , & , D >+ %  - wi  *& i* " *  ^ j9 ˆ dOi j9 ‡ wi  *" * *& &" & ,!-  dOi  ^ ROä9 ˆ ROä w9 "*&& w ˆ i wi pi  & * &!  *" * *&&   &&  , %&&+  ! % * ä  ˆ !  ROä99 !$$& *&  @ D * *& $$&+ * &-* -!    r $& ! , Od 1  d 2  . . .  d r 9 &J& á i d i ˆ äg O&& > 1(9 ;& * ÖOä9 ˆ @f HOá1  . . .  á n 9D iˆ1 ROä9 > HOU 9 ÖOä9 "*&& HOU9  *& !-& &$+ >+ ;&,+% *& ,"% &$D O9 ,  OU  V 9   & -*& -*&;% ROä9 ROä9 ˆ HOU 9 v pOv9 HOU  jV ˆ v9 O>9  , äOv9 ˆ u pOujv9dOu  v9 *& HOU jV ˆ v9 < ÖOäOv99 O-9 v pOv9ÖOäOv99 < ÖO v pOv9äOv99 < ÖOä9 3.11 O!9 , " D * *& ,!*& $$&+ * &-* row   $& !

Notes

3.12

3.13

3.14 3.15 3.16 3.17

36

 , &;&+ *& " *" * ROä9 ˆ HOU 9 ÖOä9 , ä  < ä < ä @ , *& !-&  + &- Oi9 & "  I "* $! $*>& AOi9 U  !$! $*>& A V   i  ,!- d Ou v9 i ˆ 1 2 *& product * !-& $*>& O29 O19 O29 AO19 @ "* && U 3 A U  & $*>& A V 3 A V    O19 O29 dFOu1  u2 9 Ov1  v2 9G ˆ d Ou1  v1 9 ‡ d Ou2  v2 9 O*+-+ * -&$  " &$&& $& !-&9 *" * *& & ,!- , *& $!- !-&  ROä9 ˆ ‡ fRO19 Ot9 ‡ RO29 Oä t9g "*&& RO19  RO29 & *& & ,!- , !-& 1  2 &$&-;&+ *! *" * *& ROä9 -!;&  >& >+ % >* -& , *& -!;& RO19  RO29  $  *& " -!;& *;% *&  & $& O&! !&  *9 O&,&- -&  *& && &-: 9 , q    ! >&  , AU ˆ AV ˆ f+ dOu v9 ˆ fju vj jq u ‡ vjg , q ˆ 2t 2 ‡ 2t ‡ 1 -& * !-& -& , &%* k ˆ 2 "* M ˆ q -&"D C ˆ fOv1  v2 9D v2  O2t ‡ 19v1 O  q9g *" * *& $*&& , ! t ! *&& -& " - $&&+ -;& *& & , q 2 $ Ou1  u2 9 "*! ;&$ O $*&& , ! t ! Ov1  v2 9  *& & fOu1  u2 9D dOu1  v1 9 ‡ dOu2  v2 9 < tg:9 *" * *& ;&%&  , * -&  16
O2t 6 ‡ 6t ‡ t9=O2t 2 ‡ 2t ‡ 19 '&,+ * dOC9 ˆ 1< 3 , *& !-& -& ,
@ $& 62 O $ (79 *" * *&& 6. &  !& , --! O9  &$-& >+ NNM < 2,b kROä9c IIL  , ä . ä   *  &  !& , O9  !-*%& >! O>9  &$-& >+ NNdOC9 < äII ;& * , < < x y < 1 M > & *" +! -! &@$-+ -*&;& "*  $ & >+ *& !-& -% *&& O *&& 6.9  ä ˆ ä @ 

Notes 1 O $ :79 &-*-+ dOu v9  -&  single-letter  &!&  %!*  ,  &!& * & &J&  -& k!$& Ou v9 ˆ OOu1  . . .  uk 9 Ov1  . . .  v k 99 &-+ *& * >+
H O619 2 O $ :)9 * &@ $&  !&  * O&& F27G $$ 2.)B2.:  % 2  $ 2)29  &->% * *& $&-! !-& *& "& NN. . . *& F!-&G $*>& - , *&& $>& &% 1 & , " && 12  ‡12 & %* *& *;& *& @ *  *"  F
@ $& 62GII 6 O $ :)9 *& &J * ," O
H O669  O6799 - >& %;& ,  >+ + !-&L &&
H O769  O7.9 . O $ ::9 && & 6 *$& 2 7 O $ :(9  ;&" , *&& 62 *& && + "& "*+ *& ,!- Rk Oä9 "&& !-&  *& J $-& *& &  ",  ,  + !-& "* & + &J O669  O679 #& &&  +$-+ Rk Oä9   &-&%

3.

Discrete memoryless sources and their rate-distortion functions

,!- , k , J@& ä O"&;& "& *  !+ !-& "* & +  * >#9 &- O &  $ , ! $!$&9 *& &J %;&  $,+ *& $, , *& -;&& , *& !-&-*& -% *&&  O&& *$& 7 &$&-+
H O71 +   "*  %    k," , *& !-& $>>& & &&D p1 < p2


< pr  &J& Sk ˆ iˆ1 pi  ä k ˆ S k 1 ‡ Or k9 pk   H k ˆ HO p k‡1 =O1 Sk 9 . . .  pr =O1 Sk 99 *& , ä k 1 < ä < ä k  ROä9 ˆ O1 S k 1 9 O H k 1 HOOä S k 1 9=O1 Sk 99 Oä Sk 9=O1 Sk 9%Or k 199 , k ˆ ?&- && &&# $ F1.G $$ 6!&    ;>& "* & <  ;-& ó 2  * -*&  ,& &$-&   % .1 !*& & *&&   NN-II -& "* &-* $! xL   bOx9 ˆ x 2   * &- "& * *" * *& -$-+B- ,!- , * -*&  %;& >+ CO â9 ˆ 12 %O1 ‡ â=ó 2 9:

O.:19

0!  , *& -&& -*& , *$& 2 "& * *" * CO â9 &$&& *& @ ! &  "*-* *& -*& -   &,&& ,  , *& ;&%& $! -  &-&  >& < â

Figure 4.1 *& ! -*&

37

3)

The Gaussian channel and source

&,& &;% , ! O.19 *"&;& "& $!&  %;&  >&, ,  &-$ , *" !-*  -*& %* &  $--& !$$& "& "*     &H!&-& , n & ! >& x1  x2  . . .  xn , & $  *&  T &-  & &H!&   * >+ -;&% *& xi I   -!! ,!- ,  & xOt9 O &J&& &I ! & xOt9 &$&& *& ;%& -  1* 9 & "+   *   J n ,!- ö i Ot9 i ˆ 1 2 . . .  n "*-* & *   *& &; F& xi & &-;&>& , xOt9D T xi ˆ xOt9ö i Ot9dt:

O.:.9

iˆ1


&-!& , * + O.29  iˆ1 x i  &-& "& ! *;& n 1
PT  x2 < n iˆ1 i n

O.:79

"*-*   + * *& $! ;&- x ˆ Oxp . . .  xn 9 - >& >+L  1   ! & "* 
!-& $*&& , ! PT 

*& - $-  * "*& xOt9   & *& &-&;& % " ,& >& , *& , x^Ot9 ˆ xOt9 ‡ zOt9 "*&& zOt9   & # , & $-& -  +$& , &  0* O*& 9 & "*-*  -!& >+ *& *&  % , *& &&-  *& &-&;&  * -&   &>&  & zOt9   "*& ! & $-&L , ! $!$& * & * *&&   ! >& N & T O-, O..99 < x^Ot9ö i Ot9dt *& " > *& & & x^i ˆ xi ‡ zi   ! +D "&   x ˆ Ox1  . . .  xn 9 "*&& x ! ,+ O.79  &-&;& ^x ˆ Ox1 ‡ z1  . . .  xn ‡ zn 9 "*&& z1  . . .  zn & &$&& &

4.1 The Gaussian channel

3:

& ! , -!& *  ?! *& ! && >+ *& ! -*& !-&  *& >&%% , * &- "* & ;-& ó 2 ˆ N < =2  $! - ⠈ PT =n

--%  O.19 *& -$-+ , * -*&  12 %2 O1 ‡ 2PT =nN < 9 > $& + > , " "& &J& *&   NN>"*II2 >+ W ˆ n=2T  >&;& * "& &  % n=T ˆ 2W + > $& &- *& -$-+ >&- & C ˆ W %2 O1 ‡ P=N < W 9 > $& &-

O.:)9

 ! O.)9  *I , ! &@$& , *& -$-+ ,  >  & $"& & ! -*& & * , W  P=N<  *& &! % NN"&>II ! -*& * -$-+ Cˆ

1 P  2 N
 $& &-

O.::9

& ! " &!  *&  --& , * &- *& &; , , ! O.19

%!+ "*
H O269 & ! &J& *& n* -$-+B- ,!- Cn O â9 , *& ! -*& >+   n
2 Cn O â9 ˆ !$ IOXL Y9 D EOX i 9 < nâ  O.:(9 iˆ1

"*&& *& !$& !  #& ;&  $ X ˆ OX 1  . . .  X n 9 Y ˆ OY1  . . .  Yn 9 , n &  ;&- !-* *D X *  -!! &+ ,!- pOx9 n
iˆ1

EOX 2i 9 < nâ

Yi ˆ X i ‡ Z i 

i ˆ 1 2 . . .  n

O.:39 O.:3>9 O.:3-9

"*&& Z 1  Z 2  . . .  Z n & &$&& O, &-* *&  , *& X i I9 & & *& ;& -$-+B- ,!- , *& ! -*&  " &J&  CO â9 ˆ !$ n

/& 

1 Cn O â9: n

&&+ $;& *& ,"% *&& 

O.:1& + & !-& ,+% O.39  O.3>9

*& >+ O.3-9 *& ? &+ , X  Y  %;& >+ pOx y9 ˆ pOx9 gOz9 "*&& z ˆ O y1 * 

x1  . . .  yn

xn 9  gOz9  *& ? &+ , Z 1  . . .  Z n 

1 gOz1  . . .  zn 9 ˆ &@$ O2ðó 2 9 n=2

   z2i  2ó 2

O-,
H O12:99 & Ai ˆ EOX 2i 9 -& X i  Z i & &$&& EOY 2i 9 ˆ EOX 2i 9 ‡ EO Z 2i 9 ˆ Ai ‡ ó 2    >+ *&& 111  n 1=n  n 2 hOY9 < % 2ðe OAi ‡ ó 9 : O.:119 2 iˆ1 n 2 2 " >+ O.3>9 iˆ1 OAi ‡ ó 9 < nO ⠇ ó 9   >+ *& *& &-B %& &- & &H!+ *& $!-  O.119   %& * O ⠇ ó 2 9 n  &-& n hOY9 < % 2ðeO ⠇ ó 2 9: O.:129 2 + *&& 11! hOYjX9 ˆ hOZ9 ˆ On=29% 2ðeó 2 O&&
@ $& 116   > 12:9 *! IOXL Y9 ˆ hOY9

hOZ9

  n â < % 1 ‡ 2  2 ó

O.:169

"*-* $;& * Cn O â9 < On=29%O1 ‡ â=ó 2 9  $;& *& $$&  &H!+ & X 1  X 2  . . .  X n >& &$&& & & *& O.39  O.3>9 " >& J&  Y1  Y2  . . .  Yn " >& &$&& & & O-, > 1239  IOXL Y9 ˆ hOY9 hOZ9 ˆ On=29%O1 ‡ â=ó 2 9

* - $&& *& $, , *&& .1 h Note Another way to de®ne Cn O â9 is to replace condition (4.9a) with X ! & + ,&+

+ ;!&

In Prob. 4.11 it is shown that these two de®nitions are equivalent.

O.:399

4.2 The Gaussian source

33

*& J &!  * &-  *& -% *&& , *& ! -*&   - $&&+ %!  *& -&& -% *&& 2.D

O-*&9 -& , &%* n  ?!  & , M n & ;&- fx1  . . .  x M g ,  -&" x ˆ Ox1  . . .  xn 9   & ;& *& -*&   &-&;&  y ˆ x ‡ z "*&& *& - $& , z & &$&& !  ;>& "* & <  ;-& ó 2  &-% !& , !-*  -&   $$% f , *& & ,  n & ;&- y  *& -& *& & $>>& POi9 E &$&& *& $>>& , &-& & %;& * *& i* -&" " & *  POi9 E ˆ Pf f O y9 6ˆ x i jx i  &S "*&& y ˆ x i ‡ z  >;& Theorem 4.2 Ocoding theorem for Gaussian channels9 Fix â > 2  O-9 POi9 , å for all i ˆ 1 2 . . .  M E Proof --%  *& $&-&% &   $>&  -*&  J&+ >&  ;>& X "* EOX 2 9 < â !-* * IOX L Y 9  >+ -&  CO â9 *& -&$% Y "  >& -&& >! -& IOX L Y 9 ˆ  IOX L FY G9  *& H!4 FY G >&- & J&  J& O&&
H O11:99L  " >& $>&  J  -&&  ;>& FY G !-* * IOX L FY G9 . R *& -&& & +& -*& "*-* *  $! *& ;!& ! & >+ X   !$! *& ;!& ! & >+ FY G "* $! -&  ,+ EOX 2 9 < â *! *  -$-+ %&& * R   *& &@&-& ,  -& ,+% O9 O>9  O-9 ,"  &&+ ,

*&& 2. h 4.2 The Gaussian source

* !-& "*& ,!  &  *& NN-&& & & +& ! !-&II *  !-& $*>& AU *& & ,  & ! >&  *& !-& !$!  && >+  &H!&-& U 1  U 2  . . . , &$&& &-+ >!&    ;>& "* & <  ;-& ó 2  ! >?&-  * &-   - $!& *& & ,!- , * !-& &;&  *& NNH!&&II  -&  "*-* *& & $*>& AV  % *& & , & ! >&  *&  >&"&&  !-& + > u   & + > v  %;& >+

1& & 2 ' 1 % ó , ä < ó 2  2 ROä9 ˆ ä ( < , ä > ó 2 :

O.:1.9

O.:179

  $& O ! , >9  % .2 , -!&  ! !-&  #&+  >& &-!&&    + # , %*&% &@$& & -&  J& ! >& , > & &H!&  &$&&  >+ & ! >& u "* $&,&- J&+   , &&  #" *& &,, >&"&& *& ! >& , > !&  &$&& *& &@$& & !- &  *& &!%   &J&  O.179 ROä9 &  ,- &$&& *&  ! $>& ! >& , > !,J-&  &$&&  ! O;-& ó 2 9  ;>& , *& @ ! $& >& & H!& &  ä * ,- " >& ;&J& >+ *& !-& -% *&& 

*&& .7  >+ *& &! , *$& 7

 &>* O.179 "& J &J& *& k* & ,!- Rk Oä9 , *& ! !-& "* &$&-  *& &H!& & -& >+ O&&
H O6699 Rk Oä9 ˆ , fIOUL V9 D EOkU

Vk2 9 < käg

"*&& *& J !  #& ;&  $ , k &  U ˆ OU 1  . . .  Uk 9  V ˆ OV1  . . .  Vk 9 !-* *D U1  U2  . . .  Uk & &$&&  ;>&

O.:1)9 ;&-

& ! , U  V  %;& >+  -!! &+ ,!- pOuv9

O.1:-9

*& ;& & ,!- ROä9  *& &J& O-,
H O6799 >+ ROä9 ˆ , k

1 Rk Oä9: k

O.:1(9

 *$$&& , -&& & +& !-&  *$& 6  " ! ! * *& J !  O.1(9  &+ -*&;&  k ˆ 1 &,& $;% * ,- *"&;& "& %;&  -& , &@$& , R1 Oä9 "*-*  , -!& *& ;!& , ROä9  "& Theorem 4.3

& 2 ' 1 % ó 2 R1 Oä9 ˆ ä (
ó 2 :

Proof -# ä å . & !-* *D

IOU L V 9 , R1 Oä9 ‡ å U  ! "* EFOU

& 9 O.:13-9 O.:139

* *  $>& ," , &J O.1)9  *& - O.1:9 + O.13-9  O.139  ä> pOu v9Ou v92 u v  ˆ

 pOv9 pOujv9Ou

v92 u v

O.:2+ O.2 12 % ó 2 =äL >! -& *  !& ,  å . & ! "* ;-& ó 2 å *& % O.1:9  J&   R1 Oä9 < IOU L V 9 ˆ hOU 9 hOG9 ˆ 12 % 2ðeó 2 1 2 å9 ˆ 12 %F1 ‡ å=Oó 2 å9G -& *  !& ,  å . &  *&  $ , *& $, , *&& .6  ,& -& *& NN>-#"II & -*& , - $!% ROä9 >&-!&  - >& &$-&   % .6  $& , *& ,- * U  *& !-&  V  *& &M   NN,"II ;& , * & -*& && > .1(9

4.2 The Gaussian source

1-#" & -*& ,  ! !-&

/& - " - $!& ROä9 Theorem 4.4 Rk Oä9  ˆ kR1 Oä9 @ 12 % ó 2 =ä < 

for

all

k,

and

so

ROä9 ˆ R1 Oä9 ˆ

Proof -# å . &  $ ,  ;&- ,+% O.1:9  IOUL V9 , Rk Oä9 ‡ å: O.:279 k

*& >+ *&& 1( IOUL V9 > iˆ1 IOUi L Vi 9 , "& &J& ä i ˆ EFOUi Vi 92 G *& >+ *& &J , R1 Oä9 IOUi L Vi 9 > R1 Oä i 9  >+ k k O.1:>9 iˆ1 ä i < kä !*& & iˆ1 R1 Oä i 9 > kR1 Oä9 > kR1 Oä9 "*&&  k ä ˆ k 1 iˆ1 ä i  -& R1 Oä9  -;&@ [  -+ &-&% >+

*&& .6 &-& Rk Oä9 ‡ å . IOUL V9 >

k


IOUi L Vi 9

iˆ1

>

k


R1 Oä i 9

iˆ1

> kR1 Oä9 > kR1 Oä9: -& *  !& ,  å . ó 2 *& $,      &,  > .2& $&-J& "* d%2 M e >  *& &!% $&+ > &H!&   dOC9  Theorem 4.5 Let ROä9 ˆ @ 12 % ó 2 =ä < denote the rate-distortion function of the Gaussian source with respect to the mean-squared error distortion criterion. Fix ä > 9 dOC9 , ä9 Proof , ä > ó 2  *& >+ -*% *& %& -&" < , &%* 1 "& " -*&;& "*  $ & ,  *& *& * ä , ó 2 & OU  V 9 >& *& >-#" & -*& -*&;% ROä9 &->&  *& $, , *&& .6L && % .6 *& ä1  ä2  * ä , ä1 , ä2 , ä9: &@ $-# ,!- á  â &-* ! % + J&+ * EfFáOU 9 EfFU "*&&

âOV 9G2 g , ä1  áOU 9G2 g ˆ å

O.:2:9 + ;!& !-* O.:2(9 O.:239

Problems  å ‡ 2 åä2 ‡ ä2 , ä9:

1 .229 /& " -& *& -&& & +& !-& áOU 9 "* & $*>& &H!  *& %& , â   ,!- dOu v9 ˆ Ou v92  -& >+ *&& 17 IFáOU 9L âOV 9G < IOU L V 9 ˆ R1 Oä9 , "& && *& & ,!- , * !-& >+ R  ," , O.2(9 * ROä1 9 < ROä9 , R9:

O.:619

*! >+ *& !-& -% *&& , -&& & +& !-& O *&& 6.9 *&& &@  !-& -& , &%* k , *& !-& áOU 9 ,+%D M < 2b kR9c 

O.:629

dOC9 , ä2 :

O.:669

/& " -& !% * -& , *& % !-& U , u ˆ Ou1  . . .  uk 9   >+ ;&- , &%* k  , v i  *& !-& -&" -&  áOu9 ˆ OáOu1 9 . . .  áOuk 99 *& ku v i k ˆ kFu áOu9G ‡ FáOu9 v i Gk   >+ -*"4I &H!+6 EOkU

Vi k2 9 < EFkU

áOU9k2 G ‡ EFkáOU9

‡ 2EFkU

Vi k2 G

áOU9k2 G1=2 EFkáOU9

Vi k2 G1=2 

O.:6.9

"*&& Vi && *& -&" -&  *&  ;&- U ! , O.239 EFkU áOU9k2 G ˆ kå  , O.669 EFkáOU9 Vi k2 9 , kä2  &-& - >% O.6& "* O.629 $;& *& *&& 

h

EFkU

Problems 4.1 &  -*& "* AX ˆ AY ˆ *& & & , "*-* *& !$! Y  *& ! , *& $! X  $ &$&& &  ;>& Z "*-*  % !, + >!&  12 12  !$$& *& $! X  -&  ,+ jX j < â , â   &%& *" * *& -$-+ O&J& %!+ 
H O.(9 O.39 O.1>+ PE , 2 nO R< R9  O *  %& * *&& .2  *  %;&  &@$- & & , *"   PE - >& &   ,!- , n   "&#&  * R< , CO â9  * , & R< , R , CO â9  &  *" * PE ! <  $>&9 4.3  *" * ,  -& fx1  x2 g "* + " -&" , *& &-& -*& *& & && O
!-& -&9  *& &-&;& ;&- y *& & $>>+  %;& >+ P2 Fx1  x2 G ˆ QOkx1 x2 k=2ó 9 p  1 s 2 =2 "*&& QOá9 ˆ O1= 2ð9 á e ds 4.4 &@ *" *  %&& , *& -& fx1  . . .  x M g  &-& >+ *& && -&" &%+  , POi9 E && *& $>>+ , &-% & , x i   & *& POi9 E
 .. > *& ,"% & & , *& ;&%& ;!& EFPOi9 -& &  &&-& &$&&+ E G "*&& *& -&"    --%  *& >! Pfx ˆ ‡ âg ˆ Pfx ˆ âg ˆ 12D  n 1‡ã Oi9 EFP E G , M
: 2

Problems 4.8 " - $&& *& $, , *& R< *&& 4.9 + *" *D O9 

R< O â9 ˆ % 2

O>9 

R< O â9 1 ˆ : CO â9 2

â!‡1

â!&  "& * !+ *& &,,&- , H!4  *& ! -*& && >+ C O rs9 O â9 *& -$-+ , *& ! -*& , "&  * *& $! X ! &   r - ;!&  * *& !$! Y * >&& H!4&  & ,   s ,,&& ;!&  + "& &J& , &-* n C Onrs9 O â9 ˆ !$fIOX 1  . . .   X n L f OY1 9 . . .  f OYn 99g "*&& &-* X i n - ! &   r - ;!& iˆ1 EOX 2i 9 < nâ Yi ˆ X i ‡ Z i  "*&& 2 *& Z i I & &$&& & &  f  + ,!- "*-* ! &   s ,,&& ;!& *& C O rs9 O â9 ˆ !$ n n 1 C Onrs9 O â9 4.10 *" * C Ors9 O â9 ˆ C 1O rs9 O â9 4.11 *" *  rs!1 C O rs9 O â9 ˆ 12 %O1 ‡ â=ó 2 9 F#&-* , $,D  *" * ,  J@& ;!& , r  s!1 C O rs9 O â9 ˆ C O r19 O â9  %;& >+ *& , ! C O r19 O â9 ˆ !$fIOX L Y 9g "*&& *& !$& !  #& ;&   ;>& X ! %   r ,,&& ;!& EOX 2 9 < â  Y ˆ X ‡ Z "* Z  &$&& & & % *& , ! IOX L Y 9 ˆ hOY 9 hO Z9 O-, > 12:9 *" * C O r19 O â9 < 12 %O1 ‡ â=ó 2 9 &@ & X >&  & & & I 1  . . .  I r >&  $ , *& & &  r &;  , &-* i ˆ 1 2 . . .  r -*& xi 2 I i "* jxi j     $>& &J& *& &"  ;>& X 9 >+ PfX 9 ˆ xi g ˆ PfX 2 I i g i ˆ 1 2 . . .  r *& C O r19 O â9 > IOX 9L X 9 ‡ Z9 "  r ! 1  *& $ >&- &  -&%+ J& hOX 9 ‡ Z9 ! hOX ‡ Z9 ˆ 12 %Oó 2 ‡ â9 >+ &>&%!&I   & -;&%&-& *&&  &-& 

r!1

C O r19 O â9 ˆ 12 %O1 ‡ â=ó 2 9:

4.12  J@& r  s *" *  â=ó 2 !1 C O rs9 O â9 ˆ O% r % s9 4.13 *" * , r > s *& C O rs9 O â9 ˆ C Oss9 O â9 F&!  1 !&   ! 2&+ 0G 4.14 , â=ó 2 ˆ á2  *" * C O219 O â9 ˆ á2 1 gO y9% -*Oá ‡ á y9 dy 2 "*&& gO y9 ˆ O2ð9 1=2 e y =2  *" * ,   á C O219 O â9 ˆ á2 =2 á. =. ‡ &;& * *  ;!+ *&  &  *& !H!4& -$-+ CO â9 ˆ 12 %O1 ‡ á2 9 &-& *& ,# *&& NN   %&  >+ $! H!4 &I *!II O>! && > .3>9 && , á < 1 *&  C O219 O â9=CO â9  "+ > & ó 2 %& ,  i ó 2i  â 4.18 *& >?&- , * &@&-&  , +!  &;&  NN,"II & -*& * -*&;& ROä9 , *& ! !-&  &H!& &  -& &- "*  &&&D U   & & ä < ó 2   V ! >& ,!  * EFOU V 92 G ˆ ä  IOU L V 9 ˆ 12 % ó 2 =ä *" *   $>&  #& V ˆ áU ‡ Z "*&& á   -  Z  ! "* ;-& ó 2  >+ -*% ä  ó 2 $$&+L   #& V ˆ âOU ‡ Z9 "*&& â  -  Z  !

*&& " NN,"II & -*& - >& &$-& #& *D

4.19 &  !-& U 1  U2  . . . "*&& *& Ui I & &$&&  ;>& "* -  &+ ,!- pOu9  ;-& ó 2  *" * *& &  ,!-  , * !-& O&& $ 1 ó 2  O&& $, , *&& .. & & >& * OU V9 ! *;&  -!! ? &+  &J% V  < "  9

Problems

1?&- , * &@&-&   *" * , - O.1:-9 "&& &$-& >+ - O.1:-99 *& ;!& , Rk Oä9 "! >& !-*%&  *& ,"% #&-* ,  $,   ! & * k ˆ 1  ä , ó 2  O *& %&&4  %& k  ä  H!& &+   * " >& %;&9 ! ?>   !$$+ *& & *! & R91 Oä9 && *&  ! $>& ;!& , IOU L V 9 !>?&-  O.1:9 O.1:>9 O.1:-99  *" * R91 Oä9 < 12 % ó 2 =ä & OU  V 9 >&  & -*& -*&;% R1 Oä å9  &->&  *& $, , *&& .6 *& å . &  && - ,!- , V ! % + J&+ + ;!& !-* * EFOU V 992 G , ä *& IOUL V 99 < IOUL V 9 ˆ R1 Oä å9   R91 Oä9 < 12 % ó 2 =Oä å9 -& *  !& ,  å . 12 % ó 2 =ä & OU  V 9 >&  & -*& ,+% O.1:9 O.1:>9 O.1:-99 , "*-* IOU L V 9 , R91 Oä9 ‡ å & V 9 ˆ V ‡ G "*&& G   & & &$&& , V *& *& OU  V 9 & -*& J& O.1:9  O.1:-9  EFOU V 992 G < ä ‡ å -& IOU L V 99 < IOUL V 9  ," * R91 Oä9 ‡ å . 12 % ó 2 =Oä ‡ å9 -& *  !& ,  å R91 Oä9 > 12 % ó 2 =ä 4.22 *" * *&& & ,!- áOu9 âOv9 "*-* ,+
H O.2(9 O.239  O.6!&  ;>& "*  -  &+ ,!- pOu9   -  ,,&& &$+ hOU 9 & f Ox9 >&  >+ &%;& &;& ,!- ,  & ;>& x  &J& *& &  ,!- , *& !-& "* &$&-  *&  &!& dOu v9 ˆ f Ou v9  ROä9 ˆ ,fIOU L V 9g *& J ! >&% #& ;&  $ ,  ;>& OU V 9 "*&& U  >!& --%  *& &+ pOu9 OU V 9 *;&  ? &+ ,!-  EF f OU V 9G < ä O -!+  >& -& "* *& $&& , *$& . "& *! &J& Rk Oä9 %!+ "*
H O.1)9  ROä9 ˆ , k k 1 Rk Oä9 "&;&  "! ! ! * Rk Oä9 ˆ k R1 Oä9  * *& &J , * $>&  &+ *&  &9 &J& öOä9 ˆ !$fhOX 9 D EF f OX 9G < äg ;& * ROä9 > RL Oä9 ˆ hOU 9 öOä9 HintD  *" * öOä9  -;&@ \  ä *& , äOv9 ˆ f Ou v9 pOujv9du IOUL V 9 ˆ hOU9 hOU jV 9   hOU jV 9 ˆ qOv9 pOujv9%  < qOv9öFäOv9G dv

1 du dv pOujv9

  < ö qOv9äOv9 dv < öOä9: 4.24 O!9 /* *&  , > 167 &;!& *& * "& >! RL Oä9  *& ,"% -&D

11
9 pOu9 ˆ e juj= A  f Ox9 ˆ jxj: 2A 2 O-9 pOu9 ˆ O1 ‡ u 2 9 2  f Ox9 ˆ jxj: ð 4.25 + -!-%  $$$& & -*& *" * ROä9 ˆ RL Oä9 , *& !-& , > .2.> 4.26 FNoteD *& &! , * $>& - >& !&   % $  *& !+ , *& & ,!- ,  ! !-& "* & + && &%& F16G *$& .G & k NN$&II ! !-& U1  U 2  . . .  Uk  *  *& Ui & &$&& & & &J& ROä9 ˆ , f OUL J !  ;& *& $ OUL V9 "*  ?  k V9g "*&& *& &+  iˆ1 EFOUi Vi 92 G < kä *" * ROä9  %;& $ &-+ >+   k
1 ó2 ROä9 ˆ % 2i  2 ó ‡ iˆ1 äˆ

k
iˆ1

Oó 2  ó 2i 9

"*&& *& $ && ó 2 %& , <  @ i ó 2i  4.27 '&,+ * *& ! -*&  & +&  *& && , *& &J  $ ..

Notes 1 O$ 3)9  0* & *& $&- &+  N< ˆ kT  "*&& k ˆ 4 0  - ˆ 1:6( 3 1< 26 ?!&=85  T  *& &,,&-;& & & $&!& -!+ *& $&- &+  &$&&  *& ,&H!&-+   %;& >+ 0 PO f 9 ˆ hf =Oe hf =kT 19 "E*&4 "*&& f  *& ,&H!&-+   *&4  h ˆ -#  - ˆ ):) 3 1< 6. ?!&&- *& & * zOt9ö i Ot9dt   !  ;>& , ;-& N< =2  !& + , *& % ö i Ot9 *  ,  &&%+ -J&  ,&H!&-& "*&& *& $$@  PO f 9  N<  ; O&& & 29 "&;& * $$@   +$-+ ; !$  ;&+ *%* ,&H!&-& &&L , &@ $& , f , & , - !-% *& !$! ,  ,  !-& ;&  + -*& *!%*! * -*$& "& * #& *& >-# % *"  % 71  ! $%  '!+ + & - !- +& - >& !>! & !& *& & , % 71 *& &-& >-# &$&&  *&  $-&% O"*-* + -!& H!4 ! &-  "&  -% , & -9 $&, &  *& !-& !$! >&,&     ! & * *&& &@ &%& k  n !-* * *& +& $-&& !--&;& >-# , k !-& + > &$&&+  -;& *&  !--&;& >-# , n -*& $! + > O &@ $& k -! >& #&  *&  ! >& , !-& + > & & !% *& ,& & , *& +&  >! ,  +&  k  !-*  & * *9 *& -*&  ! &  --&$ -*& $! + >  -&& &;   &$&  &  -*& !$! + >  && -&$&-&  *& $!1 *& ! $ * *& -*&  NN-&& &II  &-;& >!  ;&+ -&  NN-!! &II -*& - >& ;&"&  NN-&& &II "* ;&+ &  O-, $$ 3)B3:9 *& NN&-&II >-# &$&&  *&  $-&%

Figure 5.1 %&& -

112

!- +& 

The source±channel coding theorem

116

$&, &  *& -*& !$! >&,& &;&+  *& &   ! & * *& &-& $-&& !--&;& >-# , n -*& !$! + > &$&&+  -;& *&  >-# , k & + > *& & &H!&-& V ˆ OV1  V2  . . .  Vk 9 >&% *& +& I & & , *& !-& &H!&-& U ˆ OU1  U2  . . .  Uk 9 /& " $!&  &;&" *& >- --&$ ,  &  *&+ $$+  *& >-# % , % 71  *& ,"% -! "& * &,&  ;&+ ;%!&  %&& -*&  !-& &L *& && *! >& "& *"&;& * "& *;& $;& *& &H!& -% *&&  + , -& $&- -&  "& -& *& -*& /& & %;& *& -*&I $! $*>& AX  *& !$! $*>& AY L "* &-* $! + > x 2 AX *&&  -&  &%;& & ! >& bOx9 *& NN- , &% xII & %&&+ , &-* x ˆ Ox1  . . .  xn 9 2 A nX  bOx9  *& - , &% x  "& & %;& *& -*& - *  , &-* n "& & %;&  - $>>+ >!2 * " !  - $!& *& >! , *& !$! Y ˆ OY1  . . .  Yn 9 -& "& #" *& $! X ˆ OX 1  . . .  X n 9  &-* $;& &%& n  â > + Cn O â9 ˆ !$fIOXL Y9D EFbOX9G < nâg:

O7:19

 O719 *& !$& !  #& ;&  n &  ;&- X ˆ OX 1  . . .  X n 9 #% ;!&  A nX  ,+% EFbOX9G < nâ  Y  *& n &  ;&- #% ;!&  A Yn "*-* &->& *& -*&I !$! , X  !&  *& $!  * -&@ X  -&  n  & â >& test source *& -$-+B- ,!- CO â9 , *& -*&  &J& >+   1 CO â9 ˆ !$ Cn O â9D n ˆ 1 2 . . . : O7:29 n  %&& CO â9   -!! -;&@ \  -+ -&% ,!- , â > â  ˆ , fbOx9D x 2 AX g *& -!- ,- >! CO â9  *& ,"% O  !!   $& % & >& 29 Channel Coding TheoremD @ â< > â   *& , + $& O â C9 å9 "* â . â<  C9 , CO â< 9 å . & AU   & $*>& AV   , &-* $ Ou v9 2 AU 3 AV  &%;& & ! >& dOu v9 *& NNII * &! , *& !-& + > u ;&  *& &  v & %&&+ , &-* $;& &%& k  &-* $ Ou v9 2 A Uk 3 A Vk  dOu v9 &!& *&   , *& !-& &H!&-& u ˆ Ou1  . . .  uk 9 ;&  *& & &H!&-& v ˆ Ov1  . . .  v k 9  "& & %;& *& !-& - *  , &-* k "& & %;& *& >! , U ˆ OU 1  . . .  Uk 9 *&  ;&- * &->& k !--&;& !-& !$!2  &-* $;& &%& k  ä > ä  &J& Rk Oä9 >+ Rk Oä9 ˆ , fIOU V9D EFdOU V9G < käg

O7:69

"*&& *& J !  #& ;&  $ OUL V9 ˆ OU1  . . .  U k L V1  . . .  Vk 9 , k &  ;&- #% ;!&  A Uk 3 A Vk !-* * *& % >! , U  * %;& >+ *& !-& -  EFdOUL V9G < kä *! "* "& & &+ ;+%  O769  *& - >! , V %;& U  * - >!  -&  k  & test channel *& J !  O769    #& $-& ;&  k & ä >& & -*& *& rate-distortion function , *& !-&  &J& >+   1 ROä9 ˆ , Rk Oä9D k ˆ 1 2 . . . : O7:.9 k  %&& ROä9   -!! -;&@ [  -+ &-&% ,!- , ä . ä   *& -!- ,- >! ROä9  *& ,"% *&&  Source Coding TheoremD @ ä . ä   *& , + $ Oä9 R99 "* ä9 . ä  R9 . ROä9 ,  !,J-&+ %& k *&& &@  !-& -& C ˆ fv1  v2  . . .  v M g , &%* k !-* *D O9 M < 2b R9 kc  O>9 dOC9 , ä9 O&- * dOC9 *& ;&%&  , *& !-& -& C  &J& >+ dOC9 ˆ k 1 EFd  OU9G "*&& , &-* u 2 A Uk  d  Ou9 ˆ fdOu v j 9D j ˆ 1 2 . . .  Mg:9 /& " &!  *& >-# % , % 71 >&;& * -& *& &-&  *& &-& *;& >&& $&-J& "& - ;&" U X Y  V   ;&- , U  &J&  >&   ;&-  *& J $-& 

The source±channel coding theorem

117

*& - $>>& , X %;& U , Y %;& X  , V %;& Y & &$&-;&+ %;& >+ *& &-&I &% *& -*& -  *& &-&I &% /& " &J& *&& $ && * " & ! !-* >! *" "& *& +& $&,  The average costD 1 EFbOX9G: n

O7:79

1 EFdOUL V9G: k

O7:)9

k : n

O7::9

∠The average distortionD äˆ The rate of transmissionD

rˆ

*& $*+- %J--& , *&& $ && *! >& >;!D â & ! *" !-* *& +& -  ! O  $&-*& $! >9L ä -& *" &>+ *& +&   *& !-& !$!L  r &!& *" , O ! , !-& + > $& -*& + >9 *& +&   % ,    %;& !-&  -*& "& "! #&  &%  +& "*   â   ä  %& r >! , -!& *&& & -K-% %

*& ,"% *&&  "*-*  *& -& &! , ,  *&+ & ! &@-+ "*     $>& Theorem 5.1 (the source±channel coding theorem). For a given source and channel: O9 The parameters â ä  r must satisfy r
9 Conversely, given numbers â . â   ä . ä   and r , CO â9=ROä9 it is possible to design a system of the type depicted in Fig. 5.1 such that â < â ä < ä and r > r Proof /& * J $;& O9  >&% >+ & #% * *& $, " &$& +  &J O729  O7.9    *& -% *&& 

*! *& J $ , *& *&& O & & -& *& NN-;&&II  NN"&# -;&&II , *& -% *&& 9 " >& &&  >& !&  &@& &+ %&& --! -&

11)

The source±channel coding theorem

*! !$$& "& & %;&  +& , *& +$& &$-&  % 71 ! J >&;  * *& &H!&-& OU X Y V9 ,  ;&-   #; -* OY &$&  U + *!%* X V &$&  X + *!%* Y9   >+ *& $-&% *&& O
H O11799 IOUL V9 < IOXL Y9:

O7:(9

&@ >&;& * -& EFbOX9G ˆ nâ O&& O7799  ," , O719 * IOXL Y9 < Cn O â9  *& *& * , O729 Cn O â9 < nCO â9   IOXL Y9 < nCO â9:

 -& EFdOUL V9G ˆ kä O&& O7)99  ," , > Rk Oä9  , O7.9 * Rk Oä9 > kROä9 &-& IOUL V9 > kROä9:

O7:39 O769 * IOUL V9 O7:1% O7(9 O739  O719 *L >! "* *& *&$ , *& -*& -% *&&  *& !-& -% *&&     *  $;&  , "  "& ! & * â . â   ä . ä    r , CO â9=ROä9 & %;& ! ?>   &%  &-&  &-&  * *& &!% â ä  r ,+ â < â ä < ä r > r  *& &-*- $& & &&- ! >& â<  ä<  ä1  C9  R9 ,+%D â



< â< , â

O7:119

ä



< ä< , ä1 , ä

O7:129

C9 , CO â< 9

O7:169

R9 . ROä< 9

O7:1.9

r , C9=R9:

O7:179

O * * - "+ >& &  *& -& , > 729 ! &-& " >& &%&  " %&  !-& &-&   -*& &-&  *"  % 72 /& " &->& *& !-& &-& , % 72 --%  *& !-& -% *&&  , !,J-&+ %& k < *&& &@  !-& -& C , &%* k < "* M 1 -&" "*&&

The source±channel coding theorem

11:

Figure 5.2 *& %&& &% , *& &-&.

M 1 < 2b R9 k < c

O7:1)9

dOC9 , ä1 :

O7:1:9



  -& &%& m O >& $&-J& &9 "& #& k ˆ k < m *& !-& &-& , % 72 $ *& !-& &H!&-& U , &%* k  m >-# , &%* k <   !$! *& m !-& -&" -&$%  * &H!&-& , !-& >-# *! *& & && ;&- W ˆ OW 1  . . .  W k 9 " "+ >&  &H!&-& , m -&" , *& %;& !-& -& C  $-! W ! &   M 1m < 2 mk < R9 - ;!& * - $&& *& &-$ , *& !-& &-& O&@-&$ , *& $&-J- , m9

 &->& *& -*& &-& "& J &&  &J& *& worst-case distortion , *& !-& -& C  &-* !-& &H!&-& u 2 A Uk <  & d @ Ou9 ˆ @fdOu v i 9D v i 2 Cg *! , *& !-& &  u  *& !-& &-% $-&  &*" %& *+"& d @ OU9 &$&& *& " $>& &!%  *& "-&  , *& -& C  &J&  >& *& ;&%& ;!& , *  4&   $&+ > >D DOC9 ˆ

1 EFd k
&;& && *&& &   2 mk < R9 - WI  >+ O7229  & * + - -&" ;% - $&&+ &->& *& &-&  % 71 "& *;&  &&+ , O72.9 * r ˆ k=n ˆ k < m=nm > r  , O7219 * n⠈ EFbOX9G < nâ  &   *" * *& ;&%&  ä  < ä    * "& ! , -!& &% *& &-&  % 71  %&& &%  &$-&  % 76 *& -*& &-& !&  *& &-& "*& &@&-&  $ & >+ *& -*& -% *&& L  $ *& + -&" Y ˆ OY1  . . .  Yn 9  & , *& -*& -&" + Z ˆ O Z 1  . . .  Z n 9 + *& !-& &-&  % 76 --&$  $! *& -*& -&" Z  !$! *& &H!&-& V ˆ OV1  . . .  Vk 9 , m !-& -&" -&$%  Z , *&&  & O * -&$&-&  *& & %;& >+ *& !-& &-& , % 729 ,  &H!&-& , m !-& -&" -! *;& %;& &  Z & ! ! & *& !-& &-& !$!  J@& NN! &H!&-&II v< ˆ Ov& AV  ;% - $&&+ &->& *& - !- +& , % 71B76 ! & % #   - $!& *& ;&%&  ä ˆ k 1 EFdOUL V9G

  * "& !-&  &"  ;>& B "*-* & ! "*&*&   *& -*& &-&  % 76 !--&& *!  < , &-& !--&&L ZˆX Bˆ 1 , &-& ,L Z 6ˆ X:

Figure 5.3 *& %&& &% , *& &-&.

The source±channel coding theorem

113

*&D EFdOUL V9G ˆ EFdOUL V9jB ˆ % O7279 O72)9  O72:9 "& > ä ˆ k $ &

O7:2:9 1

EFdOUL V9G , ä  h

Discussion /& ! & *& ;&%& - â  J@& O * *& - ,!- bOx9  &-+ 4&  "*-* -& *& -$-+ ,!- CO â9   - C @  *& -*& -$-+9  !+ *& &,, >&"&& *& & r  *& ;&%&  ä , *& - !- +& , % 71 *& & , *& !-&B-*& -% *&& & ! 4&  % 7. & O9 , *& !-&B-*& -% *&&  * , + &4>& +& *&

12
& J& "* & *;% $ && Or äO1 ‡ 1< 7< 99M

Problems 5.1  &@$& && "*&  &%  +& , - !-% >&; ,  !  $-& ;  >+ + &- -*& "*-*D --&$ 1 1+  $% *& $-&   & , R  $& $& &- O*&  $& & & &9  &-% *&  $& >&,&   !$$& *& - &&  ;&%& &H!& & ,   ä  - $&  ;&%& ,   B  $& +  *& -*& /*-* , *& ,"% *&& & , OB ä R9 &  $-$& &4>&P B T (). 2732 .62
+ + &- -*& "* " > & $>>+ p &,&%  % 7. *" * &;&+ $  *& >!+ ,+% ä ˆ p  &4>& O& * , p ˆ < * $ , *& >!+   -!! 9 *" * *& - â . â   $ O>9 , *& !-&B-*& -% *&& - >& & ;& *" * *& H!+ DOC9 &J& ,  !-& -& 
H O71(9  J& $;& * , &-* v 2 A Vk < *& &@$&- EFdOUL v9G  J& *& *" * * -  J& , *& ! !-& , *$& . O
  &9 * $>& !& *& $>& $, ;;&  >!%  NN,-&II - !- +&  -*&;& "*  $ & >+ *& !-&B -*& -% *&&  !$$& "& "  - !-& *& !$! ,  >+ + &- !-& ;&  >+ + &- -*& "*& " & 1 $>>+  1<  O ! & *&&   - -& "* -*& !&9 !*& ! & *   &H!&  *;& ä < + J &%%  !-& -& , *&  "* dOC9  & $&-& -& *& ,"% J%!&D

&& U   & & "*-* $& *!%*  &;-& O $J&9 "*-* !$&  >+  - ë *&  & & Z &$&& , U  &    J+ *& &!  $& *!%*  &;-& O&!9 "*-* !$& >+ *& - ì *& $! X  *& -*& ! ,+ EOX 2 9 < â *" * *&& &@   ä  - ë ì !-* *D O9 ROä9 ˆ CO â9 O>9 EFOU V 92 G ˆ ä "*&& ROä9  *& & ,!- , *& !-&  CO â9  *&

122

The source±channel coding theorem -$-+B- ,!- , *& -*& &-& *" * *& &H!+ k=n < CO â9=ROä9 - >& -*&;& "* &H!+ "* k ˆ n ˆ 1

Notes 1 O$ 1129 -& * -*& "*-* -*&;!+ &  &&& + >  *&  & & & *! >&+ ! $%  2 O$$ 116 11.9  & , *& %;& &J , CO â9  ROä9  >& ! >%!! "& ! ! & * *& -*&  !-& & stationary *  *& -   &$&  "*& "& >&%    *& !-&I !$!  "*& "& >&%  !& *& -*& 6 O$ 11)9 -!+    &-&+  ! & * *& &-&  *& &-& & && - &;-&  &  $;& *& &H!+ k=n < CO â9=ROä9  *  &H!&  * *& &-&  &-& >& !-* * *& &H!&-& OU X Y V9 ,   #; -*L , &@ $&  &-& !-* * X &$&  +  U >!   & -+ %&&&  ;>&  "& "! >& --&$>& . O$$ 11: 11(9 *& $&-! -!- "& !&  *& $, , *& !-&B -*& -% *&&   "*-* *& &-&  &-& "&& ,-&  !-&  -*& &-&  &-& &&;& - &  - &>+ >& %!& * >+ % *& !-&  -*& -% &$&+ "& + & !$ "*  +& & - $&@ * &-&+ O&& > 7:  7( &%9 "&;& , *& $ , O â ä r9 *%   -& &;&+ !-* $& -*&;>& >+ + &   -*&;>& >+  ,-& +& 

6 !;&+ , ;-& $- ,  &

6.1 Introduction  * -*$& "& >&K+ ! 4&  & , *&  $ &!  ,  *&+ "*-* "& *;&  >&& >&  &  & /& * %;&  $, >! & &,& *& &&& && &&"*&& !!+   &@>#  & &   % $$& , & /& -*&  &- ! & &+  %&&4  &@& , *& " $& , ,  *&+ *I channel coding theorem O *&& 2.   -+9  * source coding theorem O *&& 6.9 /& & &-*   &$& &- 6.2 The channel coding theorem /& && *& *&&

, &,&&-& O&& +  *&&

2.9

Associated with each discrete memoryless channel, there is a nonnegative number C (called channel capacity) with the following property. For any å . < and R , C, for large enough n, there exists a code of length n and rate > R (i.e., with at least 2 Rn distinct codewords), and an appropriate decoding algorithm, such that, when the code is used on the given channel, the probability of decoder error is , å /& * " -!-  %!& ! *!%* *& *&&  $% !  "& % $-& "*&& *& *+$*&& - >& "&#&&  *& --! &%*&& *& $ , && " >& *& $*& discrete memoryless channel, a nonnegative number C, for large enough n  there exists a code . . . and . . . decoding algorithm /& *  >&K+ -! ;! - ;&&  *& -% *&&  126

12.

Survey of advanced topics for Part one

· Discrete memoryless channel *& *&& *  >&& $;& , + *& -*& & &*$ *&  $& &@ $& , !-*  -*&  & , "*-* *& $! $*>& AX  *& !$! $*>& AY & >* &H!   J& >& %!$ A  "*&& *& i* -*& !$! Yi  &&  *& i* -*& $! X i >+ *& &H! Yi ˆ X i ‡ Z i  "*&& Z 1  Z 2  . . . ,  &%-  $-& #% ;!&  *& %!$ A *& -*& -$-+ - >& &J& >+   1 @ IOXL Y9  C ˆ !$ O):19 n X n "*&& *& & @ !  #& ;&  n &  ;&- X #% ;!&  A n   Y ˆ OY1  . . .  Yn 9  *&  ;&- * &->& *& -*& !$! , X  *& $! *  Y ˆ X ‡ Z "*&& Z ˆ O Z 1  . . .  Z n 9 - , *& J n - $& , *& & $-&   &+  && * *& & @ !  -*&;& "*& X  !, + >!&  A n    -& IOXL Y9 ˆ HOY9 HOZ9 O&& > 1169 C ˆ % q

, n

1 HO Z 1  Z 2  . . .  Z n 9 n

"*&& q  *& ! >& , && &  A  ! ! * , n

1 1 HO Z 1  . . .  Z n 9 ˆ  HO Z 1  . . .  Z n 9 ˆ H < : n!1 n n

* H!+  -& *& entropy , *& $-& Z 1  Z 2  . . . :

$, , *& -% *&& , * ;& &%- & -*& - >& %;& % *& & , *& $, "& %;& , *&& 2. *& + ,J-!+ & "*& "& +  & * , &-* å . >+    1  P  IOxL y9 C  . å n $$-*& < , n  %& !&+ *&&   *&&  - I asymptotic equipartition property "*-* & * , &-* å . >+    1    P  HOz1  . . .  zn 9 H <  . å n $$-*& < , n  %& -& IOxL y9 ˆ  % q "*  &&&

HOz1  . . .  zn 9 *  ?!

6.2 The channel coding theorem

127

*&& & + *& -&& -*& "* & + , "*-* *& -% *&& * >&& $;&L -! %& F1:G  /,"4 F2:G , &

*& ! -*& , *$& . - >& %&&4&  ," & AX ˆ AY ˆ *& & ! >&  & Yi ˆ X i ‡ Z i  "*&& " Z 1  Z 2  . . .   &;!&  $-&  "*&& *&& +  >&  & # , $! - *&  !--&,! *&&  , * # --! "*& *& Z i I & &$&& O& *& -*&  & +& >!  -&&9  "*& *& Z i I ,  ! $-&L && %& F1:G *$& :  (

 *& -*& "& *;& -!&  , *;& >&& *&  &  *& && *  %& & , ,  K"  & &- +  & , *&  &&% &@& , *& -% *&&  *"&;& *;& >&&  multiterminal -*& *& && !-* &! !&  * * &, --& -*& "* ,&&>-# *  -*&  "*-* *&  & - J ! *& !$! yi -&$%  * $! xi  * O&& F27G $$ 113B12-#   -&& & +& -*& &  -*%&  -$-+ /*  &    #& - !-  & & * -$-+ !-* && O&& *& &%* $$&  $$ 6:6B.6)  F27G   > 22 , f&D x1 < < 1 1

x2 < 1 < 1

y < ? ? 1

O&& NNPII   $&- &!& + >9   ,!$! *&&!$! -*& * -*&  &&  * -$-+ % 6 "&;& , *& " && & $    &$&& ,  &   !&!+ ;& *& -*& *&+ " &,&& "* &-* *& *!   $ H!& D   %;& $ OR1  R2 9 , &   $>& , && 1  J  -& , & R1  , && 2  J  -& , & R2   , *& &-&;&  J  &-% !& !-* * *& - -&-+ &-& >* &%& "*   

12)

Survey of advanced topics for Part one

 & $>>+  &&P *& "& ! !  >& NN&II ,, *& $ OR1  R2 9 &  *& capacity region , *& -*& "*-*  * -&  *"  % )1  & %&& & +& "$! &!$! -*& *& !  &&+ *&  & *&&   -;&@ &%  *& J H! % -& *& -$-+ &% !-* * >+ &>& - !-  >& , + $ , &  *& &%  +  broadcast channel  & "* & $!  & * & !!+ " !$! %   $>&  $;& *& &@&-& ,  -$-+ &% O! /+& F.3G ,  & && --! , *&& !&  -*&9 · A nonnegative number C
;& ,  -&& & +& -*& "& "  *$& 2 * *& --! , -*& -$-+ - >& ;&+ ,J-! !& *& -*& &?+  *%* &%&& , + &+ &- * C  %;& >+ Cˆ

@ IOX L Y 9

X

*& @ ! >&% #& ;&  O& &9 & !-& X  *  ;&  $>& $>>+ >!  *& -*& $! $*>& AX 

* ; $>&   ; >!   &"* && * %*  J $$& -& IOX L Y 9   -;&@ \ ,!- , X   *&-& + - @ !    %> @ !  % * ,-   % $ %& F1:G &;& " &-  *& $>& , J% -*& -$-+ ,    & &-&+   F23G  *! F62G *;& %;& &,J-& ! &- $-&!& , - $!% C  -*& *& *  I *& --! , -$-+  !!+ !-* & ,J-! , *& &J
H O729   %&&  &;& &,,&-;&  -*& "* ;& &%- & *& --! , C  "& *;& &&  &H!;&  *& --! , *& & &$+ , *& & $-& 

Figure 6.1 *& -$-+ &% ,   $&

!$&--& -*&

6.2 The channel coding theorem

12:

#;   &+  - $!& *& &$+ O&& > 12!  %&& ;&+ &  #"  !&  -*& *& -&$% $>&  *  *& --! , *& -$-+ &%   * "% ;!+ !!-*& · For large enough n  -&& & +& -*& *& $>& , && % *& &*$ >&"&& å *& && &-& & $>>+ R *& && &  *&  & n !-* * *&& &@  -& , &%* n * & *& ?> * >&& *& !>?&- ,  & &@& &+ &;& &&-* !%*+ $&#%  * >&& -;&& * , &-* < < R , C *& >& -& , &%* n  & R *  &-& & $>>+ %;& $$@ &+ >+ PE  2

nEO R9



"*&& EOR9 *& -*&I reliability exponent   -;&@ [ ,!- , R * #  &*% #& *& -!;& *"  % )2  >& & $&-& && >+ PE OR n9 *& & $>>+ , *& >& -& , &%* n  & > R  &J& EOr9 ˆ 

n!1

1 % PE OR n9: n

O -!+ >+ * &;& $;& * *   &@ >!  *& ,"% -! "& " %& * &-*-+  >&% !& * !$$& >!  EOr9 & &+ !$$& >!  EOR9 ˆ  !$ n 1 % PE OR n9 &-9 *!%* EOR9   #" &@-+ , + ;   *&& &@ % !$$&  "& >!  EOR9 "*-*  %&& #  *"  % )6 *! *&&   NN-- &II R-  >;& "*-* *& !$$&  "& >!  EOR9 %&& >! , < , R , R- *&&   %$  &@ $& -&   "* " & $>>+ p ˆ :& #" !$$&  "& >!  EOR9 & $&  % ). O& *& %* & !$$% *& "& >!  % ). * -&$  *& R< -% *&&  > 221B22)9

Figure 6.2 *& $$@ & *$& , *& &>+ &@$& ,   

12(

Survey of advanced topics for Part one

Figure 6.3 *& $$@ & *$& , *& #" >!  EOR9

Figure 6.4 *& >& #" >!  EOR9 ,   "* p ˆ :?&- "& *;& -& NN-% *&+II O &;& *& &- $ , * ># 9 - >& ;&"&   &-* ,  -!-;&  $-- -*& -% *&&  "&;& & , *& &@$- -% -*& & " #" - >& !&  -*&;& ;&+ " & $>>&  & ;&+ -&  -*& -$-+ *& -% *&  *& " * >&& "%  -J-& *& NNPE ! < , 

6.2 The channel coding theorem

123

R , CII --! , *& -% *&& , *& #& , -!-;&&   ,  *& *"&;& * --!  -&    -& >&  ! , &,, * >&& &;&  *& $>& , &%  "*&  *&  &  & &%*&% *& NN*&& &@  -&  &-% %* II $ , *& --! & ! &->&  & , *& &,,  *& J $-&   $>&  &- &   &;&+   - , -&   $;& *& -% *&&   &@ $& *& - , linear codes O&& *$& :9 - -& * & %  *& && , *& -% *&& O&& &% %& F1:G &- )29 -& & -& & ;&+ &+  &-& * *" * &-% %*  , % -& &&  >&  $--+ - $&@ *& -% *&& *  >&& $;& , timevarying convolutional codesL && &+ F21G &- )  : &*& , *&& -& , -& *"&;& *  $-- &-% %*  /* &@-+  "& & >+  NN$--II %* P & ! $ *& ;&" "&+ --&$& >+ - $!& -& O&& &% * $-,    F1G *$& 1 R, such that: O9 PE n , 2 nE c O R9 (here PE n is the error probability of the nth code and Ec OR9 is a monotonically decreasing function of R which is positive for all R , C), O>9 The complexity of encoding and decoding Cn is -# % #& *& & *"  % )7 + -*%  & -&  +  #%  $-! # , !& -& O &&B

16
;& &+I &! %&+  *& *& %$ >&"&& ,  *&+ O"*-* *" "*  *! >& $>&  9  -% *&+ O"*-* *" "*   $--  9 ,!&+ "*& &+I *&&  NN$--II     NN-!-;&II  *& && *  &  & ! *"  J *& $$$& & -& *! *& $>& , J%  &@$-+ -!->& $-- &H!&-& , -& , "*-* PE $$-*& < &@$&+ ,  R , C &  $& ,  && &;& $!-& !-*  *&&   " !& *& !>?&- , ,  *&+  -% *&+ O $>>+ $! >* ! , >!&9 · Converses to the coding theorem + -;&&  *& -% *&& *;& >&& -J&  NN"&# -;&&II  NN% -;&&II >! *  &-!&  &% -& *& % -;&&    $+ *& "&# & /& *;& &+ && O$ 1179 *& "&# -;&&  + * , "& & $  - !-& *& !$! , +  >+ + &- !-& ;&  -*&   & &@-&&% *& -*&I -$-+ *& &!% > & $>>+  >!& "+ , 9 *&  k=n &!& *& & ,  L -&+ , k=n &@-&& C *& ROä9 , 1 "*-* ,-& ä *& > & $>>+ "+ , 1

2

nE A O R9



161

R . C

"*&& EA OR9  Arimoto's error exponent "*-* * *& %&& *$& *"  % ))  *& $&- -& ,   "* " & $>>+ p ˆ &"&& -*& -%  !-& -% * "& *;& & $*4& *& !>?&- , !-& -%  rate-distortion theory  >+  "& % & ,J-! &-& *& !>?&-  -&>+ & ;-&  *  &

Figure 6.6 *& %&& *$& ,  I &@$& EA OR9

Figure 6.7 EA OR9 ,   "* p ˆ &K+ -! converses  *& !-& -% *&&  · Discrete memoryless source *&  ! "+  &@& *& !-& -% *&&   $;&  ,  "& - , !-& * *& -&& & +& & *& *&& *  ,- >&& &@&&  * "+   ; ! >& , !-& && $--+ + !-& "*-* - >& &&   +  $-& . . .  x 1  x<  x1  x2  . . . * >&& !& ,! &+ *"&;& *& & ,!- ROä9 - >& --!& &,,&-;&+ , &;&+ ,&" , *&& !-& & O&& &%& F16G   $ 6.9 /& "  &- )2 * H!&    #" >! *& -$-+ &% , -& !&  -*& !-* &  #" >! !-& -% , !$& !& >! & &! *& &$B/, *&&  && -% , -&& !-&  "* &%  &->& *& &$B/, *&&  "& ! $!&  $ !   $  $- , *& !-& -% *&& O "*-* -&+ *$& 11  &&+ &;&M9 & X 1  X 2  . . . >&  &$&& &-+ >!& &H!&-& ,  ;>& #% ;!&   & A *# , * &H!&-&   ,  !-& &;&  *&  %  ,!- *  #& *& & $*>&  A   &J& dOx y9 ˆ 1 , x 6ˆ y ˆ < , x ˆ y *&  "& "  $ (& &&+ $&,&-+ &$&&& !% + >! HOX 9 > $&  $& * !-*  , -!&   $& -&H!&-& , *& !-& -% *&&    *$& 11 "& " %;& &@$- &-*H!& , % * noiseless coding    -& " & ! !  *& &! , &$  /, & OX i  Yi 9 i ˆ 1 2 6 . . .  >&  $&& &-+ >!& &H!&-& , $ , -&&  ;>& "* -  >! ,!- pOx y9 - 

6.3 The source coding theorem

166

 &$& HOX 9 HOX  Y 9 HOX jY 9 &- *& $>&   J &$&& !-& -& , *& X  Y &H!&-& !-* * *& &!%   &%%>& $&-J-+ "& "  -J%! !-*  *& & &$-&  % )( *& ;&- X  &-&  *& -&" f OX9 "*-* - ! & + & , M X ;!&  + gOY9 - ! & + & , M Y ;!& *& rates & &J&  RX ˆ

1 % M X  k

RY ˆ

1 % M Y  k



 *& error probability  ^ 6ˆ X  Y ^ˆ PE ˆ PfX 6 Yg:

$ , & ORX  RY 9    >& admissible ,, , &;&+ å . < *&& &@  -J%! , *&  &$-&  % )( "* PE , å &$  /, O&& F27G $$ .7&  & *& $>& "! &!-&  *& &,&& &-% ,  &H!&-& , &$&&  ;&- "* &$+ HOX  Y 9   *&  >& &% "! >& *& *,$& >!& >+ *& & RX ‡ RY ˆ HOX  Y 9 , *& &-& & *& -&-& *& 4& , *&  >& &% - -&&   *& B/  >& &% ! & &&+ >;& * &  & * , , &@ $& Y  #" $&,&-+  *& &-& *& & % !-&+ >! X  HOX jY 9   &;&+ $  *&  >& &% ! ,+ RX > HOX jY 9   + RY > HOY jX 9 *! *&  >& &% ! >&  !>& , *& &% -&  % )3 *& & #>& ,-  *   * %& & &-&+ /+&  =; F7& $>&

Figure 6.8 -# %

, *& &$B/, *&& 

16.

Survey of advanced topics for Part one

Figure 6.9 *& & ,  >& &

$&, -& , *& -J%!  % )( "*& *& & RX  RY &  *& &% #& NN/B= &%II  % )3  /B= &% X - >& &$!-& &@-+ -& RX > HO Z9 ! -& RY , HOY jX 9 &;>+ *&& " >&   *& &-&$ , Y  /+&  =; *;& && & *&  ! $>& 

*& &$B/,  /+&B=; &! ! >&  &&$ !!& %&&4 , & *&+ · A functon ROä9 /& *;& &+ &&  *$& 6 * &;& ,  -&& & +& !-& !&  ;&+ %&  ! , + &+  $&&   H!& ,J-!  - $!& *& & ,!- ROä9 "&;& *! F62G O&&  '&>   ! F2)G $$&@ : 9 * -;&&  $+ -;&%& %* , - $!% ROä9 ! &-+   *& $>& , - $!% ROä9  * -&  "&+ &%&  ;&  & %&& !-&   &!& *"&;& *& $>& , - $!% ROä9  !-* & ,J-! &*$ *&   $&;& %&& - , !-& , "*-*  ->& , ! , ROä9  #"  *& - , disccrete-time stationary Gaussian !-& &;&  *& &H!& & -& && *& !-&  &&   + ! &H!&-& ,  ;>& . . .  X 1  X <  X 1  X 2  . . . : *& &  ,!- ROä9  %;& $ &-+ >+  1 ð äOè9 ˆ Oè SOù99 dù 2ð ð    1 ð SOù9 ROè9 ˆ @ +
H O7.99  &@ $& *&&  $$&+  #" -&& !-& "* & + , "*-* ROä9 - >& - $!& &@-+ "&;& *&& &@ + &-*H!& , >!% ROä9  "& % &,& *& &&& &&  &%& F16G · For large enough k /& "  &- 2 * ,    *& -*& & $>>+ PE - >& &  $$-* < &@$&+ "* -&% >-# &%* *& %! H!& , *& !-& -% *&&  " H!-#+ & *&  ,  $ ! !-& -& , &%* k  & < ROä9 $$-* äP  * &- - F.6G O&&  %& F1:G &- 369 * *" ,    * , d  Ok9 && *&  ! $>& ;&%&  ,  !-& -& , &%* k  & < ROä9 *&   % k d  Ok9 ˆ ä ‡ < : k

*! *& -;&%&-& , ä  Ok9  ä  !-* & $ * *& -;&%&-& , PE  &  ! , &&-* * >&& &;&  *& $>& , J%  - , &+  $& &>& !-& -& , "*-* *& !-& -% *&& - >& $;& * &&-* * &  !-& -% *&&  , *& , NN + å . <  ä > ä  *&& &@  !-& -& , +$& T "* & < ROä9 ‡ å  ;&%&  < ä ‡ åII !-* *&&  *;& >&& $;& "*&& +$& T  *& & , linear codes O&& *$& :9  tree codes O&& *$& 1& - $&&+ &->& >+ + & , k &+ &$&& -&" x1  x2  . . .  x k  , k &;&+ -&"  & , *& q k & - > iˆ1 á i x i  á i 2 Fq  , "& %& *& -&"   k 3 n @ G G  -&  generator matrix , C & %&&+D De®nition Let C be an On k9 linear code over Fq. A matrix G whose rowspace equals C is called a generator matrix for C. Conversely, if G is a matrix with entries from Fq , its rowspace is called the code generated by G. + $&-,+%  %&& @   &+ $>&  %;& - $- &-$ ,  & &&% -& *& *&& &@ $& * ," O "* q ˆ 2L *&+ & binary -&9 " >& !& , !;& $!$& *!%*! * -*$& Example 7.1 O7 19 & -& C1 "* %&&

@

G1 ˆ F1 1 1 1 1 1G: Example 7.2 O7 69 & -& C2 "* %&& @

 1 1 1 < < G 2 ˆ  < < 1 1 < : 1 1 1 1 1 Example 7.3 O: .9 & -& C6 "* %&& @

 1 < < < < 1 1 < 1 < < 1 < 1  G6 ˆ   < < 1 < 1 1 < : < < < 1 1 1 1

h

h

h

OC1 - + *& " -&" ! G2   *& !H!& 
%&& @ , C2   ,-

 1 1 < < 1 G92 ˆ  < < 1 < 1  < < < 1 1 "*&& , -+ "& *;& --& *& &,  1  &-* " % G92 *& * G2  *& &-% O:19 , C2 >&- &D Ou1  u2  u6 9 ! O u1  u1  u2  u6  u1 ‡ u2 ‡ u6 9:

* &-% , C2 * *& &>& ,&!& * *& ,  + > u1  u2  u6 $$&  *& -&  *& -&"L  %&& *& + > u i " $$&  *& t i * - $& , *& -&" x ˆ uG , *& &,  &+ , *& i* " , G --!  -!  t i  + -& &   "*-* * *&

1.2

Linear codes

$$&+ * *&& &@  &-% !& !-* * *& ,  + > $$&  *& -&    >& systematic /& *;& *! & & * every linear code is systematic -& &@ * *& 
%&& -& , C1  C6 *;& *& , G ˆ FI k AG "*&& I k  *& k 3 k &+ @ &+  &;&+ & -& &?+ * $$&+ O&% C2 9L >! , *& -&  , !&   & +& -*& *& $&, %  -!  $& !  G "  & *& -&I $&, -&  "& - "+ ! & G ˆ FI k AG  * -&  &@ $& >+ &&% *& -!  , G92  *& & O16.279 "& >

 1 < < 1 1 G 20 ˆ  < 1 < < 1  < < 1 < 1 "*-* %&&&  -& "*& $&, -&  *&  &  * , C2  + & +& -*&  !-* ,  "*& , *& %&& @ *&&  *& $&*$ &;& & !&,! @ -& "* &;&+ & -& -& *& paritycheck matrix "*-* "& " &->& , C   On k9 & -& ;& Fq  parity check ) , C   &H! , *& , a1 x1 ‡ a2 x2 ‡


‡ a n x n ˆ $-& , V n OFq 9   && >+ C ? OC NN$&$II9  -& *& dual code , C +   &!  & %&> C ? *  & n  OC9 *  C ? is an On n k9 linear code over Fq $+-*&-# @ , C  " &J&   %&& @ , C ? & &-+D De®nition Let C be an On k9 linear code over Fq. A matrix H with the property that Hx T ˆ 0 iff x 2 C is called a parity-check matrix for C.: , -!& >+ *& %! & >;&  ," * &;&+ On k9 & -& *  !H!& On k9 3 n 
$+-*&-# @ !   & -   !&  %*+ ,,&& -- , , H  &@ $& , G ˆ FI k AG  ," * H - >& #&  H ˆ F AT I n

k G:

O::69

, G   , * ,  H - >& >& >+ J $&, %  -! 

7.2 Syndrome decoding on q-ary symmetric channels

1.6

$& !  $! G  *& , FI k AG  *& $&, % *& ;&& $& !  F A T I n k G(  &@ $& *& $+-*&-# -& , &  * "+ , G1  G2   G6 O$ 16.9 &

 1 1 < < <   1 < 1 < & & &- O>! -, > :11  :2! *& + &  *  &$& +  *& & $& z    *&  & -&"D s ˆ Hy T ˆ HOx ‡ z9 T ˆ Hx T ‡ Hz T ˆ Hz T  -& Hx T ˆ 0 , -!& *& &-&;&   &-+ &&&  zL *& "  #" x ! -& *& #" y  x ˆ y z *& - ,&+ ,-!  *& $>& , J% z

*& + & $;&  & ,  >! z >!  &!%* *  >&-!& ,  J@& s 2 V n k OFq 9 *& & , !  Hz T ˆ s ,   coset 1< , *& -& C *   !>& , V n OFq 9 , *& , C ‡ z< ˆ fx ‡ z< D x 2 Cg:

O::.9

*&& & q n k -& , C -&$%  *& q n k $>& + & sL &-* -& - &@-+ q k && & *! -& *& &-&;& - $!& s *& * &!-& * &-* , z , q n  q k $>&  &+ *& && & , *& -& -&$%  s  &  %!* >&"&& *&& q k -& , z *"&;&   &-&+  #"  &*% & >! *& -*&  * &- "& * ! & * *& -*&   q-ary symmetric channel OH9 *  , X    ;&- &% *& -*& $!  Y    ;&- &% *& -*& !$! *& Y ˆ X ‡ Z "*&& Z ˆ O Z 1  Z 2  . . .  Z n 9    ;&- "*& - $& & &$&& &-+ >!&  ;>& "* -  >! Pf Z ˆ & *& ! >& , 4& - $& , z &;&+ w H Oz9  *& ! >& , & --!%  z , å < 1=q *& %* & , O:)9   &-&% ,!- , w H Oz9   *&  $>>& z  *& & ,  & "&%* O *& -& 1=q , å , 1=Oq 19 && > :69 %!& :1 *" *" *& + & &-& "#  &  $-$& , -!& &$ 2  * %* &$&&  , >&  ! , "#L && *&  >?&- , *$& (  3  *& &% ,  & & -& , *& H , "*-* &$ 2  ->& "&;& , k  n k & >* &;&+     $>&   $& & &$ 2 ;  NN>& #!$II $-&!& "*-* "& " &->& & % -& C2   $+-*&-# @  %;&  $ 1.6    1 1 < < < Hˆ : 1 < 1 1 1  *  $& -& *&& & + ,! $>& + &D &"&& " ;&- x  y  d  Ox y9 ˆ *& ! >& , - $& , "*-* xi 6ˆ yi ˆ w Oy

x9Ocf: $: 1.79:

O * -& J& *& ,  $$&& &H!& ,  &-L && > :.9 *&&   &&% &*$ >&"&& *&  % %& &+ ,  -&   >+  -&- &   H "*-* "& " ;&%&

*! & C ˆ fx1  x2  . . .  x M g >&  -& , &%* n  &-&+ & , !&   H !$$& * "& " C  >& -$>& , -&-%  & $& ,  % "&%* < eL *  , x i  & y ˆ x i ‡ z  &-&;&  w Oz9 < e "& " ! &-&I !$!  >& x^ ˆ x i    &+  && * , &-* -&"  & "* $>>+ 1=M *& *& &-&;&I >& &%+ , %!&% "*-* -&" " &   $-# *& -&" -&  y *  *& & , "*-* d  Ox i  y9   & O&& > 2169 FNoteD -& d  Ox i  y9 ˆ w Oz9  ," * *& + & &-% , & -& &->&  % :1  &H!;&  NNJ *& -& -&"II &-%G   -& * , * %& &- &-% &%+  !& *& -& " >& -$>& , -&-%  $& , "&%* < e ,, *& -& >&"&& &-* $ , -&"  > 2e ‡ 1  O% :69 , d  Ox i  x j 9 > 2e ‡ 1 *  , *&  % $*&& , ! e ! x i  x j & ? *& , x i  &  d  Ox i  y9 < e y - >& -& 

7.3 Hamming geometry and code performance

Figure 7.3 

1.:

% $*&& , ! e ! ?-& -&"

x j *    x i     %& &- &-& "  $&,& x j  x i  ;&&+ , d  Ox i  x j 9 < 2e *  , *&  % $*&& , ! e &&- O% :6>9 *&   -& * , x i  & *&& &@  y * * d  Ox i  y9 < e >!   &  -&  x j     x i   "& & &  &J& *& minimum distance , *& -& C  d

 OC9

ˆ

fd  Ox x99D x x9 2 C x 6ˆ x9g

 "& *;& $;& *& ,"% *&&  Theorem 7.2 A code C ˆ fx1  x2  . . .  x M g is capable of correcting all error patterns of weight < e iff d  OC9 > 2e ‡ 1  &@ $&  -& "* d  ˆ : - -&-  & $& , "&%* < 6L , d  ˆ 22  $& , "&%* < 1&;  * -& d  Ox x99 ˆ w Ox x99  x x9 ! >&  O4&9 -&" , C  & O x 6ˆ x99 *&  ! -& ,  & -&  *&  &   minimum weight w  OC9 "*&& w

 OC9

ˆ

fw Ox9D x 2 C x 6ˆ & &$&& >! *& ! ,  -!   0L   d  OC1 9 ˆ 7  H 2   -& * d  OC2 9 ˆ 2 -& , &@ $& *& *  ,!* -!  , H 2 & &- *& !+ , H 6    $ "& * &;& ! &@ &-   7.4 Hamming codes  &,&&-& "& % $+ *& $+-*&-# @ , C6 D

 < 1 1 1 1 < < H 6 ˆ  1 < 1 1 < 1 < : 1 1 < 1 < < 1 /& " $$+ *&& :6  *& $>& , && % d   + *& & ,"% *& & & , *& *&&  d   *&  & ! >& , -!  , H "*-* !  0 &+ d  6ˆ 1 2 -& *& -!  , H 6 & 4&  - "&;& *&& & + !>& , *&& -!  , H 6 ! %  0 , &@ $& -!  1 2  6 *! d  ˆ 6   C6   singleerror-correcting -& *    -$>& , -&-%  & $& , "&%* <  1 + >&;& * , C  + On n 69 %&&-&-% -& *& n < : -& , *& 6 3 n $+-*&-# @ H * n > (  "! &*& *;&  4& -!  Od  ˆ 19   $ , &- -!  Od  ˆ 29

*& %&& &J ,  >+  % -& ,"

7.5 Syndrome decoding on general q-ary channels

1.3

De®nition Let H be an m 3 O2 m 19 binary matrix such that the columns of H are the 2 m 1 nonzero vectors from V m OF2 9 in some order. Then the n ˆ 2 m 1, k ˆ 2 m 1 m linear code over F2 whose parity-check matrix is H is called a (binary) Hamming code of length 2 m 111 /& #& " & # >!  % -&  -& * + & &-%  &$&-+ &+ , *& & $&  z ˆ ! , w Oz9 ˆ 1 + z i ˆ 1 *& s ˆ c i  *& i* -!  , H &-& *& + & &-+ &J& *& & -  *& %&& &-% %* , % :1 >&- & *& $&- %* , % :. &- &- *  -& C - -&-  $& , "&%* < 1 ,, *&  % $*&& , ! 1 ! *& -&" & ? !   % $*&& , ! 1  V n OF2 9 - n ‡ 1 ;&-  *  %&&-&-% -& - *;&   2 n =On ‡ 19 -&"  $-! , n ˆ 2 m 1 m m *&& - >&   22 1 =2 m ˆ 22 1 m -&" *& &@- ! >&  *&  % -&M *! *&  % -& *;& *& >&!,! %& &$$&+ * *& $*&& , ! 1 ! *& -&" &@-+ J V n OF2 9 "*! ;&$ * & * *&  % -& >&%   &@& &+ &@-!;& - , -& *& perfect codes *& + *& >+ & $&,&- -& & *& &$& -& O&& > :1(9  *& O26 129 + -& O&& &- 3(9 O  && *& -! , $&,&- -&  *$& 129 &;& ; ,  % -& -!% >+  % -& & %;&  > :1:  :13 "&;& *& & &&%  !&,! %&&4 ,  % -&    - , -& "*-* -&- e & , e . 1L * %&&4  *&  $- &&  *$& 3 7.5 Syndrome decoding on general q-ary channels  &- :2 "& &->&  % &-% %* , & -& >&% !&   q+ + &- -*& "&;& ,&" q$! q!$! -*& & "& && >+ H    * &- "& * >&K+ -& *& $>& , &-% & -&  & - $-& -*&

Figure 7.4 + & &-% ,  

% -&

17
&  On k9 & -& ;& Fq "*-*  %%  >& !&   -*& "*& $!  !$! $*>& & >* &H!  Fq  /& ! & * *& -*& &  ;& *  , x ˆ Ox1  . . .  x n 9  *&  & -&" *& &-&;& ;&- y  %;& >+ y ˆ x ‡ Z "*&& Z ˆ O Z 1  . . .  Z n 9    & ;&- "* >! %;& >+ PfZ ˆ zg ˆ pOz9 z 2 V n OFq 9 O& * *& H , &- :2 J * &-$ "* pOz9 %;& >+
H O:799 "  "& &-& C  * -*&P  >&,& *& J &$   - $!& *& + & s ˆ Hy T L * " &,+ *& -& * *& & $& z >&%  ! " *&  #&+ & $& "* *& -& -*-&4& >+ *& + & s  *& & "* *& %& ;!& pOz9 *& * *&  & "&%* &-& *& &-% %*  * %&& ! O-, % :19  %;&  % :7  &@ $& -& !% -& C2   -*& , "*-* *& &%* 7 & $& *;& $>>& %;& >+ *& ,"% >&D z !  " &&- + & $& z !& z   4& -&" *& &-&I & $>>+  *& ?! *& $>>+ * z   4& -&" , C , "& ! & *& -*&   H *& *& $>>+ * z &H!  $-! -&" , "&%* w  FO1 Oq 19åG n w å w O&&
H O:)99 &-& , "& && >+ A i *& ! >& , -&"  C , "&%* i *& & $>>+ , * &&-+ -*& & 

7.6 Weight enumerators and the MacWilliams identities PE ˆ

n


A i å i F1

19åG n

Oq

176

i

iˆ1

ˆ F1

Oq

19åG n FAOä9

1G

O:::9

"*&& ä ˆ å=F1 Oq 19åG  AOz9 ˆ A< ‡ A1 z ‡


‡ A n z n O& * A< ˆ 1 , + -&9 *& %&&% ,!- AOz9  -& *& weight enumerator , C /& * !+  ,!*&  *& &@ &- h 7.6 Weight enumerators and the MacWilliams identities

 &$&D , C   On k9 & -&  "&%* &! &  *& $+  AOz9 ˆ A< ‡ A1 z ‡


‡ A n z n  "*&& A i && *& ! >& , -&"  C ,  % "&%* i k &+ A< ˆ 1  AO19 ˆ q   "& *;& && O
H O::99 AOz9 - >& !&  --!& *& & $>>+ "*& C  !& , & &&- +

*& ,"% *&& *" * AOz9 - >& !&  >! *& & $>>+   !-* & &&% !  &+ "*& *& -&  >&% !&   -&& & +& -*& "*  @ ! #&* &-% !& F& + & &-%  @ ! #&*L && $ 1.)G   $-+ "& * & + >+ -& >! && > :1& " !$$& x<   & *& &-& " &J&+ 

17.

Linear codes

!$! x i , pOyjx< 9 . pOyjx i 9   , Y i ˆ fyD pOyjx i 9 > pOyjx< 9g  ," * PO Qi
>+  < AOã9 1  $ & , x<   & ,  & *& -&" + x j   & &% &-  * ?!

7.6 Weight enumerators and the MacWilliams identities 177  n %;& *" * *& &!% & $>>+ POEj9 < iˆ1 AOi j9 ã i "*&& O j9 A i ˆ *& ! >& , -&" 6ˆ x j "*  % -& i , x j  !  -& *& -&  & AOi j9 ˆ AO :129  *& *&& $;& h Example 7.6 & C1 , * -*$& * + " -&" & ;   $& & ,  , *& "&%* &! & , *& ! -& C ? O&& $ 1.29 Theorem 7.6 Othe MacWilliams identities9 Let AOz9 be the weight enumerator of an On k9 linear code C, and let BOz9 be the weight enumerator of the dual code C ? , that is, BOz9 ˆ

n


B j z j

jˆ
 :2:9  "& &>*  & -;& , x ˆ Ox1  . . .  x m 9   ;&- , + &%* &;& m ˆ 1 ;&  J& F wOx9 &&   % "&%* *  *& ! >& , 4& - $&  $-! , x   - O& x 2 F9 *&  < , x ˆ + &J *& ! >& , - $& xi , x &H!  1 h

7.6 Weight enumerators and the MacWilliams identities

17:

/& " $-&&  $;& *&& :) *& &   - $!& *& !

z wOy9 hy x x9i O::1:9 xx92C y2V

 " "+ , "& --!& *& & ! &  2


O1

z9 wOx

x99

O1 ‡ z9 n

wOx x99

ˆ

 O:1:9 J "& > >+ n


A i O1

z9 i O1 ‡ z9 n i 

iˆ
+ "&%* %* && *$&& ! *& ! -& * + 2 m -&"  *  ;&+  $& "&%*

17(

Linear codes

&! &   >+ *&& :) AOz9 , *&  % -& - >& - $!&  %&& O&& *& ,"% &@ $&   > :23  :6+ *&& :) $ % : 1 z9. O1 ‡ z96 ˆ 1 ‡ :z 6 ‡ :z . ‡ z : : h ( O1 ‡ z9 ‡ :O1

Problems 7.1 & C >&  & -& ;& *& J& F6 ˆ f+ & -& C O9   "&!-& &-*& %&& @ , C O>9   $+-*&-# @ H , C O-9   ! "&%* -& && , &-* , *& ( -& , C O9 & A i Oi ˆ & , ;&- , "&%* i  C  *& A i  O&9 &-& *& ,"% &-&;& ;&-D 111& & "* d  OC9 ˆ 2   !H!&P  $&& *& $, , *&& :. & C >&  q+ & -& >&% !&   H "* @ ! #& * &-% *" * *& &!% & $>>+  >!& >+p AOã9 1 "*&& A  *& -&I "&%* &! &  㠈 2 åF1 Oq 19åG ‡ åOq 29 O *& &J ,  H &&
H O:799

q+ erasure channel   -&& & +& -*& "* $! $*>& A X ˆ Fq  *& J& J& "* q && &  A Y ˆ Fq [ f?g "*&& NNPII   $&- &!& + > , x ˆ Ox1  . . .  x n 9   &  y ˆ O y1  . . .  y n 9  &-&;&  error    --!  *& i* $ , yi 2 Fq >! yi 6ˆ xi   erasure , yi ˆ ? O9 & C >&  On k9 & -& ;& Fq "*  ! -& d   *" * C - -&-  - > , e &  f &!& ,, d  > 2e ‡ f ‡ 1

1)
9 !$$& *& -*&   pure erasure channel *  pO yjx9 ˆ < !& y ˆ x  y ˆ ?  * -& *" * *& - Hx T ˆ 0 , x  >&  -&" - >& !&  > n k & &H!  *& !#" && -&" - $& O-9 $$+ *& &-*H!& !%%&&  $ O>9  &-& *& ,"% -& " , *& O: .9  % -& O$+-*&-# @ H 6  $ 1.)9 "*-* *;& !,,&& &!& >!  &D 19 *" * I   ,  & , C ,, -!  i1  . . .  i k , *& %&& @ & &+ &$&& O-9 " + ,  &  -& C1  C2  C6 , * -*$& *;&P O9 & C >&  binary & -& !-* * &;&+ k&& & !>& , f1 2 . . .  ng   ,  & *" * k ˆ + -& "* * $$&+ O&% *& &&B  -& , &- 3)9G O!9 *&  , ,  & !-&  > :16 - >& !&  &%  &-% %* , & -& * ,,& #&+ , + & &-%    & & -& error trapping  decoding with multipliers12 !$$& * C   On k9 & -&  * "& "*  -&-  $& , e  ,&"& & "* C & I 1  I 2  . . .  I r >& ,   & "* *&  $$&+ * , + e&& & !>& J  f1 2 . . .  ng *&& &@  & & & I i !-* * I i \ J ˆ ö O9 !$$& * y  *& &-&;& -&"  *  - < e &  &-* i ˆ 1 2 . . .  r & x i ˆ *& !H!& -&" , C * %&& "* y  *& $ $&-J& >+ I i *" *  & & , *& x i I  %&&&  *& -!  & -&" O>9 ! % * d  OC9 > 2e ‡ 1 *" * >+ - $% *& x i I  y & >+ & *&  & -&" - >& !H!&+ &J& O-9 $$+ * &-*H!&  *& O: .9  % -& /*  *&  & ! >& , ,  & * " !,J-&  -&- 1 &P !$$& +! "&& $$-*& >+  - !- &%&& "*  +! * * O>+9 -*& --&$ " , &%* n  * *& + # , & $& &;& >&;&  & , *& n ‡ 1 $& O&  %&&-&-% -& ,,  -!  , H   9 *" * ,  J@& ;!& , m , H * *& $$&&  $ O9 *&

1)2

Linear codes n < Oq m 19=Oq 19  * * >!  *$ , n ˆ Oq m 19=Oq 19 *& &!% -&  -&  q-ary Hamming code &+  * n ˆ Oq m 19=Oq 19 k ˆ n m O-9 !- $+-*&-# -& , *& ,"%  % -&D q 6 6 7

n . 16 )

k 2 1< .

7.20 *& >?&- , * $>&   !+ *& $&, -& ,  >+ & -&   ! -*& O&& &- .19 "* &&%+ - EOX 2 9 < â  & ;-& ó 2   $-! "& * >  %!& , *&& :7 , * -*&  & C >&  On k9 >+ & -&L "& ,+  , !&  *&! -*& >+ $$%  &-* NN!  *& &*$ >&"&& d   k , *& >& -&   $-+ "& * &- ! &  binary linear codes  J@& n  d & M L On d9 && *& @ ! $>& ! >& , -&"   & -& "* &%* n   ! -& > d

Problems

1)6

7.21 O *& >&B'* ; >!9 *" * M L On d9 > 2 n =‰1 ‡ …1n † ‡


‡ … dn 1 †Š F HintD , *& >& & -& * ,&"& * * + "  & & ,  -& "! *;&  " , "&%* > d ! *& ! ,  & -& "* & ,  -&    & -&G 7.22 O *&  % >!9 *" * ,  >+ -& , &%* $ n &   % -$>& , -&-% e &  -   2 n = 1 ‡ …1n † ‡


‡ … en † -&" 7.23 O!9 *" * *&  % -& -*&;& *& >! , > :22 7.24 O *& # >!9 & ð i D Ox1  . . .  x n 9 ! xi >& *& $$% * $?&-  >+ ;&-   i* - $& *" * , x %& *!%*  On k9 -& C ð i Ox9 ˆ < * &*& 2 k  2 k 1 ! & * ,-  $;& *
w Ox9 < n
2 k 1 : x2C

-!& * d  OC9 < n
2 k 1 =O2 k 19 7.25 O!9 !$$& n > 2d 2  & C< >& *& -& >& >+ &&-%  -&" , C * & <  *& J n 2d ‡ 2 -&  &&% *&& -& $$+% *& &! , > :2.  C<  --!& * M L On d9 < 2 n

2d‡2

d:

7.26 *& ,"% &! O&  ' & &&>&%9 *" *"  &%*& *&& :7 ,   O9 *" * >! O:169 - >&  $;&  Qi < ã d  Ox< x i 9‡1 , d H Ox<  x i 9   F Hint:  > >+ ,  &$& -& , &%* 2n  *&  &  , & , &%* 2n 1G O>9 &-& *" * *&& :7 - >&  $;&  PE < 12FO1 ‡ ã9 AOã9 ‡ O1 ã9AO ã9G 1 O-9 $$+ *& >! >&  $ O>9  -& C1 , * -*$&  - $& *& &!  * >& 
@ $& :)

*& &@ " $>&  & "* %&&4 , *&& :) *& -/  && *& J &! &&+ !$$& *& & , *& $, , *&& :) "*& q 6ˆ 2 *& &- & "* & -& 7.27 ;& *&& :) , %&& q F Hint: *& $, %;&  *& &@ " "# -& *& H!+ hx yi O&&
H O:1)99  !>+ %&&4&   * & hx yi ˆ ëOx
y9 "*&& ë  + ; *  $* , *& ;& %!$ , Fq  *& - $&@ p*  , !+ "*&& q ˆ p j  p  $ &G 7.28 ;& *& & -/  &&D , C ˆ fx1  . . .  x M g   !>& , V n OFq 9 &J& A i ˆ M 1
*& ! >& , $ Ox x99 , C "* d  Ox x99 ˆ i , *& B j I & &J& >+ *& , ! n 1
A i O1 M iˆ
+  % -& , &%* n ˆ 2 m 1 * "&%* 2 m 1  O>9 &-& *" * ,  >+  % -&

AOz9 ˆ

1 $ O1 ‡ z9 n ‡ nO1 2m

z 2 9O n

19=2

% O1 ‡ z9 :

O-9 &&4&  >+  % -& O&& > :139 7.30  * $>& "& " &J&  ;&%& *&  $ Reed±Muller codes & POm d9 && *& & ,  $+  , &%&& < d  m ;>& ;& F2  & Ov<  v1  . . .  v M 1 9 M ˆ 2 m  &&   ,  *& 2 m >+ ;&- Ox1  x2  . . .  x m 9   & & *& , &-* f 2 POm d9 "& %&  >+ ;&- , &%* 2 m ; *& $$% f ! O f Ov< 9 f Ov1 9 . . .  f Ov M 1 99 *& & ,  ;&- >&  * "+ , $+   POm d9  -& *& d* order Reed±Muller code of length 2 m    Om d9 , * O9 *" *  Om d9   >+ On k9 & -& "* n ˆ 2 m  k ˆ 1 ‡ …1m † ‡


‡ … dm † O>9 *" * *&  ! -& ,  Om d9  2 m d  F Hint:  *" * d  > 2 m d  & f ˆ f Ox1  x2  . . .  x m 9 2 POm d9  -& f O :1:> O&9  $!& *& "&%* &! & ,  Om d9 , d ˆ &  On k9 & -& "* $+-*&-# @ , *& , H ˆ FBI n k G "*&& B  On k9 3 k  I n k   On k9 3 On k9 &+ @ , 1 < t < k *& -& C t "*& $+ -*&-# @  H t ˆ FB t I n k G "*&& B t  *& On k9 3 Ok t9 @ >& >+ &&% *& J t -!  , B  -& a shortened version of C O&&  > :279 O9 *" * C t - ,  -&" , C "*& J t -& & &H!  ;& - >& *!%* ,  2 >! , &%* . " & H >&  m 3 n $+-*&-# @ ,  On k9 >+ & -& C *& b-fold interleaving , C  *& Onb kb9 -& C Ob9 "* mb 3 nb $+ -*&-# @

H

Ob9

  ˆ 

Notes c1

c1

c2 ..

.

cn ..

c1



.

.. c2

.

1)7

    cn

"*&& Fc1  . . .  c n G & *& -!  , H  ># && ! , &%* b  &&-  $& , f >! , &%* b O-9 !- $+-*&-# -& , O9  O21 129 -& * -&-  %& >! , &%* 6L O9  O27 2! , &%* 7 ;&  &,J-& &-% %* , >* -&

Notes 1 O$ 1639 /& $!& *&&  > &!  *L ,  *& !>?&- , &@$- -& *  "* * -% *&&  $ & *& * >&& & 2 O$ 1639 & $&-&+ & block -&  %!* *& , *& & convolutional -& , *$& 1&  $ & $"& "& - -& $! $*>& "* ) 1 !-*  - >& ,! , &@ $&  #*,,  -& F6G *$& : 7 O$ 1.19 !+  
@  $& &  *;&  & " * & & , "  *& @ &  -*%&  *&   
,  /& * ! & *"&;& * , 4& " $$&  *& &!- *&+ & &&& ) O$ 1.29 *&  & - & , *& ,- *  *& >+ -&  $+-*&-# + * &-* -&" *  &;& ! >& , 1I   -& J@& !>& , *& -& $ : O$ 1.29 &&  &&"*&& T  *& $& $& ( O$ 1.69 *&  $-+ , *  O, G  H9  , -!&  --&L *& 
,  &%&   *  &@ $&  ;& *& +& , & * %&&! &H! &$&&& >+ G2 O$ 1.&  % -& -& *&+ &   $& 12 O$ 1)+ &-% ! &  & -& O*& * >+ !!-!& -&9 "& -! *$&  J  & % -& "*-* & &>+ &+   $& &    !& * O; + & &-% , &@ $&9  NN II & -& + "*  &  &!-+   2&  $& &&  *"& "*! !-* ,J-!+ "&;&  &  > *& $&, -& $ & >+ *I *&&    &-&+  !& %& -&   %&&  %& -& &;& ,   & " >& ,J-!   $& &  * &    >-# -& !&  $--& &  ,- cyclic codesL -+-- -& ,  ;&+    *%*+ !-!& !>& , *& & , & -&  * -*$& "& " %;&  %&& !-  -+-- -& -!% >* *& !&+% *& - *&+ O&- (19  *& >- *"& --! !&   $& & -+-- -& O&- (29  &- (6 "& " *" *  % -& - >&  $& &&  -+--&   &- (.  (7 "& " && *" -+-- -& & !&  - > burst errors ! + " >& -!&  *$& 3 "*&& "& " !+ *&   $ , + , -+-- -& +& -;&&D *& E &&B  , + /& >&% ! !& "* *& -!!$$&% &J , *& - , -+-- -& De®nition An On k9 linear code over a ®eld F is said to be a cyclic code if, for every codeword C ˆ OC<  C1  . . .  C n 1 9 the right cyclic shift of C viz. C R ˆ OC n 1  C<  . . .  C n 2 9 is also a codeword.

 "& * && *&& &

+ -+-- -& >! - $&  & -& 1):

1)(

Cyclic codes

*&+ & H!& --&  &@ $& *&& & 11(11 & O: 69 -& ;& GFO29 >! + " & -+--M Example 8.1 , F  + J&  n   &%& > 6 *&& & "+  & ,! -+-- -& , &%* n ;& F !!+ -& *& ,! trivial -+--&D ·  On + *& < 1
+ C1 C2  C6  *& 4& -&" &D C1 ˆ 1& "*& POx9  ;& >+ gOx9 & POx9 ˆ QOx9 gOx9 ‡ ROx9

O(:19

"* &% R , &% g &!-% &-* , *&& $+   x n 1  !% *& ,- * &% R , &% g < n 1 O*& & &H!+ -& gOx9   -&"9 "& > ROx9 ˆ FPOx9G n

FQOx9 gOx9G n :

! FPOx9G n   -&" >+ ! $  FQOx9 gOx9G n   -&" >+

*&& (2 *! -& C  & ROx9   -&"  ! &% R , &% g  gOx9   4& -&" , & &%&&  ROx9 ˆ &- & POx9 ˆ QOx9 gOx9 "*-* *" * gOx9 ;& POx9

h

/& - " &  $;& *&  *&& >! -+-- -&  &>*&  && -&$&-& >&"&& -+-- -& , &%* n  - ; , x n 1 Theorem 8.3 O9 If C is an On k9 cyclic code over F then its generator polynomial is a divisor of x n 1 Futhermore, the vector C ˆ OC<  C1 . . .  C n 1 9 is in the code if and only if the corresponding generating function COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1 is divisible by gOx9 If k denotes the dimension of C then k ˆ n &% gOx9 O>9 Conversely, if gOx9 is a divisor of x n 1 then there is an On k9 cyclic code with gOx9 as its generator polynomial and k ˆ n &% gOx9 namely, the set of all vectors OC<  C1  . . .  C n 1 9 whose generating functions are divisible by gOx9 Proof of (a)  & POx9 ˆ x n 1  &  2O>9L FPOx9G n ˆ + *&& (2 + ;&- , &%* n "*& %&&% ,!-   !$& , gOx9   -&" ;&&+ , COx9 ˆ C< ‡ C1 x ‡


‡ Cn x n 1   -&" *& FCOx9G n ˆ COx9   &  2  $& * gOx9 ;& COx9 +

8.1 Introduction

1:7

*& & >! *& &%&& , gOx9 ," -& COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1   !$& , gOx9 ,  + , COx9 ˆ gOx9IOx9 "*&& &% I < n 1 &% g Proof of (b) !$$& * gOx9   ; , x n 1 *& COx9 ˆ OC<  C1  . . .  C n 1 9   !$& , gOx9 ,  + , COx9 ˆ gOx9IOx9 "*&& &% g ‡ &% I < n 1 *! *& & ,  !-* "   On k9 & -& "*&& k ˆ n &% g  *" * * -&  -+-- "& ! *" * *& %* -+-- *, , + -&"    -&" *! & IOx9 gOx9 >& + -&"L >+ *&& (1  %* -+-- *,  FxIOx9 gOx9G n  ! -& gOx9 ;& x n 1 "& *;& FxIOx9 gOx9G n  gOx9 ˆ FxIOx9 gOx9G  gOx9 ˆ
+ &

"*-* $;& * FxIOx9 gOx9G n   -+-- -&

O>+ &

 1O&99

 1O>99

!$& , gOx9  *& -&  &&  h

*&& (6 *" *&  $-& , *& %&& $+  ,  -+--& -&+ &&  &H!+  $ $+   *& parity-check $+  ,  -+-- -& "*-*  && >+ hOx9  &J& >+ hOx9 ˆ

xn 1 : gOx9

*& ,"% -&  *&& (6 %;& &@$- &-$ , %&&  $+-*&-# -& ,  -+-- -&  &  , gOx9  hOx9 Corollary 1 If C is an On k9 cyclic code with generator polynomial gOx9 ˆ g< ‡ g 1 x ‡


‡ g r x r (with r ˆ n k), and parity-check polynomial hOx9 ˆ h< ‡ h1 x ‡


‡ hk x k , then the following matrices are generator and parity-check matrices for CD

  g< g1 . . . . . . gr < . . . . . . < gOx9  < g   g1 . . . . . . gr < . . . <  <    x gOx9     G1 ˆ  ˆ ..   ..  ..   .  .   .
+ *& &J  &- :. *  &&   % -& /& &;& *& ;&% , *& *& NN- &II  >& (26 h

8.2 Shift-register encoders for cyclic codes

1(1


@ $& (3 "& " * gOx9 ˆ x 6 ‡ x ‡ 1 %&&&  O: .9  % : -& ! --%  >& (1 gOx9 ;&  + x 1 >!  x 1. 1L  ,- x 6 ‡ x ‡ 1jx n 1 , + n "*-*   !$& , : O&& >& (2.9 *! >+ *&& (6O>9 gOx9 %&&&  "*& , + , -+-- -& "* $ && O: .9 O1. 119 O21 1(9 . . . : "&;&  , *&& -& &@-&$ *& J -  ;&- "*& %&&% ,!-  x : 1   *;&  ! "&%* &H!  2 &-& *&& -& &  &&% , & -&-L "& * - *& improper -+-- -& O&& >& (2)9  *& & , * -*$& "*& "& &,&   On k9 -+-- -& "* %&& $+  gOx9 "& * ! &    proper -+-- -& & & , "*-* n  *& smallest $;& &%& !-* * gOx9jx n 1 * &%&   & & -& *& period , gOx9 -&   *& $& , *& &H!&-& Fx i  gOx9G i><  && >& (2: ,  -! , -+-- -& , "*-* *& parity-check $+  * $& & * n9 /* * -;& &>*& "& & &+  -!& ! !+ , -+--&

8.2 Shift-register encoders for cyclic codes " * "& #" >+ *&& (6 * &;&+ -+-- -&  -*-&4& >+  %&& $+  "& - >&%  && "*+ -+-- -& & !-* &&   $& & * >+ & -&  * &- "& * *"  ,- * &;&+ -+-- -& - >& &-& "*   $& J&& -*& -&  shift-register encoder &- *  &-% %* ,  On k9 & -& -+--     !& , $$% *& & , &%*k ,  &H!&-& OI <  I 1  . . .  I k 1 9  *& & , &%*n -&" OC<  C1  . . .  C n 1 9  &H!;&+ , $$% *& ,  $+  IOx9 ˆ OI <  I 1 x ‡


‡ I k 1 x k 1  *& & , -& $+  COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1  , *& -&  -+-- *&& (6 & ! * COx9   -&" ,  + , COx9   !$& , gOx9    "& "  + 1  *&& (6 & "+  -*%&  ,  $+  IOx9   -& $+    $+  !$+ IOx9 >+ gOx9D IOx9 ! IOx9 gOx9:

O(:29

 ! ! * $+  !$-  &+   $& & !%  & *% -& shift-register logic  "& * " #&  >&, !+ , * !>?&-2 %!& (1   >- &$&& ,  -*& -$>& , !$+%

xn ‡ 1 ˆ

Ox ‡ 19 Ox ‡ 192 Ox ‡ 19Ox 2 ‡ x ‡ 19 Ox ‡ 19. Ox ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 192 Ox 2 ‡ x ‡ 192 Ox ‡ 19Ox 6 ‡ x ‡ 19Ox 6 ‡ x 2 ‡ 19 Ox ‡ 19( Ox ‡ 19Ox 2 ‡ x ‡ 19Ox ) ‡ x 6 ‡ 19 Ox ‡ 192 Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 19. Ox 2 ‡ x ‡ 19. Ox ‡ 19Ox 12 ‡ x 11 ‡ x 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 192 Ox 6 ‡ x ‡ 192 Ox 6 ‡ x 2 ‡ 192

1 2 6 . 7 ) : ( 3 1< 11 12 16 1.

1Oˆ x n ‡ 19  $"& , &!->& $+  ;& GFO29 , 1 < n < 61

n

Table 8.1 -4 , x n

17 1) 1: 1( 13 2< 21 22 26 2. 27 2) 2: 2( 23 6< 61

Ox ‡ 19Ox 2 ‡ x ‡ 19Ox . ‡ x ‡ 19Ox . ‡ x 6 ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 191) Ox ‡ 19Ox ( ‡ x 7 ‡ x . ‡ x 6 ‡ 19Ox ( ‡ x : ‡ x ) ‡ x . ‡ x 2 ‡ x ‡ 19 Ox ‡ 192 Ox 2 ‡ x ‡ 192 Ox ) ‡ x 6 ‡ 192 Ox ‡ 19Ox 1( ‡ x 1: ‡ x 1) ‡


‡ x ‡ 19 Ox ‡ 19. Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19. Ox ‡ 19Ox 2 ‡ x ‡ 19Ox 6 ‡ x ‡ 19Ox 6 ‡ x 2 ‡ 19Ox ) ‡ x . ‡ x 2 ‡ x ‡ 19Ox ) ‡ x 7 ‡ x . ‡ x 2 ‡ 19 Ox ‡ 192 Ox 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 11 ‡ x 3 ‡ x : ‡ x ) ‡ x 7 ‡ x ‡ 19Ox 11 ‡ x 1< ‡ x ) ‡ x 7 ‡ x . ‡ x 2 ‡ 19 Ox ‡ 19( Ox 2 ‡ x ‡ 19( Ox ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19Ox 2< ‡ x 17 ‡ x 1< ‡ x 7 ‡ 19 Ox ‡ 192 Ox 12 ‡ x 11 ‡ x 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 2 ‡ x ‡ 19Ox ) ‡ x 6 ‡ 19Ox 1( ‡ x 3 ‡ 19 Ox ‡ 19. Ox 6 ‡ x ‡ 19. Ox 6 ‡ x 2 ‡ 19. Ox ‡ 19Ox 2( ‡ x 2: ‡


‡ x ‡ 19 Ox ‡ 192 Ox 2 ‡ x ‡ 192 Ox . ‡ x ‡ 192 Ox . ‡ x 6 ‡ 192 Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 7 ‡ x 2 ‡ 19Ox 7 ‡ x 6 ‡ 19Ox 7 ‡ x 6 ‡ x 2 ‡ x ‡ 19Ox 7 ‡ x . ‡ x 2 ‡ x ‡ 19Ox 7 ‡ x . ‡ x 6 ‡ x ‡ 19Ox 7 ‡ x . ‡ x 6 ‡ x 2 ‡ 19

1(.

Cyclic codes output g0

g1

g2

gr⫺1

gr

input

Figure 8.1 *,&%& --! , !$+% >+ gOx9 ˆ g < ‡ g1 x ‡


‡ g r x r

&;&+ *   &-& ,  On k9 -+-- -& "* %&& $+  gOx9 "* r ˆ n k

 arbitrary $+  IOx9 >+  ®xed $+  gOx9 &,& $;% * *& --! "# "& * >&& &@$  ;! - $&

*& --!  %!& (1  >! >+ -&-% %&*& *&& +$& , - $&D ¯ip-¯ops, adders  constant multipliers *&   $ , *&&  *& K$K$  & & -&  delay && &D

a flip-flop

K$K$   &;-& "*-* - & & && & , *& J& F  $-!&  !  $J& --! %  >!   $ $ , *&    &@& --# "*-* %&&&   % % ONN-#II9 &;&+ t< &-6 /*& *& --# -# *& -& , *& K$K$ & *,& ! , *& K$ K$  *& &- , *& " *!%* *& --! ! *& &@ K$K$  &-*& && *& % $ ! *& &@ -# *& -& , *& *, &%& - >& J& >+ *& --! >&"&& !--&;& K$K$ "*-*  "*&& *& *& " %- && & - &  *& "$! adder "*-* # #& *

an adder

B   &;-& * - $!& *& ! ,  " $! % O *& --!  "+ %&  !-*  "+ * ,& &;&+ -# &-* $!  *& & &-&;& &@-+ & %  * *& &I !$!  "+ ! >%!!9 + "& - &  *&  $& --! && & *& constant multiplier: a constant multiplier

8.2 Shift-register encoders for cyclic codes

1(7

* &;-&  $+ !$&  $! >+ *& - a "*  &+ " &I && , "& - !& "*+ *& --! , %!& (1 - >& !& , $+  !$- /& J "& !  & *& , ! , *& -&,J-& , *& $!- COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1  "*&& COx9 ˆ IOx9 gOx9D C< ˆ I < g<  C1 ˆ I < g1 ‡ I 1 g<  C2 ˆ I < g2 ‡ I 1 g1 ‡ I 2 g<  .. .

O(:69

Cj ˆ I < gj ‡ I 1 g

1

‡


‡ I j g< 

.. . Cn

1

ˆ Ik

1 gr :

*& K$K$ , % (1 & + J& "* & OC<  C1  . . .  C n 1 9 "*&& *& Cj I & &J& >+
H O(69   * --! - >& !&   &-& , *& On k9 -+--& "* %&& $+  gOx9 Comment The preceeding description is for a ``generic'' shift-register circuit, capable of performing arithmetic in an arbitrary ®eld F The required devices (¯ip-¯op, adders, multipliers) are not ``off-the-shelf'' items, except when the ®eld is GFO29 In this case, however, these devices are very simple indeed: ,$,$ &   >& &-& *& ,! K$K$ *! >& 4&   OI <  I 1  I 2  & *"&;&  &%  +& - *,&%& &-& , + -+-- -& "*-*  + %*+ & - $&@ * *& +& - &

Out

In

1

2

3

4

Figure 8.2 +& - &-& , *& O: 69 -+-- -& "* %&& $+  gOx9 ˆ 1 ‡ x 2 ‡ x 6 ‡ x . 

8.2 Shift-register encoders for cyclic codes

*& &   !& *& &! , *&& IOx9   ,  $+  *& IOx9 ! x r IOx9

1(:

(6 + 2 "*-* + * ,

Fx r IOx9G  gOx9

O(:.9

  +& - &-% !& ,  -+-- -& "* %&& $+  gOx9 *& -& - $&  !-*  &-& " >&  NN  gOx9II --! "*&& gOx9  * & -&I %&& $+  %!& (6O9 *" !-*  --! , gOx9 ˆ x r ‡ g r 1 x r 1 ‡


‡ g <   &  && "*+ *& --! , %!& (6O9  >&  $&,  NN  gOx9II --! "& ,-!  *& *,&%& -& Fs<  s1  . . .  s r 1 G "*-* "& - *& state vector , *& -*&   *& -&$% %&&% ,!- SOx9 ˆ s< ‡ s1 ‡


‡ s r 1 x r 1  "*-* "& - *& state polynomial /& >&% "*  &  "*-* &@$ *" *& & -*%&  &$&   $! Lemma 3 If the circuit in Figure 8.3(a) has state polynomial SOx9 and the input is s, then the next state polynomial will be S9Ox9 ˆ Os ‡ xSOx99  gOx9:

⫺g0

In

⫺g1

s0

⫺gr⫺1

s1

sr⫺1

(a)

In

(b)

Figure 8.3 O9 %&& NN  gOx9II *,&%& --! "*&& gOx9 ˆ x r ‡ g r 1 x r 1 ‡


‡ g <  O>9 $&-J- NN  x . ‡ x 6 ‡ x 2 ‡ 1II *, &%& "*&&  * &-   2

1((

Cyclic codes

Proof , *& $&& & ;&-  S ˆ Fs<  . . .  sr 1 G  *& $!  s *& >+ ! !& &->% *& >!% >-# , *& --! *& &@ & ;&- " >& >+ &J S9 ˆ Fs

g < sr 1  s
+ &  1O-9 h Theorem 8.4 If the circuit of Figure 8.3(a) is initialized by setting s< ˆ s1 ˆ


ˆ s r 1 ˆ  !-*%&  -# < *!%* k 1  "! #&   !$!% *& $+-*&-# & *& -&,J-& , Fx r IOx9G  gOx9  *& k* -# >! " *;&  " , r & -# >&,& Fx r IOx9G  gOx9  &+ ! &+   $>&  ; * r-# NN"  &II >+ !% *& --! *"  %!& (.O9  *& --! , %!& (.O9 *& $! > & ,&  *& right & , *& *, &%& *& * *& &,   %!& (6 *& &!  * , *& $! &  a<  a1  . . .  ,& *& t* -# *& *, &%&I & $+     " >& tjˆ< aj x r‡ t j  gOx9 *& * tjˆ< aj x t j  gOx9 O  ;&,+ * &  >& (619 *! , *& &-& $! *& k ,  + > I k 1  I k 2  . . .  I <  *& *, &%& , %!& (. *& -&,J-& , Fx r IOx9G  gOx9 " >& &+ % "* *& k* -#  && - $&& +& - &-& >&  *& *, &%& , %!& (.O9  *"  %!& (7O9 & *  - $&  *& +& &-& , %!& (1  *& &-& , %!& (7O9 *& ,  + > & &  *& &-&  *& reverse & ;4 I k 1  . . .  I <   *& -&" - $& &  &  *& -*&  *& &;&& & ;4 C n 1  C n 2  . . .  C1  C<  Example 8.11 , "& $$+ *& %&& -!- &->&  %!& (7O9  *& $&- -& , gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 ;& *& J& GFO29 "& %& *& &-& &$-&  %!& (7O>9 , , &@ $& *& ,  &H!&-& OI 2  I 1  I < 9 ˆ O1 1 & O1 1 9 $&- -& gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1

-# < 1 2 6 . 7 )

$!

 -&

!$!

1 1 < < < <
+ J& GFO29  *! >& -& *"  !& *& *,&%& --! , %!& ()O9  >!  +& - &-& ,  -+-- -& "* $+-*&-# $+ 

⫺h1

st⫺1

⫺h2

st⫺2

⫺hk⫺1

⫺hk

st⫺k

(a)

(b)

Figure 8.6  O9 *,&%& --! *  $& & *& k*& & &-! k St ˆ jˆ1 hj S t j  O>9 *& $&- -& St ˆ S t 2 ‡ S t 6  -&$%  hOx9 ˆ 1 ‡ x 2 ‡ x 6 

13.

Cyclic codes

hOx9 ˆ 1 ‡ h1 x ‡


‡ hk x k O&& %!& (:O99 !% *& J k -# *& "-*   *& NN"II $  *& k ,  + > &  !&!+ & !  *& -*&  &  *& k%& &%&

*& *& "-*  $!  *& NN!$II $  !% *& & % r -# *& *,&%& --! --!& *& & % r -&" - $& ; k *& &-! Ci ˆ jˆ1 hj C i j  -& * *& +& - NNhOx9 &-&II , %!& (: ,,& , *& +& - NN gOx9 &-&II , %!& (7  "  $ "+D  *& gOx9 &-& *& ,  + > OI <  I 1  . . .  I k 1 9 --!$+ -&" - $& Cr  C r‡1  . . .  C n 1   *& -&" + > & &  *& -*&  *& &;&& & C n 1  C n 2  . . .  C<  "*&&  *& hOx9 &-& *& ,  + > --!$+ -&" - $& C<  C1  . . .  C k 1  *& -&" + > & &  *& -*&  *& ! & C<  C1  . . .  C n 1 Example 8.12 , "& $$+ *& %&& -!- , %!& (:O9  *& $-! O: 69 >+ -+-- -& "* gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 

first k ticks: down last r ticks: up

⫺h1

⫺h2

⫺hk⫺1

⫺hk

Out

In (a)

first 3 ticks: down last 4 ticks: up

Out

In (b)

Figure 8.7 O9 +& - *,&%& &-& ,  On k9 -+-- -& "* $+ -*&-# $+  hOx9 ˆ 1 ‡ h1 Ox9 ‡


‡ hk x k  O>9 $&- -& hOx9 ˆ 1 ‡ x 2 ‡ x 6 

8.3 Cyclic Hamming codes

137

hOx9 ˆ x 6 ‡ x 2 ‡ 1 "& %& *& &-& *"  %!& (:O>9 *! , &@ $& , *& ,  + > & OI <  I 1  I 2 9 ˆ O1 1 9  $& & *& &-! Ci ˆ C i 2 ‡ C i 6  $!-& *& -&" O1 1 &  $& "*& k , r & k , n=2 h

8.3 Cyclic Hamming codes4  &- :. "& &J&  >+  % -& , &%* 2 m 1  >& + & -& "*& $+-*&-# @ *  -!  *& 2 m 1 4& >+ ;&- , &%* m arranged in any order  &@ $& *& ,"% $+-*&-# @ &J&  O: .9  % -& O-, *& @ H 6  &- :.9D

 < 1 1 1 1 < < H ˆ  1 < 1 1 < 1 < : 1 1 < 1 < < 1

*&& & , -!& O2 m 19M "+  & *&& -!   *!%* + & , *&& &% $!-&  $&,&- %&&-&-% -&  & &% & >&& * *& ,   $& & $  ,- "& * &&  * &- *   $>&  -*&  &% * $!-&  cyclic ;& , *&  % -& "*-* &    $& *, &%&  $& & , >* *& &-&  *& &-&  &->& *& $$$& &% "& ! ! & *& &&  >& ,  "* *& J& J& GFO2 m 9  "*-* &-* && &  &$&&& >+  >+ ;&- , &%* m O  $$&@  "& %;&  ! + , *& &&& ,-9

 >  -+--  % -& , &%* 2 m 1 >&% "*  $ ;& m  á 2 GFO2 9  !&   &J&  & -& C  ," ;&- C ˆ OC<  C1  . . .  C n 1 97 "* - $&  GFO29   C ,  + , C < ‡ C 1 á ‡


‡ Cn 1 á n

1

ˆ + & -& &J& >+ *& 1 3 n $+-*&-# @ H ˆ F1

á

á2

...

á n 1 G:

O(:39

/& *  $;& * *& -& &J& >+ O((9  O(39   -+- % -&  " *&  $ *%  -& >! * &J 

13)

Cyclic codes Table 8.2 *& $"& , á  GFO(9 "*&& á6 ˆ á ‡ 1 i

ái

< 1 2 6 . 7 )

+  &$&&   m> -!  ;&-  &@ $& "* m ˆ 6 , á   $ ;&   GFO26 9 ,+% á6 ˆ á ‡ 1 *& >& (2 *" *& $"& , á &$&&&  *&& & ;&- *! *& 1 3 : GFO(9 @ H  O(39 &J& *&  & >+ -&  *& ,"% 6 3 : >+ @ H9D

 < < 1 < 1 1 1 H9 ˆ  < 1 < 1 1 1 < : 1 < < 1 < 1 1

*& ,"% *&&

 *&

 &! , * &-

Theorem 8.6 The code de®ned above (i.e., by (8.8) or (8.9)) is an On n m9 binary cyclic code with generator polynomial gOx9 the minimal polynomial of á Furthermore, its minimum distance is 6 so that it is a Hamming code Proof   -& * *& -&   >+ & -& , &%* n /& &&  *" *   -+--  * gOx9  *& %&& $+   *" *   -+-- & * , "& !$+ &H! O((9 >+ á  !& *& ,- * á n ˆ 1 "& %& Cn

1

‡ C< á ‡


‡ C n 2 á n

2

ˆ + *&& (6 gOx9  *& -&I %&& $+ 

 - $&& *& $, "& &&  *" * *& -&I  ! -& d   6 -& *& -&  & d  ˆ w   /& !& *& &J O((9L , *&& "&&  -&" , "&%* 1 *& á i ˆ < ,  & i "*-*   $>&  +  " , "&%* 2 -! &@ ,  + , á i ‡ á j ˆ < ,  & i  j "* < < i , j < n 1 ;% >+ á i * >&- & 1 ‡ á j i ˆ &  >!  $& *,&%& &-& ,  % -& >&  *& %&& $+  -& *& %&& $+  ,   % -& ! >&  $ ;& $+   &   $& &  -+--  % -& , &%* 2 m 1   &-&+  *;&  $ ;& $+  , &%&& m  >& (6 "&  & $ ;& $+   , &%&& m , 1 < m < 12   %;& ;!& , m *&& "  %&& >& + $ ;& $+  , &%&& m >! *& &  >& (6 & -*&  *;& *& ,&"& $>& 4& -&,J-& "*-* " &  *,&%& &-& "* *& ,&"& $>& 2 & &  $ * *& &-%  $-+ ,  % -& *"&;&  *& ,-    $>&  >!  $& decoders , *&  /& !& !-*  &-& , *& O: .9  % -& "* %&& $+  gOx9 ˆ x 6 ‡ x ‡ 1  %!& (( *& &-& - , *&&  $ " *, &%& --!   AND %& *& !$$& *, &%&   NN  gOx9II --!  *& "& *, &%&   NN  x n ‡ 1II --!

*&  %& !$!  1 , *& !$$& *, &%& - *& $& 1< . . . & %&& $+  ,   % -& m

 ;& $+  , &%&& m

1 2 6 . 7 ) : ( 3 1< 11 12

x‡1 x2 ‡ x ‡ 1 x6 ‡ x ‡ 1 x. ‡ x ‡ 1 x7 ‡ x2 ‡ 1 x) ‡ x ‡ 1 x: ‡ x ‡ 1 x( ‡ x: ‡ x2 ‡ x ‡ 1 x3 ‡ x. ‡ 1 x 1< ‡ x 6 ‡ 1 x 11 ‡ x 2 ‡ 1 x 12 ‡ x ) ‡ x . ‡ x ‡ 1

mod g(x) circuit

Switch A: closed for first n ticks open for next n ticks

AND gate: 10…0 recognizer Switch B: open for first n ticks Switch B: closed for next n ticks mod (xn ⫹ 1) circuit

Figure 8.8 &-% --! , *& O: .9  % -& "* %&& $+  gOx9 ˆ x 6 ‡ x ‡ 1

OR<  R1  . . .  Rn 1 9 -& ROx9 ˆ COx9 ‡ EOx9  COx9  gOx9 ˆ & x t EOx9  gOx9  *& "& *, &%& " - x t ROx9  x n ‡ 1 & *& t* %* -+-- *, ORn t  . . .  Rn t 1 9 , *& &-&;& " *& " *, &%& & -&-& ; "-*   *& AND %& "*-* !$!  1 ,  + , *& !$$& *, &%& - 1< . . . & -&-& *& ,& Oe 19  n ,!*& -# *& &-&;& " " *;& - $&&  --! ?!&+ ! *& "& *, &%&  " $$& "*  & -&-&  *& "& *, &%&  &- (7 "& " && *"  %&&4& *& --! , %!& ((  &  >!  &-& ,  -+-- burst&-&-% -& *& !& +% *&+ , >!& -&- " >& $&&&  &- (. O -&+ *& dual -& , -+--  % -& &  &&%   $ *&+ & -;&& >&K+  >& (.19

8.4 Burst-error correction  + -*& , $--  $-& & "*& *&+ --! &  --!  bursts *+-+  >! , & --! "*& ,  & & *& -*& & &;&+ -&& ,  >&,  &  *& &!     * &- "& " && *" -+-- -& - >& !&  &&-  -&- & >! /& >&% "*  $!&+ *& - &J ,  & >! ,  -&" C   &   &-&;&  R ˆ C ‡ E *& *& & ;&- E  -&  burst of length b , *& 4& - $& , E & -J&  b -&-!;& - $&  &@ $& E ˆ O  E  &% "* *&  4& + > *& >!I location  *& &@ , *& J 4& + >  *& >!  &@ $& *& ;&- E ˆ O! &-$ , w ˆ <  1 *&&,& *&&  *%  $;&  "& ! & w > 2

8.4 Burst-error correction

2! &-$ , E *& $& " -  , EI 4& - $&   *& - $& , E not -!&  *& $& " ,  -+-- ! , &%% ?! ,& *&  4& - $&  *& $&  -!% ! ?! >&,& *& J 4& - $&  *& $& *& & , -& -&$%  * ! , ! &-$  &@ $& , E ˆ O+ % , &%* b "*-* >&% "*  1 *! *&& & 2 b 1 $>& $&     , n
2 b 1 4& >! , &%* < b

% 1  * ! >&  --! , *& 4& >! "& %& n2 b 1 ‡ 1  && h

*& &@ " *&&  %;& !&,! >!  *& 4& , -& "*-* -&- >! &   $-+ "& * -  -& -$>& , -&-%  >!& $& , &%* < b  burst-b-error-correcting code Theorem 8.9 OThe Hamming bound for burst-error correction9 If 1 < b < On ‡ 19=2 a binary burst-b-error-correcting code has at most 2 n = On2 b 1 ‡ 19 codewords Proof + *&& (( *&& & n2 b 1 ‡ 1 >!& $& , &%* < b , *&& & M -&" *&& & *& MOn2 b 1 ‡ 19 " "*-* ,,& ,  -&" >+  >! , &%* < b *&& " !  >& -   MOn2 b 1 ‡ 19 < 2 n  h Corollary (the Abramson bounds). If 1 < b < On ‡ 19=2 a binary linear On k9 linear burst-b-error-correcting code must satisfy n < 2r where r ˆ n

b‡1

1 Ostrong Abramson bound9

k is the code's redundancy. An alternative formulation is

r > d%2 On ‡ 19e ‡ Ob

19 Oweak Abramson bound9:

Proof   & On k9 -& *&& & M ˆ 2 k -&"   >+

*&& (3 2 k < 2 n =On2 b 1 ‡ 19 &%% * "& %& n < 2 r b‡1 2 b‡1  -& n ! >&  &%& * >! - >&  $;&  n < 2 r b‡1 1 *& % >  >! &%% this  >  >!  r "& %& *& "&# >  >! h

8.4 Burst-error correction

2+ *& $%&*& $-$& *&& ! >& " - -&" "*-* %&&  *& J n 2b -& *&& " -&" - *& >& &$&&& -*& -+  ,"D 2b

n 2b )****************+,****************- )***********+,***********X ˆ        A A A A A A

Y ˆ       B B B B B B "*&& NNII && %&& &  *& AI  BI & >+ ! *& *& " Zˆ        A A A B B B ,,& ,

>* X  Y >+  >! , &%* < b a --

h

Corollary (the Reiger bound). If < < b < n=2 a binary On k9 linear burstb-error-correcting code must satisfy r > 2b where r ˆ n

k is the code's redundancy

Proof *& ! >& , -&"   On k9 >+ & -&  2 k  "*-* >+ *&& (1< ! >& < 2 n 2b  *  &H!;&  *& & & , *& -+ h /& & "   $  -!  && , &@ $& , >!& -&-% -&  &-* -& *& -& " >& -+--  && &*& *& % >   &%& >! O"*-* $$+   & -&  ?! -+-- -&9  * -! "*& "& + *  $-! >! O&*& *& >  >!  *& &%& >!9  tight "& & * *&& &@  -& "*& &!-+  &H!  *& ;!& , *& >! ,  !-* -& &@ "& " + * *& >!  loose Example 8.13 *& On 19 >+ &$& -& "* gOx9 ˆ x n 1 ‡ x n 2 ‡


‡ x ‡ 1 "*&& n   - -&-  & $& , "&%* < On 19=2     >!OOn 19=29&-&-% -& -& r ˆ n 1 *& &%& >!  %* h

2& -&" , &%* n   -+-- -& "* gOx9 ˆ 1   O;+9  b ˆ < >!&-&-% -&  -& r ˆ <  *& &%& >!  % %* h Example 8.15 + >+  % -& "* n ˆ 2 m 1 r ˆ m  gOx9 ˆ  $ ;& $+  , &%&& m   b ˆ 1 >!&-&-% -& O + & ;&- , "&%* 1  $ ,-  >! , &%* 19 *& % >  >!  %* ,  *&& -& h Example 8.16{ + -+--  % -& , "*-* *& -&" ,  "&%* *;& >&& & ;&   b ˆ 2 >!&-&-% -& -&  Abramson code *&& -& & -+-- -& "* %&& $+  , *& , gOx9 ˆ Ox ‡ 19 pOx9 "*&& pOx9   $ ;& $+  O&& >& (779 *&  & >  -&  *& O: 69 -+-- -& "* gOx9 ˆ Ox ‡ 19Ox 6 ‡ x ‡ 19  $+-*&-# @   1 á á2 á6 á. á7 á) Hˆ  1 1 1 1 1 1 1 "*&& á   $ ;&  ,+% *& &H! á6 ‡ á ‡ 1 ˆ <  O(9

 ;&,+ * * -&  &&  b ˆ 2 >!&-&-% -& "& &&  -*&-# * *& + & ,  >! , &%* < b < 2 & -  b ˆ < O*& 4& & $&9 + &  <   b ˆ 1  >! "* á*& i &-$ O1 i9 * + &   b ˆ 2  >! &-$ O11 i9 * 1 i + & á< Oá‡19  *&& 1 ‡ 2n + & &  -   *& -&  &&  b ˆ 2 >!&-&-% -& & J+ * , gOx9 ˆ Ox ‡ 19 pOx9 "*&& pOx9   $ ;& $+  , &%&& m *& n ˆ 2 m 1 r ˆ m ‡ 1  b ˆ 2  * *& % >  >!  %* O m ˆ 6 & *& O: 69 b ˆ 2 -& *& &%& >!  %*  >! ,  %& ;!& , m *& &%& >!  &9 h Example 8.17) *& O17 39 >+ -+-- -& "* %&& $+  gOx9 ˆ Ox . ‡ x ‡ 19Ox 2 ‡ x ‡ 19 ˆ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ 1 ! !  >&  b ˆ 6 >!&-&-% -& , "*-* *&&,& >* *& %

>   &%& >! & %*  $;& * *   "& !& *& $+-*&-# @   1 á á2 . . . á1. Hˆ  1 ù ù2 . . . ù1. "*&& á   $ ;&   GFO1)9 ,+% á. ‡ á ‡ 1 ˆ &-!& , *& NN&"&& *& $& 1 11  1& , *& , ù t L  %!* >&"&& *&  "& #  Os t9  6 , *& $&  1 *& s ˆ i t ˆ i  Os t9  6 ˆ ! &-$ 11  &@ $& *& + & á1 * s ˆ 11 t ˆ ! &-$ O11 :9 & *& & $&  O& + *&& " , * -& "*-* "& - $+   6 3 : +D A< B< C
! >+ &% *& >;& + >+ -!   -& *& interleaving , A B  CD A< B< C< A1 B1 C1 A2 B2 C2 A6 B6 C6 A. B. C. A7 B7 C7 A) B) C) : & ! !$$& * * % -&"   & ;&  >!+ -*&  !,,&  >! , &%* ) -& >+ ID A< B< C< A1 B1 C1 A2 B2 C2 A6       B7 C7 A) B) C) :   $>&  -&- * >! , &  $+ >+ NN&&&;%II * % -&"   - $& -&" >&-!& ,& &&&;%  & , *& *&& -&" " *;& !,,&&  >! %& * "D A< B< C
& &&;% , *&& -&" , *& O: 69 >  -&  -& *& depth6 interleaving , *& % -&    O21 39 & -&L  *& %! & >;& *" *    ,-  b ˆ ) >!&-&-% -& & %&&+ , + $;& &%& j *& &$* j &&;% , *& O: 69 >  -&   O: j 6 j9 b ˆ 2 j >!&-&-% -& & * &-* & , *&& -& J& *& &%& >! "* &H!+ O-& r ˆ . j  b ˆ 2 j9  *   -& &&   , &,J-&-+ &! "*&  -&  &&;& h

%*," %&&4 , *& %! & %;& 
@ $& (1( &  *& ,"%  $ *&&  Theorem 8.11 If C is an On k9 linear burst-b-error correcting code, then the depth-j interleaving of C is a Onj kj9 burst-bj-error-correcting code

2;! >!   &;&*&& !& * , "& &&;&  cyclic -&  &$* j *& &!% -&   -+-- *& &@ *&& $& * ! Theorem 8.12 If C is an On k9 cyclic code with generator polynomial gOx9 then the depth-j interleaving of C is an Onj kj9 cyclic code with generator polynomial gOx j 9



C
+ C< C1


C j 1  &+ &  "*& $, "& &;&  >& ()7 *" *"  - $!& *& %* -+-*, ,  &&;& -&"D O(:1  -& "* %&& $+  gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1  !% *& &&;% &-*H!& "& - $!-&  J& , + , -+-- >!&-&-% -& ;4 *& O: j 6 j9 b ˆ 2 j -& "* %&& $+  g j Ox9 ˆ x . j ‡ x 6 j ‡ x 2 j ‡ 1  + % "* *& -+-- O17 139 b ˆ 6 -& ,
@ $& (1: "& > *& J& , + ;4 *& O17 j 3 j9 b ˆ 6 j -+-- -& "* %&& $+  gOx9 ˆ x ) j ‡ x 7 j ‡ x . j ‡ x 6 j ‡ 1 & * &;&+ -&  &-* , *& , & && *& &%& >! O+  *&& (1+ -+-- n ˆ 17 >! &-&-% -& , b ˆ 1 2 . . .  :  &-* !-* ;!& , b "&  *& "& >!  *& &&& &!-+ !% *& O"&#9 >  >! OrA 9  *& &%& >! OrR 9 + -*-&  ! ! *  &;&+ -& *&&   -+-- -& "*& &!-+  @OrA  rR 9  "& *;& & *& %&& $+  gOx9 , !-*  -&  &-* -& *& -& b ˆ 1 2 6 "& *;& &+ -

8.4 Burst-error correction b 1 2 6 . 7 ) :

rA rR gOx9 . 7

2 x. ‡ x ‡ 1 . Ox . ‡ x ‡ 19Ox ‡ 19

) ) Ox . ‡ x ‡ 19Ox 2 ‡ x ‡ 19 : ( Ox . ‡ x ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 ( 1< Ox . ‡ x ‡ 19Ox 2 ‡ x ‡ 19Ox . ‡ x 6 ‡ 19 ˆ x 1< ‡ x 7 ‡ 1 3 12 Ox . ‡ x ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox . ‡ x 6 ‡ 19 ˆ x 12 ‡ x 3 ‡ x ) ‡ x6 ‡ 1 1. 1< 1. x ‡ x 16 ‡


‡ x ‡ 1

2  -& O
@ $& (1)9
@ $& (1: >& (7( O6 19 &&;& 3 7 >&

()
& , "+ , *& *& -+-- >!& -&-% -& "& *;& !& *&   $ ,,&&-&  *& %!&& >+  detect + & >! * &  %  correct && -& & strong >!&-&-% -&  *& && , *& ,"% &J De®nition An On k9 cyclic code with generator polynomial gOx9 is said to be a strong burst-b-error-correcting code, if, whenever Z1 and Z2 are burst-error vectors with the same syndrome, and with burst descriptions O$&1  -1 9 and O$&2  -2 9 such that &%*O$&1 9 ‡ &%*O$&2 9 < 2b then Z1 ˆ Z2 

21
!&-&-% -& & -$>& ,  !&! >!& -&-  &&- Theorem 8.13 If C is a strong burst-b-error-correcting code, then for any pair of nonnegative integers b1 and b2 such that b1 < b2 and b1 ‡ b2 ˆ 2b it is possible to design a decoder for C that will correct all bursts of length < b1  while at the same time detecting all burst of length < b2  Proof * ," , *&& :.  *& &J , % >! -&-  && * & E && *& & , & >! , &%* < b1   & F && *& & , & >! , &%* < b2  *& , Z1 2 E  Z2 2 F  "& #" * Z1  Z2 *;& >! &-$ !-* * &%*O$&1 9 < b1  &%*O$&2 9 < b2  ! b1 ‡ b2 ˆ 2b   >+ *& &J ,  % >!b&-&-% -& Z1  Z2 *;& ,,&& + & *! >+ *&& :. C * *& ;&& -$>+ h Example 8.21{  >!b&-&-% -& -!%  , *& -&  >& (6 & not %  &@ $& -& *& O: 69 b ˆ 2

>  -&  $+-*&-# @ - >& #&  >&   1 á á2 á6 á. á7 á) Hˆ  1 1 1 1 1 1 1 "*&& á6 ‡ á ‡ 1 ˆ <  GFO(9  "& + 
@ $& (1) * -& -&-  >! , &%* < 2L >!  - -&-  >! , &%* < 1 "*& &&-%  >! , &%* < 6 -& &% *& & $& 7 O111 m and ®nally that g 1 Ox9 and g2 Ox9 are relatively prime. Then gOx9 ˆ g 1 Ox9 g 2 Ox9 is the generator polynomial for an On k9 cyclic code which is a (strong) burst-b-error-correcting code, whereD n ˆ - On1  n2 9 kˆn bˆ

&%O g 1 9

&%O g 2 9

Ob1  m On1 ‡ 19=29:

Proof + *&& (6O9 g 1 Ox9jx n1 1  g 2 Ox9jx n2 1 *! -& >* x n1 1  x n2 1 ;& x n 1 "*&& n ˆ - On1  n2 9  -& g 1 Ox9  g2 Ox9 & &;&+ $ &  ," * gOx9 ˆ g 1 Ox9 g 2 Ox9jx n 1   >+ *&& (6O>9 gOx9 %&&&  On k9 -+-- -& "*&& k ˆ n &%O g9 ˆ n &%O g 1 9 &%O g 2 9 * !-*  &+ *& *& , *& $, & "* *& & >! *& >!&-&-% -$>+ , *& -& /& " ! & * *& -& %&&& >+ g 1 Ox9  %  *" * *& &!% %& -&   % *& $, ,  NN"&#II g1 Ox9  &,  >& ()3

*! & Z1  Z2 >& " & ;&- , &%* n "* *&  & + & &;&  *& -+-- -& "* %&& $+  gOx9  "* >! &-$ OP1  i9  OP2  j9 !-* * &%*OP1 9 ‡ &%*OP2 9 < 2b /& &&  *" * Z1 ˆ Z2  &I && *& %&&% ,!- , *& " >! patterns >+ P1 Ox9  P2 Ox9  ," * *& %&&% ,!- , *& & ;&- Z1  Z2 & Fx i P1 Ox9G n  Fx j P2 Ox9G n &$&-;&+   *& & & +  & & Fx i P1 Ox9G n  gOx9  Fx j P2 Ox9G n  gOx9 ! -& gOx9jx n 1 &  1O&9  $& * *& " + & -  ,- >& "&  S1 Ox9 ˆ x i P1 Ox9  gOx9 S2 Ox9 ˆ x j P2 Ox9  gOx9: -& Z1  Z2 & ! &  *;& *&  & + & "& *&&,& *;& x i P1 Ox9  x j P2 Ox9 O  gOx99: -& gOx9 ˆ g1 Ox9 g 2 Ox9 *&  x i P1 Ox9  x j P2 Ox9 O  g 1 Ox99:

O(:119

212

Cyclic codes

! &%*OP1 9 ‡ &%*OP2 9 < 2b >+ ! $  2b < 2b1 >+ *& &J  , b *! -& g1 Ox9 %&&&  % >!b1 &-&-% -&  ," * , *& ;&"$ , *& -+-- -& %&&& >+ g 1 Ox9 *& & ;&- Z1  Z2 & &- & x i P1 Ox9 ˆ x j P2 Ox9 O  x n1

19:

! &%*OP1 9 ‡ &%*OP2 9 < 2b < n1 ‡ 1   >+ *&& P1 ˆ P2  iˆ j

(: O(:129

O  n1 9:

O(:169

 ;&" , * O(119  $& Ox i

x j 9P1 Ox9 
% xi

xj 
% * ,- "* O(169  !% *& ,- * n ˆ - On1  n2 9 "& *;& i  j O  n9L >! -& i  j >* &  *& %& b and period n<  Then gOx9 is the generator polynomial for an On n 2b ‡ 1 m9 cyclic code which is a strong burst-b corrector, where n ˆ - O2b 1 n< 9 This code is called a Fire code, in honor of Philip Fire, its discoverer Proof * ,"  &&+ , *&& (1. >+ #% g 1 Ox9 ˆ Ox 2b 1 19  g 2 Ox9 ˆ f Ox9 O 
@ $& (22 "& " * *& g 1 Ox9 -&   % >!b -&-9 h Example 8.23 --%  *& -+  *&& (1. *& >+ -+--& "* gOx9 ˆ Ox 6 ‡ 19Ox 6 ‡ x ‡ 19 ˆ x ) ‡ x . ‡ x ‡ 1 %&&&  O21 179 % b ˆ 2 >!&-&-% & -& *!   $>& 

8.4 Burst-error correction

216

&;&  &-& , * -& * -&-  >! , &%* < 2L   -&-  >! , &%* < 1  &&-  >! , &%* < 6L   &&-  >! , &%* < . & * * -& && &*& *& %

>  >!  *& &%& >!L  *& *& * -& *  && *&& >! & $$&+ &;& % >! -&-  & * , "& #& g 1 Ox9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 "*-* %&&& *& O"&#9 O: 69 b ˆ 2

>  -&  g 2 Ox9 ˆ x 2 ‡ x ‡ 1 *&& (1.  $& * gOx9 ˆ g 1 Ox9 g 2 Ox9 %&&&  O"&#9 O21 179 b ˆ 2 >!&-&-% -& ! g 1 Ox9 g 2 Ox9 ˆ x ) ‡ x . ‡ x ‡ 1  "& *;& &+ && * * $+   %M h

Example 8.24 *& $+  P67 Ox9 ˆ x 67 ‡ x 26 ‡ x ( ‡ x 2 ‡ 1   $  ;& >+ $+  , &%&& 67   >+ *& -+  *&& (1. gOx9 ˆ Ox 16 ‡ 19P67 Ox9 ˆ x .( ‡ x 6) ‡ x 67 ‡ x 26 ‡ x 21 ‡ x 17 ‡ x 16 ‡ x ( ‡ x 2 ‡ 1 %&&&  % -+-- O16O267 19 16O267 19 .(9 ˆ O..)):)73(::1 ..)):)73(:269 b ˆ : & -& * $-! & -&  H!& , ! >&-!&  * !&  NN*&&II ;& ,   + ,  # ;& /*   *&& -+-- -&P  %&& , gOx9 * &%&& r  %&&&  On n r9 -+-- -& *& , + n< < n gOx9  %&&&  On<  n< r9 *&& -+-- -& * -& - ,  ;&- C , &%* n< "*& %&&% ,!- COx9   !$& , gOx9  *& -& , *&  & -& n< ˆ 172772  *& -& !&  -!+  O172772 1727! &-&-% -& *&  &-&  ;&+ -&;;&  + & $  -&- >! , &%* < .   -& *& -&  %  *&&>+ % *& >+  &&-  >! , &%* < 1& >& (7 %  & !&,! >+ & -&  &-* , *&& -& gOx9 ˆ Ox 2b 1 ‡ 19Pb Ox9 "*&& Pb Ox9   $ ;& $+  , &%&& b *& &!-+  *! r ˆ 6b 1 *!%* *& &!-+ &H!& >+ *& "&# >   &%& >!  +$-+ -&>+  & * * * NN&@II &!-+  $$&+ *& $-& * ! >& $  & * *& -& >& % >! -&-

b

O67 2:9 6 O1&% *& & $& , *& &-& - $!& *& & & + & SOx9 &J& >+ SOx9 ˆ ROx9  gOx9 *& *& ;&- >& >+ !>-% SOx9 , ^ ˆ ROx9 COx9

SOx9

ROx9 ;4 O(:1)9

^  %!&&  >&  -&" *  >&-!& COx9  gOx9 ˆ ROx9  gOx9 SOx9  gOx9 ˆ +  & $&  E * !-*  !& , + -+-- -& &I " -& *& $&- -& ,  -+-- burst&-&-% -& !$$& *& * C   >!b&-&-% -&  * *& + & SOx9 J& *& " - SO+ *& >;& -! *& &-& - ,&+ & * COx9  &J&  O(1)9  *& -!  & -&" &-%  *&&,& &+ , O(1:9  J& ,!&+ *"&;& * + *$$& , *& & ;&-   >! -&  $ !  -&   $ other * + #% ;%& , *& NN-+--&II , *& -& &&I *"D , *& & ;&- EOx9   4& >! , &%* < b *& EOx9 *  !H!& >! &-$ , *& ,

21)

Cyclic codes

OPOx9 i< 9 "* PO!  -&  $ & -&-&  &&+ , i< 6ˆ 9 *& ! >& , -+-- *, &H!&   *  *& !H!& &%& j<  *& %& < < j< < n 1 !-* * i< ‡ j<  < O  n9 "*-*  j< ˆ O i< 9  n: , " R j< Ox9 && *& j< * -+-- *, , ROx9 "& *;& R j< Ox9 ˆ C j< Ox9 ‡ E j< Ox9 "*&& C j< Ox9  Ej< Ox9 & *& j< * -+-- *, , COx9  EOx9 &$&-;&+ " C j< Ox9   -&" -& *& -&  -+--  E j< Ox9 ˆ POx9 >+ *& &J , j<    , S j< Ox9 && *& & & + & , R j< Ox9 "& *;& (a) P E(x):

1XXXX

S(x) ⫽ [xi0P(x)]n mod g(x).

i0 (b) P Ej0(x):

1XXXX

Sj0(x) ⫽ P(x).

0

Figure 8.9 O9 *& >!& ;&- EOx9 ˆ Fx i< POx9G n O>9 ,& *,% EOx9 j< !  *& %* "*&& j< ˆ On i< 9  n *& & $&  NN$$&II  - & NN$$&II   - >& -&-& && *& >!& pattern " >& %;& >+ S j  Ox9  *& >!& location " >& %;& >+ *& , ! i< ˆ O j< 9  n "*&& j<  *& ! >& , *, &H!&  $ *& &  * $ "& -! &%  ,+  $& &-& !% *&& & "&;& >&,& %  "& "*  & * *& --! , *& !--&;& S j Ox9I - >& -&>+  $J& >+ !% *& ,"% &! Theorem 8.15 OMeggitt's lemma9 For j > < de®ne Sj Ox9 ˆ Fx j ROx9G n  gOx9 i.e., Sj Ox9 is the remainder syndrome of the jth cyclic shift of ROx9 Then for j > + &

 1O&9

Sj Ox9 ˆ Fx j ROx9G  gOx9 -& gOx9jx n

1 *&

FxSj Ox9G  gOx9 ˆ FxOFx j ROx9G  gOx99G  gOx9 ˆ Fx j‡1 ROx9G  gOx9 ˆ S j‡1 Ox9:

>+ &

 1O9 h

Example 8.25 /& !& *&& & !% *& O: 69 b ˆ 2 >  -& "* gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 !$$& *& &-&;& ;&-  R ˆ

21(

Cyclic codes

F1!& $&  !$!  & 3 O *& %*  "# , *&& & no &L && >& (:79

*& %* &->&  %!& (1< & &,   $& & >+ *,&%& %- /& !& *  %!& (11  (12 , " ,,&& >!&-&-% -+-- -& *& O: 69 b ˆ 2 >  -& ,
@ $& (1)  *& O17 39 b ˆ 6 -& ,
@ $& (1:  *& &-% --!  %!& (11  (12 *&& & *&&  - $&D   gOx9 *, &%&  x n 1 "& *, &%&   NN1+ *&& (.9  *& "& *, " - ROx9 &@ "-*  opened  "-*   closed *! "% *& !$$& *, &%&  - !-&  *& "& *, &%& ; *& 1& &!&  *& % $ "* *& & >! -&-&  %&& --! #& *&&  %!& (11  (12 & -& NN,! $&--#!II &-& *&+ &H!& &@-+ 2n --# -#  -&- + >! , &%* < b "* *"& - $&@+ *  OOn9 *&+ &  "&$& !&  $--& "&;& , -+ *&& -+-- >! &-&-% -& O&%
@ $& (2.9  J- , * --! + >& &H!&Q&& >& (:: Problems 8.1 *" * *& ! >& , k & !>$-& ,  n & ;&- $-& ;& *& J& GFOq9  &@-+ Oq n 19Oq n 1 Oq k 19Oq k

1

19 . . . Oq n k‡1 19  19 . . . Oq 19

 !& * , !  ;&,+ * *&& & &@-+ 11(11 >+ O: 69 & -& 8.2 -!& *& ,"%D O9 19 x 1<  Ox 12 19L O-9 Ox 17 19  x 1< 1 8.3  *& NN II $& -;& E - !;&P *  & *& && OP  Q9  R ˆ P  OQ  R9 P  Q ˆ Q  P !&  %&&P

Problems

221

8.4 *" * , m 2 f9 ;& &  1 $ O>9 O-9 ;& &  1 $ O-9 O9 ;& &  1 $ O9 O&9 ;& &  1 $ O&9 8.6 ;&  ,  $, * , C   -+-- -&  , COx9 2 C *& ,  i > 1 x i COx9  x n 1 2 C HintD & &  1O9G 8.7  *& G1  H 1 -& , *& O( .9 -+-- -& ;& GFO69 "* %&& $+  gOx9 ˆ Ox 2 ‡ 19Ox 2 ‡ x ‡ 19 O&& + 1  *& & (69 8.8  *& G2  H 2 -& , *& O( .9 -+-- -& ;& GFO69 "* %&& $+  gOx9 ˆ Ox 2 19Ox 2 ‡ x ‡ 19 O&& + 2  *& & (69 8.9 O9 *" * *& ,"% -& & %&& $+-*&-# -& ,  On k9 -+-- -& "* $+-*&-# $+  hOx9 O NN&;&&II ~ $+-*&-# $+  hOx99D ~ n 1 G6 ˆ Fx j hOx9G jˆ< H 6 ˆ Fx i‡ k

O-! 9

~ r 1 x i‡ k hOx9G iˆ
& --& *& @ H 1 , + 1  *&& (6 O9 , *& + & S , *& ;&- R ˆ OR<  R1  . . .  Rn 1 9  --!&  S T ˆ H 1 R T  *" * *& %&&% ,!- ROx9 ˆ R< ‡ R1 x ‡


‡ Rn 1 x n 1  SOx9 ˆ S< ‡ S1 x ‡


‡ S r 1 x r 1 & && >+ SOx9 ˆ

FROx9hOx9G  x n

FROx9hOx9G  x k xk

O"*-*   ?! -! + "+ , +% * OS<  S1  . . .  S r 1 9 & *& -&,J-& , x k  x k‡1  . . .  x n 1  *& $!- ROx9hOx99 O>9 % *& &! , $ O9 J *& + & , *& ;&- R ˆ F1&-!& , *& 1) -&"  *& -& + F11&K+ improper -+-- -& & *& , "*-* *& $& , *& %&& $+  gOx9  & * n  * $>&  "& " ;&%& -+-- -& , "*-* *& parity-check

Problems

226

$+  * $& & * n *! & C >&  On k9 -+-- -&  & x n 1 ˆ gOx9hOx9 "*&& gOx9  *& %&& $+   hOx9  *& $+-*&-# $+  , C !$$& ,!*& * *& $& , hOx9  n< , n  * C<  *& On<  k9 -+-- -& "* %&& $+  gOx9 ˆ Ox n< 19=hOx9 O9 /*  *& $+-*&-# $+  , C< P O>9 , d <  *&  ! -& , C<  "*  *&  ! -& , CP O-9 *" * C ˆ On=n< 9C<  "*&& NN jC< II && *& -& , &%* n >& , C< >+ &$&% &-* -&" j  & 8.28  &-* j  *& %& 1 < j < 1) J *& &%*  &   ! -& , *& O$$&9 >+ -+-- -& "* %&& $+  Ox ‡ 19 j  F HintD *& &! , *& $&;! $>& " >& *&$,!G 8.29 , gOx9 * $& n  &%&& r *& , + m > 1 gOx9jx mn 1   >+

*&& (6O>9 gOx9 %&&&  Onm nm r9 -+-- -& O m > 2 * -&   $$&9 O9  *" *  ;&- C ˆ FC<  C1  . . .  C nm 1 G   * -& ,  + , nm 1 i  n  < O  gOx99 iˆ< Ci x O>9 *" * *& -  $ O9  &H!;&  +% * FC<  C1  . . .  C n 1 G ‡ FCn  C n‡1  . . .  C2 n 1 G ‡


‡ FCO m

19 n 

COm

19 n‡1 

. . .  C mn 1 G

  *& On n r9 $$& -+-- -& %&&& >+ gOx9 O-9 " + " , "&%* 2 &  * -&P 8.30  %!& (6 "& &&  NN  gOx9II --! ,  monic $+  gOx9
@$ *"  ,+ * --! ,   - $+  gOx9 ˆ g < ‡ g 1 x ‡


‡ gr x r  "* gr 6ˆ & St Ox9 ˆ

t


aj x r‡ t

j

 gOx9:

jˆ
9 , n && *& $& +! ,!  $ O9 J  $+-*&-# @ , *& -+-- -& , &%* n %&&& >+ gOx9 O-9 & * -& *;& + " , "&%* 2P ,  &@$ "*+  ,  &@$-+   !-* " O9 "  +& - *,&%& &-& , * -&

22.

Cyclic codes

8.33 &% *&& ,,&& *,&%& &-& , *& O17 :9 >+ -+-- -& "* gOx9 ˆ x ( ‡ x : ‡ x ) ‡ x . ‡ 1 8.34 * $>& --& *& O61 2)9 -+--  % -& "* gOx9 ˆ x 7 ‡ x 2 ‡ 1 O9   +& - O& , *& , FAjI 7 G9 $+-*&-# @ , * -& O>9 &%  &-% --! , * -& O-9 & +! &-& , $ O>9  &-& *& &-&;& " R ˆ F111111111111&  + 1  *&& (6 O9 *" * n ˆ 2 k 1 O>9 &->&  &,J-& *,&%& &-& , C k 

Problems

8.42 8.43 8.44 8.45 8.46 8.47 8.48 8.49 8.50 8.51

8.52

227

O-9 *" * *& & , 4& -&" , C  &@-+ &H!  *& & , -+-- *, , *& J " , G1  !& * &! , C. !% *& $ ;& $+  x . ‡ x ‡ 1 , >& (6 O9  *& "&%* &! & , C k    , ! , *& ! >& , ordinary O&  -+--9 >! , &%* b ;&  "&& $*>& O-, *&& ((9 , 1 < b < n &&4& *&& ((   q&& $*>&
@& *&& ((  -;& *& -& b . On ‡ 19=2 &&4& *&& (3   -+   q&& $*>& &&4& *&& (1<   -+  -;& *& -& ,  q&& $*>& /*+  *& -& b ˆ <  -;&&  *&& (3P /*+  +! !$$& *& " ;& , *& >  >! & -& NN%II  NN"&#IIP

& *& >  >! O-+  *&& (39 &;& %* , *& >+ &$& -&P  *& &%& >! &;& %* , *& >  -&P   & >!&-&-% $$-   !,J-&  -&- >! * --! +  -& $&- - *&   $ &@ $& , *  "*  -& phased burst-error correction  $*& >!& -&- *& >-# &%* n   !$& , b  >! , &%* b + --! +  - "*-* & !$& , b -& -$>& , -&-%  $*& >! , &%* b  -&  phased >!b&-&-% -&  &@ $&  $*& >!6&-&-% -& , &%* 12 ! >& >&  -&- + >! , &%* 6 "*-* --!  - & , -&-% t $*& >! , &%* b *& r > 2tb &  O2+ & -&  "*-* *& $+-*&-# & &->& >+ *& ,"% +D

 C< C. C( C12 C1)  C1: C1 C7 C3 C16     C1. C1( C2 C) C1< : C11 C17 C13 C6 C:   %;& -&" FC<  C1  . . .  C13 G *& - $& --!$+% *& !$$& &, 6 3 . $ , *& + -+ *& ,   *& - $& , % *& %* -!   > " , *& + & $+ $ *& $+ -*&-# !& & * *& ! , *& - $&  &-* "  -!  , *& +  4& *! , &@ $& C< ‡ C. ‡ C( ‡ C12 ‡ C1) ˆ < OJ " ! 9  C. ‡ C1 ‡ C1( ‡ C17 ˆ < O&- -!  ! 9 O9 *" * * -& - -&-  $*& >! , &%* . O>9 *" * *& -&$% 6 3 . + -&  not -$>& , -&-%  $*& >! , &%* 6 O-9 &&4&  *" *  b 3 Ob ‡ 19 + -& - -&-  $*& >! , &%* b ,  + , b ‡ 1   $ & ! >&

22)

Cyclic codes

8.53 O! -&9 O9 & gOx9 ˆ Ox b 19 pb Ox9 "*&& pb Ox9   &!->& $+  , &%&& b  $& n1  *" * gOx9 %&&&  On n 2b9 -+--& "* n ˆ - On1  b9 "*-*  -$>& , -&-%  b-phased bursts , &%* b O&& >& (719 O>9 & *& &H!&-& , Onj On 2b9 j9 -+-- -& >& >+  &&;% *& -&  $ O9  & bj && *& -&$% O+ & !$*&9 >!&-&-% -$>+ *" *  j!1 bj =2bj ˆ 12  * *&& -&+ $-+ && *& &%& >! 8.54 &  >+ -+-- b ˆ 6 >!&-&-% -& , &%* 61 O9 & *& >   *& &%& >!  & & *&  ! &&& &!-+ O>9  +! *# *&&  !-*  -& "* *& &!-+ $&-&  $ O9P
@$ ,!+ 8.55 *" * , gOx9  *& %&& $+  ,  On k9 -+-- -&  , Ox ‡ 19    ; , gOx9 *& g9Ox9 ˆ Ox ‡ 19 gOx9  *& %&& $+  ,  On k 19 -& "*-* &H! *& % -& !  " ,  "&%* 8.56 &-& *& ,"% + -&" , *& O: 69 >  -& ,
@ $& (1)D O9 F+ *& '& & &&  *&& O&& > 369 &-& *& -!  , B9 & & *& , B - >& &+ &$&&   *& -& & -&-  & $& , "&%* < t  ;&,+ *& >! k > n mt  *&  & >&;& * *& % $+-*&-# @ H ;&"&   @ "* && , GFO29 *& * GFO2 m 9 *  & mt 3 n

 >+ *& &! , &- :1 * & * *& ! -& *  & < mt   *& -& &, *  & > n mt h

*& -& &->&  *&& 31 & -& BCH codes  * , *& ;& & +*!*!  -H!&%*&  *&& -& &  $   !-* >&-!& , *&& 31 &, O*& -& - *;& *%*& &  %&  ! -&9 >! *& >&-!& *&& & &,J-& &-%  &$&-+ &-% %*  , *&   *&

26.

BCH, Reed±Solomon, and related codes

&@ &- "& " && * , "& -*& &@-+ *& %* &% Oá<  á1  . . .  á n 1 9  -& %-+ >&- & -+-- -&   >+ *& &! , *$& ( *& &-% ! -+ >&- &  $&  + * NN-+--II ;&" ,  -& " " !  &J& ! & & , *& -&I  & *&  &- 36B37 "& " ,!+ &->& & ;& , &&# $I , ! &-% %* ,  -&

9.2 BCH codes as cyclic codes &- *& &J ,  t&-&-%  -& , &%* n ˆ n 1 j 2 m 1D C ˆ OC<  . . .  C n 1 9   -&" ,, iˆ< C i á i ˆ < , j ˆ 1 6 . . .  2t 1 O&H!;&+ , j ˆ 1 2 6 . . .  2t9 "*&& Oá<  á1  . . .  á n 1 9    , n - 4& && & , GFO2 m 9 , *&   -*& $$&+ *& -& >&- &  -+-- -&  *&&>+ *&  *&  $& & -*&+ ;>& , -+-- -& *&& NN-+--II  & *& , *& , O1 á . . .  á n 1 9 "*&& n   ; , 2 m 1  á   && & , GFO2 m 9 , & n /* &$&-  !-*   *& &J >&- &D C ˆ OC<  C1  . . .  C n 1 9   -&" ,, n 1


C i á ij ˆ &- &  cyclic code  *& && , *$& (  && * *   & COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1 >& *& %&&% ,!- , *& -&" CL *& O3:9 >&- & COá j 9 ˆ + *&& (1 C R Ox9 ˆ xCOx9 Ox n 19 "*-* & * R C Ox9 ˆ xCOx9 ‡ MOx9Ox n 19 ,  & $+  MOx9 *! , j ˆ 1 2 . . .  2t C R Oá j 9 ˆ á j COá j 9 ‡ MOá j 9Oá jn

19:

! COá j 9 ˆ < >+ O3(9  á jn 1 ˆ < -& á n ˆ 1  ," * C R Oá j 9 ˆ < , j ˆ 1 2 . . .  2t  * C R    *&  -& &J& >+ O3:9 "*-* & * *& -&  -+--  " ," , *&& (6 * &;&+  -&  -*-&4& >+

9.2 BCH codes as cyclic codes

267

 %&& $+  gOx9 ! *" - "& - $!& gOx9P --%  *& &J gOx9  *& & &%&& $+   *& -& & *& & &%&& $+  ,+% gOá9 ˆ gOá6 9 ˆ


ˆ gOá2 t 1 9 ˆ & A ˆ fá á6  . . .  á2 t 1 g , GFO2 m 9 &-& , A  &J&  >& *& & ,  GFO29-?!%& , && &  A & i A ˆ fâ2 D â 2 A i > &- & V^ j ˆ V Oá j 9

O3:179

 Vi ˆ

1 ^ V Oá i 9: n

O3:1)9

*&& & + &&%  !&,! &*$ >&"&& *&  &   ,&H!&-+  -& ,  %;& ;&- & , *&  *  NN$*& *,II  *&  &   -&$   NN & *,II  *& ,&H!&-+    *& ,"% &&  "& !$+ *& i* - $& , V >+ á ìi & , "& &J&  &" ;&- Vì  Vì ˆ OV<  V1 á ì  . . .  V n 1 á ìO n

19

9

O3:1:9

*&    ^ ì ˆ O V^ì  V^ì‡1  . . .  V^ì‡ n 1 9 V

O3:1(9

"*&&  O31(9 *& !>-$ & #&  n /& &;& *& $, , O31(9  >& 31+ O3129 W i ˆ 1n W ^ Ox9 ˆ W ^< ‡ W ^1 x ‡


‡ W ^ n m 1 x n m 1  -& W ^ Ox9   4& $+ W   , &%&& < n m 1  ," * W i ˆ < , at most n m 1

9.3 Decoding BCH codes, Part one: the key equation

263

;!& , i   W i 6ˆ < , at least m ‡ 1 ;!& , i *! "OV9 ˆ "OW9 > m ‡ 1 h /& &   &+  !-& *& #&+ &H! >! "& &&  ,&" & &J /* *& ;&- V J@& "& &J&  support set I  ,"D I ˆ fi D < < i < n

1  V i 6ˆ & 31(9 Corollary 1 For each i 2 I, we have Vi ˆ

ái

ùV Oá i 9 : ó V9 Oá i 9

O3:279

Proof , "& ,,&&& *& #&+ &H! O3269 "& %& ó V Ox9 V^9Ox9 ‡ ó V9 Ox9 V^Ox9 ˆ ùV Ox9O nx n 1 9 ‡ ù9V Ox9O1

x n 9:

O3:2)9

& * , x ˆ á i "* i 2 I , O32& &-;&& , ó V Ox9  ùV Ox9 *& &@ -+ + * , *& J ,&" ,&H!&-+  -& , V & #" *& & - >& &-;&& , ó V Ox9 & ;   $& &-!  *& & & , *& -+ "& !$$& * *& -&,J-& , ó V Ox9 & %;& >+

9.3 Decoding BCH codes, Part one: the key equation

2.1

ó V Ox9 ˆ 1 ‡ ó 1 x ‡


‡ ó d x d :

Corollary 2 For all indices j, we have d


V^ j ˆ

ó i V^ j i 

O3:2(9

iˆ1

where all subscripts are to be interpreted mod n Proof *& #&+ &H!  $& * ó V Ox9 V^Ox9 
& #&  n  "& *;& &J& ó < ˆ 1 ! " &H! O36 , aOx9  bOx9 &H!  dOx9 &  &H! , *& , uOx9aOx9 ‡ vOx9bOx9 ˆ dOx9:

O3:6(9

*& %* ;;& ,! &H!&-& , $+ D Ou i Ox99 Ov i Ox99 Ori Ox99 Oq i Ox99 *&  - & u 1 Ox9 ˆ 1

v 1 Ox9 ˆ & *& quotient  remainder &$&-;&+ "*& ri 2 Ox9  ;& >+ ri 1 Ox9D ri 2 Ox9 ˆ q i Ox9ri 1 Ox9 ‡ ri Ox9

&% ri , &% ri 1 :

O3:.+ u i Ox9 ˆ u i 2 Ox9

q i Ox9u i 1 Ox9

O3:.19

v i Ox9 ˆ v i 2 Ox9

q i Ox9v i 1 Ox9:

O3:.29

-& *& &%&& , *& & & ri & -+ &-&% *&& " >&   4& &L -  r n Ox9  ! ! * r n Ox9  *& %- , aOx9  bOx9  ,!*& & * *& && &H! &@$&% *& %-   & - > , *& % " $+  O-,
H O36(99 

9.4 Euclid's algorithm for polynomials

2.7

Table 9.2 $&& ,
!-I %* 

  


v i ri 1 v i 1 ri ˆ O 19 i a u i ri 1 u i 1 ri ˆ O 19 i‡1 b u i v i 1 u i 1 v i ˆ O 19 i‡1 u i a ‡ v i b ˆ ri &%Ou i 9 ‡ &%Ori 19 ˆ &%Ob9 &%Ov i 9 ‡ &%Ori 19 ˆ &%Oa9

+ !-  iL && > 313O9 Example 9.3 & F ˆ GFO29 aOx9 ˆ x (  bOx9 ˆ x ) ‡ x . ‡ x 2 ‡ x ‡ 1 *& >&*; ,
!-I %*  %;&  >& 36

*& i ˆ . & , >& 36 *" * %-OaOx9 bOx99 ˆ 1 O"*-*  >;! +"+9  "* $&+  , >& 32 +& *& &H! Ox 7 ‡ x . ‡ x 6 ‡ x 2 9aOx9 ‡ Ox : ‡ x ) ‡ x 6 ‡ x ‡ 19bOx9 ˆ 1 * &@ $&  -!& 
@ $& 3. h /& " ,-! ! &  $&+   >& 32 "*-* - >& &"&  v i Ox9bOx9  ri Ox9 O  aOx99: % $&+   *& ,- * &% ri

1

. &% ri  "& %& *& & &

&% v i ‡ &% ri , &% a:

*&  &! , * &- O *&&  O3.79 /& >&% "*  & 

O3:..9 O3:.79

379   # , -;&&  O3..9

Lemma 2 Suppose Euclid's algorithm, as described above, is applied to the two polynomials aOx9 and bOx9 Given two integers ì > < and í > < with ì ‡ í ˆ &% a 1, there exists a unique index j, < < j < n, such that:

2.)

BCH, Reed±Solomon, and related codes

Table 9.3  &@ $& ,
!-I %*  i

ui 1 < 1 2 6 . 7

vi

ri

qi

1 < x( ) . 2 < 1 x ‡x ‡x ‡x‡1 1 x2 ‡ 1 x6 ‡ x ‡ 1 x6 ‡ 1 x7 ‡ x6 ‡ x2 x2 x. ‡ x ‡ 1 x) ‡ x. ‡ x6 ‡ x2 ‡ 1 x‡1 x7 ‡ x. ‡ x6 ‡ x2 x: ‡ x) ‡ x6 ‡ x ‡ 1 1 ) x ‡ x. ‡ x2 ‡ x ‡ 1 x(
+ &H!% &% r j

1

> í ‡ 1

&% r j < í:

O3:.(9 O3:.39

*& >+ $&+  "&  *;& &% vj < ì

O3:7 ì ‡ 1:

O3:719


H! O3.39  O37 ì ‡ 1 > &% v ‡ 1  &% r j 1 > í ‡ 1 ˆ &% r ‡ 1 &-& , *&&   &@ !-* * O37.9  O3779 *  ! >& !H!& " &"& $&+  
H O3729  ,"D u j a ‡ vj b ˆ r j 

O3:7)9

ua ‡ vb ˆ r

O3:7:9

"*&& u   & !$&-J& $+  !$+ O37)9 >+ v  O37:9 >+ vj D u j va ‡ vj vb ˆ r j v O3:7(9 uvj a ‡ vvj b ˆ rvj :

O3:739

%&*& O37(9  O3739  $+ r j v  rvj O  a9 ! >+ O3.:9  O3769 &%Or j v9 ˆ &% r j ‡ &% v < í ‡ ì , &% a  + >+ O3.)9  O3769 &%Orvj 9 ˆ &% r ‡ &% vj < í ‡ ì , &% a  ," * r j v ˆ rvj  * ,- - >& "* O37(9  O3739  $& * u j v ˆ uvj  ! -& $&+  %!&& * u j  vj & &;&+ $ & * & * uOx9 ˆ ëOx9u j Ox9 vOx9 ˆ ëOx9vj Ox9 ,  & $+  ëOx9 *&
H! O37:9 >&- & ëu j a ‡ ëvj b ˆ rL - $% * "*
H O37(9 "& --!& * rOx9 ˆ ëOx9r j Ox9 h

*& &! , *&& 37 " >& !& -+  ! ,*- % -! , &-% %*  ,   &&B  -&  ,-& *&& -! "& " !-& *& %* - $-&!& NNEuclidOaOx9 bOx9 ì í9II De®nition If OaOx9 bOx99 is a pair of nonzero polynomials with &% aOx9 > &% bOx9, and if Oì í9 is a pair of nonnegative integers such that ì ‡ í ˆ &% aOx9 1, EuclidOaOx9 bOx9 ì í9 is the procedure that returns the unique pair of polynomials Ovj Ox9 r j Ox99 with &% vj Ox9 < ì and &% r j Ox9 < í, when Euclid's algorithm is applied to the pair OaOx9 bOx99

2.(

BCH, Reed±Solomon, and related codes

*& ,"% *&&

!

4& *& &! , * &-

Theorem 9.6 Suppose vOx9 and rOx9 are nonzero polynomials satisfying vOx9bOx9  rOx9 O  aOx99

O3:)+ O3)69  O3).9 &% vj Ox9 < &% vOx9 < ì  &% r j Ox9 < &% rOx9 < í h Example 9.4 & aOx9 ˆ x (  bOx9 ˆ x ) ‡ x . ‡ x 2 ‡ x ‡ 1 F ˆ GFO29  
@ $& 36 % >& 32 "& - >!& *& !$! , Euclid , *& &%* $>& $ Oì í9D Oì í9

EuclidOx 8  x 6 ‡ x 4 ‡ x 2 ‡ x ‡ 1 ì í9

O& ó Ox9 ˆ x 2 ‡ 1 ùOx9 ˆ x 6 ‡ x ‡ 1 h

 * $ *& $$- , *&& 3.  *& $>& , ;% *& #&+ &H! ,  -& *! >& $$&  + &;& "& $&  !  *& &@ &- 9.5 Decoding BCH codes, Part two: the algorithms & ! &-$!& *&  &-% $>&  "*-* "& >& & $+  *& & , &- 36 /& & %;&  &-&;& ;&- R ˆ OR<  R1  . . .  Rn 1 9 "*-*   + ;& ,  !#" -&" C ˆ OC<  C1  . . .  C n 1 9 , *& t&-&-%  -& &J& >+ O3:9 & R ˆ C ‡ E "*&& E  *& & $& ! %   &-;& C , R *& J &$  *& &-% $-&   - $!& *& syndrome polynomial SOx9 &J& >+ n

SOx9 ˆ S1 ‡ S2 x ‡


‡ S2 t x 2 t 1 

O3:)79

1 ij iˆ< Ri á 

, j ˆ 1 2 . . .  2t /& "  *& & , &- 36 "*&& S j ˆ * SOx9 ˆ V^Ox9  x 2 t  "*&& V^Ox9  *& %&&% ,!- , *& !& , , *& ;&- V &J&  O36)9  * *& #&+ &H! O36:9 >&- & ó Ox9SOx9  ùOx9 O  x 2 t 9

O3:))9

"*&& ó Ox9  *& &- $+   ùOx9  *& &&;! $+ 

*& &@ &$  *& &-% $-&   !&
!-I %*    $-! *& $-&!& EuclidOaOx9 bOx9 ì í9 &J&  &- 3.  ;& *& #&+ &H! , ó Ox9  ùOx9 *  $>& -& , *& ! >& , & * -!+ --!&  < t *& >+ O32+ &  1 %-Oó Ox9 ùOx99 ˆ 1 *! *& *+$*&& , *&& 37 & & "* aOx9 ˆ x 2 t  bOx9 ˆ SOx9 vOx9 ˆ ó Ox9 rOx9 ˆ ùOx9 ì ˆ t í ˆ t 1  * , *& $-&!& EuclidOx 2 t  SOx9 t t 19  -& 

27
& && & >+ *& ,- * ó O& 31 / ``Frequency-Domain'' BCH Decoding Algorithm / { for (j ˆ 1 to 2t)  Sj ˆ in ˆ1< Ri á ij L SOx9 ˆ S1 ‡ S2 x ‡


‡ S2 t x 2 t 1 L if (SOx9 ˆˆ&"&& *& " -& , -&  *  -& & *& &;& !+  *&   , ,  characters *& * bits  * &- "& " &J&  !+ &&B  -&

*! & F >& + J& "*-* -  && & á , & n) , r   J@& &%& >&"&& 1  n *& & ,  ;&- C ˆ OC<  C1  . . .  C n 1 9 "* - $&  F !-* * n 1


C i á ij ˆ &%%  *& -& & -&  codewords *& ,"% *&& %;& *& >- ,- >!  -& Theorem 9.7 The code de®ned by (9.67) is an On n r9 cyclic code over F  with generator polynomial gOx9 ˆ rjˆ1 Ox á j 9, and minimum distance d  ˆ r ‡ 1 Proof & C ˆ OC<  C1  . . .  C n 1 9 >&  >+ ;&- , &%* n ;& F  & COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1 >& *& -&$% %&&% ,!- *& O3):9 + * C   -&" ,  + , COá j 9 ˆ + *&& (6O>9 *& -&   On n r9 -+-- -& "* %&& $+  gOx9  $;& *& & >! d   >&;& * O3):9 + * , C^ ˆ O C^<  C^1  . . .  C^ n 1 9  *&  ,  -&" *& C^1 ˆ C^2 ˆ


ˆ C^ r ˆ < O-,
H O31199

*! >+ *&  %! & O *&& 369 *& "&%* , + 4& -&"  > r ‡ 1  *& *& * *& %&& $+  gOx9 ˆ x r ‡ g r 1 x r 1 ‡


‡ g <  "*& ;&"&   -&" * "&%* < r ‡ 1 *! d  ˆ r ‡ 1  && h Example 9.6 & *& O: 69 &&B  -& ;& GFO(9 , á  

9.6 Reed±Solomon codes

277

$ ;&   GFO(9 ,+% á6 ˆ á ‡ 1 *& %&& $+  , *& -&  gOx9 ˆ Ox á9Ox á2 9Ox á6 9Ox á. 9 ˆ x . ‡ á6 x 6 ‡ x 2 ‡ áx ‡ á6  , gOx9  ;&"&   -&"   Fá6  á 1 á6  1 ! -!-;&L *"&;& ,  -& *&&   &,J-& interpolation algorithm "*-*  -&+ &&  *& %%& &$ , ! , ! &- +

*& &@ *&& $& * ! Theorem 9.10 Consider the On de®ned by (9.67), where k ˆ n between the codewords C ˆ OC<  all polynomials POx9 ˆ P< ‡ P1 x over F, given by

k9 Reed±Solomon code over the ®eld F r There is a one-to-one correspondence C1  . . .  C n 1 9 of this code, and the set of ‡


‡ Pk 1 x k 1 of degree k 1 or less

Ci ˆ á

iO r‡19

Thus apart from the scaling factors á codeword are the values of a certain Ok

POá i 9:

iO r‡19

, the components of a given RS 19st-degree polynomial

Proof & C ˆ FC1  . . .  C n 1 G >&  J@& -&" /& &J&  NN"&II ;& , C -& D ˆ FD1  . . .  D n 1 G >+ Di ˆ á

iO r‡19

Ci

, i ˆ + O3):9 "& *;& C^1 ˆ C^2 ˆ


ˆ C^ r ˆ + O31)9 "& *;& Di ˆ POá i 9 , i ˆ % * "* O3)39 "& > C i ˆ á iO r‡19 POá i 9 "*-*  "* "& "& h

*& ,"% &@ $& !& *&&

31& 
@ $& 3) -- %  *&& 33 *&&   !H!& -&" C !-* * C1 ˆ á6  C. ˆ á  C) ˆ á.  & ! -!- * -&" /& >&% >+ >&;% * , I ˆ f1 . )g *&& 33 %!&&  &&-& *& &@&-& ,  6 3 : %&& @ , C , *& , G1.)

.< 1  1 ˆ / < 
+ J& GFO29 O>& 32:9Q*& *&& >- - $& OK$K$ &  !$&9 " +$-+  >& NN,,*&*&,II &  *!%* *& &% , *&& - $& ;& *&  $ J& GFO2 m 9    $  &&% $-   >&+ *& -$& , * >#  "& " --!& ! -! ,  &-& "* %!& 36 "*-* *"  +& *,&%& &-& , *& O: 69  -& ;& GFO(9 "* gOx9 ˆ x . ‡ á6 x 6 ‡ x 2 ‡ áx ‡ á6 O&&
@ $& 3)  3:9 /& ! "  *& $>& , decoding  -& "*-* ! !  >& H!&    *& &-% ,  -&  ;&" , *&  + , *& &J O- $& O3:99 "* O3):99 * *!  >& !$% & ! >&% >+ , + % *&  &-% $>&  /& & %;&  &-&;& ;&- R ˆ OR<  R1  . . .  Rn 1 9 "*-*   + ;& , 

9.6 Reed±Solomon codes

273

first 3 ticks closed last 4 ticks open

a3

a

1

a3

first 3 ticks down last 4 ticks up

Out In

Figure 9.3 +& - *,&%& &-& , *& O: 69  -& ;& GFO(9 "* gOx9 ˆ x . ‡ á6 x 6 ‡ x 2 ‡ áx ‡ á6 

!#" -&" C ˆ OC<  C1  . . .  C n 1 9 , *& On k9  -& &J& >+ O3):9 & R ˆ C ‡ E "*&& E  *& & $& -& >+ *&& 3: d  ˆ r ‡ 1 "& - *$&  -&-+ &,+ C !& "OE9 < br=2c   , *& & , *& -! "& * & t ˆ br=2c  ! & * "OE9 < t

*& J &$  *& &-% $-&   - $!& *& syndrome polynomial SOx9 ˆ S1 ‡ S2 x ‡


‡ S r x r 1 

O3::19

n 1 "*&& S j ˆ iˆ< Ri á ij  , j ˆ 1 2 . . .  r + *& &! , &- 36 , "& &J& *& NN"& & $&II >+ V ˆ OE<  E1 á E2 á2  . . .  E n 1 á n 1 9 *& SOx9 ˆ V^Ox9  x r   *& #&+ &H! O3269 &!-& >&- &

 x r

ó Ox9SOx9  ùOx9 O  x r 9 "*&& ó Ox9  *& - $+   ùOx9  *& &;! $+  , *& ;&- V

 * $ *& &-% $>&    &@-+ *&  &   " ,  -&  &->&  &- 37  $-! , *& $-&!& EuclidOx r  SOx9 t t 19  -&  " &! *& $ , $+  OvOx9 rOx99 "*&& vOx9 ˆ ëó Ox9  rOx9 ˆ ëùOx9 ,  & 4& -  ë

*& J &$  *& &-% %*   !& ó Ox9  ùOx9  && & *& & $& E ˆ OE<  E1  . . .  E n 1 9  *&-& *& % -&"

2)
&-!&  -& & >+  * E i ˆ <  1 ,  i *! , *&&   &  $ i & E i 6ˆ &- & *& inner channel9   &%  -& ,   * &@ $& *& !& -*&     "* 1) $!L *& &! , * &- !%%& * "& &% *&& $!  !$!  && & , GFO1)9 *& *  ,! & ;&- ;& GFO29  & ! " -& !%  O17 119  -& ;& GFO1)9  &!-& *& &  *& !& -*&  !&  %!& 3:

*&  &-&  %!& 3: #& 11 ,  + > á ˆ Oá<  . . .  á1< 9 , GFO1)9 O"*-* & &+ .. > , *& % !-&9  $!-&   -&" C ˆ OC<  C1  . . .  C1. 9 *& !& -*& *& %>& C    &-&;&  R ˆ OR<  . . .  R1. 9 *&  &-& *& $!-&  & & ⠈ Oâ<  . . .  â1< 9 , á "*-* " >& &H!  á , *& !& -*& * -!&  & * " + > & *! , E Oˆ + , &-& &  %!& 3) *& 17 $>>+ , &-& &  %!& 3:   & * kˆ6 17 k E917 k ˆ & H!$+  ;& GFO1)9

*& $&-&% &@ $& !& >* *& %&& & , --&  *& & "*+  -& &  !&,!  --&& +&  Any -& - !- +& - >& &%&   + !& -*&   %!& 3( "&;& , * $ , ;&"  >& !&,! "& ! >& >&  &%  !& -& -$>& , -&-%  , *& & -!& >+ *& !& -*& "*-*  #&+  >&  ;&+ - $&@ >& -&  & & -!& >+ & &-& ,!& /*& *& & &-& , *  "*& Ov1  . . .  v k 9 6ˆ Ou1  . . .  u k 9  %!& 3( *& + > v1  . . .  v k !!+ >& $--+  && >-&  u1  . . .  u k  * & * &  *& !& -*& &  --!  >! , &%* k  "& *;& &+ && *  -& & "& !&  >!& -&- *  *& & "*+  -& &  "&$& !&  !& -&  --&& +&  9.7 Decoding when erasures are present /& *;& && *    -& - -&- !$& &  * &- "& " && * *&+ -  -&- *& - , -*& K" -& erasures  &!&   $+  -*& + > "*-*  &-&;& &%>+  &@ $& -& *&
%* " BLOCK , *& * &&  -*%& , O  A "& %& BLACKL *   error  *& * $ "&;& , *&  & " !,,&  erasure  *& * $ *& &!  BLCK "*&& NNII  *& erasure symbol  $--& &!& & H!& -  *&+ - >& &@$&-&  --! "*& *& -*& & >&- &

9.7 Decoding when erasures are present

2):

!!!+ &;&& ,  *  &  &@ $& , +! & +%  #  *& $   "K+% ?& $& ;&*& +! -;&  erased ! && "  #& "* +! & +%  +L *&+ "  $+  >& >&  !& +!  * &- "& " &  &*% >! &!& -&- /& " && *  $-$&  &!&  + *,  *  -&-   & O *&& 3119L  "& " && *"  ,+ *&    &-% %*   &  -&- >* &!&  &

 &  -*& "*-* - $!-& &!&  "&  & "&  $+ &%& *& !&+% + > & F  F ˆ F [ fg "*&& NNII   >;&  $&- &!& + > *& + "& transmitted + > & *& && & , F >! + && &  F - >& received *&  *&&- &! >!  !&! &!&& -&- ," O $& "* *&& :29 Theorem 9.11 Let C be a code over the alphabet F with minimum distance d. Then C is capable of correcting any pattern of e< erasures and e1 errors if e< ‡ 2e1 < d 1 Proof  $;& *& *&&  "& J !-& *& extended Hamming distance d H Ox y9 >&"&& + >  FD & ' < , x ˆ y d H Ox y9 ˆ 1 , x 6ˆ y  &*& x  y  NN  II (1 , x 6ˆ y  & , x  y  NN  II: 2

*! , &@ $& , F ˆ f& *& O: 69 -+-- -& , "


@ $& (2 "* -&

C< ˆ + *& %&   $ &-&-% -& O *&&    O11 )9 + -& ;& GFO69L && > 3).B3):9 /& >&% "*  4% ! >&*&&- ,-  *& 26 & ;&- $-& ;& GFO29 "*-* "& - V26    % $*&& , ! 6 -       26 26 26 ˆ 2& #&  *& -&"   O26 129 >+ -+-- -&M  -%*&&- &  *& "& &&  -!-  >+ -+-O26 129 *&&&-&-% -& & & "* d  > : /& >& *& -!-  -& $$&& , *& J& GFO211 9 -& 211 1 ˆ 2! "& "  && * &@$- ,-4  *& & , * &- /& - " &J& *& + -& De®nition The O26 129 Golay code is the binary cyclic code whose generator polynomial is gOx9, as de®ned in (9.85) or (9.88). "  OM9 "& *;&    *" * *& -&I

*& J &$  * &-  *& &+

 !

"&%*  > :

Lemma 3 Each nonzero Golay codeword has weight > 7 Proof  ;&" , *& !-!& , *& & B , 4& , gOx9 O-,
H O3(799 "& && * , &;&+ -&" C COâ9 ˆ COâ2 9 ˆ COâ6 9 ˆ COâ. 9 ˆ : Lemma 5 If C is a Golay codeword of even weight w, then w  < O  .9 Proof & *& %&&% ,!- , C >& && >+ COx9 & COx9 ˆ x e1 ‡ x e2 ‡


‡ x e w 

O3:319

"*&& < < e1 , e2 ,


, e w < 22 -& C >&%  *& + -& COâ9 ˆ &-!& w  &;&   - $! #& $-&  GFO299 *! ~  COx9 COx9

22


ìb x b

O  x 26

19

bˆ1

"*&& ì b  *& ! >& , && $ Oi j9 "* e i , ! O33)9 &-* ì b  &;&D ìb  < " , e i

O  29

e j  b *&  e j

+ -& *&& & wOw

b ˆ 1 2 . . .  22:

e i  26

ì b ˆ ì26 b 

O3:3(9

b O  269 *!

b ˆ 1 2 . . .  11:

19 &   *& ! 22


e j  b O  269 +

ì b ˆ wOw

O3:339

 *& %* & , O33:9

19:

O3:1+ appending an overall parity check & >+ &J%  2.* - $& C26  ,"D C26 ˆ C< ‡ C1 ‡


‡ C22 :3 , &;&+ + -&"  &@&&  * "+ *& &!% -&   >+ & O>!  %& -+--9 O2. 129 -& "*-*  -& *& O2. 129 extended Golay code     $& & *&  $;& *& ,"% *&&  O&& >& 3739

Theorem 9.14 The number of codewords of weight i in the O2. 129 extended Golay code is < unless i ˆ + O&& >&  373 3)29 /& --!& "*  & & # >! *&  $& & , *& + -& -& *& O26 129 -&  -+--   -& * "& -! &%  11 %& *,&%& &-& ,  O&& &- (2   >& 3)- &-% %*    &+L "& -! &+ ,+ *&  &-% %*  %!& 31  32  -&- &;&+ $& , "  ,&"& & >! *& -&  NN--&+II -$>& , -&-% *&&M !&+ *"&;& *& -&    &!%*  * *& + & NN>& #!$II %* -!&  &- :2  !!+ $-- O&& > 3)69

2(2

BCH, Reed±Solomon, and related codes Problems

9.1 /& "  &- 31 * *& ,!- fOV9 ˆ V6 #& *& @ H 2 ,
H O329  *& $+-*&-# @ ,  "&-&-% -& ;&%& "*&*&   *& ,"% -& fI "#D O9 fOV9 ˆ TV "*&& T   & ,  , V m  O>9 fOV9 ˆ a< ‡ a1 V ‡ a2 V2  "*&& V  &%&   && & , GFO29 O-9 fOV9 ˆ V 1  "*&& V 2 GFO2 m 9 9.2 !$$& F   J& J& "* q && & O9 , a   >+ && & , F &J& *& Oq 19&%&& $+  f a Ox9 ˆ Ox a9 q 1 1  *& ;!& , f a Ox9 , &-* , *& q && & , F O>9 % *& &! , $ O9  *&"& *" * &;&+ ,!- f D F ! F - >& &$&&&   $+  , &%&&   q 1 9.3 O * $>& %;&  %&&4 , *& '& & &&  *&&  "*-*  &&&  *& $, , *&& 319 & Pi Ox9 >&  - $+  , &%&& i , i ˆ & - && & *" * . 0 P< Ox1 9 . . . P< Ox n 9 6 7 P1 Ox n 9 7  n 1  n 6 P1 Ox1 9 . . . 6 7 &6 Ox j xi 9: 7ˆ . . 6 7 iˆ1 jˆi‡1 .. .. / 1 Pn 1 Ox1 9

...

Pn 1 Ox n 9

F HintD , xi ˆ x j  *& &, & ;*&G *& '& & &&  *&&  *& $&- -& Pi Ox9 ˆ x i  9.4 &&   $&!-& % ,  %* , - $!% *&  & , *& t&-&-%  -& , &%* 2 m 1D {

}

S ˆ f1 6 . . .  2t 1g; k ˆ 2 m 1; while (S is not empty) { u< ˆ least element in SL u ˆ u< ; do { delete u from SL k ˆ k 1; u ˆ 2u  2 m 1; } while (u 6ˆ u< 9 }

O9 *" * "*& *& %* & & *& &%& k  &H!  *&  & , *& t&-&-%  -& , &%* 2 m 1

Problems

9.5

9.6

9.7

9.8 9.9

9.10 9.11

O>9 & *& %*  - $!& *&  & , *& t&-&-%  -& , &%* )6 , 1 < t < 61 O9 ;& * *&  & , *& "&-&-%  -& , &%* n ˆ 2 m 1  n 2m ,  m > 6 O>9 & %&&+ *" * , + J@& t > 1 *&  & , *& t& -&-%  -& , &%* n ˆ 2 m 1  n mt ,  !,J-&+ %& m O-9 /*  *&  & ;!& , m< !-* * *& three&-&-%  -& , &%* n ˆ 2 m 1 *  & n 6m ,  n > m< P O9  &-* t  *& %& 1 < t < : - $!& *&  & , *& t& -&-%  -& , &%* 17 O>9  &-* , *& -&  $ O9 --!& *& %&& $+  ! %  $ ;&  á  GFO1)9 * J& *& &H! á. ˆ á ‡ 1 O,
@ $& 319 
@ $& 31 "& - $!& *& %&& $+  , *& *&&& -&-%  -& , &%* 17 !& *& ! $ * *& $ ;&  á , GFO1)9 J& á. ˆ á ‡ 1 , "& * -*&  $ ;&  ,+% á. ˆ á6 ‡ 1 & "* "! *& %&& $+  *;& !& !  >&P ;& *& ;&&  , !
H O3129 & *& J& GFO:9  &$&&& >+ *& & , &%& f9 % 6  *& &&& $ ;& )*  , !+ J *&  , *& ;&- V1 ˆ O1 2 6 . 7 )9  V2 ˆ O1 6 2 ) . 79 O-9 /*+  +! !$$& *&  , V2   !-* NN $&II * * , V1 P ;& * *&  , *& $*&*,& ;&- Vì  &H! O31:9  %;& >+ *& , ! O31(9 O&&4&  -&9 & gOx9 >&  $+  "* -&,J-&  GFOq9 "*-* ;& x n 1 !*& ! & * á   n*  , !+   & &@& J& , GFOq9  * gOá i 9 ˆ & *& -+-- -& "* %&& $+  gOx9 *" * *&  ! -& , C  > d F HintD & *&  %! &G  *& -;&& , *& NN %! &II O *&& 369 !&P *   , V   ;&- , "&%* w &  &-&+ ," * *&  V^ ! *;& w 1  & -&-!;& 4& - $&P , +! "&  yes %;&  $, , +! "&  no %;&  &@$- -!&&@ $& ;& * %-O V^Ox9 1 x n 9 ˆ i=2 I O1 á i x9 "*&& V^Ox9   &J&  O31.9  I   &J&  O3139 ^ & #" *& & - >& *" * , + d -&-!;& - $& , V --!& $;& "&  #" ó V Ox9 O-, + 329

*& J& GFO1)9  &$&&&  >& 31 -   $ ;& 7*  , !+  &+ á6 "*-* , *& & & , * $>& "& * && >+ â & V ˆ O1 â.  â7  &  $"& && ;&  J& F , ì  í & &%;& &%&  Oì í9 Pade approximation  AOx9    ,!- pOx9=qOx9 !-* * O9

qOx9AOx9  pOx9 O  x ì‡í‡1 9

O>9

&% qOx9 < ì &% pOx9 < í:

% *&& 37 *" * , &-* Oì í9 *&&  O$ ,  - ,-9  !H!& $ O p< Ox9 q< Ox99 !-* * , O9  O>9  * *& pOx9 ˆ ë p< Ox9  qOx9 ˆ ëq< Ox9 ,  4& - ë *& $ O p< Ox9 q< Ox99  -& the Oì í9 Pade approximant  AOx9 &,&%  >& 36 - $!& *& &C $$@   AOx9 ˆ 1 ‡ x ‡ x 2 ‡ x . ‡ x ) ‡


;& GFO29 "* ì ‡ í ˆ : 9.22 /* *&  & &!$ 
@ $& 37 &-& *& ,"% + -&" , *& O17 79 *&&&-&-%  -&D

Problems

2(7

R ˆ FR<  . . .  R1. G ˆ F11& , Pr  * "+   & -& ;& Fq m "* &%* n  & r ‡ 1   ! -& n r 9.35 *& &!$ >&% *&  &   > 36. -&$%  &-* f 2 Pr & C ˆ OC<  C9<  C1  C91  . . .  C n 1  C9n 1 9 2 GFOq m 92 n >& &J& >+ C i ˆ f Oá i 9 C9i ˆ á i f Oá i 9 *" * *& & , ;&- *! >&   & -& ;& GFOq m 9 "* &%* 2n   & r ‡ 1 !-* * "* &-* 4& -&" *&& &  & n r - $ OC i  C9i 9 9.36 & ö D GFOq m 9 ! GFOq9 m >&  && & $$% , GFOq m 9 

Problems

2(:

GFOq9 m  #& *& -& ;& GFOq m 9 &J&  > 367  #&    -& ;& GFOq9 >+ $$% *& -&" C  OöOC< 9 öOC9< 9 . . .  öOC n 1 9 öOC9n 1 99 *" * *& &!% GFOq9 & -& * &%* 2mn   & mOr ‡ 19  * "* &-* 4& -&"  % *& n !>;&- OöOC i 9 öOC9i 99 *&& &  & n r - & 9.37 O * $>&   ! , $-& &$& $$&-&9 & fx1  . . .  x M g >&  & , M - ;&- , VnOF 2 9  & w i ˆ w H Oxi 9 && *&  % "&%* , xi  & p ˆ Ow1 ‡


‡ w M 9=nM ;& * % M < nH 2 O p9 "*&& H 2  *& >+ &$+ ,!- F HintD & X ˆ OX 1  X 2  . . .  X n 9 >&   ;&- "*-*  &H!+ #&+  ! & + , *& ;!& xi   & p j && *& ,- , *& M ;&- * *;&  1  *& j* -& " ;&,+ *& ,"% % , &H!&  &H!&D % M ˆ HOX9
 367 $&-4&  q ˆ 2 n ˆ 2 m   Oá<  á1 . . .  á n 1 9 + &% , *& && & , GFO2 m 9 & r=2 m ˆ r  *" * *& &!% -& *;& O9 &%* ˆ m2 m‡1  O9 & ˆ 12 r ‡ 21m  $ % O9 d n > O1 r9 H 2 1 12 ‡ %22O1m r9 

*&& -& & *& Justesen codes && >;& F HintD  $;& O9 !& *& &! , > 36:G 9.39 + *" * , + < < R < 12 *&&   J& &H!&-& , 0!&& -& ;& GFO29 "* &%* n i   & k i    ! -& d i !-* *D 

i!1



i!1



i!1

n i ˆ 1

k i =n i ˆ R

!$ d i =n i > H 2 1 O1=29
O1 ˆ & 9.42   %;& ;!& , d *" + $ , &%;& &%& Oe<  e1 9 & *&& !-* * e< ‡ 2e1 < d 1P 9.43 , m  n & $;& &%& !-* * m ‡ n  &;& *" *

2((

BCH, Reed±Solomon, and related codes 2

3 4 5 m n m‡ n ‡ ˆ : 2 2 2

O&& *& & #  &&+ $&-&&%
H O3(.99 9.44 & *& O: .9  % -& ,
@ $& :6 *& -& * d  ˆ 6   >+ *&& 311   -$>& , -&-% + $& , e< &!&  e1 & $;& e< ‡ 2e1 < 2 &% *   &-& O, $>&9 &-* , *& "  $ O9 O>9  O-9 O9 F1 1 1 < <  9 F<  1 1 1 < 1G O-9 F< 1  1 <  1G O9 , R    + -*& ;&- , &%* : -% &@-+ & &!& "*  *& $>>+ *  " >& !H!&+ &-& >+ *&  &-% %* !-&  *& $, , *&& 311P 9.45 *" * , ó 1 Ox9  *& &- $+   ùOx9  *& & &!& &;! $+  ,  &&!& &-% O&& &H!  O3::9  O3(199 *& %-Oó 1  ù9 ˆ 1 9.46 ;&%& *& $>>+ , &-& & ,  &&B  &-& !& *& ,"% --! -& O9 r &!&  1 & O>9 r 1 &!&  1 & O-9 r ‡ 1 &!&  & 9.47 & *& O17 :9  -& ;& GFO1)9 O%&&& >+  $ ;&  ,+% á. ˆ á ‡ 19  &-& *& ,"% &-&;& " R ˆ Fá16 1  á1< á12 á)  á7 á16  á1 á( á: á2 á3 G: 9.48 % *& !%%&  *& & , &- 3: &-& *& ,"% + ;&- , *& O17 79  -& "* %&& $+  gOx9 ˆ x 1< ‡ x ( ‡ x 7 ‡ x . ‡ x 2 ‡ x 2 ‡ 1 O-,
@ $& 319 R ˆ F1 1  < < <  < < < 1  1 < 1G: O9 & *&  &  - $& O>9 & *& ,&H!&-+  - $& 9.49 &  On k9 & -& "* d  ˆ d , *&& &  & *&& 311 %!&& * *& -&  -$>& , -&-% + $& , !$  d 1 &!& *" * * &! - >&  $;& >+ *"% * *&&   & & & , d &!& * *& -& I -$>& , -&-% 9.50  &- (. "& -&& -& -$>& , -&-% %& >! , &  ! ! *   ;&+ !-* &&  -&- %& >! , erasures ,& % * $>&  +! " %&& * *   O9 , C   On k9 & -& "*-*  -$>& , -&-% + &!& >! , &%* b  & *" * n k > b O, *& &%& >! -+  *&& (19 *" * + On k9 -+-- -&  -$>& , -&-% + &!& >! , &%* n k  & O-9 & *& O: 69 -+-- -& ,
@ $& (2 &- *& ,"%

Problems

9.51

9.52

9.53

9.54

2(3

-&" "*-* *;& !,,&& &!& >! , &%* .D O1<      "* && & x  y *! >% R ˆ O1 x 1 y 1 < 19 O9 & *& ,- * &;&+ -&"  *& O: .9  % -& J& *& &H! HC T ˆ  *&& &H!  *& *&& && & &$&&% *& &!& /& -! *& ;& *&& &H! , *& && & *! -&-% *&& &!& & *&& 311 + %!&& * *& -&  -$>& , -&-% " &!& /*I "% *&&P  * $>&  "& -&  &;& $$-*  -&-% &!&  & "*-* ;;& *& & , NN%!&%II *& ;!& , *& &!& O9 ! & J * *& -&  >+ & *& J& F  *&& 311  GFO29 !$$& "& *;&  -& C "*  ! -& d  * "& *;& &-&;&  + -&" -% e< &!&  e1 & "* e< ‡ 2e1 < d 1 !$$& * "& -*%&  *& && $  & &% 
@ $& 33  * d  ˆ .   >+

*&& 31&  +! & #&  ;&%& *& : . ˆ 67 $>& &!& $& , 4& ,!   && & "*-*  *& & -&->&  $-! $&& J *& number , "&%* ,! &!& $& * & -&->&

* $>& --& *& $>>+ *   + &&-& ;&- , GFOq9 n " >& &-& >+  &-& ,  &&B  -&

23
9 & *& , ! , $ O9  - $!& *&   ,  J@& ;!& , t  q ! 1 , *& $>>+ *   + &&-& " , &%* q 1 " , "*  % -& t  & ,  & -&"   t& -&-%  -& , &%* q 1 ;& GFOq9 O ! & * *& -&I &!-+  r ˆ 2t9 & C >&  On k9 >+ -+-- -& "* %&& $+  gOx9  $+-*&-# $+  hOx9 O9 *" * , hO19 6ˆ 9 *" * , *&&   $ Oè1  è2 9 ,  , hOx9 !-* * è1 è2 ˆ 1 *& &;&+ -&"I "&%*  ;>& >+ ,! F HintD *   %&& 4 , *& &!  &  7G  *& &@ "& $;& * *& O26 129 + -& * d  > : *" *  ,- d  ˆ : , * -&  *  " "+D O9 + &@ % *& %&& $+  gOx9 O>9 + *"% * any O26 129 >+ & -& ! *;& d  < : *" * *&&   O3+ & -& "* d  ˆ 7 &  $&,&- "&-&-% -& , &%* 3& , 12> + & , 1 2  "&%* -&$%  &> & *" * *& ! >& ,  "&%* + & -&$%  "> &  rO3< r9  & $  && & rG

*& O26 129 >+ + -& &J&  &- 3( "*& - >& "* + & >& #!$ &-% * *& $$&+ * &;&+ & $& , "&%* < 6 " >& -&-& O9 &->&  & "* *& &-& "  , *& & $& * "&%* . O>9 /* , *& & $& * "&%* 7P O-9 &&4& *& &! , $ O9  O>9  &-* &%& t  *& %& . < t < 26 "* " *& &-&  , *& & $& * "&%* tP ;& *&& 31.

* $>& --& *& ! >& , -&" , "&%* (  *& O2. 129 &@&& + -& O9 *" * *& ! >& , -&" , "&%* (   4&  *  " "+D O19 + &@ % *& %&& $+  gOx9 , *& % O26 129 + -&L O29 + *"% * any O2. 129 >+ & -& ! *;& d  < ( O>9 ;& 1< * *& -& - &@-+ :73 " , "&%* ( *" * , + !>& fi1  . . .  i7 g , J;& && & , f&% !&  -&-  & $& , "&%* *&&  &9 O9 *" * *& -& - &&-  & $& , "&%* . O>9 ;& * *& "&%* &! & , *& -&  1 ‡ :73x ( ‡ 27:)x 12 ‡ :73x 1) ‡ x 2.  , &-* e  *& %& . < e < 2. - $!& *& ! >& , & $& , "&%* e * *& -& " &&- O-9 " ! & *& &-& + & $  -&- & $& , "&%* two  &  &$& $ O>9 O9 " ! & *& &-& + & $  -&- & $& , "&%* one  &  &$& $ O>9 O&9 + ! & *& &-&  !&   detect-only mode & , *& + &  4& *& &-&;& "  --&$&  -&- >! *&"&   &?&-& &$& $ O>9 9.62  * $>&  "& " >&K+ -& &-& , *& O26 129  O2. 129 + -& O9 &%  *,&%& &-& , *& O26 129 + -& O>9 + ,+% +! &%  $ O9  *&"& - & !$ "*  &% ,  &-& , *& O2. 129 &@&& + -& 9.63 -! *& 4&  - $&@+ ,  + & >& #!$ &-& , *& O2. 129 + -& In Probs. 9.64±9.67 we will investigate the ternary Golay code. Observe that in the vector space GFO611 9 a Hamming sphere of radius 2 contains     11 11 1‡2 ‡. ˆ 2.6 ˆ 67 1 2 vectors. This suggests that it might be possible to perfectly pack GFO611 9 with :23 ˆ 6) spheres of radius 2 The ternary Golay code does this. It is an O11 )9 linear code over GFO69 whose codewords, when taken as sphere centers, produce such a packing. The code is de®ned as follows. Since 67 1 ˆ 11
22, it follows that GFO67 9 contains a primitive 11th root of unity, which we shall call â Since over GFO69 the factorization of x 11 1 is x 11 1 ˆ Ox 19 gOx9 g~Ox9, where gOx9 ˆ x 7 ‡ x . x 6 ‡ x 2 1( and g~Ox9 ˆ x 7 x 6 ‡ x 2 x 1, we may assume that â is a zero of gOx9 The O11 )9 ternary Golay code is then de®ned to be the cyclic code with generator polynomial gOx9 To show that the spheres of radius 2 around the 729 codewords are disjoint, we must show that the minimum Hamming distance between codewords is > 7, that is, every nonzero codeword has weight > 7 The following problems contain a proof of this fact1< 9.64 *" * *& -&I  ! "&%*  > . F HintD & *&  %! &

*&& 36G 9.65 *" * , C< ‡ C1 ‡


‡ C1< ˆ & >+ 6 F HintD && &  . &- 3(G 9.66 ,  *& *& * C< ‡ C1 ‡


‡ C1< ˆ á 6ˆ &&+ -;&& *  ,- -H!&%*& * -$& &  +*!*! *& -& "&& &-*&& &*!*! -H!&%*&  BCH codes    $  & & >& * + *& -&  *& &-% %*  "&& -;&& >+ *&& &+ "&  *& *+ , *& &-% %*  && $$ 67.B677 6 O$ 26:9 , *& -*-&- , F  J& "& ! & * *& -*-&- &  ;& n . O$ 2..9 + -;& *& &%&& , *& 4& $+   1 *  &  * >- ,- #& &%Oab9 ˆ &%Oa9 ‡ &%Ob9 &%Oa ‡ b9 < @O&%Oa9 &%Ob99 &- " * &;& , & , a  b  *& 4& $+  7 O$ 2.:9 * $,  !&  0  *&& ) O$ 27.9    >!   $$- *& J& F " >& GFO2 m 9 ,  & m > 1 "&;& *& !&+% *&+ %& *!%* &H!+ "& , + J& J&    "& * #&  !&-&+ &-  F : O$ 2:19 & , -!&  & , *& && - $& , C "&& -!+ & :1:O>9 1< O$ 23& !& , + ,,&& $ , ;&"  * !-+ &- "& $&& *&& $$-*& "*-* "& *;& -& *& polynomial matrix $$-* *& scalar matrix $$-*  *& shiftregister $$-* O *&& *& $$-*& *& state-diagram $$-* *& trellis $$-*  *& tree $$-* " >& %;& &  *& -*$&2 9 ·The polynomial matrix approach &- , *$& : *  >+ On k9 & >-#6 -& - >& -*-&4& >+  k 3 n %&& @ G ˆ O g ij 9 ;& F2  On k9 convolutional code O9   -*-&4& >+  k 3 n %&& @ GL *& ,,&&-&  * ,  -;! -& *& && g ij & polynomials ;& F2  &@ $& *& @ G ˆ Fx 2 ‡ 1 x 2 ‡ x ‡ 1G  *& %&& @ ,  O2 19  "*-* "& >&  1 , ,!!& &,&&-&  +  Gˆ

1 < < 1

x‡1 x



 *& %&& @ ,  O6 29  "*-* "& -  2 /& " &J& *&&  $ ! >& * & -& "*  D

*& memoryD 236

23.

Convolutional codes Mˆ

@ F&%O g ij 9G:

O1-# -& - >& ;&"&    "* M ˆ  ! >& $$&  *& -&,J-& ,  k!$& , $+  I ˆ OI < Ox9 . . .  I k 1 Ox99 *& *& NN-&"II C ˆ OC< Ox9 . . .  C n 1 Ox99 "*-*   n!$& , $+   &J& >+ C ˆ I
G

O1& &-& ;
H O1& &-& ;
H O1 $& & , * -&$&-& & !+  *& - @ $$-*  I ·The scalar matrix approach *&  ! >"& O-9 &$&&  ,  $+  -&" C ˆ OC< Ox9 . . .  C n 1 Ox99  >& >+ &&;% *& $+ I -&,J-& *! , *& j* $+   Cj Ox9 ˆ C j< ‡ C j1 x‡ . . .  *& - &$&& , C  C ˆ OC-# 1 1 < 1 1 1

 1 < 1

1 1 1

Example 10.6 , "& #& L ˆ 2   %&& @

1 < 1 < < 1 < < G2 ˆ   1
 I j >!   *& $&-&% " > I j 1  I j 2  *& &-& ! & & >& " $! >    *& -!& >L *  "*+ *& -&    *;& & + M ˆ 2  , &;&+ $! > *&& & " !$! >L *! *& -&I &  1=2 + & * *& &-& , % 1& , & >+ ! % *& j* !$! & , &-* , *& *, &%& Example 10.7 *,&%& &-& ,  2  !&  % 1& &D @& &$&& *& ,! &L   , & &  *& -&$%   $! , NN+   &%&   -&$%  NN1II >+  *& &%&  *& >&   &%& &$&& *& &-&I !$!   ;& , & &  *&  &@ $& -& *& &%& %% , & cO1&*; "*& *& *, &%&  % 1+ "&% *!%* *& & %  "& + ;& *&  & &%& +  &L * #&  ,J-!  #&&$ -# , "*&& "& *;& >&&   >&- & &&>&  ,+ *& & % >+ -!% *&  & ,  & &*$ *& >& "+   *   *;&  ,,&& -$+ , *& & % , &-* -#   ? *& %&*&   trellis diagram  *"  % 1& -& *& depth  *& &$* j $$&-& ,  & " >& && >+  !>-$ j &$* j &  ?& >+  &%&   &$* O j ‡ 19 & ,, *&&   &%& ?% *& " &  *& & %  O  &@-&$ --! , j ˆ <  1 >&-!&  j ˆ < *& *, &%& , % 1+ -% *& $$$& $* *!%* *& &  &@ $& *& !$! & O111 >+ > &-* , *& ). -&"  R "&;& *&&   !-*  $& &* "*-* #& ;%& , *& ,- * *& ). -&" -&$  *& ). $* , a<  a(  *& & , % 1&"&&  % 1&  *& -&$% " > , R  &@ $& *& b6 ! c. &%& %& *& >& d H O+ S *& & , & fa b c dg &@ , s t 2 S  *&&   &%& %% , s  t  *& & %  "& &J&

Figure 10.8 *& ;& , *& & , % 1&"&& a<  a( 

6& <  1 --%  "*&*& *&  , s  t -&   <   1 $! , *&&   !-* &%& BOs t9   &J& O&& % 1I %*  %;&  % 1 %* , % 1&  * *& - $&@+ , *& %*   &@$&+ -&% ,!- , M O*!%*   + &  L9     +  $-- &* , &-% I "* &;&+   ;!& , M && "*

Figure 10.12 '&>I %*  $$&  *& & , % 1& *&   &;& , -& "* k ˆ 1 " "& -& & %&& -*&  &- !&;&  " $!  I O>! && > 1 216 *   %&&    @ ! #&* &-& ! -&  -&" C i ˆ OC i<  . . .  C i n 1 9 , "*-* *& $>>+ PfRjCi g ˆ

n 1 

pORj jCij 9

jˆ
& "*&& R ˆ OR<  . . .  Rn 1 9  &-&;&  *& pO yjx9I & *& -*&I  $>>& -& *& %*    -+ -&% ,!- ,  %! & *&  &-& + &H!;&+ # ,  -&" C i , "*-* LOC i  R9 ˆ

n 1


% pORj jCij 9

O1& /* * &  * *& '&> %* , % 1#& !$  L ‡ M n> ;&- , *&&   &%& %% , & s  & t  *& & %  && >+ C s t *& n> &-& !$! -&$%  *  *  *& >&  *& s ! t &%& O&& % 1 &-*H!&  *& $>& , &! &%  !-* $*  $$+ *& &!  > $&, -& & & , $&-J- -;!  -&  !! "& " !& *& --&$ "*  1  -! %&&4 & & % *& & % , % 1& , 1I  *& >& , *& &%& - $% *& $* O &@ $& *& $* acbaacddba * "&%* 129 /& "! $&*$ #&  -! *& ! >& , $* , a  a , "&%* i ,  & J@& &%& i >! * ! >& " "+ >& &*& <  J& >&-!& , *& $ , "&%* <  a & "+ ! * $>&   >&;& * &;&+ $* , a  a - >& !H!&+ &- $&   &H!&-& , $* , a  a "*  & && &!  a O &@ $& *& $* acbaacddba - >& &- $&  *& $* acba aa  acddba9 & ! -  $* O*& * *& ; $* aa9 >&%%  &%  a "*  & && &!  fundamental path  , &-* i && >+ Ai *& ! >& , ,! & $* , "&%* i +   & & - --!& A< ˆ A1 ˆ A2 ˆ A6 ˆ A. ˆ && & ;& * & * *&&   && -&$&-& >&"&& *& & , ,! & $*  *& % & %  *& & ,  $*

6&%   -;&& >##&&$% &;-& , &! &% $* , a<  a1 >+ "&%* , , "& &J& *& >& ,  $* P  >& *& $!- , *& >& ,  &%& *& -&+ *& &@$&  *& >& , P " &H! *& "&%* , P  &@ $& *& >& , *& $* P ˆ a< cbcdba1  x :    *& "&%* , P  : &-% * ! %  *& - $! , *& ! >& Ai O"*&& " Ai &H! *& ! >& , $* , a<  a1 , "&%* i9 "& &J& *& path weight enumerator ( ,  1  *& %&&% ,!- AOx9 ˆ A< ‡ A1 x ‡ A2 x 2 ‡




O1& *&  % "&%* , *& &-& !$! -&$%  *& &%&  *& &@$& , z &->& *&  % "&%* , *& -&$% $! *& &@$& , y  "+ 1 -&$ %  *& ,- * *& &%* , &-* &%&  1 -& % "& &J& *& >& ,  $*  >& *& $!- , *& >& ,  &%&  &@ $& *& $* P ˆ a< cbcdba1 * >& x : y ) z 6  * & * *&  % "&%* , *& -&$% &-& !$! O11>+  &-* ! >& *& %*+

Figure 10.15

& &>&+ >&& ;& , % 1&,& -!% &-*H!& , >!% *& & $>>+ "& * &->& "* "& & >+  NN&II !$$& "& & !%  $-! On k9 -;! -&   $-! -&& & +& -*& O>! && > 1+ & $+9 & , -&- $* &% & &$& >+  & , $* "*-* & &&+ >&" *& -&- $* &@-&$ , *& & $ *&& -&- $* &% & "& - error events O % 1& & &;& -&$   ,! & $*  *& &-&I & %  *& ;! &-& & $>>& "& * " -! &  $>>& &&  & &;& & >;! *% "& -! +   - $!& *& $>>+ * *&& &  & &;& ! -& "& *;& ! & L  >& ;&+ %& * $>>+ " >& & < !& *& -*&  && O&& > 1>+ * *& J & &;& >&%  &$* & * & *  - -*& x- !& POy l jxl 9 > POy l j0 l 9 -& *&"& *& $* >&

Figure 10.16  & & $*

10.3 Path enumerators and error bounds

611

Figure 10.17 &$* < & &;&

, x- >+ &$-% *& & &;& E "* Ol " >& $&,&&  x-  + &% &- "* * %;&  *& $, , *&& :7 O-,
H O:1699 *& $>>+ * E " >& $&,&&  0 l >+   &-&  >!& >+ ã wH O E9 "*&& ã  %;& >+
H O:(9  wH OE9  *&  % "&%* , E *! *& J & $>>+ PE1  >!& >+
PE1 < ã wH O E9 O1&;& *"&;& * *& & &;& & &-  *& ,! & $* *!%* *& & % L  AOx9 O
H O1>+ * *& &-&I $* &$ , *& -&- $*  &$* >+ * *& &-&  ,, *& -&- $*  &$* j %;& *    *& -&- $*  &$* j 1 * -%   &"* !!   "& " &J& *& error event probability PE  *& $>>+ * *& &-&  ,, *& -&- $*   %;& &$* j *  ?! *& $>>+ * *&&   & &;& *%% >&" *& -&- $*  &$* j O&& % 1! -&$%  O1 " NN--&+II >& %* *! *& bit error probability Pe "  %&& >& & * *& & &;& $>>+ % &%    * &% 
H O1!  Pe O&& > 1+ & ã7 & 
@ % *& &J , AOx9  AOx y z9 "& && *   >&-!& 7  *&  & "&%* , + ,! & $*   %&& -;! -& *& free distance d f  &J&  >& *&  & "&%* , + ,! & $*    *  *" O&& > 1& *&   $ %& &!& ,  -;!  -&I >+  - > &  !-* &,, * >&& &@$&&  *& $>& , J% -;! -& "* %& ,&& -& O&& &- 126   > 1 1 &-% "! >& *$&&+ - $&@ *! "& & ;&  J  &-% %* * " "#  -;! -& "* ;&+ %& ;!& , M *&&   - , !-* %*  -& sequential decoding algorithms *&+ &  H!&  %   %*  ,  J@& -& >! * &,&-  %&+ - $&& , >+ *& ,- * *&+ - >& !&  &-&  & -& "* ;&+ %& M

*& #&+  !&% &H!& &-% %*   *& tree diagram   *,&%& &-& ,  On 19 -;! -& & ,  > &&% *& &-& -!& n &-& >  &;& O-, % 1&  ;&" *& &-% $-& --&$!+   "# O- >P9 *!%*  binary tree  &$-&  % 1&%  *&   ;&&@  ,& d $!  " >&   & ;&&@  *& &&  &$* d , *& &@ $!  NN-*  *& !$" &-L ,   NN1II  *& "" &-  &@ $& , *& $!   *& &-& " !$!   ;& *& >-*

*& &&  % 1! , -!& , + --&& &4 ,  -& *& && " >& J& , *& L* !- , *& -&  >&% !& *& && " & &  &$* L ‡ ML  >&+

61.

Convolutional codes

Figure 10.19 >+ && ,  On 19 -;! -&

&$* L 1 *&& " >&  >,!- , *& $* -& *&  M $!  *& &-& " "+ >& NN&  + ;& , * $*  *& &-&I ?> " >&  %!& "*-* $* *& &-& -!+ # -& *&& " >&   , 2 L $>& $* , L  %&  "   %&& >& $>&  - $& *& &-&;& &%&  &-* , *&& $* *& $$-* #&  * $>& >+ sequential decoding algorithms   &@$&  ;&+   !>& , *& $>& $* ,   %;& %&  -& $+ &@$& $* # $ %   &@$& ,!*&L ,  &    >&  *& $*  & /& * " &->& *& " >& #" &H!& &-% %*  *& stack algorithm  Fano's algorithm  !& *& & "& >&% "*  NN*!%* &@$& &II

10.4 Sequential decoding

617

Figure 10.20 *& >+ && -&%  *& L ˆ 6 !- ,  1

Example 10.9 %& * "& & !% *& &&  % 1 &H!&-& y ˆ O y1  y2  . . .  yn 9  $& , *& ,- * *& -&" & , ,,&& &%*

61)

Convolutional codes

& *&& ! $ , y  &-&;& *& &-% & $>>+ " >&  4& , *& &-& &&- *& -&" x i , "*-* *& - $>>+ Pfx i &jy &-&;&g   %&  $>& O-, > 2169 ! -& y  J@&  Pfx i jyg ˆ Pfx i  yg=Pfyg *  &H!;&  @ 4% Pfx i  yg ˆ Pfx i gPfyjx i g "*-*  %;& >+ *& ,"% , !D Pfx i & y &-&;&g ˆ pi

ni 

pO yj jxij 9

jˆ1

n 

pO yj 9

O1+ *& $;& - njˆ1 pO yj 9 #& %*   @ 4& *& &!% &@$& "*-*  -&  metricD   ni
pO yj jxij 9 1 1 ìOx i 9 ˆ : O1 ! *;& >&& 11L *&  $ $>>+ , * &;&  12 ! % * NN "&& + 1. !%  * "+ "& ;&  *& ,"% >&D '&&@ A B C D

&" >>+ 11   &H!+ #&+  >& <  1 O&& > 1! %;&% *&    pO&"&& *& ,! $* A B C D "& *! - $!& *& -&$% ,! &- !%
H O1+ 2 d   *& & n i 1 % p i 1 ,
H O1&  &@& , &@-+ & , *&  *&  #&+ $*  *& & , "*-* *&  &- ì Ox i 9  %& F& 
H O1-* %& O , x< &@&  &$* L  >&+ + & &@& " >& $>&L && % 1&L *& $*  &-* -# & &$&&& >+ *& &  ;&-&  *& -&$% &- & %;&  $&*&&

 &@ $& -# )  >& ,

-# 7 >+ &&% C  &$-% 

Figure 10.21  L ˆ 2 M ˆ 2 && , &H!& &-%

10.4 Sequential decoding

613

"* F  G F %&  *& $ , *& -# >&-!&  &- O 19  %& * * , G I  H *& &-&I J &-  *& ;&&@ KL *& -&$% ,  > & 1&-!& , *& ;&+ " &- ,  $&&-& G    $& -*-&4 , *& $* -*& >+ *& -# %* && > 1+ - $% *& &- ì , * ;&&@ O- $!& >+
H O1-* &%  *& ;&&@

*& & , I %* & %;&  *& K"-* , % 1+ !$& , Ä  $&&&-& -    & T " >& < *& &, *& -!& ;&&@ 6  & /  O-&% &$*9  *& &&L  &  5 /   5    - $!& *& &- ì ,  ;&&@ & >-*   *& &&  '
   ;& *& &-&   ;&&@ & >-*  5   '
 *;& -&$% &% , *& %* 5  , *&   ;&&@ "& ! & *& &- ì  >& 1 *&& *   $&;&  >-#" ;& ,    . '  , '

A   %;& ' *& 
 '  * ;&&@ & >-* &&$&  *& && "* *& %& &-L *& / '  *& & "* *&  & &- 7 *& *&* T  tight   %;& ;&&@ , -&% T >+ Ä "!

62
-* , *& &@ ì > P H!&+  * &!  &!-% T  2L *! "& *;& ;&  &$ :  *& >& 12!16D  &$ 12 *& &-&   ;&&@ B "* T ˆ 2 *;% ?! ;& >-#" , E *& "&  *&  / 'P H!&+  
 -& E  *& / ' &;&  B >-#" #  AO ì ˆ & *&$,!  #"  &*% , *& *&+ , !& ;& $-$ & %L -! &% % F3G *$& A' &- 29 & F >&  J&  & FFxG && *& % , $+   *& && & x ;& F & FFxG n && *& & ,  n!$& f ˆ O f 1 Ox9 . . .  f n Ox99 , $+  , FFxG convolutional code C  &J&  >&  !>& , FFxG n  "*-*  O9 -& !& - $&"&   !>-  O>9 -& !& !$- >+ && & , FFxGL *  , f ˆ O f 1  . . .  f n 9 2 C *&   af ˆ Oaf 1  . . .  af n 9 ,  a 2 FFxG basis , C   & fg1   . . .  g k g  FFxG n !-* * &;&+ f 2 C - >& &@$&& !H!&+  k f ˆ iˆ1 ai g i ,  & a1  . . .  ak 2 FFxG *& ! >& k  -& *&  & O #9 , *& -& O9 *" * &;&+ -;! -& C * !-*  >  * &;&+ > , C * *&  & ! >& , && &L  J  > , FFxG n &, O>9 *" * *& &-% &->& >+
H O1 fg1  . . .  g k g , C  > ff 1  . . .  f n g , FFxG n   $+  a1  . . .  ak "* ai ja i‡1  i ˆ 1 2 . . .  k 1 !-* * g i ˆ ai f i  i ˆ 1 2 . . .  k *& $+  ai & !H!&  & -& *& invariant factors , *& -&  !-* >& , I 1  2 O$ 2329  , *& O2 19 -& "* $+  %&& @ G ˆ Ox ‡ 1 x 2 ‡ 19 n O9 &J& *& dot product f
g ˆ iˆ1 f i g i 2 FFxG >&"&& " && & , n FFxG   *" * f
g ˆ < ,  g 2 FFxG n ,, f ˆ 0 O&9 , ff 1  . . .  f n g   > , FFxG n  dual basis   > ff 1  . . .  f n g !-* * f i
f j ˆ < , i 6ˆ j ˆ 1 , i ˆ j *" *  ! > &@   !H!& O,9 , C   -;! -&  dual code C ?  &J&  C ˆ fh 2 FFxG n D h
g ˆ < ,  g 2 Cg  >&%   $ O-9 O9 O&9 *" * ff k‡1  . . .  f n g ,   > , C ?  *&-& * C *  & n k   > , *& ! -& , &-* , *& *&& -& -&  $ O-9 O%9 & C >&  -;! -& "* k 3 n $+  %&& @ G C    >& ;&>& , *&& &@  k 3 n $+  @ H !-* * GH T ˆ I k  *& k 3 k &+ @ *" * C  ;&>& ,, *& ; ,- a1  . . .  ak O&& $ O-99 &  &H!  1  &@$- ;&& , I 1  2 , * -*$& O*9 & %&&+  @ H  -&  øOx9 inverse , G , GH T ˆ øI k  *" * G *  øOx9 ;&& ,, ak jø   øOx9 ;&& , G ˆ Fx ‡ 1 x 2 ‡ 1G "* øOx9 ˆ x ‡ 1 /& &&  $ 236 *  On k9 >-# -& -! >& ;&"&   On k9 -;! -& "* M ˆ -# -&P   $>& *  %;& $ , &  *& & % ,  -;! -& " >& ?& >+ & * & &%& /*& " * --!P &  J@& On 19 -;! -& !-&  &%* L *" * *&  ! , "# ;;&  &-% >+ *& >!&,-& NN- $& *& &-&;& &H!&-&  &-* , *& 2 L $>& -&"II &*  >! 2 L K  &  !-*  * ;;&  !% *& '&> %*  O&& K ˆ *& -&I - &%*L &&
H O1& '&>I &-% %*  *& &@ + , -& "* k ˆ 1 &&4&  %&& k O/-* ! , !$& &%& >&"&& &Q&& > 1-# -& -& *&+ & M ˆ < -;! -&9 & C >& *& O6 29M ˆ 1 -;! -& &->& >+ *& %&& @  Gˆ

1‡x x

1 1‡x

 1‡x :
! + *;& & && &!  a9 ;&  %&& , ! , Bi 10.9 & *& ,"% >&& &-& %$*D

 $!& *&   >&"&& ;&-& a  b  &  , *& &%& >& A B C D O! $$&@  , &J9 10.10  $!& *& $* &! & AOx9  *& - $&& $* &! & AOx y z9 ,  2 10.11 & C >& *& O6 19M ˆ 2 -;! -& &->& >+ *& %&& @ Ox 2 ‡ 1 x 2 ‡ x ‡ 1 x 2 ‡ x ‡ 19  $!& AOx9  AOx y z9 , C " !$$& *& -&   >& !&  *& ,"% >+ &!& -*&D

;& !$$& >!  *& J & $>>+ PE1 *& & &;& $>>+ PE  *& > & $>>+ Pe 10.12 & !%  >+ On k9 -;! -&   J@& "$!   "*-*   && *  "*& -$-+  & * % 2 & PO&L9 && *& $>>+ *  & & & &;& " --!  *& &-% ,  -&" , *& L* !- , *& -& *" *  L!1 POL9 ˆ 1 O&& $ 6!  Pe %;& 
H O1! , &- 1 :2>+ J& PE ˆ áã d f ‡ OOã d f ‡1 9 ,   ã  &,+ *& - a &;& %! &! , PE1  Pe O-,
@ $& 1! O> :2.9  *& L* !- ,  On k9 -;! -&  *" *  ,&& -& O&& $ 6119 d f J& 2 df
& & "*  -& %*  *& & % ,  -;! -& *& %* !$!  &%;& &%&L +! & !$$&  J%!& ! "* *& &%& &$&&   ! & * &-* &%& , *& & %  >&& "* *& ! >& , 1I !$! >+ *& &-& "*& *& &  -&$%  * &%&  &  &@ $& ,  1 *& & % "! # #& * O-, % 1+ V L O29 *& >& ,  &%& -&-% & v i  & v j  lOv i  v j 9L O69 *& & &-*>&  & &$ ,  & v & -& *& successors , vQ, &@ $& *& !--& , & d  *& >;& & % & fd bgL O.9 *& 4& &  && >+ v& D O9 /* & %* X P O>9 $$+ %* X  *& -;! -& "* $+  %&& -& Fx 6 ‡ x 2 ‡ 1 x 6 ‡ x 2 ‡ x ‡ 1G Fx 6 ‡ x ‡ 1 x 6 ‡ x 2 ‡ 1 x 6 ‡ x 2 ‡ x ‡ 1G   1‡x 1 1‡x : x 1‡x
&  & "*  &! $>& * - & , *& "% # , -;! -&  -*& catastrophic error propagation  > 1 1* *& $!  !$! &H!&-& & J& 10.18 & *& O2 19M ˆ 2 -;! -& "* $+  %&& @ G ˆ Fx ‡ 1 x 2 ‡ 1G O9 " *& & %  O>9 & $  --!& *& $* &! & AOx9 /*+ & *& & $ ,P O-9 ! & *& -&  >&% !&    "* " > & $>>+ < , p , 12  !$$& * *& $! &  &;& * " -&-!;& > & & -&  --! &;&!+G 10.19  > 1& , -*& & -! -!&  J& ! >& , &-& & -& , "*-* * - *$$&  -&  catastrophic -&  * $>& "& * %;& &;& &H!;& - , &% "*&*&    %;& -&  -$*-

*! & C >&  On k9 -;! -& "* $+  %&& @ G  *" * *& ,"% - & &H!;& O >&;& * *& -& , > 1& , 1I !-* * *& -&$% !$! & * +  J& ! >& , 1I O>9 *&& &@ k  ,!- I j Ox9 ˆ pj O@9=qi Ox9 j ˆ 1 2 . . .  k O& , *& qj Ox9I >&% ;>& >+ Ox9 !-* * *& ;&-

Problems

10.20

10.21 10.22 10.23

62:

C ˆ OC1 Ox9 . . .  Cn Ox99 ˆ OI 1 Ox9 . . .  I k Ox99
G * $+  - $ & O-9 *&&   -& $*  *& & % O*& * *& & $  *& 4& &9  "*-* &;&+ &%& * >& & , &-& & F HintD & *& -*& & $& >& *& ;&- C , $ O>9G O9 *& k* ; ,- ak , *& -&  not  $"& , x O-, > 1&&  , G    $"& , x  $-! , k ˆ 1   > 1& -*& >+ *& -# %* O *&  -*-&4 , *& $* * "  >& -*&9 ! & "& *;&  && !-*  *& & &$-&  % 1& -*& >+ *& -# %*  O&& > 1+ I %* 9 10.24 % &" &-  *& && , % 19 *& $* -*& >+ *& -# %*   O-9 *& $* -*& >+ I %* "  >& ,,&&

62(

Convolutional codes

 *& &@ : $>&  "& " &;&$ &;& , *&   $ $$&& , I %* O% 1 1&*;  &->& >+  &H!&-& , $ Ov1  T1 9 Ov2  T2 9 . . .  "*&& v i   ;&&@  Ti  *& ;!& , *& *&*  v i  >&% ;& *& &- ,  ;&&@ v  && >+ ìOv9L "& ! & * *& &- *;& >&& !& ,,  * *&+ &  !$& , *& H!+ Ä , " ;&-& v  v9 &  *&  & $* *!%* *& &&  *& &$* , v9  %&& * * , v "& + v9   successor , v v   predecessor , v9  v9   immediate !--& ,  &$*  ?! & & * *& &$* , v 10.25 !$$& *&  %*   &"*&&   $& $!-& *& &H!&-& Ov T 9 Ov1  T1 9 . . .  Ov n  Tn 9 "*&& v1   !--& , v v n ˆ v >! v i 6ˆ v , i ˆ 1 2 . . .  n 1 *" * T n ˆ T  10.26 & *&  & ! $   > 1& &H!  *& &- , *& ;&&@ 10.28 *" * ,&  !!--&,! ," # ,  ;&&@ v "* *&* T O& "*& *&  >-*  #& , *& !$$& ì > T P H!&+9 &;&+ $* ," , v ! -  & ;&&@ "*& &-  , T  10.29 , *&  %*   Ov T 9 *" * &;&+ $* ," , &;&+ $&&-& , v ! -  ;&&@ "* &- < T   -+ *" * &;&+  && !--& , v "* &- . T ! &+ *;& >&& ;& 10.30 &,&  *& #&-* , > 1 1 1+ &!& -*& "* &!& $>>+ ED

&-& *& ,"% &-&;& &H!&-&D PPPPPP11   ! F2)G 7 O$ 23(9 % , % 1!   $ - $& ,  $-- &-&   !$&@% "-* * "! & && > , C<  C1 " *& -*& ) O$ 6!  -+ &-&+ , *&  $& & ,  '&> &-& *& & % & - &+ &;&   + - & , *& %* I >&*; : O$ 6>+ "& * - $!& &  ,- &$&& , *& ! &  & $* O&& *&  " &&-&  *& $, , *&& :7 $ 1779 1< O$ 6169 *& &  * &-  $&&&  *& - $&& &;&& , *& *- & , -;&+ "*-*   ,"D  %* O13)69 -# %* O13)39 +- ?!J- , *&  &- O13:29 *& $$-* #& *&&  !&  &+ F21G 11 O$ 61:9 -!+ *& NN-#II  >&& ;&"&   NN$+ H!&!&II   !-!& &J&  *$& . , * $-,    F1G

11 '>&&%* !-& -%1

11.1 Introduction &  -&& & +& !-& "* !-& - p ˆ O p<  p1  . . .  p r 1 9 *   &H!&-& U 1  U 2  . . . , &$&& &-+  >!&  ;>& "* -  >! PfU ˆ ig ˆ pi  i ˆ &  &$&& * !-& ,*,!+ !%  ;&%& , r 1 2 H 2 O p9 ˆ iˆ< pi %2 pi > $& !-& + >  * -*$& "& !+  -;& -!-;& $-&!& , % * -& variable-length source coding

 %& *& %&& & O *& &&  "& * *& ,"%  &@ $&   &"* &&-!9 -& *& $-! !-& p ˆ 12 1. 1( 1(  "*& &$+  H 2 O12 1. 1( 1(9 ˆ 1::7 > *& !-& $*>&  AU ˆ f& &-&  + &-% &;&+*%  & $$&+ &?+& >+ *& -& , >& 111  *   uniquely decodable *  *& !-& &H!&-& - >& &-!-& $&,&-+ , *& &-& &   &@ $& *& > & 1& &$&&& ,*,!+ ;& >+  ?! >+ $*>&  &- 112 "& !+ *& $!&+ - > $>& , !H!& &->+  &- 116 "& -* -&  !-&  &- 11. "& $&& !,, I , ! %* , -!-% $  ; >&&%* -& ,  %;& !-&

11.2 Uniquely decodable variable-length codes & S >&  J& & -% s && &L !& *&"& $&-J& S ˆ f+ Æ  *& % -%  + > , ó ˆ s1 s2 . . . sk  ô ˆ t1 t2 . . . tl & % *& concatenation ó  ô  *& % s1 s2 . . . sk t1 t2 . . . tl  , ó ˆ ó 1  ó 2  ó 6  ó 1  -&  pre®x ó 2  substring  ó 6  suf®x , ó  *! *& & $+ % Æ   $&J@ !,J@  !>% , &;&+ % *& length ,  % ó  && >+ jó j  ," * jó  ôj ˆ jó j ‡ jôj , + $ , % ó  ô

O;>&&%*9 code 6 ;& S   J& & , % ;& S *& % - $%  -& C & -& *& codewords , C & C1  C2 >& " -& *& product && C1  C2 - ,  % , *& , ó 1  ó 2  ó 1 2 C1  ó 2 2 C2  *& $!- ,  -& "* &, k  & " >& && >+ C k *  C k ˆ C  C  . . .  C Ok ,-9

*& -& C  -& uniquely decodable O9 , &-* %  &-* C k &  + & "+   --& , -&" * & * , + ô1  ô2 


 ô k ˆ ó 1  ó 2 


ó k  &-* , *& ôI  ó I   -&

662

Variable-length source coding

" *& ô1 ˆ ó 1  ô2 ˆ ó 2  . . .  ô k ˆ ó k  *! &;&+ %  C k - >& !H!&+ &-&   --& , -&" O  &H!;& &J ,  && > 1119 Example 11.1 & s ˆ 2  -& *& -& C1 ˆ f&-!& , *&& " &! * $&J@  -& & &+ -&&L $&J@ -& & H!& &+   $& & O&& > 1169  ,   -& &@ "*  -& & , &%*  $&J@ -& "  &@ Theorem 11.1 OMcMillan9. If C ˆ fó <  ó 1  . . .  ó r 1 g is a UD code over  r 1 ni S ˆ f&  $;& &%& *& -& C   !k !k k
n @ r 1



ni jó j s ˆ s ˆ s jó j ˆ K l k s 1  iˆ
 O1119 h Example 11.2 #& s ˆ 6 *&   -& "* &%* O1 2 2 2  2 2 6 6 6 69 -! &@ -& 6 ni ˆ 2(=2: * ,- "!  >& &+  $;& &-+ h Theorem 11.2 OKraft9. If code with lengths ni 

r

1 ni iˆ< s

< 1, there exists a pre®x Oa fortiori UD9

Proof & ! && *& ni I  * n< < n1 <


< n r 1  &J& &%&  j 1 nj ni w  j ˆ + w< ˆ 1 -& j r 1 ni nj s < 1  ," * w < s 1 ,  j " &J& *& % ó j j iˆ<  >& *& s+ &$&& , *& &%& wj "* &!%*  & %&   >& 112 -& * , "& +  $$+ * -!-  *&  
@ $& 112 "& %& w3 ˆ 2: ˆ 1 11)  11:9 h 11.3 Matching codes to sources ;% " &&  &*% >! *& $!&+ - > $$&& ,  -& "& & $&$&  -* -&  !-& , &,J-& - ! - &- O$ 6239 *  %;&   "* !-& $*>& A ˆ f+ *& $>>+ ;&- p ˆ O p<  p1  . . .  p r 1 9  s-ary code , *& !-& p  &J&  >&  $$% , A   -& fó <  ó 1  . . .  ó r 1 g ;& S ˆ f i  *& -&" ó i    ó i  , , &@ $& "& !& * &%+    k !-& + > + i1  i2  . . .  ik *& &-&;& " > *& % ó ˆ ó i1  ó i2 


 ó i k  , "& " *& &-&;&  >& >&  &-;& i1  . . .  ik , ó  "& ! ! & * *& -&  

*& average length , *& -& fó <  ó 1  . . .  ó r 1 g "* &$&-  *& r 1 !-& p ˆ O p<  . . .  p r 1 9  &J&  >& n ˆ iˆ< pi jó i j  $-- %J--&  * *& ! >& , s+ + > O& + > , S9 &H!&  &-& *& J k !-& + >  #&+  >& >! k
n , %& k *! n   &!& , *& ;&%& %  *& -*&  $& >+ *& !& , *& -& *&& 116 O"*-* -!+ ," , &!   &9 %;&  &&% "& >!  n /& ! & * ni && *& &%* , *& i* -&"  C *  ni ˆ jó i j Theorem 11.3 If the code C is UD, its average length must exceed the s-ary entropy of the source, that is, n > H s Op9 ˆ

r 1
iˆ
! *& -! ;!& , ns Op9 *& &@- --! ! >& &,  &- 11. >! *& ,"% & &  H!& !&,! Theorem 11.4 H s Op9 < ns Op9 , H s Op9 ‡ 1: FNote. The lower bound of Theorem 11.3 is repeated here for aesthetic reasons.G Proof &J& ni ˆ d% s p i 1 e i ˆ + *&& 112 *&& &@   -& "* &%* iˆ< s r 1 n<  n1  . . .  n r 1  *& ;&%& &%* , !-*  -&  iˆ< pi ni  >! -&  r 1 ni , % s p i 1 ‡ 1 * !  , iˆ< pi O% s p i 1 ‡ 19 ˆ H s Op9 ‡ 1 h Example 11.4 & p ˆ O:1 :. :79 s ˆ 2 *& H s Op9 ˆ 1:6)1 *& ni I !%%&&  *& $, , *&& 11. & O. 2 19 &!%   ;&%&  &%* , 1: "&;& *& &%* O2 2 19  ,+ 2 nk < 1    ,- ns Op9 < 1:7 *& $  * *& !$$& >! , *&& 11.  !!+  *$ O>! && > 1139L  & &   %& %&&+ 

66)

Variable-length source coding

&& $&;!+  "+  -!- $   -& ,  %;& !-& " >& %;&  &- 11. h

 * $ "& *;&  $>&   & * *& &! , *$& 6 & ! *  *! >& $>&  &-& *& !-& p ,*,!+ !%  *& ;&%& + H s Op9 s+ + > $& !-& + >  *& *& * "& *;& ?! && *    $>&   * "*   s+ -& , p -& ns Op9  !!+ -+ %& * H s Op9

*& !  * $>&   -& -& , *& extended !-& p m  m ˆ 1 2 . . .  *& !-& p m  &J&  >& *& !-& "* !-& $*>& Am O&- * A  *& !-& $*>& , p9  "*-* *& !-& + > Ou1  u2  . . .  um 9  %& *& $>>+ PfU1 ˆ u1  . . .  U m ˆ um g ˆ pu1 pu2 . . . pu m   &,,&- "*& "& -& *& !-& p m  "& & >-#% *& !-& &H!&-& U 1  U 2  . . .  -&-!;& >-# , m + > , A  &%% &-* !-* >-#   %& + > , *& $*>& Am  "   -& , p m "* ;&%& &%* nm " &H!&  ;&%& , nm s+ + >  &$&& & + > , Am  & + > , Am &$&& m + > , *& % !-&   * -& ,*,!+ &$&& *& % !-& p !% + nm =m s+ + > $& !-& + > *& ,"% *&& *" * >+ #% m %& &!%*  !% * &-*H!& "& - #& *& ;&%& ! >& , + > &&&  &$&& p ,*,!+  -&  H s Op9  && Theorem 11.5 

m!1

1 ns Op m 9 ˆ H s Op9: m

Proof   &+  && * H s Op m 9 ˆ mH s Op9 O> 11129 &-& >+

*&& 11. mH s Op9 < ns Op m 9 , mH s Op9 ‡ 1 *&& 117 ," >+ ;% >+ m  #%   h

*&& 117  ,+% *&&-+ -&  & ! * *& !-& p - && >& &$&&& ,*,!+ !% O$&*$ %*+ & *9 H s Op9 s+ + > $& !-& + >  &;&  &*%  >& && ,  -!- ;& ;&"$ *"&;& -&  &&  *& "&# -!- , *&& 11.  *& &@ &- "& * & &+ * ! >+ $&&%  &-*H!& "*-* " &>& !  -!- *& >& $>& -& , *& !-& p m 

11.4 The construction of optimal UD codes (Huffman's algorithm) 66: 11.4 The construction of optimal UD codes (Huffman's algorithm)

--%  *&& 11. ns Op9 &  &"*&& >&"&& H s Op9  H s Op9 ‡ 1  * & &  &H!& ,  & $!$& O&% *& $, ,

*&& 1179 !   !  "& >! *& &@- ;!& , ns Op9 ,  J@& s  p  * &- "& $&&  %* !&  ; !,,  * *"  + *"  - $!& ns Op9 >!  *"  -!-   O&&  $&J@9 -& "* ;&%& &%* ns Op9 &,& %;%  ,  &-$ , !,, I %*  "& * "#  &@ $& *!%*! * &- "& &,&    s+ -& , p "*& ;&%& &%*  ns Op9   optimal -& , p Example 11.5 & s ˆ . p ˆ O:2. :21 :1: :16 :1% *& *&&  & $>>&  p *! p9 ˆ O:2. :21 :1: :16 :1-#"  -!- $  -& C9  C , p9  p >+ NN&@$%II *& -& C 0    $& "+  "& -!-  $  -& , p9 -& * *&& , *& $>>& O;4 :2. :21 :1:9 "&&  -*%&  *& &!- , p9  p0 *& !&  * -&  *  *& &@$ , C 0  C9 *&

Figure 11.1 *& !--&;& &!- , p

66(

Variable-length source coding

-&$% -&"   -*%& &*& "&;& $>>+ 6(  p 0 &@$  ,! $>>& O:16 :1& !,, I %* L ! & % #Q"*-*  !$%+ -#+Q  *" *  "# & *  $!-&  s+ $&J@ -& , p "*& ;&%& &%*      $>& *& ,"% *&&  -!-L  %!&& * *&&  "+  $  $&J@ -& , p  "*-* *&  s9 " & , *&  & &%*  %&& &@-&$ , *&  - $&

 J@ ! & & ! " ! & * *& $>>& p<  p1  . . .  p r 1 & %&  &-&% &D p< > p1 >


> p r 1  /& -& $&J@ -& , p ;& *& $*>& f+ ó <  ó 1  . . .  ó r 1  "* &%* ni ˆ jó i j i ˆ 2, there exists an optimal s-ary pre®x code for p with the following two propertiesD O9 n< < n1 <


< n r 1  O>9 The last s9 Osee Eqs. O1129 and O11699 codewords are identical except for their last component, that is, there exists a string ó of length n r 1 1 such thatD ór ór

s9

ˆ ó  + *&& 112 *&& "! &@  $&J@ -& "* *&& &%* * &" -& !  >& $  O , -!& p r 1 ˆ !  * "! -- *& ! $ * ni    /& --!& * < < Ä < s 2  "*  *&  & *% s

Ä 2 f2 6 . . .  sg:

O11:79

&@ & r9 && *& ! >& , " , &%* n r 1  *& -& , r9 ˆ 1 "& -! *& *& !H!& %& -&" >+ &&%   - $& "*! ;% *& $&J@ - %  -- &-& r9 > 2:

O11:)9

, "& &@ &
H O11.9 ! s "& %& Ä  r9 O  s9 *  r9  s Ä O  s9 ! ,
H O1179  O11)9  ," * r9 ˆ ks ‡ Os

Ä9

,  & k > s9 *  *&& &  & s9 %& -&" &@ >&;& * , ó 9 ˆ ó  a  & , *&& r9 -&" , &%* n r 1  "*&& ó   % , &%* n r 1 1  a 2 f+ &$-% -& " , &%* n r 1  , *& , ó  a "* & * & "& ;&   $  -& "*-* * ó  p1 >


> p r 1  r > s *& s-ary Huffman reduction , p  &J& >+ p9 ˆ O p<  p1  . . .  p r

s9 1 

pr

s9

‡


‡ p r 1 9:

 " p9  >& , p >+ - >% *&  & s9 $>>&  p "*&& s9  &J& >+
H O1129  O1169 *& ,"% *&& *" *"  $  -& , p + >& -!-& , & , p9 -& p9 * ,&"& + > * p  +&  &-!;& $-&!& , -!-% $  -& , p

11.4 The construction of optimal UD codes (Huffman's algorithm) 6.1 Theorem 11.7 If C9 ˆ fô<  ô1  . . .  ô r then C ˆ fô<  . . .  ô r

s9 1 

s9 1 

ôg is an optimal code for p9

ô  &% >+ $;% O1139 & On<  n1  . . .  n r s9 1  n9 && *& &%* , *& -&"  C9 + &J p< n< ‡


‡ p r s9 1 n r s9 1 ‡ pn ˆ ns Op99 "*&& p ˆ p r s9 ‡


‡ p r 1  , C  !&   -& , p  ;&%& &%*  ns Op99 ‡ pL *&-& ns Op9 < ns Op99 ‡ p:

O11:1&  $  -& , p * &?+ $$&& O9  O>9 , *&& 11) *& -& fó <  . . .  ó r s9 1  ó g !& , p9 * ;&%& &%* ns Op9 p *  ns Op99 < ns Op9

p:

O11:119

 >%
H O111& , !-& &!- &H!&  dOr s9=Os 19eL ,& * + &!- *& !-& " *;& &@-+ s + >  f 11219 F&

*& >+ !-& &$+  H 2 Op9 ˆ 1:32 >G Example 11.7 & s ˆ 2 r ˆ 2 p ˆ O:3 :19 *&& H 2 Op9 ˆ :.)3 "*& !,, I %* +& *& ,"% ;!&D n2 Op9 ˆ 1:+ *& &&L && > 11139 *& $ *&&  * *& -;&%&-& , m 1 ns Op m 9  H s Op9 O&& *&& 1179  !!+ H!& $ O>! && > 1129 

*&& &@  &-$*&>& % "* &$&-  C r 1 jó i j O-9 ˆ 1 iˆ< s 11.3 ;& *  $&J@ -& C  &-&+   &%  &,J-& %* , &-% C *  , ,-%  J& --& , -&" , C   - $& -&" 11.4 *" * *& $!- , $&J@ -&  %  $&J@ -&  *& $!- ,  >+ $ ,  -& &-&+ P O&& &J $ 66& +! &  +  -!-  -& ;& S ˆ f& "&  ó ˆ r1  i  r2  ô ˆ r1  j  r6  "* i j 2 S  i , j *" * &@-%$*- &%    &% *  , + $ Oó  ô9 "* ó 6ˆ ô &*& ó ,
ô  ô ,
ó O!9 & *& ,"% && , ! , *& -!-   *&& 112D *& ó < ˆ & &@-%$*-+ & % , &%* n k‡1 * *  $&J@&  *& & fó <  ó 1  . . .  ó k g *" * * -!- "+ +& *&  & -&  *& & %;&  *&& 112 *" * ns Op9 ˆ H s Op9 ,, *&& &@ $;& &%& On<  n1  . . .  n r 1 9 !-* * pi ˆ s ni  i ˆ   $$&   J@& ! >& n , -*& $! + > * $>&  &%&  *" *"  ;>&&%* -& - >& J&  J * & *!%*! *& !-& p ˆ O p<  p1  . . .  p r 1 9 " >& J@&  *& -*& " >& 

6..

Variable-length source coding &&   "* $!  !$! $*>& >* &H!  S ˆ f>& %;& >+  pO yjx9 ˆ

11.11

11.12 11.13 11.14 11.15 11.16

1
& J@& &%& " -& *& ,"% &%+ , $$% k !-& + > U1  U 2  . . .  Uk  n + > , S  $ &-* !-& + >  *& -&$% -&" , C  , *& --& ó , *&& -&" *& &%* , ó   % ;& S    ;>& >! "& ,+ ó  *  * &%* n  ," , *& &%* , ó &@-&& n "& &&& *&  jó j n + >L ,   & * n "& ? n jó j   ;& *  *&+ "&& erased >+ *& +&  && >+ PE *& $>>+ *  & & , *& k + >  *& !-& >-#  && & å . < >& >+ O9 *" * , + í . ns Op9 *&& &@  -*& & , *& >;&  "* n=k < í  PE , å O>9 % *& &! , $ O9  *&& 117 *" * , + r , b H s Op9c 1  å . >+ >! , U " !$$& OU 1  U 2  . . .9   J& #; -* O&& > 12 2  & ;&P *" * , r > s ,& dOr s9=Os 19e s+ &!- , p ˆ O p<  p1  . . .  p r 1 9 *& &!% !-& * &@-+ s + > & $$! "+ , ;% *& $>& *  *& J !,,  &!- ,  !-& p *& ! >& , + > - >& + >& & * s   ? s s9 &@ NN! +II + >  *& !-& $*>& &-* --!% "* $>>+ >& O &@ $& 
@ $& 117 s s9 ˆ 1  *& !-& "! >&- & p ˆ O:2. :21 :1: :16 :1>+ ;&- O p1  p2  p6  p. 9 "* p1 > p2 > p6 > p. , "*-* *&&  & " -& & $   *& -;&@ *! , *&& J@& $>>+ ;&-L  J *&& *&& ;&- 11.22 &  % & , NN"&+ H!&II  "*-* +! & &H!&  && & ,& #%  -& ! >& , H!& * - >& "&& NN+&II  NNII *& !- & , &  ,  $ , -& O *& !- &  *&&,& & , *& &%& 2 6 . . .  129 /*  *&  ! ! >& , H!& +! &&  #  *& ;&%&P F HintD , +! #& NN   2PII NN   6PII &- +! "! ;&%&  & !& @ H!&   $>&   >&& *"&;&G 11.23 *& >?&- , * $>&  , +!  & *"  &%  $  binary $&J@ -& , *& + &- r+ !-& p r ˆ O1=r 1=r . . .  1=r9 O9 !- !-*  -& , p2  p6  p.  p7 O>9  %&& "* & *& $>& &%* , *& "   $  >+ $&J@ -& , p r P O-9  $!& n2 Op r 9  %&&

Notes 1 O $ 66?&- & , * -*$&  >& $&&&  &   &@  ,  *&+ -&    $-!+ &&$  &+ &  &$&  + , *& $&-&% & "&;& "& *;& -*&  $-&  *&& >&-!&  &$&&  -!-;& $$-*  *& !-& -% $>&  ?!  *& &  *$& :B1< &$&&  -!-;& $$-*  *& -*& -% $>&  2 O $ 66& & >* &H!  f>+ , å ,   å9  *& &

6.)

Variable-length source coding

-&  * -& *& & ,!- ROä9 &;!&  *& &&  Oä ˆ &&%* -& - >& , &   -& &&   >-# -& O&& > 111&"&& s+ $&J@ -&  s-ary trees "*-*  & $*&   &@  "& *! &   & >&K+  !& *& & -& *& $&J@ -& f-*  && >+  NN+ -&  &@ O&% >  F12G %& F1:G9 $;& + *& s ˆ 2 -&  &;& *& O;M9 %&&4   $>&  O&&  > 111)9

12 !;&+ , ;-& $- ,  "

12.1 Introduction

* -*$& &;& *&  & ,!- ,  "  *$& ) &;& ,  & *   ! 4&  & , *&   $ &!  -% *&+ "*-* *;&  >&& &&  *$& :B11  &- 122 126  12. "& & -*& -% O>-# -& -;! -&   - $ , *& "9 +  &- 127 "& -! !-& -%

12.2 Block codes

*& *&+ , >-# -&  &  -*& * *& *&+ , -;! -&   * &-  !-* %& * &- 126 O *  >-& &  $$+  $-- $$- *"&;&L && &- 12.9  &  %;& * &-  & & >-& , %4 "& * -,+ *& &!  >& -& --%  &&# $I F17G  , *& *&& ? $>&  , -% *&+D 1 " % & *& >& -&P 2 " - "& &% % -&P 6 " - "& &-& !-* -&P · How good are the best codes? & , *& && $>&  "*-* &  -% *&+ " * , J% perfect codes , "& ;&"  -& , &%* n ;& *& J& J& Fq   !>& fx1  x2  . . .  x M g , *& ;&- $-& Vn OFq 9 *& -&    >& $&,&- O -& $-#&9 , ,  & &%& e *&  % $*&& , ! e ! *& M -&" - $&&+ J Vn OFq 9 "*! ;&$ + 137< &;& # , $&,&- -& "&& #"D 6.:

6.(

Survey of advanced topics for Part two n

2e ‡ 1 Oq m 19=Oq 26 11

q

e

& #

2 O>+9 &$& -&L && > :1( 19 O>+ 1  % -&L && &- :. $ & > :13 $"&9 2 6 + + -&L && &- 3( 6 2

&+ + -&L && > 36& &-&-% $"&  &&-*& !+ +&&  &@& *& >;& >& "&;& ,&  % && , &@& &+ ,J-! ;&% &%  *& &+ 13:& &  -/   & F13G *$& )

*&&  &@ $&,&- -& "* *&  & $ &&  *&  % -& "*-* &  &H!;&  *& L *& *& *&& # *"&;& & !H!&

*& H!& , *& &@&-& , $&,&- -& ;& $*>& "* q + > "*&& q is not a prime power   * "%  - $&&+ &&  &&  !#&+ *"&;& * + &@ $$& >+ *& &@&-& , *& $&,&- -& &&-*& "&& *& &-*  -& * &    %D nearly perfect  uniformly packed -& ;& J& J& !-* -& & "  -J& , e > .L , e < 6 + $ &! & #"L && ; >% F.)G

*& &@&-& , *& O26 129 $&,&- + -& & ! * *& @ ! ! >& , ;&- , V26 OF2 9 "*-* - >& -*&  * &-* $ , ;&-  &$& >+   % -& > :  &@-+ .+ n  :  &$-& >+ d   "& &J&

AOn d9 ˆ *& %& &%& M !-* * *&& &@ M -&" fx1  . . .  x M g , Vn OF2 9 !-* * d H Ox i  x j 9 > d , i 6ˆ j:

&;&+ O12:19 AOn d9 ˆ *& %& ! >& , -&" $>&   -& , &%* n   ! -& d:

12.2 Block codes

6.3

*& !+ , *& ! >& AOn d9  &%& >+ +  *& -& $>& , -% *&+ *!%* AOn d9  &+ #" &@-+ !& n  d & &;&+    2d > n  %& & , J& &&-* * %&  *& $>& , J% % !$$&  "& >! , AOn d9  *& &@ ,&" $%& "& " ! 4&  & , * &&-* >! , *& "*& + *& &&  &,&&  -/   & F13G *$& 1:  &;&+   ;!& , n  d &&+ *& + #" "+ , >% "& >! , *& , AOn d9 > M   &@$-+ &@*> M ;&- , Vn OF2 9 ,+% *& - -&  O1219 /& -! !-* -& -!- >&" O&& $ 6.3 ,,9  *& *& *  &  >  !$$& >! , *& , AOn d9 , M   &-&+  *" * &;&+ !>& fx1  . . .  x M g , Vn OF2 9 -  & & $ Oi j9 !-* * d  Ox i  x j 9 , d *&& &  %& + $>& "+ , % * >! -!&+ *&  $"&,! &-*H!& $$&  >& *& linear programming $$-* "*-* "& " &->& , C ˆ fx1  . . .  x M g   >+ -& , &%* n *   !>& , Vn OF2 9 , &-* i ˆ & *& ! >& , -&"  C  -& i , x *& distance distribution , C  &J&  >& *& On ‡ 19!$& OA<  A1  . . .  An 9 , &%;& & ! >& "*&& 1
Ai ˆ Ai Ox9: M x2C & && &+ $$&& , *& Ai & 

 , *& -& *

&&D

A< ‡ A1 ‡


‡ An ˆ M:

O12:29

A< ˆ 1:

O12:69

 !

-& d   ," *

A1 ˆ A2 ˆ


ˆ A d

1

ˆ !  AOn d9 ,   &;&+ $ On d9 - & , *& & $% % >!   J- ,   &@ $& *& >!  AO16 )9 >& >+ - >% *&  &H!& O1279 "* *&  *- &H!+ A1< ‡ .A12 < . O"*-*  &+  > - >+9  AO16 )9 < 62  &;& +& >&,& *& -;&+ , *&  $$-* AO16 )9 " *&  $& !#" ;!& , AOn d9  &&-*& * !%%& "*  !% *&-+ - $-& - > %! & >! 62 < AO16 )9 < 67 " *& >& *&+ -!  , -!& *&  "& >!Q"*-* & >&-!& *&&   #" -& *& & -& "* n ˆ 16 d ˆ ) M ˆ 62Q- >& "* *& &" !$$& >! && *& &D AO16 )9 ˆ 62 &*$ *& >& #" ;& , *& AOn d9 $>&  *& asymptotic problemD , Od n 91 nˆ1   &H!&-& , &%& "* d n =n ! ä < < ä < 1 *" & AOn d n 9 >&*;&  n ! 1P & , + -& *& & ,  -& , &%* n "* M -&"  1=n %2 M "& &J& 1 %2 AOn d n 9 n!1 n 1 %2 AOn d n 9 ROä9 ˆ ,  n!1 n

ROä9 ˆ !$ 

O12:)9 O12::9

"*&& *& NN!$II  O12)9  *& NN, II  O12:9 & >* #& ;&  &H!&-& Od n 91 nˆ1 ,+% d n =n ! ä   +% * >*  !$$&   "& ;!& , ROä9 ! >& &J&L &;&+>+ >&&;& O>! >+ * $;&9 * ROä9 ˆ ROä9 ,  ä   $-+  "* ," *"&;& "& * &,& +  ROä9  >&% !& *  !$$& >!  ROä9   !$$& >!  ROä9  "& >!   "& >!  ROä9   #"  &;&+ &+  $;& * RO! ROä9  !#" , < , ä , 12 *& >& #" "& >! 2 , ROä9  * %&  ROä9 > 1

H 2 Oä9

O12:(9

 &! $;& >+ >&  1372 O * >!   + $- , , *& >! >&  > :219  & * !--&&&  % >&I "& >! >! *&& * >&&  "*& && , &+ &-&% !$$& >!  ROä9 *& -!& &-*& >& >+ -
&-& & -* ! &+  /&-* !% *& & $% % $$-*  ROä9
!  >&-!& *&+ $&   $&  &,J-& &-% %*    , *& -& "& * &->& *"&;&  % &-% %*  +& #" /& * &->&  & , *& &&-* * * >&& &;&  *& &% , &-% %*   *& &@ !>&- O&& $ 676 ,,9

Figure 12.1 *& >& #" !$$&  "& >!  ROä9

672

Survey of advanced topics for Part two

>>+ *&   $&;& &!  -!-;& -% *&+ & "* *& - , cyclic codes  On k9 & -& C ;& Fq    >& -+-- , "*&&;& c ˆ Oc<  c1  . . .  c n 1 9   C *&   *& -+-- *, c9 ˆ Oc n 1  c<  c1  . . .  c n 2 9  + ;!& , q  n > 6 *&& & "+  & ,! -+-- -& , &%* nD · · · ·

*& 4& -&" 0 ˆ O&  4& , x n 1 , n  q & &;&+ $ & O -+-- -& , "*-* *   *& -& & !!+ !&&%9 *& 4& , x n 1 & f1 â . . .  â n 1 g "*&& â   $ ;& n*  , !+ ;& Fq L *! *& 4& , gOx9 & , *& , fâ a D a 2 Ag "*&& A  f 2t ‡ 1 /& "   & , *&   $ &! #" >! -+-- -& O & , q ˆ 2 &;& *!%* %&&4  %& q & !!+ #"9 · , n ˆ 2 m 1  B ˆ f1 6 . . .  2t 1g *& d  > 2 m 1 Ot 192 m=2 O&& ;  F1(G *&& )6)9 · , n   $ & -%!&  1 O  .9  A ˆ fH!- &!& O  n9S *& k ˆ On ‡ 19=2 

d



& p ' n p > 1 ‡ .n 6 ( 2

, n  1 O  (9 , n 

1 O  (9:

*&& & *& quadratic residue -&L && -/   & F13G *$& 1) · , *&&   j!$& b1  b2  . . .  bj 2 B !-* * b1 ‡


‡ bj  < O  n9 *& &;&+ -&" * "&%* ;>& >+ 2 j O %!& ;& , *& $&- -& j ˆ 2 , * *&& $$&  &  7 $ 2:(  $,  &@&;& %&&4 -! &&  -
&-& F6.G9 · , B ˆ f1g *& -& & -& irreducible -+-- -& O>&-!& *& -*&-# $+  hOx9  &!->&9  minimal -+-- -& O>&-!& *&+ -  ; -+-- !>-&9 *&&   &>& *&+ , *& L && -/   & F13G *$& (  -
&-& F63G · , n ˆ 2 m 1 B ˆ f< < b , 2 m 1 D b - >& "&   ! , < d $"& , 2S *& &!% -&  &&+ &H!;&  *& &&B !& -&  Om d9 , > :6!     &"* $$% >&-!& *& >& -& & &+ -+-- + * "& & !%*+ * ,   + -*& $ On d9   H!& !#&+ * *&& " &@  -+-- -& "* AOn d9 -&" O&& &J O12199  ,- *& >& -& & ,&  &;& &  *&&  >&%%  >& H!&  &$&->& -&- , % & -&  & &@ $& O%  -& *;& q ˆ 29 ,"D · *& >& -& "* n < 2d & $--+  &  & &&  Hadamard matrices && -/   & F13G *$& 2

67.

Survey of advanced topics for Part two

· *& Kerdock & -& *;& $ && n ˆ 22 m  M ˆ 2.m -& " d  ˆ 22m 1 2O2 m 29=2  m > 2 *&  $& &@ $& Om ˆ 29   -& , &%* 1) "* 27) -&"  d  ˆ ) *& >& & -& "* n ˆ 1) d ˆ ) * + 12( -&" && -/   & F13G *$& 17 2m · *& Preparata -& *;& $ && n ˆ 22m  M ˆ 22 .m  d  ˆ ) m > 2 *&+ $$&  >& !  *& 5&-# -&L && -/    & F13G *$& 17 + "& & * -/   & F13G $$&@  %;&  >& , *& >& #" -& , d < 23  n < 712 + , *& -& & *&& & -&;&  *- -!- * "& - &->& *&& · How can we decode such codes? *& $>& , J% *& %& $>& -& "*  %;& n  d   $  -*&%% !  -!+ $! !-*  -&  && + -&  !&   + -*&   &-&+  *;&  $-- &-% %*  , *& ! >& , -&"    &!%*  *   ,&>&  - $& &-* , *&  *& &-&;& " *&&   $>&  , *& -&   On k9 & -&  q n k    &!%*  * *& + & >& $$-* , &- :2  :7  $>& %  $>&  "&;&   $-- !   !#&+ *  -&   &!%*  " & , *&& $ ;& &-% %*  " >& $"&,! &!%*   *& ?>  * & -&>& &,, * >&& &;&  J% % &-% %*  , %&  $"&,! -&  * !>&- "& * &->&  & , *&& %*      $  *& B &-% %* * "& &->&  &  *$& 3 -& *  !-*   $ &! "& $!& *&&  $+ >!&   -;&&  1373  13)< - & *& % $$& >+ -H!&%*& OF17G $$ :2B :.9  &  +*!*! OF17G $$ :7B:(9    $  & * *&& &+ $$& --&& *& &;& + "* *& -!- , *& -&  *& &-%  13)< && OF17G $$ 1+  $& &-% %* "*-* ;;& *& ! ,  ! &! & &H! ;& -& J& J& & +& & &&  =&& OF17G $$ (:B(39 %&&4&  *& $&;! "#  >+ -&  ,! * *& %&&4 -!& *& -& -;&& >+ &&    OF17G $$ :&,& *& &  + *!*! $$& * >&& $!>*&M & >&#*!%*  *& &-% ,  -& - &  13)( "*& &&# $ OF17G $$ 1.7B1.(9 O&&  *$& :  F1.G9 $!>*& * >&!,! &;& &-% %* 

+&  &%&& "* "*&   4& >!&+ *& - $&@+ , 

12.2 Block codes

677

 &-&   "& ;&  !& &&# $I $-&!& ! -&  $ --&$! ;-& "&& +&  - &  13:< $$ OF17G $$ 1&!,! ! %&&4 ,  -& & *"& * *&& & % $$ -& * & + $-+ %  *& && * *&+ & >;& *& >& "& >! , % 121 O * - "* *& ,- *  -&  $& , *& !&,!& & + $-+ NN>II  *& && *  + &H!&-& ,  -& "* & > R . < *&  d  =n ! $$-* &!,! ,- "*-* "& -!&  &%*  *$& 3 * & - !&
!-I %*  &-&   &&B  -& -!+ "*& $$&   -&
!-I % *  ,& >+    - ,-  &&# $I   !-* &&  !& *"&;&  *& -&>& %*  &&# $I % *   ,- "* *%*   " $>&  ;&" &&# $I %*    $;& ;& ,
!-IM +  13:7  --& * >+ #&" " >&% , & " - $&& "*& && F66G  *& *"& * $$I -& - >& >& >+ ,+% *& -& &&B  -&M

*!%* *& %&>- &-% , B -&  *&   $ &!  *& & &;& *& H!& !&,! &-% &* &  ;>& /& * " &->&  & , *& 

*& && ; &-% %* -;&& " threshold decoding "*-*   # , %&&4 , *& NN ?+ ;&II &-% &&  *& !- -;&& >+ &&  137. , &-% &&B !& -& O-, > :6+ -&% *& O: 69 >+ & -& "* $+-*&-# @

1 1 H ˆ < 1

1 < 1 1

< 1 1 1

1 < <
&% "* *$& 16 , -/   & F13G  "* &+ F2+ !$! H!4 + * "& & *& ,"% + >+ ;&- x ˆ Ox1  . . .  xn 9 2 Vn OF2 9 - >&  & 2 ;&  ! -*& "* ;&%&  $"& - EOX 9 + $$% xI 4& -&  ‡ â  xI & -&  â , + x  ,   -&  & "*&  !& *& &-% %* , *$& 3  *& &-&;& ;&- y & " *;&  >&% >+ , % *& -& , yQ"*-* & & ! >&Q + $$%

12.3 Convolutional codes

67:

$;& -&  <  &%;& &  1 *  "* "& & >+ >+ !$! H!4 ,!&+ * J &$ &-&& *& -* &I -$-+ O>+ >! 2L && > .179  , * & -& "*  *&&+ >+ &-% %* & $--+ !&&  + ! -*& "&;&  "& *;& &&  &-% , J& &&B !& -&   *&&+ >+   *&& -& & -;& , !&  ! -*& *& &&$$-& - !- -*&  "& &&   ! -*&L     !$% * , 13)3  13:)  ,   I Mariner - &&$$-& $>& "&& &H!$$& "*  O62 )9 &&B !& -& O*& &&$$-& $>& *;& >&& &H!$$& "* -;!  -&L && &- 12.9

12.3 Convolutional codes

*& *&+ , -;! -&  !-* & "& &;&$& * *& *&+ , >-# -& $+ >&-!& *& !>?&-  !-* +!%& >! + >&-!&   -+ & ,J-!  &@ $&  13:< &+ OF17G $$ 216B2629 "&  &  $$& * $;& &! , -;! -& %!  *&  $& &!  &- :1 , >-# -&M *    + * -;! -% *&+ *  $!-&  $ &&-*  & &@$ *"&;& "*+ * &-  !-* *& * *& $&-&% & O  -;! -& & ,& !$&  >-# -&  $-- !L && &- 12.9

*& free distance ,  -;! -& O&& $ 6119 &;& *&  & &  & *&  ! -& ,  >-# -&D  % -& ,  J@& & *& & "* *& >%%& ,&& -&  #&+  >& >& !-* &,, * >&& $&  *& $>& , J% -;! -& "* *& %& $>& ,&& -& *!%*  , + , -;! -& - $ >&  + *&  >-# -& * +& >&& ,! *&& &@ &@&;& -%!& , $&-J- % -&  &@ $& & F6:G %;&  >& , % O2 19 O6 19  O. 19 -;! -& "* & + M < 16L 0*& F6)G  % O2 19 -& !$  M ˆ 67   -*-&- , *& !>?&- * *&& >& *! >& &-&  On k9 ˆ O2 19 O6 19 O. 19 &$&-+ O2 19 *& &@ &%&& , ,&& $&&  -;! -&  &+ *& & + M #& *& !+ , O2 19 -;! -&  &@& &+ -* !>?&- ! &, "*&& O2 19 >-# -& & , -!& ;  *!%* *& $>+ , !+% -;! -& "* J@& M  ;+% On k9 !%%& &, >+ * +& $!!&  ;&+ ,

67(

Survey of advanced topics for Part two

*& -*& -% *&& 2. " $;& !% >-# -&   !  "& "*&*&  - >& $;& !% -;! -& *& "&  ,   R!&  - >& $;& !% tree codes O-& >& >+ %% >&   && !-!& #& *& &  % 1+ %% >&   & !-!& #& *& &  % 1-& , *&  !-* -& - >& >& , *, &%& O&& % 1& , $* * *& &H!& &-& ! ;&%& >&,&  &-& *& J L ,  >   #" * PfC . xg  Lx

á



"*&& á   $;& - * &$&  *& -*&  *& -& & >!   x  ! ! * , *& -&I &   -&  -*& -$-+ *& ;!& , á & >&"&& 1  2 *  !  $& * *!%* *&  ;>& C * J& &  ;-&  J&  * * !$& $--  $-  & & >& * "*& *& &-&  +%  J  "+ *!%* *& && & + > & >&% &-&;& *&& + > "%  >& &-& ! >& &   >!,,& , J& &%* , *& &-& &H!&  + - $! * >!,,& " ;&K"  ,  " >&  , *& ;-& , C  J& * >!,,& " >& ;&K"%  *&  &  &H!& &-%  !&&

*& &  "*-* * !$& $*& & --!  -& *& computational cutoff rateL   ?! *& & R< "& *;& &+ &-!&&  > 221B22)  .6B.3L R<    & & && >+ *& + > R- $ 
;&  & >&" R& ,& && & *& -&I $&, -&  &@ $& >+ !%  O2 19 -;!  -& "* ;&+ %& & +  ,&& -&   "&> !

12.4 A comparison of block and convolutional codes

673

-*& & - &+ #& *& > & $>>+ &@& &+   >!   ;&+ ,J-!  &-&& *& erasure $>>+ O& *& $>>+ , *&&  >&%  &  &-&  %;& >9 >&" + 1< 6 *& ,! + , &H!& &-% - $! >!   >+ '&>   ! F2)G *$& )    '&> &-%  &H!& &-% *&&   *  $ +$& , &-% , -;! -& -& threshold decoding   %!  *& *&* &-% , >-# -& "*-* "& &->&  &- 122  " &;&$&  13)6 >+ &+ F2+ %& F1:G &- )(L   ;&+ -&;& $$- , *&* &-%   --&& -;! -% +& * &-&+ >&& %;& >+ /! F.(G + "& & *  13:) -*#"?#  '-# F..G ;&&  &-% %* *   %!& , -;! -& , + & &-% O&& &- :29 , >-# -&   * !-*  -  "* '&>I &-% %* 

12.4 A comparison of block and convolutional codes  * &- "& #&  >&, & $  - $& *& &;& & , >-#  -;! -&  $-- $$-   ,+ -& * *& &&$&    $&;& *&&- &!  -% *&+ O>-#  -;!9  *& %&>- &-% , B -& "*-* "& &->&  *$& 3    $  >&   *"&;& * *&& -& "&& &%& ,  ;&+ $&- - , -*& *& q+ + &- -*& O $-! *& >+ + &- -*&  *& -& ,  -&9  * *& &-% %*  &  &+ O,  9 $>&  *& -*&  *& *& * *& "   $ &-% %*  , -;! -& '&>  &H!& &-% *!%* $&*$   *& -+ $,! & &@& &+ >!  - *& &+ >& $&   ;&+ "& - , -*&  -&  &;&+   ,- , *& - !- -*& * &  $--& & "& && >+  H    !$% *  &;&+ %& ,- , $-- $$- , -% *&+ ;;& -;!   >-# -& /& " !& * $ "* " $&-J- &@ $& & J  >+ + &- -*& "* " > & $>>+ p < , p , 12  %!& 122 "& %;&  %$* , *& $&, -& , " $-- -% +&  , * -*&  O12: ).9 1-#  -;! -&   

"* *& &-% %* &->&  % 31   O2 19M ˆ ) -;!  -& "* %&& @ G ˆ Ox ) ‡ x . ‡ x 6 ‡ x ‡ 1 ) 7 . 6 x ‡ x ‡ x ‡ x ‡ 19  ;!& , p %&& * >! p ˆ :&& >! ,   & p *&  -&  >&&L , &@ $&  p ˆ :& H!& ,J-!   $& &  & 1=2 -;! -& "* $&, -& #&+ !$&  * *"  % 122 -& *& + "+  %& >&& $&, -&   !&  -& "* & & +L >! , *& '&> %* -&% M >&+ :  (  &@& &+ ,J-!  , &H!& &-%  !&   -& "* %& M *& &!& $>>+ O& *& $>>+ * *& %* " ,  #&  &-

12.4 A comparison of block and convolutional codes

6)1

>&-!& , *& - $! $>&  &->&  &- 1269 "  & *& -&I $&, -&  *& *& *  "!  >& ,J-!  %    -& , & $$@ &+ 1=2  &%* 277  711  %& $&, -& #&+ !$&  *  % 122   & -  -&& - $&@+ /& --!& *& * ,  >+ + &- -*& !& *& -*&  &@& &+ +  >-# -&  $-!   -& " $>>+ >& $&,&>&   -;! -&

  &- &@ $& -& *& "&> ! -*& &->&  *$& . O$ 379  % 126 "& %;& $&, -& -!;& , *&  & " -&  >&,&L *& H!+ Eb  *&  P=R "*&& P  *&  & $"& O "9  R  *& !-& & O > $& &-9 *! Eb *  & ?!& $& >L   !!+ -& *& energy per bit  Eb =N <  -& *& bit signal-to-noise ratio *&  NNII & * "* 

Figure 12.3  $ , >-#  -;! -&  "&> ! -*&

6)2

Survey of advanced topics for Part two

$&   Eb =N < &-+ >! 1< %1< Eb =N <  *  4  *&  & , * -*&L && , &@ $& &&  0-> F67G  % 126 *&  -&  + *   !$&+L   " #&+ ,&  *& -;! -& *   >&-!&  &  $$+ *& %&>- &-% %* , *$& 3  *&  -&   &-&+  !& >+ !$! H!4 O-, $ 6679 "*&& *& '&> %* - >& $&  --&$ !H!4& & ! >&  $! "* &;& && O-, &- 1+ >+ !$! H!4   $+  %&  >& - $&& , >+ *& ,- * *&  -&  *&&+ & $"&,! * *& -;! -&

*&  $ *%  #" >! *&& " &@ $&  * *& "&> ! -*& &   "& ;&+ , $-- - !- !  "*&&   *+ &;& --!   $!& ,    *!  >& !$% * -;! -& *;& >&& !&   ?+ , $-- $$- , &-&-% -& O  &&% &@ $& , $-- $$-D *& - !- # >&"&&  $-& ;&*-&  *& &*  !!+ H!& "& &&   "&> ! -*&  &;& $*-& -% -*& & *;& >&& !&  &&$$-& $$- -& *& & 13)!  13:: * +& " -$$&  ,; , *& O2 19M ˆ ) -;! -& &->&  * &- *& $-&  O&%   I Pioneer   /& & +I Helios9 *;& !& &H!& &-%  % OM > 2.9 O2 19 -;! -&9 & !  & *  13:) &;& , !    I 0& $! >+ &;&  &-% %* ,  $"&,! >-# -& *& O.( 2.9 H!- &!& -& * &  &H!& >+ !$! H!4

*& $&, -& , * -& * >&& $&  *& *& -!;&  % 126 *!%*  $&, -&   !$&  * , *& -;! -&  !%%& *& ,"% $&-!D -& *& O12: ).9  -&   >&& -& * *& O.( 2.9 -& Od  ˆ 21 ;: d  ˆ 129 ,  % >+ &-% %* -! >& ,! , *&  -&  %* "& ! !  >& %J-+ !$&  *& -;! -& + "& & * *& $&, -& , *& -;! -&  % 126  *& & , *&  + , ;!& , Pe , >! 1< 6  &  &@& &+   ;!& , Pe  *& >& #" $-- +&   --&  -*& & O&& &- )2  3)9  "*-* *& & -&   -;! -&  *& !& &   &&B  O>-#9 -& O&,&&-&D &"& F.1G9

12.5 Source codes

6)6

12.5 Source codes *I !-& -% *&& 6. !,,& , *&  & ? K"  * -*& -% *&& D   -!-;& O&& !-+ & #  *$& :9   *&&,& !  !$$& * &&-*& + *;& &;&$&  -!-;& *&+ , !-& -% %!  *& -!- ;& *&+ , -*& -% $&&&  *$& :B19 *& --&$>&   $--& &    * &;&+ & - >& %&L O-9 +   !-& * &  $--& & &@& &+ ,J-!  & *& -+  & *& -&$% & !& ,  *& -- &@ $& , *&  $*& &  *& -& , ;!  , &@ $& $*%$* ,  $& /& " &->&  & , *& &  $ &! * *;& >&& >&  -!-;& !-& -% /& >&% "* *& & +& ! !-& "* &H!& & J&+ -& O-, &- .29    $  *;& &@$- !-& -% &-*H!& , * !-& >* >&-!&  & ,&H!&+  $--&  >&-!& $&,&-     $>& -& RO& L1 ,


, Ln O-& H!4 levels9  T1 ,


, T n 1 O-& H!4 thresholds9 !-* * f Ou9 ˆ Lk ,, T k 1 , u < Tk  "*&& T< ˆ 1  Tn ˆ ‡1 * $$% ,  *& -!! & +& !-& U O & &9   -&& & +& !-& f OU 9 * ! & *& ;!& Lk "* $>>+ pk ˆ PfT k 1 , U < Tk g *& ;&%&  * &! , &$&&% U >+ f OU 9  , -!& ä ˆ EFU f OU 9G2  !*& &  "& &&  *$& 11 *& ;&%& ! >& , > &H!&  &$&&  + > , *& !-& f OU 9  HF f OU 9G *! O&,&%  % .29 *& $ R ˆ HF f OU 9G ä ˆ EFU f OU 9G2  -*&;>& >+ * H!4 -*& & !$%+ @  *& *;& *" >+ ! &- &-*H!& * , *& &;&  *&* & $$&+ -*&   $>&  * "+  -*&;& $&, -& *  H!& -& O"* & >9  *& ROä9 -!;& O @I $$&   $$ 2):B2:)  F27G &%& F16G &- 71

6).

Survey of advanced topics for Part two

$;&  & && ! + , H!4 &-*H!& -!%  -! , *& $$->+  ! !-& "* & + &&  $$& 13B27  F1)G9  &  &->&  & , *& & $*-& !-& -% &-* H!& * & #" "& $!&  -! *& %  + >&"&& !-& -%  -*& &-%  -+ "& * &- !&;&  *& >+ + &- !-& &;&  *&  %  O&& $$ (2B(69 &- *  !-& -& , &%* k  & Rs   !>& Cs ˆ fv1  v2  . . .  vM s g  Vk OF2 9 ˆ Vk "* k 1 %2 M s ˆ Rs      %;& >+ äOCs 9 ˆ

1
d  Fu f s Ou9G 2 k
k u2Vk

"*&& , &-* u 2 Vk *& &-% ,!- f s Ou9  &J&  >& *& !-& -&" , Cs -&  u "* &$&-  *&  % -& d  O-,
H O61(99 " &- *  -*& -& , &%* n "* & Rc   !>& Cc ˆ fx1  x2  . . .  x M c g  Vn "* Rc ˆ n 1 %2 M c  + *& -&" x i &  & >! >&-!& , -*& & x i - >& &-&;&  + y 2 Vn  *& &-% ,!- f c ! *& $ Vn  *& -& Cc L &+ f c $ *& &-&;& y  *& -&" -&  y "* &$&-  *&  % -& d    >* -& *& ,!- f $  ;&- , *& "*& $-&  *& -& -&" * !%%& * %;&  % -*& -&  &-% %*  *&  & -& - >& !&   !-& -& "*&& " *& -*& &-% %*  !&   !-& &-% %*  O -& + *& -*& &-% %* *! >& !&  *& !-& &-% %*   +! && "*+P9 * &   ,-  % & >!   & "* $,  "& * " &@$ /& *;& &+ && &@ $& , !%  -*& -& >-#"  %&  !-& -&  *& !-D *&& "&  *& &$& -& >-#"  &  $ "&  *& O: .9  % -& >-#"  %& R ˆ & !& +   -*& , "*-* & * *&& &   &@& & +  $>>& &;& ! , !-& -%  "*-* &-* , *& ;&-  V61  &H!+ #&+  >& & &    &L , (7V , *&  & *& &-&E&-& " > > !M O *& !! ,!& & " >& * *& &- $+  ó Ox9 " *;& &%&& 6 >! " *;& ,&"& * 6 4&&  *& J& F62 9

*& >;& -& *;& & &&-*&  +  &% $-- &-% %*  ,  -& * " $!-& *& && -&"  & "* *& &-& $!   %&&-&-%  -& *&&   $>& D *& &-% %*  &+ - $&& &&# $ F1.G *$& 1) * %;&  - $&& %* , *& "&-&-% -&  J+ '&*  &%& F.:G *;& !& *& *&&& -&-% -&

*!%* "& *;& !& *& --&$ , & $+%  -*& -&   !-& -& !% >-# -& & -! ?!  "& !& -;! -&  && &;& &&-*& *;& &   &@ $& &%& F16G *$& ) -!& -& J- , &H!& &-% * %;& % &! & &-& $$& , & F2(G %;&  $ , *& -# %*  !-& -% *& *%+ F1)G && >+ ;  + - &;& & $$&  *&  & & -!%  &&% $$-  $&&-* - $&

Appendix A >>+ *&+

/& & $*4&  -& * ! >?&- *&&    &-* *& && $>>+ *&+  * $!$& "& &- & &*& *& * ># >+  $& F(G  *& ";! & !+ >+ && F.G ! >?&-  &+ ?!  &&    -& -;&      & *& "&# " , %& ! >& "*-*  *& >-  !&  *& $, , *& -% *&&  , *$& 2  6

*& >- --&$  * ,  probability space 8Ù B P9 "*&& Ù   & $+ & -& *& sample space B   & J& , !>& , Ù  P   &%;& ,!- &J& ,  A 2 B "* *& $$&+ * PfÙg ˆ 1  91  1 8
P An ˆ PfA n g nˆ1

nˆ1

$;& &-* A n 2 B  *& A n I & ? P  -& *& probability measure  &@ $& , Ù ˆ fù1  ù2  . . .g  J&  -!>& B  *& -&- ,  !>& , Ù 8 p1  p2  . . .9   &H!&-& , &%;&  ! >& "*& !  1 *& *& &J PfAg ˆ f p n D ù n 2 Ag #& 8Ù B P9  $>>+ $-&L   -&  discrete $>>+ $-&

random variable X   ,!- $$% Ù   & & R -& *& range of X 8/& * &&  ;>& >+ !$$& -& && , & *& & , *& $*>&9  + "& ! & R   !>& , *& & ! >& >!  & & R " >& *& # , &  &@ $& , R   !>& , n &
!-& $-& X " >& -&  random vector  >& $&  >,-&D X *& - $& , X " >& && >+ X 1  X 2  . . .  X n L *!   ;&- X ˆ 8X 1  . . .  X n 9 - >& ;&"&    , 8& &9  ;>&

"  ;>& X  Y &J&  *&  & $>>+ $-& & 6))

Appendix A

6):

  >& &H! almost everywhere , *& & fùD X 8ù9 6ˆ Y 8ù9g * P &!&  X ˆ Y :&: ,, PfX 6ˆ Y g ˆ & "* %& R &H!  *& & &  expectation  ;&%&  &J& >+  E8X 9 ˆ X 8ù9 dPL Ù

 " E8X 9  *& &>&%!& &% , *& ,!- X "* &$&-  *& &!& P

*&&  *& &H!;& &J , &@$&- *  ,& & -;&& , - $! *& &!& P &J&  Ù !-&  $> >+ &!& PX  R ; *& &J PX 8S9 ˆ PfùD X 8ù9 2 Sg , + & & , & * &!&  -& *& distribution of X  *& ,!- S F X 8x9 ˆ PfùD X 8ù9 < xg  -& *& distribution function of X *& &@$&- , X  &H!+ %;& >+  1 E8X 9 ˆ x dPX ˆ x dF X 8x9 8 :19 R

1

 *& && , &>&%!&B&?& /& $!&  - &  " $&- -& , 8 19  , Ù  -&&  , , &-* x 2 R "& &J& p8x9 ˆ PX 8fxg9 ˆ PfùD X 8ù9 ˆ xg &J 8 19 >&- &
E8X 9 ˆ p8x9
x x

"*&& *& %& , !   $!$&+ &, !$&-J&L  - >& ;& + -&& !>& S , & !-* * PfX 2 Sg ˆ 1 & %&&+ , f  + &;!& ,!- &J&  S f 8X 9   &"  ;>&   &@$&-  %;& >+
EF f 8X 9G ˆ p8x9 f 8x9: 8 :29 x

!  , * # $$& *!%*!  & , * >#  ,& f 8x9 " ! & *&  $$& ;!& 1  " >& !&J& , -& ;!& , x "&;&  " "+ ! ! * f 8x9 " >&*;& +   & ,

6)(

Appendix A

&!& &  $>>+ $-&  & A1  A2  . . .  A n >& & , B *& NN&;&II A1  A2  . . .  A n &   >& independent , , &;&+ !>& , *&  A i1  . . .  A i m  "& *;& PfA i1 \ A i2 \


\ A i m g ˆ PfA i1 g . . . PfA i m g: &  $  -&- X 1  . . .  X n ,  ;>&  &J&  *&  &  $& $-&    >& &$&& , *& &;& A i ˆ fùD X i 8ù9 2 S i g & &$&& , + -*-& , S1  S2  . . .  S n 2 B *    &+ -  ;&,+ &-+  "& %;&  && , !  *& -& , + "  ;>& , X  Y & + $ , &  ;>& &J&  *&  &  $& $-& &$&&   *& $$% ù ! 8X 8ù9 Y 8ù99 !-&  &!& PXY  *& J& , " & & & *  -& *& joint distribution , X  Y  *& ,!- F XY 8x y9 ˆ PfùD X 8ù9 < x Y 8ù9 < yg  -& *& joint distribution function  + * X  Y & &$&&   + * *& &!& PXY   $!- &!& *  * , + $ S  T , & & & & PXY 8S 3 T 9 ˆ PX 8S9PY 8T 9 "*&& PX  PY & *& & & &!& !-& >+ X  Y 
H!;&+ X  Y & &$&& ,, F XY 8x y9 ˆ Fx 8x9FY 8 y9 "*&& F X  FY & *& >! ,!- , X  Y   *& -&& -& , "& &J& , &-* x y  8%& , X  %& , Y 9 _ p8x y9 ˆ PfùD X 8ù9 ˆ x Y 8ù9 ˆ yg X  Y & &$&& ,, p8x y9 ˆ p8x9 p8 y9 "*&& p8x9 ˆ PfX 8ù9 ˆ xg p8 y9 ˆ PfY 8ù9 ˆ yg 8& *& &>&& !& ,  >%!! D *&  & + > p8:9  !& , " - $&&+ ,,&& -&& >! ,!-L *&+ - >& %!*&

Appendix A

6)3

+ >+ *& ,- * && &  *& %& , X & && >+ xI && &  *& %& , Y >+ yI9  *& -!! -& , X  Y >* *;& && p8x9  q8 y9 X  Y & &$&& ,, x  y F XY 8x y9 ˆ p8s9q8t9 ds dt 1

1

*  X  Y *;&  joint density p8x y9 ˆ p8x9q8 y9 /& - " & *& "&# " , %& ! >& "*-*  ;  *& $, , *& -% *&&  Weak Law of Large Numbers  &-* n & X 1  X 2  . . .  X n >& &$& &  ;>& &-* "* J& &@$&- ìQ&& &-* "* *&  & >! ,!- *& , &-* å . å ˆ & K  E n O
!-& n$-&9  -& convex , *& & &% & ?% + " $ , K  -&  KD

*& & &% & ?% x1  x2  , + &J&  fx D x ˆ tx1 ‡ O1 t9x2  t 2 F& %;&  &  , convex combinations , + J& ! >& , $  ," $ x    >&  -;&@ - > , x1   x2  . . .  x m , *&& &@ &%;& - á1  á2  . . .  á m "* á i ˆ 1   á i xi ˆ x *& & ,  -;&@ - > , x1  x2  . . .  x m  -& *& convex hull O% 19 , x1  x2  . . .  x m    &+  *" *  & K  -;&@ ,, &;&+ -;&@ - > , $  K    K

Figure B.1 ;&@ *!  "  &

6:
& -!!  >!+ $ , K D & K ˆ F    Information Theory and Coding &" #D -" #  13)6

* && &+  *%*+ &>& !-  ,  *&+ -! % "*  $, , *& -*& -% *&& , -&& & +& -*& 16 &%& >+ Rate Distortion Theory
%&" ,, 0D &-& 13:1

 ;-& ># &% "*+ "* *& !-& -% *&&   %&&4   $-- $$- 1. &&# $
"+ Algebraic Coding Theory &" #D -" #  13)(

6(.

References

6(7

&% "+ *& $+ --&&&  *& ,!*& &-*& , *& *&+ , >-# -&

tour de force >+  !*&- $* %&! 17 &&# $
"+ & Key Papers in the Development of Coding Theory &" #D


& 13:.

 *%+ , ..  $ $$&  >& , *& &@&;& & - &&# $

& >+

1) ; &&  + >& & Data Compression !>&% D "& !-*    13:)

 *%+ , .)  $ $$& >! & *&+  !-& -% &-*H!& 1: %& >& Information Theory and Reliable Communication &" #D 0* /&+   13)(   & +& *&  ;-& &@>#  *& !>?&- -!& &@&;& & & , >-# -% & &@$&  ";&, -*& 1( '  0->! Coding Theory &-!& &  *& -  2&,  !- !-  *& *&+  $$- , -;! -& 22 && /&&+ Error-Correcting Codes  >%& D    & 13)1

*& -- &@  -% *&+ & & " >!  >& ,  & & , *, &%&  $& &   >& , &!->& $+  8&& &@ &+9 26 && /&&+  /&
" Error-Correcting Codes 2 &  >%& D    & 13:2

- $&&+ &;& ;& , *& $&-&% &+ -%  *& *&+ , $+  -&

!-* &" ,  >+

2. #&   Information and Information Stability of Random Variables and Processes

& >+  &&  --D &+ 13).

6()

References


%*  , *& -- ! &@ -% *& + ;>& - $&& & & , !! ,  , >+  ;>& 27 &$ ; & Key Papers in the Development of Information Theory &" #D 


 & 13:.

 *%+ , .3  $ $$&  ,  *&+ 12 >+ * -!% * -- NN *& - *&+ , - !-II  NN% *&&  ,  -&& !-& "*  J&+ -&II 2) '&> &"   ! 0  Digital Communication and Coding &" #D -" #  13:(
-+-$&- -;&%& , ,  *&+ &$&-+ &-& ;-&  & &@$&  & *&+
$*   -;! -& *!%*! 2: /,"4 0-> The Coding Theorems of Information Theory 6
 &D $%&'&% 13:(   "& ;&+ , %&&4 , *& -*& -% *&&

  -;&&

3. Original papers cited in the text FNote  , *&& $$& $$&&  *& IEEE Transactions on Information Theory "*-* "& *&&,& >>&;&  ITG 2( & 0* NN -# %* , !-& &-% "*  J&+ -&II IT 20 813:.9 211B22) 23   !%!! NN  %* , - $!% *& -$-+ , >+ -&& & +& -*&II IT 18 813:29 1.B2+ &1E2 -;! -&II IEEE Trans. Commun. COM-24 813:)9 3::B3(7 .7 !%+  ! 5*  " *%&-*   &#" **# NN &* , ;% #&+ &H! , &-% $$ -&II Information and Control 27 813:79 (:B33 .) ' >% &-! NN, + $-#& -&II * & &-*-*& %&-*
*;& *& &*& 13:) .: '&* 0&  &%& >+ NN $&& &-% , $&&-&-% >+  -&II IT 22 813:)9 16(B1.: .( /! /  NN&" -;! -&Q II IEEE Trans. Commun. COM-24 813:)9 3.)B377 .3 /+&  NN&-& &!  *& * *&+II IT 20 813:.9 2B1! &-$ ! >& , >! $& , &%* b  % >! , >! & '! & >! , >! -&- &&;% *&& , >! -&- &&;% -+-- -& % >! & -&-

*& & -!- &%%I & 

1:1 1:2 1:. 1(( 131 13) 2&   &H!& &-% 67( !& -&- 133 ,, ! , & >&*; , &H!& &-% %*   $&&-& , 62( 8> 1 :629 &&B  -& , -&-% 2)6B2). $-+ - $! , 12) -! , %J--& 7(B73 , >+ + &- -*& 11 , -&& & +& -*& 7. :6 8& .9 , + &-   7: 8 * 269 $-+- ,!- -!+ ,  ⠈ â   )3 8> 279 -;&@+ , 72 ,   71 ,, , ! -*& 37 %&& &J 116 *& *& - $$&& 7. %J--& -!& 72

63
 1 219 ): 8> 22 22)9 62( 8> 1+ + &- 1 16 8& 29 21 7< >- 12) -&& & +& 21 7< ! > & -$-+ , 3: &J& 37 %&&4& 127 $&, -& , & -&  1)2 "&> 3: 12< 8> 769 6)1B6)2 &  --&& +&  2)7 #; :< 8> 21 :119 q+ + &- 1.. r+ + &- 7( %+ + &- )( 8> 269 + &- 7: "&#&+ + &- )( 8> 229 *& -& See & -*& *& -% *&&  See -% *&&  *-&- , J& 6:) &

>  2& )<  8*$& 3  &;&   -& &-% %* , 26: ,,L *+ , 67.B677 &J& 262B266 %&&4& 2(6 8> 3119 %&& $+  , 267 $+-*&-# @ , 262 $>& !&  ! -*& 6)1B6)2 >-# 8 -&  *$& :B3 & >-# -&9 1)7 8& 29 - $&  -;! -& 673B6)< --&& "* -;! -& 6)2 ! >-#"  !-& -& 6).B6)7 -*& %&& &J ,   )< -K$$% . 12 8> - $$&& , 622 8> 1 1 :2.9   216 &&;& 1). 8> :629 2 36.B3639 5&-# 67. & 8*$& :  &;&  & -&9 -*& -% *&& - >& $;& , 123 -;! -&  %&&4 , 236 &J& 1.< %&&  $+-*&-# @ 1.6 8 * :19 !-& -% *&& - >& $;& , 167 & 67< &+ $&,&- 6.( & 676 ,, $&,&- 36 8> 6169 1.3 6.: ,, $&J@ 662

632

Index

$&$ 67. H!- &!& 676 6)2 &&B !& 1). 8> :6  :1(9 %&& -&-% 1.( %&& -&-% !>&& -&-% 172 8
@ :.9 !-& %&& &J , (. !-& && 162 66< ,, +& - 1.2 && 167 67( See also & -;! !, + $-#& 6.( ;>& &%* 8*$& 11  &;&  ;>&&%* -&9 ;&%& &%* , 66. &J& 661 !,,  %* , -!-% 66) ,, -*&     66. $ + &J& 66: % *&&  -*& ,  (B11 -;&&  117 16< , %&&   )2 8 * 2.9 :. 8& 1 221B22)9 %&& 116 126 , ! -*& 33 8 * .29 R<  , ! -*& 1& &!& >+ !! ,  2( &+ ,  ;>& 6)( &;;& ,  2. 31(9 Deus ex machina 3<  & ,  -& %* , - $!% 2(2 8> 3.9  & , & -& 1.< -&& & +& -*& See *& -& >! , -& 6.3  ;&%& ,  %&& - !- +&  117  &!& 8,  ,!-9 ,   :7 %&&4& 117  % (2 6.7 8& 29 H!&& , ! !-& 33 >! ,!- 6): ; --! *, &%& 1(: 8% (69   See *& -&& & +& d   1.:   See !-& -&& & +& ! > , -;! -& 622 8> 1 1+ 1 +  &H!& -% 673
!& + > 22 ) :629 133
 &;! $+  2.6
 &;& 61<
 - 27<
 - $+  2.6
 $& 7 1.6 171
 $>>+ > , 8: .9  % -& ( >-# ,  % -& : , -K$$% -& . 12 8>   :1.9 217
 ;!& 2)<
!-I %* , $+  2.. ,, 677
!&I ö,!- 6:)
@$&- ,  ;>& 6):
@$!% 1)1 8> :1:9 1)) 8& 169
@&& !-& 66)

E> = See  %& 
$+ % 661
-% %* 

 *- $;% !-&-*& -% *&&  11: 8% 729 , -;! -& 233 8% 1& 1: 6:. , -&&  ;&- 66 , &%- $-& 12. f&$+ .7 8> 1:9 , !  ;>& .1B.2 8
@ 1169 ? 2( , #; -* .)B.: 8> 12>+ ;&- 1(

-& -&  &-& - &  122 8& .9 n 1 ;& H 1(2 8 >& (19 6:( -% x  %*  62< 8% 1 1 111 1129  !& ,  67) &&>-# >+ &!& -*& &H!$$& "* :1 8> 22! :. 8& 179 127 & J& 6:7 ,,  & $>>+ 611 $K$ 1(. &+  123 623 8& .9 67: " & -*& , - $!% R8ä9 , ! !-& 1 .1(9 && -& , -;! -& %* , && % 627 8> 1 3.9 , %&& -+-- -& 1:6 , &&B  -& 27. >&B'* ; >! 1)2 8> :219 67< + -& See & + $$ -& 677 &&  67. $* &J , 6(<   @ 676 %&>%& / 16 8& 79  % >! 1)6 8> :229 2 2169 :1 8> 21:9 1 .)9 1.) 173 8> :.9  % "&%* 1.7  &- 22 1 .179 See also R!4 && 0 6)2 Helios $-&-, 6)2 "  6)2 -H!&%*&   266 67. $-, 0 123 !,, I %*  , -!-% $   -& 66) ,, &-$*&>& % 6.2 8> 1129 &$&&-& ,  ;>& &J 6)( ,  *& - &J 1( ,  !! See !! ,  ,  & 1)< 8> :169 ,  !-& See !-& $! $*>& See $*>& &&;% , -& , >!& $&- 1). 8> :629 2& (19 676 6:7 0->  6)2 0&&I &H!+ 6:2B6:. 0*&  67: 0* & 3) 0!&& -& See & 0!&& 5*  677 5 16 8& 29 5&+ &H! , &-% E -& 263 8 * 3.9 5,B -  &H!+ 662 5,I *&&   -!-% $&J@ -& 666 8 * 1129 & 5 67: " , %& ! >& 6 )) 3 6169 &%* , & -& 1.< &@-%$*- &% 6.6 8> 11) 11:9 & -& See & & & $% % >! 6.3B67< ;  0 676 -
&-& 0 12( 67+ 1)6 8> :2:9 & 1)6 8> :2(9 6.3 Mariner $-&-, 67: 6)2 #; -* &J& 23 &$+ , -&& .) 8> 12&&%* -& , 6.. 8> 11119 &+ 0 123 2(: 8> 36:9 623 8& .1 2169 &%%I &  21: 8 * (179 & + , -;! -& 236 233 & +& ! $ ,   6) , %&&4& -*& .6B.. &&-! &@ $& 66< &-  I %*  61:  '&>I %*  6+ &-% :< 8> 216 21.9  ! "&%* , & -& 1.: W>! ,!- 6:7  g8x9 --! 1(: 8% (69 !& 
 1). 8> :6! 1)6 8> :2.9 627 8> 1 7:9 &J@ 661 See also & $&J@  ;& $+  13:B13( 6::  ;&  6:) >>+ $-& 6)) >>+ ;&- 1:B1( !- ,  I )3 8> 239 R!4 &,,&-  -$-+ , ! -*& 1 221B22)9 12( 8% ).9 , ! -*& 1& &J& 6))  ;&- &J& 6)) & , -;! -& 23. 23: 233 , %&& -*& -& 73 , %&& - !- +&  117 , & -& 1.< , !-& -& (. & ,!- , >+   &;&   %  3. 8& )9 -!+ ,  ä ˆ ä   31 8> 6.9 &J ,   :)B:: &J %&& 11. &,J-& - $! , 16. , ! !-& 1 & $>>+ 2 +*!*! 5 266  67. &!-& -*& , - $!% C   77 &!- , !-& 6.< &! > See +-*&-# &&  67.B67) &&B !& -& See & &&B !&

63)

Index

&&B  -& See & &&B  &>+ &@$& 12: &$& -& See & &$& & -*
 67< 
8" &!-&
-*&9 @ 1.1 1)7 8& 79 ! &+  0 1 .169 67<  $& $-& 6)) -*#"?# 0  673 -*"4I &H!+ 1!  R8ä99 32 8> 61 :619 *& $* $>&  &&;-&  &-% -;! -& 6 12< 8> 769 6)1 &$B/, *&&  162B166 & 0  6.3 676 67) !-& >+ + &- 1 -&& & +& :7 &@&& 67) ! 33 ,, 16. 6)6B6). && + 11. + &- (2B(6 See also & ,!-    1)) 8& 129 67. !-& -& See & !-&-*& -% *&&  -! , 113B12< & & , 117 8 * 719 !-& -% *&&  See % *&&  $&&-* - $& 6)7 H!&&  -& See  &!& H!&";& 22 -# %*  61:B61(  + , & -& 1.7 -*- @ .) 8> 12% 661 !,J@ 661

!%+   677 ! ,  I )3 8> 2(9 !$&-*& 123 !;;  '&>I %*  6-#" , - $!% R8ä9 ,  ! !-& 1+ !$! H!4  ! -*& 1 .1(9   0 123 -&+ See
$+ H!& &->+ 66 1119 ' & &&>&% 1)6 8> :2)9 '&* 0 6)7 '& & &&  266 2(2 8> 369 '>& &%* -& 617 ,, See also *$& 11 '* ; See >&B'* ; >! '& %  , & & >&% >- ,- >! &$+  !! ,  2( 8% 179 '-# 0 673 '&>  12( 16. 167 623 8& .9 67( '&> &-% See &-% %* 

Index /&# -;&&  -% *&& ,  I 16< /&%* &! & 176 ,, /&%* , & $& ) See also  % "&%*L  ! "&%* , & -& /&-*  67< /&> ! -*& See *& /,"4 0 127

63:

/-&  11: /! / 673 /+&  12) /+&B=; *&& , 166B16. =& & -$-+ :1 8> 2139 :. 8& 1.9 =&& 0 67.