Source Requantization Successive Degradation and Bit Stealing

Citation preview

Sour e Requantization: Su

essive Degradation and Bit Stealing Aaron S. Cohen

Stark C. Draper

Emin Martinian

Gregory W. Wornell

Massa h usetts Institute of Te hnology, Cambridge, MA 02139 fa ohen,s d,emin,gwwgallegro.mit.edu

Abstra t We onsider sour e requantization in tw oforms | su

essive degradation (i.e., sour e delity redu tion) and bit stealing (i.e., information embedding) | when no forward planning has been done to fa ilitate the requantization. We fo us on nite-alphabet sour es with arbitrary distortion measures as well as the Gaussian-quadrati s enario. F orthe su

essive degradation problem, w e show an a hievable distortion-rate trade-o for non-hierar hi ally stru tured rate-distortion a hieving odes, and ompare it to the distortion-rate trade-o of su

essively re nable odes. We further onsider sour e requantization in the form of bit stealing, whereby an information embedder a ts on a quantized sour e, produ ing an output at the same rate. Building on the su

essive degradation results, we develop a hievable distortion-rate trade-os. Two ases are onsidered, orresponding to whether the sour e de oder is informed of any bit stealing or not. In the latter ase, the embedder must produ e outputs in the original sour e odebook. F or the Gaussian-quadrati s enario, all trade-os are within 0.5 bits/sample of the distortion-rate bound. F urthermore, for bit stealing, the use of simple post-re onstru tion pro essing that is only a fun tion of the embedded rate an eliminate the loss experien ed by uninformed de oders.

1 Introdu tion We onsider aspe ts of the problem of sour e requantization when no provisions hav e been made for requantization. First, we examine the su

essive degradation, or quantization rate redu tion, problem. As a motivating example, onsider transporting a sour e through a multi-hop network, where the sour e is originally en oded at rate

Rorig

with distortion

d0

drops to a residual rate

.

If the apa ity of some intermediate link along the path

Rres < Rorig

due to fading, ongestion, et ., a trans oder is

required to redu e the rate of the quantized sour e to

Rres

, whi h in turn in reases

This w ork has been supported in part b y NSF Grant No. CCR-0073520 and a gran t from Mi rosoft Resear h.

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

the distortion to d > d0 . Ideally, we would like trans oding to be eÆ ient in the sense that Rres and d still lie on the rate-distortion urve. If the sour e was originally en oded in a su

essively re nable manner [1 ℄, one

ould easily redu e the rate by erasing the least signi ant des riptions. We onsider what an be a hieved ev enwithout requiring su h spe ial sour e oding stru ture at the original en oder b y onsidering arbitrary random rate-distortion odebooks.1 In the network ongestion example, when a sour e pa ket and data pa k et arrive at a link that annot support their ombined rate, an alternative to dropping sour e bits to a

ommodate the data pa k et is to inje t the data bits into the sour e bits via information embedding into the sour e re onstru tion. If the embedded (stolen) rate is Rem b, then the residual rate that is ee tively allo ated to the sour e representation is Rres = Rorig Remb. We examine the degree to whi h this an be eÆ ient in terms of the asso iated distortion-rate trade-os. The basi dieren e between the s enarios onsidered here and those in, e.g., [1, 2℄, is the sign of the hange in total rate. Those works onsider how best to make use of an additional sour e observation and an in rease in the total rate (abov e the rate of the rst odebook) to minimize the distortion of the nal sour e estimate(s). In this work the total rate hange is negative, and when that hange must be made there is no sour e observation to work with, nor hav e prior provisions been made for the rate de rease. In ontrast to the results of [1℄, these onstraints lead to an unav oidable ratedistortion ineÆ ien y, even in the Gaussian-Quadrati ase. F urthermore, as opposed to, e.g., [3, 4, 5℄, the order of operations onsidered herein is reversed. In those works information embedding o

urs rst, followed by ompression. We onsider the reverse s enario where information embedding must be done in an already- ompressed sour e. Se tion 2 onsiders requantization in the form of su

essive degradation, while Se tion 3 in turn onsiders requantization in the form of bit stealing, both for informed and uninformed destination sour e de oders.

2 Su

essive Degradation Fig. 1 depi ts the su

essive degradation s enario. The sour e x is en oded at rate Rorig b y a rate-distortion a hieving sour e ode using a random odebook, produ ing sour e reprodu tion x^ at distortion d0  E [D(^x; x)℄. Throughout this paper, we assume that the sour e oder is generated in an i.i.d. manner; this stru ture is suÆ ient to a hieve the rate distortion bound and we believe that the output of any good sour e oder will look approximately i.i.d. The reprodu tion x^ is pro essed at the su

essive degrader to produ e a se ond sour e reprodu tion x. We de ne Rres (d) to be the su

essive degradation rate-distortion fun tion. That is, Rres (d) is the smallest residual rate su h that we an keep the distortion level E [D(x; x)℄ < d + Æ . In Appendix A we show the follo wing: 1 By

`arbitrary random rate-distortion odebook' we mean that the odebook is generated a

ording to an i.i.d. distribution p(^ xjx), without any spe ial stru ture su h as, e.g., the Markov stru ture of a su

essively-re nable odebook whi h, in the notation of [1℄, satis es p(^ x1 ; x ^2 jx) = p(^ x 1 jx ^2 )p(^ x2 jx) where E [D(^x1 ; x )℄  E [D(^x2 ; x )℄.

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

x

Source Coder

R orig

x

Source Decoder

R res

Re−encoder

x

Source Decoder

Successive Degrader

Figure 1: Requantization for su

essive degradation: redu ing the rate of an alreadyquantized sour e x^; Rres < Rorig .

Theorem 1

Let

x

b e dr awn i.i.d.

random sour e- o debo ok with rate

 p(x) and let p(^xjx) determine the design of a

Rorig

= I (^x ; x )

d0 < E [D (^ x ; x )℄

and distortion

+

Æ . An a hievable upper b ound on the su

essive de gr adationrate-distortion fun tion Rres (d) is given by Rres (d) inf p(xjx^) I (x ; x^) where p( x; x ^; x) = p( xx ^)p(^ x x)p(x) and E [D ( x ; x )℄ d.





2.1

j

j

Gaussian-Quadrati Results

In Appendix B we show that an a hievable upper-bound on the su

essive degradation rate-distortion fun tion Rres (d) for i.i.d. Gaussian sour es under a mean-squared distortion (MSD) measure is, for d > d0 , 

1 Rres (d)  min log 2



x2

d0

d

d0



1 ; log 2



x2 d0



:

(1)

As d ! x2 , Rres ! 0. As d ! d0 , Rres ! Rorig . Be ause the approa h in Appendix B requantizes the sour e as if it is an i.i.d. Gaussian sour e, while in fa t it is a member of a nite set of 2nRorig v e tors, there is some nite range of distortions abov e d0 for whi h requantization will not save us any rate, it is better to leave the original sour e

ode as is. T o nd the point where su

essive degradation (without time-sharing) be omes useful, set the rst term in the minimization of (1) equal to Rorig to get
d = d0 + (x2 d0 )2 2Rorig = d0 2 2 2Rorig , d : (2) This sa ys that ea h time we independently requantize the sour e with an independently generated sour e ode of the same rate, the o v eralldistortion in reases b y roughly d0 . We now upper bound the rate-loss in su

essive degradation (whi h an be further de reased with time-sharing). Sin e the biggest rate loss o

urs at d = d in (2), the maximum loss is Rres (d ) R(d )



1 log 2



x2 d0



1 log 2



d0 (2

x2

2



2Rorig )

=

1 1 log[2 2 2Rorig ℄ < : (3) 2 2

Our results indi ate there is a positive rate-loss: unlike in the su

essive re nement problem [1℄, in su

essive degradation we do not hav e a

ess to the original sour e signal when degrading the sour e, only to a quantized version of it. In re ent work [6℄ we hav e shown that random odebooks an be onstru ted so that Rres (d) is also lower-bounded b y the right-hand-side of (1). In other words, in the worst ase (1) holds with equality.

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

m R emb x

Source R orig Source Decoder Coder

x

Re−encoder

Information Embedder

R orig

Source Decoder

x

Message Decoder

m

Figure 2: Requantization for bit stealing: embedding a message in to an alreadyquantized sour e; Remb < Rorig .

3 Embedding in a Quantized Sour e Fig. 2 depi ts the s enario where requantization takes the form of embedding a message into a quantized sour e. The sour e x is en oded at rate Rorig b y a rate-distortion a hieving sour e ode using a random odebook, produ ing sour e reprodu tion x^. A message m of rate Rem b is embedded in to host x^. Be ause we want to embed a message in to the quantized sour e, the rates in to and out of the embedder are identi al. ^. The de oder produ es two estimates: of the sour e, x, and of the message, m We onsider two embedding sub lasses of su h problems. In the rst lass, the sour e destination is informed of any bit stealing and thus the embedder is free to reate an entirely new sour e odebook, requantize the sour e to that ode, and transmit that des ription; the remaining bits are used for the message. In the se ond lass, the sour e destination is not informed of any bit stealing, and th us the embedder must produ e outputs in the original sour e odebook. In the Gaussian-quadrati ase we show that, ev enwith this additional onstraint, if the de oder be omes informed of the rate of the embedding Rem b, then through the use of a simple post-re onstru tion s aling, the same distortion an be a hieved as if a totally new odebook were generated | as in the rst lass. 3.1

Embedding for Informed De oders: Rate-Splitting

In this ase, it suÆ es to apply the results of Se tion 2 to the embedding problem. In ee t, we split the rate into t wo streams. One, of rate Rem b, en odes the message. The other, of rate Rorig Rem b, is the residual rate left to des ribe the sour e. Theorem 1 gives us a strategy to a hieve a parti ular rate-distortion trade-o between Rem b and d for any nite-alphabet sour e. T oexplore this trade-o for Gaussian sour es and MSD, substitute Rorig Rem b into (1) for Rres , and solve for d to see that this method a hieves distortion equal to d(Rorig

3.2

Rem b)

= d0 + (x2

d0 )2

Rorig Rem b) = d + d 1 0 0

2(


2 2Rorig 22Rem b: (4)

Embedding for Uninformed De oders

In this ase, the sour e de oders in Fig. 2 are identi al. We de ne Remb (d) to be the largest embedding rate su h that we an keep the distortion level E [D(x; x)℄ < d + Æ .

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

In Appendix C we show the follo wing: p(^xjx) determine the design of a random sour e- o deb o ok with rate Rorig = I (^ x ; x ) and distortion d0 < E [D (^ x ; x )℄ + Æ . An a hievable lower-bound on the rate-distortion embe dding fun tion is given by Rem b(d) that satisfy Rorig Rem b(d)  inf p(xjx^) I (x ; x^) where p( x; x^; x) = p(xjx^)p(^ xjx)p(x), px^ (x) = px (x), and E [D(x; x)℄ < d + Æ . Theorem 2 Let

3.2.1

x be drawn i.i.d

 p(x)

and let

Gaussian-Quadrati Results

Let x be an i.i.d. zero-mean Gaussian sequen e of v arian eP x2 . Assume we are given xi xi )2 ℄  d0 + Æ . x^, the output of a rate Rem b sour e oder with MSD n1 E [ ni=1 (^ If we wish to embed in this quantized sour e a message of rate Rem b  Rorig , then Appendix D shows that the following MSD is a hievable by an uninformed de oder: "

#

n 1 X dU(Remb ) = E (xi n i=1

= x2 2

xi )2


2 2Rorig


2 2Rorig

2x2 1

p

1

2

2(Rorig

Remb ) :

(5)

In the ase of a de oder that is informed of the rate of embedding, Appendix D further shows we an improve the distortion by multiplying x b y where =

p

1

2

2(Rorig

Remb ) :

(6)

In parti ular, this s aling is in v ertible,so it does not ee t the embedding rate, but improv es distortion to dI (Rem b) = d0 + d0 1


2 2Rorig 22Rem b

(7)

where d0 = D(Rorig ) = x2 2 2Rorig . Comparing (7) with (4) we see that the informed de oder does as well as one given a ompletely new odebook. As one limiting ase, we substitute Rem b = 0 into (7) to get [ .f. (2)℄ dI (0) = d0 (2 2Rorig 2 ) ' 2d0 . This an be in terpreted as follows. If Rem b = 0 the embedder has nothing to do and no additional distortion should be in urred, i.e E [(x x )2 ℄ = d0 . But if we wish to embed at a sligh tly positive rate, Rem b > 0, be ause the embedder is restri ted to mappings between quantization v e tors, the odeword will be mov ed from the original quantization ve tor to a dierent one in the sour e odebook to en ode the message. In a good ve tor quantizer, quantization ve tors have a minimal spa ing of approximately 2d0 . F orpositive rates then, we expe t to see a minimal distortion on the order of 2d0 . As a se ond limiting ase, we substitute Rem b = Rorig into (6) and (7) to get = 0 and dI (Rorig ) = x2 . In ontrast, in the ase of uninformed de oders ( = 1), sin e jRem b=0 6= 1, then dU (0) > dI (0) = d0 (2 2 2Rorig ). In parti ular, substituting Rem b = Rorig in to (5) yields dU (Rem b) = 2x2 d0 . This an be interpreted as follows. When Rem b = Rorig the sour e is erased, and ea h quantization v e tor is used to en ode a dierent message. Erasing all information relevant to the sour e in urs Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

distortion x2 . Mapping a message independent of the sour e to one of the odebook

odewords in urs an additional distortion (x2 d0 ), the varian e of the quantization v e tors. This results in an o v eralldistortion of 2x2 d0 . Clearly though, if all bits are used to send the embedded message, there is no data about the original sour e left, so the de oder (if informed) should set x = 0, yielding a MSD of x2 , whi h is re e ted b y jRem b=Rorig = 0. A normalized distortion measure for the informed de oder is, via (7),

dnorm (Rem b) = dI (Rem b)=x2 = 2 2Rorig + 2 I

2(Rorig

Remb ) 1


2 2Rorig :

(8)

A normalized measure for uninformed de oders, dnorm (Rem b) an be derived by dividU ing (5) by x2 . In Fig. 3 normalized distortion is plotted versus residual sour e oding rate, Rres = Rorig Remb , for Rorig = 2. The normalized distortions of the uninformed de oder, dnorm (Rem b), and of the informed de oder, dnorm (Rem b), are plotted as dashed and U I 2 dash-dotted urves, respe tively. Time-sharing, when possible, makes the points that lie along the dotted line a hievable. At a parti ular residual sour e- oding rate, the minimum a hievable distortion is lower-bounded b y the distortion-rate fun tion D(Rorig Rem b), whi h is plotted with the solid urve. In Fig. 3, note the \requantization" gap between the bit stealing urves and the distortion-rate bound at Rorig Rem b = Rorig , i.e., Rem b = 0, whi h o

urs be ause the embedder is requantizing the quantized sour e. Unless the original quantization point is mapped ba k to itself, a non-zero minimal distortion is in urred. Sin e, using the approa h detailed in Appendix C, for any Remb > 0 the probability of mapping ba k to the same point goes to zero, we observe a gap. Unfortunately, one annot always av oid this dis ontinuity via time-sharing be ause one annot time-share within a single ve tor odebook. Only if there is a sequen e of quantized sour es transmitted

an time-sharing be implemented. In this spe ial ase one an time-share between the a hievable distortions giv enb y (8) and doing nothing (Rem b = 0) to get the points along the dotted line. A

Su

essively Degrading a Finite-Alphabet Sour e

Let x1 ; x2 ; : : : ; xn be dra wni.i.d.  p(x) and let D(x; x^) be a distortion measure for this sour e. Given a randomly-generated sour e ode that a hieves distortion d0 , w e an reen ode this odebook into a se ond odebook of rate R that a hieves distortion d as long as R > Rres (d), where Rres is the rate-distortion fun tion for this sour e/ rst odebook pair. We show that an a hievable upper-bound on Rres (d) is ( )  inf I (x ; x^) p(xjx^)

Rres d

(9)

)℄  d. where p( x; x ^; x) = p( xjx ^)p(^ xjx)p(x) and E [D (x ; x 2 As shown in (7), the distortion-rate performan e of the informed de oder for Gaussian sour es is equivalen t to the performan e of the rate-splitting embedder (4). This means that the dash-dot line in Fig. 3 also shows us the distortion-rate performan e of su

essive degradation sour e oding for the quadrati Gaussian ase dis ussed in Se tion 2.1.

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

1 Bit−stealing, uninformed Bit−stealing, informed Time−sharing Lower−bound, rate−dist.

Normalized Mean−Squared Distortion (MSD)

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

0

0.2

0.4 0.6 0.8 1 1.2 1.4 1.6 Residual Source−Coding Rate: Rres(d) = Rorig − Remb (bits/sample)

Figure 3: Residual-Rate-Distortion behavior;

urves plot

d

norm ( U

R

em

b) and

d

norm ( I

essive degradation performan e.

R

em

R

orig

1.8

2

= 2. The dashed and dash-dot

b), respe tively; the latter is equivalent to su -

Time-sharing (when possible) makes the points

along the dotted line a hievable. The solid urve is the lower-bound orresponding to the Gaussian distortion-rate fun tion 2

2

2Rorig

 2.

D(R

orig

R

em b).

Note

d

norm ( U

R

emb

=

R

orig )

=

Assume the sour e is quantized using a randomly generated odebook a

ording to p(^xjx) where Rorig > I (^x ; x ) and d0  E [D(^x ; x )℄ + Æ. In parti ular, the sour e

odebook C onsists of 2nRorig n-length sequen es generated a

ording to ni=1 p(x^i ). These sequen e are labeled ^x(1); : : : ; ^x(2nRorig ). F orsour e en oding, nd the indi es i su hthat (x; x^(i)) 2 A (n) , where A (n) denotes the set of jointly strongly typi al sequen es of length n. If there is more than one su h index, hoose any one of them. T ransmit thati. Quantizer:

Q

Given that w ew an t o in rease the sour e distortion to d > d0 , at the sour e degrader sear hov erjoint distributions p( x; x ^; x) = p( xjx ^)p(^ xjx)p(x) su hthat (1) p(^ xjx) and p(x) are as de ned as abov e, and (2) E [D ( x ; x )℄ < d + Æ . Pi k the distribution that satis es these onstraints while minimizing I (x ; x^). Generate a se ond odebook C2 whi h onsists of 2nR n-length sequen es generated a

ording to ni=1 p(xi ). These sequen e are labeled x(1); : : : ; x(2nR ). For en oding, nd the indi es j su h that (^x; x(j )) 2 A(n) . If there is more than one su h index, hoose an y one of them. T ransmit thatj . Sour e Degrader:

Q

De oding:

At the de oder the sour e estimate is x(j ).

Probability of error: The probability that the sour e oder annot nd at least one ^x(i) join tlystrongly typi al with x goes to zero for n large. This follows from standard join tstrong typi ality reasoning be ause Rorig > I (^x ; x ). The probability that the sour e

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

degrader annot nd a x(j ) join tlytypi al with ^x(i) goes to zero for n large. This follows from standard joint strong typi ality reasoning be ause R > I (x ; x^). Distortion: We now analyze the expe ted distortion betw een x and x. Be ause x and ^x x are jointly typi al, and x Æx^ Æx form a Markov hain, as n are jointly t ypi al,^x(and gets large (x; x) 2 A ) with probability approa hing one by the Markov lemma. Be ause they are jointly strongly typi al, E [D(x; x)℄ = P  p(x; x)D(x; x). Sin e R > I (x ; x^) is a hievable, and R > Rres(d), then (9) provides an upper bound on the rate-distortion fun tion Rres(d). 

n

x;x

B

Su

essively Degrading a Gaussian Sour e

T oextend the results of Appendix A to ontinuous alphabets, w emust partition x ! x , and x ! x arefully, so as to preserve the Markov relationship x Æx^ Æx for ea h partition. See [7℄ for more details. De ne the follo wingtest hannel: x^ = x + eorig where  = (1 d0 =x2 ) and eorig  N (0; d0 ). This is the standard test- hannel for rate-distortion sour e oders. Next de ne a se ond standard test hannel x = x^ + eres where = (1
=x^2) and eres  N (0;
). This ompound test hannel gives us overall MSD d = E (x x )2 = d = d0 +
, and  2  x d0 1 (10) I (x ; x^) = log 2
: Substitute
= d d0 into (10) to get an upper bound on Rres(d),  2  1 x d0 Rres (d)  log ; (11) 2 d d0 where d d0 > 0. p

x^ ! x^p

C

p

p

p

p

Embedding in a Quantized Finite-Alphabet Sour e

Let x1; x2 ; : : : ; x be dra wni.i.d.  p(x) and let D(x; x^) be a distortion measure for this sour e. Given a randomly-generated sour e ode that a hieves distortion d0 , we an embed a message of rate R in to this odebook ausing overall distortion d as long as R < Rem b(d), where Rem bis the rate-distortion fun tion for this sour e/ rst odebook pair. We show that an a hievable low er-bound onRem b(d) is given by the Remb(d) that satisfy Rorig Rem b(d)  inf I (x ; x^) (12) (j^) n

p xx

where p(x; x^; x) = p(xjx^)p(^xjx)p(x), px (x) = px^ (x), and E [D(x ; x)℄  d. Assume the sour e is quantized using a randomly generated odebook a

ording to p(^xjx) where Rorig > I (^x ; x ) and d0  E [D(^x ; x )℄ + Æ. In parti ular, the sour e Q n-length sequen es generated a

ording to =1 p(x^ ). These

odebook C onsists of 2 sequen e are labeled x^(1); : : : ; x^(2 ). F orsour e en oding, nd the indi es i su hthat (x; x^(i)) 2 A( ) , where A( ) denotes the set of jointly strongly typi al sequen es of length n. If there is more than one su h index, hoose any one of them. T ransmit thati. Quantizer:

nRorig

nRorig



n



n

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

n i

i

Information Embedder: Given that we want to embed a message at distortion d, at the information embedder sear h over joint distributions p(x; x^; x) = p(xjx^)p(^ xjx)p(x) su h that (1) p(^xjx) and p(x) are as de ned as above, (2) px (x) = px^ (x), and (3) E [D(x ; x )℄ < d + Æ. Pi k the distribution that satis es these onstraints while minimizing I (x ; x^). Randomly bin C in to 2nR sub odes Cj where R = Rorig I (x ; x^) ,  > 0.. That is, for ea h x^(i) pi kan index j uniformly distributed over 1; : : : ; 2nR and assign x^(i) to sub ode Cj . On av erage there are 2n(Rorig R) x^ sequen es in ea h Cj . Relabel the x^ sequen es in Cj as x(j; k) where j 2 1; : : : ; 2nR and k 2 1; : : : ; 2n(Rorig R) . Finally, given a sour e odeword ^(i), and a message m = m, nd the indi es k su h that (^x(i); x(m; k)) 2 A (n) . If there is x more than one su h index pi k any one. T ransmit the indexl su h that x^(l) = x(m; k). De oding:

x(m; k) = ^x(l).

^ = m su hthat The sour e estimate is ^x(l), and the message estimate is m

Probability of error: The probability that the sour e oder annot nd at least one ^x(i) jointly strongly typi al with x goes to zero for n large. This follows from standard joint strong typi ality reasoning be ause Rorig > I (^x ; x ). The probability that the information embedder annot nd a x(m; k) jointly typi al with ^(i) goes to zero for n large. T o see this, note that the probability that the original sour ex quantization ve tor x^(i) falls into the sele ted bin m is 2 nR , whi h goes to zero for n large. Then, onditioned on the event that x^(i) is not in bin m, the rest of the

odewords in bin Qn m look like i.i.d. sequen es generated independent of x^(i) a

ording to i=1 px (xi ). (Sin e the elements of Cm are also in C , and those in C w ere generated independently a

ording to Qn px^ (x) = px (x) the odewords in bin m look like they w eregenerated ^ (xi ), be ause i=1 px Q a

ording to ni=1 px (xi ).) The probability that at least one of these sequen es, x(m; k), is jointly strongly typi al with x^(i) goes to one for n large be ause there are 2n(Rorig R) ^ 6= m is zero be ause

odewords in bin m and Rorig R > I (x ; x^). The probability that m there is no noise in the system.

We now analyze the expe ted distortion between x and x. Be ause (1) x and ^x are jointly t ypi al, (2) ^x and x are jointly t ypi al, and (3) x Æx^ Æx form a Markov

hain, as n gets large (x; x) 2 A(n) with probability approa hing one by the Markov lemma. P Be ause they are jointly strongly typi al, E [D(x; x)℄ = x;x p(x; x)D(x; x). Sin e R = Rorig I (x ; x^)  is a hievable for all  > 0, and R  Remb (d), then (12) pro vides a low er bound on the rate-distortion fun tionRem b(d). Distortion:

D Embedding in a Quantized Gaussian Sour e

T oextend the results of Appendix C to ontin uous alphabets, w emust partition x ! xp , x^ ! x^p and x ! xp arefully, so as to preserve the Markov relationship xp Æx^p Æxp for ea h partition. See [7℄ for more details. In this se tion we derive a ompound test hannel for Gaussian sour e. First, de ne the follo wing test hannel: x^ = x + eorig where  = (1 d0 =x2 ) and eorig  N (0; d0 ). This is the standard test- hannel for rate-distortion sour e oders. Next de ne an information embedding test hannel [8℄ x = x^ + eem b where eemb  N (0; (1   

2 )(x2 d0 )). Note that E x2 = E x^2 whi h satis es the onstraint px^ (x) = px (x) sin e

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE

both are zero-mean Gaussian random variables. Finally, de ne x^^ = x . In Appendix C w edid not allo w post-pro essing of the estimate. That was the ase of an `uninformed' de oder, when the de oder does not know embedding has o

urred. In that ase = 1. We analyze the ase = 1 below, but also analyze the ase of `informed' de oders when an be hosen as a fun tion of the embedding rate R. The multipli ation by doesnot ee t the a hievable transmission rates, sin e the multipli ation an be inv erted, hen e the a hievabilit yarguments from Appendix C still hold. The ee t of the s aling is thus restri ted to distortion al ulations. We optimize ov er all possible s alings and . From rate-distortion results for white Gaussian sour es and mean-squared distortion  2 x 1 ^ measures, w e know that to a hieve distortion d0 , Rorig > 2 log d0 = I (x ; x ). F rom Theorem  2w e know that w e an embed at a rate Rpsu h that Rorig R = I (^x ; x) = 1 log 1 1 2 . Solve for as a fun tion of R to get = 1 2 2(Rorig R)  0, whi h holds 2 with equality when R = Rorig . Next, solve for the expe ted distortion and substitute in for , , and the varian es of eorig and eem b to get i h d = E (x^^ x )2 = ( 1)2 x2 +  2 2 d0 + 2 (1 2 )(x2 d0 ) # "    s d d d 0 0 0 2 1 22Remb + 1 : (13) = x2 2 1 1 2 2 2

x

x

x

For uninformed de oders, substitute = 1 and d0 = x2 2 2Rorig in to (13) to get (5). For informed de oders, dierentiate (13) with respe t to and set equal to zero to get p 2(Rorig R) = 1 2 , whi h upon substitution in to (13) with d0 = x2 2 2Rorig yields (7). Solving (7) for R yields a low er-bound onRem b(d) be ause Rem b(d)  R.

Referen es

[1℄ W. H. Equitz and T. M. Cov er, \Su

essive re nement of information," Info. Theory, vol. 37, pp. 269{275, Mar. 1991.

IEEE T rans.

[2℄ H. Viswanathan and T. Berger, \Sequential oding of orrelated sour es," IEEE Trans. Info. Theory, vol. 46, pp. 236{246, Jan. 2000.

[3℄ D. Karakos and A. P apamar ou,\Relationship between quantization and distribution rates of digitally watermarked data," in Pr o . IEEE Inter. Symp. Info. Theory, 2000. [4℄ D. Kundur, \Impli ations for high apa ity data hiding in the presen e of lossy ompression," in Pro . IEEE Int. Conf. On Info. T e h.: Coding & Comp., pp. 16{21, 2000. [5℄ M. Ramkumar and A. N. Akansu, \Information theoreti bounds for data hiding in

ompressed images," in Pr o . IEEE Workshop on Multimedia Sig. Pro ., 1998. [6℄ A. Cohen, S. C. Draper, E. Martinian, and G. W. Wornell, \Stealing bits from a quantized sour e," Submitted to Int. Symp. Info. Theory, 2002. [7℄ A. D. Wyner, \The rate-distortion fun tion for sour e oding with side information at the de oder{II: General sour es," Information and Control, v ol. 38, pp. 60{80, 1978.

[8℄ M. H. Costa, \Writing on dirty paper," IEEE Trans. Info. Theory, vol. 29, pp. 439{441, May 1983.

Proceedings of the DATA COMPRESSION CONFERENCE (DCC’02) 1068-0314/02 $17.00 © 2002 IEEE