Algorithms for Big Data Lecture Notes (UIUC CS598CSC)

Citation preview

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

1

Lecture date: August 26, 2014 Scribe: Chandra Chekuri

Introduction/Administrivia • Course website: https://courses.engr.illinois.edu/cs598csc/fa2014/. • There is no specific book that we will follow. See website for pointers to several resources. • Grading based on 4-5 homeworks, scribing a lecture and course project. Details to be figured out.

Course Objectives Big Data is all the rage today. The goal of this course is learn about some of the basic algorithmic and analysis techniques that have been useful in this area. Some of the techniques are old and many have been developed over the last fifteen years or so. A few topics that we hope to cover are: • Streaming, Sketching and Sampling • Dimensionality Reduction • Streaming for Graphs • Numerical Linear Algebra • Compressed Sensing • Map-Reduce model and some basic algorithms • Property Testing • Lower Bounds via Communication Complexity There is too much material in the above topics. The plan is to touch upon the basics so that it will provide an impetus to explore further.

2

Streaming/One-Pass Model of Computation

In the streaming model we assume that the input data comes as a stream. More formally, the data is assumed to consist of m items/tokens/objects a1 , a2 , . . . , am . A simple case is when each token is a number in the range [1..n]. Another example is when each token represents an edge in a graph given as (u, v) where u and and v are integers representing the node indices. The tokes arrive one by one in order. The algorithm has to process token xi before it sees the next token. If we are allowed to store all the tokens then we are in the standard model of computing where we have access to the whole input. However, we will assume that the space available to the algorithm is much less than m tokens; typically sub-linear in m or in the ideal scenario polylog(m). Our goal is to (approximately) compute/estimate various quantities of interest using such limited space. Surprisingly one can compute various useful functions of interest. Some parameters of interest for a streaming algorithm are: 1

• space used by the algorithm as a function of the stream length m and the nature of the tokens • the worst-case time to process each token • the total time to process all the tokens (equivalently the amortized time to process a token) • the accuracy of the output • the probability of success in terms of obtaining a desired approximation (assuming that the algorithm is randomized) There are several application areas that motivate the streaming model. In networking, a large number of packets transit a switch and we may want to analyze some statistics about the packets. The number of packets transiting the switch is vastly more than what can be stored for offline processing. In large databases the data is stored in disks and random access is not feasible; the size of the main memory is much smaller than the storage available on the disk. A one-pass or multi-pass streaming algorithm allows one to avoid sophisticated data structures on the disk. There are also applications where the data is distributed in several places and we need to process them separately and combine the results with low communication overhead. In database applications and such it also makes sense to discuss the multi-pass model or the semi-streaming model where one gets to make several passes over the data. Typically we will assume that the number of passes is a small constant.

3

Background on Probability and Inequalities

The course will rely heavily on proababilistic methods. We will mostly rely on discrete probability spaces. We will keep the discussion high-level where possible and use certain results in a black-box fashion. Let Ω be a finite P set. A probability measure p assings a non-negative number p(ω) for each space; an event in ω ∈ Ω such that ω∈Ω p(ω) = 1. The tuple (Ω, p) defines a discrete probability P this space is any subset A ⊆ Ω and the probability of an event is simply p(A) = ω∈A p(ω). When Ω is a continuous space such as the interval [0, 1] things get trickier and we need to talk about a measure spaces σ-algebras over Ω; we can only assign probability to certain subsets of Ω. We will not go into details since we will not need any formal machinery for what we do in this course. An important definition is that of a random variable. We will focus only on real-valued random variables in this course. A random variable X in a probability space is a function X : Ω → R. P In the discrete setting the expectation of X, denoted by E[X], is defined as ω∈Ω p(w)X(ω). For R continuous spaces E[X] = X(ω)dp(ω) with appropriate definition of the integral. The variance 2 , is defined as E[(X − E[X])2 ]. The standard deviation is σ , of X, denoted by Var[X] or as σX X the square root of the variance. Theorem 1 (Markov’s Inequality) Let X be a non-negative random variable such that E[X] is finite. Then for any t > 0, Pr[X ≥ t] ≤ E[X]/t. Proof: The proof is in some sense obvious, especially in the discrete case. Here is a sketch. Define a new random variable Y where Y (ω) = X(ω) if X(ω) < t and Y (ω) = t if X(ω) ≥ t. Y is

2

non-negative and Y ≤ X point-wise and hence E[Y ] ≤ E[X]. We also see that: X X tp(ω) X(ω)p(ω) + E[X] ≥ E[Y ] = ω:X(ω)≥t

ω:X(ω) 0, Pr[X > (1 + δ)µ] ≤ (1+δ)e (1+δ) . • For any µ ≤ E[X] and any δ > 0, Pr[X < (1 − δ)µ] ≤ e−µδ

2 /2

.

The following corollary bounds the deviation from the mean in both directions. Corollary 4 Under the conditions of Theorem 3, the following hold: • If δ > 2e − 1, Pr[X ≥ (1 + δ)µ] ≤ 2−(1+δ)µ . 2

• For any U there is a constant c(U ) such that for 0 < δ < U , Pr[X ≥ (1 + δ)µ] ≤ e−c(U )δ µ . In particular, combining with the lower tail bound, 2

Pr[|X − µ| ≥ δµ] ≤ 2e−c(U )t µ . We refer the reader to the standard books on randomized algorithms [6] and [4] for the derivation of the above bounds. If we are interested only in the upper tail we also have the following bounds which show the dependence of µ on n to obtain an inverse polynomial probability. 3

Corollary 5 Under the conditions of Theorem 3, there is a universal constant α such that for any ln n c µ ≥ max{1, E[X]}, and sufficiently large n and for c ≥ 1, Pr[X > αc ln ln n · µ] ≤ 1/n . Similarly, c there is a constant α such that for any  > 0, Pr[X ≥ (1 + )µ + αc log n/] ≤ 1/n . Remark 6 If the Xi are in the range [0, b] for some b not equal to 1 one can scale them appropriately and then use the standard bounds. Some times we P need to deal with random variables that are in the range [−1, 1]. Consider the setting where X = i Xi where for each i, Xi ∈ [−1, 1] and E[Xi ] = 0, and the Xi are independent. In this case E[X] = 0 and we can no longer expect a dimension-free bound. Suppose each Xi is 1 P with probability 1/2 and −1 with probability 1/2. Then X = i Xi corresponds to a 1-dimensional √ random walk and even though the expected value is 0 the standard deviation of X is Θ( n). One √ 2 can show that Pr[|X| ≥ t n] ≤ 2e−t /2 . For these settings we can use the following bounds. Theorem P7 Let X1 , X2 , . . . , Xn be independent random variables such that for each i, Xi ∈ [ai , bi ]. Let X = i ai Xi and let µ = E[X]. Then Pr[|X − µ| ≥ t] ≤ 2e

− Pn

i=1

2t2 (bi −ai )2

.

In particular if bi − ai ≤ 1 for all i then Pr[|X − µ| ≥ t] ≤ 2e−

Note that Var[X] =

P

i Var[Xi ].

2t2 n

.

One can show a bound based of the following form −

Pr[|X − µ| ≥ t] ≤ 2e

t2 2(σ 2 +M t/3) X

where |Xi | ≤ M for all i. Remark 8 Compare the Chebyshev bound to the Chernoff-Hoeffding bounds for the same variance. Statistical Estimators, Reducing Variance and Boosting: In streaming we will mainly work with randomized algorithms that compute a function f of the data stream x1 , . . . , xm . They typically work by producing an unbiased estimator, via a random variable X, for the the function value. That is, the algorithm will have the property that the E[X] is the desired value. Note that the randomness is internal to the algorithm and not part of the input (we will also later discuss randomness in the input when considering random order streams). Having an estimator is not often useful. We will also typically try to evaluate Var[X] and then we can use Chebyshev’s inequality. One way to reduce the variance of the estimate is to run the algorithm in parallel (with separate 1 P X as the final estimator. Note random bits) and get estimators X , X , . . . , X and use X = 1 2 i h i h P that Var(X) = h1 i Var(Xi ) since the Xi are independent. Thus the variance has been reduced by a factor of h. A different approach is to use the median value of X1 , X2 , . . . , Xh as the final estimator. We can then use Chernoff-Hoeffding bounds to get a much better dependence on h. In fact both approaches can be combined and we illustrate it via a concrete example in the next section.

4

4

Probabilistic Counting and Morris’s Algorithm

Suppose we have a long sequence of events that we wish to count. If we wish to count a sequent of events of length upto some N we can easily do this by using a counter with dlog2 N e bits. In some settings of interest we would like to further reduce this. It is not hard to argue that if one wants an exact and deterministic count then we do need a counter with dlog2 N e bits. Surprisingly, if we allow for approximation and randomization, one can count with about log logN bits. This was first shown in a short and sweet paper of Morris [5]; it is a nice paper to read the historical motivation. Morris’s algorithm keeps a counter that basically keeps an estimate of log N where N is the number of events and this requires about log logN bits. There are several variants of this, here we will discuss the simple one and point the reader to [5, 3, 1] for other schemes and a more detailed and careful analysis. ProbabilisticCounting: X←0 While (a new event arrives) Toss a biased coin that is heads with probability 1/2X If (coin turns up heads) X ←X +1 endWhile Output 2X − 1 as the estimate for the length of the stream.

For integer i ≥ 0, let Xn be the random variable that denotes the value of the counter after i events. Let Yn = 2Xn . The lemma below shows that the output of the algorithm is an unbiased estimator of the count that we desire. Lemma 9 E[Yn ] = n + 1. Proof: Proof by induction on n. The case of n = 0, 1 is easy to verify since in both cases we have that Yn is deterministically equal to n + 1. We have for n ≥ 2: Xn

E[Yn ] = E[2

] =

=

∞ X j=0 ∞ X j=0

=

∞ X

2j Pr[Xn = j]   1 1 2j Pr[Xn−1 = j] · (1 − j ) + Pr[Xn−1 = j − 1] · j−1 2 2 j

2 Pr[Xn−1 = j] +

j=0

∞ X

(2 Pr[Xn−1 = j − 1] − Pr[Xn−1 = j])

j=0

= E[Yn−1 ] + 1 = n+1 where we used induction to obtain the final equality. 2 Since E[Yn ] = n + 1 we have E[Xn ] = log2 (n + 1) which implies that the expected number of bits in the counter after n events is log log n + O(1) bits. We should also calculate the variance of Yn . Lemma 10 E[Yn2 ] = 32 n2 + 32 n + 1 and Var[Yn ] = n(n − 1)/2.

5

Proof: Proof is by induction on n. Easy to verify base cases n = 0, 1 since Yn = n + 1 deterministically. For n ≥ 2: X 22j · Pr[Xn = j] E[Yn2 ] = E[22Xn ] = j≥0

=

X

2j

2

j≥0

=

X

22j

  1 1 · Pr[Xn−1 = j](1 − j ) + Pr[Xn−1 = j − 1] j−1 2 2 X
−2j Pr[Xn−1 = j − 1] + 42j−1 Pr[Xn−1 = j − 1] · Pr[Xn−1 = j] +

j≥0

= = =

j≥0

2 E[Yn−1 ]

+ 3E[Yn−1 ] 3 3 (n − 1)2 + (n − 1) + 1 + 3n 2 2 3 2 3 n + n + 1. 2 2

We used induction and the value of E[Yn ] that we previously computed. Var[Yn ] = E[Yn2 ]−(E[Yn ])2 and it can be verified that it is equal to n(n − 1)/2. 2

4.1

Approximation and Success Probability

The analysis of the expectation shows that the output of the algorithm is an unbiased estimator. The analysis of the variance shows that the estimator is fairly reasonable. However, we would like to have a finer understanding. For instance, given some parameter  > 0, what is Pr[|Yn −(n+1)| > n]? We could also ask a related question. Is there an algorithm that given , δ guarantees that the output Y will satisfy the property that Pr[|Y − n| > n] ≤ δ while still ensuring that the counter size is O(log log n); of course we would expect that the constant in the O() notation will depend on , δ. The algorithm can be modified to obtain a (1 + )-approximation with constant probability using a related scheme where the probability of incrementing the counter is a1X for some parameter a; see [5, ?]. The expected number of bits in the counter to achieve a (1 + )-approximation can be shown to be log log n + O(log 1/) bits. Here we describe two general purpose ways to obtain better approximations by using independent estimators. Suppose we run the basic algorithm h times in parallel with independent 1 Ph randomness to get estimators Yn,1 , Yn,2 , . . . , Yn,h . We then output Z = h i=1 Yn,i − 1 as our estimate for n. Note that Z is also an unbiased estimator. We have that Var[Z] = h1 Var[Yn ]. Claim 11 Suppose h = 2/2 then Pr[|Z − n| ≥ n] < 14 . Proof: We apply Chebyshev’s inequality to obtain that Pr[|Z − n| ≥ n] ≤

Var[Z] 1 Var[Yn ] 1 n(n − 1) 1 ≤ ≤ < . 2 n2 h 2 n2 h 22 n2 4

2 Now suppose we want a high probability guarantee regarding the approximation. That is, we would like the estimator to be a (1 + )-approximation with probability at least (1 − δ) for some given parameter δ. We choose ` = c log 1δ for some sufficiently large constant c. We independently and in parallel obtain estimators Z1 , Z2 , . . . , Z` and output the median of the estimators; lets call Z 0 the random variable corresponding to the median. We will prove the following. 6

Claim 12 Pr[|Z 0 − n| ≥ n] ≤ (1 − δ). Proof: Define an indicator random variable P` Ai , 1 ≤ i ≤ ` where Ai is 1 if |Zi − n| ≥ n. From Claim 11, Pr[Ai = 1] < 1/4. Let A = i=1 Ai and hence E[A] < `/4. We also observe that |Z 0 − n| ≥ n only if A ≥ `/2, that is, if more than half of the Zi ’s deviate by more than n. We can apply Chernoff-Hoeffding bound to upper bound Pr[A ≥ `/2] = Pr[A ≥ (1 + δ)µ] where δ = 1 and µ = `/4 ≥ E[A]. From Theorem 3 this is at most (e/4)`/4 . Since ` = c log 1δ , the probability is at most δ for an appropriate choice of c. 2 The total space usage for running the estimates in parallel is O( 12 · log 1δ · log log n).

5

Reservoir Sampling

Random sampling is a powerful general purpose technique in a variety of areas. The goal is to pick a small subset S of a set of items N such that S is representative of N and often a random sample works well. The simplest sampling procedure is to pick a uniform sample of size k from a set of size m where k ≤ m (typically k  m). We can consider sampling with or without replacement. These are standard and easy procedures if the whole data set is available in a random-access manner — here we are assuming that we have access to a random number generator. Below we describe a simple yet nice technique called reservoir sampling to obtain a uniform sample of size 1 from a stream. UniformSample: s ← null m←0 While (stream is not done) m←m+1 xm is current item Toss a biased coin that is heads with probability 1/m If (coin turns up heads) s ← xm endWhile Output s as the sample

Lemma 13 Let m be the length of the stream. The output of the algorithm s is uniform. That is, for any 1 ≤ j ≤ m, Pr[s = xj ] = 1/m. Proof: We observe that s = xj if xj is chosen when it is considered by the algorithm (which happens with probability 1/j), and none of xj+1 , . . . , xm are chosen to replace xj . All the relevant events are independent and we can compute: Y Pr[s = xj ] = 1/j × (1 − 1/i) = 1/m. i>j

2 To obtain k samples with replacement, the procedure for k = 1 can be done in parallel with independent randomness. Now we consider obtaining k samples from the stream without replacement. The output will be stored in an array of S of size k.

7

Sample-without-Replacement(k): S[1..k] ← null m←0 While (stream is not done) m←m+1 xm is current item If (m ≤ k) S[m] ← xm Else r ← uniform random number in range [1..m] If (r ≤ k) S[r] ← xm endWhile Output S

An alternative description is that when item xt arrives (for t > k) we decide to choose it for inclusion in S with probability k/t, and if it is chosen then we choose a uniform element from S to be replaced by xt . Exercise: Prove that the algorithm outputs a random sample without replacement of size k from the stream. Weighted sampling: Now we consider a generalization of the problem to weighted sampling. Suppose the stream consists of m items x1 , . . . , xm and each item j has a weigth wj > 0. The goal is to choose a k sample in proportion to the weights. P Suppose k = 1 then the goal is to choose an item s such that Pr[s = xj ] = wj /W where W = i wi . It is easy to generalize the uniform sampling algorithm to achieve this and k samples with replacement is also easy. The more interesting case is when we want k samples without replacement. The precise definition of what this means is not so obvious. Here is an algorithm. Obtain first a uniform sample s from x1 , . . . , xm in proportion to the weights. Remove s from the set and obtain another uniform sample in the residual set. Repeat k times to obtain a set of k items (assume that k < m). The k removed items form the output S. We now want to obtain a random set S according to this same distribution but in a streaming fashion. First we describe a randomized offline algorithm below. Weighted-Sample-without-Replacement(k): For i = 1 to m do ri ← uniform random number in interval (0, 1) 1/w wi0 = ri i endFor Sort items in decreasing order according to wi0 values Output the first k items from the sorted order

We leave it as a simple exercise to show that the above algorithm can be implemented in the stream model by keeping the heaviest k modified weights seen so far. Now for the analysis. To get some intuition we make the following claim. 8

Claim 14 Let r1 , r2 be independent unformly distributed random variables over [0, 1] and let X1 = 1/w 1/w r1 1 and X2 = r2 2 where w1 , w2 ≥ 0. Then Pr[X1 ≤ X2 ] =

w2 . w1 + w2

The above claim can be shown by doing basic analysis via the probability density functions. More precisely, suppose w > 0. Consider the random variable Y = r1/w where r is chosen uniformly in (0, 1). The cumulative probabilty function of Y , FY (t) = Pr[Y ≤ t] = Pr[r1/w ≤ t] = Pr[r ≤ tw ] = tw . Therefore the density function fY (t) is wtw−1 . Thus Z Pr[X1 ≤ X2 ] =

1

Z

1

FY1 (t)fY2 (t)dt =

tw1 w2 tw2 −1 dt =

0

0

w2 . w1 + w2

We now make a much more general statement. Lemma 15 Let r1 , r2 , . . . , rn be independent random variables each of which is uniformly dis1/w tributed random variables over [0, 1]. Let Xi = ri i for 1 ≤ i ≤ n. Then for any α ∈ [0, 1] Pr[X1 ≤ X2 . . . ≤ Xn ≤ α] = α

w1 +w2 +...+wn

·

n Y i=1

wi . w1 + . . . + wi

Proof: By induction on n. For n = 1, Pr[X1 ≤ α] = FY1 (α) = αw1 . Assuming the lemma holds for all h < n we prove it for n. Z α Pr[X1 ≤ . . . ≤ Xn−1 ≤ t]fYn (t)dt Pr[X1 ≤ . . . ≤ Xn ≤ α] = 0 ! Z α n−1 Y w i wn twn −1 dt = tw1 +w2 +...+wn−1 · w1 + . . . + wi 0 i=1 !Z n−1 α Y wi = wn tw1 +w2 +...+wn −1 dt w1 + . . . + wi 0 i=1

= αw1 +w2 +...+wn ·

n Y i=1

wi . w1 + . . . + wi

We used the induction hypothesis in the second equality. 2 Now we are ready to finish the proof. Consider any fixed j. We claim that the probability wj that Xj is the largest number among X1 , X2 , . . . , Xm is equal to w1 +...+w . Do you see why? n Conditioned on Xj being the largest, the rest of the variables are still independent and we can apply this observation again. You should hopefully be convinced that picking the largest k among the values X1 , X2 , . . . , Xm gives the desired sample. Bibliographic Notes: Morris’s algorithm is from [5]. See [3] for a detailed analysis and [1] for a more recent treatment which seems cleaner and easier. Weighted reservoir sampling is from [2]. See [7] for more on efficient reservoir sampling methods. 9

References [1] Mikl´os Cs¨ ur¨ os. Approximate counting with a floating-point counter. CoRR, abs/0904.3062, 2009. [2] Pavlos S Efraimidis and Paul G Spirakis. Weighted random sampling with a reservoir. Information Processing Letters, 97(5):181–185, 2006. [3] Philippe Flajolet. Approximate counting: a detailed analysis. BIT Numerical Mathematics, 25(1):113–134, 1985. [4] Michael Mitzenmacher and Eli Upfal. Probability and computing: Randomized algorithms and probabilistic analysis. Cambridge University Press, 2005. [5] Robert Morris. Counting large numbers of events in small registers. Communications of the ACM, 21(10):840–842, 1978. [6] Rajeev Motwani and Prabhakar Raghavan. Randomized Algorithms. Cambridge University Press, 1995. [7] Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37–57, 1985.

10

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

1

Lecture date: August 28, 2014 Scribe: Chandra Chekuri

Estimating Frequency Moments in Streams

A significant fraction of streaming literature is on the problem of estimating frequency moments. Let σ = a1 , a2 , . . . , am be a stream of numbers where for each i, ai is an intger between 1 and n. We will try to stick to the notation of using m for the length of the stream and n for range of the integers1 . Let fi be the number of occurences (or frequency) of integer i in the stream. We let f = (f1 , f2 , . . . , fn ) be the frequency vector for a given stream σ. For k ≥ 0, Fk (σ) is defined to be the k’th frequency moment of σ: X Fk = fik . i

We will discuss several algorithms to estimate Fk for various P values of k. For instance F0 is simply the number of distinct elements in σ. Note that F1 = i fi = m, the length of the stream. A k increases to ∞ Fk will concentrate on the most frequent element and we can thing of F∞ as finding the most frequent element. Definition 1 Let A)(σ) be the real-valued output of a randomized streaming algorithm on stream σ. We say that A provides an (α, β)-approximation for a real-valued function g if   A(σ) Pr | − 1| > α ≤ β g(σ) for all σ. Our ideal goal is to obtain a (, δ)-approximation for any given , δ ∈ (0, 1).

2

Background on Hashing

Hashing techniques play a fundamental role in streaming, in particular for estimating frequency moments. We will briefly review hashing from a theoretical point of view and in particular kuniversal hashing. A hash function maps a finite universe U to some range R. Typically the range is the set of integers [0..L − 1] for some finite L (here L is the number of buckets in the hash table). Sometimes it is convenient to consider, for the sake of developing intuition, hash functions that maps U to the continuous interval [0, 1]. We will, in general, be working with a family of hash functions H and h will be drawn from H uniformly at random; the analyis of the algorithm will be based on properties of H. We would like H to have two important and contradictory properties: • a random function from H should behave like a completely random function from U to the range. • H should have nice computational properties: 1

Many streaming papers flip the notation and use n to denote the length of the stream and m to denote the range of the integers in the stream

1

– a uniformly random function from H should be easy to sample – any function h ∈ H should have small representation size so that it can be stored compactly – it should be efficient to evaluate h Definition 2 A collection of random variables X1 , X2 , . . . , Xn are k-wise independent if the variables Xi1 , Xi2 , . . . , Xik are independent for any set of distinct indices i1 , i2 , . . . , ik . Take three random variables {X1 , X2 , X3 } where X1 , X2 are independent {0, 1} random variables and X3 = X1 ⊕ X2 . It is easy to check that the three variables are pairwise independent although they are not all independent. Following the work of Carter and Wegman [3], the class of k-universal hash families, and in particular for k = 2, provide an excellent tradeoff. H is strongly 2-universal if the following properties hold for a random function h picked from H: (i) for every x ∈ U, h(x) (which is a random variable) is uniformly distributed over the range and (ii) for every distinct pair x, y ∈ U, h(x) and h(y) are independent. 2-universal hash families are also called pairwise independent hash families. A weakly 2-universal family satisfies the property that that Pr[h(x) = h(y)] = 1/L for any distinct x, y. We state an important observation about pairwise independent random variables. P Lemma 1 Let Y = hi=1 Xi where X1 , X2 , . . . , Xh are pairwise independent. Then Var[Y ] =

h X

Var[Xi ].

i=1

Moreover if Xi are binary/indicator random variables then X X Var[Y ] ≤ E[Xi2 ] = E[Xi ] = E[Y ]. i

i

There is a simple and nice construction of pairwise independent hash functions. Let p be a prime number such that p ≥ |U|. Recall that Zp = {0, 1, . . . , p − 1} forms a field under the standard addition and multiplication modulo p. For each a, b ∈ [p] we can define a hash function ha,b where ha,b (x) = ax + b mod p. Let H = {ha,b | a, b ∈ [p]}. We can see that we only need to store two numbers a, b of Θ(log p) bits to implicitly store ha,b and evaluation of ha,b (x) takes one addition and one multiplication of log p bit numbers. Moreover, samply a random hash function from H requires sampling a, b which is also easy. We claim that H is a pairwise independent family. You can verify this by the observation that for distinct x, y and any i, j the two equations ax + b = i and ay + b = j have a unique a, b them simultaneously. Note that if a = 0 the hash function is pretty useless; all elements get mapped to b. Nevertheless, for H to be pairwise independent one needs to include those hash functions but the probability that a = 0 is 1/p while there are p2 functions in H. If one only wants a weakly universal hash family we can pick a from [1..(p − 1)]. The range of the hash function is [p]. To restrict the range to L we let h0a,b (x) = (ax + b mod p) mod L. More generally we will say that H is k-universal if every element is uniformly distributed in the range and for any k elements x1 , . . . , xk the random variabels h(x1 ), . . . , h(xk ) are independent. Assuming U is the set of integers [0..|U|], for any fixed k there exist constructions for k-universal hash families such that every hash function h in the family can be stored using O(k log |U|) bits (essentially k numbers) and h can be evaluated using O(k) arithmetic operations on log |U| bit numbers. We will ignore specific details of the implementations and refer the reader to the considerable literature on hashing for further details. 2

3

Estimating Number of Distinct Elements

A lower bound on exact counting deterministic algorithms: We argue that any deterministic streaming algorithm that counts the number of distinct elements exactly needs Ω(n) bits. To see this, suppose there is an algorithm A that uses strictly less than n bits. Consider the h = 2n different streams σS where S ⊆ [n]; σS consists of the elements of S in some arbitrary order. Since A uses n − 1 bits or less, there must be two distinct sets S1 , S2 such that the state of A at the end of σS1 , σS2 is identical. Since S1 , S2 are distinct there is an element i in S1 \ S2 or S2 \ S1 ; wlog it is the former. Then it is easy to see the A cannot give the right count for at least one of the two streams, < σS1 , i >, < σS2 , i >. The basic hashing idea: We now discuss a simple high-level idea for estimating the number of distinct elements in the stream. Suppose h is an idealized random hash function that maps [1..n] to the interval [0, 1]. Suppose there are d distinct elements in the stream σ = a1 , a2 , . . . , am . If h behaves like a random function then the set {h(a1 ), . . . , h(am )} will behave like a collection of d independent uniformly distributed random variables in [0, 1]. Let θ = min{h(a1 ), . . . , h(am )}; the 1 expectation of θ is d+1 and hence 1/θ is good estimator. In the stream setting we can compute θ by hashing each incoming value and keeping track of the minimum. We only need to have one number in memory. Although simple, the algorithm assumes idealized hash functions and we only have an unbiased estimator. To convert the idea to an implementable algorithm with proper guarantees requires work. There are several papers on this problem and we will now discuss some of the approaches.

3.1

The AMS algorithm

Here we describe an algorithm with better parameters but it only gives a constant factor approximation. This is due to Alon, Matias and Szegedy in their famous paper [1] on estimating frequency moments. We need some notation. For an integer t > 0 let zeros(t) denote the number of zeros that the binary representation of t ends in; equivalenty zeros(t) = max{i :| 2i divides t}. AMS-DistinctElements: H is a 2-universal hash family from [n] to [n] choose h at random from H z←0 While (stream is not empty) do ai is current item z ← max{z, zeros(h(ai ))} endWhile 1 Output 2z+ 2

First, we note that the space and time per element are O(log n). We now analyze the quality of the approximation provided by the output. Recall that h(aj ) is uniformly distributed in [n]. We will assume for simplicity that n is a power of 2. Let d to denote the number of distinct elements in the stream and let them be b1 , b2 , . . .P , bd . For a given r let Xr,j be the indicator random variable that is 1 if zeros(h(bj )) ≥ r. Let Yr = j Xr,j . That is, Yr is the number of distinct elements whose hash values have atleast r zeros. 3

Since h(bj ) is uniformaly distribute in [n], E[Xr,j ] = Pr[zeros(h(bj )) ≥ r] = Therefore E[Yr ] =

X

E[Xr,j ] =

j

(n/2r ) 1 ≥ r. n 2 d . 2r

Thus we have E[Ylog d ] = 1 (assuming d is a power of 2). Now we compute the variance of Yr . Note that the variables Xr,j and Xr,j 0 are pairwise independent since H is 2-universal. Hence Var[Yr ] =

X j

Var[Xr,j ] ≤

X

2 E[Xr,j ]=

j

X

E[Xr,j ] =

j

d . 2r

Using Markov’s inequality Pr[Yr > 0] = Pr[Yr ≥ 1] ≤ E[Yr ] ≤

d . 2r

Using Chebyshev’s inequality Pr[Yr = 0] = Pr[|Yr − E[Yr ]| ≥

d Var[Yr ] 2r ] ≤ ≤ . 2r (d/2r )2 d 0

1

Let z 0 be the value of z at the end of the stream and let d0 = 2z + 2 be the estimate for d output by the algorithm. We claim that d0 cannot be too large compared to d with constant probability. 1 Let a be the smallest integer such that 2a+ 2 ≥ 3d. √ d 2 0 Pr[d ≥ 3d] = Pr[Ya > 0] ≤ a ≤ . 2 3 Now we claim that d0 is not too small compared to d with constant probability. For this let b the 1 largest integer such that 2b+ 2 ≤ d/3. Then, √ 2b+1 2 0 Pr[d ≤ d/3] = Pr[Yb+1 = 0] ≤ ≤ . d 3 √ Thus, the algorithm provides (1/3, 2/3 ' 0.4714)-approximation to the number of distinct elements. Using the median trick we can make the probability of success be at least (1 − δ) to obtain a (1/3, δ)-approximation by running O(log 1δ )-parallel and independent copies of the algorithm. The time and space will be O(log 1δ log n).

3.2

A (1 − )-approximation in O( 12 log n) space

Bar-Yossef et al. [2] described three algorithms for distinct elements, that last of which gives a ˜ 2 + log n)) space and amortized time per element O(log n + log 1 ); the notation O) ˜ bound O(  suppresses dependence on log log n and log 1/. Here we describe their first algorithm that gives gives an (, δ0 )-approximation in space O( 12 log n) and O(log 1/ log n) time per update; via the median trick, with an additional log 1/δ factor, we can obtain a (, δ)-approximation. The algorithm is based on the Flajolet-Martin idea of hashing to [0, 1] and taking the minimum but with a small and important technical tweak. Let t = c2 for some constant c to be fixed later. 4

BJKST-DistinctElements: H is a 2-universal hash family from [n] to [N = n3 ] choose h at random from H t ← c2 While (stream is not empty) do ai is current item Update the smallest t hash values seen so far with h(ai ) endWhile Let v be the t’th smallest value seen in the hast values. Output tN/v.

We observe that the algorithm can be implemented by keeping track of t hash values each of which take O(log n) space and hence the space usage is O(t log n) = O( 12 log n). The t values can be stored in a binary search tree so that when a new item is processed we can update the data structure in O(log t) searches which takes total time O(log 1 log n). The intution of the algorithm is as follows. As before let d be the number of distinct elements and let them be b1 , . . . , bd . The hash values h(b1 ), . . . , h(bd ) are uniformly distributed in [0, 1]. For any fixed t we expect about t/d hash values to fall in the interval [0, t/d] and the t’th smallest hash value v should be around t/d and this justifies the estimator for d being t/v. Now we formally analyse the properties of this estimator. We chose the hash family to map [n] to [n3 ] and therefore with probability at least (1 − 1/n) the random hash function p h is injective over [n]. We will assume this is indeed the case. Moreover we will assume that n ≥ /24 which implies in particular that t/(4d) ≥ 1/N . Let d0 the estimate returned by the algorithm. Lemma 2 Pr[d0 < (1 − )d] ≤ 1/6]. Proof: The values h(b1 ), . . . , h(bd ) are uniformly distributed in 1..N . If d0 < (1 − )d it implies tN tN that v > (1−)d ; that is less than t values fell in the interval [1, (1−)d ]. Let Xi be the indicator Pd random variable for the event that h(bi ) ≤ (1 + )tN/d and let Y = i= Xi . Since h(bi ) is distributed uniformly in [1..N ], taking rounding errors into account, we have (1 + /2)t/d ≤ E[Xi ] ≤ (1 + 3/2)t/d and hence E[Y ] ≥ t(1 + /2). We have Var[Y ] ≤ E[Y ] ≤ t(1 + 3/2) (due to pairwise independence, Lemma 1)). We have d0 < (1 − )d only if Y < t and by Chebyshev, Pr[Y < t] ≤ Pr[|Y − E[Y ]| ≥ t/2] ≤

4Var[Y ] ≤ 12(2 t2 ) ≤ 1/6. 2 t2

2

Lemma 3 Pr[d0 > (1 + )d] ≤ 1/6]. Proof: Suppose d0 > (1 + )d, that is v < less than

tN (1+)d

tN (1+)d .

This implies that more than t hash values are

≤ (1 − /2)tN/d. We will show that this happens with small probability.

P Let Xi be the indicator random variable for the event h(bi ) < (1−/2)tN/d and let Y = di=1 Xi . We have E[Xi ] < (1 − /2)t/d + 1/N ≤ (1 − /4)t/d (the 1/N is for rounding errors). Hence E[Y ] ≤ (1 − /4)t. As we argued d0 > (1 + )d happens only if Y > t. By Cheybyshev, Pr[Y > t] ≤ Pr[|Y − E[Y ]| ≥ t/4] ≤

16Var[Y ] ≤ 16/(2 t2 ) ≤ 1/6. r t2 2

5

Bibliographic Notes: Our description of the AMS algorithm is taken from notes of Amit Chakrabarti. Flajolet and Martin proposed the basic hash function based algorithm in [5]. [2] Kane, Nelson and Woodruff [7] described an (, δ0 )-approximation for a fixed δ0 in space O( 12 + log n) (the time per update is also the same). This is a theoretically optimum algorithm since there are lower bounds of Ω(2 ) and Ω(log n) on the space required for an -approximation. We refer the reader to [4, 6] for analysis and empirical evaluation of an algorithm called HyperLogLog which seems to very well in practice.

References [1] Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximating the frequency moments. Journal of Computer and System Sciences, 58(1):137–147, 1999. Preliminary version in Proc. of ACM STOC 1996. [2] Ziv Bar-Yossef, TS Jayram, Ravi Kumar, D Sivakumar, and Luca Trevisan. Counting distinct elements in a data stream. In Randomization and Approximation Techniques in Computer Science, pages 1–10. Springer, 2002. [3] J Lawrence Carter and Mark N Wegman. Universal classes of hash functions. Journal of Computer and System Sciences, 18(2):143–154, 1979. Preliminary version in Proc. ACM STOC, 1977. [4] Philippe Flajolet, Eric Fusy, Olivier Gandouet, Fr´ed´eric Meunier, et al. Hyperloglog: the analysis of a near-optimal cardinality estimation algorithm. Analysis of Algorithms 2007 (AofA07), pages 127–146, 2007. [5] Philippe Flajolet and G Nigel Martin. Probabilistic counting algorithms for data base applications. Journal of computer and system sciences, 31(2):182–209, 1985. [6] Stefan Heule, Marc Nunkesser, and Alex Hall. Hyperloglog in practice: Algorithmic engineering of a state of the art cardinality estimation algorithm. In Proceedings of the EDBT 2013 Conference, Genoa, Italy, 2013. [7] Daniel M Kane, Jelani Nelson, and David P Woodruff. An optimal algorithm for the distinct elements problem. In Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 41–52. ACM, 2010.

6

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

1

Lecture date: Sept 2, 2014 Scribe: Chandra Chekuri

AMS Sampling

We have seen reservoir sampling and the related weighted sampling technique to obtain independent samples from a stream without the algorithm knowing the length of the stream. We now discuss a technique to sample from a stream σ = a1 , a2 , . . . , am where the tokens aj are integers from [n] and we wish to estimate a function X g(fi ) g(σ) := i∈[n]

where fi is the frequency of i and g is a real-valued function such that g(0) = 0. A natural example P k is to estimate frequency moments Fk = i∈[n] fi ; here we have g(x) = xk , a convex function for P fi k ≥ 1. Another example is the empirical entropy of σ defined as i∈[n] pi log pi where pi = m is 1 the empirical probability of i; here g(x) = x log x. AMS sampling from the famous paper [?] gives an unbiased estimator for g(σ). The estimator is based on a random variable Y defined as follows. Let J be a uniformly random sample from [m]. Let R = |{j | aj = aJ , J ≤ j ≤ m}|. That is, R is the count of the number of tokens after J that are for the same coordinate. Then, let Y the estimate defined as: Y = m(g(R) − g(R − 1)). The lemma below shows that Y is an unbiased estimator of g(σ). Lemma 1 E[Y ] = g(σ) =

X

g(fi ).

i∈[n]

Proof: The probability that aJ = i is exactly fi /m since J is a uniform sample. Moreover if aJ = i then R is distributed as a uniform random variable over [fi ]. X E[Y ] = Pr[aJ = i]E[Y |aJ = i] i∈[n]

=

X fi E[Y |aJ = i] m

i∈[n]

fi X fi X 1 m (g(`) − g(` − 1)) m fi `=1 i∈[n] X = g(fi ).

=

i∈[n]

2 One can estimate Y using small space in the streaming setting via the reservoir sampling idea for generating a uniform sample. The algorithm is given below; the count R gets reset whenever a new sample is picked. 1

In the context of entropy, by convention, x log x = 0 for x = 0.

1

AMSEstimate: s ← null m←0 R←0 While (stream is not done) m←m+1 am is current item Toss a biased coin that is heads with probability 1/m If (coin turns up heads) s ← am R←1 Else If (am == s) R←R+1 endWhile Output m(g(R) − g(R − 1))

To obtain a (, δ)-approximation via the estimator Y we need to estimate Var[Y ] and apply standard tools. We do this for frequency moments now.

1.1

Application to estimating frequency moments

We now apply the AMS sampling to estimate Fk the k’th frequency moment for k ≥ 1. We have already seen that Y is an exact statistication estimator for Fk when we set g(x) = xk . We now estimate the variance of Y in this setting. Lemma 2 When g(x) = xk and k ≥ 1, 1

Var[Y ] ≤ kF1 F2k−1 ≤ kn1− k Fk2 . Proof: Var[Y ] ≤ E[Y 2 ] X

≤

Pr[aJ = i]

fi X m2  `=1

i∈[n]

fi

`k − (` − 1)k

2

fi X fi X m2 k (` − (` − 1)k )(`k − (` − 1)k ) m fi

≤

`=1

i∈[n]

≤ m

fi XX

k`k−1 (`k − (` − 1)k )

(using (xk − (x − 1)k ) ≤ kxk−1 )

i∈[n] `=1

≤ km

X

fik−1 fik

i∈[n]

≤ kmF2k−1 = kF1 F2k−1 . We now use convexity of the function xk for k ≥ 1 to prove the second part. Note that maxi fi = F∞ . X X X X X X k−1 X k−1 F1 F2k−1 = ( fi )( fi2k−1 ) ≤ ( fi )F∞ ( fik ) ≤ ( fi )( fik ) k ( fik ). i

i

i

i

2

i

i

i

P P 1 Using the preceding inequality, and the inequality ( ni=1 xi )/n ≤ (( ni=1 xki )/n) k for all k ≥ 1 (due to the convexity of the function g(x) = xk ), we obtain that X X X X k−1 X k−1 X 1 X F1 F2k−1 ≤ ( fi )( fik ) k ( fik ) ≤ n1−1/k ( fik ) k ( fik ) k ( fik ) ≤ n1−1/k ( fik )2 . i

i

i

i

i

i

i

2 Thus we have E[Y ] = Fk and Var[Y ] ≤ We now apply the trick of reducing the variance and then the median trick to obtain a high-probability bound. If we take h independent estimators for Y and take their average the variance goes down by a factor of h. We let h = c kn1−1/k for some fixed constant c. Let Y 0 be the resulting averaged estimator. We have E[Y 0 ] = 2 2 Fk and Var[Y 0 ] ≤ Var[Y ]/h ≤ c Fk2 . Now, using Chebyshev, we have kn1−1/k Fk2 .

1 Pr[|Y 0 − E[Y 0 ]| ≥ E[Y 0 ]] ≤ Var[Y 0 ]/(2 E[Y 0 ]2 ) ≤ . c We can choose c = 3 to obtain a (, 1/3)-approximation. By using the median trick with Θ(log 1δ ) independent estimators we can obtain a (, δ)-approximation. The overall number of estimators we run independently is O(log 1δ · 12 · n1−1/k ). Each estimator requires O(log n + log m) space since we keep track of one index from [m], one count from [m], and one item from [n]. Thus the space usage to obtain a (, δ)-approximation is O(log 1δ · 12 · n1−1/k · (log m + log n)). The time to process each stream element is also the same. ˜ 1−1/k ) is not optimal for estimating Fk . One can achieve O(n ˜ 1−2/k ) The space complexity of O(n which is optimal for k > 2 and one can in fact achieve poly-logarithmic space for 1 ≤ k ≤ 2. We will see these results later in the course. Bibliographic Notes: See Chapter 1 of the draft book by McGregor and Muthukrishnan; see the application of AMS sampling for estimating the entropy. See Chapter 5 of Amit Chakrabarti for the special case of frequency moments explained in detail. In particular he states a clean lemma that bundles the variance reduction technique and the median trick.

3

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

1

Lecture date: Sept 2, 2014 Scribe: Chandra Chekuri

F2 Estimation

We have seen a generic algorithm for estimating the Fk , the k’th frequency moment of a stream ˜ 1−1/k )-space for k ≥ 1. Now we will see an amazingly simple algorithm for F2 estimation using O(n due to [2]. AMS-F2 -Estimate: H is a 4-universal hash family from [n] to {−1, 1} choose h at random from H z←0 While (stream is not empty) do aj is current item z ← z + h(aj ) endWhile Output z 2

An conceptually equivalent way to describe the algorithm is the following. AMS-F2 -Estimate: Let Y1 , Y2 , . . . , Yn be {−1, +1} random variable that are 4-wise independent z←0 While (stream is not empty) do aj is current item z ← z + Yaj endWhile Output z 2

The difference between the two is that the former one is a streaming friendly. Instead of keeping Y1 , . . . , Yn explicity we sample h from a 4-wise independent hash family so that h can be stored compactly in O(log n)-space and we can generate Yi = h(i) in O(log n) time on the fly. We will analyze the algorithm in the second description. P Let Z = i∈[n] fi Yi be the random variable that represents the value of z at the end of the stream. Note that for all i ∈ [n], E[Yi ] = 0 and E[Yi2 ] = 1. Moreover, since Y1 , . . . , Yn are 4-wiseindependent and hence also 2-wise independent, E[Yi Yi0 ] = 0 for i 6= i0 . The expected value of the output is X X X X fi fi0 E[Yi Yi0 ] = fi2 = F2 . E[Z 2 ] = fi fi0 E[Yi Yi0 ] = fi2 E[Yi2 ] + i,i0 ∈[n]

i6=i0

i∈[n]

We can also compute the variance of the output which is E[Z 4 ]. X X X X E[Z 4 ] = fi fj fk f` E[Yi Yj Yk Y` ]. i∈[n] j∈[n] k∈[n] `∈[n]

1

i∈[n]

Via the 4-wise independence of the Y ’s we have that E[Yi Yj Yk Y` ] = 0 if there is an index among i, j, k, ` that occurs exactly once in the multiset, otherwise it is 1. If it is 1 there are two cases: all indices are the same or there are two distinct indices that occur twice each. Therefore, E[Z 4 ] =

X X X X

fi fj fk f` E[Yi Yj Yk Y` ] =

i∈[n] j∈[n] k∈[n] `∈[n]

X i∈[n]

fi4 + 6

n X n X

fi2 fj2 .

i=1 j=i+1

Thus, we have Var[Z 2 ] = E[Z 4 ] − (E[Z 2 ])2 n X n X 2 = F4 − F2 + 6 fi2 fj2 = F4 − (F4 + 2

i=1 j=i+1 n X n X

fi2 fj2 ) + 6

i=1 j=i+1

= 4 ≤

n X

n X

n X n X

fi2 fj2

i=1 j=i+1

fi2 fj2

i=1 j=i+1 2F22 .

Let X = Z 2 be the output estimate. We have E[X] = F2 and Var[X] ≤ 2F22 ≤ 2E[X]2 . We now apply the standard idea of averaging O(1/2 ) estimates to reduce variance, apply Chebyshev on the average estimator to see that it is a -approximation with > 1/2 probability. Then we apply the median trick with log( 1δ )-independent averaged estimators to obtain an (, δ)-approximation. The overall space requirement is O( 12 log 1δ log n) and this is also the time to process each element.

2

(Linear) Sketching and Streaming with Updates

The F2 estimation algorithm is amazingly simple and has the following interesting properties. Suppose σ1 and σ2 are two streams and the algorithm computes z1 and z2 on σ1 and σ2 It is easy to see that the algorithm on σ = σ1 · σ2 (the concatenation of the two streams) computes z1 + z2 . Thus the algorithm retains z as a sketch of the stream σ. Note that the output of the algorithm is not z but some function of z (in the case of F2 estimation it is z 2 ). Moreover, in this special case the sketch is a linear sketch which we will define more formally later. Formally a sketch of a stream σ is a data structure z(σ) that has the property that if σ = σ · σ2 , z(σ) can be computed by combining the sketches z(σ1 ) and z(σ2 ). Ideally the combining algorithm should take small space as well. Note that the algorithm can post-process the sketch to output the estimator. The power of sketching algorithms is illustrated by thinking of more general streaming models than what we have seen so far. We have considered streams of the form a1 , a2 , . . . , am where each ai is a token, in particular an integer from [n]. Now we will consider the following model. We start with a n-dimensional vector/signal x = (0, 0, . . . , 0) and the stream tokens consists of updates to coordinates of x. Thus each token at = (it , ∆t ) where it ∈ [n] and ∆t is a number (could be a real number and be negative). The token at udpates the i’th coordinate of x: xit ← xit + ∆t . 2

We will let xt be the value of x after the updates corresponding to a1 , a2 , . . . , at . If the ∆t ’s are allowed to be negative the model is called turnstile streams; note that ∆t being negative allows items to be deleted. If xt is required to be always non-negative, we have the strict turnstile stream model. A further special case is when ∆t is required to be positive and is called the cash register model. Linear sketches are particularly simple and yet very powerful. A linear sketch corresponds to a k × n projection matrix M and the sketch for vector x is simply M x. Composing linear sketches corresponds to simply addding the sketches since M x + M x0 = M (x + x0 ). In the streaming setting when we see a token at = (it , ∆t ), updating the sketch corresponds to adding ∆t M eit to the sketch where eit is the vector with 1 in row it and 0 every where else. To implement the algorithm in small space it would suffice to be able to generate the i’th column of M efficienty on the fly rather than storing the entire matrix M . F2 estimation as linear sketching: It is not hard to see that the F2 estimation algorithm we have seen is essentially a linear sketch algorithm. Consider the matrix M with k = O( 12 log 1δ ) rows where each entry is in {−1, 1}. The sketch is simply z = M x. The algorithm post-processes the sketch to output its estimator. Note that because the sketch is linear it does not matter whether x is negative. In fact it is easy to see this from the analysis as well. In particular this implies that we can estimate ||fσ − fσ0 ||2 where fσ and fσ0 are the frequency vectors of σ and σ 0 respectively. Similarly, if x, x0 are two ndimensional signals representing a time-series then the `2 norm of their difference can be estimated by making one pass of the signals even when the signals are given via a sequence of updates which can even be interspersed (of course we need to know the identity of the signals from which the updates are coming from).

3

Johnson-Lindenstrauss Lemma and Dimension Reduction in `2

The AMS linear sketch for F2 estimation appears magical. One way to understand this is via the dimensionality reduction for `2 spaces given by the well-known Johnson-Lindenstrauss lemma which has many applications. The JL Lemma can be stated as follows. Theorem 1 (JL Lemma) Let Let v1 , v2 , . . . , vn be any n points/vectors in Rd . For any  ∈ (0, 1/2), there is linear map f : Rd → Rk where k ≤ 8 ln n/2 such that for all 1 ≤ i < j ≤ n, (1 − )||vi − vj ||2 ≤ ||f (vi ) − f (vj )||2 ≤ ||vi − vj ||2 . Moreover f can be obtained in randomized polynomial-time. The implication of the JL Lemma is that any n points in d-dimensional Euclidean space can be projected to O(ln n/2 )-dimensions while preserving all their pairwise Euclidean distances. The simple randomized algorithm that proves the JL Lemma is the following. Let M be a k × d matrix where each entry Mij is picked independently from the standard N (0, 1) normal distribution. Then the map f is givens as f (v) = √1k M v. We now sketch why this works. Lemma 2 Let Z1 , Z2 , . . . , Zk be independent N (0, 1) random variables and let Y = for  ∈ (0, 1/2), there is a constant c such that, 2

Pr[(1 − )2 k ≤ Y ≤ (1 + )2 k] ≤ 2ec k . 3

2 i Zi .

P

Then,

In other words the sum of squares of k standard normal variables is sharply concentrated around its mean which is k. In fact the distribution of Y has a name, the χ2 distribution with parameter k. We will not prove the preceding lemma. A proof via the standard Chernoff-type argument can be found in various places. Assuming the lemma we can prove the following. Lemma 3 Let in(0, 1/2) and v be a unit-vector in Rd , then (1 − ) ≤ ||M v||2 ≤ (1 + )|| with 2 probability at least (1 − 2ec k ). Proof: First we observe a well-known fact about normal distributions. Let X and Y be independent √ 2 2 N (0, 1) random √ √ variables. Then aX + bY is N (0, a + b ) random variable. Pn vj kMij . Since Let u = kM v. Note that u is a random vector. Note that ui = j=1 √ each kMij is N (0, 1) randomPvariable and all entries are independent we have that ui ' N (0, 1) because the variance of ui is j vi2 = 1 (note that v is a unit vector). Thus u1 , u2 , . . . , uk are P independent N (0, 1) random variables. Therefore ||u||22 = i =k u2i . Applying Lemma 2, we have 2

Pr[(1 − )2 k ≤ ||u||22 ≤ (1 + )2 k] ≥ 1 − 2ec k . 2 Unit-vectors are convenient for the proof but by scaling one obtains the following easy corollary. Corollary 4 Let in(0, 1/2) and v be any vector in Rd , then (1 − )||v||2 ≤ ||M v||2 ≤ (1 + )||||v||2 2 with probability at least (1 − 2ec k ). Now the JL Lemma follows easisly via a union bound. Let k = c0 ln n/2 where c0 is chosen based on c. Consider any pair vi , vj . 2

Pr[(1−)||vi −vj ||2 ≤ ||M (vi −vj ||2 ≤ (1+)||vi −vj ||2 ] ≥ (1−2ec k ) ≥ 1−2ec

2 ·c0

ln n/2

≥ 1−

2 . ncc0

If cc0 ≥ 3 then the probability of the distance between vi and vj being preserved to within a relative -approximation is at least 1 − 1/n3 . Since there are only n(n − 1)/2 pairs of distances, the probability that all of them will be preserved to this error tolerance is, via the union bound, at least (1 − 1/n). Bibliographic Notes: See Chapter 6 of Amit Chakrabarti’s lecture notes. A lot of work has been done in the algorithmic community to make the dimensionality reduction faster to evaluate. An interesting result is due to Achlioptas [1] who showed that the matrix M whose entries we chose from N (0, 1) can in fact be chosen from {−1, 0, 1}; the discrete entries create a sparser matrix and the resulting matrix multiplication is computationally easier.

References [1] Dimitris Achlioptas. Database-friendly random projections: Johnson-lindenstrauss with binary coins. Journal of computer and System Sciences, 66(4):671–687, 2003. [2] Noga Alon, Yossi Matias, and Mario Szegedy. The space complexity of approximating the frequency moments. Journal of Computer and System Sciences, 58(1):137–147, 1999. Preliminary version in Proc. of ACM STOC 1996.

4

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

1

Lecture date: Sept 9, 2014 Scribe: Chandra Chekuri

Sketch for Fp Estimation when 0 < p ≤ 2

We have seen a linear sketching estimate for F2 estimation that uses O(log n) space. Indyk [1] obtained a technically sophisticated and interesting sketch for Fk estimation where 0 < p ≤ 2 (note that p can be a real number) which uses polylog(n) space. Since the details are rather technical we will only give the high-level approach and refer the reader to the paper and related notes for more details. Note that for p > 2 there is a lower bound of Ω(n1−2/p ) on the space required. To describe the sketch for 0 < p ≤ 2 we will revisit the F2 estimate via the JL Lemma approach that uses properties of the normal distribution. F2 -Estimate: Let Y1 , Y2 , . . . , Yn be sampled independenty from the N (0, 1) distribution z←0 While (stream is not empty) do (ij , ∆j ) is current token z ← z + ∆j · Yij endWhile Output z 2

P Let Z = i∈[n] xi Yi be the random variable that represents the value of z at the end of the stream. The variable Z is a sum of independent normal variables and by the properties of the normal qP 2 distribution Z ∼ i xi · N (0, 1). Normal distribution is called 2-stable for this reason. More generally a distribution D is said to be p-stable if the following property holds: Let Z1 , Z2 , . . . , Zn be P independent random variables distributed according to D. Then i xi Zi has the same distribution as kxkp Z where Z ∼ D. Note that a p-stable distribution will be symmetric around 0. It is known that p-stable distributions exist for all p ∈ (0, 2] and not for any p > 2. The p-stable distributions do not have, in general, an analytical formula except in some cases. We have already seen that the standard normal distribution is 2-stable. The 1-stable distribution is the Cauchy distribution which is the distribution of the ratio of two independent standard normal random 1 variables. The density function of the Cauchy distribution is known to be π(1+x 2 ) ; note that the Cauchy distribution does not have a finite mean or variance. We use Dp to denote a p-stable distribution. Although a general p-stable distribution does not have an analytical formula it is known that one can sample from Dp . Chambers-Mallows-Stuck method is the following: • Sample θ uniformly from [−π/2, π/2]. • Sample r uniformly from [0, 1]. • Ouput sin(pθ) (cos θ)1/p



cos((1 − p)θ) ln(1/r) 1

(1−p)/p .

We need one more definition. Definition 1 The median of a distribution D Ris θ if for Y ∼ D, Pr[Y ≤ µ] = 1/2. If φ(x) is the µ probability density function of D then we have −∞ φ(x)dx = 1/2. Note that a median may not be uniquely defined for a distribution. The distribution Dp has a unique median and so we will use the terminology median(Dp ) to denote this quantity. For a distribution D we will refer to |D| the distribution of the absolute value of a random variable drawn from D. If φ(x) is the density function of D then the density function of |D| is given by ψ, where ψ(x) = 2φ(x) if x ≥ 0 and ψ(x) = 0 if x < 0. Fp -Estimate: k ← Θ( 12 log 1δ ) Let M be a k × n matrix where each Mij ∼ Dp y ← Mx 1 |,|y2 |,...,|yk |) Output Y ← median(|y median(|Dp |)

By the p-stability property we see that each yi ∼ kxkp Y where Y ∼ Dp . First, consider the case that k = 1. Then the output |y1 |/median(|Dp |) is distributed according to c|Dp | where c = kxkp /median(|Dp |). It is not hard to verify that the median of this distribution is kxkp . Thus, the algorithm take k samples from this distribution and ouputs as the estimator the sample median. The lemma below shows that the sample median has good concentration properties. Lemma 1 Let  > 0 and let D be a distribution with density function φ and a unique median µ > 0. Suppose φ is absolutely continuous on [(1 − )µ, (1 + )µ] and let α = min{φ(x) | x ∈ [(1 − )µ, (1 + )µ]. Let Y = median(Y1 , Y2 , . . . , Yk ) where Y1 , . . . , Yk are independent samples from the distribution D. Then 2 2 2 2 Pr[|Y − µ| ≥ µ] ≤ 2e− 3  µ α k . We sketch the proof to upper bound Pr[Y ≤ (1 − )µ]. The other direction is similar. Note that by the definition of the median, Pr[Yj ≤ µ] = 1/2. Hence Z µ Pr[Yj ≤ (1 − )µ] = 1/2 − φ(x)dx. (1−)µ

Rµ

Let γ = (1−)µ φ(x)dx. It is easy to see that γ ≥ αµ. Let IP j be the indicator event for Yi ≤ (1 − )µ; we have E[Ij ] = Pr[Yi ≤ (1 − )µ] ≤ 1/2 − γ. Let I = j Ij ; we have E[I] = k(1/2 − γ). Since Y is the median of Y1 , Y2 , . . . , Yk , Y ≤ (1 − )µ 1 . only if more than k/2 of Ij are true which is the same as Pr[I > (1 + δ)E[I]] where 1 + δ = 1−2γ 2

Now, via Chernoff bounds, this probability is at most e−γ k/3 for sufficiently small γ. We can now apply the lemma to the estimator output by the algorithm. We let φ be the distribution of c|Dp |. Recall that the median of this distribution if kxkp and the output of the algorithm is the median of k indepenent samples from this distribution. Thus, from the lemma, 2 kµ2 α2 /3

Pr[|Y − kxkp | ≥ kxkp ] ≤ 2e−

2

.

Let φ0 be the distribution of |D√ | and µ0 be the median of φ0 . Then it can be seen that µα = µ0 α0 where α0 = min{φ0 (x) | (1 − )µ0 ≤ (1 + )µ0 }. Thus µ0 α0 depends only on Dp and . Letting this be cp, we have, 2 2 Pr[|Y − kxkp | ≥ kxkp ] ≤ 2e− kcp, /3 ≤ (1 − δ), provided k = Ω(cp, ·

1 2

log 1δ ).

Technical Issues: There are several technical issues that need to be addressed to obtain a proper algorithm from the preceding description. First, the algorithm as described requires one to store the entire matrix M which is too large for streaming applications. Second, the constant k depends on cp, which is not explicitly known since Dp is not well-understood for general p. To obtain a streaming algorithm, the very high-level idea is to derandomize the algorithm via the use of pseudorandom generators for small-space due to Nisan. See [1] for more details.

2

Counting Frequent Items

We have seen various algorithm for estimating various Fp norms for p ≥ 0. Note that F0 corresponds to number of distinct elements. In the limit, as p → ∞, `p norm of a vector x is the maximum of the absolute values of the entries of x. Thus, we can define the F∞ norm to corresponds to finding the maximum frequency in x. More generally, we would like to find the frequent items in a stream which are also called “heavy hitters”. In general, it is not feasible to estimate the heaviest frequency with limited space if it is too small relative to m.

2.1

Misra-Greis algorithm for frequent items

Suppose we have a stream σ = a1 , a2 , . . . , am where aj ∈ [n], the simple setting and we want to find all elements in [n] such that fi > m/k. Note that there can be at most k such elements. The simplest case is when k = 2 when we want to know whether there is a “majority” element. There is a simple deterministic algorithm that perhaps you have all seen for k = 2 in an algorithm class. The algorithm uses an associative array data structure of size k. MisraGreis(k): D is an empty associative array While (stream is not empty) do aj is current item If (aj is in keys(D)) D[aj ] ← D[aj ] + 1 Else if (|keys(A)| < k − 1) then D[aj ] ← 1 Else for each ` ∈ keys(D) do D[`] ← D[`] − 1 Remove elements from D whose counter values are 0 endWhile For each i ∈ keys(D) set fˆi = D[i] For each i 6∈ keys(D) set fˆi = 0

3

We leave the following as an exercise to the reader. Lemma 2 For each i ∈ [n]: fi −

m ≤ fˆi ≤ fi . k

The lemma implies that if fi > m/k then i ∈ keys(D) at the end of the algorithm. Thus one can use a second-pass over the data to compute the exact fi only for the k itmes in keys(D). This gives an O(kn) time two-pass algorithm for finding all items which have frequency at least m/k. Bibliographic Notes: For more details on Fp estimation when 0 < p ≤ 2 see the original paper of Indyk [1], notes of Amit Chakrabarti (Chapter 7) and Lecture 4 of Jelani Nelson’s course.

References [1] Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. Journal of the ACM (JACM), 53(3):307–323, 2006.

4

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

Lecture date: Sept 11, 2014 Scribe: Chandra Chekuri

The Misra-Greis deterministic counting guarantees that all items with frequency > F1 /k can be found using O(k) counters and an update time of O(log k). Setting k = 1/ one can view the algorithm as providing an additive F1 approximation for each fi . However, the algorithm does not provide a sketch. One advantage of linear sketching algorithms is the ability to handle deletions. We now discuss two sketching algorithms that have a found a number of applications. These sketches can be used to for estimating point queries: after seeing a stream σ over items in [n] we would like to estimate fi the frequency of i ∈ [n]. More generally, in the turnstile model, we would like to estimate xi for a given i ∈ [n]. We can only guarantee the estimate with an additive error.

1

CountMin Sketch

We firt describe the simpler CountMin sketch. The sketch maintains several counters. The counters are best visualized as a rectangular array of width w and depth d. With each row i we have a hash function hi : [n] → [w] that maps elements to one of w buckets. CountMin-Sketch(w, d): h1 , h2 , . . . , hd are pair-wise independent hash functions from [n] → [w]. While (stream is not empty) do at = (it , ∆t ) is current item for ` = 1 to d do C[`, h` (ij )] ← C[`, h` (ij )] + ∆t endWhile For i ∈ [n] set x ˜i = mind`=1 C[`, h` (i)].

The counter C[`, j] simply counts the sum of all xi such that h` (i) = j. That is, X C[`, j] = xi . i:h` (i)=j

Exercise: CountMin is a linear sketch. What are the entries of the projection matrix? We will analyze the sketch in the strick turnstile model where xi ≥ 0 for all i ∈ [n]; note that ∆t we be negative. ˜i and Lemma 1 Let d = Ω(log 1δ ) and w > 2 . Then for any fixed i ∈ [n], xi ≤ x Pr[˜ xi ≥ xi + kxk1 ] ≤ δ. Proof: Fix i ∈ [n]. Let Z` = C[`, h` (i)] be the value of the counter in row to. We have X X1 x i0 ≤ x i + E[Z` ] = xi + Pr[h` (i0 ) = h` (i)]xi0 = xi + w 0 0 i 6=i

i 6=i

` to which i is hashed  kxk1 . 2

Note that we used pair-wise independence of h` to conclude that Pr[h` (i0 ) = h` (i)] = 1/w. 1

By Markov’s inequality (here we are using non-negativity of x), Pr[Z` > xi + kxk1 ] ≤ 1/2. Thus Pr[min Z` > xi + kxk1 ] ≤ 1/2d ≤ δ. `

2 Remark: By choosing δ = Ω(log n) we can ensure with probability at least (1 − 1/poly(n)) that x ˜i − xi ≤ kxk1 for all i ∈ [n]. Exercise: For general turnstile streams where x can have negative entries we can take the median of the counters. For this estimate you should be able to prove the following. Pr[|˜ xi − xi | ≥ 3kxk1 ] ≤ δ 1/4 .

2

Count Sketch

Now we discuss the closely related Count sketch which also maintains an array of counters parameterized by the width w and depth d. Count-Sketch(w, d): h1 , h2 , . . . , hd are pair-wise independent hash functions from [n] → [w]. g1 , g2 , . . . , gd are pair-wise independent hash functions from [n] → {−1, 1}. While (stream is not empty) do at = (it , ∆t ) is current item for ` = 1 to d do C[`, h` (ij )] ← C[`, h` (ij )] + g(it )∆t endWhile For i ∈ [n] set x ˜i = median{g1 (i)C[1, h1 (i)], g2 (i)C[2, h2 (i), . . . , gd (i)C[d, hd (i)]}.

Exercise: CountMin is a linear sketch. What are the entries of the projection matrix? Lemma 2 Let d ≥ log 1δ and w >

3 . 2

Then for any fixed i ∈ [n], E[˜ xi ] = xi and

Pr[|˜ xi − xi | ≥ kxk2 ] ≤ δ. Proof: Fix an i ∈ [n]. Let Z` = g` (i)C[`, h` (i)]. For i0 ∈ [n] let Yi0 be the indicator random variable that is 1 if h` (i) = h` (i0 ); that is i and i0 collide in h` . Note that E[Yi0 ] = E[Yi20 ] = 1/w from the pairwise independence of h` . We have X Z` = g` (i)C[`, h` (i)] = g` (i) g` (i0 )xi0 Yi0 i0

Therefore, E[Z` ] = xi +

X

E[g` (i)g` (i0 )Yi0 ]xi0 = xi ,

i0 6=i

2

because E[g` (i)g` (i0 )] = 0 for i 6= i0 from pairwise independence of g` and Yi0 is independent of g` (i) and g` (i0 ). Now we upper bound the variance of Z` .   X Var[Z` ] = E ( g` (i)g` (i0 )Yi0 xi0 )2  i0 6=i

  X X = E x2i0 Yi20 + xi0 xi00 g` (i0 )g` (i00 )Yi0 Yi00  i0 6=i

=

X

i0 6=i00

x2i0 E[Yi20 ]

i0 6=i

≤ kxk22 /w. Using Chebyshev, Pr[|Z` − xi | ≥ kxk2 ] ≤

1 Var[Z` ] ≤ 2 ≤ 1/3. 2 2  w  kxk2

Now, via the Chernoff bound, Pr [|median{Z1 , . . . , Zd } − xi | ≥ kxk2 ] ≤ e−cd ≤ δ. Thus choosind d = O(log n) and taking the median guarantees the desired bound with high probability. 2 Remark: By choosing δ = Ω(log n) we can ensure with probability at least (1 − 1/poly(n)) that |˜ xi − xi | ≤ kxk2 for all i ∈ [n].

3

Applications

Count and CountMin sketches have found a number of applications. Note that they have a similar structure though the guarantees are different. Consider the problem of estimating frequency moments. Count sketch outputs an estimate f˜i for fi with an additive error of kf k2 while CountMin guarantees an additive error of kf k1 which is always larger. CountMin provides a one-sided error when x ≥ 0 which has some benefits. CountMin uses O( 1 log 1δ ) counters while Count sketch uses O( 12 log 1δ ) counters. Note that the Misra-Greis algorithm uses O(1/)-counters.

3.1

Heavy Hitters

We will call an index i an α-HH (for heavy hitter) if xi ≥ αkxk1 where α ∈ (0, 1]. We would like to find Sα , the set of all α-heavy hitters. We will relax this assumption to output S such that Sα ⊆ S ⊆ Sα− . Here we will assume that α < α for otherwise the approximation does not make sense. Suppose we used CountMin sketch with w = 2/ and δ = c/n for sufficiently large c. Then, as we saw, with probability at least (1 − 1/poly(n)), for all i ∈ [n], xi ≤ x ˜i ≤ xi + kxk1 . 3

Once the sketch is computed we can simply go over all i and add i to S if x ˜i ≥ αkxk1 . It is easy to see that S is the desired set. Unfortunately the computation of S is expensive. The sketch has O( 1 log n) counters and processing each i takes time proportional to the number of counters and hence the total time is O( 1 n log n) to output a set S of size O( α1 ). It turns that by keeping additional information in the sketch in a hierarchical fashion one can cut down the time to be proportional to O( α1 polylog(n))).

3.2

Range Queries

In several application the range [n] corresponds to an actual total ordering of the items. For instance [n] could represent the discretization of time and x corresponds to the signal. In databases [n] could represent ordered numerical attributes such as age of a person, height, or salary. In such settings range queries are very useful.PA range query is an interval of the form [i, j] where i, j ∈ [n] and i ≤ j. The goal is to output i≤`≤j xi . Note that there are O(n2 ) potential queries. There is a simple trick to solve this using the sketches we have seen. An interval [i, j] is a dyadic interval/range if j − i + 1 is 2k and 2k divides i − 1. Assume n is a power of 2. Then the dyadic intervals of length 1 are [1, 1], [2, 2], . . . , [n, n]. Those of length 2 are [1, 2], [3, 4], . . . and of length 4 are [1, 4], [5, 8], . . .. Claim 3 Every range [i, j] can be expressed as a disjoint union of at most 2 log n dyadic ranges. Thus it suffices to maintain accurate point queries for the dyadic ranges. Note that there are at most 2n dyadic ranges. They fall into O(log n) groups based on length; the ranges for a given length partition the entire interval. We can keep a separate CountMin sketch for the n/2i dyadic intervals of length i (i = 0 corresponds to the sketch for point queries). Using these O(log n) CountMin sketches we can answer any range query with an additive error of kxk1 . Note that a range [i, j] is expressed as the sum of 2 log n point queries each of which has an additive error. So 0 for the sketches has to be chosen to be /(2 log n) to ensure an additive error of kxk1 for the range queries. By choosing d = O(log n) the error probability for all point queries in all sketches will be at most 1/poly(n). This will guarantee that all range queries will be answered to within an additive kxk1 . The total space will be O( 1 log3 n)

3.3

Sparse Recovery

Let x ∈ Rn be a vector. Can we approximate x by a sparse vector z? By sparse we mean that z has at most k non-zero entries for some given k (this is the same as saying kzk0 ≤ k). A reasonable way to model this is to ask for computing the error errkp (x) =

min kx − zkp

z:kzk0 ≤k

for some p. A typical choice is p = 2. It is easy to see that the optimum z is obtained by restricting x to its k largest coordinates (in absolute value). The question we ask here is whether we can estimate errk2 (x) efficiently in a streaming fashion. For this we use the Count sketch. Recall that by choosing w = 3/2 and d = Θ(log n) the sketch ensures that with high probability, ∀i ∈ [n],

|˜ xi − xi | ≤ kxk2 .

One can in fact show a generalization. 4

Lemma 4 Count-Sketch with w = 3k/ and d = O(log n) ensures that ∀i ∈ [n],

 |˜ xi − xi | ≤ √ errk2 (x). k

Proof: Let S = {i1 , i2 , . . . , ik } be the indices of the largest coordinates in x and let x0 be obtained from x by setting entries of x to zero for indices in S. Note that errk2 (x) = kx0 k2 . Fix a coordinate i. Consider row ` and let Z` = g` (i)C[`, h` (i)] as before. Let A` be the event that there exists an index t ∈ S such P that h` (i) = h` (t); that is any “big” coordinate collides with i under h` . Note that Pr[A` ] ≤ t∈S Pr[h` (i) = Pr[h` (t)] ≤ |S|/w ≤ /3 by pair-wise independence of h. Now we estimate   Pr[|Z` − xi | ≥ √ errk2 (x)] = Pr[|Z` − xi | ≥ √ kx0 k2 ] k k   = Pr[A` ] · Pr[|Z` − xi | ≥ √ kx0 k2 ] + Pr[|Z` − xi | ≥ √ kx0 k2 | ¬A` ] k k ≤ Pr[A` ] + 1/3 < 1/2. 2 ˜ be the approximation to x that is obtained from the sketch. We can take the k largest Now let x ˜ to form the vector z and output z. We claim that this gives a good approximation coordinates of x k to err2 (x). To see this we prove the following lemma. Lemma 5 Let x, y ∈ Rn such that  kx − yk∞ ≤ √ errk2 (x). k Then, kx − zk2 ≤ (1 + 5)errk2 (x), where z is the vector obtained as follows: zi = yi for i ∈ T where T is the set of k largest (in absolute value) indices of y and zi = 0 for i 6∈ T . Proof: Let t = of x. We have,

√1 errk (x) 2 k

to help ease the notation. Let S be the index set of the largest coordinates

(errk2 (x))2 = kt2 =

X

X

x2i =

i∈T \S

i∈[n]\S

X

x2i +

x2i .

i∈[n]\(S∪T )

We write: kx − zk22 =

X i∈T

i∈S\T

=

X

X

i∈T

X

|xi − zi |2 + |xi − yi |2 +

i∈S\T

i∈[n]\(S∪T )

x2i +

X i∈[n]\(S∪T )

We treat each term separately. The first one is easy to bound. X X |xi − yi |2 ≤ 2 t2 ≤ 2 kt2 . i∈T

i∈T

5

X

|xi − zi |2 +

x2i .

x2i

The third term is common to kx − zk2 and errk2 (x). The second term is the one to care about. Note that S is set of k largest coordinates in x and T is set of k largest coordinates in y. Thus |S \ T | = |T \ S|, say their cardinality is ` ≥ 1. Since x and y are close in `∞ norm (that is they are close in each coordinate) it must mean that the coordinates in S \ T and T \ S are roughly the same value in x. More precisely let a = maxi∈S\T |xi | and b = mini∈T \S |xi |. We leave it as an exercise to the reader to argue that that a ≤ b + 2t since kx − yk∞ ≤ t. Thus, X x2i ≤ `a2 ≤ `(b + 2t)2 ≤ `b2 + 4ktb + 4k2 t2 . i∈S\T

But we have X

x2i ≥ `b2 .

i∈T \S

Putting things together, kx − zk22 ≤ `b2 + 4ktb +

X

x2i + 5k2 t2

i∈[n]\(S∪T )

≤

X i∈T \S

x2i +

X

x2i + 4(errk2 (x))2 + 52 (errk2 (x))2

i∈[n]\(S∪T )

≤ (errk2 (x))2 + 9(errk2 (x))2 . The lemma follows by by the fact that for sufficiently small ,

√

1 + 9 ≤ 1 + 5. 2

Bibliographic Notes: Count sketch is by Charikar, Chen and Farach-Colton [1]. CountMin sketch is due to Cormode and Muthukrishnan [4]; see the papers for several applications. Cormode’s survey on sketching in [2] has a nice perspective. See [3] for a comparative analysis (theoretical and experimenta) of algorithms for frinding frequent items. A deterministic variant of CountMin called CR-Precis is interesting; see http://polylogblog.wordpress.com/2009/09/ 22/bite-sized-streams-cr-precis/ for a blog post with pointers and some comments. The applications are taken from the first chapter in the draft book by McGregor and Muthukrishnan.

References [1] Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. Theoretical Computer Science, 312(1):3–15, 2004. [2] Graham Cormode, Minos N. Garofalakis, Peter J. Haas, and Chris Jermaine. Synopses for massive data: Samples, histograms, wavelets, sketches. Foundations and Trends in Databases, 4(1-3):1–294, 2012. [3] Graham Cormode and Marios Hadjieleftheriou. Methods for finding frequent items in data streams. VLDB J., 19(1):3–20, 2010. [4] Graham Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. J. Algorithms, 55(1):58–75, 2005.

6

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

1

Lecture date: Sept 23, 2014 Scribe: Chandra Chekuri

Priority Sampling and Sum Queries

Suppose we have a stream a1 , a2 , . . . , an (yes, we are changing notation here from m to n for the length of the stream) of objects and each ai has a non-negative weight wi . We want to store a representative sample S ⊂ [n] of the items soPthat we can answer subset sum queries. That is, given a query I ⊆ [n] we would like to answer i∈I wi . One way to do this as follows. Suppose we pick S as follows: sample each i ∈ [n] independently with probability pi and if i is chosen P we set a scaled weight w ˆi = wi /pi . Now, given a query I we output the estimate for its weight as i∈I∩S w ˆi . Note that expectation of the estimate is equal to w(I). The disadvantage of this scheme is mainly related to the fact that we cannot control the size of sample. This means that we cannot fully utilize the memory available. Related to the first, is that if we do not know the length of stream apriori, we cannot figure out the sampling rate even if we are willing to be flexible in the size of the memory. An elegant scheme of Duffield, Lund and Thorup, called priority sampling overcomes these limitations. They considered the setting where we are given a parameter k for the size of the sample S and the goal is maintain a k-sample S along with some weights w ˆi for i ∈ S such that we can answer subset sum queries. Their scheme is the following, described as if a1 , a2 , . . . , an are available offline. 1. For each i ∈ [n] set priority qi = wi /ui where ui is chosen uniformly (and independently from other items) at random from [0, 1]. 2. S is the set of items with the k highest priorities. 3. τ is the (k + 1)’st highest priority. If k ≥ n we set τ = 0. 4. If i ∈ S, set w ˆi = max{wi , τ }, else set w ˆi = 0. We observe that the above sampling can be implemented in the streaming setting by simply keeping the current sample S and current threshold τ . We leave it as an exercise to show that this informatino can be updated when a new item arrives. We show some nice and non-obvious properties of priority sampling. We will assume for simplicity that 1 < k < n. The first one is the basic one that we would want. Lemma 1 E[w ˆ i ] = wi . Proof: Fix i. Let A(τ 0 ) be the event that the k’th highest priority among items j 6= i is τ 0 . Note that i ∈ S if qi = wi /ui ≥ τ 0 and if i ∈ S then w ˆi = max{wi , τ 0 }, otherwise w ˆi = 0. To evaluate 0 Pr[i ∈ S | A(τ )] we consider two cases. Case 1: wi ≥ τ 0 . Here we have Pr[i ∈ S | A(τ 0 )] = 1 and w ˆ i = wi . Case 2: wi < τ 0 . Then Pr[i ∈ S | A(τ 0 )] = wτ 0i and wˆi = τ 0 . In both cases we see that E[w ˆ i ] = wi . 2 The previous claim shows that the estimator

P

I∩S

estimate the variance of wˆi via the threshold τ . 1

w ˆi has expectation equal to w(I). We can also



τ max{0, τ − wi } if i ∈ S 0 if i 6∈ S 0 0 Proof: Fix i. We define A(τ ) to be the event that τ is the k’th highest priority among elements j 6= i. The proof is based on showing that Lemma 2 Var[w ˆi ] = E[ˆ vi ] where vˆi =

E[ˆ vi | A(τ 0 )] = E[w ˆi2 | A(τ 0 )] − wi2 . From the proof outline of the preceding lemma, we estimate the lhs of as E[ˆ vi | A(τ 0 )] = Pr[i ∈ S | A(τ 0 )] × E[ˆ vi | i ∈ S ∧ A(τ 0 )] = min{1, wi /τ 0 } × τ 0 max{0, τ 0 − wi } = max{0, wi τ 0 − wi2 }. Now we analyze the rhs, E[w ˆi2 | A(τ 0 )] = Pr[i ∈ S | A(τ 0 )] × E[w ˆi2 | i ∈ S ∧ A(τ 0 )] = min{1, wi /τ 0 } × (max{wi , τ 0 })2 = max{wi2 , wi τ 0 }. 2 Surprisingly, if k ≥ 2 then the covariance between wˆi and w ˆj for any i 6= j is equal to 0. Lemma 3 E[w ˆi w ˆj ] = 0. In fact the previous lemma is a special case of a more general lemma below. Q Q ˆi ] = ki=1 wi if |I| ≤ k and is 0 if |I| > k. Lemma 4 E[ i∈I w Proof: It is easy to see that if |I| > k the product is 0 since at least one of them is not in the sample. We now assume |I| ≤ k and prove the desired claim by inducion on |I|. In fact we need a stronger hypothesis. Let τ 00 be theQ (k − |I| + 1)’th highest Q priority among the items j 6= i. We will condition on τ 00 and prove that E[ i∈I w ˆi | A(τ 00 )] = i∈I wi . For the base case we have seen the proof for |I| = 1. Case 1: There is h ∈ I such that wh > τ 00 . Then clearly h ∈ S and w ˆh = wh . In this case Y Y E[ w ˆi | A(τ 00 )] = wh · E[ w ˆi | A(τ 00 )], i∈I

i∈I\{h}

Q ˆi | A(τ 00 )] is referring to the and we apply induction to I \ {h}. Technically the term E[ i∈I\{h} w fact that τ 00 is the k − |I 0 | + 1’st highest priority where I 0 = I \ {h}. Case 2: For all h ∈ I, wh < τ 00 . Let q be the minimum priority among items in I. If q < τ 00 then w ˆj = 0 for some j ∈ I and the entire product is 0. Thus, in this case, there is no contribution to Thus we will consider the case when q ≥ τ 00 . The probability for this event is Q thewexpectation. i But in thisQcase all i ∈ I will be in S and moreover w ˆi = τ 00 for each i. Thus the expected i∈I τ 00 .Q value of i∈I w ˆi = i∈I wi as desired. 2

P Combining Lemma 2 and 3 the variance of the estimator i∈I∩S w ˆi is X X X Var[ w ˆi = Var[w ˆi ] = E[ˆ vi ]. i∈I∩S

i∈I∩S

i∈I∩S

The advantage of this is that the variance of the estimator can be computed by examining τ and the weights of the elements in the S ∩ I. 2

2

`0 Sampling

We have seen `2 sampling in the streaming setting. The ideas generalize to `p sampling to `p sampling for p ∈ (0, 2) — see [] for instance. However, `0 sampling requires slightly different ideas. `0 sampling means that we are sampling near-uniformly from the distinct elements in the stream. Surprisingly we can do this even in the turnstile setting. Recall that one of the applications we saw for the Count-Sketch is `2 -sparse recovery. In particular we can obtain a (1+)-approximation for errk2 (x) with high-probability using O(k log n/) words. Suppose x is k-sparse then errk2 (x) = 0! It means that we can detect if x is k-sparse, and in fact identify the non-zero coordinates of x, with high-probability. In fact one can prove the following stronger version. 0

Lemma 5 For 1 ≤ k ≤ n there and k 0 = O(k) there is a sketch L : Rn → Rk (generated from O(k log n) random bits) and a recovery procedure that on input L(x) has the following feature: (i) if x is k-sparse then it outputs x0 = x with probability 1 and (ii) if x is not k-sparse the algorithm detects this with high-probability. We will use the above for `0 sampling as follows. We will first describe a high-level algorithm that is not streaming friendly and will indicate later how it can be made stream implementable. 1. For h = 1, . . . , blog nc let Ih be a random subsets of [n] where Ij has cardinality 2j . Let I0 = [n]. 2. Let k = d4 log(1/δ)e. For h = 0, . . . , blog nc, run k-sparse-recovery on x restricted to coordinates of Ih . 3. If any of the sparse-recoveries succeeds then output a random coordinate from the first sparserecovery that succeeds. 4. Algorithm fails if none of the sparse-recoveries output a valid vector. Let J be the index set of non-zero coordinates of x. We now show that the algorithm with probability (1 − δ) succeeds in outputting a uniform sample from J. Suppose |J| ≤ k. Then x is recovered exactly for h = 0 and the algorithm outputs a uniform sample from J. Suppose |J| > k. ∗ We observe that E[|Ih ∩ J|] = 2h |J|/n and hence there is a h∗ such that E[|Ih∗ ∩ J|] = 2h |J|/n is between k/3 and 2k/3. By Chernoff-bounds one can show that with probability at least (1 − δ), 1 ≤ |Ih∗ ∩ J| ≤ k. For this h∗ the sparse recovery will succeed and output a random coordinate of J. The formal claims are the following: • With probability at least (1 − δ) the algorithm outputs a coordinate i ∈ [n]. • If the algorithm outputs a coordinate i then the probability that it is not a uniform random sample is because the sparse recovery algorithm failed for some h; we can make this probability be less than 1/nc for any desired constant c. Thus, in fact we get zero-error `0 sample. The algorithm, as described, requires one to sample and store Ih for h = 0, . . . , blog nc. In order to avoid this we can use Nisan’s pseudo-random generator for small-space computation. We skip the details of this; see [2]. The overall space requirements for the above procedure can be shown

3

to be O(log2 n log(1/δ) with an error probability bounded by δ + O(1/nc ). This is near-optimal for constant δ as shown in [2].

Bibliographic Notes: The material on priority sampling is directly from [1] which describes applications, relationship to prior sampling techniques and also has an experimental evaluation. Priority sampling has shown to be “optimal” in a strong sense; see [3]. The `0 sampling algorithm we described is from the paper by Jowhari, Saglam and Tardos [2]. See a simpler algorithm in the chapter on signals by McGregor-Muthu draft book.

References [1] Nick Duffield, Carsten Lund, and Mikkel Thorup. Priority sampling for estimation of arbitrary subset sums. Journal of the ACM (JACM), 54(6):32, 2007. [2] Hossein Jowhari, Mert Saglam, and G´abor Tardos. Tight bounds for lp samplers, finding duplicates in streams, and related problems. CoRR, abs/1012.4889, 2010. [3] Mario Szegedy. The dlt priority sampling is essentially optimal. In Proceedings of the thirtyeighth annual ACM symposium on Theory of computing, pages 150–158. ACM, 2006.

4

CS 598CSC: Algorithms for Big Data Instructor: Chandra Chekuri

Lecture date: Sept 25, 2014 Scribe: Chandra Chekuri

Suppose we have a stream a1 , a2 , . . . , an of objects from an ordered universe. For simplicity we will assume that they are real numbers and more over that they are distinct (for simplicity). We would like to find the k’th ranked element for some 1 ≤ k ≤ n. In particular we may be interested in the median element. We will discuss exact and approximate versions of these problems. Another terminology for finding rank k elements is quantiles. Given a number φ where 0 < φ ≤ 1 we would like to return an element of rank φn. This normalization allows us to talk about -approximate quantiles for  ∈ [0, 1). An -approximate quantile is an element whose rank is at least (φ − )n and at most (φ + )n. In other words we are allowing an additive error of n. There is a large amount of literature on quantiles and quantile queries. In fact one of the earliest “streaming” papers is the ˜ 1/p ) space one by Munro and Paterson [5] who described a p-pass algorithm for selection using O(n for any p ≥ 2. They also considered the “random-order” streaming setting which has become quite popular in recent years. The material for these lectures is taken mainly from the excellent chapter/survey by Greenwald and Khanna [1]. We mainly refer to that chapter and describe here the outline of what we covered in lectures. We will omit proofs or give sketchy arguments and refer the reader to [1].

1

Approximate Quantiles and Summaries

Suppose we want to be able to answer -approximate quantile queries over an ordered set S of n elements. It is easy to see that we can simply pick elements of rank i|S|/k for 1 ≤ i ≤ k ' 1/ and store them as a summary and use the summary to answer any quantile query with an n additive error. However, to obtain these elements we first need to do selection which we can do in the offline setting if all we want is a concise summary. The question we will address is to find a summary in the streaming setting. In the sequel we will count space usage in terms of “words”. Thus, the space usage for the preceding offline summary is Θ(1/). We will see two algorithms. The first will create an -approximate summary [4] with space O( 1 log2 (n)) and is inspired by the ideas from the work of Munro and Paterson. Greenwald and Khanna [2] gave a different summary that uses O( 1 log(n)) space. Following [1] we will think of a quantile summary Q as storing a set of elements {q1 , q2 , . . . , q` } from S along with an interval [rminQ (qi ), rmaxQ (qi )] for each qi ; rminQ (qi ) is a lower bound on the rank of qi in S and rmaxQ (qi ) is an upper bound on the rank of qi in S. It is convenient to assume that q1 < q2 < . . . < q` and moreover that q1 is the minimum element in S and that q` is the maximum element in S. For ease of notation we will simply use Q to refer to the quantile summary and also the (ordered) set {q1 , q2 , . . . , q` }. Our first question is to ask whether a quantile summary Q can be used to give -approximate quantile queries. The following is intuitive and is worthwhile proving for oneself. Lemma 1 Suppose Q is a quantile summary for S such that for 1 ≤ i < `, rmaxQ (qi+1 ) − rminQ (qi ) ≤ 2|S|. Then Q can be used for -approximate quantile queries over S. In the following, when we say that Q is an -approximate quantile summary we will implicitly be using the condition in the preceding lemma.

1

Given a quantile summary Q0 for a multiset S 0 and a quantile summary Q00 for a multiset S 00 we would like to combine Q0 and Q00 into a quantile summary Q for S = S 0 ∪ S 00 ; by union here we view S as a multiset. Of course we would like to keep the approximation of the resulting summary similar to those of Q0 and Q00 . Here a lemma which shows that indeed we can easily combine. Lemma 2 Let Q0 be an 0 -approximate quantile summary for multiset S 0 and Q00 be an 00 -approximate quantile summary for multiset S 00 . Then Q = Q0 ∪ Q00 yields an -approximate quantile summary 0 0 00 n00 0 00 0 0 00 00 for S = S 0 ∪ S 00 where  ≤  nn0+ +n00 ≤ max{ ,  } where n = |S | and n = |S |. We will not prove the correctness but describe how the intervals are constructed for Q from those for Q0 and Q00 . Suppose Q0 = {x1 , x2 , . . . , xa } and Q00 = {y1 , y2 , . . . , yb }. Let Q = Q0 ∪ Q00 = {z1 , z2 , . . . , za+b }. Consider some zi ∈ Q and suppose zi = xr for some 1 ≤ r ≤ a. Let ys be the largest element of Q00 smaller than xr and yt be the smallest element of Q00 larger than xr . We will ignore the cases where one or both of ys , yt are not defined. We set rminQ (zi ) = rminQ0 (xr ) + rminQ00 (ys ) and rmaxQ (zi ) = rmaxQ0 (xr ) + rmaxQ00 (yt ) − 1. It is easy to justify the above as valid intervals. One can then prove that with these settings, for 1 ≤ i < a + b the following holds: rmaxQ (zi+1 ) − rminQ (zi ) ≤ 2(n0 + n00 ). We will refer to the above operation as COMBINE(Q0 , Q00 ). The following is easy to see. Lemma 3 Let Q1 , . . . , Qh be -approximate quantile summaries for S1 , S2 , . . . , Sh respectively. Then Q1 , Q2 , . . . , Qh can be combined in any arbitrary order to obtain an -approximate quantile summary Q = Q1 ∪ . . . ∪ Qh for S1 ∪ . . . ∪ Sh . Next we discuss the PRUNE(Q, k) operation on a quantile summary Q that reduces the size of Q to k + 1 while losing a bit in the approximation quality. Lemma 4 Let Q0 be an -approximate quantile summary for S. Given an integer parameter k 1 there is a quantile summary Q ⊆ Q0 for S such that |Q| ≤ k + 1 and it is ( + 2k )-approximate. We sketch the proof. We simply query Q0 for ranks 1, |S|/k, 2|S|/k, . . . , |S| and choose these elements to be in Q. We retain their rmin and rmax values from Q0 . rmaxQ (qi+1 ) − rminQ (qi ) ≤ i|S|/k + |S| − ((i − 1)|S|/k − |S|) ≤ |S|/k + 2|S|.

1.1

An O( 1 log2 (n) space algorithm

The idea is inspired by the Munro-Paterson algorithm and was abstracted in the paper by Manku et al. We will describe the idea in an offline fashion though it can be implemented in the streaming setting. We will use several quantile summaries with k elements each for some parameter k, say ` of them. Each summary of size k will be called a buffer. We will need to reuse these buffers as more elements in the stream arrive; buffers will combined and pruned, in other words “collapsed” into a single buffer of the same size k. Pruning introduces error. 2

Assume n/k is a power of 2 for simplicity. Consider a complete binary tree with n/k leaves where each leaf corresponds to k consecutive elements of the stream. Think of assigning a buffer of size k to obtain a 0-error quantile summary for those k elements; technically we need k + 1 elements but we will ignore this minor additive issue for sake of clarity of exposition. Now each internal node of the tree corresponds to a subset of the elements of the stream. Imagine assigning a buffer of size k to each internal node to maintain an approximate quantile summary for the elements of the stream in the sub-tree. To obtain a summary at node v we combine the summaries of its two children v1 and v2 and prune it back to size k introducing and additional 1/(2k) error in the approximation. The quantile summary at the root of size k will be our final summary that we output for the stream. Our first observation is that in fact we can implement the tree-based scheme with ` = O(h) buffers where h is the height of the tree. Note that h ' log(n/k). The reason we only need O(h) buffers is that if need a new buffer for the next k elements in the stream we can collapse two buffers corresponding to the children of an internal node — hence, at any time we need to maintain only one buffer per level of the tree (plus a temporary buffer to do the collapse operation). Consider the quantile summary at the leaves. They have error 0 since we store all the elements in the buffer. However at each level the error increases by 1/(2k). Hence the error of the summary at the root is h/(2k). Thus, to obtain an -approximate quantile summary we need h/(2k) ≤ . 1 And h = log(n/k). One can see that for this to work out it suffices to choose k > 2 log(2n). 1 The total space usage is Θ(hk) and h = log(n/k) and thus the space usage is O(  log2 (n)). One can choose d-ary trees instead of binary trees and some optimization can be done to improve the constants but the asymptotic dependence on  does not improve with this high-level scheme.

1.2

An O( 1 log(n) space algorithm

We now briefly describe the Greenwald-Khanna algorithm that obtains an improved space bound. The GK algorithm maintains a quantile summary as a collection of s tuples t0 , t2 , . . . , ts−1 where each tuple ti is a triple (vi , gi , ∆i ): (i) a value vi that is an element of the ordered set S (ii) the value gi which is equal to rminGK (vi ) − rminGK (vi−1 ) (for i = 0, gi = 0) and (iii) the value ∆i which equals rmaxGK (vi ) − rminGK (vi ). The elements v0 , v1 , . . . , vs−1 are in ascending order and moreover P v0 will the minimum element in S and vs−1 is the maximum element in S. Note that seen so far. With this set up we n = s−1 j=1 gj . The summary P P also stores n the number of elements note that rminGK (vi ) = j≤i gj and rmaxGK (vi ) = ∆i + j≤i gj . Lemma 5 Suppose Q is a GK quantile summary for a set |S| such that maxi (gi + ∆i ) ≤ 2|S| then it can be used to answer quantile queries with n additive error. The query can be answered as follows. Given rank r, find i such that r − rminGK (vi ) ≤ n and rmaxGK (vi ) − r ≤ n and output vi ; here n is the current size of S. The quantile summary is updated via two operations. When a new element v arrives it is inserted into the summary. The quantile summary is compressed by merging consecutive elements to keep the summary within the desired space bounds. We now describe the INSERT operation that takes a quantile summary Q and inserts a new element v. First we search over the elements in Q to find an i such that vi < v < vi+1 ; the case when v is the new smallest element or the new largest element are handled easily. A new tuple t = (v, 1, ∆) with ∆ = b2nc − 1 is added to the summary where t becomes the new (i + 1)st tuple. Note that here n is the current size of the stream. It is not hard to see if the summary Q before 3

arrival of v satisfied the condition in Lemma 5 then Q satisfies the condition after inserting the tuple (note that n increased by 1). We note that that the first 1/(2) elements are inserted into the summary with ∆i = 0. Compression is the main ingredient. To understand the operation it is helpful to define the notion of a capacity of a tuple. Note that when v arrived t = (v, 1, ∆) is inserted where ∆ ' 2n0 where n’ is the time when v arrived. At time n > n0 the capacity of the tuple ti is defined as 2n − ∆i . As n grows, the capacity of the tuple increases since we are allowed to have more error. 0 We can merge tuples ti0 , ti0 +1 , . . .P , ti into ti+1 at time n (which means we eliminate Pi+1 ti , . . . , ti ) while ensuring the desired precision if i+1 g + ∆ ≤ 2n; g is updated to g and ∆i+1 does i+1 i+1 j=i0 j j=i0 j not change. Note that this means that ∆ of a tuple does not change once it is inserted. Note that the insertion and merging operations preserve correctness of the summary. In order to obtain the desired space bound the merging/compression has to be done rather carefully. We will not go into details but mention that one of the key ideas is to keep track of the capacity of the tuples in geometrically increasing intervals and to ensure that the summary retains only a small number of tuples per interval.

2

Exact Selection

We will now describe a p-pass deterministic algorithm to select the rank k element in a stream using ˜ 1/p )-space; here p ≥ 2. It is not hard to show that for p = 1 any deterministic algorithm needs O(n Ω(n) space; one has to be a bit careful in arguing about bits vs words and the precise model but a ˜ 1/p )-space algorithm using near-linear lower bound is easy. Munro and Paterson described the O(n p passes. We will not describe their precise algorithm but instead use the approximate quantile based analysis. We will show that given space s and a stream of n items the problem can be effectively reduced in one pass to selecting from O(n log2 n/s) items. Suppose we can do the above. Choose s = n1/p (log n)2−2/p . After i passes the problem is p−i

2i

reduced to n p (log n) p elements. Setting i = p − 1 we see that the number of elements left for the p’th pass is O(s). Thus all of them can be stored and selection can be done offline. We now describe how to use one pass to reduce the effective size of the elements under consideration to O(n log2 n/s). The idea is that we will be able to select two elements a1 , b1 from the stream such that a1 < b1 and the k’th ranked element is guaranteed to be in the interval [a1 , b1 ]. Moreover, we are also guaranteed that the number of elements between a and b in the stream is O(n log2 n/s). a1 and b1 are the left and right filter after pass 1. Initially a0 = −∞ and b0 = ∞. After i passes we will have filters ai , bi . Note that during the (i + 1)st pass we can compute the exact rank of ai and bi . How do we find a1 , b1 ? We saw how to obtain an -approximate summary using O( 1 log2 n) space. Thus, if we have space s, we can set 0 = log2 n/s. Let Q = {q1 , q2 , . . . , q` } be 0 -approximate quantile summary for the stream. We query Q for r1 = k −0 n−1 and r2 = k +0 n+1 and obtain a1 and b1 as the answers to the query (here we are ignoring the corner cases where r1 < 0 or r2 > n). Then, by the 0 -approximate guarantee of Q we have that the rank k element lies in the interval [a1 , b1 ] and moreover there are at most O(0 n) elements in this range. It is useful to work out the algebra for p = 2 which shows that the median can be computed in √ O( n log2 n) space.

4

2.1

Random Order Streams

Munro and Paterson also consider the random order stream model in their paper. Here we assume that the stream is a random permutation of an ordered set. It is also convenient to use a different model where the the i’th element is a real number drawn independently from the interval [0, 1]. We can ask whether the randomness can be taken advantage of. Indeed one can. They showed that √ with O( n) space one can find the median with high probability. More generally they showed that in p passes one can find the median with space O(n1/(2p) ). Even though this space bound is better than for adversarial streams it still requires Ω(log n) passes is we have only poly-logarithmic space, same as the adversarial setting. Guha and McGregor [3] showed that in fact O(log log n) passes suffice (with high probability). Here we describe the Munro-Paterson algorithm; see also http://polylogblog.wordpress. com/2009/08/30/bite-sized-streams-exact-median-of-a-random-order-stream/. The algorithm maintains a set S of s consecutively ranked elements in the stream seen so far. It maintains two counters ` for the number of elements less then min S (the min element in S) which have been seen so far and h for the number of elements larger than max S which have been so far. It tries to maintain h − ` as close to 0 as possible to “capture” the median. MunroPaterson(s): n←0 S←∅ `, h ← 0 While (stream is not empty) do n←n+1 a is the new element if (a < min S) then ` ← ` + 1 else if (a > max S) then h ← h + 1 else add a to S if (|S| = s + 1) if (h < `) discard max S from S and h ← h + 1 else discard min S from S and ` ← ` + 1 endWhile if 1 ≤ (n + 1)/2 − ` ≤ s then return (n + 1)/2 − `-th smallest element in S as median else return FAIL.

To analyze the algorithm we consider the random variable d = h − ` which starts at 0. In the first s iterations we simply fill up S to capacity and h − ` remains 0. After that, in each step d is either incremented or decremented by 1. Consider the start of iteration i when i > s. The total number of elements seen prior to i is i − 1 = ` + h + s. In iteration i, since the permutation is random, the probability that ai will be larger than max S is precisely (h + 1)/(h + s + 1 + `). The probability that ai will be smaller than min S is precisely (` + 1)/(h + s + 1 + `) and thus with probability (s − 1)/(h + s + 1 + `), ai will be added to S. Note that the algorithm fails only if |d| at the end of the stream is greater than s. A sufficient condition for success is that |d| ≤ s throughout the algorithm. Let pd,i be the probability that |d| increases by 1 conditioned on the fact that 0 < |d| < s. Then we see that pd,i ≤ 1/2. Thus the process can be seen to be similar to a random walk on the line and some analysis shows that if we √ √ choose s = Ω( n) then with high probability |d| < s throughout. Thus, Ω( n) space suffices to find the median with high probability when the stream is in random order. 5

Connection to CountMin sketch and deletions: Note that when we were discussion frequency moments we assume that the elements were drawn from a [n] where n was known in advance while here we did not assume anything about the elements other than the fact that they came from an ordered universe (apologies for the confusion in notation since we used m previously for length of stream). If we know the range of the elements in advance and it is small compared to the length of the stream then CountMin and related techniques are better suited and provide the ability to handle deletions. The GK summary can also handle some deletions. We refer the reader to [1] for more details. Lower Bounds: For median selection Munro and Paterson showed a lower bound of Ω(n1/p ) on the space for p passes in a restricted model of computation. Guha and McGregor showed a lower bound of Ω(n1/p /p6 ) bits without any restriction. For random order streams O(log log n) passes suffice with polylog(n) space for exact selection with high probability [3]. Moreover Ω(log log n) passes are indeed necessary; see [3] for references and discussion. Bibliographic Notes:

See the references for more information.

References [1] Michael Greenwald and Sanjeev Khanna. Quantiles and equidepth histograms over streams. Available at http://www.cis.upenn.edu/~mbgreen/papers/chapter.pdf. To appear as a chapter in a forthcoming book on Data Stream Management. [2] Michael Greenwald and Sanjeev Khanna. Space-efficient online computation of quantile summaries. In ACM SIGMOD Record, volume 30, pages 58–66. ACM, 2001. [3] Sudipto Guha and Andrew McGregor. Stream order and order statistics: Quantile estimation in random-order streams. SIAM Journal on Computing, 38(5):2044–2059, 2009. [4] Gurmeet Singh Manku, Sridhar Rajagopalan, and Bruce G Lindsay. Approximate medians and other quantiles in one pass and with limited memory. In ACM SIGMOD Record, volume 27, pages 426–435. ACM, 1998. [5] J Ian Munro and Mike S Paterson. Selection and sorting with limited storage. Theoretical computer science, 12(3):315–323, 1980.

6