IEEE Transactions on Signal Processing (May 2005)

Citation preview

MAY 2005

VOLUME 53

NUMBER 5

ITPRED

(ISSN 1053-587X)

PAPERS

Multichannel Signal Processing Applications Statistical Resolution Limits and the Complexified Cramér–Rao Bound. . . . . . . . . . . . . . . . . . . . . . . . . . . . S. T. Smith Covariance, Subspace, and Intrinsic Cramér–Rao Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . S. T. Smith Optimal Dimensionality Reduction of Sensor Data in Multisensor Estimation Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y. Zhu, E. Song, J. Zhou, and Z. You Fourth-Order Blind Identification of Underdetermined Mixtures of Sources (FOBIUM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. Ferréol, L. Albera, and P. Chevalier Penalty Function-Based Joint Diagonalization Approach for Convolutive Blind Separation of Nonstationary Sources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . W. Wang, S. Sanei, and J. A. Chambers An Empirical Bayes Estimator for In-Scale Adaptive Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . P. J. Gendron Methods of Sensor Array and Multichannel Processing Robust Minimum Variance Beamforming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R. G. Lorenz and S. P. Boyd Blind Spatial Signature Estimation via Time-Varying User Power Loading and Parallel Factor Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Y. Rong, S. A. Vorobyov , A. B. Gershman, and N. D. Sidiropoulos Source Localization by Spatially Distributed Electronic Noses for Advection and Diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J. Matthes, L. Gröll, and H. B. Keller

1597 1610 1631 1640 1654 1670 1684 1697 1711

Signal and System Modeling Damped and Delayed Sinusoidal Model for Transient Signals . . . . . . . . . . . . . . . . . . . . . R. Boyer and K. Abed-Meraim

1720

Filter Design and Theory Filtering for Uncertain 2-D Continuous Systems . . . . . . . . . . . S. Xu, J. Lam, Y. Zou, Z. Lin, and W. Paszke Robust Theory and Design of Multirate Sensor Arrays. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .O. S. Jahromi and P. Aarabi Armlets and Balanced Multiwavelets: Flipping Filter Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J. Lian Optimization of Two-Dimensional IIR Filters With Nonseparable and Separable Denominator . . . . . . . . .B. Dumitrescu A New Approach for Estimation of Statistically Matched Wavelet. . . . . . . . . . . . . .A. Gupta, S. D. Joshi, and S. Prasad Phaselets of Framelets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R. A. Gopinath

1731 1739 1754 1768 1778 1794

(Contents Continued on Back Cover)

TLFeBOOK

(Contents Continued from Front Cover) Nonlinear Signal Processing Direct Projection Decoding Algorithm for Sigma-Delta Modulated Signals . . . . . . . . . . . . . . I. Wiemer and W. Schwarz

1807

Algorithms and Applications Testing for Stochastic Independence: Application to Blind Source Separation . . . . . . . . . . . . . . C.-J. Ku and T. L. Fine

1815

Neural Networks Methods and Applications Nonlinear Adaptive Prediction of Complex-Valued Signals by Complex-Valued PRNN . . . . S. L. Goh and D. P. Mandic

1827

Signal Processing for Communications Equalization With Oversampling in Multiuser CDMA Systems . . . . . . . . . . . . . . . . . . B. Vrcelj and P. P. Vaidyanathan Noise-Predictive Decision-Feedback Detection for Multiple-Input Multiple-Output Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D. W. Waters and J. R. Barry Blind Equalization for Correlated Input Symbols: A Bussgang Approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G. Panci, S. Colonnese, P. Campisi, and G. Scarano Low Noise Reversible MDCT (RMDCT) and Its Application in Progressive-to-Lossless Embedded Audio Coding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J. Li Signal Transmission, Propagation, and Recovery Convolutional Codes Using Finite-Field Wavelets: Time-Varying Codes and More . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F. Fekri, M. Sartipi, R. M. Mersereau , and R. W. Schafer BER Sensitivity to Mistiming in Ultra-Wideband Impulse Radios—Part II: Fading Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Z. Tian and G. B. Giannakis Design and Analysis of Feedforward Symbol Timing Estimators Based on the Conditional Maximum Likelihood Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Y.-C. Wu and E. Serpedin

1837 1852 1860 1870

1881 1897 1908

CORRESPONDENCE

Methods of Sensor Array and Multichannel Processing A Bayesian Approach to Array Geometry Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Ü. Oktel and R. L. Moses Computation of Spectral and Root MUSIC Through Real Polynomial Rooting. . . . . . . . . . . . . . . . . . . . . . . . . J. Selva Signal Detection and Estimation Estimate of Aliasing Error for Non-Smooth Signals Prefiltered by Quasi-Projections Into Shift-Invariant Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . W. Chen, B. Han, and R.-Q. Jia Robust Super-Exponential Methods for Deflationary Blind Source Separation of Instantaneous Mixtures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M. Kawamoto, K. Kohno, and Y. Inouye Signal Processing for Communications Iterative Decoding of Wrapped Space-Time Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A. Sezgin and H. Boche

1919 1923

1927 1933 1937

ANNOUNCEMENTS

Call for Papers—2006 IEEE International Symposium on Biomedical Engineering: From Nano to Macro . . . . . . . . . . . Call for Papers—IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING Special Issue on Progress in Rich Transcription ...................................................................................... Call for Papers—IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY . . . . . . . . . . . . . . . . . . . . . . . . . . Call for Papers—IEEE TRANSACTIONS ON SIGNAL PROCESSING Special Issue on Genomic Signal Processing . . . . . . . .

1943 1944 1945

EDICS—Editor’s Information Classification Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Information for Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1946 1947

TLFeBOOK

1942

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1597

Statistical Resolution Limits and the Complexified Cramér–Rao Bound Steven Thomas Smith, Senior Member, IEEE

Abstract—Array resolution limits and accuracy bounds on the multitude of signal parameters (e.g., azimuth, elevation, Doppler, range, cross-range, depth, frequency, chirp, polarization, amplitude, phase, etc.) estimated by array processing algorithms are essential tools in the evaluation of system performance. The case in which the complex amplitudes of the signals are unknown is of particular practical interest. A computationally efficient formulation of these bounds (from the perspective of derivations and analysis) is presented for the case of deterministic and unknown signal amplitudes. A new derivation is given using the unknown complex signal parameters and their complex conjugates. The new formula is readily applicable to obtaining either symbolic or numerical solutions to estimation bounds for a very wide class of problems encountered in adaptive sensor array processing. This formula is shown to yield several of the standard Cramér–Rao results for array processing, along with new results of fundamental interest. Specifically, a new closed-form expression for the statistical resolution limit of an aperture for any asymptotically unbiased superresolution algorithm (e.g., MUSIC, ESPRIT) is provided. The statistical resolution limit is defined as the source separation that equals its own Cramér–Rao bound, providing an algorithm-independent bound on the resolution of any high-resolution method. It is shown that the statistical resolution limit of an array or coherent integration window is about 1 2 SNR 1 4 relative to the radians (large number of Fourier resolution limit of 2 array elements). That is, the highest achievable resolution is proportional to the reciprocal of the fourth root of the signal-to-noise ratio (SNR), in contrast to the square-root (SNR 1 2 ) dependence of standard accuracy bounds. These theoretical results are consistent with previously published bounds for specific superresolution algorithms derived by other methods. It is also shown that the potential resolution improvement obtained by separating two collinear arrays (synthetic ultra-wideband), each with a fixed aperwavelengths (assumed large), is approxture wavelengths by )1 2 , in contrast to the resolution improvement of imately ( for a full aperture. Exact closed-form results for these problems with their asymptotic approximations are presented. Index Terms—Adaptive arrays, adaptive estimation, adaptive signal processing, amplitude estimation, direction-of-arrival estimation, error analysis, ESPRIT, estimation, Fisher information, image resolution, maximum likelihood estimation, MUSIC, parameter estimation, parameter space methods, phase estimation, radar resolution, signal resolution, spectral analysis, superresolution, ultra-wideband processing.

Manuscript received January 5, 2004; revised May 25, 2004. This work was supported by the United States Air Force under Air Force Contract F19628-00-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the author and are not necessarily endorsed by the United States Government. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Fulvio Gini. The author is with the Lincoln Laboratory, Massacusetts Institute of Technology, Lexington, MA 02420 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845426

I. INTRODUCTION ESOLUTION and accurate estimation of signals and their parameters are central capabilities of adaptive sensor array processing. The Cramér–Rao bound (CRB) [29], [34], [44], [54] provides the best accuracy achievable by any unbiased estimator of the signal parameters and therefore provides a fundamental physical limit on system accuracy. In this paper, deterministic CRBs are derived for a general signal model with unknown signal amplitudes, and new closed-form expressions for statistical resolution limits of arrays are derived from these bounds. The statistical resolution limit is defined as the source separation that equals its own CRB, providing an algorithm-independent resolution bound. Multiparameter CRBs for adaptive sensor array processing problems are well known [8], [10], [27], [40], [43], [45], [55]. In this paper, a new formula and new derivation for CRBs involving a complex parameter space [56] are shown to be both very general and very convenient. The new formula [see (29)] for deterministic CRBs has been used in a wide variety of applications to obtain both numerical and analytical CRB results quickly and transparently. What distinguishes the CRB results in this paper from the great majority of existing derivations is a “complexified” approach to the general problem of estimating multiple signal parameters from multiple signals in (proper [26]) complex noise, each signal with an unknown amplitude and phase. Whereas the “realified” approach to estimating complex parameters involves their real and imaginary components, the complexified approach uses the complex parameters and their complex conjugates directly, oftentimes resulting in a greatly simplified analysis. (See Appendix A for background and algebraic definitions of realified and complexified vector spaces.) The complexified approach and the efficient expressions for CRBs it yields are described by Yau and Bresler [56] and van den Bos [53]. New results employing the complexified approach have begun to appear [20], [40]. The derivation of the complexified CRB presented here is greatly simplified from previous work, and the resulting formula is more general than Yau and Bresler’s [56, Th. 2] because a more general steering vector model and parameter space are considered, as well as an arbitrary interference-plus-noise covariance. The CRB formulae are applied to computing new closed-form expressions for the statistical resolution limit of both a single uniform linear array (ULA) [see (61)] and a split-aperture used in synthetic ultra-wideband processing [see (73)]. The case of a single data snapshot or nonfluctuating (deterministic) target model is considered. Synthetic ultra-wideband processing [1], [5], [6], [22], [40], [58] refers to a family of methods using model-based interpolation/extrapolation across widely separated subbands/apertures, usually over large fractional

R

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1598

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

bandwidths. Because the statistical resolution limit is based on the CRB, it represents a high SNR bound on the resolution achievable by any asymptotically unbiased superresolution method, such as MUSIC [24], [30], [35], [46], [48] or ESPRIT [32], [48]. CRB analysis of the source separation to bound the resolution has been used by Clark [4], Smith [40], and Ying et al. [57] in a mathematically equivalent treatment. It is shown that the statistical resolution limit of an aperture is about SNR rad relative to the Fourier resolution limit rad (large number of array elements). That is, the statistical resolution limit is proportional to the reciprocal of the fourth root of the signal-to-noise ratio (SNR), in contrast to the square-root SNR dependence of standard accuracy bounds. For the application of array processing, this provides the best possible fraction of the Rayleigh resolution limit achievable by any superresolution method. The single-aperture results are comparable in many ways with the deterministic resolution limit found by Lee [23], the stochastic CRB case explored by Swingler [49]–[52], and the resolvability detectors considered by Shahram and Milanfar [37]–[39]. The analysis of ULA resolution is applied to the problem of computing resolution limits for synthetic ultra-wideband processing, in which widely separated apertures (either in the frequency or spatial domains) are coherently combined using a model-based procedure to provide enhanced resolution. It is shown that synthetic ultra-wideband processing does indeed improve the resolution above that obtained by using each aperture individually. It is also shown that the potential resolution improvement obtained by separating two collinear arrays, each with a fixed aperture wavelengths by wavelengths (assumed large), is approximately , in contrast to the resolution improvement of for a full aperture. It should be kept in mind that the accuracy and resolution results described in this paper are all derived from CRBs. Therefore, they inherit the properties of such bounds: They are local (not global) bounds and do not reflect the large errors that occur at lower SNR’s, where the ambiguity function has high sidelobe levels. The paper’s body is organized into three sections. Section II contains the description of the complexified Fisher information matrix (FIM) and the corresponding complexified CRBs. These bounds are applied to the key example of pole model estimation in Section III. The pole model accuracy bounds obtained in Section III are then applied to the problem of computing resolution limits in Section IV, which includes a treatment of both the single-aperture and split-aperture cases, as well as a comparison of the resolution improvement between the two. There are also appendices defining the algebraic concept of complexified vector spaces, deriving the complexified FIM, and applying the complexified CRB to a simple angle-angle estimation problem.

where is a proper complex [26] Gaussian interference-plusnoise vector with -by- covariance matrix (2) is a complex -by- “steering” matrix whose columns define signals impinging upon the sensor, the responses of is the -vector of real signal parameters, and is the (unknown, deterministic) complex -vector of amplisignals. Throughout the paper, it tudes corresponding to the is assumed that the unknown amplitudes are fixed (deterministic), although the extension to the stochastic case is straightforward. As explained in Appendix B, deterministic CRB results do not depend on the lack of knowledge of the covariance matrix . Note that this signal model is identical to that of Yau and Bresler’s [56] in the frequently encountered case, where ( , i.e., parameters per signals). The probability distribution of is each of the

(3) and the log-likelihood function is (4) Bounds on the estimation error of the real parameters , which are typically composed of a collection of angles of arrival, Doppler frequencies, ranges, frequencies, damping factors, etc., are to be computed. If the unknown complex amplitudes are in fact known, then the Cramér–Rao theorem [29], [34], [54] asserts that the covariance of any unbiased estimator of the (real-valued) unknown parameters is bounded below by the inverse of the FIM: (5) (in the sense that for symmetric matrices and , is positive semidefinite), where and only if

if (6)

is the FIM, and (7) (8) are the Jacobian and Hessian matrices (1-by- and -bymatrices, respectively) of derivatives of with respect to the is an elements of the parameter vector . Recall that if arbitrary mapping from to , then the Jacobian matrix and, consequently, the FIM transform contravariantly, i.e., (the chain rule). Therefore

II. COMPLEXIFIED CRB (9)

A. Preliminaries The measurement vector from a sensor array containing elements is given by the equation (1)

is the Jacobian matrix of the where transformation. Furthermore, if there are real nuisance pathat must be estimated (such rameters as the signal amplitudes) along with the desired parameters ,

TLFeBOOK

SMITH: STATISTICAL RESOLUTION LIMITS AND THE COMPLEXIFIED CRAMÉR-RAO BOUND

then the Cramér–Rao lower bound on the estimation error of is given by the inverse of the Schur complement of in the full FIM (10) This CRB on the error covariance

1599

is positive defi-

briefly discussed. The real-valued FIM nite with respect to the metric the matrix pencil

, i.e., all eigenvalues of are positive. By (12),

the metrics

is (11)

which increases the CRB due to the lack of knowledge of . (Recall that the Schur complement of the square matrix in the block matrix

is

. The inverse

.) Note of the Schur complement is denoted here as depends on the unknown parameter vector that the CRB . Equation (11) is invariant to arbitrary linear transformations , where is an invertible of the nuisance parameters matrix that may possibly have complex-valued elements. By (9), induces the following transthe change of variables formations in : and ; for an arbitrary therefore, invertible matrix . Consider the unknown complex amplitude vector Re Im appearing in (1). Traditionally, CRB results are derived in terms of the real and imaginary parts and . However, due to the invariance of (11), the CRB remains unchanged if the complexified variables (12) are used. Because the most computationally convenient form of the log-likelihood function depends on the complexified parameters and and not the real parameters and , a complexified approach is generally preferred [53], [56]. By (9), the FIM of the real and imaginary components and is related to the complexified coordinates and by the equation

(13)

(14) are equivalent. Therefore, the complexified FIM of (13) is positive definite with respect to the metric , i.e., the generalized eigenvalues of the matrix pencil (which equal the eigenvalues of

)

are all positive. Compare these facts pertaining to the effect of linear transformations on estimation accuracy with the properties of general complex random variables [36]; see also Krantz [19] for a general treatment of functions of complex variables. Note that the change of variables in (12) and the chain rule defines derivatives with respect to and [19]: (15) (16) These derivatives are consistent with the desired formulae (17) is a function of several complex variables, As an aside, if is analytic if and only if , i.e., the Cauchy-Riemann equations. Because the log-likelihood function is real-valued everywhere and therefore depends on , is not analytic, and indeed, the theory of analytic functions is irrelevant for the results of this paper. B. Complexified FIM and CRB The FIM

(Equation (13) is equivalent to Theorem 1 of Yau and Bresler [56]). The log-likelihood function of many array processing problems [e.g., (4)] depends linearly on and (i.e., sesquilinearly on ), oftentimes greatly simplifying the derivation of the complexified FIM of (13). Note that in encountered in contrast to the complex Hessian matrix the study of plurisubharmonic functions [19], the complexified FIM represents the expectation of the full Hessian matrix with . respect to all the elements of the complexified vector and usually vanish. However, the diagonal blocks typically appear in the Therefore, only the cross terms resulting CRBs. Some additional nonstandard properties of complexified FIM’s that do not appear elsewhere in the literature will be

(18) will be used with (11) to determine the CRB on the covariance of for any unbiased estimator . The details are straightforward and provided in Appendix B, as is the reason the deterministic CRB is not affected by the inclusion of the known or unknown covariance matrix in the parameter vector. The full FIM is (19) where is an -by- complex matrix (the negative expectation of the complex Hessian matrix

TLFeBOOK

1600

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

[19]),

Re

is a -by- real matrix, is a -by- complex matrix, and the subscripted notation “ ” denotes the substitution of the first derivative with respect to wherever the th index of is required (see Appendix B for specific examples). The elements and are given by of the matrices Re

th element th row

where (22) is the -by- complex matrix of the derivative of the steering matrix with respect to . Applying (11) to (19), the complexified CRB is obtained:

where

(23) -by-

Re Re

(24) -by-

-by-by-by-

SNR

(20) (21)

-by-

In the simplest case of a single source and a uniform , and letting linear array, be the array center’s phase, (23)–(28) reduce to SNR , SNR , yielding the well-known result

(25) (26) (27) (28)

The somewhat obvious notation for a subscripted “ ” is explained in the previous paragraph, and the formulae for , , and are repeated for convenience. Equations (24)–(28) may be combined to obtain the expression Re (29) In the frequently encountered case where the signal model is , (29) is readily seen to correspond to Re

(31)

which is independent of the phase center . It also is instructive to compare the deterministic CRB of (29) to the stochastic CRB [2], [14], [49], [50], where the signal amare random with unknown covariance matrix plitudes , and the output probability distribution of with co(1) is , and data matrix variance matrix . In this case, the full FIM is given by . This is an appropriate data model for multiple data snapshots with fluctuating target amplitudes. Matveyev et al. [25] provide a detailed comparison of the deterministic and stochastic CRBs. The bounds shown in (29) must be evaluated for a particular choice of amplitude vector and parameter vector . These vectors are set at some nominal values of interest. The magnitude of the vector may be set to achieve a given array-level signal-to-noise ratio (SNR) via the equation SNR

(32)

is the covariance matrix assumed for the noise where . In the specific model. Note that case considered in Section IV of two sources with identical (but unknown) amplitudes, the array SNR when the sources coaSNR , which is obtained from lesce is given by SNR , SNR , and , and SNR denotes the element-level SNR. The standard application of computing bounds on azimuth and elevation measurements is provided in Appendix C. III. EXAMPLE: POLE MODEL ESTIMATION Consider the problem of modeling frequency data using a set of damped exponentials:

(30) where “ ” and “ ” denote the Schur–Hadamard (componentwise) and Kronecker (tensor) products, respectively, denotes the -by- matrix having unity elements, and, by abuse of notation, [in (30) only]. This is simply Yau and Bresler’s [56] result for the . The matrix may be interpreted as an array of case may be interpreted “cross difference SNR” terms; the matrix as the extra loss of accuracy due to the unknown amplitudes. Application of this complexified approach is straightforward for a large class of applications; compare (29) and (30) to other deterministic CRB formulae reported in the literature [8], [25], [27], is an unbiased estimator of , the [43], [45]–[47], [55]. If CRB holds.

rad

(33) are unknown complex numbers, and the are where the samples in frequency. This may also be viewed as a time-series modeling problem of sinusoids in complex noise [43], [45], [48], [49], [52]; however, the mathematically equivalent frequency-domain version used by Cuomo et al. [5], [6] for synthetic ultra-wideband processing is considered here. Steedly and Moses [43] compute deterministic CRBs for the complex poles and the unknown amplitudes using the standard real/imaginary approach. In this section, the complexified approach will be used to obtain formulae for the identical CR bounds of significantly greater simplicity. These formulae will then be applied to computing closed-form resolution limits in Section IV.

TLFeBOOK

SMITH: STATISTICAL RESOLUTION LIMITS AND THE COMPLEXIFIED CRAMÉR-RAO BOUND

Equation (33) can be expressed compactly using (1) and the steering matrix

.. .

.. .

.. .

(34)

, Because the unknowns are complex numbers the treatment of Section II must be slightly modified to handle this case. There is a choice: Use the complexified approach for and directly, or use the more common parameterizations of the real and imaginary parts or the moduli and phases of the . In this example, the steering poles matrix depends only on the complex pole values (and not their complex conjugates ), and derivatives with respect to and , which are easily obtained from (15) and (16), are (35) (36) which may be extracted from the real and imaginary parts of the derivatives of with respect to . Therefore, it will be simplest to choose the parameterization (37) i.e., the standard real/imaginary approach, and proceed as described in Section II using a hybrid of the real and complexified approaches to compute bounds on the unknown complex poles with the unknown complex amplitudes . Given (37), the CRB for any unbiased estimator of from (23)–(28) is given by

(38) where

Re

(39) (40)

Re

(41) (42) (43) (44) (45) th column of

(46)

Note that the tensor products in (40) and (44) appear in the opposite order than that in (30) because of the different lexicographic order of the parameter vector in (37). Because the th column , the tensor of depends only on and may be efficiently represented using an -bymatrix;

1601

it should be verified that (39)–(46) are consistent with the intergiven in (21) and (22). To see the relative simpretation of plicity achieved by the complexified CRBs, compare (38)–(46) to the equivalent, real/imaginary bounds [43, pp. 1306–1309]. In Section IV, these CRBs on the estimates of the poles will be used to derive bounds on the fundamental resolution achievable by an aperture, i.e., the smallest pole separation measurable by any unbiased estimator of the poles. IV. STATISTICAL RESOLUTION LIMITS The CRB quantifies a lower bound on estimation error, but it does not directly indicate the best resolution achievable by an unbiased estimator. Nevertheless, the CRB can be used to define an absolute limit on resolution. The minimum requirement to resolve two signals is standard deviation of source separation

source separation

(47)

The statistical resolution limit is defined as the source separation at which equality in (47) is achieved, i.e., the source separation that equals its own CRB. Because this concept of resolution is based on the CRB, it has the advantage of bounding the resolution of all super-resolution methods in the regimes where the CRB holds: high SNRs, unbiased estimators, and no modeling or signal mismatch. In this sense, the statistical resolution limit provides the best-case resolution bound for any algorithm. Due to the so-called SNR threshold effect [17], [31] caused by estimation outliers in the threshold region, the resolution of specific algorithms typically exceeds the statistical resolution limit, although their qualitative behaviors may be similar. By statistical considerations alone, Swingler [51] concludes that the practically achievable resolution limit is about one tenth to one fourth of the Rayleigh resolution limit. Furthermore, mismatch and modeling errors can greatly degrade resolution [11], [17], [31], but none of these effects are taken into account by the statistical resolution limit. The CRB of the source separation has been used to bound resolution in previous work by Clark [4], Smith [40], and Ying et al. [57], who provide a mathematically equivalent treatment of (47). The CRB of the source accuracies themselves (not the CRB of the difference) has been used by Lee [23], Swingler [49]–[52], Steedly et al. [42], and several researchers in the optics community (see the references cited in the survey by den Dekker and van den Bos [7], especially the paper by Helstrom [16], as well as Lucy [21] and Bettens et al. [3]). As will be seen, the results derived in this paper are qualitatively consistent with many other published results. Equation (47) is an appealing definition of resolution in a practical sense. In the regime where the sources are resolved, the estimation accuracy is bounded by the CRB. If the poles are too close, they cannot be resolved because the two spectral peaks of each signal coalesce into one; this fact is behind much of the analysis of the resolution limits for several algorithms. In this case, the two poles are estimated to lie at the same location somewhere between the two true locations, the difference between the poles is estimated to be zero, and the

TLFeBOOK

1602

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

root-mean-square error (RMSE) of the estimated difference approximately equals the pole separation. A. Single-Aperture Resolution Limit The resolution limit for two signals sampled over an equally spaced aperture of points will be computed. Assume that sigand ( nals are modeled by the poles small), that each signal has identical but unknown amplitudes, . For this exand that the noise background is white ample problem, (38)–(46) can be expressed compactly using the Dirichlet kernel (48)

Equation (57) is Van Trees’s property 5 of CRBs of nonrandom variables [54, sec.2.4]. Symbolic algebra packages, such as Maple or Mathematica, are useful for computing the Laurent series Var

(58) SNR (59)

small, where SNR SNR is the array SNR for when the poles coalesce, and SNR is the element-level SNR. As discussed above, it is desired to compute the separation that yields at least a standard deviation of , i.e.,

Because of the special form of (33), it will be convenient to take derivatives with respect to the poles

SNR

and then

transform them to derivatives with respect to the real and imaginary parts of the poles via (35) and (36). Expressing (38)–(46) in terms of the example, it is found that Re

Solving (60) for this tical resolution limit

(49)

(60) (assumed small) determines the statis-

SNR (50) (51)

Re

(54) , and

where

for some unknown

complex parameter . In the assumed case of unit-variance SNR , where SNR is the element-level white noise, is an unknown phase. Applying (32) when the SNR, and poles coalesce , SNR SNR . To simplify the computations, a phase-centered aperture is used. As discussed above, a bound is desired for the RMSE of the difference complexified or where

real , and

(55) (56) ,

as above. Therefore, the FIM must be computed in terms of the variable or . Because real/imaginary coordinates are used in (38)–(46), the parameter vector from (56) is used, resulting in

(57)

for

large

(62)

Dividing (62) by the standard Fourier resolution limit rad yields a useful rule-of-thumb for the absolute resolution limit: resolution re. Fourier res.

(52) (53)

(61)

SNR

SNR

(63)

In addition, note that (61) implies that more than three sensors are required to superresolve the two complex poles. Equation (61) indicates that to double the resolution using a fixed aperture, the SNR must increase by 12 dB, or to increase the resolution tenfold, the SNR must increase by 40 dB. This property is shown in Fig. 1, which for the example of shows the RMSEs and the CRBs of the modulus of the difference of the estimated poles versus the pole separation. The “ringing” pattern observed in Fig. 1 when the poles are resolved occurs as the sources move in and out of the nulls and peaks of their respective array responses. The error is lowest when one pole is near a null of the other array’s response and highest when it is near a peak. Equation (61) is verified by Monte Carlo simulations (Figs. 2 and 4) of the maximum-likelihood (ML) method for pole estimation with unknown amplitudes. Fig. 2 illustrates an example with 50 samples and SNR of 48 dB. The maximum-likelihood method achieves the CRB when the poles are resolved but breaks away from the CRB curve at the predicted statistical resolution limit of about 0.08 of the Fourier resolution rad. The resolution performance of both the MUSIC and ESPRIT algorithms is also shown, using a data masamples. trix constructed from a sliding window of Both of these superresolution methods achieve resolutions of rad, with ESPRIT about 0.3–0.4 of the Fourier resolution achieving a slightly better resolution than MUSIC due to the invariance properties of pole estimation that ESPRIT exploits.

TLFeBOOK

SMITH: STATISTICAL RESOLUTION LIMITS AND THE COMPLEXIFIED CRAMÉR-RAO BOUND

= 50

Fig. 1. Cramér-Rao bound of the pole difference for N and several SNRs, its asymptote computed using (60), and the resolution limits computed using (61) expressed using a standard Fourier resolution cell (FRC) of =N  = rad. Note that halving the resolution requires increasing the SNR by 12 dB. In this example, the standard deviation of the difference between pole locations (   separation) is computed assuming two signals with poles at 1 and e and with equal but unknown amplitudes. The “ringing” pattern observed results from the sources moving in and out of the nulls and peaks of their respective array responses.

18

1603

2

1

Beneath their resolution limits, MUSIC and ESPRIT provide the identical estimate for the two different poles. Hence, the RMSE of the difference equals the pole separation of the unresolved poles. In contrast, beneath the resolvability of the ML algorithm, the random behavior of peaks in the eight-dimensional log-likelihood surface (magnitude and phase for two poles and two complex amplitudes) yields a nearly flat RMSE curve when the poles are unresolved. These resolution results are qualitatively consistent with many resolution bounds reported in the literature [40]. Lee [23, Eqs. (62) and (63)] hypothesizes a necessary condition for the resolvability of signals (deterministic CRB assumption) and or SNR for . deduces a dependence of SNR Swingler [49]–[52] (stochastic assumption) uses an approximation of the stochastic CRB to obtain resolution bounds. For large , high SNR, and a single data snapshot, Swingler’s implies that the resoapproximation [51, Eq. (3)] ) is at lution (normalized by the Fourier resolution of SNR SNR . Up to the constant least multiple of 1.7, this is comparable to the statistical resolution SNR given in (63). Ying et al. [57] define a limit of resolution limit in their (5) that is mathematically equivalent to the statistical resolution limit defined in (47), and they illustrate in their Fig. 1, as well resolution proportional to SNR provide as an alternate geometric interpretation of this limit. Shahram and Milanfar [37]–[39] use a detection theoretic concept of resolvability to arrive at a resolution limit that depends . Lee and Wengrovitz [24] (stochastic assumption) on SNR show in their Eq. (91) that the resolution threshold for MUSIC . Farina [9, Sec.5.7] confirms the reis proportional to SNR sult of Gabriel [12] that the achievable superresolution is about

Fig. 2. Full aperture predicted versus measured resolution with two poles. The RMSE of the difference of the pole estimates versus pole separation for the maximum-likelihood (ML) method (red dashed curve with s), the MUSIC algorithm (purple dashed curve with s), the ESPRIT algorithm (blue dashed curve with xs), and the CRB (dark green solid curve) are all shown using the , unit of a Fourier resolution cell (FRC) =N  = rad, where N and SNR dB in this example. For both the MUSIC and ESPRIT samples is used to generate the algorithms, a sliding window of N= data matrices. The maximum-likelihood method fails to resolve the two poles at the resolution predicted by (63) and breaks away from the CRB at this point; the statistical resolution limit (separation RMSE) is the dashed black line to the upper right. The SNR threshold behavior of both the MUSIC and ESPRIT algorithms is observed, resulting in an achieved resolution greater than the statistical resolution limit for both of these methods. As expected, the resolution of the ESPRIT method is slightly greater than that of the MUSIC method because ESPRIT exploits invariance information not used by MUSIC. A Monte Carlo simulation with 1000 trials is used to compute the RMSEs. The “ringing” pattern observed is the sources moving in and out of the nulls and peaks of their respective array responses.

+

= 48

2 2 = 25

18

= 50

=

SNR , which is a good match to the theoretical result SNR . Steedly and Moses [43] (deterministic assumption) numerically compute the CRB for the angle separation of two poles on the unit circle and illustrate a plot (Fig. 5 on p. 1312) that is qualitatively similar to Fig. 1. This result is also derived by Kaveh and Lee [17, Eq. (5.118)] (stochastic case), using asymptotic properties of the Wishart distribution. Stoica and Nehorai [46], [47] (deterministic and stochastic assumptions) illustrate numerical evidence that the standard deviation is proportional to the inverse of the separation as in (60). Stoica and Söderström (stochastic case) observe in [48, p. 1842] that the covariance elements become large when two frequencies are close but do not quantify this increase. Zhang [59, Fig. 1] illustrates numerical results for the probability of resolution dependence. that show a close match to the predicted SNR In contrast to this body of work, Kosarev [18] uses an information-theoretic approach to arrive at a superresolution bound that depends logarithmically on the SNR. If the magnitudes of the two poles are fixed on the unit circle, a similar analysis shows that the resolution is also determined . by (61) if the resolution limit is multiplied by

TLFeBOOK

1604

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

B. Split-Aperture Resolution Limit Split-aperture arrays refer to the case where instead of having a full aperture, there are two apertures separated by a large gap. The analysis of the preceding section can be modified to compute the achievable resolution of split-aperture arrays with so-called synthetic ultra-wideband coherent processing [5], [6], [40], [58]. CRBs on source direction of arrivals are well known for arbitrary array geometries [13], [28]. The CRB of the source separation for a split-aperture is considered here. Let the synthetic ultra-wideband coherent aperture samples (as above), but with the middle consist of samples removed (for convenience, assume that the difference is even so that the two widely separated apertures have equal length). The frequency samples occur at the samples , deleted band , , . The results of the preof (48) is vious section carry over if the Dirichlet kernel replaced with the modified Dirichlet kernel

Fig. 3. Band gap CRB for B = 25, M = 500 (yielding N = 550), and several SNRs expressed using a standard Fourier resolution cell (FRC) of 2=N 0:011 rad. The asymptotes from (71) and the resolution limits from (73) are also shown. The “ringing” pattern observed is the sources moving in and out of the nulls and peaks of their respective array responses.



(64) In terms of the subaperture bandwidth tion , (65) becomes

Computing the Laurent series of the variance for small Var

Var

SNR

SNR (71)

(65) where

SNR (66)

(for

large,

SNR (73) SNR

SNR SNR is the array SNR when the poles coalesce, and SNR is the element-level SNR. Note that this and and that it reduces to (59) if equation is symmetric in (as desired). The ultrawide bandwidth is more conveniently parameterized in terms of the two separated bandwidths and the band gap as (70)

(72)

fixed), where . Applying (47) to (65) [or (71)] yields the statistical resolution limit for a split-band aperture

(67) (68)

(69)

and the band separa-

(74)

(radians, large, fixed). Fig. 3 illustrates the CRB and resolution achievable for estimating two pole positions whose signal amplitudes are assumed equal (with 36 dB array SNR) but unknown for two frequency bands of 25 samples each separated by 100 samples. Fig. 4 compares the performance of the maximum-likelihood estimator to the compute CRBs. Note that at the resolution limit, the RMSE of the maximum-likelihood estimate breaks away from the resolution bound because the poles are no longer resolved and therefore cannot be estimated individually as the CRB analysis assumes. Comparing Figs. 1 and 3, it is seen that splitting and

TLFeBOOK

SMITH: STATISTICAL RESOLUTION LIMITS AND THE COMPLEXIFIED CRAMÉR-RAO BOUND

1605

(74). The array SNR is the same in both cases. Taking the ratio of these two equations yields the approximate equation (75)

resolution improvement

large), which quantifies the increase in resolution ( fixed, achieved by separating the two bands. Comparing Figs. 1 and 3 36 or 48 dB, shows that the resoat high SNRs, e.g., SNR lution improvement is about 4.57. Applying (75) to this example and yields 4.47, which is correct within with about 2% of the true value. Note that the quantities in (75) are dimensionless. An equivalent formula involving the bandwidths and band gap (in Hertz) of the two subapertures BW , where is the frequency sampling in Hertz, BW is given by the equation resolution improvement Fig. 4. Split-aperture predicted versus measured resolution with two poles. The RMSE of the maximum-likelihood estimate of the pole difference and the = 36 dB is CRB versus pole separation for M = 100, B = 25, and SNR shown using a standard Fourier resolution cell (FRC) of 2=N 0:011 rad. The ML method fails to resolve the two poles at the resolution predicted by (74) and breaks away from the CRB at this point. The RMSE of the pole differences is computed using a Monte Carlo simulation with 1000 trials. The “ringing” pattern observed is the sources moving in and out of the nulls and peaks of their respective array responses.



BW BW

(76)

(BW fixed, BW large). Again, it should be kept in mind that (75) and (76) represent the improvement for the high SNR case. As the two subapertures are moved further apart, ambiguities in the likelihood surface make accurate estimation increasingly difficult. Furthermore, modeling errors, which further degrade the resolution improvement, are not included in this analysis. V. CONCLUSIONS

separating a 50-sample-wide band by 100 samples improves the statistical resolution limit. An approximation of this improvement under the assumption of large band gaps is provided in Section IV-C. C. Effect of Split-Aperture Band Gap on Resolution It is useful to quantify the resolution improvement achieved by using split-aperture arrays. The resolution improvement obtained by increasing the band gap relative to a single coherent aperture with no band gap is computed in this subsection. To accomplish this, the asymptotic result of (74) for large band gap is compared to the single-aperture case of (62) for . Keep in mind that these CRB-based results assume unbiased estimators; as the band gap grows, interferometric ambiguities increase, and it becomes more and more difficult to compute unbiased estimates of the pole positions. Zatman and Smith [58] analyze the transition for band gaps in which the poles are resolved, and the CRB assumptions are valid, and larger band gaps, where the poles become unresolved, and more general Weiss–Weinstein bounds must be applied. is fixed and Assuming that the subaperture bandwidth small with respect to the band gap, the statistical resolution achievable by coherently processing over the full aperture (with no band gap) is given approximately by (62), whereas the resolution achievable by coherently processing over widely separated subapertures is given approximately by

New formulae for complexified CRBs are derived for parameter estimation in adaptive sensor array processing. The case of deterministic and unknown signal amplitudes is considered, which is an appropriate assumption when a single data snapshot is available, or if a nonfluctuating target model is used. These bounds, which are both general and computationally attractive, are used to analyze estimation accuracy for pole model estimation for both array processing and synthetic ultra-wideband processing. A new bound on the statistical resolution limits of single and split-apertures is derived by examining the CRBs in the case of two closely spaced poles. The statistical resolution limit is defined as the source separation that equals its own CRB. This bound is used to describe the resolution improvement achieved by separating two subapertures with a large band gap. It is shown that 1) the statistical resolution limit of an aperture is about SNR relative to the Fourier resolution rad (large ), and 2) the resolution improvement of a split-aperture used in synthetic ultra-wideband processing is quantified , where (assumed large) is the separation (in by samples or wavelengths), and is the size of each aperture (in samples or wavelengths). The fourth root dependence of the resolution limit on SNR is a useful rule of thumb and illustrates how important SNR is to resolvability: To double the resolution using a fixed aperture, the SNR must be increased by 12 dB, and to increase the resolution tenfold, the SNR must be increased by 40 dB. These resolution limits represent the best case possible theoretically. The real effects of ambiguities, bias, and modeling errors are frequently encountered and further limit the resolution achievable in practice.

TLFeBOOK

1606

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

APPENDIX A DEFINITIONS OF COMPLEXIFIED AND REALIFIED VECTOR SPACES The complexification of a real vector space is the complex , i.e., the tensor product over the real vector space field of with the complex numbers. The complex vector space contains vectors of the form . Note that . The realification of a comis a restriction of to the real numplex vector space . Note that . bers and is denoted These linear algebraic concepts [15] are encountered most frequently in the study of representations of Lie algebras [33]. If (the complexification of a real vector space ), then is isomorphic to any real subspace of (direct sum) such that , where is the complex conjugate of the vector space . Two obvious choices result in the standard decompositions for realified vecfor : Re Im (ditors . In estimation probrect sum), and lems involving complex parameters whose real and imaginary parts lie in a real vector space , the great majority of authors derive CRBs using the realified approach, i.e., pa. In the complexirameters of the form Re Im fied approach, results are obtained using parameters of the form . Both approaches use different rep, but for ease of comresentations of the real vector space putation, the latter is preferred for the general case of a (proper) complex signal model, where formulae typically involve the complex parameters and their complex conjugates and not their real and imaginary parts.

Examining the log-likelihood function of (4), its derivatives with respect to , , and may be written down by inspection (motivating the simplicity of the complexified approach), as shown in (78)–(83) at the bottom of the page, where (84) [see (27)]. Expressions involving a subscript “ ” in (79)–(83) simply denote the substitution of the first derivative with wherever the th index of is required, e.g., respect to for (only will appear in the final expressions), likewise for expressions involving a subscripted ” and the second derivatives with respect to . Specif“ ically, the quantity is an -bymatrix; therefore, is a rank 3 tensor -by- -by- . The summation implied by of dimension is efficiently expressed as the -by- matrix

(85)

.. .

Similarly, the double summation implied by is the -by- matrix whose th element is (86)

APPENDIX B DERIVATION OF THE COMPLEXIFIED FIM The derivation of the complexified FIM of (19) is provided in this appendix. The FIM

Expressions involving

are interpreted in the same way:

(77) will be used with (11) to determine the CRB on the covariance for any unbiased estimator with an associated unknown of complex parameter vector .

.. .

(87)

(78)

const.

(79) (80) (81) (82) (83)

TLFeBOOK

SMITH: STATISTICAL RESOLUTION LIMITS AND THE COMPLEXIFIED CRAMÉR-RAO BOUND

is a

-by-

matrix, and the is given by

th element of the

-by-

matrix

1607

which results in the CRB of (23): (98)

(88) Note that any expression possessing a single subscripted “ ” is in general a complex matrix with rows or columns whose th row or column involves the derivative with respect to . ” is Similarly, any expression possessing a subscripted “ a -by- complex matrix whose th elements involve the second derivative with respect to and . In the final CRB results, the important terms will contain two subscripts “ ” that represent a -by- complex matrix whose th elements involve a bilinear expression of the first derivatives with respect and [see (25)]. to (note that ) Taking the expectation of yields the FIM

APPENDIX C ANGLE–ANGLE ESTIMATION As a straightforward application of the CRB approach developed in Section II, the estimation bounds for angle–angle estimation are developed in this Appendix. Consider an -element sensor array whose elements are located at positions with respect to an arbitrary phase center . Represent the sensor array’s position by the real . At frequency and 3-by- matrix , the array’s steering vector is given by wavelength (99) where

(89)

(100)

of (19). is assumed to be unknown, this If the covariance matrix extra nuisance parameter does not affect the CRBs on the other parameters because the cross terms in the FIM containing all vanish. Indeed, from the Taylor series

and are the azimuth and elevation of the signal, and the exponential function here is intended to act component-wise (as in Matlab). As in (31), the choice of phase center is irrelevant if the signal amplitude is assumed to be unknown. Note that there is only one signal and one unknown amplitude. A bound on azimuth-elevation estimation error is obtained by applying (23)–(28) to this steering vector and its derivatives

(90)

(101) (102)

(91) it follows that (92) in (92) with respect to The expectation of the derivative of ). Thereany of , , or always vanishes (because fore, there are no nonzero cross terms in the extended FIM containing , and consequently, the CR bounds on the other parameters are not affected by lack of knowledge of the covariance in this deterministic case. of To compute the inverse of the Schur complement in the full FIM, apply (11), noting that (for )

, , and “ ” denotes the where Schur–Hadamard (component-wise) product. For any unbiased estimators and of and , the bound on their error covariance is given by (103) The intermediate quantities , , (a scalar denoted by in this case), and (a vector denoted by in this case) are given by the formulae Re

(104)

Re

(105) (106) (107)

(93) (94)

Re

(95) (96) (97)

(cf. other specific multiple parameter cases [8], [41], [55]). ACKNOWLEDGMENT The analysis and results in this paper were motivated by discussions with M. Zatman, J. Ward, K. Senne, R. Barnes, K.

TLFeBOOK

1608

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Cuomo, J. Piou, G. Hatke, and K. Forsythe. The very detailed and thorough feedback from the anonymous reviewers, the associate editor F. Gini, and A. Farina, greatly improved this paper. The author is indebted to them all for their very helpful insights and suggestions.

REFERENCES [1] L. Y. Astanin and A. A. Kostylev, “Ultra wideband signals—A new step in radar development,” IEEE Aerosp. Electron. Syst. Mag., vol. 7, pp. 12–15, Mar. 1992. [2] W. J. Bangs, “Array Processing With Generalized Beamformers,” Ph.D. dissertation, Yale Univ., New Haven, CT, 1971. [3] E. Bettens, A. J. Den Dekker, D. Van Dyck, and J. Sijbers, “Ultimate resolution in the framework of parameter estimation,” in Proc. IASTED Int. Conf. Signal Image Process., Las Vegas, NV, Oct. 1998. [4] M. P. Clark, “On the resolvability of normally distributed vector parameter estimates,” IEEE Trans. Signal Process., vol. 43, no. 12, pp. 2975–2981, Dec. 1995. [5] K. M. Cuomo, J. E. Piou, and J. T. Mayhan, “Ultra-wideband coherent processing,” Mass. Inst. Technol. Lincoln Lab. J., vol. 10, no. 2, pp. 203–222, 1997. , “Ultrawide-band coherent processing,” IEEE Trans. Antennas [6] Propag., vol. 47, no. 6, pp. 1094–1107, Jun. 1999. [7] A. J. den Dekker and A. van den Bos, “Resolution, a survey,” J. Opt. Soc. Amer., vol. 14, pp. 547–557, 1997. [8] A. Dogandˇzic´ and A. Nehorai, “Cramér-Rao bounds for estimating range, velocity, and direction with an active array,” IEEE Trans. Signal Processing, vol. 49, no. 6, pp. 1122–1137, Jun. 2001. [9] A. Farina, Antenna-Based Signal Processing Techniques for Radar Systems. Boston, MA: Artech House, 1992. [10] J. M. Francos, “Cramér-Rao bound on the estimation accuracy of complex-valued homogeneous Gaussian random fields,” IEEE Trans. Signal Process., vol. 50, no. 3, pp. 710–724, Mar. 2002. [11] B. Friedlander and J. Weiss, “The resolution threshold of a directionfinding algorithm for diversely polarized arrays,” IEEE Trans. Signal Process., vol. 42, no. 7, pp. 1719–1727, Jul. 1994. [12] W. F. Gabriel, “Spectral analysis and adaptive array superresolution techniques,” Proc. IEEE, vol. 68, no. 6, pp. 654–666, Jun. 1980. [13] A. B. Gershman and J. F. Böhme, “A note on most favorable array geometries for DOA estimation and array interpolation,” IEEE Signal Process. Lett., vol. 4, no. 8, pp. 232–235, Aug. 1997. [14] A. B. Gershman, P. Stoica, M. Pesavento, and E. G. Larsson, “Stochastic Cramér-Rao bound for direction estimation in unknown noise fields,” Proc. Inst. Elect. Eng. Radar Sonar Navig., vol. 149, no. 1, pp. 2–8, Feb. 2002. [15] P. R. Halmos, Finite-Dimensional Vector Spaces. New York: SpringerVerlag, 1987. [16] C. W. Helstrom, “Resolvability of objects from the standpoint of statistical parameter estimation,” J. Opt. Soc. Amer., vol. 60, pp. 659–666, 1970. [17] M. Kaveh and H. Wang, “Threshold properties of narrow-band signal subspace array processing methods,” in Advances in Spectrum Analysis and Array Processing, S. Haykin, Ed. Upper Saddle River, NJ: Prentice-Hall, 1991, vol. 2, pp. 173–220. [18] E. L. Kosarev, “Shannon’s superresolution limit for signal recovery,” Inverse Prob., vol. 6, pp. 55–76, 1990. [19] S. G. Krantz, Function Theory of Several Complex Variables, Second ed. Belmont, CA: Wadsworth, 1992. [20] X. Liu and N. D. Sidiropoulos, “Cramér-Rao lower bound for low-rank decomposition of multidimensional arrays,” IEEE Trans. Signal Process., vol. 49, no. 9, pp. 2074–2086, Sep. 2001. [21] L. B. Lucy, “Statistical limits to super-resolution,” Astron. Astrophys., vol. 261, pp. 706–710, 1992. [22] J. T. Mayhan, M. L. Burrows, K. M. Cuomo, and J. E. Piou, “High resolution 3D background and feature extraction,” IEEE Trans. Aerosp. Electron. Syst., vol. 37, no. 4, pp. 630–642, Apr. 2001. [23] H. B. Lee, “The Cramér-Rao bound on frequency estimates of signals closely spaced in frequency,” IEEE Trans. Signal Process., vol. 40, no. 6, pp. 1508–1517, Jun. 1992. [24] H. B. Lee and M. S. Wengrovitz, “Resolution threshold of beamspace MUSIC for two closely spaced emitters,” IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 9, pp. 1545–1559, Sep. 1990.

[25] A. L. Matveyev, A. B. Gershman, and J. F. Böhme, “On the direction estimation Cramér-Rao bounds in the presence of uncorrelated unknown noise,” Circuits, Syst., Signal Process., vol. 18, no. 5, pp. 479–487, 1999. [26] F. D. Neeser and J. L. Massey, “Proper complex random processes with applications to information theory,” IEEE Trans. Inf. Theory, vol. 39, no. 7, pp. 1293–1302, Jul. 1993. [27] M. Pesavento and A. Gershman, “Array processing in the presence of nonuniform sensor noise: A maximum likelihood direction finding algorithm and Cramér-Rao bounds,” in Proc. 10th IEEE Signal Process. Workshop Statist. Signal Array Process., Pocono Manor, PA, 2000, pp. 78–82. [28] M. Pesavento, A. B. Gershman, and K. M. Wong, “Direction finding in partly calibrated sensor arrays composed of multiple subarrays,” IEEE Trans. Signal Process., vol. 50, no. 9, pp. 2103–2115, Sep. 2002. [29] C. R. Rao, “Information and the accuracy attainable in the estimation of statistical parameters,” Bull. Calcutta Math. Soc., vol. 37, pp. 81–89, 1945. [30] B. S. Rao and K. V. S. Hari, “Performance analysis of root-MUSIC,” IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 12, pp. 1939–1949, Dec. 1989. [31] C. R. Richmond, “Capon algorithm mean squared error threshold SNR prediction and probability of resolution,” IEEE Trans. Signal Process., to be published. [32] R. Roy and T. Kailath, “ESPRIT—Estimation of signal parameters via rotational invariance techniques,” IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 7, pp. 984–995, Jul. 1989. [33] H. Samelson, Notes on Lie Algebras. New York: Springer-Verlag, 1990. [34] L. L. Scharf, Statistical Signal Processing: Detection, Estimation, and Time Series Analysis. Reading, MA: Addison-Wesley, 1991. [35] R. O. Schmidt, “Multiple emitter location and signal parameter estimation,” in Proc. RADC Spectrum Estimation Workshop, Griffiss Air Force Base, NY, 1979. [36] P. J. Schreier and L. L. Scharf, “Second-order analysis of improper complex random vectors and processes,” IEEE Trans. Signal Process., vol. 51, no. 3, pp. 714–725, Mar. 2003. [37] M. Shahram and P. Milanfar, “A statistical analysis of achievable resolution in incoherent imaging,” in Proc. SPIE Annual Meet., San Diego, CA, Aug. 2003. , “Imaging below the diffraction limit: A statistical analysis,” IEEE [38] Trans. Image Process., vol. 13, no. 5, pp. 677–689, May 2004. [39] , “On the resolvability of sinusoids with nearby frequencies in the presence of noise,” IEEE Trans. Signal Process., to be published. [40] S. T. Smith, “Accuracy and resolution bounds for adaptive sensor array processing,” in Proc. 9th IEEE Signal Process. Workshop Statist. Signal Array Process., Portland, OR, 1998, pp. 37–40. [41] , “Adaptive radar,” in Wiley Encyclopedia of Electrical and Electronics Engineering, J. G. Webster, Ed. New York: Wiley, 1999, vol. 1, pp. 263–289. [42] W. M. Steedly, C.-H. J. Ying, and R. L. Moses, “Resolution bound and detection results for scattering centers,” in Proc. Int. Conf. Radar, Brighton, U.K., 1992, pp. 518–521. [43] W. M. Steedly and R. L. Moses, “The Cramér-Rao bound for pole and amplitude coefficient estimates of damped exponential signals in noise,” IEEE Trans. Signal Process., vol. 41, no. 3, pp. 1305–1318, Mar. 1993. [44] P. Stoica and R. L. Moses, Introduction to Spectral Analysis. Upper Saddle River, NJ: Prentice-Hall, 1997. [45] P. Stoica, R. L. Moses, B. Friedlander, and T. Söderström, “Maximum likelihood estimation of the parameters of multiple sinusoids from noisy measurements,” IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 3, pp. 378–392, Mar. 1989. [46] P. Stoica and A. Nehorai, “MUSIC, maximum likelihood, and Cramér-Rao bound,” IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 5, pp. 720–741, May 1989. [47] , “Performance study of conditional and unconditional direction-ofarrival estimation,” IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 10, pp. 1783–1795, Oct. 1990. [48] P. Stoica and T. Söderström, “Statistical analysis of MUSIC and subspace rotation estimates of sinusoidal frequencies,” IEEE Trans. Signal Process., vol. 39, no. 8, pp. 1836–1847, Aug. 1991. [49] D. N. Swingler, “Frequency estimation for closely-spaced sinusoids: Simple approximations to the Cramér-Rao lower bound,” IEEE Trans. Signal Process., vol. 41, no. 1, pp. 489–494, Jan. 1993. [50] , “Simple approximations to the Cramér-Rao lower bound on directions of arrival for closely spaced sources,” IEEE Trans. Signal Process., vol. 41, no. 4, pp. 1668–1672, Apr. 1993.

TLFeBOOK

SMITH: STATISTICAL RESOLUTION LIMITS AND THE COMPLEXIFIED CRAMÉR-RAO BOUND

[51] [52] [53] [54] [55] [56] [57] [58] [59]

, “Narrowband line-array beamforming: Practically achievable resolution limit of unbiased estimators,” IEEE J. Ocean. Eng., vol. 19, no. 2, pp. 225–226, Apr. 1994. , “Further simple approximations to the Cramér-Rao lower bound on frequency estimates for closely-spaced sinusoids,” IEEE Trans. Signal Process., vol. 43, no. 1, pp. 367–369, Jan. 1995. A. van den Bos, “A Cramér-Rao lower bound for complex parameters,” IEEE Trans. Signal Process., vol. 42, no. 10, pp. 2859–2859, Oct. 1994. H. L. Van Trees, Detection, Estimation, and Modulation Theory. New York: Wiley, 1968, pt. 1. J. Ward, “Maximum likelihood angle and velocity estimation with space-time adaptive processing radar,” in Proc. 30th Asilomar Conf. Signals, Syst., Comput., 1996, pp. 1265–1267. S. F. Yau and Y. Bresler, “A compact Cramér-Rao bound expression for parametric estimation of superimposed signals,” IEEE Trans. Signal Process., vol. 40, no. 5, pp. 1226–1230, May 1992. C.-H. J. Yang, A. Sabharwal, and R. L. Moses, “A combined order selection and parameter estimation algorithm for undamped exponentials,” IEEE Trans. Signal Process., vol. 48, no. 3, pp. 693–701, Mar. 2000. M. Zatman and S. T. Smith, “Resolution and ambiguity bounds for interferometric-like systems,” in Proc. 32nd Asilomar Conf. Signals, Syst. Comput., 1998. Q. T. Zhang, “A statistical resolution theory of the AR method of spectral analysis,” IEEE Trans. Signal Process., vol. 46, no. 10, pp. 2757–2766, Oct. 1998.

1609

Steven Thomas Smith (M’86–SM’04) was born in La Jolla, CA, in 1963. He received the B.A.Sc. degree in electrical engineering and mathematics from the University of British Columbia, Vancouver, BC, canada, in 1986 and the Ph.D. degree in applied mathematics from Harvard University, Cambridge, MA, in 1993. From 1986 to 1988, he was a research engineer at ERIM, Ann Arbor, MI, where he developed morphological image processing algorithms. He is currently a senior member of the technical staff at MIT Lincoln Laboratory, Lexington, MA, which he joined in 1993. His research involves algorithms for adaptive signal processing, detection, and tracking to enhance radar and sonar systems performance. He has taught signal processing courses at Harvard and for the IEEE. His most recent work addresses intrinsic estimation and superresolution bounds, mean and variance CFAR, advanced tracking methods, and space-time adaptive processing algorithms. Dr. Smith was an associate editor of the IEEE TRANSACTIONS ON SIGNAL PROCESSING from 2000 to 2002 and received the SIAM outstanding paper award in 2001.

TLFeBOOK

1610

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Covariance, Subspace, and Intrinsic Cramér–Rao Bounds Steven Thomas Smith, Senior Member, IEEE

Abstract—Cramér–Rao bounds on estimation accuracy are established for estimation problems on arbitrary manifolds in which no set of intrinsic coordinates exists. The frequently encountered examples of estimating either an unknown subspace or a covariance matrix are examined in detail. The set of subspaces, called the Grassmann manifold, and the set of covariance (positive-definite Hermitian) matrices have no fixed coordinate system associated with them and do not possess a vector space structure, both of which are required for deriving classical Cramér–Rao bounds. Intrinsic versions of the Cramér–Rao bound on manifolds utilizing an arbitrary affine connection with arbitrary geodesics are derived for both biased and unbiased estimators. In the example of covariance matrix estimation, closed-form expressions for both the intrinsic and flat bounds are derived and compared with the root-mean-square error (RMSE) of the sample covariance matrix . The accuracy (SCM) estimator for varying sample support bound on unbiased covariance matrix estimators is shown to be 1 2 about (10 log 10) dB, where is the matrix order. Remarkably, it is shown that from an intrinsic perspective, the SCM is a biased and inefficient estimator and that the bias term reveals the dependency of estimation accuracy on sample support observed in theory and practice. The RMSE of the standard method of estimating subspaces using the singular value decomposition (SVD) is compared with the intrinsic subspace Cramér–Rao bound derived in closed form by varying both the signal-to-noise ratio (SNR) of the unknown -dimensional subspace and the sample support. In the simplest case, the Cramér–Rao bound on subspace estimation 1 2 SNR 1 2 ))1 2 accuracy is shown to be about ( ( rad for -dimensional subspaces. It is seen that the SVD-based method yields accuracies very close to the Cramér–Rao bound, establishing that the principal invariant subspace of a random sample provides an excellent estimator of an unknown subspace. The analysis approach developed is directly applicable to many other estimation problems on manifolds encountered in signal processing and elsewhere, such as estimating rotation matrices in computer vision and estimating subspace basis vectors in blind source separation. Index Terms—Adaptive arrays, adaptive estimation, adaptive signal processing, covariance matrices, differential geometry, error analysis, estimation, estimation bias, estimation efficiency, Fisher information, Grassmann manifold, homogeneous space, matrix decomposition, maximum likelihood estimaiton, natural gradient, nonlinear estimation, parameter estimation, parameter space methods, positive definitive matrices, Riemannian curva. Riemannian manifold, singular value decomposition. ture,

I. INTRODUCTION STIMATION problems are typically posed and analyzed for a set of fixed parameters, such as angle and Doppler. In contrast, estimation problems on manifolds, where no such

E

Manuscript received June 23, 2003; revised June 3, 2004. This work was supported by the United States Air Force under Air Force Contract F19628-00-C0002. Opinions, interpretations, conclusions, and recommendations are those of the author and are not necessarily endorsed by the United States Government. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Randolph Moses. The author is with the Lincoln Laboratory, Massachusetts Institute of Technology, Lexington MA 02420 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845428

set of intrinsic coordinates exists, are frequently encountered in signal processing applications and elsewhere. Two common examples are estimating either an unknown covariance matrix or a subspace. Because neither the set of covariance (positive definite Hermitian) matrices nor the set of subspaces (the Grassmann manifold) are equipped with an intrinsic coordinate system or a vector space structure, classical Cramér–Rao bound (CRB) analysis [17], [55] is not directly applicable. To address this class of problems, an intrinsic treatment of Cramér–Rao analysis specific to signal processing problems is established here. Intrinsic versions of the CRB have also been developed for Riemannian manifolds [34], [43], [44], [46], [51], [52], [68], [79], [80], statistical manifolds [5], and for the application of quantum inference [11], [53]. The original contributions of this paper are 1) a derivation of biased and unbiased intrinsic CRBs for signal processing and related fields; 2) a new proof of the CRB (Theorem 2) that connects the inverse of the Fisher information matrix with its appearance in the (natural) gradient of the log-likelihood function; 3) several results [Theorem 4, Corollary 5, (143)] that bound the estimation accuracy of an unknown covariance matrix or subspace; 4) the noteworthy discovery that from an intrinsic perspective, the sample covariance matrix (SCM) is a biased and inefficient estimator (Theorem 7), and the fact that the bias corresponds to the SCMs poor estimation quality at low sample support (Corollary 5)—this contradicts the well-known fact that because the linear expectation operator implicitly treats the covariance matrices as a convex cone included in the vector space , compared to the intrinsic treatment of the covariance matrices in this paper; 5) a generalization of the expression for Fisher information (Theorem 1) that employs the Hessian of the log-likelihood function for arbitrary affine connections—a useful tool because in the great majority of applications the second-order terms are much easier to compute; 6) a geometric treatment of covariance matrices as the quotient space (i.e., the Hermitian part of the matrix polar decomposition), including a natural distance between covariance matrices that has not appeared previously in the signal processing literature; and 7) a comparison between the accuracy of the standard subspace estimation method employing the singular value decomposition (SVD) and the CRB for subspace estimation. In contrast to previous literature on intrinsic Cramér–Rao analysis, it is shown explicitly how to compute practical estimation bounds on a parameter space defined by a manifold, independently of any particular metric or affine structure. As elsewhere, the standard approach is used to generalize classical bounds to Riemannian manifolds via the exponential map, i.e., geodesics emanating from the estimate to points in the parameter space. Just as with classical bounds, the unbiased intrinsic

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

CRBs depend asymptotically only on the Fisher information and do not depend in any nontrivial way on the choice of measurement units, e.g., feet versus meters. Though the mathematical concepts used throughout the paper are well known to the differential geometry community, a brief informal background of the key ideas is provided in footnotes for readers unfamiliar with some of the technicalities, as is a table at the end of the paper, comparing the general concepts to their more familiar counterparts in . The results developed in this paper are general enough to be applied to the numerous estimation problems on manifolds that appear in the literature. Zheng and Tse [82], in a geometric approach to the analysis of Hochwald and Marzetta [35], compute channel capacities for communications problems with an unknown propagation gain matrix represented by an element on the Grassmann manifold. Grenander et al. [30] derive Hilbert–Schmidt lower bounds for estimating points on a Lie group for automatic target recognition. Srivastava [70] applies Bayesian estimation theory to subspace estimation, and Srivastava and Klassen [71], [72] develop an extrinsic approach to the problem of estimating points on a manifold, specifically Lie groups and their quotient spaces (including the Grassmann manifold), and apply their method to estimating target pose. Bhattacharya and Patrangenaru [9] treat the general problem of estimation on Riemannian manifolds. Estimating points on the rotation group, or the finite product space of the rotation group, occurs in several diverse applications. Ma et al. [41] describe a solution to the motion recovery problem in computer vision, and Adler et al. [2] use a set of rotations to describe a model for the human spine. Douglas et al. [20], Smith [66], and many others [21]–[23], develop gradient-based adaptive algorithms using the natural metric structure of a constrained space. For global estimation bounds rather than the local ones developed in this paper, Rendas and Moura [59] define a general ambiguity function for parameters in a statistical manifold. In the area of blind signal processing, Cichocki et al. [14], [15], Douglas [19], and Rahbar and Reilly [54] solve estimation problems on the Stiefel manifold to accomplish blind source separation, and Xavier [79] analyzes blind MIMO system identification problems using an intrinsic approach. Lo and Willsky [39] analyze optimal estimation problems on Abelian Lie groups. Readers may also be interested in the central role played by geometrical statistical analysis in establishing Wegener’s theory of continental drift [18], [27], [43]. A. Model Estimation Problem Consider the problem [68] of estimating the unknown matrix , , given the statistical model

by (1)

where is a -dimensional normal random vector with zero mean and unknown covariance matrix , and is an -dimensional normal random vector independent of with zero mean and known covariance matrix . The normal random vector has zero mean and covariance matrix (2) Such problems arise, for example, when there is an interference term with fewer degrees of freedom than

1611

the number of available sensors [16], [66]. CRBs for estimation problems in this form are well known [7], [60]. The Fisher information matrix for the unknown parameters is given by the simple expression , and provides the so-called stochastic CRB. What differs in the estimation problem of (1) from the standard case is an explanation of the derivative terms when the parameters lie on a manifold. The analysis of this problem may be viewed in the context of previous analysis of subspace estimation and superresolution methods [30], [70], [72]–[74], [76], [77]. This paper addresses both the real-valued and (proper [49]) complex-valued cases, which are also referred to as the real symmetric and Hermitian cases, respectively. All real-valued examples may be extended to (proper) in-phase plus quadrature data by replacing transposition with conjugate transposition and using the real representation of the unitary group. B. Invariance of the Model Estimation Problem The estimation problem of (1) is invariant to the transformations (3) , which is the genfor any by invertible matrix in eral linear group of real by invertible matrices. That is, substituting and into (1) leaves the measurement unchanged. The only invariant of the transformation is the column span of the matrix , and the positive-definite symmetric (Hermitian) structure of covariance matrices is, of course, invariant to the transformation . Therefore, only the column span of and the covariance matrix of may be measured, and we ask how accurately we are able to estimate this subspace, i.e., the column span of , in the presence of the unknown covariance matrix . The parameter space for this estimation problem is the set of all -dimensional subspaces in , which is known as the Grassmann manifold , and the set of all by positive-definite symmetric (Hermitian) matrices , which is the so-called nuisance parameter space. Both and may be represented by sets of equivalence classes, which are known mathematically as quotient or homogeneous spaces. Although this representation is more abstract, it turns out to be very useful for obtaining closed-form expressions of the necessary geometric objects used in this paper. In fact, both the set of subspaces and the set of covariance matrices are what is known as reductive homogeneous spaces and, therefore, possess natural invariant connections and metrics [10], [12], [25], [32], [38], [50]. Quotient spaces are also the “proper” mathematical description of these manifolds. A Lie group is a manifold with differentiable group operations. For a (closed) subgroup, the quotient “ ” denotes the set of equivalence classes , where is the equivalence class for all , . For example, any positive-definitive symmetric matrix has the Cholesky decomposition , where (the general linear group) is an invertible matrix with the unique polar decomposition [28] , where is a positive-definite symmetric matrix, and

TLFeBOOK

1612

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

(the orthogonal Lie group) is an orthogonal matrix. Clearly, and the orthogonal part of the polar decomposition is arbitrary in the specification of . Therefore, for any covariance matrix , there is a corresponding equivalence class in the quotient , where is the unique positive-definitive space symmetric square root of . Thus, this set of covarience matrices may be equated with the quotient space , allowing application of this space’s intrinsic geometric structure to problems involving covariace matrices. In the Hermitian covariance matrix case, the correct identification is , where is the Lie group of unitary matrices. Another way of viewing the identification of is by the transitive group action [10], [32], [38] seen in (3) of the group acting on via the map . This action is almost effective (the matrices are the only ones that fix all , ) and has the isotropy (invariance) subgroup of at because for all orthogonal matrices . The only part of this group action that matters is the positive-definite symmetric part because . Thus, the set of positive-definite symmetric (Hermitian) matrices may be viewed as the equivalence class of invertible matrices multiplied on the right by an arbitrary orthogonal (unitary) matrix. Although covariance matrices obviously have a unique matrix representation, this is not true of subspaces, because for subspaces, it is only the image (column span) of the matrix that matters. Hence, quotient space methods are essential in the description of problems involving subspaces. Edelman et al. [23] provide a convenient computational framework for the Grassmann manifold involving its quotient space structure. Subspaces are represented by a single (nonunique) by matrix with orthonormal columns that itself represents the entire equivalence class of matrices with the same column span. Thus, for the unknown matrix , , may be multiplied on the right by any by orthogonal matrix, i.e., for , without affecting the results. The Grassmann manifold is represented by the quotient because the set of by orthonormal matrices is the same equivalence class as the set of by orthogonal matrices

where

is an arbitrary by matrix such that and . Many other signal processing applications also involve the Stiefel manifold of subspace basis vectors or, equivalently, the set of by matrices with orthonormal columns [23]. Another representation of the Grassmann manifold is the set of equivalence classes . Although the approach of this paper immediately applies to the Stiefel manifold, it will not be considered here. The reductive homogeneous space structure of both and is exploited extensively in this paper, as are the corresponding invariant affine connections and invariant metrics [viz. (65), (66), (119), and (122)]. Reductive homogeneous spaces and their corresponding natural Riemannian metrics appear frequently in signal processing and other applications [1], [33],

[43], [65], e.g., the Stiefel manifold in the singular value and QR-decompositions, but their presence is not widely acknowledged. A homogeneous space is said to be reductive [12], [32], [38] if there exists a decomposition (direct sum) such that Ad , where and are the Lie algebras of and , respectively. Given a bilinear form (metric) on (e.g., the trace of , , for symmetric matrices), there corresponds a -invariant metric and a corresponding -invariant affine connection on . This is said to be the natural metric on , which essentially corresponds to a restriction of the Killing form on in most applications. For the example of the covariance matrices, , which is the Lie algebra of by matrices [or ], , which is the sub-Lie algebra of skewsymmetric matrices [or the skew-Hermitian matrices ], and symmetric Hermitian matrices , so that (direct sum). That is, any by matrix may be expressed as the sum of its symmetric part and its skew-symmetric part . The symmetric matrices are Ad -invariant because for any symmetric matrix and orthogonal matrix , Ad , which is also symmetric. Therefore, admits a -invariant metric and connection corresponding to the bilinear form at , specifically (up to an arbitrary scale factor) (4) . See Edelman et al. [23] for the details of at arbitrary the Grassmann manifold for the subspace example. Furthermore, the Grassmann manifold and the positive-definite symmetric matrices with their natural metrics are both Riemannian globally symmetric spaces [10], [25], [31], [38], although, except for closed-form expressions for geodesics, parallel translation, and sectional curvature, the rich geometric properties arising from this symmetric space structure are not used in this paper. As an aside for completeness, is a compact irreducible symmetric space of type BD I, and is a decomposable symmetric space, where the irreducible noncompact component is of type A I [31, ch. 10, secs. 2.3 and 2.6; ch. 5, prop. 4.2; p. 238]. Elements of decompose naturally into the product of the determinant of the covariance matrix multiplied by a covariance matrix with unity determinant, i.e., . C. Plan of the Paper The paper’s body is organized into three major sections, addressing intrinsic estimation theory, covariance matrix estimation, and subspace and covariance estimation. In Section II, the intrinsic CRB and several of its properties are derived and explained using coordinate-free methods. In Section III, the well-known problem of covariance matrix estimation is analyzed using these intrinsic methods. A closed-form expression for the covariance matrix CRB is derived, and the bias and efficiency of the sample covariance matrix estimator is considered. It is shown that the SCM viewed intrinsically is a biased and inefficient estimator and that the bias term reveals the degraded estimation accuracy that is known to occur at low

TLFeBOOK

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

sample support. Intrinsic CRBs for the subspace plus unknown covariance matrix estimation problem of (1) are computed in closed-form in Section IV and compared with a Monte Carlo simulation. The asymptotic efficiency of the standard subspace estimation approach utilizing the SVD is also analyzed. II. INTRINSIC CRB An intrinsic version of the CRB is developed in this section. There are abundant treatments of the CRB in its classical form [60], [78], its geometry [61], the case of a constrained parameter space [29], [75], [80], and several generalizations of the CRB to arbitrary Riemannian manifolds [11], [34], [43], [52], [53], [68], [79] and statistical manifolds [5]. It is necessary and desirable for the results in this paper to express the CRB using differential geometric language that is independent of any arbitrary coordinate system and a formula for Fisher information [26], [60], [78] that readily yields CRB results for problems found in signal processing. Other intrinsic derivations of the CRB use different mathematical frameworks (comparison theorems of Riemannian geometry, quantum mechanics) that are not immediately applicable to signal processing and specifically to subspace and covariance estimation theory. Eight concepts from differential geometry are necessary to define and efficiently compute the CRB: a manifold, its tangent space, the differential of a function, covariant differentiation, Riemannian metrics, geodesic curves, sectional/Riemannian curvature, and the gradient of a function. Working definitions of each of these concepts are provided in footnotes; for complete, technical definitions, see Boothby [10], Helgason [31], Kobayashi and Nomizu [38], or Spivak [69]. In addition, Amari [3]–[6] has significantly extended Rao’s [55], [56] original geometric treatment of statistics, and Murray and Rice [48] provide a coordinate-free treatment of these ideas. See Table II in Section IV-F of the paper for a list of differential geometric objects and their more familiar counterparts in Euclidean -space . Higher order terms containing the manifold’s sectional [the higher dimensional generalization of the two-dimensional (2-D) Gaussian curvature] and Riemannian curvature [10], [12], [31], [34], [38], [52], [79], [80] also make their appearance in the intrinsic CRB; however, because the CRB is an asymptotic bound for small errors (high SNR and sample support), these terms are negligible for small errors. A. Fisher Information Metric be the probability density function (pdf) of a Let , given vector-valued random variable in the sample space in the -dimensional manifold1 ( not the parameter necessarily equal to ). The CRB is a consequence of a natural metric structure defined on a statistical model, defined by the . parameterized set of probability densities Let

1613

be the log-likelihood function, and let and be two tangent vectors2 on the parameter space. Define the Fisher information metric3 (FIM) as (6) where is the differential4 of the log-likelihood function. We may retrench the dependence on explicit tangent vectors and express the FIM as (9) where “ ” denotes the tensor product. The FIM is defined with respect to a particular set of coordinate functions or paon : rameters (10) Refer to footnote 4 and the “Rosetta Stone” table for an explanation of the notation “ ” for tangent vectors. Of course, the Fisher information metric is invariant to the , i.e., change of coordinates , because the FIM transforms contravariantly (11) where and are the FIMs with respect to the coordinates is the Jacospecified by their subscripts, and 2The tangent space of a manifold at a point is, nonrigorously, the set of vectors tangent to that point; rigorously, it is necessary to define manifolds and their tangent vectors independently of their embedding in a higher dimension [e.g., as we typically view the 2-sphere embedded in three-dimensional (3-D) space]. The tangent space is a vector space of the same dimension as M . This vector M is typically denoted by T M . The dual space of linear maps space at from T M to , called the cotangent space, is denoted T M . We imagine tangent vectors to be column vectors and cotangent vectors to be row vectors. If P from one manifold to another, then there exists we have a mapping : M a natural linear mapping : T M T P from the tangent space at to the tangent space at ( ). If coordinates are fixed, = @ =@ , i.e., simply the Jacobian transformation. This linear map is called the push-forward: a notion that generalizes Taylor’s theorem ( +  ) = ( ) + (@ =@ ) + + . h.o.t. = 3A Riemannian metric g is defined by a nondegenerate inner product on the T M is a definite quadratic manifold’s tangent space, i.e., g : T M form at each point on the manifold. If is a tangent vector, then the square = ; = g( ; ). Note that this inner length of is given by product depends on the location of the tangent vector. For the example of the = : =1 , sphere embedded in Euclidean space, S g( ; ) = (I ) for tangent vectors to the sphere at . For covariance matrices, the natural Riemannian metric is provided in (4); for subspaces, the natural metric is given in (118). 4The differential of a real-valued function `: M on a manifold, called d`, simply represents the standard directional derivative. If c(t) (t ) is a = (d=dt) c(t) is a tangent vector at c(0), curve on the manifold, then and the directional derivative of ` in direction is given by

2

!





!

2

h i

k k



!

f 2



0

!



d`

( ) =

j



d

( ( )) :

` c t

dt

2

g

2

(7)

Examining this definition with respect to a particular coordinate system, ` may be viewed as a function `( ;  ; . . . ;  ), the curve c(t) is viewed as (c (t); c (t); . . . ; c (t)) , the tangent vector is given by = ( ; ; . . . ; ) = (dc =dt; dc =dt; . . . ; dc =dt) , and the directional derivative is given by the equation



d`

( ) =

@` @



:

(8)

as an n by 1 column vector, then d` = (@`=@ ; . . . ; @`=@ ) is a 1 by n row vector. Furthermore, the basis (5) vectors induced by these coordinates for the tangent space T M are called (@=@ ); (@=@ ); . . . ; (@=@ ) because d`(@=@ ) = @`=@ . The dual 1A manifold is a space that “looks” like basis vectors for the cotangent space T M are called d ; d ; . . . ; d belocally, i.e., M may be parameterized by n coordinate functions  ;  ; . . . ;  that may be arbitrarily chosen cause d (@=@ ) =  (Kronecker delta). Using the push-forward concept up to obvious consistency conditions. (For example, positions on the globe are for tangent spaces described in footnote 2, `: M ! , and ` : T M ! , mapped by a variety of 2-D coordinate systems: latitude-longitude, Mercator, i.e., d` = ` , which is consistent with the interpretation of d` as a cotangent If we view @`=@ ;

and so forth.)

(row) vector.

TLFeBOOK

1614

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

bian of the change of variables. Because the Jacobian determines how coordinate transformations affect accuracy bounds, it is sometimes called a sensitivity matrix [45], [61]. Although the definition of the FIM employing first derivatives given by (9) is sufficient for computing this metric, in many applications (such as the ones considered in this paper), complicated derivations involving cross terms are oftentimes encountered. (It can be helpful to apply multivariate analysis to the resulting expressions. See, for example, Exercise 2.5 of Fang and Zhang [24]). Derivations of CRBs are typically much simpler if the Fisher information is expressed using the second derivatives5 of the log-likelihood function. The following theorem provides this convenience for an arbitrary choice of affine connection , which allows a great deal of flexibility ( is not the gradient operator; see footnotes 5 and 8). The FIM is independent of the arbitrary metric (and its induced Levi–Civita connection) chosen 5Differentiating real-valued functions defined on manifolds is straightforward because we may subtract the value of the function at one point on the manifold from its value at a different point. Differentiating first derivatives again or tangent vectors, however, is not well defined (without specifying additional information) because a vector tangent at one point is not (necessarily) tangent at another: Subtraction between two different vector spaces is not intrinsically defined. The additional structure required to differentiate vectors is called the affine connection , so-called because it allows one to “connect” different tangent spaces and compare objects defined separately at each point. [This is not the gradient operator; see (47) in footnote 8.] The covariant differential of a `, is simply d`( ). In function ` along tangent vector , written `( ) or a particular coordinate system, the ith component of ` is (@`=@ ). The ij th component of the second covariant differential `, which is the generalization of the Hessian, is given by

r



(r `) =

r r r r

@ ` @ @

0

@g @x

+

0



@` @

(12)

0 @g @x

(13)

to define the root-mean-square error (RMSE) on the parameter space. be a family of pdfs parameterized Theorem 1: Let , be the log-likelihood function, and be the Fisher information metric. For any affine connection defined on

by

(16) Proof: The proof is a straightforward application of the lemma: , which follows immediately from differentiating the identity with respect to and observing that . Applying this lemma to , which is expressed in arbitrary coordinates as in (12), it is seen that , the last equality following from integration by parts, as in classical Cramér–Rao analysis. Theorem 1 allows us to compute the Fisher information on an arbitrary manifold using precisely the same approach as is done in classical Cramér–Rao analysis. To see why this is so, consider the following expression for the Hessian of the loglikelihood function defined on an arbitrary manifold endowed with the affine structure : (17) where is a geodesic6 emanating from in the direction . Evaluating for small [12], [69, vol. 2, ch. 4, props. 1 and 6] yields (19)

where

0 =

1 2

g

@g @x

are the Christoffel symbols (of the second kind) of the connection defined by (@=@ ) = 0 (@=@ ) or defined by the equation for geodesics 0 _ _ = 0 in footnote 6. It is sometimes more convenient to  + work with Christoffel symbols of the first kind, which are denoted 0 , where 0 = g 0 , i.e., the 0 are given by the expression in (13) without the g term. For torsion-free connections (the typical case for applications), the +0 = 0. Christoffel symbols possess the symmetries 0 = 0 and 0 It is oftentimes preferable to use matrix notation 0( _ ; _ ) to express the quadratic terms in the geodesic equation [23]. For a sphere embedded in Euclidean space, the Christoffel symbols are given by the coefficients of the ( ) for tangent vectors and to quadratic form 0( ; ) = in [23] [cf. (122)]. The ij th component of the covariant the sphere at differential of a vector field ( ) T M is

r

1

2

(r ) =

@

@

+

0

:

(14)

Classical index notation is used here and throughout the paper (see, for example, Spivak [69] or McCullagh [44] for a description). A useful way to compute

= _ (0) + 0( (0); 2) for the vector field the covariant derivative is

(t) = (exp t2) (the “exp ” notation for geodesics is explained in footnote 6). on M along the In general, the covariant derivative of an arbitrary tensor tangent vector 2 is given by the limit

r

r

= lim



(t) 0 (0) t

(15)

where (t) denotes the tensor at the point exp (t2), and  denotes parallel translation from to (t) = exp (t2) [10], [31], [38], [69]. Covariant differentiation generalizes the concept of the directional derivative of along 2 and encodes all intrinsic notions of curvature on the manifold [69, vol. 2, ch. 4–8]. satisfies the differential The parallel translation of a tensor field (t) =  equation = 0.

r

This equation should be interpreted to hold for normal coordinates (in the sense of footnote 6) on . Applying (17) and (19) to compute the Fisher information metric, it is immediately seen that for normal coordinates (20) where the second derivatives of the right-hand side of (20) represent the ordinary Hessian matrix of , interpreted to be a scalar function defined on . As with any bilinear form, the 6A geodesic curve tial equation

r r 7!

_ =  +

(t) on a manifold is any curve that satisfies the differen0 _ _ = 0;

(0) =

r 

;

_ (0) = :

(18)

If is a Riemannian connection, i.e., g 0 for the arbitrary Riemannian metric g , (18) yields length minimizing curves. The map

(t) = exp ( t) is called the exponential map and provides a natural diffeomorphism between the tangent space at and a neighborhood of on the manifold. Looking ahead, the notation “exp” explains the appearance of matrix exponentials in the equations for geodesics on the space given in (67) of covariance matrices P and the Grassmann manifold G and (120). The geodesic (t) is said to emanate from in the direction. By the exponential map, any arbitrary choice of basis of the tangent space at yields a set of coordinates on M near ; the coordinates are called normal [31], [38], [69]. One important fact that will be used in Sections III and IV to determine covariance and subspace estimation accuracies is that the length of to (t) is t . For the sphere embedded in the geodesic curve from Euclidean space, geodesics are great circles, as can be seen by verifying that = = 1; the paths (t) = cos t + sin t [ = 0, cf. (120)] satisfy the differential equation  + 0( _ ; _ ) = 0, where the Christoffel symbol 0 for the sphere is given in footnote 5.

TLFeBOOK

j j1k k

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

1615

off-diagonal terms of the FIM may be computed using the standard process of polarization: (21) Compare this FIM approach to other intrinsic treatments of CRBs that employ the exponential map explicitly [34], [43], [51], [52]. Thus, Fisher information does not depend on the choice of any particular coordinates or any underlying Riemannian metric (or affine connection). We should expect nothing less than this result, as it is true of classical estimation bounds. For example, discounting a scaling factor, measurements in feet versus meters do not affect range estimation bounds. B. Intrinsic Estimation Theory Let be a parameterized pdf, where takes on values in the manifold . An estimator of is a mapping from the space of random variables to . Oftentimes, this space of random variables is a product space comprising many snapshots taken from some underlying distribution. Because addition and subtraction are not defined between points on arbitrary manifolds, an affine connection on (see footnotes 5 and 6) must be assumed to make sense of the concept of the mean value of on [51]. Associated with this connection is the exponential map from the tangent space at any point to points on (i.e., geodesics) and its inverse . Looking ahead, the notation “ ” and “ ” is explained by the form of distances and geodesics on the space of covariance matrices and the Grassmann manifold provided in (65), (67), (69), (119), and (120). If is connected and complete, the function is onto [31], but its inverse may have multiple values (e.g., multiple windings for great-circle geodesics on the sphere). Typically, the tangent vector of shortest length may be chosen. In the case of conjugate points to [12], [31], [38] (where is singular, e.g., antipodes on the sphere), an arbitrary tangent vector may be specified; however, this case is of little practical concern because the set of such points has measure zero, and the CRB itself is an asymptotic bound used for small errors. The manifold has nonpositive curvature [see (28) in footnote 7]; therefore, it has no conjugate points [38, vol. 2, ch. 8, sec. 8], and in fact, is uniquely defined by the unique logarithm of positive-definite matrices in this case. For the Grassmann manifold , is uniquely defined by the principal value of the arccosine function. Definition 1: Given an estimator of , the expectation of the estimator with respect to , which is denoted , and the bias vector field of are defined as (22) (23) Fig. 1 illustrates these definitions. Note that unlike the stan, is not (necessarily) linear dard expectation operator as it does not operate on linear spaces. These definitions are independent of arbitrary coordinates on but do

Fig. 1. Intrinsic estimation on a manifold. The estimator ^(z) of the parameter is shown, where z is taken from the family of pdfs f (z ) whose parameter set is the manifold M . The exponential map “exp ” equates points on the manifold with points in the tangent space T M at via geodesics (see footnotes 5 and 6). This structure is necessary to define the expected value of ^ because addition and subtraction are not well defined between arbitrary points on a manifold [see (22)]. The bias vector field [see (23)] is defined ^ ] and, therefore, depends on the by ( ) = exp E [ ^] = E [exp geodesics chosen on M .

j

depend on the specific choice of affine connection , just as the bias vector in depends on vector addition. In fact, because for every connection there is a unique torsion-free connection with the same geodesics [69, vol. 2, ch. 6, cor. 17], the bias vector really depends on the choice of geodesics on . is said to be unbiased if . If , the bias

Definition 2: The estimator (the zero vector field) so that is said to be parallel.

), the In the ordinary case of Euclidean -space (i.e., exponential map and its inverse are simply vector addition, i.e., and . Parallel bias vectors are constant in Euclidean space because . The proof of the CRB in Euclidean spaces relies on that fact that . However, for arbitrary Riemannian manifolds, plus second-order and higher terms involving the manifolds’s sectional and Riemannian curvatures.7 The following lemma quantifies these nonlinear terms, which are negligible for small errors (small distances between and ) and small biases (small vector norm ), although these higher order terms do appear in the intrinsic CRB. 7The sectional curvature is Riemann’s higher dimensional generalization of the Gaussian curvature encountered in the study of 2-D surfaces. Given a circle of radius r in a 2-D subspace H of the tangent space T M , let A (r ) = r be the area of this circle, and let A(r ) be the area of the corresponding “circle” in the manifold mapped using the function exp . Then, the sectional curvature of H is defined [31] to be

K (H ) =

lim 12

A (r) 0 A(r) : r A (r)

(24)

We will also write K to specify the manifold. For the example of the unit sphere S , A(r ) = 2 (1 cos r ), and K 1, i.e., the unit sphere is a space of constant curvature unity. Equation (24) implies that the area in the manifold is smaller for planes with positive sectional curvature, i.e., geodesics tend to coalesce, and conversely that the area in the manifold is larger for planes with negative sectional curvature, i.e., geodesics tend to diverge from each other. Given tangent vectors and and the vector field , the Riemannian curvature tensor is defined to be

0

(

;

)

0

=



r r 0r r 0r (25) 2 T M is the Lie bracket. The Riemannian cur-

where [ ; ] = vature ( ; ) measures the amount by which the vector is altered as it is parallel translated around the “parallelogram” in the manifold defined by and [31, ch. 1, ex. E.2]: ( ; ) = lim ( (t))=t , where (t) is the parallel translation of . Remarkably, sectional curvature and Riemannian curvature are equivalent: The n(n 1)=2 sectional curvatures completely determine the Riemannian curvature tensor and vice-versa. The relationship between the two is

TLFeBOOK

0

0

K(

^

)=

h

(

;

)

;

k ^ k

i

(26)

1616

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

^ k ^ k

where K ( ) is the sectional curvature of the subspace spanned by and , = sin  is the square area of the parallelogram formed by and , and  is the angle between these vectors. Fig. 2 illustrates how curvature causes simple vector addition to fail. There are relatively simple formulas [12], [13], [31] for the Riemannian curvature tensor in the case of homogeneous spaces encountered in signal pro(n; ) is noncompact, cessing applications. Specifically, the Lie group G = and its Lie algebra = (n; ) admits a decomposition = + [direct = Hermitian matrices, = (n)], such that sum,

k kk k

;

[

]

 ;

[

;

]

 ;

[

;

]

 :

The sectional curvature of the symmetric homogeneous space (n; )= (n) is given by [13, prop. 3.39], [31, ch. 5, sec. 3]

K

(

^

)=

0

1 2

;

1

=

2

0 41 k[ ; ]k

1

=

4

(27)

Fig. 2. Failure of vector addition in Riemannian manifolds. Sectional and Riemannian curvature terms (see footnote 7) appearing in the CRB arise ^ , which contains a quadratic form ( ) in from the expression exp

])

. This quadratic form is directly expressible in terms of the Riemannian curvature tensor and the manifold’s sectional curvatures [69, vol. 2, ch. 4]. These terms are negligible for the small errors and biases, which are (max K ) , where the domain of interest for CRBs, i.e., d( ^; ) K is the manifold’s sectional curvature.

 =

P ;

tr ([

2

0

and

R2

R 1



K and

where

0

(

^

0 2

k ; ]k

)= [

=

0 21 tr ([ ;

0 0

0

0

are skew-symmetric matrices of the form

max

jK j = 41

and

max

K

=1

:

(30)

be a family of pdfs parameterized by Lemma 1: Let , be the log-likelihood function, be an arbitrary Riemannian metric (not necessarily the FIM), be an affine connection on corresponding to , and, for any estimator of with bias vector field , let the matrix denote the covariance of , , all with respect to the arbitrary coordinates near and corresponding tangent vectors , . Then 1)

(31) is

where the term defined by the expression

(32) 2)

For sufficiently small bias norm of is

, the

j

j

where denotes the sectional curvature of the 2-D subspace , and , , and are the angles between the tangent vector and , , and , respectively. For sufficiently small covariance norm , the matrix is given by the quadratic form (34) is the Riemannian curvature tensor. where Furthermore, the matrices and are symmetric, and depends linearly on .



0

Q

r

, and

is an (n p) by p matrix. [The signs in (28) and (29) are correct because for = tr = tr .] The an arbitrary skew-symmetric matrix , sectional curvature of G is non-negative everywhere. For the cases p = 1 or 1 p = n 1, check that (29) is equivalent to the constant curvature K for the sphere S given above. To bound the terms in the covariance and subspace CRBs corresponding with these curvatures, we note that

k k

3)

(29)

])

^



(28) are orthonormal Hermitian matrices, and [ ; ] = (n) is the Lie bracket. (The scalings and arise from the fact that the matrix P corresponds to the eqivalence class H in G=H ). Note that P has nonpositive curvature everywhere. Similarly for the Grassmann manifold, the Lie group (n) ( (n) in the complex case) is compact, and its Lie algebra admits a comparable decomposition (see Edelman et al. [23, sec. 2.3.2] for application-specific details). The sectional curvature of the Grassp) (p)) equals mann manifold G = (n)=( (n where

= exp

Proof: First take the covariant derivative of the identity (the zero vector field). To differentiate the argument inside the integral, recall that for an arbitrary vector field , . The integral of the first part of this sum is simply the left-hand side of (31); therefore, . This establishes part 1). The remainder of the proof is a straightforward computation using normal coordinates (see footnote 6) and is illustrated in Fig. 2. We will compute , , using (15) from footnote and 5. Define the tangent vectors by the equations (35) Assume that the are an orthonormal basis of . Expressing all of the quantities in terms of the normal coordinates , the geodesic from to is simply the , where . The geodesic curve to satisfies (18) in footnote 6 subject to curve from the boundary condition specified in (35). Using Riemann’s original local analysis of curvature, the metric is expressed as the Taylor series [69, vol. 2, ch. 4]

th element (36) if if

where metries (33)

TLFeBOOK

, which possess the sym, and . Applying (13) in footnote 5 gives the Christoffel

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

symbols (of the first kind) geodesic equation for to second order in

. Solving the yields (37)

denotes the quadratic form in where (36). The parallel translation of along the geodesic equals . Therefore (38) (also see Karcher [37, app. C3]). Riemann’s quadratic form simply

equals

, where is the sectional curvature of the subspace spanned by , and , is the square area of the parallelogram formed by and , is the angle between these vectors, and is the Riemannian curvature tensor (see footnote 7). Taking the inner product of (38) with

1617

of square distance; therefore, the sectional curvature terms in and bias Lemma 1 are negligible for small errors norm much less than , i.e., errors and biases less than the reciprocal square root of the maximal sectional curvature. For example, the unit sphere has constant sectional curvature of unity; therefore, the matrices and may be neglected in this case for errors and biases much smaller than 1 rad. The intrinsic generalization of the Cramér–Rao lower bound is as follows. Theorem 2 (Cramér–Rao): Let be a family of pdfs parameterized by , let be the log-likelihood function, be the Fisher information metric, and be an affine connection on . 1) For any estimator of with bias vector field , the covariance matrix of , satisfies the matrix inequality

(42) where Cov covariance matrix

is the

(43)

(39) where terms

is the FIM, is the identity matrix, is the covariant differential , are the Christoffel symbols, and the matrices of and representing sectional and Riemannian curvature terms are defined in Lemma 1, all with respect to the arbitrary coordinates near . 2) For sufficiently small relative to , satisfies the matrix inequality

is the angle between and . Ignoring the higher order , the expectation of (39) equals

(44) (40) Applying polarization [(21)] to (40) using the orthonormal establishes parts 2) and 3) and shows basis vectors that and are symmetric. Using normal coordinates, the th element of the matrix is seen to be (41)

3) Neglecting the sectional and Riemannian curvature terms and at small errors and biases, satisfies (45) We may substitute “ ” for the expression “ ,” interpreting it as the component by component difference for some set of coordinates. In the trivial case , the proof of Theorem 2 below is equivalent to asserting that the covariance matrix of the zero-mean random vector

which depends linearly on the covariance matrix . As with Gaussian curvature, the units of the sectional and Riemannian curvatures and are the reciprocal

TLFeBOOK

grad

(46)

1618

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

is positive semi-definite, where grad is the gradient8 of with respect to the FIM . Readers unfamiliar with technicalities in the proof below are encouraged to prove the theorem in Euclidean space using the fact that . As usual, the matrix inequality is said to hold for positive semi-definite matrices and if , i.e., is positive semi-definite, or for all vectors . Proof: The proof is a direct application of Lemma 1 to the computation of the covariance of the random tangent vector grad where grad is the gradient of (see footnote 8). Denoting is given by

(51)

with respect to the FIM by , the covariance of

Cov

8The gradient of a function ` is defined to be the unique tangent vector T M such that g (grad `; ) = d`( ) for all tangent vectors . grad ` With respect to particular coordinates



2



G

grad ` =



(47)

d`

g (@`=@ ), where g is the i.e., the ith element of grad ` is (grad `) = . Multiplication of any tensor by g (summation ij th element of the matrix implied) is called “raising an index,” e.g., if A = @`=@ are the coefficients g A are the coefficients of grad `. The of the differential d`, then A = presence of in the gradient accounts for its appearance in the CRB. The process of inverting a Riemannian metric is clearly important for computing T M , there is a corresponding the CRB. Given the metric g : T M naturally defined by the equation tensor g : T M T M

(52) , and the covariance of , The mean of vanishes, is positive semi-definite, which establishes the first part of the theorem. Expanding into a Taylor series about the zero matrix and computing the first-order term establishes the second part. The third part is trivial. For applications, the intrinsic FIM and CRB are computed as described in Table I. The significance of the sectional and Riemannian curvature terms is an open question that depends upon the specific application; however, as noted earlier, these terms become negligible for small errors and biases. Assuming that the inverse FIM has units of beamwidth SNR for some beamwidth, as is typical, dimensional analysis of (44) shows that the Riemannian curvature appears in the SNR term of the CRB. Several corollaries follow immediately from Theorem 2. Note that the tensor product of a tangent vector with itself is equivalent to the outer product given a particular choice of coordinates [cf. (9) and (10) for cotangent vectors]. Corollary 1: The second-order moment of is given by

G

G

2

2

!

g

(

;

!

; )

(53) (viewed as a matrix with respect to given coordinates), satisfies the matrix inequality

(48)

) = g(



Cov

2

T M is defined for all tangent vectors , where the cotangent vector T M (see footnotes 2 and 4 by the equation g ( ; X ) = (X ) for all X for the definition of the cotangent space T M ). The coefficients of the metric g and the inverse metric g with respect to a specific basis are computed as follows. Given an arbitrary basis (@=@ ); (@=@ ); . . . ; (@=@ ) of the tangent space T M and the corresponding dual basis d ; d ; . . . ; d of the cotangent space T M such that d (@=@ ) =  (Kronecker delta), we have



2

g

=g

g

=g

@ @

;

@ @

(d ; d ):

, which

(49) (50)

(54) and relative to for sufficiently small . Neglecting the sectional and Riemannian curvature terms and at small errors and biases

g g g = g (tautologically, raising both indices of g ), and Then, the coefficients g of the inverse metric express the CRB with respect to this basis.

TLFeBOOK

(55)

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

1619

If

TABLE I

is unbiased, then it achieves the CRB if and only if grad

(59)

Thus the concept of estimator efficiency [26], [60], [78] depends upon the choice of affine connection, or, equivalently, the choice of geodesics on , as evidenced by the appearance of the operator and curvature terms in (58). Corollary 4: The variance of the estimate of the th coordinate is given by the th diagonal element of the matrix [plus the first-order term in (44) for larger errors]. If is unbiased, then the variance of the estimate of the th coordinate is given by the th diagonal element of the inverse of the FIM (plus the first order term in (44) for larger errors). Many applications require bounds not on some underlying paof that space to another rameter space but on a mapping manifold. The following theorem provides the generalization of the classical result. The notation from footnote 2 is used to designate the push-forward of a tangent vector (e.g., the Jacobian matrix of in ). , Theorem 3 (Differentiable Mappings): Let , , and be as in Theorem 2, and let be a differentiable mapping from to the -dimensional manifold . Given arbitrary bases , and of the tangent spaces and , respectively (or, equivalently, arbitrary coordinates), for any unbiased estimator and its mapping (60)

is the these basis vectors.

by

is the FIM, and the push-forward Jacobian matrix with respect to

Equation (60) in Theorem 3 may be equated with the change-ofvariables formula of (11) (after taking a matrix inverse) only when is one-to-one and onto. In general, neither the inverse function nor its Jacobian exist. Corollary 2: Assume that is an unbiased estimator, i.e., (the zero vector field). Then, the covariance of the estimation error satisfies the inequality Cov Neglecting the Riemannian curvature terms errors, the estimation error satisfies Cov Corollary 3 (Efficiency): The estimator Cramér–Rao bound if and only if

(56) at small

(57) achieves the

grad

III. SAMPLE COVARIANCE MATRIX ESTIMATION In this section, the well-known problem of covariance matrix estimation is considered, utilizing insights from the previous section. The intrinsic geometry of the set of covariance matrices is used to determine the bias and efficiency of the SCM. be an by matrix whose Let columns are independent and identically distributed (iid) zeromean complex Gaussian random vectors with covariance ma(also see Diaconis [18, p. trix 110, ch. 6D and 6E]; the technical details of this identification in the Hermitian case involve the third isomorphism the). orem for the normal subgroup of matrices of the form The pdf of is , where is the SCM. The log-likelihood of this function is (ignoring constants)

(58)

TLFeBOOK

(61)

1620

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

We wish to compute the CRBs on the covariance matrix and examine in which sense the SCM (the maximum likelihood estimate) achieves these bounds. By Theorem 1, we may extract in to compute the FIM. the second-order terms of These terms follow immediately from the Taylor series

the matrix exponential equivalent to the representation is a tangent vector at [ responding to [ coloring transformation

. This is , where for unit vectors] corfor unit vectors] by the (68)

(62) (63) where and are arbitrary Hermitian matrices. It follows that the FIM for is given by (64) that is, the Fisher information metric for Gaussian covariance matrix estimation is simply the natural Riemannian metric given in (4) (scaled by , i.e., ); this on also corresponds to the natural metric on the symmetric cone [25]. Given the central limit theorem and the invariance of this metric, this result is not too surprising. A. Natural Covariance Metric CRB The formula for distances using the natural metric on of (4) is given by the 2-norm of the vector of logarithms of the generalized eigenvalues between two positive-definite matrices, i.e.,

The appearance of the matrix exponential in (67), and in (120) of Section IV-C for subspace geodesics, explains the “exp” notation for geodesics described in footnote 6 [see also (69)]. From the expression for geodesics in (67), the inverse exponential map is (69) (unique matrix square root and logarithm of positive-definite Hermitian matrices). Because Cramér–Rao analysis provides tight estimation bounds at high SNRs, the explicit use of these geodesic formulas is not typically required; however, the metric of (65) corresponding to this formula is useful to measure distances between covariance matrices. In the simple case of covariance matrices, the FIM and its inverse may be expressed in closed form. To compute CRBs for covariance matrices, the following orthonormal basis vectors (Hermitian matrices in this section) for the tangent space of at are necessary: an

and

(see footnote 6). A geodesic emanating from has the form

in the direction

element is is There are total of

(70)

th elements are both

zeros elsewhere an by Hermitian matrix whose

where

(66)

symmetric matrix whose th

diagonal element is unity, zeros elsewhere an by symmetric matrix whose th

(65) are the generalized eigenvalues of the pencil or, equivalently, . If multiplied , this distance between two covariance maby trices is measured in decibels, i.e., dB ; using Matlab notation, it . This distance is expressed as corresponds to the formula for the Fisher information metric for the multivariate normal distribution [57], [64]. The manifold with its natural invariant metric is not flat simply because, inter alia, it is not a vector space, its Riemannian metric is not constant, its geodesics are not straight lines [(67)], and its Christoffel symbols are nonzero [see (66)]. on the covariance matrices Geodesics corresponding to its natural metric of (4) satisfy the geodesic , where the Christoffel symbols equation are given by the quadratic form

by

(71) th

and th element zeros elsewhere

(72)

real parameters along the diagonal of plus real parameters in the off-diagonals, for a parameters. For example, the 2 by 2 Hermitian ma-

trix

is decomposed using four orthonormal

basis vectors as

(73) and

is

therefore represented by the real 4-vector . To obtain an orthonormal basis for at , color the basis vectors by prethe tangent space of and post-multiplying by as in (68), i.e., (74)

(67) is the unique positive-definite symmetric (Hermiwhere tian) square root of , and “exp” without a subscript denotes

where the superscript “ Hermitian basis vectors

TLFeBOOK

” denotes a flag for using either the or the symmetric basis vectors

; for notational convenience,

is implied.

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

1621

Equation (84) follows from the consequence of (65) and (67): The natural distance between covariance matrices along geodesics is given by

With respect to this basis, (64) yields metric coefficients (75) . (Kronecker delta), that is, A closed-form inversion formula for the covariance FIM is easily obtained via (48) in footnote 8. The fact that the Hermitian matrices are self-dual is used, i.e., if is a linear mapping from the Hermitian matrices to , then may also be represented as a Hermitian matrix itself via the definition

which is simply the Frobenius norm of the whitened matrix (see also see footnote 6). Therefore, the CRB of the natural distance is given by

(76)

(87)

for all Hermitian matrices . The inverse of the Fisher information metric (see footnote 8) is defined by

(86)

where

is computed with respect to the coefficients and . Either by (75) or by (80) and (81)

(77) for all

, where

and

are related by the equation for all . Clearly, from (64) (78) (79)

Applying (77) and (79) to (64) yields the formula (80) To compute the coefficients of the inverse FIM with respect to the basis vectors of (74), the dual basis vectors (81)

(88) , which establishes the theorem. The real symmetric for [ real pacase follows because real parameters along the diagonal of plus rameters in the off-diagonals; the additional factor of 2 comes from the real Gaussian pdf of (110)]. B. Flat Covariance Metric CRB The flat metric on the space covariance matrices expressed using the Frobenius norm is oftentimes encountered:

are used; note that . We now have sufficient machinery to prove the following theorem.

Theorem 5: The CRB on the flat distance [see (89)] between any unbiased covariance matrix estimator and is

Theorem 4: The CRB on the natural distance [see (65)] between and any unbiased covariance matrix estimator of is

(90)

Hermitian case real symmetric case

(82) (83)

is the root mean square error, where has been neglected. and the Riemannian curvature term To convert this distance to decibels, multiply by . Proof: The square error of the covariance measured using the natural distance of (65) is (84) where each of the in (74), and

(89)

is the root (Hermitian case), where mean square error (flat distance), and denotes the th element of . In the real symmetric case, the scale factor in the . The flat numerator of (90) is . and natural CRBs of Theorem 4 coincide when Proof: The proof recapitulates that of Theorem 4. For the flat metric of (89), bounds are required on the individual coeffifor the diagonals cients of the covariance matrix itself, i.e., and for the real and imaginary off-diagonals. of Using these parameters (91)

are the coefficients of the basis vectors are the coefficients of the

orthonormal basis vectors

in the vector decomposition

where the covariance

(85)

TLFeBOOK

is decomposed as the linear sum (92)

1622

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

With respect to the orthonormal basis vectors (70)–(72), the coefficients of the inverse FIM

and are

of

(93) The CRB on the flat distance is (94) A straightforward computation shows that (95) (96) (97) establishing the theorem upon summing over

.

C. Efficiency and Bias of the Sample Covariance Matrix We now have established the tools with which to examine the intrinsic bias and efficiency of the sample covariance matrix in the sense of (23) and Corollary 3. Obviously, , but this linear expectation operation means the inte, which treats the covariance matrices as a gral ( convex cone [25] included in the vector space for the real, symmetric case; a convex cone is a subset of a vector space that is closed under addition and multiplication by positive real numbers). Instead of standard linear expectation valid is interpreted intrinsifor vector spaces, the expectation of for various cally as choices of geodesics on , as in (22). First, observe from the first-order terms of (62) and (63) that (98) establishing that maximizes the likelihood. Because the SCM is the maximum likelihood estimate, it is asymptotically efficient [17], [78] and independent of the geodesics chosen for . From the definition of the gradient (see footnote 8), grad for all ; therefore, with respect to the FIM grad

(99)

is The well-known case of the flat metric/connection on examined first. Flat distances between and are given by the as in (89), and the geodesic between Frobenius norm , . these covariance matrices is . Obviously for the flat connection,

Corollary 3, an unbiased estimator is efficient if grad , which, by (99), is true for the flat metric. The flat metric on has several undesirable practical properties. It is well known that the SCM is not a very good estifor small sample sizes (which leads to ad hoc mate of techniques such as diagonal loading [67] and reference prior methods [81]), but this dependence of estimation quality on sample support is not fully revealed in Theorem 6, which ensures that the SCM always achieves the best accuracy possible. In addition, in many applications treating the space of covariance matrices as a vector space may lead to degraded algorithm performance, especially when a projection onto submanifolds (structured covariance matrices) is desired [8]. The flat of metric has undesirable geometric properties as well. Because the positive-definite Hermitian matrices are convex, every point on straight-line geodesics is also a positive-definite Hermitian matrix. Nevertheless, these paths may not be extended indefi. Therefore, the space endowed with the nitely to all flat metric is not geodesically complete, i.e., it is not a complete metric space. Furthermore, and much more significantly for applications, the flat connection is not invariant to the group acof on the positive-definite Hermition . tian matrices, i.e., Therefore, the CRB depends on the underlying covariance matrix , as seen in Theorem 5. in (65) has none of these In contrast, the natural metric on defects and reveals some remarkable properties about the SCM, as well as yielding root mean square errors that are consistent with the dependence on sample size observed in theory and practice (see Fig. 3). The natural distance RMSE varies with the sample size relative to the CRB, unlike the flat distance RMSE, whose corresponding estimator is efficient and, therefore, equals its Cramér–Rao lower bound at all sample sizes. Furthermore, the natural metric is invariant to the group action ; therefore, it yields bounds that are independent of of the underlying covariance matrix . In addition, endowed with this metric is a geodesically complete space. Because the [see (65)] has the properties natural Riemannian metric on of accounting for the change in estimation quality of the SCM as the sample size varies, being invariant to the group action of and therefore independent of , and yielding a complete metric space for , it is recommended for the analysis of covariance matrix estimation problems. Theorem 7: The sample covariance matrix estimator with respect to the natural metric on is biased and not efficient. The bias vector field and expectation of with respect to are (100) (101) where

Theorem 6: The sample covariance matrix is an unbiased and efficient estimator of the covariance with respect to the flat metric on the space of covariance matrices . for the flat Proof: Trivially, metric; therefore, is unbiased with respect to this metric. By

(102) (Hermitian case), and tion. In the real symmetric case,

TLFeBOOK

is the digamma func-

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

1623

at the integers yields of (102). Therefore, , and thus, , establishing the first part of the theorem. The proof of the real case is left as an exercise. must be comTo prove that the SCM is not efficient, puted. Using the description of the covariant derivative in footnote 5, for any vector field along the . For the bias vector field, geodesic , and . Applying the Christoffel symbols of (66), . This is true for , and the bias of the SCM is arbitrary ; therefore, parallel. This fact can also be shown using the explicit formula and (15): for parallel translation on Fig. 3. CRB and RMSE of the distance on P (in decibels, Hermitian case) of the SCM ^ from versus sample support. A Monte Carlo simulation is used with 1000 trials and Wishart distributed SCMs. itself is chosen randomly from a Wishart distribution. The unbiased natural CRB [see (82), dotted cyan] is shown below the biased natural CRB [see (105), solid red], which is itself below the unbiased flat CRB [see (90), solid magenta]. The RMSEs of the natural and flat (Frobenius) distances of (65) and (89) are the dashed (blue below green) curves (respectively). Because the SCM w.r.t. the flat metric is an efficient estimator, the RMSE lies on top of its CRB; however, unlike the (inefficient) natural distance, the (efficient) flat distance does not accurately reflect the varying quality of the covariance matrix estimates as the sample size varies, as does the natural distance [Theorem 7 and Corollary 5]. The SCM w.r.t. the natural metric is biased and inefficient and, therefore, does not achieve the biased natural CRB. Nevertheless, the maximum likelihood SCM is always asymptotically efficient. The SCM’s Riemannian curvature terms, which become significant for covariance errors on the order of 8.7 dB, have been neglected.

R

R

R

. Furthermore, this bias is parallel, and the matrix of sectional curvature terms vanishes, i.e., and

(103)

Proof: From the definition of the bias vector in (23) and the inverse exponential in (69), , is the whitened SCM. where Therefore, the bias of the SCM is given by the col. The whitened SCM ored expectation of has the complex Wishart distribution CW . , Using the eigenvalue decomposition of , where is an arbitrary column of : the last equality following from the independence of eigenvalues and eigenvectors [47]. For the eigenvector part, because for a complex normal vector (zero mean, unit variance). For the eigenvalue part, , and the is the same as : distribution of the product of independent complex chi-squared random ; Muirhead’s Thevariables [ orem 3.2.15 [47] contains the real case]. Therefore, , where is the digamma function. Applying standard identities for evaluated

(104) The proof that the SCM is not efficient is completed by obof serving that (58) in Corollary 3 does not hold for (69), of (100), and grad of (99). Finally, computing defined in (33) at gives the matrix , because the formula for the sectional cur[see (28) of vature of , footnote 7] vanishes trivially for . By invarifor all . ance, It is interesting to note that both and vanish (conveniently) because the SCMs bias vector is tangent to the 1-D Euclidean part of the symmetric space decomposition [31, ch. 5, , where the sec. 4] part represents the determinant of the covariance matrix. That is, the SCMs determinant is biased, but it is otherwise unbiased. Theorem 7 is of central importance because it quantifies the estimation quality of the sample covariance matrix, especially at low sample support. Corollary 5: The CRB of the natural distance of the SCM estimate from is (105) is the mean (Hermitian case), where is defined in (102), and the Riemannian square error, curvature term has been neglected. Proof: The proof is an application of (54) of Corollary is parallel, the bound on is 1. Because , where the tensor product given by is interpreted to be the outer product over the -dimensional vector space and not the Kronecker product of matrices. The first part of this sum is established in (82) of Theorem 4 and the second part by the fact that the trace of the outer product of two vectors equals their inner product: . An expression for the Riemannian curvature term for the SCM, which becomes significant for errors of order dB, is possible using the sectional curvature of given in (28) of

TLFeBOOK

1624

footnote 7. At and let

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

, let be a Hermitian matrix. Then

,

(106) where that

is the Lie bracket. It can be shown if and

IV. SUBSPACE AND COVARIANCE ESTIMATION ACCURACY We now return to the analysis of estimating an unknown subspace with an underlying unknown covariance matrix given in (1). The parameter space is the product manifold , with the covariance matrix being an unknown nuisance parameter. The dimension of this product manifold is the sum and , which equals of the dimensions of in the real case and in the proper complex case. We will focus on the real case in this section; the proper complex case is a trivial extension. A. Subspace and Covariance Probability Model

if . A connection can also be made with asymptotic results for the eigenvalue spectrum of the SCM. The largest and smallest and of the whitened SCM are apeigenvalues proximated by

be a real by matrix whose Let columns are iid random vectors defined by (1). The joint pdf of this data matrix is

(107)

(111)

( large, ) [36], [62]. For , the extreme specdiffer considerably from unity, even though tral values of for all , and this estimator is efficient with respect to the flat connection. Indeed, by the deformed quarter-circle law [36], [42], [63] for the asymptotic distribution of the spec(large ), the mean square of the natural distance of trum of from is

(108) , where . For large and large sample support (i.e., small ), the SCMs (biased) CRB in (105) has the asymptotic expansion (109) whose linear term in coincides exactly with that of (108) because the SCM is asymptotically efficient. The SCM is not efficient for (finite sample support), and the quadratic terms of (108) and (109) differ, with the being strictly less than CRB’s quadratic term the mean square error’s quadratic term ; adding the SCM’s Riemannian curvature terms from (108) adds , resulting in a term of . We note in passing the similarity between the first-order term in of this CRB and the well-known Reed–Mallett–Brennan estimation loss for adaptive detectors [58], [67].

(110)

where is the sample covariance matrix, and and represent the unthe by and by matrices known subspace and covariance, respectively. The log-likelihood of this function is (ignoring constants) (112) The maximum likelihood estimate of the subspace is simply the -dimensional principal invariant subspace of the whitened . Indeed, a straightforward compuSCM tation involving the first-order terms of (62) and (63) establishes results in the invariant that solving the equation subspace equation [23]

for , such that . Choosing to be the invariant maxisubspace corresponding to the largest eigenvalues of mizes the log-likelihood. B. Natural Geometry of Subspaces and Closed-form expressions for CRBs on the parameters are obtained if points on the Grassmann manifold are represented by by matrices such that (113) and post-multiplication of any such matrix by an orthogonal by matrix represents the same point in the equivalence class. This constraint is a colored version of the convenient represenby matrices with orthonormal columns tation of points on [1], [23] and simply amounts to the whitening transformation (114) where . Given such a matrix with orthonormal satisfying (113), tangent columns or, equivalently,

TLFeBOOK

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

vectors to the Grassmann manifold are represented by by matrices or (colored case) of the form whitened case

(115)

colored case

(116)

and or, equivalently, such that is an arbitrary by matrix with orthonormal that and , columns such that , and is an arbitrary by matrix [23]. Cotan(whitened case) are also represented by gent vectors in by matrices with the identification (117)

, where the Christoffel symbols equation are given by the quadratic for the Grassmann manifold form [23] (122) The inverse exponential map is given by , where these matrices are computed with the compact SVDs (singular values in ascending order) (singular values and , in descending order), and the formulas (principal values), and . Then, ; note that and that . C. Subspace and Covariance FIM and CRB

. In the colored case, ; therefore, . As usual, dual

, and vectors are whitened contravariantly. The CRB for the unknown subspace will be computed using given by [23] the natural metric on

Theorem 1 ensures that we may extract the second-order in and to compute the terms of FIM. The Fisher information metric for the subspace plus covariance estimation problem of (1) is obtained by using the second-order terms of the Taylor series given in (62) and (63), along with (16) and the perturbation

(118)

(123)

for all

such that

1625

Distances corresponding to this metric are give by the 2-norm of the vector of principal angles between two subspaces, i.e.,

where the first- and second-order terms are (124) (125)

(119) , where are the singular values of . This distance between two subspaces is the matrix measured in radians; in Matlab notation, it is expressed as . There are several other subspace distances that are defined by embeddings of the Grassmann manifold in higher dimensional manifolds [23, p. 337]. Both the arccosine implicit in (119) and the logarithm for in (65) correspond to the inverse of the the natural metric on ” discussed in Section II-B. exponential map “ Applying (115) to (118) shows that the natural distance between two nearby subspaces is given by the Frobenius norm of . Therefore, the CRB of the the matrix natural subspace distance is given by the FIM with respect to the elements of . This fact is made rigorous by the following corobservations. Geodesics on the Grassmann manifold responding to its natural metric are given by the formula [23] (120) is the compact SVD of the tangent vector where at ( , ). For the proper complex case, the transpositions in (120) may be replaced with conjugate transpositions. Furthermore, it may be verified that for the case of geodesics provided in (120) (121) [see also (86) for covariance matrices and footnote 6 for the general case]. Geodesics for subspaces satisfy the differential

The resulting FIM is given by the quadratic form

(126) Only the first-order terms of (123) appear, and (126) is consistent with the well-known case of stochastic CRBs [60]: for the param. Applying the Woodbury formula eters (127) to (126) and the constraint of (113) yields the simplified FIM for the covariance and subspace estimation problem:

(128) In the proper complex case, (128) may be modified by removing the factor of 1/2 that appears in the front and replacing the transpose operator with the Hermitian (conjugate transpose) operator. There are no cross terms between the covariance and subspace components of the joint FIM; therefore, there is no estimation loss on the subspace in this example. CRBs on the covariance and subspace parameters are ob(as described in tained by computing the inverse metric footnote 8). This inverse is given by the equation

TLFeBOOK

(129)

1626

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

where the cotangent vectors and (colored) are defined by the equation for all tangent vectors and (colored). Using (76), (117), and (128) to solve for and as functions of and yields

matrix that represents the cross terms, which vanish in this exof (128), the ample. Using the Fisher information metric coefficients of these blocks are (138) (139) (140)

(130) (131) Finally, these expressions for and may be inserted into (129) to obtain the inverse Fisher information metric

Note that a basis for the tangent space of the product manifold must be used, analogous to standard Cramér–Rao analysis with a nuisance parameter. The CRB for the subspace , accuracy is given by the lower right subspace block of which is expressed using the inverse of the Schur complement of as

(132)

(141) The bound on the subspace estimation accuracy is then given by the formula

D. Computation of the Subspace CRB The inverse Fisher information metric of (132) provides the CRB on the natural subspace distance between the true subspace and any unbiased estimate of it. Because this distance corresponds to the Frobenius norm of the elements of the matrix in (115), the FIM and inverse FIM will be computed with respect to using these elements and the basis of the tangent space of the corresponding dual basis of the cotangent space. Using classical Cramér–Rao terminology, we will compute a lower bound on the root mean square subspace error

(133)

(142) is the RMSE defined in (133), and the sectional and where and have been neRiemannian curvature terms glected. The invariance of this result to any particular basis for the tangent space of the covariance matrices may be seen by and into (141) for an arsubstituting bitrary invertible matrix , as in standard Cramér–Rao analysis. For problems in which the cross terms of (137) are nonzero, in (141) quantifies the loss in measurement accuracy associated with the necessity of estimating the nuisance parameters. Alternatively, the formula for the inverse FIM of (132) may : be used to compute

between an estimate of , where are the elements of . are The orthonormal basis vectors (whitened case) of (134) an arbitrary by matrix such that an by matrix whose th element is unity zeros elsewhere

(135)

(143) is the (colored) dual basis vector where of in (134). SNR (but unknown), where SNR In the specific case is the signal-to-noise ratio, the blocks of the FIM in (137) with respect to the basis vectors above simplify to

(136)

For convenience, we will also use the orthonormal basis vectors for the tangent space of defined in (74), although the invariance of the subspace accuracy to this choice of basis ensures that this choice is arbitrary. The full FIM for the subspace and nuisance covariance parameters is written conveniently in block form as

SNR SNR SNR SNR (144) As a result SNR

(137) is a square matrix of order representing the where covariance block, is a square matrix of order repby resenting the subspace block, and is a

SNR

rad

(145)

where the sectional and Riemannian curvature terms and have been neglected. For large SNR, this expression is SNR rad . well approximated by

TLFeBOOK

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

Fig. 4. RMSEs of the whitened and unwhitened SVD-based subspace estimator (Section IV-E) and the CRB of (143) versus SNR for the estimation and are chosen randomly from a Wishart problem of (1) on G . distribution, and ~ is chosen randomly from the uniform distribution on G [orth(randn(n; p)) in Matlab notation]. The RMSE of the unwhitened SVD estimate is the dashed (green) curve, which exceeds the RMSE of the whitened . Below SVD estimate (dashed blue curve) because of the bias induced by these curves is the CRB in the solid (red) curve. A constant sample support of 10 = 5p snapshots and 1000 Monte Carlo trials are used. The RMSEs of the SVD-based subspace estimators are a small constant fraction above the Cramér–Rao lower bound over all SNRs shown.

Y

R

R

R

E. SVD-Based Subspace Estimation Given an by data matrix whose columns are iid random vectors, the standard method [16], [28], [40] of estimating the -dimensional principal invariant subspace of is to compute the (compact) SVD (146) where is an by orthogonal matrix, is an by ordered is a by matrix with orthonormal diagonal matrix, and columns. The -dimensional principal invariant subspace of is taken to be the span of the first columns of . Furthermore, if an estimate is desired of the subspace represented by the matrix in (1) and the background noise cois nonwhite and known, the simple SVD-based esvariance timator using the data vectors is biased by the principal in. In this case, a whitened SVD-based apvariant subspace of of the proach is used, whereby the SVD is taken to be the whitened data matrix is computed, then . As noted, this is the first columns of , and maximum likelihood estimate and is therefore asymptotically [17], [78]. efficient as

1627

Fig. 5. RMSEs of the whitened and unwhitened subspace estimators (Section IV-E) and the CRB of (143) versus sample support (divided by p) for the estimation problem of (1) on G . The RMSE of the unwhitened SVD estimate is the dashed (green) curve, which exceeds the RMSE of the , whitened SVD estimate (dashed blue curve) because of the bias induced by especially at large sample support. Below these curves is the CRB in the solid (red) curve. A constant SNR of 21 dB and 1000 Monte Carlo trials are used. Note that, as expected, the RMSE of the whitened SVD estimate approaches the Cramér–Rao lower bound as the sample support becomes large, i.e., this maximum likelihood estimator is asymptotically efficient.

R

, , , and the number of indethat may be varied). Once pendent snapshots are specified, the CRB is computed from these values as described in Section IV-C. One thousand (1000) Monte Carlo trials are then performed, each of which consists data matrix whose covariof computing a normal by , estimating from the -dimensional principal inance is variant subspace of and the whitened data matrix and then computing the natural subspace distance between these (using the 2-norm of the vector of principal estimates and angles, in radians). The results comparing the accuracy of the whitened and unwhitened SVD estimators to the CRB are shown in Figs. 4 and 5 as both the SNR and the sample support vary. As the SNR is varied, the SVD-based method achieves an accuracy that is a small constant fraction above the Cramér–Rao lower bound. Because the unwhitened SVD estimator is biased , its RMSE error is higher than the whitened SVD estiby mator, especially at higher sample support. As the sample support is varied, the accuracy of the SVD-based method asymptotically approaches the lower bound, i.e., the SVD method is asymptotically efficient. We are reminded that Table II lists differential geometric objects and their more familiar counterparts . in Euclidean -space

F. Simulation Results

V. CONCLUSIONS

A Monte Carlo simulation was implemented to compare the subspace estimation accuracy achieved by the SVD-based estimation methods described in Section IV-E with the CRB pre(chosen randicted in Section IV-C. A 2-D subspace in domly from the uniform distribution on ) is estimated given and an unknown 2 by 2 a known 5 by 5 noise covariance (chosen randomly from a Wishart districovariance matrix SNR , where SNR is a signal-to-noise ratio bution,

Covariance matrix and subspace estimation are but two examples of estimation problems on manifolds where no set of intrinsic coordinates exists. In this paper, biased and unbiased intrinsic CRBs are derived, along with several of their properties, with a view toward signal processing and related applications. Of specific applicability is an expression for the Fisher information metric that involves the second covariant differential of the log-likelihood function given an arbitrary affine connection, or,

TLFeBOOK

1628

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE II

equivalently, arbitrary geodesics. These general results are then applied to the specific examples of estimating a covariance matrix in the manifold of positive-definite matrices and estimating a subspace in the Grassmann manifold in the presence of an unknown covariance matrix in . The CRB of an unbiased covariance matrix estimator using both the natural and flat (Frobenius) covariance metrics is derived and shown for the natural metric to be dB ( by Hermitian case with sample support ). It is well known that with respect to the flat metric, the sample covariance matrix is an unbiased and efficient estimator, but this metric does not quantify the extra loss in estimation accuracy observed at low sample support. Remarkably, the sample covariance matrix is biased and not efficient with respect to the natural invariant metric on , and the SCM’s bias reveals the extra loss of estimation accuracy at low sample support observed in theory and practice. For this and other geometric reasons (completeness, invariance), the natural invariant metric for the covariance matrices is recommended over the flat metric for analysis of covariance matrix estimation.

The CRB of the subspace estimation problem is computed in closed form and compared with the SVD-based method of computing the principal invariant subspace of a data matrix. In the simplest case, the CRB on subspace estimation accuracy is shown to be about SNR rad for -dimensional subspaces. By varying the SNR of the unknown subspace, the RMSE of the SVD-based subspace estimation method is seen to exceed the CRB by a small constant fraction. Furthermore, the SVD-based subspace estimator is confirmed to be asymptotically efficient, consistent with the fact that it is the maximum likelihood estimate. From these observations, we conclude that the principal invariant subspace can provide an excellent estimate of an unknown subspace. In addition to the examples of covariance matrices and subspaces, the theory and methods described in this paper are directly applicable to many other estimation problems on manifolds encountered in signal processing and other applications, such as computing accuracy bounds on rotation matrices, i.e., the orthogonal or unitary groups, and subspace basis vectors, i.e., the Stiefel manifold.

TLFeBOOK

SMITH: COVARIANCE, SUBSPACE, AND INTRINSIC CRAMÉR–RAO BOUNDS

Finally, several intriguing open questions are suggested by the results: What is the geometric significance of the fact that only the SCM’s determinant is biased? Does this fact, along with the numerical results in Section III-C, suggest improved covariance estimation techniques at low sample support? Would any such technique be preferable to the ad hoc but effective method of “diagonal loading”? Is the whitened SVD a biased subspace estimator? Does the geometry of the Grassmann manifold, akin to the SCM’s biased determinant, have any bearing on subspace bias? Are there important applications where the curvature terms appearing in the CRB are significant? Such questions illustrate the principle that admitting a problem’s geometry into its analysis not only offers a path to the problem’s solution but also opens new areas of study. ACKNOWLEDGMENT The author benefited enormously in writing this paper from conversations with A. Hero and L. Scharf and from excellent feedback from the anonymous reviewers. In particular, Dr. Scharf’s editorial and pedagogical suggestions greatly improved this paper. The author is also greatly indebted to J. Xavier, who helped him to identify an inaccuracy in the draft. REFERENCES [1] P.-A. Absil, R. Mahoney, and R. Sepulchre, “Riemannian geometry of Grassmann manifolds with a view on algorithmic computation,” Acta Applicandae Mathematicae, vol. 80, no. 2, pp. 199–220, Jan. 2004. [2] R. L. Adler, J. P. Dedieu, J. Y. Margulies, M. Martens, and M. Shub, “Newton’s method on Riemannian manifolds and a geometric model for the human spine,” IMA J. Numer. Anal., vol. 22, pp. 359–390, 2002. [3] S. Amari, “Differential-Geometrical Methods in Statistics,” in Lecture Notes in Statistics. Berlin, Germany: Springer-Verlag, 1985, vol. 28. , “Differential geometric theory of statistics,” in Differential Geom[4] etry in Statistical Inference, S. S. Gupta, Ed. Hayward, CA: Inst. Math. Statist., 1987, vol. 10, IMS Lecture Notes—Monograph Series. , Methods of Information Geometry. Providence, RI: Amer. [5] Math. Soc., 2000. Originally published in Japanese as Joho kika no hoho (Tokyo, Japan: Iwanami Shoten, 1993). [6] , “Information geometry on hierarchy of probability distributions,” IEEE Trans. Inf. Theory, vol. 47, no. 5, pp. 1701–1711, Jul. 2001. [7] W. J. Bangs, “Array Processing With Generalized Beamformers,” Ph.D. dissertation, Yale Univ., New Haven, CT, 1971. [8] T. A. Barton and S. T. Smith, “Structured covariance estimation for space-time adaptive processing,” in Proc. IEEE Conf. Acoust., Speech, Signal Process., Munich, Germany, 1997. [9] R. Bhattacharya and V. Patrangenaru, “Nonparametric estimation of location and dispersion on Riemannian manifolds,” J. Statist. Planning Inference, vol. 108, pp. 23–35, 2002. [10] W. M. Boothby, An Introduction to Differential Manifolds and Riemannian Geometry, Second ed. New York: Academic, 1986. [11] D. C. Brody and L. P. Hughston, “Statistical geometry in quantum mechanics,” Proc. R. Soc. Lond., vol. A 454, pp. 2445–2475, 1998. [12] I. Chavel, Riemannian Geometry—A Modern Introduction. Cambridge, U.K.: Cambridge Univ. Press, 1993. [13] J. Cheeger and D. G. Ebin, Comparison Theorems in Riemannian Geometry. Amsterdam, The Netherlands: North-Holland, 1975. [14] A. Cichocki and S. Amari, Adaptive Blind Signal and Image Processing. West Sussex, U.K.: Wiley, 2002. [15] A. Cichocki and P. Georgiev, “Blind source separation algorithms with matrix constraints,” IEICE Trans. Inf. Syst., vol. E86-A, no. 1, pp. 522–531, Jan. 2003. [16] P. Comon and G. H. Golub, “Tracking a few extreme singular values and vectors in signal processing,” Proc. IEEE, vol. 78, no. 8, pp. 1327–1343, Aug. 1990. [17] H. Cramér, Mathematical Methods of Statistics. Princeton, NJ: Princeton Univ. Press, 1946. [18] P. Diaconis, “Group Representations in Probability and Statistics,” in IMS Lecture Notes—Monograph Series, S. S. Gupta, Ed. Hayward, CA: Inst. Math. Statist., 1988, vol. 11.

1629

[19] S. C. Douglas, “Self-stabilized gradient algorithms for blind source separation with orthogonality constraints,” IEEE Trans. Neural Networks, vol. 11, no. 6, pp. 1490–1497, Nov. 2000. [20] S. C. Douglas, S. Amari, and S.-Y. Kung, “On gradient adaption with unit-norm constraints,” IEEE Trans. Signal Process., vol. 48, no. 6, pp. 1843–1847, Jun. 2000. [21] A. Edelman and S. T. Smith, “On conjugate-gradient-like methods for eigen-like problems,” BIT, vol. 36, no. 3, pp. 494–508, 1996. , Linear and Nonlinear Conjugate Gradient-Related Methods, L. [22] Adams and J. L. Nazareth, Eds. Philadelphia, PA: SIAM, 1996. [23] A. Edelman, T. A. Arias, and S. T. Smith, “The geometry of algorithms with orthogonality constraints,” SIAM Matrix Anal. Appl., vol. 20, no. 2, pp. 303–353, 1998. [24] K.-T. Fang and Y.-T. Zhang, Generalized Multivariate Analysis. Beijing, China: Science Press, 1990. [25] J. Faraut and A. Korányi, Analysis on Symmetric Cones. London, U.K.: Oxford Univ. Press, 1994. [26] R. A. Fisher, “On the mathematical foundations of theoretical statistics,” Phil. Trans. Roy. Soc., ser. A, vol. 222, pp. 309–368, 1922, [Online] http://www.library.adelaide.edu.au/digitised/fisher/18pt1.pdf. , “Dispersion on a sphere,” Proc. Roy. Soc. Lond., ser. A, vol. 217, [27] no. 1130, pp. 295–305, 1953. [28] G. H. Golub and C. Van Loan, Matrix Computations, Third ed. Baltimore, MD: Johns Hopkins Univ. Press, 1996. [29] J. D. Gorman and A. O. Hero, “Lower bounds for parametric estimation with constraints,” IEEE Trans. Inf. Theory, vol. 26, no. 6, pp. 1285–1301, Nov. 1990. [30] U. Grenander, M. Miller, and A. Srivastava, “Hilbert-Schmidt lower bounds for estimators on matrix Lie groups for ATR,” IEEE Trans. Pattern Anal. Machine Intell., vol. 20, no. 8, pp. 790–802, Aug. 1998. [31] S. Helgason, Differential Geometry, Lie Groups, and Symmetric Spaces. New York: Academic, 1978. , Groups and Geometric Analysis. New York: Academic, 1984. [32] [33] U. Helmke and J. B. Moore, Optimization and Dynamical Systems. Berlin, Germany: Springer-Verlag, 1994. [34] H. Hendriks, “A Cramér–Rao type lower bound for estimators with values on a manifold,” J. Multivariate Anal., vol. 38, pp. 245–261, 1991. [35] B. M. Hochwald and T. L. Marzetta, “Unitary space-time modulation for multiple-antenna communications in Rayleigh flat fading,” IEEE Trans. Inf. Theory, vol. 46, no. 2, pp. 543–564, Mar. 2002. [36] I. M. Johnstone, “On the distribution of the largest eigenvalue in principal components analysis,” Ann. Stat., vol. 29, no. 2, pp. 295–327, 2001. [37] H. Karcher, “Riemannian center of mass and mollifier smoothing,” Commun. Pure Appl. Math., vol. 30, pp. 509–541, 1977. [38] S. Kobayashi and K. Nomizu, Foundations of Differential Geometry. New York: Interscience, 1969, vol. 1 and 2. [39] J. T.-H. Lo and A. Willsky, “Estimation for rotation with one degree of freedom—Part I: Introduction and continuous time processes,” IEEE Trans. Autom. Control, vol. AC-20, no. 1, pp. 10–21, Feb. 1975. [40] E. Lundström and L. Eldén, “Adaptive eigenvalue computations using Newton’s method on the Grassmann manifold,” SIAM Matrix Anal. Appl., vol. 23, no. 3, pp. 819–839, 2002. [41] Y. Ma, J. Koˇsecká, and S. Sastry, “Optimization criteria and geometric algorithms for motion and structure estimation,” Intl. J. Comput. Vision, vol. 44, no. 3, pp. 219–249, 2001. [42] V. A. Marˇcenko and L. A. Pastur, “Distribution of eigenvalues for some sets of random matrices,” Math. USSR-Sb, vol. 1, pp. 457–483, 1967. [43] K. V. Mardia and P. E. Jupp, Directional Statistics. West Sussex, U.K.: Wiley, 2000. [44] P. McCullagh, Tensor Methods in Statistics. London, U.K.: Chapman and Hall, 1987. [45] T. McWhorter and L. L. Scharf, “Cramér–Rao bounds for deterministic modal analysis,” IEEE Trans. Signal Process., vol. 41, no. 5, pp. 1847–1866, May 1993. [46] M. L. Menéndez, D. Morales, L. Pardo, and M. Salicrü, “Statistical tests based on geodesic differences,” Appl. Math. Lett., vol. 8, no. 1, pp. 65–69, 1995. [47] R. J. Muirhead, Aspects of Multivariate Statistical Theory. New York: Wiley, 1982. [48] M. K. Murray and J. W. Rice, Differential Geometry and Statistics. London, U.K.: Chapman and Hall, 1993. [49] F. D. Neeser and J. L. Massey, “Proper complex random processes with applications to information theory,” IEEE Trans. Inf. Theory, vol. 39, no. 7, pp. 1293–1302, Jul. 1993. [50] A. Ohara, N. Suda, and S. Amari, “Dualistic differential geometry of positive definite matrices and its applications to related problems,” Lin. Alg. Appl., vol. 247, pp. 31–53, 1996.

TLFeBOOK

1630

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

[51] J. M. Oller, “On an intrinsic bias measure,” in Stability Problems for Stochastic Models, V. V. Kalashnikov and V. M. Zolotarev, Eds. Berlin, Germany: Springer-Verlag, 1991, Lecture Notes in Mathematics 1546, pp. 134–158. [52] J. M. Oller and J. M. Corcuera, “Intrinsic analysis of statistical estimation,” Ann. Stat., vol. 23, no. 5, pp. 1562–1581, Oct. 1995. [53] D. Petz, “Covariance and Fisher information in quantum mechanics,” J. Phys. A: Math. Gen., vol. 35, pp. 929–939, 1995. [54] K. Rahbar and J. P. Reilly, “Blind source separation algorithm for MIMO convolutive mixtures,” in Proc. Third Int. Conf. Independent Component Anal. Signal Separation, San Diego, CA, Dec. 2001, pp. 242–247. [55] C. R. Rao, “Information and the accuracy attainable in the estimation of statistical parameters,” Bull. Calcutta Math. Soc., vol. 37, pp. 81–89, 1945. , Selected Papers of C. R. Rao, S. Das Gupta, Ed. New York: [56] Wiley, 1994. , “Differential metrics in probability spaces,” in Differential Geom[57] etry in Statistical Inference, S. S. Gupta, Ed. Hayward, CA: Inst. Math. Statist., 1987, vol. 10, IMS Lecture Notes—Monograph Series. [58] I. S. Reed, J. D. Mallett, and L. E. Brennan, “Rapid convergence rate in adaptive arrays,” IEEE Trans. Aerosp. Electron. Syst., vol. AES-10, no. 6, pp. 853–863, 1974. [59] M. J. D. Rendas and J. M. F. Moura, “Ambiguity in radar and sonar,” IEEE Trans. Signal Process., vol. 46, no. 2, pp. 294–305, Feb. 1998. [60] L. L. Scharf, Statistical Signal Processing: Detection, Estimation, and Time Series Analysis. Reading, MA: Addison-Wesley, 1991. [61] L. L. Scharf and T. McWhorter, “Geometry of the Cramér–Rao bound,” Signal Process., vol. 31, pp. 301–311, 1993. [62] J. W. Silverstein, “The smallest eigenvalue of a large-dimensional Wishart matrix,” Ann. Prob., vol. 13, pp. 1364–1368, 1985. [63] J. W. Silverstein and Z. D. Bai, “On the empirical distribution of eigenvalues of a class of large dimensional random matrices,” J. Multivariate Anal., vol. 54, pp. 175–192, 1995. [64] L. T. Skovgaard, “A Riemannian geometry of the multivariate normal model.,” Scand. J. Stat., vol. 11, pp. 211–223, 1984. [65] S. T. Smith, “Dynamical systems that perform the singular value decomposition,” Syst. Control Lett., vol. 16, pp. 319–327, 1991. , “Subspace tracking with full rank updates,” in Proc. 31st Asilomar [66] Conf. Signals, Syst., Comput., vol. 1, 1997, pp. 793–797. , “Adaptive radar,” in Wiley Encyclopedia of Electrical and Elec[67] tronics Engineering, J. G. Webster, Ed. New York: Wiley, 1999, vol. 1, pp. 263–289. , “Intrinsic Cramér–Rao bounds and subspace estimation accu[68] racy,” in Proc. IEEE Sensor Array and Multichannel Signal Processing Workshop, Cambridge, MA, 2000, pp. 489–493. [69] M. Spivak, A Comprehensive Introduction to Differential Geometry, Third ed. Houston, TX: Publish or Perish, 1999, vol. 1, 2. [70] A. Srivastava, “A Bayesian approach to geometric subspace estimation,” IEEE Trans. Signal Process., vol. 48, no. 5, pp. 1390–1400, May 2002. [71] A. Srivastava and E. Klassen, “Monte Carlo extrinsic estimators for manifold-valued parameters,” IEEE Trans. Signal Process., vol. 50, no. 2, pp. 299–308, Feb. 2002. , “Bayesian, geometric subspace tracking,” Advances Appl. Prob., [72] to be published.

[73] P. Stoica and A. Nehorai, “MUSIC, maximum likelihood, and Cramér–Rao bound,” IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 5, pp. 720–741, May 1989. , “Performance study of conditional and unconditional direction-of[74] arrival estimation,” IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 10, pp. 1783–1795, Oct. 1990. [75] P. Stoica and B. C. Ng, “On the Cramér–Rao bound under parametric constraints,” IEEE Signal Process. Lett., vol. 5, no. 7, pp. 177–179, Jul. 1998. [76] P. Stoica and T. Söderström, “Statistical analysis of MUSIC and subspace rotation estimates of sinusoidal frequencies,” IEEE Trans. Signal Process., vol. 39, no. 8, pp. 1836–1847, Aug. 1991. [77] R. J. Vaccaro and Y. Ding, “Optimal subspace-based parameter estimation,” in Proc. IEEE Conf. Acoust., Speech, Signal Process., vol. 4, Minneapolis, MN, USA, 1993, pp. 368–371. [78] H. L. Van Trees, Detection, Estimation, and Modulation Theory. New York: Wiley, 1968, pt. 1. [79] J. Xavier, “Blind Identification of MIMO Channels Based on 2nd Order Statistics and Colored Inputs,” Ph.D. dissertation, Instituto Superior Técnico, Lisbon, Portugal, 2002. , “Geodesic lower bound for parametric estimation with con[80] straints,” in Proc. Fifth IEEE Workshop Signal Process. Adv. Wireless Commun., Lisbon, Portugal, Jul. 2004. [81] R. Yang and J. O. Berger, “Estimation of a covariance matrix using the reference prior,” Ann. Stat., vol. 22, no. 3, pp. 1195–1211, 1994. [82] L. Zheng and D. N. C. Tse, “Communication on the Grassmann manifold: a geometric approach to the noncoherent multiple-antenna channel,” IEEE Trans. Inf. Theory, vol. 48, no. 2, pp. 359–383, Feb. 2002.

Steven Thomas Smith (M’86–SM’04) was born in La Jolla, CA, in 1963. He received the B.A.Sc. degree in electrical engineering and mathematics from the University of British Columbia, Vancouver, BC, canada, in 1986 and the Ph.D. degree in applied mathematics from Harvard University, Cambridge, MA, in 1993. From 1986 to 1988, he was a research engineer at ERIM, Ann Arbor, MI, where he developed morphological image processing algorithms. He is currently a senior member of the technical staff at MIT Lincoln Laboratory, Lexington, MA, which he joined in 1993. His research involves algorithms for adaptive signal processing, detection, and tracking to enhance radar and sonar systems performance. He has taught signal processing courses at Harvard and for the IEEE. His most recent work addresses intrinsic estimation and superresolution bounds, mean and variance CFAR, advanced tracking methods, and space-time adaptive processing algorithms. Dr. Smith was an associate editor of the IEEE TRANSACTIONS ON SIGNAL PROCESSING from 2000 to 2002 and received the SIAM outstanding paper award in 2001.

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1631

Optimal Dimensionality Reduction of Sensor Data in Multisensor Estimation Fusion Yunmin Zhu, Enbin Song, Jie Zhou, and Zhisheng You

Abstract—When there exists the limitation of communication bandwidth between sensors and a fusion center, one needs to optimally precompress sensor outputs-sensor observations or estimates before the sensors’ transmission in order to obtain a constrained optimal estimation at the fusion center in terms of the linear minimum error variance criterion, or when an allowed performance loss constraint exists, one needs to design the minimum dimension of sensor data. This paper will answer the above questions by using the matrix decomposition, pseudo-inverse, and eigenvalue techniques. Index Terms—Linear compression, minimum variance estimation, multisensor estimation fusion.

I. INTRODUCTION ODERN estimation/tracking systems often involve multiple homogeneous or heterogeneous sensors that are spatially distributed to provide a large coverage, diverse viewing angles, or complementary information. If a central processor receives all measurement data from all sensors directly and processes them in real time, the corresponding processing of sensor data is known as the centralized estimation. This approach has several serious drawbacks, including poor survivability and reliability and heavy communications and computational burdens. The second strategy is two level optimization in the estimation fusion, called the distributed estimation fusion. Every sensor is also a subprocessor. It first optimally estimates the object based on its own observation in terms of a criterion and then transmits its estimate to the fusion center. Finally, the fusion center optimally estimates the object based on all received sensor estimates (see Fig. 1). Clearly, the second one has significant practical value, such as consuming less communication bandwidth, considering the local purpose and the global purpose simultaneously, and having more survivability in a war situation since the survived sensors in the distributed system can still estimate objects even if the fusion center is destroyed. In the two kinds of multisensor estimation fusion, when communication bandwidth between sensors and a fusion center is

M

Manuscript received September 28, 2003; revised May 1, 2004. This work was supported in part by the NSF of China under Grants 60374025 and 60328305 and the SRFDP under Grant 200330610018. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Yusef Altunbasak. Y. Zhu, E. Song, and J. Zhou are with Department of Mathematics, Sichuan University, Chengdu, Sichuan 610064, China (e-mail: [email protected]; [email protected]). Z. You is with Department of Computer Science, Sichuan University, Chengdu, Sichuan 610064, China (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845429

very limited, one needs to optimally precompress sensor outputs-sensor measurements or sensor estimates before the sensors’ transmission to obtain a constrained optimal estimation at the fusion center. On the other hand, another practically valuable issue is how to optimally design the minimum communications between sensors and the fusion center while an allowed maximum loss of system performance caused by the communication compression is given. In the existing literature, many research results were focused on the so-called sensor measurement fusion, where every sensor receives and transmits its own measurement to the fusion center with no preprocessing. Then, due to limited computer load and computation capacity, the fusion center must use available methods to compress sensor measurements first to make a real-time final estimate. All the previous research results in the measurement fusion problem considered how the fusion center linearly compresses sensor measurements to obtain the optimal estimate as the same as using all uncompressed sensor measurements (see [1]–[5]). Obviously, such results are true under some restrictive conditions and are only useful to reduce the computation load at the fusion center but not to reduce the communications between sensors and the fusion center. Therefore, they are not suitable to the situation where very limited communication bandwidth can be used in the systems. Even in the distributed estimation fusion, where every sensor is a sub-estimator, if the available communication bandwidth between sensors and the fusion center is still not enough to transmit sensor estimates, precompression of all sensor estimates before the sensors’ transmission is also necessary. Since communication bandwidth limitation is a bottleneck problem in multisensor techniques, there have been extensive works on the sensor data compression, for example, [6]–[13] among others. Particularly, in 1994, Luo and Tsitsiklis presented the first result [6] on the sensor data dimension reduction in the data fusion problem. In their work, they presented a minimal bound of the dimension of the compressed sensor data, which can still get the same estimation performance as using uncompressed sensor data. In the multisensor data fusion problems, one is concerned more about the optimal compression for any given dimension of the compressed sensor data due to the communication bandwidth limitation. To the authors’ knowledge, there has been no result on such optimal dimension reduction of the sensor data so far. In terms of criterion of the Linear Unbiased (error) Minimum Variance (LUMV),1 when only the covariance of measurements can be known, the above problems reduce to the maximum 1The well-known Kalman filtering can be derived under the LUMV criterion; for example, see [15].

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1632

Fig. 1.

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Distributed multisensor estimation fusion system. C (Y ) and C (^ ) stand for the compressed observation and estimation of the ith sensor, respectively.

problem of the trace of a nonlinear matrix function with matrix inverse or pseudo-inverse. When the covariance of measurements is invertible, using the matrix decomposition technique, we derive that the optimal compression matrix space is spanned by a part of eigenvectors of the covariances of sensor measurements. It is, in fact, an extended version of the Rayleigh Quotient (see Remark 2 below). If the covariance of measurements is not invertible and the row dimensions of the compression matrix is equal to the rank of the covariance of measurements, we can find a compression matrix so that the estimation error covariance using the compressed measurements is the same as that using the uncompressed measurements. When the row dimensions of the compression matrix are less than the rank of the covariance of measurements, we choose an optimal compression matrix so that the rank of the product of the compression matrix and the covariance of measurements is equal to the rank of the compression matrix, i.e., the column vectors of conjugate transpose of the compression matrix are in the range space of the covariance of measurements. Based on the above results, we can derive similar optimal compression matrix results for the multisensor estimation fusion system and the optimal compression matrix in terms of the Optimal Weighted Least Squares (OWLS) criterion. On the other hand, precompressing sensor outputs must lose system performance. We will give some measures for the performance loss caused by sensor compression and present the optimal sensor compression design under the performance loss constraints. This paper is organized as follows. The problem formulation is presented in Section II. Then, Section III is dedicated to the main result in this paper: the optimal compression matrix. In Section IV, several different constraints for the optimal compression matrix are given. The extension to the distributed es-

timation fusion is presented in Section V. A brief conclusion is given in the Section VI. II. PROBLEM FORMULATION Consider linear observation equations of an unknown common parameter , where the observed data is contaminated with noise, namely (1) where and are the unknown random parameter and the noise, respectively. and are, respectively, the known observation and the deterministic design/input, i.e., is the dimension of the estimated parameter , and is the dimension of the observation of the th sensor. Usually, is assumed to be zero-mean noise, i.e., and Var . The observation here is not necessarily the original sensor measurement. It may also be a precan be the estimate at processed sensor data. In particular, the th sensor. This model looks like a static model without a variation in time, but it is quite general, including the following dynamic system as a special case. Suppose the dynamic system with observers is given by (2) (3) where variables.

TLFeBOOK

is a full-rank matrix of order , , , , , and are zero-mean random

ZHU et al.: OPTIMAL DIMENSIONALITY REDUCTION OF SENSOR DATA

1633

To estimate the state with the information of all measureand reduce the above dyments from time 0 to namic system to (1), we introduce the following notations: .. .

.. .

In practice, when there exists the limitation of communication bandwidth between sensors and a fusion center or computation capacity of the fusion center, one needs to optimally precompress sensor outputs-sensor observations or estimates before the sensors’ transmission. Let the linear compression matrix be ..

(4) if if if

(5)

(6)

.. .

.. .

..

(7)

.

.

(14)

where matrix , , is a full row-rank matrix. . Using matrix to compress observaDenote tion , i.e., using matrix to compress observation at the th sensor, where is the compressed dimension of the observation of the th sensor, we have the compressed model (15) (16) , the covariance , , It is well known that when the mean are known, the LUMV estimate of and the cross-covariance using is given by

.. .

(17) (8) Using (4)–(8) (cf. [15, ch. 2, pp. 21–22] or see [17]), the above model can be written as

and the estimate error variance Var

(18)

where

(9) (10) where (11) Obviously, (10) is a specific version of (1). Remark 1: The multisensor estimation system (1) and its special version—the multisensor dynamic system (2) and (3)—are typical models in the multisensor estimation fusion problems, which were extensively discussed in the literature of this field (cf. [1]–[6] and [11]–[13]) and, indeed, are used in diverse advanced systems that make use of a large number of sensors in practical applications, ranging from aerospace and defense and robotics and automation systems to the monitoring and control of a process generation plants. In addition, there have been many works on the compression of the sensor data to reduce communications between the local sensors and the processing/fusion center as well as the computation load at the processing center; for example, see [6]–[13] among others. Denote

the superscripts “ ” and “ ” stand for the pseudo-inverse and conjugate transpose, respectively. More generally, the LUMV estimate of , using the given defined in (17) and (18), holds even without model (1). Similarly, the LUMV estimate of using the compressed is given by measurement (19) and the estimate error variance Var

(20)

can be given, and it and (or is of full If only column rank) are both nonsingular, the OWLS estimate for the linear model (13) is well known to be (21) and the estimate error variance Var

(12) Thus, we can rewrite (1) as

(22)

where is actually the solution of the following weight minimization problem:

(13)

TLFeBOOK

Var

(23)

1634

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 2.

Multisensor sensor data dimensionality reduction system.

The above solutions can be derived by using matrix Schwarz inequality (for example, see [15]). Hence, the estimate given in (21) is called the OWLS. Similarly, the OWLS estimate of using the compressed measurement is given by

Then, it is easily to see

(24) and the estimate error variance Var

(25)

where is assumed to be of full column-rank. The above formulations can be seen in Fig. 2. Now, a significant problem is how to derive optimal compression matrix in terms of some criteria. III. OPTIMAL COMPRESSION MATRIX

It will be proved in Theorem 1 below that the traces of the both matrices above are the maximum of the trace of but obviously not the solution of the maximum of the matrix problem. However, when there is a matrix, namely , such that tr attains a minimum of , there exists no other matrix, namely, , tr strictly smaller which can make the matrix than . Otherwise, by the matrix theory

A. Under LUMV Criterion

tr

tr

to minimize Naturally, one may wish to find an optimal the estimate error variance in (20). This can be reduced to the following maximizing problem:

On the other hand, if the matrix extremum, namely maximum solution exists, i.e., there is a matrix such that

(26)

(27)

where is a full row-rank matrix in , , and is a positive semi-definite or definite Hermite matrix. Unfortunately, it is not guaranteed that such a solution exists. See the following example. Example 1: Denote

then, by the matrix theory, we must have tr

(28)

Based on the above analysis, instead of criterion (26), when is invertible, we consider the following criterion: tr

(29)

is not invertible, the above criterion will not work. An If alternative criterion will be proposed in (44) below. To present the solution to (29), we have the following theorem.

TLFeBOOK

ZHU et al.: OPTIMAL DIMENSIONALITY REDUCTION OF SENSOR DATA

1635

Theorem 1: Suppose that the eigenvalues of are , the corresponding normal orthogonal . Then, we have eigenvectors are

Obviously, the maximum of the trace in (34) does not exist can be arbitrarily large. Naturally, instead of whenever (33), we may consider the following maximum problem: (35)

tr

(30)

tr

From (20), is certainly upper bounded. . We present the following lemma Let rank first. are Lemma 1: Supposing that the eigenvalues of , the corresponding normal orthogonal eigenvectors are . Denote , Diag . Then, we have

tr

(36)

Proof: Since is a full row-rank matrix in , we can , where is a full row-rank matrix in decompose . Then, decompose , where is a full rank square matrix, the row vectors of are normal orthogonal . Thus, we have vectors, i.e., tr

tr

(31)

Proof: By matrix decomposition, we know

Using a result on extremum of the sums of eigenvalues by Wielandt [18], we know the following maximum: (37) On the other hand, we have

tr It is easy to see that substituting yields . Hence tr

into (38)

tr

Remark 2: If we have

in Theorem 1, i.e.,

(32)

is a row vector,

tr

which is the minimum of the Rayleigh Quotient. Hence, Theorem 1 is an extension of the extremum of the Rayleigh Quotient. is not invertible, the following counterHowever, when part of the above objective function (29) tr

(33)

The lemma implies that when , the compression does not change the estimation error covariance; therefore, it is not necessary to consider the case of . .2 By the In the following, we only consider the case of matrix theory, we can decompose

(39) Clearly (40) Using (39), (40), and matrix pseudo-inverse theory, we have

is not available because the maximum of (33) does not exist. See the following example. . Example 2: Let

(41) which motivates us to consider the maximum problem

tr

(42)

(34)

2In

fact, Luo and Tsitsiklis’ work in [6] presented a smaller lower bound of

n that one needs to consider.

TLFeBOOK

1636

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

However, similar to the analysis around Example 1, we finally consider the following trace maximum problem: tr

Using Theorem 1 and the notations therein, we know that tr

(43) tr

is an orthogonal projection, its eigenSince values are either 0 or 1. Thus, the maximum problem (43) reduces to tr

rank rank rank (44)

Using the notation defined in Lemma 1, an optimal solution of (44) is simply given by (45) . , where . Summarizing the above analysis, we have the following theorem, which is more general than Theorem 1. Theorem 2: When is not invertible, we can consider the maximum problem (44), and its optimal solution is given simply by (45). Remark 3: In fact, Theorem 2 is a more general result than Theorem 1. From (44), when the dimension of compression maof measurement trix is less than the rank of the covariance , a more essential rule to verify the optimality of is to check is of full rank, or , where whether or not is the range of matrix . This is quite a mild condiis invertible, any of full row-rank is obviously tion. When an optimal compression matrix in the sense of maximizing the trace of the matrix in (44). Theorem 1, in fact, only provides a special solution. However, the above results are only suitable to single . In general, the matrix defined sensor case, i.e., in (14) is a special block-diagonal full row-rank matrix in since In fact,

(48) consist of the corresponding eigenand therefore, vectors of , as given in Theorem 1, respectively. are not invertible, using Theorem Furthermore, if some of 2, we can similarly derive the optimal compression matrices , as given in (45). is usually random, However, since the estimated are correlated generally, i.e., , for , and we cannot use Theorems 1 or 2 directly. some , where . By the partitioned Denote matrix technique, we have ..

Diag can be any

. Now, the problem is how to derive Diag the optimal solution for the matrix defined in (14). Denote ..

tr

(49)

rank

(50)

. We still only consider the case of rank Denote rank , and . Without loss of generality, suppose that (51) is the and maximum set of the linearly independent row vectors of . Thus, take

.. .

(52)

where , whose th element is a unit and others are zeros. It is easy to see that rank

rank

(46)

.

are mutually uncorrelated, i.e., are invertible, then

.. .

Using (44), the problem reduces to

rank If , and all

.. .

.

,

tr Diag (47)

(53)

It is worth noting that there are many maximum sets of the linearly independent row vectors of . Therefore, we should holds. If there exists no such a choose those so that to guarantee choice, we can adjust from 0 through rank because of rank rank .

TLFeBOOK

ZHU et al.: OPTIMAL DIMENSIONALITY REDUCTION OF SENSOR DATA

1637

B. Under OWLS Criterion Under the OWLS criterion, one wishes to find an optimal that minimizes the estimate error variance in (25). As in the analysis before, this can be reduced to the following maximizing problem: tr

are correlated, using the analysis of (49)–(53), If so it is easy to see that we can properly choose and , that , where for as many as possible, and rank

rank (56)

(54) C. Performance Loss Constraint

where is a full row-rank matrix in , . Obviously, must be greater than or equal to the dimension of ; otherwise, it is impossible for to be of full column rank. by there, the maximizing Using Remark 3 and replacing problem (54) further reduces to finding in order to have of full column rank because is always nonsingular. Thus, using the analogous techniques in (49)–(53), such an can be found easily even if defined in (14) is a special blockdiagonal full row-rank matrix in

.

IV. OPTIMAL SOLUTIONS UNDER DIFFERENT CONSTRAINTS In practical applications, one may have various communication constraints for a multisensor estimation system. The following results include the optimal measurement compression matrices under several different constraints. A. Fixed

for Each Sensor

, The optimal solutions were given in (48) when , and all are invertible. Otherwise, we will : find the set of the linearly independent row vectors of , where , so that they belong to a maximum set of the linearly independent row vectors of with as many “ ” as possible. In this way, under the assumption that rank , the rank will approach as closely as possible. B. Only the Sum of

Fixed

When one designs a multisensor system, the first issue is how many sensors and what communication pattern are needed according to the performance requirement. Obviously, measurement compression may reduce the system performance when rank . Using the results derived in Section III, rank we can show how to measure the compression performance losses and how to optimally choose sensors and the sensor compression matrices. Let us consider general measurement compression for models , and . Usually, there (13) and (16), where are two measures of the compression performance losses. The first one is the absolute loss tr Var

tr Var

where is given in (20) [or (25)], and is given in (18) [or (22)], whereas they are all computable, or simply under the assumption in Theorem 1 that tr

tr

or under the assumption in Theorem 2 that rank

rank

The second one is the relative loss

In practice, the performance of such a system with only fixed should be better than that defined in the sum of Section IV-A because the optimal solution can be chosen with more freedom as the analysis in the last paragraph of Section III. In this case, it is possible for some of the dimento be zero, i.e., the measurement of the th sensor sion of is removed. For example, if , , , are uncorrelated, are invert, 2, and 3, and , ible, , then the optimal solution for the optimization problem (48) is given by

tr Var

tr Var tr Var

while they are all computable, or simply under the assumption in Theorem 1 that tr

tr tr

or under the assumption in Theorem 2 that (55)

This implies that the second sensor is given up. Therefore, the communication pattern given in Section IV-B cannot only have better performance but also uses fewer sensors.

rank

rank rank

If a tolerance parameter is given, then we can optimally choose local sensors and their compression matrices under

TLFeBOOK

1638

constraint Section III.

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

or

by using the results derived in

V. EXTENSION TO THE DISTRIBUTED ESTIMATION FUSION When the available communication bandwidth between sensors and the fusion center is very limited, the precompression of all sensor estimates before sensors’ transmission is also necessary. In this case, model (1) will be replaced by (57) where is of full column-rank, and usually, , is an unbiased estimate of at the th sensor. Suppose that the covariance of all sensor estimate errors is .. .

..

.

.. .

(58)

where , which is known in many cases (for example, see [1]). Then, all results derived in Sections III and IV can be extended to the optimal distributed estimate compression for model (57). VI. CONCLUSIONS The optimal compression matrix for sensor measurements has been derived in this paper. Such choices are quite relaxed. The only condition is that rank rank if only the rank is considered (for the reason, see case of rank rank Lemma 1) in the LUMV problem or rank in the OWLS problem. Obviously, it is useful to use them to reduce the communications between sensors and the fusion center. Even in the distributed estimation fusion, when the available communication bandwidth between sensors and the fusion center is very limited, the precompression of all sensor estimates before the sensors’ transmission is also necessary. We have given two measures for the performance loss caused by the sensor compression and presented the optimal sensor compression design under the performance loss constraints. It is worth noting that the objective functions in this paper are the traces of and other than the traces and . of Therefore, further work is needed in this direction. ACKNOWLEDGMENT The authors are very grateful to the editors and referees for their many valuable and constructive comments on our manuscript. REFERENCES [1] Y. Bar-Shalom, “On the track-to-track correlation problem,” IEEE Trans. Autom. Control, vol. AC-26, no. 2, pp. 571–572, Apr. 1981. [2] Y. Bar-Shalom and X. R. Li, Multitarget-Multisensor Tracking: Principles and Techniques. Storrs, CT: YBS, 1995.

[3] D. Willner, C. B. Chang, and K. P. Dunn, “Kalman Filter Configurations for Multiple Radar Systems,” Lincoln Lab., Mass. Inst. Technol., Lexington, MA, Tech. Note 1976-21, 1976. [4] Y. Bar-Shalom, Ed., Multitarget-Multisensor Tracking: Advanced Applications. Norwood, MA: Artech House, 1990, vol. 1. [5] Y. Bar-Shalom, Ed., Multitarget-Multisensor Tracking: Advanced Applications. Norwood, MA: Artech House, 1992, vol. 2. [6] Z.-Q. Luo and J. N. Tsitsiklis, “Data fusion with minimal communication,” IEEE Trans. Inf. Theory, vol. 40, no. 5, pp. 1551–1553, Sep. 1994. [7] B. Aiazzi, L. Alparone, S. Baronti, and M. Selva, “Lossy compression of multispectral remote-sensing images through multiresolution data fusion techniques,” Proc. SPIE, vol. 4793, pp. 95–106, 2002. [8] Z.-U. Rahman, D. J. Jobson, G. A. Woodell, and G. D. Hines, “Multisensor fusion and enhancement using the retinex image enhancement algorithm,” Proc. SPIE, vol. 4736, pp. 36–44, 2002. [9] S. Jagannathan, P. Nagabhushan, U. N. Das, and V. Nalanda, “New multispectral fusion and image compression,” J. Spacecraft Technol., vol. 7, no. 2, pp. 32–42, 1997. [10] L. Rong, W.-C. Wang, M. Logan, and T. Donohue, “Multiplatform multisensor fusion with adaptive-rate data communication,” IEEE Trans.Aerosp. Electron. Syst., vol. 33, no. 1, pp. 274–281, Jan. 1997. [11] Y. M. Zhu, Multisensor Decision and Estimation Fusion. Boston, MA: Kluwer, 2003. [12] Y. M. Zhu and X. R. Li, “Best linear unbiased estimation fusion,” in Proc. Second Int. Inf. Fusion Conf., vol. 2, Sunnyvale, CA, Jul. 1999, pp. 1054–1061. [13] X. R. Li, Y. M. Zhu, J. Wang, and C. Z. Han, “Optimal linear estimation fusion–Part I: Unified fusion rules,” IEEE Trans. Inf. Theory, vol. 49, no. 9, pp. 2192–2208, Sep. 2003. [14] A. Ben-Israel and T. N. E. Greville, Generalized Inverses: Theory and Applications, Second ed. New York: Wiley, 2002. [15] C. K. Chui and G. Chen, Kalman Filtering With Real-Time Applications. New York: Springer-Verlag, 1987. [16] S. Grime and H. F. Durrant-Whyte, “Data fusion in decentralized sensor networks,” Control Eng. Practice, vol. 2, no. 5, pp. 849–863, 1994. [17] Y. M. Zhu, “Efficient recursive state estimator for dynamic systems without knowledge of noise covariances,” IEEE Trans. Aerosp. Electron. Syst., vol. 35, no. 1, pp. 102–114, Jan. 1999. [18] H. Wielandt, “An extremum property of sums of eigenvalues,” Proc. Amer. Math. Soc., vol. 6, pp. 106–110, 1955.

Yunmin Zhu received the B.S. degree from the Department of Mathematics and Mechanics, Beijing University, Beijing, China, in 1968. From 1968 to 1978, he was with the Luoyang Tractor Factory, Luoyang, Henan, China, as a steel worker and a machine engineer. From 1981 to 1994, he was with Institute of Mathematical Sciences, Chengdu Institute of Computer Applications, Academia Sinica, Chengdu, China. Since 1995, he has been with Department of Mathematics, Sichuan University, Chengdu, as a Professor. During 1986 to 1987, 1989 to 1990, 1993 to 1996, 1998 to 1999, and 2001 to 2002, he was a Visiting Associate or Visiting Professor at the Lefschetz Centre for Dynamical Systems and Division of Applied Mathematics, Brown University, Providence, RI; the Department of Electrical Engineering, McGill University, Montreal, QC, Canada; the Communications Research Laboratory, McMaster University, Hamilton, ON, Canada; and the Department of Electrical Engineering, University of New Orleans, New Orleans, LA. His research interests include stochastic approximations, adaptive filtering, other stochastic recursive algorithms and their applications in estimations, optimizations, and decisions for dynamic system as well as for signal processing and information compression. In particular, his present major interest is multisensor distributed estimation and decision fusion. He is the author or coauthor of over 60 papers in international and Chinese journals. He is the author of Multisensor Decision and Estimation Fusion (Boston, MA: Kluwer, 2002) and Multisensor Distributed Statistical Decision (Beijing, China: Science Press, Chinese Academy of Science, 2000), and coauthor (with Prof. H. F. Chen) of Stochastic Approximations (Shanghai, China: Shanghai Scientific and Technical Publishers, 1996). He is on the editorial board of the Journal of Control Theory and Applications, South China University of Technology, Guangzhou, China.

TLFeBOOK

ZHU et al.: OPTIMAL DIMENSIONALITY REDUCTION OF SENSOR DATA

1639

Enbin Song was born in 1979 in Shandong, China. He received the bachelor of science degree from Shandong Normal University, Shandong, China, in 2002. He is currently a graduate student with the Department of Mathematics, Sichuan University, Chengdu, China. His main research interests are in information fusion, including probability theory, statistics, and matrix theory.

Zhisheng You received the M.S. degree in digital signal processing from Sichuan University, Chengdu, China, in 1968. He was a visiting scholar at Michigan State University, East Lansing, from 1981 to 1983. He is now a Professor and Director of the Institute of Image and Graphics, Sichuan University. His research interests include signal processing, pattern recognition, image processing and machine vision, and system simulation. He has published more than 100 papers. Dr. You won two Second Grade of National Science and Technology Advanced Awards of China.

Jie Zhou received the B.S. and M.S. degrees in pure mathematics in 1989 and 1992, respectively, and the Ph.D. degree in probability theory and mathematical statistics in 2003 from the Department of Mathematics, Sichuan University, Chengdu, China. He has been with the Department of Mathematics, Sichuan University, since 1992, where he is currently an Associate Professor. He has published more than ten technical papers in international and Chinese journals and is the co-author of one textbook for undergraduates. His main research interests include information fusion, fuzzy mathematics, knowledge discovery, data compressing, and robust estimation.

TLFeBOOK

1640

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fourth-Order Blind Identification of Underdetermined Mixtures of Sources (FOBIUM) Anne Ferréol, Laurent Albera, and Pascal Chevalier

Abstract—For about two decades, numerous methods have been ) mixtures developed to blindly identify overdetermined ( statistically independent narrowband (NB) sources received of sensors. These methods exploit the informaby an array of tion contained in the second-order (SO), the fourth-order (FO) or both the SO and FO statistics of the data. However, in practical situations, the probability of receiving more sources than sensors increases with the reception bandwidth and the use of blind identification (BI) methods able to process underdetermined mixmay be required. Although tures of sources, for which such methods have been developed over the past few years, they all present serious limitations in practical situations related to the radiocommunications context. For this reason, the purpose of this paper is to propose a new attractive BI method, exploiting the information contained in the FO data statistics only, that is able to process underdetermined mixtures of sources without the main limitations of the existing methods, provided that the sources have different trispectrum and nonzero kurtosis with the same sign. A new performance criterion that is able to quantify the identification quality of a given source and allowing the quantitative comparison of two BI methods for each source, is also proposed in the paper. Finally, an application of the proposed method is presented through the introduction of a powerful direction-finding method built from the blindly identified mixture matrix. Index Terms—Blind source identification, FO direction finding, fourth-order statistics, performance criterion, SOBI, trispectrum, underdetermined mixtures.

I. INTRODUCTION OR more than two decades and the pioneer work of Godard [30] about blind equalization in single-input single-output (SISO) contexts, there has been an increasing interest for blind identification (BI) of both single-input multiple-output (SIMO) and multiple-input multiple-output (MIMO) systems. While, in the SISO case, blind equalization or channel identification require the exploitation of higher order (HO) statistics in the general case of nonminimum phase systems [30], it has been shown recently that for SIMO systems, multichannel identification may be performed from SO statistics only under quite general assumptions [39], [43], [49]. Extensions of these pioneer works

F

Manuscript received November 26, 2003; revised May 17, 2004. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jonathan H. Manton. A. Ferréol and P. Chevalier are with Thalès-Communications, 92704 Colombes, France (e-mail: [email protected], [email protected]). L. Albera is with the Laboratoire Traitement du Signal et de L’Image (LTSI), Rennes University (INSERM U642), 35042 Rennes Cedex, France (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845431

and the development of alternative methods for both blind multichannel identification and equalization in MIMO finite impulse response (FIR) systems from SO or HO statstics are presented in [1], [23], [31], [32], and [17], [29], [35], [38], [50]–[53], respectively. Other extensions to MIMO infinite impulse response (IIR) systems or taking into account the finite-alphabet property of the sources are presented in [34], [44] and [46], [54], respectively. However, the BI or deconvolution problems in MIMO contexts are not recent but have been considered since the pioneer work of Herault and Jutten [33], [36] about blind source separation (BSS) in 1985. Since these pioneer works, numerous methods have been developed to blindly identify either instantaneous or convolutive mixtures of statistically independent sensors. Some of these NB sources received by an array of methods [5], [48] exploit the SO data statistics only, whereas other methods [6], [9], [14], [22] exploit both the SO and the FO statistics of the data or even the FO data statistics only [2]. Nevertheless, all the previous methods of either blind multichannel identification of MIMO systems or BI of instantaneous or convolutive mixtures of sources, either SO or HO, can only process overdetermined systems, i.e., systems for which the number of sources (or inputs) is lower than or equal to the . number of sensors (or outputs) , i.e., such that However, in practical situations such as, for example, airborne electronic warfare over dense urban areas, the probability of receiving more sources than sensors increases with the reception bandwidth and the use of BI methods that are able to process , may underdetermined mixtures of sources, for which be required. To this aim, several methods have been developed this last decade mainly to blindly identify instantaneous mixtures of sources, among which we find the methods [3], [4], [8], [15], [16], [19]–[21], [37], [45]. Concerning convolutive mixtures of sources or MIMO FIR systems, only very scarce results exist about BI of underdetermined systems, among which we find [18] and [47]. Some of these methods focus on blind source extraction [16], [37], which is a difficult problem since underdetermined mixtures are not linearly invertible, while others, as herein, favor BI of the mixture matrix [3], [4], [8], [15], [16], [18]–[21], [37], [45], [47]. The methods proposed in [8], [15], [18]–[21], and [47] only exploit the information contained in the FO statistics of the data, whereas the one recently proposed in [3] exploits the sixth-order data statistics only, and its extension to an arbitrary even order is presented in [4]. Finally, the method proposed in [45] exploits the information contained in the second characteristic function of the observations, whereas in [37], the probability density of the observations conditionally to the mixture matrix is maximized. Nevertheless, all

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

FERRÉOL et al.: FOURTH-ORDER BLIND IDENTIFICATION OF UNDERDETERMINED MIXTURES OF SOURCES

these methods suffer from serious limitations in operational contexts related to radiocommunications. Indeed, the method [8] and its improvements for both instantaneous [21] and convolutive [18] mixtures of sources remain currently mainly conceptual and have not yet been evaluated by any simulations. The methods [15], [19], and [20] assume FO noncircular sources and, thus, fail in identifying circular sources, which are omnipresent in practice. Besides, the theories developed in [15] and [19] confine themselves to the three source and two sensor cases. Although the method [37] succeeds in identifying the steering vectors of up to four speech signals with only two sensors, the authors need sparsity conditions and do not address the general case when all sources are always present. Moreover, the method [45] has been developed only for real mixtures of real-valued sources, and the issue of robustness with respect to an overestimation of the source number remains open. Although very promizing, powerful, and easy to implement, the methods [3] and [4] suffer a priori from both a higher variance and a higher numerical complexity due to the use of data statistics with an even order strictly greater than four. Finally, for instantaneous mixtures of sources, the method developed in [47] can only process overdetermined systems. In order to overcome these limitations for underdetermined systems, the purpose of this paper is to propose a new BI method, exploiting the information contained in the FO data statistics only that is able to process both over and underdetermined instantaneous mixtures of sources without the drawbacks of the existing methods of this family but assuming the sources have a different trispectrum and have nonzero kurtosis with the same sign (the latter assumption is generally verified in radiocommunications contexts). This new BI method, which is called the Fourth-Order Blind Identification of Underdetermined Mixtures of sources (FOBIUM), corresponds to the FO extension of the second-order blind idenification (SOBI) method [5] and is able to blindly identify the steering vectors sources from an array of sensors with of up to sources from an array of space diversity only and of up to different sensors. Moreover, this method is asymptotically robust to an unknown Gaussian spatially colored noise since it does not exploit the information contained in the SO data statistics. To evaluate the performance of the FOBIUM method and, more generally, of all the BI methods, a new performance criterion that is able to quantify the identification quality of the steering vector of each source and allowing the quantitative comparison of two methods for the blind identification of a given source is also proposed. Finally, an application of the FOBIUM method is presented through the introduction of a FO direction-finding method, built from the blindly identified mixing matrix and called MAXimum of spatial CORrelation (MAXCOR), which is shown to be very powerful with respect to SO [42] and FO subspace-based direction-finding methods [7], [13], [40]. Note that an extension of the FOBIUM method to HO statistics remains possible. After the problem formulation and an introduction of some notations, hypotheses and data statistics in Section II, the FOBIUM method is presented in Section III. The associated conditions about the identifiability of the mixture matrix are then analyzed in Section IV. The new performance criterion is

1641

presented in Section V. The application of the FOBIUM method to the direction-finding problem through the introduction of the MAXCOR method is described in Section VI. All the results of the paper are illustrated in Section VII through computer simulations. The numerical complexity of the FOBIUM method compared with the one of some existing methods is briefly presented in Section VIII. Finally, Section IX concludes this paper. Note that the results of the paper have been partially presented in [11] and [25]. II. PROBLEM FORMULATION, HYPOTHESES, AND DATA STATISTICS A. Problem Formulation the We consider an array of NB sensors, and we call vector of complex amplitudes of the signals at the output of these sensors. Each sensor is assumed to receive the contribution of zero-mean stationary and stastistically independent NB sources corrupted by a noise. Under these assumptions, the observation can be written as follows: vector (1) where is the noise vector that is assumed to be zero-mean, stationary, and Gaussian, the complex envelope of the source , , is the th component of the vector that is assumed zero-mean and stationary, corresponds to the steering vector mixture matrix whose of the source , and is the columns are the vectors . The instantaneous mixture model defined by (1) have already been considered in numerous papers [2]–[12], [14]–[16], [19]–[22], [24]–[28], [33], [36], [37], [45], [48] and is perfectly suitable for applications such as, for example, airborne or satellite electronic warfare. Under these assumptions, the problem addressed in this paper is that of FO blind identification of the mixture matrix . It consists of estimating, from the FO data statistics, the mixing invertible diagonal matrix and matrix to within a permutation matrix . a B. Statistics of the Data Under the previous assumptions, the SO statistics of the data used in the paper are characterized by the correlation or covari, which is defined by ance matrix

(2) is the power of source received by where is the mean of the noise power an omnidirectional sensor, is the spatial coherence of the noise such that per sensor, , where Tr means Trace, is Tr the correlation matrix of the source vector , and the symbol means transpose and complex conjugate. The FO statistics of the data used in the paper are charquadricovariance matrices acterized by the

TLFeBOOK

1642

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

, whose elements are defined by Cum

(3)

where ponent

means complex conjugate, and is the comof . Using (1) into (3) and assuming that is the element of the matrix , we obtain the expression of the latter, which is given, under a Gaussian noise assumption, by (4) where is the quadricovariance matrix , and is the Kronecker product. of Under the assumption of statistically independent sources, the matrix contains at least zeros, and expression (4) degenerates in a simpler one given by

(5) where Diag defined by

is the

matrix defined by , is diagonal matrix defined by , and

Cum

(2) and (4), respectively. In these conditions, using a cyclo-ercan still be estimated from godicity property, the matrix the sampled data by the SO empirical estimator [26], but the has to be estimated by a nonempirical matrix estimator presented in [26], taking into account the SO cyclic frequencies of the data. Note, finally, that this extension can also be applied to nonzero mean cyclostationary sources, such as some nonlinearly digitally modulated sources [41], provided that nonempirical statistics estimators, which are presented in [27] and [28] for SO and FO statistics, respectively, are used. Such SO estimators take into account the first-order cyclic frequencies of the data, whereas such FO estimators take into account both the first and SO cyclic frequencies of the data. D. Hypotheses In Sections III–VIII, we further assume the following hypotheses: . H1) is full rank. H2) (i.e., no source is Gaussian). H3) H4) , , (i.e., sources have FO autocumulant with the same sign). , , such that H5)

the is

(6)

Expression (5), which has an algebraic structure similar to that of data correlation matrices [5], is the starting point of the FOBIUM method, as it will be shown in Section II-C. To sim, plify the notations, we note in the following , and , and we obtain from (5) (7)

(8) Note that hypothesis H4 is not restrictive in radiocommunication contexts since most of the digitally modulated sources have negative FO autocumulant. For example, -PSK constellations for and to for [41] have a kurtosis equal to . Continuous phase modulation (CPM) [41], among which we find, in particular, that the Continuous Phase Frequency Shift Keying (CPFSK), the Minimum Shift Keying (MSK), and the Gaussian Minimum Shift Keying (GMSK) modulation (GSM . Moreover, standard) have a kurtosis lower than or equal to note that (8) requires, in particular, that the sources have a different normalized tri-spectrum, which prevents us, in particular, from considering sources with both the same modulation, the same baud rate, and the same carrier residue. III. FOBIUM METHOD

C. Statistics Estimation In situations of practical interests, the SO and FO statistics of the data, which are given by (2) and (3), respectively, are not known a priori and have to be estimated from samples of data , , where is the sample period. For zero-mean stationary observations, using the ergodicity property, empirical estimators [26] may be used since they generate asymptotically unbiased and consistent estimates of the data statistics. However, in radiocommunications contexts, most of the sources are no longer stationary but become cyclostationary (digital modulations). For zero-mean cyclostationary observations, the statistics defined by (2) and (3) become time dependent, and the theory developed in the paper can be and extended without any difficulties by considering that are, in this case, the temporal means and over an infinite interval duration of the instantaneous statistics and defined by

The purpose of the FOBIUM method is to extend the SOBI method [5] to the FO. It first implements an FO prewhitening step aimed at orthonormalizing the so-called virtual steering . vector [12] of the sources, corresponding to the columns of Second, it jointly diagonalizes several well-chosen prewhitened matrix. quadricovariance matrices in order to identify the Then, in a third step, it identifies the mixing matrix from the matrix. The number of sources able to be processed by this method is considered in Section IV. A. FO Prewhitening Step The first step of the FOBIUM method is to orthonormalize, matrices (5), the columns of , which in the can be considered to be virtual steering vectors of the sources for the considered array of sensors [12]. For this purpose, let us , consider the eigendecomposition of the Hermitian matrix

TLFeBOOK

FERRÉOL et al.: FOURTH-ORDER BLIND IDENTIFICATION OF UNDERDETERMINED MIXTURES OF SOURCES

whose rank is by

under the assumptions H1 to H3, which is given

, which is solution to the In other words, the unitary matrix previous problem of joint diagonalization, can be written as

(9)

(14)

where is the real-valued diagonal matrix of the nonzero eigenvalues of , and is the matrix of the associated orthonormalized eigenvectors. Proposition 1: Assuming sources with nonzero kurtosis having the same sign (i.e., ), it is straigthare not zero forward to show that the diagonal elements of and also have the same sign corresponding to . , which contains the We deduce from Proposition 1 that nonzero eigenvalues of , has square root decompositions , where is a such that , and . Thus, the square root of existence of this square root decomposition requires assumpprewhitening matrix tion H4. Considering the defined by

where and respectively. Noting that

are unitary diagonal and permutation matrices, , the pseudo-inverse of , we deduce from (14) that

is the inverse of

, we obtain, from (11)

is the where Diag matrix we obtain

identity matrix and where . Expression (11) shows that the is a unitary matrix

, and (12)

which means that the columns of ized to within a diagonal matrix.

have been orthonormal-

B. FO Blind Identification of The second step of the FOBIUM method is to blindly idenmatrix from some FO statistics of the data. For this tify the purpose, we deduce from (5) and (12) that

, such (15)

and using (10) and (12) into (15), we obtain (16) Span , From (7) and (9), we deduce that Span on the space which implies that the orthogonal projection of , corresponds to . spanned by the columns of Using this result in (16), we finally obtain (17)

(10) where (7) and (9)

1643

which shows that the matrix can be identified to within a diagonal and a permutation matrix from the matrix . C. Blind Identification of The third step of the FOBIUM method is to identify the . For this purpose, we note from (17) mixing matrix from that each column and the definition of of corresponds to a vector , , where , such that , is an element of the diagonal matrix . Thus, mapping the components of each of into an matrix such that column , , consists of building , . We then deduce that the matrices of the source corresponds, to within a the steering vector associated with the eigenvalue scalar, to the eigenvector of having the strongest modulus. Thus, the eigendecomposition of matrices allows the identification of to all the within a diagonal and a permutation matrix. D. Implementation of the FOBIUM Method

(13) which shows that the unitary matrix diagonalizes the matrices whatever the set of delays , and the associated eigenvalues correspond to the diagonal terms of . the diagonal matrix and a given order of the For a given set sources, is unique to within a unitary diagonal matrix if and only if the diagonal elements of the matrix are all different. If it is not the case, following the results of [5], we have to consider , , such that for each couple several sets , there exists at least a set , such of sources that (8) is verified for this set, which corresponds to hypothesis H5. Under this assumption, the unitary matrix becomes, to within a permutation and a unitary diagonal matrix, the only one matrices . that jointly diagonalizes the

The different steps of the FOBIUM method are summarized hereafter when snapshots of the observations are available. Step 1 Estimation of the matrix from the snapshots using a suitable estimator of the FO cumulants [26], [27]; Step 2 Eigen Value Decomposition (EVD) of the matrix . • From this EVD, estimation of the number of by a classical source number detection sources test; • Evaluation of the sign of the eigenvalues; • Restriction of this EVD to the principal compo, where is the diagonal manents: trix of the eigenvalues with the strongest modulus, is the matrix of the associated eigenvectors. and Step 3 Estimation of the prewhitening matrix by ;

TLFeBOOK

1644

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Step 4 Selection

of

appropriate set of delays , . For example, one may choose these sets such that and or such that , , where where may be lower than or equal to is an estimate of the observation bandwidth ; of the matrices Step 5 Estimation for the delays sets using a suitable estimator; Step 6 , • Computation of the matrices . of the unitary matrix from • Estimation matrices the joint diagonalization of the (the joint diagonalization process is decribed in [5] and [9]); ; Step 7 Computation of Step 8 of • Mapping each column into an matrix . • EVD of the matrices • An estimate of the mixing matrix to within a diagonal and a permutation matrix is obtained by considering that each of the columns of corresponds to the eigenvector of a matrix associated with the eigenvalue having the strongest modulus.

[12]. Finally, for an array with sensors having all a different [12]. angular and polarization pattern, V. NEW PERFORMANCE CRITERION Most of the existing performance criterions used to evaluate the quality of a blind identification method [14], [15], [45] are global criterions, which evaluate a distance between the true mixing matrix and its blind estimate . Although useful, a global performance criterion necessarily contains implicitly a part of arbitrary considerations in the manner of combining the , to gendistances between the vectors and , for erate a unique scalar criterion. Moreover, it is possible to find of is better than an estimate , with rethat an estimate estimate spect to the global criterion, while some columns of the associated true steering vectors in a better way than those of , which may generate some confusion in the interpretations. To overcome these drawbacks, we propose in this section a new performance criterion for the evaluation of a blind identification method. This new criterion is no longer global and allows both the quantitative evaluation of the identification quality of each source by a given method and the quantitative comparison of two methods for the blind identification of a given source. It corresponds, for the blind identification problem, to a performance criterion similar, with respect to the spirit, to the one proposed in [10] for the extraction problem. It is defined by the following -uplet (18)

IV. IDENTIFIABILITY CONDITIONS

where

Following the developments of the previous section, we deduce that the FOBIUM method is able to identify the steering vectors of sources from an array of sensors, provided hypotheses H1 to H5 are verified. In other words, the FOBIUM non-Gaussian sources method is able to identify having different trispectrum and kurtosis with the same sign, matrix has full rank , i.e., that the virprovided that the for the considered tual steering vectors array of sensors remain linearly independent. However, it has been shown in [24] and [12] that the vector can also be considered as a true steering vector but for an FO virtual array different sensors, where is directly related of to both the pattern of the true sensors and the geometry of the true array of sensors. This means, in particular, that components of each vector are redundant components rows that bring no information. As a consequence, matrix bring no information and are linear combinaof the tions of the others, which means that the rank of cannot be . In these conditions, the matrix may have greater than . Conversely, for an FO vira rank equal to only if , sources tual array without any ambiguities up to order coming from different directions generate an matrix with . Thus, the FOBIUM method a full rank as long as sources, where is the number is able to process up to of different sensors of the FO virtual array associated with the considered array of sensors. For example, for a uniform linear , whereas for array (ULA) of identical sensors, most of other arrays with space diversity only,

,

, such that

, is defined by

Min is the pseudo-distance between the vectors where , which is defined by

(19) and

(20) Thus, the identification quality of the source is evaluated by the parameter , which decreases toward zero as the identification quality of the source increases. In particular, the source is perfectly identified when . Although arbitrary, we consider in the following that a source is blindly identified , with a high quality if with a very high quality if , with a good quality if , and with a poor quality otherwise. Besides, we will say that a method M1 is better than a method M2 for the identification of the source if (M1) (M2), where (Mi) corresponds to the paramgenerated by the method Mi. Moreover, we will say that eter a method M1 is better than a method M2 if it is better for each (M2) for . Finally, we source, i.e., if (M1) diagonal matrix and perverify that, whatever the mutation matrix , we obtain (21) which means that two mixing matrix estimates that are equal to within a diagonal and a permutation matrix generate the same performance for all the sources, which is satisfactory.

TLFeBOOK

FERRÉOL et al.: FOURTH-ORDER BLIND IDENTIFICATION OF UNDERDETERMINED MIXTURES OF SOURCES

1645

VI. APPLICATION OF THE FOBIUM METHOD: DIRECTION FINDING WITH THE MAXCOR METHOD Before presenting some computer simulations in Section VII, we propose, in this section, an application of the FOBIUM method that is usable when the array manifold is known or estimated by calibration. This application consists to find the direction of arrival (DOA) of the detected sources directly from the blindly identified mixing matrix, allowing better DOA estimations than the existing ones in many contexts. Besides, for a given array of sensors, this application allows the interpretation coefficient, introduced in the previous section to of the evaluate the identification quality of the source , in terms of angular precision. A. Existing Direction-Finding Methods When the array manifold is known or estimated by calibraof the steering vector tion, each component may be written as a function of the DOA of the source , where and are the azimuth and the elevation angles of source , respectively (see Fig. 1). The function is the th component of the steering vector for . In particular, in the absence of modeling the direction can errors such as mutual coupling, the component be written, under the far field assumption and in the general case of an array with space, angular, and polarization diversity, as [12] (22), shown at the bottom of the page, where is the are the coordinates of sensor of the wavelength, array, and is a complex number corresponding to the response of sensor to a unit electric field coming from the . Using the knowledge of the array manifold direction , it is possible to estimate the DOA of the sources from some statistics of the data such as the SO or the FO statistics given by (2) and (7), respectively. Among the existing SO direction-finding methods, the so-called High-Resolution (HR) methods, which have been developed from the beginning of the 1980s, are the most powerful in multisource contexts since they are characterized by an asymptotic resolution that becomes infinite, whatever the source signal-to-noise ratio (SNR). Among these HR methods, the subspace-based methods such as the MUSIC method [42] are the most popular. Recall that after a source number estimation , the MUSIC method consists of finding the couples , minimizing the pseudo-spectrum defined by (23) where

is the steering vector for the direction , where is the

matrix, and

is the

matrix of the

and identity

orthonormalized

Fig. 1. Incoming signal in three dimensions.

eigenvectors of the estimated data correlation matrix associated with the strongest eigenvalues. One of the main drawbacks of the SO subspace-based methods such as the MUSIC method is that they are not able sources from an array of to process more than sensors. Mainly to overcome this limitation, but also to still increase the resolution with respect to that of SO methods for a finite duration observation, higher order HR direction-finding methods [7], [13], [40] have been developed during these two last decades, among which the extension of the MUSIC method to the FO [40], which is called MUSIC4, is the most popular. Recall that after a source number estimation , the MUSIC4 minimizing method consists of finding the couples the pseudo-spectrum defined by (24) where , where

is the

, and identity matrix,

and is the matrix of the orthonormalized eigen vectors of the estimated data quadricovariance matrix associated with the strongest eigenvalues. Moreover, it has been shown in [12] that the MUSIC4 method is able to process sources where corresponds to the number of up to different sensors of the FO virtual array associated with the considered array of sensors. B. Application of the FOBIUM Method: The MAXCOR Method Despite the interests of both the SO and FO HR subspace-based direction-finding methods described in Section VI-A, the latter keep a source of performance limitation in multisource contexts for a finite duration of observation since they may be qualified as multidimensionnal methods insofar as they implement a procedure of searching multiple minima of a pseudo-spectrum function. This multidimensionality character

(22)

TLFeBOOK

1646

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

of these methods generates interaction between the sources in the pseudo-spectrum, which is a source of performance limitation, for a finite duration observation, in the presence of modeling errors or for poorly angularly separated sources, for example. To overcome the previous limitation, it is necessary to transmonodiform the multidimensional search of minima into mensional searches of optima, which can be easily done from the source steering vector estimates and is precisely the philosophy of the new proposed method. More precisely, from the estimated mixture matrix , the new proposed method called MAXCOR (search for a MAXimum of spatial CORrelation), , consists of solving, for each estimated source a mono-dimensional problem aiming at finding the DOA that maximizes the square modulus of a certain spatial correlation coefficient, which is defined by (25)

Fig. 2.



as a function of

L. (a) JADE. (b) SOBI. (c) FOBIUM. P

= 2,

Fig. 3.



as a function of

L, (a) JADE. (b) SOBI. (c) FOBIUM. P

= 2,

N

= 3, ULA,  = 90 ,  = 131:76 , SNR = 10 dB.

where (26) which is equivalent to minimizing the pseudo-spectrum defined by (27) It is obvious that the performance of the MAXCOR method is directly related to those of the BI method, which generates the estimated matrix . Performance of the MAXCOR method from a generated by the FOBIUM method are presented in Section VII and compared with those of MUSIC2 and MUSIC4, both with and without modeling errors. Note that following the FOBIUM method, the MAXCOR method is able to process up statistically independent non-Gaussian sources, whereas to sources [12]. MUSIC4 can only process

VII. COMPUTER SIMULATIONS Performances of the FOBIUM method are illustrated in Section VII-A, whereas those of the MAXCOR method are presented in Section VII-B. Note that the sources considered for the simulations are zero-mean cyclostationary sources corresponding to quadrature phase shift keying (QPSK) sources, which is not a problem for the FOBIUM method, according to Section II-C, provided the sources do not share the same trispectrum. Nevertheless, for complexity reasons, the empirical estimator of the FO data statistics is still used, despite the cyclostationarity of the sources. This is not a problem since it is shown in [26] that for SO circular sources such as QPSK sources, although biased, the empirical estimator behaves approximately like an unbiased estimator. Finally, the elevation angle of the sources is assumed to be zero.

N

= 3, ULA,  = 90 ,  = 131:76 , SNR = 0 dB.

A. FOBIUM Method Performance The performances of the FOBIUM method are presented in this section both for the overdetermined and underdetermined mixtures of sources. 1) Overdetermined Mixtures of Sources: To illustrate the performance of the FOBIUM method for overdetermined mixtures of sources, we assume that two statistically independent QPSK sources with a raise cosine pulse shape filter are omnidirectional sensors spaced received by a ULA of half a wavelength apart. The two QPSK sources have the same symbol duration (where is the sample period), the , the same input SNR and have a carrier same roll-off , , and a DOA residue such that equal to and , respectively. The performance for the source , is computed and averaged over 300 realizations. Under these assumptions, Figs. 2–5 show, for several configurations of SNR and spatial correlation between the sources, ( behaves in a same way) at the output the variations of of both Joint Approximated Diagonalization of Eigenmatrices (JADE) [9], SOBI [5], and FOBIUM methods, as a function of

TLFeBOOK

FERRÉOL et al.: FOURTH-ORDER BLIND IDENTIFICATION OF UNDERDETERMINED MIXTURES OF SOURCES

Fig. 4.

N

Fig. 5.

N



as a function of L. (a) JADE. (b) SOBI. (c) FOBIUM.  = 90 ,  = 82:7 , SNR = 10 dB.

= 3, ULA,



as a function of L. (a) JADE. (b) SOBI. (c) FOBIUM.  = 90 ,  = 82:7 , SNR = 0 dB.

= 3, ULA,

P

P

= 2,

Fig. 6.

N

= 2,

the number of snapshots . For Figs. 2 and 3, the sources are , ) and such well angularly separated ( that their SNR is equal to 10 and 0 dB, respectively. For Figs. 4 , and 5, the sources are poorly angularly separated ( ) and such that their SNR is equal to 10 and 0 dB, delays, respectively. For the SOBI method, are considered such that , whereas for the FOBIUM method, delays set, are taken into account and . such that Figs. 2 and 3 show that for well angularly separated nonGaussian sources having different spectrum and trispectrum, the JADE, SOBI, and FOBIUM methods succeed in blindly identifying the sources steering vectors with a very high quality , ) from a relatively weak number of snap( for SNR dB shots and even for weak sources ( and for SNR dB). Nevertheless, in such situations, we note the best behavior of the SOBI method with respect to FO methods and the best behavior of JADE with respect to FOBIUM, whatever the source SNR and the number of snapshots, due to a higher variance of the FO statistics estimators. Fig. 4 confirms the very good behavior of the three methods even when from a very weak number of snapshots



(1 = 3, ULA,

1647

 p  5) as a function of L at the output of FOBIUM, P = 5, = 90 , 120.22 , 150.65 , 052:05 , and 076:32 .



the sources are poorly angularly separated, provided the SNR is dB for Fig. 4). However, Figs. 4 and not too low (SNR 5 show that for poorly angularly separated sources, there exists a number of snapshots increasing with the source SNR for SNR dB and for SNR dB), ( over which the FOBIUM method becomes much more efficient than the JADE and SOBI methods. In such situations, the resolution gain obtained with FOBIUM is higher than the loss due to a higher variance in the statistics estimates. In particular, for sources with an SNR equal to 0 dB, Fig. 5 shows a very high , ) with source identification quality ( , whereas the JADE and the FOBIUM method for only around 0.05 for SOBI methods generate coefficients . 2) Underdetermined Mixtures of Sources: To illustrate the performance of the FOBIUM method for underdetermined mixtures of sources, we assume first that five statistically independent QPSK sources with a raised-cosine pulse shape filter are omnidirectional sensors. The received by an array of , the five QPSK sources have the same symbol duration , the same input SNR of 20 dB, a carrier same roll-off , , residue such that , , and , and a DOA given , , , , and by , respectively. The performance for the source , is still computed and averaged over 300 realizations. For the FOBIUM method, delays set, are taken into , and . account such that Under these assumptions, Figs. 6 and 7 show the variations of at the output of the FOBIUM all the coefficients method, as a function of the number of snapshots . For Fig. 6, a ULA of three sensors spaced half a wavelength apart is considered, whereas for Fig. 7, the array of sensors corresponds to ( is the a uniformly circular array (UCA) such that radius, and is the wavelength). Note that the two considered arrays of sensors have the same aperture on the -axis if the sensors of the ULA lie on this axis.

TLFeBOOK

1648

Fig. 7.

N

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005



(1 = 3, UCA,

 p  5) as a function of L at the output of FOBIUM, P = 5, Fig. 8.  (1  p  6) as a function of L at the output of FOBIUM, P = 6, = 90 , 120.22 , 150.65 , 052:05 , and 076:32 . N = 3, ULA,  = 90 , 120.22 , 150.65 , 052:05 , 076:32 , and 66.24 .



Figs. 6 and 7 show that for both the ULA and UCA arrays, ( for the ULA and for as long as the UCA), the FOBIUM method succeeds in blindly identifying , the source steering vectors with a high quality ( ) in underdetermined contexts as soon as there are for the ULA and for enough snapshots ( the UCA). Nevertheless, the comparison of Figs. 2 and 6 shows that for a given array of sensors, the number of snapshots required to obtain a high BI quality of all the source steering vecfor tors increases with the number of sources ( and for for a ULA of three sensors). On the other hand, the comparison of Figs. 6 and 7 show that for a given number and scenario of sources, the required number of snapshots ensuring a high quality of source steering vector idendecreases. Note that tification increases as the quantity the quantity (0 for the ULA and two for the UCA) corresponds to the number of degrees of freedom in excess for the FO virtual array associated with the considered array of sensors. We now decide to add one QPSK source with a raised-cosine pulse shape filter to the five previous ones. Source 6 has the , the same roll-off , the same symbol duration , input SNR of 20 dB, a carrier residue such that . For the FOBIUM method, and a DOA given by delays set, , are still taken into account a such that , and . Under these new assumptions, Figs. 8 and 9 again show the variations of all the at the output of the FOBIUM coefficients method, as a function of the number of snapshots . For Fig. 8, a ULA of three sensors is considered, whereas for Fig. 9, the UCA of three sensors is considered. The comparison of Figs. 7 and 9 confirms, for a given array , the increasing value of of sensors and as long as required to obtain a good blind identification of all the source , ) as the number of steering vectors ( for and for sources increases ( for a UCA of three sensors). However, the comparison of Figs. 6 and 8, for the ULA with three sensors, shows off the limitations of the FOBIUM method and the poor identification ), even for large quality of some sources ( such that as soon as . values of

Fig. 9.  (1  p  6) as a function of L at the output of FOBIUM, P = 6, N = 3, UCA,  = 90 , 120.22 , 15.65 , 052:05 , 076:32 , and 66.24 .

B. MAXCOR Method Performance The performance of the MAXCOR method, which extracts the DOA of the sources from the source steering vectors blindly identified by the FOBIUM method, are presented in this section both in the absence and in the presence of modeling errors. considered 1) Performance Criterion: For each of the sources and for each of the three considered direction finding methods, two criterions are used in the following to quantify the quality of the associated DOA estimation. For a given source, the first criterion is a probability of aberrant results generated by a given method for this source, and the second one is an averaged Root Mean Square Error (RMSE), computed from the nonaberrant results, generated by a given method for this source. More precisely, for a given method, a given number of snapshots , and a particular realization of the observation vectors , the estimation of the DOA of the source is defined by Min

(28)

where, for the MUSIC2 and MUSIC4 methods, the quantities correspond to the minima of the pseudo-spec-

TLFeBOOK

FERRÉOL et al.: FOURTH-ORDER BLIND IDENTIFICATION OF UNDERDETERMINED MIXTURES OF SOURCES



Fig. 10. RMS error of the source 1 and p( ) as a function of L. (a) MAXCOR. (b) MUSIC2. (c) MUSIC4. P = 2, N = 3, ULA, SNR = 10 dB,  = 90 , and  = 131:76 ; no modeling errors.

, and , respectively, which are trum defined by (23) and (24), and where, for the MAXCOR corresponds to the minimum of , method, . To each estimate , we associate the corresponding value of the pseudo-spectrum, which is for MUSIC2, defined by for MUSIC4, and for MAXCOR, where is the integer that minimizes . In this context, the , where is a estimate is considered to be aberrant if . threshold to be defined. In the following, realizations of the observation vecLet us now consider tors . For a given method, the probability of is defined by the abberant results for a given source , is aberratio between the number of realizations for which of realizations. From the nonaberrant rant, and the number realizations for the source , we then define the averaged RMS error for the source (RMSE ) by the quantity

RMSE

(29)

1649

Fig. 11. RMS error of the source 1 and p(   ) as a function of L. (a) MAXCOR. (b) MUSIC2. (c) MUSIC4. P = 2, N = 3, ULA, SNR = 10 dB,  = 90 ,  = 82, 7 ; no modeling errors.

is the number of nonaberrant realizations for the where is the estimate of for the nonaberrant resource , and alization . 2) Absence of Modeling Errors: To illustrate the performance of the MAXCOR method in the absence of modeling errors, we consider the scenarios of Figs. 2 and 4, respectively, for which two QPSK sources that are well and poorly angularly dB, are separated, respectively, and such that SNR received by an ULA of three sensors. Under the assumptions of Fig. 2 (sources with a large angular separation), Fig. 10 shows the variations, as a function of the number of snapshots , of the RMS error for the source 1 ) and the associated probability of nonabberant results ( (we obtain similar results for the source 2), estimated realizations at the output of the MAXCOR, from MUSIC2, and MUSIC4 methods. Fig. 11 shows the same variations as those of Fig. 10 but under the assumptions of Fig. 4 (sources with a weak angular separation). Fig. 10(b) shows that the probability of aberrant realizations for source 1 is zero for all the methods as soon as becomes greater than 120. In this context, Fig. 10(a) shows that for well angularly separated non-Gaussian sources having different

TLFeBOOK

1650

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

spectrum and trispectrum and a SNR equal to 10 dB, the three methods succeed in estimating the DOA of the two sources with , ) from a relatively a high precision (RMSE weak number of snapshots ( for MUSIC2 and for MUSIC4 and MAXCOR). Nevertheless, in such situations, we note the best behavior of the MUSIC2 method with respect to HO methods MUSIC4 and MAXCOR, which give the same results, due to a higher variance of the FO statistics estimators. Fig. 11(b) shows that the probability of aberrant realizations for the source 1 is equal to 0 for MAXCOR, whatever the value for MUSIC2 of , but remains greater than 20% for and MUSIC4. Both in terms of probability of nonaberrant results and estimation precision, Fig. 11(a) and (b) shows, for poorly angularly separated sources, the best behavior of the MAXCOR method, which becomes much more efficient than the MUSIC4 and MUSIC2 methods. Indeed, MAXCOR succeeds in estimating the DOA of the two sources with a high , ) from a relatively precision (RMSE , whereas MUSIC4 and weak number of snapshots MUSIC2 require and snapshots, respectively, to obtain the same precision. In such situations, the resolution gain obtained with MAXCOR and MUSIC4 with respect to MUSIC2 is higher than the loss due to a higher variance in the statistics estimates. Besides, the monodimensionality character of the MAXCOR method with respect to MUSIC4 jointly with the very high resolution power of the FOBIUM method explain the best behavior of MAXCOR with respect to MUSIC4. 3) Presence of Modeling Errors: We now consider the simulations of Section VII-B2 but with modeling errors due for instance to a nonperfect equalization of the reception chains. In the presence of such errors, the steering vector of the source is not the known function of the DOA but becomes an unknown function of , where is a modeling error vector. In such conditions, the previous HR methods lose their infinite asymptotic resolution, and the question is to search for a method that presents some robustness to the modeling errors. To solve this is a zero-mean, problem, we assume that the vector Gaussian, circular vector with independent components such . Note that for omnidirecthat is the sum of the phase and tional sensors and small errors, amplitude error variances per reception chain. For the simulations, is chosen to be equal to 0.0174, which corresponds, for example, to a phase error with a standard deviation of 1 with no amplitude error. In this context, under the assumptions of Fig. 10 (sources with a large angular separation) but with modeling errors, Fig. 12 shows the variations, as a function of the number of snapshots , of the RMS error for the source 1 ( RMSE ) and the asso(we obtain ciated probability of nonabberant results realsimilar results for the source 2) estimated from izations at the output of the MAXCOR, MUSIC2, and MUSIC4 methods. Fig. 13 shows the same variations as those of Fig. 12 but under the assumptions of Fig. 11 (sources with a weak angular separation) with modeling errors. Fig. 12(b) shows that the probability of aberrant realizations for the source 1 is zero for all the methods as soon as becomes greater than 135. In this context, comparison of Figs. 10

Fig. 12. RMS error of the source 1 and p(  ) as a function of L. (a) MAXCOR. (b) MUSIC2. (c) MUSIC4. P = 2, N = 3, ULA, SNR = 10 dB,  = 90 , and  = 131:76 ; with modeling errors.

and 12 show a degradation of the performance of each method in the presence of modeling errors. However, for well angularly separated sources, MUSIC2 is more affected by the presence of modeling errors than FO methods as soon as the number of snapshots is sufficient. Indeed, while MUSIC2 remains better than FO methods for a relatively weak number of snapshots , due to a higher variance of HO methods, MUSIC4 and MAXCOR, which are equivalent to each other, become better than MUSIC2 as soon as the number of snapshots is suffi. In this latter case, the higher number of sensors cient with respect to that of the true of the FO virtual array array reduces the effect of modeling errors on the performances of FO methods. Fig. 13(b) shows that the probability of aberrant realizations for the source 1 is equal to 0 for MAXCOR, whatever the value for MUSIC2 of , but remains greater than 20% for and MUSIC4. Both in terms of probability of nonaberrant results and estimation precision, comparison of Figs. 11 and 13 again show a degradation of the performance of all the methods in the presence of modeling errors. However, for poorly angularly separated sources, whatever the value of the number of snapshots, MUSIC2 is much more affected by the modeling er-

TLFeBOOK

FERRÉOL et al.: FOURTH-ORDER BLIND IDENTIFICATION OF UNDERDETERMINED MIXTURES OF SOURCES

1651

Fig. 14. Minimum numerical complexity as a function of P . (a) JADE. (b) SOBI. (c) FOBIUM.

VIII. NUMERICAL COMPLEXITY COMPUTATION



Fig. 13. RMS of the source 1 and p( ) as a function of L. (a) MAXCOR. (b) MUSIC2. (c) MUSIC4. P = 2, N = 3, ULA, SNR = 10 dB,  = 90 ,  = 82, 7 ; with modeling errors.

, due to a greater aperrors than FO methods, as soon as ture and number of sensors of the FO virtual array with respect to the true array. Note again, in the presence of modeling errors, the best performance of MAXCOR with respect to MUSIC4 for poorly angularly separated sources.

Comp JADE

This section aims at giving some insight into the relative numerical complexity of the SOBI, JADE, and FOBIUM methods for given values of , , , and the number of sweeps required by the joint diagonalization process [5], [9]. The numerical complexity of the methods is presented in terms of the number of floating complex operations (Flocops) required to identify the mixture matrix from snapshots of the data. Note that a flocop corresponds to the sum of a complex multiplication and a complex addition. The number of flocops required by the JADE, SOBI, and FOBIUM methods for given values of , , , and are given is the by (30)–(32), shown at the bottom of the page, where number of correlation and quadricovariance matrices jointly diagonalized by the SOBI and FOBIUM methods, respectively. For a given number of sources , the minimum complexity of the previous methods is obtained by minimizing the values of , , , and , ensuring the good identification of the mixture matrix . It is said in [14] that the minimum value of is Int , where Int means integer part. The mindepends on the spectral difference between imum value of

Min

Min (30)

Comp SOBI (31) Comp FOBIUM (32)

TLFeBOOK

1652

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

the sources and is chosen to be equal to in the folis equal to for lowing. The minimum value of JADE and SOBI, whereas for FOBIUM, assuming an array with , space diversity only, it corresponds to the minimum value . Finally, the minimum such that value of depends on several parameters such as , , the FO autocumulant, and the SNR of the sources…. For this reason, is chosen to be the same for all the methods in the following. Under these assumptions, Fig. 14 shows the variations of the minimum numerical complexity of JADE, SOBI, and FOBIUM . Note as a function of the number of sources for the higher complexity of FOBIUM with respect to JADE and SOBI, which requires about 1 Mflocops to process four sources from 1000 snapshots. IX. CONCLUSION A new BI method that exploits the FO data statistics only (called FOBIUM) has been presented in this paper to process both overdetermined and underdetermined instantaneous mixtures of statistically independent sources. This method does not have the drawbacks of the existing methods that are capable of processing underdetermined mixtures of sources and is able to put up with any kind of sources (analogical or digital, circular or not, i.i.d or not) with potential different symbol duration … It only requires non-Gaussian sources having kurtosis with the same sign (practically always verified in radiocommunications contexts) and sources having different trispectrum, which is the only limitation of the method. The FOBIUM sources method is capable of processing up to sensors with space diversity only and up from an array of sources from an array of different sensors. A conto sequence of this result is that it allows a drastic reduction or minimization in the number of sensors for a given number of sources, which finally may generate a receiver that is much less expensive than a receiver developed to process overdetermined mixtures only. The FOBIUM method has been shown to require a relatively weak number of snapshots to generate good output performances for currently used radiocommunications sources, such as QPSK sources. Besides exploiting the FO data statistics only, the FOBIUM method is robust to the presence of a Gaussian noise whose spatial coherence is unknown. Finally, an application of the FOBIUM method has been presented through the introduction of a new FO direction-finding method that is built from the blindly identified mixing matrix and called MAXCOR. The comparison of this method to both the SO and FO HR subspace-based direction-finding methods shows better resolution and better robustness to modeling errors of the MAXCOR method with respect to MUSIC2 and MUSIC4 and its ability to process underdetermined mixtures of up to statistically independent non-Gaussian sources. ACKNOWLEDGMENT The authors gratefully acknowledge Dr. P. Comon for his contribution about the numerical complexity computation of the methods considered in the paper.

REFERENCES [1] K. Abed-Meraim, P. Loubaton, and E. Moulines, “A subspace algorithm for certain blind identification problems,” IEEE Trans. Inf. Theory, vol. 43, no. 2, pp. 499–511, Mar. 1997. [2] L. Albera, A. Ferreol, P. Chevalier, and P. Comon, “ICAR, un algorithme d’ICA à convergence rapide robuste au bruit,” in Proc. GRETSI, Paris, France, Sep. 2003. [3] L. Albera, A. Ferreol, P. Comon, and P. Chevalier, “Sixth order blind identification of underdetermined mixtures (BIRTH) of sources,” in Proc. ICA, Nara, Japan, Apr. 2003, pp. 909–914. , “Blind Identification of Overcomplete MixturEs of sources [4] (BIOME),” Linear Algebra Applicat. J., vol. 391, pp. 3–30, Nov. 2004. [5] A. Belouchrani, K. Abed-Meraim, J. F. Cardoso, and E. Moulines, “A blind source separation technique using second order statistics,” IEEE Trans. Signal Process., vol. 45, no. 2, pp. 434–444, Feb. 1997. [6] E. Bingham and H. Hyvarinen, “A fast fixed-point algorithm for independent component analysis of complex valued signals,” Int. J. Neural Syst., vol. 10, no. 1, pp. 1–8, 2000. [7] J. F. Cardoso, “Localisation et identification par la quadricovariance,” Traitement du Signal, vol. 7, no. 5, 1990. , “Super-Symetric decomposition of the fourth-order cumulant [8] tensor—Blind identification of more sources than sensors,” in Proc. ICASSP, Toronto, ON, Canada, May 1991, pp. 3109–3112. [9] J. F. Cardoso and A. Souloumiac, “Blind beamforming for non-Gaussian signals,” Proc. Inst. Elect. Eng. F, vol. 140, no. 6, pp. 362–370, Dec. 1993. [10] P. Chevalier, “Optimal separation of independent narrow-band sources—Concept and performance,” Signal Process., vol. 73, no. 1-2, pp. 27–47, Feb. 1999. [11] P. Chevalier, G. Benoit, and A. Ferreol, “DF after blind identification of the source steering vectors : The blind-Maxcor and blind-music methods,” in Proc. EUSIPCO, Trieste, Italy, Sep. 1996, pp. 2097–2100. [12] P. Chevalier and A. Ferreol, “On the virtual array concept for the fourthorder direction finding problem,” IEEE Trans. Signal Process., vol. 47, no. 9, pp. 2592–2595, Sep. 1999. [13] H. H. Chiang and C. L. Nikias, “The ESPRIT algorithm with high order statistics,” in Proc. Workshop Higher Order Statist., Vail, CO, Jun. 1989, pp. 163–168. [14] P. Comon, “Independent component analysis—A new concept?,” Signal Process., vol. 36, no. 3, pp. 287–314, Apr. 1994. , “Blind channel identification and extraction of more sources than [15] sensors,” in Proc. SPIE Conf., San Diego, CA, Jul. 1998, pp. 2–13. [16] P. Comon and O. Grellier, “Non linear inversion of underdetermined mixtures,” in Proc. ICA, Aussois, France, Jan. 1999, pp. 461–465. [17] P. Comon and L. Rota, “Blind separation of independent sources from convolutive mixtures,” IEICE Trans. Funda. Commun. Comput. Sci, vol. E86-A, no. 3, pp. 542–549, Mar. 2003. [18] L. De Lathauwer, “The canonical decomposition and blind identification with more inputs than outputs : Some algebraic results,” in Proc. ICA, Nara, Japan, Apr. 2003, pp. 781–784. [19] L. De Lathauwer, P. Comon, B. De Moor, and J. Vandewalle, “ICA algorithms for 3 sources and 2 sensors,” in Proc. Workshop Higher Order Statist., Caesarea, Israel, Jun. 1999, pp. 116–120. [20] L. De Lathauwer, B. De Moor, and J. Vandewalle, “ICA techniques for more sources than sensors,” in Proc. Workshop Higher Order Statist., Caesarea, Israel, Jun. 1999, pp. 121–124. [21] L. De Lathauwer, B. De Moor, J. Vandewalle, and J. F. Cardoso, “Independent component analysis of largely underdetermined mixtures,” in Proc. ICA, Nara, Japan, Apr. 2003, pp. 29–33. [22] N. Delfosse and P. Loubaton, “Adaptive blind separation of independent sources: A deflation approach,” Signal Process., vol. 45, pp. 59–83, 1995. [23] Z. Ding, “Matrix outer-product decomposition method for blind multiple channel identification,” IEEE Trans. Signal Process., vol. 45, no. 12, pp. 3053–3061, Dec. 1997. [24] M. C. Dogan and J. M. Mendel, “Applications of cumulants to array processing—Part I: Aperture extension and array calibration,” IEEE Trans. Signal Process., vol. 43, no. 5, pp. 1200–1216, May 1995. [25] A. Ferreol, L. Albera, and P. Chevalier, “Fourth order blind identification of underdetermined mixtures of sources (FOBIUM),” in Proc. ICASSP, Hong Kong, Apr. 2003, pp. 41–44. [26] A. Ferreol and P. Chevalier, “On the behavior of current second and higher order blind source separation methods for cyclostationary sources,” IEEE Trans. Signal Process., vol. 48, no. 6, pp. 1712–1725, Jun. 2000. Errata: vol. 50, no. 4, p. 990, Apr. 2002.

TLFeBOOK

FERRÉOL et al.: FOURTH-ORDER BLIND IDENTIFICATION OF UNDERDETERMINED MIXTURES OF SOURCES

[27] A. Ferreol, P. Chevalier, and L. Albera, “Higher order blind separation of non zero-mean cyclostationary sources,” in Proc. EUSIPCO, Toulouse, France, Sep. 2002, pp. 103–106. , “Second order blind separation of first and second order cyclosta[28] tionary sources—Application to AM, FSK, CPFSK and deterministic sources,” IEEE Trans. Signal Process., vol. 52, pp. 845–861, Apr. 2004. [29] G. B. Giannakis, Y. Inouye, and J. M. Mendel, “Cumulant based identification of multichannel moving-average processes,” IEEE Trans. Autom. Control, vol. 34, no. 7, pp. 783–787, Jul. 1989. [30] D. N. Godard, “Self-recovering equalization and carrier tracking in two dimensional data communication systems,” IEEE Trans. Commun., vol. COM-28, pp. 1867–1875, Nov. 1980. [31] A. Gorokhov and P. Loubaton, “Subspace-based techniques for blind separation of convolutive mixtures with temporally correlated sources,” IEEE Trans. Circuits Syst. I: Funda., Theory Applicat., vol. 44, no. 9, pp. 813–820, Sep. 1997. , “Blind identification of MIMO-FIR systems: A generalized linear [32] prediction approach,” Signal Process., vol. 73, pp. 105–124, Feb. 1999. [33] J. Herault, C. Jutten, and B. Ans, “Détection de grandeurs primitives dans un message composite par une architecture de calcul neuromimétique en apprentissage non supervisé,” in Proc. GRETSI, Juan-Les-Pins, France, May 1985, pp. 1017–1022. [34] Y. Inouye and T. Umeda, “Parameter estimation of multivariate ARMA process using cumulants,” IEICE Trans. Funda. Commun. Comput. Sci., vol. E77-A, pp. 748–759, 1994. [35] Y. Inouye and K. Hirano, “Cumulant-based blind identification of linear multi-input multi-output systems driven by colored inputs,” IEEE Trans. Signal Process., vol. 45, no. 6, pp. 1543–1552, Jun. 1997. [36] C. Jutten and J. Herault, “Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture,” in Signal Process., 1991, vol. 24, pp. 1–10. [37] T. W. Lee, M. S. Lewicki, M. Girolami, and T. S. Sejnowski, “Blind source separation of more sources than mixtures using overcomplete representations,” IEEE Signal Process. Lett., vol. 6, no. 4, pp. 87–90, Apr. 1999. [38] S. Mayrargue, “A blind spatio-temporal equalizer for a radio-mobile channel using the constant modulus algorithm (CMA),” in Proc. Intern. Conf. Acoust., Speech, Signal Process., Adelaide, Australia, Apr. 1994, pp. 317–320. [39] E. Moulines, P. Duhamel, J. Cardoso, and S. Mayrargue, “Subspace methods for blind identification of multichannel FIR filters,” IEEE Trans. Signal Process., vol. 43, no. 2, pp. 516–525, Feb. 1995. [40] B. Porat and B. Friedlander, “Direction finding algorithms based on higher order statistics,” IEEE Trans. Signal Processing, vol. 39, no. 9, pp. 2016–2024, Sep. 1991. [41] J. G. Proakis, Digital Communications, Third ed. New York: McGrawHill, 1995. [42] R. O. Schmidt, “Multiple emitter location and signal parameter estimation,” IEEE Trans. Antennas Propag., vol. AP-34, no. 3, pp. 276–280, Mar. 1986. [43] D. Slock, “Blind fractionally-spaced equalization, perfect-reconstruction-filter-banks and multichannel linear prediction,” in Proc. Int. Conf. Acoust., Speech, Signal Process., 1994, pp. 585–588. [44] A. Swami, G. B. Giannakis, and S. Shamsunder, “A unified approach to modeling multichannel ARMA processes using cumulants,” IEEE Trans. Signal Process., vol. 42, no. 4, pp. 898–913, Apr. 1994. [45] A. Taleb, “An algorithm for the blind identification of N independent signals with 2 sensors,” in Proc. 16th Symp. Signal Process. Applicat., Kuala Lumpur, Malaysia, Aug. 2001, pp. 5–8. [46] S. Talwar, M. Viberg, and A. Paulraj, “Blind estimation of synchronous co-channel digital signals using an antenna array: Part I, Algorithms,” IEEE Trans. Signal Process., vol. 44, no. 5, pp. 1184–1197, May 1996. [47] L. Tong, “Identification of multichannel parameters using higher order statistics,” Signal Process., vol. 53, no. 2, pp. 195–202, 1996. [48] L. Tong, R. Liu, V. C. Soon, and Y. F. Huang, “Indeterminacy and identifiability of blind identification,” IEEE Trans. Circ. Syst., vol. 38, no. 5, pp. 499–509, May 1991. [49] L. Tong, G. Xu, and T. Kailath, “Blind identification and equalization based on second order statistics: A time domain approach,” IEEE Trans. Inf. Theory, vol. 40, no. 2, pp. 272–275, Mar. 1994. [50] A. Touzni, I. Fijalkow, M. Larimore, and J. R. Treichler, “A globally convergent approach for blind MIMO adaptive deconvolution,” in Proc. Int. Conf. Acoust., Speech, Signal Process., 1998. [51] J. K. Tugnait, “Blind spatio-temporal equalization and impulse response estimation for MIMO channels using a Godard cost function,” IEEE Trans. Signal Process., vol. 45, no. 1, pp. 268–271, Jan. 1997.

1653

, “Identification and deconvolution of multichannel linear non-Gaussian processes using higher order statistics and inverse filter criteria,” IEEE Trans. Signal Process., vol. 45, no. 3, pp. 658–672, Mar. 1997. , “On linear predictors for MIMO channels and related blind iden[53] tification and equalization,” IEEE Signal Process. Lett., vol. 5, no. 11, pp. 289–291, Nov. 1998. [54] A. J. Van Der Veen, S. Talwar, and A. Paulraj, “Blind estimation of multiple digital signals transmitted over FIR channels,” IEEE Signal Process. Lett., vol. 2, no. 5, pp. 99–102, May 1995. [52]

Anne Ferréol was born in 1964 in Lyon, France. She received the M.Sc. degree from ICPI-Lyon, Lyon, France, and the Mastère degree from Ecole Nationale Supérieure des Télécommunications (ENST), Paris, France, in 1988 and 1989, respectively. She is currently pursuing the Ph.D. degree with the Ecole Normale Supérieure de Cachan, France, in collaboration with both SATIE Laboratory and THALES Communications. Since 1989, she has been with Thomson-CSFCommunications, Gennevilliers, France, in the array processing department. Ms. Ferréol co-received the 2003 “Science and Defense” Award from the French Ministry of Defence for its work as a whole about array processing for military radiocomunications.

Laurent Albera was born in Massy, France, in 1976. In 2001, he received the DESS degree in mathematics and the DEA degree in authomatic and signal processing from the University of Science (Paris XI), Orsay, France, and, in 2003, the Ph.D. degree in science from the University of Nice, Sophia-Antipolis, France. He is now Associate Professor with the University of Rennes I, Rennes, France, and is affiliated with the Laboratoire Traitement du Signal et de l’Image (LTSI). His research interests include high order statistics, multidimensional algebra, blind deconvolution and equalization, digital communications, statistical signal and array processing, and numerical analysis. More exactly, since 2000, he has been involved with blind source separation (BSS) and independent component analysis (ICA) by processing both the cyclostationary source case and the underdetermined mixture identification problem.

Pascal Chevalier received the M.Sc. degree from Ecole Nationale Supérieure des Techniques Avancées (ENSTA), Paris, France, and the Ph.D. degree from South-Paris University in 1985 and 1991, respectively. Since 1991, he has been with Thalés-Communications, Colombes, France, where he has shared industrial activities (studies, experimentations, expertise, management), teaching activities both in French engineering schools (Supelec, ENST, ENSTA), and French Universities (Cergy-Pontoise), and research activities. Since 2000, he has also been acting as Technical Manager and Architect of the array processing subsystem as part of a national program of military satellite telecommunications. He has been a Thalés Expert since 2003. His present research interests are in array processing techniques, either blind or informed, second order or higher order, spatial- or spatio-temporal, time-invariant or time-varying, especially for cyclostationary signals, linear or nonlinear, and particularly widely linear for noncircular signals, for applications such as TDMA and CDMA radiocommunications networks, satellite telecommunications, spectrum monitoring, and HF/VUHF passive listening. He is author or co-author of about 100 papers, including journals, conferences, patents, and chapters of books. Dr. Chevalier was a member of the THOMSON-CSF Technical and Scientific Council from 1993 to 1998. He co-received the “2003 Science and Defense” Award from the french Ministry of Defence for its work as a whole about array processing for military radiocommunications. He is presently a EURASIP member and an emeritus member of the Societé des Electriciens et des Electroniciens (SEE).

TLFeBOOK

1654

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Penalty Function-Based Joint Diagonalization Approach for Convolutive Blind Separation of Nonstationary Sources Wenwu Wang, Member, IEEE, Saeid Sanei, Member, IEEE, and Jonathon A. Chambers, Senior Member, IEEE

Abstract—A new approach for convolutive blind source separation (BSS) by explicitly exploiting the second-order nonstationarity of signals and operating in the frequency domain is proposed. The algorithm accommodates a penalty function within the cross-power spectrum-based cost function and thereby converts the separation problem into a joint diagonalization problem with unconstrained optimization. This leads to a new member of the family of joint diagonalization criteria and a modification of the search direction of the gradient-based descent algorithm. Using this approach, not only can the degenerate solution induced by a null unmixing matrix and the effect of large errors within the elements of covariance matrices at low-frequency bins be automatically removed, but in addition, a unifying view to joint diagonalization with unitary or nonunitary constraint is provided. Numerical experiments are presented to verify the performance of the new method, which show that a suitable penalty function may lead the algorithm to a faster convergence and a better performance for the separation of convolved speech signals, in particular, in terms of shape preservation and amplitude ambiguity reduction, as compared with the conventional second-order based algorithms for convolutive mixtures that exploit signal nonstationarity. Index Terms—Blind source separation, convolutive mixtures, frequency domain, orthogonal/nonorthogonal constraints, penalty function, speech signals.

I. INTRODUCTION HE objective of blind source separation (BSS) is to extract the original source signals of interest from their mixtures and possibly to estimate the unknown mixing channel using only the information within the mixtures observed at the output of each channel with no, or very limited, knowledge about the source signals and the mixing channel. A challenging BSS problem is to separate convolutive mixtures of source signals, where the observed signals are assumed to be the mixtures of linear convolutions of unknown sources. This is an issue in several application fields, of which the most famous is the cocktail party problem, where the name comes from the fact that we can hold a conversation at a cocktail party even though other people are speaking at the same time within an enclosed

T

Manuscript received September 11, 2003; revised May 15, 2004. This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) of the United Kingdom. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Hamid Krim. The authors are with the Cardiff School of Engineering, Cardiff University, Cardiff, CF24 0YF, U.K. (e-mail: [email protected]; [email protected]; [email protected]). Digital Object Identifier 10.1109/TSP.2005.845433

environment [1]; this classical definition can be extended to include the presence of a noise background due to the other interfering speech or music. In this case, the observed signals are the mixtures of weighted and delayed components of the sources. Convolutive BSS has been conventionally developed in either the time domain (see [2] and the reference therein) or the frequency domain, e.g., [3]–[18]. In this paper, we focus on the operation in the frequency domain. In contrast to time domain approaches, where a large number of parameters has to be estimated, frequency domain approaches usually simplify the convolutive problem into an application of instantaneous methods to each frequency bin. As a result, the frequency domain approaches generally have a simpler implementation and better convergence performance, although the downside of arbitrary permutations and scaling ambiguities of the recovered frequency response of the sources at each frequency bin remain open problems in many realistic environments, such as where the sources are moving. The representative separation criterion used in the frequency domain is the cross-power spectrum-based cost function, where (shown in Section II) generally leads to a trivial solution to the minimization of the cost function. A selective method, as has been used in some general ICA approaches, is to incorporate a term of in the cost function to be the form minimized to ensure that the determinant does not approach zero. Alternatively, one can also use a special initialization method or a hard constraint to the parameters to avoid this degenerate solution. In this paper, in contrast, we propose a new approach based on a penalty function for convolutive blind separation in the frequency domain. This new criterion is motivated by nonlinear programming techniques in optimization. As will be shown, incorporating a penalty function within the second-order statistics (SOS)-based cost function for nonstationary signals, the degenapproaching zero is automatically erate solution due to removed. Another objective of this paper is to use a penalty function to unify the concept of joint diagonalization with unitary or nonunitary constraint. Under such a framework, the constrained joint diagonalization problem is converted into an unconstrained optimization problem, and the main task is then to focus on the choice of the form of penalty functions associated with appropriate application models and the adaptation of the penalty parameters. This is particularly useful for applying some well-known optimization algorithms and therefore developing new algorithms for BSS. Moreover, such a criterion has

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

a close relation to the joint diagonalization methods with projection onto the Stefiel manifold. The state-of-the-art schemes, using either orthogonal or nonorthogonal matrix constraints, appear to be related to the new criterion and may lead to new viewpoints and algorithms for joint diagonalization regarding conventional optimization methods such as the Newton and gradient descent algorithms. As will be shown in Sections VI and VII, the penalty function may lead to improved performance for the solution of the BSS problem, and the implementation of a nonunitary constraint in a later example indicates an additional ability of the proposed scheme to reduce the scaling ambiguities. In this paper, we assume that the source signals are nonstationary. This is especially true for most real-world signals whose statistical properties are very often statistically nonstationary or (quasi-) cyclostationary, such as speech and biological signals. In [26], we have shown that the cyclostationarity of the signals can be exploited in order to generate new algorithms that outperform some conventional BSS algorithms for some real-world cyclic-like signals such as ECGs. However, as observed in [26], cyclostationarity may not be valid for speech signals whose statistics vary very rapidly with time. In this case, it is difficult to estimate accurately their cycle frequencies. A natural and better way to describe speech signals is to consider their nonstationarity. To this end, we exploit the SOS of the covariance matrices to estimate the separation matrix. In comparison to higher order statistic (HOS) methods, although using SOS is insufficient for separation of stationary signals [19], and therefore, HOS has to be considered either explicitly [20] or implicitly [3], SOS methods usually have a simpler implementation and better convergence performance and require fewer data samples for time-averaged estimates of the SOSs. Moreover, they can potentially overcome the non-Gaussianity assumption for source signals [5], which is nevertheless necessary for HOS-based methods. This paper is organized as follows. In Section II, the convolutive BSS problem and the frequency domain approach are discussed. A brief discussion on the problem of second-order nonstationarity of speech signals and the off-diagonal separation criterion is given in Section III. In Section IV, several joint diagonalization criteria are summarized, and the concept of constrained BSS is introduced. This provides the fundamental background for this work. In Section V, the penalty function-based joint diagonalization method is described in detail. In Section VI, a unifying analysis will be given to the unitary and nonunitary constraints within joint diagonalization under the penalty function framework. An implementation example with gradient adaptation is also included in this section. Section VII presents the simulation results, and Section VIII concludes the paper.

1655

modeled as a weighted sum of convolutions of the source signals corrupted by additive noise, that is (1) is the th element of the -point impulse response where from source to microphone , is the signal is the signal received by microphone , is from source , the additive noise, and is the discrete time index. All signals are assumed zero mean. Throughout the paper, both the mixing and unmixing processes are assumed to be causal finite filter models, i.e., FIR filters as in (1). We note that it is also possible to exploit noncausal models [27], [28], but this issue is, however, beyond the scope of this paper. Using a -point windowed discrete Fourier transformation (DFT), time-domain signals can be converted into frequency-domain time-series signals as (2) where denotes a window function, , and is a frequency index . We use , , and , in closed-form expressions for which does not depend on the time index due to the assumption that the mixing system is time invariant. The same assumption will be applied to the separation system as follows. As shown in [4], a linear convolution can be approximated by a , that is circular convolution if (3) and are the time-frequency representations of the source signals and the observed signals, respectively, and denotes vector transpose. Equation (3) implies that the problem of convolutive BSS has been transformed into multiple instantaneous (but complex-valued) BSS problems at each frequency bin. , we can alternatively Rather than directly estimating , i.e., at estimate a weighted pseudo-inverse of every frequency bin by using a backward discrete-time model [15], [16] where

(4) where is the time-frequency representation of the estimated source signals, and is the discrete time index. The parameters in are determined become mutually so that the elements independent. The above calculations can be carried out independently in each frequency bin.

II. CONVOLUTIVE BSS IN THE FREQUENCY DOMAIN

III. SECOND-ORDER NONSTATIONARITY AND OFF-DIAGONAL CRITERION

Assume that source signals are recorded by microphones (here, we are particularly interested in acoustic appli. The output of the th microphone is cations), where

Define a cost function as a measure of inde, for ; the separation pendence between problem then becomes an optimization problem for which we so that have to find a method to minimize

TLFeBOOK

1656

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

are as independent as possible. A necessary condition to exploit the SOS conditions for nonstationary signals is to ensure that are mutually uncorrelated. Nonstationarity of the speech signals can be generated in various ways, e.g., variation of the vocal tract filter and glottis, whereby neighboring samples have varying cross correlation, or even higher order moments. Here, we resort to the cross-power spectrum of the output signals at multiple times, i.e., (5) is the covariance matrix of , and is the covariance matrix of . The objecthat (at least approximately) jointly tive is to find a diagonalizes these matrices simultaneously for all time blocks . That is, , is an arbitrary diagonal matrix, which, where based on the independence assumptions of the source signals and the sensor noise, can be derived by the following equation: where

(6) and are the different diagonal covariwhere ance matrices of the source and noise signals for each , respectively, and denotes the Hermitian transpose operator. As in [4], an intuitive criterion is defined to minimize a function of the between and error (7) where is the squared Frobenius norm. It is straightforward to show that this separation criterion is equivalent to (denoted by problem ) (8) where

is defined as diag

(9)

where diag is an operator that zeros the off-diagonal elements of a matrix. clearly leads to the minimization of The solution , which is a degenerate solution to the minimiza. This means that a cost function sufficient for tion of joint diagonalization should enforce a solution for and, hence, a constraint on the former cost function. On the other hand, the representative approaches (mostly working in the time domain) using joint diagonalization project the separation maonto the Stefiel manifold to generate a unitary constraint trix ). Nevertheless, as shown in [22], such a (cf. [21], . Ununitary constraint is not always necessary like the method using a hard constraint on the initialization or at each iteration (see [4]), here, we present an alternatively new . This method is approach to satisfy the constraint motivated by nonlinear programming optimization theory [23] and, at the same time, unifies the orthogonal and nonorthogonal

matrix constraint methods and therefore provides new insight into joint diagonalization algorithms for BSS (see Section VI). is omitted hereFor simplicity, the frequency index in after where appropriate. IV. JOINT DIAGONALIZATION CRITERIA AND CONSTRAINED BSS A. Criteria The objective of joint (approximate) diagonalization is to find a matrix that simultaneously (approximately) diagonalizes a set of matrices. The typical application of joint diagonalization for BSS is to achieve separation by using (very often approximating) joint diagonalization of a set of covariance matrices. In comparison with the method using only one covariance matrix, approximate joint diagonalization of a set of covariance matrices increases the statistical reliability of the procedure for estimating the unmixing matrix and reduces the possibility of unsuccessful separation induced by choosing an incorrect time-lag of covariance matrices, therefore significantly increasing the robustness at the cost of increased computation. Letting the set of matrices to be jointly diagonalized be , the widely used off-diagonal criterion for the joint diagonalization stated in Section III can be expressed generally as

where is a joint diagonalizer, which is also referred to as the separation matrix in BSS, and , where is an matrix. Here, we try to use the same notation as stated in Section III. This criterion is simple to implement and can be theoretically justified. The Jacobi method, the gradient method, and the Newton method can all be used to minimize this criterion [4], [25], [34], [55], [56]. These methods can also be classified into either of those works based on SOS, as in [34], or those works based on HOS, as in [25]; most of them consider the orthogonality constraint. Unlike this criterion, a log likelihood criterion proposed in [31] is more suitable for measuring the diagonality of a set of positive definite Hermitian matrices without the orthogonality constraint diag

where is a factor measuring the distribution of . Another characteristic of this criterion is its invariance property regarding a scale change that is not held by the off-diagonal criterion. Recently, a new criterion based on the Cayley transform has been proposed in [57]. The criterion is performed in two steps:

where are the different unitary diagonalizers produced by eigendecomposition of covariance matrices for different time

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

lags, and is the inverse Cayley transform of a weighted average matrix of the individual unitary matrices. The similarity between these two matrices provides a reasonable measure for joint diagonalization since different unitary matrices, with identical dimensions, are equivalent. More recently, an interesting linear criterion was proposed in [32]

1657

data [54]. This brief overview, together with a recent contribution to a similar work in [53], justifies that imposing an approor the estimated priate constraint on the separation matrix source signals with special structure, such as invariant norm, orthogonality, geometry information, or non-negativity, provides meaningful information to develop a more effective BSS solution, especially for real-world signals and practical problems. V. PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

where diag , are constant diagonal matrices, which can be adjusted to have similar orso that the effect ders as the diagonal elements of of closely spaced eigenvalues of the matrices to be jointly diagonalized can be canceled. However, due to the order of diagat critical points being unknown onal elements of a priori, this criterion cannot work by itself. As will be elaborated in Section VI, however, the different joint diagonalization criteria, considering the case either with as in [21], [25], orthogonal (unitary) constraints [32], and [57], or with nonorthogonal (nonunitary) constraints , as in [4], [31], and [33], can be unified by using the penalty function (see Section V-A) as an unconstrained optimization problem. This makes it easier for us to carry out the theoretical analysis for the behavior of the joint diagonalization criterion for BSS problems, where the penalty function can be appropriately chosen to comply with the orthogonal and the nonorthogonal constraints. After such processing, to minimize the cost function in (16) (see Section V-A), we can resort to any unconstrained optimization algorithms such as the gradient-based (including steepest descent and conjugate gradient) methods, (quasi-)Newton methods, or any other alternatives.

A. Approach Constrained optimization problems can be effectively transformed into unconstrained ones by using the penalty function method, which adds a penalty term to represent constraint violation within the objective function [23], [24], [42]. A penalty function is a non-negative function, which is zero within the feasible area and positive within the infeasible area. A feasible point corresponds to one where all the constraints are satisfied. Typical constrained optimization problems are either subject to an equality constraint, an inequality constraint, or both. In this paper, we are particularly interested in the former type, due to the fact that the most general applications in BSS are subject to an equality constraint. Recalling a standard nonlinear equality constraint optimization problem in [24] (also see [23]) and extending the idea to the optimization for matrices, we have the following equality constraint optimization problem: s.t. where , , , and that there may exist more than one constraint. Let

(10)

indicates

B. Constraints on BSS Generally, BSS employs the least possible information pertaining to the sources and the mixing system. However, in practice, there exists useful information to enhance the separability of the mixing system, which can be exploited to generate various effective algorithms for BSS. For example, a constrained parameter space specified by a geometric structure with has been exploited in [45], where the norm restriction of the separation matrix is necessary for practical hardware implementawithin the space of tions. The algorithms for adapting , were first proorthonormal matrices i.e., posed in [48] and [49] and then developed in [46]. A merit of such algorithms lies in the possibility of extracting an arbitrary group of sources. This idea has also been addressed as the optimization problem on the Stiefel manifold or the Grassman manifold in [21], [38], [46], [50], and [51]. The key principle of such methods is to exploit the geometry of the constraint surface to develop algorithms searching either along a geodesic (the shortest line between two points on a curve surface) [51] or other paths [21]. In [52], the natural gradient procedures were devel. The geooped to maintain a nonholonomic constraint on metric information of sources was considered in [44] as a constraint on BSS. Recently, a non-negative constraint on BSS has been shown to possibly be a useful way to represent real-world

so that . This means that problem (10) can be refor, s.t. . This is equivalent to the mulated as standard formulation in [24], which can, on the other hand, be and will be deemed as a case for (10) by letting further utilized in Section VI. The transformation between the vector form and matrix form in the derivation of the algorithms can be followed as in [40]. In this paper, the complex-valued case is considered for generality. This is simply done by considering the real and imaginary parts simultaneously. can be reformulated following any In the BSS context, of the joint diagonalization criteria in Section IV-A, and can take the form of any constraint in Section IV-B. Penalty functions can be classified into two categories: exterior and interior. For the equality constraint problem (10), an exterior penalty function fits best due to the fact that interior penalty functions can only be used for sets defined by inequality constraints. We next give a self-contained definition of the exterior penalty function below. be a closed subset of . A seDefinition 1: Let , quence of continuous functions

TLFeBOOK

1658

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

is a sequence of exterior penalty functions if the following three conditions are satisfied: (11) (12) (13) A diagramatical representation of such functions is shown in Fig. 1. Suppose that is a real-valued non-negative penalty , i.e., function vector corresponding to the constraints , is a coefficient vector that controls and that the penalized values for the corresponding constraints; then, by using penalty functions, (10) is changed to (14) An important issue for the successful implementation of the penalty function method is to choose a proper penalty function for the cost function since not all penalty functions ensure the minimization of the cost function [43]. In general, there are different penalty functions that can be utilized to characterize different problems. It is unlikely that a generic penalty function exists that is optimal for all constrained optimization problems. In Section VI, two penalty functions are introduced for both the unitary and nonunitary constraint, respectively. These two penalty functions are quite suitable for our applications in terms of the theoretical basis given below in Lemma 1 and numerical experiments given in Section VII. However, it should be noted that for a practical application with desired purpose, it is difficult to define a generic scheme to guide the user in terms of choice of penalty functions, but our work demonstrates that practically useful schemes can be found. To better understand the following discussion, we consider the following lemma. Lemma 1: Suppose that is continuous . For , let and that be such that and , as , and let . If we define as

WW W

W

Fig. 1. U ( )(i = 0; 1; . . . ; 1) are typical exterior penalty functions, where U ( ) < U ( ) < 1 1 1 < U ( ) and the shadow area denotes the subset W .

in (8). In an expansion form, the criterion is denoted by . The separation problem is thereby converted into a new unconstrained joint . diagonalization problem, i.e., Minimization of is approximately equivalent to the . However, note that in fact, problem of minimization of the solution to problem (10) does not guarantee a solution for represents a set of exact problem . Only when penalty functions for (10) are the solutions equivalent [24]. Incorporating penalty functions into a joint diagonalization criterion enables us to convert constrained joint diagonalization problems into unconstrained ones and unify several methods using joint diagonalization for BSS; it can be used to construct special algorithms (including globally stabilizing locally converging Newton-like algorithms) for joint diagonalization probin (8) can be easily satislems. The constraint in (16) fied by choosing a suitable penalty function so that the degenerate solution can be automatically removed. As will be revealed in Sections V-B and VII, a proper penalty function may adapt the separation coefficients in a more uniform way, which generally indicates a better convergence performance (also see [18]). Assuming that the penalty functions are in the form of , and is the vector of (15), e.g., , where , perturbations of then minimization of the criterion with the equality constraint is equivalent to

(15) where , 2, or , then means that the exterior penalty function projects the space from one into another will form a with a possibly different dimension. Then, sequence of exterior penalty functions for the set . We omit the proof of Lemma 1 since it is straightforward to verify it by following Definition 1. The penalty function given in (15) forms a set of differentiable penalty functions that will be used in the following discussion. Using a factor vector to absorb the coefficient and incorporating exterior penalty functions (15), our novel general cost function becomes (16) where is a set of penalty functions with desired properties that can be designed to meet our requirements, are the weighting factors, and represents a basic joint diagonalization criterion, such as

(17) is the perturbation function defined as the optimal where value function for the equality constraint problem. Equation (17) implies that by adding the term , an attempt is as increases, and as , the made to convexify perturbation value approaches zero. This indicates that by increasing the value of the penalty parameter, the penalty function-based criterion has the ability to self-adjust in the presence of the perturbation. Similar discussions are given to show the stability of the algorithms for unitary and nonunitary constraints in Section VI. It is worth noting that the criterion (16) has a similar form as that of exploiting well-known Lagrangian multipliers; however, they are essentially two different approaches for the constrained optimization problem. Using the penalty function, the

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

solution to (14) approaches the same solution to (10) while satisfying the constraint simultaneously. When the penalty coeffi, , and hence, can be cients satisfied. For the nonlinearly constrained problem, local minimizers of (10) are not necessarily local minimizers of the Lagrangian-based cost function so that it is not possible to design consistent line search rules based on the Lagrangian function, which is nevertheless especially important for smoothly connecting global optimization techniques with an effective asymptotic procedure, e.g., [41]. In this case, a better choice is to construct a global strategy for problem (10) based on penalty functions. In addition, for a nonconvex problem, a penalty function-based method can tend to the optimal solution for the original problem with increasing penalty coefficients ; however, the Lagrangian approach would fail to produce an optimal solution of the primal problem due to the presence of a duality gap [23].

1659

nels in a direct-sequence code division multiple access system. This idea is based on the following corollary [24]. Corollary 1: If for some , then is an optimal solution to the problem (10). Therefore, in our practical application, we can relax the theoretical condition that , provided , and attain performance advantage with finite . This also mitigates another problem in implementation of the penalty function method, i.e., that the introduction of penalties may transform a smooth objective function into a rugged one. To avoid local minima in the objective function, the penalized joint diagonalization cost function must preserve the topology of the objective function to allow location of the optimal feasible solution; see [24] for more details. VI. PENALIZED JOINT DIAGONALIZATION WITH UNITARY/NONUNITARY CONSTRAINT A. BSS With Unitary and Nonunitary Constraint

B. Global Minima and Local Minima The equivalence between (16) and (10) follows from the results in [24], which provide the fundamental theoretical basis for our approach, including its global and local minima given by the following lemmas. We define the sets and , where , and we assume the following: A1) s.t. is compact, and A2) s.t. . As is convention with the use of penalty functions in nonlinear optimization [24], the th trial, where , corresponds to one setting of a scalar penalty coefficient, which is denoted , and the accumulation point is the minimizer when . Then, we have two lemmas. Lemma 2: If is the minimizer of at the th trial, then in the limit, as , , the accumulation point is as in assumption A2. Lemma 3: If , is a strict local minimizer of at the th trial, i.e., for some , , then in , there is an accumulation point , as the limit as in assumption A2, for which , s.t. , , and is a local minimizer of (10). Assumption A1 ensures that problem (10) has a solution, and A2) ensures that the closure of the set contains an optimal solution to problem (10). The lemmas imply that the new criterion (16) holds the same global and local properties as that without the penalty term when given large enough penalty parameters. This means that the choice of the penalty parameters usually has a major effect on the overall optimization accuracy in practical implementation. Too small parameters will lead to an inexact or even incorrect final solution, whereas too large values may create a computationally poorly conditioned or strongly nonlinear energy function [43]. Unlike the standard penalty function method ultimately requiring infinite penalty parameter , in practice, with time-varying signals, we employed a finite to obtain a performance advantage as supported by our simulation results. A similar technique is used in [39] for blind detection of desired signals from multiple chan-

For unitary constraint, the problem is formulated as the minimization of a cost function under the constraint . That is, is a function of the subspace spanned by the columns of . It has been shown that there are several potential approaches to solving the unitary constrained BSS problem, including the whitening approach (e.g., [25], [34], and [37]), projection on the Stiefel–Grassman manifold (e.g., [5], [21], [38], and [46]), and other approaches (e.g., [33] and [35]). The algorithms were developed with the aim of restricting the separation matrix to be orthogonal. From the discussion in Section IV-B, the constrained minimization of a cost function is converted into the unconstrained optimization on a manifold. Particularly, the orthogonal constrained optimization problem in [21] can be treated as a special case of the unconstrained problem in (16) so that Lemma 1 is applicable for such a case. This idea can find a number of applications in the areas of signal processing and linear algebra, such as joint diagonalization, subspace tracking, or singular value decomposition [5], [25], [29], [34], [35]. However, the methods using unitary constraint may result in a degraded performance in the context of BSS, due to the unbalanced weighting of the misdiagonalization between all the matrices being jointly diagonalized. The nonunitary methods were therefore addressed in a more general way [18], [22], [30], [31], [33]. A common characteristic of the works in [22], [31], and [30] is that no hard whitening is exerted on the separation process. In [33], a combination of several possible cost functions was proposed for BSS; however, no details were found to address the behavior and performance of such combinations. In [18], the idea of using penalty functions for nonunitary constraint with application to frequency-domain BSS was presented. This work, regardless of initialization, and similar to that in [22], is different from [30], which is typical for Gauss–Newton-type algorithms and needs the separation matrix to be properly initialized. B. Unifying by Penalty Function With Perturbation Analysis Compared with the methods discussed in Section VI-A, which all considered instantaneous mixtures in the time do-

TLFeBOOK

1660

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

main, the method addressed here operates in the frequency domain. Incorporating the penalty function discussed earlier, the total cost function becomes

(18) where is a penalty weighting factor. The key problem is now to choose a suitable penalty function. For a unitary constraint and a nonunitary constraint, the alternative but equivalent forms can be chosen as (see [46]) and diag (see [33]). If we choose the penalty function to be in the form of or diag , we will see that using a penalty function with a limited norm will control the size of the designed parameters. This property would be especially useful when the cost function is not explicitly dependent on the design parame, where the case may happen in some optiters, such as mization problems that are ill-posed in the sense that solutions with bounded controls do not exist. Note that the compact assumption A1 in Lemma 2 and 3 is addressed with respect to the in (18), regardless of the form objective function . In addition, even if of the constraint, e.g., is complex valued, assumption a) is still satisfied due to the real-valued penalty function being used. Under the penalty function-based framework, either a unitary or a nonunitary constraint problem can be deemed as an example of a penalized unconstrained optimization problem, whereas the forms of the penalty functions may be different. We may choose a penalty function with the form of (see [39] for a similar application) or Tr (see [36] for a similar application), where is a diagonal matrix containing the Lagrangian multipliers. It would be useful to examine the eigenvalue structure of the , which dominates Hessian matrix of the cost function, i.e., the convergence behavior of the algorithms used for minimizing . Following a similar procedure as in Appendix A, by and assuming that calculating the perturbation matrix of the cost function is twice-differentiable

[23]. This means . Combining (19)

as

that and (20), we have

(21) The first two terms on the right-hand side of (21) approach the Hessian of the Lagrangian function . The last term in (21) strongly relies on . It can , has some eigenvalues apbe shown that as proaching , depending on the number of the constraints, and the other eigenvalues approach finite value. The infinite eigenvalues reveal the reason behind the ill-conditioned computation problem mentioned in Section V-B. Letting be the step size in the adaptation, perturbation of the cost function in the iterative . In the presence of update is then nonlinear equality constraints, the direction may cause any to be shifted by . This reduction of needs the step size to be very small to prevent the ill-conditioned computation problem induced by large eigenvalues at the expense of having a lower convergence rate. C. Gradient-Based Adaptation In this section, we develop a gradient-based descent method to adapt the coefficients in (18). The method is actually a least squares (LS) estimation problem. Although the convergence rate of the gradient method is linear as compared to the Newton-type adaptation method, which can achieve quadratic convergence by using second-order derivatives, a gradient-based adaptation method usually has the ability to converge to a local minimum, whereas no guarantee exists for a Newton-type method, which may converge to the nearest critical point. One of the other advantages of the gradient-based adaptation method is its simplicity of implementation as well as its lower computational complexity. Let the penalty function be in (18). For a unitary constraint (see [33] and [46]), the gradients in (18) with respect to their parameters can be readily derived as (see Appendix A)

(19) where

is the Kronecker product, diag vec , and is a matrix whose elements

are all ones

(22) diag (20)

diag (23)

The

is the linear combination of and . The conditions of Lemma 2 indicate that , will approach the optimum . If as is a regular solution to the constrained problem, then there exist unique Lagrangian multipliers such that

Theoretically, incorporating the penalty term is equivalent to applying a projection operation on the gra; however, even for the latter method, the dient of penalty function is useful to attain a stable performance [59].

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

For a nonunitary constraint, the adaptation is parallel to the foregoing procedure. Letting the penalty function be in the form diag (see [33]), the adaptation of equations to find the coefficients in (18), corresponding to (22), become (see Appendix A)

diag

(24)

In this case, (23) takes the same form. It is worth noting that an alternative adaptation method to (24) is to use a modified gradient by using a Lagrangian multiplier, which can adapt in a more uniform way; see [18] for more details. Note that theoretically, when we update the algorithm by using rules (22) and (24), after it converges, the norm of in (15) approaches a finite value. When , the multiplication will approach infinity in terms of Lemma 1 and, hence, satisfy (13). However, in practical implementation, (13) is exploited approximately. Such consideration is based on the following two reasons. First, there exist numerical problems in practical implementations. Second, when the penalty function approaches infinity, it will dominate the objective function, which, however, should not happen for a desired purpose. In our implementation, the cross-power spectrum matrix remains important in the objective function, and at the same time, the separation matrix satisfies a desired property to some degree. Because the cross-correlation is time dependent for nonstationary signals, it is difficult to estimate the cross-power-spectrum with a relatively short stacan be tionary time resolution. However, in practice, estimated by the block form (25) where is the number of intervals used to estimate each cross power matrix. In implementation, the normalized step sizes for and take the forms adaptation of (26)

(27)

where , , and are scalar values adjustable for adaptation. The LS solution to (18) can be obtained using the well-known stochastic gradient algorithm. The solution will be in the form , where of denotes the complex conjugate operator. D. Scale and Permutation Indeterminacies There are possible indeterminacies in the model (1), such as sign, scale, spectral shape, and permutation [13], among which,

1661

the permutation ambiguity and the scale ambiguity are most important for the separation performance. In Section II, when we try to combine the results obtained at each frequency bin, the permutation problem occurs because of the inherent permuta. Here, we use the method tion ambiguity in the rows of in [4] to address the permutation problem with regard to the filter length constraint of the FIR model so that we can compare the performance of the proposed method with other traditional techniques. One promising advantage of using the penalty function is its potential to reduce the scaling ambiguity. Taking the form of the unitary constraint in Section VI-C as an example, we see that by using the penalty function, not only the constraint tends to be satisfied, but the norm of the separation matrix is restricted to a desired value as well. This is especially useful in the iterative update, and the amplitude of the signal at each frequency bin is preserved to have normal energy. Such an effect on the scaling ambiguity while incorporating the penalty function will be demonstrated by a simulation in Section VII. E. Implementation Pseudo-Codes According to the discussions given in the above sections, taking the penalty functions with unitary or nonunitary constraints as examples, we summarize the whole algorithms as the following steps:

, , , , , 1) Initialize parameters , , , , , , , and . 2) Read input mixtures, i.e., time samples : —For artificially mixing, is obtained using (1). is —For the real mixing environment, the signal recorded by a microphone array. 3) Calculate the cross-power spectrum matrix: to using (2); —Convert using (25). —Calculate 4) Calculate the cost function and update gradient: to —for Update and using (26) and (27) respectively; Update using (22) or (24); Update using (16) or (18); if break; —end. 5) Solve permutation problem , where is a function dealing with permutation operation (refer to [4]). according to (4). Re6) Calculate construct the time domain signals IDFT . 7) Calculate the performance index using (29) or (31). 8) End.

TLFeBOOK

1662

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 2. Convergence performance of the cross-power spectrum based off-diagonal criterion changes under the penalty function constraint with various penalty coefficients .

Fig. 3. Two source speech signals used in the experiment.

VII. NUMERICAL EXPERIMENTS In this section, we examine the performance of the proposed method by simulations when applied to both artificially mixed signals and real room recordings. The evaluations will be carried out in two aspects: convergence behavior and separation performance (see Fig. 2). A system with two inputs and two outputs (TITO) is consid(the simulations can be ered for simplicity, that is, readily extended into MIMO systems due to the general formulation of the problem in the previous sections). The two source speech signals used in the following experiments are shown in Fig. 3 (sampling rate 12 kHz), which are available from [62]. We artificially mix the two sources by a nonminimum-phase system

Fig. 4. Convergence performance of the penalty function with various penalty coefficients .

in (28) [47], shown at the bottom of the page. First, we investigate the convergence behavior of the proposed method by comparison with the method in [4], which corresponds to the case in our formulation in the previous sections, and the following simulations were specified. We develop the work in [18] and choose the penalty function to be in the form of a nonunitary constraint (see also [59]). Adaptation (24) is therefore used to examine the convergence performance of the proposed method. The parameters are set as follows. The number of intervals used . The to estimate each cross-power-matrix is set to be . The length of the fast Fourier transform (FFT) is parameters in the normalized step sizes given in (26) and (27) , , and , respectively. The are set to be . The step number of the matrices to be diagonalized is . The initial value of the size in gradient adaptation is . We applied the short term separation matrix is FFT to the separation matrix and the cross-correlation of the input data. Fig. 2 shows the behavior of the penalty coefficient affecting the convergence of the cross-power spectrum based off-diagonal criterion. Fig. 4 shows how the penalty function itself changes with the penalty coefficients when the cross-power spectrum based off-diagonal criterion was constrained by the penalty function. From Figs. 2 and 4, we see that the separation algorithm generally has increasing convergence speed with the increasing value of the penalty coefficient , as well as a quicker trend approaching the constraint. An increase in the value of introduces a stronger penalty being exerted to the off-diagonal criterion. This means, by increasing the penalty coefficient , that we may not only approach the constraint in a quicker way but also attain a better convergence performance. The convergence performance of the new criterion with penalty function, i.e., (18) is shown in Fig. 5, where the cost function converges to a lower value in a faster speed as compared with the

(28)

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

Fig. 5. Comparison of convergence performance of the new criterion (with penalty function constraint where  6= 0) and cross-power spectrum based off-diagonal criterion in [4] (without penalty function, where  = 0).

criterion in [4]. The convergence speed will be scaled by the is due penalty coefficient. The unusual behavior for to ill-conditioning and will be discussed in more detail later. More precise examination of the convergence behavior is given in Fig. 6, where the adaptation stops when a threshold is satisfied, and the current value will be taken as the stable value of the criterion. From Fig. 6, it can be seen that by an appropriate choice of the penalty coefficient (generally increasing ), we can obtain a significantly improved convergence speed, as well as a better performance, which is indicated by the lower level of the off-diagonal elements existing in the cross-correlation matrices of the output signals. Theoretically, increasing the value of the penalty coefficient will arbitrarily approach the constraint. However, as discussed in Sections V and VI, numerical problems will be induced due to large value of the penalty . Under a common step size, too large a penalty will introduce ill-conditioned calculation, and this can be observed in Figs. 2, 4, and 5 for the case of , where there is fluctuation in the adaptation. At the same time, although a larger penalty will put higher emphasis on the constraint, the theoretical analysis in Section VI showed that the constraint itself has no necessary connection to the performance improvement. This ). However, the property can be confirmed in Fig. 6 (see of performance improvement with increasing penalty appears to be promising. An alternative method to fully exploit such a property is to decrease the step size . Fig. 7 showed that the ) can numerical ill-condition problem in Fig. 5 (when be removed by reducing the step size to a smaller value. This indicates that we can choose a suitably small step size to suppress the numerical problem. However, as shown in the case of and in Fig. 7, this means that a stable numerical performance for large penalties will be achieved at a small cost of convergence speed. Therefore, an optimal penalty coefficient will be obtained by a tradeoff consideration of the step size in practical implementations. As discussed in Section VI-D, there are inherent permutation and scaling ambiguities in BSS algorithms. Here, we will show

1663

Fig. 6. Comparison of the stable values and the required iteration numbers (to reach such stable values) between the new criterion and the conventional cross-power spectrum-based criterion. (a) Corresponding to the criterion in [4] with  = 0. (b)–(g) Corresponding to the new criterion with penalty coefficient = 6 0.

Fig. 7. Influence of step size on the penalty coefficients , where the step sizes  in (a)–(c) are set to be 1, 0.1 and 0.01, respectively, whereas the penalty coefficients  keep the same to be 10. The adaptation stops when a threshold is satisfied, and the stable value of the cost function is obtained at this point.

the ability of the penalty function to suppress the scaling ambiguity by simulations and addressing the permutation problem, as in [4], to allow performance comparison. Theoretically, due to the independence assumption, the cross-correlation of the output signals in (5) should approximately approach zero. Fig. 8(a) shows the results using the off-diagonal criterion (8). From these results, we see that it is true at most frequency bins but not for very low frequency bins. From the remaining figures in Fig. 8, we see that such an effect can be significantly reduced using penalty functions by careful choice of the value of penalty coefficients used in the experiment, and the effect is almost removed when is close to 0.01 in this case. It should be noted that the imaginary part is not plotted due to its similar behavior. An alternative to this problem can be found in [58], where such an effect was restricted by a normalization process using the diagonal elements of the cross-power matrices.

TLFeBOOK

1664

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 9. Comparison of the estimate error between the proposed method and the method in [4] using waveform simularity measurement (step size  = 0:06).

difference. By assuming the signals are zero-mean and unit-variance, we have dB (29)

Fig. 8. Overlearning observation by calculating the values (real part) of the (!; k ) at each off-diagonal elements of the cross-correlation matrices frequency bin. (a) Corresponding to the case without penalty function as in [4]. (b)–(d) Corresponding to the cases with penalty functions and penalty coefficients  are 1, 0.1, and 0.01 respectively.

R

To further evaluate the performance of the proposed method, we measure the resemblance between the original and the reconstructed source waveforms by resorting to their mean squared

(this alWe still use the speech signals in Fig. 3 and set lows us to select a large penalty according to the previous simu). lations, and therefore, the convergence is smooth when Other parameters are the same as those in the previous experiments. We perform seven tests by changing the penalties. The estimate error was plotted in Fig. 9 on a decibel scale, where the estimate error in [4] is not influenced by the penalties. From this simulation, we can clearly see that the separation performance is improved with an increasing penalty and can reach . However, with further increase of up to 16 dB when the penalties, the separation performance may degrade due to the reason discussed previously. To show the improved performance of the proposed method in reconstructing the original source signals, we give another example. The two source signals are available from [61]. Both of the signals are sampled at 22.05 kHz with a duration of 9 s. The samples are 16-bit 2’s complement in little endian format. One source signal is a recording of the reading sound of a man. The other is a street and . acoustic background. The parameters are The improved separation performance can be directly perceived dB dB through Fig. 10. The estimation errors are dB dB for the method in [4] and the proand posed method, respectively (calculated through all samples). From Fig. 10, we see that the proposed algorithm reduces the amplitude ambiguity to a lower value at the same time, and this is the direct result of the fact shown in Fig. 8. Let us examine the separation quality by using a more complicated mixing process, which will involve more time delays

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

1665

Fig. 10. Separation result of (a) a reading sound and (b) a street background sound (c), (d) mixed by a nonminimum-phase system with (g), (h) the proposed algorithm, as compared to (e), (f) the algorithm in [4].

and cross-talk between independent speech signals. The mixing matrix is (30) where ;

Fig. 11. Separation results of (a), (b) two speech signals mixed with FIR systems (30) with (c), (d) the proposed algorithm, as compared to the method in (e), (f) [4] and (g), (h) [62] by spectrogram. The performance comparison can be examined with the highlight arrows.

TLFeBOOK

1666

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

; ; . The parameters in this simulation are the same as the initial . The step setting in the first simulation, except for , to allow comparison for size remains the same, i.e., the proposed method and the method in [4], which has been tuned empirically to an approximate optimum value for this test. The estimation errors of [4] and the proposed method are dB dB and dB dB, respectively. The separation results are now plotted by spectrogram, which is a clearer way to examine the time-frequency representation of speech signals, and shown in Fig. 11. To calculate the spectrogram, we use a Hamming window with length 500, the size of the overlap is 450, and the length of the FFT is 512. To evaluate the separation result more generally, the comparison was made to the method in [7] as well. By close observation to Fig. 11, we found that the cross talk in the recovered signals by the proposed method has been reduced to a lower level, as compared to the other two methods. Another method to quantify the performance is using signal to interference ratio (SIR) [4],

SIR

Fig. 12. Simulate a room environment using simroommix.m, where the sources are set in symmetric positions.

(31)

We conduct another experiment to further evaluate the proposed method by testing the SIR improvement, which will be compared with the methods in [4] and [8], respectively. The method in [4] is a representative joint diagonalization method using the off-diagonal criterion and SOS. The approach in [8] directly implements the Jacobi angle-based joint diagonalization approach in [25] to the frequency domain BSS. Note that we use a variation of this approach so that the permutation problem is addressed in the same way as in [4]. To this end, we employed the simulated room environment, which was realized by a roommix function available from [63]. One promising characteristic of this function is that one can simulate any desired configuration of speakers and microphones. In our experiment, the room is assumed to be a 10 10 10 m cube. Wall reflections are computed up to the fifth order, and an attenuation by a factor of two is assumed at each wall bounce. We set the position matrices , of two sources and two sensors, respectively, as (see Fig. 12). This setting constructs highly reverberant conditions. The parameters were set the same as those in Fig. 11. The SIR is plotted in Fig. 13, which indicates that the separation quality increases with the increasing filter length of the separation system. The performance is highly related to the data length, and it is especially clear when the filter length becomes long. Fig. 13 also indicates that incorporating a suitable penalty can increase the SIR; however, it seems that the proposed method suffers more heavily from the data length for a long filter. Additionally, incorporating a penalty may change the local minima, which can be observed from Fig. 13, as the

Fig. 13. SIR measurement for a simulated room environment with high reverberance.

SIR plots are not smooth, and the increased amplitude of the two methods is not consistent with each other. From Fig. 13, we can also see that exploiting spectral continuity of the separation matrix (the proposed method and that in [4]) may have superior performance to the method (e.g., [8]), which considers the separation at each frequency bin independently. It should be noted that although the simulations are based on the nonunitary constraint, the implementation of the unitary constraint can be readily followed in a similar way. The influence (including the situation of the penalties affecting the convergence behavior and separation performance, as discussed previously in this section) of the penalty function with unitary constraint on the conventional loss criterion complies with the same regulation discussed in Sections V and VI. However, as observed in certain literature (see [22]), considering a nonunitary constraint may outperform a unitary constraint because of the unbalanced weighting of the misdiagonalization in the unitary constraint. On the other hand, it is necessary to employ a technique such as in [46] to preserve the unitary constraint during adaptation. However, a fully unitary constraint does not necessarily indicate a good separation performance (see Section VI). In practical situations, it is not possible and not

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

necessary to totally fulfill the constraint. According to the simulations in this section, a finite penalty can normally generate a satisfactory solution (also see [39]). One of the downsides of the proposed method is that although introducing a penalty function can improve the convergence performance and separation quality, it will inevitably increase the computational complexity, depending on the penalty function being used in the algorithm. Therefore, it is required to consider the practical situations of different applications when implementing the proposed method. Another problem in the implementation is that one needs to consider how much penalty should be incorporated at every iteration, and an improper penalty may lead the algorithm to fluctuation; a possible solution to this problem has been recently addressed in [60].

1667

where is a small scalar, and value. On the other hand

is the real part of a complex

(33) where panded separately

, and

can be ex-

diag diag (34)

VIII. CONCLUSIONS A new joint diagonalization criterion proposed for separating convolutive mixtures of nonstationary source signals in the frequency domain has been presented. Using the cross-power spectrum and nonstationarity of speech signals, the algorithm incorporates a penalty function to the conventional cost function in the frequency domain, which leads to a different search direction to find the minimum of the cost function. The new criterion transforms the separation problem into the joint diagonalization problem with unconstrained optimization, which provides a unifying way to look at the orthogonal and nonorthogonal constraint joint diagonalization methods. An implementation example with nonunitary constraint and the evaluation by numerical experiments verified the effectiveness of the proposed criterion. It has been shown that a suitable penalty function may lead the algorithm to a better performance for the separation of the convolved speech signals, in particular, in terms of shape preservation and amplitude ambiguity reduction, as compared to the second-order-based nonstationary algorithm for convolutive mixtures. APPENDIX DERIVATION OF (22) AND (24) To calculate the gradient of the cost function, we can resort to the first-order Taylor expansion of the cost function with respect to a small perturbation. Suppose that the perturbation matrix of is . Then the perturbation of (18) reads separation matrix

tr

(32)

tr

Omitting the time and frequency index for simplicity and regarding (5) and (9), (34) becomes (35) and (36), shown at the , were apbottom of the page, where plied in (36). For the unitary constraint , we have the perturbation

tr

(37)

For the nonunitary constraint , we have the perturbation

diag

diag diag tr

diag diag (38)

Substituting (36) and (37) into (33) and comparing with (32), we have the adaptation (22) for penalized joint diagonalization with unitary constraint. Similarly, substituting (36) and (38) into (33) and comparing with (32), we obtain the adaptation (24) for penalized joint diagonalization with nonunitary constraint. It is

tr

diag

(35)

tr tr

diag tr

(36)

TLFeBOOK

1668

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

worth noting that the adaptation (23) with respect to the noise can be obtained by following the same procedure. ACKNOWLEDGMENT The authors acknowledge the insightful comments provided by the Associate Editor and the anonymous reviewers, which have added much to the clarity of the paper. REFERENCES [1] A. S. Bregman, Auditory Scene Analysis. Cambridge, MA: MIT Press, 1990. [2] A. Cichocki and S. Amari, Adaptive Blind Signal and Image Processing: Learning Algorithms and Applications. Chichester, U.K.: Wiley, 2002. [3] P. Smaragdis, “Blind separation of convolved mixtures in the frequency domain,” Neurocomput., vol. 22, pp. 21–34, 1998. [4] L. Parra and C. Spence, “Convolutive blind source separation of nonstationary sources,” IEEE Trans. Speech Audio Process., vol. 8, no. 3, pp. 320–327, May 2000. [5] K. Rahbar and J. Reilly, “Blind source separation of convolved sources by joint approximate diagonalization of cross-spectral density matrices,” in Proc. ICASSP, May 2001. [6] V. Capdevielle, C. Serviere, and J. L. Lacoume, “Blind separation of wideband sources in the frequency domain,” in Proc. ICASSP, 1995, pp. 2080–2083. [7] J. Anemüller and B. Kollmeier, “Amplitude modulation decorrelation for convolutive blind source separation,” in Proc. ICA, Helsinki, Finland, Jun. 2000, pp. 215–220. [8] N. Murata, S. Ikeda, and A. Ziehe, “An approach to blind source separation based on temporal structure of speech signals,” Neurocomput., vol. 41, pp. 1–24, 2001. [9] K. I. Diamantaras, A. P. Petropulu, and B. Chen, “Blind two-inputtwo-output FIR channel identification based on frequency domain second-order statistics,” IEEE Trans. Signal Process., vol. 48, no. 2, pp. 534–542, Feb. 2000. [10] B. Chen and A. P. Petropulu, “Frequency domain blind mimo system identification based on second- and higher-order statistics,” IEEE Trans. Signal Process., vol. 49, no. 8, pp. 1677–1688, Aug. 2001. [11] M. Z. Ikram and D. R. Morgan, “A multiresolution approach to blind separation of speech signals in a reverberant environment,” in Proc. ICASSP, May 2001. [12] S. Araki, S. Makino, R. Mukai, Y. Hinamoto, T. Nishikawa, and H. Saruwatari, “Equivalence between frequency domain blind source separation and frequency domain adaptive beamforming,” in Proc. ICASSP, May 2002. [13] M. Davies, “Audio source separation,” in Mathematics in Signal Separation V. Oxford, U.K.: Oxford Univ. Press, 2002. [14] W. E. Schobben and C. W. Sommen, “A frequency domain blind source separation method based on decorrelation,” IEEE Trans. Signal Process., vol. 50, no. 8, pp. 1855–1865, Aug. 2002. [15] J. Anemüller, T. J. Sejnowski, and S. Makeig, “Complex independent component analysis of frequency-domain EEG data,” Neural Networks, vol. 16, pp. 1311–23, Aug. 2003. [16] V. D. Calhoun, T. Adali, G. D. Pearlson, P. C. M. van Zijl, and J. J. Pekar, “Independent component analysis of fMRI data in the complex domain,” Magn. Resonance Med., vol. 48, pp. 180–192, Jul. 2002. [17] D. T. Pham, C. Servière, and H. Boumaraf, “Blind separation of convolutive audio mixtures using nonstationarity,” in Proc. ICA, Nara, Japan, Apr. 2003, pp. 975–980. [18] W. Wang, J. A. Chambers, and S. Sanei, “A joint diagonalization method for convolutive blind separation of nonstationary sources in the frequency domain,” in Proc. ICA, Nara, Japan, Apr. 2003, pp. 939–944. [19] S. Van-Gerven and D. Van-Compernolle, “Signal separation by symmetric adaptive decorrelation: Stability, convergence and uniqueness,” IEEE Trans. Signal Process., vol. 43, no. 7, pp. 1602–1612, Jul. 1995. [20] D. Yellin and E. Weinstein, “Multichannel signal separation: Methods and analysis,” IEEE Trans. Signal Process., vol. 44, no. 1, pp. 106–118, Jan. 1996. [21] J. H. Manton, “Optimization algorithms exploiting unitary constraints,” IEEE Trans. Signal Process., vol. 50, no. 3, pp. 635–650, Mar. 2002. [22] A. Yeredor, “Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation,” IEEE Trans. Signal Process., vol. 50, no. 7, pp. 1545–1553, Jul. 2002.

[23] M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear Programming Theory and Algorithms. New York: Wiley, 1993. [24] E. Polak, Optimization Algorithms and Consistent Approximations. New York: Springer-Verlag, 1997. [25] J. F. Cardoso and A. Souloumiac, “Blind beamforming for nongaussian signals,” Proc. Inst. Elect. Eng. F, vol. 140, pp. 362–370, 1993. [26] W. Wang, M. G. Jafari, S. Sanei, and J. A. Chambers, “Blind separation of convolutive mixtures of cyclostationary signals,” Int. J. Adaptive Control Signal Process., vol. 18, no. 3, pp. 279–298, Apr. 2004. [27] L. Zhang, A. Cichocki, and S. Amari, “Geometrical structures of FIR manifold and their application to multichannel blind deconvolution,” in Proc. IEEE Int. Conf. Neural Network Signal Process., Madison, WI, Aug. 1999, pp. 303–314. [28] T.-W. Lee, Independent Component Analysis: Theory and Applications. Boston, MA: Kluwer, 1998. [29] A. J. van der Veen and A. Paulraj, “An analytical constant modulus algorithm,” IEEE Trans. Signal Process., vol. 44, no. 5, pp. 1136–1155, May 1996. [30] A. J. van der Veen, “Joint diagonalization via subspace fitting techniques,” in Proc. ICASSP, vol. 5, May 2001. [31] D. T. Pham, “Joint approximate diagonalization of positive definite Hermitian matrices,” SIAM J. Matrix Anal. Appl., vol. 22, pp. 1136–1152, 2001. [32] G. Hori and J. H. Manton, “Critical point analysis of joint diagonalization criteria,” in Proc. ICA, Apr. 2003. [33] M. Joho and H. Mathis, “Joint diagonalization of correlation matrices by using gradient methods with application to blind signal separation,” in Proc. SAM, Aug. 2002. [34] A. Belouchrani, K. Abed-Meraim, J.-F. Cardoso, and E. Moulines, “A blind source separation technique using second order statistics,” IEEE Trans. Signal Process., vol. 45, no. 2, pp. 434–444, Feb. 1997. [35] M. Wax and J. Sheinvald, “A least-squares approach to joint diagonalization,” IEEE Signal Process. Lett., vol. 4, no. 2, pp. 52–53, Feb. 1997. [36] S. Costa and S. Fiori, “Image compression using principal component neural networks,” Image Vision Comput., vol. 19, pp. 649–668, 2001. [37] E. Moreau, “A generalization of joint-diagonalization criteria for source separation,” IEEE Trans. Signal Process., vol. 49, no. 3, pp. 530–541, Mar. 2001. [38] S. Fiori, “A theory for learning by weight flow on Stiefel-Grassman manifold,” Neural Comput., vol. 13, no. 7, pp. 1625–1647, Jul. 2001. [39] J. Ma and J. K. Tugnait, “A penalty function approach to code-constrained CMA for blind multiuser CDMA signal detection,” in Proc. ICASSP, May 2001. [40] M. Joho and K. Rahbar, “Joint diagonalization of correlation matrices by using newton methods with application to blind signal separation,” in Proc. SAM, Aug. 2002. [41] T. F. Coleman, J. Liu, and W. Yuan, “A quasi-Newton quadratic penalty method for minimization subject to nonlinear equality constraints,” Comput. Optimization Applicat., vol. 15, no. 2, pp. 103–124, Feb. 2000. [42] A. V. Fiacco and G. P. McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Technique. New York: Wiley, 1968. [43] A. Cichocki and R. Unbehauen, Neural Networks for Optimization and Signal Processing. New York: Wiley, 1993. [44] L. Parra and C. Alvino, “Geometric source separation: Merging convolutive source separation with geometric beamforming,” IEEE Trans. Speech Audio Process., vol. 10, no. 6, pp. 352–362, Sep. 2002. [45] S. C. Douglas, S. Amari, and S.-Y. Kung, “On gradient adaptation with unit norm constraints,” IEEE Trans. Signal Process., vol. 48, no. 6, pp. 1843–1847, Jun. 2000. [46] S. C. Douglas, “Self-stabilized gradient algorithms for blind source separation with orthogonality constraints,” IEEE Trans. Neural Networks, vol. 11, no. 6, pp. 1490–1497, Jun. 2000. [47] T. W. Lee, A. J. Bell, and R. Lambert, “Blind separation of delayed and convolved sources,” in Advances in Neural Information Processing Systems . Cambridge, MA: MIT Press, 1997, vol. 9, pp. 758–764. [48] J.-F. Cardoso and B. Laheld, “Equivariant adaptive source separation,” IEEE Trans. Signal Process., vol. 44, no. 12, pp. 3017–3030, Dec. 1996. [49] J.-F. Cardoso, “Informax and maximum likelihood in source separation,” IEEE Signal Process. Lett., vol. 4, no. 4, pp. 112–114, Apr. 1997. [50] U. Helmke and J. B. Moore, Optimization and Dynamical Systems. New York: Springer-Verlag, 1994. [51] A. Edelman, T. A. Arias, and S. T. Smith, “The geometry of algorithms with orthogonality constraints,” SIAM J. Matrix Anal. Applicat., vol. 20, no. 2, pp. 303–353, 1998. [52] S. Amari, T. P. Chen, and A. Cichocki, “Nonholonomic orthogonal learning algorithms for blind source separation,” Neural Comput., vol. 12, pp. 1463–1484, 2000.

TLFeBOOK

WANG et al.: PENALTY FUNCTION-BASED JOINT DIAGONALIZATION APPROACH

[53] A. Cichocki and P. Georgiev, “Blind source separation algorithms with matrix constraints,” IEICE Trans. Funda., vol. E86-A, no. 1, Jan. 2003. [54] M. D. Plumbley, “Algorithms for nonnegative independent component analysis,” IEEE Trans. Neural Networks, vol. 14, no. 2, pp. 534–543, May 2003. [55] M. Nikpour, J. H. Manton, and G. Hori, “Algorithms on Stiefel manifold for joint diagonalization,” in Proc. ICASSP, 2002, pp. 1481–1484. [56] A. Bunse-Gerstner, R. Byers, and V. Mehrmann, “Numerical methods for simultaneous diagonalization,” SIAM J. Mater. Anal. Appl., vol. 14, pp. 927–949, 1993. [57] M. Klajman and J. A. Chambers, “Approximate joint diagonalization based on the Cayley transform,” in Mathematics in Signal Separation V. Oxford, U.K.: Oxford Univ. Press, 2002. [58] S. Ding, T. Hikichi, T. Niitsuma, M. Hamatsu, and K. Sugai, “Recursive method for blind source separation and its applications to real-time separation of acoustic signals,” in Proc. ICA, Nara, Japan, Apr. 2000. [59] M. Joho, R. H. Lambert, and H. Mathis, “Elementary cost functions for blind separation of nonstationary source signals,” in Proc. ICASSP, May 2001. [60] A. E. Smith and D. W. Coit, “Penalty functions,” in Handbook of Evolutionary Computation, T. Baeck, D. Fogel, and Z. Michalewicz, Eds: A Joint Publication of Oxford University Press and Institute of Physics, 1995. [61] P. Smaragdis.. [Online]. Available: http://sound.media.mit.edu/icabench/ [62] J. Anemüller.. [Online]. Available: http://medi.uni-oldenburg.de/members/ane [63] A. Westner.. [Online]. Available: http://www.media.mit.edu/~westner

Wenwu Wang (M’03) was born in Anhui, China, in 1974. He received the B.Sc. degree in automatic control in 1997, the M.E. degree in control science and control engineering in 2000, and the Ph.D. degree in navigation guidance and control in 2002, all from Harbin Engineering University, Harbin, China. He then joined the Department of Electronic Engineering, King’s College, London, U.K., as a postdoctoral research associate and transferred to the Cardiff School of Engineering, Cardiff University, Cardiff, U.K., in January 2004. His current research interests are in the areas of blind signal processing, machine learning, and perception. Dr. Wang is a member of the IEE and of the IEEE Signal Processing and Circuits and Systems Societies.

1669

Saeid Sanei (M’97) received the B.Sc. degree in electronic engineering from Isfahan University of Technology, Tehran, Iran, in 1985 and the M.Sc. degree in satellite communication engineering from the University of Surrey, Surrey, U.K., in 1987. He received the Ph.D. degree in biomedical signal and image processing from Imperial College of Science, Technology, and Medicine, London, U.K., in 1991. He has since held academic positions in Iran, Singapore, and the United Kingdom. He is a Senior Lecturer with the Cardiff School of Engineering, Cardiff University, Cardiff, U.K. Dr. Sanei is a member of the IEEE Signal Processing, Biomedical Engineering, and Communications Societies.

Jonathon A. Chambers (SM’98) was born in Peterborough, U.K., in 1960. He received the B.Sc. (Hons) degree from the Polytechnic of Central London, London, U.K., in 1985, together with the Robert Mitchell Medal as the top graduate of the Polytechnic, and the Ph.D. degree in 1990 after study at Peterhouse, Cambridge University, and Imperial College London. He served in the Royal Navy as an Artificer Apprentice in Action, Data, and Control between 1979 and 1982. He has since held academic and industrial positions at Bath University, Bath, U.K.; Imperial College London; King’s College London; and Schlumberger Cambridge Research, Cambridge, U.K. In January 2004, he became a Cardiff Professorial Research Fellow within the Cardiff School of Engineering, Cardiff, U.K. He leads a team of researchers involved in the analysis, design, and evaluation of novel algorithms for digital signal processing with application in acoustics, biomedicine, and wireless communications. His research contributions have been in adaptive and blind signal processing. He has authored/co-authored almost 200 conference and journal publications and supervised 20 Ph.D. graduates. Dr. Chambers has served as an Associate Editor for the IEEE TRANSACTIONS ON SIGNAL PROCESSING and Circuits and Systems. He is also a past chairman of the IEE Professional Group E5, Signal Processing. He is currently serving as an Associate Editor for the IEEE SIGNAL PROCESSING LETTERS and the IEEE TRANSACTIONS ON SIGNAL PROCESSING and as a member of the IEEE Technical Committee on Signal Processing and Methods.

TLFeBOOK

1670

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

An Empirical Bayes Estimator for In-Scale Adaptive Filtering Paul J. Gendron, Member, IEEE

Abstract—A scale-adaptive filtering scheme is developed for underspread channels based on a model of the linear time-varying channel operator as a process in scale. Recursions serve the purpose of adding detail to the filter estimate until a suitable measure of fidelity and complexity is met. Resolution of the channel impulse response associated with its coherence time is naturally modeled over the observation time via a Gaussian mixture assignment on wavelet coefficients. Maximum likelihood, approximate maximum a posteriori (MAP) and posterior mean estimators, as well as associated variances, are derived. Doppler spread estimation associated with the coherence time of the filter is synonymous with model order selection and a MAP estimate is presented and compared with Laplace’s approximation and the popular AIC. The algorithm is implemented with conjugate-gradient iterations at each scale, and as the coherence time is recursively decreased, the lower scale estimate serves as a starting point for successive reduced-coherence time estimates. The algorithm is applied to a set of simulated sparse multipath Doppler spread channels, demonstrating the superior MSE performance of the posterior mean filter estimator and the superiority of the MAP Doppler spread stopping rule. Index Terms—Adaptive filters, recursive estimation, timevarying channels, time-varying filters, wavelet transforms.

I. INTRODUCTION DAPTIVE filtering schemes and system identification methods for time-varying operators have broad application in the field of signal processing [1]–[3]. From echo cancellation to active and passive sonar, as well as equalization of communication signals, adaptive filtering methods are fundamental modeling regimes for extracting information from signals. The methods employed are often driven by constraints on memory, computational resources and tolerable processing delays. Multiresolution models of time-varying operators have been promoted with the prospect that the sparsity or “economy” of representation attendant in them can be exploited [4]. This hope is not misplaced since many operators that are not shift invariant are often succinctly described in wavelet bases. Beylkin has shown that broad classes of operators are nearly diagonalized by wavelet decompositions [5]. Tewfik has demonstrated that wavelet bases approximate the Karhunen–Loeve transformation (KLT) associated with large classes of covariance matrices arising from nonstationary processes, and these can reduce

A

Manuscript received July 8, 2003; revised April 8, 2004. This work was supported by the Office of Naval Research. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Carlos H. Muravchik. The author is with the Acoustics Division, Naval Research Laboratory, Washington DC, 20375 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845442

the computational demand of solving such linear systems to , implying that solutions to nonstationary problems can be well approximated with computational resources on par with that of Toeplitz systems [6]. Advantages for estimation are concomitant, relying on the fundamental notion that sparsity of representation results in improved estimation [7], [8]. Sparsity of representation has been exploited in time-adaptive wavelet packet model selection for seismic and acoustic estimation [9], [10] and in wavelet discrimination of active sonar returns from compound [11] and moving objects [12]. Furthermore, the conceptually simple framework, like that of Fourier, has intuitive appeal [13], [14]. Regarding adaptive filtering, multiresolution models naturally allow estimation of the response at a given time instant based on data both before and after. In addition, the model circumvents the need for the locally stationary assumption of time-recursive algorithms. The result is a greater amount of information being brought to bear on the estimation problem at each time instant. Unlike in-time estimation algorithms that are, by their very nature, causal or near causal, dependencies across time in the forward-looking direction are not accounted for with the classic in-time adaptive filtering algorithms [2]. In-time estimation is a powerful paradigm because of its computational efficiencies and memory requirements [1], and it continues to find broad appeal and enjoys superior performance in many applications. Nevertheless, in areas where fading and multipath delay wander do not conform to the wide sense stationary (WSS) assumption, the multiresolution model is warranted. There are of course other practical reasons to consider multiresolution decompositions as signal processing and communication schemes are often wed to strategies based on finite duration signaling. For instance, in mobile radio and underwater acoustic communications, data is formatted in finite duration packets. Similarly, for underwater target localization by active sonar as well as radar applications, source signals consist of time localized “pings.” This paper is concerned with the estimation of time-varying channel operators in the multiresolution framework. Doroslovacki and Fan [4] gave a general framework for formulating multiresolution filter structures and presented a fast least mean square-like estimator in this framework under an assumed maximal scale of representation. When the maximal scale of representation or the most economical wavelet bases cannot be assumed, they must be jointly estimated. To address this assumption, the contribution of this paper is to exploit sparsity of representation under a priori uncertainty regarding the Doppler spread and degree of sparsity of the operator. This entails three important and necessary estimators. The first is the estimation of

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

GENDRON: EMPIRICAL BAYES ESTIMATOR FOR IN-SCALE ADAPTIVE FILTERING

wavelet coefficients of the channel response based on the flexible and adaptable Gaussian mixture model. The second is the estimation of the maximal scale of the multiresolution space in which the operator resides. The third is the estimation of nuisance parameters associated with the mixture model. It is shown that the estimation of the maximal scale synonymous with the Doppler spread is akin to model order selection, and estimators for this parameter are provided. These channel estimators are recursive in scale, each relying on the preceeding lower scale estimate as a starting point for the next higher resolution estimate. In this framework, the channel is gradually built up in scale, rather than time, and with each addition of detail, a greater Doppler spread (decreased coherence time) is hypothesized and tested against the data in a model selection framework. Thus, multiple resolutions of the operator are provided in sequence with finer scale details descending from the lower scale estimates, starting with the linear time invariant (LTI). The algorithm as presented is limited to filter operators that are underspread (i.e., , where is the delay spread of the filter, and is the maximal Doppler bandwidth associated with any of the scatterers; this requirement excludes the underdetermined parameter estimation case). This paper is organized as follows. Section II provides preliminary notation to address the time-varying channel as well as an introduction to the empirical Bayes approach. Section III reviews the multiresolution decomposition of a time-varying operator. In Section IV, the in-scale maximum likelihood estimator (MLE) is presented based on the conjugate gradient algorithm for in-scale recursions. Section IV goes on to introduce a suitable sparsity prior for the time-varying channel, and this leads to the posterior mean and variance, along with estimators of the nuisance parameters associated with the sparsity model. Finally, Section IV presents Doppler spread estimates and stopping rules based on the MAP criteria and on Laplace’s approximation as well as the simple and effective Akaike information criteria (AIC). Finally, in Section V, the usefulness of these algorithms is demonstrated on a diverse set of simulated channels with various types and degrees of Doppler spread. Improved performance of the empirical Bayes posterior mean estimator over the MLE is demonstrated. Section VI presents final conclusions and direction for future work. II. PRELIMINARIES The following definitions are provided for ease of reference and will be used throughout. Without loss of generality, all mamatrix trices and vectors are assumed real. Denote the by and the vector by . Let denote a denote a column vector of 0’s, and column vector of 1’s, denote the identity matrix of size by .

1671

stacking of ’s component columns into a supercolumn such that . with inverse denoted by is the 3) Kronecker product matrix of weighted B blocks, having an th block . entry 4) The elementwise product . 5) The vector square root . 6) The matrix operator , associated with the matrix array with row , has rows, the th . of which is Appendix A lists a few useful properties and identities that follow directly from these definitions. B. Empirical Bayes Procedures The empirical Bayes (EB) approach is useful primarily for its computational efficiency [15], [16]. Consider computation of , var and , where represents received data, a parameter set of interest, and a set of nuisance paramare often eters. Expectations over the joint density not easily computed. EB methods leverage a crude assumption to approximate expectations via itand are synonyeration mous with approximations of the posterior marginal density via . The method is versatile with other suitable estimators in place of , where their computation is faster or warranted by knowledge of the distribution. The penalty of this approximation is its underestimation var . of variance since var EB approaches therefore provide a lower bound var var that is useful when is relatively small. For adaptive filtering problems where the computational demands of Bayesian analysis are presently out of reach, the approach has merit. III. MULTIRESOLUTION MODEL OF TIME-VARYING FILTER spaces: one over Define two multiresolution and time and the other over delay. Let represent these, respectively [13]; thus, , where is the scaling function associated and highpass are with this MR space, and the lowpass a quadrature mirror filter pair. The scale index increases with increasing bandwidth, in agreement with the convention and on the wavelets that . imFollowing the filter model of [4] plies that

A. Definitions , define the diagonal matrix with diagonal 1) For vectors by diag . For square matrices, is the elements vector of diagonal elements of the matrix by diag . acts on a matrix 2) Stack operator , yielding , which is a

(1) has approximate Doppler spread and represents the time-varying filter perfectly on the domain . The model (1) implicitly specifies

TLFeBOOK

1672

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

the maximal Doppler spread and is therefore denoted with a subscript . is modFrequency selectivity in the band eled by the basis functions at delay . In this way, the frequency selectivity of the shifts channel due to diverse scatterer locations and delay spreads is modeled as a superpostion of wavelet bases. For moving scatterers, Doppler spread is induced and is synonymous with an effective channel modulation over time, corresponding to an imparted bandwidth to the impulse response process over time at any given delay. Modulations at time with durations are modeled with the basis . These modulations correspond to each functions scatterers motion such that scatterers corresponding to greater accelerations will yield a channel operator with projections onto bases at fine scales (i.e., large ). be a suitably samLet . Representing pled in time and delay version of of this two-dimensional array as a single stacked column time-invariant filters via definition (2), each operating to prosamples of the output, express the wavelet coeffiduce (nonzero up to scale ) by cients of (2) is the by matrix decimation operator The matrix space of [13]. Its adjoint associated with the is the associated interpolation operator. The columns of and are, respectively, the expansion coefficients of the and in the bases of the associated wavelets scaling functions and , respectively. The maand have maximal trices log and log , respecscales of is the Kronecker product of the two tively. The operator wavelet transforms . The wavelet coefficients are computable via the fast wavelet transform [13]. The operator is synthesized from with the fast inverse wavelet transform to scales ( interpolation/scaling operators added via zero ) appending (3) is synthesized as constant at time scales less implying that . Clearly, then, must satisfy log since than the channel is LTI at the sampling rate. The left-hand side of (3) follows from Property 4 of Appendix A, where and are from via the right-hand side of (3), unitary. Compute and denote the scaling function expansion of the channel by

with peaky features in frequency, Fourier bases or autoregressive models will be more suitable [18]. Fourier bases modeling Doppler spread will be optimal as well for the WSS channel as . Fourier bases will fail, however, to capture discontinuities or time-localized phenomena. In addition, where finite duration segments must be modeled, boundaries pose severe penalties to Fourier methods. Wavelets, on the other hand, parsimoniously capture these features. At lower frequencies associated with scatterers with smaller Doppler spread, wavelets will offer near Fourier-like performance for the WSS condition. Nevertheless, wavelets provide a means to model abrupt changes in channel conditions, localized phenomena in time and delay, as well as minimizing boundry effects associated with finite duration signaling. Wavelet models, of course, are not universally appropriate for delay localized operators. For instance, in the case of channels consisting of arrivals that have no dispersion in delay, the standard Euclidean bases (in delay) are optimal (i.e., ). In this case, the path delays and amplitudes (associated with the time changing geometry of the environment) are modeled. IV. IN-SCALE MODEL This section introduces an in-scale likelihood model from which a recursive in-scale MLE follows. A sparsity prior is then assumed, and from this in-scale posterior expectation, variance and Doppler spread estimates are derived. Empirical Bayes methods are employed throughout. A. Likelihood Model of (1) with Hz Doppler The channel response spread is observed via the source and received signal through the linear model (5) where is a white Gaussian noise process of known power . Let . Assuming is time-invariant over durations and and are suitably sampled at the Nyquist rate , that let be the received signal dimension and the delay spread dimension. Because we have the discrete-time model associated with (5) using the time-varying operator of Section II, definition 6) is (6) represents the by block convolution where in (3). Synonymously using (3), operator associated with express (6) as a set of stacked wavelet coefficients

(4)

(7)

Finally, define the time-varying convolution matrix operator asor, synonysociated with the time-varying array structure as . mously, its stacked representation A qualifying word is in order. For channels with sparse arrivals that are dispersed and nonstationary, the wavelet model offers a represention that is parsimonious. However, for filters

where is interchangeable with as in (3), and for notational simplicity, the conditioning on families and are assumed for being the Doppler time and delay, respectively. With is the block convolution operdimension, ator associated with the sampled source signal . The blocks are proportional to the coherence time of over of

TLFeBOOK

GENDRON: EMPIRICAL BAYES ESTIMATOR FOR IN-SCALE ADAPTIVE FILTERING

which the channel is LTI. operates on a long list of stacked -length LTI channel vectors to yield a channel output. ; then, has the form To illustrate, consider

1673

and

To illustrate, consider again the simple case and takes the form It follows that ..

.

.

.. . (11)

.. .

The L’s (Y’s) are lower (upper) triangular and are each of size by . The X’s are full and of size by . Together, each block represents a time-invariant filter operseconds derived from the source ator over a duration of maps the by signal of this block. The operator block convolution operator of source samples to the by block convolution operator of modulated blocks associated with coefficients in a scaling expansion of . Some useful properties are exposed by considering the case of large. Define the limiting matrix (8) which operates on the stacked coefficients of a scaling function ) to produce the exexpansion of the channel (i.e., pected value of the output of the time-varying filter according to (7). The following scaling equation for such block convolution . For a proof, operators is informative use (8), property 2), and . Further, observe that , the scaling equation for the operator takes the for form (9) and while it requires no multiplies, it exhibits abrupt discontinuities between blocks. In practice, (8) is not computable, and the upper bound on is determined by the bandwidth of the log such that modeling of the channel as anysource thing but LTI over durations smaller than the sampling period is infeasible and unnecessary. For channels with reasonable reguis sufficient in practice. For instance, with ,a larity, -Hz Doppler spread channel is modeled as LTI over durations s. Returning to (7), the implied likelihood function on is log

(10) matrix is the source where the covariance matrix associated with the time-varying filter model.

..

.

is Toeplitz, is lower triangular Hankel with where , and . zeros on the main diagonal having rank is represented either by the vectors of length correand or by the sponding to the first columns of diagonals of , each being of length . Thus, is nonzero dinon-Toeplitz and banded Hermitian with Haar will differ from in that it will agonals. For be banded with width . The vector in (10) is a block overlapped cross-correlation of source and received data vectors associated with the windowing scaling func. The weakness of the in-scale likelihood approach tion is simply that the parameter set grows exponentially with scale ), which is a severe penalty for a linear model. (i.e., Nevertheless, for the very limited set of spectrally flat signals, the approach has some merit computationally. Signals for which is small for some scalar (e.g., Gaussian will be white signals, maximal length sequences), are attained with small as well, and close approximations to the fast block correlation . B. In-Scale Computation by Conjugate Gradient A direct computation of in (10) requires , which is a computationally impractical feat. A feasible apis known to a tolerable precision. proach assumes that Compute starting from via the conjugate gradient (CG) method [19]. Initialization requires only since for , the system is Toeplitz. The CG algorithm has found wide application in solving banded and sparse systems [19], [20] as well as in adaptive filtering [21], [22], [24]. Indeed, the CG algorithm has been shown to be synonymous with the multistage Wiener filter [23], [25], [26]. Table I lists the operations necessary along with the associated computational demands. Note that only a single matrix multiplication by the is required per iteration. A multiply sparse banded matrix stored diagonals is accomplished efficiently from the . In this framework, large matrix inversions are of length avioded by solving a lower dimension problem to a given precision and using this solution as a starting point for the next higher dimension. The resulting MLE when the CG algorithm and is computed is iterated to machine precision is termed via interpolation from as

The CG algorithm also allows for flexibility in the number of to machine precision may not iterations. Computation of be necessary, and a solution slightly more coherent (closer to ) may be tolerable. For less computation, iterate to within

TLFeBOOK

1674

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE I COMPUTATION OF ~ VIA CONJUGATE GRADIENT

TABLE II TEST CASES FOR TIME-VARYING CHANNEL PARAMETERS

w

. With cohas variance . , iterate while . A conservative approximation of corresponding to is made in the simulations of Section V. efficients, this implies In this scenario, with

C. Gausssian Mixture as a Sparsity Prior for the Doppler Spread Channel

a ball of radius proportional to the covariance of the MLE, and an approximate MLE. To determine call such a solution this iteration number, an approximate ad hoc approach is taken. Define as the ratio of minimum to maximum eigenvalues of ; then, at iteration , where , the CG solution is within a ball of radius , where [19]. The computation of is not trivial and must be approxiated for the class of source signals that are considered. Further, in the case of in-scale computation proposed here, the extreme eigenat each would have to be approximated; at values of present, this is impractical, and an approximation is used for . The implementation of the CG algorithm presented makes an additional ad hoc assumption to eliminate from the stopping criteria based on the fact that the CG algorithm gives fairly smooth exponential convergence due to dominant eigenvectors not dominating the convergence. For this reason, in pracbound does not tice, the tightness of the vary greatly from iteration to iteration. Express the radius as for some (since the innovation components are not constant orthogonal). Assuming the convergence is fairly well behaved, approximate . A common scenario is that of a white nonfading source signal with variance . The variance of each coefficient is simply

The previous maximum likelihood in-scale estimate of the channel assumes that all delay lags of the channel at each time contribute to the output a priori equally. Synonymously, there is, in the likelihood model, an implicit prior of large and equal variance for each and every time-delay component of the channel operator. This assumption is simplistic for sparse channels, and a modification to the likelihood approach is warranted. The prior chosen should give the model flexibility to choose components that are associated with frequency selectivity and Doppler spread (time modulation) that are significant and reject those that are not. This problem is synonymous with variable selection, and mixture prior assignments have been useful for such problems in both the Bayesian [17] and empirical Bayes methodologies [16], [27]. For this reason, the binary mixture normal model (12) is chosen as a prior. Specify the prior conditional density of as

(13) The mixture normal model is useful because it captures the fundamental attribute of sparsity. For each of the significant arrival paths associated with an operator, we have a normal model. For those delays for which no arrival energy is present, variance. The hyperparameters we model these with small necessary to specify the model are summarized in a single pa. With the prior and likerameter lihood specified, the resulting posterior distribution to be maximized jointly over and (synonymous with and ) and is hyperparameter

TLFeBOOK

(14)

GENDRON: EMPIRICAL BAYES ESTIMATOR FOR IN-SCALE ADAPTIVE FILTERING

1675

Fig. 1. Magnitude of the simulated test cases of time-varying channel operators. Increasing Doppler spread shown clockwise starting from case 1. LTI bottom left to a Doppler spread of 4 Hz for bottom right.

In the first term, the likelihood assumes independence of wavelet on the coefficients via the near-diagonalizing property of source covariance , and in the second term, the prior is predicated on this assumption. represent an arbitrary For ease of notation, let wavelet coefficient at time-delay location associated with according to the model (1) so that according to (2). In the spirit of empirical Bayes, consider the posterior distribution of wavelet coefficients given all other parameters. This conditional density is shown in Appendix B to be

attenuates The resulting operator smaller coefficients and leaves larger ones relatively unchanged. It is similar in form to a Wiener filter with the noticeable difference of a mixture of Wiener gains, each gain being proportional to the implicit empirical posterior probability of the associated model in the mixture. sampled at the rate The posterior mean of the channel follows directly from (16) and the linearity of the wavelet transform as

(17) (15) where the posterior mixture coefficient and the gains are defined in Appendix B. This posterior (15) has the same mixture normal form as the prior (12). There are, however, two important distinctions: First, the posterior mixture coare functions of each individual empirical coefefficients . Second, the means of the mixture densities are ficient not equal. For this reason, the posterior density is not symetric, is not synonyand therefore, its first moment mous with the argument maximizing (15). D. Estimating Since the posterior is a mixture, the mean is simple to assess as

The posterior mean is useful since for the ensemble of channels associated with the mixture prior on wavelet coefficients, no other estimator has smaller mean square error. It is, however, a biased estimator. The MAP estimate for the posterior is computationally intensive, requiring iterations over every wavelet coefficient, and for this reason, an approximate MAP is worth considering. if if

(18)

is a simple approximation and has been tested against the actual MAP, computed via Nelder–Mead iteration, and gives comparable performance in terms of MSE on the simulated channels of Section V-B. E. Variance of

(16)

The conditional variance is computable directly from the posterior density (15) and gives measure of the uncertainty associ-

TLFeBOOK

1676

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 2. Comparison of MLE (left) and PM (posterior mean, right) estimates for test case 2: a path amplitude induced Doppler spread channel at SNR = 0 dB.

ated with the channel after observation of the data. Appendix C presents the following:

are uncorrelated cov Diag . The posterior covariance of is then using (3) and property 2 of Appendix A cov

var

Diag Diag (19)

The above shows that the conditional variance for each coefficient is the sum of two components. The first is a weighted average of the variances associated with the two models of the mixture. The last component represents the uncertainty of the coefficient being associated definitively with either of the two . Demodels in the mixture and is maximum for as the vector of posterior variances (19) associated with fine and, then, under the assumption that the wavelet coefficients

(20)

Determine the variance of at any given , by comDiag using Appendix A.1 to express puting Diag . This implies defined over , is that the posterior variance field (21) and do not Since the operators obey a scaling recursion, they must be computed directly an im. For practicle feat for channels of high dimension

TLFeBOOK

GENDRON: EMPIRICAL BAYES ESTIMATOR FOR IN-SCALE ADAPTIVE FILTERING

1677

Fig. 3. Scale adaptation for test case 4: a jointly path delay and amplitude induced Doppler spread channel at relatively low SNR = 0 dB.

confidence interval this reason, a fast approximate is worth pursuing. Proceed as follows: Define for and and transform via (3). , with . These Here, of course, wavelet domain bounds produce an approximate EB high-probability region for the channel as

otherwise . With the total number of coeffiscale denoted , cients at the when an asymptotically unbiased estimate of for the model of (12) is then

min

log max

(22) F. Estimating Hyperparameters In keeping with the empirical Bayes methodology, estimate as follows: First, for , use Donoho’s implies that threshold argument [7], namely, that log . Let

For , use . Last, for the ’s, using the and noise processes are inassumptions that the channel , the median absolute deviation (MAD) dependent with estimator of zero-mean independent and identically distributed variates suggests mad , leading to the max mad . estimator G. Estimating Doppler Spread In this section, inference regarding Doppler spread or maxlog is shown to be synonymous with imum scale

TLFeBOOK

1678

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

0

Fig. 4. For test case 1 LTI channel, at 6 dB SNR (above) and 3 dB (below). Typical MSE performance (left) for approximate MLE (aMLE), MLE, approximate MAP, and posterior mean (PM). Doppler estimation performance as minimum of AIC, Laplace approximation (LAP), and logp(r B ) (MAP) cost (right).

0

model order selection. An EB-MAP estimate, as well as rules based on Laplace’s approximation and AIC, are provided. It is convenient in keeping with the in-scale paradigm to place a prior on synonymous with an uninformative prior on , which is the maximum scale parameter. With this assumption, the MAP estimate of the Doppler spread is determined by maximizing the ; the objective is then likelihood argmin

Integrate over

j

analytically since

and over the paramusing the asymptotic result of Schwarz [29] (i.e., ). Under the near-diagonalizing property of , approximate , and with the estimate and , the MAP criterion is of

eters of

log argmin (23)

log

The first term in the integrand is decomposed as

log

log (24)

leading to

It is somewhat striking and counterintuitive that this empirical Bayes MAP estimate of Doppler spread (24) does not require the computation of the MAP or posterior mean (PM), the residual errors being associated with that of the MLE. To consider rules based on the residual errors associated with the posterior mean or approximate MAP channel estimate, prolog (noting cede as follows: Let that this is proportional to the log posterior of , expand about ;

TLFeBOOK

GENDRON: EMPIRICAL BAYES ESTIMATOR FOR IN-SCALE ADAPTIVE FILTERING

1679

0

Fig. 5. For test case 3 at 6 dB SNR (above) and 3 dB (below). Typical MSE performance (left) for MLE, approximate MLE (aMLE), and Empirical-Bayes posterior mean (PM). Doppler estimation performance as minimum of AIC, Laplace approximation, and MAP cost (right).

. Approximate the posterior mean and covariance

, and substitute , yielding

The associated penalty cost in this case is simply one half the number of parameters of the model argmin

log (25) V. SIMULATION RESULTS

With [29] applied to argmin

and the asymptotic result of Scharwz , the resulting Laplace approximation rule is log

log log

These in-scale algorithms were tested on a diverse set of simulated multipath channels to determine their relative performance in jointly estimating the channel operator and its Doppler spread. A. Description of Test Channels

log

The considered channels are of the form The first term log is a measure of the quality of the posterior mean channel estimate to predict the data and is synonymous with the coding complexity of the data, given the estimate of the channel. From this perspecitive, it is proportional to the length of the best code of the residuals of the data predicted by the adaptive filter estimate and sourse signal. Similary, the sum of the second and third terms is proportional to the information necessary to specify this particular channel estimate from the prior assignment to the resolution associated with the posterior variance [28]. A popular and heuristically simple alternative is AIC (for a reasonable explanation of this ad hoc measure, see Kay [30]).

(26) having independent component paths. Each path has an associated arrival spreading function that is time invariant. Doppler spread is induced by both the path gain and the path delay processes . Arrival processes and amplitudes are modeled path delay times as correlated Gaussian variates with respective covariances for , and 0 otherfor , and wise, and 0 otherwise. A realization of such a channel is then generated

TLFeBOOK

1680

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

0

Fig. 6. For test case 4 at 6 dB SNR (above) and 3 dB (below). Typical MSE performance (left) for approximate MLE (aMLE), MLE, approximate MAP, and posterior mean (PM). Doppler estimation performance as minimum of AIC, Laplace approximation (LAP), and logp(r jB ) (MAP) cost (right).

0

parameters: the weight functions and cogiven the herence times and . An associated Doppler bandwidth for each of these path processes is termed and . For each test case, the Doppler spread is termed the maximum of these constituent Doppler bandwidths. Varying the correlation times associated with the processes ’s and ’s as well as the spreading shapes ’s allows for the simulation of processes that are diverse and realistic. Four channel operators were tested, the number of paths and the maximal Doppler spread for each of the cases is listed in Table II. Fig. 1 shows the magnitude response of the four test channels under consideration. The channels share a common feature of sparsity in that while each scatterer path exhibits Gaussian uncorrelated scattering, each delay lag does not. Thus, the channels are not wide sense stationary-uncorrelated scattering (WSS-US) [3]. Gaussian source and additive noise signals of the same spectra were used in the simulations. The source signal correlafor , and tion . The channels were simulated 0 otherwise. for durations of approximately seconds and a bandwidth kHz. B. Results and were used as approximaIn these simulations, tions to and in the CG algorithm for the computational savings associated with its tight banded structure. In practice, iterations to compute starting from the number of

to a chosen precision of is approximately 5. Further iterations are used to compute the MLE. The algorithm was implemented with Daubechies [31] orthogonal in delay and in time wavelets order for under the folded wavelet assumption [13]. The interpoof (3) in this simulation was implemented lation operator as a windowed sinc for the simplicity associated with its linear phase, and was set at 4 so that for a given hypothesized , the estimate was assumed LTI over durations of . The normalized mean square error for the channel estimates

is computed for each of the estimators: the approximate MLE (aMLE), the MLE , the posterior mean (PM), and the approximate MAP. Fig. 2 presents a comparison of the channel estimates for the MLE and PM estimates. Granular noise is present in the MLE and represents the cost incured by the use of this unbiased estimator. With the sparsity prior, the PM is able to provide a more regular estimate that exhibits better MSE performance. However, this performance comes at a cost of bias in the PM estimate. This bias is pronounced in the peaks of the channel response esloss timate. Indeed, it is easily verified by considering near to approxifunctions. Tests were conducted with loss and demonstrate that the MLE often outperforms mate the PM, even for these sparse test cases.

TLFeBOOK

GENDRON: EMPIRICAL BAYES ESTIMATOR FOR IN-SCALE ADAPTIVE FILTERING

Fig. 3 depicts the evolution in scale for the PM for test case 4. The addition of detail is progressing from the bottom to the top of the figure, demonstrating the basic nature of the algorithm. It is noted first in the lower left that the algorithm starts with an LTI estimate of the channel and continues to add detail. At the highest scale, five high-resolution artifacts appear in the estimate, suggesting to the naked eye that the algorithm is begining to over fit the data. To determine the ability of the in-scale approach to accurately determine Doppler spread, the algorithm is tested on an LTI operator: test case 1. One important attribute of the in-scale approach is that LTI filters can be recognized even at quite low SNR. Fig. 4 displays the ability of the in-scale algorithm to identify LTI channels with only the computation of two iterations in the scale domain. The first iteration is that of the LTI estimate, and the second is the scale 1 iteration from which the LTI is compared favorably and the in-scale iteration is ceased. This is in stark contrast to in-time recursions where computational resources are distributed over time with no respect to the actual coherence time of the channel process. Thus, for finite duration signaling, the in-scale approach has a distinct advantage. The Doppler complexity measure is a minimum for the th scale estimate for all three criteria (AIC, LAP, MAP) and does not fall below this at greater resolution estimates. Here, each of the Doppler costs are normalized by log log . It is further noted that for sparse channels in delay, the mixture normal model is an effective sparsing model, affording a 2-dB improvement in performance over the MLE estimate at the zeroth scale estimate. The PM estimate exhibits greater gains at higher scales, demonstrating the robustness of the posterior mean and MAP estimates against variance of the Doppler spread rule. Figs. 5 and 6 display typical performance at SNRs of 6 and 3 dB with test case 3 (4-Hz Doppler spread with path delay wander only) and test case 4 (4-Hz Doppler spread due to path delay and amplitude processes), respectively. First, consider the Doppler estimation and stopping rules. The algorithms compare similarly with each other for these two channels in the high SNR 6-dB scenario. The AIC, LAP, and MAP methods all perform similarly, reaching a minimum cost at 4 Hz. In the low SNR scedB in Figs. 5 and 6, and the nario depicted below, SNR AIC and LAP become biased toward more coherent estimates while the MAP estimate rule retains its minimum at 4 Hz. This performance is typical. However, Monte Carlo simulations reveal slightly greater variability of the MAP stopping rule over that of the AIC or LAP. Channel estimation error is depicted as well in Figs. 5 and 6 and demonstrates the expected performance gain of the posterior mean over other estimators relative to MSE. At the specified Doppler spread, the PM estimate is 2 dB superior at 6 dB and 4 dB superior at 3 dB.

1681

An MLE, approximate MLE, approximate MAP, and posterior mean were derived along with AIC, Laplace approximation, and MAP Doppler estimation procedures based on empirical Bayes assumptions. These algorithms were tested and compared with one another on a number of simulated time-varying channels. As expected, the posterior mean demonstrates improvement in performance over the MLE and approximate MAP for time-varying channels relative to mean square error. All of the estimators presented for in-scale processing rely on the conjugate gradient algorithm to compute approximate MLE estimates from the previous lower scale estimate. From the MLE and conditional variance along with a sparsity prior, the posterior mean is a simple adaptive thresholding of wavelet coefficients. In this framework, computation is distributed over scale in the estimator, and this contrasts with in-time Kalman methods that distribute computation in time for an assumed coherence time. Joint estimation of channel parameters and coherence time is fundamental to in-scale estimation as it provides a stopping rule at which additional computation is unlikely to provide an improved estimate. Last, covariance estimates of the time-varying model are provided, and approximate high probability regions are shown to be easily accessible. These present useful computationally efficient empirical Bayes lower bounds on certainty bands of the channel operator. The implications for multisensor and multichannel estimation are apparent. Computational resources can be automatically and seemlessly focused on system nodes that exhibit greater Doppler bandwidth. This implies two advantages: First, channels that are more coherent will not waste computational resources. Second, performance degradation is limited since overfitting of data associated with overly complex models is less likely. Important issues relating to computation and performance relative to in-time recursions must be addressed. Future work should focus on comparison of in-scale recursions with in-time recursions (e.g. Kalman, RLS) to give bounds on signal duration, channel sparsity, and coherence time measures where the in-scale regime is to be favored. The in-scale approach could be broadened to include other scale-stepping increments rather than the classic octave band partition presented here. Mixed scale-time methods can also be envisioned for block processing over time.

APPENDIX A. Various Matrix and Kronecker Properties ;

1) 2) 3)

; , where and

VI. CONCLUSIONS This paper demonstrates that a model of the time-varying filter as a process in scale yields estimators that provide joint estimation of channel response and coherence time. Coherence time is easily modeled in this framework as model order selection.

4) 5) Diag 6) Diag

;

Diag

, and , where Diag

and , where

; ; and

; 7) 8)

TLFeBOOK

Diag Diag

; , where

and

.

1682

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

The following result is useful in Section IV when computing the posterior variance of a time-varying operator: for if then

Diag

the posterior density, the second moment is easily evaluated as

(A.1)

Diag and of course, sult after simplification is

Proof:

is taken from (16). The re-

property Diag property (C.1)

Diag property

Since confirmed that for

property

and and

var

, it is easily

var

(C.2)

property while the maximum variance attainable is approximately var log , where

B. Posterior Distribution The prior and likelihood on wavelet coefficients are and , respectively. The marignal density of empirical coefficients is then

log

(C.3)

. Since ACKNOWLEDGMENT

where is akin to a Wiener gain, a direct application of Bayes theorem confirms the posterior density to be

(B.1) where

C. Variance For the variance, start with var . For simplicity, . From denote the modulus square of a coefficient by

The author is thankful to B. Nandram for many useful discussions, to R. Hietmeyer, and the three anonymous reviewers for conscientious and helpful comments. REFERENCES [1] S. Haykin, Adaptive Filter Theory, Second ed, ser. Prentice Hall Information and Systems Sciences. Englewood Cliffs, NJ: Prentice-Hall, 1991. [2] S. Haykin, A. H. Sayed, J. R. Zeidler, P. Yee, and P. C. Wei, “Adaptive tracking of linear time-variant systems by extended RLS algorithms,” IEEE Trans. Signal Process., vol. 45, no. 5, pp. 1118–1128, May 1997. [3] J. G. Proakis, Digital Communications, Third ed. Englewood Cliffs, NJ: Prentice-Hall, 1991. [4] M. I. Doroslovacki and H. Fan, “Wavelet based linear system modeling and adaptive filtering,” IEEE Trans. Signal Process., vol. 44, no. 5, pp. 1156–1167, May 1996. [5] G. Beylkin, R. Coifman, and V. Rokhlin, “Fast wavelet transforms and numerical algorithms I,” Commun. Pure Applied Math., vol. 44, no. 2, pp. 141–183, May 1991. [6] A. H. Tewfik and M. Kim, “Fast positive definite linear system solvers,” IEEE Trans. Signal Process., vol. 42, no. 3, pp. 572–584, Mar. 1994. [7] D. L. Donoho, “Unconditional bases are optimal bases for data compression and for statistical estimation,” App. Comput. Harmonic Anal., vol. 1, Dec. 1993. [8] D. Donoho, M. Vetterli, R. DeVore, and I. Daubechies, “Data compression and harmonic analysis,” IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2435–2476, Oct. 1998. [9] P. J. Gendron and B. Nandram, “An empirical Bayes estimator of seismic events using wavelet packet bases,” J. Agri., Biol., Environ. Statist., vol. 6, no. 3, pp. 379–406, Sep. 2001.

TLFeBOOK

GENDRON: EMPIRICAL BAYES ESTIMATOR FOR IN-SCALE ADAPTIVE FILTERING

[10] [11]

[12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24]

, “Modeling heavytailed and correlated noise with wavelet packet bases,” J. Statist. Planning Inference, vol. 112, no. 1–2, pp. 99–114, Mar. 2003. L. R. Dragonette, D. M. Drumheller, C. F. Gaumond, D. H. Hughes, B. T. O’Conner, N.-C. Yen, and T. J. Yoder, “The application of two-dimensional signal transformations to the analysis and synthesis of structural excitations observed in acoustical scattering,” Proc. IEEE, vol. 84, no. 9, pp. 1249–1263, Sep. 1996. L. G. Weiss, “Wavelets and wideband correlation processing,” IEEE Signal Process. Mag., vol. 11, no. 1, pp. 13–32, Jan. 1994. S. G. Mallat, “A theory for multiresolution signal decomposition: the wavelet representation,” IEEE Trans. Pattern Anal. Machine Intell., vol. 11, no. 7, pp. 674–693, Jul. 1989. G. Strang, “Wavelet transforms versus Fourier transforms,” Bull. Amer. Math. Soc., vol. 28, no. 2, pp. 288–304, 1993. G. Archer and D. Titterington, “On some Bayesian/regularization methods for image restoration,” IEEE Trans. Image Process., vol. 4, no. 7, pp. 989–995, Jul. 1995. E. I. George and D. P. Foster, “Calibration and empirical Bayes variable selection,” Biometrika, vol. 87, no. 4, pp. 731–747, 2000. E. George and R. McCulloch, “Variable selection via Gibbs sampling,” J. Amer. Statist. Assoc., vol. 88, pp. 881–889, 1993. S. M. Kay and S. B. Doyle, “Rapid estimation of the Range-Doppler scattering function,” IEEE Trans. Signal Process., vol. 51, no. 1, pp. 255–268, Jan. 2003. G. H. Golub and C. F. Van Loan, Matrix Computations, Third ed. Baltimore, MD: John Hopkins Univ. Press, 1996. J. Kamm and J. G. Nagy, “Optimal Kronecker product approximation of block Toeplitz matrices,” SIAM J. Matrix Anal. Appl., vol. 22, no. 1, pp. 155–172. G. K. Boray and M. D. Srinath, “Conjugate gradient techniques for adaptive filtering,” IEEE Trans. Circuits Syst. I: Funda. Theory Applicat., vol. 39, no. 1, pp. 1–10, Jan. 1992. P. S. Chang and A. N. Wilson, “Analysis of conjugate gradient algorithms for adaptive filtering,” IEEE Trans. Signal Process., vol. 48, no. 2, pp. 409–418, Feb. 2000. J. S. Goldstein, “A multistage representation of the Wiener filter based on orthogonal projections,” IEEE Trans. Inf. Theory, vol. 44, no. 7, pp. 2943–2959, Nov. 1998. Z. Fu and E. Dowling, “Conjugate gradient projection subspace tracking,” IEEE Trans. Signal Process., vol. 45, no. 6, pp. 1664–1668, Jun. 1997.

1683

[25] M. E. Weippert, J. D. Hiemstra, J. S. Goldstein, and M. D. Zoltowski, “Insights from the relationship between the multistage Wiener filter and the method of conjugate gradients,” in Proc. Sensor Array Multichannel Signal Process. Workshop, Aug. 4–6, 2002, pp. 388–392. [26] G. Dietl, “Conjugate Gradient Implementation of Multistage Nested Wiener Filter for Reduced Dimension Processing,” M.S. thesis, Munich Univ. Technol., Munich, Germany, May 2001. [27] H. Chipman, E. Kolaczyk, and R. McCulloch, “Adaptive Bayesian wavelet shrinkage,” J. Amer. Statist. Assoc., Theory Methods, vol. 92, no. 440, pp. 1413–1421, Dec. 1997. [28] A. Barron and T. Cover, “The minimum description length principle in coding and modeling,” IEEE Trans. Inf. Theory, vol. 44, no. 6, pp. 2743–2760, Oct. 1998. [29] G. Schwarz, “Estimating the dimension of a model,” Ann. Statist., vol. 6, no. 2, pp. 461–464, 1978. [30] S. M. Kay, Modern Spectral Estimation: Theory and Application. Englewood Cliffs, NJ: Prentice-Hall. [31] I. Daubechies, “Orthonormal bases of compactly supported wavelets,” Commun. Pure Appl. Math., vol. XLI, pp. 909–996, 1988.

Paul J. Gendron (M’99) received the B.S. degree from the University of Massachusetts, Amherst, in 1985, the M.S. degree from Virginia Tech, Blacksburg, in 1993, and the Ph.D. degree from Worcester Polytechnic Institute, Worcester, MA, in 1999, all in electrical engineering. Since 2002, he has worked for the Signal Processing Branch, Acoustics Division, Naval Research Laboratory, Washington, DC. From 1985 to 1991, he was an engineer at the Portsmouth Naval Shipyard, Portsmouth NH, working on submarine sonar and navigation systems. He was the recipient of an ONR special research award in ocean acoustics in 2000. His research interests are in underwater acoustic communications, adaptive filtering, Doppler/dilation process estimation, and the detection and estimation of transient events. Dr. Gendron is a member of Eta Kappa Nu.

TLFeBOOK

1684

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Robust Minimum Variance Beamforming Robert G. Lorenz, Member, IEEE, and Stephen P. Boyd, Fellow, IEEE

Abstract—This paper introduces an extension of minimum variance beamforming that explicitly takes into account variation or uncertainty in the array response. Sources of this uncertainty include imprecise knowledge of the angle of arrival and uncertainty in the array manifold. In our method, uncertainty in the array manifold is explicitly modeled via an ellipsoid that gives the possible values of the array for a particular look direction. We choose weights that minimize the total weighted power output of the array, subject to the constraint that the gain should exceed unity for all array responses in this ellipsoid. The robust weight selection process can be cast as a second-order cone program that can be solved efficiently using Lagrange multiplier techniques. If the ellipsoid reduces to a single point, the method coincides with Capon’s method. We describe in detail several methods that can be used to derive an appropriate uncertainty ellipsoid for the array response. We form separate uncertainty ellipsoids for each component in the signal path (e.g., antenna, electronics) and then determine an aggregate uncertainty ellipsoid from these. We give new results for modeling the element-wise products of ellipsoids. We demonstrate the robust beamforming and the ellipsoidal modeling methods with several numerical examples.

; the expected weighted array response in direction is effect of the noise and interferences at the combined output is , where , and denotes the exgiven by pected value. If we presume that and are known, we may choose as the optimal solution of minimize subject to

Minimum variance beamforming is a variation on (2) in with an estimate of the received signal which we replace covariance derived from recently received samples of the array output, e.g., (3) The minimum variance beamformer (MVB) is chosen as the optimal solution of

Index Terms—Ellipsoidal calculus, Hadamard product, robust beamforming, second-order cone programming.

I. INTRODUCTION denote ONSIDER an array of sensors. Let the response of the array to a plane wave of unit amplitude as the array manarriving from direction ; we will refer to ifold. We assume that a narrowband source is impinging on the array from angle and that the source is in the far field of is then the array. The vector array output

C

(1) includes effects such as coupling between elements where and subsequent amplification; is a vector of additive noises representing the effect of undesired signals, such as thermal noise or interference. We denote the sampled array output by . Similarly, the combined beamformer output is given by

where is a vector of weights, i.e., design variables, and denotes the conjugate transpose. The goal is to make and small, in which recovers , i.e., . The gain of the case, Manuscript received January 20, 2002; revised April 5, 2004. This work was supported by Thales Navigation. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Joseph Tabrikian. R. G. Lorenz is with Beceem Communications, Inc., Santa Clara, CA 95054 USA (e-mail: [email protected]). S. P. Boyd is with the Department of Electrical Engineering, Stanford University, Stanford, CA 94305 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845436

(2)

minimize subject to

(4)

This is commonly referred to as Capon’s method [1]. Equation (4) has an analytical solution given by (5) Equation (4) also differs from (2) in that the power expression we are minimizing includes the effect of the desired signal plus in (4) prevents the gain in the noise. The constraint direction of the signal from being reduced. A measure of the effectiveness of a beamformer is given by the signal-to-interference-plus-noise ratio (SINR), given by SINR

(6)

where is the power of the signal of interest. The assumed may differ from the actual value value of the array manifold for a host of reasons, including imprecise knowledge of the signal’s angle of arrival . Unfortunately, the SINR of Capon’s method can degrade catastrophically for modest differences between the assumed and actual values of the array manifold. We now review several techniques for minimizing the sensitivity of MVB to modeling errors in the array manifold. A. Previous Work One popular method to address uncertainty in the array response or angle of arrival is to impose a set of unity-gain constraints for a small spread of angles around the nominal look direction. These are known in the literature as point mainbeam

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

LORENZ AND BOYD: ROBUST MINIMUM VARIANCE BEAMFORMING

1685

constraints or neighboring location constraints [2]. The beamforming problem with point mainbeam constraints can be expressed as minimize subject to

penalizes large values of and has the The parameter general effect of detuning the beamformer response. The regularized least squares problem (10) has an analytical solution given by

(7)

where is an matrix of array responses in the convector specifying the destrained directions, and is an sired response in each constrained direction. To achieve wider responses, additional constraint points are added. We may similarly constrain the derivative of the weighted array output to be zero at the desired look angle. This constraint can be expressed in the same framework as (7); in this case, we let be the deriva. tive of the array manifold with respect to look angle and These are called derivative mainbeam constraints; this derivative may be approximated using regularization methods. Point and derivative mainbeam constraints may also be used in conjunction with one another. The minimizer of (7) has an analytical solution given by

(11) Gershman [4] and Johnson and Dudgeon [5] provide a survey of these methods; see also the references contained therein. Similar ideas have been used in adaptive algorithms; see [6]. Beamformers using eigenvalue thresholding methods to achieve robustness have also been used; see [7]. The beamformer is computed according to Capon’s method, using a covariance matrix that has been modified to ensure that no eigenvalue is less than a factor times the largest, where . Specifically, let denote the eigen, where is a diagonal value/eigenvector decomposition of matrix, the th entry (eigenvalue) of which is given by , i.e., ..

(8) Each constraint removes one of the remaining degrees of freedom available to reject undesired signals; this is particularly significant for an array with a small number of elements. We may overcome this limitation by using a using a low-rank approximation to the constraints [3]. The best rank approximation to , in a least squares sense, is given by , where is a diagonal matrix consisting of the largest singular is a matrix whose columns are the correvalues, matrix sponding left singular vectors of , and is a whose columns are the corresponding right singular vectors of . The reduced-rank constraint equations can be written as or equivalently (9) where denotes the Moore–Penrose pseudoinverse. Using (8), we compute the beamformer using the reduced-rank constraints as

This technique, which is used in source localization, is referred to as MVB with environmental perturbation constraints (MVEPC); see [2] and the references contained therein. Unfortunately, it is not clear how best to pick the additional constraints, or, in the case of the MV-EPC, the rank of the constraints. The effect of additional constraints on the design specifications appears to be difficult to predict. Regularization methods have also been used in beamforming. One technique, referred to in the literature as diagonal loading, chooses the beamformer to minimize the sum of the weighted array output power plus a penalty term, proportional to the square of the norm of the weight vector. The gain in the assumed angle of arrival (AOA) of the desired signal is constrained to be unity. The beamformer is chosen as the optimal solution of minimize subject to

(10)

.

Without loss of generality, assume . We form the diagonal matrix , the th entry of which is given by ; viz,

..

.

The modified covariance matrix is computed according to . The beamformer using eigenvalue thresholding is given by (12) The parameter corresponds to the reciprocal of the condition number of the covariance matrix. A variation on this approach is to use a fixed value for the minimum eigenvalue threshold. One interpretation of this approach is to incorporate a priori knowledge of the presence of additive white noise when the sample covariance is unable to observe said white noise floor due to short observation time [7]. The performance of this beamformer appears to be similar to that of the regularized beamformer using diagonal loading; both usually work well for an appropriate choice of the regularization parameter . We see two limitations with regularization techniques for beamformers. First, it is not clear how to efficiently pick . Second, this technique does not take into account any knowledge we may have about variation in the array manifold, e.g., that the variation may not be isotropic. In Section I-C, we describe a beamforming method that explicitly uses information about the variation in the array re, which we model explicitly as an uncertainty ellipsponse soid. Prior to this, we introduce some notation for describing ellipsoids.

TLFeBOOK

1686

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

B. Ellipsoid Descriptions An -dimensional ellipsoid can be defined as the image of a -dimensional Euclidean ball under an affine mapping from to , i.e., (13) where , and . The set describes an ellipsoid whose center is and whose principal semiaxes are the unit-norm left singular vectors of scaled by the corresponding singular values. We say that an ellipsoid is flat if this mapping is not injective, i.e., one-to-one. Flat ellipsoids can be de. In this scribed by (13) in the proper affine subspaces of and with . case, will be paramUnless otherwise specified, an ellipsoid in eterized in terms of its center and a symmetric non-negas ative definite configuration matrix (14) is any matrix square root satisfying where . When is full rank, the nondegenerate ellipsoid may also be expressed as (15) is deThe first representation (14) is more natural when generate or poorly conditioned. Using the second description (15), one may quickly determine whether a point is within the ellipsoid. As in (18), we will express the values of the array manifold as the direct sum of its real and imaginary components in ; i.e., (16) While it is possible to cover the field of values with a complex , doing so implies a symmetry between the real ellipsoid in and imaginary components, which generally results in a larger ellipsoid than if the direct sum of the real and imaginary com. ponents are covered in C. Robust Minimum Variance Beamforming A generalization of (4) that captures our desire to minimize the weighted power output of the array in the presence of unceris then tainties in minimize subject to

(17)

denotes the real part. Here, is an ellipsoid that where covers the possible range of values of due to imprecise , uncertainty in the angle knowledge of the array manifold of arrival , or other factors. We will refer to the optimal solution of (17) as the robust minimum variance beamformer (RMVB). for all in (17) We use the constraint for two reasons. First, while normally considered a semi-infinite constraint, we show in Section II that it can be expressed as a second-order cone constraint. As a result, the robust MVB problem (17) can be solved efficiently. Second, the real part of

the response is an efficient lower bound for the magnitude of the is unchanged if the weight response, as the objective . This is particuvector is multiplied by an arbitrary shift larly true when the uncertainty in the array response is relatively small. It is unnecessary to constrain the imaginary part of the response to be nominally zero. The same rotation that maximizes simultaneously minithe real part for a given level of mizes the imaginary component of the response. Our approach differs from the previously mentioned beamforming techniques in that the weight selection uses the a priori uncertainties in the array manifold in a precise way; the RMVB is guaranteed to satisfy the minimum gain constraint for all values in the uncertainty ellipsoid. Wu and Zhang [8] observe that the array manifold may be described as a polyhedron and that the robust beamforming problem can be cast as a quadratic program. While the polyhedron approach is less conservative, the size of the description and, hence, the complexity of solving the problem grows with the number of vertices. Vorobyov et al. [9], [10] have described the use of second-order cone programming for robust beamforming in the case where the uncertainty in the array response is isotropic. In this paper, we consider the case in which the uncertainty is anisotropic [11], [12]. We also show how this problem can be solved efficiently in practice. D. Outline of the Paper The rest of this paper is organized as follows. In Section II, we discuss the RMVB. A numerically efficient technique based on Lagrange multiplier methods is described; we will see that the RMVB can be computed with the same order of complexity as its nonrobust counterpart. A numerical example is given in Section III. In Section IV, we describe ellipsoidal modeling methods that make use of simulated or measured values of the array manifold. In Section V, we discuss more sophisticated techniques, based on ellipsoidal calculus, for propagating uncertainty ellipsoids. In particular, we describe a numerically efficient method for approximating the numerical range of the Hadamard (element-wise) product of two ellipsoids. This form of uncertainty arises when the array outputs are subject to multiplicative uncertainties. Our conclusions are given in Section VI. II. ROBUST WEIGHT SELECTION For purposes of computation, we will express the weight vector and the values of the array manifold as the direct sum of the corresponding real and imaginary components (18) can be written as The real component of the product the quadratic form may be expressed in terms of , where

We will assume

TLFeBOOK

is positive definite.

; as

LORENZ AND BOYD: ROBUST MINIMUM VARIANCE BEAMFORMING

1687

Let be an ellipsoid covering the possible values of , i.e., the real and imaginary components of . The ellipsoid is centered at ; the matrix determines its size and shape. The constraint for all in (17) can be expressed as (19)

. To calculate the stationary points, we where with respect to and ; setting these partial differentiate derivatives equal to zero, we have, respectively (26) and (27)

which is equivalent to for all

s.t.

(20)

if and only if it holds for the Now, (20) holds for all , namely, . value of that maximizes By the Cauchy-Schwartz inequality, we see that (19) is equivalent to the constraint

which are known as the Lagrange equations. To solve for the Lagrange multiplier , we note that (26) has an analytical solution given by

Applying this to (27) yields

(21) which is called a second-order cone constraint [13]. We can then express the robust minimum variance beamforming problem (17) as minimize subject to

(22)

which is a second-order cone program. See [13]–[16]. The subject of robust convex optimization is covered in [17]–[21]. is positive definite, and the constraint By assumption, in (22) precludes the trivial minimizer of . Hence, this constraint will be tight for any optimal solution, and we may express (22) in terms of real-valued quantities as minimize subject to

(23)

In the case of no uncertainty where is a singleton whose center is , (23) reduces to Capon’s method and admits an analytical solution given by the MVB (5). Compared to the MVB, the RMVB adds a margin that scales with the size of the uncertainty. In the case of an isotropic array uncertainty, the optimal solution of (17) yields the same weight vector (to a scale factor) as the regularized beamformer for the proper the proper choice of . A. Lagrange Multiplier Methods It is natural to suspect that we may compute the RMVB efficiently using Lagrange multiplier methods. See, for example, [14] and [22]–[26]. Indeed, this is the case. The RMVB is the optimal solution of minimize subject to

(28) The optimal value of the Lagrange multiplier is then a zero of (28). We proceed by computing the eigenvalue/eigenvector decomto diagonalize (28), i.e., position (29) . Equation (29) reduces to the following where scalar secular equation: (30) where are the diagonal elements of . The values of are known as the generalized eigenvalues of and and are . Having computed the roots of the equation satisfying , the RMVB is computed the value of according to (31) Similar techniques have been used in the design of filters for radar applications; see Stutt and Spafford [27] and Abramovich and Sverdlik [28]. In principle, we could solve for all the roots of (30) and choose the one that results in the smallest objective value and satisfies the constraint , which is assumed in (24). In the next section, however, we show that this constraint is met for all values of the Lagrange multiplier greater than a minimum value . We will see that there is a single that satisfies the Lagrange equations. value of B. Lower Bound on the Lagrange Multiplier

. We define if we impose the additional constraint that associated with (24) as the Lagrangian

We begin by establishing the conditions under which (9) has a solution. Assume , i.e., is symmetric and positive definite. full rank, there exists an Lemma 1: For for which if and only if . Proof: To prove the if direction, define

(25)

(32)

(24)

TLFeBOOK

1688

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

By the matrix inversion lemma, we have

Since both block diagonal elements of

(33) , For creasing function of ; therefore, for exists a for which

is a monotonically in, there

are invertible

(36) , which is the Schur where complement of the (1,1) block in , and , which is the Schur complement of the (2,2) block in . We conclude if and only if the matrix has a negative eigenvalue. By the matrix inversion lemma

(34) This implies that the matrix Since , As in (28) and (30), let amining (28), we see

is singular. , for all

. . Ex-

(37) Inverting a scalar preserves its sign; therefore (38) if and only if has a negative eigenvalue. Remark: Applying Sylvester’s law of inertia to (28) and (30), we see that (39)

Evaluating (28) or (30), we see . For all , , and is continuous. Hence, assumes the value of 0, establishing the existence of a for which . To show the only if direction, assume that satisfies . This condition is equivalent to (35) For (35) to hold, the origin cannot be contained in ellipsoid , which implies . Remark: The constraints and in (24), taken together, are equivalent to the constraint in (23). For , full rank, and , (23) has a unique minimizer . For , is full rank, and the Lagrange equation (26)

where is the single negative generalized eigenvalue. Using , this fact and (30), we can readily verify as stated in Lemma 1. Two immediate consequences follow from Lemma 2. First, we may exclude from consideration any value of less than . Second, for all , the matrix has a single negative eigenvalue. We now use these facts to obtain a tighter lower bound on the value of the optimal Lagrange multiplier. We begin by rewriting (30) as (40) Recall that exactly one of the generalized eigenvalues secular equation (40) is negative. We rewrite (40) as

in the

(41) holds for only a single value of . This implies that there is a for which the secular equation (30) unique value of equals zero. with Lemma 2: For full rank, , and , if and only if has a negative eigenvalue. the matrix Proof: Consider the matrix

where denotes the index associated with this negative eigenvalue. A lower bound on can be found by ignoring the terms involving the non-negative eigenvalues in (41) and solving

This yields a quadratic equation in (42) We define the inertia of as the triple , where is the number of positive eigenvalues, is the number of negative eigenvalues, and is the number of zero eigenvalues of . See Kailath et al. [29, pp. 729–730].

the roots of which are given by

TLFeBOOK

LORENZ AND BOYD: ROBUST MINIMUM VARIANCE BEAMFORMING

By Lemma 2, the constraint has a negative eigenvalue since

Hence, We conclude that

1689

implies that

, where is the single negative eigenvalue. , where (43)

For any feasible beamforming problem, i.e., if has a negative eigenvalue, the parenthetical quantity in (43) is always non-negative. To see this, we note that , where is the eigenvector associated with the negative eigenvalue . Hence, can be expressed as the optimal solution of minimize subject to and responding objective value. Since

(44) , which is the cor-

(45) we conclude

.

C. Solution of the Secular Equation The secular equation (30) can be efficiently solved using Newton’s method. The derivative of this secular equation with respect to is given by (46) As the secular equation (30) is not necessarily a monotonically increasing function of , it is useful to examine the sign of the derivative at each iteration. The Newton-Raphson method enjoys quadratic convergence if started sufficiently close to the root . Se Dahlquist and Björck [30, §6] for details.

We summarize the algorithm below. In parentheses are approximate costs of each of the numbered steps; the actual costs will depend on the implementation and problem size [31]. As in [25], we will consider a flop to be any single floating-point operation.

and

.

The computational complexity of these steps is discussed as follows. is expensive; fortu1) Forming the matrix product nately, it is also often avoidable. If the parameters of the uncertainty ellipsoid are stored, the shape parameter may . In the event that an aggregate ellipsoid be stored as is computed using the methods of Section IV, the quantity is produced. In either case, only the subtraction of the quantity need be performed, requiring flops. 2) Computing the Cholesky factor in step 2 requires flops. The resulting matrix is triangular; hence, computing flops. Forming the matrix in its inverse requires step 2c) requires flops. 3) Computing the eigenvalue/eigenvector decomposition is the most expensive part of the algorithm. In practice, it flops. takes approximately 5) The solution of the secular equation requires minimal effort. The solution of the secular equation converges quadratically. In practice, the starting point is close to ; hence, the secular equation generally converges in seven to ten iterations, independent of problem size. 6) Accounting for the symmetry in and , computing requires flops. flops. In comparison, the regularized beamformer requires Hence, the RMVB requires approximately 12 times the computational cost of the regularized beamformer. Note that this factor is independent of problem size.

III. NUMERICAL EXAMPLE

D. Summary and Computational Complexity of the RMVB Computation

RMVB Computation Given , strictly feasible . 1) Calculate 2) Change coordinates.

a) Compute Cholesky factorization . . b) Compute c) . 3) Eigenvalue/eigenvector computation. a) Compute . 4) Change coordinates. a) . 5) Secular equation solution. a) Compute initial feasible point b) Find for which . 6) Compute

Consider a ten-element uniform linear array, centered at the origin, in which the spacing between the elements is half of a wavelength. Assume that the response of each element is isotropic and has unit norm. If the coupling between elements is given by is ignored, the response of the array

where , and is the angle of arrival. The responses of closely spaced antenna elements often differ substantially from this model.

TLFeBOOK

1690

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 1.

Beamformer block diagram.

In this example, three signals impinge upon the array: a and two uncorrelated interfering signals desired signal and . The signal-to-noise ratio (SNR) of the desired signal at each element is 20 dB. The angles of arrival of the and are 30 and 75 ; the SNRs of interfering signals these interfering signals are 40 dB and 20 dB, respectively. We model the received signals as (47) where

denotes the array response of the desired signal, and denote the array responses for the interfering signals, denotes the complex amplitude of the and denote the interfering desired signal, is a complex vector of additive white noises. signals, and , where is an Let the noise covariance identity matrix, and is the number of antennas, viz, 10. Similarly, define the powers of the desired signal and interfering sig, , and nals to be , where

Fig. 2. Response of the MVB (Capon’s method, dashed trace), the regularized beamformer employing diagonal loading (dotted trace), and the RMVB (solid trace) as a function of angle of arrival  . Note that the RMVB preserves greater-than-unity gain for all angles of arrival in the design specification of  2 [40; 50].

equally spaced samples of the array response at angles between 40 and 50 according to and (49) where for

If we assume the signals , , and are all uncorrelated, the estimated covariance, which uses the actual array response, is given by

(48) In practice, the covariance of the received signals plus interference is often neither known nor stationary and, hence, must be estimated from recently received signals. As a result, the performance of beamformers is often degraded by errors in the covariance due to either small sample size or movement in the signal sources. We will compare the performance of the robust beamformer with beamformers using two regularization techniques: diagonal loading and eigenvalue thresholding (see Fig. 1). In this is 45 . example, we assume a priori that the nominal AOA , The actual array response is contained in an ellipsoid whose center and configuration matrix are computed from

(50)

and

Here, , and . In Fig. 2, we see the reception pattern of the array employing the MVB, the regularized beamformer (10), and the RMVB, all computed using the nominal AOA and the corresponding covariance matrix . The regularization term used in the regularized beamformer was chosen to be one one hundredth of the largest eigenvalue of the received covariance matrix. By design, both the MVB and the regularized beamformer have unity gain at the nominal AOA. The response of the regularized beamformer is seen to be a detuned version of the MVB. The RMVB maintains greater-than-unity gain for all AOAs covered by the uncertainty . ellipsoid In Fig. 3, we see the effect of changes in the regularization parameter on the worst-case SINRs for the regularized beamformers using diagonal loading and eigenvalue thresholding and the effect of scaling the uncertainty ellipsoid on the RMVB. Using the definition of SINR (6), we define the worst-case SINR

TLFeBOOK

LORENZ AND BOYD: ROBUST MINIMUM VARIANCE BEAMFORMING

1691

Fig. 3. Worst-case performance of the regularized beamformers based on diagonal loading (dotted) and eigenvalue thresholding (dashed) as a function of the regularization parameter . The effect of scaling of the uncertainty ellipsoid used in the design of the RMVB (solid) is seen; for  = 1, the uncertainty used in designing the robust beamformer equals the actual uncertainty in the array manifold.

Fig. 4. Ambiguity function for the RMVB beamformer using an uncertainty ellipsoid computed from a beamwidth of 10 (solid), 2 (dashed), and the Capon beamformer (dotted). The true powers of the signal of interest and interfering signals are denoted with circles. In this example, the additive noise power at each element has unit variance; hence, the ambiguity function corresponds to SNR.

as the minimum objective value of the following optimization problem:

For comparison, the worst-case SINR of the MVB with (three) unity mainbeam constraints at 40 , 45 , and 50 is 1.85 dB. The MV-EPC beamformer was computed using the same 64 samples of the array manifold as the computation of the uncertainty ellipsoid (49); the design value for the response in each of these directions was unity. The worst-case SINRs of the rank-1 through rank-4 MV-EPC beamformers were found to be 28.96, 3.92, 1.89, and 1.56 dB, respectively. The worst-case response for the rank-5 and rank-6 MV-EPC beamformers is zero, i.e., it can fail completely. If the signals and noises are all uncorrelated, the sample covariance, as computed in (3), equals its expected value, and the uncertainty ellipsoid contains the actual array response, the RMVB is guaranteed to have greater than unity magnitude response for all values of the array manifold in the uncertainty ellipsoid . In this case, an upper bound on the power of the deis simply the weighted power out of the array, sired signal namely

minimize subject to where the expected covariance of the interfering signals and noises is given by

The weight vector and covariance matrix of the noise and used in its computation reflect the chosen interfering signals value of the array manifold. For diagonal loading, the parameter is the scale factor multiplying the identity matrix added to the covariance matrix, divided by the largest eigenvalue of the covariance matrix . , the performance of the regFor small values of , i.e., ularized beamformer approaches that of Capon’s method; the worst-case SINR for Capon’s method is 29.11 dB. As , . The beamformer based on eigenvalue thresholding performs similarly to the beamformer based on diagonal loading. In this case, is defined to be the ratio of the threshold to the largest eigenvalue of ; as such, the response of this beamformer is . only computed for For the robust beamformer, we use to define the ratio of the size of the ellipsoid used in the beamformer computation divided by size of the actual array uncertainty . , Specifically, if . When the design uncertainty equals the actual, the worst-case SINR of the robust beamformer is seen to be 15.63 dB. If the uncertainty ellipsoid used in the RMVB design significantly overestimates or underestimates the actual uncertainty, the worst-case SINR is decreased.

(51) In Fig. 4, we see the square of the norm of the weighted array for output as a function of the hypothesized angle of arrival the RMVB using uncertainty ellipsoids computed according to , 4 , and 0 . If the units of the array (49) and (50) with output correspond to volts or amperes, the square of the magnitude of the weighted array output has units of power. This plot is referred to in the literature as a spatial ambiguity function; its resolution is seen to decrease with increasing uncertainty ellipcorresponds to the soid size. The RMVB computed for Capon beamformer. The spatial ambiguity function using the Capon beamformer provides an accurate power estimate only when the assumed array manifold equals the actual. Prior to publication, we learned of a work similar to ours by Li et al. [32], in which the authors suggest that our approach

TLFeBOOK

1692

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

can be “modified to eliminate the scaling ambiguity when estimating the power of the desired signal.” We submit that 1) there is no scaling ambiguity, and 2) the approach suggested in [32] is counter productive. First, the array response is not an abstract quantity. The array consists of sensors, each element transforming a time-varying physical quantity such as electric field strength or acoustic pressure to another quantity such as voltage or current. The array response can then be measured and expressed in terms of SI (International System) units. The effect of signal processing electronics can be similarly characterized. The sample covariance matrix, being derived from samples of the array output, is hence unambiguous, and no scaling ambiguity exists. Second, sensor arrays do not generally have constant vector norm for all angles of arrival and for all frequencies of interest. Li et al. [32] suggest normalizing the nominal array response to a constant equal to the number of sensor elements. This normalization appears to discard useful information about the array response, namely its norm, which can serve no useful end. We summarize the effect of differences between assumed and actual uncertainty regions on the performance of the RMVB. •

If the assumed uncertainty ellipsoid is smaller than the actual uncertainty, the minimum gain constraint will generally not be met, and the performance may degrade substantially. The power estimate, which is computed using the RMVB as in (51), is not guaranteed to be an upper bound, even when an accurate covariance is used in the computation. • If assumed uncertainty is greater than the actual uncertainty, the performance is generally degraded, but the minimum gain in the desired look direction is maintained. Given accurate covariance, the appropriately scaled weighted power out of the array yields an upper bound on the power of the received signal. The performance of the RMVB is not optimal with respect to SINR; it is optimal in the following sense. For a fixed covariance matrix and an array response contained in an ellipsoid , no other vector achieves a lower weighted power out of the array while maintaining the real part of the response greater than unity for all values of the array contained in . Given an ellipsoidal uncertainty model of the array manifold and a beamformer vector, the minimum gain for the desired signal can be computed directly. If this array uncertainty is subject to a multiplicative uncertainty, verification of this minimum gain constraint is far more difficult. In Section V, we extend the methods of this section to the case of multiplicative uncertainties by computing an outer approximation to the element-wise or Hadamard product of ellipsoids. Using this approximation, no subsequent verification of the performance is required. Prior to this, we describe two methods for computing ellipsoids covering a collection of points.

IV. ELLIPSOIDAL MODELING The uncertainty in the response of an antenna array to a plane wave arises principally from two sources: uncertainty in the

AOA and uncertainty in the array manifold given perfect knowledge of the AOA. In this section, we describe methods to compute an ellipsoid that covers the range of possible values given these uncertainties. A. Ellipsoid Computation Using Mean and Covariance of Data If the array manifold is measured in a controlled manner, the ellipsoid describing the array manifold may be generated from the mean and covariance of the measurements from repeated trials. If the array manifold is predicted from numerical simulations, the uncertainty may take into account variation in the array response due to manufacturing tolerance, termination impedance, and similar effects. If the underlying distribution is multivariate normal, the standard deviation ellipsoid would be , expected to contain a fraction of points equal to where is the dimension of the random variable. We may generate an ellipsoid that covers a collection of points by using the mean as the center and an inflated covariance. While this method is very efficient numerically, it is possible to generate “smaller” ellipsoids using the methods of the next section. B. Minimum Volume Ellipsoid (MVE) be a set of samples of possible Let . Assume that is bounded. values of the array manifold In the case of a full rank ellipsoid, the problem of finding the minimum volume ellipsoid containing the convex hull of can be expressed as the following semidefinite program (SDP): minimize subject to (52) See Vandenberghe and Boyd [33] and Wu and Boyd [34]. The minimum-volume ellipsoid containing is called the LöwnerJohn ellipsoid. Equation (52) is a convex problem in variables and . For full rank (53) with and . The choice of is not unique; in fact, any matrix of the form will satisfy (53), where is any real unitary matrix. Commonly, is often well approximated by an affine set of , and (52) will be poorly conditioned numerdimension ically. We proceed by first applying a rank-preserving affine to the elements of , with transformation . The matrix consists of the left singular vectors, corresponding to the nonzero singular values, of matrix the

We may then solve (52) for the minimum volume, nondegen, which covers the image of under . erate ellipsoid in The resulting ellipsoid can be described in as as in (13), with and .

TLFeBOOK

LORENZ AND BOYD: ROBUST MINIMUM VARIANCE BEAMFORMING

1693

For an -dimensional ellipsoid description, a minimum of points are required, i.e., . Compared to an ellipsoid based on the first- and second-order statistics of the data, a minimum volume ellipsoid is robust in the sense that it is guaranteed to cover all the data points used in the description; the MVE is not robust to data outliers. The computation of the covering ellipsoid is relatively complex; see Vandenberghe et al. [35]. In applications where a real-time response is required, the covering ellipsoid calculations may be profitably performed in advance. V. UNCERTAINTY ELLIPSOID CALCULUS Instead of computing ellipsoid descriptions to represent collections of points, we consider operations on ellipsoids. While it is possible to develop tighter ellipsoidal approximations using the methods of the previous section, the computational burden of these methods often precludes their use. A. Sum of Two Ellipsoids in terms Recall that we can parameterize an ellipsoid in and a symmetric non-negative definite of its center as configuration matrix

where is any matrix square root satisfying . Let and . The range is of values of the geometrical (or Minkowski) sum contained in the ellipsoid (54) for all

, where (55)

see Kurzhanski and Vályi [36]. The value of is commonly . chosen to minimize either the determinant or the trace of Minimizing the trace of in (55) affords two computational advantages over minimizing the determinant. First, computing operations; minthe optimal value of can be done with . Second, the minimum imizing the determinant requires trace calculation may be used without worry with degenerate ellipsoids. There exists an ellipsoid of minimum trace, i.e., sum of squares of the semiaxes, that contains the sum ; it is described by , where is as in (55), (56) and denotes trace. This fact, which is noted by Kurzhanski and Vályia [36, §2.5], may be verified by direct calculation.

Fig. 5. Possible values of array manifold are contained in ellipsoid E ; the values of gains are described by ellipsoid E . The design variable w needs to consider the multiplicative effect of these uncertainties.

precisely known. The gains may be known to have some formal uncertainty; in other applications, these quantities are estimated in terms of a mean vector and covariance matrix. In both cases, this uncertainty is well described by an ellipsoid; this is depicted schematically in Fig. 5. Assume that the range of possible values of the array mani. fold is described by an ellipsoid Similarly, assume the multiplicative uncertainties lie within a . The set of possible second ellipsoid values of the array manifold in the presence of multiplicative uncertainties is described by the numerical range of the Hadamard, and . We will develop outer i.e., element-wise product of approximations to the Hadamard product of two ellipsoids. In Section V-B2, we consider the case where both ellipsoids describe real numbers; the case of complex values is considered in Section V-B3. Prior to this, we will review some basic facts about Hadamard products. 1) Preliminaries: The Hadamard product of vectors is the element-wise product of the entries. We denote the Hadamard product of vectors and as

The Hadamard product of two matrices is similarly denoted and also corresponds to the element-wise product; it enjoys considerable structure [37]. As with other operators, we will consider the Hadamard product operator to have lower precedence than ordinary matrix multiplication. Lemma 3: For any

Proof: Direct calculation shows that the , entry of the , which can be regrouped as . product is Lemma 4: Let and . Then, the field of values of the Hadamard are contained in the ellipsoid product

B. Outer Approximation to the Hadamard Product of Two Ellipsoids In practice, the output of the antenna array is often subject to uncertainties that are multiplicative in nature. These may be due to gains and phases of the electronics paths that are not

TLFeBOOK

Proof: By Lemma 3, we have

1694

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

in particular

We can expand

as

(57) The Hadamard product of two positive semidefinite matrices is positive semidefinite [37, pp. 298–301]; hence, the last three terms on the right-hand side of (57) are all positive semidefinite. Therefore Fig. 6. Samples of the Hadamard product of two ellipsoids. The outer approximations based on the minimum volume and minimum trace metrics are labeled E and E .

Lemma 5: Let . The Hadamard product of vector in the ellipsoid

, and let be any is contained in

Proof: This is simply a special case of Lemma 3. 2) Outer Approximation: Let and be ellipsoids in . Let and be -dimensional vectors taken from ellipsoids and , , we have respectively. Expanding the Hadamard product (58) By Lemmas 4 and 5, the field of values of the Hadamard product

The parameters of

are

Samples of the Hadamard product of are shown in Fig. 6 along with the outer approximations based on the minimum and , respectively. volume and minimum trace metrics 3) Complex Case: We now extend the results of Section V-B2 to the case of complex values. Again, we will compute the approximating ellipsoid using the minimum trace metric. As before, we will consider complex numbers to be represented by the direct sum of their real and imaginary and be the direct sum components. Let and , respectively, i.e., representations of

is contained in the geometrical sum of three ellipsoids (59) Ignoring the correlations between terms in the above expansion, , where we find that

We can represent the real and imaginary components of as

(61) (60) and . The values of and may be for all chosen to minimize the trace or the determinant of . In addition to requiring much less computational effort, the trace metric is numerically more reliable; if either or has a very small entry, the corresponding term in expansion (60) will be poorly conditioned. As a numerical example, we consider the Hadamard product . The ellipsoid is described by of two ellipsoids in

where

and Note that multiplications associated with matrices correspond to reordering of the calculations and not general matrix multiplications. Applying (61) to and yields (62)

TLFeBOOK

LORENZ AND BOYD: ROBUST MINIMUM VARIANCE BEAMFORMING

1695

The direct-sum representation of the field of values of the is contained in the geometrical complex Hadamard product sum of ellipsoids

in (62). zeroing the term The desired outer approximation is computed as the geometrical sum of outer approximations to the remaining five terms, i.e.,

(67) (63) As before, we compute , where the center of the covering ellipsoid is given by the sum of the first two terms of (62); the configuration matrix is calculated by repeatedly applying (54) and (55) to the remaining terms of (62), where is chosen according to (56). 4) Improved Approximation: We now make use of two facts that generally lead to tighter approximations. First, the ellipsoidal outer approximation ignores any correlation between the terms in expansion (62); hence, it is productive to reduce the number of these terms. Consider a Given’s rotation matrix of the form ..

..

.

..

.

.

..

.

(64) The effect of premultiplying a direct sum representation of a complex vector by is to shift the phase of each of component by the corresponding angle . It is not surprising, then, that for and of the form (64), we have all

(65) which does not hold for unitary matrices in general. We now compute rotation matrices and such that the entries associated with the imaginary components of products and , respectively, are set to zero. In computing , we choose the values of in (64) according to . is similarly computed using the values of , i.e., . We change coordinates according to

The rotated components associated with the ellipsoid centers have the form

(66)

Second, while the Hadamard product is commutative, the outer approximation based on covering the individual terms in the expansion (62) is sensitive to ordering; simply interchanging and results in different qualities of the dyads approximations. The ellipsoidal approximation associated with this interchanged ordering is given by

(68) Since our goal is to find the smallest ellipsoid covering the numerical range of , we compute the trace associated with both orderings and choose the smaller of the two. This determination can be made without computing the minimum trace ellipsoids be the minexplicitly, making use of the following fact. Let . The trace of is imum trace ellipsoid covering given by

which may be verified by direct calculation. Hence, determining which of (67) and (68) yields the smaller trace can be performed calculations. After making this determination, we perin form the remainder of the calculations to compute the desired configuration matrix . We then transform back to the original coordinates according to

VI. CONCLUSION The main ideas of our approach are as follows. • The possible values of the manifold are approximated or covered by an ellipsoid that describes the uncertainty. • The robust minimum variance beamformer is chosen to minimize the weighted power out of the array subject to the constraint that the gain is greater than unity for all array manifold values in the ellipsoid. • The RMVB can be computed very efficiently using Lagrange multiplier techniques. • Ellipsoidal calculus techniques may be used to efficiently propagate the uncertainty ellipsoid in the presence of multiplicative uncertainties.

TLFeBOOK

1696

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

REFERENCES [1] J. Capon, “High-resolution frequency-wavenumber spectrum analysis,” Proc. IEEE, vol. 57, no. 8, pp. 1408–1418, Aug. 1969. [2] J. L. Krolik, “The performance of matched-field beamformers with Mediterranean vertical array data,” IEEE Trans. Signal Process., vol. 44, no. 10, pp. 2605–2611, Jan. 1996. , “Matched-field minimum variance beamforming,” J. Acoust. Soc. [3] Amer., vol. 92, no. 3, pp. 1406–1419, Sep. 1992. [4] A. B. Gershman, “Robust adaptive beamforming in sensor arrays,” AEU Int. J. Electron. Commun., vol. 53, no. 6, pp. 305–314, Dec. 1999. [5] D. Johnson and D. Dudgeon, Array Signal Processing: Concepts and Techniques, ser. Signal Processing. Englewood Cliffs, NJ: PrenticeHall, 1993. [6] S. Haykin, Adaptive Filter Theory, ser. Information and System Sciences. Englewood Cliffs, NJ: Prentice-Hall, 1996. [7] K. Harmanci, J. Tabrikian, and J. L. Krolik, “Relationships between adaptive minimum variance beamforming and optimal source localization,” IEEE Trans. Signal Process., vol. 48, no. 1, pp. 1–13, Jan. 2000. [8] S. Q. Wu and J. Y. Zhang, “A new robust beamforming method with antennae calibration errors,” in Proc. IEEE Wireless Commun. Networking Conf., vol. 2, New Orleans, LA, Sep. 1999, pp. 869–872. [9] S. A. Vorobyov, A. B. Gershman, and Z.-Q. Luo, “Robust adaptive beamforming using worst-case performance optimization,” IEEE Trans. Signal Process., vol. 51, no. 2, pp. 313–324, Feb. 2003. , “Robust adaptive beamforming using worst-case performance op[10] timization via second-order cone programming,” in Proc. ICASSP, 2002. [11] R. G. Lorenz and S. P. Boyd, “An ellipsoidal approximation to the Hadamard product of ellipsoids,” in Proc. ICASSP, 2002. , “Robust beamforming in GPS arrays,” in Proc. Inst. Navigation, [12] Nat. Tech. Meeting, Jan. 2002. [13] M. S. Lobo, L. Vandenberghe, S. Boyd, and H. Lebret, “Applications of second-order cone programming,” Linear Algebra Applicat., vol. 284, no. 1–3, pp. 193–228, Nov. 1998. [14] S. P. Boyd and L. Vandenberghe, Course Reader for EE364: Introduction to Convex Optimization With Engineering Applications. Stanford, CA: Stanford Univ. Press, 1999. [15] A. Ben-Tal and A. Nemirovski, “Robust solutions of uncertain linear programs,” Oper. Res. Lett., vol. 25, no. 1, pp. 1–13, 1999. [16] H. Lebret and S. Boyd, “Antenna array pattern synthesis via convex optimization,” IEEE Trans. Signal Process., vol. 45, no. 3, pp. 526–532, Mar. 1997. [17] A. L. Soyster, “Convex programming with set-inclusive constraints and applications to inexact linear programming,” Oper. Res., vol. 21, no. 5, pp. 1154–1157, Sep.–Oct. 1973. [18] L. El Ghaoui and H. Lebret, “Robust solutions to least-squares problems with uncertain data,” SIAM J. Matrix Anal. Applicat., vol. 18, no. 4, pp. 1035–1064, Oct. 1997. [19] A. Ben-Tal and A. Nemirovski, “Robust convex optimization,” Math. Oper. Res., vol. 23, no. 4, pp. 769–805, 1998. [20] A. Ben-Tal, L. El Ghaoui, and A. Nemirovski, “Robustness,” in Handbook on Semidefinite Programming. Boston, MA: Kluwer, 2000, ch. 6, pp. 138–162. [21] A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications, ser. MPS/SIAM Series on Optimization. Philadelphia, PA: SIAM, 2001. [22] W. Gander, “Least squares with a quadratic constraint,” Numerische Mathematik, vol. 36, no. 3, pp. 291–307, Feb. 1981. [23] B. D. Van Veen, “Minimum variance beamforming with soft response constraints,” IEEE Trans. Signal Process., vol. 39, no. 9, pp. 1964–1971, Sep. 1991. [24] G. H. Golub and U. von Matt, “Quadratically constrained least squares and quadratic problems,” Numerische Mathematik, vol. 59, no. 1, pp. 561–580, Feb. 1991. [25] G. Golub and C. V. Loan, Matrix Computations, Second ed. Baltimore, MD: Johns Hopkins Univ. Press, 1989. [26] D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods. Belmont, MA: Athena Scientific, 1996. [27] C. A. Stutt and L. J. Spafford, “A “best” mismatched filter response for radar clutter discrimination,” IEEE Trans. Inf. Theory, vol. IT-14, no. 2, pp. 280–287, Mar. 1968.

[28] Y. I. Abromovich and M. B. Sverdlik, “Synthesis of a filter which maximizes the signal-to-noise ratio under additional quadratic constraints,” Radio Eng. Electron. Phys., vol. 15, no. 11, pp. 1977–1984, Nov. 1970. [29] T. Kailath, A. H. Sayed, and B. Hassibi, Linear Estimation, ser. Information and System Sciences. Upper Saddle River, NJ: Prentice-Hall, 2000. [30] G. Dahlquist and Å. Björck, Numerical Methods, ser. Automatic Computation. Englewood Cliffs, NJ: Prentice-Hall, 1974. [31] J. W. Demmel, Applied Numerical Linear Algebra. Philadelphia, PA: SIAM, 1997. [32] J. Ki, P. Stoica, and Z. Wang, “On robust Capon beamforming and diagonal loading,” IEEE Trans. Signal Process., vol. 51, no. 7, pp. 1702–1715, Jul. 2003. [33] L. Vandenberghe and S. Boyd, “Semidefinite programming,” SIAM Rev., 1995. [34] S.-P. Wu and S. Boyd, “SDPSOL: A parser/solver for semidefinite programs with matrix structure,” in Advances in Linear Matrix Inequality Methods in Control, L. E. Ghaoui and S.-I. Niculescu, Eds. Philadelphia, PA: SIAM, 2000, ch. 4, pp. 79–91. [35] L. Vandenberghe, S. Boyd, and S.-P. Wu, “Determinant maximization with linear matrix inequality constraints,” SIAM J. Matrix Anal. Applicat., vol. 19, no. 2, pp. 499–533, Apr. 1998. [36] A. Kurzhanski and I. Vályi, Ellipsoidal Calculus for Estimation and Control, ser. Systems and Control: Foundations and Applications. Boston, MA: Birkhauser, 1997. [37] R. Horn and C. Johnson, Topics in Matrix Analysis. Cambridge, U.K.: Cambridge Univ. Press, 1991.

Robert G. Lorenz (M’03) received the B.S. degree in electrical engineering and computer science from the University of California, Berkeley, in 1987 and the Ph.D. degree in electrical engineering from Stanford University, Stanford, CA, in 2003. In 1987, he co-founded Ashtech Inc., Sunnyvale, CA, a leading provider of high accuracy Global Positioning System (GPS) surveying systems. More recently, he was the Director of Engineering at SiRF Technology, San Jose, CA, where he led the development of high-sensitivity GPS receivers. Currently, he is the Vice President of Engineering at Beceem Communications, Inc., Santa Clara, CA, where he leads the physical layer development of broadband wireless systems. His current interests include space-time processing for wireless communications, numerical optimization, and array signal processing.

Stephen P. Boyd (SM’97–F’99) received the AB degree in mathematics, summa cum laude, from Harvard University, Cambridge, MA, in 1980 and the Ph.D. degree in electrical engineering and computer science from the University of California, Berkeley, in 1985. He is the Samsung Professor of engineering, Professor of electrical engineering, and Director of the Information Systems Laboratory at Stanford University, Stanford, CA. His current interests include computer-aided control system design and convex programming applications in control, signal processing, and circuit design. He is the author of Linear Controller Design: Limits of Performance (with C. Barratt, 1991), Linear Matrix Inequalities in System and Control Theory (with L. El Ghaoui, E. Feron, and V. Balakrishnan, 1994), and Convex Optimization (with L. Vandenberghe, 2003). Dr. Boyd received an ONR Young Investigator Award, a Presidential Young Investigator Award, and the 1992 AACC Donald P. Eckman Award. He has received the Perrin Award for Outstanding Undergraduate Teaching in the School of Engineering and an ASSU Graduate Teaching Award. In 2003, he received the AACC Ragazzini Education award. He is a Distinguished Lecturer of the IEEE Control Systems Society.

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1697

Blind Spatial Signature Estimation via Time-Varying User Power Loading and Parallel Factor Analysis Yue Rong, Student Member, IEEE, Sergiy A. Vorobyov, Member, IEEE, Alex B. Gershman, Senior Member, IEEE, and Nicholas D. Sidiropoulos, Senior Member, IEEE

Abstract—In this paper, the problem of blind spatial signature estimation using the parallel factor (PARAFAC) analysis model is addressed in application to wireless communications. A time-varying user power loading in the uplink mode is proposed to make the model identifiable and to enable application of PARAFAC analysis. Then, identifiability issues are studied in detail and closed-form expressions for the corresponding modified Cramér–Rao bound (CRB) are obtained. Furthermore, two blind spatial signature estimation algorithms are developed. The first technique is based on the PARAFAC fitting trilinear alternating least squares (TALS) regression procedure, whereas the second one makes use of the joint approximate diagonalization algorithm. These techniques do not require any knowledge of the propagation channel and/or sensor array manifold and are applicable to a more general class of scenarios than earlier approaches to blind spatial signature estimation. Index Terms—Blind spatial signature estimation, parallel factor analysis, sensor array processing.

I. INTRODUCTION HE USE of antenna arrays at base stations has recently gained much interest due to their ability to combat fading, increase system capacity and coverage, and mitigate interference [1]–[5]. In the uplink communication mode, signals from different users can be separated at the base station antenna array based on the knowledge of their spatial signatures [5]–[8]. In particular, known spatial signatures can be used for beamforming to separate each user of interest from the other (interfering) users. However, user spatial signatures are usually unknown at the base station and, therefore, have to be estimated.

T

Manuscript received July 21, 2003; revised March 25, 2004. The work of A. B. Gershman was supported by the Wolfgang Paul Award Program of the Alexander von Humboldt Foundation, Germany; the Natural Sciences and Engineering Research Council (NSERC) of Canada; Communications and Information Technology Ontario (CITO); and the Premier’s Research Excellence Award Program of the Ministry of Energy, Science, and Technology (MEST) of Ontario. The work of N. D. Sidiropoulos was supported by the Army Research Laboratory through participation in the ARL Collaborative Technology Alliance (ARL-CTA) for Communications and Networks under Cooperative Agreement DADD19-01-2-0011. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Constantinos B. Papadias. Y. Rong and S. A. Vorobyov are with the Department of Communication Systems, University of Duisburg-Essen, Duisburg, 47057 Germany. A. B. Gershman is with the Department of Communication Systems, University of Duisburg-Essen, Duisburg, 47057 Germany, on leave from the Department of Electrical and Computer Engineering, McMaster University, Hamilton, ON, L8S 4K1 Canada. N. D. Sidiropoulos is with the Department of Electronic and Computer Engineering, Technical University of Crete, Chania 73100, Greece, and also with the Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455 USA. Digital Object Identifier 10.1109/TSP.2005.845441

Traditional (nonblind) approaches to spatial signature estimation make use of training sequences that are periodically transmitted by each user and are known at the base station [6]. However, the use of training sequences reduces the information transmission rate, and strict coordination of the training epochs of several users in a multiuser setting requires tight synchronization. As a result, blind spatial signature estimation techniques have attracted significant attention in the literature [8]–[16]. There are several blind approaches to spatial signature estimation. The most common one is based on the parametric modeling of spatial signatures using direction-of-arrival (DOA) parameters [5], [8], [9]. For example, in [5], the coherently distributed source model is used to parameterize the spatial signature. Unfortunately, the source angular spread should be small for the first-order Taylor series expansion used in [5] to be valid. This is a limitation for mobile communications applications in urban environments with low base station antenna mast heights, where angular spreads up to 25 are typically encountered [17], [18]. Furthermore, the approach of [5] requires precise array calibration. Two other DOA-based blind spatial signature estimation methods are developed in [8] and [9]. In these papers, the source spatial signature is modeled as a plane wave distorted by unknown direction-independent gains and phases. The latter assumption can be quite restrictive in wireless communications where spatial signatures may have an arbitrary form, and therefore, such gains and phases should be modeled as DOA-dependent quantities. As a result, the techniques of [8] and [9] are applicable to a particular class of scenarios only. Another popular approach to blind spatial signature estimation makes use of the cyclostationary nature of communication signals [10], [11]. This approach does not make use of any DOA-based model of spatial signatures, but it is applicable only to users that all have different cyclic frequencies. The latter condition implies that the users must have different carrier frequencies [which is not the case for Space-Division Multiple Access (SDMA)] and/or baud rates [11]. This can limit practical applications of the methods of [10] and [11]. One more well-developed approach to this problem employs higher order statistics (cumulants) to estimate spatial signatures in a blind way [12]–[16]. Cumulant-based methods are only applicable to non-Gaussian signals. Moreover, all such algorithms are restricted by the requirement of a large number of snapshots. This requirement is caused by a slow convergence of sample estimates of higher order cumulants. The aforementioned restrictions of cumulant-based methods have been a strong motivation for further attempts to develop

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1698

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

blind spatial signature estimators that are based on second-order statistics only and do not require any DOA-related or cyclostationarity assumptions. In [15], such a method was proposed using joint approximate diagonalization of a set of spatial autoand cross-covariance matrices. This method requires an existence of a long-time coherence of the source signals to obtain enough cross-covariance matrices at multiple lags for the joint diagonalization process and to guarantee identifiability. In practical wireless communication systems, the signal time coherence is severely limited, i.e., the correlation time of the received signals typically does not largely exceed the sampling interval. For example, communication signals sampled at the symbol rate are uncorrelated,1 and hence, higher lag correlations are all zero. In such cases, multiple covariance matrices are unavailable, and the method of [15] is not applicable. Furthermore, [15] offers limited identifiability—for example, it requires that the matrix of spatial signatures be full column rank, and therefore, the number of sources should be less or equal to the number of antennas. In this paper, we develop a new bandwidth-efficient approach to blind spatial signature estimation using PARAFAC analysis [20]–[23]. Our approach does not require any restrictive assumptions on the array geometry and the propagation environment. Time-varying user power loading is exploited to obtain multiple spatial zero-lag covariance matrices required for the PARAFAC model. Blind PARAFAC multisensor reception and spatial signature estimation have been considered earlier in [21] and [23]. However, the approach of [21] is applicable to direct sequence-code division multiple access (DS-CDMA) systems only, as spreading is explicitly used as the third dimension of the data array, whereas [23] requires multiple shifted but otherwise identical subarrays and a DOA parameterization. Below, we show that the proposed user power loading enables us to give up the CDMA and multiple-invariance/DOA parameterization assumptions and extend the blind approach to any type of SDMA system employing multiple antennas at the receiver. Blind source separation of nonstationary sources using multiple covariance matrices has also been considered in [24] but, again, under limited identifiability conditions, stemming from the usual ESPRIT-like solution. Our identifiability results are considerably more general as they do not rely on this limited viewpoint. The rest of this paper is organized as follows. The signal model is introduced in Section II. Section III formulates the spatial signature estimation problem in terms of three-way analysis using time-varying user power loading. The identifiability of this model is studied in Section IV. Two spatial signature estimators are presented in Section V: PARAFAC fitting based on the trilinear alternating least squares (TALS) regression procedure and a joint approximate diagonalization-based estimator. A modified deterministic CRB for the problem at hand is derived in Section VI. Simulation results are presented in Section VII. Conclusions are drawn in Section VIII. 1Channel-coded signals, which include redundancy for error correction, are in fact interleaved before transmission, with the goal of making the transmitted signal approximately uncorrelated.

II. DATA MODEL sensors receive the signals from narLet an array of rowband sources. We assume that the observation interval is shorter than the coherence time of the channel (i.e., the scenario is time-invariant), and the time dispersion introduced by the multipath propagation is small in comparison with the reciprocal of the bandwidth of the emitted signals [5]. Under such snapshot vector of antenna array outputs assumptions, the can be written as [5] (1) where is the matrix of the user spais the spatial signatures, tial signature of the th user, is the vector of the equivalent baseband user waveforms, is the vector of additive spatially and temporally white Gaussian noise, and denotes the transpose. Note that in contrast to direction finding problems, the matrix is unstructured. Assuming that there is a block of snapshots available, the model (1) can be written as (2) is the array data matrix, is the user waveform matrix, is the sensor noise matrix. and A quasistatic channel is assumed throughout the paper. This assumption means that the spatial signatures are block time-invariant (i.e., the elements of remain constant over a block of snapshots). Assuming that the user signals are uncorrelated with each other and sensor noise, the array covariance matrix of the received signals can be written as where

(3) where is the diagonal covariance matrix of is the sensor noise variance, is the the signal waveforms, denotes the Hermitian transpose. identity matrix, and The problem studied in this paper is the estimation of the matrix from noisy array observations . III. PARAFAC MODEL Before proceeding, we need to clarify that by identifiability, we mean the uniqueness (up to inherently unresolvable source permutation and scale ambiguities) of all user spatial signatures given the exact covariance data. Identifiability in this sense is impossible to achieve with only one known covariance matrix (3) because the matrix can be estimated from only up to an arbitrary unknown unitary matrix [22]. The approach we will use to provide a unique user spatial signature estimation is based on an artificial user power loading and PARAFAC model analysis. Therefore, next, we explain how this model is related to our problem. Let us divide uniformly the whole data block of snapshots into subblocks so that each subblock contains

TLFeBOOK

RONG et al.: BLIND SPATIAL SIGNATURE ESTIMATION VIA TIME-VARYING USER POWER LOADING

snapshots, where denotes the largest integer less than . We fix the transmit power of each user within each subblock while changing it artificially2 between different subblocks. It should be stressed that the proposed artificial time-varying user power loading does not require precise synchronization among the users, but the users should roughly know the boundaries of epochs over which the powers are kept constant (this can be achieved, for example, using the standard power control feedback channel). Therefore, a certain level of user coordination is required from the transmitter side.3 We stress that the proposed user power loading can be easily implemented by overlaying a small power variation on top of the usual power control, without any other modifications to existing hardware or communication system/network parameters. In addition, as it will be seen in the sequel, the user powers need not vary much to enable blind identification. In particular, power variations that will be used are on the order of 30%. Such power variations will not significantly affect the bit error rate (BER), which is seriously affected only when order-of-magnitude power variations are encountered. If power control is fast enough (in the sense that there are several power changes per channel coherence dwell), we can exploit it as a sort of user power loading. However, power control is usually much slower than the channel coherence time, because its purpose is to combat long-term shadowing. For this reason, in practice, it may not be possible to rely on the power control variations, and we need to induce a faster (but much smaller in magnitude) power variation on top of power control. This extra power variation need not “follow the channel”, i.e., it can be pseudo-random, and hence, the channel need not be measured any faster than required for regular power control. Using the proposed power loading, the received snapshots within any th subblock correspond to the following covariance matrix: (4) is the diagonal covariance matrix of the user wavewhere forms in the th subblock. Using all subblocks, we will have different covariance matrices . Note that these matrices differ from each other only because the signal differ from one subblock waveform covariance matrices to another. In practice, the noise power can be estimated and then subtracted from the covariance matrix (4). Let us stack the matrices , together to form a three-way array , which is natural to call the covariance th element of such an array can be written as array. The

where th subblock, and the matrix

1699

is the power of the th user in the denotes the complex conjugate. Defining as .. .

..

.

.. .

we can write the following relationship between

that the effect of time-varying user powers has been exploited in [24], where an ESPRIT-type algorithm has been proposed for blind source separation of nonstationary sources. Similar ideas have been used in [15] and [25]. However, the authors of [15], [24], and [25] assume that the source powers vary because of signal nonstationarity rather than artificial power loading. 3As it will be seen from our simulations, the methods proposed in the present paper will work well, even in the case when there is no user coordination (i.e., in the unsynchronized user case).

and

: (7)

for all . In (7), is the operator that makes a diagonal matrix by selecting the th row and putting it on the main diagonal while putting zeros elsewhere. is a sum of rank-1 triple prodEquation (5) implies that is sufficiently small,4 (5) represents a low-rank deucts. If composition of . Therefore, the problem of spatial signature estimation can be reformulated as the problem of low-rank decomposition of the three-way covariance array . IV. PARAFAC MODEL IDENTIFIABILITY In this section, we study identifiability of the PARAFAC model-based spatial signature estimation. Toward this end, we discuss conditions under which the trilinear decomposition is unique. Identifiability conditions on the number of of subblocks and the number of array sensors are derived. We start with the definition of the Kruskal rank of a matrix [20]. Definition: The Kruskal rank (or -rank) of a matrix is if and only if every columns of are linearly independent has columns or contains a set of and either linearly dependent columns. Note that -rank is always less than or equal to the conventional matrix rank. It can be easily checked that if is full column rank, then it is also full rank. Using (7) and assuming that the noise term is subtracted from , we can rewrite (4) as the matrix (8) for all

. Let us introduce the matrix

.. .

.. .

(5)

2Note

(6)

(9) is the Khatri–Rao (column-wise Kronecker) matrix where product [23]. To establish identifiability, we have to obtain under which via matrices conditions the decomposition (9) of the matrix and is unique (up to the scaling and permutation ambiguities). In [20], the uniqueness of trilinear decomposition for 4Exact

TLFeBOOK

conditions for

M are given in the next section.

1700

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

the case of real-valued arrays has been established. These results have been later extended to the complex-valued matrix case [21]. In the context of our present application, which involves a conjugate-symmetric PARAFAC model, the results of [20] and [21] specialize to the following Theorem (see also [28] for a discussion of the corresponding real-symmetric model). Theorem 1: Consider the set of matrices (8). If for

•

(11) where is a permutation matrix, and scaling matrices satisfying

and

are diagonal

and

, the value of (15)

is sufficient for almost sure identifiability. Proof: The assumptions of Theorem 2 mean that the following equalities hold almost surely [29]: rank rank

(10) then and are unique up to inherently unresolvable permutation and scaling of columns, i.e., if there exists any other pair that satisfies (10), then this pair is related to the pair via

For

(16) (17)

Substituting (16) and (17) into (10), we have (18) The following cases should be considered: 1) . In this case, . Furthermore, as , . Therefore, (18) is always satisfied. we have that ; . In this case, , , and 2) (18) becomes (19)

(12) , and are always unique, irrespective of (10). For Note that the scaling ambiguity can be easily avoided by taking one of the array sensors as a reference and normalizing user spatial signatures with respect to it. The permutation ambiguity is unremovable, but it is usually immaterial because typically, the ordering of the estimated spatial signatures is unimportant. It is worth noting that condition (10) is sufficient for identior but is not fiability and is necessary only if [27]. Furthermore, for , the condinecessary if tion becomes necessary [26]. In terms of the number of subblocks, the latter condition requires that (13) The practical conclusion is that in the multiuser case, not less than two covariance matrices must be collected to uniquely identify , which means that the users have to change their powers at least once during the transmission. Similarly, it is nec. essary that The following result gives sufficient conditions for the number of sensors to guarantee almost sure identifiability.5 Theorem 2: Suppose the following. • The elements of are drawn from distribution , which is assumed continuous with respect to . the Lebesgue measure in , • The elements of are drawn from distribution which is assumed continuous with respect to the Lebesgue measure in . Then, we have the following. • For , the value of (14)

3)

,

, and (20)

This inequality is equivalent to (15).

V. ESTIMATORS We will now develop two techniques for blind spatial signature estimation based on the PARAFAC model of Section III. are unavailIn practice, the exact covariance matrices , able but can be estimated from the array snapshots . The sample covariance matrices are given by (21) These matrices can be used to form a sample three-way covariance array denoted as . , then the noise power can be estimated as the If eigenvalues of the matrix average of the smallest (22) can be subtracted from and the estimated noise component subblocks of the sample covariance array . In case , noise power can be estimated on system start-up before any transmission begins. To formulate our techniques, we will need “slices” of the matrices and along different dimensions [21]. Toward this end, let us define the “slice” matrices as (23)

is sufficient for almost sure identifiability. 5The

This inequality is equivalent to (14). ; . In this case, (18) can be written as

definition of almost sure identifiability in the context discussed is given

in [29].

TLFeBOOK

(24) (25)

RONG et al.: BLIND SPATIAL SIGNATURE ESTIMATION VIA TIME-VARYING USER POWER LOADING

where Similarly

;

; and

1701

whose analytic solution is given by

.

(34) (26) (27)

•

(28) ; ; and where For the sake of convenience, let us introduce rewrite (9) as

denotes the matrix pseudoinverse. Set . where Step 3: Find the estimate of by solving the following LS problem: (35)

. whose analytic solution is given by

and

(36) (29)

.. .

•

Set . Step 4: Find the estimate of LS problem:

by solving the following

In the same way, let us define the matrices

(37) whose analytic solution is given by (30)

.. .

(38) Set . Step 5: Repeat steps 2–4 until convergence is achieved, and then compute the final estimate of as . The complexity of the TALS algorithm is per iteration. It is worth noting that when is small relative to and , only a few iterations of this algorithm are usually required to achieve convergence [23]. •

(31)

.. . and their sample estimates

.. .

.. .

B. Joint Diagonalization-Based Estimator

.. .

(32) Note that for the sake of algorithm simplicity, we will not exploit the fact that our PARAFAC model is symmetric. For example, the algorithm that follows in the next subsection treats and as independent variables; symmetry will only be exploited in the calculation of the final estimate of .

Using the idea of [15], we can obtain the estimate of by means of a joint diagonalizer of the matrices , . The estimator can be formulated as the following sequence of steps: • Step 1: Calculate the eigendecomposition of , and find of the noise power as the average of the the estimate smallest eigenvalues of this matrix. • Step 2: Compute the whitening matrix as

A. TALS Estimator The basic idea behind the TALS procedure for PARAFAC fitting is to update each time a subset of parameters using LS regression while keeping the previously obtained estimates of the rest of parameters fixed. This alternating projections-type procedure is iterated for all subsets of parameters until convergence is achieved [19], [21], [23], [30]. In application to our problem, the PARAFAC TALS procedure can be formulated as follows. • Step 1: Initialize and . • Step 2: Find the estimate of by solving the following LS problem:

(39)

•

are the largest (signal-subspace) eigenwhere values of , and are the corresponding eigenvectors. Step 3: Compute the prewhitened sample covariance matrices as (40)

• •

(33)

TLFeBOOK

Step 4: Obtain a unitary matrix as a joint diagonalizer of the set of matrices . Step 5: Estimate the matrix as (41)

1702

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Several efficient joint diagonalization algorithms can be used in Step 4; see [31] and [32]. For example, the complexity of the per iteration. ac-dc algorithm of [32] is It should be pointed out that the joint diagonalization-based estimator requires stronger conditions in terms of the number of is sensors as compared to the TALS estimator. Indeed, required for the joint diagonalization algorithms [15] and [32], whereas this constraint is not needed for TALS. Both the TALS and joint diagonalization algorithms can be initialized randomly [23]. Alternatively, if power control is fast enough (in the sense that there are several power changes per channel coherence dwell), we can use the fact that the power changes are known at the base station to initialize the matrix in TALS. However, as mentioned in Section III, power control algorithms are usually much slower than the channel coherence time because their purpose is to combat long-term shadowing. For this reason, such an initialization of may not be possible.

our simulation results in the next section, the performance of the proposed estimators is rather close to this optimistic CRB, and therefore, this bound is relevant. In addition, note that the parameter is decoupled with other parameters in the Fisher information matrix (FIM) [34]. Therecan be excluded from the fore, without loss of generality, vector of unknown parameters. A delicate point regarding the CRB for model (42) is the inherent permutation and scaling ambiguities. To get around the problem of scaling ambiguity, we assume that each spatial signature vector is normalized so that its first element is equal to one (after such a normalization the first row of becomes [1, ,1]). To avoid the permutation ambiguity, we assume that the first row of is known and consists of distinct elements. Then, the vector of the parameters of interest can be written as (46) where

VI. MODIFIED CRAMÉR–RAO BOUND

Re

In this section, we present a modified deterministic CRB on estimating the user spatial signatures.6 The model (1) for the th sample of the th subblock can be rewritten as (42) where

Im (47)

The vector of nuisance parameters can be expressed as (48) where is the th row of the matrix . Using (46) and (48), the vector of unknown parameters can be written as (49)

(43) is the vector of normalized signal waveforms, and the normalization is done so that all waveforms have unit powers. Hence, the observations in the th subblock satisfy the following model:

Theorem 3: The Fisher Information Matrix (FIM) is given by (50), shown at the bottom of the page, where Re Im

(44) where

Im Re

Re

(52)

(45)

(53)

The unknown parameters of the model (42) are all entries of , diagonal elements of , and the noise power . Note that to make the model (42) identifiable, we assume that the signal waveforms are known. Therefore, we study a modified (optimistic) CRB. However, as follows from

.. .

..

.. .

6The deterministic CRB is a relevant bound in cases when the signal waveforms are unknown deterministic or random with unknown statistics; see, e.g., [33] and [34].

..

(51)

.. .

.

..

.

(54)

.. .

(55)

. (50) .. .

..

TLFeBOOK

.

RONG et al.: BLIND SPATIAL SIGNATURE ESTIMATION VIA TIME-VARYING USER POWER LOADING

Fig. 1. RMSEs versus synchronized users.

N for K

= 10 and SNR = 10 dB. First example,

Re

Im

Im

1703

Fig. 2. RMSEs versus the SNR for synchronized users.

K = 10 and N = 1000. First example,

(56)

Re

(57) (58) Re Im

(59) (60)

(61)

(62) and denotes the Kronecker product. spatial signature-related block The of the CRB matrix is given in closed form as

Fig. 3. BERs versus the SNR for synchronized users.

K = 10 and N

= 1000. First example,

The obtained CRB expressions will be compared with the performance of the TALS and joint diagonalization-based estimators in the next section.

CRB

VII. SIMULATIONS

Re

(63)

where the upper-left block of (50) can be expressed as ..

. Re Im

Proof: See the Appendix.

Im Re

(64)

In this section, the performance of the developed blind spatial signature estimators is compared with that of the ESPRIT-like estimator of [8], the generalized array manifold (GAM) MUSIC estimator of [5], and the derived modified deterministic CRB. Although the proposed blind estimators are applicable to general array geometries, the ESPRIT-like estimator is based on the uniform linear array (ULA) assumption. Therefore, to compare the estimators in a proper way, we assume a ULA of omnidirectional sensors spaced half a wavelength apart and binary phase shift keying (BPSK) user signals impinging on and relative to the broadside, the array from the angles where in each simulation run, and are randomly uniformly

TLFeBOOK

1704

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

joint diagonalization algorithm is the relative improvement in joint diagonalization error. The algorithms are stopped if such errors become small. Typically, both algorithms converged in less than 30 iterations. In most figures, the estimator performances are compared in terms of the root-mean-square error (RMSE) RMSE

Fig. 4. RMSEs versus unsynchronized users.

N for K

= 10 and SNR = 10 dB. First example,

Fig. 5. RMSEs versus the SNR for unsynchronized users.

K = 10 and N = 1000. First example,

drawn from the whole field of view . Throughout the simulations, the users are assumed to be synchronized (except Figs. 4 and 5, where the case of unsynchronized users is consubblocks are used in our techniques (except sidered), Fig. 10, where is varied), and the user powers are changed between different subblocks uniformly with a constant power change factor (PCF) of 1.2 (except Fig. 9, where the PCF is SNR , where SNR is varied). Note that the average user SNR in a single sensor, is the matrix whose elements are all equal to one, is a random matrix whose elements are uniformly and independently drawn from the interval . [ 0.5,0.5], and it is assumed that To implement the PARAFAC TALS and joint diagonalization-based estimators, we use the COMFAC algorithm of [30] and AC-DC algorithm of [32], respectively. Throughout the simulations, both algorithms are initialized randomly. The stopping criterion of the TALS algorithm is the relative improvement in fit from one iteration to the next. The stopping criterion of the

(65)

where is the number of independent simulation runs, is the estimate of obtained from the th run. Note and that permutation and scaling of columns is fixed by means of a least-squares ordering and normalization of the columns of . A greedy least-squares algorithm [21] is used to match to those of . We first form the (normalized) columns of distance matrix whose th element contains an the Euclidean distance between the th column of and the th column of . The smallest element of this distance matrix determines the first match, and the respective row and column of this matrix are deleted. The process is then repeated with the reduced-size distance matrix. The CRB is averaged over simulation runs as well. To verify that the RMSE is a proper performance measure in applications to communications problems, one of our figures also illustrates the performance in terms of the BER when the estimated spatial signatures are used together with a typical detection strategy to estimate the transmitted bits. Example 1—Unknown Sensor Gains and Phases: Following [8], we assume in our first example that the array gains and phases are unknown, i.e., the received data are modeled as (2) with

where is the matrix of nominal (plane-wavefront) user spatial signatures, and is the diagonal matrix containing the array unknown gains and phases, i.e., diag . The unknown gains are independently drawn in each simulation run from the uniform random generator with the mean equal to and standard deviation equal to one, whereas the unknown are independently and uniformly drawn phases . from the interval Fig. 1 displays the RMSEs of our estimators and the ESPRIT, like estimator of [8] along with the CRB versus for dB. Fig. 2 shows the performances of the same and SNR and estimators and the CRB versus the SNR for . Fig. 3 illustrates the performance in terms of the BER when the estimated spatial signatures are used to detect the transmitted . bits via the zero-forcing (ZF) detector given by sign To avoid errors in computing the pseudoinverse of the matrix , the runs in which was ill-conditioned have been dropped. The resulting BERs are displayed versus the SNR for and . Additionally, the results of the so-called clairare displayed in this figure. voyant ZF detector sign Note that the latter detector corresponds to the ideal case when the source spatial signatures are exactly known, and therefore,

TLFeBOOK

RONG et al.: BLIND SPATIAL SIGNATURE ESTIMATION VIA TIME-VARYING USER POWER LOADING

Fig. 6. RMSEs versus synchronized users.

N for K

= 4 and SNR = 10 dB. First example,

Fig. 7. RMSEs versus the SNR for synchronized users.

K = 4 and N

= 1000. First example,

it does not correspond to any practical situation. However, its performance is included in Fig. 3 for the sake of comparison as a benchmark. To demonstrate that the proposed techniques are insensitive to user synchronization, Figs. 4 and 5 show the RMSEs of the same methods and in the same scenarios as in Figs. 1 and 2, respectively, but for the case of unsynchronized users.7 To evaluate the performance with a smaller number of sensors, Fig. 6 compares the RMSEs of the estimators tested versus for and SNR dB. Fig. 7 displays the perand formances of these estimators versus the SNR for . To illustrate how the performance depends on the number of sensors, the RMSEs of the estimators tested are plotted in Fig. 8 versus . Figs. 9 and 10 compare the performances of the proposed PARAFAC estimators versus the PCF and the number 7That is, the user powers vary without any synchronization between the users.

Fig. 8. RMSEs versus synchronized users.

1705

K for SNR = 10 dB and N = 1000. First example,

Fig. 9. RMSEs versus the PCF for SNR = 10 dB and example, synchronized users.

N

= 1000. First

of subblocks , respectively. In these figures, and SNR dB. Example 2—Unknown Coherent Local Scattering: In our second example, we address the scenario where the spatial signature of each nominal (plane-wavefront) user is distorted by local scattering effects [17], [18]. Following [35], the th user spatial signature is formed in this example by five signal paths of the same amplitude including the single direct path and four coherently scattered paths. Each of these paths is characterized by its own angle and phase. The angle of the direct path is equal to the nominal user DOA, whereas the angles of scattered paths are independently drawn in each simulation run from a uniform random generator with the mean equal to the nominal user DOA and the standard deviations equal to 8 and 10 for the first and second users, respectively. The path phases for each user are uniformly and independently drawn in each simulation . run from the interval

TLFeBOOK

1706

Fig. 10. RMSEs versus synchronized users.

Fig. 11. RMSEs versus synchronized users.

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

P for SNR = 10 dB and N = 1000. First example,

N for K = 10 and SNR = 10 dB. Second example,

Note that in the second example, it is improper to compare the proposed techniques with the ESPRIT-like estimator of [8] because the latter estimator is not a relevant technique for the scenario considered. Therefore, in this example, we compare our techniques to the GAM-MUSIC estimator of [5]. Fig. 11 displays the performance of the spatial signature estiand mators tested versus the number of snapshots for dB. Note that the SNR is defined here by taking into SNR account all signal paths. The performance of the same methods and is displayed in versus the SNR for Fig. 12. Discussion: Our simulation results clearly demonstrate that the proposed blind PARAFAC spatial signature estimators substantially outperform the ESPRIT-like estimator and the GAM-MUSIC estimator. These improvements are especially pronounced at high values of SNR, number of snapshots, and number of sensors.

Fig. 12. RMSEs versus the SNR for example, synchronized users.

K

= 10 and

N

= 1000. Second

Comparing Figs. 1 and 2 with Figs. 4 and 5, respectively, we observe that the requirement of user synchronization is not critical to the performance of both the TALS and joint diagonalization-based algorithms. As a matter of fact, the performances of these techniques do not differ much in the cases of synchronized and unsynchronized users. This means that our techniques can easily accommodate intercell interference, provided that out-of-cell users also play up and down their powers, because the fact that out-of-cell users will not be synchronized is not critical performance-wise. From Fig. 9, it is clear that the performance of the proposed techniques can be improved by increasing the PCF. This figure clarifies that the performance improvements of our estimators over the ESPRIT-like estimator are achieved by means of using the power loading proposed. From Fig. 9, it follows that even moderate values of PCF (1.2 1.4) are sufficient to guarantee that the performances of the proposed PARAFAC estimators are comparable with the CRB and are substantially better than that of the ESPRIT-like estimator. From Fig. 10, we can observe that the performance of the proposed PARAFAC estimators is also improved when increasing the number of subblocks while keeping the total block length fixed. However, this is only true for small numbers of ; for , curves saturate. Note that this figure makes it clear that is sufficient even a moderate number of subblocks to guarantee that the performance is comparable with the CRB and is better than that of the ESPRIT-like estimator. We stress that the effects of the PCF and cannot be seen from the CRB in Figs. 9 and 10 because the time-averaged user powers and the total number of snapshots do not change in these figures. Figs. 11 and 12 show that both the TALS and joint-diagonalization based estimators substantially outperform the GAMMUSIC estimator if the values and SNR are sufficiently high. Interestingly, the performance of GAM-MUSIC does not imor SNR. This observation can prove much when increasing be explained by the fact that the GAM-MUSIC estimator is biased. Note that from Fig. 11, it follows that GAM-MUSIC may

TLFeBOOK

RONG et al.: BLIND SPATIAL SIGNATURE ESTIMATION VIA TIME-VARYING USER POWER LOADING

perform better than the proposed PARAFAC estimators in the case when is small because the power loading approach does not work properly if there are only a few snapshots per subblock (in this case, the covariance matrix estimates for each subblock become very poor). Interestingly, as it follows from Fig. 3, the proposed PARAFAC-based techniques combined with the zero forcing (ZF) detector have the same BER slope as the clairvoyant ZF detector, whereas the performance losses with respect to the latter detector do not exceed 3 dB at high SNRs. There are several reasons why the proposed techniques perform better than the ESPRIT-like algorithm. First of all, even in the case when the array is fully calibrated, the performance of ESPRIT is poorer than that of MUSIC and/or maximum likelihood (ML) estimator because ESPRIT does not take advantage of the full array manifold but only of the array shift-invariance property. Second, our algorithm takes advantage of the user power loading, whereas the ESPRIT-like algorithm does not. As far as the comparison GAM-MUSIC method is concerned, better performances of the proposed techniques can be explained by the above-mentioned fact that GAM-MUSIC uses the first-order Taylor series approximation, which is only adequate for asymptotically small angular spreads. As a result, the GAM-MUSIC estimator is biased. In addition, similarly to the ESPRIT-like algorithm, GAM-MUSIC does not take any advantage of the user power loading. Although the performances of the proposed estimators can be made comparable to the CRB with proper choice of PCF and system parameters, they do not attain the CRB. This can be partially attributed to the fact that the modified CRB is an optimistic one in that it assumes knowledge of the temporal source signals, which are unavailable to the blind estimation algorithms. Furthermore, the TALS estimator does not exploit the symmetry , whereas joint diagonalization reof the model lies on an approximate prewhitening step. Both methods rely on finite-sample covariance and noise-power estimates. This explains the observation that the CRB cannot be attained.

1707

APPENDIX PROOF OF THEOREM 3 th element of the FIM is given by [34]

The

Re

FIM

(66) Using (45) along with (66), we have

(67)

Re

(68)

Im

(69)

where is the vector containing one in the th position and zeros elsewhere. Using (67) and (68) along with (66), we obtain that (70)

Re Re

(71)

where (72) Similarly

VIII. CONCLUSIONS The problem of blind user spatial signature estimation using the PARAFAC analysis model has been addressed. A time-varying user power loading in the uplink mode has been proposed to make the model identifiable and to enable the application of the PARAFAC analysis model. Identifiability issues and the relevant modified deterministic CRB have been studied, and two blind spatial signature estimation algorithms have been presented. The first technique is based on the PARAFAC fitting TALS regression, whereas the second one makes use of joint matrix diagonalization. These techniques have been shown to provide better performance than the popular ESPRIT-like and GAM-MUSIC blind estimators and are applicable to a much more general class of scenarios.

Im

(73)

Therefore

TLFeBOOK

Re

Re .. .

Re Re

..

.. .

. Re

(74)

1708

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

and

Finally, using (67)–(69) along with (66), we can write for ; and

.. .

..

.. .

.

(75)

Using (74) and (75), we obtain (51). Note that the right-hand side of (51) does not depend on the index . Hence

..

Collecting all elements given by the last two equations in one matrix, we obtain

. Re Im

Im Re

Re

(76)

Im .. . Re

Next, using (69) along with (66), we can write, for and

Im (80) Observing that

Re

Re

Re

Im

Im

Im Re

Re Re

(77)

Im (81)

where we can further simplify (80) to (78)

elements given by (77) in one matrix, we have

Stacking all

(82) In addition, note that (83)

Re

Re .. .

Re Re

..

.. .

. Re

(79)

Using (76), (79), (82), and (83), we obtain the expressions (50)–(62). Computing the CRB for requires the inverse of the matrix (50). Our objective is to obtain the CRB associated with the vector

TLFeBOOK

RONG et al.: BLIND SPATIAL SIGNATURE ESTIMATION VIA TIME-VARYING USER POWER LOADING

parameter only, avoiding the inverse of the full FIM matrix. Exploiting the fact that the lower right subblock

..

.

(84)

of (50) is a block-diagonal matrix and using the partitioned matrix inversion lemma (see [34, p. 572]), after some algebra, we obtain (63) and (64), and the proof is complete.

REFERENCES [1] B. Ottersten, “Array processing for wireless communications,” in Proc. 8th IEEE Signal Processing Workshop Statistical Signal Array Process., Corfu, Greece, Jul. 1996, pp. 466–473. [2] A. J. Paulraj and C. B. Papadias, “Space-time processing for wireless communications,” IEEE Signal Process. Mag., vol. 14, pp. 49–83, Nov. 1997. [3] J. H. Winters, “Smart antennas for wireless systems,” IEEE Pers. Commun. Mag., vol. 5, pp. 23–27, Feb. 1998. [4] J. H. Winters, J. Salz, and R. D. Gitlin, “The impact of antenna diversity on the capacity of wireless communication systems,” IEEE Trans. Commun., vol. 42, pp. 1740–1751, Feb.–Apr. 1994. [5] D. Asztèly, B. Ottersten, and A. L. Swindlehurst, “Generalized array manifold model for wireless communication channel with local scattering,” Proc. Inst. Elect. Eng., Radar, Sonar Navigat., vol. 145, pp. 51–57, Feb. 1998. [6] A. L. Swindlehurst, “Time delay and spatial signature estimation using known asynchronous signals,” IEEE Trans. Signal Process., vol. 46, no. 2, pp. 449–462, Feb. 1998. [7] S. S. Jeng, H. P. Lin, G. Xu, and W. J. Vogel, “Measurements of spatial signature of an antenna array,” in Proc. PIMRC, vol. 2, Toronto, ON, Canada, Sep. 1995, pp. 669–672. [8] D. Astèly, A. L. Swindlehurst, and B. Ottersten, “Spatial signature estimation for uniform linear arrays with unknown receiver gains and phases,” IEEE Trans. Signal Process., vol. 47, no. 8, pp. 2128–2138, Aug. 1999. [9] A. J. Weiss and B. Friedlander, “‘Almost blind’ steering vector estimation using second-order moments,” IEEE Trans. Signal Process., vol. 44, no. 4, pp. 1024–1027, Apr. 1996. [10] B. G. Agee, S. V. Schell, and W. A. Gardner, “Spectral self-coherence restoral: a new approach to blind adaptive signal extraction using antenna arrays,” Proc. IEEE, vol. 78, no. 4, pp. 753–767, Apr. 1990. [11] Q. Wu and K. M. Wong, “Blind adaptive beamforming for cyclostationary signals,” IEEE Trans. Signal Process., vol. 44, no. 11, pp. 2757–2767, Nov. 1996. [12] J.-F. Cardoso and A. Souloumiac, “Blind beamforming for nonGaussian signals,” Proc. Inst. Elect. Eng. F, vol. 140, no. 6, pp. 362–370, Dec. 1993. [13] M. C. Dogan and J. M. Mendel, “Cumulant-based blind optimum beamforming,” IEEE Trans. Aerosp. Electron. Syst., vol. 30, pp. 722–741, Jul. 1994. [14] E. Gonen and J. M. Mendel, “Applications of cumulants to array processing, Part III: Blind beamforming for coherent signals,” IEEE Trans. Signal Process., vol. 45, no. 9, pp. 2252–2264, Sep. 1997. [15] A. Belouchrani, K. Abed-Meraim, J.-F. Cardoso, and E. Moulines, “A blind source separation technique using second-order statistics,” IEEE Trans. Signal Process., vol. 45, no. 2, pp. 434–444, Feb. 1997. [16] N. Yuen and B. Friedlander, “Performance analysis of blind signal copy using fourth order cumulants,” J. Adaptive Contr. Signal Process., vol. 10, no. 2/3, pp. 239–266, 1996. [17] K. I. Pedersen, P. E. Mogensen, and B. H. Fleury, “A stochastic model of the temporal and azimuthal dispersion seen at the base station in outdoor propagation environments,” IEEE Trans. Veh. Technol., vol. 49, no. 2, pp. 437–447, Mar. 2000. , “Spatial channel characteristics in outdoor environments and [18] their impact on BS antenna system performance,” in Proc. Veh. Technol. Conf., vol. 2, Ottawa, ON, Canada, May 1998, pp. 719–723.

1709

[19] R. A. Harshman, “Foundation of the PARAFAC procedure: model and conditions for an “explanatory” multi-mode factor analysis,” UCLA Working Papers Phonetics, vol. 16, pp. 1–84, Dec. 1970. [20] J. B. Kruskal, “Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics,” Linear Algebra Applicat., vol. 16, pp. 95–138, 1977. [21] N. D. Sidiropoulos, G. B. Giannakis, and R. Bro, “Blind PARAFAC receivers for DS-CDMA systems,” IEEE Trans. Signal Process., vol. 48, no. 3, pp. 810–823, Mar. 2000. [22] N. D. Sidiropoulos and R. Bro, “On the uniqueness of multilinear decomposition of N-way arrays,” J. Chemometr., vol. 14, pp. 229–239, 2000. [23] N. D. Sidiropoulos, R. Bro, and G. B. Giannakis, “Parallel factor analysis in sensor array processing,” IEEE Trans. Signal Process., vol. 48, no. 8, pp. 2377–2388, Aug. 2000. [24] M. K. Tsatsanis and C. Kweon, “Blind source separation of nonstationary sources using second-order statistics,” in Proc. 32nd Asilomar Conf, Signals, Syst. Comput., vol. 2, Pacific Grove, CA, Nov. 1998, pp. 1574–1578. [25] D.-T. Pham and J.-F. Cardoso, “Blind separation of instantaneous mixtures of nonstationary sources,” IEEE Trans. Signal Process., vol. 49, no. 9, pp. 1837–1848, Sep. 2001. [26] R. L. Harshman, “Determination and proof of minimum uniqueness conditions for PARAFAC1,” UCLA Working Papers Phonetics, vol. 22, pp. 111–117, 1972. [27] J. M. F. ten Berge and N. D. Sidiropoulos, “On uniqueness in CANDECOMP/PARAFAC,” Psychometrika, vol. 67, no. 3, Sept. 2002. [28] J. M. F. ten Berge, N. D. Sidiropoulos, and R. Rocci, “Typical rank and INDSCAL dimensionality for symmetric three-way arrays of order 1 2 2 or 1 3 3,” Linear Algebra Applicat., to be published. [29] T. Jiang, N. D. Sidiropoulos, and J. M. F. ten Berge, “Almost sure identifiability of multi-dimensional harmonic retrieval,” IEEE Trans. Signal Process., vol. 49, no. 9, pp. 1849–1859, Sep. 2001. [30] R. Bro, N. D. Sidiropoulos, and G. B. Giannakis, “A fast least squares algorithm for separating trilinear mixtures,” in Proc. Int. Workshop Independent Component Analysis and Blind Signal Separation, Aussois, France, Jan. 1999. [31] J.-F. Cardoso and A. Souloumiac, “Jacobi angles for simultaneous diagonalization,” SIAM J. Matrix Anal. Applicat., vol. 17, pp. 161–164, Jan. 1996. [32] A. Yeredor, “Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation,” IEEE Trans. Signal Process., vol. 50, no. 7, pp. 1545–1553, Jul. 2002. [33] P. Stoica and A. Nehorai, “Performance study of conditional and unconditional direction-of-arrival estimation,” IEEE Trans. Acoust., Speech, Signal Process., vol. 38, no. 10, pp. 1783–1795, Oct. 1990. [34] S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. Englewood Cliffs, NJ: Prentice-Hall, 1993. [35] S. A. Vorobyov, A. B. Gershman, and Z.-Q. Luo, “Robust adaptive beamforming using worst-case performance optimization: A solution to the signal mismatch problem,” IEEE Trans. Signal Process., vol. 51, no. 2, pp. 313–324, Feb. 2003.

2

2 2

2

Yue Rong (S’03) was born in 1976 in Jiangsu, China. In 1999, he received the Bachelor degrees from Shanghai Jiao Tong University, Shanghai, China, both in electrical and computer engineering. He received the M.Sc. degree in computer science and communication engineering from the University of Duisburg-Essen, Duisburg, Germany, in 2002. Currently, he is working toward the Ph.D. degree at the Department of Communication Systems, University of Duisburg-Essen. From April 2001 to April 2002, he was a student research assistant at the Fraunhofer Institute of Microelectronic Circuits and Systems. From October 2001 to March 2002, he was with the Application-Specific Integrated Circuit Design Department, Nokia Ltd., Bochum, Germany. His research interests include signal processing for communications, MIMO communication systems, multicarrier communications, statistical and array signal processing, and parallel factor analysis. Mr. Rong received the Graduate Sponsoring Asia scholarship of DAAD/ABB in 2001.

TLFeBOOK

1710

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Sergiy A. Vorobyov (M’02) was born in Ukraine in 1972. He received the M.S. and Ph.D. degrees in systems and control from Kharkiv National University of Radioelectronics (KNUR), Kharkiv, Ukraine, in 1994 and 1997, respectively. From 1995 to 2000, he was with the Control and Systems Research Laboratory at KNUR, where he became a Senior Research Scientist in 1999. From 1999 to 2001, he was with the Brain Science Institute, RIKEN, Tokyo, Japan, as a Research Scientist. From 2001 to 2003, he was with the Department of Electrical and Computer Engineering, McMaster University, Hamilton, ON, Canada, as a Postdoctoral Fellow. Since 2003, he has been a Research Fellow with the Department of Communication Systems, University of Duisburg-Essen, Duisburg, Germany. He also held short-time visiting appointments at the Institute of Applied Computer Science, Karlsruhe, Germany, and Gerhard-Mercator University, Duisburg. His research interests include control theory, statistical array signal processing, blind source separation, robust adaptive beamforming, and wireless and multicarrier communications. Dr. Vorobyov was a recipient of the 1996–1998 Young Scientist Fellowship of the Ukrainian Cabinet of Ministers, the 1996 and 1997 Young Scientist Research Grants from the George Soros Foundation, and the 1999 DAAD Fellowship (Germany). He co-received the 2004 IEEE Signal Processing Society Best Paper Award.

Alex B. Gershman (M’97–SM’98) received the Diploma (M.Sc.) and Ph.D. degrees in radiophysics from the Nizhny Novgorod State University, Nizhny Novgorod, Russia, in 1984 and 1990, respectively. From 1984 to 1989, he was with the Radiotechnical and Radiophysical Institutes, Nizhny Novgorod. From 1989 to 1997, he was with the Institute of Applied Physics, Russian Academy of Science, Nizhny Novgorod, as a Senior Research Scientist. From the summer of 1994 until the beginning of 1995, he was a Visiting Research Fellow at the Swiss Federal Institute of Technology, Lausanne, Switzerland. From 1995 to 1997, he was Alexander von Humboldt Fellow at Ruhr University, Bochum, Germany. From 1997 to 1999, he was a Research Associate at the Department of Electrical Engineering, Ruhr University. In 1999, he joined the Department of Electrical and Computer Engineering, McMaster University, Hamilton, ON, Canada where he is now a Professor. Currently, he also holds a visiting professorship at the Department of Communication Systems, University of Duisburg-Essen, Duisburg, Germany. His research interests are in the area of signal processing and communications, and include statistical and array signal processing, adaptive beamforming, spatial diversity in wireless communications, multiuser and MIMO communications, parameter estimation and detection, and spectral analysis. He has published over 220 technical papers in these areas. Dr. Gershman was a recipient of the 1993 International Union of Radio Science (URSI) Young Scientist Award, the 1994 Outstanding Young Scientist Presidential Fellowship (Russia), the 1994 Swiss Academy of Engineering Science and Branco Weiss Fellowships (Switzerland), and the 1995–1996 Alexander von Humboldt Fellowship (Germany). He received the 2000 Premier’s Research Excellence Award of Ontario and the 2001 Wolfgang Paul Award from the Alexander von Humboldt Foundation, Germany. He is also a recipient of the 2002 Young Explorers Prize from the Canadian Institute for Advanced Research (CIAR), which has honored Canada’s top 20 researchers 40 years of age or under. He co-received the 2004 IEEE Signal Processing Society Best Paper Award. He is an Associate Editor of the IEEE TRANSACTIONS ON SIGNAL PROCESSING and the EURASIP Journal on Wireless Communications and Networking and a Member of both the Sensor Array and Multichannel Signal Processing (SAM) and Signal Processing Theory and Methods (SPTM) Technical Committees of the IEEE Signal Processing Society. He was Technical Co-Chair of the Third IEEE International Symposium on Signal Processing and Information Technology, Darmstadt, Germany, in December 2003. He is Technical Co-Chair of the Fourth IEEE Workshop on Sensor Array and Multichannel Signal Processing, to be held in Waltham, MA, in July 2006.

Nicholas D. Sidiropoulos (M’92–SM’99) received the Diploma in electrical engineering from the Aristotelian University of Thessaloniki, Thessaloniki, Greece, and the M.S. and Ph.D. degrees in electrical engineering from the University of Maryland, College Park (UMCP), in 1988, 1990 and 1992, respectively. From 1988 to 1992, he was a Fulbright Fellow and a Research Assistant at the Institute for Systems Research (ISR), UMCP. From September 1992 to June 1994, he served his military service as a Lecturer in the Hellenic Air Force Academy. From October 1993 to June 1994, he also was a member of the technical staff, Systems Integration Division, G-Systems Ltd., Athens, Greece. He was a Postdoctoral Fellow (1994 to 1995) and Research Scientist (1996 to 1997) at ISR-UMCP, an Assistant Professor with the Department of Electrical Engineering, University of Virginia, Charlottesville, from 1997 to 1999, and an Associate Professor with the Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, from 2000 to 2002. He is currently a Professor with the Telecommunications Division of the Department of Electronic and Computer Engineering, Technical University of Crete, Chania, Crete, Greece, and Adjunct Professor at the University of Minnesota. His current research interests are primarily in signal processing for communications, and multi-way analysis. He is an active consultant for industry in the areas of frequency hopping systems and signal processing for xDSL modems. Dr. Sidiropoulos is a member of both the Signal Processing for Communications (SPCOM) and Sensor Array and Multichannel Signal Processing (SAM) Technical Committees of the IEEE Signal Processing Society and currently serves as an Associate Editor for the IEEE TRANSACTIONS ON SIGNAL PROCESSING. From 2000 to 2002, he also served as Associate Editor for the IEEE SIGNAL PROCESSING LETTERS. He received the NSF/CAREER award (Signal Processing Systems Program) in June 1998 and an IEEE Signal Processing Society Best Paper Award in 2001.

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1711

Source Localization by Spatially Distributed Electronic Noses for Advection and Diffusion Jörg Matthes, Lutz Gröll, and Hubert B. Keller

Abstract—Based on continuous concentration measurements from spatially distributed electronic noses, the location of a point source is to be determined. It is assumed that the emitted substance is transported by advection caused by a known homogeneous wind field and by isotropic diffusion. A new two-step approach for solving the source localization problem is presented. In the first is determined, on step, for each sensor , the set of points which the source can lie, taking only the specific concentration at sensor into account. In the second step, measurement an estimate for the source position is evaluated by intersecting . The new approach overcomes the problem of poor the sets convergence of iterative algorithms, which try to minimize the least squares output error. Finally, experimental results showing the capability of the new approach are presented. Index Terms—Electronic nose, source localization, spatially distributed sensors.

I. INTRODUCTION ODAY, electronic noses [1]–[11] are applied for the monitoring of process engineering plants [7] and storages for chemicals. In addition to a qualitative measurement of certain air admixtures (classification), electronic noses are increasingly able to give quantitative information, i.e., concentration measurements of the air admixtures. In order to use these concentration measurements for locating the source of an emission (e.g., leakage localization), two main approaches exist: • approaches with mobile sensors [12]–[19] and • approaches with stationary sensors [20]–[25].

T

A. Mobile Sensors Mobile sensors are mostly transported by autonomous robots. From the measured concentrations, the robot decides on the direction to go by a search strategy. In the first step of the preferred strategy, the robot moves in a predefined pattern until an odor hit (concentration above some threshold value) occurs. In the second step, the wind direction is sampled, and the robot moves upwind until the source is reached. The advantage of this approach is that only very rough assumptions on the dispersal behavior are necessary. The basis for the search strategy is the assumption that the concentration is at its maximum at the source position, which is fulfilled for continuous sources. Problems can

Manuscript received August 20, 2003; revised May 10, 2004. This work was supported in part by the “HGF Strategiefondprojekt ELMINA.” The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Jan C. de Munck. The authors are with the Institute for Applied Computer Science, Research Centre Karlsruhe, Karlsruhe, Germany 76021 (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845423

arise if an instantaneous source has to be localized. A disadvantage of this approach is the fact that mobile sensors are often not applicable in industrial environments due to accessibility problems and restrictions in explosion protection areas. B. Stationary Sensors Another approach for locating the source of an emission based on concentration measurements is to use a network of spatially distributed electronic noses. Three groups of methods exist: • Classification methods assume that only a small number of a priori known potential source positions exists. For each potential source, reference signals of the concentraare generated by experiments or simulations for tion all sensors . In the monitoring phase, the source position is classified by comparing the measurements with the reference signals (see [20], [24], and [26]). The large number of necessary simulations or experiments is the main disadvantage of these approaches. • Methods with discrete models use a state-space-model that results from space discretization of the dispersal model. It is assumed that sensors and sources are located at lattice points only. By an observer formulation, the unknown input (source position and rate) is estimated from the measured sensor data [20]. The large order of the state-space system in contrast to the small number of measured states (number of sensors) is a disadvantage of these approaches. • Methods with continuous models [21]–[23], [27] use analytical solutions of the dispersal model and formulate and solve an inverse problem for the unknown parameters source position and rate. Thus, two major challenges arise — Find a dispersal model that, on the one hand, describes the real-world problem sufficiently and, on the other hand, is simple enough to be used for inversion (at least an analytical solution exists). — Find a solution for the inverse problem that is computational robust to errors in the concentration measurements and to deviations between the dispersal model and the real-world problem. In this paper, a method for continuous models is presented. The dispersal behavior is modeled by turbulent diffusion (eddy diffusion) and advection. The analytical solution of the dispersal model results from the diffusion-advection equation in conjunction with simplified assumptions on the initial and boundary conditions, as shown in Section II. The inverse problem for the unknown source parameters, which tends to be ill-posed, is outlined in Section III. The

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1712

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

standard procedure for solving this inverse problem is presented, and it is shown that even in the undisturbed case (the model matches the real world problem exactly, concentration measurements have no errors), the standard procedure can fail. In this paper, a new two-step approach is presented, which overcomes the problems of the standard procedure. The basic idea is to split the problem into the following steps: • For each sensor , determine the set of points on which the source can be located by using only the measurement available from the specific sensor . information • Determine an estimation for the source location by estimating an intersection of all sets . Section IV shows the application of the two-step approach for the advection-free case before the advection case is treated in Section V. After a short simulative analysis of the new approach in Section VI, Section VII gives results of some experiments.

The assumption of a semi-infinite medium with an impermeable surface (the ground) is possible if the source and the sensors are placed away from boundaries. Thus, the diffusion process from the source to the sensors is not influenced significantly by these boundaries in the time of observation. for Taking into account the initial condition and for all , the solution for and arbitrary is given by [28]

erfc

(2)

with the steady solution (3) is the Euclidean distance from

II. DISPERSAL MODEL

to the source (4)

A. Diffusion In the first instance, the dispersal behavior of an emission is characterized by diffusion. The mathematical model for diffusion in a semi-infinite medium, with the source placed on the , caused by a point source with a impermeable surface , is given by the step-function source rate inhomogeneous diffusion equation

(1) , and the source The source position is denoted by start time is . is the isotropic diffusion coefficient Here, . The anisotropic case can be transformed into the isotropic case by an coordinate transformation. Equation (1) describes molecular diffusion with very small difm s for toluene in fusion coefficients (e.g., air at 20 C). In addition to this molecular diffusion, turbulence in the air leads to so-called turbulent diffusion (eddy diffusion). The turbulence is caused by thermal effects, moving of objects, wind, etc. The turbulent diffusion is very complex and is thus hard to model mathematically. However, the diffusion (1) is a reasonable approximation in many cases [22], especially if some averaging of the measured concentrations is applied. The effect of turbulent diffusion is usually much stronger than molecular diffusion, and thus, the turbulent diffusion coefficient is much m s). The larger than for molecular diffusion (up to turbulent diffusion coefficient is almost independent of the diffusing substance but depends highly on the environment [22]. The electronic nose used here works with a internal enrichment unit, which gathers the measured substance over a time of 2 min [3]. The gathered substance is analyzed for 1 min. Thus, the resulting concentration measurements represent averaged values for 3-min intervals. This improves the match of the diffusion (1) with the real-world problem of turbulent diffusion.

B. Advection The dispersal behavior of a substance is, in addition to diffusion, also characterized by advection, if wind is present. In real-world problems, the wind speed and direction is a function of space and time. To obtain an analytical solution for the dispersal model, which can be used for source localization, the simplified assumption of a homogeneous wind field is necessary. If this assumption is significantly violated for the specific problem under consideration, source localization is not at all possible with continuous model methods. Here, however, it is assumed that the homogeneous wind field assumption holds approximately • for the time interval from the start of emission to the end of the localization and • for the particular area, covering the source and those sensors that measure significant changes in concentration (only these sensors are used for the localization). By a coordinate transformation, the homogeneous wind field can be modeled as wind in the -direction with the speed for any wind direction. In that case, the behavior of the concentration at any position , and at any time is described by the inhomogeneous diffusion-advection equation

(5) With the initial condition and arbitrary the solution for

TLFeBOOK

for and for all , is given by [28]

(6)

MATTHES et al.: SOURCE LOCALIZATION BY SPATIALLY DISTRIBUTED ELECTRONIC NOSES FOR ADVECTION AND DIFFUSION

1713

Fig. 2. Logarithmic contour plot of the squared output error J dependent on the source position x . White colored areas represent large values of J (maxima at the sensor positions), and dark areas represent small values of J (global minimum at the source position and valleys).

Fig. 1.

For large

In the experiments described in this paper, the sources are ideally seen to be instantaneous. Thus, integrated concentrations over time are used. The steady value of these integrated concentrations represents the amount of substance that passed sensor .

Sensor and source positions. Simulated sensor signals.

III. STANDARD PROCEDURE FOR SOLVING THE INVERSE SOURCE LOCALIZATION PROBLEM

, a steady concentration profile (7)

is established [28], [29]. In this paper, the sensors are also placed on the impermeable , but the presented two-step approach is not limited surface to this two-dimensional (2-D) case. The sensor positions are , . A simulation example (see Fig. 1) shows that steady concentrations are reached after a short time, although a small wind m/s ( m s, mg/s) was velocity leads to an even faster convergence chosen. An increase of of . Therefore, an approach for estimating the source position and the source rate , based on steady concentrations (7), is chosen. For the source localization, it is assumed that the dispersal and are identified by prior experiments. It is parameters with anemometers. A direct meaalso possible to measure surement of is not possible. Theoretically, the dependency of from the wind speed can be used to estimate based on the anemometer measurements, but practically, it is hard to find . a function

and

The standard procedure to estimate

is to minimize (8)

minimize

is the calculated steady concentration for sensor acis the measured value. For solving this least cording to (7). squares problem, usually, gradient-based methods are applied. Depending on the choice of the unknown initial values, these methods can fail due to local minima or poor convergence. In order to show this problem, the objective function (8) is in the noise-free case, where is fixed at plotted over , its true value. The sensor positions and the source location are equal to Fig. 1, and Fig. 2 shows a minimum at the true source m m . Additionally, at each sensor position, position local maxima arise. These cause narrow valleys in the topology of the objective function shown in Fig. 2, understated by the logarithmic representation. These valleys make the search for the minimum with gradient-based methods difficult [e.g., presume m m ]. The plot becomes even worse an initial value with regard to the iteration, if also has to be estimated and if more sensors are applied.

C. Instantaneous Sources , the inteIf the source is instantaneous grated concentrations over time fulfill (6) and (7), respectively, and are equal. This results from if the absolute values of the linearity of the diffusion-advection equation. Only the didiffer ( kg/s, kg). mensions of and

IV. ADVECTION-FREE CASE A. Determining In the advection-free case, the set

TLFeBOOK

is given by (9)

1714

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 3. Potential source positions from the view of sensor i for different source rates q in the advection-free case.

Fig. 4. Intersection of all m sets P at the source position x for the undisturbed, advection-free case and the true values of q and K ( ). If q is not the true value, the circles (dashed) do not intersect in a single point.

is an estimate for the sensor-source-distance depending on the measured concentration and the unknown parameters and . Regarding (3), can be expressed as [30]

The intersection point is then an estimate for the source position [30]. To find this intersection point, the problem

(10) is a so-called scalable sensor-source-distance, which can be estimated for each sensor based on the measured concentra. is a global scaling factor covering the unknown tions parameters and . Obviously, for a fixed , each set rep. This resents a circle with the center point and the radius becomes clear if (9) is written as an implicit circle equation

has to be solved. In the undisturbed case, the sum in (12) becomes 0. Equation (12) can be reformulated with .. .

.. .

(11) Fig. 3 shows the potential source positions from the view of and fixed sensor (the set ) for different source rates and . Fig. 3 makes clear that for a certain concentration measured at sensor , the source can either be located near the sensor (assuming a small source rate ) or can be located further away from the sensor (assuming a larger ). B. Estimating the Source Position In the undisturbed case, all sets (all circles) intersect in m m , if and are at their true a single point values and if the sensors are not placed in a line (see Fig. 4). If or are not at their true values, the circles (dashed in Fig. 4) do not intersect in a single point. In the disturbed case, the estimation of is error prone. Then, even if and are at their true values, the sets do not intersect in a single point. In both cases, in the second step, an intersection point or an has to be determined approximated intersection point of all after estimating for each sensor in the first step. This is achieved by varying the global scaling factor . That means that circles with the center and the radius are all scaled until they intersect in a single point (undisturbed case) or intersect approximately in a single point (disturbed case).

(12)

minimize

.. .

.. .

(13)

into minimize

s.t.

(14)

This quadratically constrained least squares problem can be solved by an algorithm introduced in [31]. This algorithm takes approximately 0.02 s computation time on a standard pc sensors). (1 GHz, Matlab for the example of V. ADVECTION CASE A. Determining In the advection case, it is assumed that is identified by prior experiments, and is either measured online or also identified in advance. Each set is given by (15) forms an oval in the -direction. Here, for fixed and , Its form and size is determined by the nonlinear function

TLFeBOOK

(16)

MATTHES et al.: SOURCE LOCALIZATION BY SPATIALLY DISTRIBUTED ELECTRONIC NOSES FOR ADVECTION AND DIFFUSION

1715

Fig. 5. Potential source positions from the view of sensor i for different source rates q .

where is the Lambert function. Equation (16) is obtained into (7) and solving for . by inserting and In contrast to the advection-free case, in the advection case, cannot be separated into a constant scalable sensor-source-distance and a global scaling factor . Here, with the measured at sensor , only a function steady concentration can be generated for each sensor in the first step of the two-step approach. Thus, (15) can be written similarly to the implicit circle (11)

Fig. 6. Intersection of all m sets P at the source position undisturbed case and the true value of q .

x

for the

(17) Equation (17) has the same structure as the implicit circle equation, but here, the radius is not constant if is fixed, but it is a function of . This leads to ovals instead of circles (see Fig. 5). By varying , all potential source positions from the view of sensor can be calculated for a fixed source rate : (18) For each ,

, and

, two solutions are obtained with (18).

B. Estimating the Source Position Ovals can be calculated for all sensors (see Fig. 5). In the ovals intersect at the source position undisturbed case, all m m if the source rate used to generate the ovals equals the true value (see Fig. 6). are disturbed, If the measurements of the concentration is error prone. Thus, the the estimation of the steady values estimation of the sensor-source-distance contains errors. As a result, the corresponding ovals do not intersect. In both cases, an estimate for the source position can be found by formulating and solving an appropriate optimization and , for which the sum problem. Here, the idea is to find of squared distances (in the -direction) between all ovals is at its minimum: minimize (19) in dependence of for the undisFig. 7 shows turbed case with two sensors. Here, is at its true value. Vertical lines in the upper part of Fig. 7 illustrate the distance . In the -regions left and right of an oval, the sensor posi. This is reasonable because the tion is used for calculating

Fig. 7.

E

(x

;q

) for the case of two sensors with true source rate

q

.

error and, thus, the influence of a single sensor is limited to a maximal value. , the Fig. 7 makes it clear that at the source -position is zero. For the case of two senobjective function sors, generally, two intersection points of the ovals exist. These , even if is not at its true value. This lead to is clear as it is impossible to estimate the source position (two parameters in the 2-D case) and the source rate (another parameter) with only two sensors. with If all four sensors are regarded, the plot of respect to improves (see Fig. 8). For the true value, mg/s reaches its minimum at . The contour with respect to and in Fig. 9 shows that plot of now, an iterative optimization over and [minimizing (19)] is unproblematic (less than 3 s computation time). In contrast to the standard procedure (cp. Section III), here, only two paramand (not three parameters , , and ) have to be eters

TLFeBOOK

1716

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 8. E (x ; q ) for the case of four sensors with true source rate q = 1 and wrong source rates q = 2, 3, and 4.

Fig. 10. Mean value and standard deviation of the localization error if varied and v -direction, jv j, and C are disturbed with Gaussian noise.

K

is

Fig. 9. Logarithmic contour plot of E (x ; q ) for the case of four sensors with the minimum for the true values x = 6 m and q = 1 mg/s.

optimized iteratively. An increase in the number of sensors does not complicate the optimization. are estimated, an estimate for can be calOnce and used to calculate culated from the average of all in (19). Fig. 11. Mean value and standard deviation of the localization error if the v -direction is varied, and K , jv j, and C are disturbed with Gaussian noise.

VI. SIMULATION For the model-based source localization, it was assumed that the diffusion coefficient was identified by prior identification and wind direction are known by and that the wind speed measurements. With simulations (based on the sensor source configuration shown in Fig. 1), it is analyzed what influence an and in the measurement of error in the identification of has on the quality of the source localization with the proposed two-step approach. Additionally, the influence of errors in the concentration measurement is analyzed. is varied from to 30%, First, the relative error of whereas the direction of is disturbed by Gaussian noise with . The wind speed and the concentrations are disof their nominal values. turbed by Gaussian noise with For each relative error of , 100 simulations are performed. Fig. 10 shows the mean of the distances between the true and estimated source position that is dependent on the relative error of . Additionally, the standard deviation of this distance is illustrated by error bars. Fig. 11 shows the localization error if the direction of is varied from to 10 and , , and are disturbed with of their nominal values. Analoand are varied, gously, Figs. 12 and 13 show the results if respectively. To get a realistic analysis, the relative error for

was chosen contrarily to the error of and . These and simulative investigations reveal that an error in the measurement or identification of the wind direction has an important influence on the quality of the source localization (see Fig. 11). VII. EXPERIMENTAL RESULTS For chemical storage of toxic substances at the SigmaAldrich Company in Steinheim, Germany, a network of four electronic noses of the SAMONA type (developed at the Research Centre Karlsruhe) was established. The SAMONA system uses a metal oxide gradient microarray (see [6]) as well as a surface acoustic wave micro array (see [9]) with an internal enrichment unit. After a classification of the substance, a concentration calculation is performed by SAMONA. The sensors were placed as shown in Fig. 14 at a height of 0.9 m. The high-rack facilities (marked in grey in Fig. 14) are permeable and do not disturb the diffusion and advection because they are wide open. and the advection velocity The diffusion coefficient were identified by two prior experiments (P1, P2):

TLFeBOOK

MATTHES et al.: SOURCE LOCALIZATION BY SPATIALLY DISTRIBUTED ELECTRONIC NOSES FOR ADVECTION AND DIFFUSION

Fig. 12. Mean value and standard deviation of the localization error if jv varied, and K , v -direction, and C are disturbed with Gaussian noise.

j

1717

is

Fig. 14. Sensor network of four electronic noses of the SAMONA type for chemical storage at the Sigma-Aldrich Company (Germany).

Fig. 13. Mean value and standard deviation of the localization error if varied, and K , v -direction, and jv j are disturbed with Gaussian noise.

C

is

m s, m/s. By an offline optimizawere determined, which lead to tion, those values of and the best fit of simulated (source position known) and measured concentrations over time. In the first experiment, 10 ml of isopropanol are poured onto filter paper (located at a box with a height of 0.4 m) instantaneously at source position 1. In the second experiment, the same amount of toluene is used at source position 2. The four electronic noses measure the concentration of isopropanol and toluene with a sampling time of 3 min. To apply the proposed two-step approach for instantaneous sources, the concentrations have to be integrated over time (cp. Section II-C). Fig. 15 shows measured concentrations of all four electronic noses integrated over time for the two source positions illustrated in Fig. 14. Fig. 16 shows the results of the source localization for the two experiments. Here, the coordinate system is rotated to have advection in the -direction only. The estimated source positions are near the true source positions for both experiments. In experiment 1, the concentration measured at sensor 3 was higher than expected. Thus, the oval of sensor 3 is too small. Nevertheless, by regarding all four sensors, a good estimation of is

Fig. 15. Measured concentrations of isopropanol and toluene, respectively, for both experiments integrated over time.

obtained. In experiment 2, all four ovals intersect near the true source position. Table I gives a summary of the experimental results. The results show that a source localization is possible, if the wind

TLFeBOOK

1718

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE I SUMMARY OF THE EXPERIMENTAL RESULTS

Fig. 16.

True and estimated source positions for the two experiments.

direction and speed are identified correctly (exp. 3–5). In the is necescase of changing wind, another identification of sary to obtain good results (exp. 6–8). Alternatively, an online measurement of the wind speed and direction is possible. If the dispersal behavior is disturbed significantly (opened windows/doors, movement of people/vehicles, time variant air conditioning systems), a model-based source localization leads to bad results (exp. 9–10).

function of the one-step approach gets worse with respect to iterative solvers if the number of sensors increases. Motivated by these problems, a new two-step approach has been developed. In the first step, the set of points on which the source can be located is determined for each sensor based on its concentration measurements. These sets form scalable circles in the advection-free case and ovals in the advection case. The size of the sets depends on the source rate. In the second step, an intersection point of all circles/ovals can be found by varying the source rate. The intersection point is an estimate for the source position. In the disturbed case, generally, no intersection point exists, but an approximation of this point can be found by solving an optimization problem. A simulation example shows that the approach is robust to errors in the identification or measurement of the wind speed, the wind direction, and the diffusion coefficient. Finally, the two-step approach is demonstrated by means of practical experiments for industrial storage of toxic chemicals. The concentration measurements are performed by electronic noses known as SAMONA, which were developed at the Research Centre Karlsruhe. The experiments show that a source localization with the two-step approach gives good results if the wind speed and direction are known. In the case of changing wind, an online measurement of the wind is necessary. Some experiments show that the dispersal behavior can be disturbed significantly by unpredictable influences, which cannot be covered by a diffusion-advection model. In such cases, a model-based source localization will fail, regardless of whether the one- or two-step approach is applied. ACKNOWLEDGMENT

VIII. CONCLUSION In this paper, a new approach for estimating the position of an emission source, based on spatially distributed concentration measurements, is presented. It is assumed that the emitted substance is transported by homogeneous advection and isotropic diffusion. The anisotropic case can be converted into the isotropic case by a coordinate transformation. The source rate is assumed to be a step function. In the case of intantaneous source rates, the concentrations integrated with respect to time have to be regarded. Problems of one-step approaches are illustrated based on an undisturbed simulation example. It is shown that the objective

The authors would like to thank C. Arnold, M. Dirschka, V. Hartmann, and K.-H. Lubert as well as Sigma-Aldrich, Steinheim, Germany, for supporting the experimental measurements. They also thank the reviewers, whose advice has improved the paper considerably. REFERENCES [1] P. Althainz, J. Goschnick, S. Ehrmann, and H. Ache, “Multisensor microsystem for contaminants in air,” Sensors Actuators B, vol. 33, pp. 72–76, 1996. [2] C. Arnold, M. Harms, and J. Goschnick, “Air quality monitoring and fire detection with the Karlsruhe Electronic Micronose KAMINA,” IEEE Sensors J., vol. 2, no. 3, pp. 179–188, Jun. 2002.

TLFeBOOK

MATTHES et al.: SOURCE LOCALIZATION BY SPATIALLY DISTRIBUTED ELECTRONIC NOSES FOR ADVECTION AND DIFFUSION

[3] F. Bender, N. Barié, G. Romoudis, A. Voigt, and M. Rapp, “Developement of a preconcentration unit for a SAW sensor micro array and its use for indoor air quality monitoring,” Sensors Actuators B, vol. 7076, 2003, to be published. [4] J. W. Gardner and P. N. Bartlett, Eds., Sensors and Sensory Systems for an Electronic Nose. ser. International Series of Numerical Mathematics. Dordrecht, The Netherlands: Kluwer, 1992. [5] J. W. Gardner, An Introduction to Electronic Nose Technology. Stanstead, U.K.: Neotronics Scientific, 1996. [6] J. Goschnick, “An electronic nose for intelligent consumer products based on a gas analytical gradient microarray,” Microelectron. Eng., vol. 57–58, pp. 693–704, 2001. [7] A. Jerger, H. Kohler, F. Becker, H. B. Keller, and R. Seifert, “New applications of tin oxide gas sensors II. Intelligent sensor system for reliable monitoring of ammonia leakages,” Sensors Actuators B, no. 81, pp. 301–307, 2002. [8] D.-D. Lee and D.-S. Lee, “Environmental gas sensors,” IEEE Sensors J., vol. 1, no. 3, pp. 214–224, Oct. 2001. [9] M. Rapp, F. Bender, and A. Voigt, “A novel SAW micro array concept for environmental organic gas detection at low concentrations,” in Proc. First IEEE Int. Conf. Sensors, Orlando, FL, Jun. 12–14, 2002. [10] J. Reibel, U. Stahl, T. Wessa, and M. Rapp, “Gas analysis with SAW sensor systems,” Sensors Actuators B, vol. 65, no. 1–3, pp. 173–175, 2000. [11] B. A. Snopok and I. V. Kruglenko, “Multisensor systems for chemical analysis: State-of-the-art in electronic nose technology and new trends in machine olfaction,” Thin Film Solids, no. 418, pp. 21–41, 2002. [12] T. Duckett, M. Axelsson, and A. Saffiotti, “Learning to locate an odour source with a mobile robot,” in Proc. IEEE Int. Conf. Robotics Automat., Seoul, Korea, 2001, pp. 4017–4021. [13] A. T. Hayes, A. Martinoli, and R. M. Goodman, “Distributed odor source localization,” IEEE Sensors J., vol. 2, no. 3, pp. 260–271, Jun. 2002. [14] H. Ishida, Y. Kagawa, T. Nakamoto, and T. Moriizumi, “Odor source localization in the clean room by an autonomous mobile sensing system,” Sensors Actuators B, vol. 33, pp. 115–121, 1996. [15] H. Ishida, T. Nakamoto, and T. Moriizumi, “Remote sensing of gas/odor source location and concentration distribution using mobile system,” Sensors Actuators B, vol. 49, no. 1–2, pp. 52–57, 1998. [16] H. Ishida, T. Nakamoto, T. Moriizumi, T. Kikas, and J. Janata, “Plumetracking robots: A new application of chemical sensors,” Biol. Bull., vol. 200, pp. 222–226, 2001. [17] A. Lilienthal, A. Zell, M. Wandel, and U. Weimar, “Sensing odour sources in indoor environments without a constant airflow by a mobile robot,” in Proc. IEEE Int. Conf. Robotics Automat., Seoul, Korea, 2000, pp. 4005–4009. [18] R. A. Russel, D. Thiel, R. Deveza, and A. Mackay-Sim, “A robotic system to locate hazardous chemical leaks,” in Proc. IEEE Int. Conf. Robotics Automat., Nagoya, Japan, 1995, pp. 556–561. [19] R. A. Russel, Odor Detection by Mobile Robots, Singapore: World Scientific, 1999. [20] M. E. Alpay and M. H. Shor, “Model-based solution techniques for the source localization problem,” IEEE Trans. Control Syst. Technol., vol. 8, no. 6, pp. 893–902, Nov. 2000. [21] A. Y. Khapalov, “Localization of unknown sources for parabolic systems on the basis of available observations,” Int. J. Syst. Sci., vol. 25, no. 8, pp. 1305–1322, 1994. [22] A. Nehorai, B. Porat, and E. Paldi, “Detection and localization of vapor-emitting sources,” IEEE Trans. Signal Process., vol. 43, no. 1, pp. 243–253, Jan. 1995. [23] Y. Nievergelt, “Solution to an inverse problem in diffusion,” SIAM Rev., vol. 40, no. 1, pp. 74–80, 1998. [24] E. Wacholder, E. Elias, and Y. Merlis, “Artificial neural networks optimization method for radioactive source localization,” Nuclear Technol., vol. 110, pp. 228–237, 1995.

1719

[25] H. Nambo, H. Kimura, and T. Oyabu, “Estimation of gas generation point using delay of gas sensor responses,” IEEJ Trans. Sensors Micromachines, vol. 122, no. 10, pp. 480–486, 2002. [26] J. Matthes, “Lokalisierung von emissionsquellen mit ortsfesten, räumlich verteilten elektronischen Nasen,” in Proc. GMA-Kongress, BadenBaden, Germany, 2003, pp. 495–502. [27] A. Jeremic´ and A. Nehorai, “Landmine detection and localization using chemical sensor array processing,” IEEE Trans. Signal Process., vol. 48, no. 5, pp. 1295–1305, May 2000. [28] H. S. Carslaw and J. C. Jaeger, Conduction of Heat in Solids. Oxford, U.K.: Clarendon, 1959. [29] O. F. T. Roberts, “The theoretical scattering of smoke in a turbulent atmosphere,” in Proc. R. Soc. Lond., vol. 104, 1923, pp. 640–654. [30] J. Matthes and L. Gröll, “Model based source localization by distributed sensors for point sources and diffusion,” in Proc. 13th IFAC Symp. Syst. Ident., Rotterdam, The Netherlands, 2003, pp. 319–324. [31] L. Gröll, “Least squares with a single quadratic constrained,” Automatisierungstechnik, vol. 52, no. 1, pp. 48–55, 2004.

Jörg Matthes received the Dipl.-Ing. degree in mechatronics from the University of Freiberg, Freiberg, Germany, in 1999 and the Ph.D. degree from the University of Karlsruhe, Karlsruhe, Germany. His research interests are in data analysis in sensor networks and source localization with electronic noses.

Lutz Gröll received the Dipl.-Ing. degree and the Ph.D. degree in electrical engineering from the University of Dresden, Dresden, Germany, in 1989 and 1995, respectively. Since 2001, he has been head of the research group “Process Modeling and Control Engineering” at the Institute for Applied Computer Science, Research Centre Karlsruhe, Karlsruhe, Germany. His research interests are in parameter identification, nonlinear control, and optimization theory.

Hubert B. Keller received the Dipl.-Ing. degree in 1982 from the University of Karlsruhe, Karlsruhe, Germany, and the Ph.D. degree in 1988 from the University of Clausthal, Clausthal, Germany. He is head of the research groups “Innovative Process Control” and “Intelligent Sensor Systems” at the Institute for Applied Computer Science, Research Centre Karlsruhe. Until 1984, he was a Development Engineer at Siemens (real-time computer systems). His research interests are real-time systems, software engineering, machine intelligence, intelligent sensor, and process control systems.

TLFeBOOK

1720

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Damped and Delayed Sinusoidal Model for Transient Signals Rémy Boyer and Karim Abed-Meraim, Member, IEEE

Abstract—In this work, we present the Damped and Delayed Sinusoidal (DDS) model: a generalization of the sinusoidal model. This model takes into account an angular frequency, a damping factor, a phase, an amplitude, and a time-delay parameter for each component. Two algorithms are introduced for the DDS parameter estimation using a subband processing approach. Finally, we derive the Cramér–Rao Bound (CRB) expression for the DDS model and a simulation-based performance analysis in the context of a noisy fast time-varying synthetic signal and in the audio transient signal modeling context.

and the proposed estimation algorithms. In Section VII, simulation results are given, and Section VIII is dedicated to the final conclusions.

Index Terms—Cramér–Rao Bound, damped and delayed sinusoids, deflation, Fourier analysis, subband parameter estimation, transient signal.

(1)

I. INTRODUCTION ARAMETRIC models, such as the constant-amplitude sinusoidal or Exponentially Damped Sinusoidal (EDS) models are popular and efficient tools in many areas of interest including spectral-line [24] or pole estimation [15], source localization [22], biomedical signal processing [25], and audio signal compression [2], [12], [20]. In this paper, we introduce a generalization of these models, called the Damped and Delayed Sinusoidal (DDS) model, which adds a time-delay parameter to allow time-shifting of each component waveform. Note that this paper goes further into the work initiated in [3]. Properties of this model are studied, and we show that it can achieve compact representations of fast time-varying or “transient” signals. This paper also addresses the problem of the DDS model parameter estimation. Two model parameter estimation algorithms are derived, and their performances are compared on a noisy synthetic signal and on a typical audio transient signal. The paper is organized as follows: Section II introduces the DDS model. An overview of the problems and the proposed solutions is presented in Section III. In Section IV, two algorithms, called DDS-B (B stands for Block), and DDS-D (D stands for Deflation), are presented for the estimation of the DDS signal parameters. Section V presents the derivation of the Cramér–Rao Bound (CRB) for the estimation of the DDS parameters in the presence of additive white Gaussian noise. Section VI provides additional comments about the DDS model

P

Manuscript received November 20, 2002; revised May 6, 2004. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Arnab K. Shaw. R. Boyer is with the University of Paris XI-Orsay and L2S-Supélec (Signals and System Lab.), Gif-Sur-Yvette, France (e-mail: [email protected]). K. Abed-Meraim is with the GET-ENST, Paris Cedex 13, France (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845473

II. DDS MODEL A. Parametric Model Definitions The complex

-EDS model definition is given by

where is the number of complex sinusoids or the modeling order, and are the real amplitude, phase, damping factor, and angular frequency parameters. for all , we obtain the comNote that if we choose plex sinusoidal model. The -DDS model can be understood as a generalization of the previous parametric model. Its expression [3], [9] is given by (2) where we have introduced the discrete-valued time-delay paand the Heaviside function defined by rameters for and 0 otherwise. Note that the complex -DDS model is formally similar to the -EDS model of expression (1) by supposing that the amplitude varies with time according to (3) with and . Real formulation of the previous complex -DDS model can be written in terms of the complex amplitude and the pole , according to (4) is a real where 1-DDS component. In Fig. 1, different 1-DDS waveforms are presented. B. Damping Factor Sign and Compact Representation of fiIn this paper, we consider the real discrete space nite energy signals. The 1-EDS and 1-DDS signals for belong to this space. However, sinusoidal components with are required to model signals with strong onset when

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

BOYER AND ABED-MERAIM: DAMPED AND DELAYED SINUSOIDAL MODEL FOR TRANSIENT SIGNALS

d
3.

TLFeBOOK

1864

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 3. Magnitude of the triplet-based estimator.

Fig. 4.

small. As far as the phase of (14) is concerned, it is given by of the observed noisy symbol so correcting the phase that the phase differences measured after the estimation, namely , result more close to those pertaining to the more likely triplet. We must also outline that (14) can be further simplified when some admissible triplets differ only by a phase rotation; in result in equal confact, all the indices such that tribution in the summations. B. Graphical Discussion The magnitude- and the phase-correction terms of the nonlinear estimator (14) are illustrated in Figs. 3 and 4 for the reference case described in the following. The nonlinearity is computed in the case of correlated input symbols generated according to (2), at SER dB. For this generation model, the admissible phase jumps between consecutive symbols . In Figs. 3 and 4, the magnitudeand are only are, respectively, plotted the phase-correction term and , for versus the phase differences . The magnitude of the nonlinear estimator achieves its maximum value when the phase differences equal that of an admis. sible triplet, i.e., when When the observed phase differences are far from the admissible jumps , the magnitude decreases. is applied when the Analogously, no phase correction . When phase jumps are close to the admissible values of the observed phase differences differ from those of an admisor sible triplet, i.e., when , the phase correction may assume the maximum , thus restoring two consecutive phase jumps as value of close as possible to . No phase correction is applied also for phase jumps such that . The sensitivity of this phase correction, however, significantly varies. It is kept because very low for

Phase correction of the triplet-based estimator.

the observations are close to an admissible triplet. On the other hand, the sensitivity becomes very high for because both the phase corrections are equally likely. An example of this situation is further discussed in Fig. 5(d). C. Estimate Examples The overall (magnitude and phase) correction operated by the Bayesian estimator is represented in Fig. 5 for the same reference case of Figs. 3 and 4. In Fig. 5, the linearly (triangle down), the two adjacent symestimated symbol (square and circle), and the nonlinear estimate bols (empty triangle up) are depicted. On the basis of the observation , the nonlinear estimator attempts of the phases of to restore the “more likely” triplet by suitably correcting the symbol . If the observed phase difference pairs are “admissible”, as in Fig. 5(a), the correction term is very small, and the triplet-based estimator behaves like a symbol-by-symbol estimator, i.e., it tries to correct the observed symbol toward the closest point of the source alphabet. When the phase differences are not admissible, as in Fig. 5(b) and (d), the phase correction achieves the maximum value of , and the triplet-based estimator performs substantially better than a symbol-by-symbol estimator, i.e., it does not correct toward the closest point of the source alphabet but toward the closest “admissible” point. In Fig. 5(c), even though the phase differences are not admissible, the estimator corrects only the magnitude since in this very particular case, it cannot choose between equally likely phase corrections. D. High SER Behavior In high SER, the estimator behavior can be analyzed by resorting to the asymptotic expansion of the Bessel functions for . Adopting this

TLFeBOOK

PANCI et al.: BLIND EQUALIZATION FOR CORRELATED INPUT SYMBOLS

1865

(15) For increasing SER, the relative weight of the term corresponding to increases, and for SER , only the term survives in (15) so that the estimator corresponding to becomes

Fig. 5. Nonlinear estimate (empty triangle up) corresponding to different , s^ , and s^ , depicted by a square, triangle down, triplet of symbols s^ and circle, respectively.

approximation, and multiplying numerator and denominator by , yields

Hence, for increasing SER, the nonlinear estimator acts as a (magnitude) hard-detector with memory since it returns a comwith magnitude of the cenplex value and with tral symbol of the more “likely” triplet indexed by of the central linearly equalized symbol . phase The transition of the estimator behavior from a soft detector with memory for low SER to a hard detector with memory in high SER is driven by the power of the residual equalization . A coarse estimate of the power of the error residual equalization error is given by , and the estimator behavior can be controlled during the iterations.

V. EXPERIMENTAL RESULTS

Let us compactly represent the exponents, which are functions of and , as follows:

The term represents a distance measure between the estimated triplet and the admissible triplet , where both are normalized in amplitude. From the Cauchy-Schwartz inequality , we observe that

Now, let be the index for which the term is minimum, that is, the index of the more likely triplet; then, we can write the estimator as follows:

Performance of the blind Bussgang equalization presented in the previous sections is shown here, with reference to different transmission schemes. The first scheme refers to a scenario where correlated QPSK input symbols are generated according to (2). The symbols are observed after two different channels. The first channel is the SPIB1 channel, which is taken from the SPIB database at http://spib.rice.edu. The impulse response and the square magnitude of the frequency response are, respectively, shown in Figs. 6 and 7. The second channel is a 28 (complex) coefficient channel representing a multipath urban microwave link channel with severe frequency-selective fading [25], which will be referenced to as an URBAN channel in the following. The impulse response and the square magnitude of the frequency response of the URBAN channel are shown in Figs. 8 and 9. Different equalizer lengths have been considered. For each equalizer length, the normal equations (5) have been solved using sample statistics calculated from samples picked up at the channel output. In Figs. 10 and 11, the MSE measured at the output of the triplet-based Bussgang (TB) equalization algorithm is plotted versus the number measured at the of iterations, for values of the SNR output of the channel SPIB1 equal to 15 and 25 dB, respectively. Each MSE value is obtained by averaging over 50 Monte Carlo runs. For comparison, in addition, the results achieved by the Super Exponential (SE) algorithm [26] and by a classical

TLFeBOOK

1866

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 6. Impulse response of the SPIB1 channel.

Fig. 7.

Fig. 8. Impulse response of the URBAN channel [25].

Square magnitude of the frequency response of the SPIB1 channel.

symbol-based Bussgang (SB) equalizer [19] are plotted. This latter is conducted using the symbol-by-symbol estimator

(16)

that exploits only the marginal symbol constellation pdf

The estimator (16) has been derived according to the model (10), which properly takes into account a residual unknown phase rotation. As a matter of fact, it is a specialization of the estimator . (11) to the one-dimensional case Finally, we report also the performance bound obtained by a trained equalizer, i.e., an equalizer obtained by solving the

Fig. 9. Square magnitude of the frequency response of the URBAN channel [25].

normal equations (5) using sample statistics that make use of the “true” transmitted symbols. On the SPIB1 channel, all the equalization algorithms show dB and SNR dB, good performances for both SNR but the triplet-based equalizer is more stable and effective. In Figs. 12 and 13, the MSE achieved by the three equaldB izers on the URBAN channel are plotted for SNR dB, respectively. In both cases, the novel tripletand SNR based Bussgang equalizer presents a significant MSE reduction with respect to both the SE and the SB equalizers, even for very , 16. In addition, convergence short equalizers of length is attained very fast. The second simulation scheme refers to a GMSK modulated signal observed after the URBAN channel. As shown in [12], this nonlinear modulation scheme can be well approximated by a linear QPSK modulation of a sequence of correlated symbols generated according to (2). This model is exploited to design

TLFeBOOK

PANCI et al.: BLIND EQUALIZATION FOR CORRELATED INPUT SYMBOLS

1867

Fig. 10. MSE measured at the Bussgang equalizer output versus the iteration number i. Correlated QPSK input symbols as in (2), SNR = 15 dB, SPIB1 channel, 300 observed samples, equalizer length 2L = 12, 16, 32. (TB solid line, SB dashed line, SE dotted line, Trained Equalizer dash-dot line).

Fig. 12. MSE measured at the Bussgang equalizer output versus the iteration number i. Correlated QPSK input symbols as in (2), SNR = 15 dB, URBAN channel, 300 observed samples, equalizer length 2L = 12, 16, 32. (TB solid line, SB dashed line, SE dotted line, Trained Equalizer dash-dot line).

Fig. 11. MSE measured at the Bussgang equalizer output versus the iteration number i. Correlated QPSK input symbols as in (2), SNR = 25 dB, SPIB1 channel, 300 observed samples, equalizer length 2L = 12, 16, 32. (TB solid line, SB dashed line, SE dotted line, Trained Equalizer dash-dot line).

Fig. 13. MSE measured at the Bussgang equalizer output versus the iteration number i. Correlated QPSK input symbols as in (2), SNR = 25 dB, URBAN channel, 300 observed samples, equalizer length 2L = 12, 16, 32. (TB solid line, SB dashed line, SE dotted line, Trained Equalizer dash-dot line).

the Bussgang nonlinearity. The linearly estimated values at the equalizer output are interpreted as a sequence of soft estimates of QPSK symbols, which are associated with a sequence of soft estimates of the transmitted binary symbols by simply inverting . the relation (2), i.e., The performance obtained on Typical Urban (TU) COST 207 channel (see [27]) with maximum doppler frequency Hz, at SNR dB, is shown in Fig. 14, where the MSE measured on the softly estimated binary symbols is plotted versus the number of iterations for different equalizer lengths. Simulation results are obtained averaging over 1000

independent realizations of the channel. In the same figure, the bit error probability (BER) achieved substituting the soft estimates with the corresponding detected binary symbols is also shown. The curve of the BER is calculated under the hypothesis of Gaussian error distribution. Even though this approximation is too optimistic for low MSE and BER (at least one order of magnitude), it allows us to compare the performance of different algorithms. We observe that the triplet-based Bussgang equalizer converges to an MSE value significantly lower than that achieved by the SE and by the SB equalizers of the same length.

TLFeBOOK

1868

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 15.

Fig. 14. MSE measured at the Bussgang equalizer output versus the number of iterations k . GMSK signal, TU-COST207 channel (1000 independent realizations) with maximum doppler frequency f = 50 Hz, SNR = 15 dB, 300 observed samples, equalizer length 2L = 16, 20, 32. (TB solid line, SB dashed line, SE dotted line, Trained Equalizer dash-dot line).

VI. CONCLUSION In this paper, we have shown how the input symbol sequence probability distribution can be directly exploited in a blind Bussgang equalization scheme. We have obtained the analytical expression of the nonlinear (Bayesian) estimator accounting for the memory underlying the sequence of transmitted symbols. The nonlinearity generates a soft estimate of the transmitted symbol using the observation of three signal samples, namely, the actual sample at time and the two adjacent samples at times and . The computational complexity of the algorithm for correlated input symbols remains basically unchanged with respect to the classical Bussgang algorithm. Numerical results pertaining to both a QPSK and a GMSK modulation scheme show that significant performance improvement is gained, both in precision and convergence speed. APPENDIX A Here, we show that the output of the Wiener filter is a sufficient statistic for the estimation of the input symbols . With reference to Fig. 1, it is not difficult to show that the frequency response of the IIR Wiener filter is given by (see also [28] and [29]):

where is the Power Spectral Density (PSD) of the additive noise , is the overall denotes channel plus shaping filter frequency response, and complex conjugation. can be As depicted in Fig. 15, let us observe that decomposed in the cascade of the following two filters:

On the statistical sufficiency of the Wiener filter output.

•

the filter matched to the overall channel seen by the input symbols

•

a periodic modulo

filter

The input–output equivalence of Fig. 15(a) and (b) is . due to the periodicity modulo of Now, from Fig. 15(b), we observe that the Wiener filter output is one-to-one with the matched filter output because the can be inverted. Since it is well known that is filter a sufficient statistic, it follows that also is a sufficient statistic. REFERENCES [1] Y. Sato, “A method of self-recovering equalization for multilevel amplitude-modulation systems,” IEEE Trans. Commun., vol. COM-23, no. 6, pp. 679–682, Jun. 1975. [2] J. R. Treichler and B. G. Agee, “A new approach to multipath correction of constant modulus signals,” IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-31, no. 4, Apr. 1983. [3] O. Shalvi and E. Weinstein, “Universal methods for blind deconvolution,” in Blind Deconvolution, S. Haykin, Ed. Englewood Cliffs, NJ: Prentice-Hall, Jun. 1999. [4] C. R. Johnson Jr., P. Schniter, T. J. Endres, J. D. Behm, D. R. Brown, and R. A. Casas, “Blind equalization using the constant modulus criterion: A review,” Proc. IEEE, vol. 86, no. 10, pp. 1927–1950, Oct. 1998. [5] L. Tong, G. Xu, and T. Kailath, “Blind identification and equalization based on second order statistics: A time domain approach,” IEEE Trans. Inf. Theory, vol. 40, pp. 340–349, Mar. 1994. [6] H. H. Zeng and L. Tong, “Blind channel estimation using the secondorder statistics: asymptotic performance and limitations,” IEEE Trans. Signal Process., vol. 45, no. 8, Aug. 1997. [7] G. Panci, G. Jacovitti, and G. Scarano, “Bussgang-zero crossing equalization: an integrated HOS-SOS approach,” IEEE Trans. Signal Process., vol. 49, no. 11, pp. 2798–2812, Nov. 2001. [8] J. Yang, J. Werner, and G. A. Dumont, “The multimodulus blind equalization and its generalized algorithms,” IEEE J. Sel. Areas Commun., vol. 20, no. 5, pp. 997–1015, Jun. 2002. [9] G. H. Im, D. D. Harman, G. Huang, A. V. Mandzik, M. H. Nguyen, and J. J. Werner, “51.84 Mb/s 16-CAP ATM-LN standard,” IEEE J. Sel. Areas Commun., vol. 13, pp. 620–632, May 1995. [10] G. H. Im and J. J. Werner, “Bandwidth efficient digital transmission over unshielded twisted pair wiring,” IEEE J. Sel. Areas Commun., vol. 13, pp. 1643–1655, Dec. 1995. [11] D. D. Harman, G. Huang, G. H. Im, M. H. Nguyen, J. J. Werner, and M. K. Wong, “Local distribution for IMIV,” IEEE Multimedia, vol. 2, pp. 14–23, Fall 1995. [12] Z. Ding and G. Li, “Single channel blind equalization for GSM cellular systems,” IEEE J. Sel. Areas Commun., vol. 16, no. 8, pp. 1493–1505, 1998.

TLFeBOOK

PANCI et al.: BLIND EQUALIZATION FOR CORRELATED INPUT SYMBOLS

1869

[13] M. Sirbu, J. Mannerkoski, V. Koivunen, and Y. Zhang, “On the feasibility of blind equalization for EDGE systems,” in Proc. Third IEEE Workshop Signal Process. Adv. Wireless Commun., Taoyuan, Taiwan, R.O.C., Mar. 2001, pp. 90–93. [14] K. H. Afkhamie and Z. K. Luo, “Blind identification of FIR systems driven by Markov-like input signals,” IEEE Trans. Signal Process., vol. 48, no. 6, pp. 1726–1736, Jun. 2000. [15] R. Lopez-Valcarce and S. Dasgupta, “Blind channel equalization with colored sources based on second order statistics: a linear prediction approach,” IEEE Trans. Signal Process., vol. 49, no. 9, Sep. 2001. [16] P. Jung, “Laurent’s representation of binary digital continuous phase modulated signals with modulation index 1/2 revisited,” IEEE Trans. Commun., vol. 42, no. 2/3/4, pp. 221–224, Feb./Mar./Apr. 1994. [17] W. H. Gerstacker and R. Schober, “Equalization concepts for EDGE,” IEEE Trans. Wireless Commun., vol. 1, pp. 190–199, Jan. 2002. [18] A. Furuskär, S. Mazur, F. Müller, and H. Olofsson, “EDGE: enhanced data rates for GSM and TDMA/136 evolution,” IEEE Trans. Wireless Commun., vol. 1, pp. 190–199, Jan. 2002. [19] G. Panci, S. Colonnese, and G. Scarano, “Fractionally spaced Bussgang equalization for correlated input symbols,” in Proc. Fourth IEEE Workshop Signal Process. Adv. Wireless Commun., Rome, Italy, Jun. 2003. , “Fractionally spaced Bussgang equalization for GMSK modulated [20] signals,” in Proc. Seventh Int. Symp. Signal Process. Applicati., Paris, France, Jul. 2003. [21] S. Bellini, “Bussgang techniques for blind deconvolution and equalization,” in Blind Deconvolution, S. Haykin, Ed. Englewood Cliffs, NJ: Prentice-Hall, 1994. [22] R. Godfrey and F. Rocca, “Zero memory nonlinear deconvolution,” Geophys. Prospecting, vol. 29, no. 2, Apr. 1981. [23] G. Panci, P. Campisi, S. Colonnese, and G. Scarano, “Multichannel blind image deconvolution using the Bussgang algorithm: spatial and multiresolution approaches,” IEEE Trans. Image Process., vol. 12, no. 11, pp. 1324–1337, Nov. 2003. [24] P. Campisi and G. Scarano, “Multiresolution approach for texture synthesis using the circular harmonic function,” IEEE Trans. Image Process., vol. 11, no. 1, pp. 37–51, Jan. 2002. [25] J. J. Shynk, R. P. Gooch, G. Krishnamurthy, and C. K. Chan, “A comparative performance study of several blind equalization algorithms,” in Proc. SPIE Adaptive Signal Process., San Diego, CA, Jul. 1991. [26] O. Shalvi and E. Weinstein, “Super-exponential methods for blind deconvolution,” IEEE Trans. Inf. Theory, vol. 39, no. 2, pp. 504–519, Mar. 1993. [27] CEPT/COST 207 WG1, Proposal on Channel Transfer Functions to be Used in GSM Tests Late 1986, vol. TD86, 51, Rev. 3, Sep. 1986. [28] C. B. Papadias and D. T. M. Slock, “Fractionally spaced equalization of linear polyphase channels and related blind techniques based on multichannel linear prediction,” IEEE Trans. Signal Process., vol. 47, no. 3, pp. 641–654, Mar. 1999. [29] R. D. Gitlin, J. F. Hayes, and S. B. Weinstein, Data Communications Principles. New York: Plenum, 1992.

Gianpiero Panci was born in Palestrina, Rome, Italy. He received the “Laurea” degree in telecommunications engineering in 1996 and the Ph.D. degree in communication and information theory in 2000, both from the Università “La Sapienza” Rome. He has worked in the area of signal processing for communication and radar systems, and in the 1999, he was involved in the design of the Signum Coded Sar Processor for the Shuttle Radar Topographic Mission. He currently holds an Associate Research position with the Dipartimento di Scienza e Tecnica dell’Informazione e della Comunicazione (INFO-COM), University “La Sapienza,” where he is also Lecturer for a graduate course in communications. His current research interests include statistical signal processing, blind identification and equalization, and array processing.

Stefania Colonnese was born in Rome, Italy. She received the “Laurea” degree in electronic engineering from the Università “La Sapienza,” Rome, in 1993 and the Ph.D. degree in electronic engineering from the Università degli Studi di Roma “Roma Tre” in 1997. In 1993, she joined the Fondazione Ugo Bordoni, Rome, first as a scholarship holder and later as Associate Researcher. During the MPEG-4 standardization activity, she was involved in the MPEG-4 N2 Core Experiment on Automatic Video Segmentation. In 2001, she joined the Dipartimento di Scienza e Tecnica dell’Informazione e della Comunicazione (INFO-COM), University “La Sapienza,” as Assistant Professor. Her research interests lie in the areas of video communications and image and signal processing.

Patrizio Campisi (M’99) received the “Laurea” degree in electronic engineering from the Università “La Sapienza,” Rome, Italy, in 1995 and the Ph.D. degree in electrical engineering from the Università degli Studi di Roma “Roma Tre,” Rome, in 1999. He is an Assistant Professor with the Department of Electrical Engineering, Università degli Studi di Roma “Roma Tre,” where he has taught the graduate course “Signal Theory” since 1998. From September 1997 until April 1998, he was a visiting research associate at the Communication Laboratory, University of Toronto, Toronto, ON, Canada, and from July 2000 until November 2000, he was a Post Doctoral fellow with the same laboratory. From March 2003 until June 2003, he was a visiting researcher at the Beckman Institute, University of Illinois at Urbana-Champaign. From October 1999 to October 2001, he held a Post Doctoral position at the Università degli Studi di Roma “Roma Tre.” His research interests are in the area of digital signal and image processing with applications to multimedia.

Gaetano Scarano (M’00) was born in Campobasso, Italy. He received the “Laurea” degree in electronic engineering from the Università “La Sapienza” Rome, Italy, in 1982. In 1982, he joined the Istituto di Acustica of the Consiglio Nazionale delle Ricerche, Rome, as Associate Researcher. Since 1988, he has been teaching Digital Signal Processing at the University of Perugia, Perugia, Italy, where in 1991, he became Associate Professor of signal theory. In 1992, he joined the Dipartimento di Scienza e Tecnica dell’Informazione e della Comunicazione (INFO-COM), University of Roma “La Sapienza,” first as Associate Professor of image processing and then as Professor of signal theory. His research interests lie in the area of signal and image processing, communications, and estimation and detection theory, and include channel equalization and estimation, image restoration, and texture analysis, synthesis, and classification. Prof. Scarano has served as Associate Editor of the IEEE SIGNAL PROCESSING LETTERS.

TLFeBOOK

1870

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Low Noise Reversible MDCT (RMDCT) and Its Application in Progressive-to-Lossless Embedded Audio Coding Jin Li, Senior Member, IEEE

Abstract—A reversible transform converts an integer input to an integer output, while retaining the ability to reconstruct the exact input from the output sequence. It is one of the key components for lossless and progressive-to-lossless audio codecs. In this work, we investigate the desired characteristics of a high-performance reversible transform. Specifically, we show that the smaller the quantization noise of the reversible modified discrete cosine transform (RMDCT), the better the compression performance of the lossless and progressive-to-lossless codec that utilizes the transform. Armed with this knowledge, we develop a number of RMDCT solutions. The first RMDCT solution is implemented by turning every rotation module of a float MDCT (FMDCT) into a reversible rotation, which uses multiple factorizations to further reduce the quantization noise. The second and third solutions use the matrix lifting to implement a reversible fast Fourier transform (FFT) and a reversible fractional-shifted FFT, respectively, which are further combined with the reversible rotations to form the RMDCT. With the matrix lifting, we can design the RMDCT that has less quantization noise and can still be computed efficiently. A progressive-to-lossless embedded audio codec (PLEAC) employing the RMDCT is implemented with superior results for both lossless and lossy audio compression. Index Terms—Integer transform, low-noise reversible transform, matrix lifting, multiple factorization reversible rotation, progressive to lossless embedded audio codec, reversible FFT, reversible fractional-shifted FFT, reversible MDCT (RMDCT), reversible transform.

I. INTRODUCTION IGH-performance audio codecs bring digital music into practical reality. Popular audio compression technologies, such as MPEG-1 layer 3 (MP3), MPEG-4 audio, Real Audio, and Windows Media Audio (WMA), are lossy in nature. The audio waveform is distorted in exchange for higher compression ratio. In applications where audio quality is critical, such as professional recording and editing, the compromise of trading distortion for compression is not acceptable. These applications must preserve the original audio. Any audio compression should be performed in a lossless fashion. An especially attractive feature of the lossless audio codec is the progressive-to-lossless, where the audio is compressed into a lossless bitstream, which may be further truncated at an

H

Manuscript received November 17, 2003; revised June 3, 2004. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Sheila S. Hemami. The author is with the Microsoft Research, Communication, Collaboration and Signal Processing, Redmond, WA 98052 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845480

arbitrary point to a lossy bitstream of lesser bitrate without re-encoding. The progressive-to-lossless media codec offers the greatest flexibility in compression. During initial encoding, the media may be compressed to lossless, which preserves all the information of the original media. Later, if the transmission bandwidth or the storage space is insufficient to accommodate the full lossless media, the compressed media bitstream may be effortlessly truncated to whatever bitrate is desired. The state-of-the-art image compression algorithm (JPEG 2000 [1]) has the progressive-to-lossless compression mode. No existing audio codec operates in the progressive-to-lossless mode. Most lossless audio coding approaches, such as those in [5]–[7], are built on a lossy audio coder. The audio is first encoded with an existing lossy codec; then, the residue error between the original audio and the lossy coded audio is encoded. The resultant compressed bitstream has two rate points: the lossy base bitrate and the lossless bitrate. It may not be scaled at the other bitrate points. Since the quantization noise in the lossy coder is difficult to model, such approaches usually lead to a drop in the lossless compression efficiency. Moreover, it is also more complex, as it requires the implementation of a base coder and a residue coder. Some other approaches, e.g., [8], build the lossless audio coder directly through a predictive filter and then encode the prediction residue. The approaches may achieve good lossless compression performance. However, there is still no scalability of the resultant bitstream. To develop a progressive-to-lossless embedded audio codec, there are two key modules: the reversible transform and the lossless embedded entropy coder. The reversible transform is usually derived from the linear transform of a traditional psychoacoustic audio coder. By splitting the linear transform into a number of modules and implementing each module with a reversible transform module, we can construct a larger reversible transform module whose output resembles that of the linear transform, except for the rounding errors. The reversible transform establishes a one-to-one correspondence between its input and output and converts the input audio to a set of integer coefficients. The lossless embedded entropy coder then encodes the resultant coefficients progressively all the way to lossless, often in a sub-bitplane by sub-bitplane fashion. By incorporating both modules in the audio codec, we can achieve progressive-to-lossless. If the entire compressed bitstream is delivered to the decoder, it may exactly reconstruct the original audio. If the bitstream is truncated at a certain bitrate, the decoder may reconstruct a high perceptual quality audio at that bitrate. In this paper, we focus on the design of the reversible transform. See [12] for details of the embedded entropy coder.

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

LI: LOW NOISE REVERSIBLE MDCT (RMDCT)

1871

For a transform to be reversible, it must convert the integer input to the integer output and be able to exactly reconstruct the input from the output. These two properties are essential to guarantee reversibility. Nevertheless, there are other desired properties of the reversible transform. Low computational complexity is certainly one of them. Another desired property is the normalization. Consider the following two reversible transforms, which are the candidates of the stereo mixer used in the progressive-to-lossless audio codec (PLEAC):

and

step step step step

(1)

(6) It is easier to find the corresponding reversible transform for the simple module . We may then concatenate the reversible modules to form the reversible transform of . For such reversible transform design, another desired property concerns the quantization noise of the reversible transform, which is the deviation of the output of the reversible transform from that of the linear transform of its characterization matrix rev

(2)

where is a rounding to integer operation, and are integer inputs, and and are integer outputs. Both transforms are reversible; however, the output of transform (1) generates a sparse output set because all points with equal to odd are not occupied. In comparison, the output of transform (2) is dense. We notice that if the rounding operation in (2) is removed, it will become a linear transform. In general, let a reversible transform be rev

(3)

where is the input integer vector, is the output integer vector, and rev( ) denotes the reversible transform operator. If omitting all the rounding operators in the transform, the transform can be converted into a linear transform represented with matrix multiplication (4) the characterization matrix of the we call the matrix reversible transform. The characterization matrices of the reversible transforms (1) and (2) are and

the reversible transform design is to factor the original linear transform into a series of simple linear transform modules

respectively

(5) If is the set of all possible input data points, the output of the reversible transform occupies a volume roughly determined by , where is the volume of the input data set, and is the absolute determinant of the matrix . A valid reversible transform cannot have a characterization matrix with absolute determinant smaller than 1 because such a transform will compact the data and cause multiple input integer vectors mapping to one output integer vector, which contradicts the reversibility. A reversible transform with absolute determinant greater than 1 expands the input data set and creates holes in the output data. It is extremely difficult to design a lossless entropy coder to deal with the holes in the output dataset, particularly if the reversible transform is complicated. As a result, a desired property of the reversible transform is that the absolute determinant of its characterization matrix is one, i.e., the reversible transform is normalized. In audio compression, we already know a good linear transform (the FMDCT) and need to design an RMDCT whose characterization matrix is the FMDCT. A common strategy of

(7)

The quantization noise results from the rounding errors of various stages of the reversible transform. It is unavoidable because it is the byproduct of reversibility, which forces the intermediate result and the output to be integers. The rounding error in each stage of the reversible transform can be considered as an independent random variable with no correlation with the input and output of that stage. Thus, the aggregated quantization noise of the reversible transform also has likewise little correlation with the input and the output. Put another way, the output of the reversible transform rev can be considered to be the sum of the output of the linear transform and a random quantization noise . It is preferable to design the reversible transform with as low quantization noise as possible. This is because the random quantization noise increases the entropy of the output, which reduces the lossless compression performance. Moreover, the quantization noise also creates a noise floor in the output of the reversible transform, which reduces the audio quality in the progressive-to-lossless stage as well. As a result, reduction of the quantization noise can improve the lossy compression performance. The correlation between the quantization noise level of the reversible transform and its lossless and lossy compression performance is confirmed by the experiments in Section VII. The FMDCT can be factored into a series of rotations. One way to derive an RMDCT is thus to convert each and every rotation into a reversible rotation, as shown in [4]. It is common knowledge that a normalized rotation can be factored into a three-step lifting operation via

(8) By using rounding in each of the lifting steps, the rotation becomes reversible: step step (9) step where and are lifting parameters. Existing research on the reversible DCT [3] and the RMDCT [4] uses the factorization in (9) as the basic operation for the reversible transform. Although reversibility is achieved, the quantization noise of the approach (8) can be fairly large and may lead to poor signal representation and poor lossless and lossy compression performance. An alternative method is to factor a large component of the linear transform into the upper and lower unit triangular matrices (UTM), which are triangular matrices with diagonal en-

TLFeBOOK

1872

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

tries 1 or 1. It is shown in [9] that an even sized real matrix with absolute determinant of 1 can be factored into (10) and are lower UTMs, where is a permutation matrix, and is an upper UTM. Matrices , , and can be reversibly implemented via lifting with rounding operations, where is the size of the matrix. The implementation of (10) leads to fewer rounding operations and, thus, smaller quantization noise. Nevertheless, unlike a structured transform such as the FFT, there is usually no structure in matrix , , and , and thus, there is no fast algorithm to compute the multiplication by matrix , , and . The computational complexity of the UTM factorization approach is hence high. In the following, we describe an RMDCT implementation with normalized characterization matrix, low computational complexity, and low quantization noise. This is achieved through two technologies: 1) the multiple factorization reversible rotation and 2) the matrix lifting.1 Through the multiple factorizations, the quantization noise of the reversible rotation, which is an important building block of the RMDCT, is greatly reduced. Through the matrix lifting, we demonstrate that an FFT or a fractional-shifted FFT may be reversibly implemented with reduced quantization noise and an efficient computation method. We then use the reversible FFT or the reversible fractional-shifted FFT to implement the RMDCT. The rest of the paper is organized as follows. The structure of the FMDCT is reviewed in Section II. In Section III, we describe a low noise reversible rotation through the multiple factorizations. Then, in Section IV, we investigate the matrix lifting and its application in the reversible transform. We derive the reversible FFT and the reversible fractional-shifted FFT through the matrix lifting and use them to implement a lownoise RMDCT. For cross-platform reversibility, we implement the RMDCT with only integer arithmetic. A number of integer arithmetic implementation issues are examined in Section V. The PLEAC that incorporates the RMDCT is described in Section VI. Experimental results are shown in Section VII.

Fig. 1. FMDCT via (a) type-IV DST or (b) type-IV DCT.

where

(12) and

diag

(13)

is the MDCT matrix, and is a window function. In MP3 audio coding, the window function is (14) According to [11], the FMDCT can be calculated via a type-IV discrete sine transform (DST) shown in Fig. 1(a). The input signal is first grouped into two data pairs and , with . Each pair is then treated as a complex number and rotated according to an angle specified by . We call this the window rotation. The middle section of the signal is then transformed by a type-IV DST with

II. BACKGROUND-FLOAT MDCT The modified discrete cosine transform (MDCT) is often used in the psychoacoustic audio coder, e.g., MPEG-1 layer 3 (MP3) audio coder, to compact the energy of an audio signal into a few large coefficients. An N-point MDCT transform takes in 2N signal points and outputs N coefficients. The N-point float MDCT (FMDCT) can be expressed as

(15) The sign of the odd index coefficients are then reversed. The N-point type-IV DST is further implemented by an N/2-point complex fractional-shifted FFT with , as (16)

(11) 1During the review process of the paper, the author noticed that Geiger et al. [10] had independently developed an RMDCT (called as IntMDCT) solution with the matrix lifting. Geiger called his approach “multidimensional lifting.”

where

T 0 , where The solution was based on the factorization of the matrix 0 T T could be any nonsingular matrix. In comparison, our approach uses a more general factorization of any nonsingular matrix supported by Theorem 1 of Section IV-A. The float transform used in Geiger’s approach is a type-IV DST in the real domain. Our approach uses the FFT in the complex domain. The computational complexity of Geiger’s approach is N reversible rotations and three N/2-point type-IV DCTs. The computational complexity of our approach is N reversible rotations, four N/4-point FFT, and 1.75N float rotations. Geiger’s approach requires 4N rounding operations for an N-point MDCT, while our approach requires 4.5N rounding operations.

TLFeBOOK

with and is even and is odd otherwise

..

.

(17)

(18)

LI: LOW NOISE REVERSIBLE MDCT (RMDCT)

1873

The general form of an N-point fractional-shifted FFT [double the length of what is used in (16)] is

with

(19)

where and are shifting parameters, and is a complex rotation. Note that the fractional-shifted FFT is a complex matrix, whereas the other matrices in (16) are real matrices. This is interpreted by expanding every element of a complex matrix into a 2 2 submatrix of the form

transform the middle section of the signal through a type-IV DCT with

(26) The implementation can be shown in Fig. 1(b). It is easily verified that an N-point type-IV DCT can be converted into an N/2-point inverse fractional-shifted FFT

(27) with

(20) and are the real and imaginary part of the where complex value , respectively. Like the FFT, the fractional-shifted FFT is an orthogonal transform. This can be easily verified as the Hermitian inner product of any two vectors of the fractional-shifted FFT is a delta function

(21) As a corollary, the inverse of the fractional-shifted FFT is (22) The fractional-shifted FFT can be decomposed into a prerotation , FFT , and a post-rotation . The fractional-shifted FFT can thus be implemented via a standard FFT as

..

(28) .

With the FMDCT, the two implementations of Fig. 1 lead to the same result. However, they lead to slightly different derived reversible transforms. The FMDCT transform has other alternative forms and implementations, with different phases and window functions. Some alternative FMDCTs, termed modulated lapped transforms (MLTs), are shown in [11]. Nevertheless, all FMDCTs and alternative forms can be decomposed into the window rotations and the subsequent type-IV DST/DCT. In this work, we derive the RMDCT from the FMDCT in the form of Fig. 1(a). Nevertheless, the result can be easily extended to the RMDCT derived from the other FMDCT forms. For example, if an alternative form FMDCT uses the type-IV DCT implementation, we only need to implement the inverse fractional-shifted FFT instead of the forward fractional-shifted FFT.

(23) III. REVERSIBLE MDCT I—REVERSIBLE ROTATION THROUGH MULTIPLE FACTORIZATIONS

where diag is a diagonal matrix of

(24)

rotations, and (25)

is the standard FFT. We notice that the matrices and are permutation matrices and may be implemented as such in the reversible transform. To derive the RMDCT from the FMDCT above, we simply need to turn the window rotation into the reversible rotation and implement the fractional-shifted FFT with a reversible fractional-shifted FFT. An alternative implementation of the FMDCT is to first group the signal into pairs of with , rotate them according to an angle specified by , and then

In Fig. 1(a), we show that the FMDCT consists of the window rotation , the type-IV DST, and the sign change. The type-IV DST can be implemented via the fractional-shifted FFT, which in turn consists of the prerotation, the FFT, and the post-rotation. The FFT can be implemented via the butterfly operations; more specifically, the 2N-point FFT can be implemented via first applying the N-point FFT on the odd and even index signal and then combine the output via the butterfly

with

(29)

and are the 2N- and N-point FFT, is a where permutation matrix that separates the 2N complex vector into

TLFeBOOK

1874

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

the size-N vector of even indices and the size-N vector of odd indices, and is the butterfly operator. Note in the standard FFT [2, ch. 12], the butterfly operator is (30) and a normalizing operation of is applied after the entire FFT has been completed. However, this is not feasible in the reversible FFT, as normalizing by is not reversible. We thus need to adopt (29) as the basic butterfly. The matrix can be implemented via the conjugated rotation of . The matrix are complex rotations. As a result, the entire FMDCT can be implemented via a series of rotations. By implementing each and every rotation through the three-step lifting operation of (8) and (9), we can derive one implementation of the RMDCT. The problem of such an implementation is that the quantization noise of certain rotation angles could be fairly large, which leads to large quantization noise of the RMDCT. One of our contributions is to factor the rotation operation with multiple forms. We notice that (8) is not the only form in which a reversible rotation may be factorized. There are three other factorizations of the rotation in the forms of

(31)

(32)

(33) The core of the factorization is still the three-step lifting operation of (9). However, the pair of input/output variables may be swapped before [as in (32)] and after [as in (31)] the lifting operation. The sign of the input/output may be changed as well. The additional forms of the factorization lead to different lifting parameters and for the same rotation angle , with a certain form of the factorization having a lower quantization noise than the others. In the following, we select the optimal factorization form for the different rotation angle that achieves the lowest quantization noise in the mean square error (MSE) sense. Let and be the quantization noise of the reversible rotation. The goal is to minimize the MSE: . We notice that it is the rounding operation that introduces the quantization noise into the reversible transform. The coefficient swapping and sign changing operations do not introduce additional quantization noise. Let be the quantization noise of one rounding operation (34)

Fig. 2. Quantization noise versus the rotation angle of the different factorization forms. The correspondence between the legend and the -(31), -(32) and -(33). The bottom factorization forms are o-(8) solid line is the quantization noise with the combined factorization.

2

+

}

We may model the quantization noise in the reversible transform as (35) where , , and are the quantization noise at the lifting steps 0, 1, and 2 of (9), respectively. Assuming the quantization noise at each step is independent and identically distributed, with being the average energy of the quantization noise of a single rounding operation, the MSE of the quantization noise of the reversible transform can be calculated as (36) We plot the quantization noise versus the rotation angles for different factorization forms (8) and (31)–(33) in Fig. 2. We observe that with any single factorization, the quantization noise can be fairly large at certain rotation angles. By switching among different factorization forms or, more specifically, by using the factorization forms (8) and (31)–(33) for the rotation angles in range , , , and , respectively, we may control the quantization noise to be at most . The magnitude of depends on the rounding operation used. If we use rounding toward the nearest integer, the average energy of the quantization noise of one rounding step is (37) If rounding toward zero is used, the average energy becomes (38) It is apparent that rounding toward the nearest integer is preferred as it generates smaller quantization noise per rounding operation. By using the multiple factorization reversible rotation to replace each rotation in the FMDCT, we may derive an RMDCT with relatively lower noise than simply using the reversible rotation of form (8). The question is, can we further improve upon the scheme?

TLFeBOOK

LI: LOW NOISE REVERSIBLE MDCT (RMDCT)

1875

IV. REVERSIBLE MDCT II—THE MATRIX LIFTING In this section, we show that it is possible to derive a reversible transform through the matrix lifting, which will further lower the quantization noise. A. Matrix Lifting Theorem 1: Every nonsingular even-sized matrix or complex) of size can be factored into

(real Fig. 3.

Forward reversible transform via the matrix lifting.

Fig. 4.

Inverse reversible transform via the matrix lifting.

(39) where and are permutation matrices of size , is the identity matrix, , , and are matrices, and is a nonsingular matrix. Proof: Since is nonsingular, there exist permutation matrices and so that (40) with

being nonsingular. Observing that

(41) by taking a determinant of and using the distributive property of the determinant, we have (42) is thus nonsingular as well. Let

The matrix be the matrix

(43) By assigning matrices

,

,

, and

to be

(44)

substituting (44) into (40), we may easily verify that the (39) holds. Using the (39), we can derive a reversible transform from the linear transform of . The operation flow of the resultant reversible transform can be shown in Fig. 3. The input of the reversible transform is a size 2N (for real transform) or 4N (for complex transform, as each complex consists of an integer real part and an integer imaginary part) integer vectors. After the permutation operation , it is split into two size N (real) or size 2N (complex) integer vectors and , which are transformed through

(45)

where , , and are float transforms, and represents a vector rounding operation. Under the Cartesian coordinate, can be implemented via rounding every element of . For a complex vector of , we may individually round the real and imaginary part of every element of . “Rev ” is a reversible transform to be derived from the linear nonsingular transform . Finally, another permutation operation is applied on the resultant integer vectors and . Because each of the above operations can be exactly reversed, the entire transform is reversible. The inverse of the transform can be shown in Fig. 4. We call the operation in (45) matrix lifting because it bears similarity to the lifting used in (9), except that the multiplication operation now is a matrix multiplication, and the rounding operation is a vector rounding. Note that our approach is different from the approach of [9], where the matrix is factored into UTM matrices, each row of which is still calculated via scalar lifting. Using different permutation matrices and , we may derive different forms of the reversible transforms from the linear transform with different lifting matrices , , and and reversible core . The trick is to select the permutation matrices and so that we have the following. is as simple as possible. In The reversible core a) the sub-optimal case, as in the reversible FFT, the reversible core consists of reversible rotations, which can be implemented via lifting steps. In the best-case scenarios, the reversible core consists of only permutations. In such scenarios, we may derive a reversible transform with only 3N (for real matrix ) or 6N (for complex matrix ) rounding operations. Compared to turning every small module, e.g., rotation, into the reversible rotation, which requires roundings, the matrix lifting may greatly reduce the number of rounding operations required in the reversible transform and lower the quantization noise of the reversible transform. The computation complexity of the transforms , b) and is as low as possible.

TLFeBOOK

1876

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

B. Reversible FFT via the Matrix Lifting Let us derive the reversible FFT with the matrix lifting tool. Inspired by the radix-2 FFT of (29), a 2N-point FFT can be factored into

The proof is rather straightforward. Letting , the element of may be calculated through the matrix rotation as

(46) Setting the matrix (39), with

and and using (39), can be factorized into the matrix lifting form of

mod

(50)

We thus have (47)

is the inverse FFT, In (47), in the form of

is a permutation matrix

.

..

(48)

is a permutation followed by The reversible core rotations , which can be implemented via the multiple factorization reversible rotation we have developed in Section III. The float transform consists of an inverse FFT, N rotations, and a vector addition.2 The transform is an inverse FFT. The transform consists of a forward FFT, an inverse FFT, and N rotations. An N-point reversible FFT (with 2N input integers, as each input is complex with real and imaginary parts) can thus be implemented via (47), with the computation complexity being four N/2-point complex FFT, N float rotations, and N/2 reversible rotations. It requires 4.5N roundings, with N roundings being used after each matrix lifting , , and , and 1.5N roundings being used in the N/2 reversible rotations in . Compared with using reversible rotation to directly implement the reversible FFT, which requires O(NlogN) roundings, the matrix lifting approach greatly reduces the number of rounding operations. C. Reversible Fractional-Shifted FFT via the Matrix Lifting We may implement the RMDCT with the reversible FFT developed above. Yet, there is an even simpler implementation of the RMDCT. Observing that the type-IV DST, which is the most important component of the FMDCT, is directly related to the fractional-shifted FFT with through (16), we may derive a reversible fractional-shifted FFT directly with the matrix lifting. We notice that the fractional shifted FFT with has the following properties: (49) where is a permutation matrix with only the element (0,0) and elements , being nonzero.

3

p

by 2 can be rolled into either the FFT or the rotation operations (; ), with no additional complexity required.

2Scale

..

(51)

.

To derive the reversible transform from the fractional-shifted FFT with , we again use the radix-2 FFT structure. Using (50), we factor the fractional shifted FFT as follows: (52) with (53) and (54) and expanding the fractional-shifted Substituting FFT via (23), we factor the transform into the matrix lifting form of (39), with

..

.

(55) In (55), the reversible core is a permutation matrix, as multiplying by just swaps the real and imaginary part and changes the sign of the imaginary part. The reversible fractional-shifted FFT of is thus the matrix lifting of (55) plus the post reversible rotations of , which again can be implemented via the multiple factorization rotations described in Section III. Using (55), an N-point RMDCT can be implemented via N/2 reversible window rotations of , a reversible matrix lifting of , and N/2 reversible rotations of (noticing that the N-point FMDCT consists of an N-point type-IV DST, which can be further converted to an N/2-point fractional-shifted FFT). The total computational complexity is the sum of N reversible rotations, four N/4-point float FFTs, and 1.75N float rotations. The implementation complexity is about double that of an FMDCT, which requires two N/4-point FFTs

TLFeBOOK

LI: LOW NOISE REVERSIBLE MDCT (RMDCT)

1877

and 1.5N float rotations. Altogether, the RMDCT requires 4.5N roundings, with three 0.5N roundings after each matrix lifting of (55), and 3N roundings for the N reversible rotations.

TABLE I BIT PRECISION OF THE TRANSFORM COEFFICIENT

m

V. REVERSIBLE MDCT: INTEGER ARITHMETIC Most operations of the reversible transform, e.g., the add/subtract operation in the lifting, the permutation, and the sign change operation, are integer operations. The only place that requires floating-point operation is in the lifting, where the input integer value (vector) is multiplied by a float value (or transformed through a float matrix) and then rounded. The lifting operation can certainly be implemented via float arithmetic, e.g., with double precision, which provides high precision and large dynamic range. However, float arithmetic is inconsistent across machines, and therefore, reversibility cannot be guaranteed across different platforms. Moreover, float arithmetic is also more complex. Since the float result is ultimately rounded after the lifting, high precision floating-point operation is not essential in the reversible transform. Floating-point operation of the reversible transform may thus be implemented with integer arithmetic. The key is to keep the calculation error caused by integer arithmetic negligible compared to the quantization noise of the rounding operation. To implement the lifting operation with integer arithmetic, each operand of floating-point operation is interpreted as a fixed precision float number (56) is the number of bits of the fractional part, and is where the number of bits of the integer part. The representation in (56) requires a total of bits (with one bit for sign). We call the dynamic range and the precision as a fixed precision float in (56) may represent values with absolute magnitude up to and with precision down to . To perform a floating-point operation (57) where is the input, is the output, and is the multiplication value, the following operations are performed. Assuming that the input and output and are represented with bit dynamic range and bit precision, and the transform coefficient/multiplication value is represented with bit dynamic range and bit precision, we may treat , , and as integer values, perform the multiplication, and then right shift the result by bits. The only component left is the required dynamic range and bit precision of the input, the output, and the transform coefficient. In this paper, these parameters are derived empirically. First, we investigate the dynamic range and bit precision needed to represent the transform coefficient, which in the RMDCT is mainly the rotation angle . The rotation can be implemented either via a 2 2 matrix multiplication, where the values and are used, or via a three-step multiple factorization lifting developed in Section III, where the values and are used. In the multiple factorization reversible rotation, the absolute value of can reach 2.414, which needs bit dynamic range. Thus,

if the transform coefficient is represented with a 32-bit integer, it can have a bit precision of at most bits. To investigate the impact of the bit precision of the transform coefficient on the quantization noise level of the reversible transform, we measure the magnitude of the quantization noise of the RMDCT versus that of the FMDCT under different bit precisions of the transform coefficients. The quantization noise is measured in terms of the MSE, the mean absolute difference (MAD), and the peak absolute difference (PAD), where

and

MSE

(58)

MAD

(59)

PAD

(60)

is the FMDCT coefficient, and is the In the above, RMDCT coefficient. The test audio waveform is the concatenated MPEG-4 sound quality assessment materials (SQAM) [13]. The result is shown in Table I. The RMDCT in use is derived via the fractional-shifted FFT of Section IV-C. We first show in the second column of Table IV the quantization noise level of the RMDCT implemented via float arithmetic. Then, we show in the following columns the quantization noise level of the RMDCT implemented via integer arithmetic, with the bit precision of the transform coefficients being 29, 20, 16, and 12 bits. We observe that with a bit precision above 16 bits, the RMDCT implemented via integer arithmetic has a quantization noise level very close to that of float arithmetic. Less bit precision significantly increases the quantization noise level of the reversible transform, as there is not enough accuracy to correctly represent the multiplicative value/transform coefficient. In the rest of the paper, we choose the bit precision for the transform coefficient to be 29 bits, as this still allows the transform coefficient to be represented with a 32-bit integer. For the remaining 3 bits, 2 bits are used for the dynamic range of the transform coefficients , and 1 bit is used for the sign. required to repNext, we investigate the bit precision resent the input and output of the matrix lifting operations. We again compare the quantization noise level of the RMDCT versus that of the FMDCT, with different bit precisions of the input and output. The result can be shown in Table II. It is evident that the quantization noise level starts to increase with fewer than bits to represent the intermediate result of the float transform. needed to repFinally, we investigate the dynamic range resent the input and output of the matrix lifting. We notice that all operations used in the RMDCT, whether the float rotation,

TLFeBOOK

1878

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE II BIT PRECISION OF THE INPUT/OUTPUT OF THE MATRIX LIFTING

m

TABLE III MAXIMUM BIT DEPTH OF THE INPUT AUDIO THAT CAN BE FED INTO AN RMDCT WITH 32-BIT INTEGER

Fig. 5. PLEAC encoder framework.

the reversible rotation, or the FFT, are energy-preserving operations. Thus, the maximum magnitude that a coefficient can reach is (61) where bitdepth is the number of bits of the input audio, and is the size of the MDCT transform. We need one bit3 to further guard against overflow in the RMDCT. The dynamic range needed for the matrix lifting is thus bitdepth

(62)

In Table III, we list the maximum bitdepth of the input audio that can be fed into the RMDCT with 32-bit integer arithmetic implementation, with different bit precision and RMDCT size. We have verified this table by feeding a pure sine wave of maximum magnitude into the RMDCT module and by making sure that there is no overflow in the RMDCT module. With the RMDCT block size N being 1024, the RMDCT with 32-bit integer arithmetic may accommodate a 20-bitdepth input audio with bits left to represent the fractional part of the lifting. VI. PLEAC WITH THE RMDCT Using the RMDCT and the lossless embedded entropy coder developed in [12], we develop PLEAC. The encoder framework of PLEAC can be shown in Fig. 5. If the input audio is stereo, the audio waveform first goes through a reversible multiplexer (MUX), whose formulation can be shown with (2), and separates into the L R and L R components, where L and R represent the audio on the left and right channel, respectively. If the input audio is mono, the MUX simply passes through the audio. The waveform of each audio component is then transformed by an RMDCT module with switching windows. The RMDCT window size switched between 2048 and 256 samples. After the RMDCT transform, we group the RMDCT coefficients of a number of consecutive windows into a timeslot. In the current configuration, a timeslot consists of 16 long windows or 128 short windows, which in either case include 32 768 samples. The coefficients in the timeslot are then entropy encoded by a highly efficient psychoacoustic embedded entropy coder. The entropy encoder generates a bitstream that, if delivered in its entirety to 3Noticing

p2 in (55).

Fig. 6. PLEAC decoder framework.

the decoder, may losslessly decode the input coefficients. Yet, the bitstream can be truncated at any point with graceful degradation. Finally, a bitstream assembly module puts the bitstreams of both channels together and forms the final compressed bitstream. For the details of the sub-bitplane entropy coder and the bitstream assembly module, see [12]. The framework of the PLEAC decoder can be shown in Fig. 6. The received bitstream is first split into individual channel bitstreams by a disassembly module. Then, it is decoded by the embedded entropy decoder. If the decoder finds the received bitstream to be lossless or be close to lossless, i.e., the coefficients can be decoded to the last bitplane, the decoded coefficients are transformed by an inverse RMDCT module. Otherwise, an inverse FMDCT module is used. The reason to apply the inverse FMDCT for the lossy bitstream is that the inverse FMDCT not only has lower computational complexity but also generates no additional quantization noise. In comparison, the inverse RMDCT generates quantization noise, which, when decoded to lossless, serves to cancel the quantization noise of the encoding stage but, when decoded to lossy, is just additional noise, which degrades the audio playback quality. After the inverse MDCT transform, the individual audio channels are then demultiplexed to form the decoded audio waveform. Notice that the RMDCT module is always used in the PLEAC encoder but only used in the lossless decoding mode in the PLEAC decoder. The computational penalty of the RMDCT thus only resides with the PLEAC encoder and lossless PLEAC decoder.

TLFeBOOK

LI: LOW NOISE REVERSIBLE MDCT (RMDCT)

1879

TABLE IV QUANTIZATION NOISE LEVELS OF DIFFERENT RMDCT MODULES

TABLE V LOSSLESS AND LOSSY COMPRESSION PERFORMANCE OF PLEAC

VII. EXPERIMENTAL RESULTS To evaluate the performance of the proposed RMDCT, we put the RMDCT modules into the PLEAC. We then compare the following RMDCT configurations: Through the reversible fractional-shifted FFT via the a) matrix lifting described in Section IV-C; same as (a), except that the reversible rotation is imb) plemented with only the factorization form of (8); same as (a), except that the rounding operation is imc) plemented as truncation toward zero; through the reversible FFT via the matrix lifting ded) scribed in Section IV-B; with only multiple factorization reversible rotations e) described in Section III. All the other modules of the PLEAC codec are the same. We first compare the output difference of the RMDCT modules versus that of the FMDCT module, in terms of the MSE, the MAD, and the PAD calculated in (58)–(60). We then compare the lossless compression ratio of the PLEAC codecs using the specific RMDCT configuration with the state-of-the-art lossless audio compressor: Monkey’s Audio [8]. Finally, we compare the lossy coding performance. The lossy compressed bitstream is derived by truncating the losslessly compressed bitstream to the bitrate of 64, 32, and 16 kb/s. We then decode the lossy bitstream and measure the decoding noise-mask-ratio (NMR) versus that of the original audio waveform (the smaller the NMR, the better the quality of the decoded audio). The NMR results are then compared with those of the lossy EAC codec, which is the PLEAC with the FMDCT module in both the encoder and the decoder. The test audio waveform is formed by concatenating the MPEG-4 SQAMs [13]. The aggregated comparison results are shown in Tables IV and V. Because the configuration b) only differs from the configuration a) in the implementation of the reversible rotations, their difference in Table IV demonstrates the effectiveness of the multiple factorization reversible rotations. By intelligently selecting the proper factorization form under different rotation angles, multiple factorization greatly reduces the quantization noise in the rotations. The MSE of the quantization noise is reduced by 78%. There is also a noticeable improvement in the lossless (5%

less bitrate) and lossy (on average 4.5 dB better) coding performance of the PLEAC codec by replacing the single factorization reversible rotation with the multiple factorization reversible rotations. The configuration c) only differs from the configuration a) in the implementation of the rounding operation. The configuration a) uses rounding toward the nearest integer, while the configuration c) uses the rounding toward zero. Although the two schemes only differ slightly in implementation, rounding toward the nearest integer is proven to be a better choice for rounding. As shown in Tables IV and V, configuration a) reduces the MSE by 84%, with slightly better lossless (1% less bitrate) and lossy (on average 0.4 dB better) compression performance. From the configuration e) to d) to a), the matrix lifting is used to implement an ever-larger chunk of the FMDCT into a reversible module. Comparing configuration e) (only reversible rotations) with the configuration a), which uses the matrix lifting on the fractional-shifted FFT, the quantization noise of the MSE is reduced by 73%, while there is a reduction of 4% of lossless coding bitrate. We also observe that the NMR of lossy decoded audio improves by an average of 1.4 dB. Overall, the RMDCT configuration with lower quantization noise leads to better lossless and lossy audio compression performance. The anomaly lies in configuration c). Although using truncation toward zero results in a big increase in the quantization noise in term of MSE, MAD, and PAD, it does not incur as much penalty in the lossless and lossy compression performance as compared with configurations b) and e). This anomaly may be explained by the fact that the truncation toward zero reduces the absolute value of the transform coefficients. Most entropy coders including the sub-bitplane entropy coder employed by PLEAC generate a shorter compressed bitstream with smaller transform coefficients. This partly mitigates the rising quantization noise caused by using the truncation toward zero. Nevertheless, we notice that rounding toward the nearest integer still leads to superior performance in lossless and lossy compression. It is observed that with the matrix lifting, the output of the RMDCT becomes very close to the FMDCT. The best RMDCT configuration a), which is implemented via the reversible fractional-shifted FFT with the matrix lifting and the multiple-factorization reversible rotations, results in the

TLFeBOOK

1880

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

MSE of the quantization noise of only 0.48. Therefore, a large number of RMDCT coefficients are the same as the FMDCT coefficients after rounding. Incorporating the RMDCT module a), the PLEAC codec achieves a lossless compression ratio of 2.88:1, whereas the state-of-the-art lossless audio compressor (Monkey’s Audio [8]) achieves a lossless compression ratio of 2.93:1 of the same audio waveform. PLEAC with the RMDCT is thus within 2% of the state-of-the-art lossless audio codec. Moreover, the PLEAC-compressed bitstream can be scaled, from lossless all the way to very low bitrate, whereas such a feature is nonexisting in Monkey’s Audio. Comparing with the EAC codec, which uses the FMDCT in both the encoder and the decoder, PLEAC only results in an NMR loss of 0.8 dB. PLEAC is thus a fantastic all-around scalable codec from lossy all the way to lossless. VIII. CONCLUSIONS A low-noise reversible modified discrete cosine transform (RMDCT) is proposed in this paper. With the matrix lifting and the multiple factorization reversible rotation, we greatly reduce the quantization noise of the RMDCT. We demonstrate that the RMDCT can be implemented with integer arithmetic and thus be ported across platforms. A progressive-to-lossless embedded audio codec (PLEAC) incorporating the RMDCT module is implemented, with its compressed bitstream capable of being scaled from the lossless to any desired bitrate. PLEAC has decent lossless and lossy audio compression performance, with the lossless coding bitrate of PLEAC within 2% of the state-of-the-art of the lossless audio codec. ACKNOWLEDGMENT The author wishes to acknowledge H. S. Malvar and J. D. Johnston for their insightful comments and discussions, A. Colburn for proofreading the paper, and the associate editor and the anonymous reviewers for their close reading and thorough critique of the paper. REFERENCES [1] D. S. Taubman and M. W. Marcellin, JPEG 2000: Image Compression Fundamentals, Standards and Practice. Boston, MA: Kluwer, 2001. [2] W. H. Press, S. A. Teukolsky, W. T. Vertterling, and B. P. Flannery, Numerical Recipes in C++. Cambridge, U.K.: Cambridge Univ. Press, 2002. [3] K. Komatsu and K. Sezaki, “Reversible discrete cosine transform,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., vol. 3, Seattle, WA, May 1998, pp. 1769–1772.

[4] R. Geiger, J. Herre, J. Koller, and K. Brandenburg, “IntMDCT—a link between perceptual and lossless audio coding,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 2, Orlando, FL, May 2002, pp. 1813–1816. [5] M. Purat, T. Liebchen, and P. Noll, “Lossless transform coding of audio signals,” in Proc. 102nd AES Convention, München, Germany, 1997. [6] T. Moriya, A. Jin, T. Mori, K. Ikeda, and T. Kaneko, “Lossless scalable audio coder and quality enhancement,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 2, Orlando, FL, May 2002, pp. 1829–1832. [7] A. Wegener, “MUSICompress: lossless, low-MIPS audio compression in software and hardware,” in Proc. Int. Conf. Signal Process. Applicat. Technol., San Diego, CA, 1997. [8] Monkey’s Audio: A Fast and Powerful Lossless Audio Compressor, http://www.monkeysaudio.com/ [Online] [9] J. Wang, J. Sun, and S. Yu, “1-D and 2-D transforms from integers to integers,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 2, Hong Kong, Apr. 2003, pp. 549–552. [10] R. Geiger, Y. Yokotani, and G. Schuller, “Improved integer transforms for lossless audio coding,” in Proc. 37th Asilomar Conf. Signals, Syst., Comput., Nov. 9–12, 2003, pp. 2119–2123. [11] H. S. Malvar, “Lapped transform for efficient transform/subband coding,” IEEE Trans. Acoust., Speech, Signal Process,, vol. 38, no. 6, pp. 969–978, Jun. 1990. [12] J. Li, “Embedded audio coding (EAC) with implicit auditory masking,” in Proc. ACM Multimedia, Nice, France, Dec. 2002. [13] Sound Quality Assessment Material Recordings for Subjective Tests, http://www.tnt.uni-hannover.de/project/mpeg/audio/sqam/ [Online]

Jin Li (S’94–A’95–M’96–SM’99) received the B.S, M.S, and Ph.D. degrees in electrical engineering, all from Tsinghua University, Beijing, China, in 1990, 1991 and 1994, respectively. From 1994 to 1996, he served as a Research Associate at the University of Southern California, Los Angeles. From 1996 to 1999, he was a member of the technical staff at the Sharp Laboratories of America (SLA), Camas, WA, and represented the interests of SLA in the JPEG2000 and MPEG4 standardization efforts. He was a researcher/project leader at Microsoft Research Asia, Beijing, from 1999 to 2000. He is currently a researcher at Microsoft Research, Redmond, WA. Since 2000, he has also served as an adjunct professor with the Electrical Engineering Department, Tsinghua University. He is an active contributor to the ISO JPEG 2000/MPEG4 project. He has published 16 journal papers and more than 50 conference papers in a diversified research field, with interests cover audio/image/video compression, virtual environment and graphic compression, real-time audio/video communication, audio/video streaming, peer-to-peer content delivery, etc. He holds ten issued US patents, with many more pending. Dr. Li is an Area Editor for the Academic Press Journal of Visual Communication and Image Representation and an Associate Editor of the IEEE TRANSACTIONS ON MULTIMEDIA. He was the recipient of the 1994 Ph.D. thesis award from Tsinghua University and the 1998 Young Investigator Award from SPIE Visual Communication and Image Processing.

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1881

Convolutional Codes Using Finite-Field Wavelets: Time-Varying Codes and More Faramarz Fekri, Senior Member, IEEE, Mina Sartipi, Student Member, IEEE, Russell M. Mersereau, Fellow, IEEE, and Ronald W. Schafer, Fellow, IEEE

Abstract—This paper introduces a procedure for constructing convolutional codes using finite-field wavelets. This provides novel insight into the study of convolutional codes and permits the design of the new convolutional codes that is not possible by conventional methods. Exploiting algebraic properties of the wavelet convolutional wavelet codes, we show that a rate encoder is a basic encoder that is noncatastrophic. In addition, wavelet convolutional encoder is we prove that any rate 1 minimal-basic and that every -band orthogonal wavelet system generates a rate 1 self-orthogonal code. As an application of wavelet convolutional codes, we construct time-varying convolutional codes. These codes have unique trellis structures that result in fast and low computational complexity decoding algorithms. As examples, we present some time-varying wavelet convolutional codes that can be decoded faster than comparable time-invariant convolutional codes. We construct 16- and 32-state time-varying wavelet convolutional codes with minimum-free distances of seven and eight, respectively. These codes have the same minimum-free distance as the best time-invariant codes of the same rate and state complexity, but they can be decoded almost twice as fast. We show that a 32-state time-varying wavelet convolutional code is superior to the Lauer code in performance while having almost the same decoding complexity. Although the 32-state wavelet code is inferior to the 16-state Golay convolutional code as far as computational complexity, it outperforms this code in the performance. We also prove that orthogonal filterbanks generate self-dual time-varying codes. We give a design for doubly even self-dual time-varying convolutional codes by imposing some constraints on the filters that define the -band orthogonal wavelets. As another application of wavelet convolutional codes, we propose a new scheme for generating rate-adaptive codes. These codes have the property that multiple rates of the code can be decoded on one trellis and its subtrellises. Index Terms—Bipartite trellis, finite-field wavelets, rate-adaptive codes, time-varying convolutional codes.

I. INTRODUCTION INITE-FIELD wavelet transforms connect two areas: wavelets and finite fields. Wavelet transforms have been demonstrated to provide an efficient signal representation that is localized in both time and frequency, which make them popular for widespread applications in such areas as audio and video compression [1], [2] and time-frequency analysis

F

Manuscript received March 5, 2003; revised April 9, 2004. This material is based on work supported by the National Science Foundation under Grant CCR–0093229. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Alexi Gorokhov. The authors are with the School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332-0250 USA (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). Digital Object Identifier 10.1109/TSP.2005.845484

[3]. Finite-field wavelets provide a similar representation for sequence of numbers, such as digital data, that are defined over finite-fields. About 20 years ago, Blahut introduced the idea of using the finite-field Fourier methods to study Reed–Solomon and other block codes. The application of finite-field wavelets to error control coding can be seen as a natural evolution of Blahut’s idea. However, finite-field wavelets have the advantage that we can design both block [4], [5] and convolutional codes [6]. Additionally, as opposed to the Fourier basis, which is unique, with wavelets, we have more than one choice for the basis functions. Therefore, it is reasonable to expect that better codes can ultimately be constructed by using wavelets. In [7], Costello derived a lower bound on the free distance of time-varying convolutional codes that is greater than the upper bound on the free distance of time-invariant codes. As an application of wavelet convolutional codes, we thus construct timevarying convolutional codes. Because of the unique trellis structures of wavelet convolutional codes, their decoding is fast. In [8], Hu et al. in presented a provably best time-invariant 16-state convolutional code with a minimum free distance of seven and a 32-state convolutional code with a minimum free distance of eight. In this paper, we introduce a time-varying 16-state wavelet convolutional code that has the same minimum-free distance as the time-invariant convolutional code of [8], but it can be decoded almost twice as fast. As another example, we compare the 32-state wavelet convolutional code [9] with the 32-state time-invariant convolutional code. The wavelet convolutional code has the advantage of having a faster decoding algorithm, whereas it has the same minimum-free distance. We also compare the 32-state time-varying wavelet convolutional code with another nonwavelet 16-state time-varying convolutional code that is found by Calderbank et al. [1] as well as with the partial-unit-memory (PUM) convolutional code constructed by [10]. Although the wavelet code has 32 states, it is slightly more complex than the 16-state Calderbank code. However, the performance of the wavelet code is better. The wavelet time-varying code is superior to PUM code in the performance, while it has almost the same encoding complexity. The design of an error control system usually consists of selecting a fixed code with a certain rate and correction capability matched to the protection required. However, in many cases, the system needs to be more flexible because the channel is time varying. Cain et al. [11] introduced punctured convolutional that simplify the design of Viterbi decoders codes of rate . because only two branches arriving at each node instead of The concept of rate-compatible punctured convolutional codes was first introduced by Hagenauer [12]. He later modified the

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1882

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

concept of punctured convolutional codes to generate a family , where can be varied between of codes with rate . To change the rate of a code, they puncture 1 and (delete) a subset of codeword bits. They start from a low-rate and puncture the code periodically with period . As an alternative approach for flexible channel encoding with an adaptive decoder, we propose to use wavelet convolutional codes. In rate adaptive wavelet convolutional codes, the achievable rates , where can be varied between 1 and . are In Section II, we introduce the structure of wavelet convolutional codes, and in Section III, we derive the algebraic properties of these codes. In Section IV we demonstrate syndrome generators and dual encoders. The syndrome generators are useful to generate the parity check matrix for performing syndrome decoding. Since self-dual codes have received a great deal of attention, we also present a method to construct self-dual and self-orthogonal convolutional codes. Time-varying convolutional codes and their fast decoding are presented in Section V. In Section V, we introduce some time-varying wavelet convolutional codes that have the same minimum distance as the time-invariant convolutional codes of the same rate and constraint length, while they have faster decoding. In Section VI, we show an alternative method for constructing rate-compatible error control systems.

Using this block matrix, we can represent (1) in the following matrix form: .. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. . .. . (3) .. .

in which the vector

is defined as

(4)

.. . The wavelet coefficients

are the inner products

for II. STRUCTURE OF WAVELET CONVOLUTIONAL CODES

(5)

In a similar fashion, (5) can be represented in matrix notation

Convolutional coding deals with infinite-length sequences. In the following, we describe the expansion of signals by wavelet basis functions and the relationship of that expansions to filterbanks. Next, we provide a structure for generating convolutional codes. Consider an -band orthogonal wavelet system in which are the scaling function and the multiple mother wavelets. In addition, let be the support (length) of these functions. Now, any signal can be represented by

as .. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. . .. . (6) .. .

(1) We can also express this signal expansion by using an infinite matrix notation. Let , and define an block matrix, as in (2), shown at the bottom of the page.

.. .

.. .

in which is the matrix defined by (7), shown at the . Because of the bottom of the next page, for perfect reconstruction constraint built into the wavelet system, one can verify that . Furthermore, the paraunitary , where property of the wavelet system implies that the superscript denotes transposition. One can easily show that (1) can be realized by the synthesis bank of an -band filterbank, as shown in Fig. 1, where the im-

.. .

TLFeBOOK

.. .

for

(2)

FEKRI et al.: CONVOLUTIONAL CODES USING FINITE-FIELD WAVELETS

Fig. 1.

1883

Fig. 3.

Filterbank structure of the rate 1=L encoder.

Fig. 4.

Simplified structure of the rate 1=L encoder.

L-channel maximally decimated filterbank.

The structure in Fig. 3 is convenient for searching for the . The prefilters are introduced to keep the genercode of rate ality of the encoder structure. However, once the code has been found, we implement it using the simplified encoder structure shown in Fig. 4. The structure in Fig. 3 is also useful for studying the algebraic properties of the encoder. Notice that the -transin Fig. 4 is given by form of the filter (9) Fig. 2. Filterbank structure of the rate K=L encoder.

It is important not to confuse the -transform function of the code. the generator matrix

pulse responses of the filters are equal to the scaling function and mother wavelets. Furthermore, the wavelet coefficients in (5) can be obtained by using the -band maximally decimated analysis filterbank shown in Fig. 1 with the impulse responses defined as

III. ALGEBRAIC PROPERTIES OF WAVELET CONVOLUTIONAL ENCODERS Before describing the algebraic properties of wavelet convolutional encoders, we need to provide a short background on the algebraic properties of standard convolutional codes. The definitions and notation (with slight changes) are from [15]. A Laurent , polynomial matrix contains elements with only finitely many positive powers of and vector coefficients: . The coefficients form a Laurent series. Let denote the field of binary Laurent series. Note that binary Laurent series form a field in which division is defined by expanding the long division into negative exponents of . Now, we are ready for the formulation of convolutional codes [15]. convolutional code over the Definition 1: A rate is an injective linear mapping field of Laurent series of the -dimensional vector Laurent series into the -dimensional vector Laurent series . in which This mapping can be expressed as is called the generator polynomial matrix. In the design of convolutional codes, we need to avoid encoders that are catastrophic. A catastrophic encoder is a type of encoder whose decoder might make an infinite number of errors

and (8) In [13] and [14], Fekri et al. developed a procedure for designing these orthogonal filterbanks over fields of characteristic two. Fig. 2 shows the structure of a wavelet convolutional en. This requires an -band synthesis coder at rate polyphase bank, and the message sequence is divided into sequences, as shown in Fig. 2. Each of the polyphase compois encoded by a rate wavelet encoder. This renents . As shown in Fig. 2, sults in disjoint subcodes in the code is the binary sum of the codeevery codeword words from these subcodes. By this setup, it is clear that the rate convolutional code is a direct sum of these subcodes, i.e., . As depicted in Fig. 3, for each together with the set subcode , we only use the filter . To do that, each of the message polyphase components is connected (possibly through arbitrary prefilters) to one or more of the input nodes 1 to , as shown in Fig. 2. It is clear that the code is a linear code.

.. .

with

.. .

TLFeBOOK

.. .

.. .

(7)

1884

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 5.

Inverse of the rate K=L encoder of Fig. 2.

in response to a finite number of errors in transmission. The following definition is from [16]. Definition 2: A generator matrix for a convolutional code is catastrophic if there exists an information sequence with infinitely many nonzero values that results in a codeword with only finitely many nonzero elements. It is important to note that a catastrophic situation is caused by the encoder and not by the code itself. In fact, every convolutional code has both catastrophic and noncatastrophic generator matrices. In the following, we show that wavelet convolutional encoders are always noncatastrophic. First, we quote the definition of a basic encoder [17]. is Definition 3: A convolutional encoding matrix called basic if it is a polynomial and it has a polynomial left inverse. Note that in the original definition, it was stated that should have a right inverse because the authors used the nota, as opposed to our notation tion . Proposition 1: A rate wavelet convolutional encoder is a basic encoder, regardless of the choice of the impulse responses of the prefilters. Proof: From the definition of the basic encoder, it is understood that any encoder whose inverse is a feedback-free system is a basic encoder. Thus, it is sufficient to prove that the wavelet encoder has a feedforward inverse. Consider the wavelet convolutional encoder structure shown in rate Fig. 2. It is easy to prove that the system given in Fig. 5 extracts the original message sequence, provided that the filters are chosen to be the analysis filters associated with the synthesis filters in the encoder of Fig. 2. This is true for any arbitrary choice of the prefilters. Since this inverse system is a feedback-free circuit, we conclude that the wavelet encoder is a basic encoder. Next, we show that wavelet encoders are noncatastrophic. Corollary 1: A rate wavelet convolutional encoder is always noncatastrophic, regardless of the choice of the impulse responses of the prefilters. Proof: The proof follows from Proposition 1 and the fact that all basic encoding matrices are noncatastrophic [18]. For a further discussion of the properties of the wavelet encoder, we need to recall a few definitions. Definition 4: The constraint length for the th input of a rate polynomial convolutional encoding matrix is defined as (10) and the overall constraint length is obtained by [17].

Definition 5: The physical state of a realization of an encoding matrix at some time instant is the contents of its memory elements [15]. Definition 6: The abstract state associated with an input sequence is the sequence of outputs at time zero and later, that occurs up to time , which are due to that part of with zero inputs thereafter [15]. It is worth noting that the abstract state depends only on the encoder matrix and not on its realization. Different abstract states correspond to different physical states, but different physical states might correspond to the same abstract state [15]. We are now ready to address two questions. The first question is concerned with what we can say about wavelet convolutional encoders in terms of their number of abstract states. Since the abstract states are all we need to track in the decoding process, finding an encoder that has the smallest number of abstract states among all equivalent generators for that code is very desirable. The second question is whether wavelet convolutional encoders are minimal in both the number of abstract states and the number of physical states. To answer these questions, we need to recall the following Lemma. Lemma 1: If is basic, is minimal (in the sense of the has an anticausal number of abstract states) if and only if inverse [17]. Proposition 2: For any arbitrary choice of the impulse rewavelet convolutional ensponses of the prefilters, a rate coder matrix that is constructed using paraunitary filterbanks contains a minimal number of abstract states over all equivalent encoding matrices for that code. wavelet convolutional enProof: Consider the rate coder shown in Fig. 2 that is constructed by the synthesis filters of an -band paraunitary filterbank with polyphase matrix given by .. .

.. .

.. .

(11)

Here, the causal FIR system is the th polyphase component of the filter . Using the polyphase system, we can generator matrix for the rate show that the wavelet code in its most general form is the following: .. .

.. .

.. .

.. .

(12)

in which the filter , whose -transform is , connects the th input sequence (the th polyphase component of ) to the filter . Since the matrix is the message paraunitary, we have . Thus, we conclude matrix is a left inverse for that the is independent of the encoder matrix . First, note that . Second, since the elements the choice of the prefilters of are polynomials in the variable with only negative

TLFeBOOK

FEKRI et al.: CONVOLUTIONAL CODES USING FINITE-FIELD WAVELETS

1885

powers, the elements of are polynomials in the variable with only positive exponents, which means that the inverse is anticausal. The proof follows using Lemma 1 and Proposition 1. The next important problem is to address the minimality of the encoder matrix in the number of physical states. Let us first quote the following definition. Definition 7: A minimal-basic encoding matrix is a basic encoding matrix whose overall constraint length is minimal over all equivalent basic encoding matrices [15]. It is known that every minimal-basic encoding matrix results in a minimal encoder, but the reverse is not necessarily true. If is a minimal-basic encoding matrix, then the number of physical states is equal to the number of abstract states. Thus, minimal-basic encoding matrices are very desirable because they are minimal in every sense. The above definition of minimalbasic is equivalent to Forney’s definition of a minimal encoder matrix [17]. That is, a basic encoder is minimal if its overall constraint length in the obvious realization is equal to the maximum degree of its sub-determinants. From Forney’s encoder matrix definition, it can be concluded that any rate that is a basic encoder is also minimal-basic. Therefore, because of Proposition 1, we have the following fact. wavelet encoder of the form shown in Fact 1: Any rate Fig. 3 is minimal-basic, regardless of the choice of the prefilters. This guarantees that a controller canonical realization of wavelet encoder requires the smallest number any rate of memory elements over all the other encoding matrices that generate an equivalent code. Unfortunately, the above fact when ) wavelet does not hold for arbitrary rate ( encoders in general. In other words, the number of delay elements required to implement arbitrary rate wavelet encoders in controller canonical form is not necessarily minimal. To prove this, we give a counterexample. Example 1: Consider an orthogonal filterbank specified by a polyphase system: (13) Consider a rate 2/3 convolutional code constructed by the system shown in Fig. 6 in which the filters and are determined . To show that this encoder is by the first two rows of not minimal-basic, we represent the encoder in its polyphase structure. This representation results in the following generator matrix for the code: (14) It can be easily verified that the overall constraint length is , but the maximum degree of all 2 2 sub-determinants is 2. Thus, this encoding matrix is not minimal-basic by the definition. As we notice from Example 1, wavelet encoders do not have minimal-basic encoding matrices in general, although they do result in a minimal number of abstract states. and As examples, we give the minimum-free distance the generators for a few binary short constraint-length wavelet

Fig. 8.

Fig. 6.

Example of a rate 2/3 wavelet convolutional encoder.

Fig. 7.

Example of a half-rate wavelet convolutional encoder.

Polyphase representation of a half-rate wavelet convolutional encoder.

convolutional codes for different code rates. These binary codes are optimal in the sense that they have the largest possible for a given rate and constraint length. It is worth noting that the minimum-free distance is one of the important parameters for measuring the performance of a code in the maximum likeliis dehood decoding method. The minimum-free distance fined as the minimum Hamming distance between all pairs of the convolutional codewords. The following codes are found by a search through different wavelets and prefilters in the diagram of Fig. 7. Example 2: Consider the half-rate encoder shown in Fig. 7. to have an all-zero impulse response, If we set the prefilter then a code with a constraint length of two and a minimum -free distance of five is generated. To verify this, we represent the encoder in its polyphase structure, as shown in Fig. 8. Since , the polyphase components are and . Thus, the constraint length is equal to two. By drawing the trellis diagram using the . polyphase representation, it can be verified that Example 3: Let consider the encoder shown in Fig. 7 constructed by a two-channel orthogonal filterbank in which , and . If we choose , the resulting convolutional code has the prefilter . This code has the same constraint length 3, and minimum distance as the best time-invariant convolutional code of constraint length 3 constructed by the generators [19]: (Note that the numbers are in octal. Furthermore, these generators should not be confused with wavelet generators). Fig. 9 shows the encoder of this conventional convolutional code. In the above two examples, we found the best convolutional code for a given constraint length and rate. Other optimum codes for different constraint lengths and different rates can be found by searching through different wavelets and prefilters. It is important to notice two advantages of wavelet encoding systems with respect to conventional encoding methods. First, because of the orthogonality constraint, the wavelet search space for good convolutional codes is smaller than the traditional (nonwavelet) way of searching for the codes. Second, as shown by

TLFeBOOK

1886

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 9. Example of a time-invariant conventional convolutional code.

Fig. 10. (a) Filterbank structure of the syndrome generator for rate 1=L wavelet codes. (b) Simplified structure of the syndrome generator.

Corollary 1, the wavelet search method always leads to a noncatastrophic encoder, but the traditional search technique does not have this property. IV. DUAL ENCODERS AND SELF-DUAL CONVOLUTIONAL CODES is the encoder of a rate convolutional Suppose code . The syndrome generator for this code is a many-to-one -tuples, whose kernel homomorphism from -tuples to is the code . Fig. 10(a) shows the syndrome generator for rate wavelet convolutional codes. One can show that Fig. 10(b) also gives the structure of the syndrome generator for rate wavelet convolutional codes if we do some modification. Equafor a wavelet tion (15) gives the syndrome generator filter convolutional code.

can be readily studied through the dual codes. Furthermore, in the construction of self-dual and self-orthogonal codes, understanding dual codes is of interest. We first recall the definition of the dual code [20]. to a rate convolutional Definition 8: The dual code code is the set of all -tuples of sequences such that for all . The dual code is a vector space of dimension and encoder such may be generated by any rate . Suppose that specify that an -band orthogonal filterbank. We limit our study of the dual codes of wavelet-convolutional codes only to the rate whose encoder is constructed by using bands out of -band paraunitary filterbanks, as shown by Fig. 11. In other words, are used for generating the code . In this only case, one can prove that the rate code generated by specifies the dual code . To verify this fact, we represent the -band filterbank by its polyphase structure, as in (16), shown at the bottom of the page, in which is the th polyphase component of the th filter . We use columns of to form the encoder matrix the first of the code . Since , we conclude that remaining columns of specify the encoder the matrix of . In other words, . It can be verified that the syndrome generator that corresponds to the convolutional code is formed by those branches of that are associated with the analysis bank shown in Fig. 11. A linear code is called self-orthogonal if it is contained in its dual. Self-dual codes are a special case of self-orthogonal codes in which the code and its dual are the same. Thus, all selfdual codes are half rate. Self-orthogonal and, in particular, selfdual block codes have received a great deal of attention [21], [22]. Here, we present a method to construct self-dual and selforthogonal convolutional codes. Proposition 3: Every -band orthogonal wavelet system, for selfan even number , can be used to construct a rate orthogonal code. Proof: From the development in Section II, we can write the generator matrix for the encoder in Fig. 3 as in which is a semi-infinite matrix with scalar entries defined by

(15) Studying the duals of the convolutional codes is of great interest for many reasons. We expect that problems concerned with the weight, distance, and structure of convolutional codes

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. .

.. . (17)

.. .

.. .

.. . (16)

TLFeBOOK

FEKRI et al.: CONVOLUTIONAL CODES USING FINITE-FIELD WAVELETS

1887

Fig. 12. Diagram of a half-rate periodically time-varying wavelet convolutional encoder with period two.

V. TIME-VARYING WAVELET CONVOLUTIONAL CODES AND BIPARTITE TRELLISES

Fig. 11. Diagram of the wavelet encoder, its dual, and its syndrome generator. (a) Structure of the encoder (special case). (b) Generator for the dual code. (c) Syndrome generator for the code in part (a).

is the th row in the matrix defined in (2). Here, Because of the orthogonality property of wavelets, we have if if

.

(18)

Notice that the generator matrix is a constant matrix, and it should not be confused with the discrete convolution form of the generator matrix. The codeword associated with is obtained by in which and are message vector representations of the codeword and message sequences. and (each Now, we show that any two finite codewords as sequences of -tuples) are orthogonal to each other, i.e., . To prove this, note that , and are the two message sequences correin which and , respectively. Using (18), we conclude sponding to that . Hence, we have , which is equal to zero in fields of characteristic two if is an even implies that the code contains its dual. number. The special case of Proposition 3 are self-dual codes that have a rate equal to . We have the following corollary. Corollary 2: Any two-band orthogonal wavelet system (the synthesis part) results in a self-dual convolutional code.

Thus far, we have studied time-invariant convolutional codes in which the generator matrices are time-invariant. In his paper for non[7], Costello showed that the lower bound on systematic time-varying codes is greater than the upper bound for fixed (time-invariant) or time-varying systematic codes over all rates. It is also conjectured that the same result can be extended to all rates for time-invariant convolutional codes. This led several researchers to search for such codes [10], [23]. For example, a (3,2) time-varying code of constraint length one has , which is larger than the been found by [23] with best time-invariant code with the same constraint length that has . Calderbank et al. [1] used the (24,12,8) Golay code that to form a time-varying convolutional code with than the contains only sixteen states. This code has a larger maximum free distance of any time-invariant 16-state rate-1/2 . code, which is known to be In the following, we present a method to generate noncatastrophic time-varying wavelet convolutional codes. The noncatastrophic property is ensured by the structural constraint. These time-varying codes are quite different from traditional codes in that they have an unusual trellis structure that speeds up their decoding procedure. Here, we introduce a new family of time-varying convolutional codes using finite-field wavelets. We introduce these structures through a few examples for half-rate codes. The generalization for other rates is quite straightforward. To construct half-rate periodically time-varying wavelet convolutional codes with period two, we need a four-band wavelet system. As shown in Fig. 12, the message sequence is divided into two sequences: odd and even. Then, each of these sequences is encoded by a different time-invariant rate-1/4 wavelet convolutional encoder. This results in two different codes and . Finally, the codeword is obtained by adding the and . With this setup, two codewords we note that the time-varying code is a direct sum of the . It is important to notice that two code spaces: Fig. 12 is only one out of several configurations of a four-band wavelet system that can be used to construct time-varying codes. , , and For example, we could have used three filters to generate one code and , , and to construct another one. We also could have introduced prefilters to each of the rate-1/4 time-invariant encoders, as we did previously for time-invariant codes in Section III. The configuration in Fig. 12 is employed because we are interested in generating self-dual time-varying codes. Often, in block coding, self-

TLFeBOOK

1888

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

dual block codes attain higher minimum distances than nonself-dual ones. Thus, in our search for good convolutional codes, we are likely to find a higher minimum free distance if we search through self-dual convolutional codes. specify an orFact 2: If the filters thogonal filterbank, then the structure in Fig. 12 constructs a time-varying self-dual code. Proof: Using the argument used in Proposition 3, one can show that is a self-orthogonal code. A similar statement can be made for . Furthermore, because of (18), the two codes and are orthogonal to each other. In other words, for all and . Since the direct sum of two rate-1/4 self-orthogonal codes that are orthogonal to each other results in a self-dual code, we conclude that is a self-dual convolutional code. To find codes of constraint length three in Fig. 12, we search through all four-band orthogonal filterbanks with filters whose lengths are at most eight. We find that choosing the orthogonal filterbank as and (19) of results in a self-dual periodically time-varying code . Note that the filter coefficients are period two with binary; however, they are presented here in a hexadecimal form for notational simplicity. A code with a minimum free distance of four is attained (this information is obtained from the trellis diagram of the code), which is the best we can generate using orthogonal filterbanks. Note that other configurations (even those for this rate employing prefilters) do not result in a better and constraint length three. From Fig. 12, it is clear that the code is periodically time-varying with period two. This also becomes clear from the trellis diagram of the code. The most exciting feature of the time-varying wavelet convolutional codes is their new type of trellis structure. There exist multiple definitions for a code trellis. A conventional of rank is defined as an ordered time axis trellis . Each edge (branch) connects a vertex at depth to one at depth for . Here, is the length of the code measured in trellis sections. The set of vertices at depth is denoted by , and is the collection of all vertices in the trellis. The set of edges connecting vertices at depth to those at depth is shown by , and the set of all edges is denoted by . A convolutional code has a well-defined trellis diagram. Every codeword in a convolutional code is associated with a unique path. A trellis diagram for a -state rateconvolutional code ( is the constraint length as defined previously) has vertices (states) at each time . There are branches leaving and entering each state, and each is labeled by bits (this is true for a conventional trellis but not for the from trellis of time-varying wavelet codes). To construct the trellis diagram of the self-dual code , we first simplify the encoder of Fig. 12 as Fig. 13(a), where , and . Then, we represent the simplified encoder by its polyphase structure,

as shown by Fig. 13(b). By applying the noble identity to the configuration of Fig. 13(b), we obtain the encoder structure of Fig. 13(c), where . Using the representation of Fig. 13(c), we obtain the trellis diagram of the resulting code as in Fig. 14. In drawing this trellis, we use the convention that a solid line denotes the output generated by the input bit zero “0”, and a dotted line denotes the output generated by the input bit one “1.” The trellis has two types of states: active and neutral states. These active and neutral states are represented in the diagram by black and gray nodes, respectively. The active states are those that generate the codeword bits; in contrast, the neutral states do not produce any codeword bits. Since we can divide the states of the trellis into two groups, we call it a bipartite trellis structure. In a bipartite trellis, there is no direct path between two active states. All paths between active states must go through a neutral state. The trellis is obviously periodic with period two. Within to ), there is one active each period (from ) and one neutral section trellis section (the interval (the interval ). Similar to a (nonbipartite) standard trellis for half-rate codes, only one bit is encoded at each time transition. The code described by Fig. 12 may be considered either as a periodically time-varying rate-1/2 code with period two or as a time-invariant rate-2/4 “unit memory” code. As we will see shortly, the former viewpoint results in substantially less decoding complexity. We want to clarify that the bipartite property of the trellis is due to the downsampler by a factor of 2 in Fig. 13(c), and it does not exist in other encoders. The main advantage of time-varying wavelet convolutional codes from a practical point of view is their fast decoding capability caused by the bipartite trellis. The decoding procedure on a bipartite trellis is different from that on a standard (nonbipartite) trellis. In Viterbi decoding, as we proceed in time from left to right on the trellis, whenever we hit an active section, we perform the same computations that are normally done for a standard trellis. However, whenever we are on a neutral section, we need to perform only one two-way comparison at the input to each state, and no branch metric computation is required. Furthermore, it is crucial that the decoding procedure must end at an active state rather than at a neutral state. In the following, we show why time-varying wavelet codes can be decoded faster than existing time-invariant codes having the same state complexity profile. Consider again the half-rate time-invariant code of constraint length three (The number of states is eight.) of Example III. This code has . It is the best time-invariant code of rate-1/2 and constraint length three. The trellis for this code is a conventional trellis in which every section of the trellis is active, and each branch is associated with two bits of output. This trellis has the same state space complexity profile and edge-space cardinality as the bipartite trellis of the code described earlier by (19). However, the Viterbi algorithm for decoding the time-invariant code (conventional code trellis) takes almost twice as much time as required by the wavelet time-varying code (bipartite code trellis). To verify this, we recall the Viterbi algorithm [19], which is described on a conventional trellis as the ( is the time index, and following: Suppose each vertex is the vertex index in the set ) in the trellis is to be assigned a

TLFeBOOK

FEKRI et al.: CONVOLUTIONAL CODES USING FINITE-FIELD WAVELETS

Fig. 13.

1889

(a) Simplified version of Fig. 12. (b) Polyphase representation of the simplified encoder. (c) Modified version of (b) after applying the noble identity.

Fig. 14. Bipartite trellis diagram for the time-varying wavelet convolutional code in Fig. 12.

value . The vertex values are computed by the following procedure: and . Step 1) Set Step 2) At time , compute the partial path metrics for all paths entering each vertex. For a description of this computation, see [19]. equal to the best partial path metric Step 3) Set (the survivor path metric) entering the vertex corat time . responding to vertex , increment , and return to step 2. Step 4) If

Once all vertex values have been computed, the surviving branches specify the unique path corresponding to the maximum likelihood codeword. For a bipartite trellis of period two, step two is skipped on every other iteration. Since the computation in step two is considerably greater than that of step 3 (which requires only one comparison), we conclude that all half-rate time-varying wavelet convolutional codes of period two can be decoded almost twice as fast as time-invariant codes of the same rate and constraint length (i.e., state complexity profile). More precisely, by the following argument, we show that the wavelet code can be decoded 1.66 times faster. Assuming we can use quantized metrics, a table lookup procedure can be applied for computing the metrics. Table I summarizes the computational complexity for two consecutive sections, in which is the number of the states in each section. Using Table I, we conclude that the total operations of decoding four bits for the wavelet code are equal to 6S versus 10S for the conventional codes. Therefore, the wavelet can be decoded 1.66 times faster. has a faster Although the wavelet time-varying code decoder, its minimum free distance is smaller than the best timeinvariant code of the same rate and state complexity profile, . This results in a slightly higher bit error which has compared with the best time-invariant code. rate of Now, in the hope of finding better codes, we considered doubly even self-dual codes. Our motivation was that a doubly

TLFeBOOK

1890

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE I TRELLIS COMPLEXITY COMPARISON

in which is the intersection operation. Because of the orthogonal property of the filterbank, . This implies if we have . Therefore the field is (20)

Fig. 15. Example of a rate-1=L wavelet convolutional encoder. (a) Encoder structure. (b) Simplified structure of (a).

even self-dual code demonstrates a better bit error rate performance because it is likely to attain a superior weight enumerator to a singly even self-dual code [24]–[26]. Let us first consider the following lemma. Lemma 2: Suppose the set of functions specifies an -band orthogonal wavelet system over . Let us employ two of these functions to conconvolutional encoder shown in Fig. 15. struct the rateis doubly even orthogonal if the filter Then, the code is doubly even (i.e., has a weight that is multiple of four). Proof: Using Proposition 3, one can verify that is an orthogonal code. Now, we claim that is doubly even, provided is a doubly even filter. First note that if the binary that vector is defined as the binary sum of two vectors , , where then we have is the intersection operation, and denotes the weight of the sequence . If two vectors and are orthogonal , we have in . With a similar argument, we now prove that is doubly even. Let the such that integer be the weight of the filter (by the hypothesis). Since , we conclude that in which is the , and is one arbitrary weight of the message sequence . codeword in . This implies that In Corollary 2, we showed that every two-channel orthogonal filterbank can be used for constructing self-dual convolutional codes. However, we now prove that the structure of Fig. 7 cannot generate a binary doubly even (type II) self-dual convolutional code. Another proof for this is given by [1], where the authors have shown that there exists no binary doubly even time-invariant self-dual convolutional code. Lemma 3: No binary doubly even self-dual convolutional code can be constructed by a two-channel orthogonal filterbank, i.e., no time-invariant doubly even self-dual convolutional code exists. Proof: Consider the encoder shown by Fig. 7 that generates half-rate time-invariant codes. By Corollary 2, we know that we should remove the prefilter from the encoder strucand be the outputs of ture to obtain a self-dual code. Let and , respectively. In addition, let the filters be the weight of the codeword . Then, we have

Further, note that in any two-channel orthogonal filterbank, for some integer . Addition, we conclude that ally, since in a binary field. Here, the integer is the weight of the message sequence . Similarly, we can write . Consequently, using (20), we obtain (21) is not zero for any (i.e., if is an odd Thus, number, ). Since no time-invariant doubly even self-dual convolutional code exists, we should search through the time-varying codes to find doubly even ones. Our search of different orthogonal filterbank failed to find any doubly even self-dual code of constraint length three using the half-rate time-varying encoder in Fig. 12. We conclude that there exists no doubly even, self-dual, periodically time-varying convolutional code of constraint length three and period two. This, of course, does not exclude the possibility of the existence of such codes for higher constraint lengths. To construct doubly even self-dual wavelet convolutional codes, we look for half-rate periodically time-varying codes of period four. To do that, we need an eight-band wavelet system, as shown in Fig. 16. The message sequence is divided into four sequences, each encoded by a time-invariant rate-1/8 wavelet convolutional encoder. This results in four different codes . As shown in Fig. 16, every codein the code is the binary sum of the codewords word from these four code spaces. From this setup, it is clear that the time-varying code is a direct sum of these four code spaces, i.e., . Fig. 16 is only one out of several configurations of an eight-band wavelet system to construct half-rate periodically time-varying code of period four. The configuration in Fig. 16 is employed because we are interested in generating a doubly even self-dual time-varying code. This is explained in the following. Fact 3: The structure in Fig. 16 constructs a time-varying form an orthogonal filself-dual code if the filters terbank. Proof: Similar to the proof of Fact 2, one can show that the are self-orthogonal and that they are orthogcodes onal to each other. Hence, is the direct sum of four rate-1/8

TLFeBOOK

FEKRI et al.: CONVOLUTIONAL CODES USING FINITE-FIELD WAVELETS

1891

Fig. 17. Quadripartite trellis diagram for the time-varying wavelet convolutional code in Fig. 16. Fig. 16. Diagram of a half-rate periodically time-varying wavelet convolutional encoder with period four.

self-orthogonal codes that are orthogonal to each other. This implies that the half-rate convolutional code is self-dual. We search through all eight-band orthogonal filterbanks with filters of length at most eight. We find that choosing the orthogonal filterbank in Fig. 16 with (the filter coefficients are binary; however, they are presented in a hexadecimal form)

and

(22)

results in a doubly even, self-dual, periodically time-varying convolutional code of period four and constraint length three. The doubly even self-dual property comes from the fact is a doubly even self-orthogonal code by that each of Lemma 2. Therefore, code , which is the direct sum of four rate-1/8 doubly even self-orthogonal codes that are orthogonal to each other, is a doubly even self-dual code. , the minBecause of the doubly even property of imum-free distance of the code is either four or eight. By that has weight four, inspection of the generator of the code . It can be verified that is the we conclude that best code that we can generate using orthogonal filterbanks as in Fig. 16. Note that other configurations (employing prefilters) do for this rate and constraint length three. not result in better The trellis diagram of the self-dual code is depicted in Fig. 17, in which eight output bits (the branch labels) are represented in hexadecimal form. We note that the trellis for this code is quadripartite. It is obvious that the trellis is periodically time-varying with period four. Within each period (from to ), there is one active trellis section, , and three neutral sections (the interval the interval ). Similar to a standard (nonquadripartite) trellis for half-rate codes, only one bit is encoded at each transition in time. The code described by Fig. 16 may be considered either as a periodically time-varying rate-1/2 code with period four or a

time-invariant rate-4/8 “unit memory” code. The former viewpoint has significantly less decoding complexity. We follow a similar argument that we made earlier for to show the reduced trellis decoding complexity of . It is clear from Fig. 17 that the trellis of has the same state space complexity profile and edge-space as the conventional trellis of the cardinality (time-invariant) code described earlier by Example 3. However, in each period of the trellis diagram, three out of four trellis sections are inactive. Therefore, in the Viterbi algorithm, step two is skipped three times out of four iterations. By the same argument that we made for the time-varying code of period two, we conclude that all half-rate, periodically time-varying wavelet convolutional codes of period four are decoded almost four times as fast as time-invariant codes of the same rate and constraint length (i.e., state complexity profile). , the wavelet time-varying code Like the code has a minimum-free distance that is smaller than the best time-invariant code of the same rate and state complexity . This results in a degradation in bit profile, which has error rate performance of these two wavelet codes. This might give the impression that time-varying wavelet coding always results in a performance loss with respect to time-invariant coding. However, in the following examples, we show that the time-varying wavelet code can have the same minimum-free distance as the time-invariant code of the same rate and constraint length. Fig. 18(a) shows the general block diagram of a half-rate, periodically time-varying, wavelet convolutional encoder with pespecifies a riod two. The set of functions four-band orthogonal wavelet system. We have also introduced and to maintain the generality of the prefilters structure. We know from Proposition 1 that this encoder is noncatastrophic for any arbitrary choices of the prefilters. We found that the following choice for the wavelet system

TLFeBOOK

(23)

1892

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE II NUMBERS OF CODE SEQUENCES OF WEIGHTS d = 8, 10, 12, 14, AND 16

TABLE III COMPARISON OF THE DECODING COMPLEXITY OF THE TIME-VARYING WAVELET CODES OF PERIOD TWO WITH CONVENTIONAL CONVOLUTIONAL CODES

Fig. 18. Diagram of a half-rate, periodically time-varying, wavelet = 8, and a constraint length five. convolutional encoder with period two, d (a) Original encoder structure. (b) Simplified encoder structure.

and the prefilters with -transforms and result in a code with constraint length five and minimum-free distance eight. Note that although the prefilters are anticausal, the resulting filters and are causal in the simplified structure shown is simin Fig. 18(b). The trellis diagram of the code ilar to Fig. 14 in that the bipartite trellis is periodic with period two, but there are 32 states at each time stage in the trellis. The code has the same minimum-free distance as the best time-invariant code of the same rate and constraint length; howcode can be decoded almost twice as fast ever, the because of its inactive trellis sections. To generate a wavelet time-varying convolutional code with constraint length four, we chose the following wavelet system:

(24) and the and for prefilters. This results in a 16-state wavelet convolutional code that has a minimum distance of seven. The best time-invariant convolutional code with constraint length of four also has a minimum distance of seven [8]. However, the wavelet time-varying convolutional code can be decoded faster. An interesting comparison can be made between and the nonwavelet time-varying codes that are found by Calderbank et al. [1] and Lauer [10]. Calderbank et al. [1] used the Miracle Octad Generation (MOG) representation of the (24,12,8) Golay code to form a periodically time-varying convolutional of period four having 16 states. Code is a doubly code even self-dual convolutional code. Lauer [10] constructed a partial-unit-memory convolutional code. Both of these codes have as the 32-state wavelet the same minimum distance time-varying code. However, computing the weight distribution, we can conclude that the wavelet code has a better per-

formance. Table II shows the numbers of code sequences of . The numbers of code seweights quences of weights 8 (the multiplicity ) for the wavelet code is half that of PUM, whose multiplicity is about half that of the Golay convolutional code. As it can be observed by Table II, the PUM code outperforms the Golay convolutional code because of its weight distribution. Similarly, we can conclude that the wavelet code is better than the PUM code. Although the PUM code is nominally an eight-state code, its actual trellis complexity is somewhat greater than that of the 16-state Golay code because it cannot be expressed as a time-varying half rate code with less than 64 states [1]. To compare the trellis complexity of these codes, we use the same methodology as [1]. Table III presents the trellis complexity of the Golay convolutional code, the PUM code, and the wavelet code. From Tables II and III we can conclude that the is better than PUM code in performance and has almost the same is computational complexity. Furthermore, although slightly more complex than the Golay convolutional code, it has a better performance. VI. RATE-ADAPTIVE WAVELET CONVOLUTIONAL CODES In designing an error-correcting system for a specified channel, we choose a code with a fixed rate and correction capability that adapts to the worst channel condition. However, in many applications, the channel is time-varying. To maintain an acceptable quality of service, we need to change the coding rate during transmission to adapt to the channel condition. Wireless communication is an example for an application needs adaptive code rates to support different services. To achieve adaptive error control coding, we propose wavelet convolutional codes whose encoder is shown in Fig. 19. code, we According to Fig. 19, to construct a rate apply messages to out of channels of the -band orthogonal wavelet system and feed zero inputs to the rest of the channels. The maximum achievable rate from this system . To reduce the rate, we simply replace one or several is of the message inputs with additional zero inputs. The lowest

TLFeBOOK

FEKRI et al.: CONVOLUTIONAL CODES USING FINITE-FIELD WAVELETS

1893

Fig. 20. Syndrome generator for wavelet convolutional encoder of rate K=L in Fig. 19. Fig.

19. Rate-adaptive .

wavelet

convolutional

encoder

of

rates

[1=L; 2=L; . . . ; K=L]

and

rate is generated when only the first channel receives a message sequence. Therefore, the range of achievable rates is . We know that the Viterbi decoding algorithm is effective for low-rate convolutional codes. Since the complexity of a conincreases exponentially as volutional encoder of rate grows, the implementation of the Viterbi decoder becomes considerably complicated. We propose to use the alternative decoding technique called syndrome decoding [27], [28]. Syndrome decoding is a maximum likelihood decoding technique for convolutional codes using a trellis. Since the trellis is drawn based on the parity check matrix, the trellis is called a syndrome trellis. The wavelet code can also be defined by an parity check matrix . It can be shown that the matrix can also be written as

.. .

.. .

.. .

is defined such that The matrix overall constraint length of the dual code by

.. . (25) . The can also be obtained

(26) is equal to : the constraint In [17], the author has shown that length that is defined by the generator. The parity check matrix for a rate convolutional code is

.. .

In (27),

.. .

(27) .. .

..

is defined as

.. .

.. .

.. .

.. .

(29)

where is the coefficient of of the in (25). The syndrome trellis corresponds to the codeword . For each codeword , we construct the path of the trellis, which passes at through the node : level (30) To construct the syndrome trellis, vertex of level is confor . Hence, at each nected to vertices level of the trellis, one bit of the code-word is corrected. For ML decoding of -bits of a codeword, levels of the trellis vertices. are needed, and each level has at most The trellis is periodic with period . In the syndrome trellis, there is a one-to-one correspondence between the code sequence and the path in the trellis. By applying syndrome decoding, we , which is the corrected version of the codeword. get the Hence, to obtain the information sequence, we need to compute . To perform syndrome decoding for the wavelet codes, we need to compute the parity check matrix. Fig. 20 shows the syndrome generator, from which we obtain the parity check matrix. In the following, we explain the syndrome decoding for the adaptive rate wavelet codes by using a particular example. Example 4: Consider a three-band orthogonal wavelet system. As shown in Fig. 21, we can have wavelet codes of rates 1/3 and 2/3 using a three-band wavelet. To find codes with better performance, we also use a prefilter. Fig. 22 shows the syndrome generators of these codes. We chose the wavelet system as follows:

. (31)

is obtained by (28)

and the prefilter as

TLFeBOOK

.

1894

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 21.

Rate-adaptive wavelet convolutional encoder. (a) Rate 1/3. (b) Rate 2/3.

Fig. 22. Syndrome generator for wavelet convolutional code. (a) Rate 1/3. (b) Rate 2/3.

The parity check matrix for the wavelet convolutional code of rate 1/3 with constraint length of 2 is obtained by

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

. (32)

where is the th coefficient of the . is equal to . It can be seen that the Hence, parity check matrix is periodic with period 3, and so is the trellis. Therefore, we draw the trellis for three levels. Fig. 23(a) shows the syndrome trellis. The horizontal branches correspond to code bit zero, and the others correspond to code bit one. To obtain the trellis for rate 2/3, we first note that the parity check matrix is equal to

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

.

..

. (33)

Fig. 23. Trellis for syndrome decoder for wavelet convolutional code. (a) Rate 1/3. (b) Rate 2/3.

By comparing (32) and (33), one can easily verify that the later matrix has every other row of the former one. Since we draw the syndrome trellis based on the parity check matrix, the syndrome trellis for rate 2/3 will be a subtrellis of the lower rate trellis. It can be seen that the parity check matrix for a code rate of 1/3 is also periodic with period 3. The trellis for this rate is drawn in Fig. 23(b). It is worth noting that although the wavelet convolutional encoder produces different rates, we are still able to use one trellis for its decoding. Because of the wavelet encoder structure, we

TLFeBOOK

FEKRI et al.: CONVOLUTIONAL CODES USING FINITE-FIELD WAVELETS

1895

draw the trellis for the lowest rate, which has the maximum number of states. Then, a higher rate code is decoded by the subtrellis of the lower trellis. As we saw in the example, the number of states for rate 1/3 is the square root of the number of the states for rate 2/3. To obtain the trellis at the rate 2/3 from that of rate 1/3, we can simply keep the common vertices with their adjoint branches and eliminate the rest of the states in the original trellis (rate 1/3) with their corresponding branches. VII. CONCLUSION AND SUGGESTIONS FOR FURTHER WORK In this paper, the finite-field wavelet approach to the construction of convolutional codes has been presented. Two classes of convolutional codes (time-invariant and time-varying) have been constructed based on using finite-field filterbank systems. We explained algebraic properties of the wavelet encoders. wavelet convolutional An important property of the rate encoder is that they are basic encoders and they are nonwavelet catastrophic. It has been shown that any rateconvolutional encoder is also minimal-basic. To construct a self-orthogonal code, any orthogonal wavelet system ratecan be used. Time-varying wavelet convolutional codes have been introduced. We presented some examples that show, for some constraint lengths, the minimum distance of the wavelet convolutional code is as good as the best time-invariant convolutional code, but its decoding is twice as fast as the decoding of the time-invariant convolutional codes. We proved that an orthogonal filterbank generates a self-dual time-varying code. While no time-invariant doubly even self-dual convolutional code exists, we showed that doubly even orthogonal can be obtained time-varying convolutional codes of rate by adding some constraints to the filters of the -band orthogonal wavelet system. We also introduced rate-compatible wavelet convolutional codes. We compared the performance of these codes with punctured convolutional codes for fixed decoding complexity and observed that the punctured convolutional codes have better performance than the rate-compatible wavelet convolutional codes. However, we expect that if we replace orthogonal wavelets with biorthogonal wavelet systems, we can improve the performance of the wavelet codes. This is because the filters of the biorthogonal filterbanks are much more relaxed than the orthogonal ones. REFERENCES [1] A. R. Calderbank, G. D. Forney, and A. Vardy, “Minimal tail-biting trellises: The golay code and more,” IEEE Trans. Inf. Theory, vol. 45, pp. 1435–1455, Jul. 1999. [2] J. W. Woods, Subband Coding of Images. Boston, MA: Kluwer, 1991. [3] M. R. Portnoff, “Time-frequncy representation of digital signals and systems based on short-time fourier analysis,” IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-28, pp. 55–69, Feb. 1980. [4] F. Fekri, “Arbitrary rate maximum-distance separable wavelet codes,” in Proc. IEEE Conf. Acoust., Speech, Signal Process., vol. 3, 2002, pp. 2253–2256. [5] F. Fekri, R. M. Mersereau, and R. W. Schafer, “Two-band wavelets and filterbanks over finite fields with connections to error control coding,” IEEE Trans. Signal Process., vol. 51, no. 12, pp. 3143–3151, Dec. 2003. , “Convolutional coding using finite-field wavelet transforms,” in [6] Proc. Thirty-Eighth Annu. Allerton Conf., Urbana, IL, Oct. 2002. [7] D. J. Costello, “Free distance bounds for convolutional codes,” IEEE Trans. Inf. Theory, vol. IT-20, pp. 356–365, May 1974.

[8] Q. Hu and L. C. Perez, “Some periodic time-varying convolutional codes with free distance achieving the heller bound,” in Proc. IEEE Int. Symp. Inf. Theory, 2001, pp. 247–247. [9] F. Fekri, S. W. McLaughlin, R. M. Mersereau, and R. W. Schafer, “Wavelet convolutional codes with bipartite trellises,” in Proc. IEEE Int. Symp. Inf. Theory, 2001, pp. 245–245. [10] G. S. Lauer, “Some optimal partial unit-memory codes,” IEEE Trans. Inf. Theory, vol. IT-25, pp. 240–242, Mar. 1979. [11] J. B. Cain and G. C. Clark, Jr, “Punctured convolutional codes of rate (n 1)=n and simplified maximum likelihood decoding,” IEEE Trans. Inf. Theory, vol. IT-25, pp. 97–100, Jan. 1979. [12] J. Hagenauer, “Rate-compatible punctured convolutional codes(RCPC codes) and their applications,” IEEE Trans. Commun., vol. 36, no. 4, pp. 389–399, Apr. 1988. [13] F. Fekri, R. M. Mersereau, and R. W. Schafer, “Realization of paraunitary filter banks over fields of characteristic two,” in Proc. IEEE Int. Conf. Acoust. Speech, Signal Process., 2000, pp. 492–495. , “Theory of paraunitary filter banks over fields of characteristic [14] two,” IEEE Trans. Inf. Theory, vol. 48, no. 11, pp. 2964–2979, Nov. 2002. [15] R. Johanesson and Z. Wan, “A linear algebra approach to minimal convolutional encoders,” IEEE Trans. Inf. Theory, vol. 39, no. 4, pp. 1219–1233, Jul. 1993. [16] J. L. Massey and M. K. Sain, “Inverse of linear sequential circuits,” IEEE Trans. Comput., vol. C-17, pp. 330–337, Apr. 1968. [17] G. D. Forney, “Convolutional codes I: Algebraic structure,” IEEE Trans. Inf. Theory, vol. IT-16, pp. 720–738, Nov. 1970. [18] R. Johannesson and K. S. Ziagrov, Fundamentals of Convolutional Coding. New York: IEEE, 1999. [19] S. B. Wicker, Error Control Systems for Digital Commuinication and Storage. Englewood Cliffs, NJ: Prentice-Hall, 1995. [20] G. D. Forney, “Structural analysis of convolutional codes via dual codes,” IEEE Trans. Inf. Theory, vol. IT-19, pp. 512–518, Jul. 1973. [21] F. J. MacWilliams and N. J. A. Sloane, The Theory of Error Correcting Codes Amsterdam, The Netherlands, 1977. [22] E. Rains and N. J. A. Sloane, Self-Dual Codes, V. S. Pless and W. C. Huffman, Eds. Amsterdam, The Netherlands, 1998. [23] R. Palazzo, “A time-varying convolutional encoder better than the best time invariant encoder,” IEEE Trans. Inf. Theory, vol. 39, no. 3, pp. 1109–1109, May 1993. [24] M. Harada, “New extremal self-dual codes of lengths 36 and 38,” IEEE Trans. Inf. Theory, vol. 45, no. 7, pp. 2541–2543, Nov. 1999. [25] R. A. Brualdi and V. S. Pless, “Weight enumerators of self-dual codes,” IEEE Trans. Inf. Theory, vol. 37, no. 4, pp. 1222–1225, Jul. 1991. [26] S. T. Dougherty and M. Harada, “New extremal self-dual codes of length 68,” IEEE Trans. Inf. Theory, vol. 45, no. 6, pp. 2133–2136, Sep. 1999. [27] T. Yamada, H. Harashima, and H. Miyakawa, “A new maximum likelihood decoding of high rate convolutional codes using trellis,” Electron. Commun. Japan, vol. 66, pp. 11–16, Jul. 1983. [28] E. L. Blokh and V. V. Zyablov, “Coding of generalized concatenated codes,” Problemy Peredachi Informatsii, vol. 10, pp. 45–50, Jul.–Sep. 1974.

0

Faramarz Fekri (M’00–SM’03) received the B.Sc. and M.Sc. degrees from Sharif University of Technology, Tehran, Iran, in 1990 and 1993, respectively, and the Ph.D. degree from the Georgia Institute of Technology (Georgia Tech), Atlanta, in 2000. From 1995 to 1997, he was with the Telecommunication Research Laboratories (TRLabs), Calgary, AB, Canada, where he worked on multicarrier spread spectrum systems. Since 2000, he has been on the faculty of the School of Electrical and Computer Engineering at Georgia Tech. His current research interests lie in the general field of signal processing and communications, in particular, wavelets and filter banks, error control coding, cryptography, and communication security. In the past, he conducted research on speech and image processing. Dr. Fekri received the Best Ph.D. Thesis Award from Georgia Tech in 2000 for his work on finite-field wavelets and their application to error control coding. He also received the National Science Foundation CAREER award in 2001.

TLFeBOOK

1896

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Mina Sartipi (S’02) received the B.Sc. degree from Sharif University of Technology, Tehran, Iran, in 2001. In 2003, she received the M.S. degree in electrical and computer engineering from the Georgia Institute of Technology (Georgia Tech), Atlanta, where she is currently pursuing the Ph.D. degree. Her research interests lie in the general area of communications, in particular, wavelets and filterbanks, error control coding, and communication networks.

Russell M. Mersereau (S’69–M’73–SM’78–F’83) received the S.B. and S.M. degrees in 1969 and the Sc.D. in 1973 from the Massachusetts Institute of Technology, Cambridge. He joined the School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, in 1975. His current research interests are in the development of algorithms for the enhancement, modeling, and coding of computerized images, synthesis aperture radar, and computer vision. In the past, this research has been directed to problems of distorted signals from partial information of those signals, computer image processing and coding, the effect of image coders on human perception of images, and applications of digital signal processing methods in speech processing, digital communications, and pattern recognition. He is the coauthor of the text Multidimensional Digital Signal Processing (Englewood Cliffs, NJ: Prentice-Hall, 1984). Dr. Mersereau has served on the Editorial Board of the PROCEEDINGS OF THE IEEE and as Associate Editor for signal processing of the IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING and the IEEE SIGNAL PROCESSING LETTERS. He was the Vice President for Awards and Membership of the IEEE Signal Processing Society and a member of its Executive Board from 1999 to 2001. He is the corecipient of the 1976 Bowder J. Thompson Memorial Prize of the IEEE for the best technical paper by an author under the age of 30, a recipient of the 1977 Research Unit Award of the Southeastern Section of the ASEE, and three teaching awards. He received the 1990 Society Award of the Signal Processing Society and an IEEE Millenium Medal in 2000.

Ronald W. Schafer (S’62–M’67–SM’74–F’77) received the B.S.E.E. and M.S.E.E. degrees from the University of Nebraska, Lincoln, in 1961 and 1962, respectively, and the Ph.D. degree from the Massachusetts Institute of Technology, Cambridge, in 1968. From 1968 to 1974 he was a member of the Acoustics Research Department, Bell Laboratories, Murray Hill, NJ, where he was engaged in research on speech analysis and synthesis, digital signal processing techniques, and digital waveform coding. Since 1974, he has been on the faculty of the Georgia Institute of Technology (Georgia Tech), Atlanta, where he is now John and Marilu McCarty Professor and Regents Professor of electrical and computer engineering. His current research interests include speech and video processing, nonlinear signal processing systems, applications of signal processing in multimedia communication systems, and applications of signal processing in biology and medicine. He is co-author of six textbooks, including Discrete-Time Signal Processing (Upper Saddle River, NJ: Prentice-Hall, 1989) and DSP First: A Multimedia Approach (Upper Saddle River, NJ: Prentice-Hall, 1997). Dr. Schafer has been active in the affairs of the IEEE Signal Processing Society, having served as Associate Editor of the IEEE TRANSACTIONS ON SIGNAL PROCESSING, a member of several committees, Vice-President and President of the Society, and Chairman of the 1981 ICASSP. He currently serves as Chair of the IEEE Education Medal Committee. He is a Fellow the Acoustical Society of America and a member of the National Academy of Engineering. He was awarded the Achievement Award and the Society Award of the IEEE ASSP Society in 1979 and 1983, respectively; the 1983 IEEE Region III Outstanding Engineer Award; and he shared the 1980 Emanuel R. Piore Award with L. R. Rabiner. In 1985, he received the Class of 1934 Distinguished Professor Award at Georgia Tech, and he received the 1992 IEEE Education Medal. In 2000, he received the IEEE Third Millennium Medal, and he shared the IEEE Signal Processing Society 2000 Education Award.

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1897

BER Sensitivity to Mistiming in Ultra-Wideband Impulse Radios—Part II: Fading Channels Zhi Tian, Member, IEEE, and Georgios B. Giannakis, Fellow, IEEE

Abstract—We investigate timing tolerances of ultra-wideband (UWB) impulse radios. We quantify the bit-error rate (BER) sensitivity to epoch timing offset under different operating conditions, including frequency-flat fading channels, dense multipath fading channels, multiple access with time hopping, and various receiver types including sliding correlators and RAKE combiners. For a general correlation-based detector, we derived in Part I unifying expressions for the decision statistics as well as BER formulas under mistiming, given fixed channel realizations. In Part II, we provide a systematic approach to BER analysis under mistiming for fading channels. The BER is expressed in terms of the receiver’s energy capture capability, which we quantify under various radio operating conditions. We also develop the optimal demodulator in the presence of timing offset and show a proper design of the correlation-template that is robust to mistiming. Through analyses and simulations, we illustrate that the reception quality of a UWB impulse radio is highly sensitive to both timing acquisition and tracking errors. Index Terms—Mistiming, optimal correlator, performance analysis, RAKE receiver, synchronization, ultra-wideband radio.

I. INTRODUCTION ONVEYING information over Impulse-like Radio (IR) waveforms, ultra-wideband (UWB) IR comes with uniquely attractive features: low-power-density carrier-free transmissions, ample multipath diversity, low-complexity baseband transceivers, potential for major increase in capacity, and capability to overlay existing radio frequency (RF) systems [1]–[3]. On the other hand, the unique advantages of UWB IR technology are somewhat encumbered by stringent timing requirements because the transmitted pulses are very narrow and have low power [4]. This two-part sequel quantifies the timing tolerances of UWB IR transmissions for a broad range of radio operation settings. The goal is to investigate the sensitivity to mistiming by quantifying the BER degradation due to both acquisition and tracking errors.

C

Manuscript received April 23, 2003; revised February 27, 2004. Z. Tian was supported by the National Science Foundation under Grant CCR-0238174. G.B. Ginnakis was supported by ARL/CTA under Grant DAAD19-01-2-011 and the National Science Foundation under Grant EIA-0324804. Parts of the work in this paper were presented at the IEEE SPAWC Conference, Rome, Italy, June 2003, and the IEEE GLOBECOM Conference, San Franscisco, CA, December 2003. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Martin Haardt. Z. Tian is with the Electrical and Computer Engineering Department, Michigan Technological University, Houghton, MI 49931 USA (e-mail: [email protected]). G. B. Giannakis is with the Electrical and Computer Engineering Department, University of Minnesota, Minneapolis, MN 55455 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845485

In Part I [5], we derived a unifying signal model for analyzing the detection performance of correlation-based receivers. Selecting different values for the model parameters led to various operating conditions in terms of channel types, time-hopping codes, and receiver structures. For pulse amplitude modulation (PAM) transmissions under various operating conditions, [5] analyzed the BER degradation induced by mistiming under fixed channel realizations. In fading channels, the instantaneous BER derived in [5] can be integrated over the joint probability density function (pdf) of the random channel parameters to obtain the average BER, which is the focus of this paper (Part II). We first lay out a procedure for evaluating the BER performance of an optimum symbol-by-symbol receiver for any random channel. The BER is expressed in terms of the receiver’s energy capture capability. The results depend on a couple of key channel statistics that can be numerically obtained by averaging over a large number of channel realizations. This general approach to BER analysis applies to any channel fading type but requires computationally intensive numerical evaluation. This motivates our focus on real-valued Gaussian fading channels, for which we derive closed-form BER for any RAKE receivers, expressed with respect to the channel statistics and the RAKE parameters. The energy capture capability of correlation-based receivers is quantified and tabulated for various radio operating conditions. Even though a realistic channel may follow other fading characteristics [6]–[9], the results here provide meaningful implications on the robustness of correlation-receivers to mistiming under different system settings. The rest of this paper is organized as follows: The ensuing Section II briefly summarizes the system model presented in [5]. Section III studies the average BER performance of an optimum symbol-by-symbol detector under general fading channels, whereas Section IV derives closed-form BER expressions under Gaussian fading channels. The derivations also suggest an (optimal) correlator receiver design strategy that is robust to mistiming. Results for pulse position modulation (PPM) are summarized in Section V. Corroborating simulations are provided in Section VI, followed by concluding remarks in Section VII. II. SIGNAL MODEL In UWB impulse radios, every information symbol is transpulses over frames (one pulse per frame mitted using of duration ). Every frame contains chips, each of du. The equivalent transmit filter is ration of symbol duration ,

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1898

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 1. Correlation-based receiver.

where is an ultra-short (so-called monocycle) pulse of duration at the nano-second scale, and the chip sequence represents the user’s pseudo-random time-hopping (TH) , . By setting code with , is scaled to have unit energy. With information-bearing binary PAM symbols being independent identically distributed (i.i.d.) with zero mean and average transmit energy per symbol , the transmitted pulse stream is

with a locally A correlation-based receiver correlates , time-shifted by a nomgenerated correlation-template inal propagation delay , to produce the sufficient statistic for symbol detection

(1)

to represent a generic We select the correlation-template fingers, as shown in Fig. 1. The RAKE receiver with RAKE tap delays are design parameters that could be, but are not necessarily, chosen among the channel path de. A full RAKE arises when , and each lays is matched to one of the delayed paths, while corresponds to a “partial RAKE,” which may be less effective in energy capture but is computationally more affordable. In particular, the sliding correlator can be considered as a “RAKE-1” [9]. The RAKE weights can receiver with be selected to represent maximum ratio combining, equal-gain combining, or other linear combining techniques. For all these in (4) is replaced by combiners, the correlation template . , Let us denote the timing mismatch as where , and . The parameters and indicate the breakdown of mistiming into acquisiis limited to tion and tracking errors, respectively. Notice that finite values since timing is resolvable only within a symbol duration. Defining the normalized auto-correlation function of as , we combine (2) and (4) to reach a unifying expression for the detection statistic [5]

After propagating through a multipath channel, the received signal takes on the general form

(2) where is the total number of propagation paths, each with gain being real-valued with phase shift 0 or and delay satisfying , . The channel is random and quaand remaining invariant within sistatic, with one symbol period but possibly changing independently from consists of both symbol to symbol. The additive noise term ambient noise and multiple-access interference (MAI) and is , , and . We focus on perindependent of formance evaluation of the desired user, treating the composite noise as a white Gaussian random process with zero-mean . and power spectral density To isolate the channel delay spread, we define relative path for ; the maximum delay spread is delays thus . We select and set either or to avoid inter-symbol interference (ISI) when perfect timing can be acquired. With these definitions, the composite channel formed by convolving the physical , channel with the pulse is given by and the equivalent received symbol-waveform of duration can be expressed as , which simplifies to (3)

(4)

(5) where . For convenience, let represent the noise-free (signal) component of the decision statistic. , the When the RAKE taps are normalized by noise term is Gaussian with zero mean and variance . Equation (5) subsumes various operating conditions in , and ), TH codes , terms of channel types ( , , and ). and receiver structures ( ,

TLFeBOOK

TIAN AND GIANNAKIS: BER SENSITIVITY TO MISTIMING IN ULTRA-WIDEBAND IMPULSE RADIOS—PART II

In Part I [5], we derived a unifying BER expression under mistiming for fixed channel realizations. Condi, ), in (5) can tioned on ( , when be expressed as two consecutive binary symbols have the same sign, or , where there is a change of symbol signs. Here, denotes the portion of symbol energy that is collected by a receiver under both acquisition and tracking errors, whereas denotes the received energy subject to tracking error only, noting that transmission of two identical consecutive symbols . Thus, the reduces the impact of acquisition errors on instantaneous BER is given by [5]

1899

is given by . Let us define , times the channel energy . It is difficult which roughly has to express it in terms of in an exact form when TH is present. can then be simplified as The output

(9) , ) have the Depending on whether ( same or opposite signs, the effective sample energy conis given ditioned on one channel realization by subject to tracking errors only, and under both acquisition and tracking errors. The general BER expression in (6) still applies, which yields the instantaneous BER under any channel realization

(10) (6)

III. BER SENSITIVITY UNDER GENERAL FADING Focusing on symbol-by-symbol optimum reception,1 we now lay out a procedure for evaluating the BER performance under are treated as any fading channel where the taps random. We recall from (3) that the received signal within each -long interval is given by , which has the form

Equation (10) applies to both direct-path and multipath channels, regardless of TH. In the absence of TH, (10) can be considerably simplified by noting that . Introducing an -dependent ratio , we . This indicates that [c.f. (9)] have , the portion that is leaked among the total channel energy is proportional to the frame-level into the ISI term , subject to certain -induced variations . offset Correspondingly, the instantaneous BER in the absence of TH becomes

(7) Here,

, and has been defined to be the -long received symbol temshifted copies of the composite channel plate that consists of . The energy of is given by , where contains the channel energy in each frame. The optimal unit-energy correlation template under mistiming is then given by

(8) which is nothing but a circularly shifted (by ) version of to confine it within . The decision statistic for becomes , whose signal component 1Because mistiming gives rise to ISI, an optimal maximum likelihood (ML) detector requires sequence detection. On the other hand, when the receiver is unaware of the existence of mistiming, symbol-by-symbol detection is typically used. The receiver we discuss here is an optimum symbol-by-symbol ML detector under mistiming and a broad-sense optimal ML detector under perfect timing.

(11) Both (10) and (11) are expressed in terms of a given channel reor , regardless of the channel type. For any alization random channel (e.g., [6]–[9]), the average BER can be numerically evaluated by first computing and (or ) , followed by averaging (10) for each channel realization [or (11)] over a large number of realizations. Such a procedure yields for optimum detection. IV. BER SENSITIVITY UNDER GAUSSIAN FADING In addition to optimum reception using the continuous-time in (8), we are also motivated to correlation-template evaluate a general correlation-based RAKE receiver, whose pa, and can be set to reflect design traderameters , offs among detection performance, implementation complexity,

TLFeBOOK

1900

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

and robustness to mistiming. We seek the BER expressions for any RAKE receivers, with respect to the statistics of the random . channel parameters Characterizing a fading channel requires knowledge of the joint pdf of the tap fading gains. In a dense multipath environment, paths that are sufficiently far apart can be considered (at least approximately) uncorrelated, even though neighboring paths in a cluster may be correlated. For simplicity, we will assume that all tap gains are mutually uncorrelated, i.e., but not necessarily identically distributed. We focus on Gaussian fading channels, in which . each tap is Gaussian distributed with variance This assumption is justified by the well-cited statistical channel model in [7], where a bandpass channel is measured to have Gaussian-distributed gains in both the real and imaginary parts, resulting in Rayleigh distributed path envelopes. Since a baseband (carrier-less) UWB channel only has real-valued components, we discard the imaginary portion of the channel model in [7] to deduce that each is Gaussian distributed with may take either a positive zero mean. Being real-valued, or a negative sign. To facilitate our performance analysis, we revisit the general RAKE structure of Fig. 1 and introduce some definitions that will be instrumental to the design of maximal ratio combining (MRC) in the presence of mistiming. Consider, for example, in (5) without TH, i.e., [c.f. [5, (28) ]]

(12) . This output can be where rewritten as , where is the noise-free correlation output at the th finger, generated by corwith the time-shifted finger sliding correlator relating . We suppose the RAKE fingers , ) are sufficiently separated in time (e.g., pick out different deso that individual finger outputs layed path(s) ( , ) and, thus, are mutually independent across . At the th RAKE finger, associated with conditioned , ), , we define on (

With (13) and (14), comes

conditioned on (

,

) be-

(15) (16) Observe that gives rise to effective channel-re, which describe the energy capception gains ture capability of coherent detectors in the corresponding fingers, characterized by , , which are subject to both acquisition and tracking errors. On the other hand, the impact of the acquisition error is alleviated by transmission of identical consecutive symbol pairs , which leads to different effective channel . The energy capture index of the given finger gains . subject to the tracking error only is Note that when the RAKE fingers are matched to the channel , , it follows from (13) and (14) that path delays, i.e., when there is no mistiming. The equivand holds true under perfect timing in alence of any RAKE setup. A critical distinction between perfect timing and mistiming emerges: With perfect timing, there is no ISI when is chosen to be larger than the delay spread with a guard , and there is only one set of time of , i.e., that connects with effective channel gains . In contrast, mistiming induces ISI, resulting in two sets of effective channel gains and . In practice, channel gain estimation follows the synchronization task. We thus assume that the receiver has perfect knowledge and of the channel subject to mistiming (i.e., are available) in order to isolate the impact of mistiming from that of imperfect channel estimation. This assump, which is tion is quite different from directly knowing impossible to obtain without accurate synchronization. Keep in mind that MRC design is based on the available channel estimates. Deriving the average BER from (15) and (16) relies on the pdfs of the random channel parameters and . Note from (13) and (14) that each of the effective channel gains is a linear combination of Gaussian . Thus, they are also zero-mean Gaussian faded variables and can be fully characterized by their variand ances , , which are given, respectively, by [c.f. (13) and (14)]

(13)

(14)

TLFeBOOK

(17)

(18)

TIAN AND GIANNAKIS: BER SENSITIVITY TO MISTIMING IN ULTRA-WIDEBAND IMPULSE RADIOS—PART II

1901

TABLE I AVERAGE ENERGY CAPTURE FOR A SLIDING CORRELATOR

Having obtained the pdfs of the channel taps picked under mistiming, the average BER in fading channels can be obtained by integrating the instantaneous BER expressions in (6) over the joint pdf of these independent random parameters. An equivalent form of the complementary error function , and the results in [10], will be used to facilitate the analysis. A. Sliding Correlator We start with a sliding correlator operating in dense multipath. With a single RAKE finger, the normalized correlator , where the sign matches the phase weight is always . Hence, shift of the corresponding channel amplitude and , both of which are -distributed with means and . Both quantities can be deduced from . (17) and (18) by setting Based on the conditional BER in (6), and knowing and as Gaussian variables, the average BER can be derived as

B. RAKE Combining to To perform MRC, we select the RAKE multipliers match the effective channel gains at individual fingers, whereas the channel parameters are estimated symbol by symbol. Under , the channel gains are afmistiming, when . fected by the tracking error only, and we set , both acquisition On the other hand, when and tracking errors affect symbol detection, and we set . The coefficients and normalize to ensure . Correspondingly, the overall energy capture indices used in (6) become and , respectively, both of which are -distributed. The average BER of MRC reception is thus given by

(20) (19) For sliding-correlator reception, the result in (19) can be applied to other operating scenarios. The key is to identify the corand from (13) and (14). Such results responding are summarized in Table I.

which can be computed numerically by treating as a random variable uniformly distributed in , and evaluating the expectation of the integrand with respect to [10]. For MRC-RAKE reception, the result in (20) can be applied to other operating scenarios by identifying the corresponding

TLFeBOOK

1902

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE II AVERAGE ENERGY CAPTURE FOR AN MRC CORRELATOR

and of all the RAKE fingers . Such results are summarized in Table II. As a performance benchmark, the average BER for Gaussian fading channels under perfect timing is given from (19) and (20) and . The average energy capture by setting indices per finger become

under mistiming, as described in (8) and (13)–(16). A conventional approach is to borrow the optimal matched filter under perfect timing or directly use the channel information to construct detection statistics via correlation. Such an approach ignores the presence of mistiming and, thus, has little tolerance to acquisition errors, especially in the presence of TH. in (8) and channel inforIn contrast, our matched filter in (13) and (14) are all -dependent and, thus, are mation able to match the received waveform even under mistiming.

(21) , , we have . The average BER of an MRC combiner in (20) is reduced to

V. BER SENSITIVITY FOR PPM

When

and

In PPM, the information-bearing symbols time shift the UWB transmit filter by multiples of the modulation index (on the order of or ) to yield the transmitted waveform as follows:

(22) When , the BER of a sliding correlator is simplified from . (19) to We conclude this section with some remarks on the optimal symbol-by-symbol detector in Section III and the MRC RAKE receiver in Section IV. The optimal detector adopts a correlation-template matched to the channel-dependent receive symbol waveform; thus, it attains optimal matched filtering. The MRC RAKE receiver can be viewed as a discrete-time version of optimal matched filtering sampled by a collection of RAKE fingers, where the correlation templates are shifted versions of the channel-independent transmit symbol waveform , but the combining weights are matched to the aggregate channel gains on the corresponding RAKE fingers. The energy capture capability of an MRC RAKE receiver approaches that of an optimal detector when fingers are closely spaced in time delays, whereas a RAKE with a small number of fingers trades performance for reduced implementation complexity. Although other combining techniques may be considered in lieu of MRC, they may be costly to implement for UWB impulse radios. Any coherent receiver will have to at least recover the phase shift (0 or ) of in (13) and (14), which the aggregate channel amplitudes . Since the channel informain turn determine the signs of tion is required, it is justified to choose MRC over other suboptimal combiners for performance considerations, given the same number of RAKE fingers. In addition to the BER sensitivity results derived, another key result in this paper is the development of optimal receiver design

(23) Reception of PPM signals follows the same correlation principle as in (4), except that the receive template now becomes for a sliding correlator, and for a generic RAKE receiver. For binary is made when the correlation output PPM, a decision is positive, and when is negative. We analyze the BER sensitivity to UWB PPM reception based on generalizations from PAM transmissions. Extending (5) to the PPM signal in (23), a unifying expression for the detection statistic of PPM UWB is given by

(24) where . The noise variance of is , which is twice that of the PAM case, due to the use of a differential receiver template. Consider the no-TH case for clarity. Similar to (12) (c.f. [5, Sec. III.C]), the signal component of (24) can be simplified to

TLFeBOOK

TIAN AND GIANNAKIS: BER SENSITIVITY TO MISTIMING IN ULTRA-WIDEBAND IMPULSE RADIOS—PART II

(25) where

, and

. As evidenced by the analysis for PAM, the energy capture indices of each RAKE finger play an important role in assessing the BER performance in fading channels. In , , we regenerating the finger outputs place the finger receive-correlation templates with the PPM format . Unlike PAM, PPM transmissions entail four possible sets of effective channel-reception gains under mistiming, depending on the different symbol-induced pulse-shifting patterns of every consecu, ]. Mimicking (13) and (14) tive symbol pair [ and letting and , we deduce from (25) the , as quantities associated with the th RAKE finger

(26)

1903

is negative. Commade when the signal component parison of (26) and (27) with (28) and (29) indicates that the effective sampled channel gains of 0 and 1 are asymmetric: between There is an offset of and , and both and . This difference is determined by the pulse , which disappears only when . shape The BER performance of PPM now hinges on the pdfs of the four sets of aggregate channel taps in (26)–(29). When all the channel taps are Gaussian faded, their varibased on their linear ances can be obtained from relationships with , as in (17) and (18) for the PAM case. These variances characterize the energy capture capability of the corresponding RAKE fingers under mistiming. is the energy capSpecifically, ture index of detecting under a tracking error is subject only, whereas to both acquisition and tracking errors. The energy capis different, which ture capability of detecting is characterized by either or , depending on the impact of the acquisition error. With perfect channel estimation, an MRC receiver selects to match the effective channel gains at inRAKE weights , , , or dividual fingers, i.e., . Following (20), the average BER in multipath fading becomes

(27)

(30)

(28)

where , , , and are given by the variances of the Gaussian distributed random variables in (26)–(29) and can be deduced similarly to Table II. By choosing different channel and receiver parameters, this general expression applies to various operating conditions, including all the scenarios listed in Table I. VI. SIMULATIONS

(29) In (28) and (29), negative signs are imposed on the is channel-gain definitions because a decision

Based on the two general BER expressions (19) and (20) for RAKE-1 and RAKE-MRC receivers, we illustrate the BER sensitivity to timing offset in Gaussian fading channels, using the same simulation setup discussed in db, , ns, ns, [5], that is, , and , and for multipath: ,

TLFeBOOK

1904

Fig. 2.

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

BER for a direct-path flat fading channel. (a) No TH. (b) With TH.

, and

is Gaussian distributed with variance for all [11]. When an MRC re, , for , and the ceiver is used, we set are set to be normalized and proportional to finger weights , as explained in Secthe mistimed effective channel gains tion IV-B. The system parameters are kept reasonably simple to reduce the computational time without losing generality. BER Sensitivity in Direct-Path Flat-Fading Channels: Without TH, the BER curves for both additive white Gaussian noise (AWGN) and flat-fading channels are plotted in Fig. 2(a) to demonstrate the performance degradation induced by fading. A RAKE-MRC receiver is motivated even for flat-fading channels for the purpose of catching mistimed pulses rather than collecting diversity gain. RAKE-MRC is a discrete-time version of the optimum receiver used in (8), both of which use -dependent effective correlation templates to ensure channel matching under mistiming. They are expected to have very close performance in our system setup because RAKE fingers are closely spaced. It is confirmed in Fig. 2 that the use of a RAKE-MRC receiver is able to alleviate the stringent tracking requirements encountered by a RAKE-1 sliding

Fig. 3.

BER for a multipath fading channel. (a) No TH. (b) With TH.

correlator. With TH, the BER comparisons are depicted for RAKE-1 and RAKE-MRC receivers in Fig. 2(b) with reference to RAKE-1 without fading. It is seen that RAKE-MRC is able to collect more effectively the signal energy spread out by TH, thus resulting in better BER. The BER performance gap between AWGN and fading channels could reach levels as high , but the gap is less pronounced as mistiming as 20 dB at aggravates, and Gaussian fading appears to be more robust to mistiming when RAKE-MRC is employed. BER Sensitivity in Dense Multipath Fading Channels: For frequency-selective fading channels with multipath, comparisons are depicted for RAKE-1 and RAKE-MRC in Fig. 3(a), (b). Without TH, the flat-fading case is included as a reference in Fig. 3(a) to demonstrate the multipath spreading effect on symbol detection. The TH effects arise when comparing Fig. 2(b) for a direct-path channel and Fig. 3(b) for a multipath channel. Since TH acts like time shifting similar to multipath spreading, there is no noticeable difference between these two scenarios. On the other hand, when we range the hopping length to be , 6, the number of users accommodated increases, but the robustness to mistiming seems to remain stable. For

TLFeBOOK

TIAN AND GIANNAKIS: BER SENSITIVITY TO MISTIMING IN ULTRA-WIDEBAND IMPULSE RADIOS—PART II

Fig. 4. BER sensitivity to acquisition errors in fading channels. (a) Direct path. (b) Multipath.

the most-interesting multipath fading case with TH, we also plotted a BER curve generated by Monte Carlo simulations, which matches well with our analytic curves. BER Sensitivity to Acquisition: Suppose tracking errors ran. The BER sensitivity to acdomly appear over quisition, averaged over a uniformly distributed , is given by

1905

Fig. 5. BER sensitivity to tracking errors in fading channels. (a) Direct path. (b) Multipath.

, 2, 3, and 4 (with reference to ), respecat tively, for PAM transmissions over a 9-tap frequency-selective fading channel. With such a performance degradation, the deincreases. tection quality quickly becomes unacceptable as is perfectly acquired, the BER Sensitivity to Tracking: If effect of tracking offset is given by

(31)

The acquisition sensitivity for the direct-path and multipath scenarios is illustrated in Fig. 4. It is observed that MRC is needed to sustain acquisition errors, even for flat-fading channels. When TH is present, the BER performance improves over the no-TH , due to noise averaging, but appears to be case when , due to the code mismatch. The BER worse when values degrade approximately linearly as the acquisition error increases. Compared with perfect timing, the BER performance in the presence of mistiming is reduced by 2, 4, 7, and 10 dB

(32) The tracking sensitivity for flat-fading and frequency-selective fading environments is illustrated in Fig. 5. It is obvious that the BER degradation introduced by tracking alone is not drastic in multipath channels, whereas direct-path flat channels have little tolerance to tracking errors when a sliding correlator is employed. These results corroborate also the no-fading case in Part I [5]. BER Sensitivity of PPM versus PAM: To shed light on the comparative behavior between PPM and PAM modulation formats, we plot in Figs. 6(a) and (b) the flat-fading case and the

TLFeBOOK

1906

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE III BER GAIN INDUCED BY MULTIPATH DIVERSITY

and coherently combined at the receiver end; see the contrast between the BER curves of a RAKE-1 receiver in flat fading and that of a RAKE-MRC receiver in frequency-selective fading. Without mistiming, the multipath diversity provided by as many as nine taps in our multipath channel leads to considerable BER improvement of roughly 10 dB for PAM and 5.4 dB for PPM. Unfortunately, the performance advantage is reduced quickly as increases, until reaching a mere gain of 2.3 dB for PAM and 4 and 5. The BER diversity gain drops 2.5 dB for PPM, at slower for PPM than for PAM. VII. CONCLUDING SUMMARY

Fig. 6. BER sensitivity to PPM versus PAM in fading channels. (a) Flat fading (no TH). (b) Multipath.

frequency-selective fading case, respectively, all using a PPM . Both figures confirm that PPM is modulation index of less power efficient than PAM, due to the doubling of sample noise variance when using a differential correlation-template . For PPM, the general trends of BER sensitivity to mistiming are reminiscent of those of PAM. In the direct-path case, PPM with RAKE-1 reception is very sensitive to tracking errors, exhibiting sharp edges that confine the tracking errors to be . Under mistiming, RAKE-MRC is suggested for both flat-fading channels and frequency-selective fading channels, in order to improve robustness to mistiming in the former case and to effectively collect diversity gain under mistiming in the latter case. On the other hand, PPM seems to exhibit better robustness to acqsuitision errors. It can be observed that the increases, comBER values of PPM degrade more slowly as pared with PAM. In Table III, we list the BER gains induced by multipath diversity for both PAM and PPM. Compared with transmission over a flat-fading channel, diversity gain is induced when the same transmit energy is spread over multiple propagating paths

We have demonstrated that UWB transmissions exhibit pronounced sensitivity to mistiming, relative to narrowband singlecarrier transmissions. This is attributed in part to the low-dutycycle nature of UWB signaling, which hinders effective energy capture in the presence of mistiming. The BER performance is dependent on the richness of multipath, the multipath correlation and delay profile, the probabilistic characteristics of the random TH codes employed, as well as the transmit filter param, , and the receive correlation-template paeters , , , and . In particular, direct-path channels rameters , have little tolerance to tracking errors, while signal spreading incurred by TH and/or multipath propagation hinders the energy capture capability of optimum RAKE receivers when acquisition offset is present. As shown in Table III, mistiming quickly offsets the diversity gain of UWB as the acquisition error increases for both PAM and PPM. Future investigation could quantify the impact of imperfect channel estimation and multiple-access interference on the performance sensitivity to mistiming. Moreover, TH-UWB has been positioned very early as a possible multiple access technology for UWB. Future trends in the industry may go in different directions [12], which will call for future elaboration on their implications on timing tolerances. It is worth stressing that collection of the diversity gain of UWB in the presence of mistiming requires judicious receiver design. The optimal receivers developed in this paper select the matched filters, or, equivalently, the weighting coefficients of the RAKE, to be dependent on the timing offset. As a result, our RAKE-MRC is relatively robust to mistiming. In contrast, a conventional RAKE-MRC cannot tolerate any timing offset that is more than one pulse width. ACKNOWLEDGMENT The authors would like to thank the anonymous reviewers for their constructive comments and suggestions. REFERENCES [1] F. Ramirez-Mireles, “On the performance of ultra-wide-band signals in Gaussian noise and dense multipath,” IEEE Trans. Veh. Technol., vol. 50, no. 1, pp. 244–249, Jan. 2001.

TLFeBOOK

TIAN AND GIANNAKIS: BER SENSITIVITY TO MISTIMING IN ULTRA-WIDEBAND IMPULSE RADIOS—PART II

[2] M. Z. Win and R. A. Scholtz, “Ultra wide bandwidth time-hopping spread-spectrum impulse radio for wireless multiple access communications,” IEEE Trans. Commun., vol. 48, no. 4, pp. 679–691, Apr. 2000. [3] J. Foerster, E. Green, S. Somayazulu, and J. Leeper, “Ultra-wideband technology for short or medium range wireless communications,” Intel Corp. Tech. J., 2001. [Online] http://developer.intel.com/technology/itj. [4] W. M. Lovelace and J. K. Townsend, “The effects of timing jitter and tracking on the performance of impulse radio,” IEEE J. Sel. Areas Commun., vol. 20, no. 12, pp. 1646–1651, Dec. 2002. [5] Z. Tian and G. B. Giannakis, “BER sensitivity to mistiming in ultra-wideband impulse radios—Part I: Modeling,” IEEE Trans. Signal Process., vol. 53, no. 4, pp. 1550–1560, Apr. 2005. [6] H. Lee, B. Han, Y. Shin, and S. Im, “Multipath characteristics of impulse radio channels,” in Proc. IEEE Veh. Technol. Conf., Tokyo, Japan, Spring 2000, pp. 2487–2491. [7] A. A. M. Saleh and R. A. Valenzuela, “A statistical model for indoor multipath propagation,” IEEE J. Sel. Areas Commun., vol. JSAC-5, no. 2, pp. 128–137, Feb. 1987. [8] D. Cassioli, M. Z. Win, and A. F. Molisch, “The ultra-wide bandwidth indoor channel: from statistical model to simulations,” IEEE J. Sel. Areas Commun., vol. 20, no. 6, pp. 1247–1257, Aug. 2002. [9] M. Z. Win and R. A. Scholtz, “Characterization of ultra-wide bandwidth wireless indoor channels: a communication-theoretic view,” IEEE J. Sel. Areas Commun., vol. 20, no. 9, pp. 1613–1627, Dec. 2002. [10] M. K. Simon and M. S. Alouini, Digital Communications Over Generalized Fading Channels: A Unified Approach to Performance Analysis. New York: Wiley, 2000. [11] J. Foerster, “The effects of multipath interference on the performance of UWB systems in an indoor wireless channel,” in Proc. Veh. Tech. Conf., 2001, pp. 1176–1180. [12] IEEE 802.15 WPAN High Rate Alternative PHY Task Group 3a (TG3a). [Online]http://www.ieee802.org/15/pub/TG3a.html

Zhi Tian (M’98) received the B.E. degree in electrical engineering (automation) from the University of Science and Technology of China, Hefei, China, in 1994 and the M. S. and Ph.D. degrees from George Mason University, Fairfax, VA, in 1998 and 2000. From 1995 to 2000, she was a graduate research assistant with the Center of Excellence in Command, Control, Communications, and Intelligence (C3I) of George Mason University. Since August 2000, she has been an Assistant Professor with the Department of Electrical and Computer Engineering, Michigan Technological University, Houghton. Her current research focuses on signal processing for wireless communications, particularly on ultrawideband systems. Dr. Tian serves as an Associate Editor for IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS. She received a 2003 NSF CAREER award.

1907

Georgios B. Giannakis (F’97) received the Diploma in electrical engineering from the National Technical University of Athens, Athens, Greece, in 1981 and the M.Sc. degree in electrical engineering in 1983, the MSc. degree in mathematics in 1986, and the Ph.D. degree in electrical engineering in 1986, all from the University of Southern California (USC), Los Angeles. After lecturing for one year at USC, he joined the University of Virginia, Charlottesville, in 1987, where he became a Professor of electrical engineering in 1997. Since 1999, he has been a professor with the Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, where he now holds an ADC Chair in Wireless Telecommunications. His general interests span the areas of communications and signal processing, estimation and detection theory, time-series analysis, and system identification—subjects on which he has published more than 160 journal papers, 300 conference papers, and two edited books. Current research focuses on transmitter and receiver diversity techniques for single- and multiuser fading communication channels, complex-field and space-time coding, multicarrier, ultrawide band wireless communication systems, cross-layer designs, and distributed sensor networks. Dr. Giannakis is the (co-) recipient of four best paper awards from the IEEE Signal Processing (SP) Society in 1992, 1998, 2000, and 2001. He also received the Society’s Technical Achievement Award in 2000. He served as Editor in Chief for the IEEE SIGNAL PROCESSING LETTERS, as Associate Editor for the IEEE TRANSACTIONS ON SIGNAL PROCESSING and the IEEE SIGNAL PROCESSING LETTERS, as secretary of the SP Conference Board, as member of the SP Publications Board, as member and vice-chair of the Statistical Signal and Array Processing Technical Committee, as chair of the SP for Communications Technical Committee, and as a member of the IEEE Fellows Election Committee. He is currently a member of the the IEEE-SP Society’s Board of Governors, the Editorial Board for the PROCEEDINGS OF THE IEEE, and chairs the steering committee of the IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS.

TLFeBOOK

1908

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Design and Analysis of Feedforward Symbol Timing Estimators Based on the Conditional Maximum Likelihood Principle Yik-Chung Wu and Erchin Serpedin, Senior Member, IEEE

Abstract—This paper presents a general feedforward symboltiming estimation framework based on the conditional maximum likelihood principle. The proposed timing estimator presents reduced implementation complexity and is obtained by performing an approximation on the Fourier series expansion of the conditional maximum likelihood function. The proposed algorithm is applied to linear modulations and two commonly used continuous phase modulations: minimum shift keying (MSK) and Gaussian MSK (GMSK). For the linear modulations, it is shown both analytically and via simulations that the performance of the proposed estimator is very close to the conditional CRB and modified CRB for signal-to-noise ratios (SNRs) in the range SNR 30 dB. Furthermore, the proposed estimator is shown to be asymptotically equivalent to the classic square-law nonlinearity estimator under certain conditions. In the case of MSK and GMSK modulations, although the proposed algorithm reaches the conditional CRB at certain SNRs, however, the conditional CRB is quite far away from the modified CRB, and there exists an alternative algorithm whose performance comes closer to the modified CRB. Therefore, the proposed estimator is more suitable for linear modulations than for MSK and GMSK modulations. Index Terms—Conditional maximum likelihood, Cramér–Rao bound, feedforward, GMSK, linear modulations, MSK, symbol timing estimation.

I. INTRODUCTION N digital receivers, symbol timing synchronization can be implemented either in a feedforward or feedback mode. Although feedback schemes exhibit good tracking performances, they require a relatively long acquisition time. Therefore, for burst-mode transmissions, feedforward timing recovery schemes are more suitable. An all-digital feedforward symbol timing recovery scheme consists of first estimating the timing delay from the received samples, which is the focus of this paper, and then adjusting the timing using some sort of interpolation [1], [2]. Due to bandwidth efficiency considerations, nondata-aided or blind symbol timing estimation schemes have attracted much attention during the last decade. Most of the feedforward timing estimators proposed in the literature exploit the cyclostationarity induced by oversampling the received signal [3]–[8]. In

I

Manuscript received July 25, 2003; revised May 28, 2004. This work was supported by the National Science Foundation under Award CCR-0092901 and the Croucher Foundation. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Gregori Vazquez. The authors are with the Department of Electrical Engineering, Texas A&M University, College Station, TX 77843-3128 USA (e-mail: [email protected]; [email protected]). Digital Object Identifier 10.1109/TSP.2005.845486

[3], Oerder and Meyr proposed the well-known square nonlinearity estimator. Several extensions of this square nonlinearity estimator can be found in [5]–[7]. In [8], a low-SNR approximation was applied to the maximum likelihood function in order to derive a logarithmic nonlinearity. Reference [4] reported for the first time a detailed performance analysis of the estimators based on various types of nonlinearities. Recently, the conditional maximum likelihood (CML) principle was introduced for designing digital timing delay synchronizers by Riba et al. [9], [10]. The CML solution is especially important for symbol timing synchronization because it yields self-noise free timing estimates at medium and high SNRs. However, [9] and [10] concentrate on deriving a CML timing error detector (TED) so that the timing delay can only be tracked using a feedback loop. A first example of a feedforward symbol timing estimator based on the CML principle was reported in [11]. However, no insightful analysis pertaining to the design principles and performances has been reported. The purpose of this paper is to fill in this gap in the literature. The main design and performance characteristics of CML-based feedforward symbol timing delay estimators are established for general linear modulations and two commonly used continuous-phase modulations, namely, minimum shift keying (MSK) and Gaussian MSK (GMSK) [12], [13]. The performance of the timing estimators is analyzed analytically and through simulations, and compared with the conditional Cramér–Rao bound (CCRB) [9], [10], the modified CRB (MCRB) [14], and other existing state-of-the-art feedforward timing delay estimators [3], [14], [16], [17], [19]. In the proposed algorithm, an approximation is applied to the Fourier series expansion of the CML function so that the complexity of the proposed estimator is greatly reduced. Although the resulting estimator is not completely self-noise free (due to the approximation), the performances of the proposed estimator (for both linear and nonlinear modulations) are in general very close to the CCRB for signal-to-noise ratios (SNRs) smaller than 30 dB. For higher SNRs, a mean square error (MSE) floor occurs, but notice that at that high SNRs, the estimation MSE achieved by the proposed estimator is already very small; therefore, the effect of MSE floors becomes relatively less critical. For linear modulations, it is shown that the proposed estimator is asymptotically equivalent to the well-known square nonlinearity estimator [3]. However, the proposed estimator exhibits better performance (less self-noise/jitter) than [3] when a reduced number of data samples are available. Furthermore, it is shown that the performances of the proposed estimator for linear

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

WU AND SERPEDIN: DESIGN AND ANALYSIS OF FEEDFORWARD SYMBOL TIMING ESTIMATORS

modulations are also very close to the MCRB for SNR 30 dB. For MSK and GMSK modulations, although the performances of the proposed estimator come very close to the CCRB at certain SNR ranges, however, the CCRB is quite far away from the MCRB, and there exists an alternative algorithm whose performance comes closer to the MCRB. Therefore, it is concluded that the proposed estimator is more suitable for linear modulations than MSK and GMSK modulations. The rest of the paper is organized as follows. The signal model and the CML function are first described in Section II. The proposed feedforward timing estimator is derived in Section III. The relationship between the proposed estimator and the wellknown square nonlinearity estimator [3] is addressed in Section IV. The MSE expressions are derived in Section V. Simulation results and discussions are then presented in Section VI, and finally, conclusions are drawn in Section VII. II. SIGNAL MODEL AND THE CML FUNCTION A. Signal Model The complex envelope of a received linear modulation is given by (1) is the unknown phase offset; is the symbol enwhere ergy; stands for the zero-mean unit variance, independently and identically distributed (i.i.d.) complex valued symbols being is the transmit pulse with unit energy; is the symbol sent; is the unknown symbol timing delay to period; is the complex-valued circularly disbe estimated; and . After tributed white Gaussian noise with power density passing through the antialiasing filter, the received signal is then sampled at the rate , where . Note that the oversampling factor is determined by the frequency span of ; if is bandlimited to (an example of which is sufficient. The is the square-root raised cosine pulse), consecutive received received vector , which consists of samples (where is the observation length), can be expressed as (without loss of generality, we consider the received sequence ) start at (2) where1 (3) (4) (5) (6) 1Notation x denotes the transpose of x, and conjugate of x.

x

stands for the transpose

1909

, and denotes the number of symbols affected by the inter-symbol interference (ISI) introduced by one . side of For MSK and GMSK modulations, the complex envelope of the received signal is given by

(7) stands for the i.i.d. binary transmitted symbols, and where is the phase response of the modulator with length and satisfies (8) The derivative of is referred to as the frequency response of the modulator and takes the form of a rectangular pulse or a convolution between a rectangular pulse and a Gaussian-shaped pulse for MSK and GMSK modulations, respectively. According to the Laurent’s expansion (LE) [18] and the fact that most of the energy of the GMSK modulation is concentrated in the first component of the expansion [18]–[20] (the MSK signal has only one component in the expansion), MSK- and GMSK-received signals can be approximated by (9) where (10) (11) and (12) otherwise. Therefore, the sampled MSK and GMSK modulations can also be expressed in a form similar to (2). Since the pseudo-symbols (or equivalent data) are zero mean and unit variance, a single system model is sufficient to treat the linear modulations, MSK, and GMSK signals within a common framework. Remark 1: Notice that another formulation for the GMSK terms of the signal is to express the signal using all the LE, as is done in [10]. However, there is a disadvantage in doing this: Including more LE terms in the formulation would significantly increase the number of pseudo-symbols. Since, in the CML method, the pseudo-symbols and the unknown timing delay are jointly estimated from an observation vector of certain length, increasing the number of pseudo-symbols to be estimated would definitely degrade the overall estimation accuracy (of both pseudo-symbols and timing delay). Of course, neglecting some small LE terms (as is done in this paper) would

TLFeBOOK

1910

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

introduce interference and degrade the performance for the resulting estimator, but from the simulation examples to be presented in Section VI, the effect of the system model approximation (9) is very small and occurs only at a very high SNR region (at SNR 50 dB). Remark 2: MSK and GMSK modulations belong to a broader class of modulation called MSK-type modulation [12], [13]. The system model, the subsequent proposed estimator, and the MSE analysis can also be applied to other members of this MSK-type modulation as long as the approximation in (9) is tight (e.g., 1RC, 2RC modulations). However, in this paper, we only concentrate on two commonly used members: MSK and GMSK. B. CML Function From (2), the joint maximum likelihood estimate of is given by maximizing

and Fig. 1. Examples of CML functions.

(13) or equivalently minimizing (14) where and are the trial values for and , respectively. are modIn the CML approach, the nuisance parameters eled as deterministic and estimated from the received vector . From the linear signal model given in (2), if no constraint is imposed on the possible value of , the maximum likelihood (when is fixed) is [15] estimate for (15) Plugging (15) into (14), after some straightforward manipulations, and dropping the irrelevant terms, the timing delay is estimated by maximizing the following CML function [9]:

expect that it is not necessary to calculate the CML function for all the values of . It is possible that the CML function is first calculated for some ’s and that the values in between can be found by interpolation. More specifically, suppose we calculated uniformly spaced values of using (16) such that a sequence for is obtained (without loss of generality, we consider is even). Let us construct a periodic sequence by periodically extending . Further, denote as the continuous and periodic function with its samples given by . According to the sampling theorem, as long as the sampling frequency is higher than twice the highest frequency , then can be represented by its samples of without loss of information. The relationship between and is then given by sinc

(17)

(16) In general, the maximum of the CML function can be found by plugging different values of into (16). The value that provides is the CML estimate. Since is a the maximum value of continuous variable, this exhaustive search method requires a lot of computations and is impractical. Alternatively, a timing error detector (TED) [9] can be used in a feedback configuration. However, in burst mode transmissions, feedforward timing delay estimators [3]–[8] are preferred since they avoid the relatively long acquisition time and hang-up problem in feedback schemes. In the following, a new method for optimizing (16) is proposed so that an efficient implementation of the feedforward symbol-timing estimator results.

where sinc series

. Now, expand

into a Fourier

(18) where (19) Substituting (17) into (19) yields

III. PROPOSED ESTIMATOR Fig. 1 shows some realizations of the CML function calcu(for lated using (16), where the true timing delay is is a square-root raised cosine filter the linear modulation, with roll-off factor 0.5). It can be seen that the CML function has only one maximum. Since the CML function is smooth, we

TLFeBOOK

sinc

sinc

WU AND SERPEDIN: DESIGN AND ANALYSIS OF FEEDFORWARD SYMBOL TIMING ESTIMATORS

1911

K = 4), (b) the proposed estimator (K = 4), and (c) the squaring estimator.

Fig. 2. Block diagrams for (a) the IDFT-based CML estimator (

sinc

sinc where

(20)

denote the Fourier transform. It is clear that (21)

. For the rest of the paper, we refer to this estimator as the IDFT-based CML estimator. -point IDFT, To avoid the complexity in performing the an approximation is applied to (18). More precisely, it can be seen from Fig. 1 that the CML function for symbol timing estimation resembles a sine function with one period in the interval . It is expected that the Fourier coefficient is much larger than the Fourier coefficients associated with higher frequencies. Therefore, it is sufficient to approximate (18) as follows:

otherwise. are deFrom (18), it can be seen that once the coefficients termined, can be calculated for any . Then, the problem of maximizing (16) can now be replaced by maximizing (18). For efficient implementation, the function for can be approximated by a -point sequence as follows: for

for

(23)

stands for real part of . In order to maximize where , the following equation must hold: (24) where

denotes the phase of , or, equivalently

(22)

(25)

This is equivalent to first calculating using (21), then zero , and finally perpadding the high-frequency coefficients -point inverse discrete Fourier transform (IDFT). forming a For sufficiently large value of , becomes very close to for , and the index with the maximum amplitude can be viewed as an estimate of the unknown timing parameter . Fig. 2(a) shows the block diagram for this algorithm when

The estimated delay is the normalized (with respect to ) time difference between the first sample of the received vector and the nearest optimum sampling instant. The calculation within the operation is actually the first bin (i.e., second output) of a -point discrete Fourier transform (DFT) of the sequence (or the Fourier coefficient at symbol rate ). Based on (24), it is not difficult to check that the proposed estimator

TLFeBOOK

1912

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

(25) is asymptotically unbiased, which is a result that is independent of the approximation used in (23). From a computational viewpoint, it is worth mentioning that the proposed estimator only involves the calculation of samples of the CML function using (16), a -point DFT, operation. From the results to be presented, it and an is sufficient to yield good estimates in is found that practical applications. Therefore, the four-point DFT in (25) can be computed easily without requiring any multiplications. The main complexity comes from the calculation of the four using (16). However, notice that the matrix samples of can be precomputed for with . This greatly reduces the arithmetic complexity of implementation. Complexity can be further reduced by using approximating the precomputed sum-of-power-of-two (SOPOT) expressions [21], [22]. IV. RELATIONSHIP WITH THE SQUARE NONLINEARITY ESTIMATOR is a square-root raised In this section, we will show that if cosine pulse, the proposed estimator in (25) asymptotically reduces to the well-known square nonlinearity estimator [3]. First is a square-root raised cosine pulse and notice that when ), [10], in the asymptotic case (as if and zero otherwise. Notice that the mawhere trix is of dimension . The holds very well for the central approximation ) of . For the boundary portion (of dimension , the values are smaller than 1. As , the of boundary of becomes insignificant and can be ignored. into (16), it follows that Putting

The proposed CML feedforward timing delay estimator in (25) can then be rewritten as (29) (30) Therefore, when and , we have the well-known squaring algorithm [3]. Figs. 2(b) and 2(c) show the block diand the agrams for the proposed estimator (25) with squaring algorithm. It can be seen that the structures of the proposed algorithm and the squaring algorithm are very alike. Note that both the proposed algorithm and the squaring algorithm require four samples per symbol period to form the timing estimate. For the proposed estimator, the received signal is first sam, and then, sampled with minimum oversampling ratio ples with different phases are generated by filtering [see (27)]. For the squaring algorithm, the four different samples per symbol period are directly obtained through sampling. Notice that the squaring algorithm might work also by first sampling at , and then, the intermediate (additional two) samples are computed by interpolation before symbol timing estimation. Although the proposed estimator and the squaring algorithm have many characteristics in common, simulation results presented in Section VI show that the proposed estimator outperforms the squaring algorithm for reduced length observation records. V. ANALYTICAL PERFORMANCE ANALYSIS In this section, we derive the mean square error (MSE) ex. pressions for the proposed estimator as a function of First, express the true timing delay as follows:

(26) (31) Now consider the

element of From (25) and (31), the MSE for a specific delay is given by2 (32) where (27) (33)

, with a rectangular window of where . It is recognized that the summation in (27) is just length , through , followed then by sampling the filtering of . Notice that since is a square-root raised at cosine filter, and (27) actually correspond to the sampled matched filter output. If we define , . where denotes convolution, we have Plugging this result into (26) and noting that asymptotically, the , we have range of can be approximated by

Applying the approximation arctan have

for small , we

(34) (28)

E

2Notation [z ] denotes the mean of random variable z , whereas stands for the imaginary part of .

TLFeBOOK

fg

WU AND SERPEDIN: DESIGN AND ANALYSIS OF FEEDFORWARD SYMBOL TIMING ESTIMATORS

1913

(36) It is proved in the Appendix that

tr

tr tr tr

tr

tr

tr

tr tr

E

0

tr

tr tr

E

Fig. 3. Plots of [(  ) ] and [( +  ) ] as a function of E =N for " = 0, 0.25, 0.5, and 0.75 (g (t) is a square-root raised cosine pulse with  = 0:5, Q = 2, K = 4, L = 100, and L = 3). Note that all curves for different values of " overlap.

The last equality in (34) comes from the fact that . The second approximation in (34) can be justified using similar arguments as in [23]. A close examination of the in (34) illustrates that the mean fraction of its denominator is much larger than the mean of its numerator, and the standard deviations of its numerator and denominator are in general much smaller than the mean of denomiand as a function nator. Fig. 3 plots when is a square-root raised cosine pulse with of , , , , and for , 0.25, 0.5, and 0.75. Note that all curves for different values of overlap. It can be seen that for dB, is much smaller than . The same result can be obtained for other different pulse shapes . In addition, one can , the standard deviations check that at medium and high and are small relative to . of All these considerations justify the second approximation made in (34). From (33), we note that

(35)

(37) tr

tr tr tr

tr

tr

tr

tr tr

tr

tr tr (38)

where tr

denotes the trace of a matrix. In (37) and (38) (39)

is an

matrix with the given by

th element

(40) and (41), shown at the bottom of the page, where is the fourth-order moment of the transmitted symbols, which is for PSK and fixed for a specific constellation (e.g., for quadrature amplitude modulation). Therefore, the can be found by using (34)–(38). MSE for a specific delay

for linear modulations (41) for MSK and GMSK

TLFeBOOK

1914

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Fig. 5. Comparison between analytical MSE and simulations of the proposed estimator (QPSK, Q = 2, K = 4,  = 0:5, L = 100, and L = 3).

Fig. 4. MSE of the proposed estimator and the IDFT-based CML estimator = 4 and (b) = 8 (QPSK, Q = 2,  = 0:5, L = 100, and with (a) L = 3).

K

K

As the symbol timing delay is assumed to be uniformly distributed in [0,1), the average MSE is calculated by numerical integration of (34). Notice that the MSE expressions in this section can only be regarded as an approximated analysis for GMSK since only the principle component of LE is taken into consideration. However, from the results to be presented in next section, excellent agreement between analytical expressions and simulations can be observed (see Fig. 9); only a small deviation occurs at very high SNRs. VI. SIMULATION RESULTS AND DISCUSSIONS In this section, the performance of the proposed algorithm and other existing symbol timing estimators are assessed by Monte Carlo simulations and then compared with the analytical results derived in the last section, the CCRB [10], and the MCRB [14]. In all the simulations, the observation length is fixed to , and is uniformly distributed in the range [0,1).

is generated as a uniformly distributed random variable in the and is constant in each estimation. Each point is range obtained by averaging simulation runs. In all figures, the CCRB and the MCRB are plotted as references. First, consider the case of linear modulations. QPSK is chosen as the symbol constellation. The oversampling ratio for , is the square-root raised the proposed estimator is , and the number of cosine pulse with roll-off factor is assumed to be ISI symbols introduced by one side of . Figs. 4(a) and 4(b) show the MSE against for the proposed algorithm and the IDFT-based CML estimator for and , respectively. It can be seen [from both Figs. 4(a) and 4(b)] that the proposed algorithm has a performance similar to that of IDFT-based CML estimator . This further justifies the approximation in with , the self noise is not completely (23). Note that for eliminated for both the IDFT-based CML estimator and the proposed estimator [as seen from the MSE departure from CCRB at high SNR in Fig. 4(a)]. This can be explained as follows. For the IDFT-based CML estimator, the self-noise is due to the small value of chosen since in the derivation, it is assumed that the CML function can be completely represented by samples. However, there is no guarantee that is results in pretty good performance). sufficient (although to 8 removes the self noise of the Increasing the value of IDFT-based CML estimator (with ), as shown in Fig. 4(b). For the proposed estimator, although it is also should be large enough such that can be required that represented by its samples, the self noise is due to another more critical factor—the approximation (23) in the CML function. This can be seen from the fact that the performance of the proposed estimator does not improve by increasing from 4 to is good enough for 8 [compare Figs. 4(a) and 4(b)]. As the proposed estimator, is used for the rest of the paper. Fig. 5 illustrates the very close match between the simulation and the analytical results derived in the last section. It is also clear that for the SNRs under consideration, the performance of the proposed algorithm is very close to the CCRB, which

TLFeBOOK

WU AND SERPEDIN: DESIGN AND ANALYSIS OF FEEDFORWARD SYMBOL TIMING ESTIMATORS

Fig. 6. Comparison of the MSE of the proposed estimator and the square nonlinearity estimator (QPSK,  = 0:5, L = 100, and L = 3).

means that the proposed estimator almost reaches the ultimate performance of the CML principle. Furthermore, the CCRB is close to the MCRB. Since the MCRB is a lower bound on the variance of any unbiased estimate, this shows that the proposed . Notice algorithm is close to optimal for a wide range of around 30 dB, an MSE floor begins to occur (due that at to the approximation (23) in the CML function), but at that high SNR, the estimation MSE achieved by the proposed estimator ), and therefore, the is already very small (on the order of effect of the MSE floor becomes relatively less critical. Fig. 6 compares the performance of the proposed estimator ) with that of the square nonlinearity estimator (with [3]. It is apparent that the proposed estimator outperforms the . This square nonlinearity estimator, especially at high , and the is because for finite observation length, than . self-noise is better cancelled by the matrix Fig. 7 compares the performances of the proposed algorithm with the existing state-of-the-art feedforward algorithms that require only two samples per symbol to operate: Mengali [14, pp. 401], Zhu et al. [16], and Lee [17]. It can be seen that while the , performances for different algorithms are similar at low . the proposed algorithm has the smallest MSE at high Next, consider that MSK is the modulation format. Fig. 8 , shows the performances of the proposed estimator (with , and ) and the low-SNR approximated maximum likelihood (ML) algorithm [19] for MSK. The number is assumed to be of ISI symbols introduced by one side of 1. The following observations can be inferred from Fig. 8. First, it can be seen that for the proposed algorithm, the higher the oversampling ratio, the better the performance. This is because is time limited [18]; therefore, its frequency rethe pulse sponse is not bandlimited; a higher oversampling ratio reduces the aliasing and, thus, provides better performance. Second, the theoretical MSE analysis matches the simulation results very well. Third, although a higher oversampling ratio increases the range of SNRs over which the performance of the proposed estimator comes close to the CCRB, MSE floors still occur at

1915

Fig. 7. MSE for the proposed estimator, the algorithms in [14, pp. 401], [16], and [17] (QPSK, Q = 2,  = 0:5, L = 100, and L = 3).

Fig. 8. MSE of the proposed estimator (Q = 2, Q = 4, and Q = 8) and the low-SNR approximated ML algorithm [19] for MSK (L = 1 and L = 100).

high SNRs due to the approximation (23) assumed in the derivation of estimator. Furthermore, the CCRB is far away from the MCRB, and the simulation results show that the low-SNR approximated ML algorithm [19] approaches the MCRB. Therefore, direct application of the CML principle is not suitable for the MSK modulation. Now, let us consider the GMSK modulation. Fig. 9 shows and the performances of the proposed estimator (with ) and the low-SNR approximated ML algorithm [19] for . The number GMSK with premodulator bandwidth is assumed to be of ISI symbols introduced by one side of 2. Notice that although the proposed estimator is based on the approximated linear model (9), the GMSK signal in the simulation is generated according to (7) without approximation. The MCRB for GMSK is exact, and its expression can be found in [14]. For the CCRB, it is based on the approximated linear model (9). Although the resulting CCRB is not exact, it is still a valid lower bound for the proposed estimator since when the

TLFeBOOK

1916

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Finally, notice that the CCRB is far away from the MCRB, as in the case of MSK. Since the CCRB is a valid bound only for estimators that rely on quadratic nonlinearities [10], it is expected that algorithms exploiting higher order ( 2) nonlinearities might exist with performances closer to the MCRB. An example of such an algorithm is the low-SNR approximated ML algorithm [19], for which we already demonstrated that its performance is closer to the MCRB at low SNRs. The next question is whether there is an estimator whose performance comes close to the MCRB for a larger range of SNRs. This is a subject that is open to future investigations. VII. CONCLUSIONS

Fig. 9. MSE of the proposed estimator (Q = 2 and Q = 4) and the low-SNR approximated ML algorithm [19] for GMSK with BT = 0:5 (L = 2 and = 100). L

proposed estimator is applied to the true GMSK signal, the ignored components in LE would become interferences, and the performances would be poorer than that predicted by the CCRB, which assumes no interference from other components of LE. Note that the CCRB obtained by expressing the GMSK signal using all the LE components (as done in [10]) is not applicable here since in that case, the resultant CCRB is conditioned on the fact that all the pseudo-symbols are being jointly estimated together with the unknown timing offset, whereas in the proposed estimator, only the pseudo symbols related to the first LE component are estimated. From Fig. 9, it can be seen that for the proposed estimator, a higher oversampling ratio also results in better performance for the same reason as in the case of MSK modulation. However, by comparing Figs. 8 and 9, if the same oversampling ratio is used, it is found that the performance of the proposed estimator for GMSK modulation is better than that corresponding to MSK. This is due to the fact that the pulse is longer in GMSK than in MSK (although they both are time-limited); therefore, with the same oversampling ratio, the aliasing introduced in GMSK is smaller than that in MSK. Second, it is obvious that the analytical MSE expressions derived in the last section match very and well with the simulation results. Only for the case of at SNR 50–60 dB, the analytical MSE expressions predict a slightly better performance than simulations. Third, the perforcomes very close mance of the proposed estimator with dB. The MSE floor, which is to the CCRB for caused by the approximation (23) in the CML function, begins dB. Notice that the effect of the to occur only for approximation (9) in the system model (which results in the gap between analytical MSE and simulations) is much smaller than that of approximation (23) in the CML function (which causes the MSE floor). Compared to the low-SNR approximated ML algorithm [19], at low SNRs, the proposed estimator exhibits poorer performance, but for medium and high SNRs, the proposed estimator performs much better.

A new feedforward symbol-timing estimator based on the conditional maximum likelihood principle was proposed. An approximation was applied in the Fourier series expansion of the CML function so that the complexity of the proposed estimator is greatly reduced. It was shown, analytically and via simulations, that the performances of the proposed estimator for linear modulations are, in general, very close to the CCRB and MCRB for SNR 30 dB. For higher SNRs, MSE floors occur, but notice that at these high SNRs, the MSE achieved by the proposed estimator is already very small, and therefore, the effect of MSE floors becomes relatively less critical. Furthermore, for linear modulations where the transmit pulse is a square-root raised cosine pulse, the proposed estimator was shown to be asymptotically equivalent to the well-known square nonlinearity estimator [3]. However, in the presence of a reduced number of samples, the proposed estimator presents better performance than [3]. For MSK and GMSK modulations, it was found that although the performances of the proposed estimator come very close to the CCRB at certain SNR ranges, however, the CCRB is quite far away from the MCRB, and there exists an alternative algorithm that come closer to the MCRB. Therefore, it was concluded that the proposed estimator is more suitable for linear modulations than MSK and GMSK modulations. APPENDIX PROOF OF (37) AND (38) From the definition of

in (16), we have

(42) where on

TLFeBOOK

is the

element in . Now, we concentrate , which is given by

WU AND SERPEDIN: DESIGN AND ANALYSIS OF FEEDFORWARD SYMBOL TIMING ESTIMATORS

1917

(43) (54) First, let us consider linear modulations, in which case, we have (44) (45) (46) (55)

(47) for for for otherwise

(48)

and therefore, 10 out of the 16 terms that result from (43) vanish. With the definitions

(49) the remaining terms can be expressed as

(56) Plugging (50)–(56) back into (42) and expressing the summations using matrices, some straightforward calculations lead to (37). A similar procedure can be used to prove (38). Now, let us consider MSK and GMSK. Since the pseudosymbols in (10) are not circularly symmetric, (44) and (48) have to be modified accordingly. After some lengthy but straightforward calculations, it is found that

(57)

(50)

for for for for otherwise.

where

(58)

Due to (57), two more cross terms in the expansion of (43) have to be considered. One of them is , which is given by (51)

(52)

(53)

However, thanks to the correlation property of noise samples, . The other extra term is also zero due to the same reason. For the fourth-order moment in (58), compared to the corresponding expression for linear modulations (48), we notice , and there is an extra nonzero fourth-order moment. that in , an extra term has Therefore, apart from setting

TLFeBOOK

1918

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

to be added to in (51). The modified , can be expressed as

, which is denoted as

(59)

Plugging (59) into (42) and then expressing the multiplications using matrix notation, it can be proved that the only change is , which is given in (41). the definition of ACKNOWLEDGMENT The authors would like to thank the reviewers for carefully reading this manuscript and for their constructive comments, which greatly improved the presentation of this paper. REFERENCES [1] C. W. Farrow, “A continuously variable digital delay element,” in Proc. ISCAS, 1998, pp. 2641–2645. [2] L. Erup, F. M. Gardner, and R. A. Harris, “Interpolation in digital modems—Part II: Implementation and performance,” IEEE Trans. Commun., vol. 41, no. 6, pp. 998–1008, Jun. 1993. [3] M. Oerder and H. Meyr, “Digital filter and square timing recovery,” IEEE Trans. Commun., vol. 36, no. 5, pp. 605–612, May 1988. [4] E. Panayirci and E. Y. Bar-Ness, “A new approach for evaluating the performance of a symbol timing recovery system employing a general type of nonlinearity,” IEEE Trans. Commun., vol. 44, no. 1, pp. 29–33, Jan. 1996. [5] F. Gini and G. B. Giannakis, “Frequency offset and symbol timing recovery in flat-fading channels: a cyclostationary approach,” IEEE Trans. Commun., vol. 46, no. 3, pp. 400–411, Mar. 1998. [6] K. E. Scott and E. B. Olasz, “Simultaneous clock phase and frequency offset estimation,” IEEE Trans. Commun., vol. 43, no. 7, pp. 2263–2270, Jul. 1995. [7] Y. Wang, E. Serpedin, and P. Ciblat, “Blind feedforward cyclostationarity-based timing estimation for linear modulations,” IEEE Trans. Wireless Commun, vol. 3, no. 3, pp. 709–715, May 2004. [8] M. Morelli, A. N. D’Andrea, and U. Mengali, “Feedforward ML-based timing estimation with PSK signals,” IEEE Commun. Lett., vol. 1, no. 3, pp. 80–82, May 1997. [9] J. Riba, J. Sala, and G. Vazquez, “Conditional maximum likelihood timing recovery: estimators and bounds,” IEEE Tran. Signal Processing, vol. 49, no. 4, pp. 835–850, Apr. 2001. [10] G. Vazquez and J. Riba, “Non-data-aided digital synchronization,” in Signal Processing Advanced in Wireless and Mobile Communications, G. B. Giannakis, Y. Hua, P. Stoica, and L. Tong, Eds. Englewoof Cliffs, NJ: Prentice-Hall, 2001, vol. 2. [11] Y.-C. Wu and E. Serpedin, “Low-complexity feedforward symbol timing estimator using conditional maximum-likelihood principle,” IEEE Commun. Lett., vol. 8, no. 3, pp. 168–170, Mar. 2004. [12] P. Galko and S. Pasupathy, “On a class of generalized MSK,” in Proc. ICC, Jun. 1981, pp. 2.4.1–2.4.5.

[13] O. Andrisano and M. Chiani, “The first Nyquist criterion applied to coherent receiver design for generalized MSK signals,” IEEE Trans. Commun., vol. 42, no. 2/3/4, pp. 449–457, Feb/Mar/Apr. 1994. [14] U. Mengali and A. N. D’Andrea, Synchronization Techniques for Digital Receivers. New York: Plenum, 1997. [15] S. M. Kay, Fundamentals of Statistical Signal Processing—Estimation Theory. Englewood Cliffs, NJ: Prentice-Hall, 1993. [16] W.-P. Zhu, M. O. Ahmad, and M. N. S. Swamy, “A fully digital timing recovery scheme using two samples per symbol,” in Proc. IEEE Int. Symp. Circuits Syst., May 2001, pp. 421–424. [17] S. J. Lee, “A new nondata-aided feedforward symbol timing estimator using two samples per symbol,” IEEE Commun. Lett., vol. 6, no. 5, pp. 205–207, May 2002. [18] P. A. Laurent, “Exact and approximate construction of digital phase modulations by superposition of amplitude modulated pulse (APM),” IEEE Trans. Commun., vol. COM-34, no. 2, pp. 150–160, Feb. 1986. [19] M. Morelli and G. Vitetta, “Joint phase and timing recovery for MSKtype signals,” IEEE Trans. Commun., vol. 48, no. 12, pp. 1997–1999, Dec. 2000. [20] Y.-C. Wu and T.-S. Ng, “Symbol timing recovery for GMSK modulations based on the square algorithm,” IEEE Commun. Lett., vol. 5, no. 5, pp. 221–223, May 2001. [21] H. Samueli, “An improved search algorithm for the design of multiplierless FIR filters with powers-of-two coefficients,” IEEE Trans. Circuits Syst. II, vol. 36, no. 7, pp. 1044–1047, Jul. 1989. [22] C.-K.-S. Pun, Y.-C. Wu, S.-C. Chan, and K.-L. Ho, “On the design and efficient implementation of the Farrow structure,” IEEE Signal Process. Lett., vol. 10, no. 7, pp. 189–192, Jul. 2003. [23] T. M. Schmidl and D. C. Cox, “Robust frequency and timing synchronization for OFDM,” IEEE Trans. Commun., vol. 45, no. 12, pp. 1613–1621, Dec. 1997.

Yik-Chung Wu received the B.Eng. (honors) and M.Phil. degrees in electronic engineering from the University of Hong Kong in 1998 and 2001, respectively. He was then a research assistant in the same university from 2001 to 2002. He received the Croucher Foundation scholarship in 2002 and is currently pursuing the Ph.D. degree at Texas A&M University, College Station. His research interests include digital signal processing with applications to communication systems, software radio, and space-time processing.

Erchin Serpedin (M’99–SM’04) received (with highest distinction) the Diploma of electrical engineering from the Polytechnic Institute of Bucharest, Bucharest, Romania, in 1991. He received the specialization degree in signal processing and transmission of information from Ecole Superiéure D’Electricité, Paris, France, in 1992, the M.Sc. degree from Georgia Institute of Technology, Atlanta, in 1992, and the Ph.D. degree in electrical engineering from the University of Virginia, Charlottesville, in January 1999. From 1993 to 1995, he was an instructor with the Polytechnic Institute of Bucharest, and between January and June of 1999, he was a lecturer at the University of Virginia. In July 1999, he joined the Wireless Communications Laboratory, Texas A&M University, as an assistant professor. His research interests lie in the areas of statistical signal processing and wireless communications. Dr. Serpedin received the NSF Career Award in 2001 and is currently an associate editor of the IEEE COMMUNICATIONS LETTERS, the IEEE SIGNAL PROCESSING LETTERS, and the IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS.

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1919

Correspondence________________________________________________________________________ A Bayesian Approach to Array Geometry Design Ülkü Oktel and Randolph L. Moses, Senior Member, IEEE Abstract—In this paper, we consider the design of planar arrays that optimize direction-of-arrival (DOA) estimation performance. We assume that the single-source DOA is a random variable with a known prior probability distribution, and the sensors of the array are constrained to lie in a region with an arbitrary boundary. The Cramér–Rao Bound (CRB) and the Fisher Information Matrix (FIM) for single-source DOA constitute the basis of the optimality criteria. We relate the design criteria to a Bayesian CRB criterion and to array beamwidth; we also derive closed-form expressions for the design criteria when the DOA prior is uniform on a sector of angles. We show that optimal arrays have elements on the constraint boundary, thus providing a reduced dimension iterative solution procedure. Finally, we present example designs. Index Terms—Array design, Cramér–Rao bound, direction-of-arrival estimation, planar arrays.

I. INTRODUCTION Direction-of-arrival (DOA) estimation from the outputs of an array of sensors is an important and well-studied problem with many applications in radar, sonar, and wireless communications. A large number of DOA estimation algorithms and analytical performance bounds have been developed (see, e.g., [1]). The DOA estimation performance of an array strongly depends on the number and locations of the array elements. In this paper, we consider planar array geometry design for “good” DOA estimation performance. A number of researchers have considered the design of arrays to achieve or optimize desired performance goals. Much of the array design literature is devoted to linear arrays [2]–[6]. For planar arrays, performance comparisons of some common array geometries are given in [7]–[9]. In [10], a measure of similarity between array response vectors is introduced, and a tight lower bound for this similarity measure is derived. This bound is suggested as a performance criterion in the sense that the array with highest bound has best ambiguity resolution. In [11], differential geometry is used to characterize the array manifold and an array design framework based on these parameters is proposed. In [12], a sensor polynomial is constructed using prespecified performance levels, such as detection resolution thresholds and Cramér–Rao Bound (CRB) on error variance, and roots of the polynomial are the sensor locations of the desired linear or planar array. The Dolph–Chebyshev criterion is proposed for optimal element positioning in [13]. The method proposed in [13] minimizes the mainlobe area while satisfying the prespecified sidelobe levels. Most of the above papers consider designs for a single desired DOA, or they implicitly assume that the DOA is equally likely in all directions. In many applications, including radar, sonar, and wireless base station design, the DOA of interest may be constrained to lie in a sector Manuscript received May 19, 2003; revised May 2, 2004. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vikram Krishnamurthy. Ü. Oktel is with Aselsan, Inc., Ankara 06370, Turkey (e-mail: [email protected]). R. L. Moses is with the Department of Electrical and Computer Engineering, The Ohio State University, Columbus, OH 43210 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845487

or may be more likely in some directions than others. In this paper, we consider design of optimal planar arrays for such scenarios by modeling the DOA of the single source as a random variable whose prior probability distribution function (pdf) characterizes any prior constraints or arrival angle likelihood. To keep the paper concise, we present results for planar arrays that estimate DOA in azimuth angle only. However, the design method and main results also apply to volume arrays, as well as to arrays that estimate the DOA in both azimuth and elevation. In addition, the results apply to both narrowband and wideband arrays. We adopt a Bayesian approach and employ the average CRB and average FIM as design criteria. We relate both the average Fisher Information Matrix (FIM) and average CRB to the Bayesian CRB (also called the global CRB). The CRB gives a lower bound on the variance of any unbiased estimate of a nonrandom parameter. The Bayesian CRB is a lower bound on the mean-squared error of the estimate of a random parameter and is independent of any particular estimator [14]. Because the array locations are nonlinear functions of the resulting cost criteria, closed-form solutions are not available except in a few special cases; thus, we adopt nonlinear function minimization techniques. We show that the optimal element locations lie on the boundary of the element constraint region; therefore, the dimension of the minimization problem can be reduced from 2m to m, where m is the number of array elements. In the case of FIM criterion, the function to be minimized is a quadratic function of the array elements, and therefore, efficient quadratic optimization procedures can be used. Both the CRB and FIM are closely related to the mainlobe width of the array [13], [15], [16]. We show that average FIM and average CRB can be interpreted as the average mainlobe width of the array, averaged over the steering angle. Arrays that have small mainlobe width perform well in moderate or high signal-to-noise ratios (SNRs), and they have high resolution. An outline of this paper is as follows. In Section II, we describe the system model, state our assumptions, and give the expression for the CRB on the single source DOA. In Section III, we introduce the performance measures, and define the optimization problems. We discuss that both CRB- and FIM-based cost functions can be related to the Bayesian CRB. We also give the closed-form integrals for FIM- and CRB-based cost functions when the probability distribution of the DOA is uniform. In Section IV, we prove that sensors of the both CRB-optimal and FIM-optimal arrays should lie on the boundary of the constraint region. We give example optimal array designs in Section V. Section VI concludes the paper. II. SYSTEM MODEL AND SINGLE-SOURCE CRB We consider an array of m identical sensors on the (x; y ) plane. Each sensor is located at ri = [xi ; yi ]T for i 2 [1; m]. We define ra = [xa ; ya ]T = (1=m) m i=1 ri as the centroid of these sensors. The array is represented by the 2 2 m array location matrix r = [r1 ; r2 ; . . . ; rm ] =

x 1 x 2 . . . xm : y 1 y 2 . . . ym

(1)

A single, narrowband far-field source s(t) centered at frequency !c = 2= and coplanar with the array impinges on the array from direction  . A set of N snapshots are sampled by the array, giving the m 1

2

measurement vectors x(t) = A (t)s(t) + n(t)

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

t = 1; 2; . . . ; N

(2)

1920

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

where n(t) is the m 2 1 noise vector, s(t) is a scalar, and A

=

d ( )

ej

d ( )

; ej

; . . . ; ej

d

( )

T

The FIM cost criterion is a quadratic function of the array locations. From (4) and (10), we have

where dk () = (uT () 1 rk )=c is the propagation delay associated with the kth sensor, c is the speed of propagation, and u() = [cos(); sin()]T is the unit vector pointing toward the signal source. The noise at the sensors is assumed to be white Gaussian and independent of the source signal. Under the system model described above, the FIM for the DOA estimate from measurements fx(t)gN t=1 is given by [17] and [18] and some simple algebra:

B

(4)

du() T du() B d d

= 1 (r 0 rA )(r 0 rA )T m

(5) (6)

where P is an SNR term that is independent of the source DOA  and of the array geometry, and where rA = [ra ; ra ; . . . ; ra ] is the 2 2 m array centroid matrix (see also [19]). The CRB on the DOA estimate is the inverse of the FIM given in (4). For the purpose of array design, the narrowband signal assumption is not needed. If s(t) is wideband, the expression for the FIM is still given by (4)–(6); only the expression for P changes (see [16] and [18]). III. PROBLEM STATEMENT AND COST FUNCTIONS We are interested in array geometry designs, i.e., the selection of r, that yield good DOA estimation performance. We assume that the single-source DOA is a random variable characterized by a known prior pdf f (). We further assume that the sensor elements are constrained 2 , which is bounded by a to lie in a closed, connected region D0 2 2m denote the constraint D0 closed curve 0. Let D = D0 region for the array element location matrix; thus, an admissible array geometry satisfies r D . In determining optimal array designs, we adopt a Bayesian approach and propose two different but related cost functions. We define a CRBoptimal array rC as one whose element locations satisfy

2

= arg min r2D JC (r)

(7)

where the CRB cost function JC (r) is given by

f

JC (r) = E CRB(r; ) 

=

0

g

CRB(r; )f ()d



=

0

1

FIM(r; )

f ()d:

(8)

Similarly, we define the FIM optimal array rF by rF

= arg max r2D JF (r)

f

JF (r) = E FIM(r; )

In general, rC

g=



K

(9) 

0

=

0

1 (r 0 r )T K ( r 0 r ) A A

(11)

m

du() du() T f ()d d d

(12)

which is quadratic in r. Quadratic optimization functions are useful because they lead to closed-form solutions for certain array boundaries, and they permit the use of efficient quadratic programming techniques for iteratively solving (9) when closed-form solutions are not available.

The cost functions JC (r) and JF (r) can be related to the Bayesian CRB, as we show below. The Bayesian CRB has been proposed for random parameter estimation and is a lower bound on the mean-squared error of the estimated parameter (see [14]). The Bayesian CRB is a global bound that includes the a priori DOA information encoded in the prior probability distribution function. For our problem, the Bayesian CRB on the DOA angle  , which is denoted BCRB (r; ), is given by (13) BCRB(r; ) = [JF (r) + I2 ]01 where JF (r) is given in (10), and I2

FIM(r; )f ()d:

6= rF because of the integrations in (8) and (10).

(10)

= E

@ 2 ln f () @2

(14)

is the Fisher information of the prior. Since I2 is independent of array geometry, the FIM-optimal array rF also minimizes the Bayesian CRB on the DOA angle. The Bayesian CRB can also be related to JC (r). Since (1)01 is a convex function for positive arguments, by Jensen’s inequality, it follows that

 

212

rC

JF (r) = P tr

A. Relationship to Bayesian CRB

FIM(r; ) = G(r; ) 1 P G(r; ) =

1

(3)

f

1

E F IM (r; )

g  E

1

F IM (r; )

:

(15)

Combining (13) and (15) gives the following relation: BCRB(r; ) =

1

JF (r) + I2

 JF1(r)  JC (r):

(16)

The Bayesian CRB is thus bounded above by the CRB cost function, and the CRB-optimal array rC minimizes that bound. In [20], it is shown that the Bayesian CRB is unrealistically low for uniform distributions on  since the term I2 tends to infinity for uniform distributions. The FIM-optimal array design above minimizes the Bayesian CRB but removes the term in the Bayesian CRB that tends to infinity and that is any way independent of the array element locations; similar comments apply to JC (r). Thus, the functions JC (r) and JF (r) appear to be better suited than the Bayesian CRB for array design in scenarios where the DOA angle has uniform distribution. The FIM and CRB are derived using a small perturbation analysis. The resulting bounds are tight bounds for maximum likelihood estimates of DOA for high SNR, but they are typically not tight bounds at low SNR, mainly because they do not take into account the effects of high sidelobes or ambiguity directions in the array beampattern. Other possible bounds for random parameter estimation are the Ziv–Zakai lower bound (ZZLB) and the Weiss–Weinstein lower bound (WWLB) [15], [20], [21]. A design framework based on WWLB is presented for linear arrays in [22]. Although the ZZLB and WWLB provide tighter and more realistic bounds (especially at low SNRs), they are computationally intensive to determine. When optimizing for a single DOA, the computational expense may be acceptable, but when the optimization criterion contains a range of DOAs as in (8) or (10), the computation of the ZZLB becomes significant. For most cases, minimizing (8) [or

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1921

maximizing (10)] involves an iterative search for r. With the FIM criterion, the integral in (12) is evaluated once, whereas with the CRB criterion, the integral in (8) must be evaluated at each iteration on r. Both integrals are computationally simple. In contrast, the ZZLB involves computing an integral for every  in the support of f() and at each iteration on r; this is a significant increase the required computational load. An approximate closed-form expression for the ZZLB is derived in [15], but the approximation assumes that the array geometry is such that sidelobes of the beampattern are not significant, which is precisely the assumption we would attempt to avoid in replacing FIM(r; ) with a different bound. Thus, to keep computation tractable while maintaining a criterion that is based on a bound that is tight above a threshold SNR, we adopt the FIM and CRB criteria. B. Relationship to Beamwidth The beamwidth of the mainlobe for a delay-and-sum beamformer is proportional to the square root of the CRB (with the asymptotic standard deviation). Using the second-order Taylor series approximation of the array gain around the steering angle 0 , one can approximate the half-power beamwidth of the array as (0 =2) 1=G(B; 0 ) [15], [16], [19]. The CRB cost function can thus be thought of as the average beamwidth of the array (averaged over steering angle), and the CRB-optimal array gives the minimum average beamwidth. For high SNRs, the mainlobe width is a good indicator of the DOA estimation performance and resolution of the array. C. Cost Functions for Uniform Prior Distributions It is possible to obtain closed-form expressions for the integrals in the cost functions JC (r) and JF (r) when the DOA to be estimated has uniform probability distribution over a subset of [0; ]. Such a special case is useful in many practical scenarios. For example, when no prior knowledge is available about the DOA, one typically assumes that f() is uniformly distributed on [0; ]. In addition, for arrays that monitor a certain sector of angles, the prior DOA pdf may be chosen as uniformly distributed in that sector. Assume that f() is uniformly distributed in the interval [1 ; 2 ]  [0; ] with 1 < 2 . By suitably rotating the element constraint region, we can, without loss of generality, take 1 = 00 =2 and 2 = 0 =2. The array geometry dependent term G(r; ) in the CRB can be written as a function of the eigenvalues i and entries Bij of the array covariance matrix B given in (6). Let 1  2 be the eigenvalues of B. A straightforward calculation gives

G(r; ) = a 0 b cos(2 0 ) a = B11 + B22 = 1 (1 + 2 ) 2 2 B11 0 B22 2 + B 2 = 1 ( 0  ) b= 12 2 2 1 2  = tan01 B11 0 B22 : 2B12

(18) (19) (20)

size and increases as the sensors are moved away from the origin. The term b can be interpreted as an isotropy term—the array has constant CRB performance for all angles if and only if b = 0 (see [16]), and larger values of b correspond to larger changes in CRB performance as a function of DOA  . Thus, we see that the FIM criterion attempts to maximize the average aperture, whereas the CRB criterion also tends to favor isotropic arrays. These properties are seen in the examples in Section V. IV. BOUNDARY RESULT

For general boundaries or prior pdfs, it is not possible to find an analytic solution for either of the optimization problems in (8) or (10); therefore, iterative optimization procedures are employed. The optimal solution is found as a 2m-dimensional search for element locations fxi ; yi gmi=1 . In this section, we show that optimal solutions have all array elements on the boundary of the constraint region. If the constraint region D0 is convex, this boundary result is a direct consequence from optimization theory since G(r; ) is a convex function in 2m . In this section, we show that for the optimal array all elements are on the boundary even for nonconvex constraint regions. The boundary result not only reduces the search dimension from 2m to m but provides a convenient ordering of elements along the boundary to eliminate the nonuniqueness of solution corresponding to interchanging element locations of two or more elements. In particular, by parameterizing the boundary 0 as 0(t) for t 2 [0; 1] and correspondingly parameterizing each array element location ri as a point by r(ti ) on the boundary, we reduce the search space to the compact subset [t1 ; . . . ; tm ]T 2 m with 0 < t1 < 1 1 1 < tm  1. To establish the boundary result, we first show that moving a sensor away from the array centroid increases the term G(r; ) that appears in the CRB. We note that G(r; ) is invariant to translation of the entire array [see (5) and (6)], we will, without loss of generality, assume rA = 0. Lemma 1: Assume that r is the array location matrix of an m eler formed by ment array centered at the origin. Consider another array ~ moving the j th sensor away from the origin:

~r = [r1 ; r2 ; . . . ; (1 + )rj ; rj +1 ; . . . ; rm ] where

1 tan 0 0  2 2 1 tan 0 +  2 2

:

(25)

> 0. Then

G(~r; ) > G(r; ); for  2 [0; ) 0 f ; + g G(~r; ) = G(r; ); for  2 f ; + g where = tan01 (yj =xj ). r in (25), the array centroid is r~a = ( =m)rj , and Proof: For ~ rA = ( =m)[rj ; . . . ; rj ]. Then the corresponding centroid matrix is ~

(21)

+ tan01

=  [that is, f() is uniformly distributed on [0; )],

JF (r) = 1 (1 + 2 ) = a (23) 2 JC (r) = p 1 = p 21 2 : (24) 1 2 a 0b The term a = (1=2)tr(B) can be interpreted as an average aperture

(17)

When f() is uniform over [00 =2; 0 =2], the FIM and CRB cost functions are given by

JF (r)=a;0b cos  sin 0 0 1 1 JC (r)= p tan01 0 1 2

Note that for 0 then

(22)

TLFeBOOK

T G(~r; ) = @u (~r 0 ~rA )(~r 0 ~rA )T @u @ @ T m 0 1 2 r rT @u @u T rr + 2 + = j j @ m @ T = G(r; ) + 2 + m 0 1 2 @u rj rjT @u : m @ @  ( )

1922

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

Since > 0,  ()  0 and is equal to zero only when (@u=@) ? rj or, equivalently, when  = or  = +  . When a sensor is moved away from the array centroid, the term G(r; ) strictly increases except at two DOA angles. As long as f () has support region greater than just these two angles, the optimization criteria JC (r) and JF (r) will strictly increase. Thus, optimal arrays have all elements on the boundary of the constraint region. The following theorem establishes this result. Theorem 1: Assume the prior pdf f () has support on a set of nonzero measure. Then, elements of the FIM-optimal and CRB-optimal arrays rF and rC lie on the design constraint boundary 0. Proof: We will prove Theorem 1 using JC (r); the proof for JF (r) is nearly identical. Let r = [r1 ; . . . ; rm ] be the array location matrix of an optimal array so that r is a solution to (7). Assume without loss of generality that the centroid of this array is at the origin. Assume that sensor rj (for some 1  j  m) is not on the boundary 0. Then, there is a neighborhood around rj that lies in D0 . Consider r in (25), where > 0 another array with element locations given by ~ is chosen such that r~j 2 D0 . By Lemma 1  JC (~r) = P 01 G(~r; )01 f ()d < JC (r) 0

which contradicts the statement that r minimizes JC (r) in (7). Thus, every optimal array must have all elements on the boundary 0. From the discussion above Theorem 1, the assumption that the pdf f () has support of nonzero measure can be replaced by the (weaker) assumption that f () has nonzero measure on a set of greater than two points. Theorem 1 provides a qualitative explanation for array geometries designed according to the criteria proposed in [13] and [6]. The aim in [13] is to find the array that minimize the mainlobe area while satisfying a sidelobe constraint. In [13], it is noted that optimal designs have most elements either on or near the constraint boundary. Without the sidelobe level constraint, minimizing the mainlobe width corresponds to minimizing the CRB criterion because the single-source CRB is directly related to mainlobe width (see, e.g., [15], [16], and [19]); by Theorem 1, all array elements would be on the boundary in this case. Apparently, the sidelobe constraint does not significantly alter the array placement. In [6], an ML estimator (which asymptotically achieves the CRB) is used to estimate the DOAs, and the optimal nonuniform linear array is designed by minimizing the variance of the DOA estimates. Since the CRB describes the asymptotic ML performance, we expect elements to lie on the boundary in this case as well. V. EXAMPLES We present two examples of arrays designed using the cost functions

JC (r) and JF (r). First, consider an example in which the sensors are constrained to lie in a disk of radius R0 . By Theorem 1, all elements of the optimal array satisfy jri j = R0 . For the cases in which either f () is a uniform distribution over [0; ] or [0; ), it can be shown

that every solution for rC and rF is an isotropic array. Isotropic arrays are studied in [16]; a planar isotropic array is one whose single-source CRB is independent of the arrival angle  . If the prior DOA pdf is not uniform as above, then the optimal array is no longer isotropic. As an example, consider the circular constraint region as above, along with a DOA pdf shown in Fig. 1(a). The pdf represents a scenario where the signal can be impinging on the array from any direction, but it is expected primarily from a particular sector. Fig. 1(b) shows the resulting five-element FIM-optimal array (which is also the Bayesian CRB-optimal array); note that all elements cluster at the top and bottom to give the widest aperture for signals arriving around 0 . This is similar to the optimum linear array design in which

Fig. 1. Example array geometry design when the boundary of the constraint region is a circle. (a) The pdf f () of the single-source DOA. (b) Five-element FIM-optimal array. (c) Five-element CRB-optimal array  = 78 , = 45 .

half of the array elements are clustered at each end of the constraint line segment [23]. The CRB-optimal array geometry in Fig. 1(c) has elements spread around the boundary. While the CRB-optimal array is not isotropic (an isotropic array has equally spaced elements), it is nearly so but has lower beamwidths for arrival angles close to 0 and 180 . These observations agree with the maximum aperture character of FIM-optimal arrays and the aperture-and-isotropic character of CRB-optimal arrays, as discussed at the end of Section III-C. In addition, the CRB-optimal array designs generally have lower sidelobe levels than do FIM-optimal designs. For these reasons, the CRB-optimal designs are preferable in most applications. VI. CONCLUSIONS We have considered optimal planar array designs using the average CRB and average FIM as performance criteria. Prior information on the source DOA is encoded as a prior probability density function; this allowed us to address applications in which not all directions are equally likely. Closed-form expressions for the optimization criteria JC (r) and JF (r) were derived when the DOA prior was uniform on a sector of angles. We showed that sensors of both CRB-optimal and FIM-optimal arrays lie on the boundary of the array constraint region, even when that region is not convex. As a result, the dimension of the design optimization is reduced from 2m to m. We also related the two proposed optimization criteria to the Bayesian CRB and to average array beamwidth. We showed that the FIM-optimal array also minimizes the Bayesian CRB and that the CRB-optimal criterion bounds the Bayesian CRB. The boundary result, in conjunction with expressions relating the CRB to the array beamwidth, provided geometric interpretation of the optimality criteria and the resulting array designs. Because the CRB is a realistic bound for moderate to high SNRs and can be optimistic for low values of SNR, the design criteria we consider apply to moderate or high SNR applications. In [20], it is shown that the WWLB and ZZLB provides tighter bounds than the BCRB, and they converge to the BCRB above a threshold SNR. However, the

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1923

use of the WWLB or ZZLB increases the computational cost of the approach substantially. In practice, our geometry designs should perform well for SNRs that are above this threshold SNR. For lower SNR applications, it may be of interest to design arrays with low sidelobe levels. The methods presented in this paper are based on the FIM and CRB, whose properties relate to mainlobe width but not sidelobe level. The CRB-based designs tend to have lower sidelobe levels than the FIM-based designs and are therefore preferable in most applications. If low sidelobe level is an additional design requirement, the FIM and CRB criteria may be combined or augmented with other criteria or constraints, such as those in [10] and [13], to obtain desired sidelobe performance; alternately, a ZZLB- or WWLB-based criterion can be used. Although we have focused on design of planar arrays that estimate azimuth DOA, the methods apply to other scenarios as well. The design method applies to volume arrays, to DOA angles in azimuth and elevation, and to both narrowband and wideband arrays. The design procedure follows because in all of these cases, the expression for the FIM can be partitioned as in (4)–(6), where P is a scalar, and G(r; ) contains the array geometry terms.

[18] M. A. Doron and E. Doron, “Wavefield modeling and array processing, Part III—Resolution capacity,” IEEE Trans. Signal Process., vol. 42, no. 10, pp. 2571–2580, Oct. 1994. [19] H. Messer, “Source localization performance and the array beampattern,” Signal Process., vol. 28, pp. 163–181, Aug. 1992. [20] H. Nguyen and H. L. Van Trees, “Comparison of performance bounds for DOA estimation,” in Proc. IEEE Seventh Signal Process. Workshop Statistical Signal Array Process., Quebec City, QC, Canada, 1994, pp. 313–316. [21] K. L. Bell, Y. Steinberg, Y. Ephraim, and H. L. Van Trees, “Extended Ziv–Zakai lower bound for vector parameter estimation,” IEEE Trans. Inf. Theory, vol. 43, no. 2, pp. 624–637, Mar. 1997. [22] F. Athley, “Optimization of element positions for direction finding with sparse arrays,” in Proc. 11th IEEE Signal Statistical Signal Process. Workshop Statistical Signal Process., Singapore, 2001, pp. 516–519. [23] V. H. MacDonald and P. M. Schultheiss, “Optimum passive bearing estimation in a spatially incoherent noise environment,” J. Acoust. Soc. Amer., vol. 46, pp. 37–43, 1969.

Computation of Spectral and Root MUSIC Through Real Polynomial Rooting

REFERENCES [1] H. L. Van Trees, Optimum Array Processing. New York: Wiley, 2002. [2] V. Murino, “Simulated annealing approach for the design of unequally spaced arrays,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 5, Detroit, MI, 1995, pp. 3627–3630. [3] A. B. Gershman and J. F. Böhme, “A note on most favorable array geometries for DOA estimation and array interpolation,” IEEE Signal Process. Lett., vol. 4, no. 8, pp. 232–235, Aug. 1997. [4] D. Pearson, S. U. Pillai, and Y. Lee, “An algorithm for near-optimal placement of sensor elements,” IEEE Trans. Inf. Theory, vol. 36, no. 6, pp. 1280–1284, Nov. 1990. [5] H. Alnajjar and D. W. Wilkes, “Adapting the geometry of a sensor subarray,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 4, Minneapolis, MN, 1993, pp. 113–116. [6] X. Huang, J. P. Reilly, and M. Wong, “Optimal design of linear array of sensors,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 2, Toronto, ON, Canada, 1991, pp. 1405–1408. [7] J.-W. Liang and A. J. Paulraj, “On optimizing base station antenna array topology for coverage extension in cellular radio networks,” in Proc. IEEE 45th Veh. Technol. Conf., vol. 2, Stanford, CA, 1995, pp. 866–870. [8] Y. Hua, T. K. Sarkar, and D. D. Weiner, “An L-shaped array for estimating 2-D directions of wave arrival,” IEEE Trans. Antennas Propag., vol. 39, no. 2, pp. 143–146, Feb. 1991. [9] A. Manikas, A. Alexiou, and H. R. Karimi, “Comparison of the ultimate direction-finding capabilities of a number of planar array geometries,” Proc. Inst. Elect. Eng.—Radar, Sonar, Navigat., pp. 321–329, 1997. [10] M. Gavish and A. J. Weiss, “Array geometry for ambiguity resolution in direction finding,” IEEE Trans. Antennas Propag., vol. 44, no. 6, pp. 889–895, Jun. 1996. [11] A. Manikas, A. Sleiman, and I. Dacos, “Manifold studies of nonlinear antenna array geometries,” IEEE Trans. Signal Process., vol. 49, no. 3, pp. 497–506, Mar. 2001. [12] N. Dowlut and A. Manikas, “A polynomial rooting approach to superresolution array design,” IEEE Trans. Signal Process., vol. 48, no. 6, pp. 1559–1569, Jun. 2000. [13] M. Viberg and C. Engdahl, “Element position considerations for robust direction finding using sparse arrays,” in Conf. Rec. Thirty Third Asilomar Conf. Signals, Syst., Comput., vol. 2, Pacific Grove, CA, 1999, pp. 835–839. [14] H. L. Van Trees, Detection, Estimation, and Modulation Theory. New York: Wiley, 1968, vol. 1. [15] K. L. Bell, Y. Ephraim, and H. L. Van Trees, “Explicit Ziv–Zakai lower bound for bearing estimation,” IEEE Trans. Signal Process., vol. 44, no. 11, pp. 2810–2824, Nov. 1996. [16] Ü. Baysal and R. L. Moses, “On the geometry of isotropic arrays,” IEEE Trans. Signal Process., vol. 51, no. 6, pp. 1469–1478, Jun. 2003. [17] A. J. Weiss and B. Friedlander, “On the Cramér–Rao bound for direction finding of correlated signals,” IEEE Trans. Signal Process., vol. 41, no. 1, pp. 495–499, Jan. 1993.

J. Selva, Student Member, IEEE Abstract—We present a technique to compute the Spectral and (Unitary) Root MUSIC estimations from the roots of a real polynomial in the problem of estimating the angles of arrival to a Uniform Linear Array (ULA). The computed estimations are the actual MUSIC or (Unitary) Root MUSIC estimations, but the need for a one-dimensional (1-D) search in Spectral MUSIC is eliminated, and the number of polynomial coefficients to compute in (Unitary) Root MUSIC is halved. Besides, the proposed technique makes it possible to perform the polynomial rooting step in these methods through real arithmetic. Index Terms—Array signal processing, direction-of-arrival estimation, frequency estimation, , .

NOTATION Most of the notation in this correspondence is the usual in array processing. The exceptions are the following: • The symbol “” is used to define a new symbol or function. [ ]m;k and [ ]m denote the m,k , and m components of the • matrix and vector , respectively. • A centered dot in one of the indices refers to the full row or column: [ ]1;k and [ ]m;1 are the k th column and mth row of , respectively. • An exchange matrix of size M 2 M is a matrix of zeros with ones in its anti-diagonal: [ ]m;q = 1 if m + q = M + 1, and [ ]m;q = 0 otherwise, for m, q = 1; . . . ; M .

A

a

A

A

J

A

a A J

J

I. INTRODUCTION In Spectral MUSIC [1], a one-dimensional (1-D) search has to be performed in order to locate the spectrum local maxima. This search is usually implemented by sampling the spatial frequency range in a fine Manuscript received November 10, 2003; revised May 26, 2004. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vikram Krishnamurthy. The author is with the German Aerospace Centre (DLR), Institute of Communications and Navigation, Oberpfaffenhofen, 82234 Wessling, Germany. Digital Object Identifier 10.1109/TSP.2005.845489

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1923

use of the WWLB or ZZLB increases the computational cost of the approach substantially. In practice, our geometry designs should perform well for SNRs that are above this threshold SNR. For lower SNR applications, it may be of interest to design arrays with low sidelobe levels. The methods presented in this paper are based on the FIM and CRB, whose properties relate to mainlobe width but not sidelobe level. The CRB-based designs tend to have lower sidelobe levels than the FIM-based designs and are therefore preferable in most applications. If low sidelobe level is an additional design requirement, the FIM and CRB criteria may be combined or augmented with other criteria or constraints, such as those in [10] and [13], to obtain desired sidelobe performance; alternately, a ZZLB- or WWLB-based criterion can be used. Although we have focused on design of planar arrays that estimate azimuth DOA, the methods apply to other scenarios as well. The design method applies to volume arrays, to DOA angles in azimuth and elevation, and to both narrowband and wideband arrays. The design procedure follows because in all of these cases, the expression for the FIM can be partitioned as in (4)–(6), where P is a scalar, and G(r; ) contains the array geometry terms.

[18] M. A. Doron and E. Doron, “Wavefield modeling and array processing, Part III—Resolution capacity,” IEEE Trans. Signal Process., vol. 42, no. 10, pp. 2571–2580, Oct. 1994. [19] H. Messer, “Source localization performance and the array beampattern,” Signal Process., vol. 28, pp. 163–181, Aug. 1992. [20] H. Nguyen and H. L. Van Trees, “Comparison of performance bounds for DOA estimation,” in Proc. IEEE Seventh Signal Process. Workshop Statistical Signal Array Process., Quebec City, QC, Canada, 1994, pp. 313–316. [21] K. L. Bell, Y. Steinberg, Y. Ephraim, and H. L. Van Trees, “Extended Ziv–Zakai lower bound for vector parameter estimation,” IEEE Trans. Inf. Theory, vol. 43, no. 2, pp. 624–637, Mar. 1997. [22] F. Athley, “Optimization of element positions for direction finding with sparse arrays,” in Proc. 11th IEEE Signal Statistical Signal Process. Workshop Statistical Signal Process., Singapore, 2001, pp. 516–519. [23] V. H. MacDonald and P. M. Schultheiss, “Optimum passive bearing estimation in a spatially incoherent noise environment,” J. Acoust. Soc. Amer., vol. 46, pp. 37–43, 1969.

Computation of Spectral and Root MUSIC Through Real Polynomial Rooting

REFERENCES [1] H. L. Van Trees, Optimum Array Processing. New York: Wiley, 2002. [2] V. Murino, “Simulated annealing approach for the design of unequally spaced arrays,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 5, Detroit, MI, 1995, pp. 3627–3630. [3] A. B. Gershman and J. F. Böhme, “A note on most favorable array geometries for DOA estimation and array interpolation,” IEEE Signal Process. Lett., vol. 4, no. 8, pp. 232–235, Aug. 1997. [4] D. Pearson, S. U. Pillai, and Y. Lee, “An algorithm for near-optimal placement of sensor elements,” IEEE Trans. Inf. Theory, vol. 36, no. 6, pp. 1280–1284, Nov. 1990. [5] H. Alnajjar and D. W. Wilkes, “Adapting the geometry of a sensor subarray,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 4, Minneapolis, MN, 1993, pp. 113–116. [6] X. Huang, J. P. Reilly, and M. Wong, “Optimal design of linear array of sensors,” in Proc. Int. Conf. Acoust., Speech, Signal Process., vol. 2, Toronto, ON, Canada, 1991, pp. 1405–1408. [7] J.-W. Liang and A. J. Paulraj, “On optimizing base station antenna array topology for coverage extension in cellular radio networks,” in Proc. IEEE 45th Veh. Technol. Conf., vol. 2, Stanford, CA, 1995, pp. 866–870. [8] Y. Hua, T. K. Sarkar, and D. D. Weiner, “An L-shaped array for estimating 2-D directions of wave arrival,” IEEE Trans. Antennas Propag., vol. 39, no. 2, pp. 143–146, Feb. 1991. [9] A. Manikas, A. Alexiou, and H. R. Karimi, “Comparison of the ultimate direction-finding capabilities of a number of planar array geometries,” Proc. Inst. Elect. Eng.—Radar, Sonar, Navigat., pp. 321–329, 1997. [10] M. Gavish and A. J. Weiss, “Array geometry for ambiguity resolution in direction finding,” IEEE Trans. Antennas Propag., vol. 44, no. 6, pp. 889–895, Jun. 1996. [11] A. Manikas, A. Sleiman, and I. Dacos, “Manifold studies of nonlinear antenna array geometries,” IEEE Trans. Signal Process., vol. 49, no. 3, pp. 497–506, Mar. 2001. [12] N. Dowlut and A. Manikas, “A polynomial rooting approach to superresolution array design,” IEEE Trans. Signal Process., vol. 48, no. 6, pp. 1559–1569, Jun. 2000. [13] M. Viberg and C. Engdahl, “Element position considerations for robust direction finding using sparse arrays,” in Conf. Rec. Thirty Third Asilomar Conf. Signals, Syst., Comput., vol. 2, Pacific Grove, CA, 1999, pp. 835–839. [14] H. L. Van Trees, Detection, Estimation, and Modulation Theory. New York: Wiley, 1968, vol. 1. [15] K. L. Bell, Y. Ephraim, and H. L. Van Trees, “Explicit Ziv–Zakai lower bound for bearing estimation,” IEEE Trans. Signal Process., vol. 44, no. 11, pp. 2810–2824, Nov. 1996. [16] Ü. Baysal and R. L. Moses, “On the geometry of isotropic arrays,” IEEE Trans. Signal Process., vol. 51, no. 6, pp. 1469–1478, Jun. 2003. [17] A. J. Weiss and B. Friedlander, “On the Cramér–Rao bound for direction finding of correlated signals,” IEEE Trans. Signal Process., vol. 41, no. 1, pp. 495–499, Jan. 1993.

J. Selva, Student Member, IEEE Abstract—We present a technique to compute the Spectral and (Unitary) Root MUSIC estimations from the roots of a real polynomial in the problem of estimating the angles of arrival to a Uniform Linear Array (ULA). The computed estimations are the actual MUSIC or (Unitary) Root MUSIC estimations, but the need for a one-dimensional (1-D) search in Spectral MUSIC is eliminated, and the number of polynomial coefficients to compute in (Unitary) Root MUSIC is halved. Besides, the proposed technique makes it possible to perform the polynomial rooting step in these methods through real arithmetic. Index Terms—Array signal processing, direction-of-arrival estimation, frequency estimation, , .

NOTATION Most of the notation in this correspondence is the usual in array processing. The exceptions are the following: • The symbol “” is used to define a new symbol or function. [ ]m;k and [ ]m denote the m,k , and m components of the • matrix and vector , respectively. • A centered dot in one of the indices refers to the full row or column: [ ]1;k and [ ]m;1 are the k th column and mth row of , respectively. • An exchange matrix of size M 2 M is a matrix of zeros with ones in its anti-diagonal: [ ]m;q = 1 if m + q = M + 1, and [ ]m;q = 0 otherwise, for m, q = 1; . . . ; M .

A

a

A

A

J

A

a A J

J

I. INTRODUCTION In Spectral MUSIC [1], a one-dimensional (1-D) search has to be performed in order to locate the spectrum local maxima. This search is usually implemented by sampling the spatial frequency range in a fine Manuscript received November 10, 2003; revised May 26, 2004. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vikram Krishnamurthy. The author is with the German Aerospace Centre (DLR), Institute of Communications and Navigation, Oberpfaffenhofen, 82234 Wessling, Germany. Digital Object Identifier 10.1109/TSP.2005.845489

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1924

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

grid and then employing the fast Fourier transform (FFT) with zero padding [2]. For a Uniform Linear Array (ULA), this search is avoided in (Unitary) Root MUSIC through the use of the polynomial parameterization [3], [4]. Root MUSIC requires finding the roots of a complex polynomial of degree 2M 0 2, where M is the number of sensors. Comparing Spectral and Root MUSIC, we may realize that the free parameter in both has different domains: the upper Unit Semi-circle for Spectral MUSIC and the Complex Plane for Root MUSIC. However, the question remains of whether there is an efficient way to calculate the Spectral MUSIC estimation in a ULA possibly through a polynomial. In this correspondence, we show that the Spectral MUSIC estimations can be calculated from the real roots of a real (2M 0 2)-degree polynomial that lie inside the [01; 1] range. The proposed technique is based on a conformal mapping of the Unit Circle onto the Real Line. This same method is also applicable to Root MUSIC, and we also show that its estimations can be calculated from the complex conjugate roots of a real (2M 0 2)-degree polynomial. This fact halves the number of polynomial coefficients that have to be calculated in Root MUSIC in comparison with the conventional method and makes it possible to apply fast rooting algorithms for real polynomials. For both methods, the proposed technique only reduces the computational burden, i.e., the estimations computed are the same as with the standard MUSIC and Root MUSIC algorithms. II. ARRAY SIGNAL MODEL Assume that K signals sk (t) arrive at a ULA with spatial frequencies w1 ; w2 ; . . . wK . Let a(w) denote the array response, i.e., [a(wk )]m

 ej m 0 (

1)w

; m = 1; . . . ; M:

(1)

If we stack the signals sk (t), the spatial frequencies wk , and the array responses a(wk ) in, respectively, a vector s(t), a vector w, and a matrix A(w), the array observation is

x(t) = A(w)s(t) + n(t)

(2)

(Here, it is also possible to apply the forward-backward averaging [5], but we keep the conventional method for simplicity.) The estimation of w is given by the abscissas for which the cost function LSM

[FP ]p;q

X = A(w)S + N

(4)

where we have, respectively, stacked in X, S, and N the snapshots of x(t), s(t), and n(t), i.e., for n = 1; . . . ; N , we have defined

 n(n 0 1):

(5)

III. COMPUTATION OF SPECTRAL MUSIC The Spectral MUSIC algorithm starts with the calculation of a matrix I, of size M 2 K , that spans the invariant subspace corresponding to the K greater eigenvalues of the sample covariance matrix ^ HU ^ = ^, U U

^ R

N 1

XXH :

2 (

1)(

1)

:

(8)

(9)

^ 3 )1K is the vector of ^F U then, it can be easily verified that P 2 (U F values of (7) at all wo;p . The angular frequencies wo;p corresponding to the greatest K local maxima in this vector are used to initialize the 1-D searches: one for each local maximum. The most expensive operations in this conventional method are the 1-D searches, given that each of them requires to evaluate (8) repeatedly. The method presented in this correspondence allows us to compute the Spectral MUSIC estimations from the real roots of a real polynomial, completely avoiding these expensive 1-D searches. We proceed to introduce the new method, which is based on differentiating (7) in w, and then mapping the domain of the resulting equation into the Real Line. The local maxima of (7) correspond to some of the zeros of its differential in w , which, after straightforward manipulations on (7), can be written as

dLSM

(3)

and the components of n(t) are independent white noise processes. Assume that the receiver takes N samples of x(t) at t = 0; 1; . . . ; N 0 1. From (2), we obtain the model

and [N]1;n

 P1 e0j  p0 q0

^ U ^ F  FP U 0(P 0M )2K

[D]m;p

1

 s(n 0 1);

(7)

= 2Re

^U ^ H a( w ) a( w ) H D H U

(10)

where we have defined the diagonal matrix

 s k (t ) [w]k  wk [A(w)] ;k  a(wk )

[S]1;n

^U ^ H a( w ) U

^ F is the result of performing the Inverse DFT on the columns of U ^ If U with zero padding, i.e.,

dw

[s(t)]k

 x(n 0 1);

H

has local maxima inside [0;  ]. In the usual method [2], the local maxima are first coarsely located through an FFT with zero padding, and their exact positions are then determined through 1-D searches: one for each local maximum. Specifically, consider the frequencies wo;p  2 (p 0 1)=P , p = 1; . . . ; P with P  M and the Inverse DFT matrix of size P

where, for k = 1; . . . ; K , we have defined

[X]1;n

 a( w )

(6)



j (m 0 1); if m = p if m 6= p m = 1; . . . ; M; p = 1; . . . ; M: (11)

0;

Let us define the complex variable

z (w)  ejw

(12)

which transforms the [0;  ] range into the superior part of the Unit Circle, and introduce the following conformal transformation into a new complex variable u:

z0j : z+j

(13)

0 4

(14)

u ( z )  ( 0j )

u(z ) maps the upper Unit Semi-circle into the [01; 1] range of the Real Line (Fig. 1). Besides, its inverse transformation is itself, and it can be easily shown that u and w are related through the equation

u = tan

w 2

:

In order to introduce the u variable in (10), let us apply (12) and (13) to the components of a(w) [a(w )]m = e

TLFeBOOK

jw(m01) = z m01 = (

0j )m0

1

u 0 j m01 : (15) u+j

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1925

product of two real matrices of respective sizes K 2 M and M 2 M , (4M 2 K real flops). This is due to the fact that either the real or the imaginary part of any component of or is zero, as can be easily deduced from (16). Substituting (22) in (17) and using the properties of the “Ref1g” operator, we obtain

C DC

2)

v(u)T

RefM1 gT RefM2 g + ImfM1 gT ImfM2 g

M v

v ( u ) = 0:

(23)

Every product like Imf 2 g (u) produces a length-K vector of real polynomials. Let us define them. For k = 1; . . . ; K , we have

 [RefM1 g]  [ImfM1 g] (u)  [RefM2 g] (u)  [ImfM2 g]

p1R;k (u)

k;1

p1I;k (u) p2R;k p2I;k

k;1

k;1

k;1

v (u ) v (u ) v (u ) v (u ):

(24)

With this, we may write (23) as K

Fig. 1. Conformal transformation.

k=1

If we multiply (w) by (u + j )M 01 , then all components of the resulting vector are polynomials in u of degree M 0 1. Specifically, define the vector of polynomials

a

[ (u)]m

 (0j ) 01(u 0 j ) 01(u + j ) 0 m

m

M

m

; m = 1; . . . ; M:

(16)

Then, we have the relationship

a(w) = (u)(u + j )10

M

:

(17)

Besides, given that the components of  (u) are (M 0 1)-degree polynomials, there is an M 2 M matrix containing their coefficients for which

C

(u) = Cv(u)

v

(18)

where (u) is Vandermonde

v

[ (u)]m

 u 01 ; m

m = 1; . . . ; M:

(19)

Using (17) and (18) in (10), we achieve the equation

0 j )10 v(u)T CH DH U^ U^ H Cv(u)(u + j )10 = 0: (20) The product (u 0 j )10 (u + j )10 is real and positive for real u, 2Re (u

M

M

v

and therefore, it can be suppressed. Since (u) is a real vector, it can be taken out of the “Ref1g” operator. Performing these operations, we obtain

v(u)T RefCH DHU^ U^ H Cgv(u) = 0:

(21)

Note that this is a real (2M 0 2)-degree polynomial. Therefore, the Spectral MUSIC estimation can be obtained through (14) from the real roots of (21) that lie inside [01; 1]. IV. EFFICIENT COMPUTATION OF THE POLYNOMIAL COEFFICIENTS We may efficiently calculate the polynomial in (21) in the following steps: 1) Calculate the products

M1  U^ H (DC)

M2  U^ H C: (22) H ^ In the former, only the product of U with DC is required because DC can be precalculated. The complexity of this and

step is roughly equal to four times the one of performing the

(25)

Therefore, the final polynomial is the sum of 2K polynomials, each being the product of two polynomials of degree M 0 1. These products can be efficiently performed by convolving the polynomial coefficients. We have shown that the cost of calculating the polynomial coefficients from ^ is roughly equal to 4M 2 K real flops plus 2K times the cost of computing the convolution of two M -length real vectors.

U

V. EFFICIENT POLYNOMIAL ROOTING AND MAXIMA SELECTION The roots of (25) can be calculated using any of the efficient methods available for finding the real roots of a real polynomial [6, ch. 9]. We must note that it is only necessary to calculate the roots that lie inside the [01; 1] range. Once the roots r^1 ; r^2 ; . . . ; r^P are available, the only task left is to compute the spatial frequencies corresponding to the K greatest local maxima of the MUSIC spectrum. This can be done in the following steps: 1) Compute the spatial frequencies using the inverse of (14)

 w^p  2 arctan(^rp ) + ; p = 1; . . . ; P:

M

M

p1R;k (u)p2R;k (u) + p1I;k (u)p2I;k (u) = 0:

2

2)

(26)

Compute the spectrum value at these frequencies ^U ^ H a(w ^ p ): vp  a(w^p )H U

(27)

Find out the indices i1 ; i2 ; . . . ; iK for which the sequence vi ; vi ; . . . ; vi takes on the K maximum values. The ^i ; . . . ; w ^i . ^i ; w Spectral MUSIC estimations are w It would also be possible to reduce the complexity of step 2 by applying the conformal transformation to the spectrum in (7). 3)

VI. ROOT-MUSIC THROUGH REAL POLYNOMIAL ROOTING The same conformal transformation makes it possible to derive a 0 2)-degree real polynomial whose roots determine the Root MUSIC estimations. The procedure is very similar to the one for Spectral MUSIC. This new variant has two advantages relative to the standard one, which stem from the fact that the polynomial has real coefficients. First, in the standard method, (2M 0 2) complex polynomial coefficients have to be computed, whereas in the new method, the same number of coefficients has to be computed, but they are real. Therefore, the number of real components (real or imaginary parts) to compute is halved. Second, the roots appear in pairs of the form a + jb, a 0 jb,

(2M

TLFeBOOK

1926

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

which makes fast rooting algorithms for complex roots of real polynomials applicable, like Bairstow’s method, which only requires real arithmetic [6, ch. 9]. In MUSIC, and assuming that there is no noise present, the zeros of the noise-subspace cost function LRM

 a(w) (I 0 U^ U^ ) a(w) = 0 H

H

(28)

X

correspond to the exact w values used to construct . In Root MUSIC, this property is restored in the noisy case by multiplying (28) by jw (M 01) e in order to leave only positive powers of ejw , and substituting z for ejw . The Root MUSIC estimations are then obtained from the roots of the resulting polynomial in z . We may easily introduce the jw (M 01) e factor in (28) if we note that (w)H ejw(M 01) = (w)T , where is an exchange matrix. Therefore, we obtain

a

J

a

a(w) J(I 0 U^ U^ )a(w) = 0: T

The substitution z for e

jw

H

J

(29)

yields the Root MUSIC polynomial

a(z) J(I 0 U^ U^ )a(z) = 0 T

H

(30)

m = 1; . . . ; M:

(31)

where we have defined

a

[ (z )]m

z 0 ; 1

m

Recalling (17) and (18), we may apply the conformal transformation in (13) to (30). We obtain

0M ) v(u)T CT J(I 0 U ^U ^ H )CV(u) = 0:

(u + j )

2(1

(32)

Fig. 2. MUSIC spectrum and rooted polynomial with scale factor 1/8. The polynomial roots and the critical points of the spectrum have been marked with vertical stems.

TABLE I VALUES OF u, w , AND L AT EACH CRITICAL POINT OF THE MUSIC SPECTRUM L . THE FIRST AND THIRD ROWS CORRESPOND TO THE LOCAL MAXIMA, (FIG. 2)

Next, we eliminate the scalar to the left

v(u) C J(I 0 U^ U^ )CV(u) = 0: T

T

H

(33)

The latter operation is valid because u = 0j corresponds to z = 1 in (30). In can be easily shown that the conformal transformation maps the Unit Circle conjugate roots of the polynomial in (30) into complex conjugate roots in (33), i.e, roots of the form a + jb, a 0 jb. As a consequence, the coefficients of the latter polynomial are real. This, and the fact that (u) is a real vector, allow us to take the real part of the matrix in (33),

v

v (u )

T

C J(I 0 U^ U^ )C v(u) = 0: T

Re

H

(34)

These two steps have an equivalent in the standard Root MUSIC algorithm. Step 1 is equivalent to finding out the roots closer to the ^i + j ^ bi in step 2 is equivalent ^i and not a Unit Circle, and to use a to taking only the angle of the computed roots.

The efficient computation of the polynomial coefficients is similar to the one in Section IV. We only have to substitute by and change (23) and (25) for

D J

v (u )

T

C JCg 0 RefM g

Ref

T

1

T

RefM2 g0

ImfM1 gT ImfM2 g

v (u ) = 0

(35)

VII. SHORT NUMERICAL EXAMPLE We have numerically tested the validity of the methods above for Spectral and Root MUSIC. Fig. 2 shows a trial of the MUSIC spectrum in the following scenario:

and K

pc (u)

0p

1R;k (

u)p2R;k (u) 0 p1I;k (u) p2I;k (u) = 0

M = 10 sensors with =2 spacing: K = 2 uncorrelated signals with equal amplitudes: w = [0:3; 0:9]T : N = 10 snapshots: S=N = 25 dB:

(36)

k=1

where pc (u) is the data-independent polynomial pc (u )

 v (u )

T

C JCgv(u):

Ref

T

(37)

^q + j ^ bq , The polynomial in (36) has pairs of roots of the form a a^q 0 j^bq . The Root MUSIC estimations would be calculated as follows: 1) Find out the indices i1 ; i2 ; . . . ; iK for which the sequence j^b1 j; j^b2j; . . . ; j^bM 01j takes on the K smallest values. ^i ; w ^i ; . . . ; w ^i with 2) The Root MUSIC estimations are w  w^i  2 arctan (^ai ) + : (38) 2

We can see that there is a root of the polynomial in (21) at the abscissa of each critical point of the MUSIC spectrum and vice versa. The critical points values are given in Table I. Note that the values at the first and third row correspond to the local maxima (see Fig. 2). In order to compare the computational burdens of the conventional and the proposed method, we have coarsely located the local maxima of the spectrum through the FFT method in (8) and (9) with P =

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

blog

2

1927

c . Then, we have counted the number of flops that a stan-

(M )+1

dard 1-D search algorithm require to locate the maxima abscissas and the number of flops required by Newton’s method based on Horner’s synthetic division [6, section 9.5]. They, respectively, required 3000 and 1788 real flops. (A real flop is the cost of computing a real sum or a real product.) These values are only approximate given that in the standard search algorithm, only the flops required to evaluate (8) were accounted for. VIII. CONCLUSIONS We have presented an efficient method to compute the Spectral and Root MUSIC estimations based on a conformal transformation. They can be calculated from the real roots of a real (2M 0 2)-degree polynomial that lie inside the [01; 1] range in Spectral MUSIC and from the complex conjugate roots of a real (2M 0 2)-degree polynomial in (Unitary) Root MUSIC. The calculation of the polynomial coefficients in both cases roughly requires 4M 2 K real flops plus 2K times the computational cost of the convolution of two M -length vectors. For (Unitary) Root MUSIC, given that the resulting polynomial is real, the computational burden of the polynomial rooting step has been reduced. REFERENCES [1] R. O. Schmidt, “Multiple emitter location and signal parameter estimation,” in Proc. Int. Conf. Acoust., Speech, Signal Process., Mar. 1986. [2] S. Lawrence Marple Jr., Digital Spectral Analysis With Applications, ser. Signal Processing. Englewood Cliffs, NJ: Prentice-Hall, 1987. [3] B. D. Rao and K. V. S. Hari, “Performance analysis of root-music,” IEEE Trans. Acoust., Speech, Signal Process., vol. 37, no. 12, pp. 1939–1949, Dec. 1989. [4] M. Pesavento, A. B. Gershman, and M. Haardt, “Unitary Root-MUSIC with a real valued eigendecomposition: a theoretical and experimental performance study,” IEEE Trans. Signal Process., vol. 48, no. 5, pp. 1306–1314, May 2000. [5] M. Jansson and P. Stoica, “Forward-only and forward-backward sample covariances—A comparative study,” Signal Process., vol. 77, pp. 235–245, 1999. [6] W. H. Press et al., Numerical Recipes in C. Cambridge, U.K.: Cambridge Univ. Press, 1997.

Estimate of Aliasing Error for Non-Smooth Signals Prefiltered by Quasi-Projections Into Shift-Invariant Spaces Wen Chen, Member, IEEE, Bin Han, and Rong-Qing Jia Abstract—An ideal analog-to-digital (A/D) converter prefilters a signal by an ideal lowpass filter. Recent research on A/D conversion based on shiftinvariant spaces reveals that prefiltering signals by quasiprojections into shift-invariant spaces provides more flexible choices in designing an A/D conversion system of high accuracy. This paper focuses on the accuracy of such prefiltering, in which the aliasing error is found to behave like = ( ) with respect to the dilation of the underlying shiftinvariant space, provided that the input signal is Lipschitz- continuous. A formula to calculate the coefficient of the decay rate is also figured out in this paper. Index Terms—A/D conversion, aliasing error, lowpass filter, prefiltering, quasiprojection, sampling, shift-invariant spaces, Strang–Fix condition, Wiener amalgam spaces.

I. INTRODUCTION In digital signal processing and digital communications, an analog signal is converted to a digital signal by an A/D (analog-to-digital) converter. An analog signal f is of finite energy if kf k2 < 1, where kf k2 is the square norm of f defined by kf k2 = ( jf (t)j2 dt)1=2 . 2 We also denote by L ( ) the signal space of finite energy, that is, L2 ( ) = ff : kf k2 < 1g. f is said to be bandlimited if f^(!) = 0 whenever j! j >  for some  > 0, where f^ is the Fourier transform of f defined by f^(!) = f (t)e0i!t dt. In this case, f is called a  -band signal. An ideal A/D converter prefilters a signal of finite energy by an ideal lowpass filter (see Fig. 1). Then, the difference between the prefiltered signal and the original signal is referred to as the aliasing error. To reduce the aliasing error, one has to increase the bandwidth of the lowpass filter. For a   1, the shift-invariant space V (') generated by the generator ' 2 L2 ( ) is defined as [3], [18]

V  ( ') =

k2

ck '( 1 0k) :

k2

j ck j

2

j jglj (0)j for any other j 2 f1; 2; 1 1 1 ; ng. Then, as k ! 1, it follows that lj

g[2] (k)

(8)

j0

0; lim !1 jg (k)j = c~ 6= 0;

step. Therefore, it is possible to solve recursive formula (10), which yields

TLFeBOOK

a~ :=

m

r

h h ; r = 1; 2; 1 1 1 ; n ij

i;j =1

n

ir

jr

(17) (18)

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1935

and diagf1 1 1g denotes a diagonal matrix with the diagonal elements built from its arguments. ~ is not positive semi-definite Here, we note that the diagonal matrix 3 ^ 3 but the diagonal matrix defined by

3^ := diag fj 1 a~1 j; j 2 a~2 j; 1 1 1 ; j n a~n jg

(19)

is positive semi-definite. It is clear from the definitions (17) and (19) ~ := 3^ I_ , where the sign that there exists a sign matrix I_ such that 3 matrix I_ is defined as a diagonal matrix whose diagonal elements are either +1 or 01. ^ is full rank, then by putting 3 in (12) to 3^ , the soRemark 1: If 3 T y ^ ^ ^HT lution (H 3H ) H 3g l can be obtained from (12). However, H 3 T ^ ~ 3 H R H cannot be calculated from (15), that is, 6= . Here, we show the following theorem. ^ and 3~ are of full Theorem 2: If H is of full column rank and both 3 rank, then

(H 3^ H T )yH 3^ g l = (H 3~ H T )yH 3~ g l

(20)

~ := 3^ I_ . where 3 Proof: Let the left-hand and the right-hand sides of (20) be de^ l and w~ l , respectively. Then, we will show that w~ l can be noted by w ^ l = (H 3^ H T )yH 3^ g l . derived from w Since H has full column rank, using a property of the pseudo-inverse operation (see [3, p. 433]), we obtain y

^ l := (H 3^ H T ) H 3^ g l = H T y (H 3^ )yH 3^ g l w = H T y3^ 01H yH 3^ g l = H T yg l

(21)

where the fourth equality comes from the fact that H y H = I because H is of full column rank. From (21) and H y H = I , we have

H T yg l

= H T y3~ 01H yH 3~ g l = (H 3~ H T )yH 3~ g l = w~ l

(22)

~ l . The reverse, where the last equality comes from the definition of w ^ l can be derived from w~ l = (H 3~ H T )yH 3~ g l , can also be which w ^ l and w~ l are identical. shown in the same way. Therefore, both w Remark 2: If H is not of full column rank, Theorem 2 does not hold. Because, in such a case, H T H does not become a nonsingular ^ l and w~ l are, matrix. Moreover, it can be seen from (21) and (22) that w ^ and 3~ , that is, Theorem 2 holds for any pair respectively, irrelevant to 3 ^ l = w~ l = H T yg l , which is of full-rank diagonal matrices. In fact, w shown in (21) and (22), attains the zero minimum value of the weighted least squares function in (12) for any diagonal positive definite matrix. In general, the right-hand side of (20) is always expressed by the fourthorder cumulants or fourth- and higher order cumulants of fy (t)g. From Theorem 2 and Remark 2, it is seen that the right-hand side of (13) can be given by the right-hand side of (20) under the condition ~ (= 3^ I_ ) is full rank. This condition, however, that the diagonal matrix 3 will be satisfied by the following theorem. Theorem 3: Let H be full column rank and i (i = 1; 2; 1 1 1 ; n) be nonzero for all i. Suppose that ij = 1 for i = j and ij = 0 for i 6= j ~ in (17) becomes full rank. [see (15)]. Then, the diagonal matrix 3 ~r ’s of Proof: If ij in (15) is 1 for i = j and 0 for i 6= j , then a 3~ in (17) become a~r

=

m i=1

2 hir ;

for r = 1; 2; 1 1 1 ; n:

(23)

~ does not have full rank. Then, one of the diagonal elements Suppose 3 ~ becomes zero, that is, r a~r = 0 for some r. It implies that a~r = of 3 m m 2 2 i=1 hir = 0 because r 6= 0. If i=1 hir = 0, then hir = 0 for all i. This contradicts the assumption that H is full column rank. ~ under these conditions is of full rank. Therefore, 3 For the time being, in this correspondence, we consider (15) with ~ g l , by using (6) ij = 1 for i = j and ij = 0 for i 6= j . As for H 3 ~j in (23) and the similar way as in [1], it can be calculated with aj = a by dl := [dl1 ; dl2 ; 1 1 1 ; dlm ]T where dlj is given by dlj

(24)

= cumfzl (t); zl (t); 1 1 1 ; zl (t); yj (t)g. Then, p

(13) can be expressed as

y

~ w [1] l := R dl ;

l = 1; 2; 1 1 1 ; n:

(25)

Since the second step (7) is a normalization of g l , it is easily shown that the second step reduces to

w l[2] :=

wl[1] : l = 1; 2; 1 1 1 ; n z2

(26)

Therefore, (25) and (26) are our proposed two steps to modify w l , which becomes one cycle of iterations in the super-exponential method [1], [4]–[6], [10]. Then, since the right-hand side of (25) consists of only higher order cumulants, the modification of w l is not affected by Gaussian noise. This comes from the fact that higher order cumulants are insensitive to additive (even colored) Gaussian noise (see [8, Prop. 4, p. 2463]). This is a novel key point of our proposed super-exponential method, from which the proposed method is referred to as a robust super-exponential method (RSEM). C. Proposed RSEM For now, there are two approaches to multichannel (or MIMO) BSS, a concurrent BSS approach and a deflationary BSS approach. ~ l ’s in (4) concurrently, whereas The former is to find all the n filters w the latter finds sequentially (or iteratively with respect to source ~ l ’s one by one. It is well known that iterative signals) the filters w algorithms based on the former approach converge to a desired solution when they start in a neighborhood of the desired solution, whereas iterative algorithms based on the latter approach converge to a desired solution globally (or regardless of their initialization) [1]. The latter approach is employed in this correspondence. Let l denote the number of the sources to be extracted. At first, set ~ 1 is calculated by the two steps (25) and (26) such l = 1; then, w ~ 1 = ~1 = [0; 1 1 1 ; 0; 1(1 th element); 0; 1 1 1 ; 0]T . Next, that H T w the contribution signals i (t) = hi s (t) (i = 1; 2; 1 1 1 ; m) ~ 1T y (t). Then, are calculated by using the output signal z1 (t) = w by calculating yi (t) 0 i (t) for i = 1; 2 1 1 1 ; m, we remove the contribution signals from the outputs in order to define the outputs of a multichannel system with n 0 1 inputs and m outputs. The number of inputs becomes deflated by one. The procedures mentioned above are continued until l = n. Therefore, the proposed RSEM is summarized as shown in Table I. The procedure from Steps 5 to 7 are implemented to make it possible to obtain solutions in (4). In Step 6, the calculation of y l (t) 0 (dl =z~ )~zl (t) is equivalent to the calculations of yi (t) 0 i (t) (i = 1; 2; 1 1 1 ; n) mentioned above. (On the details of Step 6, see Section V or [5].)

TLFeBOOK

1936

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

TABLE I PROPOSED METHOD

IV. SIMULATION RESULTS To demonstrate the validity of the proposed RSEM, many computer simulations were conducted. One of the results is shown in this section. We considered a two-input and three-output system, that is, H in (1) was set to be H =

1:0

0: 6

0:7

1:0

0:2

0: 5

:

(27)

Two source signals s1 (t) and s2 (t) were sub-Gaussian and superGaussian, respectively, in which s1 (t) takes one of two values 01 and 1 with equal probability 1/2, s2 (t) takes one of three values 02, 0, and 2 with probability 1/8, 6/8, and 1/8, respectively, and they are zero-mean and unit variance. The parameter p in (5) was set to be p = 3, that is, j (j = 1; 2) in (6), were the fourth-order cumulants of the source signals. These values were set to be 1 = 02 and 2 = 1. Three independent Gaussian noises (with identical variance n2 ) were added to the three outputs yi (t)’s at various SNR levels. The SNR is, for convenience, defined as SNR := 10 log10 (s2 =n2 ), where s2 ’s are the variances of si (t)’s and are equal to 1. Initial values of w l were randomly chosen from the values between 01 and 1. As a measure of performance, we used the multichannel intersymbol interference (MISI ) defined in the logarithmic (decibel) scale by MISI = 10 log10

n l=1

2 j j2 0 jgl jmax jgl j2max n n 2 2 l=1 jglj j 0 jg j jmax + 2 jg j jmax j =1

n j =1 glj

Fig. 1. Performances for the proposed RSEM and the conventional SEM.

shows better performance, whereas the performances of the conventional SEM hardly change. This implies that the performance of the RSEM depends on the accuracy of the estimate of the higher order cumulants. We consider, however, that since, in the above three cases, the performances of the RSEM are better than the ones of the conventional SEM, the proposed RSEM is effective for solving the BSS problem. V. DISCUSSIONS

1

1

1

(28)

1

where jgl1 j2max and jg1j j2max are, respectively, defined by jgl1 j2max := 2 2 2 maxj =1;111;n jglj j and jg1j jmax := maxl=1;111;n jglj j . The value of g ~ MISI becomes 01, if the l ’s in (4) are obtained, and hence, a minus large value of MISI indicates the proximity to the desired solution. As a conventional method, the method proposed in [5] was used for comparison. Fig. 1 shows the results of performances for the proposed RSEM and the conventional SEM when the SNR levels were, respectively, taken to be 0 (n2 = 1), 2.5, 5, 10, 15, and 1 dB (n2 = 0), in which each MISI shown in Fig. 1 was the average of the performance results obtained by 50 independent Monte Carlo runs. In each Monte Carlo run, the number ~ and d of the integers k ’s in Step 4 (see Table I) was 10, in which R l were estimated by data samples in the following three cases: (Case 1) 1000 data, (Case 2) 10 000 data, and (Case 3) 100 000 data. It can be seen from Fig. 1 that as the number of data samples that are needed to estimate the cumulants increases, the proposed RSEM

In the method shown in Table I, the calculation of Step 6 is important for implementing a deflationary SEM. Let us review the calculation of  1 in (4) is obtained for l = 1. Step 6 in Table I. Suppose that g~1 = ~ Then, from (3)–(5), and (24), z~1 (t), z~ , and d1 are, respectively T ~ 1 n (t ) z~1 (t) = c1 s (t) + w p+1 z~ = c1  p p d 1 = [h1 c1  ; h2 c1  ;

(29)

1 1 1 ; hm

(30) T p c1  ] :

(31)

Therefore, from (29)–(31), y^1 (t) := (d1 =z~ )~ z1 (t) of Step 6 in Table I becomes

^ 1 (t ) = y

~ 1T n (t) h1 s (t) + 11w ~ 1T n (t) h2 s (t) + 21w

.. .

(32)

~ 1T n (t) hm s (t) + m1 w

where i1 := hi =c1 (i = 1; 2; 1 1 1 ; m). When we calculate

TLFeBOOK

~ 1 (t ) = y 1 (t ) y

0 y^1 (t)

(33)

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1937

the output y~1 (t) in (33) is the output of a system that has n 0 1 inputs and m outputs. Therefore, by the calculation of Step 6, the number of inputs becomes deflated by one. VI. CONCLUSIONS We have proposed a deflationary SEM for solving the BSS problem, ~ l ’s satisfying (4) are found one in which the solutions of the problem w by one. The proposed SEM is not sensitive to Gaussian noise, which is referred to as a robust super-exponential method (RSEM). This is a novel property of the proposed method, whereas the conventional methods do not posses it. It was shown from the simulation results that the proposed RSEM was robust to Gaussian noise and could successfully solve the BSS problem. APPENDIX DERIVATION OF (16) From the properties of the cumulant cumfyq (t); yr (t); yi (t); yj (t)g in (14) becomes

(see

[6]),

cum fyq (t); yr (t); yi (t); yj (t)g

=

l ;l ;l ;l

hql hrl hil hjl cumfsl (t); sl (t); sl (t); sl (t)g

+ cum fnq (t); nr (t); ni (t); nj (t)g

=

n

l=1

hql hrl hil hjl l

= hTq 3 h

[2] M. Kawamoto and Y. Inouye, “A deflation algorithm for the blind source-factor separation of MIMO-FIR systems driven by colored sources,” IEEE Signal Process. Lett., vol. 10, no. 11, pp. 343–346, Nov. 2003. [3] P. Lancaster and M. Tismenetsky, The Theory of Matrices, Second ed. New York: Academic, 1985. [4] M. Martone, “An adaptive algorithm for antenna array Low-Rank processing in cellular TDMA base stations,” IEEE Trans. Commun., vol. 46, no. 5, pp. 627–643, May 1998. , “Fast adaptive super-exponential multistage beamforming cel[5] lular base-station transceivers with antenna arrays,” IEEE Trans. Veh. Technol., vol. 48, no. 4, pp. 1017–1028, Jul. 1999. [6] O. Shalvi and E. Weinstein, “Super-exponential methods for blind deconvolution,” IEEE Trans. Inf. Theory, vol. 39, no. 2, pp. 504–519, Mar. 1993. [7] C. Simon, P. Loubaton, and C. Jutten, “Separation of a class of convolutive mixtures: a contrast function approach,” Signal Process., vol. 81, pp. 883–887, 2001. [8] L. Tong, Y. Inouye, and R.-w. Liu, “Waveform-preserving blind estimation of multiple independent sources,” IEEE Trans. Signal Process., vol. 41, no. 7, pp. 2461–2470, Jul. 1993. [9] J. K. Tugnait, “Identification and deconvolution of multichannel nonGaussian processes using higher order statistics and inverse filter criteria,” IEEE Trans. Signal Process., vol. 45, no. 3, pp. 658–672, Mar. 1997. [10] K. L. Yeung and S. F. Yau, “A cumulant-based super-exponential algorithm for blind deconvolution of multi-input multi-output systems,” Signal Process., vol. 67, pp. 141–162, 1998.

(34)

hr

(35)

where the second equality comes from assumptions A2) and A3) and the fact that the fourth-order cumulant of Gaussian noises ni (t)’s are equal to zero, hq := [hq1 ; hq2 ; 1 1 1 ; hqn ]T , 3 h is a diagonal matrix defined by

3 h =

0

2 hi2 hj 2

.. .

.. .

0 and h r

i;j =1

T

0 0

111

n hin hjn

Index Terms—Iterative decoding, MIMO, SOVA, space-time codes.

. From (35), we obtain

0 .. .

0

h i1 h j 1

2

0

111 111

0 0

.. .

..

.. .

i;j hi2 hj 2

0

.

111

n

i;j

I. INTRODUCTION

(36)

is a diagonal matrix defined by

i;j

Aydin Sezgin, Student Member, IEEE, and Holger Boche Abstract—We study the iterative decoding of Wrapped Space-Time Codes (WSTCs) employing per-survivor-processing with the soft-output Viterbi-algorithm (SOVA). We use a novel receiver scheme that incorporates extrinsic information delivered by the SOVA. The decision metric of the SOVA is developed, and the performance is analyzed.

.. .

.

ij cum fyq (t)yr (t)yi (t)yj (t)g = hqT 3 6h hr

where 3 6h

1

..

0

:= [hr1 ; hr2 ; 1 1 1 ; hrn ] m

111 111

0

1 h i1 h j 1

Iterative Decoding of Wrapped Space-Time Codes

:

hin hjn

~HT . It can be seen that (36) expresses the (q; r)th element of H 3 Therefore, (16) holds true. ACKNOWLEDGMENT The authors would like to thank Dr. M. Ohata, RIKEN, and M. Ito, Nagoya University for giving useful suggestions. REFERENCES [1] Y. Inouye and K. Tanebe, “Super-exponential algorithms for multichannel blind deconvolution,” IEEE Trans. Signal Process., vol. 48, no. 3, pp. 881–888, Mar. 2000.

In recent years, the goal of providing high-speed wireless data services has generated a great amount of interest among the research community. Recent information-theoretic results have demonstrated that the capacity of the system in the presence of Rayleigh fading improves significantly with the use of multiple transmit and receive antennas [1], [2]. Diagonal Bell Labs Layered Space-Time (DBLAST), which is an architecture that theoretically achieves a capacity for such multiple-input Manuscript received January 22, 2004; revised June 8, 2004. This work was supported in part by the German ministry of education and research (BMBF) under Grant 01BU150. Part of this work was presented at the IEEE International Symposium on Personal, Indoor, and Mobile Radio Communications, Beijing, China, September 7–10, 2003. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vikram Krishnamurthy. A. Sezgin is with the Department of Broadband Mobile Communication Networks, Fraunhofer Institute for Telecommunications, HHI, D-10587 Berlin, Germany (e-mail: [email protected]). H. Boche is with the Department of Mobile Communication Networks, Technical University of Berlin, D-10587 Berlin, Germany (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845493

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1937

the output y~1 (t) in (33) is the output of a system that has n 0 1 inputs and m outputs. Therefore, by the calculation of Step 6, the number of inputs becomes deflated by one. VI. CONCLUSIONS We have proposed a deflationary SEM for solving the BSS problem, ~ l ’s satisfying (4) are found one in which the solutions of the problem w by one. The proposed SEM is not sensitive to Gaussian noise, which is referred to as a robust super-exponential method (RSEM). This is a novel property of the proposed method, whereas the conventional methods do not posses it. It was shown from the simulation results that the proposed RSEM was robust to Gaussian noise and could successfully solve the BSS problem. APPENDIX DERIVATION OF (16) From the properties of the cumulant cumfyq (t); yr (t); yi (t); yj (t)g in (14) becomes

(see

[6]),

cum fyq (t); yr (t); yi (t); yj (t)g

=

l ;l ;l ;l

hql hrl hil hjl cumfsl (t); sl (t); sl (t); sl (t)g

+ cum fnq (t); nr (t); ni (t); nj (t)g

=

n

l=1

hql hrl hil hjl l

= hTq 3 h

[2] M. Kawamoto and Y. Inouye, “A deflation algorithm for the blind source-factor separation of MIMO-FIR systems driven by colored sources,” IEEE Signal Process. Lett., vol. 10, no. 11, pp. 343–346, Nov. 2003. [3] P. Lancaster and M. Tismenetsky, The Theory of Matrices, Second ed. New York: Academic, 1985. [4] M. Martone, “An adaptive algorithm for antenna array Low-Rank processing in cellular TDMA base stations,” IEEE Trans. Commun., vol. 46, no. 5, pp. 627–643, May 1998. , “Fast adaptive super-exponential multistage beamforming cel[5] lular base-station transceivers with antenna arrays,” IEEE Trans. Veh. Technol., vol. 48, no. 4, pp. 1017–1028, Jul. 1999. [6] O. Shalvi and E. Weinstein, “Super-exponential methods for blind deconvolution,” IEEE Trans. Inf. Theory, vol. 39, no. 2, pp. 504–519, Mar. 1993. [7] C. Simon, P. Loubaton, and C. Jutten, “Separation of a class of convolutive mixtures: a contrast function approach,” Signal Process., vol. 81, pp. 883–887, 2001. [8] L. Tong, Y. Inouye, and R.-w. Liu, “Waveform-preserving blind estimation of multiple independent sources,” IEEE Trans. Signal Process., vol. 41, no. 7, pp. 2461–2470, Jul. 1993. [9] J. K. Tugnait, “Identification and deconvolution of multichannel nonGaussian processes using higher order statistics and inverse filter criteria,” IEEE Trans. Signal Process., vol. 45, no. 3, pp. 658–672, Mar. 1997. [10] K. L. Yeung and S. F. Yau, “A cumulant-based super-exponential algorithm for blind deconvolution of multi-input multi-output systems,” Signal Process., vol. 67, pp. 141–162, 1998.

(34)

hr

(35)

where the second equality comes from assumptions A2) and A3) and the fact that the fourth-order cumulant of Gaussian noises ni (t)’s are equal to zero, hq := [hq1 ; hq2 ; 1 1 1 ; hqn ]T , 3 h is a diagonal matrix defined by

3 h =

0

2 hi2 hj 2

.. .

.. .

0 and h r

i;j =1

T

0 0

111

n hin hjn

Index Terms—Iterative decoding, MIMO, SOVA, space-time codes.

. From (35), we obtain

0 .. .

0

h i1 h j 1

2

0

111 111

0 0

.. .

..

.. .

i;j hi2 hj 2

0

.

111

n

i;j

I. INTRODUCTION

(36)

is a diagonal matrix defined by

i;j

Aydin Sezgin, Student Member, IEEE, and Holger Boche Abstract—We study the iterative decoding of Wrapped Space-Time Codes (WSTCs) employing per-survivor-processing with the soft-output Viterbi-algorithm (SOVA). We use a novel receiver scheme that incorporates extrinsic information delivered by the SOVA. The decision metric of the SOVA is developed, and the performance is analyzed.

.. .

.

ij cum fyq (t)yr (t)yi (t)yj (t)g = hqT 3 6h hr

where 3 6h

1

..

0

:= [hr1 ; hr2 ; 1 1 1 ; hrn ] m

111 111

0

1 h i1 h j 1

Iterative Decoding of Wrapped Space-Time Codes

:

hin hjn

~HT . It can be seen that (36) expresses the (q; r)th element of H 3 Therefore, (16) holds true. ACKNOWLEDGMENT The authors would like to thank Dr. M. Ohata, RIKEN, and M. Ito, Nagoya University for giving useful suggestions. REFERENCES [1] Y. Inouye and K. Tanebe, “Super-exponential algorithms for multichannel blind deconvolution,” IEEE Trans. Signal Process., vol. 48, no. 3, pp. 881–888, Mar. 2000.

In recent years, the goal of providing high-speed wireless data services has generated a great amount of interest among the research community. Recent information-theoretic results have demonstrated that the capacity of the system in the presence of Rayleigh fading improves significantly with the use of multiple transmit and receive antennas [1], [2]. Diagonal Bell Labs Layered Space-Time (DBLAST), which is an architecture that theoretically achieves a capacity for such multiple-input Manuscript received January 22, 2004; revised June 8, 2004. This work was supported in part by the German ministry of education and research (BMBF) under Grant 01BU150. Part of this work was presented at the IEEE International Symposium on Personal, Indoor, and Mobile Radio Communications, Beijing, China, September 7–10, 2003. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vikram Krishnamurthy. A. Sezgin is with the Department of Broadband Mobile Communication Networks, Fraunhofer Institute for Telecommunications, HHI, D-10587 Berlin, Germany (e-mail: [email protected]). H. Boche is with the Department of Mobile Communication Networks, Technical University of Berlin, D-10587 Berlin, Germany (e-mail: [email protected]). Digital Object Identifier 10.1109/TSP.2005.845493

1053-587X/$20.00 © 2005 IEEE

TLFeBOOK

1938

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

multiple-output (MIMO) channels, has been proposed by Foschini in [3]. However, the high complexity of the algorithm implementation is its substantial drawback. Vertical BLAST (VBLAST), which is a simplified and suboptimal version of the BLAST architecture using ordered successive nulling and interference cancellation at the receiver, is capable of achieving high capacity at low complexity. A drawback of VBLAST is the effect of error propagation caused by incorrect estimations of transmitted signals. To avoid this drawback, many iterative schemes with high complexity have been proposed in the literature, e.g., [4], [5]. Another approach to reduce the effect of error propagation and to improve the performance significantly was proposed by Caire et al. in [6]. They proposed a low-complexity space-time scheme called Wrapped Space-Time Coding (WSTC) for Rayleigh fading channels to achieve high spectral efficiencies [6]. In this scheme, only a single encoder is used. The coded data is diagonally interleaved and transmitted over the nT transmit antennas. At the receiver, the nulling and cancellation steps are integrated into a Viterbi algorithm employing per-survivor processing [7]. In this work, we apply the coding and decoding technique from [6] as inner coding and decoding components, respectively. However, instead of the receiver used in [6], which provides only hard decisions over the information bits, we employ a SOVA providing soft decisions to the outer decoder. Since we need the data estimations at the inner decoder for the interference cancellation with a minimum amount of delay, the application of optimum soft-input soft-output (SISO) maximum a posteriori (MAP) algorithms [8] is not feasible. Further on, we apply an outer code at the transmitter and couple this with an iterative decoding process in order to improve the performance of the architecture and in order to achieve the capacity promised by the information-theoretic results. The performance of our scheme is evaluated by simulations and compared to the scheme proposed in [6]. The rest of this paper is organized as follows. In Section II, we introduce the system model and establish notation. The novel receiver scheme is described in Section III. Section IV gives simulation results, followed by some concluding remarks in Section V. II. SYSTEM MODEL We consider a wireless multiple-input-multiple-output (MIMO) system with nT transmit and nR receive antennas, as depicted in Fig. 1. The transmitter consists of two systematic, recursive convolutional codes (CCs), which are denoted as FEC1 and FEC2 in Fig. 1, concatenated in serial via a pseudorandom interleaver. Codes of such a structure are known as serially concatenated convolutional codes (SCCCs). This interleaver is used in order to uncorrelate the log-likelihoods of adjacent bits and distribute the error events due to a deeply faded block during a transmission. We can obtain different spectral efficiencies by puncturing the parity bits of the component encoders. After encoding the whole information bit sequence with FEC1 and interleaving, the coded bit sequence is divided into  blocks. Each block is encoded separately. Let a block of coded bits be fc1 ; c2 ; . . . ; cL g, where LB is the block length. This coded bits are then mapped onto symbols from a given constellation, e.g., binary phase shift keying (BPSK) and interleaved via a diagonal (channel) interleaver, which is different from the interleaver used for concatenation of the SCCC code component encoders. The channel is constant during the transmission of one block and changes independently from block to block. Our system model is defined by

Y= X H h h

 HX + N nT

Y

(1)

where is the (nT 2 T ) transmit matrix, is the (nR 2 T ) receive = [ 1 ; 2 ; . . . ; n ] is the (nR 2 nT ) flat fading channel matrix,

h

Fig. 1. System model with binary source, SCCC encoder, modulation, diagonal interleaving (cf. Fig. 2), Rayleigh MIMO-channel, and receiver (cf. Section III, Fig. 3).

Fig. 2. Diagonal interleaver for a system with nT = 4 transmit antennas. The entries in the cells indicate the index k of the symbols in the current codeword. A cross in a cell means that at this time the given antenna is not active.

N

matrix, and is the (nR 2 T ) additive white Gaussian noise (AWGN) matrix, where an entry fni g of (1  i  nR ) denotes the complex noise at the ith receiver for a given time t(1  t  T ) (for clarity, we dropped the time index). The real and imaginary parts of ni are independent and normal N (0; 1=2) distributed. An entry of the channel matrix is denoted by fhij g. This represents the complex gain of the channel between the j th transmitter (1  j  nT ) and the ith receiver (1  i  nR ), where the real and imaginary parts of the channel gains are independent and normal distributed random variables with N (0; 1=2) per dimension. Throughout the paper, it is assumed that the channel matrix is independent and identically distributed (i.i.d.) and that every channel realization of the channel is independent, i.e., the channel is memoryless. The expected signal-to-noise ratio (SNR) at each receive antenna is independent of nT and is denoted as  in (1). It is further assumed that the transmitter has no channel state information (CSI) and that the receiver has perfect CSI. To obtain the transmit matrix , we use a special interleaver, as illustrated in Fig. 2. With the index k of the symbols in the current codeword, we get the right cell position in as follows:

N

X

X

r =k 0 c =k 0

X

k01 nT nT k01 ( nT nT

where [ ]r;c is the current cell of column and r to the row of .

X

(2)

0 1)

(3)

X. Herein, c corresponds to the

III. RECEIVER WITH ITERATIVE DECODING A. Receiver Structure Fig. 3 shows the structure of the receiver, which consists of two stages: the space-time SOVA (STS) decoder described in Section III-B, and a MAP SISO channel decoder. The two stages are separated by deinterleavers and interleavers. The receiver works as follows: Assuming equally likely bits, the resulting a priori information A;STS is zero in the first iteration. Therefore, the switch in Fig. 3 is in

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1939

with QH Q = I. The superscript (1)H denotes matrix conjugate transpose. For the feedback (interference cancellation) filter B = [bT1 ; bT2 ; . . . ; bTn ]T , bTr = [012r ; b~ Tr ], we have

b~ Tr = frH H Fig. 3. Model of the proposed SISO receiver with Space-Time SOVA decoder, channel decoder, interleaver, and deinterleaver.

0r2(n 0r) : In 0r

(7)

Note that we have to compute the filter coefficients only once during a channel realization. At each trellis step, the module called “per-survivor processing (PSP) survivor storage” gets, for each trellis state, the hard decisions of the data sequence corresponding to the survivor at that state from the SOVA. With this input, the estimated transmit matrix X is constructed for the trellis update leaving each state. This update is done by the cancellation module, which takes X and computes the interference for the current decoding step in the trellis diagram for each trellis state. Note that we need soft-decisions for the iterative decoding, but only hard-decisions of the data sequence for the cancellation, since the PSP estimates are based on (hard) hypothesized paths, leading to the given trellis state. The nulling module takes the matrix FH , which is obtained by the filter coefficients module to null out the interference from not-yet-decoded codeword symbols. B. ST SOVA Decoder

Fig. 4. Model of the proposed receiver with PSP, joint decoding, and channel estimating.

position 1. Now, the ST SOVA decoder has to compute the extrinsic information E;STS only from the observations of the channel output. The extrinsic information E;STS is now deinterleaved and fed into the MAP decoder as a priori information A;MAP . Based on this a priori information and the trellis structure of the channel code, the MAP-Decoder computes the extrinsic information E;MAP . After interleaving, this extrinsic information E;MAP is fed back to the ST SOVA decoder as a priori information A;STS for the following iterations. After the first iteration, the switch in Fig. 3 is the on position 2. At the receiver, decoding is done by processing the receive matrix according to the diagonal interleaver structure. In Fig. 4, the variable X represents the estimated transmit matrix with entries up to the current decoding step k in the trellis diagram of the SOVA, which is obtained from the survivor terminating in the code trellis state  at decoding step k . According to the Zero-Forcing (ZF) or the Minimum Mean-Square Error (MMSE) criterion for the filter design [6], [9], we have for the feedforward (or interference nulling) filter F = [f1 ; f2 ; . . . ; fn ]

fr =

qr

1

h S h

1 S0 r hr

(4)

where

Sr = (H0)(H0)H +

nT 

In =

r01 i=1

hi hH i +

nT 

In

(5)

and

0=

Ir01 : 0(n 0r+1)2(r01)

(6)

The vector qr is obtained from the QR-factorization of the channel matrix H, where the the matrix R is an upper triangular nT 2 nT matrix, and Q = [q1 ; q2 ; . . . ; qn ] is an nR 2 nT unitary matrix

In order to obtain the path metric for the ST SOVA decoder in the presence of interference from the other layers, we use the matrices F and B for nulling out the impact of the upper (not-yet-detected) layers, and combine this with PSP for cancelling the interference of the lower (already-detected) layers. To improve the decoding process, we also need a priori information about the transmitted signals in the modified path metric. Let yk be the receive vector corresponding to the symbols in Y , and let xk be the interference vector corresponding to the symbols in X at decoding time k , respectively. Furthermore, let the signal-to-interference plus noise ratio at the output of the cancellation module given by

r =

[R]r;r

01 hH r S r hr

(8)

where Sr is given in (5), and [R]r;r is the rth row and column entry of the upper triangular matrix R. Then, the modified path metric Mk ( ) of the path terminating in a state  in the code trellis at the decoding time k is given by

Mk ( ) = min fMk01 ( ) + log pk (;  )  2P ( ) + frH yk 0 bTr xk 0 r z

(9)

where P ( ) denotes the set of parent states of  , z denotes the modulated symbol on the trellis transition  !  , Mk01 ( ) is the smallest metric of the path connected to the trellis state  , and log pk ( ) is the logarithm of the a priori probability of the bit ck corresponding to the trellis transition  !  . The a priori probability is obtained from the SISO channel decoder. The ST SOVA decoder stepwise decodes the symbols at each stage of the code trellis diagram, storing the survivor terminating in each state of the trellis and, use the survivors to cancel their impact as interference on the following decoding steps. The soft output of the ST SOVA is an approximate log-likelihood ratio of the a posteriori probabilities of the information bits. The soft output can be approximately expressed as the metric difference between the maximum-likelihood path and its strongest competitor at each decoding step. The strongest competitor of the maximum-likelihood path is the path that has the minimum path metric among a given set of paths. This set is obtained by taking all paths, which have, at the current decoding

TLFeBOOK

1940

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

step, the symbol on their trellis transition complementary to the one on the maximum likelihood path. The ST SOVA decoder provides soft information, which can be expressed as

= 0jY) 3(ck ) = log PP ((cck = 1 jY ) k P (z = +1jY) = log P (z = 01jY)  = Mk01 0 Mk+1

(10)

where Mk01 is the minimum path metric corresponding to z = 01, and Mk+1 is the minimum path metric corresponding to z = +1. We can split the soft output of the SOVA into two parts: the extrinsic information E;STS (ck ) and the intrinsic or a priori information A;STS ,

3(ck ) = A;STS (ck ) + E;STS (ck )

(11)

=

where the a priori information is given as

: 3A;STS (ck ) = log ppk (0) k (1)

(12)

=4

Fig. 5. Bit error rates, ZF receiver, nT nR antennas, WSTC-ID with coded QPSK modulation with inner and outer code CC ; 8 , RWSTC0ID = , and WSTC with BPSK and RWSTC = .

=1 4

=1 2

(5 7)

Therefore, the extrinsic information, which is fed into the MAP decoder after deinterleaving is obtained from (11) and (12) as

E;STS (ck ) = 3(ck ) 0 A;STS (ck ):

(13)

Some simulation results of the proposed scheme and their interpretation are presented in the following section.

IV. NUMERICAL SIMULATION In this section, we illustrate the bit error performance of our proposed scheme, which we call WSTC with iterative decoding (WSTC-ID) in the remainder of the paper, and compare it with the performance of the WSTC in [6] and [10]. In Fig. 5, we present the bit error rate (BER) of the WSTC-ID scheme for a system with nT = nR = 4 transmit-andreceive antennas and quadrature phase shift keying (QPSK) modulation. The outer coding is performed over multiple block fading channels. After encoding with the outer code and interleaving, we divide the whole sequence into  = 8 blocks. Each block has a length of LB = 128 bits after encoding with the inner code. We assume that the channel is constant for the transmission for each block and change independently from block to block. As component codes, we use the binary linear feedback systematic convolutional code CC(7; 5)8 , where the generator polynomials are given in octal numbers with feedback polynomial Gr = 7 and feedforward polynomial Gf = 5. The overall code rate of our scheme is RWSTC0ID = 1=4. For reference, the simulated BER performance of the WSTC for BPSK modulation are also shown. As channel code for WSTC, we used the convolutional code CC(7; 5)8 with code rate RWSTC = 1=2. By comparing the curves of our proposed scheme with the one of WSTC, we observe that although the performance at the first iteration is relatively worse, there is a significant improvement with further iterations, especially for higher SNR values. Furthermore, simulation results show that there occurs a saturation of the improvement beyond iteration 3. This behavior can be explained by the fact that after each iteration, the extrinsic output of the receiver components tends to a Gaussian distribution according to the central limit theorem, but the correlation between the extrinsic information and the channel output increases after each iteration [11].

=

=4

Fig. 6. Bit error rates, MMSE receiver, nT nR antennas, WSTC-ID with coded QPSK modulation with inner and outer code CC ; 8 , code rate RWSTC0ID = , and WSTC with BPSK and RWSTC = .

=1 4

(5 7) =1 2

Hence, the improvement of the BER performance diminishes after each iteration. Even at the first iteration, we observe that the WSTC have a diversity loss in comparison to the the new WSTC-ID scheme. The loss is due to the inability of the ZF-or MMSE Decision Feedback Equalization (DFE) within the WSTC to achieve full diversity with given coded modulation over finite alphabets. The diversity gain (or change of slope) of the new WSTC-ID scheme is mainly achieved due to soft decisions used for the iterative decoding and the distribution of error events (with interleaving) due to a deeply faded block during a transmission. Note that the improvement to WSTC can further enhanced with a bigger  and through the enlargement of the whole information bit length. In addition to the BER for the ZF solution, we present the BER for the MMSE case in Fig. 6. From the figure, we observe that the new scheme outperforms the WSTC scheme. We also observe that the improvement in comparison to WSTC is higher, as in the ZF case.

TLFeBOOK

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 53, NO. 5, MAY 2005

1941

V. CONCLUSION In conclusion, we proposed a novel iterative receiver scheme for a low-complexity space-time architecture called WSTC. The receiver consists of the serial concatenation of two stages: A Space-Time (ST) SOVA decoder and a MAP decoder. We developed the decision metric for the ST SOVA decoder employing per-survivor processing. Furthermore, we analyzed the performance of our proposed scheme in terms of numerical simulations and compared it with the noniterative WSTC scheme in [10]. It is shown that the proposed scheme has significantly better performance in terms of bit error rates. REFERENCES [1] E. Telatar, “Capacity of multi-antenna Gaussian channels,” European Trans. Telecomm., vol. 10, no. 6, pp. 585–596, Nov. 1999. [2] G. J. Foschini and M. J. Gans, “On limits of wireless communications in a fading environment when using multiple antennas,” Wireless Pers. Commun., vol. 6, no. 3, pp. 311–335, Mar. 1998. [3] G. J. Foschini, “Layered space-time architecture for wireless communication in a fading environment when using multi-element antennas,” Bell Labs Tech. J., vol. 1, no. 2, pp. 41–59, Autumn 1996.

[4] M. Sellathurai and S. Haykin, “Turbo-BLAST for high-speed wireless communications,” in Proc. Wireless Commun. Netw. Conf., vol. 1, 2000, pp. 1962–1973. [5] H. E. Gamal and A. R. Hammons Jr., “A new approach to layered spacetime coding and signal processing,” IEEE Trans. Inf. Theory, vol. 47, no. 6, pp. 2321–2334, Sep. 2001. [6] G. Caire and G. Colavolpe, “On low-complexity space-time coding for quasistatic channels,” IEEE Trans. Inf. Theory, vol. 49, no. 6, pp. 1400–1416, Jun. 2003. [7] R. Raheli, A. Polydoros, and C.-K. Tzou, “Per-survivor processing: a general approach to mlse in uncertain environments,” IEEE Trans. Commun., vol. 43, no. 2, pp. 354–364, Feb. 1995. [8] L. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” IEEE Trans. Inf. Theory, vol. IT-20, pp. 284–287, Mar. 1974. [9] E. Biglieri, G. Taricco, and A. Tulino, “Decoding space-time codes with BLAST architectures,” IEEE Trans. Signal Process., vol. 50, no. 10, pp. 2547–2552, Oct. 2002. [10] G. Caire and G. Colavolpe, “Wrapped space-time codes for quasistatic multiple-antenna channels,” in Proc. Wireless Commun. Netw. Conf., Sep. 2001. [11] J. Hagenauer, E. Offer, and L. Papke, “Iterative decoding of binary block and convolutional codes,” IEEE Trans. Inf. Theory, vol. 42, no. 2, pp. 429–445, Mar. 1996.

TLFeBOOK