Image sensors and signal processing for digital still cameras [1 ed.] 0849335450, 9780849335457

Citation preview

IMAGE SENSORS and SIGNAL PROCESSING for DIGITAL STILL CAMERAS Edited by

Junichi Nakamura

Boca Raton London New York Singapore

A CRC title, part of the Taylor & Francis imprint, a member of the Taylor & Francis Group, the academic division of T&F Informa plc.

Copyright © 2006 Taylor & Francis Group, LLC

Published in 2006 by CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2006 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group No claim to original U.S. Government works Printed in the United States of America on acid-free paper 10 9 8 7 6 5 4 3 2 1 International Standard Book Number-10: 0-8493-3545-0 (Hardcover) International Standard Book Number-13: 978-0-8493-3545-7 (Hardcover) Library of Congress Card Number 2005041776 This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC) 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Library of Congress Cataloging-in-Publication Data Image sensors and signal processing for digital still cameras / edited by Junichi Nakamura. p. cm. Includes bibliographical references and index. ISBN 0-8493-3545-0 (alk. paper) 1. Image processing—Digital techniques. 2. Signal processing—Digital techniques. 3. Digital cameras. I. Nakamura, Junichi. TA1637.1448 2005 681'.418—dc22

2005041776

Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com Taylor & Francis Group is the Academic Division of T&F Informa plc.

Copyright © 2006 Taylor & Francis Group, LLC

and the CRC Press Web site at http://www.crcpress.com

Preface Since the introduction of the first consumer digital camera in 1995, the digital still camera (DSC) market has grown rapidly. The first DSC used a charge-coupled device (CCD) image sensor that had only 250,000 pixels. Ten years later, several models of consumer point-and-shoot DSCs with 8 million pixels are available, and professional digital single-lens reflex (DSLR) cameras are available with 17 million pixels. Unlike video camera applications in which the output is intended for a TV monitor and, thus, the vertical resolution of the image sensors is standardized, there is no standard output or resolution for DSCs, so the sensor pixel count continues to grow. Continuing improvements in sensor technology have allowed this ever increasing number of pixels to be fit onto smaller and smaller sensors until pixels measuring just 2.3 mm ¥ 2.3 mm are now available in consumer cameras. Even with this dramatic shrinking of pixels, the sensitivity of sensors has improved to the point that shirtpocket-sized consumer DSCs take good quality pictures with the exposure values equivalent to 400 ISO film. Improvements in optics and electronics technologies have also been remarkable. As a result, the image quality of pictures taken by a DSC has become comparable to that normally expected of silver-halide film cameras. Consistent with these improvements in performance, the market for DSCs has grown to the point that shipments of DSCs in 2003 surpassed those of film cameras. Image Sensors and Signal Processing for Digital Still Cameras focuses on image acquisition and signal-processing technologies in DSCs. From the perspective of the flow of the image information, a DSC consists of imaging optics, an image sensor, and a signal-processing block that receives a signal from the image sensor and generates digital data that are eventually compressed and stored on a memory device in the DSC. The image acquisition part of that flow includes the optics, the sensor, and front-end section of the signal-processing block that transforms photons into digital bits. The remainder of the signal-processing block is responsible for generating the image data stored on the memory device. Other technologies used in a DSC, such as mechanics, data compression, user interface, and the output-processing block that provides the signals to output devices, such as an LCD, a TV monitor, a printer, etc., are beyond the scope of this book. Graduate students in electronics engineering and engineers working on DSCs should find this book especially valuable. However, I believe it offers interesting reading for technical professionals in the image sensor and signal-processing fields as well. The book consists of 11 chapters: •

Chapter 1, “Digital Still Cameras at a Glance,” introduces the historical background and current status of DSCs. Readers will be able to understand

Copyright © 2006 Taylor & Francis Group, LLC

•

•

•

•

•

•

what DSCs are, how they have evolved, types of modern DSCs, their basic structure, and their applications. In Chapter 2, “Optics in Digital Still Cameras,” a wide range of topics explains the imaging optics used in DSCs. It is obvious that high image quality cannot be obtained without a high-performance imaging optical system. It is also true that requirements for imaging optics have become higher as pixel counts have increased and pixel sizes have decreased. Reviews of image sensor technologies for DSC applications can be found in Chapter 3 through Chapter 6. • First, in Chapter 3, “Basics of Image Sensors,” the functions and performance parameters common to CCD and complementary metaloxide semiconductor (CMOS) image sensors are explained. • Chapter 4, “CCD Image Sensors,” describes in detail the CCD image sensors widely used in imaging applications. The chapter ranges from a discussion of basic CCD operating principles to descriptions of CCD image sensors specifically designed for DSC applications. • Chapter 5, “CMOS Image Sensors,” discusses the relatively new CMOS image sensor technology, whose predecessor, the MOS type of image sensor, was released into the market almost 40 years ago, even before the CCD image sensor. • Following these descriptions of image sensors, methods for evaluating image sensor performances relative to DSC requirements are presented in Chapter 6, “Evaluation of Image Sensors.” Chapter 7 and Chapter 8 provide the basic knowledge needed to implement image processing algorithms. • The topics discussed in Chapter 7, “Color Theory and Its Application to Digital Still Cameras,” could easily fill an entire book; however, the emphasis here is on how color theory affects the practical uses of DSC applications. • Chapter 8, “Image-Processing Algorithms,” presents the algorithms utilized by the software or hardware in a DSC. Basic image-processing and camera control algorithms are provided along with advanced image-processing examples. This provides the framework for the description of the image-processing hardware engine in Chapter 9, “Image-Processing Engines.” The required performance parameters for DSCs and digital video cameras are reviewed, followed by descriptions of the architectures of signal-processing engines. Examples of analog front-end and digital back-end designs are introduced. In Chapter 10, “Evaluation of Image Quality,” readers learn how each component described in the previous chapters affects image quality. Image quality-related standards are also given. In Chapter 11, “Some Thoughts on Future Digital Still Cameras,” Eric Fossum, the pioneer of CMOS image sensor technology, discusses future DSC image sensors with a linear extrapolation of current technology and then explores a new paradigm for image sensors. Future digital camera concepts are also addressed.

Copyright © 2006 Taylor & Francis Group, LLC

I would like to extend my gratitude to the contributors to this book, many of whom work actively in the industry, for their efforts and time given to making this book possible. Also, I am grateful to all of my coworkers who reviewed draft copies of the manuscripts: Dan Morrow, Scott Smith, Roger Panicacci, Marty Agan, Gennnady Agranov, John Sasinowski, Graham Kirsch, Haruhisa Ando, Toshinori Otaka, Toshiki Suzuki, Shinichiro Matsuo, and Hidetoshi Fukuda. Sincere thanks also go to Jim Lane, Deena Orton, Erin Willis, Cheryl Holman, Nicole Fredrichs, Nancy Fowler, Valerie Robertson, and John Waddell of the MarCom group at Micron Technology, Inc. for their efforts in reviewing manuscripts, appendices, and the table of contents; creating drawings for Chapter 3 and Chapter 5; and preparing the tables and figures for publication. Junichi Nakamura, Ph.D. Japan Imaging Design Center Micron Japan, LTD

Copyright © 2006 Taylor & Francis Group, LLC

Editor Junichi Nakamura received his B.S. and M.S. in electronics engineering from Tokyo Institute of Technology, Tokyo, Japan, in 1979 and 1981, respectively, and his Ph.D. in electronics engineering from the University of Tokyo, Tokyo, Japan, in 2000. He joined Olympus Optical Co., Tokyo, in 1981. After working on optical image processing, he was involved in developments of active pixel sensors. From September 1993 to October 1996, he was resident at the NASA Jet Propulsion Laboratory, California Institute of Technology, as a distinguished visiting scientist. In 2000, he joined Photobit Corporation, Pasadena, CA, where he led several custom sensor developments. He has been with Japan Imaging Design Center, Micron Japan, Ltd. since November 2001 and is a Micron Fellow. Dr. Nakamura served as technical program chairman for the 1995, 1999, and 2005 IEEE Workshop on Charge-Coupled Devices and Advanced Image Sensors and as a member of the Subcommittee on Detectors, Sensors and Displays for IEDM 2002 and 2003. He is a senior member of IEEE and a member of the Institute of Image Information and Television Engineers of Japan.

Copyright © 2006 Taylor & Francis Group, LLC

Contributors Eric R. Fossum Department of Electrical Engineering and Electrophysics University of Southern California Los Angeles, CA, USA Po-Chieh Hung Imaging System R&D Division System Solution Technology R&D Laboratories Konica Minolta Technology Center, Inc. Tokyo, Japan Takeshi Koyama Lens Products Development Center, Canon, Inc. Tochigi, Japan Toyokazu Mizoguchi Imager & Analog LSI Technology Department Digital Platform Technology Division Olympus Corporation Tokyo, Japan Junichi Nakamura Japan Imaging Design Center, Micron Japan, Ltd. Tokyo, Japan Kazuhiro Sato Image Processing System Group NuCORE Technology Co., Ltd. Ibaraki, Japan Isao Takayanagi Japan Imaging Design Center, Micron Japan, Ltd. Tokyo, Japan Kenji Toyoda Department of Imaging Arts and Sciences College of Art and Design Musashino Art University Tokyo, Japan

Copyright © 2006 Taylor & Francis Group, LLC

Seiichiro Watanabe NuCORE Technology Inc. Sunnyvale, CA, USA Tetsuo Yamada VLSI Design Department Fujifilm Microdevices Co., Ltd. Miyagi, Japan Hideaki Yoshida Standardization Strategy Section Olympus Imaging Corp. Tokyo, Japan

Copyright © 2006 Taylor & Francis Group, LLC

Table of Contents Chapter 1 Digital Still Cameras at a Glance ............................................................................ 1 Kenji Toyoda Chapter 2 Optics in Digital Still Cameras .............................................................................. 21 Takeshi Koyama Chapter 3 Basics of Image Sensors......................................................................................... 53 Junichi Nakamura Chapter 4 CCD Image Sensors ...............................................................................................95 Tetsuo Yamada Chapter 5 CMOS Image Sensors ...........................................................................................143 Isao Takayanagi Chapter 6 Evaluation of Image Sensors................................................................................ 179 Toyokazu Mizoguchi Chapter 7 Color Theory and Its Application to Digital Still Cameras................................. 205 Po-Chieh Hung Chapter 8 Image-Processing Algorithms............................................................................... 223 Kazuhiro Sato Chapter 9 Image-Processing Engines.....................................................................................255 Seiichiro Watanabe

Copyright © 2006 Taylor & Francis Group, LLC

Chapter 10 Evaluation of Image Quality ................................................................................ 277 Hideaki Yoshida Chapter 11 Some Thoughts on Future Digital Still Cameras................................................. 305 Eric R. Fossum Appendix A Number of Incident Photons per Lux with a Standard Light Source ..................315 Junichi Nakamura Appendix B Sensitivity and ISO Indication of an Imaging System .........................................319 Hideaki Yoshida

Copyright © 2006 Taylor & Francis Group, LLC

Appendix A Number of Incident Photons per Lux with a Standard Light Source Junichi Nakamura Radiometry describes the energy or power transfer from a light source to a detector and relates purely to the physical properties of photon energy. When it comes to visible imaging in which the response of the human eye is involved, photometric units are commonly used. The relationship between the radiometric quantity Xe,λ and the photometric quantity Xv is given by λ2

Xv = K m ⋅ ∫ Xe,λ ( λ ) ⋅ V ( λ ) ⋅ d λ λ1

(A.1)

where V(λ) is the photopic eye response, Km is the luminous efficacy for photopic vision and equals 683 lumens/watt; λ1 = 0.38µm; and λ2 = 0.78µm. Table A.1 shows the photopic eye response and it corresponds to the response of y (λ) shown in Figure 7.2 in Chapter 7. A standard light source with a color temperature T can be modeled using Planck’s blackbody radiation law (see Section 7.2.5. in Chapter 7). The spectral radiant exitance of an ideal blackbody source whose temperature is T (in K) can be described as M e ( λ, T ) =

c1 ⋅ λ5

1 W ⎡ ⎤ 2 ⎢ c ⎞ ⎛ cm − µm ⎥⎦ exp ⎜ 2 ⎟ − 1 ⎣ ⎝ λT ⎠

(A.2)

c1 = 3.7418 × 104 watt-µm4/cm2 c2 = 1.4388 × 104 µm-K

315

Copyright © 2006 Taylor & Francis Group, LLC

316

Image Sensors and Signal Processing for Digital Still Cameras

TABLE A.1 Photopic Eye Response V(λ) Wavelengh (nm)

Photopic V(λ)

Wavelength (nm)

Photopic V(λ)

380 390 400 410 420 430 440 450 460 470 480 490 500 510 520 530 540 550 560 570 580

0.000039 0.00012 0.000396 0.00121 0.0040 0.0116 0.023 0.038 0.060 0.09098 0.13902 0.20802 0.323 0.503 0.710 0.862 0.954 0.99495 0.995 0.952 0.870

590 600 610 620 630 640 650 660 670 680 690 700 710 720 730 740 750 760 770 780 –

0.757 0.631 0.503 0.381 0.265 0.175 0.107 0.061 0.032 0.017 0.00821 0.004102 0.002091 0.001047 0.000520 0.000249 0.00012 0.00006 0.00003 0.000015 –

Without a color filter array, the number of photons is represented by

n ph—blackbody (T ) =

∫

λ4 λ3

−1

⎛ hc ⎞ M e ( λ, T ) ⋅ ⎜ ⎟ ⋅ d λ ⎝ λ⎠

Km ∫

λ2

λ1

M e ( λ, T ) ⋅ V ( λ ) ⋅ d λ

⎡⎣ photons / cm 2 − lux − sec ⎤⎦

(A.3)

where λ3 and λ4 are the shortest and longest wavelengths, respectively, of the light that hits a detector. If an IR cut filter is used, then λ4 ≤ λ2 . Figure A.1 shows the number of photons per cm2-lux-sec with λ3 = λ1 and λ4 = λ2, as a function of color temperature of the black body light source. With an on-chip color filter array and an IR cut filter, the numerator has to be modified to

n ph—blackbody (T ) =

∫

λ2 λ1

−1

⎛ hc ⎞ M e ( λ, T ) ⋅ ⎜ ⎟ ⋅ T ( λ ) d λ ⎝ λ⎠ λ2

K m ∫ M e ( λ, T ) ⋅ V ( λ ) ⋅ d λ λ1

Copyright © 2006 Taylor & Francis Group, LLC

(A.4)

317

Number of Photons per cm2-lux-sec

Appendix A

2.0E+12 1.8E+12 1.6E+12 1.4E+12 1.2E+12 1.0E+12 3000

4000

5000

6000

7000

8000

Color Temperature T [K]

FIGURE A.1 Number of incident photons per cm2-lux-sec as a function of color temperature of the blackbody light source.

where T(λ) is the total spectral transmittance.

REFERENCE G.C. Holst, CCD Arrays Cameras and Displays, 2nd ed., JCD Publishing, Winter Park, FL, 1998, chap. 2.

Copyright © 2006 Taylor & Francis Group, LLC

Appendix B Sensitivity and ISO Indication of an Imaging System Hideaki Yoshida The sensitivity of a photo detector is usually represented as the ratio of an output signal level to a received illuminance level. In other words, it is a proportionality constant of the linear curve of the output signal level to the input illuminance. When the detector is the charge integrating (accumulating) type, the term “illuminance” in the above sentence should be replaced by “exposure,” which is a product of illuminance and the charge integration time. The above-mentioned “sensitivity” can be defined and measured for both DSCs and image sensors since they both can be considered types of photo detectors. However, it is not a practical “sensitivity” for DSC users because it never identifies which exposure is adequate for taking a picture. The most important parameter for users is the level of exposure needed to generate an adequate output level to produce a good picture. The ISO indicator of sensitivity in a photographic system represents this “adequate exposure,” which is represented in the equation below: ISO indication value S = K / Hm

(B.1)

where K is a constant and Hm is the exposure in lux-seconds. According to the standard ISO 2240 (ISO speed for color reversal films) and ISO 2721 (automatic controls of exposure), a numeric 10 is to be used for K in electronic imaging systems. Thus, the equation can be written as S = 10 / Hm

(B.2)

For example, ISO 100 means that the adequate average exposure Hm (= 10/S) of the imaging system is 10/100 = 0.1 (lux-s).

319

Copyright © 2006 Taylor & Francis Group, LLC

320

Image Sensors and Signal Processing for Digital Still Cameras

Hereupon, there are some parameters that can be regarded as indicating the sensitivity of a DSC described in ISO 12232. Two of them, “ISO saturation speed” and “ISO noise speed,” are described in the present text published in 1998. Two new parameters, “SOS” (standard output sensitivity) and “REI” (recommended exposure index), have been added to the first revision, which is to be published in 2005. Differences among these four parameters are only caused by what is regarded as the adequate exposure in Equation B.2. 1. ISO saturation speed is the S value when the exposure level generates a picture with image highlights that are just below the maximum possible (saturation) camera signal value. The adequate average exposure Hm is regarded as 1/7.8 of the exposure level at the saturation point (saturation exposure), where 7.8 is the ratio of a theoretical 141% reflectance (which is assumed to give the saturation exposure with 41% additional headroom, which corresponds to 1/2 “stop” of the headroom (= 2 ) to an 18% reflectance (the standard reflectance of photographic subjects). Thus, Equation B.2 can be changed into Ssat = 78 / Hsat

(B.3)

where Hsat is the saturation exposure in lux-s. The saturation speed only shows the saturation exposure as a result. Suppose there are DSCs that have the same sensitivity at low-to-medium exposure levels, meaning that their tone curves at those levels are identical. If a tone curve of one of them has a deeper knee characteristic near the saturation exposure, the saturation speed of that DSC becomes lower. Thus, it is preferable to use the saturation speed to indicate the camera’s overexposure latitude. 2. ISO noise speed is the S value when the exposure generates a “clear” picture of a given S/N value. An S/N value of 40 is used for an excellent image, while 10 is used for an acceptable image. This parameter seems to be a good indicator for taking a picture in that it shows the necessary exposure to obtain a certain low-noise image. However, an actual camera often has various image capture settings, such as the number of recording pixels, compression rate, and noise reduction. In these cases, even if the same tone curve and exposure control are used, the S/N changes as the camera settings change. Thus, ISO noise speed does not fit a camera set directly. 3. SOS is the S value when the exposure generates a picture of “medium” output level corresponding to 0.461 times the maximum output level (digital value of 118 in an 8-bit system). Hm in Equation B.1 corresponds to the exposure that produces 0.461 times the maximum output level. The numeric 0.461 corresponds to the relative output level on the s-RGB gamma curve for the 18% standard reflectance of photographic subjects. SOS gives an acceptable exposure because the average output level of the picture becomes “medium.” Thus, it is convenient for a camera set. However, there is no guarantee that the exposure indicated by SOS is the

Copyright © 2006 Taylor & Francis Group, LLC

Appendix B

321

best. Also, it is not suitable for an image sensor, whose output characteristic is linear. 4. REI is the S value when the exposure generates a picture with an “adequate” output level that a camera vendor recommends arbitrarily. According to this definition, it is apparent that REI can apply only to a camera set and that the exposure indicated by REI would be adequate only if the vendor’s recommendation is appropriate. According to above considerations, the author of this appendix suggests the following to designers or manufacturers for communicating with the next users (consumers in the case of DSC manufacturers, and camera designers in the case of image sensor manufacturers): •

•

Use SOS or REI (or both) for indicating the sensitivity of a camera set. They are both effective for users when choosing an exposure level or finding a usable subject brightness. Use ISO speeds for an image sensor. Report noise speed at each noise level (S/N = 40 or S/N = 10, the preference is 40) as basic information. In this case, it is most important to address the signal processing algorithms and an evaluation method of the S/N. Preferably, they are as simple as possible. Also, some kind of standardization is needed, but regretfully there is nothing now.

Also, report the ratio of the saturation speed to noise speed as added information indicating the upper dynamic range.

Copyright © 2006 Taylor & Francis Group, LLC

1

Digital Still Cameras at a Glance Kenji Toyoda

CONTENTS 1.1 1.2

1.3

1.4

1.5

What Is a Digital Still Camera? ......................................................................2 History of Digital Still Cameras......................................................................3 1.2.1 Early Concepts .....................................................................................3 1.2.2 Sony Mavica.........................................................................................4 1.2.3 Still Video Cameras .............................................................................5 1.2.4 Why Did the Still Video System Fail? ................................................7 1.2.5 Dawn of Digital Still Cameras ............................................................8 1.2.6 Casio QV-10.........................................................................................9 1.2.7 The Pixel Number War ......................................................................10 Variations of Digital Still Cameras................................................................10 1.3.1 Point-and-Shoot Camera Type ...........................................................10 1.3.2 SLR Type ...........................................................................................13 1.3.3 Camera Back Type .............................................................................14 1.3.4 Toy Cameras.......................................................................................14 1.3.5 Cellular Phones with Cameras...........................................................15 Basic Structure of Digital Still Cameras .......................................................16 1.4.1 Typical Block Diagram of a Digital Still Camera ............................16 1.4.2 Optics .................................................................................................16 1.4.3 Imaging Devices.................................................................................17 1.4.4 Analog Circuit....................................................................................17 1.4.5 Digital Circuit ....................................................................................17 1.4.6 System Control...................................................................................17 Applications of Digital Still Cameras ...........................................................18 1.5.1 Newspaper Photographs.....................................................................18 1.5.2 Printing Press .....................................................................................18 1.5.3 Network Use ......................................................................................19 1.5.4 Other Applications .............................................................................19

1

Copyright © 2006 Taylor & Francis Group, LLC

2

Image Sensors and Signal Processing for Digital Still Cameras

In this chapter, the author describes briefly the basic concepts of digital still cameras and the history of digital and analog electronic cameras. This is followed by a discussion of the various types of digital still cameras and their basic construction. Descriptions of several key components of typical digital still cameras are given.

1.1 WHAT IS A DIGITAL STILL CAMERA? An image can be described by “variation of light intensity or rate of reflection as a function of position on a plane.” On the other hand, a camera is a piece of equipment that captures an image and records it, where “to capture” means to convert the information contained in an image to corresponding signals that can be stored in a reproducible way. In a conventional silver halide photography system, image information is converted to chemical signals in photographic film and stored chemically at the same point where the conversion takes place. Thus, photographic film has the image storage function as well as the image capture function. Another method of image capture is to convert the image information to electronic signals. In this case, an image sensor serves as the conversion device. However, the image sensor used in the electronic photography system does not serve a storage function as does photographic film in the silver halide system. This is the most significant point in which the electronic system differs from the chemical silver halide system (Figure 1.1). Naturally, the electronic photography system needs another device to store the image signals. Two primary methods have been adopted to perform this storage function: analog and digital. Analog electronic still cameras, which were once on the market, use a kind of floppy disk that electromagnetically records the image signals in the form of video signals. In digital still cameras, the image signals from the image sensor are converted to digital signals and stored in digital storage devices such as hard disks, optical disks, or semiconductor memories.

FIGURE 1.1 Difference between a silver halide photographic camera and an electronic still camera.

Copyright © 2006 Taylor & Francis Group, LLC

Digital Still Cameras at a Glance

3

FIGURE 1.2 Classification of cameras.

Thus, still cameras are divided into two groups from the viewpoint of the image capture method: conventional silver halide cameras and electronic still cameras. Electronic still cameras are again divided into two groups: analog and digital (Figure 1.2). Thus, a digital still camera is defined as “a camera that has an image sensor for image capture and a digital storage device for storing the captured image signals.”

1.2 HISTORY OF DIGITAL STILL CAMERAS 1.2.1 EARLY CONCEPTS The idea of taking pictures electronically has existed for a long time. One of the earliest concepts was shown in a patent application filed by a famous semiconductor manufacturer, Texas Instruments Incorporated, in 1973 (Figure 1.3). In this embodiment drawing, the semiconductor image sensor (100) is located behind the retractable mirror (106). The captured image signals are transferred to the electromagnetic recording head (110) and recorded onto a removable ring-shaped magnetic drum. This concept did not materialize as a real product. In 1973, when this patent was filed, image sensor technology was still in its infancy, as was magnetic recording technology. In another patent application filed in 1978, Polaroid Corp. suggested a more advanced concept (Figure 1.4). In this camera, the image signals are recorded on a magnetic cassette tape. Considering the amount of information an image contains, it must have taken a long time to record a single image. One of the advanced features of this camera was that it had a flat display panel (24) on the back to show the recorded image (Figure 1.5). Note that LCD panels at that time could only display simple numerals in a single color. Another interesting item of note on Polaroid’s concept is that this camera had a built-in color printer. Polaroid is famous for its instant camera products, so they must have thought that the output print was best done in the camera. The printing method was neither inkjet nor thermal transfer. Inkjet printers were not as popular then as they are now and the thermal dye transfer method did not even exist in 1978.

Copyright © 2006 Taylor & Francis Group, LLC

4

Image Sensors and Signal Processing for Digital Still Cameras

108 106 104

110 102 112 100 114

FIGURE 1.3 Early concept of electronic still cameras (USP 4,057,830).

FIGURE 1.4 Another early concept of an electronic still camera (USP 4,262,301).

Actually, it was a wire dot impact printer. A paper cassette could be inserted into the camera body to supply ink ribbon and sheets of paper to the built-in printer. Thus, several early digital still camera concepts were announced in the form of patent applications prior to 1981.

1.2.2 SONY MAVICA The year 1981 was a big one for camera manufacturers. Sony, the big name in the audio–visual equipment business, announced a prototype of an electronic still camera

Copyright © 2006 Taylor & Francis Group, LLC

5

Digital Still Cameras at a Glance

70 68 72 88 74 98

104 24

102 122 92

108 106 334 84

94 82

236

78 234 80

FIGURE 1.5 Rear view of the concept described in Figure 1.4.

FIGURE 1.6 Sony Mavica (prototype).

called “Mavica” (Figure 1.6). This name stands for “magnetic video camera.” As its name suggests, this prototype camera recorded the image signals captured by the semiconductor image sensor on a magnetic floppy disk. This prototype camera had a single lens reflex finder, a CCD image sensor, a signal-processing circuit, and a floppy disk drive. Several interchangeable lenses, a clip-on type of electronic flash, and a floppy disk player to view the recorded images on an ordinary TV set were also prepared. The image signals recorded to the floppy disk were a form of modified video signals. Naturally, they were analog signals. Therefore, this was not a “digital” still camera. Nevertheless, it was the first feasible electronic still camera ever announced.

1.2.3 STILL VIDEO CAMERAS The announcement of the Sony Mavica caused a big sensation throughout the camera business. Many people felt strong anxiety about the future of conventional silver

Copyright © 2006 Taylor & Francis Group, LLC

6

Image Sensors and Signal Processing for Digital Still Cameras

halide photography. Some even said that silver halide photography would die before long. Several camera manufacturers and electronic equipment makers started a consortium to advance Sony’s idea. After a significant amount of negotiation, they came out with a set of standards for an electronic photography system. It was called “still video system,” which included “still video camera” and “still video floppy.” The still video floppy (Figure 1.7) was a flexible circular magnetic disk that measured 47 mm in diameter. It had 52 coaxial circular recording tracks. The image signals were recorded on tracks 1 through 50, with each track storing signals corresponding to one field — that is, one half of a frame image (Figure 1.8). Thus, a still video

FIGURE 1.7 Still video floppy disk.

1234

7 8 V sync pulse

34 35 36 261 262

V sync signal readout position

)

θ = 7H ± 2H (9°36′ ± 2°45′)

7

ck tra k 1st rac dt nt 2n me e ov ck m h tra t 0 5 ck tra nd PG 52

he

ad

dis

(3

8

ion

at ot kr

rp

1 2 3 4 5 6

m

0 60

µm 100 ch t i p m ck 0µ tra th 6 s wid i k du trac k ra trac m t s fir 20 m

yoke

52nd tr ack rad 14.9 mm ius

disk dia 47 m meter m

FIGURE 1.8 Track configuration of still video floppy.

Copyright © 2006 Taylor & Francis Group, LLC

Digital Still Cameras at a Glance

7

floppy disk could store 50 field images or 25 frame images. The image signals were analog video and based on the NTSC standard. Once the standards were established, many manufacturers developed cameras, equipment, and recording media based on these standards. For example, Canon, Nikon, and Sony developed several SLR types of cameras with interchangeable lenses (Figure 1.9). Various point-and-shoot cameras were developed and put on sale by Canon, Fuji, Sony, Konica, Casio, and others (Figure 1.10); however, none of them achieved sufficient sales. These cameras had many advantages, such as instant play back and erasable and reusable recording media, but these were not sufficient to attract the attention of consumers.

1.2.4 WHY DID

THE

STILL VIDEO SYSTEM FAIL?

With all its advantages, why then did this new still video system fail? It is said that the main reason lies in its picture quality. This system is based on the NTSC video format. In other words, the image recorded in a still video floppy is a frame or a field cut out from a video motion picture. Thus, its image quality is limited by the number of scan lines (525). This means that the image resolution could not be finer than VGA quality (640 × 480 pixels). This is acceptable for motion pictures or even still pictures if they are viewed by a monitor display, but not for prints. Paper prints require high quality and one cannot get sufficient quality prints with VGA resolution, even if they are small type-C prints (approximately 3.5 in. × 5 in.). As a result, the

FIGURE 1.9 SLR type still video camera.

FIGURE 1.10 Point-and-shoot still video camera.

Copyright © 2006 Taylor & Francis Group, LLC

8

Image Sensors and Signal Processing for Digital Still Cameras

use of still video cameras was limited to applications in which paper prints were not necessary; these were quite limited. Another major reason for failure was price. These cameras were expensive. Point-and-shoot cameras had a price tag of $1000 to $2500. SLR cameras were as expensive as $5000. Point-and-shoot models had a poor function equivalent to the $200 silver halide counterpart, with a no-zoom single focus length lens.

1.2.5 DAWN

OF

DIGITAL STILL CAMERAS

The still video cameras can be categorized as analog electronic still cameras. It was natural that analog would transition to digital as digital technologies progressed. The first digital camera, Fuji DS-1P (Figure 1.11), was announced at the Photokina trade show in 1988. It recorded digital image signals on a static RAM card. The capacity of this RAM card was only 2 Mbytes and could store only five frames of video images because image compression technology was not yet applicable. Though this Fuji model was not put on sale, the concept was modified and improved and several digital still cameras did debut in the camera market — for instance, Apple QuickTake 100 and Fuji DS-200F (Figure 1.12). However, they did not achieve good sales, the image quality was not sufficient yet for prints, and they were still fairly expensive. Apparently, they were not significantly different from still video cameras, except for digital storage function. A rather unique model among them was the Kodak DCS-1 (Figure1.13), which was put on sale in 1991. This model was dedicated to newspaper photography. To

FIGURE 1.11 Fuji digital still camera DS-1P.

FIGURE 1.12 Early model of a digital still camera.

Copyright © 2006 Taylor & Francis Group, LLC

Digital Still Cameras at a Glance

9

FIGURE 1.13 Kodak DCS-1.

FIGURE 1.14 Casio QV-10.

satisfy the requirements of newspaper photographers, Eastman Kodak attached a CCD image sensor that had 1.3 million pixels to the body of Nikon F3, the most popular SLR for these photographers. However, the image storage was rather awkward. A photographer had to carry a separate big box that connected to the camera body by a cable. This box contained a 200-Mbyte hard disk drive to store an adequate amount of image signals and a monochromatic monitor display. Nevertheless, this camera was welcomed by newspaper photographers because it could dramatically reduce the time between picture taking and picture transmitting.

1.2.6 CASIO QV-10 Casio announced its digital still camera, QV-10 (Figure 1.14), in 1994 and put it on sale the next year. Contrary to most expectations, this model had great success. Those who had had little interest in conventional cameras, such as female students, especially rushed to buy the QV-10. Why then did this camera succeed while other similar ones did not? One thing is clear: it was not the picture quality because the image sensor of this camera had only 250,000 pixels and the recorded picture was no better than 240 × 320 pixels. The main point might be the LCD monitor display. Casio QV-10 was the first digital still camera with a built-in LCD monitor to view stored pictures. With this monitor display, users could see pictures immediately after they were taken. Furthermore, this camera created a new style of communication, which Casio named “visual communication.” QV-10 was a portable image player, as well as a camera. Right after a picture was taken, the image could be shared and enjoyed on the spot among the friends. No other equipment was necessary, just the camera. Thus, this camera was enthusiastically welcomed by the young generation, even though its pictures did not have sufficient quality for prints.

Copyright © 2006 Taylor & Francis Group, LLC

10

Image Sensors and Signal Processing for Digital Still Cameras

Another key point was, again, the price. The QV-10 sold for 65,000 yen, approximately $600. To reduce the cost, Casio omitted many functions. It had no optical finder, with the LCD display working as a finder when taking pictures. It had no shutter blade or zoom lens. The semiconductor memory for storing image signals was fixed in the camera body and was not removable. In any case, the digital camera market began growing dramatically with this camera’s debut.

1.2.7 THE PIXEL NUMBER WAR Once the prospects for a growing market were apparent, many manufacturers initiated development of digital still cameras. Semiconductor makers also recognized the possibility of significant business and began to investigate development of image sensors dedicated to digital still cameras. Although the Casio QV-10 created a “visual communication” usage that does not require paper prints, still cameras that cannot make prints were still less attractive. More than 1 million pixels were necessary to make fine type-C prints. Thus, the socalled “pixel number war” broke out. Prior to this point, most digital still camera makers had to utilize image sensors made for consumer video cameras. Subsequently, semiconductor manufacturers were more positive about the development of image sensors that had many more pixels. In 1996, Olympus announced its C-800L (Figure 1.15) digital still camera, which had an approximately 800,000-pixel CCD image sensor. It was followed by the 1.3Mpixel Fuji DS-300 and 1.4-Mpixel Olympus C-1400L the next year. In 1999, many firms rushed to announce 2-Mpixel models, and a similar situation was seen in 2000 for 3-Mpixel models. Thus, pixel numbers of digital still cameras used by consumers increased year by year up to 8 Mpixels for point-and-shoot cameras (Figure 1.16) and 16.7 Mpixels for SLR cameras in 2004. Figure 1.17 shows the increase in number of pixels of digital still cameras, plotted against their date of debut.

1.3 VARIATIONS OF DIGITAL STILL CAMERAS The various digital still cameras sold at present are classified into several groups.

1.3.1 POINT-AND-SHOOT CAMERA TYPE Most popular digital still cameras have a similar outfit to that of silver halide pointand-shoot cameras. For this type of camera, the LCD monitor display also works

FIGURE 1.15 Olympus C-800L.

Copyright © 2006 Taylor & Francis Group, LLC

Digital Still Cameras at a Glance

11

FIGURE 1.16 Point-and-shoot digital still camera with 8 Mpixels.

FIGURE 1.17 Increase in number of pixels of point-and-shoot digital cameras.

as the finder to show the range of pictures to be taken. Thus, an optical viewfinder is omitted in some models (Figure 1.18). However, the LCD monitor display has a disadvantage in that it is difficult to see under bright illumination such as direct sunlight. Many models in this category have optical viewfinders to compensate for this disadvantage (Figure 1.19). Most of these finders are the real image type of zoom finder. Figure 1.20 shows a typical optical arrangement of this type. High-end models of the point-and-shoot digital still camera, which have rather high zoom ratio lenses, have electronic viewfinders (EVF) in place of optical ones (Figure 1.21). This type of viewfinder includes a small LCD panel that shows the output image of the image sensor. The eyepiece is a kind of magnifier to magnify this image to reasonable size. Most models that belong to this point-and-shoot

Copyright © 2006 Taylor & Francis Group, LLC

12

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 1.18 Point-and-shoot digital still camera with no optical finder.

FIGURE 1.19 Point-and-shoot digital still camera with an optical finder.

Finder optics

Cover glass Infrared cut filter Shutter blade

LCD monitor

Package Sensor chip

Color filter array

Objective lens Optical low pass filter

Micro lens array

FIGURE 1.20 Typical arrangement of a point-and-shoot digital still camera.

Copyright © 2006 Taylor & Francis Group, LLC

Digital Still Cameras at a Glance

13

FIGURE 1.21 Point-and-shoot digital still camera with an EVF (electronic view finder).

category utilize the output signals of the image sensor as control signals for auto focus and automatic exposure. Recently, most models of this type have included a movie mode to take motion pictures. Usually, the picture size is as small as a quarter VGA and the frame rate is not sufficient. The file format of these movies is motion JPEG or MPEG4; the duration of the movies is limited. However, this function may generate new applications that silver halide photography could not achieve.

1.3.2 SLR TYPE Digital still cameras of the single lens reflex (SLR) type (Figure 1.22) are similar to their silver halide system counterparts in that they have an interchangeable lens system and that most models have a lens mount compatible with the 35-mm silver halide SLR system. Digital SLR cameras have instant return mirror mechanisms just like ordinary SLRs. The major difference is that they have image sensors in place of film and that they have LCD monitor displays. The LCD display built in SLR digital still cameras, however, does not work as a finder. It only functions as a playback viewer for stored images. A few digital SLR models have an image sensor image size equivalent to the 35-mm film format, but most of them have smaller image size — one half or one

FIGURE 1.22 SLR digital still camera.

Copyright © 2006 Taylor & Francis Group, LLC

14

Image Sensors and Signal Processing for Digital Still Cameras

Photo sensor for exposure control Finder eyepiece

Penta prism Shutter curtain Infrared cut filter LCD monitor

Finder screen

Package Sensor chip Color filter array Objective lens

Main mirror Autofocus module

Sub mirror Micro lens array Cover glass Optical low pass filter

FIGURE 1.23 Typical arrangement of an SLR digital still camera.

fourth of the 35-mm full frame. As a result, the angle of view is smaller than that of 35-mm SLRs by 1/1.3 to 1/2 when an objective lens of the same focal length is attached. Recent cost reduction efforts have made this type of camera affordable to the average advanced amateur photographer. Figure 1.23 shows a typical optical arrangement of an SLR digital camera.

1.3.3 CAMERA BACK TYPE Camera back type digital still cameras are mainly for professional photographers to use in photo studios. This type of unit contains an image sensor; signal-processing circuit; control circuit; image storage memory; and, preferably, an LCD monitor display. One cannot take pictures with this unit alone; it must be attached to a medium-format SLR camera with an interchangeable film back or a large format view camera (Figure 1.24). In many cases, a computer is connected with a cable to control the camera and check the image. The image size is larger than point-andshoot or SLR cameras, but not as large as medium-format silver halide cameras.

1.3.4 TOY CAMERAS A group of very simple digital still cameras is sold for about $100 or less each (Figure 1.25). They are called “toy cameras” because manufacturers who do not make cameras, such as toy makers, sell them. To reduce the cost, the image sensor is limited to VGA class and they are equipped with fixed focus lenses and nondetachable image memory. LCD monitor displays are omitted and often no electronic flash is built in. The main use of this type of camera is for taking pictures for Web sites.

Copyright © 2006 Taylor & Francis Group, LLC

15

Digital Still Cameras at a Glance

FIGURE 1.24 Camera back digital still camera.

FIGURE 1.25 Toy camera.

FIGURE 1.26 Cellular phone with a built-in camera.

1.3.5 CELLULAR PHONES

WITH

CAMERAS

Recently, cellular phones have begun to have built-in cameras (Figure 1.26). Their main purpose is to attach a captured image to an e-mail and send it to another person. Fine pictures that have many pixels tend to have large file sizes, thus increasing time and cost of communication. Therefore, the image size of these cameras was limited to approximately 100,000 pixels.

Copyright © 2006 Taylor & Francis Group, LLC

16

Image Sensors and Signal Processing for Digital Still Cameras

However, as this type of cellular phone became popular, people began to use them as digital still cameras that were always handy. To meet this requirement, cellular phone built-in cameras that have image sensors with more pixels — say, 300,000 or 800,000 — began to appear and the pixel number war started just as it did for ordinary digital cameras. In 2004, models debuted that can store an image as large as 3 million pixels. However, the camera functions are rather limited because the space in a cellular phone in which to incorporate a camera is very small.

1.4 BASIC STRUCTURE OF DIGITAL STILL CAMERAS Digital cameras are thought to be a camera in which image sensors are used in place of film; therefore, its basic structure is not much different from that of silver halide cameras. However, some differing points will be discussed in this section.

1.4.1 TYPICAL BLOCK DIAGRAM

OF A

DIGITAL STILL CAMERA

Figure 1.27 shows a typical block diagram of a digital still camera. Usually, a digital camera includes an optical and mechanical subsystem, an image sensor, and an electronic subsystem. The electronic subsystem includes analog, digital–processing, and system control parts. An LCD display, a memory card socket, and connectors to communicate with other equipment are also included in most digital cameras. Each component will be discussed in detail later in this book.

1.4.2 OPTICS Fundamental optics of digital still cameras are equivalent to those of silver halide cameras, except for the fact that the focal length is much shorter because of the smaller image size in most models. However, some additional optical elements are required (Figure 1.20 and Figure 1.23). A filter to attenuate infrared rays is attached

FIGURE 1.27 Typical block diagram of a digital still camera.

Copyright © 2006 Taylor & Francis Group, LLC

Digital Still Cameras at a Glance

17

in front of the image sensor because the sensor has significant sensitivity in infrared range, which affects the image quality. Also arranged in front of the image sensor is an optical low pass filter (OLPF) to prevent moiré artifacts.

1.4.3 IMAGING DEVICES Charge-coupled devices (CCDs) are the most popular image sensors for digital still cameras. However, CMOS sensors and other x–y address types of image sensors are beginning to be used in SLR cameras and toy cameras for various reasons. Arranged on the image receiving surface of these imaging devices are a mosaic filter for sensing color information and a microlens array to condense the incident light on each pixel (Figure 1.20 and Figure 1.23).

1.4.4 ANALOG CIRCUIT The output signals of the image sensor are analog. These signals are processed in an analog preprocessor. Sample-and-hold, color separation, AGC (automatic gain control), level clamp, tone adjustment, and other signal processing are applied in the analog preprocessor; then they are converted to digital signals by an analog-todigital (A/D) converter. Usually, this conversion is performed to more than 8-b accuracy — say, 12 or 14 b — to prepare for subsequent digital processing.

1.4.5 DIGITAL CIRCUIT The output signals of the A/D converter are processed by the digital circuits, usually digital signal processors (DSPs) and/or microprocessors. Various processing is applied, such as tone adjustment; RGB to YCC color conversion; white balance; and image compression/decompression. Image signals to be used for automatic exposure control (AE), auto focus (AF), and automatic white balance (AWB) are also generated in these digital circuits.

1.4.6 SYSTEM CONTROL The system control circuits control the sequence of the camera operation: automatic exposure control (AE), auto focus (AF), etc. In most point-and-shoot digital still cameras, the image sensor also works as AE sensor and AF sensor. Before taking a picture, the control circuit quickly reads sequential image signals from the image sensor, while adjusting the exposure parameters and focus. If the signal levels settle appropriately in a certain range, the circuit judges that proper exposure has been accomplished. For auto focus, the control circuit analyzes the image contrast, i.e., the difference between maximum and minimum signal levels. The circuit adjusts the focus to maximize this image contrast. However, this is not the case in SLR digital cameras. These models have a separate photo sensor for exposure control and another sensor for auto focus, just like the silver halide SLRs (Figure 1.23).

Copyright © 2006 Taylor & Francis Group, LLC

18

Image Sensors and Signal Processing for Digital Still Cameras

1.5 APPLICATIONS OF DIGITAL STILL CAMERAS 1.5.1 NEWSPAPER PHOTOGRAPHS Among the various users of cameras, newspaper photographers have showed the greatest interest in digital still cameras since early days when their ancestors, still video cameras, barely made their debut. It is always their most serious concern to send the picture of an event to their headquarters at the earliest possible time from the place at which the event has taken place. Until 1983, they used drum picture transmitters, which required making a print and wrapping it around the drum of the machine. When the film direct transmitter that could send an image directly from the negative film (Figure 1.28) was developed, the time to send pictures was dramatically reduced because making prints was no longer necessary. However, they still had to develop a negative film. Using electronic still cameras could eliminate this film development process and the output signals could be sent directly through the telephone line. This would result in a significant reduction of time. Many newspaper photographers tested the still video cameras as soon as they were announced, but the result was negative. The image quality of these cameras was too low even for rather coarse newspaper pictures. They had to wait until the mega-pixel digital still cameras appeared (Figure 1.29). Newspaper technologies have seen major innovation during years in which the Olympic Games are held. Still video cameras were tested at the Seoul Olympic Games in 1988, but few pictures actually were used. Mega-pixel digital still cameras were tested in Barcelona in 1992. In 1996, approximately half the cameras that newspaper photographers used in Atlanta were digital still cameras. This percentage became almost 100% at the Sydney Games in 2000.

1.5.2 PRINTING PRESS Printing press technologies were computerized fairly early under the names of CEPS (color electronic prepress system), DTP (desktop publishing), and so on. Only the

FIGURE 1.28 Film direct transmitter.

Copyright © 2006 Taylor & Francis Group, LLC

Digital Still Cameras at a Glance

19

FIGURE 1.29 Mega-pixel digital still camera for newspaper photography (Nikon D1).

picture input device, i.e., the camera, was not digital. Therefore, as the technology of digital still cameras progressed, they were gradually incorporated into the system. At first, digital still cameras were used for printing flyers or information magazines for used cars. The pictures that they use are relatively small and do not require very high resolution; they benefit greatly from the instant availability of pictures that digital cameras provide. They became popular among commercial photographers who take pictures for ordinary brochures, catalogues, or magazines because the picture quality grew higher. Many photographers nowadays are beginning to use digital still cameras in place of silver halide cameras.

1.5.3 NETWORK USE One of the new applications of digital still cameras in which conventional silver halide cameras are difficult to use is for network use. People attach pictures taken by digital still cameras to e-mails and send them to their friends or upload pictures to Web sites. Thus, the “visual communication” that Casio QV-10 once suggested has expanded to various communication methods. This expansion also gave birth to the cellular phone built-in cameras.

1.5.4 OTHER APPLICATIONS Digital still cameras opened a new world to various photographic fields. For instance, astrophotographers could increase the number of captured stars using cameras with cooled CCDs. In the medical arena, endoscopes have drastically changed. They no longer have the expensive image guide made of bundled optical fibers because the combination of an image sensor and a video monitor can easily show an image from inside the human body. Thus, digital still cameras have changed various applications related to photography, and will continue changing them.

Copyright © 2006 Taylor & Francis Group, LLC

2

Optics in Digital Still Cameras Takeshi Koyama

CONTENTS 2.1

Optical System Fundamentals and Standards for Evaluating Optical Performance ......................................................................................22 2.1.1 Optical System Fundamentals ...........................................................22 2.1.2 Modulation Transfer Function (MTF) and Resolution .....................27 2.1.3 Aberration and Spot Diagrams ..........................................................30 2.2 Characteristics of DSC Imaging Optics ........................................................31 2.2.1 Configuration of DSC Imaging Optics..............................................32 2.2.2 Depth of Field and Depth of Focus...................................................32 2.2.3 Optical Low-Pass Filters....................................................................34 2.2.4 The Effects of Diffraction..................................................................36 2.3 Important Aspects of Imaging Optics Design for DSCs...............................38 2.3.1 Sample Design Process......................................................................38 2.3.2 Freedom of Choice in Glass Materials..............................................40 2.3.3 Making Effective Use of Aspherical Lenses .....................................41 2.3.4 Coatings..............................................................................................43 2.3.5 Suppressing Fluctuations in the Angle of Light Exiting from Zoom Lenses .............................................................................46 2.3.6 Considerations of the Mass-Production Process in Design ..............47 2.4 DSC Imaging Lens Zoom Types and Their Applications.............................49 2.4.1 Video Zoom Type ..............................................................................49 2.4.2 Multigroup Moving Zooms ...............................................................50 2.4.3 Short Zooms.......................................................................................50 2.5 Conclusion......................................................................................................51 References................................................................................................................51 In recent years, the quality of images produced by digital still cameras (DSCs) has improved dramatically to the point that they are now every bit as good as those produced by conventional 35-mm film cameras. This improvement is due primarily to advances in semiconductor fabrication technology, making it possible to reduce the pixel pitch in the imaging elements and thereby raising the total number of pixels in each image.

21

Copyright © 2006 Taylor & Francis Group, LLC

22

Image Sensors and Signal Processing for Digital Still Cameras

However, other important factors are involved. These include the development of higher performance imaging optics to keep pace with the lower pixel pitches, as well as improvements to the image-processing technology used to convert large amounts of digital image data to a form visible to the human eye. These three factors — imaging optics, imaging elements, and image-processing technology — correspond to the eye, retina, and brain in the human body. All three must perform well if we are to obtain adequate image quality. In this chapter, I explain the imaging optics used in a DSC, or its “eyes” in human terms, focusing primarily on the nature of the optical elements used and on the key design issues. Because this book is not intended for specialists in the field of optics, I make my descriptions as simple as possible, omitting any superfluous parameters rather than giving a strictly scientific explanation. For a more rigorous discussion of the respective topics, I refer the reader to the specialist literature. In this chapter, lenses in an optical system that provide the image are called imaging lenses, and the entire optical system, including any optical filters, is called the imaging optics. Imaging elements such as CCDs do not form part of the imaging optics.

2.1 OPTICAL SYSTEM FUNDAMENTALS AND STANDARDS FOR EVALUATING OPTICAL PERFORMANCE The text begins by briefly touching on a few prerequisites that are indispensable to understanding the DSC imaging optics that will be discussed later. The first is a basic understanding of optical systems and the second is a familiarity with the key terms used in discussions of the performance of those systems.

2.1.1 OPTICAL SYSTEM FUNDAMENTALS First, I explain basic terminology such as focal length and F-number. Figure 2.1 shows the image formed of a very distant object by a very thin lens. Light striking the thin lens from the object side (literature on optics normally shows the object

FIGURE 2.1 Schematic diagram of single lens.

Copyright © 2006 Taylor & Francis Group, LLC

23

Optics in Digital Still Cameras

side to the left and the imaging element side to the right) enters the lens as parallel beams when the object is very distant. The light beams are refracted by the lens and focused at a single point that is at a distance f (the focal length) from the lens. Because Figure 2.1 is a simplified diagram, it only shows the image-forming light beams above the lens’s optical axis L. The focal length f for a single very thin lens can be calculated using a relational expression such as that shown in Equation 2.1, where the radius of curvature for the object side of the lens is R1; the radius of curvature for the image side of the lens is R2; and the refractive index of the lens material is n. In this equation, the radius of lens surface curvature is defined as a positive value when it is convex relative to the object and as a negative value when it is convex relative to the imaging side. 1 1   1 = (n − 1) ⋅  −   R1 R2  f

(2.1)

For example, for a very thin lens that is convex on both sides and has a refractive index of 1.5 and values of 10 and –10, respectively, for R1 and R2, the focal length is 10 mm. From this equation, we can also see that a direct correlation exists between the focal length of the lens and its refractive index minus 1. Thus, simply by increasing the refractive index from 1.5 to 2.0, we halve the focal length of the lens. Of course, actual lenses also have thickness. Here, if we express the thickness of a single lens as d, we get the following relational expression: 1 n − 1 1 − n d (n − 1)2 = + + f R1 R2 nR1 R2

(2.2)

From this equation, we can see that for a lens that is convex on both sides, the thicker the lens is, the longer the focal length will be. This equation can also be used to calculate the combined focal length of a lens that is made up of two very thin lenses. Thus, if we take the focal length of the first lens as f1, the focal length of the second lens as f2, and the gap between the lenses as d, the expression that corresponds to Equation 2.2 is shown in Equation 2.3. 1 1 1 d = + − f f1 f2 f1 f2

(2.3)

The inverse of the focal length is what is called the refractive power or, simply, the power of a lens. Saying that a lens has a high power means that it is strongly refractive, just as we talk about the lenses in a pair of spectacles as being strong. These terms will be used again in the discussion of zoom types in Section 2.4. To get a clearer grasp of the direct correlation between multiple lens configurations and focal length, see Figure 2.2 and Figure 2.3. In Figure 2.2, where the first lens is concave and the second lens is convex, the effect is equivalent to having a

Copyright © 2006 Taylor & Francis Group, LLC

24

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 2.2 Schematic diagram of retrofocus lens.

FIGURE 2.3 Schematic diagram of telephoto lens.

single lens in position A. Therefore, the overall lens is rather long in comparison with its focal length. This configuration is called a retrofocus type and is often used in wide-angle lenses and compact DSC zoom lenses. (The position of the equivalent lens is called the principal point (or more correctly, the rear principal point.) Figure 2.3 shows the reverse situation, in which the first lens is convex and the second lens is concave; the effect is equivalent to having a single lens in position B. This configuration makes it possible for the overall lens to be short in comparison with its focal length. This configuration is called a telephoto type and is widely used in telephoto lenses and in zoom lenses for compact film cameras. However, regardless of the effectiveness of this configuration in reducing the total lens length, the telephoto configuration used in zoom lenses for compact film cameras is not used in compact DSCs. The reason for this is explained in Section 2.3.5. The principal methods used to bend light, other than refraction as discussed earlier, are reflection and diffraction. For example, mirror lenses mostly use reflection to bend light. Lenses for some of the latest single-lens reflex (SLR) cameras on the market now make partial use of diffraction. Even in DSCs in which diffraction is

Copyright © 2006 Taylor & Francis Group, LLC

25

Optics in Digital Still Cameras

not actively used for image formation, the effects of diffraction are sometimes unavoidable; this is discussed further in Section 2.2.4. The F-number of a lens (F) is expressed as half the opening angle q¢ for the focused light beams in Figure 2.1, as shown as in Equation 2.4. F=

1 2 sin θ′

(2.4)

In reality, because this depends on the cross-sectional area of the light beams, the brightness of the lens (the image plane brightness) is inversely proportional to the square of this F-number. This means that the larger the F-number is, the less light passes through the lens and the darker it becomes as a result. The preceding equation also shows that the theoretical minimum (brightest) value for F is 0.5. In fact, the brightest photographic lens on the market has an F-number of around 1.0. This is due to issues around the correction of various aberrations, which are discussed later. The brightest lenses used in compact DSCs have an F-number of around 2.0. When the value of q¢ is very small, it can be approximated using the following equation in which the diameter of the incident light beams is taken as D. This equation is used in many books. F=

f D

(2.5)

However, this equation tends to give rise to the erroneous notion that we can make the F-number as small as we like simply by increasing the size of the lens. Therefore, it is important to understand that Equation 2.4 is the defining equation and Equation 2.5 should only be used as an approximation. In addition, because actual lens performance is also affected by reflection from the lens surfaces and internal absorption of light by the optical materials used, lens brightness cannot be expressed by the F-number alone. For this reason, brightness may also be discussed in terms of the T-number, which is a value that corresponds to the F-number and takes into account the transparency (T) of the imaging optics. To make the imaging optics as transparent as possible, it is necessary to suppress surface reflections from optical system elements such as the lenses. This is achieved through the use of coatings, which are discussed in Section 2.3.4. Generally speaking, the transparency of imaging lenses to the wavelengths of light typically used by compact DSCs (around 550 nm) is between 90 and 95% for most compact cameras with relatively few lenses. In cameras with high-magnification zooms that use ten or more lenses, the figure is generally around 80%. An infrared cut filter and optical low-pass filter have a combined transparency in the 85 to 95% range. This means that, given these transparency levels for lenses and filters, when the transparency of the imaging lens and filters is, for example, 90% in each case, no more than around 80% of the light entering the camera actually reaches the imaging element. Indeed, to be strictly correct, we should also include losses caused by the glass plate covering the imaging element and losses in the element.

Copyright © 2006 Taylor & Francis Group, LLC

26

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 2.4 Schematic diagram of single lens.

Next, I discuss the image formation correlations for objects at limited distances from the camera, as shown in Figure 2.4. Compare this figure with Figure 2.1. Where the object size is y and its distance from the lens is a, an image with a size of y¢ is formed in a position only a distance of q from the focal length f of the lens (distance b from the lens). Distance p is obtained by subtracting the focal length f from distance a. Here, for the sake of simplicity, we will express all these symbols as absolute values rather than giving them positive or negative values. This gives us the following simple geometrical relationship where m is the imaging magnification: m=

y′ q f b = = = y f p a

(2.6)

By substituting a – f for p, we come to the following well-known equation: 1 1 1 = + f a b

(2.7)

In situations like this, in which the distance to the object is limited, the image is formed farther away from the lens than is the case for very far-off objects, so the brightness at the image plane (the surface of the imaging element) is lower. This is similar to the way things get darker as one moves farther away from a window. In this situation, the value that corresponds to the F-number is called the “effective Fnumber” and can be approximated as F¢ using the same approach as Equation 2.5: F′ =

f +q  q f =  1 +  ⋅ = (1 + m ) F D f D 

(2.8)

Thus, an F2.8 lens that produces a life-size image will actually be as dark as an F5.6 lens. Because the F-number acts as a square, as discussed earlier, the brightness (image plane brightness) falls to one fourth that of an image of a very distant object.

Copyright © 2006 Taylor & Francis Group, LLC

27

Optics in Digital Still Cameras

The image plane brightness Ei can be expressed using Equation 2.9, where Eo is the object brightness (luminance). However, essentially, we can think of the brightness as inversely proportional to the square of the effective F-number (1 + m)F and directly proportional to the transparency of the optical system (T).   π 1 Ei = EoT   4  (1 + m ) F 

2

(2.9)

This equation gives us the brightness at the center of the image, but the brightness at the periphery of the image is generally lower than this. This is known as edge illumination fall-off. For a very thin lens with absolutely no vignetting or distortion, edge illumination falls in proportion to cos4q, where q is the angle diverging from the optical axis facing towards the photographed area on the object side (half the field of view). This is called the cosine fourth law. However, depending on the lens configuration, the actual amount of light at the edge of an image may be greater than the theoretical value, or the effects of distortion may be not inconsiderable. The former can be seen, for example, when we look from the front into a wideangle lens with a high degree of retrofocus; in this case the lens opening (the image of the aperture) appears larger when we look from an angle than it does when we look from directly in front. With wide-angle lenses, which tend toward negative distortion (barrel distortion), this corresponds to compression of the periphery of the photographed image, which has a beneficial effect on the edge illumination. Ultimately, matching this to the imaging element is an unavoidable problem for camera design, as I discuss further in Section 2.3.5. In imaging optics, the field of view is twice the angle q (half the field of view) discussed previously. Where the radius of the imaging element's recording area is y¢, the relationship between y¢ and q can be expressed by the following equation: y ′ = f tan θ

(2.10)

There is no clear definition for the terms “wide-angle lens” and “telephoto lens”; lenses with a field of view of 65∞ or more are generally regarded as wide-angle and those with a field of view of 25∞ or less are called telephoto.

2.1.2 MODULATION TRANSFER FUNCTION (MTF) RESOLUTION

AND

Modulation transfer function (MTF) is frequently used as a standard for evaluating the imaging performance of any lens, not merely those used in DSCs. MTF is a transfer function for spatial frequency and is one of the measures used to show how faithfully object patterns are transferred to the image. The graph in Figure 2.5 shows an example of MTF spatial frequency characteristics, with the vertical axis showing the MTF as a percentage and the horizontal axis showing the spatial frequency (linepairs/mm). The unit for the horizontal axis shows the number of stripes of light and

Copyright © 2006 Taylor & Francis Group, LLC

28

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 2.5 Example of MTF spatial frequency characteristics.

dark (line-pairs) photographed per millimeter in the image plane. The graph normally shows a line falling away to the right as it does in the figure because the higher the frequency (the more detailed the object pattern), the greater the decline in object reproducibility. These graphs are best seen as a gauge for reproducing contrast through a range from object areas with coarse patterning to areas with fine patterning. The term “resolution” has also often been used in this context. This is a measure for evaluating the level of detail that can be captured, and line-pairs/mm is also widely used as a unit of resolution. TV lines are also used as a unit of resolution when lenses are used for electronic imaging, such as in DSC and, particularly, video lenses. This is based on the concept of scanning lines in TVs and is calculated as line-pairs ¥ 2 ¥ the vertical measurement of the imaging element recording plane. For example, for a Type 1/1.8 CCD, a resolution of 100 line-pairs/mm equates to roughly 1000 TV lines. In general, a “high-resolution” lens refers to a sharp lens that gives excellent reproduction of finely detailed patterns. Thus, a very close correlation exists between MTF and resolution. This is discussed in more detail in Section 2.2.4, but the theoretical limit on the number of imaging lines is reached at the point at which the MTF is roughly 9%. In an actual lens, the resolution limit is generally reached at MTF levels between 10 and 20% due to the effects of factors such as aberration, which will be discussed later. The simplest approach is to think of a high MTF at high frequencies as indicating sharpness, and a high MTF at medium frequencies as indicating the reproducibility of contrast in ordinary objects. Figure 2.6 shows three patterns of MTF spatial frequency characteristics. In this figure, the MTF for C is low in the medium frequencies and is retained even at high frequencies, indicating that when detailed patterns are photographed, contrast is insufficient overall, with no modulation between light and dark. By contrast, in pattern B, the MTF is high in the medium frequencies but low for high frequencies. This equates to high contrast but low resolution, which will produce images that appear to be out of focus. Pattern A has a high MTF for high and medium frequencies

Copyright © 2006 Taylor & Francis Group, LLC

Optics in Digital Still Cameras

29

FIGURE 2.6 (See Color Figure 2.6 following page 178.) Three patterns of MTF spatial frequency characteristics.1

FIGURE 2.7 (See Color Figure 2.7 following page 178.) Images for three patterns of MTF spatial frequency characteristics: high resolution, high contrast; low resolution, high contrast; high resolution, low contrast.

and will give images with good contrast and resolution. Figure 2.7 shows the images for three patterns of MTF spatial frequency characteristics. However, the MTF is by no means an all-purpose evaluation tool; care is essential because it frequently leads to misunderstandings. For example, even when the MTF is high, differences in the actual images produced can arise in which the MTF is

Copyright © 2006 Taylor & Francis Group, LLC

30

Image Sensors and Signal Processing for Digital Still Cameras

limited by chromatic aberration or by comatic flare. Distortion has no direct impact on MTF. These types of aberration are discussed in more detail in the next section.

2.1.3 ABERRATION

AND

SPOT DIAGRAMS

Ideal image formation by a lens, expressed in simple terms, meets the following conditions: • • •

Points appear as points. Planes appear as planes. The subject and its image are the same shape.

The reasons for a lens’s failure to meet these conditions are the aberrations in that lens. Lens aberrations were classified mathematically by Seidel in 1856. When polynomials are used to approximate spherical surfaces (lens surfaces), aberrations that apply as far as third-order areas are called third-order aberrations; my explanation will be limited to these aberrations, even though consideration of higher order aberrations is indispensable in actual lens design. According to Seidel’s classifications, five basic third-order aberrations affect monochromatic light, collectively referred to as the Seidel aberrations (there are also nine fifth-order aberrations, known as the Schwarzschild aberrations): 1. Spherical aberration is an aberration caused by the fact that the lens surface is spherical. This means that an image formed of a point light source on the optical axis cannot be focused to one point. This can be corrected by reducing the aperture size. 2. Comatic aberration is an aberration that produces a comet-like flare with a tail for a point light source that is off the optical axis. This can usually be corrected by reducing the aperture size. 3. Astigmatism is an aberration that causes a point light source to be projected as a line or ellipse rather than as a point. The shape of the line changes by 90∞, depending on the focal point (e.g., a vertical line becomes horizontal). Reducing the aperture reduces the effects of this aberration. 4. Curvature of field is an aberration that causes the focal plane to curve in the shape of a bowl so that the periphery of the image is out of focus, when the object is a flat surface. Reducing the aperture size also reduces the effects of this aberration because it increases the depth of focus. 5. Distortion refers to aberration that distorts the image. The previously mentioned barrel distortion sometimes found in wide-angle lenses is an example of distortion expanding middle of the image relative to the top and bottom, reminiscent of the shape of a barrel. This aberration by itself has no effect on the MTF and is not corrected by reducing the aperture size. Some aberrations are collectively referred to as chromatic aberration:

Copyright © 2006 Taylor & Francis Group, LLC

31

Optics in Digital Still Cameras

1. Axial or longitudinal chromatic aberration refers to the fact that different colors (wavelengths of light) have different focal points. Reducing the aperture size remedies the effects of this aberration. 2. Chromatic difference of magnification or lateral chromatic aberration refers to the fact that different colors have different rates of magnification. Accordingly, point-symmetrical color bleeding can be seen from the center of the image towards the margins of the image. Reducing the aperture size does not correct this aberration. These aberrations can be recorded in a figure known as an aberration chart, but that alone does not give the viewer a complete understanding of the nature of the lens. Spot diagrams, which show the images of a point light source as a number of spots, are often used for this. Figure 2.8 shows an example of a color spot diagram. The function that describes light and shade in a spot diagram is called the point spread function. When converted to its real number component using Fourier transformation, this function gives the MTF discussed earlier.

2.2 CHARACTERISTICS OF DSC IMAGING OPTICS In this section, I examine the characteristics of the imaging optics used in DSCs, focusing particularly on how they differ from the optical systems used in conventional film cameras. Section 2.3 deals with information on some important design aspects of DSC imaging optics; this section will be confined to the external characteristics. On axis Y

Y

Z

Y

Z

Z

Y

Y

Z

Z

FIGURE 2.8 (See Color Figure 2.8 following page 178.) Example of a color spot diagram.

Copyright © 2006 Taylor & Francis Group, LLC

32

Image Sensors and Signal Processing for Digital Still Cameras

2.2.1 CONFIGURATION

OF

DSC IMAGING OPTICS

Figure 2.9 shows an example of a typical configuration for imaging optics used in a DSC. DSC imaging optics normally consist of the imaging lenses, an infrared cut filter, and an optical low-pass filter (OLPF). Added to this is an imaging element, such as a charge-couple device (CCD). As mentioned earlier, in this chapter I refer to that part of the optical system that contributes to forming the image as the imaging lens and to the total optical system, including any filters, as the imaging optics. Imaging elements such as CCDs are not included in the imaging optics. Imaging lenses such as zoom lenses are made up of several groups of lenses, and zooming is achieved by varying the spacing between these lens groups. This, together with the differences between DSCs and film cameras, is explained in more detail in Section 2.3.5 and Section 2.4. Infrared cut filters are, as the name suggests, filters that cut out unwanted infrared light. These filters are needed because imaging elements such as CCDs are by their nature highly sensitive to unwanted infrared light. The infrared cut filter is generally positioned behind the imaging lens (between the lens and the imaging element), but can in some cases be fitted in front (on the object side). Most infrared cut filters are absorption-type filters, but some are reflective, using a thin film deposited on the filter by evaporation, and some combine both methods. Optical low-pass filters (OLPFs) are normally positioned in the closest part of the imaging lens to the imaging element. OLPFs are discussed in more detail in Section 2.2.3.

2.2.2 DEPTH

OF

FIELD

AND

DEPTH

OF

FOCUS

One of the key characteristics of DSCs, particularly compact DSCs, is that they combine a large depth of field with a small depth of focus. This section looks at this characteristic in detail. Depth of field — in other words, the area (depth) within which the object is in focus — is proportional to the square of the focal length of the imaging optics. Infrared cut filter and optical low-pass filter (OLPF)

Imaging lenses

Cover glass of CCD

FIGURE 2.9 Example of a typical configuration for imaging optics used in a DSC.

Copyright © 2006 Taylor & Francis Group, LLC

33

Optics in Digital Still Cameras

Where d is the circle of least confusion, which is the area that is allowed to be out of focus on the imaging element, the hyperfocal distance, Dh (the distance which, when used to set the focus, effectively sets the focus to infinity), is given by: Dh =

f2 Fδ

(2.11)

As this equation shows, the depth of field at the object grows rapidly as the focal length is shortened. Whereas the normal image size for 35-mm film is 43.27 mm measured diagonally, the diagonal measurement of the CCD imaging area in current compact DSCs is generally between 5 and 11 mm. Given lenses with the same field of view, the focal length f is proportional to this diagonal length. For instance, if we were to compare the image quality of prints of the same size, given a sufficiently small pixel pitch in the DSC imaging element, the circle of least confusion d is also proportional to the diagonal length. Consequently, provided the F-number of the lens is the same, d is in effect proportional to the focal length f and the hyperfocal distance becomes longer. When the DSC is an SLR-type camera, the imaging element is also larger; this means that the depth of field is smaller, which is ideal for shots such as portraits when the background should be out of focus. With a compact DSC, it is difficult to soften the background in this way, but images with a large depth of field can be obtained without having to reduce the aperture size. The circle of least confusion d is generally said to be around 35 mm on a 35-mm camera. As regards the adequacy of this figure, there is a view that examples exist in which its adequacy is indicated in calculations from factors such as the camera’s ability to discriminate and resolve two points based on observations of a final printed image. The depth of focus D, which refers to the depth on the imaging element side, is expressed in simple terms by Equation 2.12. ∆ = Fδ

(2.12)

Because the depth of focus D is directly proportional to the circle of least confusion d and the F-number, compact DSCs that have imaging elements with a short diagonal also have a small depth of focus. Table 2.1 shows specific examples of calculations for the depth of field and depth of focus in 35-mm film cameras and compact DSC cameras. The examples in the table compare the hyperfocal distance and depth of focus for different image sizes for the equivalent of a 38-mm F2.8 lens when converted to 35-mm film format. For example, the hyperfocal distance for a 38-mm F2.8 lens for a 35-mm film camera is 14.7 m, but the focal length of a lens with the same field of view using a type 1/2.7 imaging element is 5.8 mm. Given a 3-Mpixel imaging element and the same F-number of F2.8, the hyperfocal distance for the lens is roughly 2 m. In other words, by setting the focus to 2 m, an image can be obtained that is in focus from a distance of 1 m from the camera to infinity, enabling the camera to be used in what is virtually pan-focus mode.

Copyright © 2006 Taylor & Francis Group, LLC

34

Image Sensors and Signal Processing for Digital Still Cameras

TABLE 2.1 Comparison of Depth between 35-mm Film Camera and DSCs (F2.8) f

No. pixels Pixel pitch Image circle

d

Dh

D

35-mm film 38 mm Type 1/1.8 7.8 mm 4 Mpixels

3.125 mm

ø 43.27 mm 35 mm 14.7 m 98 µm ø 8.9 mm 6.3–7.5 mm 3.4–2.9 m 17–21 mm

Type 1/2.7

2.575 mm

ø 6.6 mm

5.8 mm 3 Mpixels

5.2–6.2 mm 2.3–1.9 m 14–17 mm

The corollary of this is that the depth of focus is extremely small. For a 35-mm film camera, the circle of least confusion d is around 35 mm, as stated earlier, which gives a depth of focus of 98 mm when calculated for an F2.8 lens. In other words, the image is focused through a range 98 mm in size in front of and behind the film. There are various approaches to determining the circle of least confusion in DSCs, but it is generally taken to be between 2 and 2.4 units of the imaging element’s pixel pitch. For example, for the type 1/2.7 imaging element mentioned earlier, the circle of least confusion is between 5.2 and 6.2 mm, so that the depth of focus for an F2.8 lens is between 12 and 20 mm, or around one sixth the depth of the depth on a 35-mm film camera. This figure for the depth of focus is generally of little interest to users, but requires extremely precise focus control from the manufacturers. It also demands lens performance that gives a very high level of field flatness with minimal curvature of field.

2.2.3 OPTICAL LOW-PASS FILTERS As discussed earlier, optical low-pass filters (OLPFs) are currently an indispensable part of DSCs and camcorders. They are normally made using thin, birefringent plates made from liquid crystal or lithium niobate, but diffractive optical elements or special aspherical surfaces may sometimes be used instead. Birefringency refers to the property of a material to have different refractive indexes depending on the direction in which light is polarized. Put simply, a birefringent material is a single material with two refractive indexes. Many readers will be familiar with the experience of holding a calcite plate over text and seeing the text appear in two places. If we take the two refractive indexes as ne and no and the thickness of the OLPF as t, the distance separating the two points (e.g., the amount by which the text is displaced) is given by Equation 2.13. S=t⋅

ne 2 − no 2 2 ne no

(2.13)

The reason for inserting an OLPF is to mitigate the moiré effect caused by the interaction of patterns in the object and the pattern on the imaging element caused

Copyright © 2006 Taylor & Francis Group, LLC

Optics in Digital Still Cameras

35

by the arrangement of imaging element pixels at a fixed pitch. To prevent moiré (false colors) caused by the high-frequency component included in the object, frequencies that are higher than the Nyquist frequency should be eliminated. The details of this process are the subject of another chapter, but its impact on optical performance is briefly discussed here. Figure 2.10 shows an example of the MTF for an imaging lens and the MTF frequency characteristics for a typical OLPF. By combining these two sets of characteristics, we see the final optical performance of the imaging optics as a whole. Accordingly, it is essentially not necessary for lens performance at frequencies above the OLPF cut-off frequency.2 For example, if we set the Nyquist frequency as the OLPF cut-off frequency and use a CCD with a pixel pitch of 3.125 mm (equivalent to a type 1/1.8, 4-Mpixel CCD), a simple calculation shows that MTF above 160 line-pairs/mm is no longer relevant. Therefore, if our premise is that we use this sort of CCD, even with an imaging lens capable of resolutions of 200 line-pairs/mm or above, that resolving power is no longer needed. However, because lens performance is not something that suddenly disappears at high frequencies, in order to ensure a high MTF at frequencies below the Nyquist frequency, a lens with high resolving power is normally essential. The key goal here is to guarantee performance at frequencies below those cut out by the OLPF, and the important thing is the level of the MTF at the range of frequencies between 30 and 80% of the Nyquist frequency. The cut-off frequency actually used in products varies slightly depending on the manufacturer, but the norm is to set it above the Nyquist frequency. OLPFs made from materials such as liquid crystal are constructed in various different configurations, including composite filters composed of multiple liquidcrystal plates; filters with phase plates sandwiched between other layers; and more simple filters that comprise a single liquid-crystal plate. Product specifications also vary from manufacturer to manufacturer. In an actual DSC, other factors besides the imaging optics MTF affect the final overall image quality. These include MTF deterioration due to the imaging element (involving characteristics such as pixel aperture ratio and color filter positioning) and variations in the MTF caused by image processing (edge sharpening, etc.).

FIGURE 2.10 Example of the MTF for (a) an imaging lens and (b) a typical OLPF.

Copyright © 2006 Taylor & Francis Group, LLC

36

Image Sensors and Signal Processing for Digital Still Cameras

2.2.4 THE EFFECTS

OF

DIFFRACTION

The recent progress in the miniaturization of the pixel pitch in the imaging elements used in DSCs discussed at the start of this chapter has resulted in reductions in the size of the circle of least confusion. This progress is particularly marked in compact DSCs. We have now reached the level at which the pixel pitch is only five or six times the wavelength of light, and it is inevitable that the effects of diffraction will be increasingly felt. It is believed that further reduction in the pixel pitch will occur in the future, so this problem is becoming more pressing. Light has the characteristics of rays and of wave motion. The imaging optics used in conventional film cameras have been designed in the so-called geometric areas of refraction and reflection, but the design of modern DSCs with their tiny pixel pitch must be handled using so-called wave optics that also allows for diffraction. Diffraction refers to the way light bends around objects. One can still listen to the radio when standing between two buildings because the signals from the broadcasting station bend around the buildings into the spaces between them. Light, which can be thought of simply as ultrahigh-frequency radio waves, is also bent around in the same way into microscopically small areas. In a geometrical approach that treats light simply as straight lines, the image formed by an aberration-free lens of a point body (an object that is a point) will be capable of being focused to a perfect point. However, in wave optics, which takes wave characteristics into consideration, the image is not focused to a point. For example, given an aberration-free lens with a circular aperture, a bright disc surrounded by concentric dark circles is formed. This is known as the Airy disc, and the brightness of the first ring (the primary diffracted light) is just 1.75% of the brightness at the center; although it is very dim, it does exist. The radius of the first dark ring r is shown by Equation 2.14, where l is the wavelength and F is the Fnumber. r = 1.22 λF

(2.14)

This shows that the radius largely depends on the F-number. Rayleigh took this distance to the first dark ring to be the criterion of resolving power for a two-point image. This is known as the Rayleigh limit, and the intensity at the median area between these two points is roughly 73.5% of the peak intensity for the two points. As a dimension, this correlates to the reciprocal of the spatial frequency. The graph in Figure 2.11 shows the relationship between the F-number of an ideal (aberration-free) lens with a circular aperture and the monochromatic MTF frequency characteristics for the helium d-line (587.56 nm). Where the horizontal axis is the spatial frequency v, this is shown as in Equation 2.15. MTF ( ν) =

(

2 2 ⋅ cos−1 ( λF ν ) − λF ν 1 − ( λF ν ) π

Copyright © 2006 Taylor & Francis Group, LLC

)

(2.15)

37

Optics in Digital Still Cameras

FIGURE 2.11 Relationship between the F-number and the MTF.

If we substitute 1/r from Equation 2.14 for v in this equation, we see that the MTF value is 8.94%. Thus, with the Rayleigh limit as our premise, in ideal conditions the point at which the MTF is roughly 9% corresponds to the number of spatial resolution lines at the Rayleigh limit. In this way, if we take the number of resolution lines at the Rayleigh limit in ideal conditions as the number of lines at which the MTF is roughly 9%, then F11, for instance, equates to roughly 123 line-pairs/mm. This is lower than the Nyquist frequency of 160 line-pairs/mm for the type 1/1.8, 4-Mpixels CCD mentioned earlier. This leads us to conclude that small apertures should be avoided as far as possible with lenses for high-megapixel imaging elements. The usual methods for avoiding the use of small apertures include increasing the shutter speed and using an ND filter. There is also an approach to considering resolution in a situation in which absolutely no intensity is lost in the area between the two points — that is, where the two points are perfectly linked. This is called Sparrow resolution and is shown in Equation 2.16. However, this is hardly ever used in imaging optics.3 r ′ = 0.947 λF

(2.16)

In reality, due to the effects of aberrations, the resolution limit in most situations is reached at an MTF of between 10 and 20%. Figure 2.11 shows the MTF for an ideal lens with a circular aperture; however, the effects of diffraction also vary slightly depending on the shape of the aperture. This is shown in the graph in Figure 2.12, which shows the respective monochromatic MTF frequency characteristics for an ideal F2.8 lens with a circular, rectangular (square), and diamond-shaped aperture. Also, if the goal is simply to increase the resolving power, a degree of control is possible by inserting a filter with a distributed density into the imaging optics (e.g., using an apodization filter or “super resolution”).

Copyright © 2006 Taylor & Francis Group, LLC

38

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 2.12 MTF for an ideal F2.8 lens with some aperture shape.

2.3 IMPORTANT ASPECTS OF IMAGING OPTICS DESIGN FOR DSCS In this section, the design process used for designing imaging optics for DSCs is discussed as well as the key issues that must be considered in that design process.

2.3.1 SAMPLE DESIGN PROCESS We begin by describing an ordinary design process, which is not limited specifically to imaging optics for DSCs. Figure 2.13 shows a typical optical system design process. •

•

•

•

The first step is to formulate the target design specifications, as well as optical specifications, such as the focal length, F-number, and zoom ratio. This includes physical specifications such as the optical performance targets and the total length and diameter. Next, we select the lens type. As described in Section 2.4, because the optimum lens type differs depending on the specifications, efficient lens design requires that a number of lens configurations suited to the specifications be selected in advance. Once the lens type has been selected, the next step is to build a prototype that will act as the basis for the design. For a simple lens system, mathematical methods may be used to determine the initial lens shape analytically. However, currently, the starting point is determined in most cases based on the database of existing lenses and on experience. Next, we perform simulations of light passing through the lens (light tracing) and evaluate the results in terms of how well the optical performance correlates with the target specifications. Based on the results, we change parameters such as the curvature, lens thickness, lens spacing, and

Copyright © 2006 Taylor & Francis Group, LLC

39

Optics in Digital Still Cameras

Formulate the target design specifications Select the lens type Build a prototype

Light tracing

Change parameters

Optimized design Specification Aberrations MTF Simulation Trial Manufacture

Evaluation Detailed evaluation Design fix

FIGURE 2.13 Typical optical system design process.

•

the type of glass. Then we go back and repeat the light tracing evaluation. This repetition or loop is normally performed using a computer, which makes small incremental changes to the parameters while searching for the optimal balance of factors such as optical performance. For this reason, it is referred to as optimized design. Particularly recently, because aspherical lens elements are used in almost all zoom lenses, the number of factors to be evaluated and the number of parameters that can be varied have grown enormously, and high-speed computers have become an indispensable tool. It is interesting to recall that until 50 years ago, design work followed a program in which a pair of women known as “computers” turned the handles on hand-wound calculators in accordance with calculations written by designers, and their calculations were then checked to see whether they matched before the design process moved on to the next stage. In those days there were no zoom lenses or aspherical lenses and probably no way to deal with them. Evaluation at this point mainly involves factors such as the amount of various types of aberration and the lens sizes for a range of object distances, focal lengths, and image heights (location in the image). However, once design has reached a certain point, factors such as MTF, peripheral brightness (the brightness around the edge of the image), and any suspected ghosting are also simulated. The lens configuration is then varied based on the results of these simulations and the testing loop is repeated. The final stage of testing naturally includes detailed design performance evaluation, as well as simulations of problems such as the effects of aberrations in actual manufactured models. If problems are discovered even at this late stage, the lens configuration may still be modified. Of course, it goes without saying that throughout the design process, close

Copyright © 2006 Taylor & Francis Group, LLC

40

Image Sensors and Signal Processing for Digital Still Cameras

collaboration with the design of the lens barrel that will hold the lenses must take place.

2.3.2 FREEDOM

OF

CHOICE

IN

GLASS MATERIALS

DSC imaging optics have progressed to a level at which they are far more compact, more powerful, and more highly specified than was the case several years ago. Advances in design techniques have obviously played a part in this; however, a very significant role has also been played by expansion in the range of available glass types and improvements in aspherical lens technology, which will be discussed later. Figure 2.14 maps the types of glass that can be used in optical design. The vertical axis shows the refractive index, and the horizontal axis shows the Abbe number vd. The Abbe number shows the dispersion and is given by Equation 2.17, where the refractive indexes for the helium d-line (587.6 nm), F-line (486.1 nm), and C-line (656.3 nm) are nd, nF, and nC, respectively. νd =

nd − 1 nF − nC

(2.17)

The Abbe number is an index that shows the extent of the difference in the refractive index caused by the wavelength of the light and can be thought of simply as describing the relationship between the amplitudes of the seven colors of the spectrum into which white light splits when it passes through a prism. The smaller the Abbe number is, the larger the amplitude of the seven colors of the spectrum is. In Figure 2.14, the green portion indicates the existence area of the optical glasses, and the yellow portion indicates the most frequently used types of glass.

FIGURE 2.14 (See Color Figure 2.14 following page 178.) Schematic view of the glass map.

Copyright © 2006 Taylor & Francis Group, LLC

41

Optics in Digital Still Cameras

The number of types of glass used in optical systems is more than 100 just from a single glass manufacturer; if we count the total number of lens glass types from all the manufacturers, there are several hundred or more. From all these, individual glass types are chosen based on a wide range of considerations that include optical performance, durability, environmental friendliness, delivery, and cost. Recently, in addition to these, glasses that show any special dispersion tendencies or that have an ultrahigh refractive index have come to be used in DSCs. The former are effective in reducing chromatic aberration, and the latter are extremely useful in helping produce more compact lenses that offer higher performance. Fluorite is well known as a glass material that shows special dispersion tendencies. It is a crystalline substance rather than an amorphous substance like glass. It not only has an extremely high Abbe number (very low dispersion), but also has abnormal dispersal properties and splits light into the seven colors of the spectrum in a different way from glass. This makes it very effective in correcting chromatic aberration, particularly in telephoto lenses and high-magnification zoom lenses. On the other hand, it is soft and requires particular care when it is handled. Its surface is also very delicate and machining fluorite requires considerable expertise. Glass with an ultrahigh refractive index has in the past presented problems such as deterioration in its spectral transmission characteristics when exposed to strong sunlight (solarization) and staining. However, dramatic improvements in quality have occurred in recent years and it is already used in some compact DSCs to give them a refractive index of more than 2.0. The use of new materials of this sort provides greater benefits for electronic imaging devices such as DSCs than for film cameras. For instance, staining in glass materials is less of a problem for DSCs because, as long as the spectral transmission characteristics do not change over time, even a little staining from the outset can be corrected by electronically setting the white balance.

2.3.3 MAKING EFFECTIVE USE

OF

ASPHERICAL LENSES

Aspherical lenses, as well as correcting the problem of spherical aberration described in Section 2.1.3, are effective in correcting other nonchromatic aberrations. Depending on where the aspherical elements are used, the dampening effect on the respective aberrations differs. An aspherical lens is said to have the effect of two or three spherical lenses, but when minor improvements in performance are taken into account, it is not unusual for them to be even more effective than this. Because the onus is on size reductions to provide greater convenience and portability in a compact DSC, the key requirements are a combination of greater compactness and improved performance to take advantage of the smaller pixel pitches provided by the imaging element. When the only requirement is better performance, the simple solution is to increase the number of lenses, but this is counterproductive from the viewpoint of size reductions. On the other hand, simply reducing the number of lenses means that no option remains except to reduce the power (the reciprocal of the focal length) of each lens group in order to maintain the same performance. This often results in the lens getting larger. In short, there is no alternative but to include an efficient aspherical lens.

Copyright © 2006 Taylor & Francis Group, LLC

42

Image Sensors and Signal Processing for Digital Still Cameras

Aspherical lenses are generally classified into the following for types according to how they are made and the materials used: •

•

•

•

Ground and polished glass aspherical lenses are expensive to produce because they must be individually ground and polished. Altough they are unsuited to mass production, the glass allows a great deal of design flexibility and they can also be used in large diameters. Glass molded (GMo) aspherical lenses are formed by inserting glass with a low melting point into a mold and applying pressure. These lenses provide excellent precision and durability. Small-diameter GMo lenses are ideally suited to mass production. However, they are very difficult to make in large diameters and this gives rise to productivity problems. Composite aspherical lenses are spherical lenses to which a resin film is applied to make them aspherical. Factors such as temperature and humidity must be taken into account when considering how they are to be used. These lenses are relatively easy to use in large diameters. Plastic aspherical lenses are produced by inserting plastic into a mold, usually by injection molding. They are ideally suited to mass production and are the least costly type of aspherical lens to produce. However, care is needed because they are susceptible to variations in temperature and humidity.

Of these four, all but the first type are used in the compact DSCs currently on the market. However, with the increasingly tiny pixel pitches in imaging elements in recent years, demand for glass-molded aspherical lenses has increased enormously due to their excellent precision, reliability, and suitability for mass production. The problem with this is that the materials used in glass-molded lenses are special and often differ from those used in ordinary spherical lenses. This is because the glass must have a low melting point and be suitable for molding. Few types of glass offer these characteristics, so the range of optical characteristics available is far narrower than that offered by the materials used in ordinary spherical lenses. Figure 2.15 shows an example of the effect of aspherical lenses in the PowerShot S100 DIGITAL ELPH (Canon’s compact digital still camera using the type 1/2.7CCD). In order to make the imaging optics smaller, it is vital to improve areas of the optics that have a significant impact on the optical system volume. In the case of DSCs like the PowerShot S100 DIGITAL ELPH, where the goal is greater compactness, the largest component in the optical system is usually the first lens group, so the aim should be to slim down that group somehow. Here, the technological genesis for the reduction of the group 1 size is the technology for mass producing concave meniscus-shaped aspherical GMo lenses with a high refractive index. (Meniscus-shaped lenses are convex on one side and concave on the other.) In Figure 2.15, the group 1 configuration in a conventional lens is shown on the left and the group 1 configuration in the PowerShot S100 DIGITAL ELPH is shown on the right. The conventional lens also uses an aspherical lens element, but by using glass with a higher refractive index (nd > 1.8), the current two concave lenses (one of which is aspherical) can be successfully replaced by a single concave aspherical

Copyright © 2006 Taylor & Francis Group, LLC

43

Optics in Digital Still Cameras

Conventional Lens

PowerShot S100 Digital ELPH

Shorter

nd > 1.8

Narrower

FIGURE 2.15 Example of the effect of aspherical lenses.

lens. This is an excellent example of an improvement that makes the lens shorter and narrower while at the same time making it higher performance.4 The level of precision required in the surfaces of the current aspherical lenses is so strict that if the surface of a small-diameter aspherical lens to be used in a compact DSC were a baseball field, a bulge the size of the pitcher's mound would be unacceptable, and even errors in the length to which the grass was cut would ordinarily be cause for rejection.

2.3.4 COATINGS Like conventional film cameras, DSCs are susceptible to problems involving unwanted light in the image. This is called ghosting and is caused by light reflecting off the surfaces of the lens elements inside the lens. However, DSCs differ significantly from conventional film cameras in that they include elements such as the infrared cut filters discussed earlier that have surface coatings that reflect some of the infrared light. They also include an imaging element, such as a CCD, that has relatively high surface reflectance. To minimize the negative impact of these elements, it is vital to have appropriate coatings on the lens element surfaces. Coatings are applied in a variety of ways. The most widely used method is vacuum deposition, in which the coating material is heated to a vapor in a vacuum and deposited as a thin film on the surface of the lens. Other methods, which are not discussed in detail here, include sputtering and dipping, which involves immersing the lens in a fluid. The history of lens coatings is relatively short, with the first practical applications occurring in Germany in the 1940s. In those days, the initial aim appears to have been to improve the transparency of periscopes. Periscopes use at least 20 lenses, so the drastic loss of transparency due to lens surface reflections was a major stumbling block.5 The technology was later applied to photographic lenses, starting with single-layer coatings and gradually progressing to the current situation in which a range of different multilayer coatings have been developed. However, it has been known for quite a long time that applying a thin, clear coating has the effect of

Copyright © 2006 Taylor & Francis Group, LLC

44

Image Sensors and Signal Processing for Digital Still Cameras

reducing surface reflections. The inhabitants of some South Pacific islands knew to spread oils such as coconut oil on the surface of the water when they were hunting for fish from boats because it made it easier to see the fish. This was, in effect, one type of coating applied to the ocean’s surface. With lenses also, people over the years have noticed that the surface reflectance of a lens was lower when a very thin layer of the lens surface was altered. This alteration was called tarnish, which is one type of defect in lens glass. This accidental discovery is tied up with the development of coating technology. The surface reflectance for perpendicular incident light on an uncoated lens varies depending on the refractive index of the glass used, as shown in Equation 2.18. In the equation, ng is the refractive index of the glass.  ng − 1  R=  ng + 1 

2

(2.18)

For example, the reflectance for perpendicular incident light on glass with a refractive index of 1.5 is 4%. However, for glass with a refractive index of 2.0, the reflectance rises to 11%. This shows why coatings are particularly essential for glass that has a high refractive index. The surface reflectance for perpendicular incident light when a single-layer coating is applied is given by Equation 2.19, where nf is the refractive index of the coating and the product of the coating’s refractive index and the coating thickness is one fourth the wavelength of the light.  ng − n f 2  R′ =  2  ng + n f 

2

(2.19)

Given the preceding equation, the solution in which there is no reflection is given by Equation 2.20. n f = ng

(2.20)

From this equation, for glass with a refractive index of 1.5, we can see that we must use a coating material with a refractive index of 1.22 to eliminate the reflection of light completely with a specific wavelength. However, the coatings with the lowest refractive indexes are still in the 1.33 (Na3AlF6) to 1.38 (MgF2) range, so no solution offers zero reflectance using a single coating. For glass with a refractive index of 1.9, reflections of some wavelengths of light can be completely eliminated even with a single-layer coating, but, of course, reflectance still exists for other wavelengths of light. The reflectance when a two-layer coating is applied is given by Equation 2.21, where n2 is the refractive index of the inner coating applied directly to the glass; n1

Copyright © 2006 Taylor & Francis Group, LLC

45

Optics in Digital Still Cameras

is the refractive index of the outer coating exposed to the air; and the product of the coatings’ refractive indexes and the coating thickness is one fourth the wavelength of the light.  n12 ng − n2 2  R′′ =  2 2  n1 ng + n2 

2

(2.21)

From this equation, the solution in which there is no reflection is given by: n2 = ng n1

(2.22)

This shows that it is possible to eliminate reflections in at least the specific wavelengths from glass with a refractive index of 1.5 by using two coatings with refractive indexes that differ by a factor of 1.22. This sort of combination of coating materials offers a highly practicable solution. Also, by using multilayer coatings in which very thin layers of vaporized material are deposited in alternating layers, techniques have been developed for suppressing reflectance across a wide range of wavelengths for glass materials with widely divergent refractive indexes. Almost all the current DSC imaging lenses include lens element surfaces on which multilayer coatings are used. Figure 2.16 shows the correlation between the refractive index of glass and its reflectance, along with an example of the reflectance when a single-layer coating with a refractive index of 1.38 is applied. From the graph it is clear that, as the refractive index of the glass increases, its reflectance rises rapidly if no coating is used. Figure 2.17 shows the wavelength dependency of reflectance for single-layer and multilayer coatings. From this, we can see that although reflectance is lowest

FIGURE 2.16 (See Color Figure 2.16 following page 178.) Correlation between the refractive index of glass and its reflectance.

Copyright © 2006 Taylor & Francis Group, LLC

46

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 2.17 (See Color Figure 2.17 following page 178.) Wavelength dependency of reflectance for coatings.

for a single-layer coating for only one wavelength, there are several for the multilayer coating. It is also clear that the multilayer coating suppresses reflectance across a wider range of wavelengths than the single-layer coating does. My discussion so far has looked at the contribution that coatings make to reducing reflections from a lens surface so as to increase the transparency of the imaging optics and prevent ghosting. However, coatings are also used in a number of other roles, such as preventing degeneration of the glass. A typical instance of this degeneration is the tarnishing mentioned earlier in this section. In another example, as discussed in Section 2.2.1, the purpose of the film deposited on some infrared cut filters is to actively reflect some of the infrared light.

2.3.5 SUPPRESSING FLUCTUATIONS IN THE ANGLE OF LIGHT EXITING FROM ZOOM LENSES The image side (imaging element side) of a DSC imaging lens has an OLPF made of a material such as liquid crystal, as discussed earlier, and an imaging element fitted with microlenses for each pixel to ensure that the image is sufficiently bright. If zooming or focusing makes significant changes to the angles of the light exiting from the imaging lens, this can cause problems such as vignetting in the imaging element microlenses and changes in the low-pass effect. The vignetting in the imaging element is particularly undesirable because it leads to shading of the peripheral light in the resulting image. No matter how much thought has been given in the design stages to the provision of ample peripheral light, large variations in the exiting light angle due to zooming will result in a final image that is dark around the edges. This is a major restriction in terms of optical design and imposes limits on the types of zoom that can be used in the design. Figure 2.18 compares the variations in the angle of the exit light during zooming in a zoom lens for a compact film camera and a zoom lens for a compact DSC. From this figure, it is clear that in most cases, the zoom lens group configurations

Copyright © 2006 Taylor & Francis Group, LLC

47

Optics in Digital Still Cameras

Zoom lens for a compact film camera

Zoom lens for a compact DSC

Diverges markedly on the image side

Nearly parallel to the optical axis

Wide-angle

Wide-angle

Marked difference between the wide-angle and telephoto settings Telephoto

No marked difference between the wide-angle and telephoto settings Telephoto

FIGURE 2.18 Comparison of the variations in the angle of the exit light during zooming.

are reversed front to back in zoom lenses for compact film cameras compared with zoom lenses for compact DSCs. In zoom lenses for compact film cameras, the convex lens group is at the object end of the lens while the concave lens group is at the film end of the lens. This is the retrofocus type configuration described in Section 2.1.1. The result of this is that in zoom lenses for compact film cameras, light beams passing through the center of the aperture at the wide-angle setting, diverge markedly on the image (film) side. In the zoom lens for a compact DSC, there is no marked difference between the wide-angle and telephoto settings, and the angle of the light is always nearly parallel to the optical axis. This is called a telecentric lens.6 Given that most imaging elements are now fitted with microlenses to ensure sufficient light levels directly in front of the imaging elements, this telecentric requirement means that it is basically impossible to transfer zoom lenses for compact film cameras directly to DSCs. However, in the interchangeable lenses used on SLR cameras, the distance from the rearmost lens to the film plane (the back focus) is long to avoid any interference with the quick-return mirror; the result is that the light beams are necessarily almost telecentric. Consequently, many of the lenses developed for SLR film cameras can be used on DSCs without causing problems.

2.3.6 CONSIDERATIONS IN DESIGN

OF THE

MASS-PRODUCTION PROCESS

As has already been mentioned in Section 2.2.2., the depth of focus in imaging optics for DSCs is extremely small due to the very small pixel pitch of the imaging elements. Also, because image-forming performance is required at very high spatial frequencies, the level of difficulty in the manufacturing process is generally extremely high. For this reason, the conversion to mass production not only requires

Copyright © 2006 Taylor & Francis Group, LLC

48

Image Sensors and Signal Processing for Digital Still Cameras

higher standards of manufacturing precision at the individual component level, but also means that efficient and highly precise adjustment must also form a key part of the assembly process. Even at the optical design stage, designers must consider how to reduce the likelihood that errors in the final manufacturing and assembly stages will affect the product. Normally, greater compactness is achieved by increasing the power (the reciprocal of the focal length) of the lens groups. However, the use of this means alone makes achieving basic performance more difficult and also increases the susceptibility of the lens to optical system manufacturing errors. Then problems such as eccentricity caused by very tiny manufacturing errors have significant adverse effects on the final optical performance. This is why every effort is made to eliminate such factors through measures introduced at the design stage. Figure 2.19 shows an example of one such measure taken at the design stage. This figure compares the configurations of the second lens group in the PowerShot A5 Zoom (also Canon’s compact digital still camera using the type1/2.7CCD) and the PowerShot S100 DIGITAL ELPH, along with their respective vulnerabilities to eccentricity within the group. The vertical axis shows the amount of image plane curvature in the meridional image plane for an image with a height that is 70% of the diagonal measurement of the imaging plane. In other words, the length of this bar graph effectively indicates the likelihood that blurring will occur around the edges of the image. In the PowerShot A5 Zoom, groups G4 to G6 are made up of an independent three-lens configuration in which one concave lens is sandwiched between two convex lenses in what is known as a triplet-type configuration. Although this is not an unusual configuration, any shift in the relative positions of the respective lens elements will have a major impact on optical performance, as the bar graph in Figure

FIGURE 2.19 (See Color Figure 2.19 following page 178.) Comparison of decenter sensitivity.

Copyright © 2006 Taylor & Francis Group, LLC

Optics in Digital Still Cameras

49

2.19 shows. The configuration in the PowerShot S100 DIGITAL ELPH uses two pairs of joined lens elements (G3-G4 and G5-G6). This configuration takes the triplet-type configuration used in the PowerShot A5 Zoom and links the convex and concave lenses together. The measure introduced here was to reduce the vulnerability of the lenses that were particularly sensitive to changes in their relative positions by connecting together the lenses involved. Accordingly, we can see that the second lens group in the PowerShot S100 DIGITAL ELPH has far lower vulnerability to eccentricity than previous models and has very highly stable manufacturing characteristics. Among the current DSCs, the cameras with the smallest pixel pitch in the imaging elements have now reached the 2-mm level, so even very slight manufacturing errors will have a major impact on the final optical performance. For this reason, from the design stage on down, it is vital that careful thought be given to manufacturing quality issues such as mass-production performance.

2.4 DSC IMAGING LENS ZOOM TYPES AND THEIR APPLICATIONS At present, the zoom lenses for DSCs are not divided into wide-angle zooms and telephoto zooms, as is the case for the interchangeable lenses for SLR cameras; all fall into the standard zoom range. Nevertheless, a wide range of options is available, with number of pixels supported ranging from the submegapixel level through to several megapixels, and the zoom magnifications also ranging from two times up to ten times or higher. Thus, it is necessary to choose the zoom type that best matches the camera specifications. The illustration in Figure 2.20 shows the correlation between zoom types and the zoom magnifications and their supported pixel counts. Bear in mind that the classifications used here are for the sake of convenience and are not intended to be strictly applied. The following sections will look at the characteristics of the respective zoom types.7

2.4.1 VIDEO ZOOM TYPE In this type of zoom, the first lens group is a relatively high-powered convex group, and almost all the magnification during zooming is normally done by the relatively powerful concave second group. In total, the optics most often consist of four lens groups with the final convex group used for focusing, although this type of lens can also be configured with five or six lens groups. This zoom type shows little variation in the angle at which light exits from the lens during zooming and is the type best suited to high-magnification applications. Also, by remaining fairly flexible in the allocation of power in the groups and increasing the number of lens elements used in the configuration, these lenses can be designed so that their optical performance can cope with large numbers of pixels. However, because it is difficult to reduce the total number of lens elements in the configuration and the diameter of the first group is large, it is difficult to reduce the lens size; they have a tendency to increase in cost. All of the 10¥ zoom lenses currently on the market are of this type.

Copyright © 2006 Taylor & Francis Group, LLC

50

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 2.20 (See Color Figure 2.20 following page 178.) The correlation between zoom types and the zoom magnifications and their supported pixel counts.

2.4.2 MULTIGROUP MOVING ZOOMS These are zoom lenses with four or five lens groups that do not fall into the preceding category. In these lenses, the first group may be convex or concave, depending on the specifications and aims of the lens. Lenses of this type can be miniaturized to some extent; however, they are also capable of achieving the twin goals of zoom magnifications at the four-times or five-times level and high performance capable of supporting large numbers of pixels. If the first group is a relatively weak convex group, they can be designed with a relatively bright F-number (around F2). If the first lens group is concave, they are ideal for zoom lenses that begin at a wide-angle setting. However, these lenses are not suitable for ultracompact applications, and in terms of cost they are looked upon as being towards the high end of the scale. Of the zoom lenses currently on the market, those at around F2 with a magnification of between 3¥ and 4¥ are probably of this type.

2.4.3 SHORT ZOOMS These are the type of zoom that is currently most widely used as the imaging optics for compact DSCs. They are generally composed of a concave first group, a convex second group that takes care of most of the changes in magnification, and a convex third group that handles the focusing. Short zooms with a two-group configuration have also been around for a while. Recently, lenses have appeared on the market that consist of four lens groups and fall somewhere between short zooms and the multigroup moving zooms discussed previously. An aspherical lens is normally included in the first group to correct distortion, but when the first group is composed entirely of spherical lens elements, the first lens (the lens closest to the object) is made convex. At least one aspherical lens element is normally used in the second group. This is the zoom type most suited to

Copyright © 2006 Taylor & Francis Group, LLC

Optics in Digital Still Cameras

51

ultracompact applications because it is the most amenable to reductions in the number of component lens elements. However, this type of zoom is not capable of particularly high magnifications due to the large variations in the angle of light exiting the lens during zooming and the large changes in the F-number. The current limit on the magnification for short zooms is around 4¥.

2.5 CONCLUSION In this chapter, I have discussed imaging optics which, if we look at the components of a DSC in human terms, would represent the eyes of the camera. Of course, as far as the quality of the image finally produced by a DSC goes, the characteristics of the imaging element, which acts like a net to catch the light, and the image processor, which is the brain of the camera, make a huge difference in terms of the MTF, the amount of light close to the optical axis (center brightness), and the peripheral brightness. Thus, even when the same imaging optics are used, the quality and character of the final image may be very different. Nonetheless, just as a person who is near or far sighted is unable to see the details of an object, there is no doubt that the capacity of the imaging optics that provide the initial input is crucially important to the quality of the final image on a DSC. In recent years, increasing miniaturization of the pixel pitch in imaging elements has brought us to the threshold of 2-mm resolutions. However, as discussed in Section 2.2.4, as long as we give sufficient thought to the size of the F-number (and do not reduce the aperture size too much), we can probably build optical systems that will cope with imaging elements that have a pixel pitch of as little as 2 mm or thereabouts. In the future, with the emergence of new optical materials and the effective use of new technologies such as diffraction, we can look forward to further advances in DSC imaging optics. Cameras are irreplaceable tools for capturing memorable moments in life. As optical engineers, it is our job to actively cultivate new ideas that will make DSCs a more integral part of our lives and allow us to continue providing the world with new optical systems.

REFERENCES 1. 2. 3. 4.

EF LENS WORK III (Canon Inc.), 205–206, 2003. T. Koyama, The lens for digital cameras, ITE 99 Proc., 392–395, 1999. K. Murata, Optics (Saiensu-sha), 178–179, 1979. M. Sekita, Optical design of IXY DIGITAL, J. Opt. Design Res. Group, 23, 51–56, 2001. 5. I. Ogura, The Story of Camera Development in Japan (Asahi Sensho 684), 72–77, 2001. 6. T. Koyama, J. Inst. Image Inf. TV Eng., 54(10), 1406–1407, 2000. 7. T. Koyama, Optical systems for camera (3), Optronics, (11), 185–190, 2002.

Copyright © 2006 Taylor & Francis Group, LLC

3 Basics of Image Sensors Junichi Nakamura CONTENTS 3.1

3.2

3.3

3.4

Functions of an Image Sensor ....................................................................... 55 3.1.1 Photoconversion ................................................................................. 55 3.1.2 Charge Collection and Accumulation................................................ 56 3.1.3 Scanning of an Imaging Array .......................................................... 56 3.1.3.1 Charge Transfer and X–Y Address .................................... 56 3.1.3.2 Interlaced Scan and Progressive Scan................................ 59 3.1.4 Charge Detection................................................................................ 59 3.1.4.1 Conversion Gain ................................................................. 60 Photodetector in a Pixel................................................................................. 61 3.2.1 Fill Factor ........................................................................................... 61 3.2.2 Color Filter Array............................................................................... 62 3.2.3 Microlens Array ................................................................................. 63 3.2.4 Reflection at the SiO2/Si Interface..................................................... 64 3.2.5 Charge Collection Efficiency ............................................................. 65 3.2.6 Full-Well Capacity ............................................................................. 66 Noise............................................................................................................... 66 3.3.1 Noise in Image Sensors ..................................................................... 66 3.3.2 FPN..................................................................................................... 67 3.3.2.1 Dark Current ....................................................................... 68 3.3.2.2 Shading ............................................................................... 72 3.3.3 Temporal Noise .................................................................................. 72 3.3.3.1 Thermal Noise .................................................................... 73 3.3.3.2 Shot Noise........................................................................... 74 3.3.3.3 1/f Noise.............................................................................. 74 3.3.3.4 Temporal Noise in Image Sensors ..................................... 74 3.3.3.5 Input Referred Noise and Output Referred Noise ............. 77 3.3.4 Smear and Blooming ......................................................................... 77 3.3.5 Image Lag .......................................................................................... 77 Photoconversion Characteristics .................................................................... 78 3.4.1 Quantum Efficiency and Responsivity .............................................. 78 3.4.2 Mechanics of Photoconversion Characteristics ................................. 79 3.4.2.1 Dynamic Range and Signal-to-Noise Ratio....................... 79 3.4.2.2 Estimation of Quantum Efficiency ..................................... 81 3.4.2.3 Estimation of Conversion Gain .......................................... 81

53

Copyright © 2006 Taylor & Francis Group, LLC

54

Image Sensors and Signal Processing for Digital Still Cameras

3.4.2.4 Estimation of Full-Well Capacity....................................... 82 3.4.2.5 Noise Equivalent Exposure ................................................ 83 3.4.2.6 Linearity.............................................................................. 83 3.4.2.7 Crosstalk ............................................................................. 83 3.4.3 Sensitivity and SNR........................................................................... 84 3.4.4 How to Increase Signal-to-Noise Ratio............................................. 84 3.5 Array Performance......................................................................................... 85 3.5.1 Modulation Transfer Function (MTF) ............................................... 85 3.5.2 MTF of Image Sensors, MTF Imager .................................................... 86 3.5.3 Optical Black Pixels and Dummy Pixels .......................................... 87 3.6 Optical Format and Pixel Size....................................................................... 88 3.6.1 Optical Format ................................................................................... 88 3.6.2 Pixel Size Considerations .................................................................. 89 3.7 CCD Image Sensor vs. CMOS Image Sensor............................................... 90 References................................................................................................................ 90 A solid-state image sensor, also called an “imager,” is a semiconductor device that converts an optical image that is formed by an imaging lens into electronic signals, as illustrated in Figure 3.1. An image sensor can detect light within a wide spectral range, from x-rays to infrared wavelength regions, by tuning its detector structures and/or by employing material that is sensitive to the wavelength region of interest. Moreover, some image sensors can reproduce an “image” generated by charged particles, such as ions or electrons. However, the focus of this chapter is on “visible” imaging, corresponding to the spectral response of the human eye (from 380 to 780 nm). Silicon, the most widely used material for very large-scale integrated circuits (VLSIs), is also suitable for visible-image sensors because the band gap energy of silicon matches the energy of visible wavelength photons. To reproduce an image with acceptable resolution, a sufficient number of picture elements or “pixels” must be arranged in rows and columns. These pixels convert

Photons

Lens

Sensor

Output

Camera Image Sensor

FIGURE 3.1 (See Color Figure 3.1 following page 178.) Image sensor.

Copyright © 2006 Taylor & Francis Group, LLC

55

Basics of Image Sensors

the incoming light into a signal charge (electrons or holes, depending on the pixel’s structure). The number of applications using image sensors is growing rapidly. For example, they are now found in state-of-the-art cellular phones, making the phones serious competition for the low-end digital still camera (DSC) market. It is also expected that they will make serious inroads into the automotive industry, which will become a major market for electronic cameras. In fact, some more expensive vehicles already have built-in cameras. Until recently, the charged-coupled device (CCD) image sensor was the technology of choice for DSC applications. However, complementary metal-oxide semiconductor (CMOS) image sensors are rapidly replacing CCDs in low-end camera markets such as toy cameras and PC cameras. In addition, large-format CMOS image sensors have been used for higher end digital single lens reflex (DSLR) cameras. Obviously, image sensors for DSC applications must produce the highest possible image quality. This is achieved through high resolution, high sensitivity, a wide dynamic range, good linearity for color processing, and very low noise. Also, special operation modes such as fast readout for auto exposure, auto focus, and auto white balance are needed, as well as viewfinder and video modes. CCD and CMOS image sensors are discussed in detail in Chapter 4 and Chapter 5, respectively; therefore, this chapter describes overall image sensor functions and performance parameters for both technologies.

3.1 FUNCTIONS OF AN IMAGE SENSOR 3.1.1 PHOTOCONVERSION When a flux of photons enters a semiconductor at energy levels that exceed the semiconductor’s band gap energy, Eg, such that E photon = h ⋅ ν =

h⋅c ≥ Eg λ

(3.1)

where h, c, n, and l are Planck’s constant, the speed of light, the frequency of light, and the wavelength of light, respectively, the number of photons absorbed in a region with thickness dx is proportional to the intensity of the photon flux F(x), where x denotes the distance from the semiconductor surface. Because the band gap energy of silicon is 1.1 eV, light with wavelengths shorter than 1100 nm is absorbed and photon-to-signal charge conversion takes place. On the other hand, silicon is essentially transparent to photons with wavelengths longer than 1100 nm. The continuity of photon flux absorption yields the following relationship1: d Φ( x ) = −α ⋅ Φ( x ) dx

Copyright © 2006 Taylor & Francis Group, LLC

(3.2)

56

Image Sensors and Signal Processing for Digital Still Cameras

where a is the absorption coefficient and depends on wavelength. Solving this equation with the boundary condition F(x = 0) = F0, one can obtain Φ( x ) = Φ 0 ⋅ exp(−αx )

(3.3)

Thus, the photon flux decays exponentially with the distance from the surface. The absorbed photons generate electron-hole pairs in the semiconductor with densities that follow Equation 3.3. Figure 3.2 shows the absorption coefficient of silicon2 and how the flux is absorbed. The figure shows that the penetration depth, 1/a, the depth at which the flux decays to 1/e, for blue light (l = 450 nm) is only 0.42 mm, while that of red light (l = 600 nm) reaches 2.44 mm.

3.1.2 CHARGE COLLECTION

AND

ACCUMULATION

This section outlines how the generated signal charge is collected at a charge accumulation area inside a pixel. Figure 3.3 illustrates a simple photodiode as a charge collection device. In this example, the p region is grounded and the n+ region is first reset at a positive voltage, VR. It then becomes electrically floating with the reverse bias condition being held. Electrons excited by photons tend to collect at the n+ region, reducing this region’s potential; holes flow to the ground terminal. In this case, the electrons are the signal charge. All CCD and CMOS image sensors for DSC applications operate in this charge-integrating mode first proposed by G. Weckler in 1967.3 Figure 3.4 shows another photoelement, a metal-oxide semiconductor (MOS) diode. When a positive voltage is applied to the gate electrode, the energy bands bend downward, and the majority of carriers (holes) are depleted. The depletion region is now ready to collect free charge. As described in Chapter 4, the MOS diode is a building block of the surface CCD. A buried channel MOS diode is used in the pixels of a frame transfer CCD (see Section 4.1.3 and Section 4.2.1). In both cases (a reverse-biased photodiode and MOS diode), electrons generated inside the depletion region are fully utilized as signal charge. However, only a fraction of the electrons generated in the neutral region deep in the bulk can reach the depletion region through diffusion because no electric field exists at the neutral region; some of the electrons are lost by the recombination process before reaching the depletion region. This issue is revisited in Section 3.2.5.

3.1.3 SCANNING

OF AN IMAGING

ARRAY

3.1.3.1 Charge Transfer and X–Y Address The accumulated charge or the corresponding signal voltage or current must be read out from a pixel in an image sensor chip to the outside world. The signals distributed in two-dimensional space should be transformed to a time-sequential signal. This is

Copyright © 2006 Taylor & Francis Group, LLC

57

Basics of Image Sensors

1.E+06 25ºC

Absorption (cm-1)

1.E+05

1.E+04

1.E+03

1.E+02

1.E+01 200

400

600

800

1,000

1,200

Wavelength (nm) (a) 1 850nm

Photon Flux

0.8

0.6 600nm

0.4 550nm 0.2 450nm 0 0

1

2

3

4

5

Depth (µm) (b)

FIGURE 3.2 Absorption of light in silicon: (a) absorption coefficient; (b) intensity vs. depth.

called “scanning,” and an image sensor should have this capability. Figure 3.5 shows two types of scanning schemes. Several CCD readout architectures, such as the full-frame transfer (FFT), interline transfer (IT), frame transfer (FT), and frame-interline transfer (FIT) architectures, are

Copyright © 2006 Taylor & Francis Group, LLC

58

Image Sensors and Signal Processing for Digital Still Cameras

VR

φR

n+

p

Ephoton n+

Depletion region

p + (a)

(b)

FIGURE 3.3 Reverse biased photodiode: (a) cross-sectional view; (b) energy band diagram.

φ

Depletion region

p

(a)

(b)

VERTICAL SCANNER

FIGURE 3.4 Reverse biased MOS diode: (a) cross-sectional view; (b) energy band diagram.

OUT

HORIZONTAL SCANNER (a)

OUT

(b)

FIGURE 3.5 Imaging array scanning scheme: (a) charge transfer scheme; (b) X–Y address scheme.

discussed in Chapter 4. Figure 3.5(a) illustrates the IT CCD charge transfer scheme, in which each signal charge stored at a pixel’s photodiode is shifted to a vertical CCD (V-CCD) simultaneously over the entire imaging array and is then transferred from the V-CCD to the horizontal CCD (H-CCD). The charge within the H-CCD is transferred to the output amplifier, which converts it into a voltage signal. The charge transfer readout scheme requires almost perfect charge transfer efficiency, which, in turn, requires highly tuned semiconductor structures and process technology.

Copyright © 2006 Taylor & Francis Group, LLC

59

Basics of Image Sensors

Charge Integration

Charge Integration ROW 1

ROW 1

ROW 2

ROW 2

ROW 3

ROW 3

ROW N

ROW N time (a)

time (b)

FIGURE 3.6 Operational timing of CCD and CMOS image sensors: (a) CCD image sensor; (b) CMOS image sensor.

Figure 3.5(b) shows an X–Y addressing scheme used in CMOS image sensors. In most of them, the signal charge is converted to a voltage or a current by an active transistor inside a pixel. As the X–Y address name suggests, a pixel signal addressed by a vertical scanner (a shift register or a decoder) selects a row (Y) to be read out, and a horizontal scanner selects a column (X) to be read out. As is apparent when comparing the two diagrams, the X–Y addressing scheme is much more flexible in realizing several readout modes than the charge transfer scheme. Because CCD and CMOS image sensors are charge-integrating types of sensors, the signal charge on a pixel should be initialized or reset before starting the charge integration. The difference in scanning schemes results in operational timing differences, as shown in Figure 3.6. In the CCD image sensor, the charge reset is done by transferring the charge from a photodiode to a V-CCD. This action occurs at the same time over the entire pixel array. Alternately, the charge reset and the signal readout occur on a row-by-row basis in most CMOS image sensors. 3.1.3.2 Interlaced Scan and Progressive Scan In conventional color television systems, such as National Television Systems Committee (NTSC), Phase Alternating Line (PAL), and Sequential Couleur Avec Memoire (SECAM), the interlace scanning mode is used in which half of the total lines (rows) are scanned in one vertical scan and the other half are scanned in a second vertical scan. Each vertical scan forms a “field” image and a set of two fields forms a single “frame” image, as shown in Figure 3.7(a). Figure 3.7(b) shows a progressive scanning mode, which matches the scanning scheme of a PC monitor. Although progressive scan is preferable for DSC applications, the V-CCD structure becomes complicated, and thus it is more difficult to keep the photodiode area sufficiently large in CCDs. (This issue is addressed in Section 4.3.2 in Chapter 4.) CMOS image sensors for DSC applications operate in the progressive scanning mode.

3.1.4 CHARGE DETECTION The charge detection principle is basically identical for CCD image sensors and most CMOS image sensors. As illustrated in Figure 3.5, CCD image sensors perform charge detection at an output amplifier, and CMOS image sensors accomplish it inside the pixel. Figure 3.8 shows a conceptual diagram of the charge detection

Copyright © 2006 Taylor & Francis Group, LLC

60

Image Sensors and Signal Processing for Digital Still Cameras

Horizontal direction

Vertical direction

Vertical direction

Horizontal direction

Odd field Even field (a)

(b)

FIGURE 3.7 Interlaced scan and progressive scan: (a) interlaced scan; (b) progressive scan.

AV

VOUT

∆ VFD = QSIG/CFD

FIGURE 3.8 Charge detection scheme.

principle. Signal charge, Qsig, is fed into a potential well, which is monitored by a voltage buffer. The potential change, DVFD, caused by the charge is given by ∆VFD =

Qsig CFD

(3.4)

where CFD is the capacitance connecting to the potential well and acts as the chargeto-voltage conversion capacitance. The output voltage change is given by ∆VOUT = AV ⋅ ∆VFD

(3.5)

where AV represents the voltage gain of the voltage buffer. 3.1.4.1 Conversion Gain Conversion gain (mV/e–) expresses how much voltage change is obtained by one electron at the charge detection node. From Equation 3.4 conversion gain is obtained as C .G. =

Copyright © 2006 Taylor & Francis Group, LLC

q [mV/electron] CFD

(3.6)

61

Basics of Image Sensors

pixel

On-Chip Microlens Array On-Chip Color Filter Array

Aperture (Apd)

Dielectric Material Light Shield n+ P

n+

n+

Photodiode

Substrate Apixel (b)

(a)

FIGURE 3.9 A simplified pixel structure: (a) cross-sectional view; (b) plane view.

where q is the elementary charge (1.60218 ¥ 1019 C). Obviously, Equation 3.6 represents the “input referred” conversion gain, and it is not measured directly. The “output referred” conversion gain is obtained by multiplying the voltage gain from the charge detection node to the output and is given by C .G.output _ referred = AV ⋅

q CFD

(3.7)

The most commonly used charge detection scheme is floating diffusion charge detection.4 The charge detection is performed in a CCD image sensor by a floating diffusion structure located at the end of the horizontal CCD register; in CMOS active-pixel sensors (APSs), it is performed inside a pixel. In combination with correlated double sampling (CDS) noise reduction,5 extremely low-noise charge detection is possible. These schemes in CCD image sensors are addressed in Section 4.1.5 in Chapter 4, and those in CMOS image sensors are described in Section 5.1.2.1 and Section 5.3.1 in Chapter 5.

3.2 PHOTODETECTOR IN A PIXEL A simplified pixel structure is shown in Figure 3.9. The following chapters provide details of pixel structures in CCD and CMOS image sensors.

3.2.1 FILL FACTOR Fill factor (FF) is defined as the ratio of the photosensitive area inside a pixel, Apd, to the pixel area, Apix. That is, Fill factor = ( Apd / Apix ) × 100 [%]

Copyright © 2006 Taylor & Francis Group, LLC

(3.8)

62

Image Sensors and Signal Processing for Digital Still Cameras

G

R

G

R

Mg G

B

G

B

G

Cy Ye Cy Ye

G

R

G

R

Mg G

B

G

B

G

Cy Ye Cy Ye

1

Mg G

Mg G

1

B

Relative Response

Relative Response

Ye R G

0.5

0

Cy G 0.5

Mg 0

400

500

600

700

400

500

600

Wavelength (nm)

Wavelength (nm)

(a)

(b)

700

FIGURE 3.10 Color filter arrangement and spectral transmittance: (a) Bayer primary color filter pattern and its responses; (b) CMY complementary color filter pattern and its responses.

Without an on-chip microlens, it is defined by the aperture area not covered with a light shield in typical IT and FIT CCDs. In IT and FIT CCDs, the portion of the pixel covered with the light shield includes the area that holds a transfer gate, a channel stop region that isolates pixels, and a V-CCD shift register. The FF of an FT CCD is determined by the nonphotosensitive channel-stop region separating the V-CCD transfer channels and the CCD gate clocking. In active-pixel CMOS image sensors, at least three transistors (a reset transistor, a source follower transistor, and a row select transistor) are needed and are covered by a light shield. If more transistors are used, the FF degrades accordingly. The area required for them depends on the design rules (feature sizes) of the process technology used. The microlens condenses light onto the photodiode and effectively increases the FF. The microlens plays a very important role in improving light sensitivity on CCD and CMOS image sensors.

3.2.2 COLOR FILTER ARRAY An image sensor is basically a monochrome sensor responding to light energies that are within its sensitive wavelength range. Thus, a method for separating colors must be implemented in an image sensor to reproduce an image of a color scene. For consumer DSC applications, an on-chip color filter array (CFA) built above the photodiode array provides a cost-effective solution for separating color information and meeting the small size requirements of DSCs.* Figure 3.10 shows two types of color filter arrays and their spectral transmittances. *

In some high-end video cameras, three (sometimes four) separate image sensors are used. They are attached on a dichroic prism with each image sensor detecting a primary color. This configuration is usually used for high-end video applications.

Copyright © 2006 Taylor & Francis Group, LLC

63

Basics of Image Sensors

DSC applications mainly use the red, green, and blue (RGB) primary color filter array. RGB CFAs have superior color reproduction and higher color signal-to-noise ratio (SNR), due to their superior wavelength selectivity properties. The most commonly used primary color filter pattern is the “Bayer” pattern, as shown in Figure 3.10(a). Proposed by B.E. Bayer,6 this pattern configuration has twice as many green filters as blue or red filters. This is because the human visual system derives image details primarily from the green portion of the spectrum. That is, luminance differences are associated with green whereas color perception is associated with red and blue. Figure 3.10(b) shows the CMY complementary color filter pattern consisting of cyan, magenta, and yellow color filters. Each color is represented in the following equation: Ye = R + G = W – B Mg = R + B = W – G

(3.9)

Cy = G + B = W – R G=G The transmittance range of each complementary color filter is broad, and higher sensitivity can be obtained compared to RGB primary color filters. However, converting complementary color components to RGB for display can cause a reduction in S/N due to subtraction operations. Also, color reproduction quality is usually not as accurate as that found in RGB primary filters. Material for on-chip color filter arrays falls into two categories: pigment and dye. Pigment-based CFAs have become the dominant option because they offer higher heat resistance and light resistance compared to dye-based CFAs. In either case, thicknesses ranging from submicron to 1 mm are readily available.

3.2.3 MICROLENS ARRAY An on-chip microlense collimates incident light to the photodiode. The on-chip microlens array (OMA) was first introduced on an IT CCD in 1983.7 Its fabrication process is as follows: first, the surface of the color filter layer is planarized by a transparent resin. Next, the microlens resin layer is spin-coated on the planarization layer. Last, photolithographic patterning is applied to the resin layer, which is eventually shaped into a dome-like microlens by wafer baking. Recent progress in reducing pixel size and increasing the total number of pixels has been remarkable. However, sensitivity becomes poorer as a pixel shrinks. This can be countered by a simple on-chip microlens array, but it produces shading due to the positional dependence of incident light angles from an imaging lens to the image sensor, as illustrated in Figure 3.11. Decreasing the distance between the microlens and the photodiode surface reduces this angular response dependency.8,9 A technique has also been introduced

Copyright © 2006 Taylor & Francis Group, LLC

64

Position

Image Sensors and Signal Processing for Digital Still Cameras

Image sensor

Imaging lens Response Microlens array

FIGURE 3.11 Shading caused by positional dependence of incident light angle. Air (n= 1.0) n Color Filter n SiO 2 (n= 1.5) Light Shield

Photodiode

FIGURE 3.12 Double-layer microlens.

in which the positions of the microlenses at the periphery of the imaging array are shifted to correct shading.10,11 FT CCDs provide a wider angular response than IT CCDs because the former have inherently larger fill factors than the latter.12 To increase the light-collection efficiency even further, the gap between each microlens has been reduced.13,14 Also, a double-layer microlens structure, which has an additional “inner” microlens beneath the conventional “surface” microlens, as shown in Figure 3.12, has been developed.15 The inner microlens improves the angular response, especially when smaller lens F-numbers are used, as well as smaller pixel sizes.16 In addition to enhancing sensitivity, the microlens helps reduce smear in CCD image sensors and crosstalk between pixels caused by minority carrier diffusion in CCD and CMOS image sensors.11,17

3.2.4 REFLECTION

AT THE

SIO2/SI INTERFACE

Incident light is reflected at the interface of two materials when the refractive indices are different. Reflectivity (R) of light rays that are incident perpendicular to the materials is given by18

Copyright © 2006 Taylor & Francis Group, LLC

65

Basics of Image Sensors

 n − n2  R= 1  n1 + n2 

2

(3.10)

Thus, with the refractive indices of 1.45 for SiO2 and 3 to 5 for Si, more than 20 to 30% of the incident light is reflected at the silicon surface in the visible light range (400 to 700 nm). To reduce the reflection at the SiO2/Si interface, antireflective films formed above the photodiode have been introduced. A 30% increase in photosensitivity was reportedly obtained with an antireflective film consisting of optimized SiO2/Si3N4/SiO2 layers.19

3.2.5 CHARGE COLLECTION EFFICIENCY Although the upper structure above a detector has been the focus of this chapter, photoconversion at the detector must also be examined. A simplified pixel structure that uses a p+-substrate and a potential profile along the surface to the substrate is shown in Figure 3.13(a). This structure corresponds to the photodiode structure shown in Figure 3.3(a). Because the p+-substrate wafer is widely used for CMOS VLSI or memory devices such as dynamic random access memory (DRAM), most CMOS image sensors, in which signal-processing circuits can be integrated on-chip, use the p+-substrate. In this structure, the depth of the p-region and the minority carrier lifetime (or the diffusion length) of the p-region and the p+-substrate affect the response in long wavelength regions of the spectrum. In general, the response from red to NIR is much higher with this structure than that with an n-type substrate. Figure 3.13(b) shows another detector structure, in which an n-substrate is used. The n-type substrate is biased at a positive voltage, and the p-region is grounded. This structure is commonly used for IT CCDs and is also an option for CMOS image sensors. In it, electrons generated deep below a depth, xp, are swept away to the n-

hv

hv

n+

p

n+ p

p-sub.

n-sub. VSUB

GND

xp

Depth

Potential

Potential

Depth

(a)

(b)

FIGURE 3.13 n+-p photodiode on different substrate types: (a) P-type substrate; (b) N-type substrate.

Copyright © 2006 Taylor & Francis Group, LLC

66

Image Sensors and Signal Processing for Digital Still Cameras

substrate and do not contribute to the signal. Thus, the spectral response at long (red to NIR) wavelengths is reduced. From the preceding discussion, the charge collection efficiency, h(l),is defined by η(λ ) =

Signal charge Photo − generated charge

(3.11)

Charge collection efficiency is determined by the substrate type, impurity profile, minority carrier lifetime in the bulk, and how the photodiode is biased. The psubstrate and n-substrate structures are discussed in Section 4.2.2 in Chapter 4 and Section 5.2.2 in Chapter 5.

3.2.6 FULL-WELL CAPACITY The photodiode operates in the charge-integrating mode, as described in Section 3.1, and, therefore, has a limited charge handling capacity. The maximum amount of charge that can be accumulated on a photodiode capacitance is called “full-well capacity” or “saturation charge” and is given by N sat =

1 q

∫

Vmax

CPD (V ) ⋅ dV [electrons]

(3.12)

Vreset

where CPD and q are the photodiode capacitance and the charge of an electron, respectively. The initial and maximum voltages, Vreset and Vmax, depend on photodiode structures and the operating conditions.

3.3 NOISE 3.3.1 NOISE

IN IMAGE

SENSORS

Table 3.1 summarizes noise components in image sensors. Noise deteriorates imaging performance and determines the sensitivity of an image sensor. Therefore, the term “noise” in image sensors may be defined as any signal variation that deteriorates an image or “signal.” An image sensor for still pictures reproduces two-dimensional image (spatial) information. Noise appearing in a reproduced image, which is “fixed” at certain spatial positions, is referred to as fixed-pattern noise (FPN). Because it is fixed in space, FPN at dark can be removed, in principle, by signal processing. Noise fluctuating over time is referred to as “random” or “temporal” noise. In this book, we use “temporal” when referring to noise fluctuating over time because “random” can also be associated with FPN; for example, “pixel-random” FPN is seen randomly in two-dimensional space. Temporal noise is “frozen” as spatial noise when a snapshot is taken by a DSC so its peak-to-peak value can appear in a reproduced image. Although temporal noise is fixed spatially in a particular shot, it will vary in sequential shots. Temporal noise in

Copyright © 2006 Taylor & Francis Group, LLC

67

Basics of Image Sensors

TABLE 3.1 Noise in Image Sensors Illuminated Dark Temporal Noise Fixed Pattern Noise (FPN)

Below saturation

Above saturation

Photo-response nonuniformity Pixel random Shading

Dark signal nonuniformity Pixel random Shading Dark current nonuniformity (Pixel-wise FPN) (Row-wise FPN) (Column-wise FPN)

Defects Dark current shot noise

Photon shot noise

Read noise (Noise floor) Amplifier noise, etc. (Reset noise) Smear, Blooming Image Lag

video images, on the other hand, is more or less filtered out by the human eye, which cannot respond accurately within a field time (1/60 sec) or a frame time (1/30 sec). Table 3.1 shows noise under dark conditions and under illumination. Under illumination, the noise components seen at dark still exist. The magnitude of FPN at dark and under illumination is evaluated as dark signal nonuniformity (DSNU) and photoresponse nonuniformity (PRNU). Also, smear and blooming are seen beyond the saturation level.

3.3.2 FPN FPN at dark can be treated as an offset variation in the output signal and is evaluated as DSNU. FPN is also seen under illuminated conditions, where it is evaluated as PRNU. If the magnitude of the FPN is proportional to exposure, it is observed as sensitivity nonuniformity or gain variation. The primary FPN component in a CCD image sensor is dark current nonuniformity. Although it is barely noticeable in normal modes of operation, it can be seen in images that have long exposure times or that were taken at high temperatures. If the dark current of each pixel is not uniform over the whole pixel array, the nonuniformity is seen as FPN because the correlated double sampling (CDS) cannot remove this noise component. In CMOS image sensors, the main sources of FPN are dark current nonuniformity and performance variations of an active transistor inside a pixel.

Copyright © 2006 Taylor & Francis Group, LLC

68

Image Sensors and Signal Processing for Digital Still Cameras

n+

iii)

i)

p

ii)

i) Generation in the depletion region ii) Diffusion from a neutral bulk iii) Generation at the surface of Si

FIGURE 3.14 Dark current component in a pixel.

In this section, only the dark current is discussed as a source of FPN. Other sources of FPN in CCD and CMOS image sensors are described in Section 4.2.4 in Chapter 4 and Section 5.1.2.3, Section 5.3.1, and Section 5.3.3 in Chapter 5. 3.3.2.1 Dark Current Observed when the subject image is not illuminated, dark current is an undesirable current that is integrated as dark charge at a charge storage node inside a pixel. The amount of dark charge is proportional to the integration time and is represented by N dark =

Qdark I dark ⋅ t INT = q q

(3.13)

and is also a function of temperature. The dark charge reduces the imager’s useable dynamic range because the full well capacity is limited. It also changes the output level that corresponds to “dark” (no illumination). Therefore, the dark level should be clamped to provide a reference value for a reproduced image. Figure 3.14 illustrates three primary dark current components. Each will be examined and mechanisms of dark current generation discussed. 3.3.2.1.1 Generation Current in the Depletion Region Silicon is an indirect-band-gap semiconductor in which the bottom of the conduction band and the top of the valence band do not occur at the same position along the momentum axis in an energy-momentum space. It is known that the dominant generation-recombination process is an indirect transition through localized energy states in the forbidden energy gap.20 In the depletion region formed at the interface of a reverse-biased p–n junction, the minority carriers are depleted, and the generation processes (electron and hole emissions) become the dominant processes for restoring the system to equilibrium.* *

In a forward-biased diode, the minority carrier density is above the level of the equilibrium, so the recombination processes take place. In the equilibrium condition (zero bias diode), the recombination and the generation are balanced to maintain the relationship of p ◊ n = ni2, where p, n, and ni denote the electron density, hole density, and intrinsic carrier density, respectively.

Copyright © 2006 Taylor & Francis Group, LLC

69

Basics of Image Sensors

From the Shockley–Read–Hall theory,21,22 the rate of electron–hole pair generation under the reverse bias condition can be expressed as     σ p σ n vth N t G=  ni  σ n exp  Et − Ei  + σ p exp  Ei − Et    kT   kT   

(3.14)

where sn = electron capture cross section sp = hole capture cross section vth = thermal velocity Nt = concentration of generation centers Et = energy level of the center Ei = intrinsic Fermi level k = Boltzmann’s constant T = absolute temperature Assuming sn = sp = s0, Equation 3.14 can be rewritten as G=

(vth σ 0 N t ) ⋅ ni n = i  Et − Ei  τ g 2 cosh   kT 

(3.15)

This equation indicates that only the energy states near midgap contribute to the generation rate, and the generation rate reaches a maximum value at Et = Ei and falls off exponentially as Et moves from Ei. The generation lifetime and the generation current are thus given by  E − Ei  2 cosh  t  kT  τg = vth σ 0 N t

J gen =

∫

W

0

qGdx ≈ qGW =

qniW τg

(3.16)

(3.17)

3.3.2.1.2 Diffusion Current At the edges of the depletion regions, the minority carrier density is lower than that of equilibrium and it approaches the equilibrium density of np0 in the neutral bulk region by the diffusion process. Here, our interest is the behavior of the minority

Copyright © 2006 Taylor & Francis Group, LLC

70

Image Sensors and Signal Processing for Digital Still Cameras

carriers (electrons) in the p region. The continuity equation in the neutral region is given by d 2 np np − np0 − =0 dx 2 Dn τ n

(3.18)

where Dn and tn denote the diffusion coefficient and the minority carrier lifetime, respectively. Solving this equation with a boundary condition of np (x = infinite) = np0 and n(0) = 0, yields the diffusion current as

J diff =

qDn n p 0 Dn ni 2 =q ⋅ Ln τn N A

(3.19)

3.3.2.1.3 Surface Generation Because of the abrupt discontinuity of the lattice structure at the surface, a much higher number of energy states or generation centers tends to be created. A discussion similar to the one on generation current can be applied to the surface generation current, which is expressed as J surf =

qS0 ni 2

(3.20)

where S0 is the surface generation velocity.23 3.3.2.1.4 Total Dark Current Based on previous discussion, the dark current, Jd, is expressed as

Jd =

qniW Dn ni 2 qS0 ni [A/cm2] +q ⋅ + 2 τg τn N A

(3.21)

Among these three major components, using typical values at room temperature for the parameters, it can be shown that Jsurf >> Jgen >> Jdiff. However, the surface component can be suppressed by making an inversion layer at the surface of the n region. The inversion empties the midgap levels of electrons through recombination with the holes, thus reducing the chance for electrons trapped in the midgap levels to emit to the conduction band. This is accomplished by introducing a pinned photodiode structure24 that is used in most IT and FIT CCD and CMOS image sensors (see Section 4.2.3 in Chapter 4 and Section 5.2.2 in Chapter 5).

Copyright © 2006 Taylor & Francis Group, LLC

71

Basics of Image Sensors

3.3.2.1.5 Temperature Dependence As seen in Equation 3.17, Equation 3.19, and Equation 3.20, the generation current and the surface generation current are proportional to ni, the intrinsic carrier density, and the diffusion current is proportional to ni2. Because  Eg  ni 2 ∝ T 3 ⋅ exp  −  kT 

(3.22)

the temperature dependence of dark current is expressed as  Eg   Eg  I d = Ad ,gen ⋅ T 3/2 ⋅ exp  − + Bd ,diff ⋅ T 3 ⋅ exp  −   kT   2 kT 

(3.23)

where Ad,gen and Bd,diff are coefficients. Figure 3.15 illustrates a typical temperature dependence of the dark current. In real devices, the temperature dependence of total dark current varies, depending on the magnitude of the coefficients, Ad,gen and B d,diff. Also, the temperature dependence is expressed as exp(–Eg/nkT), where n is between 1 and 2 and Eg/n corresponds to the activation energy of the dark current. 3.3.2.1.6 White Spot Defects As design and process technologies have progressed, dark currents have decreased to very low levels. Therefore, pixels that have extremely high dark currents with an extra generation center become visible as a white spot defect. These defects determine the quality of the image sensor. The causes of white spot defects include 80

70

60

50

40

30

20

10

(°C )

Dark Current (Arbitrary)

10,000

1,000

Total dark current 100

Diffusion current

10

Generation current 1 2.7

2.8

2.9

3

3.1

3.2

3.3

1,000/T (K-1) Generation current ∝ exp(-Eg / 2kT) Diffusion current ∝ exp(-Eg / kT)

FIGURE 3.15 Temperature dependence of dark current.

Copyright © 2006 Taylor & Francis Group, LLC

3.4

3.5

3.6

3.7

72

Image Sensors and Signal Processing for Digital Still Cameras

contamination by heavy metals, such as gold, nickel, cobalt, etc., and crystal defects induced by stress during fabrication.25 3.3.2.1.7 Dark Current from CCD Registers So far, we have focused on the dark currents generated inside a pixel, but dark currents are also generated in the CCD transfer channels of CCD image sensors. As described in Section 4.2.3 in Chapter 4, a negative voltage is applied momentarily to appropriate CCD gates to reduce surface-oriented dark currents. The technique is called “valence band pinning”; a negative voltage inverts the surface for a short period of time, creating a hole layer. Valence band pinning empties the generation centers at the surface, and it takes some time for the centers to start to generate again.26 A detailed description of the process is found in Section 4.1.3. 3.3.1.2.8 Dark Current from a Transistor in an Active Pixel of a CMOS Image Sensor In CMOS image sensors, it is reported that an additional dark current component originates at the active transistor inside a pixel. This is due to hot-carrier effects in the high electric field region near the drain end of an amplification transistor.27,28 A careful pixel layout and proper transistor length and bias setting are required to suppress this dark current component. 3.3.2.2 Shading Shading is a slowly varying or low spatial frequency output variation seen in a reproduced image. The main sources of shading in CCD/CMOS image sensors include: • •

•

Dark-current-oriented shading: if a local heat source exists, the resultant thermal distribution in an imaging array produces dark current gradients. Microlens-oriented shading: if the light collection efficiency of a microlens at the periphery of an imaging array is reduced due to an inclined light ray angle, the output of pixels at the periphery decreases (see Figure 3.11). Electrical-oriented shading: in CCD image sensors, the amplitude of driving pulses to V-CCDs may change spatially due to resistance of the polySi gates that carry the driving pulses. This may cause local degradation of charge transfer efficiency, thus yielding shading.

In CMOS image sensors, nonuniform biasing and grounding may cause shading.

3.3.3 TEMPORAL NOISE Temporal noise is a random variation in the signal that fluctuates over time. When a signal of interest fluctuates around its average, which is assumed to be constant, the variance is defined as

Copyright © 2006 Taylor & Francis Group, LLC

73

Basics of Image Sensors

Variance = < ( N − < N >)2 > = < N 2 > − < N > 2

(3.24)*

where < > expresses the ensemble average or statistical average, which is an average of a quantity at time, t, from a set of samples. When the system is “ergodic” or stationary, an average over time from one sample is considered equal to the ensemble average. The variance of a signal corresponds to the total noise power of the signal.** When several noise sources exist that are uncorrelated, the total noise power is given by N

< ntotal 2 > =


(3.25)

i =1

From the central limit theorem, the probability distribution of a sum of independent, random variables tends to become Gaussian as the number of random variables being summed increases without limit. The Gaussian distribution is represented by

p( x ) =

 ( x − m )2  1 exp  −  2σ 2  2 πσ 

(3.26)

where m is the mean or average value and s is the standard deviation or root mean square (rms) value of the variable x. In this case, the standard deviation, s, can be used as a measure of temporal noise. Three types of fundamental temporal noise mechanisms exist in optical and electronic systems: thermal noise, shot noise, and flicker noise. All are observed in CCD and CMOS image sensors. 3.3.3.1 Thermal Noise Thermal noise comes from thermal agitation of electrons within a resistance. It is also referred to as Johnson noise because J.B. Johnson discovered the noise in 1928. Nyquist described the noise voltage mathematically using thermodynamic reasoning the same year. The power spectral density of the thermal noise in a voltage representation is given by SV ( f ) = 4 kTR [V2/Hz]

(3.27)

where k is Boltzmann’s constant; T is the absolute temperature; and R is the resistance.

*

Hereafter, we will use an upper case N (or V) for an average and a lower case n (or v) for temporal noise. The square root of variance is the deviation.

**

Copyright © 2006 Taylor & Francis Group, LLC

74

Image Sensors and Signal Processing for Digital Still Cameras

3.3.3.2 Shot Noise Shot noise is generated when a current flows across a potential barrier. It is observed in a thermionic vacuum tube and semiconductor devices, such as pn diodes, bipolar transistors, and subthreshold currents in a metal-oxide semiconductor (MOS) transistor. In CCD and CMOS image sensors, shot noise is associated with incident photons and dark current. A study of the statistical properties of shot noise shows that the probability that N particles, such as photons and electrons, are emitted during a certain time interval is given by the Poisson probability distribution, which is represented as

PN =

( N ) N ⋅ e− N N!

(3.28)

where N and N are the number of particles and the average, respectively. The Poisson distribution has an interesting property that the variance is equal to the average value, or nshot 2 = < ( N − N )2 > = N

(3.29)

The power spectral densities of thermal noise and shot noise are constant over all frequencies. This type of noise is called “white noise,” an analogy to the white light that has a flat power distribution in the optical band. 3.3.3.3 1/f Noise The power spectral density of 1/f (one-over-f) noise is proportional to 1/fg, where g is around unity. Obviously, the average over time of 1/f noise may not be constant. The output amplifier of CCD image sensors and the amplifier in a CMOS image sensor pixel suffer from 1/f noise at low frequencies. However, 1/f noise is mostly suppressed by correlated double sampling (CDS) as long as the CDS operation is performed in such a way that the interval between the two samples is short enough that the 1/f noise is considered an offset. Discussion of noise in the CDS process can be found in Section 5.3.3.1 in Chapter 5. 3.3.3.4 Temporal Noise in Image Sensors 3.3.3.4.1 Reset Noise or kTC Noise When a floating diffusion capacitance is reset, noise called “reset” or “kTC” noise appears at the capacitance node when the MOS switch is turned OFF. This noise comes from the thermal noise of the MOS switch. Figure 3.16 shows an equivalent circuit of the reset operation. An MOS transistor is considered resistance during the ON period, and thermal noise appears, as shown in Equation 3.27. This noise is sampled and held by a capacitor. The resulting noise power can be calculated by integrating the thermal noise power over all frequencies, with R in Equation 3.27

Copyright © 2006 Taylor & Francis Group, LLC

75

Basics of Image Sensors φ φ

R

C

Z

vn

FIGURE 3.16 kTC noise.

replaced by the real part of the complex impedance of the RC low-pass filter, as follows: vn 2 =

∫

∞

4 kT ⋅

0

R kT ⋅ df = 1 + (2 πfRC )2 C

(3.30)

The noise charge is given by qn 2 = C 2 ⋅ vn 2 = kTC

(3.31)

It can be concluded that the noise is a function only of the temperature and the capacitance value, and thus is called kTC noise. The kTC noise that appears in the floating diffusion amplifier in CCD image sensors can be suppressed by a CDS circuit. In CMOS image sensors, the kTC noise appears at the reset of the charge-detecting node. Suppressing kTC noise through CDS in CMOS sensors depends on the pixel’s configuration, as described in Section 5.2 in Chapter 5. 3.3.3.4.2 Read Noise Read noise, or noise floor, is defined as noise that comes from the readout electronics. Noise generated in a detector is not included. In CCD image sensors, the noise floor is determined by the noise generated by the output amplifier, assuming that the charge transfer in the CCD shift registers is complete. In CMOS image sensors, the noise floor is determined by the noise generated by readout electronics, including the amplifier inside a pixel. In the noise model of an MOS transistor shown in Figure 3.17, two voltage noise sources, thermal noise and 1/f noise, are modeled in series with the gate. The thermal noise is represented by veq 2 =

Copyright © 2006 Taylor & Francis Group, LLC

4 kT α ⋅ ∆f [V2] gm

(3.32)

76

2

veq =

veq,1/f

veq

4kTα ∆f gm

2 Kf ∆f veq,1/f = COX WL f

log power density [V2/Hz]

Image Sensors and Signal Processing for Digital Still Cameras

1/f noise

Thermal noise

log frequency [Hz]

FIGURE 3.17 Noise in MOS transistor.

where gm is the transconductance of an MOS transistor and a is a coefficient that depends on the modes of MOS transistor operation. The value of a is equal to 2/3 for long-channel transistors and a larger value for submicron transistors. The 1/f noise is modeled as veq,1/ f 2 =

Kf ∆f ⋅ [V2] CoxWL f

(3.33)

where Kf is a process-dependent constant and Cox, W, and L denote the gate capacitance per unit area, width, and length of the gate, respectively.29 The noise floor of an image sensor can be estimated using Equation 3.32 or Equation 3.33, depending on the specific amplifier configurations. If an image sensor has additional circuits, such as the gain amplifier and/or FPN suppression circuit often seen in CMOS image sensors, noise generated by those circuits should be added (using Equation 3.25). If the kTC noise mentioned earlier cannot be suppressed by the CDS process, this component should also be included in the read noise. 3.3.3.4.3 Dark Current Shot Noise and Photon Shot Noise Referencing Equation 3.29, dark current shot noise and photon shot noise are given by ndark 2 = N dark

(3.34)

n photon 2 = N sig

(3.35)

where Ndark and Nsig are the average of the dark charge given by Equation 3.13 and the amount of signal charge given by Equation 3.11, respectively. Referencing Equation 3.25, the total shot noise under illumination is given by

Copyright © 2006 Taylor & Francis Group, LLC

77

Basics of Image Sensors

nshot _ total 2 = N dark + N sig

(3.36)

3.3.3.5 Input Referred Noise and Output Referred Noise Obviously, the previous discussion refers to “input referred” noise at the charge detection node, which is obtained from a measured “output referred” noise, in the manner discussed in Section 3.1.4, so that vn,output 2 = ( AV ⋅ C .G.)2 ⋅ nn, pix 2 + vn,sig _ chain 2

nn,input 2 = nn, pix 2 +

vn,sig _ chain 2 ( AV ⋅ C .G.)2

(3.37)

(3.38)

where nn,pix and vn,sig_chain are noise generated at a pixel and noise voltage generated in a signal chain, respectively.

3.3.4 SMEAR

AND

BLOOMING

These phenomena occur when a very strong light illuminates a sensor device. Smear, which appears as white vertical stripes, occurs when stray light impinges on a VCCD register or a charge generated deep in the silicon bulk diffuses into the V-CCD. Blooming occurs when the photogenerated charge exceeds a pixel’s full-well capacity and spills over to neighboring pixels and /or to the V-CCD. To suppress blooming, an overflow drain should be implemented in a pixel. Examples of smear are shown in Figure 4.28 in Chapter 4. For CCD image sensors, see Section 4.2.4.2 in Chapter 4, and for CMOS image sensors, see Section 5.3.3.4 in Chapter 5. Because smear noise can be considered signal chain noise (vn,sig_chain in Equation 3.38) in CMOS image sensors, its contribution is effectively reduced by a factor of (C.G.◊AV). In CCD image sensors, smear noise directly deteriorates CCD image quality because Av◊C.G. in equations 3.37 and 3.38 is equal to 1. (See the architectural differences shown in Figure 3.5.)

3.3.5 IMAGE LAG Image lag is a phenomenon in which a residual image remains in the following frames after the light intensity suddenly changes. Lag can occur if the charge transfer from the photodiode to the V-CCD in an IT CCD is not complete. In a CMOS image sensor with a four-transistor pixel (see Section 5.2.2), this can be caused by an incomplete charge transfer from the photodiode to the floating diffusion. In a CMOS sensor with a three-transistor pixel (see Section 5.2.1), its origin is the soft reset mode when the photodiode reset is performed in a subthreshold mode of an MOS transistor.

Copyright © 2006 Taylor & Francis Group, LLC

78

Image Sensors and Signal Processing for Digital Still Cameras

3.4 PHOTOCONVERSION CHARACTERISTICS 3.4.1 QUANTUM EFFICIENCY

AND

RESPONSIVITY

Overall quantum efficiency (QE) is given by QE (λ ) = N sig (λ ) / N ph (λ )

(3.39)

where Nsig and Nph are the generated signal charge per pixel and the number of incident photons per pixel, respectively. As described earlier, part of the incident photons are absorbed or reflected by upper structures above the photodiode. The microlens and photodiode structure (from the surface to the bulk) determine the effective FF and the charge collection efficiency, respectively. Thus, Equation 3.39 can be expressed as QE (λ ) = T (λ ) ⋅ FF ⋅ η(λ )

(3.40)

where T(l), FF, and h(l) are the transmittance of light above a detector, the effective FF, and the charge collection efficiency of the photodiode, respectively. Nsig and Nph are represented by N sig =

I ph ⋅ Apix ⋅ t INT q

(3.41)

N ph =

P ⋅ Apix ⋅ t INT hν

(3.42)

where Iph is the photocurrent in [A/cm2]. Apix is the pixel size in [cm2]. P is the optical input power in [W/cm2]. tINT is the integration time. q is the electron charge. Responsivity, R(l), is defined as the ratio of the photocurrent to the optical input power and is given by R=

I ph [ A / cm 2 ] qN sig qλ = = QE ⋅ hc P[W / cm 2 ] hνN ph

(3.43)

Referencing Equation 3.43, spectral response can be represented two ways: using responsivity or quantum efficiency. An example is shown in Figure 3.18, in which

Copyright © 2006 Taylor & Francis Group, LLC

79

Basics of Image Sensors

0.4

0.6

QE = 1.0 0.5

0.7 0.5

0.4

Responsivity (A/W)

Quantum Efficiency

0.3

0.3 0.2

0.3 0.2

0.1 0.1

0.1 0 400

500

600

700

800

900

1,000 1,100 1,200

0 400

500

600

700

800

900

1,000 1,100 1,200

Wavelength (nm)

Wavelength (nm)

(b)

(a)

FIGURE 3.18 Spectral response: (a) spectral quantum efficiency; (b) spectral responsivity.

a virtual image sensor response with a constant QE value of 0.5 in the range of 400 to 700 nm, is assumed to highlight the differences between two representations. Also, the relative response, in which the response is normalized by its peak value, is often used. Overall color response is obtained by multiplying the color filter response shown in Figure 3.10 with the image sensor’s response.

3.4.2 MECHANICS

OF

PHOTOCONVERSION CHARACTERISTICS

In this section, the photoconversion characteristics that demonstrate the relationship between the output voltage and exposure will be examined. In DSC applications, exposure, using a standard light source, is most often expressed in lux-seconds. Because the procedure for estimating the number of incident photons coming from a standard light source is somewhat complicated, the photoconversion characteristics will be presented for monochrome light, for which the number of photons per incident light energy can be obtained easily and the mechanics of photoconversion characteristics analyzed. The method for estimating the number of incident photons from a standard light source is provided in Appendix A. Figure 3.19 shows an example of photoconversion characteristics, illustrating signal, photon shot noise, and read noise (noise floor) as a function of incident photons.30 To plot this figure, a virtual image sensor with a pixel size of 25 mm2, C.G. = 40 mV/e–, a full-well capacity of 20,000 electrons, a noise floor of 12 electrons, and a detector’s QE of 0.5 are assumed. Dark current shot noise is not included in the plot. 3.4.2.1 Dynamic Range and Signal-to-Noise Ratio Dynamic range (DR) is defined as the ratio between the full-well capacity and the noise floor. Signal-to-noise ratio (SNR) is the ratio between the signal and the noise at a given input level. They are represented by N  DR = 20 log  sat  [dB]  nread 

Copyright © 2006 Taylor & Francis Group, LLC

(3.44)

80

Image Sensors and Signal Processing for Digital Still Cameras

Dynamic range 100,000

Electrons

10,000

Signal

1,000

Dynamic range Total noise

100 Read noise 10 Photon shot noise 1 1

10

100

1,000

10,000

100,000

Input Photons

FIGURE 3.19 Example of photoconversion characteristics. Apix = 25 mm2; C.G. = 40 mV/e–, Nsat = 20,000 e–, nread = 12 e–.

 N sig  SNR = 20 log  [dB]  n 

(3.45)

In the example of Figure 3.19, DR is calculated as 20◊log(20,000/12) = 64.4 dB. For SNR, the noise, n, is the total temporal noise at the signal level Nsig. When the read noise is dominant in the total noise, SNR is given by  N sig  SNR = 20 log   nread 

(3.46)

and in cases in which the photon shot noise is dominant, it is represented by  N sig   N sig  SNR = 20 log  = 20 log   = 20 log N sig  n photon   N sig 

(3.47)

Figure 3.20 shows the SNR as a function of the number of incident photons. From Equation 3.47, it is understood that the maximum SNR is determined by the full-well capacity only and is given by SNRmax = 20 log N sat = 10 log( N sat )

Copyright © 2006 Taylor & Francis Group, LLC

(3.48)

81

Basics of Image Sensors

60 50 40

SN R (dB)

30 20

Photon shot noise limited 10 0

-10 -20

Read noise limited

-30 1

10

100

1,000

10,000

100,000

Input Photons

FIGURE 3.20 SNR as a function of incident photons.

3.4.2.2 Estimation of Quantum Efficiency QE is readily obtained from Figure 3.19, which is 0.5 in this example. Also, from Equation 3.47, QE can be estimated using the shot noise dominant portion of the SNR plot shown in Figure 3.20: QE =

N sig ( S / N )2 = N photon N photon

(3.49)

where (S/N) is Nsig /nphoton. Next, the number of incident photons (the horizontal axis of Figure 3.19) must be converted to the exposure value, and the number of signal electrons (the vertical axis of Figure 3.19) to the output voltage (in this case, the sense-node voltage or the input referred voltage). For monochrome light, the number of incident photons is given by N photon =

λ ⋅ P ⋅ Apix ⋅ t INT hc

(3.50)

where P is the face-plate irradiance in W/cm2, Apix the pixel area in cm2, and tINT the integration time in seconds. 3.4.2.3 Estimation of Conversion Gain Converting an image sensor’s signal charge to signal voltage is accomplished using the following relationship:

Copyright © 2006 Taylor & Francis Group, LLC

82

Image Sensors and Signal Processing for Digital Still Cameras

Vsig = C .G. ⋅ N sig

(3.51)

where C.G. is the conversion gain (see Equation 3.6). To estimate the conversion gain, use the photon shot noise, expressed in Equation 3.35 as v photon = C .G. ⋅ N sig

(3.52)

Equation 3.51 and 3.52 provide the following relationship: v photon 2 = (C .G.) ⋅ Vsig

(3.53)

Thus, the conversion gain can be obtained as a slope of Vsig – vphoton2 plot. In this technique, the photon shot noise is used as a “signal” that provides useful information, allowing the relationship between the exposure value and the output voltage, as shown in Figure 3.21, to be obtained. 3.4.2.4 Estimation of Full-Well Capacity Equation 3.48 implies that the full-well capacity can be obtained experimentally by measuring the maximum SNR as N sat = 10 SNRmax /10

(3.54)

Also, because the intersection of the signal line and the photon shot noise line in Figure 3.21 occurs at Nsig = 1 (where the signal voltage corresponds to the conversion 1.E+01

Output Voltage (V)

1.E+00

Nsat

Signal

1.E-01

1.E-02

Photon shot noise

N=1

1.E-03

Read noise 1.E-04

Noise equivalent exposure 1.E-05 1.E-06 1.E-05

1.E-04 1.E-03

Input Energy

1.E-02 1.E-01

1.E+00

1.E+01

(µJ/cm2)

FIGURE 3.21 Photo-conversion characteristics. Exposure vs. signal voltage at the charge detection node for monochrome light with wavelength 550 nm; QE = 0.5; Apix = 25 mm2; C.G. = 40 mV/e–; Nsat = 20,000 e–; and nread = 12 e–.

Copyright © 2006 Taylor & Francis Group, LLC

Basics of Image Sensors

83

gain), the full-well capacity or saturation charge, Nsat, can be estimated from the figure. 3.4.2.5 Noise Equivalent Exposure Noise equivalent exposure can be defined as the exposure at which the signal level is equal to the read noise level, which corresponds to SNR = 1. In reality, finding the relationship between the incident photons and signal charge requires a reverse path, starting with a measured relationship between the exposure and the output signal of an image sensor. This method assumes that the photoconversion characteristic is linear with no offset voltage. The nonlinear conversion characteristic and offset, such as that caused by dark current, if any, should be corrected before applying the preceding method to obtain the conversion gain. Also, in actual devices, the SNR, including FPN, actually limits the real image SNR because a snapshot includes both sources. In addition, PRNU limits the total maximum SNR because it grows linearly, while the shot noise grows proportionally to the square root of the total signal electrons. In cases in which PRNU is linear at 1%, the SNR maximum, including PRNU, can never exceed 40 dB, no matter how large the full well becomes. 3.4.2.6 Linearity The photon-to-electron conversion is inherently a linear process. However, the electron-to-signal charge conversion (i.e., the charge collection efficiency) and the signal charge-to-output voltage conversion could be nonlinear processes. In a CCD image sensor, the nonlinearity may originate from the voltagedependent floating diffusion capacitance and the nonlinearity of the output amplifier. However, these contributions are typically very small because the operating range is relatively limited (> t1)

Thermal Diffusion

(b) Thermal diffusion force

Potential

Fringing Field E (y) min.

(c) Fringing field force

FIGURE 4.3 Charge transfer mechanism: (a) self-induced drift force; (b) thermal diffusion force; (c) fringing field force.

where L and W are the length and the width of the electrode G2, respectively; m is the carrier (electron) mobility; and Ceff is the effective storage capacitance per unit area, which corresponds to the gate oxide capacitance for the previously mentioned prototype CCD. V1 – V0 = Q0/LWCeff is the initial voltage to move the carriers into the next electrode G3. Equation 4.1 and Equation 4.2 mean that the decay speed is proportional to the initial charge density, Q0/LW. When the channel voltage of the remaining charge under G2 becomes as small as thermal voltage kT/q (26 mV at room temperature), as shown in Figure 4.3(b), the thermal diffusion occupies the transfer process, which decreases the remaining charge under G2 exponentially with time. The time constant, tth, of the thermal diffusion is expressed by next equation. tth = 4L2/p2D,

(4.3)

where D is the carrier diffusion constant. In case the fringing field is not considered, the thermal diffusion determines the charge transfer performance because the final remaining charge should be as extremely small as several electrons. Actually, the fringing field, Ey, is caused by a voltage difference between two electrodes, as shown in Figure 4.3(c), and accelerates the transfer for remaining charge at the final process. The fringing field intensity and profile depend on the gate oxide thickness, the impurity profile in Si, and the electrode voltage difference. The unit carrier transit time, ttr, through an L length electrode is expressed as

ttr =

Copyright © 2006 Taylor & Francis Group, LLC

1 µ

∫

L

0

(1 / E y )dy .

(4.4)2

99

CCD Image Sensors

Trap Levels Insulator (SiO2)

Vs

Signal Electrons

VG-VFB

Electron Energy

p-type Si Substrate

Metal Electrostatic Potential

Depth (x)

FIGURE 4.4 Energy band diagram in surface channel CCD with electron storage.

The fringing field is a most important force for high-speed operations such as over 10-MHz clocking. It is therefore important to consider how to intensify the fringing field in designing a CCD. The transfer efficiency, h, is useful to evaluate a performance of CCD,4 which is defined as h = [Qt /Qi ]1/N ¥ 100 [%],

(4.5)

where Qi is the input impulse signal charge; Qt is the transferred charge in the leading charge packet; and N is the total number of transfer stages.

4.1.3 SURFACE CHANNEL

AND

BURIED CHANNEL

In the case of a previously introduced CCD, a charge packet is stored in and transferred along the Si surface of an MOS structure, as shown in Figure 4.1 and Figure 4.2. This type is called the surface channel CCD. Figure 4.4 shows the onedimensional energy bands bent by applying positive voltage, VG, on the electrode with storing a charge packet, Qs. The surface potential, Vs, can be obtained by solving the Poisson equations with the depletion approximation as follows5: Vs = qNA xd2/2es

(4.6)

VG – VFB =Vs + Vox = Vs + qNA xd/Cox + Qs/Cox,

(4.7)

where NA is the doping density of acceptor ions. xd is the depletion region width. es is the dielectric constant of Si.

Copyright © 2006 Taylor & Francis Group, LLC

100

Image Sensors and Signal Processing for Digital Still Cameras

Vox is the voltage drop across the oxide. VFB is the flat band voltage. Qs is the stored charge per unit surface area. Cox is the oxide capacitance (oxide dielectric constant eox/oxide thickness tox). By solving Equation 4.6 and Equation 4.7 for Vs, the next equation can be obtained: Vs = VG¢ + V0 – (2 VG¢ V0 + V02)1/2,

(4.8)

VG¢ = VG – VFB – Qs/Cox

(4.9)

V0 = VG – VFB – qNAes/Cox2.

(4.10)

where

From Equation 4.8, the surface potential, Vs, is plotted for VG in Figure 4.5 with NA and tox as parameters. As NA becomes lower and VG rises higher, the curve approximates linear with a slope of one. The surface channel has a serious disadvantage: the high density of carrier trap energy levels is introduced in the forbidden gap of the Si surface; this is called the surface state or the interface state, due to the drastic irregularity of crystal lattice at Si surface. Thus, signal electrons (rarely holes) transferred along the Si surface are 24 dox = 100 nm NA = 5 ⫻ 1014 cm−3 d = 200 nm ox

Surface Potential (V)

20

dox = 300 nm dox = 100 nm NA = 5 ⫻ 1015 cm−3 d = 200 nm ox

16

dox = 300 nm

12

8

4

0

0

4

8 12 Gate Voltage (V)

16

20

FIGURE 4.5 Surface channel potential vs. gate (electrode) voltage with three gate-oxide thicknesses and two substrate impurity densities.

Copyright © 2006 Taylor & Francis Group, LLC

101

CCD Image Sensors Input Diode +20 V SiO2

Output Diode +20 V

Transfer Electrodes +15 V

n+

n-Layer

+15 V OV p-Substrate

Transfer Channel

Potential Contour

FIGURE 4.6 Cross-sectional view of a buried channel CCD. (After Walden, R.H. et al., Bell Syst. Tech. J., 51, 1635–1640, 1972.)

trapped in the surface states, with the probability determined by the trap level distribution, and suffer significant losses of electrons through the overall transfer process.6 In other word, the surface channel cannot transfer charge packets with high transfer efficiency, i.e., higher than 99.99% per unit transfer stage, and is not a good fit for a large-scale CCD. To overcome this problem, the buried channel CCD (BCCD), which has essentially highly efficient transfer capability, was developed.7 The cross-section is shown in Figure 4.6. The buried channel consists of an n-type impurity layer on a p-type Si substrate and is completely depleted by applying a reverse bias between the nlayer and the p-substrate at the initial condition. The channel potential of BCCD is controlled by applying a voltage on the electrodes formed over the n-layer in the same manner as that for the previously mentioned surface channel CCD. The one-dimensional energy band diagram of BCCD is shown in Figure 4.7. Figure 4.7(a) and (b) show band diagrams for a zero reverse bias condition and that of a completely depleted condition by applying sufficient reverse bias voltage on the n-layer with a roughly zero electrode voltage, respectively. As shown, the energy band is pulled up toward the Si surface by the electrode potential. Figure 4.7(c) is a case in which a signal charge packet is in the n-layer. Electrons are stored in the minimum potential region as shown and are separated from the Si surface by the surface depletion layer. Because no electrons interact with the interface states (trap levels), excellent charge transfer efficiency is achieved with the BCCD. The one-dimensional potential profile can also be derived analytically from Poisson equations with depletion approximation for the uniform impurity profiles in the n-layer and p-substrate as8: d2VB/dx2 = –qND/es (0 £ x £ xj)

4.11

d2VB/dx2 = qNA/es (xj < x)

4.12

where VB is channel potential of BCCD; ND is the impurity concentration of the donor ion in the n-layer; and xj is the p–n junction depth. The simple model for the analysis is given in Figure 4.8. By solving Equation 4.11 and Equation 4.12 for the

Copyright © 2006 Taylor & Francis Group, LLC

102

Image Sensors and Signal Processing for Digital Still Cameras

Metal

SiO2

n-Layer

p-Substrate φ pn

xj Depth Potential

(a) Zero bias condition

Buried Channel

VGB

Potential

(b) BCCD is completely depleted by strong reversed bias

Signal Charge

VGB

VBC Potential

(c) Signal charge is introduced in the BCCD.

FIGURE 4.7 One-dimensional energy band diagram of BCCD: (a) zero bias condition. (b) BCCD is completely depleted by strong reversed bias. (c) Signal charge is introduced in the BCCD.

boundary condition that the electrode is biased at VGB (the potential and the dielectric displacement should be continuous at x = xj and x = 0, respectively), the maximum (minimum for electrons) channel potential, VMB, can be expressed as a function of the gate electrode potential, VGB – VFB. VMB = VK − [VK + (VGB − VFB − VI ) ⋅ ( N A + N D ) / N D ]

1/ 2

,

(4.13)

where VK = qN A ( N A + N D )(tOX ε S / εOX + x j )2 / N D

(4.14)

VI = qN D x j (2tOX ε S / εOX + x j ) / 2 ε S .

(4.15)

Figure 4.9 shows calculated VMB – VGB curves with the experimental values for three doping densities of the n-layer as device parameters. In this figure, each curve has a knee point at the negative voltage of VGB. These knee curves are caused by

Copyright © 2006 Taylor & Francis Group, LLC

103

CCD Image Sensors Impurity Density ND

NA xj

O

Depth X (a)

Electric Field tox

Ej

n O

Es Eox

P

xj

Depth X xa

xd1 xd2 (b)

Potential VMB Vs VGB

n O

P xj

xd1 xd2

xa

Depth

X

(c)

FIGURE 4.8 One-dimensional analysis model of BCCD: (a) impurity profiles with box approximation; (b) electric field; and (c) electrostatic potential in BCCD without signal charge and with depletion approximation.

holes injected into the surface of the n-layer from a p-type region surrounding the BCCD such as p-type channel stops. The injected holes terminate electric field (flux) lines and pin the surface potential to the substrate potential. This phenomenon, illustrated in Figure 4.10, is called the valence band pinning; it improves the BCCD characteristics drastically.9 Actually, the holes accumulated in the surface suppress the generation current generated thermally through the surface states distributed near the midband in the forbidden gap. The thermally generated current is called the dark current, and appears as a temporal noise or a fixed pattern noise (FPN) for image sensor applications (explained in Section 4.2). In the valence band pinning condition, the dark current generated from the surface is expressed as10:

Copyright © 2006 Taylor & Francis Group, LLC

104

VMB (V)

Image Sensors and Signal Processing for Digital Still Cameras

16 14 12

: calculated

10 8

ND

6

(⫻016cm−3)

(µm)

: 1.39 : 1.24 : 1.19

1.07 1.06 1.01

4

8

4

- 10

-8

-6

-4

-2

0

: experimental xj

2

6

10

VGB (V)

FIGURE 4.9 Calculated and experimental VMB–VGB characteristics of three BCCD samples under zero-charge condition. Depletion Region

Holes

VGB ( 0.008856 Z0 Z ≤ 0.008856 Z0

a* = 500( Xn − Yn )

(7.5)

b* = 200(Yn − Z n ) u*v*: u* = 13L * ( u ′ − u0′ ) , v* = 13L * ( v ′ − v0′ )

(7.6)

C*(chroma), h (hue angle): 1

Cab * = (a*2 + b*2 ) 2 , hab =

180° −1 b * tan ( ) π a*

(7.7)

where X0, Y0, and Z0 denote the tristimulus values of white point, and u'0 and v'0 are its u'v' chromaticity. These were intended to use simple equations and the normalization by white point does not well model the visual adaptation; therefore, color appearance models such as CIECAM022 have been developed. Color appearance models take into account the adaptation of the visual system into their equations in addition to uniform color space. The model is designed to predict lightness, chroma, hue, etc. in arbitrary observing conditions.

7.1.4 COLOR DIFFERENCE The geometrical difference of two colors in a uniform color space should be proportional to the apparent, or perceived, color difference. The color difference is denoted by DE* (delta E). In the L*a*b* color space, DE is written as DE*ab and is computed as follows (For DE*uv, a* and b* are substituted by u* and v*): 1

2 2 2 ∆E *ab = ( L *1 − L *2 ) + ( a *1 − a *2 ) + ( b *1 −b *2 )  2  

(7.8)

The color difference is a typical measure to evaluate color quality against a target color. Although the CIE 1976 L*a*b* color space and the CIE 1976 L*u*v* color space are geometrically different, it is thought that a color difference value of two to three in these color spaces may be a target color difference when the original and the reproduced pictures are observed side by side. Because DE*ab and DE*uv are

Copyright © 2006 Taylor & Francis Group, LLC

211

Color Theory and Its Application to Digital Still Cameras

calculated by simple formulae, the color differences are not homogenous. For more accuracy, DE*94, DE*2000, and other formulae are recommended.3,4

7.1.5 LIGHT SOURCE

AND

COLOR TEMPERATURE

Chromaticity of light sources observed in life can be approximately plotted along the black body locus as shown in Figure 7.4. However, it is necessary to pay attention to the fact that the actual spectral distributions of light sources with the same chromaticity are not similar. The spectral distributions of black body sources, several natural light sources, and artificial lamps are shown in Figure 7.5 and Figure 7.6. 0.6 Standard white in photography and graphic arts

0.5 v⬘

HDTV TV in Japan

F5

Blue sky

Incandescent 3,000 K

2,000 K 1,500 K

1,000 K

F6 4,000 K 5,000 K 6,000 K 8,000 K 10,000 K

Daylight Blackbody Fluorescent Sky

30,000 K

0.4

50,000 K

0.3 0.1

0.2

0.3 u⬘

0.4

0.5

FIGURE 7.4 (See Color Figure 7.4 following page 178.) Light sources on the u¢v¢ chromaticity diagram. 3

30,000K

Relative power

2.5 2

10,000K 9,000K 1.5 8,000K 7,000K

1 6,000K 5,000K

0.5 4,000K 3,000K

0 2,000K 350 400

450

500 550 600 Wavelength (nm)

650

700

750

FIGURE 7.5 (See Color Figure 7.5 following page 178.) Spectral distribution of black body.

Copyright © 2006 Taylor & Francis Group, LLC

212

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 7.6 (See Color Figure 7.6 following page 178.) Spectral distribution of CIE standard illuminants (left) and fluorescent lamps (right).

Human beings have been evolving under natural light sources, which are close to the black body locus in chromaticities and spectral distributions; thus, the human visual system gives natural color recognition under these light sources. On the other hand, artificial light sources such as fluorescent lamps do not have similar characteristics to black body light sources as shown in Figure 7.6. Nevertheless, the colors of objects under such artificial light sources would be recognized as natural because most objects do not have steep spectral changes.5 As tristimulus values are calculated with different types of such light sources having the same chromaticity, there would be no significant differences. When a color is carefully observed, however, color changes are sometimes noticed. For example, under a low-cost fluorescent lamp, skin color might appear dark and yellowish. This is caused by the spectral difference between artificial and natural light sources, and this deficiency can be evaluated by the CIE 13.3 color rendering index.6 Table 7.1 depicts the indices of typical light sources. Ra indicates an index for averaged color, and Rn indicates an index for specific color. Some types of fluorescent lamps with a low Ra index (average color rendering index) tend to cause the previously mentioned phenomenon. For instance, illuminant F2 produces a low index value for skin color at R15.7 The result is that object colors cannot be observed satisfactorily under such a light source, and neither can a digital still camera, in general. Correlated color temperature is an index to represent the chromaticity of light sources based on the fact that the chromaticities of most light sources lie along the black body locus. A residual small difference from the black body locus is approximated to be the nearest point in terms of color difference. Therefore, if two different light sources have the identical correlated temperature, the color of an object under each light source will not be identical in terms of spectral distribution and chromaticity. Because the small difference of chromaticity can be compensated by visual adaptation and may be neglected, it is suggested that the color rendering index be noted along with color temperature to avoid such misunderstanding.

7.2 CAMERA SPECTRAL SENSITIVITY The simplest target color of a camera system is the scene. (Targeting color will be discussed later.) In order to realize this, the camera sensitivity curves must be a linear transformation of the color-matching functions; otherwise, two objects

Copyright © 2006 Taylor & Francis Group, LLC

213

Color Theory and Its Application to Digital Still Cameras

TABLE 7.1 Color Rendering Index for Some Light Sources Sample light source Chromaticity

x y Reference light source (P: black body; D: daylight) Correlated color temperature Average color rendering Ra index Special color rendering R1 7.5 R 6/4 index R2 5 Y 6/4 R3 5 GY 6/8 R4 2.5 G 6/6 R5 10 BG 6/4 R6 5 PB 6/8 R7 2.5 P 6/8 R8 10 P 6/8 R9 4.5 R 4/13 R10 5 Y 8/10 R11 4.5 G 5/8 R12 3 PB 3/11 R13 5 YR 8/4 R14 5 GY 4/4 R15 1 YR 6/4

F2 F7 F11 A D65 D50 D55 0.3721 0.3129 0.3805 0.4476 0.3127 0.3457 0.3324 0.3751 0.3292 0.3769 0.4074 0.3290 0.3585 0.3474 P D P P D D D 4200 64

6500 90

4000 83

2856 100

6500 100

5000 100

5500 100

56

89

98

100

100

100

100

77 90 57 59 67 74 33 –84 45 46 54 60 94 47

92 91 91 90 89 93 87 61 78 89 87 90 94 88

93 50 88 87 77 89 79 25 47 72 53 97 67 96

100 100 100 100 100 100 100 100 100 100 100 100 100 100

100 100 100 100 100 100 100 100 100 100 100 100 100 100

100 100 100 100 100 100 100 100 100 100 100 100 100 100

100 100 100 100 100 100 100 100 100 100 100 100 100 100

observed identically by human eyes will result in different color signal outputs from the same camera system. This phenomenon is called sensitivity (or observer) metamerism. Unless the characteristics of the objects are known in priority, it is impossible to exactly estimate the tristimulus values of the original scene in such a case. This criterion of camera sensitivity is called the Luther condition.8 In practice, however, it is difficult to adjust a set of spectral sensitivity curves to conform to the Luther condition due to the practical production of filters, sensors, and optical lenses. Real reflective objects have limited characteristics; the spectral reflectance of an object in general does not change steeply with respect to wavelength. This characteristic allows cameras to estimate tristimulus values for the object even though they do not satisfy the Luther condition. Theoretically speaking, as long as the spectral reflections of an object are always composed of three principal components, a threechannel camera having any three types of sensitivity curves can exactly estimate the tristimulus values of the object. (Images observed on TV displays have such characteristics.) For the evaluation of sensitivity metamerism, the DSC/SMI (digital still camera/sensitivity metamerism index) was proposed in ISO/CD 17321-1. Because it takes into account the spectral reflectance of normal objects, the index correlates well with subjective tests.9 Unfortunately, at the time of this writing, the document is not publicly available. It is expected to be published in 2006.

Copyright © 2006 Taylor & Francis Group, LLC

214

Image Sensors and Signal Processing for Digital Still Cameras

7.3 CHARACTERIZATION OF A CAMERA A typical methodology to characterize a camera with a linear matrix colorimetrically is to use test patches whose spectral responses are similar to those of real objects. Suppose that the target tristimulus values for color targets are given by:  X1  T =  Y1  Z1 

⋯

Xi

⋯

⋯

Yi

⋯

⋯

Zi

⋯

Xn   Yn  Z n 

(7.9)

and the estimated tristimulus values are given by:  Xˆ1  Tˆ =  Yˆ1 ˆ  Z1

⋯

⋯

Xˆ i Yˆi

⋯

Zˆ i

⋯

⋯

⋯

Xˆ n   a11   Yˆn  =  a21  Zˆ n   a31

a13   r1   a23  ⋅  g1 a33   b1

a12 a22 a32

ri ⋯ gi ⋯ bi ⋯

rn   ⋯ gn  = A ⋅ S ⋯ bn  ⋯

(7.10) where matrix S is measurement data through the camera. To obtain 3 ¥ 3 matrix A, simple linear optimization or recursive nonlinear optimization can be applied. The simple linear solution can be calculated by: A = T ⋅ S T ⋅ ( S ⋅ S T )−1

(7.11)

However, the resultant approximation tends to yield large visual errors in dark areas due to the nature of optimization in the linear domain and to visual characteristics; cubic roots are approximately proportional to one’s recognition, as seen in Equation 7.4 and Equation 7.5. An alternate method minimizes the total visual color difference, J, using a recursive conversion technique. Delta E may be calculated in a CIE uniform color space described earlier. n

J=

∑ w ∆E(X , Y , Z , Xˆ , Yˆ , Zˆ ) i

i

i

i

i

i

i

(7.12)

i =1

where wi is a weight coefficient for each color patch. Matrix A may be optimized mainly for important colors. It should be noted that any minimization techniques converge to the identical result as long as the Luther condition is satisfied. In practice, preparing a test chart is most problematic. A typical misconception is to use a test chart made by a printer that uses three or four colorants. When the

Copyright © 2006 Taylor & Francis Group, LLC

Color Theory and Its Application to Digital Still Cameras

215

∆

FIGURE 7.7 (See Color Figure 7.7 following page 178.) Characterization by linear matrix.

test chart is produced by such a device, the principal components of the spectral reflectances of the color patches will be limited to the characteristics of the printer, which are not the appropriate representation of a real scene. Figure 7.7 illustrates the color characterization using the Gretag Macbeth color checker, which is designed to simulate the spectral reflectances of real objects.10 The reflectances of most reflective objects can be accurately composed of five to ten principal components. Thus, the set of color patches should cover these characteristics but does not necessarily have many color patches. It should be noted that, because the resultant characterization is suitable for reflective objects and not for self-emitting light sources such as a neon bulb, color LEDs, and color phosphors, there is room for improvement.

7.4 WHITE BALANCE One of the most challenging processes in a digital camera is to find an appropriate white point and to adjust color. As described in Section 7.1.5, real scenes contain many light sources. In such a situation, the human visual system adapts to the circumstances and recognizes the objects as if they were observed in typical lighting conditions, while a camera’s sensor still outputs raw signals. For instance, they recognize white paper as white in the shadow of a clear sky even though its tristimulus values give a bluish color because the paper is illuminated by sky blue. It is known that the major adjustment is performed in the retina by adjusting each cone’s sensitivity. This process is called chromatic adaptation.

7.4.1 WHITE POINT A digital camera needs to know the adapted white that the human vision system determines to be achromatic on site. Primarily, three types of approaches are used to estimate the white point, or chromaticity, of a light source in a camera.

Copyright © 2006 Taylor & Francis Group, LLC

216

Image Sensors and Signal Processing for Digital Still Cameras

•

•

•

Average of the scene. The first approach assumes that the average color of the entire scene is a middle gray, typically at 18% reflectance. This approach was traditionally used in film-processing labs to adjust the color balance of negative film for prints. Many movie cameras for consumers do not accumulate color over just one image, but rather over the entire sequence of images spanning a couple of minutes in the recent past. When the averaged scene is supposed to be achromatic, the resultant color should be that of the light source. Brightest white. The second approach presumes that the brightest color is white. Because the light source should be brightest in the scene by its nature, brighter objects are expected to encompass more characteristics of the light source than others. Typically, the brightest point is presumed to be the same color as the light source. However, the real scene may have self-emitting objects such as traffic signals, so the wrong light source may be estimated by mistake. To reduce the chance of this happening, only bright colors with chromaticity near the black body locus may be chosen; the other colors, even though brighter, will be eliminated from consideration. Color gamut of the scene. The last approach is to estimate the light source from the distribution of colors captured by a camera. This theory supposes that the scene has several color objects that, when illuminated by a light source, are supposed to cover all of the objects’ spectral distributions statistically, thus producing the observed color gamut of the scene. That is, the light source would be estimated by comparing the correlations between the color distribution of a captured image and the entries in a color gamut database built from expected scene spectral reflectances and typical light sources.

A practical algorithm will be constructed with a mixture of these approaches along with other statistical information. Optimization should be performed for typical scenes and typical lighting conditions encountered by typical users of the camera system.

7.4.2 COLOR CONVERSION Once the white point is found, the next step is to convert all colors into the desired ones; this includes transforming the estimated white to be achromatic. Two policies are typically used: chromatic adaptation and color constancy. 7.4.2.1 Chromatic Adaptation Chromatic adaptation in the eye is mostly controlled by the sensitivity of the cones. Analogously, camera signals can be approximated into the tristimulus values of the cones, and the camera can control the gain of these signals. The typical calculation is performed as follows:

Copyright © 2006 Taylor & Francis Group, LLC

Color Theory and Its Application to Digital Still Cameras

 LW′ L  W  r′    −1 −1   g′  = A B ⋅  0   b ′    0

0 M W′ MW 0

 0   r     0  ⋅ B ⋅ A ⋅ g   b  SW′  SW 

217

(7.13)

where r  L     M  = B ⋅ A ⋅ g  b   S 

(7.14)

and r, g, and b are original camera signals. LW , MW , SW , and L¢W , M¢W , L¢W denote cone tristimulus values of white points at the original scene and at the RGB color space, respectively. This adaptation is called the von Kries model. Matrix A may be calculated by Equation 7.11. Matrix B is used to transform CIE tristimulus values into cone responses. An example of matrix B is shown as follows:11  L   0.4002     M  =  −0.2263  S   0. 7.4.2.2

0.7076 1.116532 0.

−0.08081  X     0.04570  ⋅ Y  0.91822   Z 

(7.15)

Color Constancy

It is known that the color of an object under a variety of light sources is recognized as if it were under day light as long as the Ra of the light source is high,12 although no information about the spectral distribution of the light sources and the spectral reflections of the objects is available. This is called color constancy. A simple linear approach is to use Equation 7.13 with matrix B optimized in terms of color constancy; this would estimate the corresponding color under a standard light source. It is known that the resulting equivalent sensitivity curve will be negative in places. Figure 7.8 illustrates this characteristic. In a nonlinear approach, more than one matrix optimized for each light source may be prepared. The camera could choose one of them, depending upon the type of light source. However, this approach increases the risk that an incorrect matrix is chosen.

Copyright © 2006 Taylor & Francis Group, LLC

218

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 7.8 (See Color Figure 7.8 following page 178.) Equivalent color sensitivity in white balancing optimized for chromatic adaptation and color constancy.

7.5 CONVERSION FOR DISPLAY (COLOR MANAGEMENT) Because the digital values of image data explicitly do not contain color definition, it is necessary that the image data be interpreted by a receiver without any ambiguity. Therefore, the relationship between color signal values (digital count) and their physical meanings (i.e., colorimetry) should be well defined. A typical approach to color management in digital cameras is to use a standard color space scheme. Two key points should be considered: colorimetric definition and its intent.

7.5.1 COLORIMETRIC DEFINITION Image data from a digital camera may be sent to a display without any conversions. This means that the color of RGB data is defined based on the color appearance on a display. Therefore, the color space based on a specific display is used as the standard color space. The most popular standard color encoding is sRGB (standard RGB), which is defined for an average CRT with a certain observing condition. The following is the definition of sRGB for an 8-bit system.13 RsRGB ′ = R8 bit ÷ 255 GsRGB ′ = G8 bit ÷ 255

7.16

BsRGB ′ = B8 bit ÷ 255 RsRGB ′ , GsRGB ′ , BsRGB ′ ≤ 0.04045 RsRGB = RsRGB ′ ÷ 12.92 GsRGB = GsRGB ′ ÷ 12.92 BsRGB = BsRGB ′ ÷ 12.92

Copyright © 2006 Taylor & Francis Group, LLC

(7.17)

Color Theory and Its Application to Digital Still Cameras

219

RsRGB ′ , GsRGB ′ , BsRGB ′ ≥ 0.04045 RsRGB = [( RsRGB ′ + 0.055 ) / 1.055 ]2.4 GsRGB = [(GsR ′ GB + 0.055 ) / 1.055 ]2.4

(7.18)

BsRGB = [( BsRGB ′ + 0.055 ) / 1.055 ]2.4  X   0.4124    Y  =  0.2126  Z   0.0193

0.3576 0.7152 0.1192

0.1805   RsRGB    0.0722  GsRRGB  0.9505   BsRGB 

(7.19)

The white point of sRGB is the chromaticity of D65, and the primaries of sRGB are the ones of HDTV (high-definition television). Because negative values are not allowed and the maximum value is limited, the colors represented by the standard are limited to the color gamut of the virtual display. To overcome the gamut problem, two more color spaces are often used: sYCC and Adobe RGB. (In the specification, Adobe RGB is formally defined as the “DCF optional color space.”) The sYCC14 color space is used for image compression to improve the efficiency for better compression ratio at equivalent image quality. Because the sYCC is a superset of sRGB, it can represent more chromatic colors. The Adobe RGB color space has a wider color gamut compared with sRGB by about 15% in the u'v' chromaticity diagram. For further information, please refer to Exif v2.2115,16 and DCF.17

7.5.2 IMAGE STATE The important concept to clarify in digitization of images is the image state. Because sRGB defines only the observing condition including the display, there is no guideline for the kind of color image that should be encoded into sRGB color space, no matter how beautiful or dirty the colors appear. At first sight, readers may think that the tristimulus values or the white balanced tristimulus values of the real scene are the good target. However, most users desire pleasing colors rather than correctly reproduced colors. When the color is intended to reproduce the color of the real scene, the data are called scene-referred image data. In all other cases, for example, when the color is intended to adjust for the user’s memory or preferred colors, they are called outputreferred image data. ISO 22028-1 defines this concept.18 Figure 7.9 illustrates the flow of signals and their image state in a digital camera. The process of converting scene-referred image data to output-referred image data is called color rendering. TV standards use scene-referred image data, although the term was not defined at the time that these standards were published. This kind of color image can be seen on display monitors in TV broadcasting studios. Once the image is transmitted to a home, the image is seen with some color adjustments

Copyright © 2006 Taylor & Francis Group, LLC

220

Image Sensors and Signal Processing for Digital Still Cameras Sensor signal

Reproduction of scene color Scene-referred image data

The order is arbitrary

White balancing

Preferred reproduction

Output-referred image data

Conversion to standard color space

Recording

FIGURE 7.9 Conceptual flow of color image processing in a digital camera.

— typically, increasing contrast and chroma, manually by the consumer. Thus, TV is enjoyed with preferred colors. Digital cameras using sRGB should calculate output-referred colors internally and encode them into image files. Thus, the user enjoys preferred colors on monitor displays without any further adjustment. It is obvious that the preferred colors are not unique. The desired result can be influenced by cultural background such as region, race, profession, and age. The method of color rendering is ill defined, and further study is encouraged to determine the best conversion — perhaps, for specific users.

7.5.3 PROFILE APPROACH Another way to control color is to use a profile to establish the relationship between digital data and colorimetric values. The ICC (International Color Consortium)19 offers the specification of a profile format. This approach is especially effective when it is used for the conversion of RAW data output by the camera because each camera has its own sensor characteristics that, in turn, determine the color characteristics as well.

7.6 SUMMARY In this chapter, the basics of color theory and the concepts necessary for the design of digital cameras were described. Colorimetry is the key theory to quantify color. In order to reproduce colors as our eyes see them, the Luther condition must be considered. For the exact reproduction of a scene under a standard light source, there is a systematic way to characterize a camera. For nonstandard light sources, a white point must be found and color should be converted accordingly. Many camera users wish to see preferred colors rather than true colors; in order to generate preferred colors, we still need empirical adjustments and studies of users’ preferences. Finally,

Copyright © 2006 Taylor & Francis Group, LLC

Color Theory and Its Application to Digital Still Cameras

221

for the transmission of color image data captured by a digital camera, standard color spaces like sRGB are typically used. In conventional cameras, these kinds of processing were done by silver halide film and the photofinishing process. A digital camera must handle all of the processing inside. The knowledge of color described in this chapter is essential to improving the image quality of digital cameras.

REFERENCES 1. Publication CIE 15.2-1986, Colorimetry, 2nd ed., 1986. 2. Publication CIE 159: 2004 A Colour Appearance Model for Colour Management Systems: CIECAM02, 2004. 3. CIE 116-1995, Industrial colour-difference evaluation, 1995. 4. M.R. Luo, G. Cui, and B. Rigg, The development of the CIE 2000 colour difference formula, Color Res. Appl., 25, 282–290, 2002. 5. J. Tajima, H. Haneishi, N. Ojima, and M. Tsukada, Representative data selection for standard object colour spectra database (SOCS), IS&T/SID 11th Color Imaging Conf., 155–160, 2002. 6. CIE 13.3-1995, Method of measuring and specifying colour rendering properties of light sources, 1995. 7. JIS Z 8726: 1990, Method of specifying colour rendering properties of light sources, 1990. 8. R. Luther, Aus dem Gebiet der Farbreizmetrik, Zeitschrift fur Technische Physik, 12, 540–558, 1927. 9. P.-C. Hung, Sensitivity metamerism index for digital still camera, Color Sci. Imaging Technol., Proc. SPIE, 4922, 1–14, 2002. 10. C.S. McCamy, H. Marcus, and J.G. Davidson, A color-rendition chart, J. Appl. Photogr. Eng., 2, 3, 95–99, 1976. 11. Y. Nayatani, K. Hashimoto, K. Takahama, and H. Sobagaki, A nonlinear colorappearance model using Estevez–Hunt–Pointer primaries, Color Res. Appl., 12, 5, 231–242, 1987. 12. P.-C. Hung, Camera sensitivity evaluation and primary optimization considering color constancy, IS&T/SID 11th Color Imaging Conf., 127–132, 2002. 13. IEC 61966-2-1 Multimedia systems and equipment – Colour measurement and management – Part 2.1: Colour management – Default RGB colour space – SRGB, 1999. 14. IEC 61966-2-1 Amendment 1, 2003. 15. JEITA CP-3451, Exchangeable image file format for digital still cameras: Exif Version 2.2, 2002. 16. JEITA CP-3451-1 Exchangeable image file format for digital still cameras: Exif Version 2.21 (Amendment Ver 2.2), 2003. 17. JEITA CP-3461, Design rule for camera file system: DCF Version 2.0, 2003. 18. ISO 22028-1: 2004, Photography and graphic technology – extended colour encodings for digital image storage, manipulation and interchange – part 1. 19. http://www.color.org, specification ICC. 1:2003-09, file format for color profiles (version 4.1.0).

Copyright © 2006 Taylor & Francis Group, LLC

8

Image-Processing Algorithms Kazuhiro Sato

CONTENTS 8.1

8.2

8.3

Basic Image-Processing Algorithms............................................................224 8.1.1 Noise Reduction...............................................................................225 8.1.1.1 Offset Noise ......................................................................225 8.1.1.2 Pattern Noise.....................................................................226 8.1.1.3 Aliasing Noise ..................................................................226 8.1.2 Color Interpolation...........................................................................226 8.1.2.1 Rectangular Grid Sampling ..............................................228 8.1.2.2 Quincunx Grid Sampling..................................................229 8.1.2.3 Color Interpolation............................................................229 8.1.3 Color Correction ..............................................................................232 8.1.3.1 RGB ..................................................................................232 8.1.3.2 YCbCr ...............................................................................232 8.1.4 Tone Curve/Gamma Curve ..............................................................233 8.1.5 Filter Operation ................................................................................235 8.1.5.1 FIR and IIR Filters ...........................................................236 8.1.5.2 Unsharp Mask Filter .........................................................237 Camera Control Algorithm ..........................................................................238 8.2.1 Auto Exposure, Auto White Balance ..............................................238 8.2.2 Auto Focus .......................................................................................239 8.2.2.1 Principles of Focus Measurement Methods .....................239 8.2.2.2 Digital Integration.............................................................240 8.2.3 Viewfinder and Video Mode ............................................................241 8.2.4 Data Compression ............................................................................242 8.2.5 Data Storage.....................................................................................243 8.2.6 Zoom, Resize, Clipping of Image ...................................................243 8.2.6.1 Electronic Zoom ...............................................................243 8.2.6.2 Resize................................................................................246 8.2.6.3 Clipping, Cropping ...........................................................247 Advanced Image Processing; How to Obtain Improved Image Quality ....248 8.3.1 Chroma Clipping..............................................................................248 8.3.2 Advanced Color Interpolation..........................................................248

223

Copyright © 2006 Taylor & Francis Group, LLC

224

Image Sensors and Signal Processing for Digital Still Cameras

8.3.3 Lens Distortion Correction ..............................................................251 8.3.4 Lens Shading Correction .................................................................251 References..............................................................................................................252 In this chapter, we will describe image-processing algorithms used in digital still cameras (DSC) from a theoretical point of view, as well as the many peripheral functions needed to work in a real camera, such as viewfinder, focus and exposure control, JPEG compression, and storage media. This chapter includes simple descriptions to give the reader an outline of these items. The physical aspects of CCD (charge coupled device) and CMOS (complementary metal-oxide semiconductor) sensors and color space are not included here; they are described in Chapter 3 through Chapter 6. Because digital video cameras (DVCs) use similar image-processing technologies and functions, the majority of this chapter is relevant to them also. However, DVCs use other image-processing technology based on the time axis, such as noise reduction in time domain processing. This chapter does not include DVCspecific technologies that are out of the scope of this book.

8.1 BASIC IMAGE-PROCESSING ALGORITHMS DSCs or DVCs acquire data using a CCD or CMOS sensor, reconstruct the image, compress it, and store it on a storage medium (such as a compact flash) in less than a second. In recent years, the number of pixels of the image sensor for DSCs has become large. Also, many kinds of image-processing algorithms are used in a DSC, such as color interpolation; white balance; tone (gamma) conversion; false color suppression; color noise reduction; electrical zoom; and image compression. Figure 8.1 shows a configuration of a DSC from the viewpoint of the image-processing technology used in it. The figure is a simplified view intended to show how imageprocessing algorithms are applied in the camera. The details of the actual imageprocessing hardware/software are described in Chapter 9. The lens focuses incident light on a CCD sensor to form an image, which is converted to an analog signal. This signal is read out, digitized, and sent to the digital signal-processing block. A CMOS sensor may output an analog signal, in which case it is treated the same way as the output from a CCD sensor. Alternatively, a CMOS sensor may output a digital signal that can be fed to the digital signal-

FIGURE 8.1 Digital still camera and its typical configurations.

Copyright © 2006 Taylor & Francis Group, LLC

Image-Processing Algorithms

225

processing block directly. There is no difference between the two types of sensor from the point view of the image-processing algorithms. A complete working camera has several other important blocks, including: focus; iris and optical zoom controls; optical finder; LCD display; and storage media. Usually, these are controlled by a general-purpose MPU in the camera. The digital signal-processing block of a DSC must execute all the imageprocessing algorithms on over 5 Mpixels of data and construct the final image. Recent CCD sensors generate three to five full frames of data within a second. Also, recent high-speed CMOS sensors can generate more than ten frames per second. If images are to be generated with same output speed of sensor signal data, the required processing speed for the image processor becomes over 50 Mpixels/sec. The two possible approaches for the signal processor for the DSC to achieve the speed are: (1) to base the design on a general-purpose DSP (digital signal processor) with image-processing software; or (2) to construct hard-wired logic to achieve maximum performance. Each approach has certain advantages and disadvantages, as discussed in Chapter 9.

8.1.1 NOISE REDUCTION The number of photons striking the sensor is linearly proportional to the incident light intensity. The number of electrons displaced, and thus the output signal, is proportional to the number of photons. The output signal level from a sensor is in the order of 1 V. It is very important to provide the digital image-processing hardware with the best quality input signal to get the best processed image from measured data. Thus, it is necessary to pay attention to keeping signal accuracy and quality of the input data. 8.1.1.1 Offset Noise It is best to digitize the input data value between black and maximum levels so as to utilize the dynamic range of a sensor fully. Most CCDs generate some noise signal like thermal noise even when no input light signal is present. Also, the dark noise level of a CCD drifts gradually from beginning to end of readout. Figure 8.2 is a raw data of dark signal acquired with no incident light to a lens. As can be seen on

FIGURE 8.2 Vertical offset drift of CCD sensor.

Copyright © 2006 Taylor & Francis Group, LLC

226

Image Sensors and Signal Processing for Digital Still Cameras

the figure, there is vertical brightness drift from top to bottom. A noise drift between start and end of signal readout is characteristic of this type of noise, so it is called offset drift noise of a sensor. Subtracting an offset that depends on the vertical position of a pixel can compensate for this noise. The compensation is performed at the early stage of the signal-processing path in a camera. 8.1.1.2 Pattern Noise “Pattern noise” is caused by an imbalance of pixel readout from the sensor. The human eye has very high sensitivity for patterns in a picture. When some feature in a picture is recognized as a pattern or pattern noise, the pattern will stand out very strongly. For example, if a circular pattern (ring) of noise is recognized in a picture, then a solid line circle will be seen or imagined, even if it is only very short broken arcs. To avoid this kind of effect, the pattern noise should be reduced as much as possible. Insufficient accuracy for the data calculation process may generate pattern noise. 8.1.1.3 Aliasing Noise Each cell spatially samples the image projected onto the sensor. The sampling pitch governs this sampling characteristic. It is defined by cell pitch, the spatial arrangement of the entire cell, and the aperture function of each cell. In many cases, the MTF (modulation transfer function) of a lens has higher response than a sensor, so the output of the sensor will contain alias signals over the Nyquist frequency limit defined by the cell pitch. Once this type of alias signal is mixed in the raw data, it cannot be eliminated. Many cameras use an optical low-pass filter (OLF) between the lens and sensor to limit the spatial frequency of the input image focused on a sensor. Table 8.1 summarizes the types of noise that are generated inside a DSC. It is important to reduce these noise types at an early stage of data acquisition; nevertheless, they will remain in the raw data. It is necessary to eliminate or reduce this noise in the image processing using adaptive image-processing technology.

8.1.2 COLOR INTERPOLATION The full-color digital image is displayed on a soft copy device such as a CRT monitor so that each pixel has a digital information of red (R), green (G), and blue (B) on a rectangular coordinate grid. In the case of a DSC with three sensors, each sensor has different color filters and does not need color interpolation operation because R, G, and B values for each position are measured by each sensor. However, in cameras with three CCD sensors, there is a method called half-pitch shift of G in reference with R and B sensors in horizontal. This is not used widely for DSC because highly accurate alignment of three sensors is very costly. Some professional DVCs use this type of three-sensor design. The half-pitch shift requires interpolation in the middle of each position of R, G, and B in the horizontal direction to double the horizontal resolution. After operation of color correction and other image-processing operations, the horizontal pixel size is reduced by half to get the correct aspect ratio. This chapter will focus on single-sensor systems because the major

Copyright © 2006 Taylor & Francis Group, LLC

227

Image-Processing Algorithms

TABLE 8.1 Noise and its Causes Noise type Cross color noise

False color noise

Color phase noise

Digital artifacts

Temporal noise

Fixed pattern noise Line noise

Root cause Caused by mixing of adjacent color pixels signals; typical for single CCD sensor Generated when using a digital white balance, which provides insufficient resolution in shadow areas Shows up as color blotches in dark gray areas where there was no color; can also show up as a shift in the original color Multiple causes; in an AFE, can be caused by insufficient A/D resolution, which causes step to be seen in smooth tonal areas Caused by shot noise in the CCD or insufficient SNR in the AFE Caused by a measurement mismatch in the sampling of alternate pixels Shows up as horizontal streaks in dark areas of the image; caused by the black level calibration using adjustment steps that are too large and, therefore, visible

Solutions Calibrate for each pixel

Use more bit length or analog white balance Higher resolution (e.g., 16-b) ADC and analog white balance so as to provide high resolution in dark areas of the picture Higher resolution (e.g., 16-b) ADC to provide smooth tonal response and provide an effective 42 to 48 b of color depth Use a low-noise CDS, PGA, ADC, and black level calibration so as not to add any noise Balanced CDS Use an ultraprecision (e.g., 16-b) digital black level calibration circuit

difference is the color interpolation block only and the rest of the processing is common for single- and three-sensor systems. In a single-sensor system, each cell on the sensor has a specific color filter and microlens positioned above it. The array of color filters is referred to as a “color filter array” (CFA). Each cell has only information for the single color of its color filter. The raw data obtained from this sensor do not have full R/G/B information at each cell position. Color interpolation is required when a full-color image is wanted from a single-sensor system. There are several color filter sets and filter arrays for DSC and DVC sensors. Figure 8.3 shows three sets and their spatial arrangements. Figure 8.3(a) is called “Bayer pattern color filter array” and widely used for DSCs. The ratio among R, B, and G is 1:1:2. Figure 8.3(b) is called “complementary color filter pattern” and is also used in DSCs. The advantage of this color filter is better light sensitivity than an R/G/B color filter. Figure 8.3(c) is another complementary color filter pattern and is mostly used in DVCs. It is similar to the layout (b) except that the G and Mg are changed at each line. More details for sensor, color filter array, and microlens can be found in Chapter 3.

Copyright © 2006 Taylor & Francis Group, LLC

228

Image Sensors and Signal Processing for Digital Still Cameras

(a)

(b)

(c)

FIGURE 8.3 Sensor color filter array for DSC and DVC. 1

2

2 B

G

R

G

R

G

Reconstruction area

2 G

B

G

Green sampling

B Red/Blue sampling

G

R

G

R

G

B

G

B

G

B

G

R

G

R

G

(a)

(b)

FIGURE 8.4 Rectangular grid sampling and frequency coverage for Bayer CFA.

From the viewpoint of spatial sampling geometry used for actual sensors, two types of cell arrangements are realized in CCD sensors: rectangular and quincunx or diamond-shaped sampling. The basic concepts of color interpolation theory are the same for all the color filter arrays in Figure 8.3, although actual color interpolations for the various color filter spatial and sampling cell layouts differ. In this book, rectangular sampling patterns are mainly considered when describing color interpolation algorithms for the Bayer pattern color filter array. 8.1.2.1 Rectangular Grid Sampling Figure 8.4 shows the normalized geometry and dimensions of the Bayer rectangular color filter array and its frequency coverage. There are several ways to determine which color starts from the upper-left corner position, but the grid dimensions and color pattern cycle are the same. In part (a), the distance among vertical, horizontal, and diagonal cells is normalized with minimum distance as 1.0. In the rectangular grid, the G sampling distances for horizontal and vertical directions have minimum value so that the highest Nyquist frequency is in the horizontal and vertical directions. Compared with the G signal, R and B sampling distances are highest in the diagonal directions, but their sampling distances are 2 = 1.4 . This means that the Nyquist frequency for R and B sampling

Copyright © 2006 Taylor & Francis Group, LLC

229

Image-Processing Algorithms

is 1 / 2 ≈ 0.7 times less than G. R and B cover the same frequency range. Figure 8.4(b) shows frequency coverage of R, G, and B, respectively. Also, the maximum frequency range that can be reconstructed from Bayer filter with rectangular sampling is indicated by a broken line that surrounds the G and R/B solid lines. 8.1.2.2 Quincunx Grid Sampling Figure 8.5 shows the geometry and dimensions of the alternative spatial sampling that uses quincunx or diamond grid location and frequency coverage. The geometry is the 45∞ rotated layout of a rectangular grid. As before, the dimensions are expressed with respect to the minimum sampling distance of the G signal. In the quincunx grid sampling, sampling frequency is highest in the diagonal direction (45 and 135∞). R and B signals have the same sampling interval with a distance of 2 ≈ 1.4 times larger than the G sampling distance. Based on this geometry, Figure 8.5(b) shows frequency coverage of R, B, and G and the reconstruction area for quincunx grid sampling. It is possible to reconstruct a larger reconstruction area than those described in Figure 8.4 and Figure 8.5; however, the resulting image has less information. This type of expansion can be considered as a magnification operation on the raw data. 8.1.2.3 Color Interpolation Color interpolation is the process of estimating a value at a location that does not have a measured value by using other actual measured values. It is clear that better estimation can be obtained if more measured pixels are used for the interpolation calculation. However, in an actual implementation for a DSC, the trade-off among hardware cost, calculation speed, and image quality must be considered. 1/ 2

2 G

G

Reconstruction area

G

1

1 R

B

R

G

G

G 2

B

B

R G

G

Green sampling Red/Blue sampling

G

R

B

(a)

R

(b)

FIGURE 8.5 Quincunx grid sampling and frequency coverage for Bayer CFA.

Copyright © 2006 Taylor & Francis Group, LLC

230

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 8.6 Bayer CFA color interpolation method.

Figure 8.6 shows the basic concept of color interpolation for a rectangular grid having Bayer CFA. The interpolation operation is the equivalent of inserting zero points between each two actual existing points and then applying a low-pass filter. In Figure 8.6(a), P(x,y) = Gi has a measured value of green color only, so it is necessary to calculate the nonexisting values, R and B, from the measured values of R and B points that surround P(x,y). Better estimation can be obtained if more points are used for calculation. An area shown by the broken line that has a center at the interpolation point in Figure 8.6 is defined. The number of points contained within this area affects the quality of the reconstructed image. Figure 8.6(b) is the color interpolation point that has a measured value for B but no values for G and R; thus they must be interpolated from other G and R values. The simplest interpolation method is a nearest neighbor or zero order interpolation. This is based on a very simple algorithm that the value of an interpolated point will take the same value as the nearest measured point. There are no multiplication/addition arithmetic operations for pixel data that are used for interpolation, but the image quality is not good enough for a commercial camera. The next simplest interpolation algorithm is a linear interpolation (in one dimension) or bilinear interpolation (in two dimensions). In the linear interpolation algorithm, the middle point takes the arithmetic mean of the two values of the adjacent points. Bilinear interpolation is expressed as a serial calculation of linear interpolation for vertical and horizontal directions. It does not depend on the order of calculation. In the bilinear color interpolation shown in Figure 8.6(a), R and B values at the location of P(x,y) are calculated with the equation R = (R1 + R2)/2 and B = (B1 + B2)/2. This is equivalent to one-dimensional interpolation (linear interpolation). The frequency response of the linear interpolation of this equation has a response as shown in Figure 8.7. This is the vertical response for R pixel value and horizontal response for B pixel value interpolation. There is no low-pass effect for the orthogonal direction of each pixel. If the interpolation location P(x,y) is moved to the right one position to B2 as shown in Figure 8.6(b), the geometric relationship of the colors changes and the

Copyright © 2006 Taylor & Francis Group, LLC

231

Image-Processing Algorithms

FIGURE 8.7 Frequency response of linear interpolation.

frequency response is interchanged between R and B. If focus is on the G value, P(x,y) in (b) has no measured value for G, so the G value must be interpolated using the four G values located at the diagonal positions. The bilinear interpolation operation described here is simple and gives better image quality than the nearest neighbor method. However, it still generates cyclic pattern noise because of the cyclic change of direction of the interpolation filter and also the change in its frequency response. To avoid pattern noise caused by them, it is better to use more data points for the interpolation calculation. Also, the adaptive interpolation (described in a later section) gives better image quality. The interpolation algorithm is explained in more detail in Section 8.2.6.1 and Section 8.3.2. The color interpolation method, which reconstructs a full color image from the raw data, has been briefly outlined in this section. In the color interpolation calculation, it is assumed that no aliasing noise is present in the raw data. As mentioned in Section 8.1.1.3 on aliasing noise, many DSCs have an optical low-pass filter (OLF) between lens and sensor so as to limit the spatial frequency of a real scene to match the Nyquist limit defined by the sensor cell pitch. Figure 8.8 shows the OLF, which is composed of a single or multiple layer of a thin crystal plate. The plate has a characteristic to split input light into two different paths: normal (no) and abnormal light (ne). The distance, S, between two separated light paths is given in Equation 8.1, where d is a thickness of the thin plate and no and ne are diffraction index for normal and abnormal light: S = d⋅

ne2 − no2 2 ⋅ ne ⋅ no

(8.1)

A single plate can split the light in one direction only, so four slices are required to split the light in all four directions. How many OLFs will be used in a DSC depends on a lens design and lens/sensor combination.

Copyright © 2006 Taylor & Francis Group, LLC

232

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 8.8 Optical low-pass filter.

8.1.3 COLOR CORRECTION 8.1.3.1 RGB RGB color correction is used to compensate for cross-color bleeding in the color filter used with the image sensor. It uses a set of matrix coefficients (ai, bi, ci) in the matrix calculation shown in Equation 8.2. However, with only nine elements, the matrix shown is not enough to correct the entire color space. The matrix parameters must satisfy 3

3

3

∑ ∑ ∑c = 1 . ai =

i =1

bi =

i =1

i

i =1

Increasing the main diagonal values results in richer color in the corrected image. The RGB CFA has better color representation characteristics than the complementary color filter array. The latter has the advantage of better light sensitivity (usability) because of less transmitted light loss than RGB. Many DSCs use RGB CFA, but DVC uses complementary CFA.  R'   a1  '  G  =  b1  B'   c1  

a2 b2 c2

a3   R    b3  G  c3   B 

(8.2)

8.1.3.2 YCbCr YCbCr color space is also used in DSCs. The YCbCr and RGB color spaces have a linear relation with transformation Equation 8.3 and Equation 8.4, known as ITU standard D65. Equation 8.5 is a fixed-point number representation of the transformation matrix in Equation 8.3 with 10-b resolution. By converting an image from RGB to YCbCr color space, it becomes possible to separate Y, Cb, and Cr information. Y is luminance data that do not include color information; Cb and Cr are chrominance data that contain color information only. From Equation 8.3, it can be seen that G (green) is the biggest contributor to luminance. This is as a result of the human eye’s

Copyright © 2006 Taylor & Francis Group, LLC

233

Image-Processing Algorithms

higher sensitivity to green than to the other colors, red and blue. Blue is the biggest contributor for Cb and red contributes most to the Cr.  Y   0.2988    Cb  =  −0.1689  Cr   0.5000  R  1    G  = 1  B  1

0.5869 −0.3311 −0.4189

0 −0.3441 1.772

 306   −173  512

601 −339 −429

0.1143   R    0.5000  G  −0.0811  B 

1.402   Y    −0.7141 Cb  0.00015   Cr  117   512  −83

(8.3)

(8.4)

(8.5)

The spatial response of the human eye for chrominance is very low compared to that for luminance signal. From this characteristic, the data rate can be reduced when the chrominance signals are processed. The image compression that will be described later in this chapter uses this characteristic and reduces spatial resolution of chrominance to one half or one fourth of the original data without severe degradation of image quality. The reduced bandwidth signal format is known as 4:2:2 and 4:1:1 for the JPEG standard. The RGB and YCbCr color spaces are used for DSC and DVC, although some other color spaces exist. YCbCr, RGB, and other color spaces are described in detail in Chapter 7.

8.1.4 TONE CURVE/GAMMA CURVE Many imaging devices have nonlinear characteristics when it comes to the process of capturing the light of a scene and transforming it into electronic signals. Many displays, almost all photographic films, and paper printing have nonlinear characteristics. Fortunately, all of these nonlinear devices have a transfer function that is approximated fairly well by a simple power function of the input, of the general form given in Equation 8.6: y = xγ

(8.6)

This is called the tone or gamma curve. The conversion of an image from one gamma curve function to another is called “gamma correction.” In most DSCs, gamma correction is done in the image acquisition part of the signal-processing chain: the intensity of each of the linear R, G, and B components is transformed to a nonlinear signal by an RGB gamma correction function.

Copyright © 2006 Taylor & Francis Group, LLC

234

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 8.9 RGB gamma with table look-up method.

FIGURE 8.10 Y and chroma gamma with table look-up method.

The gamma curve converts each input pixel value to other value with a nonlinear function that modifies the histogram distribution of the source image. The value g (gamma) in the Equation 8.6 lies around 0.45 for most CRT display systems. However, it is not constant for a DSC. Gamma conversion in the DSC is often used in conjunction with bit depth compression such as 12- to 8-b compression of bit depth resolution. Gamma correction also adjusts the noise floor and contrast of the original data near the black level. For this purpose, many DSCs use a table lookup method to get high flexibility and better accuracy of bit depth conversion. There are two implementations of tone curves for DSCs, one in RGB and the other in YCbCr space. Usually, the former is used in DSCs that use an RGB CFA sensor and the latter in DSCs/DVCs that use a complementary CFA sensor. In some applications, RGB gamma uses three channels of gamma curves for each of R, G, and B. Also, YCbCr uses Y, Cb, and Cr gamma curves for the three components. Figure 8.9 shows an example of RGB gamma curves and Figure 8.10 shows Y (luminance) and CbCr (chrominance) gamma curves. RGB and Y curves affect the contrast of the image. The curvature near the origin controls the dark level of the image. Granularity at low-level input changes the contrast and black level of an image greatly.

Copyright © 2006 Taylor & Francis Group, LLC

235

Image-Processing Algorithms

8.1.5 FILTER OPERATION Filtering operations on images can be viewed as modifications in the two-dimensional spatial frequency domain. The low-pass filter functions used in image manipulation should have a smooth impulse response to avoid excess ringing or artifacts on edge boundaries within an image. Actual filter calculations can be performed using arithmetic multiplication in the spatial frequency domain of Fourier transformed filter function and image. Actual Fourier transform uses the FFT (fast Fourier transform) algorithm that needs only of the order of M ◊ N ◊ (log2 M + log2 N) multiplications and additions for a two-dimensional array with the image size of M ¥ N. However, it requires complex hardware and large amounts of memory to store temporary results. The mathematically equivalent operation in the spatial domain is a circular convolution. Using a K1 ¥ K2 matrix of coefficients, called the filter kernel or convolution kernel, the convolution operation takes K1 ¥ K2 ¥ M ¥ N multiplications and accumulations (MAC), where M ¥ N is the image size. Most DSCs use convolution operations because the hardware implementation is easier and a relatively small kernel size can be used. The number of multiplies is directly proportional to K1 ¥ K2 ¥ M ¥ N , which further restricts the choice of kernel size. When a filter is to be designed, it is necessary to specify its features, such as cut-off frequency, roll-off characteristics, stop-band attenuation, and filter taps. Usually, the filter synthesis starts from a one-dimensional filter design, which satisfies a desired response; it is then converted to a two-dimensional filter. The time domain response of a unit impulse input is called its impulse response and its Fourier transform shows the frequency response of a filter. In other words, the impulse response is equivalent to a filter kernel. There are good books that describe the theoretical background and actual synthesis of one- and two-dimensional filters.1,3,4,8,10 Two-dimensional filter synthesis from a one-dimensional filter is classified into two types: separable and nonseparable. Figure 8.11 shows how to construct a separable two-dimensional filter from a one-dimensional one. The two-dimensional filter kernel is constructed from two one-dimensional filters, f1(x,y) and f2(x,y), that have different impulse responses. Equation 8.7 is a serial synthesis that is expressed as f1(x,y) ƒ f2(x,y), where ƒ denotes convolution operation of two filters, f1 and f2. Equation 8.7 means filtered output from the f2 filter is then convoluted by f1 to get the final result, g(x,y): g( x, y) = f1 ( x, y) ⊗ f2 ( x, y) ⊗ forg ( x, y)

(8.7)

Equation 8.8 is a nonseparable filter structure: g( x, y) = f ( x, y) ⊗ forg

(8.8)

In this case, the filter, f, cannot separate into two filters, f1 and f2; therefore, it is necessary to calculate a two-dimensional convolution with the source image and a two-dimensional filter kernel. Note that, in Equation 8.7, the two filters, f1 and

Copyright © 2006 Taylor & Francis Group, LLC

236

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 8.11 Construction of two-dimensional filter from one-dimensional filter.

FIGURE 8.12 Separable and nonseparable low-pass filters.

f2, can be combined into a two-dimensional filter, f3, by convolving them together: f3 = f1 ƒ f2. The resulting filter, f3, is then applied to the input data. Figure 8.12 shows the two-dimensional frequency response of the two different kinds of filter implementation. These two configurations use the same one-dimensional impulse response. Figure 8.12(a) is a serial or separable construction as described in Figure 8.11. Figure 8.12(b) is a nonseparable configuration that was designed directly in the two-dimensional Fourier domain. 8.1.5.1 FIR and IIR Filters Digital filters are classified by their structure as finite impulse response (FIR or nonrecursive) filters and infinite impulse response (IIR or recursive) filters. Figure 8.13 shows the basic structures of these two kinds of filters with time domain representation and also Z-transform notation.3,4,12 The FIR filter is always stable for

Copyright © 2006 Taylor & Francis Group, LLC

Image-Processing Algorithms

237

FIGURE 8.13 FIR and IIR filters

any input data stream and filter parameters because of its nonrecursive structure; however, it needs much longer filter coefficients (taps) than an IIR to achieve the same specification. On the other hand, the IIR has much shorter filter structures as a result of its recursive structure. In image-processing applications, care must be taken not to introduce phase distortion and instability when IIR is used. FIR structure is widely used in the image-processing field; IIR is only used for the image application in which phase distortion can be neglected.3,4,9 8.1.5.2 Unsharp Mask Filter Unsharp mask filter operation is a kind of high-pass filter. It is used widely in DSCs to boost the high-frequency component of the image. Equation 8.9 describes the theoretical background. The basic algorithm is that filtered output, g(x,y), is obtained by multiplying a blurred image by a coefficient, a, and subtracting the result from the original image, forg. Blurring is equivalent to a low-pass filter operation. The blurred image is generated by convolution with a rectangular filter kernel such as 5 ¥ 5, 9 ¥ 9, which has a constant coefficient. A nonrectangular kernel may also be used. The convolution operation is simply a summation of pixels within an area that corresponds to the kernel size. The h in Equation 8.9 is called the “mask” and is equivalent to a simple arithmetic mean. Parameter a controls amplitude of high-frequency emphasis of g(x,y); a is set between 0 and 1. A larger value of a increases the high-frequency content of the image. g( x, y) = { forg − α ⋅ (h ⊗ forg )} / (1 − α )

(8.9)

The Fourier transform is applied to both sides of Equation 8.9, yielding: G (ω x , ω y ) = { Forg − α ⋅ H ⋅ Forg } / (1 − α ) = {(1 − α ⋅ H ) ⋅ Forg } / (1 − α ) (8.10)

Copyright © 2006 Taylor & Francis Group, LLC

238

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 8.14 Frequency response of 5 ¥ 5 and 9 ¥ 9 unsharp mask filters.

The H in Equation 8.10 is the Fourier transform of the mask h. Figure 8.14 shows frequency response with 5 ¥ 5 and 9 ¥ 9 mask sizes and a set to 0.2 and 0.5. There is a very low-frequency bump in the response curve in the figure, but it does not affect the filtered results, g(x,y). It is possible to use other masks that have different values in the kernel than a simple constant valued mask. To do this requires a convolution instead of simple summation to generate the blurred image and consequently increased calculation time. A trade-off must be made among processing time, hardware cost, and required filter response.

8.2 CAMERA CONTROL ALGORITHM 8.2.1 AUTO EXPOSURE, AUTO WHITE BALANCE In a DSC, an auto exposure (AE) block adjusts the amount of incident light on the sensor so as to utilize its full dynamic range. Exposure is typically handled by electronic devices in a camera that simply record the amount of light that will fall onto the image sensor while the shutter is open. This amount of light is then used to calculate the correct combination of aperture and shutter speed at a given sensitivity. An analog or digital gain amplifier that is set before nonlinear operation and color control executes the AE information sensing and control. The luminance value, Y, is used as an index to control exposure time. The entire image-sensing area is divided into nonoverlapped sub-blocks, called AE windows. Figure 8.15 shows a typical layout of AE window within an image. This window layout is also used for calculation of auto white balance (AWB). However, the index of AWB control is different from AE. The AWB uses mean value of each R, G, and B pixel instead of Y for AE. For an RGB CFA sensor, color information is converted to Y value to measure incoming light intensity. The AE signal-processing block calculates mean and peak values for each AE window and passes them to the MPU (as shown in Figure 8.1).

Copyright © 2006 Taylor & Francis Group, LLC

239

Image-Processing Algorithms

FIGURE 8.15 AE/AWB window layout.

TABLE 8.2 Classification of Automatic Focus Control Algorithms Measurement method Range measurement Focus detection

Detection algorithm Active method Passive method External data Internal data

Actual implementation IR(Infrared), ultrasonic Image matching Phase detection Digital integration

The AE control application program running on the MPU will evaluate each value and decide the best exposure time. There are many kinds of evaluation algorithms; some use the variance of each window. Another important camera control parameter is the white balance. The purpose of the white balance is to give the camera a reference to white for captured image data. The white balance adjusts average pixel luminance among color bands (RGB). When the camera sets its white balance automatically, it is referred to as auto white balance (AWB). For an RGB CFA sensor, an image processor in the DSC controls each color gain for incident image light at an early stage of the signal flow path. Details of white balance are described in Chapter 7. The AWB gain correction is adjusted in the raw data domain or just after the color interpolation.

8.2.2 AUTO FOCUS 8.2.2.1 Principles of Focus Measurement Methods Auto focus (AF) control is one of the principal functions of a DSC. Table 8.2 summarizes several kinds of focus control algorithms and their actual implementations in DSCs. In addition, Figure 8.16 illustrates the principles of three focus measurement methods other than the digital integration that is described next.

Copyright © 2006 Taylor & Francis Group, LLC

240

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 8.16 Focus measurement.

8.2.2.2 Digital Integration Many DSCs use a digital integration method for auto focus that uses the acquired image from the sensor and a digital band-pass filter because the algorithm is simple and easy to implement in a digital signal-processing block. The digital integration focus method works on the assumption that high-frequency (HF) components of the target image will increase when in focus. The focus calculation block is composed of a band-pass filter and an absolute value integration of the output from the filter. Digital integration AF uses the acquired image data only so that it does not use any other external active or passive sensors. The host processor of the DSC uses the AF output value and adjusts the lens to get peak output from the AF block. Usually, the MPU analyzes multiple AF windows within an image. Figure 8.17 shows a typical AF window layout within an image. The layout and size of each AF window depends on the type of scene to focus. Five windows and 3 ¥ 3 layout are common for many scenes. The single-window layout is useful to get easy focus calculation for moving objects. The AF output is calculated for each window every frame. Each window generates AF data every 1/30 sec using draft mode frames of the CCD sensor and sends them to the host processor of the DSC. This method does not give absolute range distance value from the DSC to the object as an active sensor does, so the host processor will search for the peak of the AF data by using lens movement. Also, the host processor must judge which direction of lens movement will improve the focus based on the past history of lens movement and AF output data. Figure 8.18 shows how to calculate AF data within a window and the AF filter response. In Figure 8.18(a), the AF block receives horizontal pixel data and calculates the filter output. Then it accumulates the absolute values of the AF filter outputs. It is desirable to calculate in vertical and horizontal directions in each window. From the viewpoint of hardware implementation, horizontal line scan is easy but vertical

Copyright © 2006 Taylor & Francis Group, LLC

Image-Processing Algorithms

241

FIGURE 8.17 Multiple window layout for AF operation.

FIGURE 8.18 Auto-focus filter and output.

line scan requires some amount of line buffer memory. Figure 8.18(b) shows a typical band-pass filter for AF use. The resonant frequency depends on the characteristics of the lens system in the DSC. Usually, a low resonant frequency gives better response at the initial stage when a scene is out of focus. Figure 8.18(c) shows how the output from the AF block will vary with lens movement around the point of focus.

8.2.3 VIEWFINDER

AND

VIDEO MODE

Most DSCs have an electronic viewfinder composed of thin film transistor (TFT) liquid crystal display as well as a conventional optical viewfinder. The electronic viewfinder image must refresh very quickly for the photographer to find and focus on the subject. For this purpose, the DSC uses a draft mode output of a CCD sensor. Details of a draft mode readout mechanism are described in Section 4.3.5 and Section 5.4.3. The draft mode data are full size in the horizontal direction but decimated in the vertical direction; the output speed is 30 frames per second. The DSC uses draft

Copyright © 2006 Taylor & Francis Group, LLC

242

Image Sensors and Signal Processing for Digital Still Cameras

mode data for auto-focus operation and for the viewfinder by resizing it in the horizontal direction using a decimation low-pass filter.

8.2.4 DATA COMPRESSION In a single-sensor DSC, the raw data are spatially compressed by the CFA on the sensor by one third. The amount of raw data is given by X ¥ Y ¥ (12 – 14) b/pixel, where X and Y are number of pixels for horizontal and vertical directions. However, once the full-color image is reconstructed by color interpolation, it becomes X ¥ Y ¥ 3 bytes. For example, in the case of a 5-Mpixel sensor having 12 b/pixel depth, the raw data have 7.5 Mbytes; however, the reconstructed full-color image has 15 Mbyte, which uses 15 Mb of storage area for every single image. The image compression technology plays an important role. There are two data compression categories; one is reversible or nondestructive and the other is irreversible or destructive compression. The former recovers 100% of the original image and the latter loses a certain amount of information of original data. In the ideal case, the former can achieve 30 to 40% compression from original data (reduction to 0.7 to 0.6 from 1.0), but this is not enough for most DSC systems. Reversible compression is used for raw data saving and offline image processing on a personal computer. Irreversible compression can achieve a large amount of size reduction that depends on how much image degradation is acceptable. Usually, the image has a large amount of redundancy, so irreversible compression techniques have been used to reduce its redundancy. Many image compression algorithms exist today, but the need for performance and a widely used standard favors the choice of JPEG (joint photographic expert group) compression. Figure 8.19 shows the DCT-based JPEG image compression flow. JPEG technology stands on many kinds of technology. Because the human eye has highest spatial resolution for luminance but very low for chrominance, the JPEG handles YCbCr image data. It also uses discrete cosine transform (DCT) and Huff-

FIGURE 8.19 DCT-based JPEG image compression.

Copyright © 2006 Taylor & Francis Group, LLC

243

Image-Processing Algorithms

TABLE 8.3 Removable Data Storage Name HDD card Compact flash Smart media (SSFDC) SD card Mini SD card Memory stick Memory stick duo XD-picture card

Media size 20 g, 42.8 ¥ 36.4 ¥ 5.0 mm 15 g, 36 ¥ 43 ¥ 3.3 mm 2 g, 45 ¥ 37¥ 0.76 mm 1.5 g, 32 ¥ 24 ¥ 2.1 mm 21.5 ¥ 20 ¥1.4 mm 4g, 50 ¥ 21.5 ¥ 2.8 mm 31 ¥ 20 ¥1.6 mm 24.5 ¥20 ¥1.8 mm

Memory Hard disk Flash memory Flash memory Flash memory Flash memory Flash memory

man coding. The standard and related technologies are described in Pennebaker and Mitchell.2 The image file format for DSCs is standardized as the digital still camera image file format standard (exchange image file format for digital still camera: exif) by JEIDA (Japan Electronics Industry Development Association Standard).6,7

8.2.5 DATA STORAGE Thanks to the recent spread of personal data assist (PDA) equipment and the demand for large-capacity compact storage media in many fields, a wide variety of small removable media has been developed and introduced to the DSC-related market. Table 8.3 summarizes current removable data storage media used in DSC systems. Some media have storage capacity over the GB (109 byte) boundary. All of them have a capability to transfer data to a personal computer (PC) using an adapter. Read/write speed is also important for application in future DSCs. All these media achieve at least 2 Mb/sec and some over 10 Mb/sec R/W speed.

8.2.6 ZOOM, RESIZE, CLIPPING

OF IMAGE

8.2.6.1 Electronic Zoom Many DSCs have an optical zoom that uses a zoom lens system. However, the zoom described in this section is based on the digital signal processing in the DSC, which is called “electronic zoom” or “digital zoom” in some books. This section describes the principles of the electronic zoom algorithm and the underlying two-dimensional interpolation algorithm in detail. Unlike optical zoom, electronic zoom magnifies the image with digital calculation so that it does not expand the Nyquist limit of the original image. The electronic zoom operation inserts pixels between the original pixels, which are calculated by interpolation. Discussion will start with one-dimensional interpolation theory and then expand to two dimensions. A basic, special case of interpolation was explained in Section 8.1.2 as color interpolation in which a value at nonexisting location is estimated using values of surrounding existing locations. In this section, a more general form will be explained. Equation 8.11 describes the general interpolation calculation using convolution.

Copyright © 2006 Taylor & Francis Group, LLC

244

Image Sensors and Signal Processing for Digital Still Cameras

FIGURE 8.20 Nearest neighbor interpolation.

p( x ) =

∑ f ( x − x ) ⋅ g( x ) i

i

(8.11)

i

The function, f(x), of Equation 8.11 is an interpolation function and describes the entire characteristics of the interpolation results. It is a kind of low-pass filter and affects the image quality of zoomed results. A number of interpolation functions have been reported by several authors5; three well-known interpolation functions will be introduced here: nearest neighbor, linear, and cubic spline interpolation functions. 8.2.6.1.1 Nearest Neighbor Interpolation Function In the nearest neighbor interpolation, each interpolated pixel is assigned the value of the pixel value that is the nearest from the original data. Figure 8.20 shows nearest neighbor interpolation in two-dimensional space. In the figure, the interpolation result, Pi, has the same value as the original image pixel, P1. The interpolation function is shown in Equation 8.12. Nearest neighbor interpolation is the simplest method and requires little computational resource. The drawback is the poor quality of the interpolated image. f (x) = 1

0 ≤ x < 0.5

f (x) = 0

0.5 ≤ x < 1

f (x) = 0

1≤ x

(8.12)

8.2.6.1.2 Linear Interpolation Function Linear interpolation uses two adjacent pixels to obtain the interpolated pixel value. The interpolation function is expressed in Equation 8.13. When linear interpolation is applied to an image, it is called bilinear interpolation (Figure 8.21). Four surrounding points are used to estimate a pixel. Bilinear interpolation is a relatively simple interpolation method, and the resulting image quality is good. The calculation

Copyright © 2006 Taylor & Francis Group, LLC

245

Image-Processing Algorithms

FIGURE 8.21 Bilinear interpolation.

can be divided into each direction (x and y) and the calculation order of x or y makes no difference. f (x) = 1 − x

0 ≤ x