3D Video Technologies: An Overview of Research Trends 0819480975, 9780819480972

Citation preview

Bellingham, Washington USA

Onural, Levent. 3D video technologies: an overview of research trends / Levent Onural. p. cm. Includes bibliographical references and index. ISBN 978-0-8194-8097-2 1. 3-D television--Research. 2. Three-dimensional display systems-Research. I. Title. II. Title: Three-dimensional technologies. TK6658.O58 2010 006.6'96--dc22 2010036654 Published by SPIE P.O. Box 10 Bellingham, Washington 98227-0010 USA Phone: +1 360.676.3290 Fax: +1 360.647.1445 Email: [email protected] Web: http://spie.org Copyright © 2011 Society of Photo-Optical Instrumentation Engineers (SPIE) All rights reserved. No part of this publication may be reproduced or distributed in any form or by any means without written permission of the publisher. The content of this book reflects the work and thoughts of the author(s). Every effort has been made to publish reliable and accurate information herein, but the publisher is not responsible for the validity of the information or for any outcomes resulting from reliance thereon. Printed in the United States of America. About the cover: The original human model in the cover is produced using the computer graphics tool Topmod3D Topological Mesh Modeler (http://www.topmod3d.org/) by Ergun Akleman. It is then converted to holographic signals, and displayed as a small ghostlike 3D image visible by naked eye at the Bilkent University Holographic 3DTV Laboratory by Fahri Yaraş and Dr. Hoonjong Kang. A photograph of this holographic image is then incorporated into the cover graphic design. The author thanks Ergun Akleman, Fahri Yaraş and Dr. Hoonjong Kang for their contributions.

         

To Canan, Engin, and Deniz.

Contents Preface .................................................................... xi Acknowledgment ................................................... xv List of Acronyms ................................................. xvii 1

Introduction .................................................................1

2

An Overview of 3D Imaging and Visualization Technologies...............................................................5 2.1 Stereoscopy .....................................................................6 2.2 Autostereoscopic Viewing ..............................................9 2.3 Multiview Autostereoscopy ..........................................11 2.4 Integral Imaging ............................................................13 2.5 Holography ...................................................................14 2.6 Volumetric 3D Display Devices ....................................15 2.7 Comparative Assessment of 3D Imaging Techniques...18

3

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display ........................................................23 3.1 The Need for Decoupling Capture and Display ............24 3.2 State-of-the-Art in 3D Video Capture ...........................26 3.2.1 Single-camera techniques .................................... 26 3.2.2 Multicamera techniques ...................................... 27 3.2.3 Holographic capture devices ................................29 vii

viii

Contents

3.2.4 Pattern projection techniques .............................. 30 3.2.5 Time-of-flight techniques .....................................31 3.3 State-of-the-Art in 3D Representation Techniques....... 32 3.3.1 Dense depth representation ..................................32 3.3.2 Surface-based representations .............................33 3.3.3 Point-based representations ..................................34 3.3.4 Volumetric representations...................................34 3.3.5 Texture mapping ..................................................34 3.3.6 Pseudo-3D representations ...................................35 3.3.7 Light-field representations ...................................35 3.3.8 Object-based representations ................................36 3.3.9 Standards for 3D scene representation .................37 3.4 State-of-the-Art in 3D Video Coding Techniques ........ 37 3.4.1 Stereoscopic video coding ................................... 38 3.4.2 Multiview video coding........................................39 3.4.3 Video-plus-depth ................................................. 41 3.4.4 3D mesh compression ..........................................41 3.4.5 Multiple description coding..................................43 3.5 State-of-the-Art in 3D Video Streaming Techniques ....44 3.5.1 Analog broadcast ..................................................44 3.5.2 Digital broadcast ..................................................44 3.5.3 3DTV-over-IP networks .......................................45 3.5.4 Streaming protocols..............................................45 3.5.4.1 Multiview video streaming...................46 3.5.4.2 Error correction and concealment ........46 3.5.4.3 3D video-streaming experiments and demonstrations ....................................47 3.6 State-of-the-Art in 3D Video Display Techniques ........48 3.6.1 Multiview displays ...............................................50 3.6.2 Head-tracking displays ........................................ 52 3.6.3 Volumetric displays..............................................52 3.6.4 Holographic displays ............................................53 3.6.5 Signal processing issues associated with holographic displays ............................................54

Contents

ix

3.7 End-to-End 3DTV Systems .......................................... 56 3.8 3D-Video-Related Communities and Events ................ 58 4

Current Research Trends.........................................61 4.1 3DTV ............................................................................ 61 4.2 2020 3D Media: Spatial Sound and Vision................... 69 4.3 3DPHONE .................................................................... 70 4.4 MOBILE3DTV ............................................................. 71 4.5 Real 3D .........................................................................73 4.6 MUTED and HELIUM3D ............................................ 74 4.7 3D4YOU ....................................................................... 74 4.8 3D Presence .................................................................. 75 4.9 VICTORY..................................................................... 75

5

The Future of 3D-Media-Related Research Trends ........................................................................77 5.1 Near-Term Research Activities..................................... 78 5.2 Medium-Term Research Activities ................................80 5.3 Long-Term Research Activities .....................................83 5.4 Future Internet and 3D Media ........................................85

References ......................................................................87 Index ................................................................................91

 

 

Preface This small book is intended to provide a broad perspective on research trends in 3D video and related issues. At the intersection of many diverse technical fields, 3D video is certainly a difficult topic. The technical details are avoided in this book; rather, the text is developed to meet the needs of a larger reader group who desire to understand the issues, concerns, technical problems, and their currently proposed solutions, as well as the interactions among different components of the entire 3DTV chain. Current state-of-the-art is presented with a brief overview of the technological span of concepts related to 3D video. Current research activities are then outlined along with goals and results. Finally, the research direction in the field is predicted for the near, medium, and long term. Expected mingling of media in general, and in our case 3D-video-based media in particular, with the future Internet is highlighted. The work leading to this book is an outcome of the 3D Media Cluster activities to which I also contribute; the cluster consists of 3D-media-related projects that are funded by the European Commission. Currently, there are 10 projects in the cluster; the number changes as projects complete their lifetime and as new projects emerge and enter the cluster. (The cluster has recently changed its name to 3D Immersive Interactive Media Cluster.) One of the cluster tasks was to write a white paper in 3D media technologies. As the content exceeded the

xi

xii

Preface

appropriate length for a typical white paper, the idea of presenting the material as a small book to a wider readership emerged. I thank the Cluster members for their contributions, comments, and support while this book was prepared. Special thanks go to Dr. Atanas Gotchev and Dr. Aljoscha Smolic, who have provided extensive comments and remarks, which I used to improve the text and the referencing. Some of the material presented in Chapter 4 is derived from public disseminations of the 3DTV Network of Excellence (NoE) and thus is built upon the work of about 200 researchers, including myself, who contributed to the Network activities.† Key sources are given in the list of references. Some of these are archival documents such as books and journal articles. However, some of the references are publicly disseminated technical reports of the Network that can be accessed via the project website www.3dtv-research.org. I thank all 3DTV NoE researchers for their excellent contributions to 3DTV NoE activities and their willingness to share their findings with the public. The content in Chapter 5 was developed primarily from announcements about the research goals and results of the individual projects. Such announcements are made mostly through project web sites, which can be accessed via the 3D Media Cluster page at www.3dmedia-cluster.eu. I am grateful to the hundreds of scientists with whom I have had the chance to interact during my 29 years of active research in 3D video and holographic 3D displays. It would be impossible to understand the broad range of issues without their expertise and ideas, and I am especially grateful for their willingness to share these with me and others. I also thank my students who had the desire and the motivation to learn about the topics and related issues. Their                                                             

 Integrated 3D Television—Capture, Transmission and Display (3DTV) project was funded by the European Commission within FP6 under the grant number 511568.  †

 

Preface

xiii

intriguing questions—and sometimes answers—amassed over the years surely contributed to my own understanding of this exciting field of 3D video. I believe that this book will be useful to those who are willing to delve into the broad range of technical issues within the 3D video field. The book is concise and presents information in an efficient manner. Even though a limited number of references are listed in the book, those references and the literally thousands of papers cited in them collectively provide an almost complete literature for any researcher who would also like to contribute to the field. Levent Onural Bilkent, Ankara, Turkey December 2010  

 

Acknowledgment The research conducted for the projects undertaken by the 3D Media Cluster is funded by the European Commission under the scope of the ICT theme Seventh Framework Programme (FP7) of the European Community for research, technological development, and demonstration activities (2007–2013) and under the scope of the IST Thematic Priority of the Sixth Framework Programme (FP6) of the European Community for research, technological development, and demonstration activities (2002–2006). The Cluster project acronyms and the grant numbers, as of 2009, are: 3DTV - FP6 511568 (Ended) 3DPHONE - FP7 213349 MOBILE3DTV - FP7 216503 Real 3D - FP7 216105 HELIUM3D - FP7 215280 MUTED - FP6 0340990 (Ended) 3D4YOU - FP7 215075 3D Presence - FP7 215269 VICTORY - FP7 044985 (Ended) 2020 3D Media - FP7 215475 i3DPost - FP7 211471 xv 

List of Acronyms 2DTV 3DTV AFX ARQ AVC CCD CRT DBV-H DCCP EBU EC FEC HDTV IBC ICT IEC IEEE IMU IP IPTV ISO ITU

two-dimensional television three-dimensional television animation framework extension automatic repeat request advanced video coding charge-coupled device cathode ray tube digital video broadcasting - handheld datagram congestion control protocol European Broadcasting Union European Commission forward error correction high-definition television International Broadcasting Convention information and communication technologies International Electrotechnical Commission Institute of Electrical and Electronics Engineers inertial measurement unit Internet protocol Internet protocol television International Organization for Standardization International Telecommunication Union xvii

xviii

LCD LED MDC MPEG MVC NoE NTSC PAL PDLC RGB RTP SDP SECAM SLM SMPTE T-DMB UDP VLSI VRML X3D XMP YUV  

List of Acronyms

liquid crystal display light-emitting diode multiple description coding Moving Picture Experts Group multiview video coding network of excellence National Television System Committee phase alternate line polymer-dispersed liquid crystal red green blue real-time transport control session description control séquentiel couleur à mémoire (sequential color with memory) spatial light modulator Society of Motion Picture and Television Engineers terrestrial digital multimedia broadcasting user datagram protocol very large-scale integration virtual reality modeling language extensible 3D extensible metadata platform luminance (Y) and two color (U and V) signals to represent color pictures

 

Chapter 1

Introduction There is no doubt that in the future the Internet will be more densely integrated at the infrastructure level, and thus, be an integral extension of our physical lives and environment. Such an extension will affect all aspects of individual and social lifestyles. Many expert studies have been conducted to predict those effects and the subsequent changes. The scope of this book is not quite so broad; instead, it will focus on a particular technology—3D media—and its interaction with the Internet. Among the various novel modalities of content delivery via the Internet, 3D video is expected to be the choice for visual communications. The future Internet will bring a new user experience by delivering rich media in 3D, and in turn, 3D technologies will be influenced by developments in such an infrastructure. The general public is well aware of the ultimate goal of 3D video: ghostlike, moving 3D images have already been depicted in many science fiction films. The ultimate goal in visual recording and communications is to create an exact (except perhaps in size) optical replica of a 3D environment with 3D objects in it at another time or another place. Optical receivers, including our eyes and cameras, sense the light they receive. In other words, we “see” only the light that enters through our pupils. Consider two optical environments: one of them is the original illuminated 3D environment. The light, which is generated by artificial light sources, falls onto the objects and reflects off of them. This reflected light carries the information 1

2

Chapter 1

about the geometry and optical properties of the environment and fills the 3D space. The observer moves in this light-filled space and sees the environment via the light that enters his visual receptors (eyes) at his position. The other environment does not have the same physical objects but has exactly the same light distribution as the original environment in a 3D space. Since the two light distributions are the same, any observer immersed in either environment will see exactly the same scene; those who are observing the recreated optical environment will see the true 3D optical replica of the original. This physical duplication of light is the key to ghostlike visual reproduction. Such moving video images would be floating in space or standing on a tabletoplike display, and viewers would be able to peek or walk around the images to see them from different angles or perhaps even from behind (Fig.1.1).

Figure 1.1 Artist’s impression of a futuristic 3D television display unit (Artist: Erdem Yücel) (H.M. Ozaktas and L. Onural, Eds., ThreeDimensional Television: Capture, Transmission, Display. Springer, 2008).

Introduction

3

As in all other recording systems, such a visual duplication system needs two key elements: a means of recording the original space-filling light, and then, physical devices that can recreate that recorded light. Of course, the success depends on the fidelity of the recreated light distribution to the original; ideally, all detectable physical properties must be recorded and exactly recreated. All 3D video techniques attempt to achieve such a goal. However, some of these, such as stereoscopy, are quite distant from the ideal case, whereas other techniques, such as integral imaging and holography, are closer to it. Another basic component of any visual reproduction system is the delivery medium. As usual, content is delivered either in stored form via different transportable hard devices or by electronic or optic means. Thus, capture or creation of content, its transportation, and eventually its display, are the three key functional elements of an end-to-end 3D video system (Fig.1.2). The 3D Media Cluster, which consists of EC-funded projects in 3D-related topics, collectively covers almost all related technical issues in this field. This includes content capture and creation, its transportation, and its display. During the past decade the Internet has become a fast-growing delivery

Figure 1.2 Functional units of a possible end-to-end 3DTV chain. (From L. Onural, H. M. Ozaktas, E. Stoykova, A. Gotchev, and J. Watson, “An Overview of the holographic related tasks within the European 3DTV project,” Proc. SPIE 6187, 61870T, 2006.)

 

4

Chapter 1

medium for video, and the Internet will likely be the choice for any kind of data delivery—including video in 3D—surpassing all other delivery alternatives such as the conventional broadcast modes for TV. This book is a spinoff of the 3D Media Cluster activities and covers a broad range of issues in 3D media and the future Internet. The scope of the book is 3D content generation and its delivery and display, together with interaction with such content. Therefore, capture of visual information associated with dynamic 3D scenes, processing of such content, creation of synthetic 3D content via computer graphics procedures, techniques associated with abstract representation of such data, storage and transmission of such content, and different forms of display are all within this book’s scope. Compression and coding of such data for digital delivery are included as well. The technological details of each of these functional components are also of fundamental importance and within the scope. End-toend systems and their associated integration issues are also discussed. Since our focus is primarily visual 3D content, other forms of 3D imaging, such as MRI, tomography, etc., which are used primarily for medical purposes, are not within the scope of this book. However, this exclusion is only for the technical details of the associated imaging (image capture) devices; once the captured data is converted to an abstract 3D representation, the delivery, display, and interaction are still within this book’s scope. Although audio usually accompanies visual content, our interest in audio within the 3D Media Cluster is quite limited and therefore, audio is not included in the book. The book starts with a brief overview of 3D imaging and visualization. Then, an overview of major successes in the field that collectively make up the current state-of-the-art is presented. Ongoing research activities in Europe are outlined; research goals and recent results are briefly presented. The book then gives an overview of short-, medium-, and long-term research trends. The book concludes with predictions for the position and role of 3D-media technologies within the future Internet, together with some strategic goals.

 

Chapter 2

An Overview of 3D Imaging and Visualization Technologies In this chapter, an overview of techniques available in the past and at present in the field of 3D imaging and visualization are presented. The purpose is to give the reader a clear idea of the basic principles of different techniques, their evolution in time, and the advantages and problems of each in a comparative manner. Although 3D imaging and visualization technologies are commonly considered to be the most novel, recent, and advanced form of visual content delivery, 3D photography, cinema, and TV actually have a long history.1 Surprisingly, stereoscopic 3D versions of these common visual media are almost as old as their 2D counterparts. Stereoscopic 3D photography was invented as early as 1839. The first examples of 3D cinema were available in the early 1900s. Various forms of early 2D television were developed in the 1920s, and by 1929, stereoscopic 3DTV was demonstrated. Commercial stereoscopic products also emerged in this era. However, while the 2D versions of photography, cinema, and TV have flourished to become an important component of twentieth century culture, their 3D counterparts 5

6

Chapter 2

have almost disappeared after many peaks of temporary success. For example, stereoscopic 3D cinema had a peak around 1950. Photography is a technique that captures the intensity distribution of focused light on a plate. Conventional motion pictures are essentially based on the capture of many photographic still images one after another in short intervals, faster than the ability of the human visual system to follow them as a sequence of isolated pictures but rather as a continuous motion picture; again, it is the intensity of the focused light that is recorded. Chemical recording has been popular for nearly a century for both still and motion pictures. Consumer products for electronic video recording using analog techniques have been around for more than fifty years. Digital electronic recording of still pictures became popular during the past two decades, and digital electronic recording of motion pictures became a common practice after the 1990s. The cinema industry converted more slowly to digital recording and delivery, but that transition is now taking place at an increasing pace. Analog TV transmission is now being phased out all over the world, and full adoption of end-to-end digital technology in cinema is expected during the next few years. Color recording is based on capture of intensities of more than one color component; typically, there are three colors: red, green, and blue. Intensity of focused light on a screen is just one feature of light, and such a recording carries only 2D information. Other physical properties are also needed for 3D capture. In particular, not only the intensity, but also the directional distribution of propagation of light rays, is crucial for 3D information.

2.1 Stereoscopy The earliest form of 3D video is stereoscopy. Stereoscopy is based on capture and delivery of two simultaneous conventional 2D videos. The basis of stereoscopy lies in the fact that the human visual system has two eyes. Due to the different positions of our eyes, while observing the environment, two slightly different 2D images of the 3D scene fall onto the retina of each eye. The human visual system is based on (a) capturing light via

An Overview of 3D Imaging and Visualization Technologies

7

our pupils and (b) processing it via the lens and the variable size of the pupil before receiving a focused color image on the retina. The visual stimuli on both retinas are then sensed and carried to the brain where they are interpreted as the seen environment. The lens and the pupil size adaptively interact with the observed scene, as do the directions of the two eyeballs, which are slightly different due to triangularization while focusing on a nearby object. Therefore, if the two 2D images that would fall onto the retinas of the two eyes are simultaneously captured from a scene and then somehow presented separately but simultaneously to the two eyes such that the retina images are replicas of the originals, the brain should see the same 3D scene. Stereoscopy is based on the capture of those two slightly different 2D images that mimic the images naturally obtained by two eyes at slightly different positions, followed by the simultaneous delivery of each image to the corresponding eye. Different modes of separation techniques have been used. The capture is usually accomplished by two parallel cameras. Ideally the two cameras should physically be the same, and the alignment should be perfect. Usually, special glasses are needed for separating the two images. Early versions of these glasses were based on anaglyphs, which are two intensity images, each having a different color whose color spectra do not overlap. Each of the two images is printed (photography), or projected (cinema), or electronically displayed (TV) by overlapping them properly. Eyewear consists of different color filters on each eye that filter, and thus separate, these overlapped images. The color filters should match the spectrum of the displayed images so that each eye receives only its single 2D intensity image. This technique is still used in printed stereoscopic photography. Due to the restrictions on the color, full-color anaglyph-based stereograms are impossible. However, more sophisticated means of filtering are adopted for cinema and TV. In current 3D stereoscopic cinema, either polarization-based filtering or electronically controlled shutter-based delivery is preferred. Polarization-based filtering is based on the fact that  

8

Chapter 2

light is an electromagnetic wave where electric and magnetic fields are vectors and thus have directions. The polarization direction may be controlled by filters that pass only the component of the light that possesses a particular polarization property. Linear and circular polarizations are two commonly used forms. Full-color stereoscopy is made possible by polarization-based filtering. The images for the right and the left eyes are differently polarized and simultaneously displayed in superposed form, as in anaglyphs. The eyewear with matching polarization properties requires different polarizers for each eye in order to filter out each image and deliver it to the corresponding eye. Shutter-based systems are used to separate time-multiplexed right and left eye images. Therefore, the two images are not simultaneously displayed as in polarizers (or anaglyphs) but are displayed alternately by switching between one another (typically 140–200 pictures per second). Electronically remote-controlled eyewear has a shutter control that is synchronized to the displayed picture. The shutter blocks one eye by turning opaque and lets the other eye receive the image by turning transparent in synchronization with the displayed image. The opaque/transparent switching is usually accomplished through use of liquid crystal shutters. Other modes of stereoscopic video are also possible. One example is the Pulfrich effect, in which the eyewear has transparent glass (or no glass) for one eye and semitransparent glass (as in sunglasses) for the other eye. The recording is accomplished by capturing a conventional 2D video while continuously rotating around the 3D scene at a specified speed. An observer looking at the captured scene through the abovementioned eyewear sees a 3D scene because the interpretation of the visual stimulus from the eye that receives less light (due to the darker glass) is delayed by the brain. Additionally, due to the continuous rotation, that eye sees the same 3D scene at a slightly earlier time frame, which corresponds to a different angle during the rotation. Thus, it is the two eyes receiving the image from two slightly different angles that creates the stereoscopic vision.

An Overview of 3D Imaging and Visualization Technologies

9

2.2 Autostereoscopic Viewing Autostereoscopy is the term used to describe a class of technologies for the display and visualization of stereoscopic 3D images. What distinguishes autostereoscopy from conventional stereoscopy is that the former does not require any special eyewear. Indeed, some authors call all eyewear-free 3D viewing, including integral imaging and holography, autostereoscopic.2 However, for the purposes of this section, this terminology will be reserved for eyewear-free viewing of stereoscopic (two-view) images. There are two common technologies for autostereoscopic viewing: lenticular and barrier. Both technologies are based on creating a single image by fusing the two captured stereo images. The fusing procedure is called interzigging, and the exact form of interzigging is closely related to the chosen technology and the specific pixel array geometry on the display panel. For still stereoscopic photography, the fused (interzigged) image is printed. For electronic stereoscopy, either still or motion, the interzigged image is electronically displayed on a conventional video display device that is used to display 2D video. However, there is an additional layer mounted on the conventional display (or picture). In lenticular technology, this additional layer consists of tiny cylindrical lenses whose geometry and optical parameters closely match the geometry of the underlying display device (see Fig. 2.1). The lenses are like tiny stripes that are barely visible, but the texture on the lenticular sheet due to cylindrical lenses can be easily felt by fingers. The critical match of the geometry is not a problem for printed stereograms, or for pixel-based electronic display devices, such as LCD monitors. Vertical stripe geometry is common, but slanted lenticular lenses are also used. The choice is based on the way in which the vertical and horizontal resolution between the two images of the stereo pair are distributed. As a consequence of the geometry of the display device, the form of interzigging, the geometry of the lenticular  

10

Chapter 2

Figure 2.1 The optical principle associated with lenticular screens: (a) viewing setup (top view); (b) visible subpixels from a particular viewing direction (front view). (From R.-P. M. Berretty, F. J. Peters, G. T. G. Volleberg, “Real-time rendering for multiview autostereoscopic displays,” Proc. SPIE 6055, 60550N, 2006.)

sheet, and the optical nature of the overall design, the eyes of the observer at a particular location in front of the display receive their own different left and right images. In barrier technology, the additional layer is either in front of or behind the device that displays the matching interzigged picture. The barrier is like a fence consisting of opaque and transparent stripes. Furthermore, this layer is mounted with a slight gap between the display surface. The geometry is adjusted so that for an observer whose position is at a specified position in front of the display, some of the pixels on the display are blocked by the barrier for one eye and the other pixels are blocked for the other eye. Even though neither lenticular- nor barrier-technology-based stereoscopic viewing requires any eyewear (autostereoscopy), they both suffer from the same problem: there is a predefined “sweet spot” for both designs. That is, there is a specific range of positions where an observer's eyes receive the intended separate right and left images correctly. There is no satisfactory 3D perception if the observer is not in this sweet spot. Furthermore, the eyes must lie on a predetermined line, naturally horizontal, for stereoscopic vision.

An Overview of 3D Imaging and Visualization Technologies

11

Both lenticular and barrier technologies are quite old. For example, printed lenticular autostereoscopic images were commonly used for various purposes, including toys, for more than fifty years. However, lenticular- or barrier-based electronic displays are rather novel devices that became popular after LCD went into mass production.

2.3 Multiview Autostereoscopy Autostereoscopy is a technology in which the display device sends different optical content to two different angles out of the display panel. In other words, a monocular observer will see a different video at different angles while looking at the display device. The angles are adjusted so that a human located at a particular distance receives those two different images to his two eyes. In almost all designs, each view propagates away from the display panel within its own narrow horizontal angle, but there is no vertical variation. Multiview autostereoscopy is the extension of this principle to more than two views. Typically, there are five to nine views. However, there are commercial devices with up to 50 or more such views. As in autostereoscopy, each view propagates away from the display device within a narrow horizontal angle allocated to it; there is no vertical variation. Multiview video capture is similar to stereoscopic recording; however, instead of two 2D video capture, there are many cameras, each shooting at a slightly different position. Each camera corresponds to one of the views of the multiview design. An observer positioned at a particular distance depending on the design (i.e., at the sweet spot) receives one of these views into his one eye, and the next view into his other eye, and thus observes 3D as a consequence of stereoscopy. If the observer moves to the right or left, another pair of images among many that are propagating out of the display are received. The advantage of multiview over two-view stereoscopy is the horizontal parallax visible in the former. Two-view stereoscopy lacks any parallax; in other words, an observer  

12

Chapter 2

moving horizontally while viewing the scene will always see the 3D view as if looking at it from the same angle. This creates an unnatural viewing experience, and this is one of the major reasons for the very well-known viewing discomfort experienced in stereoscopic viewing. On the other hand, multiview versions provide a parallax within a viewing angle. Typically, the viewing angle is 20–30 degrees. However, it may go up to much larger angles, especially if the number of views gets larger. Multiview systems are rather novel devices. Research is still continuing to improve such systems. Head-tracking or eyetracking multiview autostereoscopic displays are available.2 These systems continuously detect the position of the eyes, and then provide to two eyes the two views that would have been seen from that particular position when looking at a 3D scene. Multiview display designs are also usually based on the barrier or lenticular technologies described in Section 2.2. The particular form of interzigging to fuse many views into one 2D image over the display panel behind the lenticular or barrier layer is different for each design and geometry. There are also commercial systems based on light-field rendering. In one such design, each pixel of the 3D display device is associated with an array of light-pointing devices, each pointing at a different horizontal angle and position. A diffuser converts the narrow rays to vertical planes of light by keeping the horizontal direction unchanged. None of the multiview display devices described in this section posses vertical variation. However, some horizontal parallax exists. Therefore, an observer gets the feeling of moving around a 3D scene within the viewing angle, together with occlusion and nonocclusion effects, but if he moves up and down in an attempt to see the scene from different vertical angles, that will be ineffective. Thus, the multiview displays reduce the artificial perception associated with stereoscopy while moving horizontally within an angle, but the problem remains for vertical motion.

An Overview of 3D Imaging and Visualization Technologies

13

2.4 Integral Imaging Integral imaging is a 3D imaging technique that has been used since 1905. In a sense, the integral imaging display is an extension of lenticular multiview video display where the lenticular sheet is no longer a 1D (cylindrical) lens array, but an array of very small spherical lenses; such an array is called a microlens array. Therefore, not only does the display consist of different 2D views of the 3D scene separated horizontally (as described in Section 2.3), but vertical parallax is also achieved. Therefore, natural 3D video viewing is possible by integral imaging (see Fig. 2.2).

Figure 2.2 Integral imaging display. (From J. Kim, K. Hong, J-H. Jung, G. Park, J. Lim, Y. Kim, J. Hahn, S-W. Min, and B. Lee, “High-definition integral floating display with multiple spatial light modulators,” Proc. SPIE 7237, 72370T, 2009.)

 

14

Chapter 2

The capture side of integral imaging consists effectively of a 2D array of cameras. In practice, the way to achieve this camera array is to use for recording the same microlens array that is used for display. Simply, a photographic intensity recording device, ideally with a very high resolution, is placed at the focal plane of the microlenses that form the array. Each image recorded by a different microlens is called an elemental image. As the lens array gets larger, with smaller lenses, the imaging device acts essentially as a light-field-capture and rendering device. Since the light-field-capturing and display phases of integral imaging attempt to replicate the 3D physical light distribution, the process may also be called incoherent holographic imaging. Principles of holographic 3D imaging are presented in Section 2.5. As the display device gets closer to an ideal light-field-rendering device, the focus–accommodation mismatch associated with the human visual system diminishes and therefore, such devices are called true-3D devices. There are presently no commercial integral-imaging-based video systems.

2.5 Holography Holography is an imaging technique in which the intensity and the directional information of light are recorded as an interference pattern that is obtained when a reference beam interferes with the information-carrying light. Since interference is only possible with coherent light, lasers are used during recording. The principles of holography have been known since 1948, and practical holograms were first demonstrated in 1960. In some forms of holography, the information-carrying beam simply interferes with itself (diffraction). Early holograms were recorded on very high-resolution photographic films. As digital electronic capture and display devices advanced, electroholography emerged, where highresolution CCDs are used for capture,3 and high-resolution spatial light modulators are used for display.2 However, even the highest-resolution electronic devices available today are far from adequate for writing the fine details of interference fringes. Computer-generated holography is a field where holograms are

An Overview of 3D Imaging and Visualization Technologies

15

generated by computation based on the geometry and optical properties of the underlying 3D scene.4 A holographic 3D display is achieved when a recorded hologram is simply illuminated by proper light. Either coherent light (lasers) is used, or some form of coherence is indirectly achieved by conventional illumination and the self-filtering properties of some types of holographic recordings, such as thick holograms or volume holograms. Coherence requirements are much less stringent during display. As a recording of diffraction or an interference pattern with rich local variations, a hologram is essentially a space-varying diffraction grid that converts the incoming plain light into a propagating light field that, ideally, replicates the original light field of the original 3D scene. Therefore, holography is also a true-3D imaging technique. As the range (angle, color, etc.) of the rendered light field gets closer to the original, the reconstructed 3D image gets closer—in all optical properties—to the original. There are some experimental electroholographic capture and display devices for holographic video;2,3 however, due to limitations of the underlying electronic devices (primarily the size, geometry, and number of pixel arrays) the reconstruction angles are very small and thus at present prohibit any comfortable viewing with the naked eye. Successful singleviewer holographic display devices have been demonstrated that yield a 3D image whose size is comparable to the size of a typical TV picture and have a depth of a few meters (based on eye tracking and accompanying holographic rendering of the light field limited to only around the pupils of the viewer).

2.6 Volumetric 3D Display Devices Volumetric displays form another 3D display mode (Figs. 2.3 and 2.4).2 Such displays employ a mechanical volume-sweeping device that has display elements on it. As the device is moved (usually in a cyclic manner) within the sweeping volume, the  

16

Chapter 2

Figure 2.3 A volumetric 3D display output. (From D. Miyazaki, K. Shiba, K. Sotsuka, and K. Matsushita, “Volumetric display system based on 3D scanning of inclined optical image” Optics Express 14(26) 12760, 2006. © OSA 2006. Reprinted with permission.)

An Overview of 3D Imaging and Visualization Technologies

17

Figure 2.4 A piston-type volumetric 3D display. (From V. Yücesoy, D. Tunaoglu, M. Kovachev, and L. Onural, “Design and implementation of a DMD-based volumetric 3D display" 3DTV-CON 2008, © IEEE, 2008. Reprinted with permission.)

display elements are electronically excited to yield appropriate brightness and color variations at those particular positions at particular moments in time. If the motion is faster than the tracking abilities of the human visual system (as in cinema or TV), the perceived image becomes a 3D volume image. There are plenty of different designs in terms of geometry and sweeping patterns.2 Either the display elements are self-luminous and directly mounted on the sweeping apparatus, or the sweeping structure acts as a moving screen that is illuminated by projection. Volume scanning may also be achieved by variable focal length lenses or mirrors. Volumetric devices provide full-parallax images and therefore do not create the unnatural and disturbing effects as in stereoscopy. However, occlusion effects cannot be implemented with this technology; the generated 3D images are rather  

18

Chapter 2

transparent. Another major drawback of most of the volumetric displays is the demanding mechanical constraints: because the device is continuously in fast motion, the display devices are bulky and quite noisy.

2.7 Comparative Assessment of 3D Imaging Techniques Each imaging technique outlined in the previous sections has its characteristic advantages and problems. As described in Chapter 1, the ultimate goal is to record and recreate the physical light distribution in a 3D space. Stereoscopy is the simplest and oldest technique, but it is very far from capturing and creating original physical light distribution, and therefore, the quality of the resultant 3D effect is quite inferior compared to other techniques. The main problem is the lack of parallax if one moves from one position to another. Another intrinsic problem is the mismatch of focus distance of the observing eye, where this focus is always on the display screen, and the triangulation of the eyes to the 3D location of the observed object point that may be in front of or behind the screen. This triangulation is commonly called convergence or vergence. Other practical problems are associated mainly with the misalignment of the two images on the screen. This includes the mismatch in camera parameters related to lens positioning, structural differences in the lenses, and the direction of the cameras. Additional alignment problems may also arise during the display. Such irregularities result in a seemingly strange and unnatural viewing experience. These irregularities, which do not exist while directly looking at a natural 3D scene, are perceived by the brain as contradictory or unusual stimuli. The consequence is a very uncomfortable feeling similar to motion sickness, and so the observer tends to quit watching the stereo image. This feeling is also called “eye fatigue.” Indeed, it is believed that this negative experience is the primary reason for the commercial failure of stereoscopic 3D imaging. Many cycles of booming trends in stereo 3D movies faded out, the latest being in the 1950s, and the

An Overview of 3D Imaging and Visualization Technologies

19

main reason may be attributed to this viewer discomfort. As a consequence of end-to-end digital-video-recording technologies, the alignment mismatch is now almost eliminated. This in turn seems to be contributing to the successful commercialization of stereo 3D as evidenced in the recent boom in 3D motion picture releases. The major advantage of stereoscopy is its relative simplicity in both recording and display. Multiview video provides some horizontal parallax, which does not exist in ordinary stereoscopy. This is its primary advantage over simple stereoscopy; however, the parallax is still limited to a small angle due to the limited number of horizontal views. Typically, each view is separated by an angle of about two–three degrees, and thus, a display with nine views gives a field of view of 20–25 degrees. If a viewer moves out of this region, usually the directionally separate views are repeated in the same order, and thus a jump at the borders of the proper viewing angle is observed. Though less disturbing, similar discrete jumps are also inevitable within the viewing zone since the finite (and quite small) number of views do not render a smooth angular transition as the observer moves. However, other problems that exist in stereoscopy, such as the vergence– accommodation conflict, also exist in multiview systems. Alignment problems, which usually exist, should be avoided for comfortable viewing. Another severe problem with multiview viewing is the resolution of the display device. Usually, the resolution is not a problem during recording, since many 2D cameras are used at once and each has sufficient resolution. However, these videos must be blended into a single video at the display side (as explained in Sections 2.2 and 2.3), and that blending will be lossless only if the total number of pixels at the display is equal to the sum of pixel numbers of recording cameras. This problem exists both in stereoscopy and in multiview systems, but it becomes more severe as the number of video views increases. Therefore, the problem may be bearable for stereoscopy, but quickly becomes unacceptable when the number of views grows (multiview), since the resulting image becomes severely blurred.  

20

Chapter 2

The primary superiority of integral imaging to stereoscopy and multiview imaging is the added vertical parallax. Furthermore, as the number of lenses—and thus the number of elemental images—increases, the system gets closer to a lightfield-rendering device and therefore, brings the potential of true3D viewing. Horizontal-parallax-only multiview systems also get closer to ideal light-field-rendering devices as the number of views increases, but only along the horizontal direction. The practical problems associated with integral imaging are also severe. The alignment problem is still present but somewhat eased if the same microlens array is used during recording and display. However, this is impossible if an integral-imaging-based TV system is launched. The resolution problem is immensely more severe since the number of views is now much larger; instead of a 1D (horizontal) array, a 2D array of elemental images must be blended. Another problem in integral imaging is the leakage (crosstalk) of neighboring elemental images with each other during recording. This is also a problem in lenticular or barrier-based autostereoscopic viewing (two-view or multiview), but it is much more severe in integral imaging due to the need to pack many more elemental images into a limited-size device. Leakage of neighboring images in integral imaging is a problem during both recording and display. In terms of duplicating the physical 3D light distribution of an original scene, holography is far superior to other techniques. However, holography has a multitude of problems. A characteristic problem occurs during recording: interference patterns are very sensitive to motion; therefore, the object must be still during the exposure. Maximum tolerable motion is approximately a fraction of a micrometer, which requires very short exposure times. In turn, this necessitates high-power light sources or sensitive capture devices. Another problem is the coherence requirement of light, since interference without coherence is impossible, and coherence brings another type of problem, called speckle noise, during viewing. Illumination requirements are also stringent in holography. Furthermore, the resolution requirement is much more demanding in holography since the fringe variations are in the order of wavelength of light,

An Overview of 3D Imaging and Visualization Technologies

21

which is smaller than a micrometer. Adding to this the requirement of a larger recording for higher quality, the signal even for a modest hologram becomes very rich, and so a digital device for holographic use must have hundreds of millions of pixels. Furthermore, the physical nature of a holographic display device is quite different from that of a conventional display. Since holographic displays operate by diffracting the incoming light, and the angle of diffraction is a direct consequence of local spatial frequency of the fringe pattern written on the hologram, large angles are only possible if the fringe pattern is a high-frequency spatial signal. Ideally, the spacing between fringes should be around a micrometer; current electronic technologies are far from such a resolution and therefore, the angle of diffraction is quite small. This results in narrower viewing zones than required for comfortable direct viewing.

 

 

Chapter 3

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display In this chapter, the current state and major technological accomplishments that are within the scope of this book are highlighted, accompanied by some open problems. As in any other imaging modality, 3D imaging involves the capture of 3D information, its delivery and storage, and eventually its display. The interaction of the end user is primarily with the display device; therefore, for the consumer the terms “3D Video,” “3D Games,” “3D Cinema,” or “3DTV” are always associated with a display device. Typically, this is a conventional TV-like monitor in stereoscopy and multiview 3D systems; a screen for the cinema as in conventional movie theaters; a game console with a TV-like monitor; or a ghostlike image in a hypothetical true-3D ideal display case. 3D graphics and 3D games are also commonly displayed on conventional 2D monitors. However, this is essentially due to the very limited availability of 3D monitors compared to their 2D counterparts. End-to-end imaging systems are much more involved than just the display: there is very sophisticated technology throughout the chain, and that includes capturing or artificially generating 3D content, delivery to the user, ability of the user to interact with and probably modify the content, etc. The display is only one end of the chain. Each of the chain components is important, and the overall success is possible only if these 23

24

Chapter 3

components function properly. Therefore, it is important to understand the current state-of-the-art in all of these functional components of an end-to-end 3D system. The future Internet is envisioned as an integral system that comprises not only digital content delivery capability, but also end units for both capturing and displaying content. Furthermore, authoring and organizational tools may also be considered as parts of the future Internet, since such services may come to the end user via the Internet, and thus their actual physical location and form will be transparent. One goal is the seamless integration of those physical end units to the network, so it is not only networking, but also terminals and their coupling to the network, that are the focus of research. The importance of highlighting current state-of-the-art in 3D imaging and delivery technologies from a holistic perspective is stressed from a holistic perspective that includes all three major functional units at once, namely, the capture/creation, storage/delivery, and display/interaction. However, each such unit also has its own specific problems, and a detailed separate coverage of these functional units is necessary and unavoidable. One other issue that prompts the examination of the abovementioned functional units separately is related to the issue of the decoupling of input technology and formats from the display side. This is described in the following section with the conclusion that tight coupling is undesirable.

3.1 The Need for Decoupling Capture and Display It is a natural requirement that any captured—or created— content should be displayable. This is assured by defining a common format at both lower and higher levels, so that the capturing side stores or delivers content in that particular format, and the displaying side reads and renders the content accordingly; however, it is highly desirable to match a wide range of possible content formats to a wide range of different display structures. This may look easy at first glance, but it actually is not so—even for classical 2D video. Analog TV has been operational in compliance with strict delivery standards

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

25

such as NTSC, PAL, and SECAM in different parts of the world. Display formats have significantly changed as the CRT-based analog displays were quickly replaced by digital LCD or plasmabased monitors. This also triggered different scanning formats for the display (for example, interlaced and progressive at different speeds). The delivery of color (YUV, RGB, etc.), the geometry of the pixel at the micro level, the number of pixels per line, and the number of lines, aspect ratio, etc., have gone through significant changes. The content for conventional TV has already deviated from a single standard analog TV format: many digital TV stations now transmit different frame sizes (standard definition versus high definition digital TV). TV receivers can adapt to different aspect ratio frames received via analog transmission, as well as different forms of digital content. The ability to display different forms of captured content on different forms of display units is also highly desirable for the future of 3DTV delivery. Content intended for cinema will also require this ability. This adaptability issue is a much more severe problem for 3D than conventional 2D, because even some basic forms of adaptability that are so trivial and natural for 2DTV do not translate as easily in 3D. One example is the different physical sizes of TVs that are used in homes: provided that the TV sets have the same number of pixels, the simple change in physical size of a pixel means physically smaller or larger picture sizes on different TV sets; however, different physical sizes for stereoscopic TVs results in distorted 3D perception and perspectives. Therefore, a need—as well as a challenge—is to develop a common 3D video format that supports a wide range of input (capture) and output (display) devices and couples them as end units via, for example, the Internet. It is highly desirable to relieve any capture unit and any display unit from stringent constraints; this is at the heart of decoupling capture and display. Such decoupling is also the basis for accommodating future developments in the field. As outlined in Chapter 2, 3D imaging types are quite different, and each demands a totally different physical structure for the input and the display equipment. It  

26

Chapter 3

would be undesirable to come up with an end-to-end 3DTV system that can function with only one set of input and display modes with rigid parameters. An ultimate decoupling might not be possible, but it is believed that a high degree of decoupling may be achieved.

3.2 State-of-the-Art in 3D Video Capture Precise acquisition of 3D information associated with 3D dynamic scenes is crucial. There has been significant research in capturing, processing, and, if needed, analysis of 3D scene information.3,5,6 Different methods have emerged as a result of recent advances in image sensor technologies and the wider availability of massive computation power. 3.2.1 Single-camera techniques These techniques have been well known in computer science for decades.3 Despite this, the capture of 3D content with one stationary 2D camera is fundamentally an ill-posed problem. Therefore, many assumptions about the captured object and the scene must be made for a successful result. These techniques are generally named as ‘shape-from-X,’ as in shape-from-shading, shape-from-texture, shape-from-defocus/focus, and shape-frommotion.3 Each of these techniques has its own advantages and problems.3,5,6 Shape-from-motion, which involves both object motion and camera motion, is generally accepted as a better method than others in terms of the accuracy of the captured 3D information. While other techniques are usually limited to operating in controlled environments for successful results and are probably not suitable for practical 3DTV, the shape-frommotion technique is not a universally valid solution, either, since it cannot capture 3D information under all conditions. This should be expected, and indeed, it is unreasonable to expect to solve the problem of 3D capture using a single camera unless there is a sufficient amount of motion that will reveal an adequate amount of 3D information. However, these techniques

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

27

are still the only ones available to convert conventional existing 2D content to 3D. Such conversion methods are highly desirable for adapting the existing video content to future 3D video environments. It is desirable to automate the procedure as much as possible, but it looks as if some human intervention is inevitable even though currently most sophisticated methods are adopted. There are companies within Europe and elsewhere that provide conversion service from existing 2D to 3D. Such services do employ recently generated technologies that automate the process to a certain extent. However, it is understood that almost all of those techniques, except under very restricted conditions, do require human intervention and input during the conversion and therefore, the task is still labor intensive. There has been recent activity in this field within the EC-funded FP6 projects, and successful results have been reported. 3.2.2 Multicamera techniques It appears that the most equipped technology for the capture of 3D content in the near future (the next few years) will be based on multicamera systems. These systems use conventional 2D cameras, but many of them shoot the same 3D environment from different positions simultaneously. A multicamera videocapturing setup is shown in Fig. 3.1. The simplest form is a linear array of identical cameras all in a parallel orientation. More complicated settings have different cameras (different zoom and lens properties, etc.) facing the same 3D scene from different angles. Omnidirectional cameras are also used. One major issue in such systems is the calibration and synchronization of these cameras. Both geometric and photometric/radiometric camera parameters vary significantly, not only from one camera to another—even if they are the same brand and model—but also for the same camera over time. Calibration is usually accomplished by posing a known 3D pattern to all cameras simultaneously before each shot. This  

28

Chapter 3

Figure 3.1 A multicamera video capturing setup. (From K. Mueller, A. Smolic P. Merkle, M. Kautzner, and T. Wiegand, “Coding of meshes and video textures for 3D video objects” Proceedings of the Picture Coding Symposium, 2004. Reprinted with permission.)

reference recording is then used to process each captured video and store or deliver them after conversion, a process which is usually called rectification. Alternatively, this reference recording is stored and delivered together with the raw data to give the receiver a chance to perform its own correction. The correction involves not only different zoom factors, lens mounting inaccuracies, lens aberrations, etc., but also color mismatches, light detection levels, etc. A line of research is dedicated to generating so-called “freeviewpoint video” from captured multicamera recordings. Such techniques are also called “virtual camera” methods. The goal is to synthesize a completely new video from the captured ones that mimics a recording from a virtual camera whose location, angle, and other parameters are different from the physical cameras that recorded the original videos. Finding the matching points in parallel video frames is a key research activity associated with multicamera techniques.

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

29

Construction of depth information, and thus formation of a 3D video from multiple 2D recordings, requires those correspondences to be accurate. Many techniques have been developed for searching a video frame from one camera for a match to a given small part of a captured scene in another camera.4 Such a search is also used in single-camera techniques among subsequent frames. One supporting technique, both for single- and multicamera techniques, is to project a controlled pattern on the 3D object/scene being recorded.3 Usually, fencelike patterns of known color and geometry are projected, and the scene is then recorded. The deviation of the projected curves from the original pattern is then used to extract the 3D shape information. Since the outlined multicamera techniques make up the dominant 3D-capture mode at present, and will continue to do so during the next few years, there are many companies and research institutions with products in this field. These are mainly in Europe, as well as in North America, East Asia, and Australia. Some of these activities are the outcomes of the EC-funded research within the Fifth Framework Programme (FP5) and the Sixth Framework Programme (FP6); these activities are also continuing within the Seventh Framework Programme (FP7) projects. It is relevant to mention that multicamera techniques have also been used for many years in conventional 2D cinema to achieve unusual effects during postprocessing. Such techniques have demonstrated commercial successes, supporting the interest in multicamera systems for 3D applications. 3.2.3 Holographic capture devices State-of-the-art in holographic capture is based on utilization of the largest possible CCD arrays to capture the interference fringes.3 As the size of the CCD chip increases, the quality of the 3D reconstructions from such images also improves. Ongoing research activities in Europe, North America, and East Asia employ multiple CCD arrays at different positions in order to  

30

Chapter 3

achieve effectively larger arrays. There are holographic microscopic 3D capture devices that are commercially available from a company in Switzerland. Related research investigates other fundamentally different techniques to capture holograms of dynamic scenes. One such study is on polymer-dispersed liquid crystals, and the target is to produce rewritable photosensitive films to capture holographic video. 3.2.4 Pattern projection techniques Projection of structured light patterns onto 3D objects to capture shape information is a known practice, as shown in Figs. 3.2(a) and (b). Lately, techniques that use sophisticated patterns and

Figure 3.2(a) A pattern-projection-based video capturing setup. (From T. Bothe, A. Gesierich, W. Li, C. v. Kopylow, N. Köpp, and W. Jüptner, “3D Camera for Scene Capturing and Augmented Reality Applications,” 3DTV-CON 2007, © IEEE 2007. Reprinted with permission.)

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

31

Figure 3.2(b) The resultant depth image. The color map (side bar) indicates the depth value where blue represents far and yellow represents near. (From T. Bothe, A. Gesierich, W. Li, C. v. Kopylow, N. Köpp, and W. Jüptner, “3D camera for scene capturing and augmented reality applications,” 3DTV-CON 2007, © IEEE 2007. Reprinted with permission.)

subsequent computation have been developed for smaller objects as well.3 For example, one technique employs color patterns composed of many monochromatic light components. Extraction of phase, utilization of phase-retrieval techniques, and specific noise removal steps are some recent advances in this area.3 3.2.5 Time-of-flight techniques Time-of-flight methods are used to directly measure the depth variations in a scene. The principle is based on older techniques used in radar and lidar.3 Typically, a modulated light source illuminates the scene, which reflects back from it. The reflected light is received by a sensor pixel array, and based on some  

32

Chapter 3

parameter variations of the received light, the distance variations (i.e., the depth information) are computed and recorded.

3.3 State-of-the-Art in 3D Representation Techniques A 2D still image is naturally represented as a 2D array of pixels that can be stored and processed by a computer. The representation for a 2D video is just a sequence of consecutive frames. Unfortunately, there is no such natural representation for 3D scenes. A generic representation is highly desirable for many reasons. One of those reasons is the decoupling of input and display phases as outlined in Section 3.1. Simply, the input data is converted to this abstract generic form that carries complete data about the 3D dynamic scene, and this data can be stored, transmitted, or further processed. On the display side, this generic form is mapped to the rendering device, which would generate the specific form of data for that particular 3D display device. Such generic 3D dynamic scene representations are usually based on computer graphics techniques.7–9 An ideal representation format should possess some desirable features, such as universal applicability to handle arbitrary geometries and topologies and preservation of image quality and accuracy. Furthermore, it should support progressive resolutions, including lower-level details. Since it is common to postprocess captured content, a versatile representation format should support high-level authoring tools for modifications.7 Some 3D representation techniques are presented below. 3.3.1 Dense depth representations One technique—which is also adopted in the computer graphics extension of MPEG-4—called animation framework extension (AFX), represents the 3D scene as a collection of many images where each such image has two components: one of these is the conventional color picture, called the texture image, and the second accompanying image is the depth map.7 Therefore, every

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

33

pixel has a color and a depth. The collection of such image pairs provides a complete description for the entire 3D scene (Fig. 3.3). 3.3.2 Surface-based representations Polygonal meshes are the most common form of surface representations.7 In a mesh format, the 3D scene is typically represented as a collection of 3D vertex coordinates and a list of edges that connect these vertices to describe a surface that consists of planar patches. As the number of such patches increases, the surface fits better to the usually smooth 3D scene surface. Most of the state-of-the-art mesh representations are progressive meshes where a coarser representation is refined by adding finer and finer stages.7 A surface representation technique that is successful for still 3D scenes might not be appropriate for time-varying scenes. Different, more appropriate variants for dynamic 3D scenes are proposed. A surface representation technique, called NURBS, is similar to a higher-dimensional version of conventional spline representation for 1D curves and fits smooth parametric surfaces based on restrictions imposed by control points.7 There are

Figure 3.3 Dense depth representation: (a) the texture image, (b) the depth image (nearer pixels are darker). (From C. Cigla and A. A. Alatan, “Depth assisted object segmentation in multiview video,” 3DTV-CON 2008, © IEEE 2008. Reprinted with permission.) (“Uli Sequence” is courtesy of Fraunhofer Institute-HHI.)

 

34

Chapter 3

techniques that allow for arbitrarily fine resolution by providing smoothness at all levels; one such technique is called subdivision surfaces.7 3.3.3 Point-based representations In point-based representations, the 3D scene is represented simply by a set of points that are obtained by sampling the underlying surface at discrete 3D points.7 Each point may carry the color and surface normal data at that sampling coordinate. No other connectivity or topology information is stored. The sampling scheme may or may not be uniform; however, uniform grid sampling is easier to handle. Such point-based representations are also called point clouds. 3.3.4 Volumetric representations In a sense, this representation is the direct counterpart of pixel representation for the 2D images: now there are voxels instead of pixels.7 A voxel is simply an element in a 3D regular grid over the 3D space. Empty voxels are transparent (carry no data). The voxel corresponding to a surface point has the same properties such as intensity and the color of the light from that surface point. Such a representation is simple and accurate but not efficient, since all voxels in a 3D volume should be stored. However, immediate compression can be achieved if voxels are represented in a hierarchical manner by grouping finer empty voxels into a coarser empty voxel (octree representations). 3.3.5 Texture mapping A 3D object is described by its shape, together with its color and other optical surface properties covering the shape. Optical properties of such a ‘skin’ of a 3D object are called its texture. It is quite common to represent the shape and the texture separately. Therefore, appropriate representations for the texture are also needed. Single-texture representations provide a single 2D image that represents the entire surface texture of the 3D object.7 This is

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

35

achieved by defining a mapping from the surface coordinates to the 2D texture image. At the rendering stage, the 2D texture image is mapped back to the 3D surface, i.e., the surface is wrapped with the texture. Multiple-texture techniques have also been developed.7 Such techniques decompose the single texture described above into a number of separate 2D texture representations. Such a separation may, for example, yield one texture image for the intrinsic optical properties of the object surface, and produce another texture image that represents the illumination that falls onto the object. Further separation for different effects and properties is possible. 3.3.6 Pseudo-3D representations There are some forms of descriptions with limited 3D capabilities that still give a sense of depth. Such methods can be grouped under pseudo-3D representations. One commonly used example is the packing of many 2D objects over each other at different depths. There are studies that indicate that the human visual system is quite coarse in depth selectivity and therefore, a coarse depth quantization may still work. Even though the 3D scene and the associated parallax is not correct in such representations, the depth perception for different objects as a consequence of rendering of the planar packings of 2D objects at different depths on a 3D display unit may be still satisfactory. Such representations may be quite suitable for cinema. The algorithmic and computational burden of such representations, both during content creation and during storage, transmission, and display, are much less compared to other methods that provide closer representations to true 3D. 3.3.7 Light-field representations A true-3D representation for a scene cannot be achieved simply by taking the structure and optical surface properties of the scene, but by providing a quasi-physical description of light that  

36

Chapter 3

fills the entire 3D space. Such representations are called lightfield representations. A typical light-field representation is a 4D function that indicates the color-separated light power from any point over a plane to another arbitrary point on another parallel plane. Eventually, the light field-representation can be described as the complete knowledge of color and directional distribution of light rays that cross an imaging surface subtending the 3D volume. Together with the direction of propagation, such a representation is a complete 3D description, and all views with any parameter can be constructed from such data except the occluded parts when viewed from the plane that subtends the 3D volume of interest. Such a representation is highly desirable since it also links the various 3D technologies that are outlined in Chapter 2. This includes multiview techniques as well as more directly physically based integral imaging and holographic approaches. However, such a rich representation might get prohibitively computation intensive, and thus, is usually avoided. 3.3.8 Object-based representations Any 3D representation restricted to a particular known object, or a collection of such objects, might benefit from associated knowledge. For example, if a 3D representation is solely designed for teleconferencing, the knowledge of the shape, flexibility, physiological structure, etc., of the human torso can be blended with the algorithms for increased accuracy and efficiency.7 Such approaches will lead to more successful representation, animation, and rendering. Additional constraints based on the shape of the human skeleton, its mobility, and the physiology of muscles and other tissue attached to the skeleton will result in more effective representation. Similarly, a 3D representation for an urban area could benefit from the associated assumptions that use our common information regarding buildings, roads, cars, etc.7 Furthermore, physically modeled objects can be more successfully animated by considering their weight, elasticity, layers and couplings of materials, along with the laws of physics.

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

37

3.3.9 Standards for 3D scene representation When considering the future of the Internet, one of the main issues is the seamless connection of content providers and content consumers to this mutual delivery infrastructure, which is only possible if commonly accepted and used standards are in place. Various standardization bodies, such as ISO, ITU, and IEEE, have worked on related standardization topics and have issued many recommendations that have already been adopted by the industry. The main components of the 3D video chain that will benefit most from the standardization are the representations that are summarized in this section, along with the coding, compression, and transmission, which are dealt with in Sections 3.4 and 3.5. Some already-existing standards that are readily applicable to 3D video are outlined herein. One such ISO standard for 3D scene representation is a description language called virtual reality modeling language (VRML), which was released in 1997.7 However, it has limited real-time capabilities and therefore, is not quite suitable for interactive communications. A successor is the so-called Extensible 3D (X3D), which was also released by ISO and developed essentially by the Web3D Consortium.7 MPEG-4, with its extensions such as AFX, provides a very powerful basis for multimedia representations, which integrate natural video, audio, and 3D computer graphics. The 3DAV group of MPEG covers the extensions needed for a successful 3D audio and video support.

3.4 State-of-the-Art in 3D Video Coding Techniques Even though an ultimate intermediate 3D representation is needed to effectively decouple content creation and consumption (as described in Section 3.1), the current status is far from this ideal case and therefore, different coding techniques have been developed and are used for different kinds of captured data.10–12 In this section an overview of the status of coding and  

38

Chapter 3

compression of input data related to different 3D capturing modes is provided. 3.4.1 Stereoscopic video coding Stereoscopy is outlined in Section 2.1. Simply, it is a special case of multiview video with the number of views equal to two. Compression of stereoscopic video has been studied, and standards have been established for this purpose.10 Since the two images of a stereo pair are very similar to each other, i.e., there is a high level of redundancy between the two images, compression of such pairs is quite successful. Therefore, in addition to the temporal redundancy that prevails between consecutive frames, now there is additional redundancy between the two parallel video streams that represent the right- and left-eye views. Essentially, the redundancy between the right and left views is exploited by predicting one of these images from the other one; the prediction error is then coded. The prediction problem is essentially the same as in temporally related frames of conventional 2D video; instead of using dense motion vectors between consecutive frames, now the dense disparity vectors between stereo pairs are used. The way to find and code the disparity vectors could be exactly the same as finding the motion vectors; however, more efficient techniques can be employed by observing that disparity vectors possess different characteristics in comparison with motion vectors. For example, disparity vector directions are restricted to lie on the epipolar line; this makes finding the vectors and compressing them easier. On the other hand, disparity vectors may have quite large values more frequently than motion vectors. Another issue is the occluded parts, which cannot be predicted and so must be coded in a standalone manner. Balancing of color and lighting differences in the two captured images is another concern during compression; however, the basis of coding is the exploitation of temporal and interview similarities within a stereoscopic video sequence. A corresponding standard specification was defined in ITU-T Rec. H.262/ISO/IEC 13818-2 MPEG-2 Video Multiview Profile in 1996; essentially, the left and right video frames are

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

39

interleaved and encoded as a single video stream.10 It is quite common to code the left-eye view in a standalone manner using a standard 2D video codec. This ensures backward compatibility with, for example, the Main Profile of H.262/MPEG-2 Video, since it is possible to decode only the left-eye-view bit stream and to display it as a 2D video. The right-eye view is then coded by using interview prediction and using the already coded lefteye view as a reference. However, compared to independent compression of left- and right-eye sequences, the gain is rather limited due to the successful standalone temporal coding techniques.10 A stereo codec that utilizes both temporal and left– right view redundancies is shown in Fig. 3.4. 3.4.2 Multiview video coding Multiview video is outlined in Section 2.3 as a video capture modality for 3D video. One immediate coding possibility is to code each such 2D video independently using one of the well developed 2D video coding techniques such as H.264/AVC. However, such an approach will miss the possibility to use interview dependencies; redundancy between views is high since frames from neighboring cameras are highly similar. Therefore, a straightforward, more efficient alternative is to use interview prediction as well as temporal prediction. A multiview video coding (MVC) standard has been recently published by MPEG after years of exhaustive experiments. The standard is based on so called hierarchical B-pictures as supported by H.264/AVC syntax, both for temporal and interview interactions. It is seen that MVC outperforms independent coding of multiple video streams, but the gain depends on content, camera settings, and properties.10 Other techniques for compression of multiview video have also been proposed. One such technique is view interpolation based on depth estimation from the disparity data. It is observed that the gain over MVC is marginal.10 Novel techniques that exploit the statistical properties of disparity data, compared to motion vectors, have the potential to improve the compression  

40

Chapter 3

efficiency. ISO/MPEG and ITU/VCEG have recently jointly published a recommendation for MVC as an extension of H.264/AVC (Amendment 4).10

Figure 3.4 Block diagram of a stereoscopic video encoder and decoder. (From A. Aksay, C. Bilen, E. Kurutepe, T. Ozcelebi, G. B. Akar, M. R. Civanlar, and A. M. Tekalp, “Temporal and spatial scaling for stereoscopic video,” EUSIPCO 2006. © EUSIPCO 2006. Reprinted with permission.)

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

41

3.4.3 Video-plus-depth In video-plus-depth data representation, each pixel of data consists of a color and an additional depth value. The depth value is also quantized and is restricted between maximum and minimum limits. Such data carries information about the 3D scene and therefore, conversion between different data formats, such as from video-plus-depth to multiview video, is possible. Depth data quantized using eight bits (256 levels) for each pixel is identical to a conventional 2D black-and-white frame in terms of data structure, and as a result, it can be compressed in a similar way using state-of-the-art video codecs.10 It is shown that only about an additional 10–20 percent of bits are sufficient in order to add the depth data onto conventional color video with compression. Backward-compatible video-plus-depth bit streams, which can be decoded by an MPEG-2 decoder as a classical color video by omitting the additional depth information, are shown to be feasible.10 MPEG specified a format “ISO/IEC 23002-3 Representation of Auxiliary Video and Supplemental Information” (MPEG-3 Part 3) for video-plus-depth data. Similarly, H.264/AVC contains an option to transmit the depth images through its auxiliary picture syntax.10 3.4.4 3D mesh compression Even though it is a novel compression mode compared to the above-mentioned coding procedures, there is plenty of literature on static and dynamic compression of meshes that represent 3D objects and scenes as described in Section 3.3.2.10 Predictability of unknown vertex locations from the known ones is the key approach in many schemes. A layered approach has also been demonstrated to be feasible; such an approach will allow hierarchical delivery of coarse-to-fine content (Fig. 3.5).10 MPEG-4 3D mesh coding standards are being developed (3DMC tools).10  

42

Chapter 3

Figure 3.5 A frame from an original mesh-based object video (top) and coding errors using different codecs for mesh-based compression at the same bit rate (center and bottom). (From A. Smolic, K. Müller, P. Merkle, M. Kautzner, and T. Wiegand, “3D video objects for interactive applications,” EUSIPCO 2005, © EUSIPCO 2005. Reprinted with permission.)

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

43

Even though most of the 3D mesh coding techniques are based on vertex and connectivity representations, there are reported methods that utilize other representation schemes as well. For example, geometry-based coding, which is based on a voxel representation (Section 3.3.4), is built upon the statistical observation that there are large volumes that are empty in most scenes.10 There are also methods that convert the mesh data first into 2D data, which is similar to 2D video data in format, and then employ a state-of-the-art 2D compression algorithm. An example of such algorithms is the projection of height data onto a 2D grid as a black-and-white picture before a conventional 2D compression. Compression of dynamic meshes exploits the predictability in both temporal and spatial domains simultaneously.10 Either transform-based or prediction-based techniques are used. Trajectories of vertices in time constitute the input data in some compression schemes. Wavelet-based approaches have also become popular. Interpolation between anchor frames to obtain predicted meshes is another approach, and such methods are already adopted in MPEG-4 (AFX-IC). Both non-linear and linear predictors have been proposed.10 3.4.5 Multiple-description coding Joint source-channel coding approaches are also applied to 3D video coding. For example, multiple-description coding (MDC) is an error-resilient joint-source-channel coding approach.10 The data source is encoded into separate bit streams, where each such bit stream is independently decodable. As these separate descriptions are received, a single recovery is obtained by fusing the results of each description. Therefore, each received description refines the result. The overall bit length is generally longer than other techniques as a consequence of deliberately added redundancy between the descriptions; but, in turn, more robust, error resilient end-to-end operation is achieved. MDC procedures have been applied to single video, as well as to multiview video (Section 2.3) and mesh data (Section 3.3.2).10  

44

Chapter 3

3.5 State-of-the-Art in 3D Video Streaming Techniques The evolution of 3D video transport technology follows the same path as in its 2D counterpart: analog broadcast, digital broadcast, and finally, streaming over the Internet Protocol (IP).13–15 There is no doubt that streaming over IP provides a more flexible means of delivery. Widespread opinion indicates that the success of 3DTV is initially based on its delivery in a backwardcompatible manner to conventional 2DTV. 3.5.1 Analog broadcast The first known experimental analog broadcast of stereoscopic 3DTV was in 1953.13 However, the first commercial broadcast occurred almost 30 years later in 1980. An experimental broadcast of such a stereoscopic 3DTV in anaglyph format took place in Europe in 1982.13 Since then, there have been many more such 3DTV broadcasts, but each was limited to a single movie or event, and was conducted occasionally and sporadically. Demonstrations of stereoscopic 3DTV broadcast almost always took place in major trade shows and exhibitions. 3.5.2 Digital broadcast As the analog-to-digital transition started in 2DTV broadcast in the early 1990s, the interest in stereoscopic 3DTV increased. Many groups, including many European-funded projects proceeded to develop standards, technologies, and production facilities for 3DTV and 3D cinema. Together with emerging 3Dvideo-related compression standards (Section 3.4) such as MPEG-2 multiview profile (MVP), these activities facilitated digital transmission of 3DTV.13 Live stereoscopic HDTV broadcasts of popular international sports events took place in Japan in 1998 and in Korea in 2002.13 Recent developments in the cinema and broadcast industries indicate a strong inclination toward stereoscopic 3D. Well-known movie directors and producers are now targeting 3D content as digital projection in movie theaters is becoming a standard procedure, and major

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

45

commercial broadcasters now routinely broadcast in stereo, especially when airing major sporting events. Commercial products in stored media (discs, for example) are also becoming available in video-plus-depth (Section 3.4.3) format. It has also been demonstrated through using H.264/AVC that video-plusdepth data can be successfully transmitted where the additional capacity requirement due to depth data is only about 10–20 percent.10,13 3.5.3 3DTV-over-IP networks As more and more services are provided through IP networks every day, there is an increasing trend for video delivery, both for broadcast and interactive applications via such infrastructure. Video-on-demand services over the Internet are becoming popular both for news and entertainment. Mobile network operators are also utilizing IP for wireless video services. There is no doubt that delivery of 3DTV signals over IP networks is a natural choice. IP, together with other layer protocols, offers a flexible design for communications systems. Consequently, improvements for specific needs of 3DTV signals are feasible.13 Some feasible near-future scenarios include unicast and multicast transmission, where peer-to-peer, as well as server-toclient delivery, are considered. The state-of-the-art protocol is RTP/UDP/IP; however, it is expected that RTP/DCCP/IP will take over in popularity.13 3.5.4 Streaming protocols The most widely used protocol today is RTP over UDP. The lack of congestion control mechanisms in such protocols creates problems, especially when large volumes of video data are considered. For this reason, DCCP is a more desirable alternative. It is possible to implement bidirectional unicast connections without reliability using DCCP.13

 

46

Chapter 3

3.5.4.1 Multiview video streaming

As in other forms of large data delivery via the Internet, the adopted protocols should allow adaptation based on the congestion parameters and must be friendly to the rest of the traffic. Such an adaptation control is different for multiview video compared to conventional 2D video.13 The main advantage for the multiview case is based on studies on human perception of multiview quality. There are many studies and applications that indicate that the human visual system is highly sophisticated while fusing the right- and left-eye views. For example, if one of the views is blurred, the overall perception is still sharp; in other words, the blurred vision via one of the eyes is compensated for by the sharp vision through the other eye. Such observations prompt the congestion control, as well as the coding algorithms outlined in Section 3.4, to shift the bit allocation in a more flexible manner among the multiple views in a multiview environment.13 An example of such a codec, where the right view is blurred compared to the left view due to downsampling, is shown in Fig. 3.4. The slowing down in the bandwidth of one or more views might not cause significant degradation, provided that some other views are delivered almost perfectly. Several such adaptation schemes for multiview video delivery are reported in the literature for UDP and DCCP protocols. Typical adaptation procedures are based on spatial subsampling, temporal subsampling, quantization step alterations, and contentbased adaptation.13 3.5.4.2 Error correction and concealment

Due to limited bandwidth, especially over wireless channels, packet losses are inevitable in streaming media applications. The way to handle such losses depends on the details of the implemented protocols. There are joint-source and channelcoding strategies developed for video streaming over channels where packets may be lost (Section 3.4.5). In addition, error concealment methods are developed at the receiving side to limit the observed damages due to the losses.13

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

47

Two common methods for such correction actions are retransmission requests (ARQ) and forward error correction (FEC). FEC techniques are reported for stereoscopic video.13 Some of the techniques developed for 2D video are not directly applicable to 3D. For example, a lost patch of a frame is usually interpolated from a priori knowledge and from the correctly delivered surrounding data. Such an interpolation for the stereoscopic case will not be successful if the two videos are processed independently; instead, an interpolation based on the underlying 3D structure is needed.13 3.5.4.3 3D video-streaming experiments and demonstrations

There are several end-to-end 3D video-delivery systems reported in the literature (see Section 3.7), and an illustration is presented in Fig. 3.6. A multicamera input system coupled with a multiprojector 3D display system, with a broadband network in between, was demonstrated by Mitsibushi Electronic Research Laboratories (MERL). Another end-to-end prototype for pointto-point streaming of stereoscopic video was developed within the EC-funded 3DTV Project and demonstrated in various exhibitions. The prototype was continuously upgraded: its initial versions operated over a LAN with no packet losses, whereas later versions have been demonstrated to operate over the Internet between locations across Europe with full errorconcealment protocols implemented. The server operates on RTP/UDP/IP protocols and can serve multiple clients at the same time. Another demonstration was the VIRTUE 3D- videoconferencing system, which was based on delivery of videoplus-depth (Section 3.4.3) over IP; the coding was based on MPEG-4 techniques, and the streaming was accomplished by using RTP/UDP protocols.13

 

48

Chapter 3

Figure 3.6 Block diagram of a 3D streaming system. (From A. M. Tekalp, E. Kurutepe, and M. R. Civanlar, “3DTV over IP,” IEEE Signal Processing Magazine 24(6), 77, 2007. © IEEE 2007. Reprinted with permission.)

3.6 State-of-the-Art in 3D Video Display Techniques As stated earlier, 3D displays have a long history dating back to the 19th century.2,16 Stereoscopic photographs arose immediately after photography was invented. Similarly, stereoscopic movies and stereoscopic TV came about quite rapidly following the inventions of their 2D counterparts. The display is the crucial element in the 3D video chain, and it is the direct visible component in almost all applications such as entertainment, scientific visualization, telepresence, medical imaging, and gaming. There are many different methods for displaying 3D video; however, none of these methods are yet a consumer item with the exception of simple stereoscopic 3D television sets, which only recently emerged. Stereoscopic display devices that can be watched with the help of eyewear have been known for about 170 years. The rightand left-eye views are projected onto the same screen either simultaneously or in a time-multiplexed manner, and a filter in the form of eyewear separates two such images and allots them to the corresponding eye. Earlier methods were mostly based on

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

49

color filtering, but later methods utilized polarization or shutter glasses. (See Chapter 2 for further description of stereoscopic technology.) The issues associated with the display itself are rather easy to handle: the colors must have the desired spectrum to match the eyewear filters in anaglyphs; the screen should not alter the polarization properties of incident light in an uncontrollable manner in the polarization-based methods. The persistence on the screen should match the intended refresh rates in a multiplexed system. Amount of brightness and flicker are other parameters of concern. These are not severe issues for projection systems as in cinemas, or on older CRT-based TV monitors. However, LCD-based systems may not be easily adopted to polarization-based viewing since the intrinsic operation of LCDs alters the polarization properties of each pixel over the screen. There are reported works that examine different types of pixellated TV (or computer) monitors for suitability for particular eyewear-based stereoscopic 3D viewing. Current focus is on proper adaptation of content to a particular geometry (size and aspect ratio) of the screen and the viewer position, and so it is not directly related to the display itself, but rather to what is projected on it. Therefore, attention is shifted to other novel display systems than eyewear-based stereoscopic viewing. Novel 3D display systems are classified into three categories as (a) autostereoscopic displays, (b) volumetric displays, and (c) holographic displays.2 Literally speaking, all three of these systems are autostereoscopic viewing systems, since none of them require eyewear.2 However, it is common to reserve the term “autostereoscopic” for those systems that yield different simple 2D views at different angles; the same convention is followed here. On the contrary, volumetric displays are physically 3D volume screens, in a sense, where a simpler screen (2D or even 1D) sweeps across a volume in a fast and repetitive manner. Holographic displays attempt to physically generate optical wavefronts instead of isolated simple 2D views. The human visual system should be well understood in order to construct successful 3D displays. There are many physiological cues that must be presented to the visual system  

50

Chapter 3

for comfortable 3D viewing; some of these are binocular disparity, motion parallax, ocular convergence, and accommodation.2 Any stimulus that creates conflicting cues will immediately result in severe viewer discomfort; this is usually similar to the feeling of motion sickness. Other cues include the correct perspective, shading, shadowing, interposition, retinal image size, texture, and color.2 Cue mismatch creates more severe discomfort as the depth of the intended 3D image covers a larger range. Among the mismatches, the most dominant ones in terms of discomfort severity are vertical disparity mismatch (which forces one eye to look up and the other one down in a totally unnatural way) and crosstalk between the right and left images. 3.6.1 Multiview displays A 3D display procedure is expected to produce a large set of 2D views and somehow project each such 2D image in the direction it would be seen from. Thus, different views are seen from different directions. If the views can be closely packed so that a new view can be seen by a single eye even for small viewing angle changes, there will be a pleasant 3D perception with a smooth variation as one moves around while looking at the display. The success depends on the angle and distance adjustments so that each eye receives a different view among the large set, as in the case of looking at an actual 3D scene. Another factor affecting the success is the degree of crosstalk among the views; ideally there should be no crosstalk at all. There are many designs that provide a 3D display unit based on the principles described above.2 However, the number of views from different directions is limited for practical reasons; this results in a so-called jumping artifact: as one moves, the scene changes abruptly instead of smoothly.2 In addition, some crosstalk usually exists, especially in autostereoscopic displays, further degrading the quality.2 The simultaneous delivery of different 2D views to different directions is achieved by different techniques, and actually, each such technique is the differentiating characteristic of each

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

51

design. In some implementations, moving slits are used in front of rapidly displayed video (or cinema) frames. In integral imaging, the projection of different views to different directions is achieved by a large array of microlenses and a large 2D display device that can simultaneously concatenate a large collection of many 2D elemental images into one single array in such a way that each lens receives light from one elemental image during projection. Holographic stereograms are based on another technique that can provide different views at different viewing angles. However, the projection angles are usually limited to a horizontal direction with accompanying vertical-only diffusers, and thus only horizontal parallax is provided. The most popular, and rather inexpensive, multiview 3D displays are based on pixellated monitors that are covered with lenticular sheets, where the cylindrical lenses on the sheets are either vertical or slanted. There are many pixels per lens. Slanted lenticular displays more evenly distribute the blurring effect horizontally and vertically. The blur is due to a reduced resolution-per-view that is a consequence of displaying many views simultaneously using a single monitor. The packing of many views into a single monitor is different for each design and depends on the geometry of the lens array, monitor, and their relative positions; such techniques, which generate a single fused 2D image pattern from many frames, are called interzigging. Commercial lenticular designs are usually limited to a few horizontal views—typically four to nine. Light sources that illuminate the LCD panels may also be altered to yield better 3D viewing in lenticular designs. Two-view (stereoscopic) displays are simpler special cases of multiview displays. Commonly used techniques are lenticular screens, twin projectors, parallax barrier methods, holographic optical elements, and prismatic screens.

 

52

Chapter 3

3.6.2 Head-tracking displays Autostereoscopic viewing with nonadaptive display systems, as explained above, is possible if the viewer is in the “sweet spot.” Adaptive display systems are designed to overcome this problem: the viewer’s eye locations are detected by a supplementary system and are dynamically fed back to the display system to alter some of the features that control the direction of the simultaneous separate views. Thus, the limited viewing position is significantly expanded. Furthermore, it is possible to provide more than one pair of stereo views for multiple viewers located at different positions simultaneously. Head tracking in addition to—or instead of—eye tracking is also used. Tracking is usually achieved by video-based systems or with the help of inertial measurement units (IMU).2 3.6.3 Volumetric displays A visible surface in a 3D space is either a reflector of light that illuminates it, or it is self-luminous. Reflection properties are also important: matte surfaces are more easily perceived when compared to shiny ones. A surface may be modeled as a continuum of tiny light sources where each has a 3D location and color. Volumetric displays attempt to duplicate the original surface properties by placing equivalent light sources in the same (within a geometric scale) positions. This is achieved in various ways. These techniques are usually classified into two groups: virtualimage systems and real-image systems.2 There are virtual-image-based systems using either mirrors or lenses with controllable focal length. As the focal length is altered, the image distance changes, and a perception of depth is achieved.2 Real-image systems are based on sweeping devices that move in a volume with a faster speed than the human visual system can follow.2 Either passive screens are vibrated while being illuminated by a time-varying calibrated projection, or active (such as LED arrays) devices are moved in a volume in a

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

53

cyclic manner as their light intensities and colors are modified. In either case, a smooth volume-filling 3D object is observed. There are many commercial volumetric displays available on the market. Since the illuminated points have true-3D positions, there is no focus–convergence mismatch in volumetric displays. A largeangle and true-perspective viewing is possible. However, the major drawback of volumetric displays is the lack of occlusion: rear surfaces are also visible behind the front surfaces. The mechanical size and nature of volumetric displays are usually annoying. 3.6.4 Holographic displays Holograms are nothing but interference patterns that diffract incoming light according to the shape of the pattern. Therefore, a holographic display is simply a device on which such a pattern can be written. Still recordings are “write-only” devices, whereas holographic video or cinema requires either rewritable devices or a sequence of still recordings. Conventionally, still holograms are recorded on photographic films. There were also experimental holographic cinema systems where a sequence of many still frames was used as in conventional cinema. Recent advances in electronics prompted the development of many experimental holographic video systems based on spatial-lightmodulator arrays. A holographic recording is the best 3D display, since it yields the actually recorded light field, but the underlying fringe patterns are spatially very high-frequency patterns and therefore, require very high-density patterns to be recorded. Typical holograms can have 1000 lines per millimeter. Considering the desired size of the 3D image, the number of pixels in a pixilated device has to be very large, and the currently available spatiallight-modulator array sizes are inadequate. Acousto-optic devices may solve the resolution issue, but they still have problems. The speckle that is associated with coherent imaging is a drawback; using LEDs instead of lasers during holographic  

54

Chapter 3

reconstructions diminishes speckle, but may bring about other concerns. There are experimental holographic video devices (see Fig. 3.7). It was recently reported that multiple SLMs can be simultaneously illuminated to get color 3D video reconstructions.17 The size of the reconstructions is on the order of a few square centimeters. 3.6.5 Signal-processing issues associated with holographic displays The fringe patterns that carry the 3D scene information in the form of a 2D pattern may be directly recorded using the holographic capture techniques outlined in Section 3.2.3. However, an alternative is to compute such fringes directly from a given 3D scene. 3D scenes may be captured as described in Section 3.2 and then represented in an abstract manner as outlined in Section 3.3. Such a representation can then be directed to a display device. As discussed in Section 3.1, the decoupling of capture and display is desirable. If the eventual display device is a holographic display, signal-processing techniques are needed to convert a 3D scene—as represented in its abstract form—to a holographic pattern. Such techniques involve detailed mathematical models for diffraction and its discretization.18–20 Furthermore, sophisticated signal-processing techniques are also needed for satisfactorily fast computation. Fast computation is essential for real-time displays. Various techniques are reported in the literature, and novel techniques are being investigated for such purposes.21 Computation-based methods have many desirable features; for example, elimination of the recording step eliminates the need for interference-based techniques, which are needed essentially for intensity-based holographic recordings over chemical films or CCD arrays. It is highly desirable to work with diffraction patterns directly instead of their interfered versions with a reference beam, but diffraction patterns are complex-valued functions. However, there is no problem in dealing with complex-valued functions in computational environments, but a direct recording of a complex

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

55

Figure 3.7 A 3D optical reconstruction from an SLM-based holographic display. The 3D image consists of three planar words floating at different depths (real image): (a) 2D slice of the 3D image at a depth of 45 cm; (b) at a depth of 50 cm; (c) at a depth of 55 cm. (From F. Yaras, M. Kovachev, R. Ilieva, M. Agour, and L. Onural, “Holographic reconstructions using phase-only spatial light modulators,” 3DTV-CON 2008, © IEEE 2008. Reprinted with permission.)

 

56

Chapter 3

light distribution is a problem. Another issue is developing signal-processing techniques that compute driving signals for display devices. Holographic- or diffraction-related signals are not only high-frequency fringe patterns; they are also complicated in terms of other features. Available display devices such as SLMs may not be able to support such signals to a full extent. For example, there are amplitude-only or phase-only SLMs, but construction of an SLM that can fully support complex-valued signals is difficult. However, such restrictedcapability devices can still be used as holographic displays, although specifically designed signal-processing techniques are needed to compute the best driving signals for such devices in order to obtain the best holographic reconstructions of 3D scenes. Such signal-processing techniques have been reported in the literature, and there are ongoing research activities along those lines.18 Currently, the application of signal processing in the reverse direction is also being studied: signal-processing methods have been developed to extract the 3D information from holograms for subsequent display of such 3D scenes using nonholographic displays. That includes conventional 2D displays as well as 3D displays. While the fundamental disadvantages of holographic capture remain, opportunities may exist in niche applications, such as 3D holographic microscopy. There are few research groups in this area in the world, some of which are in Europe.

3.7 End-to-End 3DTV Systems Most of the end-to-end 3DTV systems are designed to deliver and display stereoscopic content. Commercial activities are becoming more and more frequent. Major popular sporting events have been captured and broadcast in stereoscopic format in 1998, 2002, 2007, 2009, and 2010. There are also a few known end-to-end multiview-videoformat-based 3D video systems as listed below:22

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

  



57

One such system for delivery over IP networks was developed, demonstrated, and maintained by the ECfunded 3DTV Project, which is outlined in Section 4.1. The Mitsubishi Electric Research Laboratories (MERL) group was one of the earliest to report an end-to-end 3DTV system.22 The Electronics and Telecommunications Research Institute (ETRI, Korea) has developed an HDTVcompatible 3DTV for delivery over T-DMB. They have a full demo system that includes a transmitter and a receiver equipped with a parallax-barrier stereoscopic display. The system is based on coding of stereoscopic video. There is no special error-resilience scheme.22,23 The Video Coding and Architectures (VCA) group at the Eindhoven University of Technology has proposed a 3D-IPTV. They have designed and implemented a stereoscopic and multiple-perspective system.22

A brief telepresence-flavored TV broadcast took place in November 2008 for conventional 2DTV audiences. The broadcast, which gave the feeling of 3D teleportation to the newsroom, attracted a lot of public attention. The display at the studio was not 3D; however, the effect was possible thanks to the capture of 3D multiview content by using approximately 40 cameras. By controlling the position and other parameters of these cameras through direct feedback from the main studio, it was possible to render 2D content that gave the audience a feeling of a live 3D teleportation of a reporter. The 3D-capture technology and the delivery of rich multiview video content to the studio, together with geometric corrections and appropriate rendering, was a good example of the utilization of such underlying 3D technologies. It is highly desirable to couple such applications with displays that can provide true 3D, or at least limited forms of 3D visualization.

 

58

Chapter 3

3.8 3D-Video-Related Communities and Events The interest in 3D-video-related technologies and applications both from the public and research communities prompted the formation of many organizations, communities, associations, conferences, exhibitions, and similar activities throughout the world. The following paragraphs provide a brief outline of some of those activities that have already resulted in a major impact in the field, and those that are expected to do so in the near future. A major platform with a significant impact and visibility is the 3D Media Cluster, which was formed on 16 April 2008 by the voluntary action of EC-funded 3D-related projects. Since each project is in itself a consortium of active companies, research centers, and universities in the field, the Cluster may be considered as the summit where the highest-level research activities are discussed, and future research policies are made and reviewed; the idea of writing this book was prompted by such activities. Many projects that form the Cluster are outcomes of the recently completed EC-funded 3DTV Project, which is outlined in Section 4.1. The 3DTV-CON series of annual conferences was started as an outcome of 3DTV Project activities in 2007, and the fourth of this series was conducted in May 2010. Eight of the Cluster projects are currently active as scheduled in their research plans. Many other EC-funded and 3D-related projects are expected to join the Cluster in the near future. Another major platform, the 3D Consortium, which brings together affluent companies and other research entities, was formed on March 4, 2003. The consortium organized many meetings and sessions, and formed technical committees that served as working groups concentrating on 3D-related issues. At present, the number of members is on the order of 100. Members hail from all over the world, but most of the activities are in Japan. The consortium has organized major one-day 3D-videorelated conferences. Yet another body is the 3D@Home Consortium, which was officially launched in April 2008. The objectives of the consortium are mainly to speed up the commercialization of 3D

State-of-the-Art in 3D Imaging, Delivery, and 3D Content Display

59

with home-consumer products. The focus of the consortium includes roadmaps, education for the entire 3D industry, and the means for facilitating the development of standards. Content, hardware, and software providers—as well as consumers—are targeted. The number of members is now 41, and that includes many large and small companies from the United States, Canada, Korea, and Taiwan. The consortium management is located in the United States. The consortium is a platform for members to communicate, collaborate, and coordinate with each other on a quarterly basis, and it has formed steering teams to drive the agenda. 3D-related associations are also active in Korea with regularly scheduled seminars and conferences. Another activity is related to 3D cinema and is led by SMPTE. Activities involve major interest groups in the cinema industry and focus primarily on replacing the chemical-filmbased production and distribution with end-to-end digital techniques. During the past few years, the focus has been classical 2D cinema and has resulted in a set of recommendations on formats and various parameters that are then adopted as industry standards. The transition to an alldigital cinema chain started about four years ago and is ongoing. A natural consequence of end-to-end digital cinema is the relative simplicity of marginal efforts to deliver high-quality digital stereoscopic 3D cinema. With digital techniques and devices, it is easier to control the alignment parameters associated with stereoscopic cameras, to conduct proper postprocessing techniques that achieve the correct stereoscopic content, and finally to display the end product in movie houses. This is well observed by leading players in the cinema industry, and many popular titles have already appeared in 3D as more and more movie theaters gain 3D capabilities. The digital transition will result in 3D movies not only in specialized facilities, but also in neighborhood movie theaters. This has prompted SMPTE to form a 3D Home Entertainment Task Force whose charter is to define the parameters of a stereoscopic-3Dmastering standard for content viewed at home. The goal is to pave the way for the delivery of such stereoscopic feature films  

60

Chapter 3

to homes via different channels such as terrestrial, cable, and satellite broadcast, and packaged hard media. The first meeting of the task force was on 19 August 2008. Various matters for production and commercialization of stereoscopic 3D are regularly discussed in major industry events; one such series is the NAB Digital Cinema Summit, which increasingly has a 3D flavor; the focus was primarily on 3D cinema in 2009. More and more 3D-related exhibitions are appearing in events such as IBC as well. There are other technical conferences and exhibitions that focus primarily on 3D-related issues. Furthermore, many prestigious conferences have been regularly organizing special sessions on 3D. Many prestigious journals have published special issues on 3D-related topics.24 Similar future events and publications are scheduled. Periodic technical newsletters related to the field are compiled and distributed. There are also technology reports written by experts in the field and distributed by professional companies. As a multidisciplinary topic, the broad range of research issues outlined above is rarely consolidated. One major divide is observable between the optics-related community, which primarily focuses on the display technologies, and the computer graphics, compression, and streaming communities, which deal instead with content creation, representation, and delivery. The research community dealing with the capture of 3D video can be identified as quite separate from the community whose main interest is in associated hardware. Very few events and activities, such as the 3D Media Cluster, 3DTV Project, and 3DTV-CON, are able to bring these diverse communities together.

 

Chapter 4

Current Research Trends Research activities in core areas of 3D media are increasing and gaining importance throughout the world. This chapter presents the current research trends in these fields, and the content of this chapter is primarily based on the research interests of ten ECfunded research consortia that collectively make up the 3D Media Cluster. Even though the coverage is not exhaustive, the span of these ten projects is quite broad and provides an excellent overview of current research trends in the field. The content is based primarily on public announcements made by the individual projects related to their research goals and results.

4.1 3DTV This was a four-year project (2004–2008), the technical focus of which was 3D video communications, along with all of the functional components associated with capture, representation, coding, transmission, and display (see Fig 1.2). Furthermore, the applications of such a technology to a large number of potential areas were also investigated. The consortium adopted a strategy with a wide scope and conducted joint research in various alternative potential technologies for each functional component of the broader 3DTV topic. The technical activities covered all aspects of the outlined issues in Chapter 2, and research outcomes contributed to the state-of-the-art in 3D technologies presented in Chapter 3. 61

62

Chapter 4

The primary objective of the project was to align the interests and efforts of European researchers with diverse experiences in distinct yet related and complementary areas in order to establish an effective research community working on all aspects of 3D video, as well as its seamless integration with a more general information technology base (such as the Internet). Therefore, the main activities planned within the project were primarily targeted to reach this objective. The technical activities of the project were conducted by forming technical committees on the capture, representation, coding, transportation, and display of 3D visual data; there was another technical group whose focus was signal-processing issues in diffraction and holography. The technical achievements of the project are summarized below. (a) Capture: 





Many different candidate technologies were comparatively assessed. It was concluded that multiple synchronized video recordings are currently the most promising technology. Other alternatives such as singlecamera techniques, pattern-projection-based approaches, and holographic cameras have also been investigated and found to be more sensitive to errors and artifacts, or require special setups for scene recording. Several partners also investigated different sensor technologies to record 3D scenes. Cameras based on time-of-flight techniques are among the alternative capture devices that were also investigated. A robot equipped with a laser scanner and an omnidirectional camera was developed to capture the 3D structure of an outdoor environment as it travels. The system is based on a stereo technique and calculates dense depth fields. The robot was used to reconstruct small parts of a village (Tübingen, Germany), and the recorded data have been used for building city models. Several experimental multicamera capture systems were designed and tested. They were used for indoor scene

Current Research Trends







63

reconstruction and the analysis of natural phenomena, such as 3D models of fire, smoke, or air temperature. Such effects drastically improve the realism of animations and are therefore essential in scene representation. Many techniques were developed to generate automated 3D personalized human avatars from multicamera video input. This is an important topic, since human observers are well trained to differentiate natural and unnatural faces or animations. Image-based methods were developed for surface reconstructions of moving garments from multiple calibrated video cameras. Such techniques are highly important in animation of virtual avatars with sufficient realism. Several methods based on synthetic-aperture-radar techniques were developed to increase the resolution of CCD-based holographic recordings. Furthermore, algorithms were developed for denoising interference patterns in order to improve the accuracy of reconstructed 3D scenes from holographic recordings.

(b) Representation: 



Leading activities in ISO MPEG to make point-based (dense) 3D representation a standard for autostereoscopic 3D displays and freeview TV applications were conducted. Contributions from project partners were found to be superior in terms of their novel high-quality view-generation capabilities and coding efficiencies. Comparisons between point-based (dense depth) and surface-based (mesh) representations in terms of their coding efficiency were conducted, and the quality of generated novel views from these representations was assessed.  

64

Chapter 4

  

Constant connectivity time-varying mesh representations were developed to efficiently represent the geometry of a dynamic scene explicitly in 3D using multicamera video data. The effect of volumetric scene representation via space/angle-sweeping techniques to obtain precise reconstruction of 3D scenes was investigated. Natural-looking lip animation synchronized with incoming speech was developed. Furthermore, methods to relate the time-varying posture of a dancing human to music with multimodal modeling of both visual and audio clues were proposed.

(c) Coding: 



Partners of the project played a leading role in the development of MPEG standards for 3D; this includes both management and technical issues. A 3D video compression and coding specification, which is known as MPEG-C Part 3, was released in January 2007. The specification is based on the video-plus-depth format. This standard was developed with significant contributions from this project. A more generic format, known as multiview video coding (MVC), was provided by an extension of H.264/AVC (i.e., MPEG-4 Part 10 Amendment 4). The winning proposal of the related MPEG Call for MVC Technology that now forms the basis for further development of the standard was developed within the project. Several technical contributions were made to further develop the standard. Possible extensions for coding of multiview video-plusdepth were initiated. Extensions were initiated by partners for coding multiview video-plus-depth (MVD) and free-viewpoint television (FTV). Researchers of the project played a leading role in establishing a framework for development of such a new standard for 3D video. The goal is to efficiently support multiview autostereoscopic

Current Research Trends









65

displays and other 3D video applications that require rendering of multiple output views at the decoder. Requirements were formulated, test data were generated, a reference software framework including depth estimation and view synthesis was established, and experiments were conducted. Input from the project partners triggered new activities about dynamic 3D mesh compression in MPEG. Technology developed by partners now forms an integral part of related new amendments [MPEG-4 Animation Framework eXtension, (AFX)]. Pioneering research was conducted in the area of multiple description coding (MDC) for 3D. Contributions to the watermarking of 3D mesh models were made. In addition, watermarking of image-based rendered data was investigated for the first time within the project. Such algorithms will be important for protection of intellectual property rights associated with image-based 3DTV data. Researchers within the project performed successful pioneering work in compression of holographic data.

(d) Transport:



A state-of-the-art, end-to-end platform for streaming 3D video over IP was developed and maintained. This platform also served as a testbed to evaluate the latest research results such as forward error correction (FEC) and error concealment methods. The platform has been demonstrated in exhibitions such as IBC, Amsterdam (September 2007), and ICT Information Day, Luxembourg (December 2007). Both server and client codes utilize live555 streaming library. Both multiview (MPEG MVC) and video-plus-depth 3D video formats are supported. In the case of multiview, up to nine views are decoded in real time. Forward error correction (FEC) is used to handle packet losses. Several different 3D  

66

Chapter 4













display systems, including stereo projectors and commercially available stereoscopic and multiview displays are supported. Lossy channel effects on the resultant 3D video quality are also investigated using the developed test platform. New methods for video and 3DTV streaming over DCCP were developed. Extensions to RTP and SDP protocols for 3DTV streaming were proposed. An optimal cross-layer packet scheduling for 3D video streaming was developed. Packet loss resilience over wired and wireless channels and concealment techniques were developed. Different approaches for error concealment in stereoscopic images were developed and compared. Applications of turbo codes to 3D were investigated.

(e) Display: 

  







Road maps for different display technologies were prepared. RGB laser/LED sources for 3D displays were evaluated. SLM technologies for 3DTV and related applications were evaluated. SLM-based holographic reconstructions were demonstrated. VLSI technology targets for the future were investigated. New materials, such as polymer-dispersed liquid crystals (PDLCs) for SLMs were investigated. Human factors related to autostereoscopic and holographic displays were evaluated. Hardware/software requirements for multi-user headtracking systems were evaluated. An advanced headtracking interactive autostereoscopic multiview display was implemented and demonstrated. Interactivity of 3D application systems and their usability, along with related human factors, were investigated.

Current Research Trends



67

Applications of developed technologies to various areas, such as football, virtual tour systems for archaeological sites, and cellular phones were investigated.

(f) Signal-Processing Issues in Holography and Diffraction: 

 

 

  

Fast techniques for the computation of diffraction between tilted planes were developed; this is important in extending the known algorithms between parallel planes to a more general case. Procedures for the computation of diffraction patterns from distributed data points in space were developed and implemented; thus, the computation of diffraction patterns for more realistic scenarios became possible. Phase-retrieval techniques were developed; this is important in measurement of 3D object profiles. Speckle noise reduction techniques were developed; such techniques are important for higher-quality reconstructions from holographic data. Mathematical tools to facilitate the solution of diffraction-related problems were developed; the developed tools pave the way for easier solutions for rather difficult signal-processing problems in diffraction and holography. Exact analysis of sampling effects of the diffraction field was presented; the result is significant in the development and analysis of digital processing algorithms related to propagating fields. Algorithms were developed and tested to generate holograms to be optically displayed on available SLMs; thus the developed algorithms were also physically tested. Fast and efficient algorithms to digitally synthesize holograms of 3D objects and to reconstruct images from these holograms in different hardware platforms were developed and tested; such fast algorithms are essential for real-time holographic 3DTV operations.  

68

Chapter 4

 

 

   

Optical methods, including pattern-projection approaches, for measuring contours and shapes of 3D objects were developed; such methods are critical in 3D object recording [see Figs. 3.2(a) and (b)]. Synthetic aperture techniques for resolution enhancement in digital holography were developed; such techniques are essential in using multiple, or moving, available low-resolution digital capture and display devices to generate the desired higher-resolution operation. A wave field reconstruction and design technique, based on discrete inverse problems, was developed; this is directly related to the generation of a given diffraction field in space. A fast and efficient photorealistic hologram synthesis algorithm was developed, implemented in different hardware platforms, and tested both numerically and optically; fast implementations are essential for a realtime holographic 3DTV operation. Phase unwrapping techniques under noisy conditions were developed; such techniques are critical when capturing 3D object information based on pattern projection techniques. Multiwavelength pattern projection techniques for capturing 3D object shapes were developed and assessed; such techniques are important for 3D shape recording. Crosstalk measurement methods for autostereoscopic displays were developed and applied; such techniques are essential to assess, and therefore further enhance, autostereoscopic display performance. Methods to measure optical characteristics of slantedparallax-barrier multiview 3D displays were developed; such techniques are essential to assess and improve the performance of autostereoscopic displays.

Current Research Trends



69

Fast procedures to reduce the crosstalk in multiview displays were developed; these techniques are essential to improve the performance of such 3D displays.

4.2 2020 3D Media: Spatial Sound and Vision The focus of this project (2008–2012) is to develop and demonstrate novel forms of entertainment; this will be built on new 3D audio and video technologies. Capture, production, networked delivery, and display of 3D content are all covered. Existing techniques will be explored, but most of the activity will be devoted to development of novel techniques in these fields. Both stereoscopic and immersive audio visual content is targeted; furthermore, both home and public entertainment are among the goals. It is expected that the developed technologies will be used by both the media industry and the public. The consortium lists some of the potential advantages of stereoscopic or immersive entertainment modes as:  



A superior feeling of reality and presence. An ability to navigate in a virtualized world that has a good sense of reality. An ability to process and present multidimensional content for different purposes.

The project has so far been able to deliver significant technical results. Some of these results are presented below: 





New 3D capture camera architectures, based on multifocal techniques and pattern projection methods, were developed. Video-plus-depth recording was targeted. The device has a central cinematographic camera and two or four smaller satellite side cameras. Resolution enhancement techniques were developed for resolutions higher than 2K. Contributions to standards and formats associated with multiview video were made. The potential of existing  

70

Chapter 4

 





metadata standards, such as SMPTE380M, MPEG-7, EBU Core, EBU P-Meta, Dublin Core, and XMP were investigated. A new 3D-recording architecture was developed. The developed architecture has an interface that is compatible with standard Ethernet protocols. 2D-to-3D conversion algorithms were developed. The algorithms are based on initial segmentation of 2D video objects and subsequent conversion to 3D. It was observed that automated segmentation algorithms may lead to a well-automated conversion. However, the performance of tested automated segmentation algorithms was found to be inadequate for a good conversion to 3D. Therefore, the current segmentation step is mostly manual. Concealment methods for multiview images were developed. Incomplete 3D data as a consequence of occlusions and transmission problems may be filled in using these methods. Specification of an automated 3Dcontent distribution system was completed. A proposal for a spatial audio format was completed. The format allows for interactivity in choosing efficient decoding procedures for multiloudspeaker environments.

4.3 3DPHONE The 3DPHONE project (2008–2011) aims to develop technologies and core applications that enable a new level of user experience by developing an end-to-end all-3D imaging mobile phone. Its objective is to have all fundamental functions of the phone in 3D: media display, user interface, and personal information management applications will all be realized through use of 3D technologies. Techniques to be integrated to the all-3D phone include mobile stereoscopic video, 3D user interface, 3D capture and content creation, compression, rendering, and 3D display. The delivery of 3D content will be over cellular phone channels such as GSM and 3G; multiview video delivery is targeted. Both a multiview video encoder and a

Current Research Trends

71

decoder modified for handheld device hardware environments are being implemented. The 3DPHONE project has already delivered some results, as outlined below: 









Requirements and specifications for 3D video applications for mobile devices have been identified. Different video coding algorithms were studied for their suitability for mobile devices. Phone platform software specifications have been identified. Specifications for the 3D hardware modules have been identified. User requirements have been identified.

4.4 MOBILE3DTV The MOBILE3DTV project (2008–2010) is concentrating on robust delivery of stereoscopic video content over DVB-H channels. Research targets include stereo-video content creation techniques, scalable and flexible stereo-video encoders with error-resilience and error-concealment capabilities, and stereoscopic video displays for handheld devices. The project has already announced results that can be outlined as follows: 



Video content formats for 3DTV delivery to mobile devices were developed. Rendering properties, associated user satisfaction parameters, and compressibility requirements for mobile environments are quite different from those for stationary devices. Video codecs that are suitable for 3DTV delivery to mobile devices were developed. The spatial resolution of mobile devices along with supported frame rates, and the viable decoder complexities that can be supported by such devices necessitate specific considerations for mobile device codecs. Stereoscopic video-coding methods such as simulcast (independent right and left coding), multiview coding (MVC) and video-plus-depth  

72

Chapter 4

    

  

using H.264 were all considered. The following specific stereoscopic video codecs were evaluated: H.264/AVC simulcast, H.264 Stereo SEI message, H.264/MVC, MPEG-C Part 3 using H.264 for both video and depth, and H.264 auxiliary picture syntax for video-plus-depth. Different prediction schemes were compared. Noise characteristics of DVB-H channels and their influence on stereoscopic video delivery were investigated. The results guide the design of tools for error-resilient 3DTV transmission over DVB-H. Internet Protocol streaming procedures together with DVB-H specific tools were used for end-to-end delivery of stereoscopic content. User-experience experiments related to mobile 3DTV content are being conducted. Such mostly subjective tests are essential for subsequent usability and success of such devices for 3DTV delivery. Metrics for objective quality assessment were developed. These metrics consider specific artifacts associated with stereoscopic video compression and transmission. Video processing tools to enhance the perceived quality of video for mobile users were developed. A prototype portable device that can receive and display stereoscopic video over DVB-H was implemented and demonstrated in various public exhibitions. The device is backward compatible for receiving 2D video. An end-to-end system that can broadcast compressed and stored stereoscopic video content over DVB-H channel was targeted. A simulator to generate specified stereoscopic video artifacts was developed. This tool enables experiments in a controlled manner by allowing arbitrary combinations of potential artifacts. A taxonomy of related artifacts was developed; this includes capture, coding, conversion, transmission, and display artifacts. Typical errors associated with DVB-H were characterized.

Current Research Trends



73

A database that contains video-plus-depth data was established.

4.5 Real 3D This project started in 2008 and is planned to continue until 2011. Its objective is to eliminate the current obstacles to achieving a holographic 3D video capture and display setup for viewing of real-world objects. The objectives of the project include: 

 







Design and implementation of a 3D digital holographic acquisition system with sensors arranged in a circular configuration around a real-world 3D scene or 3D object. Design and implementation of a 3D holographic display system based on SLMs arranged in a circular configuration, capable of displaying holographic video of the captured 3D scene. Further developments in the signal processing, image processing, and information science theories, techniques, and tools required for processing, analysis, and synthesis of the data from capture to display. This includes adapting the captured data for different display requirements. Results related to functionality, performance, resolution, restrictions, data quality, and human perception experiments; such results are expected to play a key role in further commercialization efforts of this 3D holographic capture and display technology. Investigation of the capabilities and fundamental limits of the digital holography technology and principles. Design of a 360-degree primary hologram capture arrangement. The design is already in its late stages and has been partially implemented.  

74

Chapter 4



Processing of holograms to provide solutions for fundamental capture and display problems. Stillhologram data compression techniques have been extended for hologram video compression. Modification and display of digital holograms using conventional display technology have been demonstrated, and visual perception studies are planned.

4.6 MUTED and HELIUM3D The primary focus of the MUTED project (2006–2008; ended) was to produce an autostereoscopic 3DTV display that supports multiple mobile viewers simultaneously. The goal of HELIUM3D (2008–2010), however, is to develop more advanced autostereoscopic 3D displays based on a new technology called “direct-view RGB laser projection.” These projects collectively aim to provide sophisticated 3D display devices that can track and support multiple viewers, at least in a limited space. Proper motion parallax is intended for each viewer with a large 3D depth-of-field. A large screen size is the target. User interaction with the 3D content is among the goals: natural gestures and hand movements will be used for user interaction. A high-quality bright display with a superior color gamut is also targeted. The MUTED display uses a steerable backlighting system that changes direction of illumination as the observer position is tracked. This is quite different from classical backlight illumination of LCD panels. The illumination source is an RGB laser projector.

4.7 3D4YOU This project (2008–2010) focuses on specifying a 3D delivery format and providing guidelines for the 3D content generation processes. Improvements in 3D capture techniques, development of processing techniques for conversion of captured content to broadcast-ready video, and subsequent coding for broadcast are included among the project goals. Delivery of an end-to-end 3D-

Current Research Trends

75

media system is among the primary project objectives. Achievements of this ongoing project include a mixed-reality system that handles both real and superposed virtual content for 3D media. The system generates video-plus-depth data for such content. Occlusion handling procedures are developed. CCD cameras are backed with time-of-flight cameras to capture content. The system models the captured environment, determines the camera positions, and uses this data to properly mix, track and align content. A camera system that consists of a stereoscopic HDTV camera set, two satellite cameras, and a depth camera is developed and tested, and test sequences are generated.

4.8 3D Presence This project started in 2008, and the activities will continue until 2010. The objective is to implement a multiparty, high-end 3D videoconferencing system. The target is to provide the videoconferencing participants a feeling of physical presence at multiple remote locations in real-time. Therefore, the project targets the transmission, efficient coding, and accurate representation of optical presence cues that include multi-user stereopsis, multiparty eye contact, and multiparty gesture-based interaction. A shared table perception, where the remote participants would have the feeling of sitting around the same table, is the main target.

4.9 VICTORY This project (2007–2009; ended) aimed to develop an innovative, distributed search engine based on a novel software object structure. The objects are defined as 3D objects together with their key properties, such as their 2D views, related text, and audio and video properties. The primary objective of this project was to develop methods and technologies for the extraction of 3D content models from video. The intention was to achieve high-quality segmented 3D video objects that can be  

76

Chapter 4

used in digital cinema and interactive games. The target was to pave the way for efficient and automatic handling and reuse of 3D video objects. 3D data capture techniques with 3D motion estimation procedures were integrated into postproduction processes and semantic approaches to achieve high-quality object extraction. The resultant objects have semantic tagging and can be manipulated in a media production pipeline and reused in different applications.

 

Chapter 5

The Future of 3D-MediaRelated Research Trends In this chapter, the expectations of future research activities and their outcomes are discussed. While the general climate in which research activities in 3D media take place should be investigated and taken into consideration when making such predictions, such a broad overview is beyond the intentions of this book. As researcher interest in 3D technologies is rising, so too is public interest in 3D cinema and TV. Creation, end-to-end delivery, and display of 3D content are three major challenges. There seems to be a consensus that the 3D media will be the next endall application, and delivery of 3D video over the Internet to stationary and mobile environments will be a hot topic. Underlying technologies are diverse; a range of technologies from low-end (stereoscopy) to high-end, ultra-realistic displays (holography) promise an array of different quality commercial products over time. While European companies and researchers have a strong basis in 3D-related areas, competition is strong with interest from research centers and companies all over the world (especially from the United States, Japan, and Korea). Although 3D technologies have been around in some form for the past 170 years, extensive research momentum in its sophisticated forms, especially with true 3D video in mind, has not yet reached a significant level. Current research activities are instead concentrated on multiview 3D video systems; multiview video is an extension of stereoscopy with many views instead of 77

78

Chapter 5

two, and provides a performance between classical two-view stereoscopy and true 3D imaging modalities. Another research topic with current momentum is eyewear-free visualization of stereoscopic and multiview video. Based on these observations, research activities can be divided into three categories in terms of the timing of expected major results. The categorization is presented as near-, medium-, and long-term research activities.

5.1 Near-Term Research Activities This section outlines those research activities that target and have the potential of delivering commercial products, or results that can be quickly commercialized, in the near term, i.e., within a year or so. The first item in this category is a group of products related to the digital capture and projection of stereoscopic content. A major concern is the minimization of the viewer discomfort that is usually called the “eye fatigue” and results in a feeling similar to motion sickness. Digital techniques have already provided successful results. However, further research in enhancement of stereoscopic content for viewer comfort, which in turn results in a better viewer experience and satisfaction, is still going on, and there is still room for improvement and innovation. A major technical problem is better alignment at camera and display ends. Higher-level activities involve development and application of image processing techniques to refine captured stereoscopic content for better viewer satisfaction.  Another goal is the development of equipment for end-toend stereoscopic 3D. This includes cameras, production and postproduction equipment, and display units. Digital stereo cameras that facilitate even simple tasks like zoom and pan, mutual automated calibration, better ergonomics, etc., are needed; user-friendly alignment and calibration are desirable. A typical processing-stage activity will be the correction of calibration variations between simultaneous right and left views as well as the tracking of calibration over time. That includes baseline corrections for realistic perspectives, misalignment corrections, geometric corrections, and implementation of

The Future of 3D-Media-Related Research Trends

79

various enhancement effects. Limited versions of such tools that are suitable for a limited range of devices are expected to emerge as a consequence of near-term research activities. However, more adaptive versions for fully automated operation that can accommodate a wide range of devices in different environments will more likely be available in the medium term. Various image processing tools for stereoscopy are expected to emerge. This will include basic processing tools for enhancement and restoration as well as higher-level authoring tools. Ease of blending natural scene contents with synthetically generated 3D content will also be the focus. A related topic is the direct rendering of stereoscopic content from primary abstract 3D computer graphics content. This is particularly important for video game applications. Indeed, commercial products in the form of stereoscopic video display cards for currently available high-end computers are now readily available; it is possible to render stereoscopic content from any 3D video game content in some recognized formats for subsequent viewing using integrated shutter glasses and control devices. Another immediate task is the education of industry workers for better and more effective use of stereoscopic equipment. This includes artistic features as well as technological aspects and the successful blending of the two. Conversion from stored content created for one purpose to other forms for different purposes will be an active field. For example, stereoscopic content ready for 2K- or 4K-resolutioncinema format will need reprocessing for home viewing. This is a much more complicated task compared to similar tasks for 2D. Automation of such tasks will be highly desirable. An important group of activities, whose outcomes will have a significant impact on all other tasks listed above, is the evaluation of the quality of user experiences. Usability of products and their commercial acceptance by consumers depend primarily on the success of such perceptional and behavioral evaluations of human observers.  

80

Chapter 5

Content creation for stereoscopic 3D visual delivery is a challenge, since the artistic features of such a medium are quite different from that of its 2D counterpart. A new generation of artists who will have a better understanding and a better command of the new media is needed. An important phenomenon that can steer short-term activities will be a consequence of wider penetration of 3D cinema. As movie-goers experience more and more 3D viewing as a consequence of the technological and commercial evolutions outlined above, and as 3D becomes a common default cinema experience, consumers will also want to carry the same experience to their homes. This is expected to create a critical mass of consumers and producers for such devices (players and display devices) for the home-oriented stereoscopic video market and related content.

5.2 Medium-Term Research Activities It is easy to predict that the activities outlined under near-term activities in Section 5.1 will also continue in the medium term. The term “medium-term” refers to research activities in 3D video that describe those activities that are expected to yield end results that can become commercial within the next few years; a three- to five-year horizon is envisioned. Based on the overview and the state-of-the-art presented in previous chapters, it is not difficult to predict that the mainstream medium-term activities will be multiview-video-based systems intended for eyewearfree viewing. This includes various end-to-end multiview 3D delivery systems intended for different platforms such as 3D cinema, 3D broadcast TV over different physical channels, 3Dpackaged content in different hard-delivery forms, and mobile devices. A group of particular applications will be related to limited forms of telepresence and 3D videoconferencing, for which there are already existing commercial products. However, there are still many technical difficulties that prevent widespread acceptance and use of such tools. Applications include interactive communications for personal, business, and other

The Future of 3D-Media-Related Research Trends

81

purposes, as well as TV programming. For example, live reporting to TV studios, with a telepresencelike subsequent rendering, as outlined in Section 3.7, is of interest. Multiview 3D video topics will be a major medium-term research activity in all aspects of the delivery chain as outlined in Chapter 3. This includes cameras and camera systems, as well as algorithms to drive such camera systems to control and achieve a meaningful 3D video output. Automated calibration is an important goal. Storage of 3D content captured from such cameras will continue to be a challenge; this requires research efforts in modeling and representing 3D environments, and preferably automated mapping of captured content in different possible formats into such intermediate representations. A natural parallel activity is the development and testing of various multiview-video-compression algorithms. Already existing know-how and recently emerged standards in this area will further mature in the medium-term with incorporation of novel techniques. Both the intermediate representation techniques and the subsequent coding algorithms, together with related standardization activities, will be the backbone of research efforts that will directly guide and facilitate associated commercial products. Research on comparative analyses of such delivery formats will also be the focus of policy makers, since related decisions will have a major impact on the commercial success of many underlying (most likely patented) products from different sources. As indicated in Section 3.1, a very desirable end result is a complete decoupling of input devices from display devices; this can be accomplished by providing a universal interface in the form of standardized representations and coding procedures. Therefore, a significant amount of research efforts will be devoted to achieving such decoupling. However, the degree of decoupling will be quite limited in the medium term and therefore, it is likely that input and display devices will be  

82

Chapter 5

affected, at least to a certain degree, by the choices of the delivery formats. In turn, this will create further pressures on policy decisions, and will eventually steer commercialization choices that will affect the profitability of related products. Technical activities that are targeted to generate synthesized 2D video from any arbitrary viewing position using already captured multiview content will gain momentum in the medium term. Indeed, the success of such research activities may pave the way for true-3D displays, since a dense set of 2D representations may then easily be converted to such true-3D forms, provided that the physical nature of the display is adequate. Medium-term display-related research activities will be targeted toward better autostereoscopic displays. However, the resultant displays will be rather improved variants of currently available lenticular- and barrier-based devices. Such improvements will diminish the differences between multiview displays and low-end light field rendering displays as the number of views in multiview displays increases. Major accomplishments are expected, particularly in the quality of light sources used in such displays; in turn, this will improve the quality of experience on the user side. Another expected major improvement will be easily implemented head-tracking techniques; such effective and practical passive head-tracking technologies may be the key for consumer acceptance of 3D video at home. Direct light-beam scanning 3D projection displays may also emerge. Another medium-term research activity will be further investigation of efficient delivery of digital content via different channels. That includes terrestrial broadcast channels, variants of such channels intended for broadcast to mobile (especially handheld) devices, cable networks, satellites, and the Internet. It is important to emphasize the delivery over the Internet utilizing different present and future protocols, since it is expected that such a form of delivery will probably dominate other forms. 3D counterparts of current 2D-video-related activities over the Internet will emerge in the medium term.

The Future of 3D-Media-Related Research Trends

83

A line of medium-term research activities will target immersive environments and interaction with such environments. Significant research activity and subsequent improvements along such directions are expected. Specific applications will include virtual tours of remote attractions; these include cultural sites as well as underwater and space exploration. Entertainment and gaming applications with successful 3D experience are expected to emerge. Other application areas are limited only by the imagination, and include medicine, dentistry, education, scientific visualization, aviation and air traffic control, defense-related applications, land use including construction, architecture, art, and more. Understanding that end-to-end 3D video systems are highly multidisciplinary, it is expected that research activities and outputs in each such underlying field that collectively contribute to the basics will significantly affect and steer 3Dvideo-related research. Therefore, such basic research is indispensable and has direct consequences in the success of 3Dvideo-related activities. Particular fields include electronics for novel optical camera devices, electronics for novel display devices, optics- and photonics-based research targeting 3D cameras and displays, electronic and photonic improvements for the basic telecommunications infrastructure, new techniques in digital communications, and basic research in signal-processing techniques that target different issues and devices in the 3D video chain.

5.3 Long-Term Research Activities The main focus of long-term research in 3D video technologies is “ultra-realistic displays.” Ultra-realistic displays are defined as futuristic devices that can deliver optical content that is indistinguishable from real life. Literally speaking, these are ghost environments, so successful that the observer will not be able to distinguish between real environments and their optical  

84

Chapter 5

replicas. (See Fig 1.1) The basis for such optical duplication is outlined in Chapter 1. A line of research that targets such success in delivering optically duplicate content in the long-term is based on holography. Therefore, holographic video display research is expected to gain momentum during the upcoming years. This includes not only research in optics, optical devices and components, electronic and photonic devices, but also fundamental research in related mathematics and signalprocessing techniques. Data-capturing devices that are suitable for capturing holographic data will also be a research focus. Holography in its classical form is based on optical interferences, recording of such interference patterns, and subsequent light field creation from such recordings. Such interference-based operations need coherent light sources and therefore, holography and lasers are almost always related. However, coherent-light-based imaging systems suffer from various other problems. Therefore, other techniques that can record and recreate incoherent light fields should also be investigated. Such efforts to record and replay incoherent light fields are expected to gain further momentum during the next decade. The term “holography” may be extended to cover such incoherent imaging techniques, based on the observation that the primary goal in such systems, whether they are based on coherent or incoherent methods, is the recording and recreation of physical light distributions with complete underlying physical properties. In other words, these true-3D techniques aim to eliminate the heavy reliance on human-perception-based features that exist in other forms of imaging techniques and focus on physical duplication of all optical properties that make up a 3D scene. With this definition, holography and integral imaging may be merged into the same category of imaging techniques. Indeed, it is quite appropriate to label such techniques collectively as true-3D imaging techniques. Research activity that will diminish the border between classical holography and integral imaging is expected. Even though the basics of integral imaging and its capabilities toward achieving a form of true-3D display are

The Future of 3D-Media-Related Research Trends

85

understood, more research is needed for further improvements in the field. This includes many practical issues, as well as fundamental modifications in the basic nature of this form of imaging. Therefore, research activities with a longer-term focus will likely concentrate primarily on true-3D imaging, including capture and display sides, as well as processing, coding, and transport of such rich data.

5.4 Future Internet and 3D Media When computers first began communicating with each other, the interconnection networks among them were just simple hardware, software, and protocols that linked them. As the connectivity became user friendly and widespread, virtually all computers became interlinked, and value-added services were also integrated to the network. Connectivity eventually became mobile. Appliances, vehicles, and other objects also became connected, after computers. People are effectively connected via various devices for work, social interaction, business, etc. With such a widespread connectivity, and with all of the value-added services attached to and delivered by and with the support of this infrastructure, it is a natural to see the future of the Internet not as a bare telecommunications infrastructure, but as an integration of bundled services inseparable from such an infrastructure. In other words, the Internet will be perceived by the general public, service providers, and consumers as a multitude of services whose existence cannot be distinguished separately from the interconnectivity network where they are acquired from. With such a vision of the future Internet, the 3D-media community will also likely focus on delivery and consumption of services—collectively referred to as “3D-media services”—via the Internet. Such services will be perceived as an integral part of the future Internet, like many other services. Therefore, the interaction between 3D media services and the Internet will be quite tight. In particular, being the richest form of data among all other forms of media content, 3D-media services will be pushing  

86

Chapter 5

the technical limits of the telecommunications services that the Internet intrinsically provides. There will be a major demand for higher bandwidth and more stringent quality for online communications. Interactive services built around basic 3Dmedia services will be adding to that pressure. Such highbandwidth and high-quality connectivity, in turn, will further boost the interest in more sophisticated forms of 3D-media delivery and interaction. Therefore, those services now considered excessively futuristic will quickly become feasible, and this will in turn further boost related research activities. In other words, 3D-media services will demand more bandwidth and quality, and this will in turn foster more advanced 3D-media services. In the end, 3D-media services will be tightly coupled to the future Internet infrastructure, and thus, will be an indispensable and integral part of intensely utilized daily services in all aspects of life that come with the future Internet connectivity. The future Internet will be an immense leap forward in the history of civilization with a plethora of services attached to it. The connectivity infrastructure and services delivered over it will be generally perceived as a single integral form of abundant support in and for all aspects of individual lives and social interaction. 3D-media-related services and applications are envisioned as among the primary set of indispensable features of the future Internet in terms of their impact and volume. Research in such broad 3D-media topics with near-, medium-, and longterm goals in mind is the key, and the need for integration of such research activities in the field should be stressed, since such research activities within the scope of 3D media span a large multidisciplinary scientific and technological basis.

References 1. O. Schreer, P. Kau, and T. Sikora, Eds., 3D Videocommunication: Algorithms, Concepts and Real-time Systems in Human Centric Communication, Wiley, 2005. 2. P. Benzie, J. Watson, P. Surman, I. Rakkolainen, K. Hopf, H. Urey, V. Sainov, and C. von Kopylow, “A survey on 3DTV displays: Techniques and technologies,” IEEE Tr. on Circuits and Systems for Video Technology 17(11), 1647– 1658 (2007). 3. E. Stoykova, A. A. Alatan, P. Benzie, N. Grammalidis, S. Malassiotis, J. Ostermann, S. Piekh, V. Sainov, C. Theobalt, T. Thevar, and X. Zabulis, “3D time-varying scene capture technologies—A survey,” IEEE Tr. on Circuits and Systems for Video Technology 17(11), 1568–1587 (2007). 4. L. Onural, A. Gotchev, H. M. Ozaktas, and E. Stoykova, “A survey of signal processing problems and tools in holographic 3D television,” IEEE Tr. on Circuits and Systems for Video Technology 17(11), 1631–1646 (2007). 5. C. Theobalt, Editor, “3D time-varying scene capture technologies—TC1 WP7 technical report 2,” 3DTV Network of Excellence: Tech. Rep. D26.2, (March 2007). 6. B. Rosenhahn, Editor, “3D time-varying scene capture technologies—TC1 WP7 technical report 3,” 3DTV Network of Excellence: Tech. Rep. D26.3, (March 2008). 87

88

References

7. A. A. Alatan, Y. Yemez, U. Güdükbay, X. Zabulis, K. Müller, C. E. Erdem, C. Weigel, and A. Smolic, “Scene representation technologies for 3DTV—A survey,” IEEE Tr. on Circuits and Systems for Video Technology 17(11), 1587– 1605 (2007). 8. A. A. Alatan, Ed., “3D time-varying scene representation technologies—TC1 WP8 technical report 2,” 3DTV Network of Excellence: Tech. Rep. D28.2, (Feb. 2007). 9. A. A. Alatan, Ed., “3D Time-varying scene representation technologies—TC1 WP8 technical report 3,” 3DTV Network of Excellence: Tech. Rep. D28.3, (March 2008). 10. A. Smolic, K. Müller, N. Stefanoski, J. Ostermann, A. Gotchev, G. B. Akar, G. Triantafyllidis, and A. Koz, “Coding algorithms for 3DTV—A survey,” IEEE Tr. on Circuits and Systems for Video Technology, 17(11), 1606– 1621 (2007). 11. A. Smolic, Editor, “3D coding techniques—TC2 technical report 2,” 3DTV Network of Excellence: Tech. Rep. D30.2, (Feb. 2007). 12. A. Smolic, Editor, “3D coding techniques—TC2 technical report 3,” 3DTV Network of Excellence: Tech. Rep. D30.3,

(March 2008). 13. G. B. Akar, A. M. Tekalp, C. Fehn, and M. R. Civanlar, “Transport methods in 3DTV—A survey,” IEEE Tr. on Circuits and Systems for Video Technology 17(11), 1622– 1630 (2007). 14. M. Tekalp, Editor, “3D telecommunication issues—TC3 technical report 2,” 3DTV Network of Excellence: Tech. Rep. D32.2,

(Feb. 2007). 15. A. M. Tekalp, Editor, “3D telecommunication issues—TC3 technical report 3,” 3DTV Network of Excellence: Tech. Rep. D32.3,

(Aug. 2008).

References

89

16. J. Watson, Editor, “3DTV display technologies and potential applications—TC5 technical report 3,” 3DTV Network of Excellence: Tech. Rep. D36.3,

(July 2008). 17. F. Yaraş, H. Kang and L. Onural, “Real-time phase-only color holographic display system using LED illumination,” Applied Optics 48(34), H48–H53 (Dec. 2009) 18. L. Onural, A. Gotchev, H. M. Ozaktas, and E. Stoykova, “A survey of signal processing problems and tools in holographic 3D television,” IEEE Tr. on Circuits and Systems for Video Technology 17(11), 1631–1646 (2007). 19. L. Onural and H. Ozaktas, Eds., “Signal processing issues in diffraction and holography—TC4 technical report 2,” 3DTV Network of Excellence: Tech. Rep. D34.2, (Feb. 2007). 20. L. Onural, G. B. Esmer, E. Ulusoy and H. Ozaktas, Eds., “Signal processing issues in diffraction and holography— TC4 technical report 3,” 3DTV Network of Excellence: Tech. Rep. D34.3, (June 2008). 21. H. Kang, F. Yaraş and L. Onural, “Real-time phase-only color holographic display system using LED illumination,” Applied Optics 48(34), H137–H143 (Dec. 2009). 22. A. Krutz, Editor, “Report on other ongoing research activities 4,” 3DTV Network of Excellence: Tech. Rep. D21.4,

(Sept. 2008). 23. H. M. Ozaktas and L. Onural, Eds., 3D Television: Capture, Transmission, Display, Springer, 2008. 24. L. Onural and T. Sikora, “Introduction to the special section on 3DTV,” IEEE Tr. on Circuits and Systems for Video Technology 17(11), 1566–1567 (2007).

Index   3D cinema, 62, 84 3D holographic microscopy, 59 3D media, 81, 89 3D Media Cluster, 61 3D mesh compression, 45 3D video capture, 28 3D video coding, 41 3D video display, 52 3D video streaming, 51 3D video streaming techniques, 48 3D videoconferencing, 85 3D@Home Consortium, 62 3DTV over IP 3DTV systems, 60 3DTV-CON, 61

ARQ, 51 autostereoscopy, 9 binocular disparity, 54 CCD, 15 coherence, 22 coherent light, 15 convergence, 20 crosstalk, 22 DCCP, 49 dense depth representation, 35 diffraction, 16, 71 digital broadcast, 48 digital TV, 27 DVB-H, 75

accommodation, 54 anaglyphs, 7 analog broadcast, 48 analog TV, 26 animation framework extension (AFX), 35

electroholography, 15 error concealment, 50 error correction, 50 extensible 3D (X3D), 40 eye fatigue, 21, 82 91

92

forward error correction (FEC), 51 free-viewpoint, 30 full-parallax, 20 handheld devices, 75 head-tracking displays, 56 holograms, thick, 16 holograms, volume, 16 holographic 3D video, 77 holographic cameras, 66 holographic capture devices, 31 holographic cinema, 57 holographic displays, 57 holography, 15, 88 holography, computergenerated, 15 human visual system, 54 IEEE, 40 inertial measurement units (IMU), 56 integral imaging, 13, 88 interference fringes, 15 interference pattern, 15 interzigging, 9 ISO, 40 ITU, 40 lasers, 15 lenses, variable focal length, 19 lenticular, 9

Index

lenticular, slanted, 9 light field, 16, 14, 39 microlens array, 13, 14 motion parallax, 54 multicamera techniques, 29 multiple-description coding, 47 multiview autostereoscopy, 11 multiview displays, 54 multiview video, 11 multiview video coding, 42 multiview video streaming, 50 NURBS, 36 octree, 37 ocular convergence, 54 omnidirectional camera, 66 pattern projection techniques, 32 phase-retrieval techniques, 34 point-based representations, 37 polarizers, 8 polygonal meshes, 36 progressive meshes, 36 pseudo 3D, 38 Pulfrich effect, 8 RTP, 49

Index

93

vectors, disparity, 42 vectors, motion, 41 vergence, 20 video-plus-depth, 44 virtual camera, 30 virtual reality modeling language (VRML), 40 volumetric displays, 16, 56 volumetric representations, 37

segmentation algorithms, 74 shape-from-motion, 28 shape-from-shading, 28 shape-from-texture, 28 shutter, 8 signal processing, 58, 71 simulcast, 76 single-camera techniques, 28 SLMs, multiple, 58 SLMs, phase-only, 59 SMPTE, 62 spatial light modulators, 15 speckle, 58 speckle noise, 23 standards, 40 stereo 3D movies, 21 stereoscopic 3D, 82 stereoscopic cinema, 7 stereoscopic video coding, 41 stereoscopy, 6 structured light patterns, 32 sweet spot, 10

watermarking, 69

texture mapping, 38 time-of-flight techniques, 34 true-3D imaging, 16 ultra-realistic displays, 87 93

Levent Onural received his Ph.D. degree in electrical and computer engineering from State University of New York at Buffalo in 1985; his BS and MS degrees were received from Middle East Technical University in 1979 and 1981, respectively. He was a Fulbright scholar between 1981 and 1985. He joined the Electrical and Electronics Engineering Department of Bilkent University, Ankara, Turkey, in 1987 where he is a full professor and dean of engineering at present. His current research interests are in the area of image and video processing, with emphasis on video coding, 3DTV, holographic 3DTV, and signal processing aspects of optical wave propagation. He was the coordinator of European Commission-funded 3DTV Project (2004–2008). Dr. Onural received an award from TUBITAK of Turkey in 1995. He also received a Third Millenium Medal from IEEE in 2000. Dr. Onural is a fellow of IEEE. He served IEEE as the Director of IEEE Region 8 (Europe, Middle East and Africa) in 2001–2002, as the Secretary of IEEE in 2003. He was a member of IEEE Board of Directors (2001–2003), IEEE Executive Committee (2003), and IEEE Assembly (2001–2002).