3d Imaging for Safety and Security 1402061811, 978-1-4020-6181-3

Citation preview

3D Imaging for Safety and Security

Computational Imaging and Vision Managing Editor

MAX VIERGEVER Utrecht University, The Netherlands Series Editors GUNILLA BORGEFORS, Centre for Image Analysis, SLU, Uppsala, Sweden RACHID DERICHE, INRIA, France THOMAS S. HUANG, University of Illinois, Urbana, USA KATSUSHI IKEUCHI, Tokyo University, Japan TIANZI JIANG, Institute of Automation, CAS, Beijing REINHARD KLETTE, University of Auckland, New Zealand ALES LEONARDIS, ViCoS, University of Ljubljana, Slovenia HEINZ-OTTO PEITGEN, CeVis, Bremen, Germany JOHN K. TSOTSOS, York University, Canada This comprehensive book series embraces state-of-the-art expository works and advanced research monographs on any aspect of this interdisciplinary field. Topics covered by the series fall in the following four main categories: • Imaging Systems and Image Processing • Computer Vision and Image Understanding • Visualization • Applications of Imaging Technologies Only monographs or multi-authored books that have a distinct subject area, that is where each chapter has been invited in order to fulfill this purpose, will be considered for the series.

Volume 35

3D Imaging for Safety and Security

Edited by

Andreas Koschan The University of Tennessee, Knoxville, TN, USA

Marc Pollefeys University of North Carolina at Chapel Hill, NC, USA and

Mongi Abidi The University of Tennessee, Knoxville, TN, USA

A C.I.P. Catalogue record for this book is available from the Library of Congress.

ISBN 978-1-4020-6181-3 (HB) ISBN 978-1-4020-6182-0 (e-book)

Published by Springer, P.O. Box 17, 3300 AA Dordrecht, The Netherlands. www.springer.com

Printed on acid-free paper

All Rights Reserved © 2007 Springer No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without written permission from the Publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work.

Contents

Contributing Authors

vii

Preface

xiii

PART I: BIOMETRICS

1

3D Assisted Face Recognition: A Survey M. HAMOUZ, J. R. TENA, J. KITTLER, A. HILTON, AND J. ILLINGWORTH

3

A Survey on 3D Modeling of Human Faces for Face Recognition S. HUQ, B. ABIDI, S. G. KONG, AND M. ABIDI

25

Automatic 3D Face Registration without Initialization A. KOSCHAN, V. R. AYYAGARI, F. BOUGHORBEL, AND M. A. ABIDI

69

A Genetic Algorithm Based Approach for 3D Face Recognition: Using Geometric Face Modeling and Labeling Y. SUN AND L. YIN

95

Story of Cinderella: Biometrics and Isometry-Invariant Distances A. M. BRONSTEIN, M. M. BRONSTEIN, AND R. KIMMEL

119

Human Ear Detection from 3D Side Face Range Images H. CHEN AND B. BHANU v

133

vi

Contents

PART II: SAFETY AND SECURITY APPLICATIONS

157

Synthetic Aperture Focusing Using Dense Camera Arrays V. VAISH, G. GARG, E.-V. TALVALA, E. ANTUNEZ, B. WILBURN, M. HOROWITZ, AND M. LEVOY

159

Dynamic Pushbroom Stereo Vision: Dynamic Pushbroom Stereo Vision for Surveillance and Inspection Z. ZHU, G. WOLBERG, AND J. R. LAYNE 3D Modeling of Indoor Environments for a Robotic Security Guard P. BIBER, S. FLECK, T. DUCKETT, AND M. WAND 3D Site Modelling and Verification: Usage of 3D Laser Techniques for Verification of Plant Design for Nuclear Security Applications V. SEQUEIRA, G. BOSTRÖM, AND J.G.M. GONÇALVES Under Vehicle Inspection with 3D Imaging: Safety and Security for Check-Point and Gate-Entry Inspections S. R. SUKUMAR, D. L. PAGE, A. F. KOSCHAN, AND M. A. ABIDI

173

201

225

249

Colour Plate Section

279

Index

307

Contributing Authors

Besma Abidi Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 334 Ferris Hall Knoxville, TN 37996, USA Mongi A. Abidi Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 334 Ferris Hall Knoxville, TN 37996, USA Emilio Antunez Geometric Computing Group Computer Science Department Stanford University, Stanford, CA 94305, USA Venkat R. Ayyagari Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 334 Ferris Hall Knoxville, TN 37996, USA Bir Bhanu Center for Research in Intelligent Systems University of California Riverside, CA 92521, USA

vii

viii

Contributing Authors

Peter Biber Graphical-Interactive Systems Wilhelm Schickard Institute for Computer Science University of Tübingen Sand 14, 72076 Tübingen, Germany Gunnar Boström European Commission - Joint Research Centre TP210, I-21020 Ispra, Italy Faysal Boughorbel Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 334 Ferris Hall Knoxville, TN 37996, USA Alexander M. Bronstein Department of Computer Science Technion – Israel Institute of Technology Haifa 32000, Israel Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology Haifa 32000, Israel Hui Chen Center for Research in Intelligent Systems University of California Riverside, CA 92521, USA Tom Duckett AASS Research Center Department of Technology Örebro University SE-70182 Örebro, Sweden Sven Fleck Graphical-Interactive Systems Wilhelm Schickard Institute for Computer Science University of Tübingen Sand 14, 72076 Tübingen, Germany

Contributing Authors Gaurav Garg Computer Graphics Laboratory Department of Electrical Engineering Stanford University, Stanford, CA 94305, USA João G.M. Gonçalves European Commission - Joint Research Centre TP210, I-21020 Ispra, Italy Miroslav Hamouz Centre for Vision, Speech and Signal Processing University of Surrey Guildford, GU2 7XH, United Kingdom Adrian Hilton Centre for Vision, Speech and Signal Processing University of Surrey Guildford, GU2 7XH, United Kingdom Mark Horowitz Computer Systems Laboratory Computer Science Department Stanford University, Stanford, CA 94305, USA Shafik Huq Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 209 Ferris Hall Knoxville, TN 37996, USA John Illingworth Centre for Vision, Speech and Signal Processing University of Surrey Guildford, GU2 7XH, United Kingdom Ron Kimmel Department of Computer Science Technion – Israel Institute of Technology Haifa 32000, Israel

ix

Contributing Authors

x

Josef Kittler Centre for Vision, Speech and Signal Processing University of Surrey Guildford, GU2 7XH, United Kingdom Seong G. Kong Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 310 Ferris Hall Knoxville, TN 37996, USA Andreas Koschan Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 334 Ferris Hall Knoxville, TN 37996, USA Jeffery R. Layne Air Force Research Laboratory 2241 Avionics Circle, WPAFB, Ohio 45433-7318, USA Marc Levoy Computer Graphics Laboratory Computer Science Department Stanford University, Stanford, CA 94305, USA David L. Page Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 334 Ferris Hall Knoxville, TN 37996, USA Vitor Sequeira European Commission - Joint Research Centre TP210, I-21020 Ispra, Italy Sreenivas R. Sukumar Imaging, Robotics, and Intelligent Systems Laboratory The University of Tennessee, 334 Ferris Hall Knoxville, TN 37996, USA Yi Sun Computer Science Department State University of New York at Binghamton Binghamton, New York 13902 USA

Contributing Authors Eino-Ville Talvala Computer Graphics Laboratory Department of Electrical Engineering Stanford University, Stanford, CA 94305, USA Jose Rafael Tena Centre for Vision, Speech and Signal Processing University of Surrey Guildford, GU2 7XH, United Kingdom Vaibhav Vaish Computer Graphics Laboratory Computer Science Department Stanford University, Stanford, CA 94305, USA Michael Wand Graphical-Interactive Systems Wilhelm Schickard Institute for Computer Science University of Tübingen Sand 14, 72076 Tübingen, Germany Bennett Wilburn Computer Graphics Laboratory Department of Electrical Engineering Stanford University, Stanford, CA 94305, USA George Wolberg Department of Computer Science The City College of New York New York, NY 10031, USA Lijun Yin Computer Science Department State University of New York at Binghamton Binghamton, New York 13902 USA Zhigang Zhu Department of Computer Science The City College of New York New York, NY 10031, USA

xi

Preface

The past decades have seen significant improvements in 3D imaging where the related techniques and technologies have advanced to a mature state. These exciting developments have sparked increasing interest in industry and academia in the challenges and opportunities afforded by 3D sensing. As a consequence, the emerging area of safety and security related imaging incorporates these important new technologies beyond the limitations of 2D image processing. This book is so far the first that covers the current state of the art in 3D imaging for safety and security. It reports about selected contributions given at the “Workshop on Advanced 3D Imaging for Safety and Security” held in conjunction with the International Conference on Computer Vision and Pattern Recognition CVPR 2005, June 2005, San Diego, CA. The workshop brought together pioneering academic and industrial researchers in the field of computer vision and image analysis. Special attention was given to advanced 3D imaging technologies in the context of safety and security applications. Comparative evaluation studies showing advantages of 3D imaging over traditional 2D imaging for a given computer vision or pattern recognition task were emphasized. Moreover, additional experts in the field of 3D imaging for safety and security were invited by the editors for a contribution to this book. The book is structured into two parts, each containing five or six chapters on (1) Biometrics and (2) Safety and Security Applications. Chapter 1 introduces a survey on 3D assisted face recognition which is followed by a survey of technologies for 3D modeling of human faces in Chapter 2. Chapter 3 explains automatic 3D face registration which overcomes the traditional initialization constraint in data registration. Chapter 4 presents a xiii

xiv

Preface

genetic algorithm based approach for 3D face recognition using geometric face modeling and labeling. Chapter 5 looks into biometrics and isometryinvariant distances starting from the story of Cinderella as an early example of 3D biometric identification and biometric frauds. Chapter 6 reports on human ear detection from 3D side face range images considering ear as a viable new class of biometrics since ears have desirable properties such as universality, uniqueness and permanence. The second part of the book is devoted to safety and security applications introducing synthetic aperture focusing with dense camera arrays in Chapter 7. This chapter demonstrates practical applications of surveillance in addition to the theory. Chapter 8 presents a dynamic pushbroom stereo geometry model for both 3D reconstruction and moving target extraction in applications such as aerial surveillance and cargo inspection. Autonomous mobile robots play a major role in future security and surveillance tasks for large scale environments such as shopping malls, airports, hospitals and museums. The challenge of building such a model of large environments using data from the robot's own sensors: a 2D laser scanner and a panoramic camera is addressed in Chapter 9. In Nuclear Security it is important to detect changes made in a given installation or track the progression of the construction work in a new plant. Chapter 10 describes a system accepting multi-sensory, variable scale data as input. Scalability allows for different acquisition systems and algorithms according to the size of the objects/buildings/sites to be modeled. The chapter presents examples of the use in indoor and outdoor environments. A modular robotic “sensor brick” architecture that integrates multi-sensor data into scene intelligence in 3D virtual reality environments is introduced in Chapter 11. The system is designed to aid under vehicle inspection with 3D imaging for check-point and gate-entry inspections. Last, but not least, we wish to thank all the members of the Program Committee of the “Workshop on Advanced 3D Imaging for Safety and Security” 2005 for their valuable work, and all the authors who contributed to this book.

Andreas Koschan Marc Pollefeys Mongi Abidi Knoxville and Chapel Hill, January 2007

Part I Biometrics

Chapter 1 3D ASSISTED FACE RECOGNITION: A SURVEY M. Hamouz, J. R. Tena, J. Kittler, A. Hilton, and J. Illingworth Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2 7XH, United Kingdom, {m.hamouz,j.tena,j.kittler,a.hilton,j.illingworth}@surrey.ac.uk

Abstract:

3D face recognition has lately been attracting ever increasing attention. In this chapter we review the full spectrum of 3D face processing technology, from sensing to recognition. The review covers 3D face modelling, 3D to 3D and 3D to 2D registration, 3D based recognition and 3D assisted 2D based recognition. The fusion of 2D and 3D modalities is also addressed. The chapter complements other reviews in the face biometrics area by focusing on the sensor technology, and by detailing the efforts in 3D face modelling and 3D assisted 2D face matching. A detailed evaluation of a typical state-of-theart 3D face registration algorithm is discussed and conclusions drawn.

Key words:

3D sensors, 3D face models, 3D face registration, 3D face recognition

1.

INTRODUCTION

Face recognition and verification have been at the top of the research agenda of the computer vision community for more than a decade. The scientific interest in this research topic has been motivated by several factors. The main attractor is the inherent challenge that the problem of face image processing and recognition poses. However, the impetus for better understanding of the issues raised by automatic face recognition is also fuelled by the immense commercial significance that robust and reliable face recognition technology would entail. Its applications are envisaged in physical and logical access control, security, man-machine interfaces and low bitrate communication. To date, most of the research efforts, as well as commercial developments, have focused on 2D approaches. This focus on monocular imaging has partly been motivated by costs but to a certain extent also by the need to retrieve faces from existing 2D image and video databases. Last but 3 A. Koschan et al. (eds.), 3D Imaging for Safety and Security, 3–23. © 2007 Springer.

4

M. Hamouz et al.

not least, it has been inspired by the ability of human vision to recognise a face from single photographs where the 3D information about the subject is not available and therefore the 3D sensing capability of the human perception system cannot be brought to bear on the interpretation task. The literature addressing the problem of 2D face recognition and verification is truly extensive, offering a multitude of methods for each of the major steps of the process: 1) Detection and localization of faces 2) Geometric normalization 3) Photometric normalisation 4) Feature extraction and 5) Decision making. It is beyond the scope of this chapter to do a proper justice to all the contributions made to this topic. Accordingly, we shall refer the reader to the major surveys that have recently appeared in the literature. These include the work of Zhao48 on face representation and decision making, the face detection review of Yang et al.42, and the overview of photometric face normalisation methods in Short et al.36. The literature also includes reports of major evaluation studies of 2D face recognition technology such as the Face Recognition Vendor Test32 and face verification competition29. There is a general consensus that the existing 2D face recognition technology performs quite well in controlled conditions, where the subject is presented in a frontal pose under good illumination which is the same for images used for training and those acquired during the subsequent (test) operation. However, it has been observed that the performance can degrade very rapidly when the imaging conditions change. This lack of robustness renders current systems unsuitable for many applications where invariance of the imaging conditions cannot be guaranteed. The above sensitivity of 2D face recognition solutions can be attributed to several factors. First of all, the reliance on holistic approaches of the currently favoured recognition methods, whereby the face image pixels define the input measurement space for feature extraction and decision making, makes the recognition process very sensitive to registration errors. Basically two images can be compared only when they have been transformed so that certain landmarks in the respective pair coincide. If the images are misregistered, the decision making process will tend to compare measurements that do not correspond to each other and the result of the comparison will be meaningless. The second major contributor is the pose. If subject's pose deviates from the frontal, different parts of the face are imaged and this again destroys comparability. This problem can, to a certain degree, be mitigated by transforming the probe image into a canonical position22. However, this solution is riddled with difficulties. It again relies on accurate landmark detection. The true transformation is nonlinear and subject dependent and therefore unknown. Most importantly, the method cannot recover the

3D Assisted Face Recognition: A Survey

5

information lost due to lack of visibility. The alternative is to design a separate system for each pose26. However, this would require a huge quantity of training data, and even if that was available it could not be done without the quantisation of the view sphere which would again lead to inaccuracies. The use of statistical models11 to capture the variability due to pose changes has also proved not fully successful. Third, and perhaps most influential on performance, is the effect of illumination. The reflected light captured by the camera is a complex function of the surface geometry, albedo, illumination and the spectral characteristics of the sensor. Even for pose registered faces, a change in illumination dramatically affects the pixel measurements and therefore their comparability. This problem has again been tackled in a number of different ways including training over different illumination scenarios, illumination modelling, image synthesis, and photometric normalisation methods to mention just a few, but so far only with a limited success. The above list of problems motivated a radically different approach to face recognition, which is based on 3D properties of the face. This is the only information that is invariant in face imaging and should therefore constitute a solid basis for face recognition. The aim of this chapter is to review the methods of 3D face data acquisition and modelling as well as the various ways the 3D model can be used for face recognition and verification. The review differs from previous efforts in a number of respects. First of all, it focuses on the 3D sensing technology, discussing the principles of active sensing and advantages and drawbacks of the currently available solutions. Second, it includes a review of 3D face representation methods. Finally, in contrast to Bowyer et al.5, in addition to discussing separate 2D and 3D based recognition and the fusion of these modalities, we also address the problem of 3D assisted 2D recognition. The chapter is organised as follows. In the next section we briefly review the various methods of sensing 3D facial biometrics. Modelling and representation of 3D facial data is discussed in Section 3. The role of 3D in landmark detection and face registration is discussed in Section 4. Section 5 expounds on the use of 3D in face recognition. The review is drawn to conclusion in Section 6.

2.

3D SENSING FOR FACIAL BIOMETRICS

Facial biometrics can utilise 3D reconstruction of faces in two contexts: Enrolment: Offline database capture of individuals for use in training and as exemplars for verification and/or recognition. Identification: Online face capture for identity recognition or verification.

M. Hamouz et al.

6

Table 1-1 summarises important requirements for 3D face capture in each of these contexts. All applications require the simultaneous capture of 3D shape and 2D appearance with known alignment between them. A primary difference for identification is the requirement for instantaneous single shot capture in a relatively uncontrolled environment (variable illumination, subject movement etc.). Face capture for enrolment purposes can be performed in a controlled environment with the subject asked to remove glasses and remain static or perform set expressions. To model intra and inter person variability database capture requires acquisition of multiple expressions and also at least the appearance of elements such as facial hair (many active 3D sensing systems fail to reliably capture hair). For identification purposes the subject can not be assumed to be cooperative, therefore the capture system should be able to cope with movement, variation in expression and additional elements such as glasses. Sensor accuracy and spatial resolution for both enrolment and identification will also impact on performance. Accuracy of shape measurement required to resolve facial features is expected to be in the range 1-5 mm. In this section we review how current visual reconstruction approaches meet the requirements for 3D facial biometrics. Visual reconstruction techniques are reviewed in two categories: active sensing where a structured illumination pattern is projected onto the face to facilitate reconstruction; and passive sensing where reconstruction is performed directly from the facial appearance in images or video. Table 1-1. Requirements of 3D reconstruction for facial biometrics (• firm, p=possible). Shape Registered appearance Single-shot capture Robust to variable light Variation in shape Variation in appearance Facial hair Glasses Sequence Capture

2.1

Enrolment • p

p p p p p

Identification • • • •

• p

Passive sensing

Reconstruction of shape from multiple view images and video has produced a number of Shape-from-X techniques. The principal limitation in the application of these approaches to faces is the relatively uniform appearance resulting in low-accuracy reconstruction. Potential advantages of

3D Assisted Face Recognition: A Survey

7

passive techniques include reconstruction of faces from image sequences with natural illumination, simultaneous acquisition of colour appearance and video-rate shape capture. To overcome limitations on accuracy, model-based approaches14, 47 have been developed to constrain face reconstruction in regions of uniform appearance. Introduction of prior models to regularize the reconstruction has the potential to incorrectly reconstruct detailed features in facial shape which are uniform in appearance. The relatively low-accuracy and reliability of passive facial reconstruction approaches has resulted in active sensing being widely used for acquisition of face shape. Increases in digital camera resolution offer the potential to overcome limitations of uniform appearance of the face allowing accurate passive reconstruction.

2.2

Active sensing

A number of active sensing technologies have been developed for 3D surface measurement, which operate on the principle of projecting a structured illumination pattern into the scene to facilitate 3D reconstruction. 3D acquisition systems work on two principles: time-of-flight; and triangulation. Time-of-flight sensors measure the time taken for the projected illumination pattern to return from the object surface. This sensor technology requires nanosecond timing to resolve surface measurements to millimetre accuracy. A number of commercial time-of-flight systems are available based on scanning a point or stripe across the scene (Riegl, SICK). Area based systems have also been developed which project a light pulse which covers the entire scene allowing single-shot capture of moving scenes (Z-cam). The requirement for high-speed timing has limited the application of time-of-flight to systems for large-scale environment measurement with centimetre accuracy. Active systems which reconstruct depth by triangulation between projector-to-camera or camera-to-camera allow accurate reconstruction of depth at short range. Projection of a structured illumination pattern facilitates correspondence analysis and accurate feature localisation. Point and stripe patterns provide visually distinct features with accurate localisation allowing reconstruction of depth to µ m accuracy for short range measurement (