Image and Graphics, 9 conf, ICIG 2017, Part 3 978-3-319-71597-1, 3319715976, 978-3-319-71598-8

Citation preview

LNCS 10668

Yao Zhao · Xiangwei Kong David Taubman (Eds.)

Image and Graphics 9th International Conference, ICIG 2017 Shanghai, China, September 13–15, 2017 Revised Selected Papers, Part III

123

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, Lancaster, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Zurich, Switzerland John C. Mitchell Stanford University, Stanford, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Dortmund, Germany Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbrücken, Germany

10668

More information about this series at http://www.springer.com/series/7412

Yao Zhao Xiangwei Kong David Taubman (Eds.) •

Image and Graphics 9th International Conference, ICIG 2017 Shanghai, China, September 13–15, 2017 Revised Selected Papers, Part III

123

Editors Yao Zhao Beijing Jiaotong University Beijing China

David Taubman UNSW Sydney, NSW Australia

Xiangwei Kong Dalian University of Technology Dalian China

ISSN 0302-9743 ISSN 1611-3349 (electronic) Lecture Notes in Computer Science ISBN 978-3-319-71597-1 ISBN 978-3-319-71598-8 (eBook) https://doi.org/10.1007/978-3-319-71598-8 Library of Congress Control Number: 2017960877 LNCS Sublibrary: SL6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics © Springer International Publishing AG 2017 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Printed on acid-free paper This Springer imprint is published by Springer Nature The registered company is Springer International Publishing AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

These are the proceedings of the 8th International Conference on Image and Graphics (ICIG 2017), held in Shanghai, China, during September 13–15, 2017. The China Society of Image and Graphics (CSIG) has hosted this series of ICIG conferences since 2000. ICIG is the biennial conference organized by the China Society of Image and Graphics (CSIG), focusing on innovative technologies of image, video, and graphics processing and fostering innovation, entrepreneurship, and networking. This time, Shanghai Jiaotong University was the organizer, and the Nanjing Technology University and Zong Mu Technology Ltd. Company were the co-organizers. Details about the past eight conferences, as well as the current one, are as follows: Conference First (ICIG 2000) Second (ICIG 2002) Third (ICIG 2004) 4th (ICIG 2007) 5th (ICIG 2009) 6th (ICIG 2011) 7th (ICIG 2013) 8th (ICIG 2015) 9th (ICIG 2017)

Place Tianjin, China Hefei, China Hong Kong, China Chengdu, China Xi’an, China Hefei, China Qingdao, China Tianjin, China Shanghai, China

Date August 16–18 August 15–18 December 17–19 August 22–24 September 20–23 August 12–15 July 26–28 August 13–16 September 13–15

Submitted 220 280 460 525 362 329 346 345 370

Proceeding 156 166 140 184 179 183 181 170 172

This time, the proceedings are published by Springer in the LNCS series. The titles, abstracts, and biographies of the three invited speakers of plenary talks are presented first. At ICIG 2017, 370 submissions were received, and 160 papers were accepted. To ease in the search of a required paper in these proceedings, the 160 regular papers have been arranged in alphabetical order according to their titles. Another 12 papers forming a special topic are included at the end. Our sincere thanks go to all the contributors (around 200), who came from around the world to present their advanced works at this event. Special thanks go to the members of Technical Program Committee, who carefully reviewed every single submission and made their valuable comments for improving the accepted papers. The proceedings could not have been produced without the invaluable efforts of the publication chairs, the web chairs, and a number of active members of CSIG. September 2017

Yao Zhao Xiangwei Kong David Taubman

Organization

Honorary Chairs Guanhua Xu Yuan F. Zheng

MOST, China Ohio State University, USA

General Chairs Tieniu Tan Hongkai Xiong Zixiang Xiong

Chinese Academy of Sciences, China Shanghai Jiaotong University, China Texas A&M University, USA

Organizing Committee Chairs Weiyao Lin Huimin Ma Bo Yan

Shanghai Jiaotong University, China Tsinghua University, China Fudan University, China

Technical Program Chairs David Taubman Yao Zhao

UNSW, Australia Beijing Jiaotong University, China

Finance Chairs Zhihua Chen Zhenwei Shi

ECUST, China Beihang University, China

Special Session Chairs Jian Cheng Zhihai He Z. Jane Wang

Chinese Academy of Sciences, China University of Missouri, USA University of British Columbia, Canada

Award Chairs Xin Li Shiqiang Yang

West Virginia University, USA Tsinghua University, China

VIII

Organization

Publicity Chairs Mingming Cheng Moncef Gabbouj

Nankai University, China TUT, Finland

Exhibits Chairs Zhijun Fang Yan Lv

Shanghai University of Engineering Science, China Microsoft Research, China

Publication Chairs Xiangwei Kong Jun Yan

Dalian University of Technology, China Journal of Image and Graphics, China

International Liaisons Xiaoqian Jiang Huifang Sun

UCSD, USA MERL, USA

Local Chairs Wenrui Dai Junni Zou

UCSD, USA Shanghai Jiaotong University, China

Registration Chair Chen Ye

Shanghai Jiaotong University, China

Webmasters Chenglin Li Yangmei Shen

EPFL, Switzerland Shanghai Jiaotong University, China

Technical Program Committee Ping An Ru An Xiao Bai Lianfa Bai Xiang Bai Chongke Bi Hai Bian Xiaochun Cao

Shanghai University, China Hohai University, China Beijing University of Aeronautics and Astronautics, China Nanjing University of Science and Technology, China Huazhong University of Science and Technology, China Tianjin University, China Hangzhou Dica3d Technology Co., Ltd., China Institute of Information Engineering, Chinese Academy of Sciences, China

Organization

Yan-Pei Cao Chong Cao Qi Chen Kang Chen Mingkai Chen Mingming Cheng Yue Dong Zhijun Fang Qianjin Feng Xiaoyi Feng Dongmei Fu Junying Gan Lin Gao Yue Gao Xinbo Gao Zexun Geng Guanghua Gu Lin Gu Yanwen Guo Hu Han Xiaowei He Qiming Hou Dong Hu Hua Huang Haozhi Huang Yongfeng Huang Rongrong Ji Yunde Jia Sen Jia Xiuping Jia Zhiguo Jiang Zhaohui Jiang Xiaoqian Jiang Lianwen Jin Bin Kong Xiangwei Kong Dengfeng Kuang Jianhuang Lai Congyan Lang Changhua Li Chenglin Li Hua Li Jiming Li

IX

Tsinghua University, China Tsinghua University, China Hainan University, China Tsinghua University, China Nanjing University of Posts and Telecommunications, China Nankai University, China MSRA, China Shanghai University of Engineering Science, China Southern Medical University, China Northwestern Polytechnical University, China University of Science and Technology Beijing, China Wuyi University, China ICT, CAS, China Tsinghua University, China Xidian University, China Information Engineering University, China Yanshan University, China National Institute of Informatics, Japan Nanjing University, China Nanyang Technological University, Singapore Northwest University, China Zhejiang University, China Nanjing University of Posts and Telecommunications, China Beijing Institute of Technology, China Tsinghua University, China Tsinghua University, China Xiamen University, China Beijing Institute of Technology, China Shenzhen University, China University of New South Wales, USA Beijing University of Aeronautics and Astronautics, China Central South University, China University of California, San Diego, USA South China University of Technology, China Institute of Intelligent Machines, Chinese Academy of Sciences, China Dalian University of Technology, China Nankai University, China Sun Yat-Sen University, China Beijing Jiaotong University, China Xi’an University of Architecture and Technology, China Swiss Federal Institute of Technology in Lausanne, Switzerland Institute of Computing Technology, Chinese Academy of Sciences, China Zhejiang Police College, China

X

Organization

Qi Li Shutao Li Xi Li Jie Liang Pin Liao Chunyu Lin Xiaojing Liu Changhong Liu Bin Liu Bin Liu Chenglin Liu Wenyu Liu Yue Liu Qingshan Liu Hongbing Lu Hanqing Lu Jiwen Lu Jianhua Ma Huimin Ma Weidong Min Xuanqin Mou Taijiang Mu Feiping Nie Yongwei Nie Zhigeng Pan Yanwei Pang Yuxin Peng Yuntao Qian Bo Ren Jun Sang Nong Sang Yangmei Shen Yuying Shi Huifang Sun Jiande Sun Linmi Tao Lei Tong Yunhai Wang Qi Wang Cheng Wang Meng Wang Hanzi Wang Peizhen Wang Tianjiang Wang

Peking University, China Hunan University, China Zhejiang University, China China Aerodynamics Research and Development Center, China Nanchang University, China Beijing Jiaotong University, China Qinghai University, China Jiangxi Normal University, China University of Science and Technology of China, China Tsinghua University, China Institute of Automation, Chinese Academy of Sciences, China Huazhong University of Science and Technology, China Beijing Institute of Technology, China Nanjing University of Information Science and Technology, China Fourth Military Medical University, China Institute of Automation, Chinese Academy of Sciences, China Tsinghua University, China Southern Medical University, China Tsinghua University, China Nanchang University, China Xi’an Jiaotong University, China Tsinghua University, China Northwestern Polytechnical University, China South China University of Technology, China Hangzhou Normal University, China Tianjin University, China Peking University, China Zhejiang University, China Nankai University, China Chongqing University, China Huazhong University of Science and Technology, China Shanghai Jiaotong University, China North China Electric Power University, China Mitsubishi Electric Research Laboratories, USA Shandong University, China Tsinghua University, China Beijing University of Technology, China Shandong University, China Northwestern Polytechnical University, China Xiamen University, China Hefei University of Technology, China Xiamen University, China Anhui University of Technology, China Huazhong University of Science and Technology, China

Organization

Bin Wang Lili Wang Shigang Wang Miao Wang Yunhong Wang Chunhong Wu Hongzhi Wu Xiaojun Wu Fei Wu Zhongke Wu Dingyuan Xia Hongkai Xiong Mingliang Xu Chunxu Xu Kun Xu Zengpu Xu Jianru Xue Xiangyang Xue Bo Yan Ling-Qi Yan Xiao Yan Jingwen Yan Jun Yan Jinfeng Yang Sheng Yang Yongliang Yang Shiqiang Yang Tao Yang Hongxun Yao Yong Yin Shiqi Yu Nenghai Yu Yinwei Zhan Aiqing Zhang Wei Zhang Daoqiang Zhang Jiawan Zhang Lei Zhang Song-Hai Zhang Shiliang Zhang Xinpeng Zhang Yanci Zhang Yongfei Zhang Fang-Lue Zhang Guofeng Zhang

XI

Tsinghua University, China Beihang University, China Jilin University, China Tsinghua University, China Beijing University of Aeronautics and Astronautics, China University of Science and Technology Beijing, China Zhejiang University, China Jiangnan University, China Zhejiang University, China Beijing Normal University, China Wuhan University of Technology, China Shanghai Jiaotong University, China Zhengzhou University, China Tsinghua University, China Tsinghua University, China Tianjin University of Science and Technology, China Xi’an Jiaotong University, China Fudan University, China Fudan University, China UC Berkeley, USA Tsinghua University, China Shantou University, China Institute of Remote Sensing and Digital Earth, Chinese Academy of Sciences, China Civil Aviation University of China, China Tsinghua University, China Bath University, UK Tsinghua University, China Tsinghua University, China Harbin Institute of Technology, China Dalian Maritime University, China Shenzhen University, China University of Science and Technology of China, China Guangdong University of Technology, China Anhui Normal University, China Shandong University, China Nanjing University of Aeronautics and Astronautics, China Tianjin University, China Beijing Institute of Technology, China Tsinghua University, China Peking University, China Shanghai University, China Sichuan University, China Beijing University of Aeronautics and Astronautics, China Victoria University of Wellington, New Zealand Zhejiang University, China

XII

Organization

Qiang Zhang Yun Zhang Liangpei Zhang Shengchuan Zhang Xiaopeng Zhang Sicheng Zhao Yao Zhao Jieyu Zhao Chunhui Zhao Ying Zhao Wei-Shi Zheng Ping Zhong Quan Zhou Jun Zhou Liang Zhou Linna Zhou Tao Zhou Wengang Zhou Zhe Zhu Wang-Jiang Zhu Yonggui Zhu

Dalian University, China Zhejiang University of Media and Communications, China Wuhan University, China Xiamen University, China Shanghai Jiaotong University, China Tsinghua University, China Beijing Jiaotong University, China Ningbo University, China Harbin Engineering University, China Central South University, China Sun Yat-Sen University, China National University of Defense Technology, China China Academy of Space Technology, Xi’an, China Griffith University, Australia Nanjing University of Posts and Telecommunications, China University of International Relations, China Ningxia Medical University, China University of Science and Technology of China, China Duke University, USA Tsinghua University, China Communication University of China, China

Contents – Part III

Computational Imaging A Double Recursion Algorithm to Image Restoration from Random Limited Frequency Data . . . . . . . . . . . . . . . . . . . . . . . . . . . Xiaoman Liu and Jijun Liu

3

RGB-D Saliency Detection with Multi-feature-fused Optimization . . . . . . . . . Tianyi Zhang, Zhong Yang, and Jiarong Song

15

Research on Color Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . Jingwen Yan, Xiaopeng Chen, Tingting Xie, and Huimin Zhao

27

Depth-Based Focus Stacking with Labeled-Laplacian Propagation . . . . . . . . . Wentao Li, Guijin Wang, Xuanwu Yin, Xiaowei Hu, and Huazhong Yang

36

A Novel Method for Measuring Shield Tunnel Cross Sections . . . . . . . . . . . Ya-dong Xue, Sen Zhang, and Zhe-ting Qi

47

Image Noise Estimation Based on Principal Component Analysis and Variance-Stabilizing Transformation . . . . . . . . . . . . . . . . . . . . Ling Ding, Huying Zhang, Bijun Li, Jinsheng Xiao, and Jian Zhou Efficient High Dynamic Range Video Using Multi-exposure CNN Flow . . . . Yuchen Guo, Zhifeng Xie, Wenjun Zhang, and Lizhuang Ma Depth Image Acquisition Method in Virtual Interaction of VR Yacht Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qin Zhang and Yong Yin An Image Segmentation Method Based on Asynchronous Multiscale Similarity Measure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Min Li, Zhongwai Xu, Hongwen Xie, and Yuhang Xing

58

70

82

93

Lagrange Detector in Image Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . Feilong Ma, Linmi Tao, and Wu Xia

102

New Tikhonov Regularization for Blind Image Restoration . . . . . . . . . . . . . Yuying Shi, Qiao Liu, and Yonggui Zhu

113

XIV

Contents – Part III

Real-Time Multi-camera Video Stitching Based on Improved Optimal Stitch Line and Multi-resolution Fusion. . . . . . . Dong-Bin Xu, He-Meng Tao, Jing Yu, and Chuang-Bai Xiao Image Quality Assessment of Enriched Tonal Levels Images . . . . . . . . . . . . Jie Zhao, Wei Wen, and Siamak Khatibi

124 134

Computer Graphics and Visualization A Variational Model to Extract Texture from Noisy Image Data with Local Variance Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tao Zhang and Qiuli Gao

149

Joint Visualization of UKF Tractography Data . . . . . . . . . . . . . . . . . . . . . . Wen Zhao, Wenyao Zhang, Na Wang, and Pin Liao

158

Semantic Segmentation Based Automatic Two-Tone Portrait Synthesis . . . . . Zhuoqi Ma, Nannan Wang, Xinbo Gao, and Jie Li

170

Parameters Sharing Multi-items Non-parametric Factor Microfacet Model for Isotropic and Anisotropic BRDFs . . . . . . . . . . . Junkai Peng, Changwen Zheng, and Pin Lv SRG and RMSE-Based Automated Segmentation for Volume Data . . . . . . . . Wang Li, Xiaoan Tang, and Junda Zhang Shape Recovery of Endoscopic Videos by Shape from Shading Using Mesh Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhihang Ren, Tong He, Lingbing Peng, Shuaicheng Liu, Shuyuan Zhu, and Bing Zeng Lazy Recoloring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Guanlei Xu, Xiaotong Wang, Xiaogang Xu, and Lijia Zhou Similar Trademark Image Retrieval Integrating LBP and Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tian Lan, Xiaoyi Feng, Zhaoqiang Xia, Shijie Pan, and Jinye Peng Adaptive Learning Compressive Tracking Based on Kalman Filter . . . . . . . . Xingyu Zhou, Dongmei Fu, Yanan Shi, and Chunhong Wu Online High-Accurate Calibration of RGB+3D-LiDAR for Autonomous Driving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tao Li, Jianwu Fang, Yang Zhong, Di Wang, and Jianru Xue

182

194

204

214

231

243

254

Contents – Part III

Run-Based Connected Components Labeling Using Double-Row Scan . . . . . Dongdong Ma, Shaojun Liu, and Qingmin Liao A 3D Tube-Object Centerline Extraction Algorithm Based on Steady Fluid Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dongjin Huang, Ruobin Gong, Hejuan Li, Wen Tang, and Youdong Ding

XV

264

275

Moving Objects Detection in Video Sequences Captured by a PTZ Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Li Lin, Bin Wang, Fen Wu, and Fengyin Cao

287

Fast Grid-Based Fluid Dynamics Simulation with Conservation of Momentum and Kinetic Energy on GPU . . . . . . . . . . . Ka-Hou Chan and Sio-Kei Im

299

Adaptive Density Optimization of Lattice Structures Sustaining the External Multi-load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Li Shi, Changdong Zhang, Tingting Liu, Wenhe Liao, and Xiuyi Jia

311

Hyperspectral Image Processing Hyperspectral Image Classification Based on Deep Forest and Spectral-Spatial Cooperative Feature . . . . . . . . . . . . . . . . . . . . . . . . . . Mingyang Li, Ning Zhang, Bin Pan, Shaobiao Xie, Xi Wu, and Zhenwei Shi Hyperspectral Image Classification Using Multi Vote Strategy on Convolutional Neural Network and Sparse Representation Joint Feature. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daoming Ye, Rong Zhang, and Dixiu Xue Efficient Deep Belief Network Based Hyperspectral Image Classification . . . . Atif Mughees and Linmi Tao Classification of Hyperspectral Imagery Based on Dictionary Learning and Extended Multi-attribute Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qishuo Gao, Samsung Lim, and Xiuping Jia Deep Residual Convolutional Neural Network for Hyperspectral Image Super-Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chen Wang, Yun Liu, Xiao Bai, Wenzhong Tang, Peng Lei, and Jun Zhou

325

337 347

358

370

XVI

Contents – Part III

Multi-view and Stereoscopic Processing Stereoscopic Digital Camouflage Pattern Generation Algorithm Based on Color Image Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Qin Lei, Wei-dong Xu, Jiang-hua Hu, and Chun-yu Xu

383

Uncertain Region Identification for Stereoscopic Foreground Cutout . . . . . . . Taotao Yang, Shuaicheng Liu, Chao Sun, Zhengning Wang, and Bing Zeng

390

Map-Build Algorithm Based on the Relative Location of Feature Points . . . . Cheng Zhao, Fuqiang Zhao, and Bin Kong

400

Sparse Acquisition Integral Imaging System . . . . . . . . . . . . . . . . . . . . . . . . Bowen Jia, Shigang Wang, Wei Wu, Tianshu Li, and Lizhong Zhang

412

Marker-Less 3D Human Motion Capture in Real-Time Using Particle Swarm Optimization with GPU-Accelerated Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bogdan Kwolek and Boguslaw Rymut Warping and Blending Enhancement for 3D View Synthesis Based on Grid Deformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ningning Hu, Yao Zhao, and Huihui Bai A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm Based on Ship-Shaped Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Wang, Chunyu Lin, Yi Gao, Yaru Li, Shikui Wei, and Yao Zhao A Quality Evaluation Scheme to 3D Printing Objects Using Stereovision Measurement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Li-fang Wu, Xiao-hua Guo, Li-dong Zhao, and Meng Jian

423

436

445

458

Representation, Analysis and Applications of Large-Scale 3D Multimedia Data Secure Image Denoising over Two Clouds . . . . . . . . . . . . . . . . . . . . . . . . . Xianjun Hu, Weiming Zhang, Honggang Hu, and Nenghai Yu

471

Aesthetic Quality Assessment of Photos with Faces. . . . . . . . . . . . . . . . . . . Weining Wang, Jiexiong Huang, Xiangmin Xu, Quanzeng You, and Jiebo Luo

483

Contents – Part III

Sensitive Information Detection on Cyber-Space . . . . . . . . . . . . . . . . . . . . . Mingbao Lin, Xianming Lin, Yunhang Shen, and Rongrong Ji Optimization Algorithm Toward Deep Features Based Camera Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Han Chen, Feng Guo, Ying Lin, and Rongrong Ji

XVII

496

505

Security A Robust 3D Video Watermarking Scheme Based on Multi-modal Visual Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . Congxin Cheng, Wei Ma, Yuchen Yang, Shiyang Zhang, and Mana Zheng Partial Secret Image Sharing for (n, n) Threshold Based on Image Inpainting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xuehu Yan, Yuliang Lu, Lintao Liu, Shen Wang, Song Wan, Wanmeng Ding, and Hanlin Liu A New SMVQ-Based Reversible Data Hiding Scheme Using State-Codebook Sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Juan-ni Liu, Quan Zhou, Yan-lang Hu, and Jia-yuan Wei

517

527

539

An Efficient Privacy-Preserving Classification Method with Condensed Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Xinning Li and Zhiping Zhou

551

Cross-Class and Inter-class Alignment Based Camera Source Identification for Re-compression Images . . . . . . . . . . . . . . . . . . . . . . . . . . Guowen Zhang, Bo Wang, and Yabin Li

563

JPEG Photo Privacy-Preserving Algorithm Based on Sparse Representation and Data Hiding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wenjie Li, Rongrong Ni, and Yao Zhao

575

Surveillance and Remote Sensing An Application Independent Logic Framework for Human Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wengang Feng, Yanhui Xiao, Huawei Tian, Yunqi Tang, and Jianwei Ding

589

XVIII

Contents – Part III

An Altitude Based Landslide and Debris Flow Detection Method for a Single Mountain Remote Sensing Image . . . . . . . . . . . . . . . . . . . . . . Tingting Sheng and Qiang Chen

601

Improved Fully Convolutional Network for the Detection of Built-up Areas in High Resolution SAR Images . . . . . . . . . . . . . . . . . . . Ding-Li Gao, Rong Zhang, and Di-Xiu Xue

611

SAR Image Registration with Optimized Feature Descriptor and Reliable Feature Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yanzhao Wang, Juan Su, Bichao Zhan, Bing Li, and Wei Wu

621

An Improved Feature Selection Method for Target Discrimination in SAR Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yanyan Li and Aihua Cai

633

The Detection of Built-up Areas in High-Resolution SAR Images Based on Deep Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yunfei Wu, Rong Zhang, and Yue Li

646

SAR Automatic Target Recognition Based on Deep Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Xu, Kaipin Liu, Zilu Ying, Lijuan Shang, Jian Liu, Yikui Zhai, Vincenzo Piuri, and Fabio Scotti Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

656

669

Computational Imaging

A Double Recursion Algorithm to Image Restoration from Random Limited Frequency Data Xiaoman Liu and Jijun Liu(B) School of Mathematics, Southeast University, Nanjing 211189, People’s Republic of China [email protected]

Abstract. One of the main tasks in image restoration is to catch the picture characteristics such as interfaces and textures from incomplete noisy frequency data. For the cost functional with data matching term in frequency domain and the total variation together with Frobenius norm penalty terms in spatial domain, the properties of the minimizer of cost functional and the error estimates on the regularizing solution are established. Then we propose an algorithm with double recursion to restore piecewise smooth image. The Bregman iteration with lagged diffusivity fixed point method is used to solve the corresponding nonlinear EulerLagrange equation. By implementing recursion algorithms a few times, the satisfactory reconstructions can be obtained using random band sampling data. Numerical implementations demonstrate the validity of our proposed algorithm with good edge-preservations. Keywords: Image restoration · Random sampling · Total variation Bregman iteration · Adjoint conjugate gradient method

1

Introduction

A lot of engineering backgrounds such as objects detections for military purposes and magnetic resonance imaging (MRI) for medical applications, lead to the studies on image restorations. The main tasks of image restorations aim to the recovery of an image from given noisy measurement data. However, in most engineering configurations, the specified measurement data are limited frequency data instead of the full data in spatial domain. Therefore the image restorations are essentially of the nature of ill-posedness [1]. Mathematically, the image restorations require the stable approximations to the target image from insufficient noisy measurement data, which are also one of the most important research areas of applied mathematics by dealing with the ill-posedness of the image restorations. In recent decades, various image restoration models and mathematical techniques have been developed due to the great importance of image restoration c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 3–14, 2017. https://doi.org/10.1007/978-3-319-71598-8_1

4

X. Liu and J. Liu

problems, such as level set methods, wavelet-based frameworks, nonlinear PDEs models and optimization schemes [2–4]. In all these studies, the basic ideas are the reconstructions of an image by some denoising process, while the key information about the image should be kept in terms of the regularization techniques. The mathematical framework dealing with ill-posed problems is the regularizing scheme with some appropriate penalty terms incorporating into the cost functional. The key issue for this scheme is that the suitable weight called the regularizing parameters between the data matching term and the penalty terms should be specified artificially to keep the balance between the data matching and smoothness of the sought solution. In the cases that the exact solution to be sought is smooth, the penalty terms for the solution can be measured by some standard norms such as L2 or H 2 norm for which the choice strategy for the regularizing parameters have been studied thoroughly [1, 4]. However, for nonsmooth exact solution such as those in the image restorations with the sharp jump of the grey level function of the image, instead of the standard differential norms, the other non-differential penalty term such as total variation (TV) or L0 -norm sparsity penalty term should be applied [2, 3]. Motivated by the above engineering backgrounds, we consider an image recovery problem using the incomplete noisy frequency data by minimizing a cost functional with penalty terms in Sect. 2. Based on the derivatives expressions of the cost functional, an iterative scheme with outer and inner recursions are proposed in Sect. 3 to solve the minimizing problem. Finally, some numerical experiments are presented to show the validity of the proposed scheme in Sect. 4.

2

Optimization Modeling with Error Analysis

As the standard configurations in signal processing, we define the Fourier Transform matrix F ∈ RN ×N with the components 2π

Fi,j = e−i N ij ,

i, j = 1, · · · , N.

(2.1)

It is well-known that the matrix F is an unitary matrix [10] satisfying F ∗ F = I, where the superscript ∗ denotes the conjugate transpose of an matrix, I is identity matrix. In most computer vision problems, the two-dimensional image f := (fm,n )(m, n = 1, · · · , N ) can be represented as a vector f . Here we introduce some symbols for this representation as follows: vect : RN ×N  operator 2

2

→ RN ×1 : vect[f ] := (f1 , f2 , · · · , fN 2 )T = f , where the N -elements are generated by re-ordering the N column vectors of f sequently.  operator array: the inverse of the vect, i.e., array[f ] = f .  Two-dimensional discrete Fourier transform (DFT) matrix F: ˆ f := vect[F T f F ] = (F ⊗ F )f := Ff , where ⊗ is the tensor product of two matrices.

A Double Recursion Algorithm to Image Restoration

5

By partial frequency data of fˆ, we mean that only parts of the elements of fˆ are sampled, with other elements considered as 0. Denoted by P the N × N matrix generating from the identity matrix I by setting its N –M rows as null vectors, i.e., P = diag(p11 , p22 , · · · , pN N ) with pii being 1 or 0. Then P fˆ means that we only take M (≤ N ) rows of fˆ as our partial data. To the vector form f , the sampling matrix should be modified as an N 2 × N 2 matrix. Then we have the following notations with tensor product: vect[P fˆ] = (I ⊗ P )vect[fˆ] = (I ⊗ P )ˆ f := Pˆ f.

(2.2)

Obviously, the matrix P is the sampling matrix in algorithm domain which is chosen before reconstruction. Generally, the frequency data of an image f are obtained by some scanning process, which are of unavoidable error, i.e., our inversion data for image recovery are in fact P gˆδ with noisy data gˆδ of fˆ satisfying P fˆ − P gˆδ F ≤ fˆ − gˆδ F ≤ δ,

(2.3)

where  · F is the Frobenius norm for an N × N matrix, corresponding to the 2-norm of an N 2 -dimensional vector after stacking the elements of an N × N matrix into an N 2 -dimensional vector. Hence the data matching term can be written as P fˆ − P gˆδ 2F = P(Ff ) − Pˆ gδ 22 .

(2.4)

There are various sampling matrices, like those in radial lines sampling method [12], band sampling method [6] which is much efficient in numerical experiments. In this paper, we apply the random band sampling process which samples some rows randomly. Denoted by cenR the central ratio, i.e., to the frequency image there are only cenR × N rows in the central parts of the frequency image (the centre is between No. N/2−1 and N /2 rows) in Cartesian coordinate system or in the natural domain coordinate which is shown in Fig. 1(b), and the others sampling rows are the random ones. In the algorithm domain shown in Fig. 1(a), the sampling rows are distributed on the four corners, i.e., it is the same distribution as the mask as usual.

Fig. 1. (a) Algorithm domain coordinate; (b) Natural domain coordinate.

Recall that the most important issue for image restorations is the edgepreservation property of the image, which means that we are essentially interested in the efficient reconstruction for a piecewise constant image with the

6

X. Liu and J. Liu

main interests in detecting image edges. Since the total variation (TV) of a twodimensional function can describe the function jumps efficiently, we are then led to the following constraint optimization problem   |f |T V : P F T f F − P gˆδ 2F ≤ δ , (2.5) min f ∈RN ×N

where P gˆδ is the incomplete noisy frequency data, and TV penalty term |f |T V for f is defined in a standard way [6, 12]. Since |f |T V is not differentiable at f = Θ (zero matrix), we approximate |f |T V by |f |T V,β =

 N 

2

2

2 1 f) + β f ) + (∇xm,n (∇xm,n

(2.6)

m,n=1

  2 1 for small constant β > 0, where ∇m,n f := ∇xm,n f with two components f, ∇xm,n 1 ∇xm,n f

=



fm+1,n − fm,n , if m < N, x2 ∇m,n f = f1,n − fm,n , if m = N,

 fm,n+1 − fm,n , fm,1 − fm,n ,

if n < N, if n = N

for m, n = 1, · · · , N due to the periodic boundary condition on f . However, the constraint optimization problem (2.5) in the case of P = I has no restrictions on the size of f , notice that P F T XF = Θ may have nonzero solution X arbitrarily large for singular matrix P . To exclude this uncertainty, our image recovery problem is finally reformulated as the following unconstraint problem  ∗ f := arg min Jβ (f ), f (2.7) Jβ (f ) := 12 P F T f F − P gˆδ 2F + α1 f 2F + α2 |f |T V,β . where α1 , α2 > 0 are regularizing parameters. The theorems below illustrate the existence of the minimizer and establish the choice strategies for the regularizing parameters α1 , α2 . Theorem 1. For α1 > 0, α2 , β ≥ 0, there exists a local minimizer to the optimization problem (2.7). Proof. Since Jβ (f ) ≥ 0 for f ∈ RN ×N , there exists a constant J ∗ ≥ 0 such that J ∗ = inf Jβ (f ). So there exists a matrix sequence {f k ∈ RN ×N : k = 1, 2, · · · } f

such that lim Jβ (f k ) = J ∗ , which means α1 f k 2F ≤ Jβ (f k ) ≤ C0 for k = k→∞

1, 2, · · · , i.e., f k 2F ≤ C0 /α1 . Therefore there exists a subsequence matrix of {f k : k = 1, 2, · · · }, still denoted by {f k : k = 1, 2, · · · }, such that lim f k = f ∗ . k→∞

Notice that |f |T V,β is also continuous with respect to f by (2.6), the continuity of Jβ (f ) with respect to f yields Jβ (f ∗ ) = lim Jβ (f k ) = J ∗ = inf Jβ (f ), k→∞

i.e., f ∗ is the minimizer of Jβ (f ). The proof is complete.

f

⊓ ⊔

A Double Recursion Algorithm to Image Restoration

7

Theorem 2. Denote by f † ∈ RN ×N the exact image. Then the minimizer f ∗ = fα∗1 ,α2 ,β,δ satisfies the following estimates √ P F T fα∗1 ,α2 ,β,δ F − P gˆδ 2F ≤ δ 2 + α1 f † 2F + 2α2 N 2 β + 2α2 |f † |T V , (2.8) √ α2 † α2 δ2 (2.9) +α |f |T V + α N 2 β + f † 2F , fα∗1 ,α2 ,β,δ 2F ≤ 2α 1 1 1 √ α1 δ2 ∗ † † 2 2 (2.10) |fα1 ,α2 ,β,δ |T V,β ≤ 2α2 + α2 f F + N β + |f |T V . Proof. Since fα∗1 ,α2 ,β,δ is the minimizer, we have

≤ ≤ = ≤

1 P F T fα∗1 ,α2 ,β,δ F − P gˆδ 2F + α1 fα∗1 ,α2 ,β,δ 2F + α2 |fα∗1 ,α2 ,β,δ |T V,β 2 1 P F T f † F − P gˆδ 2F + α1 f † 2F + α2 |f † |T V,β 2 1 2 δ + α1 f † 2F + α2 (|f † |T V,β − |f † |T V ) + α2 |f † |T V 2 N  1 2 β   δ + α1 f † 2F + α2 + α2 |f † |T V 2 † † m,n=1 |fm,n |2 + β + |fm,n |2 1 2 δ + α1 f † 2F + α2 N 2 β + α2 |f † |T V . (2.11) 2

Since |fα∗1 ,α2 ,β,δ |T V ≤ |fα∗1 ,α2 ,β,δ |T V,β , the proof is complete by the triangle inequality.  The above decompositions are important for seeking the minimizer of our cost functional, which is taken as our reconstruction of image. This result generates the resolution analysis for our reconstruction scheme in terms of the data-matching and regularity-matching for the image, i.e., the quantitative error descriptions on these two terms are given. We can adjust the parameters α1 , α2 analytically such that our reconstruction fits our concerns for either image details (data-matching) or image sparsity (TV difference).

3

The Iteration Algorithm to Find the Minimizer 2

Take the image vector f = (f1 , f2 , · · · , fN 2 )T ∈ RN ×1 as the equivalent variables, and each components fi has one-to-one correspondence relationship with fm,n , i.e. fm,n = f(n−1)N +m . For the optimization problem 

1 P F T f F − P gˆδ 2F + α1 f 2F + α2 |f |T V,β (3.1) min Jβ (f ) = min f f 2 finding the minimizer f ∗ approximately, the Bregman iterative algorithm is given in [13], which is established in terms of Bregman distance [14]. In order to solve the optimization problem (3.1) iteratively, f (k+1) is yielded by solving its EulerLagrange equation [15]. Due to the penalty terms f F and |f |T V,β , the corresponding Euler-Lagrange equation for the minimizer is nonlinear. So we propose

8

X. Liu and J. Liu

to find the minimizer by the lagged diffusivity fixed point method [16]. Considering the optimization problem with respect to the image vector f as

 1 δ 2 2 P(Ff ) − Pˆ g 2 + α1 f 2 + α2 |f |T V,β . (3.2) min Jβ (f ) := min f f 2 In order to solve the Euler-Lagrange equation of (3.1), we need the derivatives of data matching term and the penalty terms in (3.2). By straightforward computations, these derivatives have the following expressions: ⎧ gδ , ⎨ ∇f 12 P F T f F − P gˆδ 2F = F∗ P∗ PFf − F∗ P∗ Pˆ 2 ∇f f F = 2I ⊗ If , (3.3) ⎩ ∇f |f |T V,β = L[f ]f , where P = I ⊗ P and the N 2 × N 2 matrix

L[f ] := (I ⊗ D)T Λ[f ](I ⊗ D) + (D ⊗ I)T Λ[f ](D ⊗ I)

(3.4)

with ⎧   1 1 ⎨ Λ[f ] := diag d1 [f ] , · · · , dN 2 [f ] ,  ⎩ d [f ] := (N 2 (I ⊗ D) ′ f ′ )2 + (N 2 (D ⊗ I) ′ f ′ )2 + β, i i,l l i,l l l′ =1 l′ =1

where i = i(m, n) = (n−1)N +m, l′ = l(m′ , n′ ) = (n′ −1)N +m′ for m, n, m′ , n′ = 1, · · · , N , and the N × N circulant matrix D := circulant(−1, 0, · · · , 0, 1), Based on (3.3), we can find the approximate minimizer by the following Bregman iterative algorithm. Algorithm 1. Bregman iterative algorithm for minimizing Jβ (f ) δ ′ ′ N ×N Input: frequency input {ˆ gm , ′ ,n′ : m , n = 1, · · · , N }, sampling matrix P ∈ R and parameters α1 , α2 , β. Do iteration from l = 0 with g (0) = Θ, f (0) = Θ. While l < L0 { Compute:   gˆ(l+1) = gˆδ + gˆ(l) − F T f (l) F ,

f (l+1) = arg

min

f ∈RN ×N

α1 f 2F + α2 |f |T V,β + 21 P F T f F − P gˆ(l+1) 2F ,

l ⇐ l + 1. } End do f ∗ := f L0 End

According to (3.3), the stacking vector f (l+1) of the minimizer f (l+1) of Jβ (f ) at the l−th step satisfies the following nonlinear equation:   (3.5) ˆδ − Ff (l) , N 2 (I ⊗ P )f + 2α1 (I ⊗ I)f + α2 L[f ]f = F∗ I ⊗ P g

A Double Recursion Algorithm to Image Restoration

9

with sampling data P gˆδ and the spatial approximation f (l) at the (l − 1)th step, which constitutes the standard Bregman iterative algorithm. Now we propose a new algorithm based on the Bregman iteration by introducing an inner recursion. Notice, the real symmetric matrix I ⊗ P may not be invertible due to our finite sampling matrix P . Therefore an efficient algorithm should be developed 2 for solving the nonlinear system (3.5) with unknown f ∈ RN ×1 . We apply the lagged diffusivity fixed point method [16]. 1 1 , · · · , d(n−1)N ), then Λ[f ] = diag(Λ1 [f ], Define Λn [f ] := diag( d(n−1)N +1 [f ] +N [f ]

· · · , ΛN [f ]). Since

L1 [f ] := 2α1 (I ⊗ I) + α2 (I ⊗ D)T diag(Λ1 [f ], · · · , ΛN [f ])(I ⊗ D) is a real positive block diagonal matrix and L2 [f ] := α2 (D ⊗ I)T Λ[f ](D ⊗ I) ⎛ 1 1 − d11[f ] ··· d1 [f ] + dN 2 [f ] 1 1 1 ⎜ − d1 [f ] d1 [f ] + d2 [f ] · · · ⎜ ⎜ . .. .. .. = α2 ⎜ . . ⎜ ⎜ 0 0 ··· ⎝ 0 ··· − d 12 [f ] N

1 dN 2 −2 [f ]

−d

0 0 .. . + 1

−d

1

N 2 [f ]

1 dN 2 −1 [f ]

N 2 −1 [f ]

−d

0 .. . 1

N 2 −1 [f ]

1 dN 2 −1 [f ]

+

1 dN 2 [f ]

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

is a symmetric block matrix, we construct the inner iteration scheme from l-step for the nonlinear system (3.5) as   L1 [f (l) ]f (l+1) = − N 2 (I ⊗ P ) + L2 [f (l) ] f (l)   (3.6) + I ⊗ P F∗ g ˆδ − F∗ Ff (l)

for l = 0, 1, · · · . Since L1 [f (l) ] is a known block diagonal matrix being symmetric positive, the computational costs for solving f (l+1) are affordable by solving each column vector of f (l+1) separately, which meets an N −dimensional linear equations with symmetric positive coefficient matrix. In the numerical experiments, we choose regularization of adjoint conjugate gradient method (ACGM) as the inner iteration scheme to solve (3.6). Let b(l) be the right term in (3.6) which is the known part from the l−step in exterior recursion, and µ is the prior regularizing parameter in ACGM. So we have   (l+1) (l) (l) (l) (l) (3.7) fk+1 = fk − κk µfk + L1 [f (l) ]T (L1 [f (l) ]fk − b(l) ) , (l)

where κk is the step-size at k-step in inner recursion defining as (l)

κk :=

(l)

(l)

−rk , (µI + L1 [f (l) ]T L1 [f (l) ])(−rk )T  (l)

(µI + L1 [f (l) ]T L1 [f (l) ])(−rk )22

,

from the classic successive over relaxation method (SOR),  ,  is the L2 inner (l) (l) (l) product, and rk := µfk + L1 [f (l) ]T (L1 [f (l) ]fk − b(l) ).

10

X. Liu and J. Liu (l)

Notice, the initial value f0 in the inner recursion can be chosen as 0 or f (l) , and the stopping criterion may be the maximum iteration number K0 or some others related to the small values of the cost function or the small difference of the iterative sequences. Here we stop the iteration process when the difference between f (l+1) and f (l) is smaller than 10−3 . Finally we have the scheme to find the approximate minimizer by the following iterative algorithm with inner recursion. Algorithm 2. Bregman iterative algorithm with inner recursion ′ ′ N ×N δ , Input: frequency input {ˆ gm ′ ,n′ : m , n = 1, · · · , N }, sampling matrix P ∈ R and parameters α1 , α2 , β, µ, L0 . Do exterior recursion from l = 0 with g (0) = Θ, f (0) = Θ While l < L0   { Compute: g ˆ(l+1) = g ˆδ + g ˆ(l) − PFf (l) , (l)

Do inner recursion from k = 0 with f0 = f (l) . (l+1) (l) While fk+1 − fk 22 > 10−3 (l)

(l)

{ Compute: b(l) , rk , κk . (l+1) (l) (l) (l) Compute: fk+1 = fk − κk rk with (3.7) k ⇐ k + 1. } End do l ⇐ l + 1. } End do f ∗ := f L0 End

4

Numerical Experiments

All the numerical tests are performed in MATLAB 7.10 on a laptop with an Intel Core i5 CPU M460 processor and 2 GB memory. We consider a model problem with Ω = [0, 1]2 and N = 128. Define   2

2

1 1 1 , + x2 − ≤ D1 := x = (x1 , x2 ) : x1 − 4 2 64        3 1  1 1 , (4.1) D2 := x = (x1 , x2 ) : x1 −  ≤ , x2 −  ≤ 4 8 2 4 and

⎧ ⎪ ⎨1, x ∈ D1 , f (x) := 2, x ∈ D2 , ⎪ ⎩ 0, x ∈ Ω \ (D1 ∪ D2 ).

(4.2)

The functions f (x) together with its frequency function log(|fˆ(ω)|) in algorithm domain (i.e., after shifting as Fig. 1) is shown in Fig. 2(a) and (b). Obviously, the

A Double Recursion Algorithm to Image Restoration

11

frequency data in the center (or in the four corners before shifting) consist of the main information about the image, so we should sample these data as much as possible.

Fig. 2. (a) f (x); (b) Frequency function log(|fˆ(ω)|) after shifting; (c) and (d) Sampling g δ (ω)|) and P90 log(|ˆ g δ (ω)|) after shifting. noisy frequency data P60 log(|ˆ

δ Firstly we yield the full noisy data gm,n from the exact image fm,n by δ = fm,n + δ × rand(m, n), gm,n

where m, n = 1, · · · , N and rand(m, n) are the random numbers in [−1, 1]. The mesh image of initial image and the noisy image are shown in Fig. 3. Then the full noisy frequency data are simulated by δ δ ′ ′ gˆm ′ ,n′ = F[gm,n ], m , n = 1, · · · , N.

(4.3)

So with the random band row sampling method using sampling matrix P , δ P gˆm ′ ,n′ is the input incomplete noisy data by row sampling process.

Fig. 3. The mesh of initial image f (x) and noisy image g δ (x).

Take α1 = 1000, α2 = 0.001, β = µ = 0.0001, and the noise level δ = 0.1. To the row sampling processes, we consider two schemes by taking M0 = 60, cenR = 0.3 and M0 = 90, cenR = 0.3, so the sampling ratios are M0 /M = 60/128 = 46.88% and 70.31%, respectively. To compare the restoration performances by applying more sampling data, we require that the data for M0 = 60 be included in the data set for M0 = 90. In order to ensure the validity of tests, the random number rand(m, n) and sampling rows should be fixed in each parts. Then we obtain the sampling matrix P60 , P90 with pii = 1 only at the following locations: i ∈ {1 − 5, 16, 17, 18, 23, 34, 37, 39, 40, 43, 44, 45, 47 − 55, 58, 60, 61, 63, 64, 70, 76 − 79, 81, 82, 83, 86, 88 − 91,

94, 95, 97, 98, 100, 101, 103, 105, 107 − 112, 125 − 128}

(4.4)

12

X. Liu and J. Liu

and i ∈ {1 − 14, 16, 17, 18, 23, 27 − 32, 34, 37, 39, 40, 43, 44, 45, 47 − 55,

58, 60, 61, 63, 64, 70, 72, 73, 74, 76 − 79, 81, 82, 83, 86, 88, 89 − 91, 94, 95, 97, 98, 100, 101, 103, 105, 107 − 112, 114 − 128} (4.5)

respectively. Figure 2(c) and (d) show the two-dimensional image of full frequency data, the incomplete noisy frequency data with P60 , P90 after shifting respectively. In our iteration process, the Bregman iterative number is L0 = 20, and the initial value in ACGM inner recursion is f (l) . We compare Algorithm 2 with Algorithm 1, i.e., comparing the proposed scheme to the Bregman iterative algorithm without inner recursion. Figure 4(a) and (b) give the reconstructed image f ∗ with P60 , P90 by our proposed algorithm, while Fig. 4(c) shows the reconstructed image f ∗ with P90 by standard Bregman iterative algorithm.

Fig. 4. (a), (b) The reconstruction of f ∗ with P60 , P90 by our Bregman iterative algorithm with ACGM inner recursion; (c) The reconstruction of f ∗ with P90 by standard Bregman iterative algorithm without inner recursion.

From our numerical implementations, the algorithm based on random band sampling method can reconstruct the piecewise smooth image with good edgepreservation. Considering we apply the noisy data with relative error 10% and the unused sampling data (the lost data) are more than 50% and 30%, the image restorations based on Bregman iterative algorithm with ACGM inner recursion are satisfactory. However, the reconstruction could only restore the relative grey level in the whole image, the exact value cannot be recovered efficiently. The numerical evidences for this phenomena are that the reconstructed image f ∗ with sampling matrix P90 has interfaces clearly, but the interfaces of f ∗ with P60 is worse.

5

Conclusion

An efficient algorithm to restore image based on L2 − T V regularization penalty terms is established. The data matching term of the optimizing model is only used limited data in frequency domain, which are obtained by random band sampling process. The new idea is that the model is included with two iteration: Bregman iteration and adjoint conjugate gradient method as inner recursion. In order to solve the optimizing problem, the Bregman iteration with lagged

A Double Recursion Algorithm to Image Restoration

13

diffusivity fixed point method is used to solve the nonlinear Euler-Lagrange equation of modified reconstruction model. For the inner recursion, the initial value getting from the l-th exterior recursion can decrease the inner iteration time. The experimental results demonstrate that proposed algorithm with random band sampling is very efficient for recovering the piecewise smooth image with limited frequency data, compared with the standard Bregman iterative algorithm. Acknowledgments. This work is supported by NSFC (Nos. 11531005, 11421110002, 11671082), and Postgraduate Research & Practice Innovation Program of Jiangsu Province (No. KYCX17 0038). We thank the referees for their comments and suggestions which make the paper much improved.

References 1. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer Academic Publishers, Dordrecht (1996) 2. Shi, Y.Y., Yang, X., Chang, Q.: The Total Variation Model and Numerical Methods for Image Restoration. Science Press, Beijing (2013) 3. Xu, C.: The Application on Image Inpainting with Beyond Wavelets and Variation Analysis. Science Press, Beijing (2013) 4. Jia, Y., Liu, P., Niu, S.: Partial Differential Equation on Image Processing and Program Design. Science Press, Beijing (2012) 5. Gills, A., Luminita, V.: A variational method in image recovery. SIAM J. Numer. Anal. 34, 1948–1979 (1997) 6. Zhu, Y.G., Liu, X.M.: A fast method for L1–L2 modeling for MR image compressive sensing. J. Inverse Ill-posed Prob. 23, 211–218 (2015) 7. Wang, X.D., Feng, X., Wang, W., Zhang, W.: Iterative reweighted total generalized variation based poisson noise removal model. Appl. Math. Comput. 223, 264–277 (2013) 8. Candes, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52, 489–509 (2006) 9. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006) 10. Vogel, C.R.: Computational Methods for Inverse Problems. SIAM Frontiers in Applied Mathematics, Philadephia (2002) 11. Shahrasbi, B., Rahnavard, N.: Model-based nonuniform compressive sampling and recovery of natural images utilizing a wavelet-domain universal hidden Markov model. IEEE Trans. Sign. Process. 65, 95–104 (2017) 12. Liu, X.M., Zhu, Y.G.: A fast method for TV-L1-MRI image reconstruction in compressive sensing. J. Comput. Inf. System. 2, 1–9 (2014) 13. Yin, W.T., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for L1-minimization with applications to compressed sensing. SIAM J. Imag. Sci. 1, 143–168 (2008) 14. Bregman, L.: The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phy. 7, 200–217 (1967) 15. Chen, B.L.: Theory and Algorithms of Optimization, 2nd edn. Tsinghua University Publishing, Beijing (2005)

14

X. Liu and J. Liu

16. Vogel, C.R., Oman, M.E.: Iterative methods for total variation denoising. SIAM J. Sci. Comput. 17, 227–238 (1996) 17. Yao, M.: Digital Image Processing. China Machine Press, Beijing (2006) 18. Ng, M., Chan, R., Tang, W.: A fast algorithm for deblurring models with Neumann boundary conditions. SIAM J. Sci. Comput. 21, 851–866 (1999)

RGB-D Saliency Detection with Multi-feature-fused Optimization Tianyi Zhang1, Zhong Yang1,2 ✉ , and Jiarong Song1 (

1

)

College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, People’s Republic of China [email protected] 2 Unmanned System Laboratory, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, People’s Republic of China

Abstract. This paper proposes a three-stage method using color, joint entropy and depth to detect salient regions in an image. In the first stage, coarse saliency maps are computed through multi-feature-fused manifold ranking. However, graph-based saliency detection methods like manifold ranking often have prob‐ lems of inconsistency and misdetection if some parts of the background or objects have relatively high contrast with its surrounding areas. To solve this problem and provide more robust results in varying conditions, depth information is repeatedly used to segment and refine saliency maps. In details, a self-adaptive segment method based on depth distribution is used secondly to filter the less significant areas and enhance the salient objects. At last, the saliency-depth consistency check is implemented to suppress the highlighted areas in the back‐ ground and enhance the suppressed parts of objects. Qualitative and quantitative evaluation on a challenging RGB-D dataset demonstrates significant appeal and advantages of our algorithm compared with eight state-of-the-art methods. Keywords: RGB-D saliency detection · Multi-feature fusion Manifold ranking · Saliency-depth consistency check

1

Introduction

Salient object detection is a subject aiming to detect the most salient attention-attracted object in a scene [1]. As an important part of computer vision and neuroscience, its application has been widely expend to various fields [2] such as image segmentation [3], retargeting [4, 5], compressing [6], object recognition [7] and object class discovery [8, 9]. Over the past decades, a lot of algorithms have been proposed for detecting salient regions in images, most of which aim to identify the salient subset from images first and then use it as a basis to segment salient objects from the whole image [10]. Generally, these approaches can be categorized into two breakdowns: bottom-up and top-down approaches. Bottom-up saliency approaches use some pre-assumed priors like contrast prior and texture prior to compute saliency of image while top-down approaches usually use high-level information to guide the detection. In this paper, only bottom-up saliency detection methods are considered. © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 15–26, 2017. https://doi.org/10.1007/978-3-319-71598-8_2

16

T. Zhang et al.

To our best knowledge, bottom-up saliency detection methods can be categorized as patch-based saliency detection methods and graph-based saliency methods. Patchbased saliency detection utilizes pixels and patches as the basic components for saliency calculation while graph-based saliency detection regards each superpixel segmented from image as a node in graph and calculates saliency according to graph theory. Compared with pixel-based methods, graph-based saliency detection methods can more easily and directly represent the spatial relationship between superpixels. Among all studies in patch-based saliency detection, Chen et al. [11] provided a simple but power approach to compute the saliency of each sparse-represented color. Duan et al. [12] presented a saliency detection method based on the spatially weighted dissimilarity which integrated dissimilarities, dimensional space and the central bias to determine the contrast of a patch. Other methods on pixel-based saliency detection also have made great contribution to detect attention-attracted regions. However, their results are sensitive to small regions with strong contrast, especially on the edge of image. For graph-based approaches, Chuan Yang et al. [13] proposed a graph-based manifold ranking algorithm to measure the difference between four sides and the rest parts of image. Yao Qin et al. [14] constructed a background-based map using colour and space contrast with the clustered boundary seeds to initial saliency maps. Other graph-based methods also show an outstanding performance on boundary suppressing. However, in order to enhance the local contrast, most of graph-based methods only assign edges between superpixels and their adjacent nodes. Because of this, some regions could be ununiformed, which will significantly decrease the accuracy of detection. Although the previous methods mentioned above showed good results in some scenes, there are still a lot of problems remaining to be solved. The Saliency regions of an image often have great contrast in color, texture or other features. However, some parts of background can also have high-contrast regions while the contrast of some salient parts can be relatively low. This phenomenon will cause inconsistency and misdetection in saliency maps and cannot be removed by neither graph-based nor pixelbased algorithm due to the nature of contrast comparison. The depth information of object has an indispensable role in the vision ability of human beings according to cognitive psychology [15]. Recently, with the development of stereo vision, smoothed and accurate depth of an image can be easily acquired through Kinect, ZED camera or other stereo cameras. Because of this, there is an increasing number of methods [16–18], both pixel-based and graph-based, that introduce depth as an important basis for saliency detection. To take advantage of it, this paper repeatedly uses depth information as a basis to generate, filter and refine saliency maps. Firstly, a multi-feature-fused manifold ranking algorithm is suggested to generate coarse saliency maps. Compared with traditional manifold ranking [13], this algorithm is more robust to varying scenes by taking depth and texture as references for ranking. Then, a selfadaptive segmentation method based on depth distribution is used to filter the less salient areas. This step segments possible salient regions from candidates based on the depth information of each superpixel. At last, we propose a further method of saliency-depth consistency check to remove or enhance ununiformed regions in background and sali‐ ency regions respectively. In the first step of this algorithm, multiple features are taken as references for ranking, some of which are unnecessary and can cause error

RGB-D Saliency Detection with Multi-feature-fused Optimization

17

highlighting or suppressing. The error saliency of these areas will be refined in the third step due to their saliency inconsistent with depth values.

2

Proposed Method

Our algorithm mainly includes three steps. Firstly, we generate coarse saliency maps through multi-feature-fused manifold ranking [13]. In this process, color, depth and joint entropy of superpixels are taken as references for ranking. Then, a self-adaptive segmen‐ tation method based on depth distribution is used to furtherly erase the saliency of back‐ ground. Finally, we implement saliency-depth consistency check to remove small speckles on the background and enhance ununiformed regions on objects. The main stages of our algorithm are demonstrated in Fig. 1. In the figure, the result of every stage is listed to illustrate how our algorithm optimizes the saliency map step by step.

Fig. 1. Illustration of the main stages of proposed algorithm

2.1 Multi-feature-fused Manifold Ranking Suppose X = {x1 , ⋯ xl , xl+1 , ⋯ , xn } ∈ Rm×n is a data set, some of which are labelled as queries and others are supposed to be ranked according to their similarity of the queries. In our case, X is a set of superpixels segment via simple linear iterative clustering (SLIC) method [19]. f:X → Rn is a ranking function which assigns each superpixel xi to a ranking score ri. Then, a graph G = (V, E) is constructed, where nodes V are superpixels in X. W ∈ Rn×n denotes the adjacency matrix with element wij to store the edges E, which is the weight between superpixel i and j. Finally, y = [y1 , ⋯ , yn ]T is an initial vector in which yi = 1 if xi is a query and yi = 0 otherwise. The cost function associated with ri is defined to be: ‖ ‖2 n n ∑ fi fj ‖ 1 ∑ ‖ ‖f − y ‖2 ) ‖ + � f = arg min ( wij ‖ √ − √ ‖ i‖ ‖i ‖ f 2 i,j=1 ‖ dii djj ‖ i=1 ‖ ‖ ∗

(1)

18

T. Zhang et al.

The first and second term of cost function is smoothness constrain and fitting constrain, respectively. In this paper, a unormalized Laplacian matrix is used to construct f ∗:

f ∗ = (D −

1 W)−1 y �+1

where D = diag{d11 , ⋯ , dnn } is the degree matrix of G, dii =

(2) ∑

wij. μ is the parameter

i

which controls the balance between smoothness constrain and fitting constrain. Traditional manifold ranking algorithm just assigns the Euclidean distance of colour in CIElab space to adjacency matrix, which does not consider other features such as depth. In this paper, we define W as: ⎧ 1 1 1 ⎪ exp(− 2 cij − 2 dij − 2 hij ) j ∈ Ni 2� 2�d 2�h wij = ⎨ c ⎪ 0 otherwise ⎩

(3)

where Ni is the set of superpixels which are adjacent to i . �c , �d , �h are weighting factors for the corresponding features cij , dij , hij, which have been normalized to [0, 1] before ‖ ‖ ‖ ‖ fusion. cij = ‖ci − cj ‖ is the Euclidean distance of colour in CIElab space. dij = ‖di − dj ‖ ‖ ‖ ‖ ‖ is the Euclidean distance of depth between two superpixels. hij refers to the joint entropy of two adjacent superpixels. The entropy can indicate the information content between two image patches. In other words, if the joint entropy hij is low, one image patch can be predicted by the other, which corresponds to low information [20]. The joint entropy is calculated as a sum of joint probability distribution of corresponding intensities of two superpixels: hij =

∑

Pij (a, b) log Pij (a, b)

a,b

(4)

where Pij (a, b) is the marginal probability distribution of two superpixels: Pij (a, b) =

Hi,j (a, b) ∑ ∑

L−1 M−1

Hi,j (a, b)

(5)

a=0 b=0

where Hi,j (a, b) is the joint histogram of two image patches. L and M is the length and width of each image patch respectively. To calculate the joint entropy of two superpixels, the sizes of two superpixels need to be the same. Hence, we fit the compared superpixels in one minimum enclosing rectangle before calculation. Specifically, we use the nodes on the image boundary as background seeds and construct four saliency maps using nodes on four sides of image as queries respectively.

RGB-D Saliency Detection with Multi-feature-fused Optimization

19

As shown in Fig. 1, the final saliency map for stage one is integrated by multiplying four saliency maps: Sc (i) = St (i) × Sb (i) × Sl (i) × Sr (i)

(6)

where St (i), Sb (i), Sl (i) and Sr (i) is the saliency map using nodes on the top, bottom, left and right side of image as queries respectively. Taking the top as an example St (i) can be calculated by: St (i) = 1 − f

∗

(7)

∗

where f denotes the normalized vector. The others can be calculated in the same way. 2.2 Saliency Optimization Using Depth Distribution While most regions of salient objects are highlighted in the coarse saliency map computed from multi-feature-fused manifold ranking, some superpixels which belong to background are also highlighted. To solve this problem, we firstly segment depth map using an adaptive threshold to roughly select the superpixels of foreground. Since most regions of foreground have been highlighted in the coarse saliency map, the threshold is set to be the mean saliency of the coarse saliency map. The saliency less than mean saliency are set to be zero and otherwise is set be one. Then we use this binary map Scb (i) to segment depth map: Ds = Scb × D

(8)

where Ds is the segmented depth map and D is the original depth map. In physical world, salient objects usually have discrete depth values while back‐ ground objects like walls, grounds or sky often have continuous depth values. In other words, the depth value of salient object often aggregate in a small interval. According to this finding, we use a nonlinear mapping function to refine depth value as a method of suppressing the depth value in background: D′s = 90 arctan(

255 1 D − Td ) + 20 s 2

(9)

where D′s is the new depth map after mapping. Td is the threshold value to control the enhancing interval and can be gained by choosing a reasonable distance for peak value of histogram. At last, the saliency result of stage two Sc′ is gained by: Sc′ = D′s × Sc

(10)

According to our observation, depth value does not naturally reflect the salient level of an object. An object with continuous depth like wall or ground usually has relatively close distance than salient object while their saliency is not that high. Thus, we use

20

T. Zhang et al.

Eq. (9) to guarantee that the utilization of depth value in stage two does not change the existing saliency of object. It is just used as a reference of segmentation to separate foreground objects with background. 2.3 Saliency-Depth Consistency Check Saliency detection based on graph could be adversely affected if some background regions have relatively high contrast with its surrounding regions. Meanwhile, some parts of the saliency region could be assigned low saliency even if its surrounding parts have high saliency. These two weaknesses could result in inconsistency of salient regions, as Fig. 2 demonstrates. The saliency of some small regions in the background can be very high while the saliency of some object regions, especially those embedded in the object center can be low due to their low contrast to the four sides.

Fig. 2. Procedure of saliency-depth-consistency check

Since this problem cannot be solved by simply adjusting the weights of features in Eq. (3), a method of saliency-depth consistency check is introduced here to refine sali‐ ency map furtherly. The idea of this method is based on the fact that if regions share the same (or similar) depth value, their saliency should be the same (or similar) as well. According to this hypothesis, we firstly check if the saliency of a superpixel is consistent with its surrounding regions. If the saliency is inconsistent with its surrounding regions, we use a self-adaptive method to identify which of its surrounding superpixels are consistent with it according to their depth values. At last, we refine the inconsistent saliency with the mean of its consistent surrounding regions. We iterate all superpixels until all of them are consistent with its surrounding regions. The procedure of this algo‐ rithm is shown is Fig. 2. In details, the saliency of superpixel Sc′ (i) is defined to be consistent with its adjacent superpixels if: S′ (i) D 1 Di ≤ c′ ≤ a i , j ∈ Ni a Dij Sc (j) Dij

(11)

RGB-D Saliency Detection with Multi-feature-fused Optimization

21

where a > 1 is the threshold factor. Ni is the set of superpixels which are adjacent to i. Di and Dij is the depth value of superpixel i and its adjacent superpixels respectively. In other words, if there exists a superpixel which does not meet Eq. (11), the saliency of superpixel i is inconsistent and should be refined by its adjacent superpixels. The new saliency of superpixel i can be calculated by: ⎧ Mean(S′ (j)) if STD[ Di , Di ⋯ Di ] ≤ T c ⎪ Di1 Di2 Dij Sc′′ (i) = ⎨ D D D ⎪ Mean(Sc′ (h)) if STD[ i , i ⋯ i ] > T ⎩ Di1 Di2 Dij

(12)

where Mean(Sc′ (j)) is the mean of all saliency of superpixels adjacent to i . Mean(Sc′ (h)) is the mean of saliency of superpixels whose depth are considered to be consistent with i. The threshold value T can be gained by minimizing σ:

� = STD[

D D Di D Di Di , , ⋯ , i ] + STD[ i , , ⋯ , i ], j ∈ Ni Di1 Di2 Dih Dih Di(h+1) Dij

D D Di D Di > i >, ⋯ , i > >, ⋯ , > i Di1 Di2 Dih Di(h+1) Dij

3

(13)

Experimental Results

To demonstrate the effectiveness of proposed method, we evaluate it on a RGB-D data set [18] captured by Microsoft Kinect which contains 1000 RGB-D images selected from 5000 natural images. Compared with other 3D data set, it has plenty of important characteristics such as high diversity and low bias. Moreover, both RGB and depth images in this data set share variable colour or depth ranges, which makes the salient object extremely difficult to be extracted by a simple thresholding. We compared our algorithm with eight state-of-the-art saliency detection algorithms: ASD [16], DMC [18], SF [21], GS [22], MF [13], OP [23], PCA [24] and SWD [25]. Among these algorithms, both RGB and RGB-D, patch-based and graph-based methods were considered. The results of these algorithms are all provided by their original open source codes. 3.1 Evaluation Measures To conduct a quantitative performance evaluation, we select the method of precisionrecall curve (PRC) and F-measure. The precision value indicates the proportion of salient pixels correctly assigned to all the pixels of detected regions, while the recall value correspond to the ratio of extract pixels be related to the ground-truth numbers. Like all state-of-the-art, the precision-recall curves are obtained by segmenting the saliency maps using threshold from 0 to 255. The F-measure is an overall performance meas‐ urement, which is defined as follow with the weighted harmonic of precision and recall:

22

T. Zhang et al.

F� =

(1 + � 2 ) × precision � 2 precision + Recall

(14)

where β2 = 0.3 is the emphasizing factor of precision. To evaluate the precision of detected pixels and salient pixels in ground truth, the mean absolute error (MAE) [21] is introduced in this paper, which is defined as: MAE =

W H 1 ∑∑ |s(x, y) − GT(x, y)| W × H x=1 y=1

(15)

where W and H is the width and height of image. s(x, y) indicates the saliency map and GT(x, y) is the image of ground truth. 3.2 Quantitative Comparison with Other Methods We examine our algorithm with eight state-of-the-art methods. The PR curve, F-measure and MAE of eight methods as well as ours are shown is Fig. 3. In saliency detection, the higher the precision and recall value is, the better performance of the method achieves. In this perspective, our algorithm has the best performance among all the eight methods despite some samples with high recall but relatively low precision. Although the precision-recall curve can give a qualitative evaluation of algorithms, the quantitative analysis still remains to be done. That is the reason we implement F-measure evaluation shown is Fig. 3(c). In that respect, the results of ours fully exceed the others except the recall is next only to OP. The MAE shown is Fig. 3(d) indicates SF, DMC and OP have relatively low MAE than other methods while our result is still the best among all eight methods. In detail, our final result (MAE = 0.216276) is around 17% better than the second performance DMC (MAE = 0.260925). This is mainly due to the saliency-depth consistency check which highlights the shadow part on object and removes some speckles in the background. In a word, as shown is Fig. 3 our algorithm is effective on the perspective of PR-curve on the RGB-D data base, only with the recall next to OP, but outperformances all the state-of-the-art methods in both F-measure and MAE. It is worthwhile noticing that the performance of other methods measured by PRcurve, F-measure and MAE on this RGB-D data set are all poorer than the results presented in their original papers, which indicates that this RGB-D data set is more challenging compared with other data sets such as ASD and MSRA. Thus, throughout the evaluation on this RGB-D data set, it can be concluded that our algorithm can process more complex scenes and is more robust to the variation in the natural environments.

RGB-D Saliency Detection with Multi-feature-fused Optimization

23

Fig. 3. Performance comparisons of proposed algorithm (a) PRC of proposed algorithm versus ASD [16], MF [13], OP [23] and PCA [24], (b) PRC of proposed algorithm versus DMC [18], GS [22], SF [21] and SWD [25], (c) F-measure scores, (d) MAE scores

3.3 Vision Comparison with Other Methods The vision comparison of the results presented by our algorithm and eight state-of-theart methods are shown in Fig. 4. It is obvious that 3D saliency detection methods like DMC can suppress the saliency of background more effectively. However, graph-based 2D saliency detection methods like MF and OP have stronger ability to highlight the saliency region and provide easier way to segment salient object from background. By taking advantage of both these two kinds of methods, our algorithm can highlight the intact boundary of object as well as suppress the complex interference in the background. Moreover, our algorithm can provide more inner-consistent saliency map due to the saliency-depth consistency check, which makes it much easier for further segmentation.

24

T. Zhang et al.

Fig. 4. Vision comparison of results computed by different methods. The proposed algorithm has better inner-consistency than others

4

Conclusion

In this paper, a novel three-stage saliency detection method using multi-feature-fused manifold ranking and depth optimization is presented. By fusing the depth information and joint entropy with color contrast in CIElab space, our algorithm is more robust in varying natural environments. To reduce the saliency of isolated background regions and highlight the saliency regions uniformly, a segment strategy as well as saliencydepth consistency check is proposed. Moreover, our algorithm takes advantage of both the graph-based saliency detection method and 3D saliency detection method, which makes detected salient objects have intact boundaries as well as inner-consistent sali‐ ency. Experimental results comparing with eight state-of-the-art approaches on a chal‐ lenging RGB-D data set illustrates that the proposed method brings out desirable performance and produces more robust and visual favorable results in terms of the precision and recall analysis, F-measure and MAE. Despite of few limitations, our algo‐ rithm indeed promotes the quality of saliency detection in natural scenes and has a great application prospect.

RGB-D Saliency Detection with Multi-feature-fused Optimization

25

Acknowledgments. The authors would like to thank the anonymous reviewers for their comments. This research was supported in part by the National Natural Science Foundation of China (Grant No. 61473144) and the Innovation Experiment Competition Funding for Graduates of Nanjing University of Aeronautics and Astronautics.

References 1. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998) 2. Li, H., Wu, W., Wu, E.: Robust salient object detection and segmentation. In: Zhang, Y.-J. (ed.) ICIG 2015. LNCS, vol. 9219, pp. 271–284. Springer, Cham (2015). https://doi.org/ 10.1007/978-3-319-21969-1_24 3. Schade, U., Meinecke, C.: Texture segmentation: do the processing units on the saliency map increase with eccentricity? Vis. Res. 51(1), 1–12 (2011) 4. Fang, Y., et al.: Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans. Image Process. 21(9), 3888–3901 (2012) 5. Chen, Y., Pan, Y., Song, M., et al.: Image retargeting with a 3D saliency model. Signal Process. 112, 53–63 (2015) 6. Guo, C., Zhang, L.: A novel multi-resolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans. Image Process. 19(1), 185–198 (2010) 7. Ren, Z., et al.: Region-based saliency detection and its application in object recognition. IEEE Trans. Circuits Syst. Video Technol. 24(5), 769–779 (2014) 8. Zhu, J.Y., et al.: Unsupervised object class discovery via saliency-guided multiple class learning. In: IEEE Conference on Computer Vision and Pattern Recognition 2012, pp. 3218– 3225. IEEE, Rhode Island (2012) 9. Sharma, G., Jurie, F., Schmid, C.: Discriminative spatial saliency for image classification. In: IEEE Conference on Computer Vision and Pattern Recognition 2012, pp. 3506–3513. IEEE, Rhode Island (2012) 10. Borji, A., et al.: Salient object detection: a survey. Eprint Arxiv, vol. 16, no. 7, pp. 1–26 (2014) 11. Cheng, M.M., et al.: Global contrast based salient region detection. In: IEEE Conference on Computer Vision & Pattern Recognition 2015, pp. 409–416. IEEE, Boston (2015) 12. Duan, L., et al.: Visual saliency detection by spatially weighted dissimilarity. In: IEEE Conference on Computer Vision and Pattern Recognition 2011, pp. 473–480. IEEE, Colorado (2011) 13. Yang, C., et al.: Saliency detection via graph-based manifold ranking. In: IEEE Conference on Computer Vision and Pattern Recognition 2013, pp. 3166–3173. IEEE, Oregon (2013) 14. Qin, Y., et al.: Saliency detection via cellular automata. In: IEEE Conference on Computer Vision and Pattern Recognition 2015, pp. 110–119. IEEE, Boston, (2015) 15. Lang, C., et al.: Depth matters: influence of depth cues on visual saliency. In: European Conference on Computer Vision 2012, pp. 101–115. IEEE, Florence (2012) 16. Ju, R., et al.: Depth saliency based on anisotropic center-surround difference. In: IEEE International Conference on Image Processing 2014, pp. 1115–1119. IEEE, Paris (2014) 17. Wang, A., et al.: Salient object detection with high-level prior based on Bayesian Fusion. IET Comput. Vis. 11(2), 161–172 (2017) 18. Peng, H., et al.: RGBD salient object detection: a benchmark and algorithms. In: European Conference on Computer Vision 2014, pp. 92–109. IEEE, Zurich (2014)

26

T. Zhang et al.

19. Achanta, R., Shaji, A., Smith, K., et al.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012) 20. Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008) 21. Perazzi, F., et al.: Saliency filters: contrast based filtering for salient region detection. In: IEEE Conference on Computer Vision and Pattern Recognition 2012, pp. 733–740. IEEE, Rhode Island (2012) 22. Wei, Y., et al.: Geodesic saliency using background priors. In: European Conference on Computer Vision 2012, pp. 29–42. IEEE, Florence (2012) 23. Zhu, W., et al.: Saliency optimization from robust background detection. In: IEEE Conference on Computer Vision and Pattern Recognition 2014, pp. 2814–2821. IEEE, Columbus (2014) 24. Margolin, R., Tal, A., Zelnik-Manor, L.: What makes a patch distinct. IEEE Trans. Comput. Vis. Pattern Recogn. 9(4), 1139–1146 (2013) 25. Duan, L., et al.: Visual saliency detection by spatially weighted dissimilarity. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 473–48. IEEE, Colorado, (2011)

Research on Color Image Segmentation Jingwen Yan1,3(&)

, Xiaopeng Chen1,3 and Huimin Zhao2,3

, Tingting Xie1,3

,

1

3

Department of Electronic Engineering, Shantou University, Shantou, Guangdong, China [email protected] 2 School of Electronic and Information, Guangdong Polytechnic Normal University, Guangzhou, China Key Laboratory of Digital Content Processing and Security Technology of Guangzhou City, Guangzhou, Guangdong, China

Abstract. The traditional Ncut algorithm has the disadvantages of large amount of computation and insufficient ability of resisting noise interference because it needs to solve the eigenvector and eigenmatrix. In order to solve this problem, a color image segmentation algorithm based on the improved Ncut algorithm is proposed in this paper. Firstly, we apply the Mean Shift for pre-segmentation, and then the pixel blocks produced from pre-segmentation replace the raw image pixel blocks to form a new block diagram. Next, Ncut algorithm is employed to final segmentation. We implemented several experiments and the experimental results show that the improved Ncut algorithm can not only improve the efficiency of the segmentation, but also has excellent ability to resist noise interference. Keywords: Image segmentation


Mean shift
Ncut algorithm

1 Introduction Image segmentation means that classifies all pixels to form sub regions based on certain classification rules. So far it has played an important role in the field of pattern recognition and computer vision. Image segmentation usually is the first step of the image processing, which the segmentation results will influence the final results of the whole image processing. Now image segmentation has been widely used in many fields, such as intelligent transportation, remote sensing image and medical image processing and so on. At present, there are plenty of common image segmentation algorithms, including threshold segmentation method [1], regional growth or Split Merge segmentation method, common edge detection operator segmentation method [2], morphological segmentation method [3] and clustering segmentation algorithm. In addition, there are also genetic algorithm segmentation method and neural network segmentation method which are combined with bioinformatics. The image segmentation method based on clustering is very suitable to be extended to color space, which has been attracted many scholars.

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 27–35, 2017. https://doi.org/10.1007/978-3-319-71598-8_3

28

J. Yan et al.

Graph cut theory is one of the most widely used image segmentation methods. It is mainly form an energy function, and then optimize the energy function to minimize. Ncut algorithm is a mature technology based on the graph cut theory in the field of image segmentation. Its basic idea is that an image is indicated as an undirected weighted and the pixels are regarded as nodes, following the segmentation of graph-cut theory. However, there are two drawbacks in the traditional Ncut algorithm: the first one is the expensive calculation due to the eigenvector and the eigenmatrix; the second one is the inadequate ability of resisting noise. Many scholars have improved the Ncut algorithm in the application process. For example, Cui [4] presented a method that combined Ncut segmentation with two watershed algorithm to image pre-processing twice, which generated the area block instead of raw image pixels. This method reduced the following computation of Ncut segmentation as well as improved the efficiency of segmentation, but the results were easily influenced by human interference and the segmentation was limited. Huang et al. [5] utilized classical Merge-Split method to improve the segmentation algorithm based on the Ncut segmentation, which compared the similarity of texture and color in the adjacent regions of the image. This method eliminated most situations of over segmentation and under segmentation. However, the selection of value K and the setting of some thresholds will affect the results in different degrees. Moreover, the segmentation results and effects are not yet solved well. For solving the disadvantages of traditional Ncut algorithm, in this paper, we present an improved Ncut algorithm based on the Mean Shift pre-segmentation. Firstly, Mean Shift pre-segmentation is used on raw image, which helps construct some regional block diagrams. Then, Ncut algorithm is applied to segment image to get the final segmentation results. The experimental results indicate that the improved Ncut algorithm makes the segmentation more efficiently since the number of pixel blocks is much less than the raw image pixels. Furthermore, the improved method has excellent ability to resist noise interference because of the Mean Shift algorithm.

2 Related Work There are some drawbacks in traditional Ncut algorithm. For example, the algorithm is a kind of NP-hard problems, so it needs expensive computation, which lead to the limitations of the application of real-time image segmentation. Besides, Ncut is chosen as K-means clustering algorithm. In this situation, the selection of parameter K will have pretty more influence on the final segmentation results, so some rules of thumb are needed to ensure how the parameter K should be set. In addition, the algorithm is very sensitive to noise. In this paper, an improved Ncut algorithm based on Mean Shift pre-segmentation is presented to reduce the computational complexity of the algorithm. The core idea is to pre-segment the original image through Mean Shift, and then use the Ncut algorithm to segment the undirected weighted graph formed by the pixel blocks. Mean Shift is a spatial analysis method based on the feature offset vectors. One of its great advantages is that it has the property of clustering segmentation. Moreover, the Ncut can be regarded as a classification algorithm. It can satisfy most image

Research on Color Image Segmentation

29

segmentation applications. However, it will need expensive calculations when there is a large amount of raw image pixels. Instead, our improved algorithm is based on the pixels block formed by the image block, which can greatly reduce the amount of calculations. Our method has two advantages, which are as follows: ① The process greatly reduces the amount of computation. As the number of regional blocks formed by Mean Shift pre-segmentation is less than the number of raw image pixels, the dimension of eigenmatrix will be reduced greatly. At the same time, the vertices and edges of the graph are also reduced a lot after transforming into a graph. ② The effect of image segmentation will be better. Because Ncut algorithm is pretty sensitive to noise, even a little amount of noise will have great influence on the final image segmentation results. The Mean Shift algorithm has a fixed inhibitory effect on the noise, and reduces the edge of the mutation to some degree. In addition, due to the Ncut is a global clustering algorithm [6], it can effectively eliminate the over-segmentation phenomenon of Mean Shift [7]. Furthermore, the pre-processing of Mean Shift can make up for the disadvantages of the Ncut algorithm which is overly dependent on the classification parameter K. Mean Shift divides the pixels into different classes through the distance. Therefore, it is necessary to transform the RGB to the LUV before Mean Shift pre-segmentation. In order to eliminate the interference of noise and get more smoothing segmentation results, the median filter is needed to pre-process the image, and then an improved adaptive Mean Shift algorithm is applied to cluster the processed image.

Fig. 1. System design.

3 The Proposed Algorithm In general, segmenting image is an ill-conditioned problem. In order to achieve better results, we need to preprocess the image before segmentation. In this section, we first operate space transformation and smoothing on the image, and then present our improved method. Figure 1 shows the schematic of our approach.

30

3.1

J. Yan et al.

Color Space Transformation

The full name of LUV is CIE 1976 (L*, u*, v*), where L* represents object luminance, u* and v* are chroma. In general, u* and v* are range from −100 to +100, and luminance L* is range from 0 to 100. It is relatively simple to transform RGB to LUV. That is, after transforming RGB to XYZ, XYZ is then transformed to LUV. Lena picture in Fig. 2 is selected to show the results of the space transformation. RGB to CIE XYZ: 2 32 3 2 3 b11 b12 b13 R X 1 6 76 7 6 7 4 b21 b22 b23 54 G 5 4Y 5 ¼ b21 b31 b32 b33 B Z 2 0:49 0:31 1 6 ¼ 4 0:17697 0:81240 0:17697 0:00 0:01

32 3 0:20 R 76 7 0:01063 54 G 5 0:99

ð1Þ

B

CIE XYZ to LUV: 8  1 > < 116  Y 3 16; Yn L ¼ > 29
3 Y : 3 Yn ; 



u ¼ 13L  u

v ¼ 13L  v0

0

Y Yn

[


6 3 29

Y Yn




6 3 29

u0n
v0n




ð2Þ

Fig. 2. Space transformation from RGB to LUV.

3.2

Median Filter

The median filter is a typical nonlinear smoothing technique. In some situation, it can overcome the problem of blurred images brought by the minimum mean square filter and mean filter. Moreover, it can efficiently suppress the interference of filter pulse and the noise of image scanning.

Research on Color Image Segmentation

31

The equation of 2D median filter is as follow: Gðx; yÞ ¼ med ff ðx

k; y

lÞ; ðk; l 2 W Þg

ð3Þ

Where “med” indicates the median operation, F (x, y) and G (x, y) are the original and processed images respectively. W is a two-dimensional sliding template, which the template size is generally selected as 3. The median filtering method is to sort the size of the pixels in the sliding window, and then the output pixel values of the filtering result are defined as the median value of the sequence. An example is given in Fig. 3.

Fig. 3. An example of median filter

3.3

Mean Shift Pre-segmentation

According to the paper presented by Li et al. [8], an improved adaptive Mean Shift is used to segment the image. In the Mean Shift algorithm, the bandwidth of H is the only parameter which affects the final clustering result. The method is to combine the advantages of the fixed bandwidth selection and the adaptive bandwidth selection. And the bandwidth selection is in formula (4): 8 < Hmin ; hi \ Hmin

ð4Þ Hmin  hi  Hmax ; hi ¼ xi xi; k h i ¼ hi ; : Hmax ; hi [ Hmax We choose fixed bandwidth Hmin and Hmax as the upper and lower limits of bandwidth. When the value of the adaptive bandwidth is in the interval ½Hmin ; Hmax Š, it will choose the adaptive bandwidth value, otherwise choose the fixed

bandwidth

value. The

method of adaptively determining the bandwidth, that is hi ¼ xi xi;k , is presented by Wang et al. [9], where xi represents the sample, and xi;k is the k nearest neighbor point. This method is simple in calculation and has little dependence on value K. 3.4

Ncut Clustering Segmentation

After the pre-segmentation of Mean Shift, the pixels of the image will form several regional blocks in which pixels are regarded as pixel blocks.

32

J. Yan et al.

We construct an undirected weighted graph with nodes formed by pixel blocks. After Ncut is used to segment the resulting undirected weighted graph. The corresponding weight matrix is formula (5): kFðiÞ Wij ¼ e

FðjÞ r2 I

2

k2

8 2 < kXðiÞ XðjÞ k2 2 r  e I ; if i is neighbor j : 0; else

ð5Þ

The weight matrix considers the information of spatial location and eigenvector. In the upper formula, FðiÞ ¼ fRðiÞ ; GðiÞ ; BðiÞ g indicates the information of spatial. That is, if the original image is gray image instead of color image, then FðiÞ represents the gray value of pixel blocks. XðiÞ represents the spatial location information of the pixel block. As mentioned in the previous section, the definition of the pixel block is as follows: the spatial information of eigenvector is represented by the average RGB of pixel blocks or gray values, while the spatial coordinate information is represented by the central point of the blocks. At the same time, the boundary of the image pretreated by Mean Shift has been saved, which can be used to judge whether the pixel block i and j are adjacent.

4 Experiments 4.1

Simulation Platform

The goal of Simulation platform is to produce better experimental results. In this paper, the experimental simulation platform of all algorithms we use is Matlab R2014b. We analyze the performance of the algorithm from two aspects: the segmentation effect and efficiency. To verify the effectiveness of our improved algorithm, we also compare our algorithm with the traditional Ncut algorithm. 4.2

Experimental Treatment

Firstly, an original image of 256  384 pixels in the (a) of Fig. 4 is imported. The segmentation results of the Mean Shift, traditional Nut and our improved algorithm are showed in Fig. 4 from (b) to (e) respectively, in which the corresponding parameters of Mean Shift is hr ¼ 6, hs ¼ 8, M ¼ 2000 and the parameter of traditional Nut is K ¼ 4ðrI ; rX Þ ¼ ð90; 110Þ. In our experiment, Mean Shift segments the image into 12 regional blocks. The running time of Mean Shift pre-segmentation is 2.59 s. The Ncut segments the combined pixel blocks, which runs 0.61 s, so the total time of improved Ncut is 3.20 s. Since the running time of the classical Ncut segmentation algorithm is 7.86 s, the total time of improved Ncut algorithm is half less than that of the classical Ncut. Follow the same operation as Fig. 2, we import an different original image of 277  384 pixels in the (a) of Fig. 5, thus we can also get the segmentation results of the Mean Shift, traditional Nut and our improved algorithm respectively, in which the corresponding parameters of Mean Shift is hr ¼ 8; hs ¼ 8; M ¼ 2000 and the parameter of traditional Nut is K ¼ 2ðrI ; rX Þ ¼ ð90; 110Þ. Here, Mean Shift segments

Research on Color Image Segmentation

33

Fig. 4. Comparison between the improved algorithm and traditional Ncut algorithm. (a) Original image. (b) Mean Shift. (c) Ncut clustering segmentation. (d) Improved algorithm for grayscale image segmentation. (e) The effect of improved algorithm segmentation.

Fig. 5. Comparison between the improved algorithm and traditional Ncut algorithm. (a) Original image. (b) Mean Shift. (c) Ncut clustering segmentation. (d) Improved algorithm for grayscale image segmentation. (e) The effect of improved algorithm segmentation.

34

J. Yan et al.

the image into 8 regional blocks, and the running time of Mean Shift pre-segmentation is 3.69 s. The Ncut segments the combined pixels blocks with the running time of 0.75 s, so the total time of improved Ncut is 4.44 s. Because the total running time of classical Ncut segmentation algorithm is 10.56 s, we can see that the total time of improved Ncut algorithm is half less than that of the classical Ncut, too. 4.3

Effect Analysis

It can be seen from the segmentation results that Mean Shift is a good segmentation of the target, but there will be an obvious situation of over-segmentation. Furthermore, the segmentation effect of classical Ncut algorithm is not so ideal. Instead, our improved algorithm is proved to be relative perfect to image segmentation. With the increase of the number of pixels in the original image, the time of the improved algorithm is much less than that of the classical Ncut algorithm. A comparable table is showed in the following Table 1: Table 1. Time comparison of the improved Ncut and traditional Ncut. Original image Mountain Pigeon

Resolution ratio 256 * 384 277 * 384

Mean Shift (s) 2.59 3.69

Ncut divides pixel blocks (s) 0.61 0.75

Improved Ncut (s) 3.20 4.44

Traditional Ncut (s) 7.86 10.56

5 Conclusions Image segmentation is an essential step in image recognition and understanding. Because of the drawbacks of traditional Ncut, such as the expensive calculation and the insufficient ability of resisting noise, an improved Ncut is proposed in this paper to segment color images. First of all, we apply LUV space transform and median filter to pre-process the original image. Then, we adapt Mean Shift to segment the processed image to construct the regional block. Here, the segmentation efficiency will be greatly improved due to the reduction of the number of the pixels in original image. After that the Ncut algorithm is used to get the final segmentation result. Several experimental results show that the accuracy and efficiency of our proposed algorithm are all good. Furthermore, and it can implement the effective segmentation of the color images. However, the segmentation effect is largely affected by the pre-segmentation of Mean Shift. The final segmentation effect of the improved Ncut algorithm will be disturbed when the effect of the pre-segmentation of Mean Shift is not ideal, which is what we need to solve in our future work. Acknowledgments. This work was funded by the National Natural Science Foundation of China (No. 61672335) and the National Science Foundation of China (No. 61772144).

Research on Color Image Segmentation

35

References 1. Qiang, Y., Zhi, H., Hongzhe, Z., et al.: Improved threshold image segmentation based on PCNN model. China Sci. Technol. Expo 3, 357 (2016) 2. Haffner, O., Ravas, R.: Edge detection in image. In: Elitech 2014: Conference of Doctoral Students, vol. 18, no. 1, pp. 91–101 (2014) 3. Viveroescoto, J.L., Chiang, Y.D., et al.: Recent progress in mesoporous titania materials: adjusting morphology for innovative applications. Sci. Technol. Adv. Mater. 13(1), 13003 (2012) 4. Cui, Y.: Research on image segmentation algorithm based on improved normalized cut. Laser Mag. 37(02), 122–124 (2016) 5. Yicen, H., Yifan, S., et al.: An improved image segmentation algorithm based on Normalized Cut. Comput. Eng. Appl. 44(34), 179–181 (2008) 6. Zhao, L.: Research and Improvement on K-mean Clustering Algorithm. Xi’an Electronic and Science University (2013) 7. Maria, F.: Oversegmentation reduction by flooding regions and digging watershed lines. Int. J. Pattern Recognit. Artif. Intell. 20(1), 15–38 (2012) 8. Guang, L., Zhaoying, W., et al.: Improved mean shift algorithm and its application in color image segmentation. Softw. Guide 09(1), 53–54 (2010) 9. Wang, Y.Z., Wang, J., Peng, N.S., et al.: Unsupervised color-texture segmentation based on soft criterion with adaptive mean-shift clustering. Pattern Recogn. Lett. 27(5), 386–392 (2006)

Depth-Based Focus Stacking with Labeled-Laplacian Propagation Wentao Li, Guijin Wang(B) , Xuanwu Yin, Xiaowei Hu, and Huazhong Yang Department of Electronic Engineering, Tsinghua University, Beijing 100084, China [email protected]

Abstract. Focus stacking is a promising technique to extend the depth of field in general photography, through fusing different images focused at various depth plane. However, existing depth propagation process in depth-based focus stacking is affected by colored texture and structure differences in guided images. In this paper, we propose a novel focus stacking method based on max-gradient flow and labeled Laplacian depth propagation. We firstly extract sparse source points with max-gradient flow to remove false edges caused in large blur kernel cases. Secondly, we present a depth-edge operator to give these sparse points 2 different labels: off-plane edges and in-plane edges. Only off-plane edges are then utilized in our proposed labeled-Laplacian propagation method to refine final dense depthmap and the all-in-focus image. Experiments show that our all-in-focus image is superior to other state-of-the-art methods. Keywords: All-in-focus Sparse-to-dense

1

· Labeled-Laplacian · Max-gradient flow

Introduction

In general photography, optical imaging systems always have limited depth-offield: optical lenses focus on a specific plane, while leaving other regions of the scene blurred. Although decreasing the aperture size could extend the DOF in some extent, this would lead to lower signal-to-noise ratio and longer exposure time. To overcome this limitation, focus-stacking has become more popular with the development of digital imaging technology [2, 5,14]. It captures a sequence of images focused at various planes and fuses them into a single all-in-focus image. The focus stacking technique has attracted a lot of attentions in the last decade, which could be divided into 2 categories: transform domain fusion approaches and depth-based approaches. For transform domain fusion approaches, source images are converted in transform domain, then corresponding transform coefficients (DWT [10], DSIFT [7], DCT [3]) are fused, finally the all-in-focus image is reconstructed by the inverse transform. These methods are usually complicated and unstable with variation of transform coefficients. In depth-based methods [9, 11, 15], they firstly extract some sparse pixels whose depth values are the sharpest index across the stack, then propagate them c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 36–46, 2017. https://doi.org/10.1007/978-3-319-71598-8_4

Depth-Based Focus Stacking with Labeled-Laplacian Propagation

37

to the dense depthmap, finally the all-in-focus image is generated by fusing pixels in the stack according to the depthmap. Suwajanakorn et al. [11] used sharpness measurement and formulated the fusing problem as a multi-labeled MRF optimization problem. Moeller chose well-known modified Laplacian (MLAP) function as the measure of contrast, and propagated the resulting depth estimates in a single variational approach (VDFF [9]). Aguet et al. [1] also estimated the all-in-focus image with a model based 2.5D deconvolution method. In all methods above, the depth values of extracted sparse points are affected and noised by false edges occurred in large blur kernel case. To remove false edges, we proposed max-gradient flow [15] to extract true source points, and gave an iterative anchored rolling filter to estimate the all-in-focus image. However, in all the sparse-to-dense propagation processes in these depth-based methods, the final depthmap is affected by colored texture and structure differences in guided images. In this paper, we propose a novel focus stacking method based on maxgradient flow and labeled Laplacian depth propagation. Firstly, we construct sparse depthmap with the max-gradient-flow proposed in our previous MGFARF method [15]. Then we design a depth-edge operator to give these sparse points 2 different labels: off-plane edges and in-plane edges. Here in-plane edges are image edges at the same depth plane, while off-plane edges are image edges at boundaries of different depth planes. Only off-plane edges are then utilized in the labeled-Laplacian depth propagation to generate final dense depthmap which is smoothed at textures in the same depth plane and strengthened at boundaries between different depth planes. Experiments show that our depthmap is smoothed at textures in the same depth plane and sharpened at depth boundaries, while the all-in-focus image is refined and superior to other state-of-the-art methods.

2

Sparse Depthmap with Max-Gradient Flow

In this section, we introduce the max-gradient flow [15] briefly to extract sparse depthmap. Max-gradient flow could model the propagation of gradients in the stack and remove the false edges produced in large blur kernel cases. To introduce max-gradient flow in detail, We capture a sample stack with Imperx B4020 mono camera equipped with a SIGGMA 50 mm/F1.4 lens. This stack consists of 14 images with large blur kernels, and is utilized to describe our method in the rest of our paper. Figure 1 shows 3 images focused at different depth planes from our stack. With focal stacks I1 , I2 ,..., In , an all-in-focus image could be produced by selecting the sharpest pixels across the focal stack. Several different measures of pixel sharpness have been defined in some shape-from-focus literature [8–10]. In this paper, without loss of generality, magnitude of gradients is calculated as sharpness measurement, which is defined as  ∂I 2 ∂I 2 (1) Gi = |∇I(x, y)| = ( ) + ( ) ∂x ∂y

38

W. Li et al.

Fig. 1. Three images captured from real scene focused at 3 focusing plane respectively

where Gi is the gradient magnitude of Ii , the i-th image in the stack. Then depth value of sparse points could be calculated as: D(x, y) = argmaxGi (x, y)

(2)

Here D(x, y) stores the depth value that gives the sharpest gradient across the stack. However, traditional methods following Eqs. (1) and (2) would produce ‘false edges’ [15]. False edges, the production of which has been explained in detail in [15], are those image edges with false depth values because of spreading of blur kernels of neighbouring strong edges in large blur kernel cases. To remove these false edges, max-gradient flow is utilized to analysis the propagation of gradients. The max-gradient flow from [15] is defined as: T

M GF (x, y) = [fx (x, y), fy (x, y)]

Here the two elements are calculated as: ⎡ max G (x+∆x,y)−max G (x,y) ⎤ j i   j i fx (x, y) ∆x ⎦ = ⎣ max Gk (x,y+∆y)−max Gi (x,y) fy (x, y) i k

(3)

(4)

∆y

The flow describes the propagation of gradients in the stack and is valid to divide points in the stack into 2 categories: source points and trivial points. Source points are points whose depth value calculated by Eqs. (1) and (2) is true and valid, while trivial points are points with false depth value from Eq. (2). Points whose max-gradient flow changes its direction oppositely are chosen as source points, formulated as: ∇ · M GF (x, y) > 0

(5)

Otherwise, the points are defined as trivial points if ∇ · M GF (x, y) < 0

(6)

We only preserve the depth value of source points to get the sparse depthmap. Figure 2 shows the comparison of performance of sparse depthmap with and without applying max-gradient flows. We could find that with max-gradient flow, the false edges are effectively suppressed and true edges are preserved as many as possible.

Depth-Based Focus Stacking with Labeled-Laplacian Propagation

39

Fig. 2. (a) Depth values without applying max-gradient flow. (b) Depth value for source points extracted with max-gradient flow

3

Labeled-Laplacian Depth Propagation

Laplacian matting is a traditional sparse-to-dense propagation method. Similar with other propagation methods, it causes depth artifacts and noises at textures on the same depth plane because of color and structure differences of guided image. Therefore, to generate a refined depthmap, it is critical to differentiate image edges on the same depth plane with those points at boundaries of depth planes to propagate these 2 labels of sparse points respectively. In this section, we propose a novel two-step depth propagation process. Firstly, we construct a novel L-matrix to get a coarse dense depthmap which removes effects of colored texture and structure differences of guided images. Secondly, 2 different labels are given to sparse points by our depth-edge operators extracted from the coarse dense depthmap: off-plane edges and in-plane edges. Then, in the second propagation process, only off-plane edges are utilized to update L-matrix. In this way, two labels of points are propagated differently to refine the dense depthmap: in-plane edges are smoothed while off-plane edges are strengthened and sharpened. 3.1

Coarse Dense Depthmap

In traditional Laplacian propagation methods [6, 16], the depth propagation problem could be formulated as minimizing the following cost energy: ˆ T D(d − d), ˆ E(d) = dT Ld + λ(d − d)

(7)

D is a diagonal matrix whose element D(i, i) is equal to 1 if the pixel i has valid depth value. d and dˆ are the dense depthmap and the sparse depth map which only has valid depth values at source points. Decomposing the Eq. (7), ˆ T D(d − d) ˆ denotes the dT Ld denotes the fidelity of source points while (d − d)

40

W. Li et al.

smoothness of depth propagation. The scalar λ controls the balance of these two parts. L is the Laplacian matrix calculated from color and structure differences of guided images, and is traditionally calculated as below: 1 ε T (1 + (Ii − µk ) (Σk + U3 )−1 )(Ij − µk )) (δij − L(i, j) = k|(i,j)∈ωk |ωk | |ωk | (8) where δij is the Kronecker delta, U3 is identity matrix, Σk is the covariance matrix of the colors in patch ωk , Ii and Ij are colors of all-in-focus image as guided image. From the equation above, differences of RGB-values of patches of guided image would affect the construction of L-matrix and the cost energy of Eq. (7). Therefore depthmap would produces depth artifacts and noises at locations of colored textures of guided images on the same depth plane. To remove these depth noises, we assume that all pixels in each patch ωk are constant, which makes Ii = µk and modify the L into: 1 ) (9) L(i, j) = (δij − |ωk | k|(i,j)∈ωk

From this equation, the construction of L matrix has nothing to do with colored textures Ii , Ij of guided image. Furthermore, the cost energy in Eq. (7) only depends on the sparse depthmap d shown in Fig. 3(a) and its distribution D. In this way, depth noises caused by colored textures of guided images in traditional propagation methods are removed and the coarse dense depthmap shown in Fig. 3(b) is produced only according to depth values of sparse source points. This dense depthmap is utilized to extract depth-edge operators in the next section. 3.2

Labeled-Laplacian Depth Propagation

From Fig. 3(a) and (b), the coarse dense depthmap, which is blurry at edges of different depth planes, is not satisfying. Therefore, we propose a labeledLaplacian depth propagation which sharpens edges of different depth planes to refine its estimation. Firstly, we design a novel operator to give source points 2 labels: off-plane edges and in-plane edges. For each source point as centered, we spread both N pixels along and against the rising direction of gradient in the coarse dense depthmap to construct a (2N + 1) * 1 pixels patch ω as our operator. In our operator, the depth value increases along with the increase of pixel index. From the definition above, we know that off-plane edges locate at boundaries of objects belonging to different depth plane, and are usually sharp when the image is focused at the nearer object. Therefore only the points with relatively small depth value and whose neighbouring depth value vary in large range should be classified as true off-plane edges. Therefore, we apply the equation below to calculate the value Ω for each depth-edge operator. The source point k is labeled as off-plane edges if Ωk =

ωk (2N + 1) − ωk (N + 1) > ΩT H ωk (N + 1) − ωk (1)

(10)

Depth-Based Focus Stacking with Labeled-Laplacian Propagation

41

Figure 3 shows the process of setting labels for source points in this section. Three different example points are displayed in Fig. 3(a). Only the red one is located at the boundaries of different depth planes, while the green point and the blue point are both at the same depth plane. Figure 3(b) shows the coarse dense depthmap generated from Eq. (12), from which we extract operators ω, and Fig. 3(c) presents the depth-edge operators of three example points. Observing Fig. 3(c), only the red point is divided as true off-plane edges with Eq. (10).

Fig. 3. (a) The close-up view and the entire source points result, (b) close-up view and entire performance of coarse dense depthmap (c) the value of feature vectors of three different points across the (2N + 1) * 1 patch (Color figure online)

From the labels of off-plane edges, the energy minimization equation for depth-propagation could be updated again as: ˆ ˆ T D(d − d), ˆ + λ(d − d) E(d) = dT Ld

(11)

ˆ is modified as: where L 1 ε T ˆ j) = (1+(Ii − χ(i, k)) (Σk + U3 )−1 )(Ij −χ(j, k))) L(i, (δij − k|(i,j)∈ωk |ωk | |ωk | (12) here (13) χ(i, k) = (1 − Πi )µk + Πi Ii where Πi = 1 when the point is classified as off-plane edge, and 0 if in-plane edges. From the equation above, in our labeled-Laplacian propagation method, only off-plane edges’ color and structure differences of guided image are utilized to update the modified L-matrix. This is because that only at off-plane edges, depth boundaries are aligned with edges of guided image. In this way, we generate the refined dense depthmap, where sparse points with different labels are propagated differently: depth differences of off-plane edges are strengthened while depth values of in-plane edges are smoothed.

42

W. Li et al.

4

Experiments

4.1

Setup

Performance of our method is tested on the focal stack we introduced in Sect. 2. The movement of the focusing plane when capturing the focal stack would cause the change of field of view. It is corrected with the image registration technique [4, 8,12]. In our experiments, parameters are set as follows: GT H = 0.05, ΩT H = 20, λ = 0.1, N = 10. 4.2

All-in-Focus Comparison

We first compare our all-in-focus performance with state-of-the-art methods. Figure 4 shows the ground truth depthmap of the 3 evaluation patches. The yellow patch and the red patch both contain off-plane edges between different planes, while the blue patch only contains on-plane edges on the same depth plane. We manually set the groundtruth depthmap to produce all-in-focus patches shown in Fig. 4 by extracting the corresponding content from the focal stack.

Fig. 4. Left: the focal stack with the evaluation patches (red, blue and yellow rectangle). Upper right: manually set ground truth depthmap of the patch. The embedded number represents the index of the in-focus image. Lower right: manually set ground truth of all-in-focus patch according to the groundtruth depthmap of three different patches (Color figure online)

The comparison on our test data is presented in Fig. 5. It shows quantitative evaluation on three extracted patches and whole content of the composited image of the all the compared methods (DCT, DSIFT, 2.5D deconvolution and MGFARF). The performance is evaluated with Structural SIMilarity (SSIM) [13] index, the higher SSIM value indicates more similarity between two images. In Fig. 5, for each method, the left part shows the composited all-in-focus image. For the right part, the upper row shows the constructed all-in-focus images for different extracted patches while the lower row presents the local SSIM value map (error map). To make the error map visualize more distinguishable, we choose

Depth-Based Focus Stacking with Labeled-Laplacian Propagation

43

Fig. 5. Comparison of our method with state-of-the-art methods. (a) Our method (b) DCT (c) DSIFT (d) MGF-ARF (e) 2.5D deconvolution

44

W. Li et al.

different δSSIM for corresponding patches according to different distributions of SSIM in different patches, to map the value of [δSSIM , 1] of SSIM to [0, 1] of brightness of the displayed error map. From the comparison, we could find that our method (Fig. 5(a)) gives the highest SSIM over compared methods. Our methods preserve both off-plane edges and in-plane edges to make the strong edges and weak edges both sharpest free of artifacts and ghost edges. Whereas, the DSIFT-based method produces artifacts near both off-plane edges and in-plane edges and enhance the noise, as shown in Fig. 5(c). The DCT-based method, Fig. 5(b) and the 2.5D deconvolution method, Fig. 5(f) both produce ghost edges in the off-plane edges, which makes the strong edges blurry and the weak edges near them disappeared in the red patch and the yellow patch. The MGF-ARF method, which is presented in Fig. 5(d), although is free of artifacts of ghost edges, produces noises subject to colored texture from the guided image on the blue patch belonging to the same depth plane. 4.3

Depthmap Comparison

Figure 6 presents performance of our final dense depthmap with our labeledLaplacian propagation method and the comparison with state-of-the-art depth propagation methods(Laplacian propagation, ARF [15] and DVFF [9]). Figure 6(d) presents our refined dense depthmap, where depth values in the same depth plane are smoothed and depth boundaries are strengthened. We also choose one patch (red) to display the advantage of our method more clearly. In Fig. 6(c), depth values are totally wrong because of false edges. In small patches of Fig. 6(a) and (b), the depth value of the farther box and the boundaries

Fig. 6. Close-up views of red region and entire depthmap generated by: (a) MGF-ARF (b) traditional Laplacian optimization (c) VDFF (d) our method (Color figure online)

Depth-Based Focus Stacking with Labeled-Laplacian Propagation

45

between two different depth plane are affected by the colored texture. In Fig. 6(d) produced by our method, however, noises in the farther box are removed and depth value are smoothed with our labeled-Laplacian propagation.

5

Conclusion

In conclusion, we propose a novel focus stacking method based on max-gradient flow and labeled-Laplacian depth propagation. We utilize max-gradient flow to extract true source points to generate sparse depthmap. Then we design a depthedge operator to give these sparse points 2 different labels: off-plane edges and in-plane edges. Only off-plane edges are then utilized in the following labeledLaplacian depth propagation to generate final dense depthmap. Experiments show that our method achieve an all-in-focus image with higher quality than state-of-the-art methods. Acknowledgement. This work was partially supported by National Science Foundation of China (No. 61271390) and State High-Tech R&D Program of China (863 Program, No. 2015AA016304).

References 1. Aguet, F., Van De Ville, D., Unser, M.: Model-based 2.5-D deconvolution for extended depth of field in brightfield microscopy. IEEE Trans. Image Process. 17(7), 1144–1153 (2008) 2. Goldsmith, N.T.: Deep focus; A digital image processing technique to produce improved focal depth in light microscopy. Image Anal. Stereology 19(3), 163–167 (2011) 3. Haghighat, M.B.A., Aghagolzadeh, A., Seyedarabi, H.: Real-time fusion of multifocus images for visual sensor networks. In: 2010 6th Iranian Machine Vision and Image Processing (MVIP), pp. 1–6. IEEE (2010) 4. He, B., Wang, G., Lin, X., Shi, C., Liu, C.: High-accuracy sub-pixel registration for noisy images based on phase correlation. IEICE TRANS. Inform. Syst. 94(12), 2541–2544 (2011) 5. Huang, W., Jing, Z.: Evaluation of focus measures in multi-focus image fusion. Pattern Recogn. Lett. 28(4), 493–500 (2007) 6. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008) 7. Liu, Y., Liu, S., Wang, Z.: Multi-focus image fusion with dense sift. Inform. Fusion 23, 139–155 (2015) 8. Miao, Q., Wang, G., Lin, X.: Kernel based image registration incorporating with both feature and intensity matching. IEICE TRANS. Inform. Syst. 93(5), 1317– 1320 (2010) 9. Moeller, M., Benning, M., Sch¨ onlieb, C., Cremers, D.: Variational depth from focus reconstruction. IEEE Trans. Image Process. 24(12), 5369–5378 (2015) 10. Sroubeka, F., Gabardab, S., Redondob, R., Fischerb, S., Crist´obalb, G.: Multifocus fusion with oriented windows. In: Proceedigs of SPIE, vol. 5839, p. 265 (2005)

46

W. Li et al.

11. Suwajanakorn, S., Hernandez, C., Seitz, S.M.: Depth from focus with your mobile phone. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3497–3506 (2015) 12. Thevenaz, P., Ruttimann, U.E., Unser, M.: A pyramid approach to subpixel registration based on intensity. IEEE Trans. Image Process. 7(1), 27–41 (1998) 13. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004) 14. Yin, X., Wang, G., Li, W., Liao, Q.: Iteratively reconstructing 4D light fields from focal stacks. Appl. Opt. 55(30), 8457–8463 (2016) 15. Yin, X., Wang, G., Li, W., Liao, Q.: Large aperture focus stacking with maxgradient flow by anchored rolling filtering. Appl. Opt. 55(20), 5304–5309 (2016) 16. Zhuo, S., Sim, T.: Defocus map estimation from a single image. Pattern Recogn. 44(9), 1852–1858 (2011)

A Novel Method for Measuring Shield Tunnel Cross Sections Ya-dong Xue1,2(&), Sen Zhang1,2, and Zhe-ting Qi2 1

Key Laboratory of Geotechnical and Underground Engineering of Education Ministry, Tongji University, Shanghai 200092, China [email protected] 2 Department of Geotechnical Engineering, Tongji University, Shanghai 200092, China

Abstract. With more metro tunnels being constructed and operated, the task of measuring tunnels’ deformation becomes more imperative. This article proposes a novel method for measuring shield tunnel cross sections based on close range photogrammetry. Direct Linear Translation (DLT) method is suitable for non-metric photography, requiring several control points on the linings, which is time-consuming. A new method of setting control points was put forward to overcome the shortcoming. A laser source forms a bright outline on the tunnel’s inner surface. The polar coordinates of control points on the outline are gained by a laser range-finder installed on a 360° protractor. These coordinates are used to solve the unknown parameters of DLT equations. Then the precise outline of the tunnel cross sections can be obtained. A series of tests in the subway tunnel of Shanghai Metro Line 1 were carried out to validate the method being precise and effective. Keywords: Tunnel lining deformation
Cross sections Close range photogrammetry
Laser orientation
Laser range-finding

1 Introduction With a large number of metro tunnels being constructed and operated in China, the fast and effective tunnel deformation measuring method becomes imperative as tunnel’s deformation condition is important for evaluating its performance. Most of the metro tunnels are constructed using the shield method. Shield tunnels’ deformation is generally induced by cyclic train loads, foundation deformation and adjacent construction activities, not only impacting tunnels’ durability but also reducing their safety factors (Wang et al. 2009) [1]. In soft soil regions such as Hangzhou city and Shanghai city, tunnels’ deformation is usually large. The maximum deformation observed in Shanghai metro was 148 mm (2.4%D) (Zhang et al. 2014) [2]. Traditional deformation measuring techniques mainly include Bassett convergence instruments and total stations. The convergence instruments, which measure tunnels’ deformation through the change of the measuring lines’ length, are typical contacting measurements. Setting an enormous number of measuring lines is time-consuming, costly and tedious (Bassett et al. 1999) [3]. Measurement of total station is discrete and © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 47–57, 2017. https://doi.org/10.1007/978-3-319-71598-8_5

48

Y. Xue et al.

inflexible even though it implements precise and non-contact measurements (Hope et al. 2008; Yang et al. 2005; Yang et al. 2006) [4–6]. These two methods cannot satisfy the increasing measuring requirements any longer due to the shortcomings aforementioned. Some new measuring methods have appeared in recent years such as light detection and ranging (LiDAR), which is able to obtain the complete data of cross sections. Han et al. (2013) [7] used Minimum-Distance Projection (MDP) algorithm to establish point correspondences to measure tunnels’ deformation. Some researchers established 3D models of tunnels using terrestrial laser scanning to measure tunnels’ geometry deformation and settlement (Xie et al. 2017; Argüelles et al. 2013) [8, 9], but for most subway companies it’s still too expensive. The close range photogrammetry technology advances rapidly as digital single-lens reflex cameras mature. Study on photogrammetry using non-metric cameras has become a research hotspot during the past years. The shapes, dimensions and locations of the vertical cross-sections of tunnels can be derived from the photographs based on the control data provided by two targets’ locations on the plates (Ethrog et al. 1982) [10]. Zhang et al. (2001) [11] used a CCD area array camera to measure areas of irregular plane objects. Yang et al. (2001) [12] proposed a novel algorithm to improve CCD cameras’ measuring precision. Lee et al. (2006) [13] found that the displacement and strain data obtained with close range photogrammetric technique in a model tunnel showed a remarkable agreement with the physical data. Hu (2008) [14] used close range photogrammetry to measure a mountain tunnel’s deformation, but the precision was unsatisfactory. Chen et al. (2014) [15] proposed using a mobile metal bracket to offer control points, which improved the practicability and maneuverability of DLT method in tunnel engineering. This article tries to create a novel method for measuring shield tunnel cross sections based on laser orientation, laser range-finding and close range photogrammetry. Based on a novel way of setting control points and target points by using laser, this article studies on how to apply DLT method in tunnel monitoring. A specially-designed device was used in the field experiment. The results proved the practicability and accuracy of the proposed method.

2 Close Range Photogrammetry 2.1

Imaging Principle

The pinhole imaging is the fundamental theory of close range photogrammetry (see Fig. 1). Based on the similar triangles principle, we have:

Fig. 1. The theory of pinhole imaging

A Novel Method for Measuring Shield Tunnel Cross Sections

49

0

h h ¼ S2 f f

ð1Þ

0

h h ¼ S1 S2

ð2Þ

0

Where h stands for the object height, h stands for the image height, f represents the focal length, S1 represents the distance between the object plane and the optical center and S2 represents the distance between the imaging plane and the optical center. Rearranging the two equations, we can get: 1 1 1 þ ¼ f S1 S2

ð3Þ

As S1 is usually far greater than S2 in engineering projects, it is easy to get: 0

h ¼

2.2

fh S1

ð4Þ

General Coordinate Systems

Four general coordinate systems are used in DLT method: world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system. World coordinates are absolute coordinates that objectively describe the three-dimensional space, signified with (Xw, Yw, Zw). Camera coordinates are signified with (Xc, Yc, Zc) and the origin point of them is located at the lens’ optical center. Zc represents the lens’ main optical axis. Image coordinates and pixel coordinates are signified with (x, y) and (u, v) separately. The main difference between them lies in the measurement unit. The former one is measured in millimeter or centimeter whereas the later one is measured in pixel. Figure 2 shows the conversion relations of the four general systems. It should be pointed out that the conversion between camera coordinates and image coordinates is based on Eq. (4).

Fig. 2. General coordinate systems

50

Y. Xue et al.

The transformation equation between pixel coordinates and world coordinates is: 0 1 01 u dx ZC @ v A ¼ @ 0 1 0

10 u0 f 00 v0 A@ 0 f 0 0 01 1

0 1 dy

0

1 0  R 0A T 0 0

1 XW T B YW C A @ 1 ZW 1 

0

ð5Þ

Where R represents a certain 3  3 rotation matrix and T represents a 3  1 translation matrix. The parameters dx and dy are the physical dimensions of one pixel in x-axis direction and y-axis direction in the image plane respectively. 2.3

Direct Linear Translation Method

Among the commonly used parsing methods for close range photogrammetry, DLT is especially suitable for non-metric photography. It does not involve initial approximations for the unknown parameters of inner and outer orientation of the camera (Abdel-Aziz et al. 2015) [16]. DLT contains a direct linear transformation from pixel coordinates into world coordinates with Eq. (6) which is based on Eq. (5). A
m¼B 2

Xw1 Yw1 6 0 0 6 6 Xw2 Yw2 6 0 0 A¼6 6 .. 6 6 . 4 X Y wn wn 0 0

Zw1 0 Zw2 0 Zwn 0

1 0 1 0 1 0

0 Xw1 0 Xw2 0 Xwn

0 Yw1 0 Yw2 .. . 0 Ywn

ð6Þ

0 Zw1 0 Zw2

0 1 0 1

0 Zwn

0 1

u1 Xw1 v1 Xw1 u2 Xw2 v2 Xw2 un Xwn vn Xwn

u1 Yw1 v1 Yw1 u2 Yw2 v2 Yw2 .. . un Ywn vn Ywn

u1 Zw1 v1 Zw1 u2 Zw2 v2 Zw2 un Zwn vn Zwn

3 7 7 7 7 7 7 7 7 5

2n11

ð7Þ B ¼ ½ u1

v1

u2

v2






un

vn ŠT2n1

ð8Þ

Where m represents the coefficient matrix with Eq. (9). Interior and exterior orientation elements are implicit in the m matrix. m ¼ ½ m1

m2

m3

m4






m10

m11 ŠT111

ð9Þ

In general, pixel coordinates in a photo are easily obtained. Then Eq. (6) could be used to calculate the world coordinates of any target point if the m matrix is calculated. To solve the m matrix, several control points are required, of which the world coordinates are known. Three-dimensional DLT method requires at least six control points as there are 11 unknown Parameters in the m matrix. In tunnel engineering, these points are normally set on the inner surface of the linings (Ma et al. 2006) [17]. To improve the measuring precision, they are generally more than six. Actually, it’s difficult to set many control points in a shield tunnel, which has restricted the application of DLT method in tunnel projects.

A Novel Method for Measuring Shield Tunnel Cross Sections

51

The calculating process of DLT method is divided into four steps: (1) identifying and positioning control points; (2) computing the m matrix (camera calibration); (3) extracting pixel coordinates of target points; (4) calculating world coordinates of target points. Chen’s research is an example of applying DLT method in tunnel engineering (Chen et al. 2014) [15]. A metal bracket was placed in the tunnel with predefined target points on the linings when taking a photo. The bracket established a reference coordinate system in photos. The m matrix could be calculated through these points’ coordinates. However, solving any target point needed at least two photos captured from different vision angles because only two equations can be built from one photo for each target point which contains three unknowns: (Xw, Yw, Zw). 2.4

Laser-Based Calibration Method

This article proposes a novel calibration method based on laser range-finding, which aims to achieve higher precision at lower cost. A laser source is fixed on a self-made inspection device (MTI-200A) to create a bright outline on the lining surface. On the same device, a laser range-finder is placed on a 360° protractor, which is coplanar with the outline plane (see Fig. 3). The distances and their corresponding angles of control points are recorded by rotating the laser range-finder (see Fig. 4). The recorded data will be used to calculate world coordinates of control points with Eqs. (10) and (11). Xwi ¼ Li  sinhi

ð10Þ

Ywi ¼ Li  coshi

ð11Þ

Where Li stands for the distance and hi stands for the corresponding angle.

Fig. 3. The related devices

Fig. 4. A control point on the outline

52

Y. Xue et al.

The parameter Zw in 3D DLT equations is assumed as zero on the hypothesis that the outline and control points are located in the same plane. Then the matrix A in Eq. (7) transforms into 2D form as shown in Eq. (12). 2

6 6 6 A¼6 6 4

Xw1 Yw1 1 0 0 0 .. .. .. . . . Xwn Ywn 1 0 0 0

0 0 Xw1 Yw1 .. .. .. . . . 0 0 Xwn Ywn

0 1 0 1

u1 Xw1 v1 Xw1 .. .. . . un Xwn vn Xwn

3 u1 Yw1 v1 Yw1 7 7 7 7 7 un Ywn 5 vn Ywn 2n8

ð12Þ

Solving the m parameters of 2D DLT requires at least 4 control points due to the unknown number of m has decreased from 11 to 8. To achieve higher measuring precision, more control points are needed. Least squares method as shown in Eq. (13) is used to process the data if there are extra control points. m ¼ ðAT AÞ 1 AT B

ð13Þ

The most significant improvement of this method is that solving each target point needs only one photo captured from any vision angle. The measuring system frame is shown in Fig. 5. The relevant algorithms were programed with Mathematica.

Fig. 5. The measuring system frame

3 Field Test and Verification 3.1

Preparation

The main advantages of non-metric digital cameras are listed as follows: (1) The interior and exterior orientation elements of them can be calculated accurately; (2) They do not need negatives like film cameras do; (3) They storage data in digital form and that data can be easily taken use of by a computer; (4) They are fairly cheaper than

A Novel Method for Measuring Shield Tunnel Cross Sections

53

metric cameras. (5) Specially-designed programs are able to complete the calculating process automatically and correctly. Nikon D7000 digital camera and Tokina AT-X PRO SD 11-16 F2.8 IF DX IINikon lens were chosen as the test equipment as shown in Fig. 6. Full frame cameras were not considered as they had not been in widespread use. The image sensor of D7000 has sixteen million pixels, which is enough for the experiments. A wide-angle lens was required and Tokina 11–16 met the requirements perfectly. The 11 mm focus was preferred because it allowed the experimenter to approach closer to the outline than the 16 mm focus. The only problem lied in the noticeable barrel image distortion at the 11 mm focus as shown in Fig. 7, released by the famous image quantity reference website DXOMARK [18]. Distortion means that the original straight lines are distorted into curved lines which will undoubtedly lead to some errors in measurements. To avoid that, the Adobe software Lightroom was used to correct the distortion. Figure 8 shows the original image of a gridding board shot by D7000 and Fig. 9 shows the modified image using the Lightroom software. Although the distortion in the image edges is still detectable, the maximum deviation of straight lines has decreased from 75 pixels to 15 pixels. Furthermore, the outline range seldom ever appears on the image edges as Fig. 10 shows, in fact. In terms of correcting distortion, lots of papers talked about different correcting algorithms. For example, Jin et al. (2011) [19] used BP neural network model to correct image distortion.

Fig. 6. The D7000 camera

Fig. 7. The image distortion

To demonstrate the feasibility of DLT method, four randomly selected points E, F, G, H in Fig. 9 were chosen as control points to solve the relative coordinates of other points. Each little square was 17.5 mm in width and 24.8 mm in height. The maximum error appeared at the point S which was 1.1315 mm. The pixel distances between target points and the image center were defined as PD. The percentage errors were defined as absolute errors divided by measuring distances. Then the curves of PE in x-axis and y-axis to PD were obtained as shown in Fig. 11. It is clearly indicated that errors in the image edges are larger than those in the image center.

54

Y. Xue et al.

Fig. 8. The original image

Fig. 9. The corrected image

Fig. 10. The outline range

Fig. 11. The curve of percentage errors to pixel distances

3.2

Field Experiment

A series of tests were carried out in the subway tunnel of Shanghai Metro Line 1 using the specially-designed device. Specific experiment steps were as follows: (1) The laser range-finder and the protractor were installed on the tunnel inspection vehicle. (2) The laser source was powered and it formed a bright red circular outline on the inner surface of the lining. (3) The laser range-finder was rotated with a 10-degree interval. (4) The real-time distances and their corresponding angles were recorded. (5) During the test, a total station was used to get the precise coordinates of these recorded points. The model of the inspection device was MTI-200A (see Fig. 10). The laser range-finder’s model was HZH-80 with an accuracy of 1 mm. The total station’s model was SOKKIA CX-102. Its accuracy was (3 + 2 ppm*D) mm without reflection prism.

A Novel Method for Measuring Shield Tunnel Cross Sections

3.3

55

Test Results

Pixel coordinates of tens of thousands of target points on the outline were extracted automatically on the basis of color differences using Mathematica software. In total, about seventeen thousand pixel points were extracted of each outline. Twenty uniformly-distributed control points were taken into consideration. Two fitting outlines are shown in Fig. 12. The red one was fitted based on the absolute coordinates measured by the total station while the blue one was fitted according to the data obtained by the proposed method. The horizontal diameter of the red outline was 5539 mm whereas that of the blue one was 5533 mm, meaning that the error was below 6 mm. According to the design document as Fig. 13 shows, the designed dimension of the tunnel’s inner diameter was 5500 mm. The maximum magnitude of horizontal convergence was around 39 mm, which was in accordance with the monitoring data on site. Compared with the total station, the precision of the proposed method was acceptable and the method was more flexible and more effective. The specially-designed mobile device was not only low-cost but also handy. With the device, the DLT method was potential to be applied in tunnel deformation monitoring. The abundant measurement results would provide basing data for stress analysis and performance analysis.

Fig. 12. Two fitting outlines (Color figure online)

3.4

Discussion

As Fig. 12 shows, there are still some deviations between the red outline and the blue one. Error sources in close range photogrammetry were discussed thoroughly by Song et al. (2010) [20]. In this test, there were various factors that led to errors. They were analyzed as follows: (1) The image distortion still had a little influence on the results. (2) The laser range-finder was rotated manually, leading to inaccuracy in angles. (3) The measuring equipment had systemic errors. (4) The sensor’s pixels were not abundant enough.

56

Y. Xue et al.

Fig. 13. The geometry of shield tunnel’s linings

More control points will lead to higher measuring precision intuitively. However, it means a higher cost meanwhile. The optimal option should satisfy the precision requirements at minimum costs. An important question is determining that how many control points are optimal. Assuming that eight control points are optimal, then eight laser range-finders are installed uniformly on a protractor. It is fixed on the inspection device in practical applications. When the device moves along the tracks, the only thing necessary is to photograph the outline continually. Each photo records a certain outline and hundreds of outlines are recorded. Photoelectric encoder is used to record the real-time distances the device has moved. Once the outlines and their corresponding locations are combined correctly, the complete 3D shape of a tunnel will be gained, which is extremely helpful for engineers to evaluate the tunnel’s performance.

4 Conclusions A novel method for measuring tunnel cross sections was put forward. It was based on laser orientation module, laser range-finding module and the theory of close range photogrammetry. A series of tests in the subway tunnel of Shanghai Metro Line 1 were carried out to validate the method being low-cost and precise. Measurement results were satisfied with twenty uniformly-distributed control points for the tunnel around 5.5 m in diameter. The maximum error between the calculated outline and the actual outline was just 6 mm. Acknowledgements. The authors acknowledge the support of National Natural-Science Foundation of China (Grant No. 40772179), Science and Technology Commission of Shanghai Municipality (Grant No. 16DZ1200402) and the support of Science and Technology Plan of Department of Communication of Zhejiang province (Grant No. 2010H29).

A Novel Method for Measuring Shield Tunnel Cross Sections

57

References 1. Wang, R.L.: Analysis on influencing factors and deformation characteristics of shanghai soft soil subway tunnel deformation. Undergr. Eng. Tunn. 1(1), 7 (2009) 2. Zhang, D.M., Zou, W.B., Yan, J.Y.: Effective control of large transverse deformation of shield tunnels using grouting in soft deposits. Chin. J. Geotech. Eng. 36(12), 2203–2212 (2014) 3. Bassett, R., Kimmance, J., Rasmussen, C.: An automated electrolevel deformation monitoring system for tunnels. Proc. Instit. Civil Eng. Geotech. Eng. 137, 117–125 (1999) 4. Hope, Colin, Marcelo, C.: Geotechnical instrumentation news-manual total station monitoring. Geotech. News 26(3), 28 (2008) 5. Yang, S.L., Wang, B., Ji, S.Y., Liu, W.N., Shi, H.Y.: Non-contact monitoring and analysis system for tunnel surrounding rock deformation of underground engineering. Trans. Nonferrous Met. Soc. China (English Edition) 15, 88–91 (2005) 6. Yang, S., Liu, W., Shi, H., Huang, F.: A study on the theory and method of non-contact monitoring for tunnel rock deformation based on free stationing of a total station. China Civil Eng. J. 39, 100–104 (2006) 7. Han, J.Y., Guo, J., Jiang, Y.S.: Monitoring tunnel profile by means of multi-epoch dispersed 3-D LiDAR point clouds. Tunn. Undergr. Space Technol. 33, 186–192 (2013) 8. Xie, X.Y., Lu, X.Z.: Development of a 3D modeling algorithm for tunnel deformation monitoring based on terrestrial laser scanning. Undergr. Space 2, 16–29 (2017) 9. Argüelles-Fraga, R., Ordóñez, C., García-Cortés, S., Roca-Pardiñas, J.: Measurement planning for circular cross-section tunnels using terrestrial laser scanning. Autom. Constr. 31(3), 1–9 (2013) 10. Ethrog, U., Shmutter, B.: Tunnel calibration by photogrammetry. Photogrammetria 38(3), 103–113 (1982) 11. Zhang, Q.F., Hang, Y.X., Yang, H.B.: Area measurement of irregular plane object using area array CCD. Instrum. Tech. Sens. 2, 36–39 (2000) 12. Yang, L.F., Hang, J.W., Li, Y.Z.: Application of high resolution measurement technique with area CCD. J. Taiyuan Univ. Technol. 32(5), 455–458 (2001) 13. Lee, Y.J., Bassett, R.H.: Application of a photogrammetric technique to a model tunnel. Tunn. Undergr. Space Technol. 21(1), 79–95 (2006) 14. Hu, T.Y.: Study on Close Range Photogrammetry in Tunneling Engineering’s Application. Beijing Jiaotong University, Beijing (2008) 15. Chen, Z.H., Zhao, M., Qiao, D.L.: Research on monitoring method of metro tunnel deformation based on close - range photogrammetry. Spec. Struct. 5, 61–65 (2014) 16. Abdel-Aziz, Y.I., Karara, H.M.: Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. Photogramm. Eng. Remote Sens. 81, 103–107 (2015) 17. Ma, G.L., Zhang, Q.Y., Li, S.C.: Application of digital photogrammy in deformation measurement for 3D geo-mechanics model test of large tunnel driving. J. Geotech. Investig. Surv. 6, 50–52 (2006) 18. DXOMARK. http://www.dxomark.com 19. Jin, Y., Meng, J.B., Wang, K.: The research of plane array camera geometric distortion correction method. Microelectron. Comput. 28(10), 36–38 (2011) 20. Song, M.: Analysis on precision of multi-lens area array CCD mapping camera. Chin. J. Space Sci. 30(6), 589–595 (2010)

Image Noise Estimation Based on Principal Component Analysis and Variance-Stabilizing Transformation Ling Ding1,2, Huying Zhang1(&), Bijun Li3, Jinsheng Xiao4, and Jian Zhou3 1

2

4

School of Computer, Wuhan University, Wuhan 430072, China {dn0715dn,zhy2536}@whu.edu.cn College of Computer Science and Technology, Hubei University of Science and Technology, Xianning 437100, China 3 State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sencing, Wuhan University, Wuhan 430079, China School of Electronic Information, Wuhan University, Wuhan 430072, China

Abstract. Image denoising requires taking into account the dependence of the noise distribution on the original image, and the performance of most video denoising algorithms depend on the noise parameters of noisy video, which is particularly important for the estimation of noise parameters. We propose a novel noise estimation method which combines principal component analysis (PCA) and variance-stabilizing transformation (VST), and extend the noise estimation to mixed noise estimation. We also introduce the excess kurtosis to ensure the accuracy of noise estimation and estimate the parameters of VST by minimizing the excess kurtosis of noise distribution. Subjective and objective results show that proposed noise estimation combining with classic video denoising algorithms obtains better effects and make video denoising more widely in application. Keywords: Video denoising
Noise estimation
Principal component analysis Variance-stabilizing transformation

1 Introduction The noise signal is a pollution signal, which will seriously affect the image quality of the video picture, bring a lot of difficulties to the follow-up process of the video image and affect people’s visual experience of video images, the traditional video image noise reduction algorithms are implemented based on the known noise levels [1–3], the accurate noise level estimation is very necessary and can further improve the performance of noise

This work is supported by the National Natural Science Foundation of China (61540059, 41671441, 91120002); the Plan Project of Guangdong Provincial Science and technology (2015B010131007). © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 58–69, 2017. https://doi.org/10.1007/978-3-319-71598-8_6

Image Noise Estimation Based on Principal Component Analysis

59

reduction algorithm. The current noise estimation algorithm can be divided into three types of space domain, transform domain and matrix domain. The spatial noise estimation algorithm is to deal with the noisy image directly and mainly dependent on the weak texture area of the image for the estimation of the noise variance, which is divided into the image block based noise estimation, the filtering based noise estimation as well as mixed noise estimation. Liu et al. [4] have proposed based on the image block based noise estimation, which uses the gradient covariance matrix to solve the weak texture region of the image and estimate the noise level of the weak texture region, which the estimation is only for the Gaussian noise level. Pei et al. [5] proposed a filtering based noise estimation method, which processes the texture details of the image through an adaptive filter and estimates the noise level combining the noisy image blocks and their filtered image blocks. For the noise estimation in the transform domain, Ponomarenko et al. [6] proposed the noise estimation algorithm based on DCT transform, which transforms the image into the domain with DCT transform to field to separate the signal and noise, achieves a more accurate noise estimation in dealing with some simple images with less textures. Based on Yu, Li et al. [7] proposed the method of noise estimation relying on test, and made noise estimation by reducing the value of the image by wavelet transform, as well as the relationship between the noise signal and its variance. Donoho [8] proposed to use the wavelet soft value for noise estimation, made mean absolute deviation (MAD) in the HH sub-band of the wavelet transform and estimated the noise standard deviation. Noise estimation of space and transform domains are both requiring the image block is smooth enough, but in many cases, the given video image contains a lot of random textures, then the above-mentioned noise estimation algorithms cannot deal with the video images with more complex textures. While the matrix domain noise estimation uses the idea of matrix decomposition to distinguish the video image signal and the noise signal in the matrix domain, so it also applies to the video images with more complex textures [9]. Matrix domain noise estimation is mainly divided into two types of the principal component analysis and singular value decomposition. The principal component analysis (PCA) algorithm based on image blocks proposed by Pyatykh et al. [10] can not only deal video images with a smooth area, but get a more accurate level of noise for the video images with more complex textures. Liu and Lin [11] proposed a noise estimation algorithm based on singular value decomposition (SVD) to estimate the noise level by the singular value decomposition of the noisy video image, while the above noise estimation algorithms are only for the Gaussian noise estimation, and the estimation effect of sensor noise, i.e. Poisson-Gaussian noise, is not ideal. Based on PCA, this paper combines variance stability transformation (VST) to estimate the noise of the noisy images, which can estimate not only the simple Gaussian noise but the sensor noise level. Meanwhile, the concept of excessive peak is introduced to further determine the accuracy of the mixed noise estimation parameters, and estimate the parameters of the VST transform with the excessive peak of the minimum noise distribution. The subjective and objective results have shown that the combination of the noise estimation algorithm in this paper and the classical video ablation algorithm has achieved good results and made the video denoising application gain a wider range of applications.

60

L. Ding et al.

2 Related Works 2.1

PCA Noise Estimation

For the known model in image block, we need to select the appropriate image block for the main component analysis. First define a positive integer m. When all the image blocks corresponded by the original video image xi are all in the sub-space VM m 2 RM , the information of the original video image x has a redundancy characteristic, where the dimension M–m of the subspace is smaller than the vector dimension M, so we are select the desired image block based on this assumption and the standard selection formula as follows: di ¼ Disðxi ; VM

mÞ

i ¼ 1. . .. . .:K

ð1Þ

In Eq. (1), Dis () represents the distance, that is, when the distance between the image block xi and the subspace VM m 2 RM meet a certain range, the image block xi is determined as the appropriate image block. Since the noise signal and the video image signal are independent of each other, thus the variance of the original image block, the noise variance and the variance of the noisy image block have the following relationship: sðxi Þ ¼ sðyi Þ

r2 i ¼ 1. . .. . .:K

ð2Þ

According to the above formula, r is standard deviation, we can know that he noisy image block yi is positively correlated with the image block distance di. Thus we can eventually be select the appropriate image block for principal component analysis based on the standard deviation of the noisy image block. After the eigenvalue decomposition, the main constituent of the vector Y is obtained as vTY;i Y, and satisfies the following relationship: s2 ðvTY;i YÞ ¼ kY;i i ¼ 1; 2; . . .:; K

ð3Þ

Where s2 represents the sample variance, kY;i represent the eigenvalues of the covariance matrices SY, respectively, the corresponding eigenvectors are vY;i . According to the selection criteria of the image block and vector Y composed by the selected image block, m characteristic values satisfies the following relationship:
Eð
kY;i

pffiffiffiffi
r2
Þ ¼ oðr2 = N Þ

N!1

ð4Þ

Where i has the range of [M–m + 1, M]. Since the original video image signal and the noise signal are independent of each other, the covariance matrix of X and N is 0, then the following formula holds:

Image Noise Estimation Based on Principal Component Analysis

SY ¼ SX þ SN

S N ¼ r2 I

61

ð5Þ

Where SY, SX, and SN represent the covariance matrix of Y, X, and N, respectively. As the m minimum eigenvalues of the covariance matrix SX are zero, the corresponding m m minimum eigenvalues of SY is r2 . When the number of samples N is large, the following formula is satisfied:
lim Eð
kY;M

N!1


r2
Þ ¼ 0

ð6Þ

As Formula (6) shoes, when the number of samples is large enough, the minimum eigenvalue of the sample covariance matrix SY for the noisy video image is approximately equal to the variance of the noise signal. Therefore, we can calculate the eigenvalues of the sample covariance matrix for the noisy image to approximate the estimated noise variance. 2.2

VST-Based Noise Estimation

Noise estimation algorithm based on the principal component analysis (PCA) is only for the estimation of noisy images of Gaussian noise, and has a poor effect on of treating the sensor noise, that is, Poisson-Gaussian noise, based on this, the noise estimation of noisy images combining variance stability transformation (VST) could not only estimate the simple Gaussian noise, but estimate the sensor noise level, with a relatively accurate estimation. The following equation is obtained by solving the variance of the noisy image model: varðyðpÞÞ ¼ a2 kðpÞ þ b ¼ axðpÞ þ b

ð7Þ

From the above equation, Parameter a represents the multiplicative factor, the noise variance of the noisy image is linearly related to the original image pixel value. According to the characteristics of Poisson distribution, when k(p) is large enough, x(p) approximately obeys the normal distribution with mean k(p) and variance of k(p). We know that the pixel image y(p) approximately obeys the normal distribution with mean x(p) and variance of axðpÞ þ b, the sensor noise can be approximated as the additive white Gaussian noise, and the following relationship is satisfied between the pixel image y(p) and the original image pixel value x(p): yðpÞ  xðpÞ þ

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi axðpÞ þ bnðpÞ

ð8Þ

The above equation is to solve the noise level of the noisy image, and firstly we must solve the noise parameters a, b. Therefore, the solution of the noise level according to the noise model is turned into the parameter of the noise model.

62

L. Ding et al.

Define f(y(p); a, b) as a function of the noisy image y(p), that is, the variance stability transformation function, the transformed image f(y(p); a, b) is independent of the original image, and the definition and its standard deviation are as follows: stdðf ðyðpÞ; a; bÞÞ ¼ r

ð9Þ

A first-order Taylor expansion for the transformed image f(y(p); a, b) is performed at x(p), with the expression as follows: 0

f ðyðpÞ; a; bÞ  f ðxðpÞ; a; bÞ þ f ðxðpÞ; a; bÞðyðpÞ

xðpÞÞ

ð10Þ

According to the expanded formula, we can get the approximate expression of the formula (9) as follows: 0

f ðxðpÞ; a; bÞ
stdðyðpÞÞ ¼ r 0

f ðxðpÞ; a; bÞ ¼

r r ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi stdðyðpÞÞ axðpÞ þ b

ð11Þ ð12Þ

Solve the integral on n both sides of the formula (12) to get the following expression: 2r pffiffiffiffiffiffiffiffiffiffiffiffi f ðt; a; bÞ ¼ at þ b ð13Þ a In the above equation, t represents a random variable, that is, the ariance stability transformation of random variable t. Variance stability transformation (VST) is a smoothing function, the noisy image y(p) approximately obeys the normal distribution, therefore, when y(p) has a sufficiently small variance, the transformed image f(y(p); a, b) also approximately obeys the normal distribution, that is, for all pixel values, stdðf ðyðpÞ; a; bÞÞ  r holds. So for the transformation of the image, the noise signal can be approximated as the additive white Gaussian noise. 2.3

PCA-Based Image Block Transformation

As the transformation characteristics of the above image variance stability show, the noise signal of the transformed image f(y(p); a, b) is approximated as the additive Gaussian white noise signal with a standard deviation of r, so the transformed image can be considered as the original noise-free image Z by adding noise, therefore, the expectation of transforming the image is the noise-free image Z, with the expression as follows: Eðf ðyðpÞ; a; bÞÞ ¼ Z

ð14Þ

Define N as the number of image blocks in the transformed image, K is the size of the image block, the image block is transformed into a vector of size K by removing the unnecessary elements of each image block, where the N vectors of the transformed image are v1, … vN and the N vectors of the image Z are u1, … uN. In order to

Image Noise Estimation Based on Principal Component Analysis

63

effectively separate the noise signal from the image signal, the image Z is assumed to have the redundant characteristics, that is, the dimensions of the vectors u1, … uN are less than K, and use PCA to show as follows: (1) Calculate the mean vectors of vectors v1, … vN, with the calculation formula as follows: v¼

N 1X vi N i¼1

ð15Þ

(2) Calculate the sample covariance matrix of vectors v1, … vN, with the calculation formula as follows: S¼

1 N

1

N X i¼1

ðvi

vÞðvi

vÞT

ð16Þ

(3) We get the normalized eigenvectors a1, … aK of the sample covariance S, and these eigenvectors form a set of orthogonal bases and obey the following relations: s2 ðaT1 vi Þ  s2 ðaT2 vi Þ 


 s2 ðaTK vi Þ

ð17Þ

Where s2 ð
Þ represents the sample variance. (4) The expression for the weight calculation is as follows: xk;i ¼ aTk ðvi

vÞ

ð18Þ

Where k ranges from [1, K], the value range of i is [1, N]. xk;i represents the kth weight of the center vector ðvi vÞ , expressed as x, since the distribution of the noise vector is a multivariate Gaussian distribution, its expression is as follows: s2 ðaTi vi Þ  s2 ðaTi ui Þ þ r2

ð19Þ

We can know that the sample variance s2 ðaTk vi Þ is equal to the eigenvalue of the sample covariance matrix S. when the principal component analysis (PCA) utilizes the redundancy of the noise-free image Z, this allows the sample vectors u1, … uN to be linearly represented by the previous M feature vectors, for the last feature vector, the sample vectors u1, … uN are orthogonal. Therefore, the distribution of the weight xK is the same as the noise distribution [5], and in practice, the distribution of the noise signal can be replaced by analyzing the distribution characteristic of the weight xK , and from (19) we can know that when s2 ðaTK ui Þ ¼ 0, the noise variance can be approximated as the weight variance, and the expression is as follows:

64

L. Ding et al.

s2 ðxK Þ  r2

ð20Þ

3 VST Correction Based on Excessive Peak In estimating the noise level, we must consider the distribution characters of noise signals, so the authenticity of the noise parameters is unknown, and the parameters a and b obtained by VST deviates from the real parameters, therefore, VST transform may not have the variance stabilizing effect, and the distribution of the noise signal at this time also deviates from the normal distribution category. Therefore, we need to measure the normal distribution of noise signals to evaluate the resulting transformation parameters. Here, we use the excess peak to carry out the detection, and the expression for the excessive peak of the random variable X is expressed as follows: cX ¼

EðXÞÞ4 Þ

EððX

ð21Þ

3

EðXÞÞ2 Þ2

EððX

In the above formula, when the random variable follows a normal distribution, its excessive peak cX is zero. Therefore, the reduction of excessive peak is the necessary condition for noises to obey the normal distribution. For the noisy image expression of the noisy model, its noise obeys the sufficient condition of the normal distribution. Define x1 \


\xM as the pixel value of image x, and the corresponding probabilities are h1 ; . . .; hM , we assume that the parameters in 0 the VST transform are a , b0 , which are not equal to the true noise model parameters a and b. According to the first-order Taylor expansion, the standard deviation of the 0 0 transformed image f ðyðpÞ; a ; b Þ can be obtained as follows: 0

0

0

0

0

stdðf ðyðpÞ; a ; b ÞÞ  f ðxðpÞ; a ; b Þ
stdðyðpÞÞ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi axðpÞ þ b ¼ r pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi a0 xðpÞ þ b0

ð22Þ

According to the above equation, the noise variance of the pixel value x1 \


\xM of the image x can be obtained. The calculation expression is as follows: r2i ¼ r2

axi þ b i ¼ 1; . . .; M a0 x i þ b0 0

ð23Þ 0

It can be obtained that the transformed image f ðy; a ; b Þ obeys the normal distri0 0 bution with the variance of r2i , so in the transformed image f ðy; a ; b Þ, the noise signal can be expressed as a multivariate Gaussian distribution Nð0; r2i Þ with the weight of hi , and its excessive peak expression is as follows:

Image Noise Estimation Based on Principal Component Analysis

c¼3

M P

ð

i¼1 M P

i¼1

65

hi r4i 3

hi r2i Þ2

ð24Þ

As Eq. (24) shows, the excess peak is a nonnegative number, only in cases where 0 0 all ri are equal, that is, the parameters a ; b and a; b proportionate, it will be zero, and the formula is as follows: a b 0 ¼ a b0

ð25Þ

0

0

If the above formula holds, then f ðyðpÞ; a ; b Þ and the original image pixel value xðpÞ are independent of each other, that is, the noise signal of the transformed image 0 0 f ðyðpÞ; a ; b Þ is the additive white noise. Therefore, the reduction of excessive peak is a necessary and sufficient condition for the additive white gaussian noise. Assume that the excessive peak of the sample is GðXi Þ, the corresponding sample variables are X1 ; . . .; XN , when X1 ; . . .; XN obey the normal distribution, then the sample over-peak GðXi Þ multiplied by a certain coefficient obeys the standard normal distribution, expressed as follows: pffiffiffiffiffiffiffiffiffiffiffi ð26Þ GðXi Þ N=24  Nð0; 1Þ The above equation indicates that there is a certain threshold value Tc , when pffiffiffiffiffiffiffiffiffiffiffi GðXi Þ N=24 is less than the threshold, it obeys the normal distribution, where the threshold Tc is nonnegative. As shown in the above-described distribution characteristic of the detected noises, we estimate the parameters of the VST transform by minimizing the excessive peak of the noise distribution, in order to effectively minimize the excessive peaks, we make conversion of the parameters a and b into a function of the parameters r and /, and the expression is as follows: a ¼ r2 cos / b ¼ r2 sin /

ð27Þ

Where the parameter r is a nonnegative number, the value range of parameter / is ½0; p=2Š. It can be derived that the following formula holds: cos / sin / 0 ¼ 0 cos / sin / , sinð/ ,/¼/

0

0

/Þ¼0

ð28Þ

Taken into the VST transformation formula, the following expression can be obtained:

66

L. Ding et al.

f ðt; a; bÞ ¼ f ðt; r2 cos /; r2 sin /Þ ffi 2r pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi tr2 cos / þ r2 sin / ¼ 2 r cos / 2 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ t cos / þ sin / cos /

ð29Þ

From the above equation we can see that the VST transform is only related to the 0 0 parameter /, therefore, the estimated value a and b of the noise model parameters is finally obtained by repeated iterations and comparing the calculated excess peak to the given threshold, and the noise variance r2i ði ¼ 0; . . .; 255Þ of each gray value is obtained according to the noise model, with the expression as follows: 0

0

r2i ¼ a i þ b i ¼ 0; . . .; 255

ð30Þ

The final average noise variance is obtained according to the above equation to be the noise variance yield of the noisy image, with the expression as follows: r2avg ¼

255 1 X r2 256 i¼0 i

ð31Þ

Therefore, this formula can be used to accurately estimate the Gaussian and mixed noises, which is not affected by the video image texture and more stable than PCA, making video denoising have a greater application range.

4 Test Results To verify the effectiveness of the noise estimation algorithm in this paper, four groups of videos are selected, akiyo, foreman, salesman and football respectively [12]. Add the Gaussian noise or Poisson-Gaussian noise into the video images, and compare the estimated noise level with true values and other noise estimation algorithms. The noise ^ rj, which is the absolute value of the difference estimate error is defined as dðrÞ ¼ jr between the estimated error and the true error, and the error is estimated by each algorithm to measure the accuracy of the noise estimation. Respectively remove the 3rd frame in the test videos for comparison. The 3rd frame images the four sequences of non-noise videos are as follows (Fig. 1):

Fig. 1. The 3rd of the original video frames

Image Noise Estimation Based on Principal Component Analysis

67

(1) The error comparison results of adding the Gaussian noises 10, 20, 30 are shown in the following Table 1: Table 1. Gaussian noise estimation error comparison r Algorithm Akyio Foreman 10 PCA 0.1600 0.0039 0.0951 0.0364 Literature [6] Literature [11] 0.1794 0.1926 Algorithm in this paper 0.0568 0.3246 20 PCA 0.2643 0.2683 0.3327 0.1216 Literature [6] 0.3028 0.2613 Literature [11] Algorithm in this paper 0.0459 0.0889 30 PCA 0.8385 0.0178 0.7271 0.2519 Literature [6] Literature [11] 0.6724 0.3651 Algorithm in this paper 0.6116 0.3393

Salesman 0.0100 0.1129 0.1435 0.0977 0.1719 0.3731 0.2813 0.0511 0.1732 0.3681 0.3317 0.1454

Football 0.1375 0.3632 0.3067 0.0347 0.0188 0.1832 0.2725 0.0567 0.0985 0.2106 0.4021 0.4435

From the above table we can see that in the case of simply adding Gaussian noises, the noise variance estimated by the noise estimation algorithm in this paper has a very small difference with the true noise variance, and it is more accurate than the noise level estimated by the compared algorithms in most cases. (2) The error comparison results for Poisson-Gaussian mixed noises of adding parameters a=b are shown in the following Table 2: Table 2. Comparison of Gaussian-Poisson noises estimation errors a=b=ravg

Algorithm

5/5/25.739

PCA Literature [6] Literature [11] Algorithm in this 5/10/27.156 PCA Literature [6] Literature [11] Algorithm in this 10/5/36.055 PCA Literature [6] Literature [11] Algorithm in this 10/10/37.080 PCA Literature [6] Literature [11] Algorithm in this

paper

paper

paper

paper

Akyio

Foreman Salesman Football

15.0540 4.1080 0.1734 0.1410 13.3487 4.0269 0.3637 0.1943 21.5333 5.8594 0.3549 0.2642 21.0025 6.2379 1.2357 0.4205

13.1872 8.7416 7.4630 1.2097 0.1956 0.6771 0.1689 0.3765 11.2435 7.7648 7.0491 1.1198 0.6948 0.2256 0.5455 0.1068 20.3788 14.8848 10.5698 1.2330 0.4683 0.9862 0.1518 0.6440 19.0014 13.3717 10.0850 1.2147 1.3138 0.7263 0.6124 0.2253

10.6535 1.5402 0.6942 0.4434 10.5201 0.9475 0.4736 0.3505 16.4083 1.8694 1.2574 0.5304 15.7649 1.8665 0.8588 0.5009

68

L. Ding et al.

The comparison in the above table can clearly tell that when adding GaussianPoisson mixed noise, the average variance of the noise estimation algorithm in this paper is very close to the average variance of the real noise, and more accurate than the compared algorithms, and has a higher accuracy especially than the PCA noise estimation algorithm. From the above comparison results, we can see that the noise estimation algorithm in this paper can achieve a more accurate noise level, which could not only make a more accurate estimation on Gaussian noise, but obtain accurate results for the Gaussian-Poisson mixed noises. Therefore, the noise estimation algorithm in this paper has a good applicability.

5 Conclusions This paper has improved the traditional noise estimation algorithms, proposed the mixed noise estimation algorithm combining PCA and variance stabilization transform, and meanwhile introduces the concept of excessive peak to further determine the accuracy of the mixed noise estimation parameters, and the parameters of the VST transform are estimated by minimizing the excessive peak of the noise distribution. In addition, the noise estimation algorithm is combined with the classical video denoising algorithm to achieve a better video denoising effect and bring video denoising a wider range of applications.

References 1. Wen, Y.-W., Ng, M.K., Huang, Y.-M.: Efficient total variation minimization methods for color image restoration. IEEE Trans. Image Process. 17(11), 2081–2088 (2008) 2. Tang, Q.X., Jiao, L.C.: Image denoising with geometrical thresholds. Electron. Lett. 45(8), 405–406 (2009) 3. Portilla, J., Strela, V., Wainwright, M.J.: Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Trans. Image Process. 12(11), 1338–1351 (2003) 4. Liu, X., Tanaka, M., Okutomi, M.: Noise level estimation using weak textured patches of a single noisy image. In: 2012 19th IEEE International Conference on Image Processing (ICIP), pp. 665–668 (2012) 5. Pei, Z., Tong, Q., Wang, L., et al.: A median filter method for image noise variance estimation. In: 2010 Second International Conference on Information Technology and Computer Science (ITCS), pp. 13–16. IEEE (2010) 6. Ponomarenko, N.N., Lukin, V.V., Zriakhov, M.S., et al.: An automatic approach to lossy compression of AVIRIS images. In: IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2007, pp. 472–475. IEEE (2007) 7. Li, T.-Y., Wang, M.-H., Chang, H.-W., Chen, S.-Q.: An entropy-based estimation of noise variance in wavelet domain. J. Beijing Univ. Posts Telecommun. 05, 1–5 (2011) 8. Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(3), 613–627 (1995) 9. Zhu, X., Milanfar, P.: Automatic parameter selection for denoising algorithms using a no-reference measure of image content. IEEE Trans. Image Process. 19(12), 3116–3132 (2010)

Image Noise Estimation Based on Principal Component Analysis

69

10. Pyatykh, S., Hesser, J., Zheng, L.: Image noise level estimation by principal component analysis. IEEE Trans. Image Process. 22(2), 687–699 (2013) 11. Liu, W., Lin, W.: Gaussian noise level estimation in SVD domain for images. In: 2012 IEEE International Conference on Multimedia and Expo (ICME), pp. 830–835. IEEE (2012) 12. Xiao, J.-S., Li, W.-H., Jiang, H., Peng, H., Zhu, S.-T.: Three dimensional block-matching video denoising algorithm based on dual-domain filtering. J. Commun. 09, 91–97 (2015)

Efficient High Dynamic Range Video Using Multi-exposure CNN Flow Yuchen Guo1, Zhifeng Xie1,3 ✉ , Wenjun Zhang1, and Lizhuang Ma2,3 (

1

)

Department of Film and Television Engineering, Shanghai University, Shanghai 200072, China [email protected] 2 Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China 3 Shanghai Engineering Research Center of Motion Picture Special Effects, Shanghai 200072, China

Abstract. High dynamic range (HDR) imaging has seen a lot of progress in recent years, while an efficient way to capture and generate HDR video is still in need. In this paper, we present a method to generate HDR video from frame sequence of alternating exposures in a fast and concise fashion. It takes advantage of the recent advancement in deep learning to achieve superior efficiency compared to other state-of-art method. By training an end-to-end CNN model to estimate optical flow between frames of different exposures, we are able to achieve dense image registration of them. Using this as a base, we develop an efficient method to reconstruct the aligned LDR frames with different exposure and then merge them to produce the corresponding HDR frame. Our approach shows good performance and time efficiency while still maintain a relatively concise framework. Keywords: High dynamic range video · Convolutional neural networks Optical flow

1

Introduction

Due to the limit of sensors used by most image capturing device currently on trade, they lack the capability to capture the wide range of luminance in real world as human eyes can perceive. Thus high dynamic range (HDR) imaging techniques are developed to address this problem. While methods to capture still HDR images have been extensively researched, HDR video is still a comparably less popular subject. Large portion of HDR video applications up to date have been focused on specialized HDR camera systems [1–4]. These custom hardwares are often either expensive or inconvenient to use, making them hard to be ported for practical use or common consumer market. On the other hand, it’s already a common function of digital cameras to capture still HDR image. Utilizing camera’s exposure bracketing function, we can take several LDR pictures of same scene with different exposures and merge them to recover larger dynamic range than that of sensors, thus obtaining a HDR image [5, 6].

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 70–81, 2017. https://doi.org/10.1007/978-3-319-71598-8_7

Efficient High Dynamic Range Video Using Multi-exposure CNN Flow

71

Similarly, we can also use off-the-shelf cameras to capture a LDR video sequence with alternating exposures. And the aim of HDR video methods is to reconstruct the missing LDR frame of different exposure for each frame in the sequence so they can be merged into a HDR sequence. A sample of the process with our method is shown in Fig. 1. The reconstructed LDR frame should be well-aligned to the original frame of different exposure and temporally coherent to other frames of same exposure, otherwise there will be artifacts like ghosting or jittering in the results. Therefore the process often requires accurate image registration between frames of different exposure due to motions in the sequence. The problems of multi-exposures image registration poses the main challenge for most HDR video application, as traditional motion estimation methods like optical flow often fail in such scenario.

Fig. 1. Sample of HDR reconstruction process. (Top Row): Input sequence of two alternating exposures generated from ‘showgirl’ sequence of HdM-HDR-2014 dataset [18]. (Bottom Row): HDR sequence (tone-mapped) reconstructed with our method.

On the other hand, recently convolutional neural networks (CNN) have become quite popular in the fields of computer vision after achieve state-of-art performance in prob‐ lems like object detection, classification, segmentation, etc. Inspired by the research of FlowNet [7] that uses CNN in optical flow estimation, we propose to train an end-toend CNN model that can handle motion estimation under illumination change using custom-built synthetic dataset. In this paper, we present a new method to reconstruct HDR video from sequence of alternating exposures using the trained CNN model for motion estimation across different exposure. Leveraging the CNN model’s good estimation performance and fast speed, we are able to obtain dense registration between frames of different exposures. A fine registration combined with our occlusion fixing and refinement process, we can achieve good reconstruction results in an efficient way while maintain a relatively simple framework. In summary, our paper intends to present two main contribution: (1) an end-to-end CNN model trained on custom dataset that can handle multi-exposure motion estima‐ tion; (2) an efficient and concise approach to reconstruct HDR video from sequence of alternating exposures by utilizing the above CNN model. We will demonstrate our method and results more specifically in the rest of the paper.

72

2

Y. Guo et al.

Related Work

HDR imaging is a frequently studied subject in this field while there are only a few that are developed specifically for HDR video. And as we mentioned above, a lot of these application are based on custom hardwares like special sensors [1, 2] or devices that register two cameras to capture one scene with different exposures simultaneously [3, 4]. For brevity, in this section we only discuss methods that reconstruct HDR video from LDR sequence of alternating exposures captured by single conventional camera. Kang et al. [8] propose first practical HDR video approach using sequences of alter‐ nating exposures as input. It is a optical flow based method that unidirectionally warp the previous/next frames towards target frame using a variant of the Lucas and Kanade [9] technique in a Laplacian pyramid framework. As for over/under-exposed regions where the optical flow estimation is unreliable, they bidirectionally interpolate the previous/next frames using optical flow between them and further refine the alignment using a hierarchical homography-based registration. Mangiat and Gibson [10] instead choose a block-based motion estimation method in order to overcome the problems of gradient-based method Kang et al. used. They also present a refinement stage that use filtering methods to remove artifacts of mis-registration or block boundary. However, these methods still suffer from the accuracy of motion estimation between multi-expo‐ sures frames and often fail when non-rigid or fast motion is present. The more recent research of Kalantari et al. [11] probably represents the state-of-art result of HDR video reconstruction. They propose a patch-based HDR synthesis method that combines optical flow with a patch-based synthesis approach similar to Sen et al. [12]. Their method enhance temporal coherency using patch-based synthesis and enforce constraints from optical flow estimation to guide patch-based synthesis with a search window map. In this way, they are able to handle more complex motion in the sequence and produce high-quality HDR video output. Although perceptually insignificant, it is still reported that the unstable performance of optical flow estimation may result in artifacts around motion boundaries such as blurring or distorting [13]. Besides, the iter‐ ation and optimization process required by the method result in higher running time and complexity compared to other methods. As the main challenge for HDR video reconstruction is finding correspondences between frames with different exposures, the performance of reconstruction can benefit a lot from an improvement in motion estimation method like optical flow. One of the reasons most variation-based optical flow methods fail when dealing multi-exposures data is the brightness constancy assumption they hold, which was introduced in classical optical flow literature by Horn and Schunck [14]. There were also many attempts to gain robustness against illumination change. Brox et al. [15] added a gradient constancy assumption to the original variational optical flow framework. Mileva et al. [16] tried to make use of photometric invariants in computing an illumination-robust optical flow. Still, the challenge posed by registering frames of different exposure may combine dramatic illumination change, large displacement motion and loss of information due to saturation. It is difficult to design a framework that handles all these issue. Meanwhile, deep learning techniques, especially CNN, have demonstrated remark‐ able performance in many computer vision tasks. It is shown to be able to extract features

Efficient High Dynamic Range Video Using Multi-exposure CNN Flow

73

that otherwise hard to represent in normal ways by learning from large training datasets. Recently, Dosovitskiy et al. [7] first constructed an end-to-end CNN which are capable of solving the optical flow estimation problem as a supervised learning task. However, there is still no learning-based methods developed to overcame the incapability of most motion estimation methods that they can’t deal with multi-exposures data.

3

Multi-exposure Optical Flow Based on CNN

Inspired by previous works about CNN-based optical flow, we construct an end-to-end CNN with three main components to predict dense motion vector field between images with different exposures. Besides, in order to supply the networks with sufficient training data to learn from, we build a custom dataset from available flow datasets for multiexposures motion estimation. 3.1 Network Structure As shown in Fig. 2, our end-to-end model consists of three main components: low-level feature network, fusion feature network, and motion estimation network.

Fig. 2. Our end-to-end CNN consist of three main components: low-level feature network, fusion feature network, and motion estimation network. Given enough training data of multi-exposure image pairs and ground truth flows, our end-to-end model can be trained to predict dense optical flow fields accurately from input images with different exposures.

The low-level feature network contains three convolution layers for each input image. It constructs two separated processing streams for them, which can effectively promote the feature representation and the deep training in the different exposures. While low-level feature network only focuses on the respective features of the input images rather than their correspondences, we introduce the correlation layer of FlowNet [7] to perform the matching and fusion of two low-level features, and construct a fusion feature network to finally obtain the representation of multi-exposure motion features.

74

Y. Guo et al.

Taking the outputs of the correlation layer from low-level feature network as input, the fusion feature network itself consists of the correlation layer and the convolution layers, which can efficiently handle the matching process between two groups of low-level features and obtain the motion features in the different exposures. With the entire contracting part completed, we then introduce the motion estimation network in the expanding part, which uses upconvolution layers including unpooling and convolution. It contains seven combination layers, which not only include upcon‐ volution layers but also integrate the outputs of the low-level feature network and the fusion feature network respectively. Each of the combination layers can predict a corre‐ sponding coarse flow with 2 outputs, and then upsample the flow as the input of its next layer. In a word, the various features are fused in the motion estimation network, and they are effectively processed by a set of upconvolution and upsampling operations. 3.2 Training Data In order to effectively train a large-scale CNN, sufficient training data is needed. Besides, neural networks require data with ground truth to learn to perform a prediction task from scratch. These requirements make it difficult to prepare training data for our multiexposure application as it’s quite hard to capture ground truth motion flows from real world scenes. While there are several public optical flow datasets that contain ground truth flow, most of them are generated from synthetic scenes and, more importantly, maintain the same exposure setting. Therefore we choose to build a custom multi-exposure optical flow dataset using available datasets as bases. First we choose the public datasets to build on. There are three state-of-art candidates: the Middlebury dataset, the Kitti dataset and MPI-Sintel dataset. The Middlebury dataset is widely used for optical flow evaluation. But it only contains 8 synthetic image pairs with small displacement motions, and thus is too small for learning. The Kitti dataset is a real world scenes dataset captured by automobile platform. Its complexity in lighting and texture makes it a challenging benchmark. Yet due to the limit of capturing device, its ground truth flows are sparse, which makes it unsuitable for our need. MPI-Sintel dataset consists of sequences from an animation movie which include various motion types and scenes. All things considered, we choose the ‘final’ version of MPI-Sintel dataset with realistic rendering effects such as motion blur and atmospheric effects to get closer to the more complex real world scenes. After that we need to generate multi-exposure data from the selected dataset. As exposure value (EV) of camera is a number that represents the combination of shutter speed and f-number, with a difference of 1 EV corresponding to a standard power-of-2 exposure step. We utilize gamma correction to synthesize the multi-exposure effect. By increase one frame’s exposure while decrease another’s, the process create image pairs with drastic brightness change similar to that of exposure difference while maintain same ground truth motion. By comparing results of our post processing with real image with different exposures, it can be observed that our simple simulation can effectively reflect the change between different exposures though not perfectly accurate.

Efficient High Dynamic Range Video Using Multi-exposure CNN Flow

75

Using the new multi-exposure dataset, we trained our networks on a computer with a CPU of Intel Xeon E5-2620, 16 GB memory and an NVIDIA Tesla K20 GPU. The resulting model converged well and demonstrated good performance on the task of multi-exposure motion estimation, which effectively support our HDR reconstruction application.

4

HDR Video Reconstruction

As mentioned above, the raw data input used for HDR video reconstruction is a LDR video captured with conventional camera that alternates between different exposures for each frame. We will take a two-exposure sequence as example here. The goal of our method is to reconstruct the missing LDR frame of different exposure for each of the frame in the sequence. As the each frame has different exposure from its neighboring frames, the reconstruction process requires drawing information from its next/previous frame which may be of the same exposure. And that’s where an accurate pixel correspondences or motion estimation come into play. Figure 3 shows a brief structure of our method’s process. For certain frame Fn in the alternating exposure sequence, we try to reconstruct the missing LDR image L with a different exposure, shown with dashed red square. Other HDR video methods often use optical flow result as a rough estimation or initiation for the registration of correspond‐ ences between frames with different exposures. While by taking advantage of our trained CNN model, we can directly estimate a good motion field as optical flow between Fn and its neighboring frame Fn−1 ∕Fn+1, which are different in exposure. The improvement

Fig. 3. Reconstruction process of frame n from a sequence alternating between two different exposure levels, which only capture certain exposure at each frame (shown with solid black squares). Our method reconstructs the missing exposure for the current frame Fn using a warp and refine scheme based on optical flow f (shown with solid blue circles) between current frame and its neighboring frame, computed by our CNN model for multi-exposure motion estimation. Once the missing LDR image has been reconstructed, it can be merged together with current frame to produce the HDR frame, which will then form the entire HDR video. (Color figure online)

76

Y. Guo et al.

in quality of motion estimation between frames of different exposure enables us to utilize a more concise and straightforward scheme in the reconstruction of missing LDR frame L. Moreover, in this way we don’t need to linearize the image and boost its intensity for better registration like many other methods require, which may involve camera response function (CRF) estimation and therefore limit the application. To actually reconstruct L with the motion estimation results, we generate two inter‐ mediate results by warping the previous/next frame Fn−1 ∕Fn+1 towards current frame Fn to obtain two warped frames Wn−1 ∕Wn+1. However, we can’t directly generate target frame L from the two warped frames, even though the good motion estimation result may yield high quality warped results. Due to occlusion, large-displacement motion or small amount of unreliable flow, it is usually necessary to further refine the results. Therefore we introduce a refinement process to obtain the final reconstructed L with higher quality. The refinement process uses two main constraints to ensure a satisfactory result. They can be formulated as energy functions below: ( ) E = Ec Fn , L + Et (Fn , Fn−1 , Fn+1 )

(1)

In Eq. (1), first term Ec represent the consistency between Fn and L, as they are supposed to be the same frame with different exposures. To measure the consistency in content or structure between two images with different exposures, we employ two metrics. As the two images are supposed to contain the same content and geometry, we assume that there are similar details or gradient where the two images are both well exposed. Besides, to further utilize the performance of our multi-exposure CNN model, we estimate optical flow between the two frames with the model, which can be used to ensure there are no motion between them where the flow are reliable. These two constraints enforce the consistency the original and reconstructed frames and thus help to avoid the ghosting artifacts in HDR merge process. Their formula is shown in the function below: ( ) ( ) Ec Fn , L = α ∗ d ∇Fn , ∇L + � ∗ m(Fn , L)

(2)

where α resemble the approximation map of how well a pixel is exposed in both image and d(x, y) is L2 distance. While � measures how reliable a motion vector in the optical flow map is, and m(a, b) is the motion distance of each pixel between the two image. Second term Et in Eq. (1) maintain the time coherence between reconstructed frame and its previous/next frame with the same exposure. Our refinement procedure approaches this with two main operations. On one hand, we enhance the smoothness of optical flow by comparing all flow fields between the three frames and also the warped results in a bidirectional way to verify the motion’s reliability and continuity, which helps to avoid video jittering caused by erroneous motion. On the other hand, sometimes due to large-displacement motion, there are noticeable region of occlusion present which would cause ghosting from previous/next frames in the warped images. To deal with occlusion, we first extract regions of occlusion by comparing motion vectors’ origin and destination of flow from neighboring frame to current frame, the difference of which

Efficient High Dynamic Range Video Using Multi-exposure CNN Flow

77

can be used to extract the occlusion map. Then we fix the region of occlusion in one warped image by drawing information from the other warped image which contains content from another neighboring frame with different occlusion area. The process of handling occlusion is shown in the above Fig. 4. In summary, these operations enforce the constraint of time coherence between reconstructed frame and its neighboring frames with the same exposure, which can be formulated as the function below:

( ) ∑ Et Fn , Fn−1 , Fn+1 =

i∈pixels

( ( i i+u ) ( )) i+u d Fn , Fn−1 + d Fni , Fn+1

(3)

where i is a pixel location in Fn, while u represent the motion displacement at i between Fn and its neighboring frame Fn−1 and Fn+1. This ensures similarity and coherence between frames and thus solves the jittering artifacts.

Fig. 4. Example of occlusion fixing process. (Left Column): current reference frame (Left-Top) and its next frame (Left-Bottom). (Middle-Top): optical flow estimated by our CNN model from current frame to its next frame. (Middle-Bottom): directly reverse warped result using the flow, which shows ghosting at regions of occlusion. (Right-Top): occlusion map extracted from optical flow map. (Right-Bottom): warped LDR result after our occlusion fixing process.

Finally, after the refinement process we combine the two refined warped images to obtain the reconstructed LDR frame of different exposure at current frame time as result. With that, we merge them to achieve the HDR frame and tone-map it for display. Besides, the reconstructed LDR frame can also help to refine the reconstruction process of its neighboring frame.

5

Results and Discussion

We demonstrate and analyze some results of our HDR reconstruction method in this section. All results displayed here are fused and tone-mapped using the exposure fusion method by Raman et al. [17]. In order to obtain sequences with alternating exposures as input data for our method, we make use of the high-quality HDR video sequences dataset by Fröhlich et al. [18]. These sequences are captured using two cameras mounted on a mirror-rig and contained

78

Y. Guo et al.

various scenes with different challenges such as complicate illumination setting, high contrast skin tones and saturated colors, etc. By extracting multiple exposures from original HDR data, synthetic sequences of alternating exposures can be acquired in this

Fig. 5. Examples of test sequences and results. For each group, (Top Row): input triplet of consecutive frames of two alternating exposures, with middle one as current reference frame; (Bottom Row): reconstructed LDR result of (Bottom-Left) Kalantari et al. [11] (without corresponding CRF) and (Bottom-Middle) ours; (Bottom-Right): our HDR result (tone-mapped). (Color figure online)

Efficient High Dynamic Range Video Using Multi-exposure CNN Flow

79

way. Moreover, available ground truth data also offer a better evaluation and comparison for the performance of our method. We test our method using different dynamic scenes from the HdM-HDR-2014 dataset [18], which are extracted into sequences with two alternating exposures of −2EV and +1EV at resolution of 1920*1080 as input. The three scenes in Fig. 5 were chosen to be displayed here due to the unique and representative features they demonstrate. As shown in Fig. 5, the first scene ‘carousel fireworks’ is filmed at an annual fair where color-saturated highlight and fast moving, self-illuminated objects are present the dark nighttime surroundings. While second scene ‘bistro’ features a dark bistro-chamber combined with local bright sunlight at the window, creating a high-contrast scene with difficult lighting situation. And the third scene ‘showgirl’ shows partially illuminated skin and specular highlights on various reflecting props together in a glamorous tone. These scenes can demonstrate the performance of our method when faced with different challenges. For each frame to be processed, it is combined with its neighboring frames with different exposures to form a triplet of consecutive frames as input, producing the reconstructed LDR frame of missing exposure as output, which is then merged with original frame into the final HDR frame. For brevity, in Fig. 5 we take single triplet from each sequence as example and display the reconstructed results. Besides, we also run these test cases with the method of Kalantari et al. [11], which is considered one of the state-of-art methods of HDR video reconstruction in regards of reconstruction quality while using conventional camera. Yet it’s shown in Fig. 5 that their method fail to reconstruct the correct missing LDR image due to the lack of corre‐ sponding camera response function (CRF) for our test data. Though this doesn’t affect the good performance of their method when CRF are provided, the comparison demon‐ strates our method’s robustness and wider applicability. In order to achieve a better evaluation for our method, we compare our reconstructed frame results with the ground truth data generated from original HDR sequence. Using PSNR as main metric, the evaluation results and running of our method for each test sequence are listed in Table 1. From it we can see that our method shows good and stable performance in HDR reconstruction quality as well as high processing speed, much faster than that of Kalantari et al. [11] which may require nearly 10 min to run. It should also be noted that the operation of motion estimation with our multi-exposure CNN model only takes only about one second to run, which implies there is still much room for improvement in time efficiency given better optimization and implementation in refinement stage. Table 1. Evaluation results Testing sequence Carousel fireworks Bistro Showgirl Average (over all scenes)

PSNR/dB (Average) 38.83 42.19 41.56 32.16

Running time/s (Per frame) 103.5 99.7 84.3 98.4

80

Y. Guo et al.

Nevertheless, several limitations are still observed during experiments. When current reference image present large region of glare/saturation due to high exposure time or motion blur caused by fast movement, the optical flow result of our motion estimation may not be accurate because lack of coherence in content between frames, which then leads to a decrease in performance. In addition, sometimes there will be regions of occlusion in current frame that are not present in neighboring frames, causing the algorithm unable to draw information from adjacent frames by using motion as cue, which may require other matching method to fix. Moreover, we also observed that the performance of current CNN model is somehow sensitive to image scale and motion type possibly due to the training data we provided. To address these problems, our future work will focus on trying different CNN structure design and training scheme in order to solve the current limitations in a more unified framework. And other plans include making better use of similarity between frames in same sequence for achieving better time efficiency.

6

Conclusion

In this paper, we present a new method for HDR video reconstruction from sequence of alternating exposures, which utilize a CNN model with capability of motion estimation across multiple exposures. By training a CNN end-to-end to learn predicting optical flow from image pairs with different exposures, we manage to overcome the problems of image registration between different exposures where many other motion estimation methods failed, and thus use a more concise framework for HDR video reconstruction. With effective refinement process, the results of our method demonstrate competitive performance in both reconstruction quality and efficiency. It also shows the potentials of further application of CNN in the field of HDR synthesis. Acknowledgements. This work was supported by the National Natural Science Foundation of China (No. 61303093, 61402278, 61472245), the Innovation Program of the Science and Technology Commission of Shanghai Municipality (No. 16511101300), and the Gaofeng Film Discipline Grant of Shanghai Municipal Education Commission.

References 1. Nayar, S., Branzoi, V.: Adaptive dynamic range imaging: optical control of pixel exposures over space and time. In: 9th IEEE International Conference on Computer Vision (ICCV 2003), vol. 2, pp. 1168–1175. IEEE, Nice (2003) 2. Unger, J., Gustavson, S.: High-dynamic-range video for photometric measurement of illumination. In: Proceedings of the SPIE, vol. 6501 (2007) 3. Tocci, M.D., Kiser, C., Tocci, N., Sen, P.: A versatile HDR video production system. ACM Trans. Graph. 30(4), 41 (2011) 4. Kronander, J., Gustavson, S., Bonnet, G., Unger, J.: Unified HDR reconstruction from raw CFA data. In: IEEE International Conference on Computational Photography (ICCP 2013), pp. 1–9. IEEE, Cambridge (2013)

Efficient High Dynamic Range Video Using Multi-exposure CNN Flow

81

5. Mann, S., Picard, R.W.: On being ‘undigital’ with digital cameras: extending dynamic range by combining differently exposed pictures. In: IS&T’s 48th Annual Conference, pp. 422– 428. Society for Imaging Science and Technology, Washington, D. C. (1995) 6. Debevec, P., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: Proceedings of SIGGRAPH 1997, pp. 369–378. ACM SIGGRAPH, Los Angeles (1997) 7. Dosovitskiy, A., Fischery, P., Ilg, E., et al.: Flownet: learning optical flow with convolutional networks. In: 2015 IEEE International Conference on Computer Vision (ICCV 2015), pp. 2758–2766. IEEE, Santiago (2015) 8. Kang, S.B., Uytendaele, M., Winder, S., Szeliski, R.: High dynamic range video. ACM Trans. Graph. 22(3), 319–325 (2003) 9. Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI 1981), vol. 2, pp. 674–679. IJCAI, Vancouver (1981) 10. Mangiat, S., Gibson, J.: Spatially adaptive filtering for registration artifact removal in HDR video. In: 18th IEEE International Conference on Image Processing (ICIP 2011), pp. 1317– 1320. IEEE, Brussels (2011) 11. Kalantari, N.K., Shechtman, E., Barnes, C., Darabi, S., Goldman, D.B., Sen, P.: Patch-based high dynamic range video. ACM Trans. Graph. 32(6), 202 (2013) 12. Sen, P., Kalantari, N.K., Yaesoubi, M., Darabi, S., Goldman, D.B., Shecheman, E.: Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. 31(6), 203 (2012) 13. Tursun, O.T., Akyüz, A.O., Erdem, A., Erdem, E.: The state of the art in HDR deghosting: a survey and evaluation. Comput. Graph. Forum 34(2), 683–707 (2015) 14. Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981) 15. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3 16. Mileva, Y., Bruhn, A., Weickert, J.: Illumination-robust variational optical flow with photometric invariants. In: Hamprecht, Fred A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 152–162. Springer, Heidelberg (2007). https://doi.org/ 10.1007/978-3-540-74936-3_16 17. Raman, S., Chaudhuri, S.: Bilateral filter based compositing for variable exposure photography. In Proceedings of Eurographics 2009. European Association for Computer Graphics, Munich (2009) 18. Froehlich, J., Grandinetti, S., et al.: Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. In: Proceedings of the SPIE, vol. 9023 (2014)

Depth Image Acquisition Method in Virtual Interaction of VR Yacht Simulator Qin Zhang ✉ and Yong Yin (

)

Dalian Maritime University, Dalian, China [email protected]

Abstract. The paper investigates depth image acquisition to realize virtual natural interaction of the VR yacht simulator. Because the existing image denoising algorithms have the disadvantages of poor robustness and unstable edge of image, an improved bilateral filtering algorithm is proposed. Firstly, the algo‐ rithm obtains the depth information in the scene to generate depth image by using Kinect sensor. Secondly, the background removal is applied to keep the hand data from the depth image. Finally, median filtering and bilateral filtering are used to smooth the depth image. Experimental results show that the proposed method can obtain better depth images after removing background, thus the algorithm has good robustness. Keywords: Kinect · Depth image · Background removal · Median filter Bilateral filter

1

Introduction

At present, yacht industry is developing rapidly in China. The ability training and valu‐ ation of yacht operators are a key part of yacht industry economy chain. If part of the yacht’s actual operation training is completed in the full task yacht simulator, it will significantly improve training efficiency, save training costs and avoid the risk of training. With the development of virtual reality technology, it has the advantages of much cheaper, easier network and smaller occupied space as a complement to full mission simulator. So it will gradually occupy a certain share of market. Virtual natural interaction technology is the key technology of yacht simulator based on VR. The key point of realizing virtual natural interaction is to acquire and process 3D scene information. In computer vision system, 3D scene information is possible for computer vision applications such as image segmentation, object detection and object tracking. However, the restoration of scene depth information has the disadvantage of poor robustness. In this paper, an improved depth image de-noising method is proposed to improve the robustness of depth image de-noising algorithm.

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 82–92, 2017. https://doi.org/10.1007/978-3-319-71598-8_8

Depth Image Acquisition Method in Virtual Interaction

2

83

Related Work

With the increase of demand, the 2D images with similar color of foreground and back‐ ground cannot solve the problem in object tracking and pattern recognition. Therefore, the importance of depth images is represented. In 2010, Microsoft’s Kinect sensor, due to its good depth image features and appropriate price, attracted the attention of intelli‐ gent surveillance, medical diagnosis, human-computer interaction and other fields [1]. The research which uses the depth image to detect human body and removes the background is usually based on the method of human body detection commonly used in ordinary image, mainly based on the two methods that human body feature matching and background subtracting. This method based on human feature matching is mainly to extract features that describe the profile information of human body to detect human body in the image. Methods Spinello [2] described that based on statistical learning methods, the color image gradient direction histogram features human detection algorithm is proposed. And it has obtained good results in the measurement range of the whole sensor. Lu, Xia et al. [3] established the 3D head surface and 2D contour model, and then matched the depth image with the model to detect the human body. The disadvantages of this algorithm are that the feature classification result is greatly affected by the training samples, and the computational complexity of the feature generation and matching process is rela‐ tively high. The core of the background subtraction method is to set up the background model, and then classify the pixel whether it is the background one or the foreground one. The common algorithms include average background model [4], hybrid Gauss model (GMM) [5], Code-Book [6] and ViBe algorithm [7]. ViBe algorithm is a fast motion detection algorithm proposed by Olivier [7]. The background model is set up with the surrounding pixels for background extraction and object detection. Ye [8] tested the Vibe algorithm with 3 sets of public data sets, and proved that the algorithm has the characteristics of small computation, high processing efficiency and good detection result. In this paper, Vibe algorithm is used to separate the human body from depth image. Because the infrared measurement of scene illumination changes to a certain extent will lead to instability in the depth data and infrared reflectance characteristics of surface material and other reasons, on the edge of the object and the occluded areas are often prone to cavities. It caused a lot of noise. A variety of algorithms have been proposed for smooth image de-noising in depth image. Vijayanagar [9] used Gauss filter to esti‐ mate the hole depth value, which is a filtering method based on time-domain, this method only uses the hole depth information of surrounding pixels while ignoring the gray information. So the de-noising result is not great. Yang et al. [10] estimate the depth of the center through the distribution of the effective depth values. The method is efficient for de-noising and smoothing, but the algorithm has high complexity. In this paper, bilateral filtering and median filtering are combined to smooth noise reduction, and it improves the robustness of the algorithm.

84

3

Q. Zhang and Y. Yin

Depth Image Acquisition

In order to enhance realism of VR yacht simulator interaction, 3D data acquisition and modeling of hand is necessary in real time. The main research work of this paper is to acquire 3D scene information to generate depth image. Then extract human body infor‐ mation from the scene and separate the hand and body information. Finally, process the depth image de-noising and smoothing. 3.1 Depth Image The method of acquiring depth image can be divided into two classes: passive ranging and active ranging method. (1) Passive ranging method Passive ranging method is the most commonly used binocular stereo vision, two images of the same scene obtained by two distant cameras. The stereo matching algorithm finds the two images corresponding to the pixel. And then according to the principle of triangulation calculates the parallax information, and parallax information can be converted to display the depth information of the object in the scene. Based on the stereo matching algorithm, the depth image of the scene can be obtained by shooting a set of images from different angles in the same scene. In addition, the depth information of the scene can also be estimated indirectly through the analysis of the image’s photometric characteristics, shading features and so on. Although the disparity map obtained by stereo matching algorithm can obtain roughly 3D information of the scene, there is a large error in the parallax of some pixels. The method of obtaining parallax images in binocular stereo vision is limited by the length of the baseline and the matching accuracy of the pixels between the left and right images. The range and accuracy of the parallax images are limited. (2) Active ranging method The active ranging sensor device needs to emit energy to complete the collection of depth information. This ensures that the acquisition of depth images is inde‐ pendent of the color image acquisition. The methods of active ranging include TOF (Time of Flight), structured light and laser scanning. Kinect V2 uses TOF active ranging method to obtain depth data. 3.2 Obtain Depth Image with Kinect Sensor The Kinect sensor is a RGB-D sensor which can acquire color data(RGB) and depth of value (depth) at the same time. The depth data obtained by Kinect is valid in the range of 0.5 m–4.5 m, with a resolution of 512 × 424, and each pixel is 16-bit (see Fig. 1). This data represents the distance from the depth (IR) camera to the object, expressed in millimeters (mm).

Depth Image Acquisition Method in Virtual Interaction

85

Fig. 1. Data format of depth image

Since OpenCV can only display 8-bit image data, it is necessary to convert the depth data of 16bit into image data within the 8bit (0–255) range. In depth image, the distance from the depth (IR) camera to the object is converted to the corresponding gray value. The relation between the pixel value and the true distance in a depth image (see Eq. (1)): d = K × tan(

dgray 2842.5

+ 1.1863) − �

(1)

Where d is as actual distance, dgray is grayscale of image, K = 0.1236 m, � = 0.037 m. As shown in red circles of Fig. 2(b), there are two main problems due to the way to get the depth of the distance: one is that the depth of the image pixel values corresponding to a stationary object which will have some changes or even disappear in the sequence of adjacent frame images; the other one is that edges of object in the depth image which are not stable and prone to shake violently. In order to obtain a better quality depth image, both issues need to be further addressed.

Fig. 2. Sampling data from Kinect. (Color figure online)

86

Q. Zhang and Y. Yin

3.3 Fast Background Removal Algorithm Kinect provides a simple segmentation method between human body and background. The human detection algorithm is used to add ID to the detected pixels with the Body‐ IndexFrame data. This method supports the identification of 6 at most. Due to the Body‐ IndexFrame data and the depth data have the same spatial coordinates, we can use Eq. (2) to quickly get the depth data after the removal of the background.

{ d(x, y) =

0, others d(x, y), g(x, y) ∈ {0, 1, 2, 3, 4, 5}

(2)

Where d(x, y) is depth data, g(x, y) is BodyIndexFrame data. However, the method has low detection accuracy, the body data maybe appear to be empty and missing, and human contact or adjacent objects will be false positive as part of the human body (see red circles as in Fig. 3).

Fig. 3. Removal result of background image using Kinect SDK (Color figure online)

In order to solve the above problems, this paper uses ViBe algorithm to separate the background. ViBe algorithm is a kind of sample random clustering method. The algo‐ rithm idea is specific for each pixel storing a sample dataset, and sample value is the pixel’s past value and its neighborhood values. And then each new pixel value will be compared with every pixel in sample dataset to determine whether the point belongs to the background. The core of the algorithm is the background model initialization, pixel classification and model updating. (1) Initialization of background model The initialization of a generic detection algorithm requires a certain length of video sequence, which usually takes a few seconds and greatly affects the real-time perform‐ ance of the detection. The initialization of ViBe model is accomplished only by one frame of images. ViBe model initialization is the process of filling the sample dataset of pixels. But because the spatial and temporal distribution information of pixels cannot be contained in an image, the near pixels have similar temporal and spatial distribution characteristics. The value of the pixel �(x) is at the location x of the image in the given color space, M(x) is the background sample dataset at the location x. M(x) contains n pixel values selected from the neighborhood of the pixel x, i = 1, 2, … … , n, named the background sample value:

Depth Image Acquisition Method in Virtual Interaction

{ } M(x) = �1 , �2 , … … , �n

87

(3)

(2) Pixel classification The pixel x in the current image is classified by comparing the pixel value �(x) with the corresponding model M(x) in the background model. The distance between the new pixel value and the value of the sample dataset is calculated. If the distance is less than the threshold R, the approximate number of samples is increased. If the approximate sample number is greater than the threshold N , the new pixel is considered as the back‐ ground, as shown in Eq. (4). { } SR (�(x)) ∩ {�1 , �2 , … … , �n } ≥ min

(4)

Where SR (�(x)) indicate that the pixel x is the center, R is the area radius, N = 20, min = 2. (3) Model updating Model updating allows the background model to adapt to ever-changing in the back‐ ground, such as changes in illumination, background changes and so on. The updating method adopted in ViBe algorithm is foreground spot counting method and random sub sampling. Foreground spot counting: the pixels are counted, and if a pixel is continuously detected as foreground times, it is updated to the background point. Random sub sampling: for each new image frame, it is not necessary to update the sample value of each pixel. when a pixel is classified as background, it has 1∕ � probability to update the background model. ( −In

P(t, t + dt) = e

N N−1

) dt

(5)

When a sample value to be replaced is selected, a sample value is updated randomly so that the life cycle of the sample value in the background model is exponentially monotonically decreasing. Since it is a random update, the probability (N − 1)∕ N that such a sample value will not be updated at t time. If the time is continuous, then the probability that the sample value remains after the dt time passes is as Eq. (5). The Eq. (5) shows that whether a sample value is replaced in the model is uncon‐ cerned with time, and the random strategy is appropriate. This is the result of fast removing background by using the ViBe algorithm (see Fig. 4). It can be seen that in different positions, the algorithm can effectively remove the noise of the background, access to the depth of the human body data.

88

Q. Zhang and Y. Yin

Fig. 4. Background removal result using ViBe algorithm

In the virtual interaction of a VR yacht simulator, the hand is always in front of the body. Thus, the depth value of the hand is always smaller than the depth value of the body in the depth image. According to this characteristic, the position of the shot can be calculated from the depth image and displayed selectively:

{ dst(x, y) =

src(x, y), if src(x, y) > threshold 0, other

Using this method to get hand depth data in Fig. 5 shown here:

Fig. 5. Hand depth image using ViBe algorithm and threshold method

(6)

Depth Image Acquisition Method in Virtual Interaction

89

3.4 Improved De-noising Algorithm of Depth Image (1) Bilateral filtering When measuring the scene illumination changes to a certain extent, it will lead to unstable the depth data. And because of the surface material of the infrared reflection characteristics and other reasons, on the edge of the object and the occluded areas are prone to cavities in the depth image which obtained by Kinect sensor. Because the depth values can be considered as continuous in a relatively small field, holes can be filled with valid depth values around the holes. Bilateral filtering is a nonlinear filtering method, which is a compromise between image spatial proximity and pixel similarity. This method takes into account both spatial information and gray simi‐ larity, thus keeps the edge information of the image and achieve the effect of noise reduction. It has the characteristics of simplicity, non-iteration, locality and so on. There is the Bilateral filtering Eq. (7):

(xj − xi )2 −

Wij =

1 e Ki

�s2

(Ij − Ii )2 −

∙e

�r2

(7)

Where W is the weight, i and j are pixel index, K is the normalization constant, �s is spatial standard deviation of Gaussian Function, �r is range for the standard deviation of the Gaussian Function, I is the pixel gray value. According to the characteristics of exponential function, e should be a f (x) mono‐ tonic decreasing function, so the weight will decrease and the filtering effect will be smaller when the gray range is large (such as edge). In general, in the region where the pixel gray level is too mild, bilateral filtering has the effect similar to the Gauss filter. While in the larger gradient of the image edge, the bilateral filter has the effect of reten‐ tion. For digital image, the filtering process is eventually represented as a form of convo‐ lution using a template, which in fact indicates that the output pixel values depend on the weighted combination of the values of the domain pixels (see Eq. (8)).

∑ h(i, j) =

k,l

f (k, l)�(i, j, k, l) ∑

�(i, j, k, l)

(8)

k,l

The weight coefficient �(i, j, k, l) (Eq. (11)) depends on the product of the domain kernel (Eq. (9)) and the range kernel (Eq. (10)). (i − k)2 + (j − l)2 �s2 d(i, j, k, l) = e

(9)

‖f (i, j) − f (k, l)‖2 �r2 r(i, j, k, l) = e

(10)

−

−

90

Q. Zhang and Y. Yin

(i − k)2 + (j − l)2 ‖f (i, j) − f (k, l)‖2 − �s2 �r2 �(i, j, k, l) = e −

(11)

(2) Improved de-noising algorithm of depth image Following a hole repaired in using the bilateral filter, there are still grayscale isolated points which cannot be repaired and cannot be effectively used by the right pixel around the gray value to fill. Therefore, the median filter is used to smooth the depth image. The median filter replaces the value of each pixel with the median pixel value in the neigh‐ borhood. For larger isolated points, median values can be chosen to avoid the effects of these points. There is the Bilateral filtering Eq. (12).

D(i, j) = Med(f (i, j)) ∧K

Fig. 6. Image smoothing result (Color figure online)

(12)

Depth Image Acquisition Method in Virtual Interaction

91

Where ∧K is 3 × 3 filter window, f (i, j) is the center pixel, D(i, j) is the intermediate pixel values for the neighborhood. Through median filtering, gray value deviation and the surrounding pixels will be larger median replacement, and eliminate the noise to achieve the purpose of de-noising the image. Literature [9] Gauss filter de-noising effect is shown in Fig. 6(a), and it demonstrates that the part of the hand in the region is not correct in depth value, the repair effect is not very good as red circle shown in Fig. 6(a). Bilateral filtering results as shown in Fig. 6(b) is that the depth value is not homogeneous, and there are some isolated noises. The smoothing effect of the median filter is shown in Fig. 6(c), the gap between some fingers are disappear. After comparison, in this paper the proposed method of bilateral filtering and median filtering smoothing is better, as shown in Fig. 6(d).

4

Result

The process of the improved depth image smoothing algorithm in this paper is shown in Fig. 7.

Fig. 7. Working process

The development environment of this paper is VC++ 2012 and Kinect for Windows SDK V2.0. The PC configurations used are Inter(R) Core(TM) i5-4590 CPU @ 3.30 GHz quad core processor, 8 GB memory, NVIDIA GeForce GTX 900 graphics card, Xbox ONE Kinect 2 sensor device. The experimental results are shown in Fig. 6(d). In this paper, depth data is obtained by Kinect sensor. Hand depth data is acquired through ViBe algorithm and threshold method. And using bilateral filtering and median filtering apply to de-noise the depth image. Experimental results show that this method can effectively acquire high quality depth images in real-time.

5

Conclusion

In this paper, an improved method of depth image acquisition and processing is proposed in virtual interaction of yacht simulator. A single channel depth image obtained by a Kinect sensor’s infrared camera has a one-to-one correspondence between the gray value and the distance from the camera to the object. Based on this characteristic, we use ViBe algorithm and threshold method to segment the background. After de-noising and smoothing by using bilateral filtering and median filtering, we obtain a clear and

92

Q. Zhang and Y. Yin

seamless high-quality depth image, which lays a solid foundation for the virtual inter‐ active operation of the later VR yacht simulator. In the future, the depth data obtained from this paper can be converted into 3D point cloud data, and a 3D hand model can be created in combination with Kinect RGB images to provide real-time 3D data for the virtual natural interaction of the yacht simulator. Acknowledgment. The authors would like to acknowledge the support from the Fundamental Research Funds for the Central Universities [No. 3132016310] and Traffic Youth Science and Technology Talent Project [No. 36260401].

References 1. Han, J., Shao, L., Xu, D.: Enhanced computer vision with Microsoft Kinect sensor: a review. IEEE Trans. Cinematics 43(5), 1318–1334 (2013) 2. Spinello, L., Arras, K.O.: People detection in RGB-D data. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3838–3843 (2011) 3. Xia L., Chen, C.-C, Aggarwal, J.K.: Human detection using depth information by 3 Kinect. In: 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 15–22 (2011) 4. Zhu, T., Lu, L.B., Jin, G.D.: Obstruction on-line detection algorithm based on Kinect in-depth technical. Electron. Des. Eng. 22(12), 176–179 (2014) 5. Amara, A., Moats, T., Aloof, N.: GPU based GMM segmentation of Kinect data. In: 56th International Symposium ELMAR, pp. 1–4 (2014) 6. Murgia, J., Meurie, C., Ruichek, Y.: An improved colorimetric invariants and RGB-depthbased codebook model for background subtraction using Kinect. In: Mexican International Conference on Artificial Intelligence, pp. 380–392 (2014) 7. Barnich, O., Van Droogenbroeck, M.: ViBe: a universal background subtraction algorithm for video sequences. IEEE Trans. Image Process. 20(6), 1709–1724 (2011) 8. Yu, Y., Cao, M.W., Yue, F.E.: ViBe: an improved ViBe moving target detection algorithm. Chin. J. Sci. Instrum. 35(34), 924–931 (2014) 9. Vijayanagar, K.R., Loghman, M., Kim, J.: Refinement of depth maps generated by low-cost depth sensors. In: Soc Design Conference, pp. 355–358. IEEE (2013) 10. Yang, N.E., Kim, Y.G., Park, R.H.: Depth hole filling using the depth distribution of neighboring regions of depth holes in the Kinect sensor. In: IEEE International Conference on Signal Processing, Communication and Computing, pp. 658–661. IEEE (2012)

An Image Segmentation Method Based on Asynchronous Multiscale Similarity Measure Min Li(&), Zhongwai Xu, Hongwen Xie, and Yuhang Xing Rocket Force University of Engineering, Xi’an 710025, China

Abstract. Image segmentation as a basic operation in computer vision is widely used in object detection, feature extraction and so on. In order to improve the effects and speed of image segmentation, an asynchronous processing mechanism of image segmentation was proposed, which use the image gray histogram and spatial contiguity and can avoid multiple iteration of the traditional FCM. A multiscale similarity measure method is proposed combined with the nonlinear sensitivity of gray difference of human based on the tree structure data representation of irregular rough classification of image block, using to merge the image blocks to obtain the segmentation result. Experimental results show that the proposed algorithm outperform the FCM in terms of segmentation effect and computation speed. Keywords: Image segmentation
FCM
Asynchronous processing Data representation
Multiscale similarity measure

1 Introduction Image segmentation is a basic operation of computer vision in order to divided natural image into non overlapping meaningful region [1]. The essence of the segmentation method is how to represent each part of the image in a simple way, which makes the segmentation more meaningful and easy. The segmentation algorithm is divided into two categories according to the discontinuity and similarity of the two attributes. Both two properties are derived from the gray level. In recent years, image segmentation has been applied in many fields of computer vision, including feature extraction, object recognition, image registration and so on [2]. Gray is the most basic information of the image, which combining with clustering analysis has become the most direct segmentation method [3]. But the image segmentation is not qualitative [4], different segmentation methods will produce different results, which makes people pay special attention on the fuzzy clustering method and the most typical method is fuzzy C clustering FCM, which belongs to the unsupervised image segmentation method, of which advantages do segmentation computer automatically without human intervention. But FCM also has its obvious defects, the segmentation on the use of gray information, which leads to poor image segmentation effect, furthermore, FCM need to constantly update the center point, and calculate the

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 93–101, 2017. https://doi.org/10.1007/978-3-319-71598-8_9

94

M. Li et al.

points to the degree of deviation from the center point of the calculation is time-consuming. In view of the shortcomings of FCM, many improved algorithms have been proposed. fuzzy kernel clustering image segmentation method is used by Chaira which show [5] better effect; Ma combined with mathematical morphology method for multi regional medical image segmentation [6]; Neutrosophy theory was Introduced into image segmentation by Cheng, which provides a new description method of uncertainty, meanwhile the image segmentation algorithm based on Neutrosophy has been defined [7], which can select the threshold automatically and effectively; in the pape [8] the image of wisdom segmentation algorithm is expanded to color texture image. The the blurring effect brought by the average operator is overcome by application of a median operator; instead of Euclidean distance measure algorithm, can the to give the disadvantage of same proportion of multi-dimensional sample weight of color features was overcome by replacing Euclidean distance with Mahalanobis distance. The spatial neighborhood constraint term to the objective function and increase the correlation between the pixels neighborhood wasadded in order to improve the MNCM_S clustering algorithm [9]. Compared to FCM, the above algorithm has been improved in image segmentation effect, but it is by changing the cluster similarity measure, which in addition to considering the gray information also joined the space constraints, the edge information, increasing the calculation while improving effect. This paper proposes an asynchronous multiscale similarity measure, which through the histogram for coarse segmentation and the using of space location of the to separate sub image blocks which in the same gray level but not adjacent. Finally multi-scale comprehensive similarity measure method are used and considering the nonlinear effect on the image block difference to gray eye sensitivity with gray value changes are combined to get the final segmentation. It can solve the contradiction between the segmentation effect and calculation.

2 The Overall Structure of the Algorithm In order to solve the problem that the efficiency of fuzzy clustering is low and the image segmentation effect of complex structure is not is not satisfactory, this paper proposes a new method of image segmentation based on asynchronous multi-scale similarity measure. The traditional image segmentation based on FCM achieve the best classification by iteratively updating the center. The similarity measure is especially important in the iterative process. classification effect is not ideal if only gray information used for clustering, but if the similarity is expressed by complex formula, the computation is very huge because of many iterative times. In this paper, the main methods to solve the above problems proceed in two steps, the first step is to use simple grey level to divide image into several small classes. Then using tree data structure to represent irregular image blocks, according to the spatial relationship of the image block is further divided into the same gray adjacent; The second step is use comprehensive measurement way for small class merging. The principle of the algorithm is shown in Fig. 1.

An Image Segmentation Method Based on Asynchronous Multiscale

95

Fig. 1. Flow chart of the algorithm

3 C Mean Fuzzy Clustering Method [10] Hard clustering [11] is a common method in cluster analysis, which aim to make the objective function minimum, by specifying the initial cluster center, according to the similarity criterion, the initial data is divided into classes and each iteration the center position. The commonly used objective function is shown in formula (1): JHCM ¼

ni c X X

xij i¼1 j¼1

2 vi

ð1Þ

In the formula, ni is the number of data in the ith class. vi is the cluster center of ith class and xij is the jth data in the ith class. The center position is gotten by minimizing JHCM by iterating and the iterative formula is: vi ðt þ 1Þ ¼

ni 1X xij ðtÞ ni j¼1

ð2Þ

vi ðt þ 1Þ is the new central position. When the location of the new central point changes very little compared with the previous time, the iteration is stopped and the threshold e is chosen as the termination condition of the iteration. c X

kvi ðt þ 1Þ

vi ðtÞk2 \e

ð3Þ

i¼1

vi ðt þ 1Þ and vi ðtÞ are the ith center points of the t þ 1 and t rounds, respectively. In order to express the uncertainty of the classification of objective things, people prefer to use fuzzy clustering for clustering analysis. Fuzzy clustering is used to describe the uncertainty of of real things through the subordination matrix.

96

M. Li et al.

JFCM ¼

c X n X i¼1 j¼1

lqij xj

2 vi

ð4Þ

lij is the degree of membership of data j in the class i, q is a constant, usually 2. The fuzzy clustering in the classification using the concept of membership to avoid the classification hard clustering process which in either this or that, clustering is a special case of fuzzy clustering, which lij is only 0 or 1.

4 Asynchronous Multiscale Image Segmentation Algorithm 4.1

Image Segmentation Based on Gray Histogram

For a gray image, the gray histogram can describe the gray level distribution well, according to which can avoid the FCM iterative process, improving the efficiency of the calculation, the coarse classification of the histogram steps are as follows (Fig. 2): (1) Statistics the frequency hðiÞ of each gray level of the image to be segmented, i ¼ 0; 1; 2;


255.

Fig. 2. Sketch map of gray histogram

(2) find out all the peaks and troughs of the gray frequency. For i ¼ 0; 1; 2;


255, a set consisting satisfies the condition (

hðiÞ [ hði

1Þ

hðiÞ [ hði þ 1Þ

ð5Þ

constitutes a crest vector P, P ¼ fi j hðiÞ [ hði 1Þ & hðiÞ [ hði þ 1Þ; 0\i\255g. Similarly, For i ¼ 0; 1; 2;


255 a set consisting satisfies the condition (

hðiÞ\hði 1Þ hðiÞ\hði þ 1Þ

ð6Þ

An Image Segmentation Method Based on Asynchronous Multiscale

97

Consisting of a set of trough vectors. V ¼ fi j hðiÞ\hði 1Þ & hðiÞ\hði þ 1Þ; 0\i\255g (3) screening the trough In consideration of the pixel frequency of many pixels is only slightly fluctuating, it is necessary to further filter the trough in order to avoid excessive classification. It is obvious that the peaks and valleys appear alternately. If two pixels between the tops of too little, which satisfy the following equation: PðkÞ X

ð7Þ

hðiÞ\e

i¼Pðk 1Þ

It can be considered that the two peaks have no clear boundaries, then remove the trough between the two peaks, and merge the two peaks in the following: 0

P ðkÞ ¼

Pðk þ 1Þ þ PðkÞ 2

ð8Þ

(4) further division of image segmentation according to the trough and spatial relations According to the V (3), the image is segmented according to the gray value Regarding trough V as the threshold which get from (3) After the segmentation is completed, the adjacent blocks are divided into different classes according to the spatial relation. For a pixel ðx; yÞ satisfy a point in the four neighborhood belongs to U, then ðx; yÞ 2 U, That is U ¼ fðx; yÞ j ðx

4.2

1; y

1Þ 2 Uorðx þ 1; y

1Þ 2 Uorðx

1; y þ 1Þ 2 Uorðx þ 1; y þ 1Þ 2 U

ð9Þ

Multiscale Similarity Measure

Numerous information, such as gray, shape, contour and position can be extracted from the image by human, while the traditional segmentation method basically only little information, such as gray, edge and so on, which determines the traditional segmentation algorithm for complex image segmentation effect is not very good. In this paper, we use multiscale similarity measure to improve the image segmentation effect as much as possible. (1) Gray measure The gray level of the image is the most basic feature of the image, and the gray similarity measure is the most direct method to measure the similarity. In this paper, the gray similarity measure is expressed as formula (10) by calculating the gray difference between adjacent pixels of the adjacent image blocks: N  255

Sg ¼ P P

xi xj xi 2U xj 2Mi

ð10Þ

98

M. Li et al.

In the formula(10), U is a set of adjacent P pixel of sub image A and B. Mi is a set of Mi . adjacent pixel of xi in B. N equals xi 2U

(2) Information entropy measure The information entropy is the amount of the information source of Shannon, which can represent the degree of chaos and the probability distribution vector. Suppose that the probability distribution vector of v is ðx1 j p1 ; x2 j p2 ;


; xn j pn Þ. Then the information entropy of v can be expressed as: n X

EðvÞ ¼

pi log2 ðpi Þ

ð11Þ

i¼1

For an image block or an image, its probability distribution can be expressed by gray level histogram. That is, ðx1 j p1 ; x2 j p2 ;


; xn j pn Þ n is the gray level, xi is the gray value of the level i, pi ¼ hðxi Þ=N, N is the number of pixels. The information entropy of the image can be expressed as: n X

EðvÞ ¼

i¼1

hðxi Þ Þ log2 ð N

ð12Þ

The information entropy similarity measure Se of two images is expressed as: Se ¼

Eðv1 Þ þ Eðv2 Þ Eðv1 [ v2 Þ

ð13Þ

(3) Space adjacency measure For two adjacent image blocks whose pixels are m and n, respectively, X1 , X2 , if m [ n, the spatial adjacency measure of the image is defined as: 1 Ss ¼ n

X

yi 2X2

flagðyi Þ

!2

ð14Þ

When y is adjacent to X1 , then flagðyi Þ ¼ 1, otherwise flagðyi Þ ¼ 0, the above three measures will be integrated: S ¼ aSg þ bSe þ cSs

ð15Þ

The research shows that the nonlinear relationship between visual sensitivity to luminance difference [12]. In order to simulate visual effects, change the S in this way: 0

S ¼1

expð kSÞ

ð16Þ

An Image Segmentation Method Based on Asynchronous Multiscale

99

5 Experiment and Analysis In order to verify the effectiveness and efficiency of the algorithm, the four algorithms (plane, goose, horse, Eagle) were compared with the PCNN (Pulse coupled neural network) and PCM image segmentation methods. The experimental images derive from the standard Berkeley image library, the fuzzy clustering center number c = 2, PCNN algorithm and n = 16, using Matlab 2012b, the computer is configured to Intel (R) Core (TM) i7-4790 CPU, 3.60 GHz 64 Win7 operating system. The segmentation results of each method are shown in Fig. 3 and the computation time of each algorithm is shown in Table 1. The experimental results show that the segmentation effect is good for the simple background and single tone, and the segmentation results are better than those of PCNN and FCM for complex objects with complex tone. In terms of algorithm time efficiency, the proposed algorithm has improved significantly relative to FCNN and FCM.

Fig. 3. (continued)

100

M. Li et al.

Fig. 3. Comparison of experimental results of segmentation

Table 1. Comparison of segmentation algorithm time (time/s) Pixel FCM PCNN Proposed method

Plane 643*963 8.1618 21.6294 2.9150

Goose 643*963 5.4877 21.8116 2.9298

Horse 643*963 6.3925 20.2540 1.3152

Eagle 643*963 7.7344 21.8268 2.9412

6 Conclusion In order to solve the problem that the segmentation effect is not good and the amount of computation is too large in the fuzzy C mean clustering, this paper proposes an asynchronous multi-scale processing method. First of all, Gray histogram is used to segment the image roughly and the use spatial relations for further classification, and then propose a multiscale similarity measure method, considering the gray level, position,

An Image Segmentation Method Based on Asynchronous Multiscale

101

information entropy, combining the nonlinear characteristics of the visual system, to the first step of rough classification results by cluster merging. The simulation results show that the algorithm has some improvement in time and segmentation results.

References 1. Zhu, H., Meng, F., Cai, J., Lu, S.: Beyond pixels: a comprehensive survey from bottom-up to semantic, image segmentation and cosegmentation. J. Vis. Commun. Image R. 34, 12–27 (2016) 2. Biswas, S., Ghoshal, D., Hazra, R.: A new algorithm of image segmentation using curve fitting based higher order polynomial smoothing. Optik 127, 8916–8925 (2016). 0030-4026 3. Cui, Z., Sun, S., Chen, S., et al.: Mean shift based FCM image segmentation algorithm. Control Decis. 29(6), 1130–1134 (2014) 4. Mei, W., Yu, L.: A survey on graph theory approaches of image segmentation. Comput. Appl. Softw. 31(9), 9 (2014) 5. Chaira, T., Panwar, A.: An atanassov’s intuitionistic fuzzy kernel clustering for medical image segmentation. Int. J. Comput. Intell. Syst. 7(2), 360–370 (2014) 6. Ma, Y.Z., Chen, J.X.: A new medical image segmentation method based on Chan-Vese model. Appl. Mech. Mater. 513–517, 3750–3756 (2014) 7. Cheng, H.D., Guo, Y.: A new neutrosophic approach to image thresholding. New Math. Natural Comput. 4(3), 291–308 (2008) 8. Guo, Y.: Modified neutrosophic approach to color image segmentation. J. Electron. Imaging 22(1), 4049–4068 (2013) 9. Chengmao, W., Shangguan, R.: Robust colour image segmentation algorithm based on neutrosophic fuzzy clustering. J. Xi’an Univ. Posts Telecommun. 22.1 01 (2017) 10. Dunn, J.C.: A fuzzy relative of the ISODATA process and its use indetecting compact, well-separated clusters. J. Cybern. Inf. Sci. 3, 32–57 (1974). (S0146-5090) 11. Xuming, F., Xiaoxian, W.: The airborne solid recording system based on PC104. Comput. Eng. Appl. 41(24), 225–227 (2005) 12. Hongbo, T., Zhiqiang, H., Rong, L.: Region grow image segmentation based on human visual model. J. Image Graph. 15(9), 1352–1356 (2010)

Lagrange Detector in Image Processing Feilong Ma1,2, Linmi Tao1,2 ✉ , and Wu Xia1,2 (

)

1

2

Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China [email protected] Pervasive Computing Division, Tsinghua National Laboratory for Information Science and Technology, Beijing 100084, China

Abstract. Edge detection is a basic operation in the field of image processing and computer vision. However, to the best of our knowledge, there is less math‐ ematical work has been proposed beyond the first- and the second-order derivative operator for edge detection in the past decays. We propose a mathematical model called Lagrange detector for edge detection. Based on Lagrange polynomial interpolation theory, this detector calculates Lagrange remainder as the strength of edge and points at the features in various orders of discrete data or signal. Lagrange remainder combines the first-order derivation and the first- and secondorder derivative of neighborhood by multiplication. We use the truncation error of polynomial interpolation to estimate Lagrange remainder. Lagrange detector performs well in detecting both outlines and tiny details. Furthermore, Lagrange detector can be used to detect high-frequency information like as corner, point, Moiré pattern and etc. The research of Lagrange detector opens an new window for low level image processing, and will be used as the basis for further studies on image processing. Keywords: Edge detection · Lagrange detector · Polynomial interpolation Laplace operator · Canny detector

1

Introduction

Edge detection is a first-step operation used in image processing and computer vision applications, resulting in significantly reducing the amount of data, filtering out useless information and preserving the important structural properties in an image. Edge detec‐ tion refers to the process of identifying and locating sharp discontinuities in an image. The discontinuities are obviously changes in pixel intensity which characterize boun‐ daries of objects in a scene. This field possesses many mathematical models, including Sobel operator, Roberts operator, Prewitt operator, Laplace operator, LoG detector [1], Canny detector [2] and etc. [3]. These operators are based on first-order derivative or second-order derivative. The methods based on first-order derivative detect edges by first computing a measure of edge strength, usually first-order derivative expression such as the gradient magni‐ tude, and then searching for local directional maxima of the gradient magnitude using a computed estimate of the local orientation of the edge, usually the gradient direction. © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 102–112, 2017. https://doi.org/10.1007/978-3-319-71598-8_10

Lagrange Detector in Image Processing

103

The methods based on second-order derivative detect edges by searching for zero cross‐ ings in a second-order derivative expression, usually the zero-crossings of the Laplace operator or the zero-crossings of a non-linear differential expression. Many methods and techniques have been developed based on these mathematical theories. Data driven methods is the newly and hottest studies, by learning the possibility distributions of features, including methods based on CNN [4]. However, to the best of our knowledge, there is less mathematical work has been proposed beyond the first- and the second-order derivative operator for edge detection in the past decays. We propose a mathematical theory called Lagrange detector, which is the Lagrange form of the remainder of Taylor’s theorem. Polynomial interpolation theory provides basis for the calculation of Lagrange reminder. Lagrange detector identifies the highorder components in images, charactering, extracting and processing points that present edges or curves.

2

Related Work

2.1 Edge Detection Edge detection aims at capturing boundaries and curves by detecting sharp changes in image brightness. All of mathematical operators detecting edge are based on first-order derivative or second-order derivative. Essentially we need to get the extremum value of brightness variations that is the first-order derivative. The methods based on calculating the second-order derivative are just to get the extremum point of first-order derivative [3] (Fig. 1). 2.2 First-Order Methods The methods based on first-order derivative include Sobel operator, Roberts operator, Prewitt operator and Canny detector. One of them Sobel operator has concise mathe‐ matical expressions and Canny detector uses Sobel operator as pretreatment. So we mainly explore the mathematical theory of Sobel operator [3]. Sobel operator has clear and definite mathematical expressions. Sobel operator is defined as Sx,y =

√ dx2 + dy2

(1)

where dx is the gradient in x, dy is the gradient in y. Practically we use two 3 × 3 matrices to calculate the differences Δx , Δy as approx‐ imate the values of dx, dy. We have that ⎛ −1 0 +1 ⎞ Δx = ⎜ −2 0 +2 ⎟ × A ⎜ ⎟ ⎝ −1 0 +1 ⎠

(2)

104

F. Ma et al.

(a)

(c) (b)

Fig. 1. The figure indicates that the mathematical model of edges. (a) The black curve: the function of the original signal; (b) the blue curve: the first-order derivative of the signal; (c) the red curve: the second-order derivative of the signal. The green cross-shaped mark is the point that edge operators detect, of which the first-order derivative takes the extremum value and the secondorder derivative crosses over zero. (Color figure online)

⎛ +1 +2 +1 ⎞ Δy = ⎜ 0 0 0 ⎟ × A ⎜ ⎟ ⎝ −1 −2 −1 ⎠

(3)

where A is the data matrix of the image. While the gradient orientation can be estimated as ( / ) � = tan−1 Δy Δx

(4)

2.3 Second-Order Methods The methods based on second-order derivative include Laplace operator and LoG detector. Laplace operator also has clear and definite mathematical expressions [3]. LoG detector combines Laplace operator and Gaussian filter [1]. Laplace operator is defined as ∇2 f (x, y) =

�2f �2 f + 2 2 �x �y

(5)

We also use a 3 × 3 matrix to calculate the second-order difference as the value of ∇ f (x, y). So we have 2

Lagrange Detector in Image Processing

⎛ 0 +1 0 ⎞ ∇2 f ≈ ⎜ +1 −4 +1 ⎟ × A ⎜ ⎟ ⎝ 0 +1 0 ⎠

105

(6)

where A is the data matrix of the image.

3

Lagrange Detector

Lagrange detector is the Lagrange form of the remainder of Taylor’s theorem. The reminder can be calculated via the difference between the polynomial interpolation and the true value. For polynomial interpolation, Lagrange Polynomial is Ln (x) =

∑n

k=0

f (xk )lk (x)

(7)

n x − xj ( ) ∏ (k = 0, 1, 2, … , n), given xi , f (xi ) (i = 0, 1, 2, … , n). The j=0,j≠k xk − xj Lagrange remainder is

where lk (x) =

Rn (x) = f (x) − Ln (x) =

f (n+1) (�) ∏n (x − xi ) i=0 (n + 1)!

(8)

where ∀x ∈ (a, b), f (n+1) (x)exists, ∃� ∈ (a, b)makes the formula (8) be workable [5–7]. We have proved that Rn has obviously positive correlation with signal frequency. So we are able to use Rn to characterize and extract the high-frequency components. After attaining the high-frequency data, we can do the works well about detecting edge, corner, point, Moiré pattern [8] and etc. 3.1 1D Signal From Taylor’s Formula, a 1-dimension signal S(t) can be decomposed into a set of polynomial expressions: S(t) =

∑+∞ S(n) (t0 ) ⋅ (t − t0 )n n=0 n!

(9)

with S(n) (t0 ) is the n-order derivative of S(t) in t = t0 and S(n) (t0 ) exists. The Lagrange remainder during Lagrange polynomial interpolation is

Rn (t) =

S(n+1) (�) ∏n (t − ti ) i=0 (n + 1)!

(10)

( ) ( ) In t ∈ � and � ∈ (min t, t0 , … , tn , max t, t0 , … , tn ), the n + 1-order derivative of S(t) exists. When n = 1, the remainder is

106

F. Ma et al.

R1 (t) =

S(2) (�) (t − t0 )(t − t1 ) 2

(11)

We take three adjacent points as subject investigated. By linear interpolation with points on side, we get the valuation of the middle point. Figure 2 indicates that errors are significantly different between edges and smooth parts. The truncation error δ: | | S′′ (�) (t − t0 )(t − t1 )|| � = ||R1 (t)|| = || | | 2

(a)

(b)

(12)

(c)

Fig. 2. The figure indicates that the mathematical model of Lagrange detector. The polynomial curves pass through three adjacent points. The black circles mean the value of the middle point that is got by linear interpolation with points on side. (a) and (b) indicate the deviation is large, corresponding to edges; (c) the deviation is tiny, corresponding to smooth parts. (Color figure online)

We take the difference between true value and interpolation value as

diff = ||Smid,n=1 − Smid ||

(13)

We can use diff to detect edges and curves. The method is the formula (13). 3.2 Experiment on 1D Signals Exp.01: We take sin(2��t) as source signal and �s = 100.0 as sampling rate. In an interval ω ∈ [1, 9], we calculate � by the formula (13) and get the relationship curves of � and ω. The results follow as Fig. 3. When � is larger, the feature of edges is more obvious.

Lagrange Detector in Image Processing

107

Fig. 3. The figure indicates the relationship curves of � and ω, that is the relationship of diff and edges. (a) source signals; (b) the relationship curves of � and ω. Using linear interpolation and (13), we get � . (Color figure online)

3.3 Experiments on 2D Signals Exp. 02: Use a Gabor filter [9] to get the source image that is Fig. 4(a). After processing this image by l = 1 Lagrange detector, we get the result Fig. 4(b). To give firm analyses, we calculate � that is the average of diff �D. Figure 4(b) indicates the relation of � and the frequency. The results indicate that our approach is an isotropy detector.

108

F. Ma et al.

Fig. 4. The figure indicates the results of processing images via Gabor filter. (a) a matrix of the source images; (b) a matrix of the results in which top-to-bottom the column correspond to 0, π⁄ 4, π⁄2, 3π⁄4 on phase.

| ′ | diff �D = |Il − I|(l = 0, 1, … , N) | |

(14)

Figure 4(b) indicates Lagrange detector has the feature of isotropy on capturing edges, so we can just take one direction to contrast results of using Sobel operator, Laplace operator and Lagrange detector to process these images. Follow as Fig. 5:

Lagrange Detector in Image Processing

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

(11)

(12)

(13)

(14)

(15)

(16)

109

Fig. 5. Figures (1), (2), (3) and (4) are source images of which frequencies increase successively, that means the features of these patterns change from smooth to sharp. Figures (5), (6), (7) and (8) are the results by Sobel operator; for based on first-order derivative, operation region is two adjacent points, Sobel operator is sensitive on both smooth components and edges. Figures (9), (10), (11) and (12) are the results by Laplace operator of which processing on three adjacent points to calculate second-order derivative, resulting in losing some important information. Figures (13), (14), (15) and (16) are the results by Lagrange detector. From formula (12), Lagrange detector combines first-order derivative and second-order derivative. As a result, Lagrange detector can filter smooth components to some extent and capture as much edge information that change sharply as possible.

110

4

F. Ma et al.

Lagrange Detector Processes Images to Capture Edges

4.1 Approach Description After above all the analyses and experiments, we have the approach (Fig. 6):

(1)

(2)

(3)

(4)

Fig. 6. The figure is the flow chart of Lagrange detector. (1) Get the gray image or RGB single channel from the source image, and use Gaussian filter to depress the noise; (2) take 1-order Lagrange interpolation at four directions 0, �∕4, �∕2, 3�∕4; (3) then get the sum of their absolute values and remove weak points by taking a threshold; (4) search edges at directions 0, �∕4, �∕2, 3�∕4, to remove short and tiny curves or points. (Color figure online)

(a)

(b)

(c)

(d)

Fig. 7. These figures are test results of Canny detector, LoG detector and Lagrange detector. (a) source images; (b) results of using Canny detector; (c) results of using LoG detector; (d) results of Lagrange detector. (Color figure online)

Lagrange Detector in Image Processing

111

4.2 Experimental Results We perform some tests for making comparison between different edge detectors and Lagrange detector on the images, of which from left to right 1 is from MCM dataset [10], 2 is from Kodak dataset [11], 3 is from Sandwich dataset [12], 4 is from McGill dataset [13] and 5 is standard Lena image. The results are Fig. 7. Canny detector and LoG detector have been developed for many years, improved by researchers and developer for lots of times. So they perform well in presenting edges by curves, which means their strategies at processing the data from Sobel operator or Laplace operator. To a certain extent, Lagrange detector performs well in both outlines and tiny details.

5

Lagrange Detector Analysis and Conclusion

We propose a novel mathematical theory named Lagrange detector for edge detection. Based on Lagrange polynomial interpolation theory, this detector takes Lagrange remainder as an operator. The formulas (12) and (13) are the mathematical basis of detecting edges. By combining the first-order derivative and the second-order derivative, Lagrange detector can filter smooth components to some extent and capture as much edge information that change sharply as possible. Like as Sobel operator and Laplace operator, Lagrange detector also has solid and definite mathematical expression shown in formula (2). While used to detect edge, Lagrange detector can presented as formula (12). At this case, the expression of Lagrange detector is equivalent to the truncation error S′′ (�), which is the second-order derivation of neighborhood. Lagrange detector includes the characteristics of the firstorder derivative and the second-order derivative. The distinction between traditional edge detection operators and Lagrange detector is that the traditional operators aim at get the extremum value of brightness variations that is the first-order derivative, but Lagrange detector uses Lagrange remainder to combine the first-order derivation and the first- and second-order derivative of neighborhood by multiplication. By calculating the truncation error of polynomial interpolation, we can estimate Lagrange remainder. Lagrange detector performs well in detecting both outlines and tiny details. Lagrange detector provides a new theoretical approach for edge detection, which open a new window for low level image processing, and will be used as the basis for further studies on image processing.

References 1. Marr, D., Hildreth, E.: Theory of edge detection. Proc. Roy. Soc. London Ser. B Biol. Sci. 207, 187–217 (1167). https://doi.org/10.1098/rspb.1980.0020 2. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–714 (1986) 3. Saket, B., Ajay, M.: A survey on various edge detector techniques. Procedia Technol. 2, 220–226 (2012). Elsevier 4. Liu, Y., Cheng, M.M.: Richer convolutional features for edge detection. In: CVPR (2017)

112

F. Ma et al.

5. Meijering, E.: A chronology of interpolation: from ancient astronomy to modern signal and image processing. Proc. IEEE 90(3), 319–342 (2002) 6. Jean-Paul, B., Lloyd, N.: Barycentric lagrange interpolation. SIAM Rev. 46(3), 501–517 (2004) 7. Nicholas, J.: Highman.: the numerical stability of barycentric lagrange interpolation. IMA J. Numer. Anal. 24(4), 547–556 (2004) 8. Moiré pattern. https://en.wikipedia.org/wiki/Moiré_pattern 9. Movellan, J.R.: Tutorial on Gabor Filters. Archived from the original on 19 April 2009. Accessed 14 May 2008 10. Acude-images. http://www.eecs.qmul.ac.uk/~phao/CFA/acude/IMAX/rad-n5-images/ 11. Kodak Lossless True Color Image Suite. http://r0k.us/graphics/kodak/ 12. Meiqing, Z., Linmi, T.: A Patch Aware Multiple Dictionary Framework for Demosaicing (ACCV 2014), Singapore, 1–5 November 2014 13. McGill Calibrated Colour Image Database. http://tabby.vision.mcgill.ca/

New Tikhonov Regularization for Blind Image Restoration Yuying Shi1(B) , Qiao Liu1 , and Yonggui Zhu2 1

Department of Mathematics and Physics, North China Electric Power University, Beijing 102206, China [email protected] 2 School of Science, Communication University of China, Beijing 100024, China

Abstract. Blind image restoration is a challenging problem with unknown blurring kernel. In this paper, we propose a new algorithm based on a new Tikhonov regularization term, which combines three techniques including the split Bregman technique, fast Fourier transform and spectral decomposition technology to accelerate the computation process. Numerical results demonstrate that the proposed algorithm is simple, fast and effective for blind image restoration.

1

Introduction

Blind image restoration has been widely applied in remote sensing, astronomy, medical imaging, video cameras and so on (see, e.g. [8,10, 18]). For example, when taking the photograph of a moving object, the shutter speed and the speed of the object are unknown. The process of degradation can be modeled as: g = h∗f +n, where g stands for the degraded image, f represents the original image, h is the blurring kernel (also called as point spread function (PSF)), n expresses the noise, and ∗ denotes convolution operation. As the PSF is unknown, a lot of restoration techniques have been proposed (see, e.g. [1, 3, 5,11–14, 16,17, 19, 26]). It is becoming one of the most challenging problems for its complication and difficulty. Regularization is one way to avoid the problems due to the ill-posed nature of blind image restoration. You and Kaveh [26] used the minimizing formulation with H 1 regularization terms for both the image and the PSF: 1  h ∗ f − g22 + λ1 f 2H 1 + λ2 h2H 1 . (1) min f,h 2 Chan and Wong [5] regularized both the image and the PSF by the famous total variation (TV) regularization terms (see, e.g. [5, 21]) instead of the H 1 -norm: 1  h ∗ f − g22 + λ1 T V (f ) + λ2 T V (h) . min (2) f,h 2 The TV regularization is considered to be one of the best approaches to recovering edges of image, but also one of the hardest to computing because c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 113–123, 2017. https://doi.org/10.1007/978-3-319-71598-8_11

114

Y. Shi et al.

of the nonlinearity and non-differentiability of the TV regularization term. The split Bregman (SB) method is shown to be efficient to handle the TV term with notable stability and fast convergence rate (see, e.g. [4, 5, 7,9]). Based on the SB method, the minimization problem can be divided into several subproblems. The important and difficult subproblems are to effectively find the solutions f and h alternatively, so we need to solve large linearized systems of equations. To solve large equations, the cosine preconditioned conjugate gradient method and the fixed-point method are proposed in [5]. Coupled systems of f and h are evolved by a time marching method, which is based on the gradient descent method [9]. Chang et al. [25] applied an algebraic multigrid (AMG) method and Krylov subspace acceleration technique to solve the linearized systems of equations [25]. We also noticed that the 2D fast Fourier transform (FFT) [4,14] is fast and simple. Moreover, the spectral decomposition technology (SDT) [24] is efficient for handling a new Tikhonov regularization term [6, 15]. The SDT not only reduces the amount of calculation, but also saves the storage space. The purpose of this paper is to briefly describe a combined algorithm of the SB method, FFT and SDT, which agglomerates advantages of the above three methods, and realize blind image restoration with a TV regularization term and a new Tikhonov regularization term. The organization of this paper is given as follows. Section 2 exhibits our new combined algorithm based on the SB method, FFT and SDT. Computational results are shown in Sect. 3. Finally, some conclusions are given in Sect. 4.

2

Proposed Algorithm

In this section, we propose a fast combined algorithm to solve blind image restoration problem with the TV regularization term for h and the new Tikhonov (NT) regularization term for f [6] using the SB technique, FFT and SDT. We denote the proposed algorithm as NT-SB algorithm. 2.1

NT-SB Algorithm

To induce our new algorithm, we need to modify the model (2) as follows: 1   λ1 h ∗ f − g22 + Lλ f 22 + λ2 (∇x h)2i + (∇y h)2i , min (3) f,h 2 2 i where λ1 , λ2 > 0 are regularization parameters, Lλ f 22 is a regularization term, which can filter high frequency information such as noise. We will show the detailed definition of Lλ in (12) later. Introducing two auxiliary variables c1 , c2 similar to [7], we make the following substitutions for the model (2), ∇x h → c1 , ∇y h → c2 . This yields the following equivalent constrained problem:  1 λ1 h ∗ f − g22 + Lλ f 22 + λ2 (c1 , c2 )2 , min f,h,c1 ,c2 2 2 (4) s.t. ∇x h = c1 , ∇y h = c2 .

New Tikhonov Regularization for Blind Image Restoration

115

Notice that here we only need two auxiliary variables, since there is only one TV regularization term. While for the following TV-SB algorithm in Sect. 3, we have to introduce four variables. Thus the iterative scheme of our new algorithm based on the SB technique is as follows: , ck+1 )= (f k+1 , hk+1 , ck+1 1 2 1 λ1 h ∗ f − g22 + Lλ f 22 + λ2 (c1 , c2 )2 arg min f,h,c1 ,c2 2 2  γ2 + (c1 − ∇x h − sk1 22 + c2 − ∇y h − sk2 22 ) , 2

(5)

and = sk1 + ∇x hk+1 − ck+1 , sk+1 1 1 = sk2 + ∇y hk+1 − ck+1 , sk+1 2 2

(6)

where the parameters λ1 , λ2 > 0 fit the fidelity term and the regularization terms, γ2 > 0 regulates the penalty function terms. The minimization problem (5) can also be decoupled into the following subproblems: 1: h-subproblem: for fixed f k , ck1 , ck2 , sk1 , sk2 , we need to solve   γ2 hk+1 = arg min H(h) + (ck1 − ∇x h − sk1 22 + ck2 − ∇y h − sk2 22 ) , (7) h 2 where H(h) = 21 f k ∗ h − g22 . According to the optimality condition, which requires us to solve (f k )T ∗ (f k ∗ hk+1 − g) + γ2 ∇T ∇hk+1 +γ2 (∇Tx (sk1 − ck1 ) + ∇Ty (sk2 − ck2 )) = 0.

(8)

That is, hk+1 = F −1 [

F((f k )T g + γ2 (∇Tx (ck1 − sk1 ) + ∇Ty (ck2 − sk2 )) ], F((f k )T f k − γ2 )

where ∇T ∇ = −, ∇T = −div. 2: f -subproblem: for fixed hk+1 , we need to solve   f k+1 = arg min F1 (f ) + F2 (f ) , f

(9)

(10)

˜ − g2 , F2 (f ) = λ1 Lk+1 f 2 , and H ˜ is the BCCB (Block where F1 (f ) = 21 Hf 2 2 λ 2 Circulant with Circulant Blocks) blur matrix got by the blurring kernel h with periodic boundary conditions [20, 22]. We apply the spectral decomposition tech˜ and have nology for the blurring matrix H, ˜ = F ∗ ΣF, H

(11)

116

Y. Shi et al.

where F is a 2D unitary discrete Fourier transform (DFT) matrix, the conjugate transpose of a complex matrix F , denoted F ∗ and the diagonal matrix Σ = ˜ ˆ2 , · · ·, σ ˆn ] (see, e.g. [6]), where σ ˆ1 , σ ˆ2 , · · ·, σ ˆn are eigenvalues of H. diag[ˆ σ1 , σ Construct the regularization matrix Lλ as: Lλ = Dλ F,

(12)

where Dλ2 = ⎛ ˆ12 , 0) max(λ2 − σ ⎜ ˆ22 , 0) max(λ2 − σ ⎜ ⎜ .. ⎝ .

⎞ max(λ2 − σ ˆn2 , 0)

⎟ ⎟ ⎟. ⎠

(13)

According to the NT method proposed in [15], we have f = F ∗ (ˆ σ∗ σ ˆ + λ1 (Dλk+1 )∗ Dλk+1 )−1 σ ˆ ∗ F g. 3: c1 , c2 -subproblems: for fixed f k+1 and hk+1 , we need to solve  , ck+1 ) = arg min λ2 (c1 , c2 )2 (ck+1 1 2 c1 ,c2  γ2 k+1 − sk1 22 + c2 − ∇y hk+1 − sk2 22 ) . + (c1 − ∇x h 2

(14)

(15)

By shrinkage formulation (see, e.g. [7, 23]), the solutions of (15) are λ2 ∇x hk+1 + sk1 max{W k − , 0}, k W γ2 ∇y hk+1 + sk2 λ 2 = max{W k − , 0}, Wk γ2

= ck+1 1 ck+1 2

 where W k = (∇x hk+1 + sk1 )2 + (∇y hk+1 + sk2 )2 . We summarize the NT-SB algorithm as follows: NT-SB Algorithm 1. Initializing f 0 , c01 , c02 , s01 , s02 , 2. While f k+1 − f k 2 /f k 2 > tol, do a: solve (9) to get hk+1 , b: solve (14) to get f k+1 , , ck+1 , c: solve (16) to get ck+1 1 2 k+1 k+1 d: update s1 , s2 by (6). end do

(16)

New Tikhonov Regularization for Blind Image Restoration

117

Here, tol denotes the tolerance value for iteration scheme, and the order of the h and f subproblems can not be transposed.

3

Experimental Results

To show the superiority of the proposed algorithm, we compare the NT-SB algorithm with the TV-SB algorithm that solves the original model (2) using split Bregman method. First, we simply show the TV-SB algorithm. Using several auxiliary variables b1 , b2 , c1 , c2 , we need to solve the following equivalent constrained problem:   1 h ∗ f − g22 + λ1 (b1 , b2 )2 + λ2 (c1 , c2 )2 , min 2 f,h,b1 ,b2 (17) c1 ,c2 s.t. ∇x f = b1 , ∇y f = b2 , ∇x h = c1 , ∇y h = c2 . For the h-subproblem, we can get the same solution (9). For the f -subproblem, we get f k+1 = F −1 [

F((hk+1 )T g + γ1 (∇Tx (bk1 − tk1 ) + ∇Ty (bk2 − tk2 )) ]. F((hk+1 )T hk+1 − γ1 )

(18)

By shrinkage formulation as above, we obtain λ1 ∇x f k+1 + tk1 max{V k − , 0}, Vk γ1 k+1 k ∇y f λ1 + t2 = max{V k − , 0}, Vk γ1 k+1 k λ2 ∇x h + s1 max{W k − , 0}, = Wk γ2 k+1 k ∇y h λ2 + s2 = max{W k − , 0}, Wk γ2

bk+1 = 1 bk+1 2 ck+1 1 ck+1 2 where k

(19)



(∇x f k+1 + tk1 )2 + (∇y f k+1 + tk2 )2 ,  W k = (∇x hk+1 + sk1 )2 + (∇y hk+1 + sk2 )2 . V =

And the iterative parameters , = tk1 + ∇x f k+1 − bk+1 tk+1 1 1 k+1 , t2 = tk2 + ∇y f k+1 − bk+1 2 = sk1 + ∇x hk+1 − ck+1 , sk+1 1 1 k+1 k+1 k k+1 − c2 . s2 = s2 + ∇y h We summarize the TV-SB algorithm as follows:

(20)

118

Y. Shi et al.

TV-SB Algorithm 1. Initializing f 0 , b01 , b02 , t01 , t02 , c01 , c02 , s01 , s02 , 2. While f k+1 − f k 2 /f k 2 > tol, do a: solve (9) to get hk+1 , b: solve (18) to get f k+1 , , bk+1 , ck+1 , ck+1 , c: solve (19) to get bk+1 1 2 1 2 k+1 k+1 k+1 k+1 d: update t1 , t2 , s1 , s2 by (20). end do Some remarks are in order. (a) Comparing with the TV-SB algorithm, our proposed NT-SB algorithm uses less variables (missing b1 , b2 , t1 , t2 ) and less initial values need to be set (missing b01 , b02 , t01 , t02 ). (b) It is seen that, the computational complexity is obviously reduced for steps c and d.

(a)

(b)

Fig. 1. Original Images Table 1. ISN R values and computing times using the TV-SB algorithm and the NT-SB algorithm with different Gaussian Blurs (GB) and Moffat Blurs (MB). Blur(kernel size) Algorithm Iteration Time(s) ISN R GB (20)

TV-SB NT-SB

100 40

34.1719 1.4910 23.8906 2.2532

MB (20)

TV-SB NT-SB

44 24

16.0625 0.8769 14.4063 1.3146

GB (30)

TV-SB NT-SB

100 40

34.1719 1.4910 23.8906 2.2532

MB (30)

TV-SB NT-SB

45 23

17.2969 0.8717 13.8281 1.1383

New Tikhonov Regularization for Blind Image Restoration

(a) GB: 20 × 20

(b) TV-SB

(c) NT-SB

(d) GB: 30 × 30

(e) TV-SB

(f) NT-SB

(g) MB: 20 × 20

(h) TV-SB

(i) NT-SB

(j) MB: 30 × 30

(k) TV-SB

(l) NT-SB

119

Fig. 2. Comparisons of the TV-SB algorithm and the NT-SB algorithm. Column 1: Blurred images contaminated by different blurs with different blurring kernel sizes with the variance of σ 2 = 1; Column 2: Restored images by the TV-SB algorithm; Column 3: Restored images by the NT-SB algorithm.

120

Y. Shi et al.

In this section, we test two gray images (satellite and cameraman images) in Fig. 1 which are both of size 256 × 256 pixels to show the effectiveness and feasibility of the TV-SB algorithm and the NT-SB algorithm. In the following examples, we mainly compare the visual quality of the restored images and the improvement in signal to noise ratio (ISN R) value [2]. The larger the ISN R value is, the better the restored result is. The elements of the noise vector n are normally distributed with zero mean, 2 and the standard deviation is chosen such that n g2 = 0.01. In this case, we say that the level of noise is 1%. Moreover, the numerical examples are all implemented with MATLAB (R2010a) and the computer of test has 1G RAM and Intel(R) Pentium(R) D CPU @2.80 GHz. Table 2. ISN R values and computing times using the TV-SB algorithm and the NT-SB algorithm with different Gaussian Blurs (GB) and Moffat Blurs (MB) and 1% Gaussian noise. Blur

Algorithm Iteration Time(s) ISN R

GB (σ 2 = 1)

TV-SB NT-SB

50 20

18.8125 1.4161 12.0625 1.6982

MB (σ 2 = 1)

TV-SB NT-SB

50 15

18.2969 1.3533 9.4844 1.4057

GB (σ 2 = 1.5) TV-SB NT-SB

50 20

17.8438 0.7389 12.4375 0.8107

MB (σ 2 = 1.5) TV-SB NT-SB

50 15

17.1719 0.8776 8.7500 0.8971

Then, to compare the properties of the TV-SB algorithm and the NT-SB algorithm, we consider the degraded images contaminated by Gaussian or Moffat blur and Gaussian noise. First of all, we consider the cameraman image contaminated by only Gaussian or Moffat blur, the restored images are shown in Fig. 2. The parameters in the algorithms are set to be λ1 = 13, λ2 = 0.1, γ1 = 0.1e − 6, γ2 = 0.2. Blurred images contaminated by the size of 20 × 20 and 30 × 30 Gaussian blurs with σ 2 = 1 are displayed in Figs. 2(a) and (d), blurred images contaminated by the size of 20 × 20 and 30 × 30 Moffat blurs with σ 2 = 1 are depicted in Figs. 2(g) and (j); restored images by the TV-SB and NT-SB algorithms are shown in the second and third columns of Fig. 2, respectively. The ISN R values, number of iterations and computing times are listed in Table 1. We can see that the NT-SB algorithm has better restoration effect with larger ISN R values and needs fewer number of iterations and fewer computing times (cf. Table 1). Next, we consider the blurred-noisy satellite image contaminated by Gaussian or Moffat blur and 1% Gaussian noise in Fig. 3. Since the background color of satellite image is black, the noise is not so obvious in the contaminated images

New Tikhonov Regularization for Blind Image Restoration

(a) GB: σ 2 = 1

(b) TV-SB

(c) NT-SB

(d) GB: σ 2 = 1.5

(e) TV-SB

(f) NT-SB

(g) MB: σ 2 = 1

(h) TV-SB

(i) NT-SB

(j) MB: σ 2 = 1.5

(k) TV-SB

(l) NT-SB

121

Fig. 3. Comparisons of the TV-SB algorithm and the NT-SB algorithm. Column 1: Blurred-Noisy images contaminated by 10 × 10 Gaussian or Moffat blur with different ambiguous degrees and noise of 1%; Column 2: Restored images by the TV-SB algorithm; Column 3: Restored images by the NT-SB algorithm.

122

Y. Shi et al.

(the first column of Fig. 3). The parameters in the algorithms are set to be λ1 = 1, λ2 = 0.05, γ1 = 0.1e − 4, γ2 = 8. Blurred-noisy images contaminated by Gaussian and Moffat Blurs with σ 2 = 1 and 1% Gaussian noise are exemplified in Figs. 3(a) and (g), respectively, blurred-noisy images contaminated by the size of 10 × 10 Gaussian and Moffat Blurs with σ 2 = 1.5 and 1% Gaussian noise are exemplified in Figs. 3(d) and (j), respectively; restored images by the TV-SB and NT-SB algorithms are shown in the second and third columns of Fig. 3. We tabulate the ISN R values, number of iterations and computing times of the two algorithms in Table 2, which shows that the NT-SB algorithm has almost the same ISN R values as the TV-SB algorithm, but the NT-SB algorithm needs less number of iterations and fewer computing times (see Table 2). By observing Figs. 2 and 3, the NT-SB algorithm behaves better for the blurred-noisy images in visual and faster than the TV-SB algorithm with the same parameters.

4

Conclusions

In this paper, we introduced a new Tikhonov regularization term to replace the TV regularization term for blind restoration problem, and applied the split Bregman technique to separate the minimum formulation with the TV regularization term for the blurring kernel and the new Tikhonov regularization term for the image into several subproblems. In the process of solving the subproblems, we combined the FFT and the SDT to accelerate the computation. The TV-SB and NT-SB algorithms were shown to be effective by several numerical experiments. The NT-SB algorithm needed less space, fewer number of iterations, shorter computing times and better restored effect comparing with the TV-SB algorithm. Acknowledgments. The authors wish to thank the referees for many constructive comments which lead to a great improvement of the paper. This research is supported by NSFC grants (No. 11271126, 11571325).

References 1. Ayers, G.R., Dainty, J.C.: Iterative blind deconvolution method and its applications. Opt. Lett. 13(7), 547–549 (1988) 2. Babacan, S.D., Molina, R., Katsaggelos, A.K.: Total variation image restoration and parameter estimation using variational posterior distribution approximation. In: IEEE International Conference on Image Processing 2007, ICIP 2007, vol. 1, pp. 1–97. IEEE (2007) 3. Biemond, J., Tekalp, A.M., Lagendijk, R.L.: Maximum likelihood image and blur identification: a unifying approach. Opt. Eng. 29(5), 422–435 (1990) 4. Chan, R.H., Tao, M., Yuan, X.M.: Constrained total variation deblurring models and fast algorithms based on alternating direction method of multipliers. SIAM J. Imaging Sci. 6(1), 680–697 (2013)

New Tikhonov Regularization for Blind Image Restoration

123

5. Chan, T.F., Wong, C.-K.: Total variation blind deconvolution. IEEE Trans. Image Process. 7(3), 370–375 (1998) 6. Fuhry, M., Reichel, L.: A new tikhonov regularization method. Numer. Algorithms 59(3), 433–445 (2012) 7. Goldstein, T., Osher, S.: The split bregman method for l1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009) 8. Hansen, P.C., Nagy, J.G., O’leary, D.P.: Deblurring Images: Matrices, Spectra, and Filtering, vol. 3. Siam (2006) 9. He, L., Marquina, A., Osher, S.J.: Blind deconvolution using TV regularization and bregman iteration. Int. J. Imaging Syst. Technol. 15(1), 74–83 (2005) 10. Jin, C., Chen, C., Bu, J.: A new approach of blind image restoration. In: IEEE International Conference on Systems, Man and Cybernetics 2003, vol. 1, pp. 245– 250. IEEE (2003) 11. Kundur, D., Hatzinakos, D.: Blind image deconvolution. IEEE Signal Process. Mag. 13(3), 43–64 (1996) 12. Kundur, D., Hatzinakos, D.: A novel blind deconvolution scheme for image restoration using recursive filtering. IEEE Trans. Signal Process. 46(2), 375–390 (1998) 13. Lagendijk, R.L., Biemond, J., Boekee, D.E.: Identification and restoration of noisy blurred images using the expectation-maximization algorithm. IEEE Trans. Acoust. Speech Signal Process. 38(7), 1180–1191 (1990) 14. Li, W., Li, Q., Gong, W., Tang, S.: Total variation blind deconvolution employing split bregman iteration. J. Vis. Commun. Image Represent. 23(3), 409–417 (2012) 15. Liu, J., Shi, Y., Zhu, Y.: A fast and robust algorithm for image restoration with periodic boundary conditions. J. Comput. Anal. Appl. 17(3), 524–538 (2014) 16. Marquina, A.: Nonlinear inverse scale space methods for total variation blind deconvolution. SIAM J. Imaging Sci. 2(1), 64–83 (2009) 17. McCallum, B.C.: Blind deconvolution by simulated annealing. Opt. Commun. 75(2), 101–105 (1990) 18. Moffat, A.F.J.: A theoretical investigation of focal stellar images in the photographic emulsion and application to photographic photometry. Astron. Astrophys. 3(1), 455 (1969) 19. Nayakkankuppam, M.V., Venkatesh, U.V.: Deblurring the gaussian blur using a wavelet transform. Pattern Recognit. 28(7), 965–976 (1995) 20. Ng, M.K., Chan, R.H., Tang, W.C.: A fast algorithm for deblurring models with neumann boundary conditions. SIAM J. Sci. Comput. 21(3), 851–866 (1999) 21. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1), 259–268 (1992) 22. Shi, Y., Chang, Q.: Acceleration methods for image restoration problem with different boundary conditions. Appl. Numer. Math. 58(5), 602–614 (2008) 23. Shi, Y., Chang, Q.: Efficient algorithm for isotropic and anisotropic total variation deblurring and denoising. J. Appl. Math. 2013, 14 (2013). Article ID 797239 24. Wang, Z.F., Zhang, Z.J., Li, X.M., Tang, Y.D.: An adaptive image denoising algorithm based on SVD and energy minimization. J. Image Graph. 12(4), 603–607 (2007) 25. Xu, J., Chang, Q.: A robust algorithm for blind total variation restoration. Acta Math. Appl. Sinica Engl. Ser. 24(4), 681–690 (2008) 26. You, Y.L., Kaveh, M.: A regularization approach to joint blur identification and image restoration. IEEE Trans. Image Process. 5(3), 416–428 (1996)

Real-Time Multi-camera Video Stitching Based on Improved Optimal Stitch Line and Multi-resolution Fusion Dong-Bin Xu1,2(B) , He-Meng Tao3 , Jing Yu3 , and Chuang-Bai Xiao3 1

3

Research Institute of Highway Ministry of Transport, Beijing 100080, China 2 RIOH Transport Consultants LTD., Beijing 100191, China [email protected] College of Computer Science and Technology, Beijing University of Technology, Beijing 100124, China

Abstract. In this paper, we propose a multi-camera video stitching method based on an improved optimal stitch line and multi-resolution for real-time application. First, phase correlation is used to estimate overlapping fields between multiple videos, where SURF feature points are extracted for image registration to save computation time and improve the matching accuracy. Then, a fusion algorithm combining the improved optimal stitch line and multi-resolution algorithms is devised to improve visual effects by eliminating ghosting and any visible seams. In the fusion stage, GPU acceleration is employed to speed up the video stitching. Experiments show that the proposed algorithm has better and real-time performance compared with traditional video stitching methods. Keywords: Multi-camera stitching

1

· SURF feature · Stitch line · GPU

Introduction

Video stitching is one of the most popular topics in digital image processing and computer vision, where real-time video stitching still remains a challenge. Video images being stitched are often with significantly varying viewpoints and viewing directions [1]. In real scene, video stitching algorithms should also be sufficiently automatic, robust and fast. Video stitching generally falls into two categories, namely, video sequence stitching and multi-camera real-time video stitching. In video sequence stitching, image processing techniques are used to stitch an existing sequence of videos with the same view into a panorama video. Because of offline processing and allowing a long period of background processing, it usually gives satisfactory results but it takes too much time. Xu and Mulligan [2] attempted to stitch a 45-minute video sequence (1920 × 1080) from three cameras. Even with his improved video sequence stitching method, it takes five days to complete the stitching. Multicamera real-time video stitching aims to stitch videos collected from multiple live cameras in real-time and produces a panorama video. The stitching algorithms c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 124–133, 2017. https://doi.org/10.1007/978-3-319-71598-8_12

Real-Time Multi-camera Video Stitching

125

should be fast enough for real-time. For example, to outputs a 10-fps video, the fusion process must be completed within 100 ms. The paper mainly deals with the feature-based real-time stitching techniques for panorama videos, which involves the use of image processing and multipipeline parallel programming model. To overcome the seams in overlapping regions, a method based on improved optimal stitch line is proposed. Some algorithms are used to improve the real-time performance including matching points obtained from a circular region, multi-resolution fusion algorithms and especially GPU acceleration of CUDA [3] programming in fusion stage.

2

Determining Overlapping Regions

Most traditional SURF [4] algorithms are time-consuming because they detect features from the whole image. To solve this problem, the phase correlation algorithm is firstly used to calculate the approximate overlapping region between the two stitched images. The procedures of phase correlation are described as follows: If a displacement (△x, △y) exists between image I1 (x, y) and I2 (x, y), then the relations between them can be written as: I1 (x, y) = I2 (x − △x, y − △y)

(1)

After normalization, the cross power spectrum is: ∗ Iˆ1 (u, v)Iˆ2 (u, v) = e−j2π(u△x+v△y) ∗ |Iˆ1 (u, v)Iˆ2 (u, v)|

(2)

where Iˆ1 (u, v) and Iˆ2 (u, v) are the results of Fourier transform of I1 (x, y) and ∗ ∗ I2 (x, y),while Iˆ1 (u, v) and Iˆ2 (u, v) is the complex conjugate of I1 (x, y) and I2 (x, y). By performing inverse Fourier transform for Eq. (2), the results are shown as (3) δ(x − △x, y − △y) = F −1 [e−j2π(u△x+v△y) ] The position of the peak value of the function corresponds with the translational parameter (△x, △y) about the two images. If there is only a translational relationship between the two images, then the peak value of the function shows the extent of correlation between them, where the range of its value is [0, 1]. Through the translational parameters (△x, △y), the approximate overlapping region between the two images can be obtained. By detecting feature points only in the overlapping region, the feature points detection algorithm is accelerated because of fewer feature points being searched, and the accuracy is also improved in subsequent matching. After the images are matched, the RANSAC algorithm is used to eliminate mismatched pairs of points and calculate the coordinate transformation matrix between video images.

126

3 3.1

D.-B. Xu et al.

Image Fusion Exposure Adjustment

A problem of automatic image stitching is exposure differences between images, which will bring seams when the images are blended within overlapping regions. To eliminate exposure differences between the stitched images, a block-based exposure adjustment method is adopted, which is proposed by Uyttendaele in 2001 [5]. Image is divided into 32 × 32 blocks. Within each block, compute a quadratic transfer function in a least-squares sense, this image block to the composite of images overlapping this block. To smooth the variation in the transfer function distributions, use a combination of two techniques. First, average the functions in each patch with those of their neighbors, use an iterated separable kernel of (1/4, 1/2, 1/4), and typically use 2 iterations. Second, for each pixel, the results are blended by applying the transfer function from neighboring patches, which can be performed using bilinear interpolators as shown in Eq. (4). The results are the same as interpolating a separate function for each pixel, but since we implement the transfer function using a look-up table, the computational cost is much smaller. p(x, y) =

v=1 

Cv (

v=−1

u=1 x y  ) Cu ( )fm+u,n+v (p(x, y)) N u=−1 N

(4)

where p(x, y) is the pixel value at block location, N is the block size, Cn (x) is the bilinear coefficient, fu,v (x) is the transfer function of block u and v. 3.2

Optimal Stitch Line

The optimal stitch line [6–8] is based on a notion that there exists a stitch line in the overlapping region, which can be used to minimize the difference in both color and geometry between the two images, so that only pixels on one side of this stitch line are needed for producing a panorama image. Hence, the principle of finding an optimal stitch line is: E(x, y) = Ecolor (x, y)2 + Egeometry (x, y)

(5)

where Ecolor represents the color difference of pixels while Egeometry represents the geometric difference of pixels in the overlapping region. According to the principle of dynamic programming, the detailed steps are as follows: (1) Establish stitch lines with all pixels in the first row as the starting points. Calculate the intensity value E of each pixel. Then, the stitch line extends to the next row of pixels. (2) Compare the intensity values of three pixels in the next row neighboring to the current point of the stitch line. Take the pixels with the smallest intensity value as the direction of the extension. And then, this point becomes

Real-Time Multi-camera Video Stitching

127

the current point of the stitch line. Update the total intensity value of the stitch line and make the current point the one corresponding to the smallest intensity value of the next row. (3) Select a stitch line with the smallest total intensity value out of all stitch lines as the best stitch line. To reduce the visibility of the seam, we propose an improvement to the basic algorithm based on the optimal stitch line between the neighboring images I1 (x, y) and I2 (x, y). First, define binary images R1 (x, y) and R2 (x, y)denoting the initial weight matrix for the two neighboring images I1 (x, y) and I2 (x, y). Let the values of the pixels on the two sides of the stitch line be 1 and 0, respectively. Second, define a distance transform function to calculate the distances from all non-zero pixel points to the nearest zero pixel point, where p is the set of non-zero pixels and q is the set of zero pixels. dis(p(x1 , y1 ), q(x2 , y2 )) = |x1 − x2 | + |y1 − y2 |

(6)

Then, calculate the new transition fusion weight α1 (x, y) and α2 (x, y) for corresponding images I1 (x, y) and I2 (x, y), a threshold value ε ∈ (0, 1] is used to set the size of the smooth transitional zone. The equation is written:  α1 (x, y) = ε∗ R1 (x, y) if ε∗ R1 (x0 , y0 ) > 1, α1 (x0 , y0 ) = 1 (7) α2 (x, y) = ε∗ R2 (x, y) if ε∗ R2 (x0 , y0 ) > 1, α2 (x0 , y0 ) = 1 Finally, calculate the fused image: Ires =

α1 (x, y) ∗ I1 (x, y) + α2 (x, y) ∗ I2 (x, y) α1 (x, y) + α2 (x, y)

(8)

Bias in the registration process before stitching and object movements may lead to ghosting and distortion in the overlapping region. Using the optimal stitch line method can avoid generating a stitch line in the region with high contrast. Instead, the optimal stitch line often exists in a region with a smooth transition. Hence, pixels are selected for value assignment from a region on each side of the optimal stitch line, respectively, which helps avoid generating ghost. 3.3

Multi-resolution Image Fusion

To ensure that the transition in the overlapping region is as smooth as possible, the paper proposes multi-resolution fusion method based on Gaussian pyramid. Gaussian pyramid is a series of low-pass filtering images obtained by convoluting an image and a weight function. Let G0 be the initial image, then Gl (i, j) =

2 2  

m=−2 n=−2

w(m, n)Gl−1 (2i + m, 2j + n)

(9)

128

D.-B. Xu et al.

where 1 ≤ l ≤ N, 0 ≤ i ≤ Ri , 0 ≤ j ≤ Cl , Ri and Cl represent the numbers of rows and columns in level l, respectively. The two-dimensional discrete Gaussian convolution kernel function is denoted by w(m, n). In order to get the multi-band-pass filtered image required for multi-band fusion, we subtract the next level from each level of the pyramid. Since these images have different sampling precision, image interpolation is required to obtain new samples: G′l (i, j) = 4

2 2   2i + m 2j + n , ) ( 2 2 m=−2 n=−2

(10)

Suppose that the image on Level l of the Laplacian pyramid is Ll and the toplevel image is LN , then the formula for constructing the Laplacian pyramid is:  Ll = Gl − G′l+1 0≤l θbg P (yk = skin) > θs

(2)

here, θh , θbg and θs denotes the threshold values for hair, background, and skin respectively and are set to 0.8, 0.9 and 0.9 in our work. After these high confidence regions are selected, we use white scribbles to indicate foreground region, and black scribbles background region. As the methods are similar, we use hair region labeling as an example. The inscribed rectangles of the superpixels whose hair label likelihood P (yi = hair) surpasses the threshold value θh in set i are selected as the foreground regions. The inscribed rectangles of the superpixels from the other two sets are selected as the background regions. Then the fine hair segmentation result is obtained via employing automatic alpha matting [8]. The background and face region can be segmented in the same way by simply set the superpixels with high confidence of the target label as the foreground and others as the background.

4

Two-Tone Portrait Synthesis

This section is a detailed introduction of our two-tone portrait synthesis method. After the background, face and hair regions have been segmented, we handle different regions with different strategies and combine them together to form the final portrait.

Semantic Segmentation Based Automatic Two-Tone Portrait Synthesis

175

Background region. Background region includes not only the shooting scene but also the clothes. In our target style, portraits should have clean and white background, and the clothes region should be painted in black and white colors to depict its patterns. We first compute the per-plane composite gradient of the background region to obtain the edge map. Taking the two-tone style into consideration, we then take inversion of the edge map to generate a sketch in which deep colors denotes the edge of the image. We further enhance the sketch to darken the lines and colors using histogram equalization method. Then we use dynamic thresholding method to obtain the binary image. Hair region. Hair region is one of the most identifying part in the automatic synthesis of portrait, so the detection and representation of hair have great influence on resemblance. As hair region only contains one component and artists normally paint hair in black in two-tone style portraits, we simply binarize the extracted RGB hair region image. However, the binarized image usually comes with sawtooth effect around the edge, which is highly unlikely to appear in a freehand portrait. In order to alleviate this noise, we use gaussian filter to smooth the image and then posterize the image with specific levels to create fluidity for the edge. Face region. Face has very regular yet complicate structure, so we can not directly use binarization strategy to produce two-tone portrait as it would cause severe noise and lose some details, such as lines depicting chins and cheeks. For synthesis of face region, our method proceeds in the following steps: firstly, the face region is decomposed into components; then, these components are matched to the templates in the dictionary according to respective features; finally, we use a global model to organize the composition of the components.

Fig. 3. Example two-tone facial components templates

Two-tone templates. We have asked several professional artists to create black-and-white portraits for some photos using tablets, then manually decompose them into facial components, as shown in Fig. 3. Keypoints of these components are also manually labeled on the portraits. We then construct a dictionary including the example photo components and the corresponding templates pairs. Templates are usually drawn based on various distinct photos for a greater power of representation. A. Face decomposition After face region is segmented, we first use the active appearance model (AAM) [4] to localize the facial structure keypoints. These facial landmarks are

176

Z. Ma et al.

used to extract face regions of eyebrows, eyes, nose, mouth and face contour. These components are clipped from the image in a rectangular shape, while face contour is represented by the polygon defined from the landmarks. B. Component matching For each component extracted from the earlier step, we search the templatecomponent dictionary for the most similar realistic photo components. Since different facial components exhibit diverse visual features, we handle each kind of component with different matching strategies. After matching the input components with the dictionary components, we use their corresponding two-tone templates for the subsequent composition step. Eyebrows, Eyes and Nose. These components could be differentiated from each other in the perspective of shape and texture. The shape feature is represented using the length, width of the bounding box and its ratio. For eyebrows and eyes, the texture feature is obtained via using SIFT feature descriptor at the facial landmarks. For nose, we use RGB-SIFT descriptor to describe the color pattern change in this place. We denote the shape vector as S = [slbrow , srbrow , snose , · · ·], the texture vector as T = [tlbrow , trbrow , tnose , · · ·]. For a component of the input photo c and one of the dictionary c′ , we can define the distance function of their appearances as d (c, c′ ) = λ S − S ′  + (1 − λ) T − T ′ 

(3)

where λ is the weight for shape and texture when measuring different components. After we compute the shape vector S and the texture vector T for different components, we then find the most alike candidate from the template dictionary by minimizing the distance: (4) min d (c, cj ′ ) j

′

where cj is the jth component example in the template-component dictionary. Mouth. The appearance of mouth in portrait is largely determined by the subject’s expression. The mouth templates in our dictionary could be classified into 2 primary classes: open and shut, and then be subdivided into 4 classes: laugh and no expression under open category, smile and no expression under shut category. We train a two layer classifier to determine the expression on the subject’s face. Then we use feature matching to match the component to template in corresponding categories. Face contour. The shape of face contour can be perfectly represented simply by its landmarks. Besides shape, the texture feature should also be considered to distinguish single-chin from double-chin. The Hausdorff distance dH (VI , VD ) between the input face contour landmarks VI the components’ face contour landmarks VD from the dictionary are used to measure the shape dissimilarity.   (5) dH (VI , VD ) = max sup inf d (x, y) , sup inf d (x, y) x∈VI y∈VD

y∈VD x∈VI

Semantic Segmentation Based Automatic Two-Tone Portrait Synthesis

177

The texture distance dT (tI , tD ) are represented using SIFT descriptor, tI and tD denotes SIFT feature computed at keypoints for input face contour and the face contour from the dictionary. We then select the most similar face contour component by minimizing the distance: min dH (VI , VD ) + dT (tI , tD )

(6)

C. Composition of facial component After obtaining the best matching template for each input component, we compose them together to synthesize the face region for the final output. We use a global model to organize the arrangement of each component on the canvas. Our global model includes a set of control points {Plbrow , Pleye , Prbrow , Pnose , · · ·} for arranging the templates and resizing and warping parameters for templates. Previous studies have demonstrated that humans have a preference for symmetric faces. In the meantime, tests have revealed that the original slightly asymmetric faces were rated as more attractive than completely symmetric versions [10]. Previous work [3, 20] focus on learning the styles that the artists use to arrange the face components, however, here we emphasize more on the balance between high resemblance to the input photos and the attractiveness of the portraits by building symmetry. For a symmetric structure, our model would set up a coordinate system for the face and model the relative placement of the input face components and then adjust them using axis-symmetry protocols to determine where to put the templates. To reach the high resemblance of the original slightly asymmetric face, we warp and resize the templates in accordance with the input components. D. Additional Component In compositional models, ears are always neglected. The existence of ears could be detected by examining whether there is a sudden change area of the segmented face region mask, and if there is an outward curve with certain length in edge map, we set the ear template there. By this step, the face region is synthesized. Then we combine the processed background region and hair region together to synthesize the final output portrait.

5

Experimental Results

In this section, we first show that our segmentation method can reach a very high accuracy on labeled CUHK dataset (Section-A). We then demonstrate that our method can generate vivid two-tone portraits with high resemblance and freehand-like features (Section-B). A. Semantic segmentation Datasets. We use the public dataset from Kae et al. [7], consisting of 2927 LFW (funneled) images, and supplement the dataset with manually labeled photos from the CUHK student database (see Fig. 4).

178

Z. Ma et al.

Fig. 4. Labeled CUHK dateset.

Evaluation. The segmentation accuracy of our method tested on LFW (funneled) images has reached 94.25%, compared to 90.7% [7] and 92.69% [2]. The segmentation accuracy tested on CUHK photos with classifiers trained on our manually labeled CUHK student database are over 96%. Figure 5 shows some segmentation results on CUHK dataset. The comparison of automatically labeled results and manually labeled results shows that our method can segment background, hair and skin region accurately for the next synthesis phase.

Fig. 5. Segmentation result for labeled CUHK dataset. Top row: segmentation result. Bottom row: manually labeled result.

Fig. 6. Comparison with previous methods for sketch and cartoon synthesis. From left to right: (a) input photo; (b) the result of [3]; (c) the result of [20]; (d) our result.

B. Portrait Synthesis Our system aims at synthesizing two-tone styles portraits with both attractiveness and resemblance. In Fig. 6, we compare with the component-based methods for synthesizing profile sketch and cartoon face. These methods can generate stylized portraits yet not balance the resemblance and attractiveness of the output result. For example, The nose and mouth parts in Fig. 6(b) fail to reflect the distinct feature of the original input photo. Figure 6(c) has very stylistic

Semantic Segmentation Based Automatic Two-Tone Portrait Synthesis

179

appearance, but their over dependence on templates causes the lack of similarity in hair region. In Fig. 7, we compare with the shading sketch synthesis result. We can see that these methods would easily cause noise and blurring in the result, which should not be occurred in freehand portraits.

Fig. 7. Comparison with previous methods for shading sketch synthesis. From left to right: (a) input photo; (b) the result of [21]; (c) the result of [18]; (d) the result of [19]; (e) our result.

Fig. 8. Two-tone portraits generated by our system.

Figure 8 demonstrates some two-tone portrait results rendered by our system. We can see that our method can render two-tone portraits with distinct freehand features such as clear backgrounds and continuous lines. Also, it can maintain distinct figure features such as hair style, expression and face contour, which guaranteed the resemblance between the input photo and the output synthesis portraits. Third, our global model can synthesize portraits with symmetric facial structure meanwhile preserve the similarity to the original slightly asymmetrical face. More results are available online: http://www.ihitworld.com/ZQM/T2P. zip.

6

Conclusion and Future Work

In this paper, we have presented a framework for synthesizing two-tone portraits based on semantic segmentation. In the segmentation phase, our method based on multiple segmentations and image matting can precisely divide the input

180

Z. Ma et al.

photo into background, hair and skin regions. For the after-segmentation synthesis stage, we handle different regions with appropriate methods respectively. For background and hair regions, our processing strategy can render clean and smooth results with freehand-like features. For face region, we select the best match for each input component and use a global model to warp and arrange them to reach the balance between resemblance and attractiveness. Our experimental results show that our system is capable of synthesizing visually desired two-tone portraits with distinct freehand-like features. The main limitations of this method are the segmentation accuracy and the lack of diversity for component templates. In future work, we plan to improve the segmentation accuracy via existed deep neural networks and enrich the component dictionary with more distinctive templates. We are also trying to extend this after-segmentation synthesis method to other portrait styles, such as shading sketch, by exploring proper strategies to process hair and background region into pencil sketch style. Besides, synthesizing two-tone portraits with complex backgrounds is another worthy topic, for example, for photographs taken at landscapes, people would like to have the complete photo in two-tone style, not only the face part. Lastly, to realize the application of this framework in industrial production, the speed of segmentation is another place to improve in the future. Acknowledgement. This work was supported in part by the National Natural Science Foundation of China (under Grants 61501339, 61671339, 61432014, U1605252, 61601158, and 61602355), in part by Young Elite Scientists Sponsorship Program by CAST (under Grant 2016QNRC001), in part by Young Talent fund of University Association for Science and Technology in Shaanxi, China, in part by the Fundamental Research Funds for the Central Universities under Grant JB160104, in part by the Program for Changjiang Scholars, in part by the Leading Talent of Technological Innovation of Ten-Thousands Talents Program under Grant CS31117200001, in part by the China Post-Doctoral Science Foundation under Grants 2015M580818 and 2016T90893, and in part by the Shaanxi Province Post-Doctoral Science Foundation.

References 1. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., S¨ usstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012) 2. Ahn, I., Kim, C.: Face and hair region labeling using semi-supervised spectral clustering-based multiple segmentations. IEEE Trans. Multimedia 18(7), 1414– 1421 (2016) 3. Chen, H., Liu, Z., Rose, C., Xu, Y., Shum, H.Y., Salesin, D.: Example-based composite sketching of human portraits. In: Proceedings of the 3rd International Symposium on Non-Photorealistic Animation and Rendering, pp. 95–153. ACM (2004) 4. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001) 5. Gastal, E.S., Oliveira, M.M.: Domain transform for edge-aware image and video processing. In: ACM Transactions on Graphics (ToG), vol. 30, p. 69. ACM (2011) 6. Gooch, B., Reinhard, E., Gooch, A.: Human facial illustrations: creation and psychophysical evaluation. ACM Trans. Graph. (TOG) 23(1), 27–44 (2004)

Semantic Segmentation Based Automatic Two-Tone Portrait Synthesis

181

7. Kae, A., Sohn, K., Lee, H., Learned-Miller, E.: Augmenting CRFS with boltzmann machine shape priors for image labeling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2019–2026 (2013) 8. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008) 9. Meng, M., Zhao, M., Zhu, S.C.: Artistic paper-cut of human portraits. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 931–934. ACM (2010) 10. Mentus, T., Markovi´c, S.: Effects of symmetry and familiarity on the attractiveness of human faces. Psihologija 49(3), 301–311 (2016) 11. Rosin, P.L., Lai, Y.K.: Non-photorealistic rendering of portraits. In: Proceedings of the Workshop on Computational Aesthetics, pp. 159–170. Eurographics Association (2015) 12. Selim, A., Elgharib, M., Doyle, L.: Painting style transfer for head portraits using convolutional neural networks. ACM Trans. Graph. (TOG) 35(4), 129 (2016) 13. Tang, X., Wang, X.: Face sketch synthesis and recognition. In: Proceedings of the Ninth IEEE International Conference on Computer Vision, pp. 687–694. IEEE (2003) 14. Wang, N., Gao, X., Sun, L., Li, J.: Anchored neighborhood index for face sketch synthesis. IEEE Trans. Circ. Syst. Video Technol. (2017) 15. Wang, N., Gao, X., Sun, L., Li, J.: Bayesian face sketch synthesis. IEEE Trans. Image Process. 26(3), 1264–1274 (2017) 16. Wang, N., Tao, D., Gao, X., Li, X., Li, J.: Transductive face sketch-photo synthesis. IEEE Trans. Neural Netw. Learn. Syst. 24(9), 1364–1376 (2013) 17. Xu, Z., Chen, H., Zhu, S.C., Luo, J.: A hierarchical compositional model for face representation and sketching. IEEE Trans. Pattern Anal. Mach. Intell. 30(6), 955– 969 (2008) 18. Zhang, L., Lin, L., Wu, X., Ding, S., Zhang, L.: End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 627–634. ACM (2015) 19. Zhang, S., Gao, X., Wang, N., Li, J., Zhang, M.: Face sketch synthesis via sparse representation-based greedy search. IEEE Trans. Image Process. 24(8), 2466–2477 (2015) 20. Zhang, Y., Dong, W., Ma, C., Mei, X., Li, K., Huang, F., Hu, B.G., Deussen, O.: Data-driven synthesis of cartoon faces using different styles. IEEE Trans. Image Process. 26(1), 464–478 (2017) 21. Zhou, H., Kuang, Z., Wong, K.Y.K.: Markov weight fields for face sketch synthesis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1091–1097. IEEE (2012)

Parameters Sharing Multi-items Non-parametric Factor Microfacet Model for Isotropic and Anisotropic BRDFs Junkai Peng1,2(B) , Changwen Zheng1 , and Pin Lv1 1

Science and Technology on Integrated Infomation System Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China {junkai2015,changwen,lvpin}@iscas.ac.cn 2 University of Chinese Academy of Sciences, Beijing, China

Abstract. The reflection model of object surfaces is an essential part in photorealistic rendering. While analytical models are unable to express all significant effects of all materials, we turn to data-driven models, which, however, cost a large amount of memory and more computational resources. The non-parametric factor microfacet model designed by Bagher et al. [1] is intended to solve these problems. In this paper, we present a new non-parametric factor microfacet model which has triple specular lobes but retains the original number of parameters by sharing factors among three color channels. The fitting method called AWLS is also extended to solve for the G factor, which makes the fitting process more robust. Moreover, we use the D factor of our model for importance sampling as in the case of analytical models and find it effective for the specular materials. Finally we generalize our model and our fitting method to fit the 150 anisotropic materials. With only 2010 parameters (8KB), it can reconstruct the original data (2 MB) well, which further proves the expressiveness of our microfacet model. Keywords: BRDF compression and factorization Isotropic · Anisotropic · Importance sampling

1

· Microfacet model

Introduction

In computer graphics, bidirectional reflection distribution function (BRDF) [2] has been extensively used to represent material appearance. For a specific wavelength λ, BRDF is a 4D function fλ (θi , φi , θo , φo ), which returns the ratio of outgoing radiance to the incoming irradiance incident on the surface. All the notations used in this paper are shown in Fig. 1. BRDF models are simply divided into analytical models and data-driven models. Analytical models give exact analytical forms with only a few parameters to fit different materials. The microfacet model is one kind of analytical models, which is derived from the microfacet theory. The microfacet theory [3] assumes that rough surface consists of adequate microfacets, which have the c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 182–193, 2017. https://doi.org/10.1007/978-3-319-71598-8_17

Parameters Sharing Multi-items Non-parametric Factor Microfacet Model

183

Z N L θd θ h θi

H θo

V

dA −φi

φh

T

S φo

Y

X

Fig. 1. Notations used in this paper.

same reflection properties and whose orientations obey some kind of distribution. For a given wavelength λ (or color channel), the basic structure of the microfacet model is simple: ρ(θh , θd , φd ) = d + s(

D(θh , φh )F (θd )G2 (θi , θo ) ) . cos θi cos θo

(1)

If both the index of refraction and the extinction coefficient of the material of interest are known, then F (θd ) is determined. The indices of refraction and the extinction coefficients of some materials can be checked in the handbook [4]. Additionally, the masking (or shadowing) factor G can be derived from D, which is first proposed by Smith et al. [5] and generalized by Brown et al. [6] and Walter et al. [7]. If a simple kind of masking-shadowing function G2 , e.g. G2 (θi , θo ) = G(θi )G(θo ), is chosen, then the biggest difference among the microfacet models is normal distribution function D(θh , φh ). There are lots of choices for D(θh , φh ). Some of the most commonly used distributions can be found in [7–12]. It should be noted that the GGX distribution is the same as the Trowbridge-Reitz distribution. The SGD distribution is a mix of the Trowbridge-Reitz distribution and the Beckmann-Spizzichino distribution. Generally speaking, the shape of the Blinn-Phong distribution is very close to the Beckmann-Spizzichino’s. The Trowbridge-Reitz, SGD and ABC distributions have a narrower peak with a stronger tail but the ABC distribution is only suitable for glossy surfaces. By observing measured data, we can design a more accurate normal distribution function. Actually, with the method Ashikmin et al. [13] introduced, we can arbitrarily design normal distribution functions regardless of whether the corresponding materials exist or not. However, all the factors derived above are subject to some assumptions, which limit their capability to express some special reflection effects as well as the range of materials their corresponding models suitable to be used. In order to express all the reflection effects and not be bounded to the categories of materials, we can directly use the measured reflection data. However, acquiring accurate and dense

184

J. Peng et al.

measured reflection data is a challenging and time-consuming work. Fortunately, there is a large enough isotropic BRDF database called MERL [14], which has been analyzed by a lot of researchers. Moreover, UTIA1 BRDF database [15] contains measured reflection data for 150 anisotropic materials. Although it is not that dense, it still can be directly used in a renderer with an appropriate interpolation method. However, directly using the measured data not only needs a large amount of memory storage, but also loses the flexibility for users to edit the material appearance and the simplicity for importance sampling. One possible way to solve the problems aforementioned is using 1D arrays to substitute all the factors in the basic structure of the microfacet model. The model after substitution is called non-parametric factor microfacet model in Bagher et al. [1]. As for G2 , they used the simplest but inaccurate one, G2 = G(θi )G(θo ). Bagher et al. [1] used two slightly different models, the best of which was called Independent-G model. As its name suggests, the G factor is independent and is not derived from the D factor in the Independent-G model. Following the format of the MERL database, each of the factors D(θh ), F (θd ) and G(θi ) (or G(θo )), has 90 elements. The fitting objective is  wj (ρ(θh , θd , φd )j − ρ∗j )2 (2) E= j

ρ∗j

where is a BRDF measurement and ωj is the compressive weight. Those who are interested in the weighting scheme can find detailed formulations in [1]. By fitting the MERL measured data, each element of the 1D array and the other two coefficients can be determined. Then using the Independent-G model, original reflection data of each material is approximated with 816 floating-point parameters. It should also be noticed that the Independent-G model is always better than the compared analytical models. So we only use the IndependentG model to compare with our model in the experiments. In addition, we also compare our model with the so-called Bivariate model, which use 10× more parameters and each parameter has no physical meaning. Taking the images rendered with the original reflection data as the reference images, the PSNRs between the images rendered with the Independent-G model and the reference images are shown in Fig. 8 in [1]. Although the result seems good enough, there is still much room for improvement. We change the structure of the Independent-G model from a single specular lobe to three specular lobes. However, by sharing factors among three color channels, our model only uses six extra parameters. Moreover, sharing factors conforms to the microfacet theory, which makes our model more explainable and intuitive. More importantly, it’s more suitable for user-editing and importance sampling. In addition, we still can use the simple and easily implemented fitting method which is called alternating weighted least-squares (AWLS) in Bagher et al. [1] to fit the data. The AWLS repeatedly updates each factor in sequence until convergence. Its basic idea is finding the zero point(s) of the first derivative of the fitting objective 2 in order to get the 1

http://btf.utia.cas.cz/.

Parameters Sharing Multi-items Non-parametric Factor Microfacet Model

185

minimum of it. It is easy to prove that its second derivative is positive so the zero point must be the minimum. We do not introduce the AWLS in detail here, but in the next section, we extend it to solve for the G factor. Bagher et al. [1] thought the AWLS was not suitable for the G factor because of the non-trival factor dependency and used the GSS (golden section search) to guarantee to reduce the fitting objective. After extended, the AWLS is also able to reduce the fitting objective steadily while updating the elements of the G factor without using GSS. Finally, microfacet theory does not confine to isotropic materials. So we also use our modified model to fit the anisotropic data. In the following sections, we first introduce our models for the isotropic and anisotropic measured data respectively. Then we talk about the extended AWLS for the G factor and the simple importance sampling method which helps dramatically decrease the number of samples needed to render some specular materials. After that, we show the fitting results and further analyses about the resulting parameters. The limitations of our model are concluded in the end of the paper.

2 2.1

Parameters Sharing Multi-items Model Our Model for the Isotropic Materials

In the Independent-G model, the D, F and G factors are different for each color channel. Although such a model can fit the measured data well, it loses the original physical meaning. For example, the D factor represents normal distribution of object surfaces. Therefore, for different color channels (or wavelengths), the D factor is the same. The same conclusion can also apply to the G factor. Only the F factor, the fresnel factor, is dependent on the wavelength of the incident light so it is different for different color channels. Therefore, we share the D and G factors among different color channels, which helps reduce the number of parameters. Furthermore, experiments conducted by Ngan et al. [16] indicated that two specular lobes helped reduce the fitting error. Lafortune et al. [10] used three specular lobes but Ngan et al. [16] indicated that it made the fitting process very unstable. However, with the extended fitting method AWLS, it is not a problem. In order to compare with the Independent-G model fairly, our model uses three specular lobes, which only adds 6 unavoidable parameters. Obviously, the total number of parameters our model uses is 3 + 3 ∗ 3 + 90 ∗ 3 ∗ 3 = 822. The resulting model is ρC (θh , θd , φd ) = dC +

m B, R > B> G, R > B = G, R = B > G, G > R >B, G > R = B, G > B > R, G > B = R, G = B > R, B > R > G, G > B = R, B > G > R, G > B = R, B = G = R. In fact, in order to simplify the discussion, the sixteen relations can be classified into the following six types despite of the special equal case B = G = R: Type I: R  G  B; Type II: R  B  G; Type III: G  R  B; Type IV: G  B R; Type V: B  R  G; Type VI: B  G  R. In natural case, all kinds of practical colors can be grouped into one of the above six cases (see Fig. 3 for a classical example of six types of color from Type I to Type VI and we can find six classical relations between the intensity of R, G and B). Inspired by the sparse modes of colors in RGB space, we can derive some extremal cases. So, from the six types, some extremal cases can be derived. We can deduce the following cases:

1.5

Intensity of Three Channels

R G B

II

I

III

V

IV

VI

1

0.5

0 0

84

168

252

336

420

500

Column

Fig. 3. A classical example of six types of color from Type I to Type VI. (Color figure online)

Type I: R  G  B ) R  G >> B (Sparse Color Mode 1) and R  G  B (Sparse Color Mode 2); Type II: R  B  G ) R  B>> G (Sparse Color Mode 3) and R B  G (Sparse Color Mode 4); Type III: G  R  B ) G  R> > B (Sparse Color Mode 5) and G R  B (Sparse Color Mode 6); Type IV: G  B  R ) G  B >> R (Sparse Color Mode 7) and G B  R (Sparse Color Mode 8); Type V: B  R  G ) B  R>> G (Sparse Color Mode 9) and B R  G (Sparse Color Mode 10); Type VI: B  G  R ) B  G >> R (Sparse Color Mode 11) and B G  R (Sparse Color Mode 12). We call the above cases as the twelve extremal recolored ones if the source color image is manipulated to these twelve degrees in recoloring. Here, the so called “sparse” means that this mode denotes all the colors under such case. For example, “Sparse Color Mode 1” means all the cases of pixels satisfying R  G >> B. So, given a source color image, we can recolor it to the twelve extremal cases despite of its original type by optimization. We set the recolored image (target image) as frec ðx; yÞ ¼ fRrec ðx; yÞ; Grec ðx; yÞ; Brec ðx; yÞg: Thus, we recolor the source color image (the given color image arbitrarily) to get the target image (e.g., of Sparse Color Mode 2) by optimizing the following equation

220

G. Xu et al.

arg min

kf ðx; yÞ

frec ðx; yÞk þ k1 kGrec ðx; yÞk þ k2 kBrec ðx; yÞk þ k3 krf ðx; yÞ

rfrec ðx; yÞk;

frec ðx;yÞ

s:t:

ð1Þ

Rrec  Grec  Brec ;

where “kk” denotesthe l2 -norm. “r” is the gradient operator. k1 ; k2 ; k3 are the regularization parameters for the second, the third and the fourth terms in the optimization Eq. (1) respectively used for balance between different terms. In Eq. (1), the first term kf ðx; yÞ frec ðx; yÞk is employed to preserve the global intensity of the target image as close to the source color image as possible. When the global intensity changes, the naturalness of the color image will change as shown in above, i.e., one of the three rules. Consequently we maybe loss the sense of recoloring. Generally speaking, in recoloring the global intensity of the recolored target color image cannot change two much compared with the source color image. The second term and the third term in Eq. (1) are employed for the aim of satisfying the constraint condition Rrec  Grec  Brec , which corresponds with Sparse Color Mode 2. In addition, we generally set k1  k2 because Grec  Brec . On the other hand, since Rrec  ðGrec ; Brec Þ is needed, it is natural to minimize Grec and Brec under the constraint condition. In this paper, in order to simplify our computation, we set k1 ¼ k2 and find that the results still satisfy our needs as well. The fourth term in Eq. (1) is employed for the aim of texture and gradient preserving. Moreover, this term plays the important role of preserving reflection and irradiance (e.g., the reflection and irradiance in peppers of Fig. 5). So, we need to optimize Eq. (2) to obtain the extremal case of Sparse Color Mode 2. arg min

kf ðx; yÞ

frec ðx; yÞk þ k1 ðkGrec ðx; yÞk þ kBrec ðx; yÞkÞ þ k2 krf ðx; yÞ

rfrec ðx; yÞk;

frec ðx;yÞ

s:t:

ð2Þ

Rrec  Grec  Brec :

Similarly, we need to optimize the following Eq. (3) to obtain the extremal case of Sparse Color Mode 1. arg min

kf ðx; yÞ

frec ðx; yÞk þ k1 kBrec ðx; yÞk þ k2 krf ðx; yÞ

frec ðx;yÞ

s:t:

Rrec  Grec  Brec :

rfrec ðx; yÞk;

ð3Þ

The other extremal cases of sparse modes can be estimated in the same manner. Finally, after optimization (see computation details in Sect. 3.5) we obtain twelve extremal cases of recolored versions, like the small recolored images for fast reviewing as shown in Fig. 2, from the given target color image. 3.3

Refining the Classification in Mode Types

However, in many cases there are some pixels (especially for the pixels near the edges of different colors) that are hard to classify although they satisfy one of the six types based on the pixel intensity. When maxfjRðx; yÞ Gðx; yÞj; jBðx; yÞ Gðx; yÞj; jRðx; yÞ Bðx; yÞjg  l (l is a given small number, and in this paper we take l ¼ 0:02  0:10), we use K-NN

Lazy Recoloring

221

algorithm (K-Nearest Neighbors algorithm) [42] to classify the pixel ðx; yÞ to one of the six types by arg min Label

XK qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  2ffi     Label Label 2 Label 2 ; x x þ k f ðx; yÞ f ðx; yÞ þ y y l l l l¼1

ð4Þ

where Label ¼ Type I, Type II,…, Type VI and K is the number of neighbors (in this paper we take K = 3*9) and k is the balance parameter between the position distances and intensity distances. 3.4

Consecutive Recoloring and Transition Number Control

Once the two favorite recolored images are determined by users, how to create the consecutive recoloring series of images including the number of the transition recolored images will be discussed in this section. We set the two favorite recolored images as 1 2 ðx; yÞ; frec ðx; yÞ with the original source color image f ðx; yÞ. frec 1 First, we transform the two images to HSV space from RGB space frec ðx; yÞ ) 1 1 1 2 2 2 2 Hrec ðx; yÞ; Srec ðx; yÞ; Vrec ðx; yÞ, frec ðx; yÞ ) Hrec ðx; yÞ; Srec ðx; yÞ; Vrec ðx; yÞ by V ¼ maxfR; G; Bg; S ¼

8

V minfR; G; Bg > > > > 60ðR GÞ > > : 240 þ V minfR; G; Bg 8 > > > > > > >
0 0, x ≤ 0

(5)

And then the clockwise direction of an 8-bit binary number is calculated, corre‐ sponding to a binary mode for every pixel. The following formula for each symbol function is converted to a decimal number, which describes the local image texture. The LBP code value of the spatial structure characteristic is expressed as: ) ∑P ( LBP xc , yc = s(gi − gc )2i i=0

(6)

With the increase of the neighborhood size, the pattern of LBP feature would grow exponentially. It is more unfavorable to extract texture features when increasing more binary patterns. For the purpose of solving this problem, Ojala [2] adopted “uniform patterns” to improve LBP features. It can be expressed as LBPu2 when the binary P,R numbers corresponding of local binary mode change from 0 to 1 or from 1 to 0 up to twice [6]. So it is called a uniform pattern class. Except for uniform patterns, other modal classes are classified as another. When using uniform mode and using LBPu2 operator P,R for LBP coding, its expression is: ( ) LBP xc , yc =

{ ∑P

( ) s gi − gc 2i if U(LBP) ≤ 2 p+1 otherwise i=0

(7)

The uniform pattern accounts for more than 90% of all modes when using (8,1) neighborhood and accounts for more than 70% in the (16,2) neighborhood. The exper‐ imental results show that the uniform patterns can effectively describe most of the images of the texture features, and can greatly reduce the number of features.

Similar Trademark Image Retrieval Integrating LBP and CNN

235

So we choose uniform LBP which can reduce the dimension of features, the number of pattern drops from the original 2p down to p(p + 1) + 2 categories. The Uniform LBP improves the classification ability and enhances robustness, but meanwhile reduces the impact of high-frequency noise and vector dimension. 2.3 Trademark Retrieval After that, we extract LBP features based on the intermediate results of CNN, the more applied texture features can be extracted which combine the advantages with CNN and uniform LBP. Principal Component Analysis (PCA) is one of the most commonly used dimen‐ sionality reduction methods. PCA transforms the multivariable problem in high-dimen‐ sional space into low-dimensional space, forming a new minority of variables. This approach can reduce the dimensions of multivariable data system, but also simplify statistics of system variables. In our paper, we put forward a new method to extract Uniform LBP features based on CNN. Firstly, we use the pre-trained CNN model, i.e., imagenet-vgg-f [1], to extract features by its full connection shown in Fig. 1. We send a trademark image into the net. After convolutions and max-pooling, features can be extracted through full connection. Then we extract the features of the penultimate layer (The layer marked by the red circle in Fig. 1). Although the retrieval results of this method is similar to human vision than other traditional methods, this method does not apply to the situation that similar trade‐ mark images which have different background color. Furthermore, this approach will be affected by the overall graphics structure and ignore the local texture similarity. So we put forward an improved method to optimize it.

Fig. 1. Extracting features from full connection

236

T. Lan et al.

The proposed method is extracting Uniform LBP features from feature maps of every convolutions shown in Fig. 2. Following the same procedure of the first method, we send a trademark image into the net, but now we extract feature maps from every convo‐ lution layers respectively. The LBP algorithm is based on images in which the value of each pixel should be 0 to 255 while the value of feature maps is not in this range. So we need to normalize the value of feature maps into 0 to 255. Then extract Uniform LBP features from feature maps of five convolutions, which we choose the number of neighbor pixel is 8 and the value of radius is 1. Then we cascade LBP features of the same convolution so that we can get 5 LBP features by five convolution layers. After that we reduce the dimensions of cascade LBP features by Principal Component Anal‐ ysis (PCA). Through experiments, we select which is the best LBP features of different convolutions. The concrete flow chart of this process is shown in Fig. 3.

Fig. 2. Extracting Uniform LBP features from feature maps

Similar Trademark Image Retrieval Integrating LBP and CNN

237

Send images into the CNN

Extract Uniform LBP features from feature maps of every convolution layers

Reduce the dimension of feature vectors by PCA

Choose the best performance vector of different layers as retrieval features

Use the features to retrieve

Fig. 3. The concrete steps of extracting Uniform LBP features by our proposed method

3

Experimental Results and Analyses

3.1 Experimental Dataset In our experiments, we collect 7,139 trademark images including some transformed images by artificial process, cut, rotation, etc. In this way, we collect and create 317 similar trademark groups and every group has two similar trademarks at least. We use these 317 groups as out test set. Figure 4 is shown us a part of similar trademark dataset. In order to verify the effectiveness of our methods, we also have experiments in METU dataset [5] which contains about 930,328 trademark images. The test set of METU dataset contains 32 sets including 10 similar trademark images per set, which showed in Fig. 5 partially.

Fig. 4. Some samples of similar trademark

Fig. 5. Partial test sets of METU dataset

238

T. Lan et al.

Owing to the size of input image must be fixed to 224 × 224, while collected trade‐ mark images have different sizes. If we normalize the size of the image directly, some of geometries will be distorted. So we should firstly fill the image into a square, then normalize the size of the image into 224 × 224. 3.2 Feature Extraction In our experiment, we use pre-trained CNN model, i.e., imagenet-vgg-f [1], to extract the images features. Firstly, we feed all the trademark images to the imagenet–vgg-f model to extract their features. Then we feed the query image to the imagenet-vgg-f to extract its features. Finally, we measure the similarity distance between features of query image and features of every trademark image. Table 1 shown the structure and concrete parameters of imagenet-vgg-f model. Table 1. The structure of imagenet-vgg-f [1] Arch. CNN-F

conv1 64@ 11 × 11 st.1 pad 2 × 2pool

conv2 256@

conv3 256@

conv4 256@

conv5 256@

5×5

3×3

3×3

3×3

st. 1 pad 2 × 2pool

st. 1 pad 1

st. 1 pad 1

-

-

st. 1 pad 1 × 2pool

full6 4096 dropout

full7 4096 dropout

full8 1000 softmax

CNN contains convolutions, full connections and classification. In this part, we have tried two methods to extract features by CNN: 1. extracting features through the full connection of CNN as shown in Fig. 1; 2. extracting Uniform LBP features from every feature maps of per convolution as shown in Fig. 2. We use the first method as shown in Fig. 1 to get a 4096 dimensions vector from the full connection as the feature of the input trademark image, and use our proposed method as shown in Fig. 2 to extract Uniform LBP features from 5 convolutions, then we reduce the dimension of features by PCA. In CNN, we can extract primary visual features from the first fewer layers (inflection point, end point, corner) and extract more advanced features (texture, global shape) from the subsequent layers. Therefore, we consider that we can extract LBP features through the subsequent layers. Through a large number of experimental verification, we find the features from the forth convolution can enhance the robustness of LBP and can be much closer to human vision, so we choose forth convolution to extract LBP features. Because the forth convolution has 256 feature maps, we extract LBP features of every feature map so that we can get 256 × 59 dimensions vector per image. Afterwards, through PCA the dimensions of feature can be reduced to 6224. 3.3 Similarity Measurement The essence of the image similarity calculation is the calculation of the similarity between the corresponding features of the image. The feature of the image is expressed

Similar Trademark Image Retrieval Integrating LBP and CNN

239

by the vector, and the similarity between the two images is calculated by calculating the distance between the two vectors. In this experiment, we use Euclidean distance to measure it. Let a and b be two n-dimensional vectors, ai and bi are the values on the i-th dimension of the vector, and the formula of the Euclidean distance can be expressed as: dE (A, B) = (

∑n

i=1

|a − b |2 )1∕2 i| | i

(8)

3.4 Experimental Results In this section, firstly we test our proposed method in our own dataset. We extract features from the full connection and obtain a 4096-dimensional vector for every trade‐ mark image. The Fig. 5 (a), (b) and (c) are shown to us the retrieval consequent of this method. The query image is the first image of every results. Compared to the groundtruth in Fig. 4, the Fig. 6(a) indicates that features extracted by CNN could get better consequences which is close to human vision. However, the Fig. 6(b) and (c) show that this method cannot adjust the effect with background color, and focus on the overall shape but ignore the local texture similarity.

Fig. 6. Retrieval results by features from CNN full connection

In order to solve these problems above, we seriously consider the examples which cannot be searched in a good way and find that these examples which have poor results by CNN can be retrieve excellently by LBP features, because the pairs of these trademark images have the most of the same texture. The dimension of LBP feature is much smaller than the dimension of CNN feature extracted from full connection, so integrating these two features directly is not feasible. In view of this situation, we try our new method: extracting LBP feature from feature maps of per convolution. The Fig. 7(a), (b) and (c) show us the retrieval consequent of the proposed method. The query image is also the first image of every result which is the same test image to Fig. 6(a), (b) and (c). Comparing the two sets of experiment between Figs. 6 and 7, we can find that the result of Fig. 7(a) keeps the same good retrieval accuracy to the results of Fig. 6(a). And Fig. 7(b) and (c) get the favorable results that reflect this proposed method can solve the drawback of using CNN features from the full connection, which will be affected by background color and pay little attention to local texture.

240

T. Lan et al.

Fig. 7. Retrieval results by uniform LBP features from CNN convolutions

It is worth noting that the situations of Fig. 7(b) and (c), which changes the back‐ ground color or cut a part of original trademark, are the most common forms of infringe‐ ment of trademark copyright. So the method we proposed not only can retrieve similar trademark images as same as human vision, but also can credibly search the trademark which is suspected to plagiarism. 3.5 Evaluation In order to validate the effectiveness and accuracy of the proposed method quantitatively, we use Recall (R), Precision (P) and F-Measure to describe it. They are defined as:

Recall =

The number of associated images in the output × 100% The number of associated images in the database

(9)

The number of associated images in the output × 100% The number of images retrieved for all outputs

(10)

Precision =

F − Measure = 2 ×

Precision × Recall Precision + Recall

(11)

We test our proposed method in our trademark database, because the number of every similar trademark group is no more than 10. We set the total number of retrieved image is 10. In Table 2, we show that our proposed method compared with other common methods. Every data in Table 2 is an average value. Table 2. Recall, Precision and F-Measure of compared methods Methods SIFT HoG LBP Features from Full Connection Proposed Method

Recall 71.54% 63.12% 65.83% 79.36%

Precision 36.58% 33.87% 32.11% 39.01%

F-Measure 0.48 0.43 0.43 0.50

89.63%

45.34%

0.59

Similar Trademark Image Retrieval Integrating LBP and CNN

241

From Table 2 we can concern that our proposed method has much better performance than other traditional methods in the Table 2. And the recall, precision, and F-measure of our method are all higher than any other methods. Through the experimental results of METU dataset tested by Tursun [5], we can obtain the PVR curve of our proposed method on METU dataset which showed in Fig. 8. As can be seen from Fig. 8, our proposed method gets better retrieval results than any other methods when the recall is less than 0.5; when the recall is greater than 0.5, our method. Our method is probably the same as the best method (SIFT) for retrieval results. In summary, our approach is shown to be effective in large-scale trademark image retrieval.

Fig. 8. Retrieval results in METU dataset

4

Conclusion

In this paper, a new method is proposed to retrieve similar trademark images, which extracts Uniform LBP features based on CNN. Comparing the recall, precision, Fmeasure and PVR curve on METU dataset with other traditional methods, our method would get better retrieval results. In social area, the proposed method on the one hand could help trademark applicants to avoid infringement behavior, on the other hand could help trademark reviewers to quickly and effectively review the applications for regis‐ tration of the trademark whether it is a malicious cybersquatting, deliberately imitating the behavior of the registered trademark or not. Our method not only has a very good retrieval results, but also solve the case in the absence of adequate samples and how to use mature CNN to solve our own problems. Acknowledgement. This work is partly supported by the National Aerospace Science and Technology Foundation, the National Nature Science Foundation of China (No. 61702419) and H3C foundation of ministry of education in China (No.2017A19050).

242

T. Lan et al.

References 1. Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the Devil in the Details: Delving Deep into Convolutional Nets. Computer Science (2014) 2. Ojala, T., Pietikäinen, M., Mäenpää, T.: Gray scale and rotation invariant texture classification with local binary patterns. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1842, pp. 404–420. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45054-8_27 3. Wei, C.H., Li, Y., Chau, W.Y., Li, C.T.: Trademark image retrieval using synthetic features for describing global shape and interior structure. Pattern Recogn. 42(3), 386–394 (2009) 4. Goyal, A., Walia, E.: Variants of dense descriptors and zernike moments as features for accurate shape-based image retrieval. Signal Image Video Process. 8(7), 1273–1289 (2014) 5. Tursun, O., Kalkan, S.: METU dataset: a big dataset for benchmarking trademark retrieval. In: International Conference on Machine Vision Applications, pp. 514–517. IEEE (2015) 6. Ahonen, T., Hadid, A., Pietikäinen, M.: Face recognition with local binary patterns. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3021, pp. 469–481. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24670-1_36 7. Rusiñol, M., Aldavert, D., Karatzas, D., Toledo, R., Lladós, J.: Interactive trademark image retrieval by fusing semantic and visual content. In: Clough, P., Foley, C., Gurrin, C., Jones, Gareth J.F., Kraaij, W., Lee, H., Mudoch, V. (eds.) ECIR 2011. LNCS, vol. 6611, pp. 314– 325. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20161-5_32 8. Vasic, B.: New content based forensic metrics for judicial disputes concerning the graphic symbols similarity. In: International Conference on Telecommunication in Modern Satellite, Cable and Broadcasting Services, vol. 01, pp. 341–344. IEEE (2014) 9. Yan, Y., Ren, J., Li, Y., Windmill, J., Ijomah, W.: Fusion of dominant colour and spatial layout features for effective image retrieval of coloured logos and trademarks. In: IEEE International Conference on Multimedia Big Data, vol. 106, pp. 306–311. IEEE Computer Society (2015) 10. You, F., Liu, Y.: Research on colorful trademark images retrieval based on multi-feature combination and user feedback. In: Shen, G., Huang, X. (eds.) ECWAC 2011. CCIS, vol. 144, pp. 139–145. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20370-1_24 11. Qi, Y.L.: A relevance feedback method to trademark retrieval based on SVM. In: International Symposium on Computer Network and Multimedia Technology, pp. 1–4. IEEE (2009) 12. Wang, Z., Hong, K.: A novel approach for trademark image retrieval by combining global features and local features. J. Comput. Inform. Syst. 8(4), 1633–1640 (2012)

Adaptive Learning Compressive Tracking Based on Kalman Filter Xingyu Zhou, Dongmei Fu(&), Yanan Shi, and Chunhong Wu School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, China [email protected]

Abstract. Object tracking has theoretical and practical application value in video surveillance, virtual reality and automatic navigation. Compressive tracking(CT) is widely used because of its advantages in accuracy and efficiency. However, the compressive tracking has the problem of tracking drift when there are object occlusion, abrupt motion and blur, similar objects. In this paper, we propose adaptive learning compressive tracking based on Kalman filter (ALCT-KF). The CT is used to locate the object and the classifier parameter can be updated adaptively by the confidence map. When the heavy occlusion occurs, Kalman filter is used to predict the location of object. Experimental results show that ALCT-KF has better tracking accuracy and robustness than current advanced algorithms and the average tracking speed of the algorithm is 39 frames/s, which can meet the requirements of real-time. Keywords: Kalman filter Adaptive learning


Compressive tracking
Confidence map

1 Introduction Object tracking has been widely used in video surveillance and robotics which is a very popular topic in computer vision [1]. In recent years, although many tracking methods have been proposed and much success has been demonstrated, robust tracking is still a challenging task due to factors such as occlusion, fast moving, motion blur and pose change [2]. In order to deal with these factors, how to build an effective adaptive appearance model is particularly important. In general, tracking algorithms can be categorized into two classes: generative and discriminative algorithms [3]. The generative tracking algorithms aim at modeling the target and finding the location of the target by searching the image blocks which are the most similar to the target model. Kumar et al. [4] combine Kalman filter with geometric shape template matching method and can solve multi-target segmentation and merging. Zhan et al. [5] propose a combination of mean shift algorithm and Kalman filter tracking algorithm which can avoid the model update error. Wang et al. [6] use partial least squares (PLS) to study low dimensional distinguishable subspaces, and the tracking drift problem is alleviated by online update of the apparent model. Hu et al. [7] introduce the sparse weight constraint to dynamically select the relevant template in the global template set and use the multi-feature joint © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 243–253, 2017. https://doi.org/10.1007/978-3-319-71598-8_22

244

X. Zhou et al.

sparse representation for multi-target tracking under occlusion. However, the above generative algorithms ignore the background information. When the other object with similar texture to the target or the object is occluded, the tracking algorithm is easily interrupted or tracking failure. Discriminative tracking algorithms consider that target tracking as a two element classification problem, the purpose is to find a boundary that divide the target from the background [8]. Babenko et al. [9] propose a multiple instance learning (MIL) approach, because of its feature selection calculation complexity, it leads to poor real-time performance. Kaur and Sahambi et al. [10] propose an improved steady-state gain Kalman filter. By introducing a fractional feedback loop in the Kalman filter, the proposed algorithm solves the problem of abrupt motion. Zhang et al. [11] make full use of hybrid SVMs for appearance models to solve the blurring problem of the former background boundary and avoid the drift problem effectively. But these discriminative methods involve high computational cost, which hinders their real-time applications. In order to take advantage of the above two kinds of methods, this paper proposes an adaptive learning compressive tracking algorithm [12, 13] based on Kalman filter (ALCT-KF), which is used to solve the problems of severe occlusion, fast moving, similar object and illumination change. The adaptive learning compressive tracking algorithm uses CT algorithm to track the target, and calculate the value of the Peak-to-Sidelobe (PSR) by confidence map to update Bayesian classifier adaptively. When the PSR is less than a certain threshold, the object is considered to be heavy occlusion, then uses the Kalman filter to predict the location of object. The rest of this paper is organized as follows. Section 2 gives a brief review of original CT. The proposed algorithm is detailed in Sect. 3. Section 4 shows the experimental results of proposed and we conclude in Sect. 5.

2 Compressive Tracking As shown in [12, 13] are based on compressive sensing theory, a very sparse random matrix is adopted that satisfies the restricted isometry property (RIP), facilitating projection from the high-dimensional Haar-like feature vector to a low-dimensional measurement vector V ¼ Rx;

ð1Þ

where R 2 Rnm ðn  mÞ is sparse random matrix, feature vector x 2 Rm1 , compressive feature vector V 2 Rn1 . 8 1 > with probability > >1 > 2s > pffiffi < 1 Rði; jÞ ¼ ri;j ¼ s  0 with probability 1 > s > > > > : 1 with probability 1 2s

ð2Þ

Adaptive Learning Compressive Tracking Based on Kalman Filter

245

where s ¼ m=ða log10 ðmÞÞ, m ¼ 106  1010 , a ¼ 0:4. R becomes very sparse, and the number of non-zero elements for each row is only 4 at most, further reducing the computational complexity. The compressed features v are obtained by (1) and (2) which inputs to the naive Bayesian classifier and the position of the target is determined by the response value. Assuming that the elements in v are independently distributed, the naive Bayesian classifier is constructed: Qn  pðvi j y ¼ 1Þpðy ¼ 1Þ i¼1 HðvÞ ¼ log Qn i¼1 pðvi j y ¼ 0Þpðy ¼ 0Þ   n X pðvi j y ¼ 1Þ ¼ log pðvi j y ¼ 0Þ i¼1

ð3Þ

where pðy ¼ 1Þ ¼ pðy ¼ 0Þ ¼ 0:5, and y 2 f0; 1g is binary variable which represents the sample label. The conditional distributions pðvi j y ¼ 1Þ and pðvi j y ¼ 0Þ in HðvÞ are assumed to be Gaussian distributed with four parameters ðl1i ; d1i ; l0i ; d0i Þ pðvi j y ¼ 1Þ  Nðl1i ; r1i Þ ; pðvi j y ¼ 0Þ  Nðl0i ; r0i Þ

ð4Þ

where l1i ðl0i Þ and l0i ðd0i Þ are mean and standard deviation of the positive (negative) class. These parameters can be updated by l1i r1i

kl1 þ ð1 kÞl1i qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi kðr1 Þ2 þ ð1 kÞðr1i Þ2 þ kð1 kÞðl1i l1 Þ2

ð5Þ

where k is the learning parameter, and rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 Xn 1 ðv ðkÞ l1 Þ2 r ¼ k¼0jy¼1 i n 1 Xn 1 v ðkÞ l1 ¼ k¼0jy¼1 i n 1

ð6Þ

Negative sample parameters l0i and r0i are updated with the similar rules. Compressive tracking is simple and efficient, but the problem still exits: the classifier is updated by Eq. (5) which uses a fixed learning rate k. When the occlusion and other conditions occur, it may cause the classifier update error.

3 Proposed Algorithm 3.1

Adaptive Learning Compressive Tracking (ALCT)

Compressive tracking algorithm is difficult to re-find the right object when it tracks drift or failure. One of the main reasons is that pðvi j y ¼ 0Þ and pðvi j y ¼ 1Þ are determined

246

X. Zhou et al.

by the four parameters l0i , l1i , r0i , r1i while a fixed learning parameter k is used in Eq. (5). When the occlusion or other conditions occur, k may cause the classifier to update incorrectly. According to Eq. (3), we define the non-linear function for the naive Bayes classifier HðvÞ as objective confidence cðxÞ ¼ pðy ¼ 1 j xÞ ¼ rðHðvÞÞ

ð7Þ

where rð
Þ is a sigmoid function, rðxÞ ¼ ð1=1 þ e x Þ. The Peak-to-Sidelobe(PSR) [14], which measures the strength of a correlation peak, can be used to detect occlusions or tracking failure. PSRðtÞ ¼

maxðct ðxÞÞ rt

lt

ð8Þ

where ct ðxÞ denotes the classifier response value for all the candidate positions at the tth frame, split into the peak which is the maximum value maxðct ðxÞÞ and the sidelobe which is the rest of the search position excluding an 11  11 window around the peak. lt and rt are the mean and standard deviation of the sidelobe. Taking the Cliff bar sequence as an example, the PSR distribution is shown in Fig. 1. 2 1.8

PSR

1.6 1.4 1.2

D

1 0.8 0

A B 50

100

150

200

F

E

C 250

300

350

400

450

500

Frame Number

A-Motion Blur

B-occlusion

C-Fast Motion

D-Scale Chang

E-Fast Motion

F-Rotation

Fig. 1. Analysis of PSR in sequence of Cliff bar

Figure 1 shows the PSR can locate the most challenging factors of that video. In the first 75 frames, object has few interference factors and PSR stabilized at about 1.6. The object moves fast and causes the target area to be blurred, PSR is down to point A in

Adaptive Learning Compressive Tracking Based on Kalman Filter

247

the 75–90 frames. When there is no longer moving blur, PSR gradually returns to the normal level. In the same way, when the object undergoes occlusion, fast motion, scale change, rotation, which cause the values of PSR down to the valley point, corresponding to B, C, D, E, F respectively in Fig. 1. The value of PSR can reflect the influence of factors. The higher PSR is, the higher confidence of target location. Therefore, when the PSR is less than a certain threshold, the classifier should be updated with a smaller learning rate, which can improve the anti-interference ability of the model. Experiments (see Fig. 1) show that when the value of PSR is higher than 1.6, the tracking results are completely credible. If PSR is less than 1.6, the object may be occlusion, pose and illumination change. So we can determine the update weight of the classifier according to the PSR of each frame. The new update formula is shown in Eq. (9): 8 8 > PSRt \PSR0 > >

> > 2 > > < wt ¼ > exp½ ðPSRt PSR1 Þ Š PSR0 \PSRt \PSR1 : ð9Þ 1 other > > 1 1 1 > l ð1 kw Þl þ kw l t i t > i qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi > > > : r1 ð1 kwt Þðr1 Þ2 þ kwt ðr1 Þ2 þ kwt ð1 kwt Þðl1 l1 Þ2 i

i

i

where PSRt represents the PSR at the t-th frame, PSR0 and PSR1 are two thresholds. When PSR0 \PSRt \PSR1 , it is considered that the object may undergo partial occlusion, fast motion, pose change. When PSRt \PSR0 , it is considered that the object is completely occluded, and classifier is not updated at this time. At this time, Kalman filter is used to predict the position of object. 3.2

Heavy Conclusion

In the process of object tracking, occlusion, illumination change, fast moving and similar target can not be avoided. If the above factors occur, the accuracy of many algorithms are obviously decreased. The adaptive learning compressive tracking algorithm proposed in this paper can meet the factors of partial occlusion and slow illumination change, but it needs to improve the algorithm for heavy occlusion. Kalman filter algorithm [15] is mainly used to estimate the target location and state. The algorithm uses the position and velocity of the object as the state vector to describe the change of the object state. Kalman filter algorithm can also effectively reduce the influence of noise in the object tracking process. The state equations and the observation equation of the Kalman filter are as follows: xt þ 1 ¼ /xt þ wt

ð10Þ

Zt ¼ Hxt þ Vt

ð11Þ

where xt ðxt þ 1 Þ is the state vector of the tðt þ 1Þ moment, Zt is the observation vector of the t moment, / is state transition matrix, H is observation matrix, wt is state noise vector of system disturbance, vt is observed noise vector.

248

X. Zhou et al.

In Sect. 3.1, it is proved when PSRt \PSR0 considers the target to be heavy occlusion. Because the Kalman filter can predict the position of the target in the next frame and effectively reduce the influence of noise, we use it to solve the above problems. Kalman filter can be divided into two phases: prediction and updating. The prediction phase is mainly based on the state of the current frame target to estimate the state of the next frame. In the update phase, the estimate of the prediction phase is optimized by using the observations of the next frame to obtain more accurate new predictions. Assuming that the target has a serious occlusion at (t + 1)-th frame, the Kalman filter is used to re-estimate the object position. (1) prediction phase State prediction equation: xt þ 1 ¼ /xtþ

ð12Þ

where xtþ is the tracking result of the ALCT algorithm at the t-th frame. Error covariance prediction equation: Pt þ 1 ¼ /Ptþ /T þ Q

ð13Þ

where Ptþ is the covariance matrix at t frame, Q is the state noise covariance matrix and the value is constant. (1) updating phase Gain equation: Kt þ 1 ¼ Pt þ 1 H T ðHPt H T þ RÞ

1

ð14Þ

where Kt þ 1 is the Kalman gain matrix, R is the measurement noise covariance matrix and the value is constant. Error covariance modification equation: Ptþþ 1 ¼ ð1

Kt þ 1 HÞPt þ 1

ð15Þ

State modification equation: xtþþ 1 ¼ xt þ 1 þ Kt þ 1 ðZt þ 1

Hxt þ 1 Þ

ð16Þ

where Zt þ 1 is the object position that the ALCT algorithm tracks when it is at (t + 1)-th frame, xtþþ 1 is the position of the estimated object at the (t + 1)-th frame. Then, using the ALCT algorithm to track the position of object at the t-th frame. If PSRt þ 2 \PSR0 then re-estimate the target in the frame position by Eqs. (12)–(16), otherwise, ALCT is used to track the object at next frame. The flow chart of adaptive learning compressive tracking algorithm based on Kalman filter (ALCT-KF) is shown in Fig. 2. Firstly, the position of the target in the first frame is manually calibrated and the object is tracked by the ALCT algorithm. Then, the PSR is calculated by the target confidence map and the Bayesian classifier is updated by the PSR. If the PSR is less

Adaptive Learning Compressive Tracking Based on Kalman Filter

249

Input image sequences Manually appoint the region of object at the first frame ALCT for object tracking

No

Heavy occlusion? Yes

Predict the position of the object by Kalman filter Get new target position Fig. 2. Flow of ALCT-KF algorithm

than a certain threshold, it is considered that the target is serious occlusion. At this time, the Kalman filter is used to predict the position of the target in the current frame and the predicted position is assigned to ALCT algorithm for object tracking in next frame.

4 Experiment In order to validate the proposed algorithm, 6 different challenging video sequences are adopted, including occlusion, illumination, pose change, fast motion, and similar object. We compare the proposed ALCT-KF algorithm with the state-of-art methods, Compressive tracking(CT) [13], Online discriminative feature selection(ODFS) [16], Spatio-temporal context learning(STC) [17], Tracking-Learning-Detection(TLD) [18]. Implemented in MATLAB2013a, Core(TM)i5-4570CPU and 4 GB RAM. As is shown in Fig. 1, the thresholds of Eq. (9) are set to PSR0 2 ½1:2; 1:4Š and PSR1 2 ½1:6 1:8Š. The Kalman filter parameter is set to: 1 0 60 1 6 u¼6 40 0 0 0 2 400 6 0 6 P¼6 4 0 2

0

1 0 1 0 0

3 0 17 7 7; 05

R¼



1

0:2845 0:0045

0:0045



2

1

60 6 ; Q¼6 40 0:0045 0

0

0

400

0

0

400

0

0

0 7 7 7 0 5

400

3

0 0 0

3

1 0 07 7 7; 0 1 05 0 0 1

H¼



1 0 0

0

0 1 0

0



250

X. Zhou et al.

Table 1. Success rate (SR) (%) and average frames per second (FPS). (Top two result are shown in Bold and italic). Sequences Dudek FaceOcc2 Motocross Cliff bar David Pedestrian Average SR Average FPS

ALCT-KF 97.2 96.3 92.4 91.1 96.1 83.1 93.2 39

CT 73.1 86.2 70.7 82.0 89.8 63.0 77.5 42

ODFS 71.2 98.5 60.6 89.2 93.1 8.5 70.2 38

STC 61.6 93.0 77.8 86.4 97.6 2.5 69.8 53

TLD 72.6 70.2 69.2 63.1 92.9 32.9 66.8 16

Two metrics are used to evaluate the experimental results. (1) The first metric is the success rate which is defined as, score ¼

areaðROIT \ ROIG Þ areaðROIT [ ROIG Þ

ð17Þ

where ROIG is the ground truth bounding box and ROIT is the tracking bounding box. If score is larger than 0.5 in one frame, the tracking result is considered as a success. Table 1 shows the comparison of success rate in the test video. The proposed algorithm achieves the best or second best performance. Compared with the CT algorithm, the average tracking success rate of M-ALCT is improved by 15.7%. The last row of Table 1 gives the average frames per second. ALCT-KF performs well in speed (only slightly slower than CT method) which is faster than ODFS, TLD methods. The second metric is the center location error which is defined as the euclidean distance between the central locations of the tracked object and the manually labeled ground truth. CLE ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðxT xG Þ2 þ ðyT yG Þ2

ð18Þ

The tracking error of the proposed algorithm is smaller than other algorithms, which can be maintained within 15 pixels (see Fig. 3). The object in the Dudek and FaceOcc2 sequences (see Fig. 4) is subject to partial occlusion and heavy occlusion. In the Cliff bar and Motocross sequences (see Fig. 5), the object is abrupt motion and rotation which lead to the appearances of objects change significant and motion blur. The David and Pedestrian sequences (see Fig. 6) are challenging due to illumination variation and similar object. Through the above experiments the proposed algorithm effectively avoids the tracking failure when occlusion, abrupt motion, motion blur, similar target and other situations occur.

Adaptive Learning Compressive Tracking Based on Kalman Filter Dudek

500

350 300 250 200 150 100 50 0

0

100

200

300

350

500

30 25 20 15 10 5 0

600

250

200

150

100

50

0

0

20

40

60

80

100

0

100

200

300

120

140

160

180

Davi d

20

15

10

5

0

0

50

100

150

200

250

800

900

50

0

50

100

150

200

250

300

350

400

450

500

Pedest r i an

140

Posi t i on Er r or ( pi xel )

Posi t i on Er r or ( pi xel )

25

700

Fr ame#

ALCT-KF CT ODFS STC TLD

30

600

100

Fr ame#

35

500

ALCT-KF CT ODFS STC TLD

0

200

400

Fr ame# Cl i f f bar

150

ALCT-KF CT ODFS STC TLD

Posi t i on Er r or ( pi xel )

Posi t i on Er r or ( pi xel )

400

35

Fr ame# Mot ocr oss

300

ALCT-KF CT ODFS STC TLD

40

Posi t i on Er r or ( pi xel )

Posi t i on Er r or ( pi xel )

400

FaceOcc2

45

ALCT-KF CT ODFS STC TLD

450

251

300

350

400

450

500

ALCT-KF CT ODFS STC TLD

120

100

80

60

40

20

0

0

50

Fr ame#

100

150

200

250

300

350

Fr ame#

Fig. 3. Error plots in terms of center location error for 6 test sequences. (Color figure online)

#105

#182

#380

#490

#573

(a) Dudek sequence

(b) FaceOcc2 sequence

Fig. 4. Tracking results of the occlusion sequences. (Color figure online)

252

X. Zhou et al.

(a) Cliff bar sequence

(b) Motocross sequence

Fig. 5. Some sample tracking results of abrupt motion and rotation sequences. (Color figure online)

(a) David sequence

(b) Pedestrian sequence

Fig. 6. Tracking results of illumination variation and similar object. (Color figure online)

5 Conclusion In this paper, an adaptive learning compressive tracking algorithm based on Kalman filter is proposed to deal with the problem of occlusion, fast moving, rotation and similar object. Tracking drift problem is alleviated by using tracking confidence map to adaptively update the classifier model and Kalman filter algorithm is used to predict the object location and reduce the impact of noise. Experiments show that the proposed algorithm has better tracking accuracy and robustness and it is easy to implement and achieves real-time performance.

Adaptive Learning Compressive Tracking Based on Kalman Filter

253

References 1. Li, X., Hu, W., Shen, C., et al.: A survey of appearance models in visual object tracking. ACM Trans. Intell. Syst. Technol. 4(4), 48–58 (2013) 2. Smeulders, A.W.M., Chu, D.M., Cucchiara, R., et al.: Visual tracking: an experimental survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1442–1468 (2014) 3. Liu, W., Li, J., Chen, X., et al.: Scale-adaptive compressive tracking with feature integration. J. Electron. Imaging 25(3), 3301801–3301810 (2016) 4. Kumar, P., Ranganath, S., Sengupta, K., et al.: Cooperative multitarget tracking with efficient split and merge handling. IEEE Trans. Circ. Syst. Video Technol. 16(12), 1477– 1490 (2006) 5. Zhan, J.P., Huang, X.Y., Shen, Z.X., et al.: Object tracking based on mean shift and kalman filter. J. Chongqing Inst. Technol. 3, 76–80 (2010) 6. Wang, Q., Chen, F., Xu, W., et al.: Object tracking via partial least squares analysis. IEEE Trans. Image Process. 21(10), 4454–4465 (2012) 7. Hu, W., Li, W., Zhang, X., et al.: Single and multiple object tracking using a multi-feature joint sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 37(4), 816–833 (2015) 8. Wu, Y., Lim, J., Yang, M.H.: Online object tracking: a benchmark. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland, pp. 2411–2418 (2013) 9. Babenko, B., Yang, M.H., Belongie, S.: Robust object tracking with online multiple instance learning. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1619–1632 (2011) 10. Kaur, H., Sahambi, J.S.: Vehicle tracking in video using fractional feedback kalman filter. IEEE Trans. Comput. Imaging 2(4), 550–561 (2016) 11. Zhang, S., Sui, Y., Yu, X., et al.: Hybrid support vector machines for robust object tracking. Pattern Recognit. 48(8), 2474–2488 (2015) 12. Zhang, K., Zhang, L., Yang, M.-H.: Real-time compressive tracking. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 864–877. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_62 13. Zhang, K., Zhang, L., Yang, M.H.: Fast compressive tracking. IEEE Trans. Pattern Anal. Mach. Intell. 36(10), 2002–2015 (2014) 14. Bolme, D.S., Beveridge, J.R., Draper, B.A., et al.: Visual object tracking using adaptive correlation filters. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, pp. 2544–2550 (2010) 15. Chen, S.Y.: Kalman filter for robot vision: a survey. IEEE Trans. Ind. Electron. 59(11), 4409–4420 (2012) 16. Zhang, K., Zhang, L., Yang, M.H.: Real-time object tracking via online discriminative feature selection. IEEE Trans. Image Process. 22(12), 4664–4677 (2013) 17. Zhang, K., Zhang, L., Liu, Q., Zhang, D., Yang, M.-H.: Fast visual tracking via dense spatio-temporal context learning. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 127–141. Springer, Cham (2014). https://doi.org/10. 1007/978-3-319-10602-1_9 18. Kalal, Z., Mikolajczyk, K., Matas, J.: Tracking-learning-detection. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1409–1422 (2012)

Online High-Accurate Calibration of RGB+3D-LiDAR for Autonomous Driving Tao Li1 , Jianwu Fang1,2 , Yang Zhong1 , Di Wang1 , and Jianru Xue1(B) 1

Laboratory of Visual Cognitive Computing and Intelligent Vehicle, Xi’an Jiaotong University, Xi’an, People’s Republic of China [email protected] 2 School of Electronic and Control Engineering, Chang’an University, Xi’an, People’s Republic of China

Abstract. Vision+X has become the promising tendency for scene understanding in autonomous driving, where X may be the other nonvision sensors. However, it is difficult to utilize all the superiority of different sensors, mainly because of the heterogenous, asynchronous properties. To this end, this paper calibrates the commonly used RGB+3DLiDAR data by synchronization and an online spatial structure alignment, and obtains a high-accurate calibration performance. The main highlights are that (1) we rectify the 3D points with the aid of differential inertial measurement unit (IMU), and increase the frequency of 3D laser data as the same as the ones of RGB data, and (2) this work can online high-accurately updates the external parameters of calibration by a more reliable spatial-structure matching of RGB and 3D-LiDAR data. By experimentally in-depth analysis, the superiority of the proposed method is validated.

1

Introduction

Calibration of multiple sensors is the prerequisite of robust fusing them. The coexistence of multiple sensor has become the standard configuration for mobile robot systems, especially for autonomous driving. Based on the investigation for sensor utilization for autonomous driving [1], in addition to the visual camera system when perception [13, 14], other non-vision sensors play very important role for scene understanding, even with a more emphasis. The main reason is that vision camera is vulnerable to dynamic environment, but with a stable perception by other non-vision ones, such as laser range finders. However, vision camera has the most intuitive and richest representation for scene knowledge. Therefore, calibration for multiple sensors is an inevitable work for effectively scene perception. However, the calibration of camera and a laser range finder is challenging because of the different physical meaning of data gathered by distinct sensors [4]. In addition, the frame frequency and point sparsity of two sensor planes are of imparity significantly, seeing Fig. 1 for a demonstration. c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 254–263, 2017. https://doi.org/10.1007/978-3-319-71598-8_23

Online High-Accurate Calibration of RGB+3D-LiDAR

255

Fig. 1. One frame of RGB and 3D-LiDAR data.

1.1

Related Works and Limitations

Facing this issues, many works carried out the calibration for RGB and Laser data owning to that these two kind of sensors are the most common in the existing mobile robot systems [5, 8]. To summarize the calibration methods, they can be divided into two categories: hand-operated calibration and automatic calibration. Hand-operated calibration: Hand-operated calibration for a long time is the main module of calibration for different sensors. That is because the calibration with an human auxiliary has a controllable condition for better key point detection and correlation, by which better extrinsic calibration parameters can be obtained. For the early calibration works, the chessboard is usually utilized for a better keypoint detection. For example, Zhang and Pless [11] proposed a calibration method by adding the planar constraints of camera and Lasers with a manual selection of chessboard region. Unnikrishnan and Hebert [10] designed a toolbox for offline calibration of 3D laser and camera. Afterwards, Scaramuzza et al. [7] relaxed the condition that the calibration can be conducted by natural images instead of chessboard preparation, whereas the key points still needed to be selected manually. Because the calibration with individual keypoints is easy to generate misalignment, Sergio et al. [9] utilized a circular object to obtain a structure-based calibration. Park et al. [6] adopted a polygon constructed by several keypoints to fulfill a structure alignment. These hand-operated methods can achieve a relatively accurate calibration under controllable condition, whereas need laborious manual operation. Hence, some automatic modules for calibration appeared in recent years. Automatic calibration: Automatic calibration aims to accomplish an automatic calibration in different scenarios. For instances, Kassir and Peynot [3] automatically extracted the chessboard region in camera and laser sensors, and achieved a keypoint based calibration. Geiger et al. [2] utilized one image and the corresponding laser data to automatically fulfill the calibration with more than one chessboards in the image plane. Scott et al. [8] automatically chosen the natural scenes for a better calibration condition selection and reduced the misalignments. These automatic calibrations are all based on specific circumstances without the consideration of dynamic factors of scenes, and vulnerable to camera jitter. Aiming at this issue, online module begins to attract the attention recently. Within this domain, Levinson and Thrun [5] proposed an online

256

T. Li et al.

e tim

Intrinsic parameters

Online Calibration Searching with Natural RGB+3D-LiDAR Data

spatial-structure alignment

Initialization of Calibration Parameters with Manually Selected RGB+3D-LiDAR Data

Synchronization

automatic calibration with an updating for calibration parameters with the latest several frames. However, the calibration parameters are easy to generate bias with the optimal ones because of the improper optimization process. To this end, inspired by the work of [5], this paper re-formulates the online calibration problem, and using a more reliably spatial structure matching for parameter updating. In the meantime, we rectify the 3D points with the aid of differential inertial measurement unit (IMU), and increase the frequency of 3D laser data as the same as the ones of RGB data. The experimental analysis demonstrates that the proposed method can generate a high-accurate calibration performance even with a tough initialization of calibration parameters. The flowchart of the proposed calibration method is demonstrated in Fig. 2.

ˆ tˆ R,

tim

e

Extrinsic parameters Ro, t o

Online calibration searching with subsequent frames

Fig. 2. The flowchart of the proposed calibration method. Firstly, we compute the initialization calibration parameters by manually selecting several images and the corresponding laser data with a chessboard auxiliary. Secondly, the synchronization of RGB and 3D-LiDAR data is conducted for a calibration data preparation. Thirdly, the online high-accurate calibration with a more reliable spatial-structure matching of natural RGB and 3D-LiDAR data is given.

2

Synchronization of RGB and 3D-LiDAR Data

The purpose of the calibration in this work is to automatically determine the six dimensional transformation with a series of image frames and corresponding 3D point sets. Because of the demand of the same object being targeted by two different sensors, this work firstly fulfills the synchronization of the RGB and 3D-LiDAR data, and increases the frame frequency of laser scanning data as the same as the ones of RGB data with an aid of differential internal measurement unit (IMU). IMU in this work provides a 6-dimensional pose state of vehicle, including the location p = (x, y, z) and its corresponding rotation pose (roll, yaw, pitch). Influenced by the scanning module of 3D-LiDAR, all of the 3D points have the different timestamps and pose state. Consequently, the state of each 3D point is distinct. For a calibration task, we need rectify the pose state of the each 3D point as the same as the one of image. Specifically, for a circle of scanning by 3D-LiDAR, we can obtain the starting and ending time of one

Online High-Accurate Calibration of RGB+3D-LiDAR

257

round of scanning, and the indexes of the user datagram protocol (UDP) bags with a constant time interval. By these two kinds of information, we can calculate the timestamps and the pose state of each 3D point. Given the timestamps and pose state vector of each image frame, the synchronization is carried out by transforming the pose state of each 3D point into the one of image. Assume the timestamps of a 3D point and the image is t0 and t1 . Denote the location of the 3D point at timestamps of t0 and t1 is p0 = (x0 , y0 , z0 ) and p1 = (x1 , y1 , z1 ), and the rotation matrix of t0 and t1 corresponding the origin pose state is R0 and R1 , respectively. The transformation of p0 → p1 is computed by: ⎞ ⎛ ⎞ ⎛ x1 x0 ⎝ y 0 ⎠ = RT1 R0 ⎝ y1 ⎠ + R0 (p1 − p0 ), (1) z1 z0 where R0 and R1 are calculated by the multiplication of three rotation matrixes respectively in roll -axis, yaw -axis, and pitch-axis. With that, we achieve the synchronization of RGB and 3D-LiDAR Data. One remaining issue is that the frame frequency of RGB data and 3D-LiDAR Data is different. Usually, the capturing frequency of camera is larger. In other words, there is no corresponding 3D points for some video frames. To this end, we take a principle of proximity that we assign the 3D points to the RGB frame with a minimum interval of timestamps of them. As thus, we increase the frequency of 3D LiDAR data as the same as the one of RGB data, which forms the basis for subsequent spatial calibration.

3

Online High-Accurate Calibration

In this section, we will present the online high-accurate calibration. For online calibration, there are three steps: (1) initialization of calibration parameters containing the intrinsic parameter of camera and the extrinsic parameter of camera+3D-LiDAR; (2)online calibration with an adequate spatial alignment optimization. In the following, the initialization of the calibration parameters is firstly described, and then the online high-accurate calibration approach is given. 3.1

Initialization of Calibration Parameters

For the initialization of calibration parameters, it contains intrinsic parameter of camera and extrinsic parameter of camera+3D-LiDAR. As for the intrinsic parameter of camera, this work introduces the commonly work proposed by Zhang [12], where the best intrinsic parameter of camera is selected from 20 grayscale images with a chessboard auxiliary. With respect to the extrinsic parameter of camera+3D-LiDAR, we extract the image plane of chessboard region in 20 images and the corresponding laser plane. Then, the extrinsic parameters, i.e., translation vector t = (△x, △y, △z) for each point and the corresponding rotate matrix R ∈ R3×3 for roll, yaw, and pitch, are calculated by the Laser-Camera Calibration Toolbox (LCCT) [10].

258

3.2

T. Li et al.

Online Calibration

Online calibration aims to update the extrinsic parameter by newly observed camera frames and a series of corresponding laser scans for a better adaptation to scene variation. Inspired by the work of [5], it achieved the online calibration by the edge alignment of several newest camera images and the frames of laser scanning. However, this method is easy to generate bias for the calibration parameters, and the calibration results drift away from the accurate situation. To this end, given a calibration of t and R, this work firstly projects the 3D LiDAR sensor onto the image plane, and then formulates online calibration problem as: max : ER,t =

V f| | n 

f Vpf Si,j ,

(2)

f =n−w p=1

where w is the frame number for optimization (set as 9 frames in this work), |V f |

n is the newest observed video frame, p is the index for 3D point set {Vpf }p=1 f obtained by 3D LiDAR sensor, Si,j is the point (x, j) in the f th frame S. The f in points in both sensors are all the edge points. Similar to [5], the point Si,j image is extracted by edge detection (the method for edge detection is not the focus) appending an inverse distance transformation (IDT), and 3D point {Vpf } is obtained by calculating the distance difference of the scene from the scanner of 3D LiDAR. This formulation is similar to the work of [5], but with a better strategy for optimal calibration searching. Optimization. For the solving of Eq. 2, this work adopts the grid search algorithm. Specifically, we set the initialization calibration as {Ro , to }, which is ˆ ˆ treated as the optimal calibration {R, t} at the beginning of searching. Because there are six values in the transformation of calibration, i.e., x, y, z translations, and roll, yaw, pitch rotations. When performing the grid search with radius 1, there will be 36 = 729 configurations (denoted as perturbations by [5]) of {R, t} ˆ ˆ centered around {R, t} for the k th grid of searching, denoted as {Rki , tki }729 i=1 . We k ˆk ˆ define {R , t } as the temporary optimal calibration at the k th grid of searching. With these configurations, this work utilizes them to project the 3D point onto the image plane, and computes ERki ,tki for each configuration {Rki , tki } at the ˆ k, ˆ tk }, we select k th grid of searching. For the temporary optimal calibration {R it by: ˆ k, ˆ (3) tk } = arg max ERki ,tki . {R k Rk i ,ti

ˆ k, ˆ To evaluate the quality of {R tk }, the work of [5] defines it as the fraction that the configurations with ER k relative to all the configurations, ˆ k ,ˆ tk > ERk i ,ti . Then, they terminate the searching if FR denoted as FR k ˆ k ,ˆ ˆ k ,ˆ ˆ k−1 ,ˆ t tk −FR tk−1 < ε, k ˆ k ˆ ˆ ˆ where ε is a small constant value. Otherwise, R ← R , t ← t . However, this quality evaluation of [5] is vulnerable to noise configuration, and easy to drift away from the accurate calibration.

Online High-Accurate Calibration of RGB+3D-LiDAR

259

Fig. 3. The demonstration of the spatial-structure alignments under the best calibration and several surrounding configurations. Among them, the points with cyan color, representing the edge points of 3D-LiDAR, have a best spatial alignment with the edge of the image after inverse distance transformation (IDT). In the meanwhile, we also show some configurations neighboring the best one by other colors. From this figure, we can observe that configurations around the best calibration almost generate the similar alignments. Note that this figure should be exhibited with RGB color mode. (Color figure online)

ˆ k, ˆ In the end, this paper defines the quality of {R tk } by computing the varik k 729 ance of {Ri , ti }i=1 , denoted as V (ERki ,tki ). Note that the variance is calculated after a L2-normalization of ERki ,tki . We terminate the searching if V (ERi ,ti ) < τ , where τ is a small constant for evaluating the smoothness of ERki ,tki . Otherwise, ˆ ←R ˆ k, ˆ R t←ˆ tk . That means that the smaller value of the V (ERi ,ti ) has, the better the calibration is. The behind meaning of this strategy is that if we obtain ˆ ˆ ˆ ˆ the best calibration {R, t}, the configurations around {R, t} can generate a relative similar value of ER,t . Taking Fig. 3 as an example, because of the inverse distance transformation, the spatial-structure matchings of RGB and 3D-LiDAR data under the calibrations of {Ri , ti }729 i=1 have a similar results when the optimal calibration is searched. When the optimal calibration is obtained, it is treated as {Ro , to } for online calibration of subsequent image frames and the corresponding series of laser scans. Actually, the calibration in this work accomplishes a truly structure alignment of the data collected by camera and 3D-LiDAR. In the following, we will give the experiments to validate the performance of the proposed method.

4 4.1

Experiments and Discussions Dataset Acquisition

The experimental data is collected by an autonomous vehicle named as “Kuafu”, which is developed by the Lab of Visual Cognitive Computing and Intelligent Vehicle of Xian Jiaotong University. In this work, we utilize the equipments

260

T. Li et al.

containing a Velodyne HDL-64E S2 LIDAR sensor with 64 beams, and a highresolution camera system with differential GPS/inertial information. The visual camera is with the resolution of 1920 1200 and a frame rate of 25. In addition, the scanning frequency of the 3DLiDAR is 10 Hz. Thousands of frames for calibration are evaluated. 4.2

Implementation Details

To evaluate the performance of the proposed method, two other calibration approaches are selected. One is the camera-laser calibration in the LCCT [10], and another is the online calibration method of [5]. Because of the dynamic scene and unpredictable camera jitter, it is difficult to obtain the ground-truth of the calibration parameters. In order to give a fair comparison, this work evaluates the performance from two aspects: (1) demonstrating calibration results when searching the optimal calibration by the work of [5] and the proposed method; (2) giving some snapshot comparisons with the calibration parameters by different methods.

Fig. 4. The iteration steps of (a) the work of [5] and (b) the proposed methods. The suspension points in this figure means that the searching procedure has reached the end.

Online High-Accurate Calibration of RGB+3D-LiDAR

261

Fig. 5. The snapshots of typical calibration results. The first column is the results generated by LCCT [10]. The second column is the ones obtained by [5], and the results outputted by the proposed method shown in the third column.

4.3

Performance Evaluation

Figure 4 demonstrates the iteration process for searching the optimal calibration by [5] and the proposed method. From the shown results, we can see that the searching process of [5] drifts away, while our method can obtain a more accurate

262

T. Li et al.

calibration. For finding the reason to this phenomena, we have checked each iteration step, and found that the criterion of the terminal condition in [5] is not adequate. That is because they terminate the searching when the fraction that all other configurations generate lower value for Eq. 2 than the center configuration remains unchanged. This terminal condition is vulnerable to noise calibration with larger value for Eq. 2. Actually, this phenomena is universal within the work [5]. As for the proposed method, we emphasis that all the configurations in the grid of searching should generate large values for Eq. 2 when the best calibration is searched, which is a truly spatial-structure alignment for RGB+3D-LiDAR data. Therefore, the best calibration is outputted after iterating. In addition, we also give several snapshots for the calibration comparison, shown in Fig. 5. From this figure, it is manifestly that the proposed method obtains the best calibration for the demonstrated images. Actually, these observation is general in the comparing methods. From the above analysis, the superiority of the proposed method is validated. 4.4

Discussions

When performing the calibration, there is an universal phenomena that the accuracy of the calibration is different for the entire scene. It is common in all the calibration methods. That is to say when the near scene is correctly calibrated, the calibration results for the far scene may appear skewing, and vice versa. The main reasons are that: (1) the RGB and Laser data themselves have distortion to some extent; (2) the calibration parameters are searched from the entire scene, which cannot avoid the influence of the regional distortion data in RGB+3D-LiDAR sensors. This problem may be tackled from a local calibration view in the future.

5

Conclusion

This paper presented an online high-accurate calibration method for RGB+3DLiDAR sensors in the autonomous driving circumstance. Through a synchronization, we rectified the 3D points gathered by 3D-LiDAR data with a aid of differential inertial measurement unit (IMU), and increased frame frequency of laser data as the same as the one of visual camera. Then, this work obtained an online high-accurately calibration via a more reliable spatial-structure matching of RGB and 3D-LiDAR data. The superiority of the proposed method was verified in the experiments. Acknowledgement. This work is supported by the National Key R&D Program Project under Grant 2016YFB1001004, and also supported by the Natural Science Foundation of China under Grant 61603057, China Postdoctoral Science Foundation under Grant 2017M613152, and is also partially supported by Collaborative Research with MSRA.

Online High-Accurate Calibration of RGB+3D-LiDAR

263

References 1. Buehler, M., Iagnemma, K., Singh, S.: The DARPA Urban Challenge: Autonomous Vehicles in City Traffic. Springer, Heidelberg (2009). https://doi.org/10.1007/9783-642-03991-1 ¨ Schuster, B.: Automatic camera and range 2. Geiger, A., Moosmann, F., Car, O., sensor calibration using a single shot. In: Proceedings of the IEEE Conference on Robotics and Automation, pp. 3936–3943 (2012) 3. Kassir, A., Peynot, T.: Reliable automatic camera-laser calibration. In: Proceedings of the 2010 Australasian Conference on Robotics and Automation (2010) 4. Le, Q.V., Ng, A.Y.: Joint calibration of multiple sensors. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3651– 3658 (2009) 5. Levinson, J., Thrun, S.: Automatic online calibration of cameras and lasers. In: Proceedings of the Robotics: Science and Systems (2013) 6. Park, Y., Yun, S., Won, C.S., Cho, K., Um, K., Sim, S.: Calibration between color camera and 3D LiDAR instruments with a polygonal planar board. Sensors 14(3), 5333 (2014) 7. Scaramuzza, D., Harati, A., Siegwart, R.: Extrinsic self calibration of a camera and a 3D laser range finder from natural scenes. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4164–4169 (2007) 8. Scott, T., Morye, A.A., Pinis, P., Paz, L.M.: Choosing a time and place for calibration of LiDAR-camera systems. In: Proceedings of the IEEE Conference on Robotics and Automation, pp. 4349–4356 (2016) 9. Sergio, A.R.F., Fremont, V., Bonnifait, P.: Extrinsic calibration between a multilayer LiDAR and a camera. In: Proceedings of the IEEE Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 214–219 (2008) 10. Unnikrishnan, R., Hebert, M.: Fast extrinsic calibration of a laser rangefinder to a camera. Carnegie Mellon University (2005) 11. Zhang, Q., Pless, R.: Extrinsic calibration of a camera and laser range finder (improves camera calibration). In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2301–2306 (2005) 12. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000) 13. Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., Hu, S.: Traffic-sign detection and classification in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2110–2118 (2016) 14. Zhu, Z., Lu, J., Martin, R.R., Hu, S.: An optimization approach for localization refinement of candidate traffic signs. IEEE Trans. Intell. Transp. Syst. 18(11), 3006–3016 (2017)

Run-Based Connected Components Labeling Using Double-Row Scan Dongdong Ma1,2 ✉ , Shaojun Liu1,2, and Qingmin Liao1,2 (

1

)

Department of Electronic Engineering, Tsinghua University, Beijing, China [email protected] 2 Graduate School at Shenzhen, Tsinghua University, Shenzhen, China

Abstract. This paper presents a novel run-based connected components labeling algorithm which uses double-row scan. In this algorithm, the run is defined in double rows and the binary image is scanned twice. In the first scan, provisional labels are assigned to runs according to the connectivity between the current run and runs in the last two rows. Simultaneously, equivalent provisional labels are recorded. Then the adjacent matrix of the provisional labels is generated and decomposed with the Dulmage-Mendelsohn decomposition, to search for the equivalent-label sets in linear time. In the second scan, each equivalent-label set is labeled with a number from 1, which can be efficiently accomplished in parallel. The proposed algorithm is compared with the state-of-the-art algorithms both on synthetic images and real image datasets. Results show that the proposed algo‐ rithm outperforms the other algorithms on images with low density of foreground pixels and small amount of connected components. Keywords: Connected Components Labeling (CCL) · Image segmentation · Sparse matrix decomposition · Parallel algorithm

1

Introduction

Connected components labeling (CCL) in the binary image is a fundamental step in many fields such as image segmentation [1], computer vision, graph coloring, object recognition, etc. As the development of image techniques, number of pixels in a natural image increases dramatically. Consequently, number of objects in an image is also increasing and fast CCL algorithms have drawn attention of many researchers. Existing CCL algorithms can be classified into three categories: pixel-based algorithms, blockbased algorithms and run-based algorithms. Early researches about CCL were mainly focused on iterative pixel-based algorithm [2]. In [3], Suzuki et al. introduced a labeling algorithm whose execution time is linear with the number of total pixels. In [4, 5], an implementation of pixel-based algorithm was designed for applications where FPGA was used. In [6] Rosenfeld and Pfaltz proposed a two-scan algorithm. To save the memory allocated to the equivalent-label table, Lumia Rosenfeld et al. proposed an algorithm which only needs to store the equivalent labels in the current and last rows [7]. Through the analysis of values of adjacent pixels, He et al. used the Karnaugh map to design an efficient procedure in © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 264–274, 2017. https://doi.org/10.1007/978-3-319-71598-8_24

Run-Based Connected Components Labeling Using Double-Row Scan

265

assigning the provisional labels in the first scan [8]. Instead of checking values of four adjacent pixels in [8], algorithm in [9] only needs to check three. In [10] Chang et al. introduced a contour-tracing algorithm. The above algorithms are all characterized by checking values of adjacent pixels of the current pixel. They have advantages of saving storage and operation simplicity. However, these algorithms need to scan all the pixels one by one, which is inefficient. Moreover, many of them need to scan the whole image more than twice. To cope with the low efficiency of pixel-based algorithms, in [11], a labeling algo‐ rithm scanning the whole image through the two-by-two blocks was introduced. A deci‐ sion tree was built to determine the provisional label of the current block. Based on [11], algorithm in [12] uses the information of checked pixels to avoid the same pixels are checked for multiple times. Similarly, two-by-one block-scanning was also introduced in algorithm in [13]. Instead of checking all the 16 pixels in the adjacent blocks, in [14] Chang et al. designed an algorithm which only needs to check 4 pixels to determine the provisional label of the current block. In [15], Santiago et al. gave three labeling algo‐ rithms which all scan the whole image through the two-by-two blocks. Recently in [16], several state-of-the-art CCL algorithms were compared with each other on synthetic images and public datasets. Average results show that the BBDT [12], CCIT [14], and CTB [13] are the fastest three algorithms. Nevertheless, the weakness of these algorithms is that they are complex to implement and hard to be executed in parallel. In addition to the two types of labeling algorithms mentioned above, there is another type which is efficient while simple to implement. They are based on the run. As defined in [17], a run is a segment of contiguous foreground pixels in a row. In [17], He et al. introduced a run-based two-scan labeling algorithm. However, the mergences of the equivalent-label sets in this algorithm are time-consuming. Besides, it needs to find the smallest label in each merged set, which is inefficient. Similar to [17], a run-based onescan algorithm was proposed in [18], which essentially needs to scan the whole image twice. In [19], He et al. introduced a one and a half scan algorithm which only checks the foreground pixels in the second scan. It should be noted that all the run-based algorithms above scan only one row of pixels at each time and they all need to dynamically adjust the equivalent-label sets. These weaknesses lead to a reduction of time efficiency. Therefore, to overcome these short‐ comings, we introduce a new run-based algorithm which scans double rows of pixels at each time. Besides, the Dulmage-Mendelson (DM) decomposition is applied to search for the equivalent-label sets in linear time.

2

Proposed Algorithm

2.1 The First Scan In the following discussions, component connectivity is defined in the sense of eightadjacent regions in binary images. The proposed algorithm contains the first scan, equivalent-label sets searching and the second scan. Detail implementations are stated as follows.

266

D. Ma et al.

In the first scan, unlike the existing run-based algorithms, the run in our algorithm is defined in double rows, which consists of a succession of two-by-one pixel blocks. Each block contains one foreground pixel at least. For example, runs in the double rows in Fig. 1(a) as the red bold boxes show. Under the above definition, it is easy to see that all the foreground pixels in the same run are connected to each other. So, we give each run a single label. Then a record table is built to record the row numbers, start column numbers, end column numbers and provisional labels of all the runs, as shown in Fig. 1(b). For each run, we give it a provisional label according to its connectivity to runs in the last two rows. Through analysis we conclude that there are three cases when two runs are considered to be connected to each other, as shown in Fig. 2, from which we can see that if there is one pair of connected pixels between two runs, both runs are connected to each other. According to Fig. 2, we summarize { } {the}detail computation procedures of the provisional label in Table 1. In Table 1, si , ei , i = 1, 2, ⋯ , K are the start and end column number sets, where K is the number of runs in the last two rows. s and e are the start and [end]column[ numbers of the current run. Li represents the ] ith run in the last two rows. Li 1, j and Li 2, j are the two pixels with column number j [ ] [ ] in Li. C 1, j and C 2, j are the two pixels with column number j in the current run. MAX and MIN are functions calculating the maximum and minimum numbers. 1

1

1

1

4

2

1

0

1

0

0

1

1

0

1

1

0

1

3

1

0

0

1

0

0

1

1

1

0

0

1

6 10 3

(a)

(b)

Fig. 1. Runs in double rows and the record table. (Color figure online)

Run in last two rows

...

...

... 1

...

1

...

Run in last two rows

...

...

...

... 1 1

...

...

Run in last two rows

...

...

...

... 1 1

...

... ...

Current run

Current run

Current run

(a)

(b)

(c)

Fig. 2. Three cases of connected runs.

Run-Based Connected Components Labeling Using Double-Row Scan

267

Table 1. Pseudocode for computing the provisional label.

Algorithm: Computation of the provisional label. Input: C , s , e , {si } , {ei } , { Li } . Output: A provisional label for the current run. 1: flag = 0, pro_label = 0; 2: for each pair ( si , ei ) do 3:

if si ≤ e + 1 and ei ≥ s − 1 then

4:

start = MAX( si , s − 1 );

5:

end = MIN( ei , e + 1 );

6:

for j = start to end do

7:

if C [1, j ] =1 and L [ 2, k ] = 1, k = j − 1 or j , j + 1 then

8: if flag = 0 then 9: pro_label = the ith run’s provisional label; 10: flag = 1; 11: break; 12: else 13: temp = ith run’s provisional label; 14: add (pro_label, temp) into the equivalent-label table 15: break; 16: end if 17: end if 18: end for 19: end if 20: end for 21: return: pro_label.

In determining the provisional label of the current run, if multiple runs in the last two rows are connected to the current run, the provisional label of the current run will be the same as the first connected run. If there are no runs in the last two runs being connected to the current run, then a new label will be assigned to the current run. In Table 1 we also point out how to build the equivalent-label table. In the last two rows, if there are more than one runs being connected to the current run, the provisional labels of these runs are considered to be equivalent and stored in a two-column equivalentlabel table. In each row of the equivalent-label table, the first member is the provisional label of the first connected run, and the second member is a provisional label of one of other connected runs. For an example, in Fig. 3(a) there are three connected runs with provisional labels l1, l2 and l3 respectively, so l1 will be given as the provisional label to ( )( ) the current run. l1, l2 and l3 are considered to be equivalent and two pairs l1 , l2 , l1 , l3 are stored in the equivalent-label table as shown in Fig. 3(b).

268

D. Ma et al.

l2 1 1

...

...

... 1 1

l3

...

...

...

1 1

... ...

...

l1

... ...

l1 l2 l1 l3

l1

...

(a)

(b)

Fig. 3. Building the equivalent-label table

Compared with the conventional run-based algorithms, the proposed algorithm has two advantages. First, the number of runs to store is reduced, which reduces the subse‐ quent processing time. This can be seen by Fig. 1(a), for which in conventional one-row scan algorithms it needs to store seven runs while in ours it only needs to store three. Second, as the example in the third run in Fig. 1(a) shows, the size of the equivalentlabel table is reduced because different runs in one row are unified into a run due to the usage of double-row scan. Since the time spent on the subsequent steps is directly proportional to the sizes of record table and the equivalent-label table, the proposed algorithm (is time) efficient. The time complexity of the first scan in the proposed algo‐ rithm is O W 2 H , where W and H are the width and height of the input image. The primary goal of the first scan is to build the record table and equivalent-label table. Since the order of labels in both tables has no effect on the subsequent steps, the first scan can be executed in parallel. A simple way is to divide the image into several parts and scan them simultaneously, and then combine all the record tables and equiv‐ alent-label tables respectively, to obtain the whole record table and equivalent-label table. 2.2 Equivalent-Label Sets Searching After getting the equivalent-label table, the key step is to find out all the equivalent-label sets. In these sets, none of them contains the same label with other sets. Based on the provisional labels, an adjacent matrix can be built as Fig. 4(a) shows. The number of provisional labels in Fig. 4(a) is 9. If two labels are equivalent, then the element on the cross position is set to be 1, otherwise 0. The adjacent matrix is symmetric and generally sparse. It has been proved that a symmetric matrix can be decomposed into a block triangular form using the DM decomposition [20–22]. As Fig. 4(b) shows, vertices in the same block are connected to each other while disconnected with all the vertices in other blocks. Therefore, the equivalent-label sets can be obtained through the extraction of vertices from these triangular blocks.

Run-Based Connected Components Labeling Using Double-Row Scan

269

1

2

3

4

5

6

7

8

9

1

2

3

4

7

8

5

6

9

1

1

0

0

0

0

0

0

0

0

1

1

0

0

0

0

0

0

0

0

2

0

1

1

1

0

0

0

0

0

2

0

1

1

1

0

0

0

0

0

3

0

1

1

0

0

0

1

0

0

3

0

1

1

0

1

0

0

0

0

4

0

1

0

1

0

0

0

0

0

4

0

1

0

1

0

0

0

0

0

5

0

0

0

0

1

1

0

0

1

7

0

0

1

0

1

1

0

0

0

2,3, 4, 7,8

6

0

0

0

0

1

1

0

0

0

8

0

0

0

0

1

1

0

0

0

5, 6,9

7

0

0

1

0

0

0

1

1

0

5

0

0

0

0

0

0

1

1

1

(c)

8

0

0

0

0

0

0

1

1

0

6

0

0

0

0

0

0

1

1

0

9

0

0

0

0

1

0

0

0

1

9

0

0

0

0

0

0

1

0

1

Decomposition

(a)

1 Extraction

(b)

Fig. 4. Procedures of equivalent-label sets searching. (a) Adjacent matrix. (b) Block triangular form. (c) Extraction of equivalent-label sets.

An implementation of the DM decomposition using the bipartite graph was proposed in [23], while it was used for system reduction. Here, we design a new algorithm in Table 2 to search for the equivalent-label sets. Suppose the number of the provisional labels is M , and T denotes the equivalent-label table. V represents the vertex set {1, 2, ⋯ , M} and M ′ is the number of rows in the equivalent-label table. Detail imple‐ mentations are summarized in Table 2. Table 2. Equivalent-label sets searching algorithm.

Algorithm: Equivalent-label sets searching. Input: V , T . Output: All the equivalent-label sets. Step1: Add equivalent pairs (1,1) , ( 2, 2 ) ,

, ( M , M ) into T and T ′ denotes

the new table; Step2: Build up the bipartite graph using V and T ′ , G = (V , T ′ ) denotes the bipartite graph; Step3: Search for all the strong connected vertices in G using the tarjan algorithm [24]. Step4: Assign numbers from 1 to all the strong connected vertex set.

Because the adjacent matrix is generally sparse, to save the memory space, only the 1 s in the adjacent matrix are stored. Execution time spent on the step1, step2 and step4 in Table 2 are all linear with M . The time complexity of step3 is O(M + M ′ ) [24]. Since M ′ is close to M when the adjacent matrix is sparse, the time complexity of the whole procedure of equivalent-label sets searching is about O(M). 2.3 The Second Scan In the second scan, a number from 1 is assigned to each equivalent-label set. For example, in Fig. 4(c) three numbers 1, 2, and 3, are given to the equivalent-label sets

270

D. Ma et al.

respectively. Here, we use an array R as a lookup table to relate the provisional labels to the numbers. For each provisional label we have R[i] = the number for the equivalent − label set which contains i .

Then all the foreground pixels in the ith run are labeled with R[i]. As shown in Fig. 1, all the runs are independent with each other. So, the labeling procedure in the second scan can be executed in parallel. The time complexity of the second scan is O(S), where S is the number of foreground pixels.

3

Experimental Results

In the following experiments, all the algorithms are accomplished by C++ on a PC with 8 Intel Core i7-6700 CPUs, 3.4 GHz, 8 GB RAM, and a single core is used for the processing. We compare our algorithm with four state-of-the-art algorithms BBDT [12], CCIT [14], CTB [13], and RBTS [17], among which the first three algorithms are blockbased and the last one is run-based with single-row scan. First, we compare the efficiency of five algorithms on 19 4096 × 4096 synthetic images with different densities of fore‐ ground pixels. The densities range from 0.05 to 0.95. The number of connected compo‐ nents in each image is fixed to 900. The result in Fig. 5(a) shows that our algorithm outperforms the other algorithms when the density is lower than about 0.8. We also tested 19 2048 × 2048 synthetic images with different densities of foreground pixels and the result was similar.

(a)

(b)

Fig. 5. Algorithm performance on synthetic images. (a) Execution time per image on images with different densities of foreground pixels. (b) Execution time per image on images with different number of connected components.

To study the relationship between the time efficiency of the five algorithms and the number of connected components, 22 4096 × 4096 synthetic images with different number of connected components are tested. Here, the density of foreground pixels in each image is fixed to 0.49. N is the number of connected components. The result is shown in Fig. 5(b), from which we can see that the proposed algorithm outperforms the

Run-Based Connected Components Labeling Using Double-Row Scan

271

other algorithms when the number of connected components is lower than about 10^4. Combining Figs. 5(a) and (b), we can conclude that when the foreground pixels are sparse and the number of connected components is small, the proposed algorithm will perform best. In natural images, generally the region of interest (ROI) consists of several connected components. While the number of ROIs is small, the number of foreground pixels may be considerable. To study the performance of the proposed algorithm further, four real image datasets introduced in [16] and two additional datasets are tested. These datasets include the MIRflickr dataset [25], Hamlet dataset, Tobacco800 dataset [26], 3DPeS dataset [27], Medical dataset [28] and the Fingerprints dataset [29]. Images in these datasets are of various sizes and foreground pixel densities. Samples in each dataset as Fig. 6 shows.

(a) MIRflickr

(b) Hamlet

(c) Tobacco800

(d) 3DPeS

(e) Medical

(f) Fingerprints

Fig. 6. Sample images in six datasets

Table 3 presents the average execution time of the five algorithms. From the table we can see that our algorithm outperforms the other algorithms on Hamlet, Tobacco-800, 3DPeS, and Medical datasets. While on MIRflickr and Fingerprints datasets, it is inferior to BBDT and CCIT. The reason why the proposed algorithm is less efficient on these two datasets is that the foreground pixels in images of both datasets are dense, which leads to a large size of record table. On the other side, in BBDT and CCIT there are no use of record table, so when the foreground pixels are dense, BBDT and CCIT run faster. Nevertheless, compared with the conventional run-based algorithm RBTS, our

272

D. Ma et al.

algorithm keeps ahead all the time. In addition, our algorithm can be easily executed in parallel, both in the first scan and the second scan. While for block-based algorithms such as BBDT, CCIT and CTB, parallel execution is difficult to be conducted in the first scan since the steps in the first scan in these algorithms are closely linked and interde‐ pendent. Therefore, easy parallelism is another advantage of our algorithm. Table 3. Execution time per image on real datasets (ms). Dataset MIRflickr Hamlet Tobacco800 3DPeS Medical Fingerprints Average

4

BBDT [12] 0.67 9.64 14.25 1.18 4.18 0.60 5.087

CCIT [14] 0.75 11.63 17.18 1.39 5.07 0.66 6.113

CTB [13] 1.00 15.50 25.13 2.13 6.88 0.83 8.578

RBTS [17] 0.98 11.08 15.89 1.32 5.04 0.84 5.858

Ours 0.79 9.48 13.22 1.07 4.16 0.73 4.908

Conclusions

A new run-based CCL algorithm is proposed in this paper. Compared with the conven‐ tional algorithms, it reduces the sizes of record table and the equivalent-label table using double-row scan. In addition, a fast equivalent-label sets searching method using the sparse matrix decomposition is designed to improve the time efficiency. Since all the runs are independent with each other, both the first scan and the second scan can be executed in parallel. Comparative experiments are conducted on synthetic images and real image datasets. Results demonstrate that our algorithm outperforms the state-ofthe-art algorithms, especially when the foreground pixels are sparse and the number of connected components in the image is small. Future work will focus on the optimization of the current algorithm.

References 1. Hu, S., Zhang, F., Wang, M., et al.: PatchNet: a patch-based image representation for interactive library-driven image editing. ACM Trans. Graph. 32(6), 196 (2013) 2. Haralick, R.M., Shapiro, L.G.: Computer and Robot Vision, vol. I. Addison-Wesley, Boston (1992) 3. Suzuki, K., Horiba, I., Sugie, N.: Linear time connected component labeling based on sequential local operations. Comput. Vis. Image Underst. 89(1), 1–23 (2003) 4. Bailey, D.G., Johnston, C.T., Ma, N.: Connected components analysis of streamed images. In: International Conference on Field-Programmable Logic and Applications, pp. 679–682. IEEE, New York (2008) 5. Klaiber, M.J., Bailey, D.G., Baroud, Y.O., Simon, S.: A resource-efficient hardware architecture for connected component analysis. IEEE Trans. Circ. Syst. Video 26(7), 1334– 1349 (2016)

Run-Based Connected Components Labeling Using Double-Row Scan

273

6. Rosenfeld, A., Pfaltz, J.L.: Sequential operations in digital pictures processing. J. Assoc. Comput. Mach. 13(4), 471–494 (1966) 7. Lumia, R., Shapiro, L.G., Zuniga, O.: A new connected components algorithm for virtual memory computers. Comput. Graph. Image Process. 22(2), 287–300 (1983) 8. He, L., Chao, Y., Suzuki, K., Wu, K.: Fast connected-component labeling. Pattern Recognit. 32(9), 1977–1987 (2009) 9. He, L., Chao, Y., Suzuki, K.: An efficient first-scan method for label-equivalence-based labeling algorithms. Pattern Recognit. Lett. 31(1), 28–35 (2010) 10. Chang, F., Chen, C.-J., Lu, C.-J.: A linear-time component-labeling algorithm using contour tracing technique. Comput. Vis. Image Underst. 93(2), 206–220 (2004) 11. Grana, C., Borghesani, D., Cucchiara, R.: Optimized block-based connected components labeling with decision trees. IEEE Trans. Image Process. 19(6), 1596–1609 (2010) 12. Grana, C., Baraldi, L., Bolelli, F.: Optimized connected components labeling with pixel prediction. In: Blanc-Talon, J., Distante, C., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2016. LNCS, vol. 10016, pp. 431–440. Springer, Cham (2016). https://doi.org/ 10.1007/978-3-319-48680-2_38 13. He, L., Zhao, X., Chao, Y., Suzuki, K.: Configuration-transition-based connected-component labeling. IEEE Trans. Image Process. 23(2), 943–951 (2014) 14. Chang, W.-Y., Chiu, C.-C., Yang, J.-H.: Block-based connected-component labeling algorithm using binary decision trees. Sensors 15(9), 23763–23787 (2015) 15. Santiago, D.J.C., Ren, T.I., Cavalcanti, G.D.C., Jyh, T.I.: Efficient 2×2 block-based connected components labeling algorithms. In: IEEE International Conference on Image Processing, pp. 4818–4822. IEEE, New York (2015) 16. Grana, C., Bolelli, F., Baraldi, L., Vezzani, R.: YACCLAB - Yet Another Connected Components Labeling Benchmark. In: Proceedings of International Conference on Pattern Recognition. IEEE, New York (2016) 17. He, L., Chao, Y., Suzuki, K.: A run-based two-scan labeling algorithm. IEEE Trans. Image Process. 17(5), 749–756 (2008) 18. He, L., Chao, Y., Suzuki, K., Itoh, H.: A run-based one-scan labeling algorithm. In: Proceedings of International Conference on Image Analysis and Recognition, pp. 93–102 (2009) 19. He, L., Chao, Y., Suzuki, K.: A run based one and a half scan connected component labeling algorithm. Int. J. Pattern Recognit. Artif. Intell. 24(4), 557–579 (2011) 20. Dulmage, A.L., Mendelsohn, N.S.: Coverings of bipartite graphs. Can. J. Math. 10, 517–534 (1958) 21. Pothen, A., Fan, C.J.: Computing the block triangular form of a sparse matrix. ACM Trans. Math. Softw. 16(4), 303–324 (1990) 22. Duff, I.S., Erisman, A.M., Reid, J.K.: Direct Methods for Sparse Matrices. Clarendon Press, Oxford (1986) 23. Ait-Aoudia, S., Jegou, R., Michelucci, D.: Reduction of constraint systems. In: Compugraphic, pp. 331–340 (1993) 24. Tarjan, R.: Depth first search and linear graph algorithms. In: Symposium on Switching and Automata Theory, vol. 1, no. 4, pp. 114–121 (1971) 25. Huiskes, M.J., Lew, M.S.: The MIR flickr retrieval evaluation. In: Proceedings of ACM International Conference on Multimedia Information Retrieval, pp. 39–43 (2008). http:// press.liacs.nl/mirflickr/ 26. The Legacy Tobacco Document Library (LTDL): University of California (2007). http:// legacy.library.ucsf.edu/

274

D. Ma et al.

27. Baltieri, D., Vezzani, R., Cucchiara, R.: 3DPeS: 3D people dataset for surveillance and forensics. In: Proceedings of Joint ACM Workshop on Human Gesture and Behavior Understanding, pp. 59–64 (2011) 28. Dong, F., Irshad, H., Oh, E.-Y., et al.: Computational pathology to discriminate benign from malignant intraductal proliferations of the breast. PLoS One 9(12), e114885 (2014) 29. Maltoni, D., Maio, D., Jai, A., Prabhakar, S.: Handbook of Fingerprint Recognition. Springer, London (2009). https://doi.org/10.1007/978-1-84882-254-2

A 3D Tube-Object Centerline Extraction Algorithm Based on Steady Fluid Dynamics Dongjin Huang1,2 ✉ , Ruobin Gong1, Hejuan Li1, Wen Tang3, and Youdong Ding1,2 (

)

1

Shanghai Film Academy, Shanghai University, Shanghai 200072, China [email protected], [email protected] 2 Shanghai Engineering Research Center of Motion Picture Special Effects, Shanghai 200072, China 3 Department of Creative Technology, University of Bournemouth, Fern Barrow, Poole BH12 5BB, UK

Abstract. Three-dimensional tubular objects are widely used in the fields of industrial design, medical simulation, virtual reality and so on. Because of the complex tubular structure with bifurcation, irregular surface and uneven distri‐ bution of inner diameter, creating the centerlines of tubular objects is accurately a challenge work. In this paper, we propose a novel two-stage algorithm for efficient and accurate centerline extraction based on steady fluid dynamics. Firstly, the liquid pressure cloud data is obtained by Finite Volume Method (FVM) to simulate Newtonian fluid in the inner space of 3D tube. And the Delaunay Tetrahedralization and the Marching Tetrahedra Method are used to extract isobaric surfaces. Secondly, the selected center points of these isosurfaces are orderly organized for constructing the centerline directed tree, from which the final continuous, smooth centerline is automatically generated by Catmull-Rom spline. The experimental results show that our approach is feasible for extracting the centerlines of tubular objects with high accuracy and less manual interven‐ tions, especially has good robustness on complex tubular structures. Keywords: Tubular object · Centerline extraction · Fluid dynamics Isobaric surface

1

Introduction

Centerline, known as skeleton or medial axis, can directly reflect shape feature and topological characteristic of original graphs. In 1964, the concept of centerline was first put forward by Blum [1], defining centerline as the trajectory of the center of maximum inscribed circle of the object. Thereafter, the new research direction on 2D/3D centerline extraction was founded. With advantage of retaining complete information of original models, centerline is extensively used in object detection, feature extraction, path plan‐ ning, virtual navigation, pattern recognition and so on. As one of the most common 3D objects, the tubular models are widely applied in industrial design, medical simulation, virtual reality, et al. For example, in industrial design, 3D tubular models are mostly used in design of pipe and tunnel, due to the © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 275–286, 2017. https://doi.org/10.1007/978-3-319-71598-8_25

276

D. Huang et al.

centerlines could visually display the overall shape of them, which makes it easy to calculate the length of the pipe and tunnel. The centerline information also provides guidance mainly involved in vascular model reconstruction and surgery simulation. In particular, the centerline of the blood vessel is used as the basic path for surgical path planning and virtual blood vessel navigation. In recent years, centerline extraction becomes a hot research field. There are two impediments making it difficult to extract the centerline of a 3D tubular model accu‐ rately: (1) the structures of some 3D tubular models, such as vein, blood vessels, etc., are complex tree-like structures with many small sub-branches. Especially, in the case of the vascular stenosis and occlusion that makes the internal structure of vessels even more complicated do exist in disease vessels (2) traditional centerline extraction approaches usually require lots of manual interventions to obtain accurate results, thus leading to low efficiency, and sometimes even may generate a non single-pixel center‐ line. To address the problems above, we first introduce the hydrodynamics theory into centerline extraction and propose a novel approach to extract the centerline of 3D tubular objects. We discover the correlation between the center positions of isobaric surfaces with the center of tube shaped structure. And our algorithm is based on the pressure layer characteristics caused by steady fluid in the tubular model. FVM was adopted to calculate N-S formula to simulate fluid dynamics. The pressure data was structured by Delaunay Tetrahedralization and isobaric surfaces were extracted using Marching Tetra‐ hedra Method by presetting the inlet and outlets boundaries. The tube directed tree was constructed for optimizing the feature points and the continuous and smooth tube center‐ line was fitted by Catmull-Rom spline. At last, we set two different kinds of experiments to validate the accuracy of our algorithm, and use this algorithm to obtain the centerlines of complex tube structures with good experimental results.

2

Related Work

Among the researches related to centerline extraction, many algorithms aiming at 2D images and 3D models have been proposed. The traditional algorithms can be classified into topological thinning method [2, 3], potential energy field method [4–6], distance transformation method [7, 8], etc. Although these methods can extract the centerline of tubular objects accurately, most of them have plenty of shortcomings like requiring manual intervention, high computation complexity, poor robustness and so on. In recent years, the segmentation-based method, the minimal path method and the Voronoi diagram method have gained widespread attention. In segmentation-based method, Kumar et al. [9] proposed to segment blood vessel and extract centerline by tracking a user-initiated vascular seed cross section. Smistad et al. [10] adopted GPU parallel computing to divide blood vessel model and extract centerline, which greatly improved the extraction speed. Schneider et al. [11] pre-assumed the position of center‐ line using multivariate Hough voting and oblique random forests and then adopted fast matching method to precisely determine the centerline of the blood vessel. However, these methods require a large amount of computation and manual intervention. With

A 3D Tube-Object Centerline Extraction Algorithm

277

regard to the minimal path method, Wink et al. [12] used Dijikstra algorithm and A* algorithm to calculate the minimum cost path through the pre-defined cost image. Jin et al. [13] introduced a minimum cost path method to extract the centerline of 3D treelike objects, which avoided spurious branches without requiring post-pruning. In addi‐ tion, the improved ant colony algorithm was utilized by Gao et al. [14] to extract the centerline of human thigh blood vessel. Jia et al. [15] combined multi-model and multiresolution within the minimal path framework, enabling the minimal path process to track the key centerline points at different resolution of the images. This kind of methods are mainly used in 2D images, which obtains the centerline by searching the shortest path to certain points in target objects. Even though it can achieve high precision, the method can not guarantee the connectivity and single-pixel of the extracted centerline, furthermore, corners may appear in the places with high curvature. The extraction algo‐ rithm based on Voronoi diagram utilized the Voronoi diagram to calculate the nearest neighbor of the spatial point set to achieve the centerline. Yang et al. [16] determined the position of the centerline by using the Voronoi diagram to integrate the minimum path. Bucksch and Lindenbergh [17] proposed a graph-based approach to extract the centerline from point clouds using collapsing and merging procedures in octree-graphs. Ma et al. [18] put forward a nearest neighbor approach to extract the centerline of the points sample on 3D object boundary where each point is associated with a normal vector. However, most of the methods are only suitable for calculating the centerline of simple polyhedron, but not for discrete models. While dealing with internal hollows are also difficult for them. Different from the above-mentioned approaches, we utilize the characteristics of isobaric surface under the circumstances when the object filled with steady fluid and present a novel centerline extraction algorithm, which can accurately extract the center‐ line of 3D tubular objects with only a little of manual intervention and is able to solve most problems of the traditional approaches.

3

Centerline Extraction Algorithm

3.1 Method Overview Figure 1 depicts a block diagram of the proposed centerline extraction method, mainly including steady fluid simulation and centerline extraction. In our method, a 3D tubular model is adopted as the input for the following algorithm. Firstly, after definition the inlet and outlets of 3D tubular model, we adopt the advancing-front method for the generation of tetrahedral mesh inner this model, and the FVM based fluid method iter‐ atively executes over this meshed model until the pressure of fluid tends to stable. Secondly, the discrete pressure point clouds data are processed by delaunay tetrahedr‐ alization and the isobaric surface set in the 3D tubular model is generated. Thirdly, the center points of isobaric surfaces are calculated for constructing the direct tree as the control points. Then the directed tree is optimized by removing the wrong bifurcation, repairing deficiency and removing dense feature points. Finally, the final 3D tubular model with the centerline result fitted by Catmull-Rom spline is output.

278

D. Huang et al.

Fig. 1. Block diagram of centerline extraction method

3.2 Steady Fluid Computation The simulation of the object fully filled with steady fluid will be executed before extracting the isobaric surfaces. There may be subtle differences on the distribution of isobaric surfaces under the circumstances of injecting different fluid. Therefore, Newto‐ nian fluid is utilized in our experiment and simulated by adopting viscous incompressible Navier-Strokes (N-S) formula. The momentum equations of three dimensions in space and the continuity equation are: ( ) �p � �u �v �w − + + �x �x �y �z �x

(1)

) ( 2 ) ( ) ( �p � v �2 v �2v �v � �u �v �w �v �v �v =� + + − � − +u +v +w + + y:� �t �x �y �z �x2 �y2 �z2 �x �x �y �z �y

(2)

( x:�

( z:�

�u �u �u �u +u +v +w �t �x �y �z

�w �w �w �w +u +v +w �t �x �y �z

)

( =�

)

( =�

�2u �2u �2 u + + 2 �x2 �y2 �z

)

�2w �2w �2 w + 2 + 2 �x2 �y �z

−�

) −�

( ) �p � �u �v �w + + − �x �x �y �z �z

�� �(�u) �(�v) �(�w) + + + =0 �t �x �y �z

(3) (4)

Where u, v, w are the fluid flow velocity components in the x-y-z directions respec‐ tively; �, �, p are density, viscosity coefficient, and pressure of the fluid field respectively. The wall of tube object is then set as a Lipschitz continuous boundary, the fluid motion equation satisfies the no-slip condition and the velocity on the boundary is zero. When the above-mentioned conditions satisfied, a fluid velocity field uunflow perpendic‐ ular to the boundary plane is set at the inlet position, meanwhile the pressure at boundary of the outlets position is set as zero. There are commonly three approaches to calculate numerical solution of control equation based on Eulerian grid: Finite Difference Method (FDM), Finite Element Method (FEM) and Finite Volume Method (FVM). FDM can not achieve high precision although it is simple and easy to calculate; FEM, having high accuracy but also high computational complexity, thus is suitable for complicated shape, but has bad real-time performance. While FVM is a combination of these two methods, making it suitable for fluid simulation and irregular grids. Taking into account these considerations, FVM is chosen to discretize the control equation, of which the basic idea is to divide the compu‐ tational domain into several non-overlapping control volumes to surround each node of grids, and integrate each control volumes to obtain a discrete set of equations.

A 3D Tube-Object Centerline Extraction Algorithm

279

The prerequisite of using FVM to solve the N-S equation is to divide the Eulerian grids of fluid flow space inside the objects. We adopt the advancing-front method [19] to perform tetrahedral meshing in spatial domain. To guarantee the calculation accuracy, we set more dense grids in the position close to the object wall. In addition, we use the staggered grid method to compute the pressure and velocity on the meshed space and set the control volume distribution around the grid nodes. Finally, by setting the grids to control the volume side length, fluid coefficient, etc., we can iterate the calculation of the pressure distribution inside objects. 3.3 Isobaric Surface Extraction We acquire isobaric surfaces in the object through isosurface extraction. Foremost, the discrete pressure point clouds data is processed by delaunay tetrahedralization. This step can be accomplished by TetGen [20], an available tetrahedral generation tool. It is then necessary to remove the wrong and abnormal tetrahedrons from the generated tetrahe‐ dron set, using the following tetrahedral quality formula to eliminate the useless tetra‐ hedrons [21]:

lrms =

12 V qtetrahedron = √ 3 2 lrms √ l12 + l22 + l32 + l42 + l52 + l62

(5)

(6)

6

Where V is the volume of tetrahedron, lrms is the square root of the six edges of tetrahedron. A quality threshold q′ is set to reserve the tetrahedron satisfying qtetrahedron > q′, by which we could remove those wrong tetrahedrons being flat and slender. The structure of tetrahedron data set we reserve are shown in Fig. 2, where Fig. 2(a) represents the tetrahedralized and optimized pressure data in elbow-like tube, and Fig. 2(b) in Y-type tube.

280

D. Huang et al.

(a)

(b) Fig. 2. Result of tetrahedralization and optimization of tetrahedrons: the left side is the original tetrahedralized data, the right side is the optimized data.

After removing the false tetrahedrons, we execute isosurface extraction on the reserved tetrahedrons by means of the Marching Tetrahedra Method. Firstly, we calcu‐ late the pressure on each isosurface, then stratify the pressure according to the predefined maximum pressure Pmax and the minimum pressure Pmin. Suppose that n layers pres‐ sure need being extracted. The Pmax and Pmin are ignored due to too little data, hence, the pressure of the m layer is:

Pm =

Pmax − Pmin ∗ m + Pmin , 1 ≤ m ≤ n n+1

(7)

Then we can extract n isobaric surfaces according to the value Pm obtained from the previous step. Multiple isobaric surfaces may be extracted under the same pressure, thus need to be separated in a certain way. To be specific, we select a triangular patch from the extracted isobaric surfaces and find all the triangular patches being connected with it according to the characteristics of successive triangular patches on the same isobaric surface. The triangular patches found in the previous step should be treated equally to form recursive operation until no adjacent triangular patches can be found, that is to say, we can separate this isobaric surface from others that own the same pressure value. Repeating the previous steps until all the triangular patches extracted under the same pressure are searched, which means the separation of the isobaric surfaces under the same pressure are finished. The isobaric surfaces extracted from elbow and Y-type tube are shown in Fig. 3(a) and (e).

A 3D Tube-Object Centerline Extraction Algorithm

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

281

Fig. 3. The whole centerline extraction process of elbow-like tube and Y-type tube.

3.4 Centerline Extraction Following the isobaric surface extraction is the generation of centerline. The first step is to calculate the center points of all isobaric surfaces as the control points of the centerline, of which the formula is as follows:

∑n Pc (x, y, z) =

i=0

Pi (xi , yi , zi ) n+1

(8)

Where, Pi (xi , yi , zi ) is the center point of the ith triangular patch on the isobaric surface. While the pressure of the center point is equal to the isobaric surface. The result is shown in Fig. 3(b) and (f).

282

D. Huang et al.

The connection of centerline is conditional on building the directed tree of the center points according to the order of the pressure layer. As the order of pressure from large to small, that is, beginning to traverse from the control points at the inlet where fluid being injected, we search the control point Pi+1 (xi+1 , yi+1 , zi+1 ) in the next pressure layer Pi+1 which is continuously connected to the control point Pi (xi , yi , zi ) till the last layer. Chances are that there will be multiple tube branches containing pressure layer Pi+1, leads to more than one control point under the same pressure layer. Thus we set a distance threshold R′: • When the distance between Pi+1 (xi+1 , yi+1 , zi+1 ) and Pi (xi , yi , zi ) is less than R′, we determine that Pi+1 (xi+1 , yi+1 , zi+1 ) and Pi (xi , yi , zi ) are in the same branch and they are continuous in the directed tree. And the former is set as the child node of the latter. • When the distance between Pi+1 (xi+1 , yi+1 , zi+1 ) and Pi (xi , yi , zi ) is more than R′, we determine that Pi+1 (xi+1 , yi+1 , zi+1 ) and Pi (xi , yi , zi ) are in the different branches and they are discontinuous in the directed tree. The directed tree of the elbow and Y-type tube is shown as Fig. 3(c) and (g). 3.5 Centerline Optimization From the above approaches, we can generate a centerline directed tree. However, there may be some flaws in the structure, due to which we need to optimize its structure. Optimization is needed in the following situations: Situation 1: As a result of the instability of the steady-state fluid isobaric surfaces, there might be redundant branches of the centerline. Through the determination of the outlet position of tube, we retrieve the tree to see whether the terminal node position is corresponding to the outlet. If not, this branch is deleted. Situation 2: Some deficiency may exist on the centerline at the inlet and outlet posi‐ tion. Detecting whether the leaf node of directed tree is accurately at the inlet or outlet position, if not, connecting the leaf node to the center of the nearest inlet or outlet to complement the centerline. Situation 3: The nodes of centerline generated by previous steps may be too dense in local, which will cause the fitting speed slow and some local fluctuation on the fitting curve. In Fig. 4, pi−2, pi−1, pi, pi+1, pi+2 are the five continuous feature nodes, the angle � is the included angle of p̂ i−2 pi−1 and p̂ i+1 pi+2. If � > Thres1 (Thres1 is a threshold), the node pi needs to be removed. If dist(pi−1 , pi ) < Thres2 (Thres2 is a threshold), node pi needs to be removed. The value of Thres1 and Thres2 depends on the input model.

A 3D Tube-Object Centerline Extraction Algorithm

283

Fig. 4. Feature nodes optimization

The Catmull-Rom spline is adopted to match the control points in order that the final path is smoother. After completing the optimization steps, the final directed tree is connected and we can achieve the centerline of three-dimensional tubular objects. The centerline directed tree and the final result of extraction are shown as Fig. 3(d) and (h).

4

Experiment Results

4.1 Validation In order to verify the correctness of our algorithm, we extract the centerline of two regular tubular structures (Straight tube and C-type tube), and calculate the errors between the standard centerline and our results. Regular Straight Tube Structure. In this experiment, we compare and analyze the accuracy error of the extracted centerline in regular straight tube under different number of pressure level. Figure 5(a) shows the original isobaric surface extraction result with 20 levels, Fig. 5(b) shows the optimized center point, and Fig. 5(c) shows the final fitting centerline.

(a)

(b)

(c)

(e)

(f)

(g)

Fig. 5. Centerline extraction result of regular straight and C-type tube.

284

D. Huang et al.

Then we try to extract the centerline of regular straight tube under different pressure levels. The number of points which C-R spline generates each time is 50. We make a comparison between the extracted centerline and the ground truth of centerline, and consider the error of each spline point as the nearest distance to the ground truth, the validation result under different pressure levels are indicated in Table 1, where the Average Error is average error of all spline points. From the result we can see that the more pressure levels we divide, the higher precision of generated centerline we will get. Table 1. The error value of straight tube. Pressure level Average error

10 20 30 40 0.024 0.022 0.019 0.018

Regular C-Type Tube Structure. Applying our algorithm on the regular C-Type tube, and the extraction result under 80 levels is shown in Fig. 5(e), (f) and (g). The number of points which C-R spline generates each time is still 50 and the errors under different pressure levels are indicated in Table 2. From which we can draw the same conclusion as regular straight tube, and the accuracy of centerline tends to be stable when reaching the certain pressure levels. Table 2. The error value of C-type tube. Pressure level Average error

15 30 60 80 0.023 0.020 0.018 0.018

4.2 Complex Tubular Structure In this section, we apply our algorithm on some tubular structures that are more complex. We first extract the centerline of a complex regular tube which has several bends with high curvature, and it still performs well, the result is shown as Fig. 6(a). Then a segment of human vessel with several branches is extracted, of which the surface is also rough and not regular, the result is shown in Fig. 6(b). From the results of these two experiments, we confirm that our algorithm is universal and robust, which can be used to extract the centerline of complex tubular object with high accuracy.

A 3D Tube-Object Centerline Extraction Algorithm

(a)

285

(b)

Fig. 6. Centerline extraction result of complex regular tube and human vessel.

5

Conclusion

In this paper, a novel approach of 3D tubular object centerline extraction based on steady-state fluid dynamics is proposed. In fluid simulation, we adopt the Finite Volume Method to compute the N-S formula to simulate the steady-state fluid in the objects and obtain the pressure point clouds data. In isobaric surface extraction, the Marching Tetra‐ hedra Method is used to extract the isobaric surfaces from the tetrahedralized pressure data. In directed tree construction, the construction of the directed tree and the removal of the excess branches are carried out by using the characteristics of the pressure distri‐ bution as well as the inlet and outlets positions. Finally, a smooth, single-pixel centerline is fitted through the C-R spline. In the future work, we will apply this centerline extraction method to the path plan‐ ning and navigation of minimally invasive vascular surgery to achieve a high-precision, eye-coordinated surgical path planning system.

References 1. Blum, H.: A transformation for extracting descriptors of shape. Models Percept. Speech Vis. Form 19, 362–380 (1967) 2. Paik, D.S., Beaulieu, C.F., Jeffrey, R.B., Rubin, G.D., Napel, S.: Automated flight path planning for virtual endoscopy. Med. Phys. 25(5), 629–637 (1998) 3. Wang, S., Wu, J., Wei, M., Ma, X.: Robust curve skeleton extraction for vascular structures. Graph. Models 74(4), 109–120 (2012) 4. Ahuja, N., Chuang, J.H.: Shape representation using a generalized potential field model. IEEE Trans. Pattern Anal. Mach. Intell. 19(2), 169–176 (1997)

286

D. Huang et al.

5. Chuang, J.H., Tsai, C.H., Ko, M.C.: Skeletonisation of three-dimensional object using generalized potential field. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1241–1251 (2000) 6. Sharf, A., Lewiner, T., Shamir, A., Kobbelt, L.: On-the-fly curve-skeleton computation for 3D shapes. Comput. Graph. Forum 26(3), 323–328 (2007). Blackwell Publishing Ltd. 7. Peng, Y., He, M., Lu, X., Shi, J.: Realization technologies in a virtual endoscopy system. In: 2010 Second International Workshop on Education Technology and Computer Science (ETCS), vol. 1, pp. 43–46. IEEE, March 2010 8. Hernández-Vela, A., Gatta, C., Escalera, S., Igual, L., Martin-Yuste, V., Sabate, M., Radeva, P.: Accurate coronary centerline extraction, caliber estimation, and catheter detection in angiographies. IEEE Trans. Inf. Technol. Biomed. 16(6), 1332–1340 (2012) 9. Kumar, R.P., Albregtsen, F., Reimers, M., Edwin, B., Langø, T., Elle, O.J.: Threedimensional blood vessel segmentation and centerline extraction based on two-dimensional cross-section analysis. Ann. Biomed. Eng. 43(5), 1223–1234 (2015) 10. Smistad, E., Elster, A.C., Lindseth, F.: GPU accelerated segmentation and centerline extraction of tubular structures from medical images. Int. J. Comput. Assist. Radiol. Surg. 9(4), 561–575 (2014) 11. Schneider, M., Hirsch, S., Weber, B., Székely, G., Menze, B.H.: Joint 3-D vessel segmentation and centerline extraction using oblique Hough forests with steerable filters. Med. Image Anal. 19(1), 220–249 (2015) 12. Wink, O., Niessen, W.J., Viergever, M.A.: Minimum cost path determination using a simple heuristic function. In: Proceedings of 15th International Conference on Pattern Recognition, 2000, vol. 3, pp. 998–1001. IEEE (2000) 13. Jin, D., Iyer, K.S., Chen, C., Hoffman, E.A., Saha, P.K.: A robust and efficient curve skeletonization algorithm for tree-like objects using minimum cost paths. Pattern Recogn. Lett. 76, 32–40 (2016) 14. Gao, M.K., Chen, Y.M., Liu, Q., Huang, C., Li, Z.Y., Zhang, D.H.: Three-dimensional path planning and guidance of leg vascular based on improved ant colony algorithm in augmented reality. J. Med. Syst. 39(11), 133 (2015) 15. Jia, D., Shi, W., Rueckert, D., Liu, L., Ourselin, S., Zhuang, X.: A multi-resolution multimodel method for coronary centerline extraction based on minimal path. In: Zheng, G., Liao, H., Jannin, P., Cattin, P., Lee, S.-L. (eds.) MIAR 2016. LNCS, vol. 9805, pp. 320–328. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-43775-0_29 16. Yang, F., Hou, Z.G., Mi, S.H., Bian, G.B., Xie, X.L.: Centerlines extraction for lumen model of human vasculature for computer-aided simulation of intravascular procedures. In: 2014 11th World Congress on Intelligent Control and Automation (WCICA), pp. 970–975. IEEE, June 2014 17. Bucksch, A., Lindenbergh, R.: CAMPINO—a skeletonization method for point cloud processing. ISPRS J. Photogrammetry Remote Sens. 63(1), 115–127 (2008) 18. Ma, J., Bae, S.W., Choi, S.: 3D medial axis point approximation using nearest neighbors and the normal field. Visual Comput. 28(1), 7–19 (2012) 19. Ito, Y., Shih, A.M., Erukala, A.K., Soni, B.K., Chernikov, A., Chrisochoides, N.P., Nakahashi, K.: Parallel unstructured mesh generation by an advancing front method. Math. Comput. Simul. 75(5), 200–209 (2007) 20. Si, H.: TetGen, a Delaunay-based quality tetrahedral mesh generator. ACM Trans. Math. Softw. (TOMS) 41(2), 11 (2015) 21. Müller, M., Chentanez, N., Kim, T.Y., Macklin, M.: Air meshes for robust collision handling. ACM Trans. Graph. (TOG) 34(4), 133 (2015)

Moving Objects Detection in Video Sequences Captured by a PTZ Camera Li Lin, Bin Wang ✉ , Fen Wu, and Fengyin Cao (

)

School of Communication and Information Engineering, Shanghai University, Shanghai 200072, China [email protected]

Abstract. To solve the problem of detecting moving objects in video sequences which are captured by a Pan-Tilt-Zoom (PTZ) camera, a modified ViBe (Visual Background Extractor) algorithm, which is a pixel-based background modelling algorithm, is proposed in this paper. We divide a changing background scene into three parts. The first part is the new background region if a PTZ camera’s field of view has been changed and we re-initialize background model of this part. The second is the disappeared area in the current frame and we decide to discard their models to save memory. Then the third part is the overlapping background region of consecutive frames. Via matching SURF feature points which are extracted only in background region we obtain an accurate homography matrix between consecutive frames. To ensure that the corresponding model from the former frame can be used in the current pixel, the homographic matrix should show a forward mapping relationship between the adjacent frames. Efficiency figures show that compared with origin ViBe algorithm and some other state-of-the-art background subtraction methods, our method is more affective for video sequences captured by a PTZ camera. More importantly, our method can be used in most of pixel-based background modelling algorithms to enhance their performance when dealing with videos captured by a moving camera. Keywords: Detecting moving objects · PTZ camera · Background subtraction

1

Introduction

Moving objects detection is widely exploited as being the first layer of many computer vision applications, such as vehicle tracking [1], people counting [2] and many other related fields [3, 4]. In the last few years, various state-of-the-art background subtraction methods to detecting moving objects are proposed for video surveillance system with static camera. Simple moving objects detection algorithms regard a static frame as background reference. While finding an exact correct background reference is almost impossible due to the dynamic nature of real-world scenes. In order to adjust dynamic background and segment more accurate moving objects (foreground) from scenes, building a background model becomes the ‘mainstream’ approach. This is the basic

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 287–298, 2017. https://doi.org/10.1007/978-3-319-71598-8_26

288

L. Lin et al.

principle of background modelling: the current pixel or region compares with its back‐ ground model, after that, unmatched areas will be labeled as foreground. Finally it will generate a binary mask to distinguish background and foreground. Many well-known background modelling algorithms, like ViBe, Amber, SuBSENSE, etc. have achieved high-quality motion detection in video sequences captured by a stationary camera. However, when the stationary camera or PTZ cameras change their viewing area, these approaches are not suitable anymore. Several difficulties in detecting motion based on PTZ cameras are listed as follows. (a) Motion Estimation Error of camera. Motion between consecutive frames includes two independent parts: active motion of camera and motion of objects. Error is inevitable when estimating movement information of PTZ camera from video sequences. Such accumulative errors may have a badly influence on subsequent detection. (b) Multiresolution. PTZ cameras have zoom-in zoom-out functions so that the same scene can be scanned by different resolutions. The background pixels undergoing these complex changes tend to be misclassified as foreground objects. (c) Real-time. Many attempts have been accomplished to detect motion in a moving background by building a panoramic background image. Such background refer‐ ence may perform well in a static scene because it can cover the whole area shoot by PTZ cameras. However, in order to store and make use of this large model, more memory and computational power will be required. In this paper, we present a background modelling method to detect motion in video sequences which are captured by a PTZ camera. A basic background modelling algo‐ rithm, ViBe in [5], is employed to illustrate that our method can enhance performance of most pixel-based modelling algorithm when dealing with a moving scene. It changes the situation that most existing background modelling algorithms can not be applied to PTZ camera-based systems due to the presence of varying focal lengths and scene changes. The remainder of this paper is organized as follows: In Sect. 2, we have a review on some typical background subtraction algorithms based on stationary cameras and PTZ cameras. The review introduces the main principle and relative merits of each algorithm briefly. Section 3 explains three key issues about background modelling and describes a modified ViBe algorithm in detail. Then we discuss experimental results and compare our results to other algorithms in Sect. 4. Section 5 concludes the paper.

2

Related Work

Over the recent years, numerous background modelling methods [5–9] have been devel‐ oped. Most of these methods are just based on a static background. Gaussian Mixture Models (GMM) [6, 7] is widely used in real-life scenarios to handle a dynamic complex background (e.g. rain, swaying tree leaves, ripples). Non-parametric model based on Kernel Density Estimation (KDE) [8] also estimates background probability density functions. But differing from GMM, its background probability density functions

Moving Objects Detection in Video Sequences Captured

289

depend directly on the very recent observations at each pixel location. The feature is also important to building a background model. SuBSENSE (Self-Balanced SENsitivity SEgmenter) [9] proposed that individual pixels are characterized by spatiotemporal information based on color values and Local Binary Similarity Pattern (LBSP) features which describe local textures for background modelling. To detect motion in video sequences captured by a PTZ camera, we should have knowledge about existing methods aim at a moving background. In general, the methods in the literatures of PTZ camera contain two main types: frame-to-frame (F2F) and frame-to-global. Frame-to-frame methods focus on the relationship between the consec‐ utive frames. The current frame can reuse the information of overlapping regions from the previous frame. Kang et al. [11] present an adaptive background generation algo‐ rithm using a geometric transform-based mosaicking method. A homogeneous matrix, which describes a relation between adjacent images that have different pan and tilt angles, is used to project the existing background into the new image. This method, which differs from obtaining camera parameters by sensors directly, does not have to know the internal parameters of the PTZ camera. An algorithm proposed in [12] esti‐ mates parameters of the PTZ camera from meta data and frame-to-frame correspond‐ ences at different sampling rates. Besides using the low sampling frequency of meta data, two extended Kalman filters which uses the high frequency F2F correspondences are designed to enhance estimation accuracy. Beyond that, some methods are proposed to detect and track moving objects in a moving scene by applying of optical flow infor‐ mation [13]. Frame-to-global methods emphasize building and maintaining a panoramic background image of the whole monitored scene. Generating a panoramic background image based on image mosaic then finding the same scene in the panoramic background image by image registration, finally detecting moving objects by background subtraction is the most common approach. The problem of how to produce a background image is always discussed. The simplest case is to pan the camera 360-degree around its optical center, after that a panoramic mosaic image can be constructed on a cylindrical, squared, or spherical manifold [14]. In [15], the method extracts and tracks feature points throughout the whole video stream, and then make use of reliable background point trajectories to generate a background image. Sudipta N. Sinha et al. [16] describe a hierarchical approach for building multi-resolution panoramas by aligning hundreds of images captured within a 1–12× zoom range. In [17], a panoramic Gaussian mixture model (PGMM) covering the PTZ camera’s field of view is generated off-line for later use in on-line foreground detection. In this paper, we build a pixel-based background model with a frame-to-frame method. Compared with frame-to-global methods, our approach does not need the prior information or the off line computation process, which is hard to satisfy the requirement of real-time.

3

The Modified ViBe Algorithm

Each method has its own advantages and disadvantages. A background modelling algo‐ rithm should deal with at least three key issues. (1) Initialization. The initial process

290

L. Lin et al.

determines elements in background models. Pixels are always characterized by color information. For more accurate results, texture information or mathematics operators can be added to background model, even if consuming more memory. Besides, the speed of initialization is another main factor to estimate algorithm performance. Methods, such as Gaussian mixture model and kernel density estimation, spend some time on training. But if a training process is set up in a model to detect motion in a moving background, it won’t generate an appropriate model. Scenes captured by a PTZ camera are not fixed, so it is almost impossible to obtain enough static images for training. When it comes to the appearance or disappearance of background scenes, training becomes more difficult. Therefore, rapid and simple initialization process should be adopted to build a moving background model. (2) Classification. The similarity between new pixel and its back‐ ground model decides whether the pixel belongs to background or foreground. In most cases, the decision threshold plays a key role in classification process. A high threshold causes false background pixels and many true background pixels will be omitted by a low threshold accordingly. Making thresholds adaptively is a good choice for different areas in a static scene. (3) Update. Changes in real-life scenes are inevitable. Each algo‐ rithm needs to choose its proper update policy to fit these changes. Which background model should be updated or how long can make one update? All kinds of background methods explain such problems in its update process. In the following, we will describe the modified ViBe algorithm in detail according to the above three aspects. The overall flow of the proposed approach is illustrated in Fig. 1. Initialize New BG

SURF

Homography

Extractor

Matrix

Delete Old BG

Overlapping BG

Transformation

Detection Results

Fig. 1. Flow chart of the proposed approach. At first, matching the SURF feature points and computing the homography matrix between adjacent frames, then dividing the observed image into new background and overlapping background and doing corresponding managements, finally using the detection result which serves as feedback to delete the feature points in foreground when the next frame is captured.

3.1 Initialization Compared with other classical algorithms, like Gaussian mixture model and kernel density estimation which initialized by some training frames, ViBe algorithm uses only one frame to initialize. Rapid and simple initialization is one of the remarkable advan‐ tages of ViBe algorithm.

Moving Objects Detection in Video Sequences Captured

291

First, a background model, noted by B, contains a set of N background samples for pixel located at x. { } B(x) = v1 , v2 , … , vN

(1)

where vi is the ith background samples value. Due to every pixel value has a similar distribution with its adjacent pixels, ViBe algorithm fills background model B(x) with eight neighborhoods pixel values of center pixel x. Usually the number of samples is N = 20. If size of video sequence is E × F, then the total size of background samples is E × F × N. It is random to select samples for a background model, so one of eight neighborhoods pixel value vy of center pixel x may appear in B(x) several times, or not even once. When the perspective of PTZ camera changes, the adjacent frames can be divided into three parts. Distinctions between consecutive frames are not obvious, so we choose the images across twenty frames in Fig. 2. As is shown, region A is a new background scene. B is the overlapping region which appeared in the former image. And C represents a disappeared place. Obviously, the current image is composed of region A and B. In the same way, region B and C comprise the former image.

(a)

(b)

Fig. 2. Three parts of (a) a latter frame and (b) a former frame: region A (new background), region B (overlapping background), region C (disappeared background).

Background models only in region A need to be reinitialized according to the initial approach above. It’s unnecessary to preserve background samples in disappeared areas, so models in region C are directly abandoned to save memory. In region B, consecutive frames share a same background samples. The approach to apply the previous frame’s background models to the current frame is described in Sect. 3.3. Therefore, initialization may operate in the whole modelling process, as long as background scene has spatial changes. 3.2 Classification After initialization, we start to detect motion. Moving objects detection, regarding as a classification process, labels every pixel as one foreground pixel or a background pixel by a certain kind of rules. Then through the post-processing, a binary mask, where white

292

L. Lin et al.

(pixel gray level = 255) represents foreground and black (pixel gray level = 0) represents background, is generated eventually. If the distance between the pixel value vx at location x and a given background sample value is smaller than the maximum distance threshold R, in other words, if inequality (2) is satisfied, we consider these two pixels are similar. |vx − vi | < R

(2)

When a pixel finds thmin or more similar samples in its background model, the pixel will be classified as background.

{ } NUM v1 , v2 , … , vN

{

≥ thmin background < thmin foreground

(3)

{ } where we fixed R = 20, thmin = 2. NUM v1 , v2 , … , vN returns the number of similar samples in background model.

3.3 Update Even if in static scene, the expectation, background without changes, almost never holds. Camera jitter, illumination or other background changes are unavoidable. But beyond that, translation, rotation, scaling of background scene captured by a PTZ camera lead to the most difficult problem. Reasonable update policy gives a hand to adapt such changes. In our case, consecutive frames have overlap region B. The current pixel x located at (i, j) cannot use the previous model at the same location directly. We need to transform the former model into the current based on the following way. The first major work figures out homography matrix between consecutive frames when the PTZ camera rotates around its optical center. We parameterize the pin angle of the PTZ camera by �, the tilt angle by � and the focal length by � . Hn+1,n presents a mapping relation from (n + 1)th frame to nth frame. Hn+1, n = Kn+1 Rn+1 RTn Kn−1

(4)

where

and

⎡ �n 0 0 ⎤ Kn = ⎢ 0 �n 0 ⎥ ⎢ ⎥ ⎣ 0 0 1⎦ ⎡ cos�n 0 sin�n ⎤ Rn = ⎢ sin�n sin�n cos�n cos�n sin�n ⎥ ⎥ ⎢ ⎣ sin�n cos�n −sin�n cos�n cos�n ⎦

(5)

(6)

Moving Objects Detection in Video Sequences Captured

293

In our method, we use SURF (Speeded Up Robust Features) to represent the corre‐ spondences between two images with the same scene or object. Such a series of detectors and descriptors are achieved by relying on integral images for image convolutions detailed in [18]. SURF is scale and rotation invariant. It outperforms many previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster. We extract and match SURF descriptors in two adjacent images. Note that these feature points are located only in background according to the classification results of the previous frame. When matches located in moving objects occur, homography matrix, which is computed through the correspondence rela‐ tionship among matching points, can’t express background transformation relation precisely. Shown in Fig. 3, Outliers are filtered by RANSAC algorithm, which achieves goals via repeatedly selecting a set of random date subset.

(a)

(b)

Fig. 3. RANSAC algorithm removes some mismatching points. (a) Original matching images. (b) Matching images after filtering.

To enable every current pixel in overlap region have a certain model from history, Homography matrix, noted H, indicates the mapping from the current image to the ′

′

former image. As shown in Fig. 4, the previous location (it−1, jt−1) of pixel x which located at (it, jt) may be a non-integral type. Thus we use bilinear interpolation to select back‐ ground sample values from previous models for x. Formulization (7) explain the way to figure out one of the background sample values. At last background model at x is updated through calculating bilinear interpolation for N times. vxt = mnvat−1 + m(1 − n)vbt−1 + (1 − m)(1 − n)vct−1 + (1 − m)nvdt−1

a

b n m

d

c

Fig. 4. The previous pixel and the current pixel based on forward mapping

(7)

294

L. Lin et al.

Meanwhile, we also need to update background model by inserting the current pixel x. Our method incorporates three of important components from ViBe algorithm: a conservative update policy, a random update policy and spatial propagation of back‐ ground samples. A conservative update policy considers a pixel, only if it has been classified as back‐ ground, qualified to update its background model. It means samples in foreground are never included in background model. Conversely, there is a blind update policy using not only background but also foreground to update models. The principal shortcoming of blind update is that it may lead to more false background and poor detection of slow moving objects. Many background modelling methods use first-in first-out policy to update. It holds that the recent background sample has more efficacies but the oldest does not. In spite of ignoring the importance of temporal relationship, updating background samples randomly is still simple but effective in our methods. Observation classified as back‐ ground will replace one of its background samples. The replaced sample is selected randomly. In other words, the probability of every sample being abandoned is 1/N. Considering together with spatial propagation of background samples, a sample in the model of a pixel y, which is one of eight connected neighborhood of x, is also replaced by the current observation. Such update policy takes into account spatial relationships among incoming pixel with its surrounding. The most difference of our method from original ViBe algorithm is the model update rate. ViBe algorithm sets its time subsampling factor as 16. But in terms of detection in moving background scenes, it is necessary to update each background pixel model more quickly. So that we update background model of pixel x for every new frame as long as x is classified as background.

4

Experiments

This section reports the performance of our approach with the experiment results on the PTZ video sequences from the public Change Detection benchmark 2014. The percentage of correct classification (PCC), which wants to be as high as possible, is used to evaluate our approach. PCC =

TP + TN TP + TN + FP + FN

(8)

where TP (True Positives) counts the number of correctly pixels classified as foreground, TN (True Negatives) counts the number of correctly pixels classified as background, FP means the number of background pixels incorrectly classified as foreground and FN accounts for the number of foreground pixels incorrectly classified as background. From detailed discussion from ViBe algorithm in [5], we fix the radius R = 20 and the decision threshold thmin = 2. The only difference of our method from original param‐ eters is the time sampling factor T, which set as 16 formerly. Detection results and PCCs for model time subsampling factor T ranged 1 to 7 are displayed in Fig. 5. Obviously the best results are obtained for T = 1. It seems that a smaller time subsampling factor,

Moving Objects Detection in Video Sequences Captured

295

which indicates a faster update of background model, may implement a more accurate result for moving background scenes.

(a)

(b)

(c)

(d)

(e)

Fig. 5. Detection results and PCCs for time subsampling factor T ranged 1 to 7. (a) Input image. (b) T = 1. (c) T = 3. (d) T = 5. (e) T = 7. (f) PCCs.

(a)

(b)

(c)

(d)

(e)

Fig. 6. Comparative pure segmentation results of three background modelling techniques for continuous pan video sequence. (a) Input images. (b) Ground truth. (c) Our methods results. (d) Original ViBe algorithm. (e) Gaussian Mixture Models

296

L. Lin et al.

Figure 6 shows input images, ground truth and pure detection result without any morphological operations or noise filtering. Edges as shown are almost eliminated by a post-process. Visually, the result of our methods is better. Background modelling algo‐ rithms, like GMM, are not proper anymore. Yet every method, so does ours, have a problem that foreground pixels will be initialized into background model when there are any moving objects in new background, furthermore such error lasts for a long time. Just in the same way to remove ghosts, if the current scene not going away immediately, this mistake will be resolved by spatial propagation of background samples in the end. We combine our idea with ViBe algorithm to illustrate that the performance after our processing is improved when dealing with videos captured by a moving camera. More importantly, our method can be used in most of pixel-based background modelling algorithms to enhance their performance. To compare our methods in handing such a moving background mathematically with original ViBe algorithm and other several methods, other metrics proposed in the change detection website are also considered here. F-Measure, which is the weighted harmonic mean of ‘precision’ and ‘recall’, indi‐ cates overall performance well. The ‘precision’ is the ratio between the number of correctly classified as foreground and the pixels which are classified as foreground regardless of the correct. The ‘recall’ is used to describe the accuracy of whether the true foreground pixels are correctly classified or not. So we use F-Measure to obtain a single measure to evaluate different methods then rank them. Precision = Recall =

F − Measure =

TP TP + FP

(9)

TP TP + FN

(10)

2 ∗ Precision ∗ Recall Precision + Recall

(11)

From the experimental result shown in Table 1, it can be clearly seen that our approach after sample post-process achieves much better performance than original ViBe algorithm and other pixel-based algorithms in detecting continuous panning back‐ ground. This indicates that our method is extremely beneficial to the original background modelling algorithm to adapt the difficult moving scenarios captured by a PTZ camera. Table 1. Average performance comparison of different models. Ours ViBe GMM KDE

PCC 0.9679 0.7822 0.7929 0.7516

Precision 0.1324 0.0148 0.0139 0.0181

Recall 0.7424 0.5135 0.4578 0.7261

F Measure 0.2247 0.0288 0.0270 0.0353

Moving Objects Detection in Video Sequences Captured

5

297

Conclusion

Over the recent years, numerous background modelling methods have been developed. However, most existing work proposed for fixed cameras can not be directly applied to PTZ camera-based systems due to the presence of varying focal lengths and scene changes. Furthermore, there is much less research work for PTZ camera-based back‐ ground modelling. Most methods generate a background mosaic image then use the simplest background difference method to obtain a binary mask. In this paper, we have presented a modified Vibe algorithm to detecting moving objects in video sequences which are captured by a PTZ camera. More importantly, our method can be used in most of background modelling algorithms to suit a moving scene. We tested the performance of the method in comparison with classical existing methods. It outperforms these methods in motion detection when the background scene keeps moving. As for future extension, we are trying to combine our method to other more complex pixel-based background modelling algorithms. In addition, a detailed analysis of different application with respect to a faster moving scene, which may shoot by car cameras or unmanned aerial vehicles, is also our future consideration. Acknowledgments. This work was supported by the National Natural Science Foundation of China (grant number: 61601280) and Key Laboratory for Advanced Display and System Applications (Shanghai University), Ministry of Education of China (grant number: P201606).

References 1. Lipton, A.J., Fujiyoshi, H., Patil, R.S.: Moving target classification and tracking from realtime video. In: IEEE Workshop on Applications of Computer Vision. IEEE Computer Society, p. 8 (1998) 2. Hou, Y.L., Pang, G.K.H.: People counting and human detection in a challenging situation. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 41(1), 24–33 (2011) 3. Cutler, R., Davis, L.: Real-time periodic motion detection, analysis, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 781–796 (1999) 4. Wren, C.R., Azarbayejani, A., Darrell, T., et al.: Pfinder: real-time tracking of the human body. In: International Conference on Automatic Face and Gesture Recognition, pp. 51–56. IEEE Xplore (1996) 5. Barnich, O., Van, D.M.: ViBe: a universal background subtraction algorithm for video sequences. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc. 20(6), 1709–1724 (2011) 6. Stauffer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. In: 1999. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, p. 252. IEEE Xplore (1999) 7. Zivkovic, Z.: Improved adaptive gaussian mixture model for background subtraction, vol. 2, pp. 28–31 (2004) 8. Elgammal, A., Harwood, D., Davis, L.: Non-parametric model for background subtraction. In: Vernon, D. (ed.) ECCV 2000. LNCS, vol. 1843, pp. 751–767. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45053-X_48

298

L. Lin et al.

9. Wang, B., Dudek, P.: AMBER: adapting multi-resolution background extractor. In: IEEE International Conference on Image Processing, pp. 3417–3421. IEEE (2014) 10. St-Charles, P.L., Bilodeau, G.A., Bergevin, R.: SuBSENSE: a universal change detection method with local adaptive sensitivity. IEEE Trans. Image Process. 24(1), 359–373 (2014) 11. Kang. S., Paik, J.K., Koschan, A., et al.: Real-time video tracking using PTZ cameras. In: Proceedings of SPIE - The International Society for Optical Engineering, vol. 5132, pp. 103– 111 (2003) 12. Wu, S., Zhao, T., Broaddus, C., et al.: Robust pan, tilt and zoom estimation for PTZ camera by using meta data and/or frame-to-frame correspondences. In: International Conference on Control, Automation, Robotics and Vision, pp. 1–7. IEEE (2007) 13. Doyle, D.D., Jennings, A.L., Black, J.T.: Optical flow background subtraction for real-time PTZ camera object tracking. In: Instrumentation and Measurement Technology Conference, pp. 866–871. IEEE (2013) 14. Mann, S., Picard, R.W.: Virtual bellows: constructing high quality stills from video. In: Image Processing, 1994, Proceedings, ICIP-94, IEEE International Conference, vol. 1, pp. 363–367. IEEE (2002) 15. Jota, K., Tsubouchi, T., Sugaya, Y., et al.: Extracting moving objects from a moving camera video sequence. IPSJ SIG Notes CVIM 2004, 41–48 (2004) 16. Sinha, S.N., Pollefeys, M.: Pan–tilt–zoom camera calibration and high-resolution mosaic generation. Comput. Vis. Image Underst. 103(3), 170–183 (2006) 17. Xue, K., Liu, Y., Ogunmakin, G., et al.: Panoramic gaussian mixture model and large-scale range background substraction method for PTZ camera-based surveillance systems. Mach. Vis. Appl. 24(3), 477–492 (2013) 18. Bay, H., Tuytelaars, T., Gool, L.V.: SURF: speeded up robust features. Comput. Vis. Image Underst. 110(3), 404–417 (2006)

Fast Grid-Based Fluid Dynamics Simulation with Conservation of Momentum and Kinetic Energy on GPU Ka-Hou Chan(B) and Sio-Kei Im MPI-QMUL Information Systems Research Centre, Macao Polytechnic Institute, Macao, China [email protected] Abstract. Since the computation of fluid animation is often too heavy to run in real-time simulation, we propose a fast grid-based method with parallel acceleration. In order to reduce the cost of computation keeping a balance between fluid stability and diversity, we consider the NavierStokes equation on the grid structure with momentum conservation, and introduce the kinetic energy for collision handling and boundary condition. Our algorithm avoids the mass loss during the energy transfer, and can be applied to the two-way coupling with a solid body. Importantly, we propose to use the forward-tracing-based motion and design for parallel computing on Graphics Processing Unit (GPU). In particular, these experiments illustrate the benefits of our method, both in conserving fluid density and momentum. They show that our method is suitable to solve the energy transfer when object interaction is considered during fluid simulation. Keywords: Computational Fluid Dynamics Momentum conservation · Kinetic energy

1

Introduction

Fluid effects play an important role in the computer games industry: these are the complex interplay of various phenomena, such as convection, diffusion, turbulence and surface tension. Considerable research has taken place to improve the behaviour and performance and with the ever-increasing performance of hardware, simulations of the underlying physics can better approximate natural dynamics for representing water, waves, fire and gas. Hence, currently researches propose to simulate the interaction of multiple materials (fluid and solid objects). Physically, the Navier-Stokes Equation describes the state of the fluid, and there are many methods for solving incompressible flow by this equation. 1.1

Physical Equation

It is well known in Computational Fluid Dynamics (CFD) that the NavierStokes Equation that precisely describes the fluid’s acceleration by governs the behaviour of the fluid c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 299–310, 2017. https://doi.org/10.1007/978-3-319-71598-8_27

300

K.-H. Chan and S.-K. Im

∇·v =0

(1)

1 ∂v = − (v · ∇) v − ∇p + μ∇2 v + f ∂t ρ

(2)

where v is the fluid’s velocity at a grid’s centre, p is pressure, ρ is density, μ is the coefficient of viscosity and f includes any other external forces such as gravity or boundary confinement. These equations give a precise description of the evolution of a velocity field over time, and tell exactly how the velocity will change over an infinitesimal time-step as a function of the four terms (advection, pressure, viscosity and external-forces) in Eq. 2, but these equations do not consider the mass evolving in the computational physics transferring. 1.2

Related Work

The Eulerian grid-based fluid implementations have been popular for real time solutions as they provide a better description of the fluid properties. However, they do have a major disadvantage in that the grid must be fixed in the space, so making it difficult to track and depict the detailed behaviours (such as mist, foam and spray). The Navier-Stokes Equation was first used in [7] to animate gases and it produced good results on relatively coarse grids. In order to increase the details with lower memory consumption for grid-based simulation, [17] used a dynamic grid that had a low memory footprint when representing a highresolution Level Set method. In [12], all quantities at cell centres were stored for an iterative solver, although a staggered MAC grid is robust and can make it easier to define boundary conditions [11, 20]. In order to reduce the effect of numerical dissipation caused by the use of an implicit semi-Lagrangian integration scheme [6], physical based approaches have considered to use momentum conservation in incompressible flow and the work in [14] can conserve the energy with the semi-Lagrangian method used in the simulation. For the coupling between fluid and solid objects, there exist alternative concepts to incorporate boundary conditions for Eulerian fluids. [2] improved the FLIP method of [21] for two-way solid-fluid interaction. In that approach, the intra-pressure is formulated as a kinetic energy for the coupling problem. [10] presented a two-way coupling for deformable and solid thin objects, the algorithm using ray-casting to avoid fluid leaking through thin solids represented by mesh. [15] presented a GPU approach for the semi-Lagrangian scheme of [19] with arbitrary boundary conditions for the fluid simulation being generated and traced directly in Real-time. For the GPU acceleration, NVIDIA’s Compute Unified Device Architecture (CUDA) has been applied to a large number of GPU-based fluid dynamics implementations, primarily in the engineering and scientific computing fields [9]. [13] proposed a CPU-GPU multigrid Poisson solver that exploits both the CPU and GPU to improve the performance and accuracy of the advection step. Later, the work in [3] presented a hybrid grid of two kinds of cell composition, and [4] described a novel gas simulation system that dynamically translates the fluid simulation domain to track the object and fluid surface.

Fast Grid-Based Fluid Dynamics Simulation

1.3

301

Contribution

In this paper, we propose to maximize the performance of fluid simulation, focusing on a grid-based solution that can be efficient with parallel acceleration. For the Navier-Stokes Equation, we introduce a stable handling for the viscosityfriction term while updating the mass and momentum. The pressure-gradients term can be obtained by the Ideal Gas Law as suggested by [16]. Different from the traditional backwards advection method, we propose to use the forwardtracing-based motion and the evolving of fluid component (A) can be obtained by A′ =

1 ∂A + (A · ∇) A = − ∇p + μ∇2 A ∂t ρ

(3)

Consider the velocity evolving in Navier-Stokes Equation, the transmissions of velocities would be replaced by the transporting of momentum and mass (explain in Sect. 2). Further, we propose a fast component collection method for the fluid transport handling. This idea is designed for parallel implementation and the processing time taken is O (n log n) which is suitable for real-time simulations. In addition, the kinetic energy transferring and conservation of momentum can calculate the coupling status, when solid object interaction is considered during fluid simulation.

2

Solution Method

Commonly, the Navier-Stokes Equation in Eq. 2 can describe the velocity field, but it is not enough to describe the density field only by the changing velocity. [19] provided an improved solution to this problem and implemented a gas simulation with the density being described by the Navier-Stokes Equation. We extend this idea into both the momentum and mass fields. We know that the relation between mass and velocity can be considered as momentum, and the law of conservation of momentum describes the energy transfer in nature. Thus, we use mass and momentum instead of the velocity then apply to Eq. 3 as follows, 1 E ′ = − ∇p + μ∇2 E ρ

(4)

1 m′ = − ∇p + μ∇2 m ρ

(5)

where E = mv is momentum. In these equations, we focus on the internal (viscosity, pressure) status and external coupling forces as described in Sect. 2.3. After solving, the new velocity can be obtained by the new momentum and mass as E +∆t (6) v +∆t = +∆t m

302

2.1

K.-H. Chan and S.-K. Im

Viscosity-Friction

The viscosity is an internal friction force that describes a fluid’s internal resistance to flow. This resistance results in diffusion of the momentum (also velocity, density, etc.). It causes the fluid’s components to move towards the neighbourhood balance. To solve the mass and momentum fields we use:  (7) E +∆t = E + E ′ dt +∆t

m

=m+



m′ dt

(8)

where E ′ = ∂E/∂t and m′ = ∂m/∂t. However, this method is unstable when the viscosity is large, so we refer to an efficient method in [19] for a discussion on the Gauss-Seidel Relaxation iterative technology, given by,  2 E i + μ (∆x) · ∆t · J E j +∆t (9) = E 2 1 + μ (∆x) · ∆t · |J| 2

m+∆t =

mi + μ (∆x) · ∆t · 2



J

mj

1 + μ (∆x) · ∆t · |J|

(10)

where J is the set of neighbouring grids of current i, j is the element index of J, |J| is the number of elements in set J. This method can avoid the density becoming a negative value, and the new velocity can be more stable and realistic than that obtained by only solving the velocity field directly in Eq. 6. 2.2

Pressure-Gradients

Commonly, the pressure is computed by solving an adequately built equations system using a Conjugate Gradient solver [20], which is heavy in computational load and memory consumption. Focusing on the fluid sample taken on a cell, pressure can be sampled on the centre of the cell. Such a scheme is chosen due to it having better stability properties than a scheme where the samples are taken from the same location. Thus, we invoke the Ideal Gas Law to calculate the recent pressure. It can be obtained from the current density as p = kρ

(11)

where k is the gas stiffness that depends on the temperature. We know that there is a constant rest density in the material and this state is more obvious in liquid behaviour. Thus, we use a modified Ideal Gas Law suggested in [16], where the fluid internal pressure can be obtained from the current density as  k (ρ − ρrest ) , ρ > ρrest (12) p= 0, otherwise

Fast Grid-Based Fluid Dynamics Simulation

303

where ρrest is the rest density in the material. The pressure will be minimized as the density approaches to this value. The condition ρ > ρrest recognizes that the pressure is an internal repulsion force, and ignores the attraction force between the two nearest grids: this result/effect is more obvious in liquid behaviour. Consequently, the fluid should exhibit some internal cohesion formation, resulting in attraction-repulsion force as fi = − 2.3

1 ∇pi ∆x · ρi

(13)

Energy Transfer

In addition to model the boundary conditions, we must handle the momentum along both the fluid and solid object. Since the conservation of the total momentum demands that, the total momentum before the collision is the same as the total momentum after the collision, +∆t E +∆t fluid + E rigid = E fluid + E rigid

(14)

In order to make sure the overall energy is conserved during coupling, we have to find the relation between momentum and energy items. Moreover, the conservation of the total kinetic energy can be expressed by 1 1 1 +∆t +∆t 1 +∆t +∆t E fluid v fluid + E rigid v rigid = E fluid v fluid + E rigid v rigid 2 2 2 2

(15)

Specially, note that gravity should directly add momentum to fluid as potential energy then be converted into kinetic energy. The bottom boundary forces can offset it symmetrically. Thus, we ignore the momentum changes based on gravity in these equations. By solving Eqs. 14 and 15, the evolved fluid velocity v +∆t fluid in the new frame of reference can be determined by +∆t = v fluid

E fluid + E rigid − C (mfluid v rigid − E fluid ) mfluid + mrigid

(16)

where C is the coefficient of the restitution and slip belonging to the normal and tangential directions at the collided face. We can demonstrate the scale of coefficient (C ∈ [0.0, 1.0]) for the handling various boundary conditions, as well as complex external constraints such as perfect elastic and free slip effects.

3

Implementation

In order to do simulation in real-time, we prefer to implement the fluid model in a GPU to accelerate our computation by parallelism.

304

3.1

K.-H. Chan and S.-K. Im

Fluid Diffusion and Distribution

In the Eulerian grid-based approach, the fluid behaviour is described in terms of a fixed grid. Fluid components cannot be transported by moving these grids directly. To consider the cell size and diffusing the fluid mass (m) would lose the details in vorticity, so we change to diffuse the density (ρ = m/∆xd , d is the dimension) instead. Consider the advection term − (v · ∇) v in Eq. 2, the minus symbol means these fluid components should be found by backward tracing [19]. However, this method may cause some mass loss. If mass (also momentum) is lost and there are few areas of coupling to exchange energy to, changes in energy may cause undesirable noise. To avoid this issue we introduce a forward tracing method that can conserve the total ρ throughout the simulation.

Fig. 1. These figures illustrate the forward tracing steps: after these components have been transported, they should be assigned to their nearest 4 (in 2D, 8 in 3D) grids.

As shown in Fig. 1 in a 2D case, fluid material moves (forward) to a new position. These source components (density, momentum) have been divided by Linear-Interpolation and added to the nearest four grids (e.g. ρa will distribute to grid 8, 9, 13 and 14). Assume ρa is the current density in grid 21, the source 14 8 9 13 14 ρa will divide to ρ8a , ρ9a , ρ13 a and ρa respectively, with ρa + ρa + ρa + ρa = ρa . 8 8 8 Additionally, the final density of grid 8 should be ρa + ρb + ρc . Every grid must repeat this operation to obtain the final state. 3.2

Fluid-Grid Relation Table

In the forward tracing method, all grids must wait for fluid movement to finish, then each grid should evolve to the sum of the fluid components that have arrived, but it is difficult to distribute momentums and density to the grids efficiently. In our proposal, we define a pair of index arrays to represent the Fluid-Grid relation between the grid and the arrived fluid after diffusion (see Fig. 2). As shown in Fig. 2, array (a) stores the index of the grid that receives the fluid density and array (b) stores the sorted index of the density relevant to

Fast Grid-Based Fluid Dynamics Simulation (First, Number )

Fluid-Grid relation table ρ8a ρ9a

8

ρ2c

2 3

(0,1)

2

(1,2)

3

9

ρ3b

ρ13 a

13

ρ3c

3

ρ14 a

14

ρ4b

4

(3,1)

4

7

(4,1)

7

(5,3)

8

(8,2)

2

3

ρ7c

ρ4b

4

ρ8a

8

ρ8b

8

ρ8b

8

9

ρ8c

8 9

ρ3b

ρ9b ρ2c

2

ρ9a

ρ3c

3

ρ9b

9

ρ7c

7

ρ13 a

13

(10,1)

13

8

ρ14 a

14

(11,1)

14

ρ8c (a)

Sort =⇒

(b)

305

Group =⇒

(c)

Fig. 2. Flow chart of sorting and grouping the fluid density in the same grid.

the corresponding grids. Sorting is by the parallel radix algorithm in the GPU environment and the performance is O (n log n). According to these two arrays, we must provide the (First, Number ) pair for each cell coordinate to record the table index of the first related density and the number of related densities in the sorted table. Finally, every cell and its received density are grouped into the same region of the table, thus there is only one loop of the sum of densities and we know exactly how many densities must be queried in our method. This makes it suitable for parallel programming design and implementation. Note that using our conservative advection with diffusing can conserve the total mass throughout the simulation.

4

Experiment Results and Discussion

These experiments has been implemented on a PC with 2 GB memory with video card NVIDIA Quadro 6000 GPU. These simulations are implemented by CUDA 7.5. The scene was rendered by OpenGL 4.5, GLSL 4.5 in 2D and OptiX 3.9 in 3D. 4.1

Results

As shown in Fig. 3, the smoke is running in 2D and the cycle time-step is less than 1/60 s with the Courant-Friedrichs-Lewy (CFL) condition. Note that there is no

306

K.-H. Chan and S.-K. Im

Fig. 3. These figures show the density of smoke at different time-steps. This simulation uses our method with the smoke injected from below. It processes at a resolution of 500 × 900 grids scale and coupling with two rotating stars.

gravitational effect in smoke, thus external forces only include the boundary and coupling forces. We use Eq. 11 for computing the smoke pressure; it illustrates that while the smoke rises by buoyancy, the potential energy will decrease with increasing kinetic energy and that will apply to the energy transference. However, there is no mass loss in our method, but it does cause some areas to contain undesirable dispersion-density/noise that is obvious in the smoke simulation. To alleviate this problem we should adjust neighbouring smoke-grid (For example in Fig. 1, very low density of smoke ρ7c in grid 7 has prorated to its neighbouring smoke-grid 2 and 8). As shown in Fig. 4, water is running in a 3D scene. We use Eq. 12 for computing the water pressure, and we also add some vorticity confinement [18] to the system for momentum conservation. Note that a particle level set method [5] is used to advect and treat the boundary tracing. For forward advection we need to treat the level set as a solid object, since there is no guarantee that the particle level set method had been also used in both time t and t+∆t . Furthermore, the water would be affected by gravity and the upper grids energy will increase so that the energy becomes very large at the bottom. This is non-conservative from a global perspective. To solve this we produce a reaction vertical momentum after the gravity is added to these bottom grids. These GPU-based fluid simulations with momentum conservation can be simulated in real time. The results show that our method can be used in gas and liquid with conservation of energy. The time performance of the experimental results is shown in Table 1. 4.2

Comparison

For rendering purposes, we improve the visual quality of the simulation by generating particles that penetrate the level set surface, then applying a local momentum conservation force to slightly alter the level set velocities in fluid surface regions. The comparison between our method, the methods using the GPUbased stable fluid in [1], and the mass and momentum conservation fluid [14] is shown in Fig. 5.

Fast Grid-Based Fluid Dynamics Simulation

307

(a) Water is poured onto solid balls with different densities.

(b) Red ball is low density; green ball is equal to water density; blue ball has higher density.

(c) The interactive force by fluid does not significantly move the blue ball, and the red ball is floating on the water. The green ball can be pushed by the water but is not floating on the water.

Fig. 4. These figures show the interaction between water and solid balls in different time-steps. This simulation processes at resolution 128 × 128 × 128 grids scale in 3D, and the fluid surface is constructed by the marching cube algorithm [8]. (Color figure online)

Compared with [14], a speedup of around 10% to 20% can be achieved. Although the speed is slower than in [1], a reason is that the fluid surface rendering is also time-consuming, but there is not much difference and our method can avoid the mass loss and converse the energy of interaction.

308

K.-H. Chan and S.-K. Im

Table 1. Performance results of GPU-based of fluid simulation with momentum conservation. Grid Size

Fluid Solver (ms)

Diffusion & Surface Distribution (ms) Render (ms)

Overall (ms)

32 × 32 × 32

4.29

0.42

2.31

7.02

32 × 64 × 32

9.94

0.83

4.57

15.34

64 × 32 × 64

19.01

0.95

5.23

25.19

64 × 64 × 64

21.01

1.19

6.55

28.75

64 × 128 × 64

39.06

1.48

8.14

48.68

128 × 64 × 128

75.76

2.07

11.39

89.22

128 × 128 × 128 149.25

3.12

17.16

169.53

45 Our Method Amador et al. [1] Lentine et al. [14]

Frame Per Second (FPS)

40

35

30

25

20

15

10

0.3

0.4

0.5

0.6 0.7 Grid Size

0.8

0.9

1 ·106

Fig. 5. Comparison of the simulation speed in FPS.

5

Conclusion

In this paper, we have presented a novel momentum conservation method for simulating a wide variety of fluid behaviours. Our algorithm can avoid mass loss during the energy transfer, and the solution, designed for parallel computing,

Fast Grid-Based Fluid Dynamics Simulation

309

can be implemented in a Graphics Processing Unit (GPU). Moreover, a fast forward-tracing-based motion has been presented to handle the distribution of the fluid components. With the momentum conservation, we also considered the conservation of energy for various boundary conditions, including perfect elastic and free slip effects. The experiment results showed that our method can be around 10% to 20% faster than the previous momentum conservation fluid simulation and this is significant.

References 1. Amador, G., Gomes, A.: A Cuda-based implementation of stable fluids in 3D with internal and moving boundaries. In: 2010 International Conference on Computational Science and Its Applications (ICCSA), pp. 118–128. IEEE (2010) 2. Batty, C., Bertails, F., Bridson, R.: A fast variational framework for accurate solidfluid coupling. ACM Trans. Graph. (TOG) 26, 100 (2007). ACM 3. Chentanez, N., M¨ uller, M.: Real-time Eulerian water simulation using a restricted tall cell grid. ACM Trans. Graph. (TOG) 30(4), 82 (2011) 4. Cohen, J.M., Tariq, S., Green, S.: Interactive fluid-particle simulation using translating eulerian grids. In: Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pp. 15–22. ACM (2010) 5. Enright, D., Marschner, S., Fedkiw, R.: Animation and rendering of complex water surfaces. ACM Trans. Graph. (TOG) 21, 736–744 (2002). ACM 6. Fedkiw, R., Stam, J., Jensen, H.W.: Visual simulation of smoke. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 15–22. ACM (2001) 7. Foster, N., Metaxas, D.: Modeling the motion of a hot, turbulent gas. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 181–188. ACM Press/Addison-Wesley Publishing Co. (1997) 8. Geiss, R.: Generating complex procedural terrains using the GPU. In: GPU Gems 3, pp. 7–37 (2007) 9. Goodnight, N.: Cuda/OpenGL fluid simulation. NVIDIA Corporation (2007) 10. Guendelman, E., Selle, A., Losasso, F., Fedkiw, R.: Coupling water and smoke to thin deformable and rigid shells. ACM Trans. Graph. (TOG) 24(3), 973–981 (2005) 11. Harlow, F.H., Welch, J.E.: Numerical calculation of time-dependent viscous incompressible flow of fluid with free surface. Phys. Fluids 8(12), 2182–2189 (1965) 12. Harris, M.J.: Fast fluid dynamics simulation on the GPU. In: SIGGRAPH Courses, p. 220 (2005) 13. Jung, H.R., Kim, S.T., Noh, J., Hong, J.M.: A heterogeneous CPU-GPU parallel approach to a multigrid Poisson solver for incompressible fluid simulation. Comput. Anim. Virtual Worlds 24(3–4), 185–193 (2013) 14. Lentine, M., Aanjaneya, M., Fedkiw, R.: Mass and momentum conservation for fluid simulation. In: Proceedings of the 2011 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 91–100. ACM (2011) 15. Liu, Y., Liu, X., Wu, E.: Real-time 3D fluid simulation on GPU with complex obstacles. In: 2004 Proceedings of the 12th Pacific Conference on Computer Graphics and Applications, PG 2004, pp. 247–256. IEEE (2004)

310

K.-H. Chan and S.-K. Im

16. M¨ uller, M., Charypar, D., Gross, M.: Particle-based fluid simulation for interactive applications. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 154–159. Eurographics Association (2003) 17. Nielsen, M.B., Museth, K.: Dynamic tubular grid: An efficient data structure and algorithms for high resolution level sets. J. Sci. Comput. 26(3), 261–299 (2006) 18. Selle, A., Rasmussen, N., Fedkiw, R.: A vortex particle method for smoke, water and explosions. ACM Trans. Graph. (TOG) 24, 910–914 (2005). ACM 19. Stam, J.: Stable fluids. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 121–128. ACM Press/Addison-Wesley Publishing Co. (1999) 20. Tome, M.F., McKee, S.: Gensmac: A computational marker and cell method for free surface flows in general domains. J. Comput. Phys. 110(1), 171–186 (1994) 21. Zhu, Y., Bridson, R.: Animating sand as a fluid. ACM Trans. Graph. (TOG) 24, 965–972 (2005). ACM

Adaptive Density Optimization of Lattice Structures Sustaining the External Multi-load Li Shi1, Changdong Zhang2 ✉ , Tingting Liu2, Wenhe Liao2, and Xiuyi Jia1,3 (

1

)

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China [email protected], [email protected] 2 School of Mechanical Engineering, Nanjing University of Science and Technology, Nanjing 210094, China [email protected], {liutingting,cnwho}@mail.njust.edu.cn 3 Key Laboratory of Oceanographic Big Data Mining and Application of Zhejiang Province, Zhoushan 316022, China

Abstract. In recent years, additive manufacturing have attracted increasing attention and promoted the development of lightweight modeling methods. Some studies have been carried out using internal filling structure to optimize the 3D model, while reducing the weight of the model, it can also satisfy some physical properties, such as to withstand the external loads. This paper presents an adaptive infilling structure optimization method based on the triply periodic minimal surface (TPMS), which can be represented by a pure mathematical implicit func‐ tion. The morphology of these lattice structures within different stress regions can be adaptively adjusted, so as to reduce the weight of 3D printed objects while sustaining the external multi-load constraints. Firstly, finite element method is used to analyze the stress distribution of the original model infilled with uniform lattice structure. According to its stress value, the internal lattice structure is divided into three regions consists of high region (HR), transition region (TR) and low region (LR). Then, the inner structure within different stress regions is adaptively adjusted to relieve the stress concentration. Finally, we demonstrated that the proposed algorithm can effectively reduce the weight of 3D model while sustaining its mechanical strength. Keywords: Geometric model · Infilling structure · Multi-loads Structure optimization · Additive manufacturing

1

Introduction

Additive manufacturing (AM) refers to fabricate the part layer by layer, it enables the manufacture of novel, highly complex geometric models possible [1]. To optimize a model’s physical attributes, such as strength, stability and weight [2–5]. A typical prac‐ tice in AM is to use a uniform infilling structure, which allows a balance between the part strength and the amount of material. However, most of the optimized internal struc‐ tures are prone to create large overhangs, which make the structure impossible to be directly manufactured by AM without deformation [6–8], so it is inevitable to generate © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 311–322, 2017. https://doi.org/10.1007/978-3-319-71598-8_28

312

L. Shi et al.

the support structures to assist manufacturing [9]. When the processing is finished, the support structures need to be removed, which is a hard and cumbersome process. It not only increases the workload but also causes waste of materials. So, there is a great demand for the study of the internal filling structures with both self-supporting functions and lightweight features [10]. This paper puts forward an adaptive density optimization method for the selfsupporting lattice infilling structures sustaining the external multi–load. The lattice structures are distributed in relation to a density function defined by the stress map, allowing simple control to trade off between the strength and material consumption for AM. The contributions of this method are as follows: • We adapt three types of self-supporting lattice model, which derived from triply periodic minimal surfaces [11], to generate the internal filling structures. The morphology of the lattice structure can be effectively controlled, such as its size, horizontal angle and pillar thickness, which makes the design of the internal structure more flexible. Meanwhile, the self-supporting characteristic could reduce the mate‐ rial consumption and improve the forming efficiency [12]. • We introduce a material optimization algorithm with respect to the exterior multiload. Based on the finite element analysis, the stress of the object can be computed and the initial lattice infilling structures will be divided into three regions according to its stress value. Then the morphology of lattice structures within different stress regions can be adaptively adjusted, which makes the optimization of the internal filling structures reasonable and executable. • Our method allows a simple control to effectively reduce the weight of 3D model while sustaining its mechanical strength. The rest of this paper is organized as follows. Section 2 illustrates the main meth‐ odology. Section 3 introduces the lattice infilling structures derived from TPMS and explores its structural features. Section 4 indicates a stress analysis method for the 3D model and an adaptive optimization method formula. Section 5 provides three examples to express the optimization process. Section 6 summarizes the conclusions of this paper and lists some ideas for future work.

2

Methodology

The basic perspective embedded within the optimization is increasing the volume of high stress region, decreasing the volume of the low stress region and progressively changing the rest of the region. At first, original 3D model preprocessing should be done, which contains three steps, (1) Using the contour offset method to generate a hollowed shell structure from the input 3D model [13]. (2) Using the marching cube (MC) algo‐ rithm to generate a uniform internal structure, then filling the minimum Axis-Aligned Bounding Box (AABB) [14] that belongs to the inner shell. (3) Using Boolean operation to fill the inner shell with homogeneous lattice structure (Figs. 1 and 2).

Adaptive Density Optimization of Lattice Structures Sustaining

313

Input 3D model

Preprocessing and initialize design variables

f2 f1 (a)

Finite element analysis

Divide stress interval

(b)

(c)

(d)

(e)

Adjust the density of lattice structure

Update stress and volume of the 3D model

NO

Satisfy criterion? Yes Output 3D model

(f) Fig. 1. Flow chart of optimization process

Fig. 2. A bead head optimized example

Then, the finite element analysis need to implement to calculate the stress distribution of the original model infilled with uniform lattice structure. Some conditions should be initialized including external force, constraint conditions, material properties and so on. This paper computes an initial stress map using public OOFEM finite element library [15]. Finally, by means of the stress diffusion method proposed in Sect. 4.1, the internal structure is divided into three regions and the density of the lattice structure within different stress regions is adaptively adjusted.

314

3

L. Shi et al.

TPMS Internal Structure

A TPMS structure is a surface that is locally area-minimizing, that is, a small piece has the smallest possible area for a surface spanning the boundary of that piece. Wu et al. [3] studied the self-supporting internal structure, which showed that it is no need to add the auxiliary support structure when the direction of the structure is horizontally at a ◦ certain angle (≥ 45 ). Lattice structures derived from TPMS in this paper also have the self-supporting characteristics. Figure 3 gives three types of TPMS lattice structures we used. Figure 3(a) is the Schwarz’ P-lattice structure, which meets the self-support requirement between the growth direction and the horizontal angle. Figure 3(b) is the Schoen’s G-lattice structure, which has zero mean curvature, i.e. the sum of the principal curvatures at each point is zero. The angle between the direction of stretching and the horizontal axis can also be adjusted. Figure 3(c) is called Schwarz’ D-lattice structure, whose skeletons have a diamond lattice pattern and its porous structure is about 45 degrees with horizontal direction.

(a)

(b)

(c)

Fig. 3. Three types of TPMS lattice structure, (a) P-lattice, (b) G-lattice and (c) D-lattice

From the viewpoint of Yoo [16], the implicit surface functions of the three kinds of TPMS in Fig. 3 can be expressed as Eqs. (1), (2) and (3), respectively, where X = 2�x∕ L, Y = 2�y∕ L, Z = 2�z∕ L, x, y, z refer to the Cartesian coordinates. Param‐ eter L determines the edge length of the cubic unit cell, which will set as a constant value in this paper, parameter t determines the volume fraction of the regions that are separated by the surface [5, 17]. Based on the two parameters, this paper studies the relationship between the edge length and the volume fraction of the three kinds of TPMS lattice structures. As shown in Fig. 4, where t ∈ [−0.5, 0.9], the vertical axis represents the percentage of TPMS structure to the same scale in a solid structure. Besides, there is a linear relationship between the t-value and the volume fraction. Equations (4, 5 and 6) illustrated the linear functions of the three kinds of surfaces according to t-value respec‐ tively. In this paper, they are called as density function which indicates the percentage of the lattice relative to the same size of a solid volume.

F(X, Y, Z) = cos(X) + cos(Y) + cos(Z) − t

(1)

F(X, Y, Z) = sin(X) cos(Y) + sin(Y) cos(Z) + sin(Z) cos(X) − t

(2)

Adaptive Density Optimization of Lattice Structures Sustaining

F(X, Y, Z) = sin(X) sin(Y) sin(Z) + sin(X) cos(Y) cos(Z) + cos(X) sin(Y) cos(Z) + cos(X) cos(Y) sin(Z) − t { �P (t) = 0.5 − 0.288t s.t. − 0.5 ≤ t ≤ 0.9

315

(3)

(4)

{

�D (t) = 0.5 − 0.418t s.t. − 0.5 ≤ t ≤ 0.9

(5)

{

�G (t) = 0.5 − 0.327t s.t. − 0.5 ≤ t ≤ 0.9

(6)

Fig. 4. The relationship between the t-value of three TPMS structure and its volume fraction (Color figure online)

4

Adaptive Density Optimization of Lattice Structure

To balance the mechanical strength of the lattice structure and reduce the material consumption effectively, this paper proposed an adaptive optimization method which can change the local morphology of the lattice structure while sustaining the given multiple forces. Starting from the model infilled with uniform lattice structure, the finite element method will be implemented so to analyze the stress value of the initial input model. Then the stress analysis can be associated with the internal lattice volume. As presented in Sect. 3, the parameter t deciding the lattice volume, when it changing, the diameter of the lattice pillars will be changed. Based on this phenomenon, we can adjust

316

L. Shi et al.

the parameter t to change the morphology of the internal lattice structure, so as to meet the specific strength and weight requirements. 4.1 Lattice Stress Analysis To facilitate the execution of finite element calculations, we use the TetGen library to generate the tetrahedral meshes firstly, then by means of OOFEM library to carry out the finite element analysis. As described in Sect. 2, we will calculate the stress distri‐ bution under multiple loads. For simplicity, we assume the direction of each load is perpendicular to each other and use the Von-Mises stress to indicate the stress of the object, as shown in Eq. (7), �xy , �yz , �zx are x-axis principal stress, y-axis principal stress and z-axis principal stress, respectively. Similarly, �xy , �yz , �zx are XOY surface shear, YOZ surface shear and ZOX surface shear, respectively. �=

√

1 2 + � 2 + � 2 )] [(� − �yy )2 + (�yy − �zz )2 + (�zz − �xx )2 + 6(�xy yz zx 2 xx

(7)

Stress values of each tetrahedron in the model are obtained by the above formula. After dispersing the tetrahedrons into points, the stress regions of the model are then divided into HR, TR and LR according to the stress value of the points, as shown in Fig. 5. Among which, the high stress region and the low stress region are divided by the stress diffusion method proposed in this paper, and the remaining part is automatically classified as the transition region. Stress diffusion steps are as follows(using low region as an example to explain). 1. Sort the stress points of the model in ascending order, then the lowest stress (LS) will be found; 2. Search the stress points inside the lattice and take the top 10 points as initial points set S0; 3. Define the scale value as SV and then collect the points whose stress value } { ‖ ‖2 � ∈ [LS, SV ∗ LS] as candidate set S1, if p1j = ∃p0i ∈ S0 |‖p0i − p1j ‖ ≤ D , where ‖ ‖ p1j ∈ S1, D is distance threshold, p1j will be added to point set S0 to form a new set ′

S0. Points in set S1 won’t be visited until the size of set S0 is not increasing any more; 4. Use the length L to express the voxel unit, then construct a minimum AABB bounding box containing the model by this unit, where L value is related to parameter t in lattice; 5. Calculate the point number Ni inside the voxel unit i , when Ni ≥ CN (CN is the default threshold), the boundary of voxel is marked as low stress region. The process will continue until all the voxels have been visited.

Adaptive Density Optimization of Lattice Structures Sustaining

317

AABB

LR Point Set Stress Diffuse

Extract Point

Discrete Tetrahedron

Interval Partition

f HR Point Set

Fig. 5. Stress interval extracting process

The division of the high stress region can be completed by using the same method, where the AABB follows the bounding box in the low stress step, searching for high stress regions, which ensures that the voxel cell size is same. Inside the bounding box, voxels which are not marked as high stress regions or low stress regions are automati‐ cally divided into transition stress region. 4.2 Lattice Density Optimization After Sect. 4.1, the stress distribution of the model can be generated. From Sect. 3, we know the parameter t directly influences the volume fraction and the physical property of the structure. The material cost of the lattice structure will be reduced with the decrease of the volume fraction. In this paper, the relationship between the stress distri‐ bution and the volume fraction also can be established by the parameter t, so that the lattice structure can change the morphology characteristics of the lattice in different stress regions according to its stress value. Therefore, if the model need to meet strength and weight requirement at the same time, the objective function can be defined as follows, Minimize

E=

�1 T u K(�TPMS )u + �2 V(�TPMS ) 2

(8)

318

L. Shi et al.

Subject to K(�TPMS )u = f

V=

∑

�iTPMS Vsolid ≤ Vthreshold

(9) (10)

i

⎧ � (t) ⎪ P �TPMS (t) = ⎨ �D (t) −0.5 ≤ t ≤ 0.9 ⎪ � (t) ⎩ G

(11)

1 The first term uT K(�TPMS )u represents the strain energy. The second term V(�TPMS ) is 2 the volume of the internal structure, where �1, �2 are the balance factors during each itera‐ tion. Equation (9) describes the static state of the object under the given external force f , where u means the displacement vector. Equation (10) represents the termination condition of the model volume, where Vthreshold is volume threshold, �iTPMS is the density value of TPMS lattice of voxel unit i, Vsolid is a volume of unit cube. Equation (11) expressed the lattice density function which has three choices. In practical, we use the power-law relationship ∑ −p to compute the stiffness matrices K(�TPMS ) = e Ke (�TPMS ) = (�iTPMS ) K0, where K0 is the stiffness matrix of a solid element and p is a penalization parameter [18].

5

Results and Discussion

To demonstrate the feasibility of the proposed method, several models are studied in this paper. In the work, the finite element analysis result of the model before and after optimization under the same load are compared, and the variables of the strength and volume of the interior structure in the iterative process are analyzed. Strength Optimization. The first model is a circular sandwich structure with a skin thickness of 1 mm, an internally filled D lattice structure. The external loads are the top of the model with four constraint points shown in Fig. 6(a). It shows the initial model and the stress of the interior lattice in Fig. 6(b) and (c). Figure 6(d) and (e) are optimized model and stress map of the interior lattice. Comparing the initial stress map, it can be found that the optimized lattice structure under the same load can effectively reduce the high stress region. In addition, we show the change of the high stress region and volume with the increase of iterations in the optimization process, as shown in Fig. 9(a), where high stress point (HSP) percent represents the percentage of the optimized high stress point and the initial high stress point (black square), and the volume ratio represents the percentage of the optimized lattice volume and the initial volume (red circle). It can be found that the proportion of high stress points decreases with the increase of iterations, which illustrates that the optimization process can reduce the high stress region and achieve the model strength optimization. At the end of iterations, the final HSP only has

Adaptive Density Optimization of Lattice Structures Sustaining

319

one third of the initial value. As the strength of the main optimization, in the optimization process, the black curve is located in the red curve below the trend.

f1 f2 f3

(b)

(c)

f4 (a)

(d)

(e)

Fig. 6. Optimization of circular sandwich structure filled with D lattice structure

Volume Optimization. Here is a model of the letter ‘E’ with a skin thickness of 1 mm, which is filled with P lattice structure. Figure 7(a) shows the external forces and constraint points. It shows the transparent map of the primitive model in Fig. 7(b). Figure 7(c) is the stress result of the interior lattice of initial model. Figure 7(d) and (e) are optimized model and its stress map of the internal lattice. Before and after optimi‐ zation, the model morphology changed distinctly. Meanwhile, we show the change of the volume and high stress region with the increase of iterations in the optimization process, as showed in Fig. 9(b). It can be learned that the volume percentage gradually decreases and tends to be stable with the increase in the number of iterations, HSP percentage remains stable with the number of iterations increases. At the end of iteration, it achieves nearly 40% reduction in the volume. As the volume of the main optimization, in the optimization process, the red curve is located in black curve the below the trend.

f2 f1

(b)

(c)

(d)

(e)

(a)

Fig. 7. Optimization of the model of the letter ‘E’ filled with P lattice structure

320

L. Shi et al.

Balance Optimization. In Fig. 8, it is a cuboid model filled with G lattice structure which has a skin thickness of 2 mm. Figure 8(a) to (d) show the morphological change of the lattice within the model during the optimization process. Figure 8(e) to (h) indicate the change of the stress during the optimization process. Combining both of them, it can be noted that the volume of the internal lattice which locates in the low stress region is obviously reduced. At the same time, the volume located in the high stress region is increased. Furthermore, we have calculated the high stress point percentage and the volume ratio of the model in the optimization process. It can be seen from the Fig. 9(c) that with the increase of the number of iterations, the volume ratio of the internal lattice of the model is obviously reduced, and the proportion of the high stress point is also decreased. At the end of iteration, the volume of optimized lattice is less than the original one-half, the HSP reduction is nearly 35%. As the balance optimization, both of the curves gradually decrease and tend to be stable with the increase of the number of iter‐ ations.

f1

f2 (a)

(b)

(e)

(f)

(c)

(g)

(d)

(h)

Fig. 8. Optimization of cuboid model filled with G lattice structure

(a)

(b)

(c)

Fig. 9. HSP percent and volume ratio of the three models, (a) circular sandwich structure, (b) the letter ‘E’, (c) cuboid model (Color figure online)

Adaptive Density Optimization of Lattice Structures Sustaining

6

321

Conclusion

In this paper, we propose an adaptive density optimization method for the selfsupporting lattice infilling structures sustaining the external multi–load. It divides the internal region into three parts according to the stress values which are computed by finite element method. In the different stress region, the morphology of the lattice struc‐ ture can be adjusted adaptively, so the stress of the infilling structures will be relieve and the volume of the 3D model can be reduced. We demonstrated the effectiveness of the proposed method by several examples. However, there are still some shortcomings in the paper. The optimized result need to be verified by multi-load mechanics experiment if the measuring instrument can apply multiple forces in a plurality of different directions. Therefore, new measuring method needs to be considered for the further work. In addition, although the method can calcu‐ late the stress distribution more accurately, the finite element method for the larger model will cost too much time. In the subsequent work, we will simplify the analysis of objects and optimize the analysis process to accelerate the overall optimization process. Acknowledgement. This work was supported by Key Research and Development Program of Jiangsu Province (No. BE2015165), National Natural Science Foundation of China (No. 51405234) and Key Laboratory of Oceanographic Big Data Mining & Application of Zhejiang Province (Grant No. OBDMA201602).

References 1. Gibson, I., Rosen, D., Stucker, B.: Additive Manufacturing Technologies. Springer, US (2010). https://doi.org/10.1007/978-1-4419-1120-9 2. Lu, L., Sharf, A., Zhao, H., et al.: Build-to-last: strength to weight 3D printed objects. ACM Trans. Graph. 33, 1–10 (2014) 3. Wu, J., Wang, C.C.L., Zhang, X., et al.: Self-supporting rhombic infill structures for additive manufacturing. Comput. Aided Des. 80, 32–42 (2016) 4. Yamanaka, D., Suzuki, H., Ohtake, Y.: Density aware shape modeling to control mass properties of 3D printed objects, pp. 1–4 (2014) 5. Li, D., Dai, N., Jiang, X., et al.: Density aware internal supporting structure modeling of 3D printed objects. In: International Conference on Virtual Reality and Visualization. pp. 209– 215. IEEE (2015) 6. Musialski, P., Auzinger, T., Birsak, M., et al.: Reduced-order shape optimization using offset surfaces. ACM Trans. Graph. 34, 1–9 (2015) 7. Wu, J., Kramer, L., Westermann, R.: Shape interior modeling and mass property optimization using ray-reps. Comput. Graph. 58, 66–72 (2016) 8. Christiansen, A.N., Schmidt, R., Bærentzen, J.A.: Automatic balancing of 3D models. Comput. Aided Des. 58, 236–241 (2015) 9. Vanek, J., Galicia, J.A.G., Benes, B.: Clever support: efficient support structure generation for digital fabrication. Comput. Graph. Forum 33, 117–125 (2014)

322

L. Shi et al.

10. Aremu, A.O., Maskery, I.A., Tuck, C.J., Ashcroft, I.A., Wildman, R.D., Hague, R.J.M.: Effects of net and solid skins on self-supporting lattice structures. In: Antoun, B. (ed.) Challenges in Mechanics of Time Dependent Materials. CPSEMS, vol. 2, pp. 83–89. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-22443-5_10 11. Meeks, W.H.I.: The theory of triply-periodic minimal surfaces. Indiana Univ. Math. J. 39, 877–935 (1990) 12. Strano, G., Hao, L., Everson, R., et al.: A new approach to the design and optimization of support structures in additive manufacturing. Int. J. Adv. Manuf. Technol. 66, 1247–1254 (2013) 13. Hussein, A., Hao, L., Yan, C., et al.: Advanced lattice support structures for metal additive manufacturing. J. Mater. Process. Technology. 213, 1019–1026 (2013) 14. Ericson, C.: Real-Time Collision Detection. CRC Press Inc., Boca Raton (2004) 15. Patzak, B., Rypl, D.: Object-oriented, parallel finite element framework with dynamic load balancing. Elsevier Science Ltd (2012) 16. Yoo, D.J.: Computer-aided porous scaffold design for tissue engineering using triply periodic minimal surfaces. Int. J. Precis. Eng. Manuf. 12, 61–71 (2011) 17. Scherer, M.R.J.: Gyroid and Gyroid-Like Surfaces Double-Gyroid-Structured Functional Materials, pp. 7–19. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3319-00354-2_2 18. Bendsøe, M.P., Sigmund, O.: Material interpolation schemes in topology optimization. Arch. Appl. Mech. 69, 635–654 (1999)

Hyperspectral Image Processing

Hyperspectral Image Classification Based on Deep Forest and Spectral-Spatial Cooperative Feature Mingyang Li1 , Ning Zhang2 , Bin Pan1 , Shaobiao Xie3 , Xi Wu1 , and Zhenwei Shi1(B) 1

2 3

Image Processing Center, School of Astronautics, Beihang University, Beijing, China [email protected] Shanghai Aerospace Electronic Technology Institute, Shanghai, China School of Astronautics, Harbin Institute of Technology, Harbin, China

Abstract. Recently, deep-learning-based methods have displayed promising performance for hyperspectral image (HSI) classification. However, these methods usually require a large number of training samples, and the complex structure and time-consuming problem have restricted their applications. Deep forest, a decision tree ensemble approach with performance highly competitive to deep neural networks. Deep forest can work well and efficiently even when there are only smallscale training data. In this paper, a novel simplified deep framework is proposed, which achieves higher accuracy when the number of training samples is small. We propose the framework which employs local binary patterns (LBPS) and gabor filter to extract local-global image features. The extracted feature along with original spectral features will be stacked, which can achieve concatenation of multiple features. Finally, deep forest will extract deeper features and use strategy of layer-by-layer voting for HSI classification. Keywords: Deep-learning · Deep forest Local binary patterns (LBPS) · Gabor filter

1

Introduction

Depending on the development of sensor technology, hyperspectral sensors could provide images owning hundreds of bands with high spatial and spectral information. Hundreds of spectral band values which are continuous and narrow are recorded as a data cube, with the spectral resolution of nanometer level. Due to these advantages, the applications of hyperspectral data have been widely used in many fields such as spectral unmixing [15] and environmental monitoring [13]. Hyperspectral image classification is one of the most important technologies for these applications. However, hyperspectral image classification is still a challenge problem owing to its complex characteristic. The high dimensionality may c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 325–336, 2017. https://doi.org/10.1007/978-3-319-71598-8_29

326

M. Li et al.

produce curse of dimensionality [5], and the abundant spectral bands may also bring noise to decrease performance of the classification. Therefore, we could not simply use spectral signals for hyperspectral sensing image classification. During the last decade, many traditional methods based on feature extraction have been proposed to solve the problem. These methods use spectral and spatial information, and the classification algorithms [4, 11,16]. In recent years, deep learning has been implemented in various fields such as image classification [7]. In [2], Chen et al. proposed applying deep-learning-based methods to handle HSI classification for the first time, where autoencoder (AE) is utilized to learn deep features of HSI. After that, several simplified deep learning methods are developed [12,14]. However, methods based on deep learning could suffer from the complex framework of neural networks, and the performance is limited by the small number of training samples. Furthermore, the methods also suffer from a time-consuming training process, and final experimental results are not easy to reproduce. According to the motivations and problems mentioned above, we introduce a method based on deep forest [18] that can handle HSI classification with limited training samples. Compared to deep neural networks, deep forest achieves highly comparable performance efficiently. What is more, the hyper-parameters of deep forest is quite small, and the result is less sensitive to parameter setting. It will spend less time in training process and perform well on small-scale samples. In this letter, we propose a deep framework combining with spectral-spatial cooperative features for deeper HSI features extraction and classification, which achieves better performance with much less training samples than deep learning methods and traditional methods. To take fully into account the globality in the feature extraction, spectral-spatial cooperative feature combines local and global features with the original spectral feature. Furthermore, considering the feature learned from the last layer of deep forest may not be the best representative feature, we improve the framework of deep forest. We add a voting mechanism in the framework, and have a better experimental result. The remainder of this paper is organized as follows. In Sect. 2, we first present a roughly process about the method, then we present a detailed description of spectral-spatial cooperative feature. At last, we introduce deep forest in detail, which also includes adding a voting mechanism in the framework. In Sect. 3, the experimental results and comparison experiments are displayed. Finally, we come to a conclusion in Sect. 4.

2

The Proposed Method

The proposed features follow three parallel strategies: local feature extraction, global feature extraction and the original spectral feature. Figure 1 shows the overall implementation of the classification method. LBP operator is applied to the calculation of the entire band images to obtain LBP feature map, and the local feature is obtained by counting the histogram of each neighbourhood of the center pixel. The global feature is produced by using gabor filter which captures

Hyperspectral Image Classification Based on Deep Forest

327

Fig. 1. Simple flowchart of DFCF method

texture features at different angles, and the original spectral feature of the central pixel is drawn out. Then we combine the features to get the spectral-spatial cooperative feature, while deep forest with a voting mechanism is employed to deeper feature extraction and the final classification predictions. Here, we present a brief introduction of Gabor filter and LBP operator at first, then, we describe feature fusion about spectral-spatial cooperative feature. Finally, the framework of deep forest is introduced. 2.1

Gabor Filter

A gabor filter, which can extract the relevant features in different scales and directions in the frequency domain, and its frequency and direction of expression are similar to the human visual system. The research shows that gabor filters are very suitable for texture expression and separation [3]. Therefore, it’s widely used for computer vision and image processing. In the 2-D gabor filter model, the filter consists of real component and imaginary component. The mathematical expressions can be expressed as      ′2 x′ x + γ 2 y ′2 + ψ (1) exp i 2π g(x, y; λ, θ, ψ, σ, γ) = exp − 2σ 2 λ real component:  ′2    x + γ 2 y ′2 x′ g(x, y; λ, θ, ψ, σ, γ) = exp − cos 2π + ψ 2σ 2 λ

(2)

328

M. Li et al.

imaginary component:     ′2 x′ x + γ 2 y ′2 + ψ sin 2π g(x, y; λ, θ, ψ, σ, γ) = exp − 2σ 2 λ

(3)

x′ = x cos θ + y sin θ; y ′ = −x cos θ + y sin θ

(4)

where: where λ is wavelength of the sinusoidal function, θ specifies the direction of the parallel stripes of the gabor function, which takes a value of 0 to 360◦ . ψ represents phase offset. γ, spatial aspect ratio, usually setting to 0.5, specifies the ellipticity of the gabor function support. σ is only determined by λ and b as  λ ln 2 2b + 1 · (5) σ= π 2 2b − 1 2.2

Local Binary Pattern

LBP (Local Binary Pattern)[10] is an operator used to describe the local texture features of an image, and it has significant advantages such as rotational invariance and gray-scale invariance. For each given center pixel vc (scalar value), we find the corresponding pixel vi on the specified radius r to compare with vc , where r determines the distance from the center pixel. If the pixel value is higher than the center pixel value, set to 1, otherwise set to 0. After selecting p neighbors {v0 , v1 , . . . , vp−1 }, the LBP calculating the center pixel vc is as followed LBPp,r (vc ) =

p−1 

S(vp − vc )2p

(6)

i=0

where S(vp − vc ) = 1 if vp > vc and S(vp − vc ) = 0 if vp < vc . Through the above formula, given an image, we can get the LBP value of each pixel (The direction of the calculation process is clockwise). While each pixel does not fall to an integer position, its gray value is estimated by bilinear interpolation based on the pixel gray value of the nearest two integer positions within the radius track. The LBP value reflects local texture information and smoothness. Then, an histogram is computered over a local patch, which represents LBP feature of the center pixel. 2.3

Spectral-Spatial Cooperative Feature

By utilizing gabor to filter the spectral image of each spectral band in a given HSI, we can extract global texture features in different directions and different scales. At the same time, the lbp feature map is obtained by computing the spectral image of each spectral band in a given HSI through the LBP operator. The histogram feature is calculated for the fixed size neighborhood of each central pixel, thus obtaining local texture features. In this paper, before performing the above calculations, we use PCA to reduce the spectral dimension due to

Hyperspectral Image Classification Based on Deep Forest

329

many of spectral bands containing redundant information. We stack the local texture and the global texture feature with its original spectral feature to form the initial extracted feature. The feature contains local and global features, also including spectral and spatial information. Therefore, it is called Spectral-Spatial Cooperative Feature.

Fig. 2. The structure of Deep Forest. Each level outputs a distribution vector, then, the vector is concatenated with input feature vector, which is formed as a new feature to input the next level. Finally, output of the last level is averaged to get a probability distribution, and set the prediction label as the highest probability of the label.

2.4

Deep Forest

Cascade Forest. The structure of the network in deep neural networks mostly bases on the layer-by-layer stacking, and it is utilized to process features. In view of this, Zhou and Feng [18]. proposed deep forest, a novel decision tree ensemble method, which employs a cascade structure. The illustration is shown in Fig. 2, where the input received by each level is obtained by the preceding level, and the processing result of this level is outputted to the next level. Each level represents an ensemble of decision tree forests. To encourage the diversity, each level has different decision tree forest. In Fig. 2, complete-random tree forests and random forests [1] is used in the structure. In this paper, we simply use three complete-random tree forests to structure each level. Each tree forest contains 900 complete-random trees. With randomly selecting a feature for split at each node of the tree, the tree forest is generated. Then grow the tree until each leaf node contains only the same class of instances or no more than 10 instances. For each forest, it will estimate the distribution of the class by counting the percentage of different class of the training samples at the leaf node where the instance concerned falls into. Finally, the distribution of the classes will be averaged across all trees, and each forest will output distribution of each class.

330

M. Li et al.

Each forest outputs a distribution of each class, which forms a class vector. Then, the original vector is concatenated with the class vector to form a new feature vector that inputs to the next level of deep forest. For instance, based on a classification task with 16 classes, each of the three forests will output a sixteen-dimensional class vector. Thus, the input feature of the next level will augment 3 × 16 = 48 dimensions. In contrast to most deep neural networks, deep forest can handle different scales of training data, including small-scale ones. The reason is that, for a validation set, the model complexity of deep neural networks is fixed. However, after expanding a new level, the training process of deep forest will be terminated when the performance is not significantly improved. Therefore, the number of levels will be automatically determined. Furthermore, in order to reduce the risk of over-fitting, the output of each cascade level will be generated by a number of cross-validations to ensure that the output is sufficient to characterize the input.

Fig. 3. Cascade forest with the voting mechanism. The gray scale in a small box represents a probability value of the class. When the box is black, the probability is 1.

A Voting Mechanism. In this paper, we structure four levels deep forest to handle HSI classification task whose number of training data is poor. There are three complete-random tree forests in each level. Each tree forest outputs a distribution of each class and a prediction label. For HSI classification task, the distribution each level outputs and the prediction labels are produced by studying the feature this level inputs. In each layer, the feature of input is not identical so that it will lead to the prediction of each layer is also diverse, as it is well known that diversity [17] is crucial for ensemble construction. Therefore, we employ each layer of the prediction results, joining a voting mechanism in

Hyperspectral Image Classification Based on Deep Forest

331

deep forest model. The illustration is shown in Fig. 3. Given a feature as input, we can get three predictions in each level. Then, the final prediction of this level is obtained by using the three predictions to vote. Finally, the final predictions of each level will vote to form a result prediction. Compared with the original model, the model not only relies on the output distribution of the previous level, but also uses the prediction label to affect the next level, which increases the diversity of the structure and the final classification can get more accurate label results.

3 3.1

Experimental Results Data Sets and Experimental Setup

Two popular HSI datasets are used in our experiments: Indian Pines and Pavia University. Indian Pines: The Indian Pine data set is one of the most common data sets in the HSI classification and is obtained from the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) in Northwestern Indiana, India. The image size is 145×145, with wavelengths ranging from 0.4 to 2.5 µm. The data set originally had 220 spectral bands. In order to reduce the effect of the water absorption, 20 water absorption bands were removed, and the remaining 200 bands were used in the experiments. There are 16 classes of interests in the data set, with totally 10249 pixels which are labeled. Pavia University: The data of Pavia University were collected in the city, Italy Pavia, through the reflective optical system imaging spectrometer (ROSIS-3) sensor. The data set originally had 115 spectral bands. Due to the effect of noise interference, 12 bands were removed and 103 bands is used in the experiment. The image size is 610×340, with wavelengths range from 0.43 to 0.86 µm. Spatial resolution is approximately 1.3 m. In this data set, a total of 42776 pixels are labeled, which are classified into 9 classes.

Fig. 4. Indian Pines. (a) False color composite image choosing R-G-B=bands 15-25-59. (b)Ground truth. (c)The prediction of DFCF.

332

M. Li et al.

Fig. 5. Pavia University. (a) False color composite image choosing R-G-B=bands 15-40-65. (b)Ground truth. (c)The prediction of DFCF.

In comparison experiments, limited by space limitations, we refer three methods to compare the method which proposed in this paper, including GCK, NRS and EPF-G. GCK [8] is a composite-kernel-framework based method. NRS [9], based on gabor filter, extract global texture feature for HSI classification, while EPF-G [6] takes advantage of edge-preserving filtering to classify hyperspectral images. In order to be able to evaluate the performance of the methods more comprehensively, overall accuracy (OA), average accuracy (AA) and Kappa coefficient (κ) are employed in the experiments. In the experiment, some important parameters are set in the way of cross-validation. Due to space limitations, this article will not elaborate. Indian Pines: At first, 30 principal components are used for operations. we use b = 1 and 8 orientations to set gabor filter, and patch size of LBP operator is 21 × 21. Pavia University: At first, 30 principal components are used for operations. we use b = 5 and 8 orientations to set gabor filter, and patch size of LBP operator is 21 × 21. Deep Forest: When we apply deep forest, the results are not sensitive to parameter settings. We build 3 complete-random forests on each layer of cascade forest and 4 levels are employed to structure deep forest for both of Pavia University and Indian Pines. 3.2

Classification Results

This section presents the results of each method, and also shows the superiority of the method we proposed. Each data set is randomly selected for 20 samples

Hyperspectral Image Classification Based on Deep Forest

333

Table 1. INDIAN PINES: the classification accuracy of different methods (%). Class

Samples Methods Train Test GCK

Alfalfa

20

26 96.94 ± 3.71

99.92 ± 0.55

62.07 ± 20.5 98.00 ± 1.92

Corn-notill

20

1408 75.67 ± 5.80

70.28 ± 7.43

74.83 ± 9.59 74.94 ± 6.64

Corn-mintill

20

810 81.59 ± 4.47

75.47 ± 7.71

74.68 ± 10.96 86.23 ± 5.87

Corn

NRS

EPF-G

DFCF

20

217 93.17 ± 5.04

94.58 ± 4.59

42.07 ± 11.72 93.88 ± 4.67

Grass-pasture 20

463 89.70 ± 4.29

87.70 ± 5.18

94.83 ± 5.18 91.34 ± 4.06

Grass-trees

710 97.57 ± 1.29

92.55 ± 4.19

94.39 ± 3.90 96.89 ± 3.02

14 97.54 ± 3.94

100 ± 0.00

91.01 ± 12.62 99.57 ± 1.70

458 99.42 ± 0.23

98.21 ± 2.31

99.85 ± 0.41 99.95 ± 0.10

20

Grass-pasture- 14 mowed Haywindrowed

20

Oats

10

Soybean-notill 20

10 100.00 ± 0.00 100.00 ± 0.00 78.18 ± 19.56 99.00 ± 7.00 952 80.98 ± 4.11

72.46 ± 8.17

69.21 ± 6.70 85.55 ± 5.50

20

2435 79.87 ± 4.58

71.76 ± 8.39

84.40 ± 6.03 85.70 ± 5.36

Soybean-clean 20

573 84.07 ± 6.37

81.25 ± 7.01

59.43 ± 13.22 89.61 ± 5.36

185 99.56 ± 0.29

Soybeanmintill Wheat

20

99.18 ± 1.28

99.50 ± 1.33 99.06 ± 0.80

Woods

20

1245 93.74 ± 2.88

87.09 ± 5.12

97.27 ± 2.53 97.47 ± 1.53

BuildingsGrass-TreesDrives

20

366 93.16 ± 3.66

90.43 ± 6.10

71.08 ± 15.00 95.20 ± 5.32

Stone-SteelTowers

20

73 95.71 ± 4.69

98.97 ± 2.03

81.16 ± 7.79 98.16 ± 1.39

OA

85.54 ± 1.42

79.87 ± 1.82

78.45 ± 2.75 88.56 ± 1.82

AA

91.17 ± 0.85

88.74 ± 1.03

79.62 ± 2.35 93.16 ± 1.09

κ × 100

83.61 ± 1.58

77.14 ± 2.03

75.64 ± 3.07 86.99 ± 2.04

per class for training and the rest for testing. All methods are performed 50 times, and the average value and standard deviation are presented in Tables 1 and 2. (1) The result of Indian Pines: Indian data set is randomly selected 20 samples per class for training and the rest for testing. If half number of the samples is less than 20, we randomly select half number of the sample for training and the rest for testing. In Table 1, quantitative experimental results are listed. In all methods, the DFCF achieve the highest OA, AA and κ, and the standard deviation is relatively small, which indicates that the method has excellent classification accuracy and good stability for Indian Pines data set. In addition, in a total of 16 classes, there are 7 best class accuracies obtained by using DFCF, and 11 class accuracies of all classes are higher than 90%. Figure 4 shows the classification result obtained by utilizing DFCF method.

334

M. Li et al.

Table 2. PAVIA UNIVERSITY: the classification accuracy of different methods (%). Class

Samples Train Test

Methods GCK

Asphalt

20

6611 80.91 ± 8.55 84.17 ± 5.64 96.54 ± 2.82 87.24 ± 6.49

Meadows

20

18629 97.78 ± 1.29 84.09 ± 7.35 94.67 ± 3.05 84.77 ± 5.65

Gravel

20

2079 74.60 ± 8.30 83.69 ± 7.89 80.75 ± 13.03 90.30 ± 5.82

Trees

20

3044 80.00 ± 12.53 91.69 ± 4.54 73.78 ± 14.59 92.56 ± 4.62

NRS

EPF-G

DFCF

Painted metal 20 sheets

1325 99.73 ± 0.54 99.99 ± 0.05 94.61 ± 3.90 99.66 ± 0.21

Bare

20

5009 87.69 ± 6.87 86.45 ± 4.89 60.08 ± 12.21 94.60 ± 4.09

Bitumen

20

1310 79.03 ± 11.13 86.30 ± 6.68 76.99 ± 11.96 96.28 ± 3.43

Self-Blocking 20 Bricks

3662 70.02 ± 7.54 77.74 ± 8.29 84.96 ± 6.57 94.39 ± 4.20

Shadows

20

927 62.36 ± 10.52 92.91 ± 2.87 98.39 ± 1.30 100.00 ± 0.00

OA

86.36 ± 3.08 85.11 ± 3.08 83.54 ± 5.33 89.12 ± 2.64

AA

81.35 ± 3.38 87.45 ± 1.26 84.53 ± 3.82 93.31 ± 1.26

κ × 100

82.30 ± 3.85 80.85 ± 3.67 79.12 ± 6.32 85.99 ± 3.25

(2) The result of Pavia University: Figure 5 and Table 2, respectively, show intuitional and quantitative experiment results on Pavia University data set. In the case of the equal training numbles, each method achieves a better result than the result achieved on Indian Pine data set. The reason may be that the spatial resolution of Pavia University data set so that it achieves higher classification result. In all methods, the DFCF still achieve the highest OA, AA and κ, and the smallest standard deviation indicates the stability of the method. When the task is limited to the number of training samples, AA will become an important indicator. We have almost 7% advantages compared to the highest result of the contrast methods. In addition, in all the 9 classes, DFCF work best in 6 classes, and exceeds 99% in 2 classes.

4

Conclusion

In this letter, we have introduced deep forest combining with spectral-spatial cooperative features (DFCF) for deeper HSI feature extraction and classification. DFCF is based on deep framework, thus, it can be considered as a simple deep learning method. Through deep forest, the features are extracted into more representative features, which further increase the accuracy of the classification. Furthermore, in order to develop deep forest that is more suitable for HSI classification tasks, we have added a voting mechanism in its framework to get a significant classification result. Finally, we have used some of the most advanced methods on two popular datasets for comparison experiments, and the result has indicated that DFCF method works well and outperforms other methods.

Hyperspectral Image Classification Based on Deep Forest

335

Further research has the following points: (1) Further study of the application of deep forest in HSI classification. (2) Reduce the number of training samples and further improve classification accuracy. Acknowledgement. This work was supported by the Shanghai Association for Science and Technology under the Grant SAST2016090, and the National Natural Science Foundation of China under the Grants 61671037, and the Beijing Natural Science Foundation under the Grant 4152031, and the Excellence Foundation of BUAA for PhD Students under the Grants 2017057.

References 1. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001) 2. Chen, Y., Lin, Z., Zhao, X., Wang, G., Yanfeng, G.: Deep learning-based classification of hyperspectral data. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 7(6), 2094–2107 (2014) 3. Clausi, D.A., Ed Jernigan, M.: Designing Gabor filters for optimal texture separability. Pattern Recogn. 33(11), 1835–1849 (2000) 4. Ghamisi, P., Mura, M.D., Benediktsson, J.A.: A survey on spectral-spatial classification techniques based on attribute profiles. IEEE Trans. Geosci. Remote Sens. 53(5), 2335–2353 (2015) 5. Hughes, G.: On the mean accuracy of statistical pattern recognizers. IEEE Trans. Inf. Theory 5(1), 55–63 (1968) 6. Kang, X., Li, S., Benediktsson, J.A.: Spectral-spatial hyperspectral image classification with edge-preserving filtering. IEEE Trans. Geosci. Remote Sens. 52(5), 2666–2677 (2014) 7. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems, pp. 1097–1105 (2012) 8. Li, J., Marpu, P.R., Plaza, A., Bioucas-Dias, J.M., Benediktsson, J.A.: Generalized composite kernel framework for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 51(9), 4816–4829 (2013) 9. Li, W., Qian, D.: Gabor-filtering-based nearest regularized subspace for hyperspectral image classification. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 7(4), 1012–1022 (2014) 10. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002) 11. Pan, B., Shi, Z., Xu, X.: Hierarchical guidance filtering-based ensemble classification for hyperspectral images. IEEE Trans. Geosci. Remote Sens. 55(7), 4177–4189 (2017) 12. Pan, B., Shi, Z., Xu, X.: R-VCANet: a new deep-learning-based hyperspectral image classification method. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 10(5), 1975–1986 (2017) 13. Pan, B., Shi, Z., An, Z., Jiang, Z., Ma, Y.: A novel spectral-unmixing-based green algae area estimation method for GOCI data. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 10(2), 437–449 (2017) 14. Pan, B., Shi, Z., Zhang, N., Xie, S.: Hyperspectral image classification based on nonlinear spectral-spatial network. IEEE Geosci. Remote Sens. Lett. 13(12), 1782– 1786 (2016)

336

M. Li et al.

15. Xu, X., Shi, Z.: Multi-objective based spectral unmixing for hyperspectral images. ISPRS J. Photogrammetry Remote Sens. 124, 54–69 (2017) 16. Zhong, Z., Fan, B., Ding, K., Li, H., Xiang, S., Pan, C.: Efficient multiple feature fusion with hashing for hyperspectral imagery classification: a comparative study. IEEE Trans. Geosci. Remote Sens. 54(8), 4461–4478 (2016) 17. Zhou, Z.H: Ensemble Methods: Foundations and Algorithms. Taylor and Francis (2012) 18. Zhou, Z.H., Feng, J.: Deep forest: Towards an alternative to deep neural networks (2017)

Hyperspectral Image Classification Using Multi Vote Strategy on Convolutional Neural Network and Sparse Representation Joint Feature Daoming Ye1,2, Rong Zhang1,2 ✉ , and Dixiu Xue1,2 (

1

)

Department of Electronic Engineering and Information Science, USTC, Hefei 230027, China [email protected], [email protected], [email protected] 2 Key Laboratory of Electromagnetic Space Information, Chinese Academy of Sciences, Hefei, China

Abstract. Classification is one of the most popular topics in hyperspectral image (HSI). This paper proposes a method that uses multi vote strategy on convolu‐ tional neural network and sparse representation joint feature in hyperspectral image classification. First, the labeled spectral information was extracted by Principal Component Analysis (PCA) as well as the spatial information, at the same time, we feed the convolutional neural network and sparse representation joint feature to SVM. Then, we use multi-vote strategy to get the final result. Experimental results based on public database demonstrate that the proposed method provides better classification accuracy than previous hyperspectral clas‐ sification methods. Keywords: Convolutional neural networks · Sparse representation Support Vector Machines · Multi vote

1

Introduction

Hyperspectral image (HSI) contain hundreds of continuous narrow spectral bands, which span from visible to infrared spectrum. They are widely used in mineralogy, agriculture and surveillance. However, the high dimensionality and limited training samples of hyperspectral data increase the difficulty. It is different from usual image classification that hyperspectral image classification focus on solving problem of pixellevel, which is similar to image segmentation. To get better classification accuracy, we should add environmental factors. Several traditional methods have been used in hyperspectral image classification, such as KNN, maximum likelihood, Support Vector Machine (SVM), logistic regres‐ sion, sparse representation and so on. By building a low-dimensional subspace, Sparse representation generates a dictionary, where an unknown test pixel can be sparsely represented by a few training samples, then the sparse representation vector will include the class information. Dictionary-Based Sparse Representation in [1] and Kernel Sparse

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 337–346, 2017. https://doi.org/10.1007/978-3-319-71598-8_30

338

D. Ye et al.

Representation in [2] are useful in classification in HSI. SVM [3, 4] can handle large input space efficiently as well as deal with noisy samples robustly. In last several years, convolutional neural network methods achieve promising performance on many vision-related tasks, including image classification, scene labeling and face recognition. This model learns a good representation of features. The idea of CNNs was firstly introduced in [5], improved in [6], and refined in [7]. To deal with classification issue of HSI, the one dimension pixels have been treated as input in [8], simple feature maps generated by original pixels are also used in [9], Graph-based method has been proposed in [10]. By linear transformation, Principal Component Analysis (PCA) can extract the main characteristic components of the data as well as reduce the dimensionality. Convolu‐ tional Neural Network (CNN) implicitly learn feature from the training data which is closer to the actual biological neural network. The majority of sensory data can be represented as a superposition several atoms with associated weights by solving an optimization problem constrained by the sparsity level and reconstruction accuracy. In this paper, we propose a method which uses SVM to classify the joint feature generated by sparse representation and convolutional neural network and uses spatial labels to vote for the result, which consists of spatial information twice. First, we get spatial informa‐ tion and generate spatial-spectral samples by PCA. Second, we use classification results around to vote for the final result.

2

Algorithm Architecture

In this section, we first use PCA to extract spatial and spectral information from original HSI. Second, we respectively get CNN feature and sparse representation feature, and splice them into the joint feature as the input of SVM. Last, we vote for the output of SVM result with classification result of neighbors. The framework of the proposed method is shown in Fig. 1.

Fig. 1. Algorithm framework.

2.1 PCA To get better classification results, we take spatial information into consideration [11]. Due to the hundreds of channels along the spectral dimension, to reduce the data dimen‐ sion to a reasonable scale is necessary. We choose PCA approach to retain main

Hyperspectral Image Classification Using Multi Vote Strategy

339

information as well as reduce the spectral dimension to specified length L. After using PCA on the whole image regardless of whether the pixel has been marked, we select and flatten the eight tangent neighbors to form the spatial data. Then we can extract feature from labeled pixel samples with spatial and spectral information. Figure 2 shows the detailed extraction process.

Fig. 2. PCA framework.

2.2 Convolutional Neural Networks Convolutional neural networks which are designed to learn features directly from the images consist of various combination of convolutional layers, max pooling layers and fully connected layers. CNN learn from training data implicitly instead of extracting the feature explicitly. CNN reduce the number of parameters significantly with the help of weight sharing and local perception. Each HSI pixel sample can be regarded as a 2D image whose height is equal to 1, so the input layer is (n, 1), where n is the length of spatial-spectral information. The framework of CNN is shown in Fig. 3. The first hidden convolutional layer filters the n × 1 input data with n1 kernels of size k1 × 1, then the max pooling layer is the second

Fig. 3. CNN framework.

340

D. Ye et al.

hidden layer with the kernel size of m1 × 1. The later 3 convolutional layers with param‐ eters n2 , n3 , n4 , k2 , k3 , k4 and 2 max pooling layers with parameters m2 , m3 are similar. The fully connected layer which follows by the flatten layer has p nodes, and the output layer has the same nodes as the number of the classification. We use the result of the fully connected layer as generated feature where the length is p while the output layer assists training. 2.3 Sparse Representation The pixels belonging to the same class approximately lie in the same low-dimensional subspace. For an unknown test pixel y, there is a similar expression of linear combination of all of the training pixels as: y = Dx

(1)

Where D is an over-complete dictionary generated by training pixels from all classes, and x is the sparse vector with few non-zero entries. To avoid each feature to be assigned to too many clusters, the sparse constraint on the weight vector x is usually added to the objective. We can obtain x by solving the problem: x = argmin‖y − Dx‖2 + �‖x‖0

(2)

Where � > 0 is used to balance sparsity and reconstruction accuracy. We can solve the problem approximately by Orthogonal Matching Pursuit. During the training period, we calculate the dictionary D, which can be used to get the sparse vector x for the test samples. In this way, we transform the original spectral-spatial pixel samples into a few sparse feature where the length is q, then we can combine them with the feature generated by CNN to participate in later classification. 2.4 SVM

{ }N Given a training set of N data points xk , yk k=1, where xk ∈ Rn is the kth input pattern and yk ∈ Rn is the kth output pattern [12], the support vector method approach aims at constructing a classifier of the form: [ y(x) = sign

N ∑

(

)

]

ak yk � x, xk + b

(3)

k=1

where ak are positive real constants and b is a real constant. The classifier is constructed as follow: [ ( ) ] yk wT � xk + b ≥ 1 − �k , �k ≥ 0, k = 1, … , N,

(4)

where �(.) is a nonlinear function which maps the input space into a higher dimensional space. SVM solution is not applied directly on original spectral-spatial data, instead, we

Hyperspectral Image Classification Using Multi Vote Strategy

341

prefer to extract deep feature first. Feature generated by CNN and sparse coding plays a role as the input of SVM method. 2.5 Multi Vote The pixel samples to be classified are not real independent. We can reconstruct a prelabeled image with classification results as well as untagged pixel samples which are marked as zero. As it shows in Fig. 4, the central location (j, i) is the classification result of test sample. The central value is 5 while most results around are 2, then the test sample may be classified by mistake. There are many isolated point which are different from neighbors. We use the spatial label around each test pixel’s neighborhood to smooth it. To reduce smoothing error, we use samples nearby both in geometric space and in spectral space.

i-2

i-1

i

i+1

i+2

j-2

1

1

2

2

2

j-1

2

2

2

2

1

j

2

2

5

2

2

j+1

1

2

2

2

2

j+2

1

2

0

0

0

Fig. 4. Multi vote framework.

The proposed framework takes all the tagged pixels in a flat neighbor into consid‐ eration, where the k nearest neighbors in spectral dimensional space may vote for the final result.

3

Experiments

In this section, we use two hyperspectral databases, including Salinas, and University of Pavia scenes to verify the feasibility of our algorithm. All the programs are imple‐ mented using Python language and Theano library, which is efficient and convenient on CNN computation. 3.1 The Databases The Pavia University scene is acquired by the ROSIS sensor during a flight campaign over Pavia, north Italy. Pavia University is a 610 * 610 pixels image with 103 bands, but some of the samples contain no information and have to be discarded before the analysis. The geometric resolution is 1.3 m. Image ground truth differentiates 9 classes,

342

D. Ye et al.

including Asphalt, Meadows, Gravel, Trees, Painted metal sheets, Bare Soil, Bitumen, Self-Blocking Bricks, Shadows and so on (Table 1). Table 1. Classes and samples number for the Pavia University scene 1 2 3 4 5 6 7 8 9

Class Asphalt Meadows Gravel Trees Painted metal sheets Bare Soil Bitumen Self-Blocking Bricks Shadows

Samples 6631 18649 2099 3064 1345 5029 1330 3682 947

The Salinas scene is collected by the 224-band AVIRIS sensor over Salinas Valley, California, and is characterized by high spatial resolution (3.7-m pixels). The area covered comprises 512 lines by 217 samples. We discard the 20 water absorption bands, and the left 204 bands are used to describe 16 classes (Table 2). Table 2. Classes and samples number for the Salinas scene 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Class Samples Brocoli_green_weeds_1 2009 Brocoli_green_weeds_2 3726 Fallow 1976 Fallow_rough_plow 1394 Fallow_smooth 2678 Stubble 3959 Celery 3579 Grapes_untrained 11271 Soil_vinyard_develop 6203 Corn_senesced_green_weeds 3278 Lettuce_romaine_4wk 1068 Lettuce_romaine_5wk 1927 Lettuce_romaine_6wk 916 Lettuce_romaine_7wk 1070 Vinyard_untrained 7268 Vinyard_vertical_trellis 1807

Hyperspectral Image Classification Using Multi Vote Strategy

343

3.2 Parameter Setting For each scene, we choose 30% from all tagged pixels randomly as training samples, and the rest are testing samples. To spectral-spatial information, we set different length L in PCA method to keep the spectral and spatial information are not far-off. The Pavia University scene has 103 bands, we set L = 10, then the length of spatial information is 3 * 3 * 10. The Salinas scene has 204 bands, we set L = 20, then the length of spatial information is 3 * 3 * 20. In CNN architecture, we set n1 = 12, n2 = 32, n3 = 64, n4 = 128, m1 = 2, m2 = 2, m3 = 2, k1 = 4, k2 = 5, k3 = 4, k4 = 5, p = 1000. In sparse representation, we set the length of sparse vector q = 200, so the combined feature as the input of SVM classifier has total length of 1200. In multi vote architecture, we take 5 * 5 window around each testing sample into consideration, while the k nearest neighborhoods contain 11 labels with smaller Eucli‐ dean distance in spectral dimensional space. If there are more than 6 labels with the same value not equal to the classification result of the testing sample, we will cover the previous classification result. 3.3 Result and Analysis Figures 5 and 6 show the experiment results of proposed method with different public databases. Each figure includes three parts: the original image, the ground truth and the result of proposed method. most test samples are correctly classified with proposed method.

(a) Original image

(b) ground truth

(c) proposed method.

Fig. 5. Experiment on Pavia University image.

344

D. Ye et al.

(a) Original image

(b) ground truth

(c) proposed method.

Fig. 6. Experiment on Salinas image.

(a) Original image

(b) before multi vote

(c) after multi vote.

Fig. 7. Experiment on Pavia University image.

(a) Original image

(b) before multi vote

(c) after multi vote.

Fig. 8. Experiment on Salinas image.

Figures 7 and 8 show the result before and after multi vote method as follows. The result of feeding the sparse representation and convolutional neural network joint feature to SVM is quite good, the labels of pixels around each text sample are basically correct,

Hyperspectral Image Classification Using Multi Vote Strategy

345

then the multi vote method effective removes isolated points, which are probably esti‐ mated by mistake. We design several comparative experiments. The CNN result uses only 1-D CNN to extract feature, while the SSDCNN transform 1-D information into 2-D feature maps as the input of CNN. The best performance has been highlighted in bold. As shown in Tables 3 and 4, the proposed model outperforms other methods in each indices. Overall accuracy (OA), average accuracy (AA) and Kappa coefficient (�) are used as measure‐ ment indices. Table 3. Classification accuracy for University of Pavia OA AA �

RBF-SVM 0.9052 0.8927 0.8964

CNN 0.9256 0.9128 0.9143

SSDCNN 0.9518 0.9351 0.9364

Proposed 0.9836 0.9792 0.9784

Table 4. Classification accuracy for Salinas OA AA �

4

RBF-SVM 0.9166 0.9503 0.9056

CNN 0.9260 0.9581 0.9146

SSDCNN 0.9408 0.9743 0.9339

Proposed 0.9802 0.9920 0.9780

Conclusion

In this paper, we proposed a classification method for HSI. We use spatial information twice. In PCA part, spatial information is used to form spectral-spatial samples, and in multi vote part spatial information is used to smooth isolated point. SVM is applied to generate the previous classification map with CNN and joint feature. Through further smooth, classification problem was transformed into segmentation problem. The proposed method could achieve higher accuracy on each public database.

References 1. Chen, Y., Nasrabadi, N., Tran, T.: Hyperspectral image classification using dictionary-based sparse representation. IEEE Trans. Geosci Remote Sens. 49(10), 3973–3985 (2011) 2. Chen, Y., Nasrabadi, N., Tran, T.: Hyperspectral image classification via kernel sparse representation. IEEE Trans. Geosci Remote Sens. 51(1), 217–231 (2013) 3. Gualtieri, J.A., Cromp, R.F.: Support vector machines for hyperspectral remote sensing classification. In: Proceedings SPIE 27th AIPR Workshop: Advances in Computer Assisted Recognition, Washington, DC, October 1998, vol. 3584, pp. 221–232 (1998) 4. Melgani, F., Bruzzone, L.: Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans. Geosci. Remote Sens. 42(8), 1778–1790 (2004) 5. Fukushima, K.: Neocognitron: a hierarchical neural network capable of visual pattern recognition. Neural Netw. 1(2), 119–130 (1988)

346

D. Ye et al.

6. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2323 (1998) 7. Cireşan, D.C., Meier, U., Masci, J., Gambardella, L.M., Schmidhuber, J.: Flexible, high performance convolutional neural networks for image classification. In: Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI 2011), vol. 22, pp. 1237– 1242, July 2011 8. Hu, W., Huang, Y., Wei, L., Zhang, F., Li, H.: Deep convolutional neural networks for hyperspectral image classification. J. Sensors 2015, 12 (2015). Article ID 258619, Hindawi Publishing Corporation 9. Yue, J., Zhao, W., Mao, S., Liu, H.: Spectral–spatial classification of hyperspectral images using deep convolutional neural networks. Remote Sens. Lett. 6(6), 468–477 (2015) 10. Cao, J., Chen, Z., Wang, B.: Graph-based deep Convolutional networks for Hyperspectral image classification. In: IGARSS 2016 (2016). ISBN 978-1-5090-3332-4/16 11. Tarabalka, Y., Benediktsson, J.A., Chanussot, J.: Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques. IEEE Trans. Geosci. Remote Sens. 47(8), 2973–2987 (2009). View at Publisher, View at Google Scholar, View at Scopus 12. Suykens, J.A.K., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9, 293–300 (1999)

Efficient Deep Belief Network Based Hyperspectral Image Classification Atif Mughees(B) and Linmi Tao Computer Vision and Graphics Lab, Department of Computer Science and Technology, Tsinghua University, Haidian, Beijing 100084, China [email protected], [email protected]

Abstract. Hyperspectral Image (HSI) classification plays a key role remote sensing field. Recently, deep learning has demonstrated its effectiveness in HSI Classification field. This paper presents a spectral-spatial HSI classification technique established on the deep learning based deep belief network (DBN) for deep and abstract feature extraction and adaptive boundary adjustment based segmentation. Proposed approach focuses on integrating the deep learning based spectral features and segmentation based spatial features into a framework for improved performance. Specifically, first the deep DBN model is exploited as a spectral feature extraction based classifier to extract the deep spectral features. Second, spatial contextual features are obtained by utilizing effective adaptive boundary adjustment based segmentation technique. Finally, maximum voting based criteria is operated to integrate the results of extracted spectral and spatial information for improved HSI classification. In general, exploiting spectral features from DBN process and spatial features from segmentation and integration of spectral and spatial information by maximum voting based criteria, has a substantial effect on the performance of HSI classification. Experimental performance on real and widely used hyperspectral data sets with different contexts and resolutions demonstrates the accuracy of the proposed technique and performance is comparable to several recently proposed HSI classification techniques. Keywords: Hyperspectral image classification Deep belief network

1

· Segmentation

Introduction

Recent advances in remote sensing technology has enabled the sensors to acquire hyperspectral images (HSI) in hundreds of continuous narrow spectral channels captured in the wide range of electromagnet spectrum ranging from visible to infrared. Each pixel in HSI is a representation of the spectral characteristics of the spatial location in the scene [29] and composed of a vector of spectral entries form available spectral channels. Rich spectral and spatial information leads to c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 347–357, 2017. https://doi.org/10.1007/978-3-319-71598-8_31

348

A. Mughees and L. Tao

extensive applications such as land cover mapping, target detection, classification, mineral detection, surveillance and so on. However rich information and extensive application comes with lot of challenge connected to high dimensionality and limited available samples [2] together with Hughes phenomena [13] and heterogeneity [4]. Large number of spectral channels but limited number of training samples leads to curse of dimensionality [13]. Therefor exploiting spatial/structural features along with the spectral features and design of a classifiers plays a crucial role in HSI classification. Many techniques has been proposed so far to deal with the HSI classification. Traditional pixel-wise classifiers deal each pixel autonomously without taking spatial information into account. K-nearest neighbour classifier (K-NN), conditional random fields [32], neural networks [24], support vector machine [9, 18] have been investigated. Out of these pixel-wise classifiers, SVM performed better due to its ability to handle high dimensional data. Majority of the above mentioned classifiers suffers from curse of dimensionality and limited training data [1]. Moreover spatial information is not taken into account as there is strong association between adjacent pixels [29] but pixel-wise classifiers deals each pixels independently. Dimensionality reduction approaches were also proposed to handle the higher dimensionality and limited training samples. Principal component analysis (PCA) [25], independent component analysis (ICA) [28] are some of the well known approaches. These approaches reduces the features/dimensions from hundreds to just few bands hence results into loss of spectral information. Band/feature selection [23] is another technique of handling the above mentioned issues. Integration of spatial information along with pure spectral information for improved performance in HSI classification has been getting more and more attention of the researchers in recent years [3]. It is widely established that the complement of spectral and spatial features can result into more effective classification [22]. It is therefore necessary to incorporate the spatial features into a spectral-spatial classifier. Spectral spatial techniques can broadly be divided into three categories in which the spatial information is incorporated along with the spectral features (a) before the classification (b) during the classification process (c) after the classification process. In the first category, many techniques extracts the spatial features and integrate it with spectral features before the classification process such as spatial feature extraction through morphological profiles [7, 8,15] and through segmentation [22]. Similarly composite kernel methods concatenate spatial features with other spectral features [17, 33]. However, in most of the cases these features requires human knowledge and are mostly handcraft. In the second category of spectral-spatial classification, spatial features are incorporated into a classifier during the classification process such as statistical learning theory (SLT) [4], simultaneous subspace pursuit (SSP) [5]. In the third category of spectral-spatial classification, spatial features are incorporated after the classification process. Authors in [26] first utilized SVM for pixel-wise spectral classification and watershed segmentation for spatial feature extraction followed by majority voting within the result of pixel-wise classification and watershed

Efficient Deep Belief Network Based Hyperspectral Image Classification

349

segmentation. Authors in [16] utilized augmented Lagrangian multilevel logistic with a multilevel logistic (MLL) prior (LORSAL-MLL). Similarly authors in [21] integrates the results from segmentation and SAE based classification through majority voting.

(a) I. Spectral stage II. Spatial stage III. Combined stage (b) Spectral-Spatial Classification Framework

Fig. 1. Spectral-Spatial Classification stages and Framework.

Recently, a latest development in neural network, deep learning has proved its efficiency and efficacy in many fields particularly in computer vision such as image classification [11], speech recognition [30], language processing [19]. Deep learning based architectures has also performed well in HSI classification [31]. However incorporating spatial features into a deep network is still a persistent issue. In this paper, spectral-spatial HSI classification based on deep learning based deep belief network (DBN) and hyper-segmentation based spatial feature extraction is proposed. Spectral feature extraction is exploited through deep learning based DBN architecture [10] and logistic regression (LR) is applied as a pixelwise classifier while spatial features are extracted through structural boundary adjustment based hyper-segmentation [20] which adaptively segments the HSI image. Proposed approach is based on the third category of spectral-spatial HSI classification where spectral and spatial information is effectively incorporated

350

A. Mughees and L. Tao

after the classification. Decision to label the target pixel for a specific class is simultaneously based on the DBN based pixel-wise classification and additional spatial features obtained from effective segmentation. Accurate segmentation approach to exploit the spatial features makes this approach more effective.

2

Proposed Methodology

It is strongly believed in HSI research community that incorporating spatial contextual features can significantly improve the classification performance [34]. Proposed method first exploits multi-layer DBN for effective deep and abstract feature extraction and ML is utilized for subsequent pixel-wise classification. For contextual spatial features, adaptive boundary adjustment based hypersegmentation [20] is employed. In the third phase, majority voting [14] base process is utilized to fully exploit and integrate the spectral and spatial features for final spectral-spatial classification. Detailed description of each phase is depicted in Fig. 1.

Fig. 2. Framework of the DBN based pixel-wise classification.

2.1

Spectral Feature Extraction via DBN

Deep belief network is composed of neural network based Restricted Boltzmann Machine (RBM) learning module that consists of input data layer or visible layer x and a hidden layer y that learn to distinguish features with higher correlations in the input data as shown in Fig. 1. The energy function can be described as: E(x, y, θ) = −

m m  n n   (xj − bj )2  xj wij yi a y − − i i 2 2σ σi j=1 j=1 i=1 i=1

The conditional distributions are given by:  m  wji xj + ai P (yj |x; θ) = h

(1)

(2)

i=1

⎞ ⎛ m  P (xj |y; θ) = V ⎝ wji yj σj2 + bj ⎠ j=1

(3)

Efficient Deep Belief Network Based Hyperspectral Image Classification

351

where σ is the standard deviation (SD)of a Gaussian visible unit, and V (.) is the Gaussian distribution. A deep belief network is mainly comprised of restricted Boltzmann machine stacks, and the learning of RBM plays an essential role in DBN. The block diagram of image classification using DBN, in general, is shown in Fig. 2. At learning stage, training dataset is processed in order to get the spatial and spectral information from hyperspectral images. After that, the parameters of DBN model are adjusted by learning which includes back propagation for fine tuning. In the classification stage, the learned network is used to classify the test sample set and output the classification results. We have used DBN-LR in which DBN is use for feature extraction from spectral images and classification is made by logistic regression. 2.2

Spatial Feature Extraction via Hyper-segmentation

Two spatial constraints must be incorporated while spatial feature extraction, (1) There is a high probability that pixels with the same spectral signatures shares the same class label (2) There is a high probability that neighbouring pixels with the similar spectral signatures share the same class label. In order to full fill the above constraints an effective adaptive boundary adjustment based approach [20] is exploited to segment the HSI. The tri-factor based energy function is given by: A(q, Pi ) =

|xq − gi |2 + λ˜ ni (q)|Grad(q)|

(4)

where xq is the spectral vector at the boundary pixel, gi is the majority vector, n ˜ i (q) is the straightness factor, |Grad(q)| is the local gradient at target pixel q. Detailed implementation of the algorithm for spatial segmentation can be viewed in [20] (Fig. 3).

Fig. 3. Framework of the hyper-segmentation process.

352

2.3

A. Mughees and L. Tao

Majority Voting

The individual classification results obtained from DBN-LR classifier and Segmentation based spatial classification are integrated through majority voting (MV) [14]. In MV each pixel in the segmentation region is assigned to a most repetitive class allocated by the DBN-LR classifier. Hence both spectral and spatial features are taken into account.

3

Experimental Results and Performance Comparison

To validate the performance proposed technique for HSI classification, experiments are conducted on well known and challenging datasets which are

Fig. 4. Hyperspectral image datasets.

Efficient Deep Belief Network Based Hyperspectral Image Classification

353

widely used by other well known HSI classification techniques to validate the results (Fig. 4). 3.1

Hyperspctral Dataset

Two popular datasets Pavia University and Houston University are utilized for performance evaluation due to their distinction and difficulty. Houston University dataset was acquired by AVIRIS sensor in 1992. It consists of 144 spectral channels with spatial dimension of 349 × 1905. Indian Pine is considered difficult for classification due to small spatial structures and presence of mixed pixels. It consists of 15 classes. Pavia University dataset was collected by ROSIS sensor over the Pavia University, Italy. It comprises of 103 spectral channels with spatial dimension of 610 × 340. This dataset includes both main made structures and natural plants. It consists of 9 classes. Mostly, in the literature two datasets are considered to demonstrate the validation and accuracy of proposed techniques for HSI classification. Classification performance is estimated using the evaluation criteria based on overall accuracy (OA), Average accuracy (AA) and kappa Coefficient (k). OA is the percentage of pixels correctly classified. AA is the mean of all the class specific accuracy over the total number for classes for the specific image. Kappa is a

Fig. 5. Classification Results of Houston and Pavia University datasets using proposed method.

354

A. Mughees and L. Tao

degree of agreement between predicted class accuracy and reality. Generally, it is considered more robust than OA and AA. 3.2

Spectral-Spatial DBN-HS Classification

We conducted experiments on windows 7 system, with 4.0 GHz processor and NVIDIA GeForce GTX 970. The code was implemented in Theano. Number of hidden layers also known as depths plays a significant role in the classification performance as it characterizes the quality of the learned features. For each dataset, we choose randomly 10% of each class as the training data as training data. For Pavia University and Houston University datasets we selected depth of size 2 and number of hidden units for each hidden layer is 50 as suggested by the experiments in [6]. The performance of the proposed DBN-HS technique is compared with well known existing techniques such as support vector machine (SVM) [18], orthogonal matching pursuit (OMP) [27], deep belief network with logistic regression(DBN-LR) [6] and newly developed deep CNN (CNN) [12]. In case of DBN-LR, only spectral data as an input was considered. Individual class level accuracy results of Pavia university and Houston University dataset and their comparison with mentioned well known existing techTable 1. Classification accuracy(%) of each class for the Houston University dataset obtained by the SVM [18], OMP [27], CNN [12] using 10% training samples Class Training Test

SVM

OMP

CNN

DBN-LR DBN-HS

1

125

1126 97.47

98.27

81.20

99.20

99.0

2

125

1129 98.32

98.10

83.55

99.60

99.20

3

70

627 99.37

99.68

99.41

100.0

100

4

124

1120 98.01

96.70

91.57

99.60

99.60

5

124

1118 96.01

98.48

94.79

99.60

99.60

6

33

292 99.83

97.95

95.10

97.2

98.05

7

127

1141 91.23

86.90

63.53

97.0

98.16

8

124

1120 86.23

89.82

42.64

97.8

98.0

9

125

1127 86.99

79.37

58.17

94.0

95.25

10

123

1104 91.42

89.68

41.80

97.4

97.95

11

124

1111 91.67

82.77

75.71

97.3

98.1

12

123

1110 87.05

81.94

84.15

95.2

96.26

13

47

422 78.16

35.55

40.00

88.0

91.1

14

43

385 97.42

98.18

98.79

100

100

15

66

594 99.49

98.40

97.89

100

100

Overall accuracy

93.06

89.70

85.42

97.70

98.98

Average accuracy

93.25

88.78

76.55

97.50

98.46

Kappa coefficient

0.925

0.889

0.7200 0.975

0.9875

Efficient Deep Belief Network Based Hyperspectral Image Classification

355

Table 2. Classification accuracy of each class for the Pavia University dataset obtained by the SVM [18], OMP [27], CNN [12] using 10% training samples Class Training Test

SVM OMP CNN

6034 97.50 64.16 87.34

DBN-LR DBN-HS

1

597

2

1681

16971 97.70 82.23 94.63 92.10

87.37

94.01

3

189

1910 78.53 71.04 86.47 85.57

88.50

4

276

2788 89.29 93.43 96.29

95.11

97.40

5

121

1224 98.77 99.90 99.65

99.74

99.89

6

453

4576 83.04 69.47 93.23

91.94

94.30

7

120

1210 64.58 87.31 93.19 92.21

93.96

8

331

3351 86.90 71.57 86.42

87.02

88.14

9

85

862 99.92 97.27 100

100

100

89.78

Overall accuracy

92.04 78.07 92.56

91.18

93.98

Average accuracy

88.47 81.82 93.02

92.34

93.46

Kappa coefficient

0.903 0.711 0.9006 0.8828

0.9175

niques is shown in Tables 1 and 2. Mixed pixel is the major challenge in Houston dataset due to its low spatial resolution and small spatial size. In Houston University dataset, proposed technique performed well in classes with small spatial regions as effective segmentation plays a very important role in segmenting those small spatial regions and making them available for effective classification. The complete HSI classification result of proposed method is shown in Fig. 5. Each color characterizes a specific type of ground cover area which is the same as aforementioned ground truth image. Results confirm that spectralspatial classification using contextual feature extraction has significant effect on the classification accuracy because spatial features help prevent the salt and paper noise. Overall, experimental results demonstrates the significant improvement in HSI classification by incorporating spatial information and spectral feature selection. The algorithm has performed significantly well on the low spatial resolution dataset.

4

Conclusion

In this paper a new hyperspectral image classification DBN-HS approach based on Deep Belief Network and hyper-segmentation is proposed by taking spectral and spatial information into account. DBN based logistic regression (DBN-LR) is used for extraction of deep spectral features and hyper-segmentation is utilized for exploiting the spatial features. In the final step, DBN-LR based spectral features and hyper-segmentation based spatial features are integrated through majority voting (MV) for the efficient spectral-spatial classification of HSI. Hyper-segmentation based segmentation defines an adaptive neighborhood for

356

A. Mughees and L. Tao

each pixel. Experimental results and comparisons with the well known existing methods demonstrates that the spectral-spatial classification, based on majority voting within the regions obtained by the hyper-segmentation algorithms, led to higher classification accuracy as compare to pixel-wise classification. Use of MV for the fusion of local spectral information through DBN-LR and spatial information through effective hyper-segmentation based segmentation has a significant effect on the accuracy of the final HSI classification.

References 1. Ambikapathi, A., et al.: Convex geometry based outlier-insensitive estimation of number of endmembers in hyperspectral images. Signal 1, 1–20 (2012) 2. Benediktsson, J.A., Chanussot, J.C., Moon, W.M., et al.: Advances in Very highresolution Remote Sensins. IEEE (2013) 3. Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013) 4. Camps-Valls, G., et al.: Advances in hyperspectral image classification: earth monitoring with statistical learning methods. IEEE Signal Process. Mag. 31(1), 45–54 (2013) 5. Chen, Y., Nasrabadi, N.M., Tran, T.D.: Hyperspectral image classification via kernel sparse representation. IEEE Trans. Geosci. Remote Sens. 51(1), 217–231 (2013) 6. Chen, Y., Zhao, X., Jia, X.: Spectral-spatial classification of hyperspectral data based on deep belief network. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 8(6), 2381–2392 (2015) 7. Ghamisi, P., Benediktsson, J.A., Sveinsson, J.R.: Automatic spectral-spatial classification framework based on attribute profiles and supervised feature extraction. IEEE Trans. Geosci. Remote Sens. 52(9), 5771–5782 (2014) 8. Ghamisi, P., et al.: Automatic framework for spectral-spatial classification based on supervised feature extraction and morphological attribute profiles. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 7(6), 2147–2160 (2014) 9. Gualtieri, J.A., Chettri, S.: Support vector machines for classification of hyperspectral data. In: IEEE 2000 International Geoscience and Remote Sensing Symposium, Proceedings, IGARSS 2000, vol. 2, pp. 813–815. IEEE (2000) 10. Hinton, G.E., Osindero, S.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006) 11. Hinton, G.E.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006) 12. Hu, W., et al.: Deep convolutional neural networks for hyperspectral image classification. J. Sens. 2015 (2015) 13. Hughes, G.: On the mean accuracy of statistical pattern recognizers. IEEE Trans. Inf. Theory 14(1), 55–63 (1968) 14. Lam, L., Suen, S.Y.: Application of majority voting to pattern recognition: an analysis of its behavior and performance. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 27(5), 553–568 (1997) 15. Li, J., Zhang, H., Zhang, L.: Supervised segmentation of very high resolution images by the use of extended morphological attribute profiles and a sparse transform. IEEE Geosci. Remote Sens. Lett. 11(8), 1409–1413 (2014)

Efficient Deep Belief Network Based Hyperspectral Image Classification

357

16. Li, J., Bioucas-Dias, J.M.: Hyperspectral image segmentation using a new Bayesian approach with active learning. IEEE Trans. Geosci. Remote Sens. 49(10), 3947– 3960 (2011) 17. Li, J., et al.: Generalized composite kernel framework for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 51(9), 4816–4829 (2013) 18. Melgani, F., Bruzzone, L.: Classification of hyperspectral remote sensing images with support vector machines. IEEE Trans. Geosci. Remote Sens. 42(8), 1778–1790 (2004) 19. Mohamed, A., Dahl, G., Hinton, G.E.: Deep belief networks for phone recognition. In: NIPS Workshop on Deep Learning for Speech Recognition and Related Applications, vol. 1, no. 9, p. 39 (2009) 20. Mughees, A., Chen, X., Tao, L.: Unsupervised hyperspectral image segmentation: merging spectral and spatial information in boundary adjustment. In: 2016 55th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), pp. 1466–1471. IEEE (2016) 21. Mughees, A., Tao, L.: Efficient deep auto-encoder learning for the classification of hyperspectral images. In: 2016 International Conference on Virtual Reality and Visualization (ICVRV), pp. 44–51. IEEE (2016) 22. Mughees, A., Tao, L.: Hyper-voxel based deep learning for hyperspectral image classification. In: 2017 IEEE International Conference on Image Processing (ICIP) (2017, Accepted) 23. Mughees, A., et al.: AB3C: adaptive boundary-based band-categorization of hyperspectral images. J. Appl. Remote Sens. 10(4), 046009–046009 (2016) 24. Ratle, F., Camps-Valls, G., Weston, J.: Semisupervised neural networks for efficient hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 48(5), 2271– 2282 (2010) 25. Rodarmel, C., Shan, J.: Principal component analysis for hyperspectral image classification. Surveying Land Inform. Sci. 62(2), 115 (2002) 26. Tarabalka, Y.: Classification of hyperspectral data using spectral-spatial approaches. PhD thesis, Institut National Polytechnique de Grenoble-INPG (2010) 27. Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inf. Theory 53(12), 4655–4666 (2007) 28. Wang, J., Chang, C.-I.: Independent component analysis-based dimensionality reduction with applications in hyperspectral image analysis. IEEE Trans. Geosci. Remote Sens. 44(6), 1586–1600 (2006) 29. Willett, R.M., et al.: Sparsity and structure in hyperspectral imaging: sensing, reconstruction, and target detection. IEEE Signal Process. Mag. 31(1), 116–126 (2014) 30. Yu, D., Deng, L., Wang, S.: Learning in the deep-structured conditional random fields. In: Proceedings of NIPS Workshop, pp. 1–8 (2009) 31. Zhang, L., Zhang, L., Bo, D.: Deep learning for remote sensing data: a technical tutorial on the state of the art. IEEE Trans. Geosci. Remote Sens. 4(2), 22–40 (2016) 32. Zhang, L., Zhang, L., Du, B.: Learning conditional random fields for classification of hyperspectral images. IEEE Trans. Image Process. 19(7), 1890–1907 (2010) 33. Zhou, Y., Peng, J., Chen, C.L.P.: Extreme learning machine with composite kernels for hyperspectral image classification. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 8(6), 2351–2360 (2015) 34. Zhu, Z., et al.: Assessment of spectral, polarimetric, temporal, and spatial dimensions for urban and peri-urban land cover classification using Landsat and SAR data. Remote Sens. Environ. 117, 72–82 (2012)

Classification of Hyperspectral Imagery Based on Dictionary Learning and Extended Multi-attribute Profiles Qishuo Gao1(&)

, Samsung Lim1

, and Xiuping Jia2

1

School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW, Australia [email protected] 2 School of Engineering and Information Technology, The University of New South Wales, Canberra, ACT, Australia

Abstract. In recent years, sparse representation has shown its competitiveness in the field of image processing, and attribute profiles have also demonstrated their reliable performance in utilizing spatial information in hyperspectral image classification. In order to fully integrate spatial information, we propose a novel framework which integrates the above-mentioned methods for hyperspectral image classification. Specifically, sparse representation is used to learn a posteriori probability with extended attribute profiles as input features. A classification error term is added to the sparse representation-based classifier model and is solved by the k-singular value decomposition algorithm. The spatial correlation of neighboring pixels is incorporated by a maximum a posteriori scheme to obtain the final classification results. Experimental results on two benchmark hyperspectral images suggest that the proposed approach outperforms the related sparsity-based methods and support vector machine-based classifiers. Keywords: Extended multi-attribute profiles
Markov Random Field Hyperspectral imagery
Sparse representation classification

1 Introduction Hyperspectral imagery (HSI) has been widely used in the field of remote sensing for the past decade. Its capability to acquire hundreds of images with a wide range of wavelengths makes HSI a powerful tool in many areas, such as military surveillance, natural resources detection, land cover classification, etc. [1]. However, the unique properties of HSI have posed difficult image processing problems; for instance, it has been identified as a challenging task to analyze the spectral and spatial information simultaneously for HSI classification [2]. Many attempts have been made to solve this problem. In [3], the authors proposed a new sampling strategy to extract both spectral and spatial information. In addition, Markov Random Field (MRF) is considered to be a powerful method for integrating both spectral and spatial information [4]. However, its efficiency and effectiveness are questionable due to the high computational complexity and the uncertainty of the smoothing parameter to be chosen. Another category of approaches to deal with © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 358–369, 2017. https://doi.org/10.1007/978-3-319-71598-8_32

Classification of Hyperspectral Imagery

359

contextual information is based on attribute profiles (APs) which are constructed by a set of attribute filters (AFs). AFs operate only on the connected components based on a criterion that evaluates the attribute against a threshold. When multiple layers of HSI are considered, the stack of individually computed APs can be referred to as an extended attribute profile (EAP) [5]. Moreover, if different attributes are considered and multiple EAPs are stacked together, an extended multi-attribute profile (EMAP) can be constructed [6]. EMAP can effectively deal with both spectral and contextual information [7]. Sparse representation-based classifiers (SRCs) have been found to be efficient tools in many image processing areas in the last few years [8–10]. SRC assumes that each signal can be expressed as a linear combination of prototypes selected from a dictionary. The advantages of applying SRC as a classification method have been investigated [11–13]. SRC can achieve good performance on HSI classification because the pixels with highly correlated bands can be sparsely represented. Usually a SRC dictionary is directly constructed by training atoms, which limits the HSI classification accuracy due to the large number of training atoms. Hence it is sensible to construct a reliable dictionary for the classification problem. Based on the aforementioned knowledge, a novel framework using both EMAP and SRC is developed and presented in this paper particularly for HSI classification, which we name it as extended SRC (ESRC) method. Apart from spectral information, EMAPs have been constructed to initialize the dictionary for SRC. Thus, both spectral and spatial context can be considered to maximize among class separability. Subsequently, we optimize the dictionary using an effective method known as k-singular value decomposition (K-SVD) [14]. Similar to Jiang [15], we add a classification error term to the SRC model, then the reconstruction error and the classification error can be modelled simultaneously. Finally, the class label can be derived via the MRF-based maximum a posteriori (MAP) method, where the spatial energy term is improved by a Gaussian framework in this paper. It should be noted that the spatial information is utilized via EMAPs and then regularized by the MRF-MAP method, therefore our ESRC can further improve the classification results. The remainder of this paper is organized as follows. The proposed framework is described in Sect. 2. The effectiveness of the framework is demonstrated by the experiments in Sect. 3. Finally, we conclude and provide some remarks in Sect. 4.

2 Design of Framework 2.1

EMAP Feature Extraction

EAPs are built by concatenating many attribute profiles (APs), and each AP is generated for each feature in a scalar hyperspectral image. That is: EAP ¼ fAPðf1 Þ; APðf2 Þ; . . .; APðfn Þg

ð1Þ

360

Q. Gao et al.

APs are a generalized form of morphological profiles, which can be obtained from an image by applying a criterion T. By using n morphological thickening (uT ) and n thinning (;T ) operators, an AP can be constructed as: APðf Þ ¼ fuTn ðf Þ; uTn 1 ðf Þ; . . .; uT1 ðf Þ; f ; /T1 ðf Þ; . . .; /Tn 1 ðf Þ; /Tn ðf Þg

ð2Þ

Generally, there are some common criteria associated with the operators, such as area, volume, diagonal box, and standard deviation. According to the operators (thickening or thinning) used in the image processing, the image can be transformed to an extensive or anti-extensive one. In this paper, area, standard deviation, the moment of inertia, and length of the diagonal are used as the attributes to compute EMAP features for classification tasks. The stack of different EAPs leads to EMAPs, and the detailed information of EMAPs can be found in the report [7]. 2.2

Dictionary Learning for SRC

Suppose fxi gNi¼1 represents N training samples from a L-dimensional hyperspectral dataset, and x belongs to c labelled classes while yi 2 f1. . .:cg is the label of each P c observed pixel xi . For each class c, there exists a matrix DC 2 Rmn N ¼ ci¼1 nc containing nc prototype atoms for columns. Each pixel with a c-th label can be represented approximately as follows: xi  DC rci ;

 c r   K; i

8yi ¼ c

ð3Þ

where rc 2 Rm is the representation coefficient of signal x, k
k0 is a l0 norm which counts the number of nonzero atoms in a coefficient vector, and K is a predefined sparsity constraint. Assuming that the global dictionary D ¼ ½D1 ; D2 . . .Dc Š is known, the corresponding representation coefficients r ¼ ½r1 ; r2 . . .rc Š can be computed by solving the following equation:  ric ¼ arg minxi r

2 Dric 2

s:t: 8i; kri k0  K

ð4Þ

There exist many algorithms to optimize this problem, for example, orthogonal matching pursuit (OMP) [16] is one of the most efficient methods. OMP is a greedy method which simply selects the dictionary prototypes in sequence. The pixels belong to the class which has the minimum class-wise reconstruction error eci , where eci ¼ xi DC rci :  2 ^yi ¼ arg mineci 2 ð5Þ c

The main goal of this paper is to find a dictionary D which can help maximize classification accuracy. In order to minimize the reconstruction error and the classifi 2 cation error simultaneously, a classification error term H WT r which has the 2

sparse code directly as a feature for classification, is included in the objective function.

Classification of Hyperspectral Imagery

361

Following the solution by Jiang et al. [15], the objective function for learning optimal dictionary and sparse code can be redefined as follows:  Dr k22 þ aH

\D; r; W [ ¼ arg min kX D;r;W

s:t:8i

k r i k0  K

2 W T r 2

ð6Þ

where H ¼ ½h1 ; h2 . . .hN Š 2 RcN represents the class labels of the training samples; hi ¼ ½0; 0. . .1. . .0; 0Š 2 Rc , where the nonzero c-th position represents the c-th class that contains xi ; Wr is a linear classification function that supports learning an optimal dictionary; and a is a scalar controlling corresponding terms. In order to use K SVD as the efficient solution, Eq. (6) can be rewritten as the following:   X \D; r; W [ ¼ arg min  pffiffiffi D;r;W  aH s:t:8i

kri k0  K



D pffiffiffi T aW

 2  r  2

ð7Þ

The initial dictionary is obtained by EMAPs; given the initialized D, the original K SVD is employed to obtain r, and r can be used to compute the initial W with linear support vector machine (SVM); both D and W are updated by the K SVD algorithm. pffiffiffi pffiffiffi Let Xnew ¼ ðX T ; aH T ÞT , Dnew ¼ ðDT ; aW ÞT , then Eq. (7) can be rewritten as: \Dnew ; r [ ¼ arg min fkXnew Dnew ;r

s:t:8i;

Dnew r k22 g

kr i k0  K

ð8Þ

Then K SVD algorithm is employed to optimize this problem. Let dk and rk represent the kth row in D and its corresponding coefficients, respectively. The overall processing steps of K SVD is summarized in the report by Ahron et al. [14]. Dnew and r are computed by K SVD, and then D ¼ fd1 ; d2 . . .dk g and r ¼ fr1 ; r2 . . .rk g can be obtained from Dnew . The representation error vector can be computed from eic ¼ xi Dc ric . Additionally, the posterior probability qðyic =xi Þ is inversely proportional to eic [17]: qðyic =xi Þ ¼

1 keic k22 l

ð9Þ

where yic refers to labelled class c for the pixel xi and l is a normalized constant. 2.3

Spatial Information Regularization

We have described the mechanism that is used to obtain the class probability in the previous section. In this section, we will show how to implement the spatial information characterization.

362

Q. Gao et al.

By utilizing the MAP theory, a pixel is likely to have the same label with its neighboring pixels. According to the Hammersley-Clifford theory [18], the MAP estimation of y is represented as follows: X ^y ¼ arg minð logqðyi =xi Þ log qðyÞÞ y

i2NðiÞ r

qðyÞ ¼ e

P

i2NðiÞ

ð10Þ

dðyi yj Þ

where the term qðyi =xi Þ is the spectral energy function and can be estimated from previous Eq. (8). qðyÞ is regularized by a unit impulse function dðyi yj Þ, where dð0Þ ¼ 1 and dðyÞ ¼ 0 for y 6¼ 0; additionally, r is a smoothing parameter. This term attains the probability of one to equal neighboring labels and zero to the other way around. In this way, non-probability might result in a misclassification for some mixing regions. In this paper, we modify this term by a Gaussian radial basis network to make it a more efficient way, and the entire image is decomposed into local patches with a neighborhood size N  N. For j 2 NðiÞ, the function applied on the pixels constrained by the neighborhood size can be represented as: dðyi

yj Þ ¼

(

expð

1;  2  2 uij suij

vij svij

if Þ

if

yi ¼ yj yi 6¼ yj

ð11Þ

We improve the unit impulse function by optimizing the weight of different class probability using a smoothing function. dðyi yj Þ is a function of standardized Euclidian distance between pixels xi and xj , where uij and vij are the horizontal and vertical distances from xi and xj , respectively. suij and svij represent the stand deviation in each direction. The range of Eq. (11) can meet the definition of probability constrained by (0,1]. Given the prior class information and spatial locations of pixels, this improvement can be trained quickly and efficiently. The value 1 is assigned to qðyi ¼ yj Þ which indicates that the pixels tend to appear around those of the same class. Since homogenous areas are dominant, the improved function will yield a good approximation for the regions, especially for the edge area.

Training Samples

Hyperspectral Image

EMAP Features Feature Extraction

SRC Model

Posterior Probability

MAP Segmentation

Spatial Regularization

Fig. 1. Flowchart of the proposed framework

Final Classification MAP

Classification of Hyperspectral Imagery

363

As shown in Eq. (11), the spatial relationship is modelled directly by the spatial locations. In this paper, we utilize the / expansion algorithm to optimize this combinatorial problem. To better understand the main procedures of presented framework, the flowchart is shown as in Fig. 1.

3 Experiment Analysis 3.1

Experimental Setup

Two benchmark hyperspectral images are used to evaluate the proposed method. The attribute values used for EMAPs transformations are described as follows. Area of regions: 5000, length of the diagonal: 100, moment of inertia: 0.5, and standard deviation: 50. Smoothing parameter r is set as 0.2 in this paper. The best parameters are chosen via cross-validation for classifiers in this paper, and the results are compared with those acquired by kernel SVM (KSVM), sparsity representation model using OMP (SRC), Kernel SVM with EMAP features (EKSVM), sparsity representation model using OMP with EMAP features (ERAP), SVM probability with EMAP features and original MRF (SVM_AP), and original MRF with probability obtained with ERAP (referred as ERAP_MRF). All experiments are implemented with Matlab 2015b. Average accuracy (AA), Overall accuracy (OA), kappa coefficient (k) are calculated as the accuracy assessment, which are commonly used for classification tasks. 3.2

Experiments on Indian Pines Data Set

The Indian Pines data set was acquired by Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) in 1992. It covers 145  145 pixels with a spatial resolution of 20 m. After removing 20 water absorption bands, 200 spectral bands from 0.2 to 2.4 µm as the original features. There are 16 labelled classes, which are shown in Table 1, as well as the numbers of training and testing datasets. The average results of ten experiments repeatedly run on different randomly chosen training and testing datasets are shown in Table 2. The classification maps as well as the ground truth are shown in Fig. 2. The listed results in Table 2 and Fig. 2 show that our framework outperforms most techniques and is especially better than KSVM which is known as a state-of-the-art method. The visual results also show that the spatial information involved techniques lead to a much smoother classification map than algorithms with only spectral information involved. One can observe that EMAP with normal classifiers have already provided high classification accuracy, however, the transform of a probabilistic work and including a MAP segmentation work improve the classification accuracies as it can be particularly observed in the ERAP_MRF, SVM_AP and ESRC. This confirms the ability of the work in a probabilistic sense that MAP segmentation can indeed correct the results by

364

Q. Gao et al. Table 1. Class information for Indian Pines image Class No. Class name Train Test 1 Alfalfa 25 29 2 Corn-no till 50 1384 3 Corn-min till 50 784 4 Corn 50 184 5 Grass/trees 50 447 6 Grass/pasture 50 697 7 Grass/pasture-mowed 13 13 8 Hay-windrowed 50 439 9 Oats 10 10 10 Soybeans-no till 50 918 11 Soybeans-min till 50 2418 12 Soybeans-clean till 50 564 13 Wheat 50 162 14 Woods 50 1244 15 Buildings-grass 50 330 16 Stone-steel towers 45 50 Total 693 9673

Table 2. Classification accuracies (%) for Indian Pines image Method 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 OA AA k

SRC 54.17 77.68 78.91 50.48 86.43 94.91 95.65 98.34 73.08 69.55 83.82 72.60 97.17 97.89 60.58 83.04 80.90 79.64 78.44

KSVM 48.57 74.10 73.84 45.43 83.94 93.82 91.67 97.93 72 65.52 80.83 68.90 97.17 97.56 54.92 80.87 77.63 76.69 74.78

ERAP EKSVM SVM_AP ERAP_MRF ESRC 64.63 61.45 70.67 71.23 85.71 85.12 84.60 90.46 90.36 96.22 84.49 83.70 92.17 91.00 95.96 60.33 57.18 68.87 74.74 83.09 90.51 88.76 92.26 94.53 97.41 96.34 96.25 97.92 97.66 98.92 95.83 100.00 96.00 100.00 100.00 98.56 97.96 99.59 98.77 99.79 82.61 79.17 90.48 90.00 90.91 78.23 75.70 84.35 87.33 91.93 89.85 88.81 93.63 93.51 96.85 81.56 78.31 87.31 86.63 93.79 98.12 98.10 98.59 99.05 99.53 98.48 98.80 99.18 99.27 99.45 69.81 68.54 78.35 80.00 89.05 86.24 86.11 90.29 94.00 95.92 86.90 85.71 91.38 91.98 95.76 85.05 83.97 89.38 90.50 94.56 85.17 83.84 90.23 90.90 95.17

Classification of Hyperspectral Imagery

365

Fig. 2. Classification maps of Indian Pines. (a) Ground truth. (b) SRC. (c) KSVM. (d) ERAP. (e) EKSVM. (f) SVM_AP (g) ERAP_MRF. (h) ESRC.

regularizing the spatial information. It also should be noted that our proposed ESRC obtains the best result, which classifies most of regions accurately. As for a SRC based method, the proposed ESRC exhibits the best performance especially in the edge areas, which can be observed from the classification maps. In addition, ESRC has also shown its potential effectiveness in dealing with the small training data sets, which is meaningful for practice applications. ESRC achieves the best result when compared to EKSVM and SVM_AP, which further confirms that our method can learn a discriminative dictionary and implement a more accurate spatial regularization. The results show that the proposed ESRC method is more accurate than original MRF-based spatial regularization methods. Particularly, ESRC performs well on the minority classes (e.g. Class 1 and Class 7). The visual comparison of ERAP_MRF and SVM_AP also confirms the competitive efficiency of optimization model for SRC used in this paper. However, both of them fail to identify Class 9. This is due to the insufficient training samples for this class. The experiments also indicate that the improved method has a potential to obtain a more accurate result with a smaller training set. For ERAP_MRF and ESRC, a window size 8  8 is applied. The former method over-smooths the oat-covered region and misclassifies Oats as Grass/Trees or Corn-min. This is because each oat pixel is dominated by Class 3 (Corn-min) and Class 6 (Grass/Trees). The improvement of spatial regularization gives a weight of pixels far away from the central pixel via spatial locations, which is helpful for the dominated regions. The accuracy varies with different sparsity constraint factor K and neighborhood size N. The effect of K and N on the classification accuracy is shown in Fig. 3. The sparsity constraint factor plays an important role in the experiment, which produces less

366

Q. Gao et al.

sparse codes when set too small, and makes the dictionary no longer sparse and more time-consuming when set too large. The neighborhood size is also an important parameter in the spatial regularization network. As shown in Fig. 3, a too large N may cause over-smoothing and produce noisy information, while a too small N cannot preserve enough spatial features. N is set as 8  8 throughout the experiment to achieve the best accuracy. 100

95

90

Overall Accuracy (%)

85

80

K=5

75

K=10 70

K=15 K=20

65

K=30 60

K=40 K=60

55

K=80 50 3

4

5

6

7

8

9

10

11

12

13

Neighborhood Size N

Fig. 3. The effect of neighborhood size N and sparsity constraint factor K

We apply SRC, ERAP, ERAP_MRF and ESRC with different sparsity level factor K; ranging from K ¼ 5 to K ¼ 80, and the best accuracy is chosen for this experiment. The parameter a which controls the contribution of the classification error is determined by cross validation experiments on training images. For Indian Pines data set, it is set to 0.001. This small value for a is due to the high dimension sparse vector of the training samples. The normalized large scale sparse vector results in small Table 3. Class information for Pavia University image Class No. Class Name Train Test 1 Asphalt 50 6631 2 Meadows 50 18649 3 Gravel 50 2099 4 Trees 50 3064 5 Meta sheets 50 1345 6 Bare soil 50 5029 7 Bitumen 50 1330 8 Bricks 50 3682 9 Shadows 50 947 Total 450 42776

Classification of Hyperspectral Imagery

367

Table 4. Classification accuracies (%) for Pavia University image Method 1 2 3 4 5 6 7 8 9 OA AA k

SRC 92.76 97.06 71.93 51.47 91.64 56.74 75.62 83.58 99.89 79.32 80.08 74.20

KSVM 92.24 96.79 70.74 50.05 91.52 55.24 74.54 82.59 99.89 78.14 79.29 72.83

ERAP 94.42 98.09 79.57 59.82 94.31 64.90 81.42 88.32 99.89 84.99 84.53 80.97

EKSVM 95.95 98.74 85.31 66.28 95.79 71.92 86.15 91.29 99.89 88.87 87.93 85.74

SVM_AP ERAP_MRF ESRC 96.60 97.70 97.86 98.89 99.21 99.53 86.89 90.56 93.00 69.76 75.67 81.29 94.84 97.18 98.61 74.32 80.09 85.59 87.71 89.82 93.22 92.89 94.25 96.32 99.89 100.00 100.00 90.27 92.84 94.98 89.09 91.61 93.94 87.47 90.71 93.44

component values for the extracted features, therefore the weight of this controlling term is lower, compared to the sparse reconstruction errors. 3.3

Experiments on ROSIS Pavia University Data Set, Italy

The image was collected by the Reflective Optics Systems Imaging spectrometer (ROSIS), and the sensor generates 115 spectral bands covering from 0.43 to 0.86 µm with a spatial resolution of 1.3 m. In our experiments, 103 bands are used with 12 noisy

Fig. 4. Classification maps of Pavia University. (a) Ground truth. (b) SRC. (c) KSVM. (d) ERAP. (e) EKSVM. (f) SVM_AP (g) ERAP_MRF. (h) ESRC

368

Q. Gao et al.

bands removed from both data sets. It consists of 610  340 pixels with 9 labelled classes. As discussed above, the sparsity constraint factor T is set to range from 5 to 80 for SRC, ERAP, ERAP_MRF and ESRC. N is set as a 9  9 and a is set as 0.005 for this dataset. Table 3 shows the class information of this data set. The classification accuracies and classification maps are summarized in Table 4 and Fig. 4. The classification results for this imagery are consistent with Indian Pines imagery. Our proposed method (i.e. ESRC) performs better than the other methods in most cases.

4 Conclusion In this paper, a novel framework is proposed for HSI classification. This framework is based on EMAP and SRC. HSI pixels are considered as sparse representation by the atoms in a selected dictionary. In the proposed algorithm, a classification error term is added to the SRC model and is solved by the K-SVD algorithm. To improve the classification accuracy, we also have taken into account the influence of neighboring pixels of the pixel of interest in the MAP spatial regularization model. Experiments conducted on two different hyperspectral images show that the proposed method yields high classification accuracy, which is especially better than the state-of-the-art SVM classifiers.

References 1. Chang, C.-I.: Hyperspectral Data Exploitation: Theory and Applications. Wiley, New York (2007) 2. Camps-Valls, G., et al.: Advances in hyperspectral image classification: earth monitoring with statistical learning methods. IEEE Signal Process. Mag. 31(1), 45–54 (2014) 3. Liang, J., et al.: On the sampling strategy for evaluation of spectral-spatial methods in hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 55(2), 862–880 (2017) 4. Aghighi, H., et al.: Dynamic block-based parameter estimation for MRF classification of high-resolution images. IEEE Geosci. Remote Sens. Lett. 11(10), 1687–1691 (2014) 5. Dalla Mura, M., et al.: Extended profiles with morphological attribute filters for the analysis of hyperspectral data. Int. J. Remote Sens. 31(22), 5975–5991 (2010) 6. Song, B., et al.: Remotely sensed image classification using sparse representations of morphological attribute profiles. IEEE Trans. Geosci. Remote Sens. 52(8), 5122–5136 (2014) 7. Dalla Mura, M., et al.: Classification of hyperspectral images by using extended morphological attribute profiles and independent component analysis. IEEE Geosci. Remote Sens. Lett. 8(3), 542–546 (2011) 8. Wright, J., et al.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 210–227 (2009) 9. Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2008) 10. Wu, Z., et al.: Exemplar-based sparse representation with residual compensation for voice conversion. IEEE/ACM Trans. Audio Speech Lang. Process. 22(10), 1506–1521 (2014)

Classification of Hyperspectral Imagery

369

11. Mallat, S.: A Wavelet Tour of Signal Processing: The Sparse Way. Academic Press (2008) 12. Wright, J., et al.: Sparse representation for computer vision and pattern recognition. Proc. IEEE 98(6), 1031–1044 (2010) 13. Ye, M., et al.: Dictionary learning-based feature-level domain adaptation for cross-scene hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 55(3), 1544–1562 (2017) 14. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006) 15. Jiang, Z., Lin, Z., Davis, L.S.: Learning a discriminative dictionary for sparse coding via label consistent K-SVD. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1697–1704 (2011) 16. Chen, S., Billings, S.A., Luo, W.: Orthogonal least squares methods and their application to non-linear system identification. Int. J. Control 50(5), 1873–1896 (1989) 17. Li, J., Zhang, H., Zhang, L.: Supervised segmentation of very high resolution images by the use of extended morphological attribute profiles and a sparse transform. IEEE Geosci. Remote Sens. Lett. 11(8), 1409–1413 (2014) 18. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)

Deep Residual Convolutional Neural Network for Hyperspectral Image Super-Resolution Chen Wang1 , Yun Liu2 , Xiao Bai1(B) , Wenzhong Tang1 , Peng Lei3 , and Jun Zhou4 1

School of Computer Science and Engineering, Beihang University, Beijing, China [email protected] 2 School of Automation Science and Electrical Engineering, Beihang University, Beijing, China 3 School of Electronic and Information Engineering, Beihang University, Beijing, China 4 School of Information and Communication Technology, Griffith University, Nathan, Australia

Abstract. Hyperspectral image is very useful for many computer vision tasks, however it is often difficult to obtain high-resolution hyperspectral images using existing hyperspectral imaging techniques. In this paper, we propose a deep residual convolutional neural network to increase the spatial resolution of hyperspectral image. Our network consists of 18 convolution layers and requires only one low-resolution hyperspectral image as input. The super-resolution is achieved by minimizing the difference between the estimated image and the ground truth high resolution image. Besides the mean square error between these two images, we introduce a loss function which calculates the angle between the estimated spectrum vector and the ground truth one to maintain the correctness of spectral reconstruction. In experiments on two public datasets we show that the proposed network delivers improved hyperspectral super-resolution result than several state-of-the-art methods. Keywords: Hyperspectral image super-resolution Deep residual convolutional neural network

1

Introduction

Hyperspectral imaging acquires spectral representation of a scene through capturing a large number of continuous and narrow spectral bands. The spectral characteristics of the hyperspectral image have been proven useful for many visual tasks, including tracking [15], segmentation [19], face recognition [16] and document analysis [10]. However, in each narrow band only a small fraction of the overall radiant energy reaches the sensor. To maintain a good signal-to-noise ratio, the imaging system increases the pixel size on the chip and uses long exposures, which however, results in low spatial resolution of hyperspectral images. c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 370–380, 2017. https://doi.org/10.1007/978-3-319-71598-8_33

Deep Residual Convolutional Neural Network

371

Recently, several matrix factorization based approaches [1, 7, 9, 13, 14] have been proposed for hyperspectral image super-resolution. All these methods need auxiliary data of RGB image of the same scene, and these methods cannot be directly applied to hyperspectral images which are normally obtained beyond the visible range. Convolutional neural networks have recently been intensively explored due to their powerful learning capability. Motivated by this property, we propose a deep convolutional neural network method which learns an end-to-end mapping between low- and high-resolution images. This network does not need any high resolution RGB image to provide additional information. Furthermore, Hyperspectral image is a data-cube and has obvious physical meaning in spectral dimension. Traditional deep networks were developed for super-resolution of grayscale images, therefore, can not be directly applied to hyperspectral image. To address this problem, we introduce a loss function which calculates the angle between the estimated spectrum vector and the ground truth one to maintain the correctness of spectral reconstruction. When the network comes deeper, the vanishing gradients problem are significantly critical, so we use residual-learning and additional supervised output to solve this problem.

2

Related Work

In this section, we review relevant hyperspectral image super-resolution methods and deep learning methods for grayscale image super-resolution. 2.1

Hyperspectral Image Super-Resolution

In early years, Pan-sharpening techniques [2, 17] were introduced to merge a high resolution panchromatic (single band) image and a low resolution hyperspectral image to reconstruct a high resolution hyperspectral image. In addition, filtering techniques [8, 12] were proposed which used high resolution edges from other images of the same scene to guide filtering process. These methods indeed improve the spatial resolution of hyperspectral images, but the reconstructed high resolution images sometimes contain spectral distortions. More recently, matrix factorization based techniques for hyperspectral image super-resolution have been proposed. Kawakami et al. [9] used matrix factorization to firstly learn a series of spectral bases. Then the sparse coefficients of these spectral bases were calculated, which best reconstructed the corresponding high resolution RGB signals. At last, they used these coefficients and the spectral bases to reconstruct the high resolution hyperspectral image. This work was extended by Akhtar et al. [1] who imposed a non-negativity constraint over the solution space. Kwon et al. [13] upsampled the image guided by high resolution RGB images of the same scene and then utilized sparse coding to locally refine the upsampled hyperspectral image through dictionary substitution. Lanaras et al. [14] proposed a method to perform hyperspectral super-resolution by jointly unmixing the RGB and hyperspectral images into the pure reflectance spectra

372

C. Wang et al.

of the observed materials and the associated mixing coefficients. To improve the accuracy of non-negative sparse coding, a clustering-based structured sparse coding method [7] was introduced to exploit the spatial correlation among the learned sparse codes. All these matrix factorization based methods need high resolution RGB images to provide extra information. 2.2

Deep Learning Methods on Grayscale Images

During the past three years, deep learning methods [4–6, 11, 18, 20] have been used for single-band grayscale image super-resolution and demonstrated great success. To the best of our knowledge, however, deep learning has not been introduced for hyperspectral image super-resolution. A hyperspectral image is not simply a concatenation of several single-band images because of its obvious physical meaning in the spectral dimension. So it is necessary to develop suitable networks for hyperspectral image.

Fig. 1. The structure of the proposed deep residual convolutional neural network.

3

Deep Residual Convolutional Neural Network

In this section, we describe the proposed deep residual convolutional neural network and the new loss function to cope with spectral dimension of the image data. 3.1

Deep Residual Convolutional Neural Network

We propose a deep residual convolutional neural network to increase the resolution of hyperspectral image. The structure of this network is outlined in Fig. 1. We cascade 6 sub-networks which have the same structure. Each sub-network

Deep Residual Convolutional Neural Network

373

is a residual convolutional neural network. The input of the sub-network is lowresolution hyperspectral image and the output is high-resolution hyperspectral image. Meanwhile, the output of each sub-network is then regarded as the input of the next sub-network. Each sub-network has three types of filter: c×f1 ×f1 ×n1 for the first layer, n1 × f2 × f2 × n2 for the second layer, and n2 × f3 × f3 × c for the last layer. In this paper, we set f1 = 9, f2 = 3, f3 = 5, n1 = 96, n2 = 64, where f∗ means the size of the convolutional kernels and n∗ means the number of feature maps, c means the number of original image channels. The first layer takes the input image and represents it as a set of feature maps (n1 channels). The second layer is used to dig deeper features. At last, the third layer is used to transform the features (n2 channels) back into the original image space (c channels). When the network comes deeper, the vanishing gradients problem can be critical. Inspired by [11], we use residual-learning to solve this problem. As shown in Fig. 1, we get the residual image after the third layer. The final output of the sub-network is the sum of residual image and the input of this sub-network. It is worth mentioning that our network is a progressive structure which gradually learns the residual components. Back propagation goes through a small number of layers if there is only one supervising signal at the end of the network. So we add supervised output at the end of the third sub-networks to make sure the weights of the first three sub-networks can be updated efficiently. The optimal weights are learned by automatically minimizing the loss function of both supervised output and the final output. We will define the loss function in the next subsection. We have not used any pooling layers or deconvolution layers. The whole network takes interpolated low-resolution image (to the size of high resolution image) as input and predicts the missed high frequency parts. Although there is no pooling layer, the size of the feature map gets reduced every time the convolution operations are applied. So we use zero-padding before each convolution operation to make sure that all feature maps have the same size. 3.2

Loss Function

We now describe the loss function of our network. Given a training dataset (i) or y (i) means it is the ith image in the dataset and N is {x(i) , y (i) }N i , where x the total number of images, our goal is to learn a model f that predicts values yˆ = f (x), where x is a low-resolution hyperspectral image (after interpolation), y is the target high-resolution hyperspectral image, Mean squared error 21 y − yˆ2 is widely used in least-squares regression setting. This favors high Peak Signal-to-Noise Ratio (PSNR) by minimizing the first loss function which is defined as N 2 1    (i) (1) l1 = y − yˆ(i)  2N i Considered that reflectance spectrum is the most important information in a hyperspectral image, we add a new loss function which calculates the angle

374

C. Wang et al.

between the estimated spectrum vector and the ground truth one. Let Nm be the number of spectrum vector of hyperspectral image, yj be the j th spectral vector. We have the second loss function (i) N Nm yj , yˆj (i)  1      l2 =  (i)  (i)  N Nm i j  y  × yˆ  j

(2)

j

The final loss function is the linear combination of these two loss functions L = αl1 + (1 − α)l2

(3)

where α = 0.5 in our work. We implement our model using the TensorFlow framework. Training is carried out by minimizing the loss function using mini-batch gradient descent based on back-propagation. After training, low-resolution hyperspectral image is used as the input to test out network, and we record the super-resolution results for comparison purpose.

4

Experiment

We used two publicly available hyperspectral datasets: CAVE [21] and Harvard [3] in the experiments. The first dataset includes 32 indoor images. The spatial resolution of the images is 512 × 512. Each image has 31 spectral bands with 10 nm spectral resolution, ranging from 400 nm to 700 nm. The second dataset has 50 indoor and outdoor images recorded under daylight illumination and 27 images under artificial or mixed illumination. The spatial dimension of the images is 1392 × 1040 pixels, with 31 spectral bands covering the visible spectrum from 420 nm to 720 nm at 10 nm spectral resolution. For convenience, we used only the top left 1024 × 1024 pixels of each image to make the spatial dimension of the ground truth a multiple of 32. Figure 2 shows some representative images from these CAVE datasets. Figure 3 shows some representatives image from Harvard datasets. 4.1

Implementation Detail

In our experiments, the original images served as the ground truth. To obtain low-resolution hyperspectral images, we blurred the original images using a Gaussian kernel (standard deviation = 3), downsampled it by a scale factor (= 2,3,4) and then upscaled it to the desired size using bicubic interpolation with a scale factor (= 2,3,4). These images have the same size with ground truth but lose the high frequency components, so we still call them low-resolution images. The testing set included 7 images from the CAVE dataset (Balloons, Chart and stuffed toy, Faces, Flowers, Jelly beans, Real and fake apples, and Oil painting) and 10 images from the Harvard dataset (Img1, Img2, Img3, Img4, Img5,

Deep Residual Convolutional Neural Network

375

Fig. 2. Selected examples of hyperspectral image from CAVE database.

Img6, Imga1, Imga2, Imga3 and Imga4). The rest 92 images were used for training. For testing, we used the whole interpolated low-resolution images as the input and compared the output with the corresponding ground truth. For training, we used 32 × 32 × 31 cubic-patches randomly cropped from the training images. In total, 30,000 cubic-patches were generated from the training set. We cascaded 6 sub-networks and added supervised output to the end of the third one. In our experiments, adding more layers does not bring obvious improvement to the result. So we finally used 18 convolution layers. After each convolution layer, we used ReLU as the nonlinear mapping function. For training,

376

C. Wang et al.

Fig. 3. Selected examples of hyperspectral image from Harvard database.

we used mini-batch gradient descent based on back propagation and the batch size was set to 128. We set the momentum parameter to 0.9 and the weight decay to 0.0001. Learning rate was initially set to 0.01 and then decreased by a factor of 10 if the validation error did not decrease for 3 epochs. If the learning rate was less than 10−6 , the procedure was terminated. We trained three independent networks for factor = 2, 3, 4. It took 2 days to train one network on a GTX 980 Ti, but it took only 4.2 s to process a testing hyperspectral image on average. 4.2

Experimental Results

All hyperspectral image super-resolution methods reviewed in Sect. 2 used extra high resolution image to help the estimation process, so it is unfair to directly compare our method with these methods. We used another comparing strategy. Firstly, we set bicubic interpolation method as the baseline to evaluate the learning ability of our network. Then we compared our method with three singleband image super-resolution neural networks [4, 11, 18] to show that our network Table 1. Results on the CAVE dataset Method

CAVE Database PSNR

SAM

Scale

x2

x2

Bicubic

31.73 31.58 30.14 6.00 5.99 6.36

x3

x4

x3

x4

Dong [4] 36.28 35.10 34.67 4.23 4.20 4.52 Shi [18]

36.65 35.39 34.91 4.12 4.16 4.43

Kim [11] 36.98 35.87 35.02 4.35 4.21 4.39 Ours

38.24 37.86 37.14 1.56 1.73 1.85

Deep Residual Convolutional Neural Network

377

is more suitable for hyperspectral images. We considered hyperspectral images as a series of independent gray-level images and used these images to train and test the neural networks. To be fair, all methods used the same training set and we followed the network settings in their original paper for each method being compared. Table 2. Results on the Harvard dataset Method

Harvard Database PSNR SAM

Scale

x2

Bicubic

36.67 36.52 36.39 3.09 3.10 3.17

x3

x4

x2

x3

x4

Dong [4] 38.98 38.41 38.10 2.57 2.61 2.74 Shi [18]

39.35 38.63 38.29 2.55 2.57 2.61

Kim [11] 39.54 39.02 38.59 2.46 2.49 2.53 Ours

40.13 39.68 39.14 1.14 1.21 1.35

In this work, we used Peak Signal-to-Noise Ratio (PSNR) and spectral angle mapper (SAM) [22] as the evaluation measurements. Peak Signal-to-Noise Ratio

Fig. 4. Spectral images at 550 nm for sample CAVE [21] data. Estimated high resolution images are shown along with their absolute difference with the ground truth. The scale factor is 3.

378

C. Wang et al.

Fig. 5. Spectral images at 550 nm for sample Harvard [3] data. Estimated high resolution images are shown along with their absolute difference with the ground truth. The scale factor is 3.

(PSNR) is used as the primary evaluation measure for the estimated highresolution hyperspectral image Zˆ and the ground truth image Z in an n-bit intensity range. Let B be the number of bands of hyperspectral image and Nm be the number of spectral vector of hyperspectral image, PSNR is calculated by ⎞ ⎛ ⎜ P SN R = 10 × log10 ⎝

2n − 1  2 ⎠ .   ˆ 1 BNm Z − Z 

(4)

F

To evaluate the correctness of spectral responses, we used the spectral angle mapper (SAM) [22], which is defined as the angle between the estimated spectrum vector zˆi and the ground truth spectrum vector zj , averaged over the whole image. The SAM is given in degrees SAM =

zˆjT zj 1  arccos . Nm ˆ zj 2 zj 2

(5)

Tables 1 and 2 show the average PSNR and SAM values on the two datasets. Our approach achieves higher PSNR than all four methods. Notably, in terms of SAM, our method is clearly the best among all the methods. Considering

Deep Residual Convolutional Neural Network

379

hyperspectral image as a series of gray-level images instead of a whole data-cube always leads to spectral distortion. The loss function proposed in this paper solves this problem well. To complement the tabulated results, we also visualize the experimental results in Figs. 4 and 5. Due to space limitation, we only show one spectral image at 550 nm. The fourth column of Figs. 4 and 5 shows the absolute difference between the estimated high resolution hyperspectral image and the ground truth. It shows that with our method, a significantly larger number of pixels have very small reconstruction errors below 1 in grayscale value.

5

Conclusion

We have introduced a deep residual convolutional neural network for hyperspectral image super-resolution. The input to this network is a single low-resolution hyperspectral image and no extra RGB or other high-resolution images are needed. A new loss function is used to make the framework more suitable for hyperspectral image. Residual-learning and additional supervised output are used to solve the vanishing gradients problem. Experimental results show that the proposed method performs well under both PSNR and SAM measurements. Acknowledgements. This work is supported by NSFC project No. 61370123 and BNSF project No. 4162037. It is also supported by funding from State Key Lab of Software Development Environment in Beihang University.

References 1. Akhtar, N., Shafait, F., Mian, A.: Sparse spatio-spectral representation for hyperspectral image super-resolution. In: European Conference on Computer Vision. pp. 63–78 (2014) 2. Laben, C.A., Brower, B.V.: Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. Websterny Uspenfieldny US (2000) 3. Chakrabarti, A., Zickler, T.: Statistics of real-world hyperspectral images. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 193– 200 (2011) 4. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi. org/10.1007/978-3-319-10593-2 13 5. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016) 6. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/ 978-3-319-46475-6 25 7. Dong, W., Fu, F., Shi, G., Cao, X., Wu, J., Li, G., Li, X.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25(5), 2337–2352 (2016)

380

C. Wang et al.

8. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013) 9. Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y.W., Ikeuchi, K.: High-resolution hyperspectral imaging via matrix factorization. In: CVPR 2011, pp. 2329–2336 (2011) 10. Khan, Z., Shafait, F., Mian, A.: Hyperspectral imaging for ink mismatch detection. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 877–881 (2013) 11. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1637–1645 (2016) 12. Kopfand, J., Cohen, M., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. ACM Trans. Graph. 26(3), 96 (2007) 13. Kwon, H., Tai, Y.W.: RGB-guided hyperspectral image upsampling. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 307–315 (2015) 14. Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3586–3594 (2015) 15. Nguyen, H.V., Banerjee, A., Chellappa, R.: Tracking via object reflectance using a hyperspectral video camera. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 44–51 (2010) 16. Pan, Z.H., Healey, G., Prasad, M., Tromberg, B.: Face recognition in hyperspectral images. IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1552–1560 (2003) 17. Shah, V.P., Younan, N.H., King, R.L.: An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Trans. Geosci. Remote Sens. 46(5), 1323–1335 (2008) 18. Shi, W., Caballero, J., Huszr, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874–1883 (2016) 19. Tarabalka, Y., Chanussot, J., Benediktsson, J.A.: Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 40(5), 1267– 1279 (2010) 20. Wang, Z., Liu, D., Yang, J., Han, W., Huang, T.: Deep networks for image superresolution with sparse prior. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 370–378 (2015) 21. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. 19(9), 2241–2253 (2010) 22. Yuhas, R.H., Boardman, J.W., Goetz, A.F.H.: Determination of semi-arid landscape endmembers and seasonal trends using convex geometry spectral unmixing techniques (1993)

Multi-view and Stereoscopic Processing

Stereoscopic Digital Camouflage Pattern Generation Algorithm Based on Color Image Segmentation Qin Lei1(&), Wei-dong Xu2, Jiang-hua Hu2, and Chun-yu Xu3 1

2

Xi’an Communications Institute, Xi’an 710106, China [email protected] PLA University of Science and Technology, Nanjing 210007, China 3 Nanjing Artillery Academy, Nanjing 211132, China

Abstract. Pattern painting camouflage is an important method to improve the survivability of the military target. However, the existing flat pattern painting is not enough to confront the stereophotograph reconnaissance. So the stereoscopic digital pattern painting which can distort the 3D appearance of the target has been researched in this paper. The 3D models of the stereoscopic digital camouflage pattern were introduced. The design principles of the parallax in the pattern was analyzed and used in designing the sequence images producing algorithm. The result shows that the stereoscopic camouflage pattern can distort the appearance of the target in three dimensions. Keywords: Stereoscopic processing Digital camouflage pattern


Image processing

1 Introduction Camouflage pattern is an important kind of technique to improve the survivability of the military target [1]. With the rapid improvement of the aerospace stereoscopic imaging reconnaissance and the robot binocular imaging, the battle field imaging reconnaissance is extending from two-dimension to three-dimension. The existing camouflage patterns are two dimensional images. So they can hardly confront the stereoscopic imaging reconnaissance. To design stereoscopic camouflage pattern, using the modern stereoscopic image based on lenticular raster and combined with the digital camouflage pattern is a new technical path. The model of the stereoscopic camouflage pattern is shown in Fig. 1.

2 Parallax Design Principles of Stereoscopic Camouflage Pattern Parallax design is to determine the location of the pixels in each serial image. Figure 2 shows the sketch map of parallax design principles.

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 383–389, 2017. https://doi.org/10.1007/978-3-319-71598-8_34

384

Q. Lei et al.

Fig. 1. Model of stereoscopic camouflage

xyz is the world coordinate system, x0 y0 z0 is the coordinate system in which the number 0 image of the serial images projects through the lenticular raster. The two coordinate systems are coincident. x1 y1 z1 is the coordinate system in which the number 1 image of the serial images projected through the lenticular raster with an angle h. Point O is the original point of all the coordinates and the angle between z1 and z0 is h. Supposed that coordinate of whatever point P in the world coordinate system is ðx0 ; y0 ; z0 Þ and the coordinate of it in the x1 y1 z1 is ðx1 ; y1 ; z1 Þ, geometric relations is: 3 2 cos h x1 4 y1 5 ¼ 4 0 sin h z1 2

Fig. 2. Parallax design principles

0 1 0

32 3 sin h x0 0 54 y0 5 cos h z0

ð1Þ

There are two points Pðx; y; zÞ and P0 ðx; y; z þ HÞ. They have the same coordinates in the first two dimensions and have different coordinate in the third dimension. The difference of their depths is H. Define their coordinates in x1 y1 z1 are P1 ðx1 ; y1 ; z1 Þ and P01 ðx01 ; y01 ; z01 Þ. According to Eq. (1) we can get that: 3 2 cos h x1 4 y1 5 ¼ 4 0 z1 sin h 2

0 1 0

32 3 sin h x 0 54 y 5 cos h z

ð2Þ

Stereoscopic Digital Camouflage Pattern Generation Algorithm

3 2 x01 cos h 4 y0 5 ¼ 4 0 1 z01 sin h 2

0 1 0

32 3 sin h x 0 54 y 5 cos h zþH

385

ð3Þ

According to Eqs. (2) and (3), we can find the horizontal location of the two points. x1 ¼ cos h
x x01 ¼ cos h
x

sin h
z

sin h
z

sin h
H

ð4Þ ð5Þ

Two points Pðx; y; zÞ and P0 ðx; y; z þ HÞ in the world space have the same coordinates in the x axis and y axis and have the depth difference H in the z axis. The parallax of them in image x1 y1 plane is: Dx ¼ x

x01 ¼ sin h
H

ð6Þ

Equation (6) shows the relations of different spot layers of the stereoscopic camouflage pattern in one serial image.

3 Stereoscopic Digital Camouflage Pattern Generation Algorithm The serial images generation algorithm is the key technique of the stereoscopic camouflage pattern. The process of the algorithm is shown in Fig. 3. According to the functions of the steps, the algorithm can be divided into 3 modules. They are Layer Segmentation Module (1), Parallax Adjust Module (2) and Layer Merging Module (3). 3.1

Layer Segmentation Module

The function of the Layer Segmentation Module is to read the original camouflage pattern, split the spots into different layers based on colors, and allocate the layers into different depths. It includes the following steps. Fig. 3. Process of serial image generation algorithm

386

Q. Lei et al.

Step 1, read the original pattern and turn it into indexed image, save the color map and the color index matrix. Step 2, set the parameter of the stereoscopic camouflage pattern. Set the location of the number 0 layer (The layer in the middle depth of the stereoscopic pattern). Set the minimum parallax (The minimum parallax is, when the depth between the neighboring layers is H, the parallax between the neighboring layers in the neighboring serial images). The minimum parallax between the serial images Dxmin ¼ sin h
H
I=2:54 (pixels). Step 3, segment and allocate the spots in different depths [4]. Adjust the color index matrix and the color map, to match the color index number with the depth order of the layers and adjust the location of the colors in the color map at the same time. In the default condition, the order of the layers is in accordance with the brightness of the color. Extract the spots of the camouflage pattern, according to the color the spots, and save them in different layers as shown Fig. 4. Then fill all the pixels of the last layer with the same color of the spots in order to avoid the blank pixel the in the process of parallax adjustment.

Fig. 4. Results of spots segmentation

3.2

Parallax Adjustment Module

The parallax adjustment module adjusts the parallax of the spots in serial images according to the depth order of the layers. It includes the following steps: Step 1, set the serial number i of the serial image that being processing. Serial number is used to mark the order of angle when the lenticular raster is imaging, the sum total of the serial images is n.

Stereoscopic Digital Camouflage Pattern Generation Algorithm

387

Step 2, generate the depth index matrix according to the color index matrix. Change the color index matrix into depth index matrix by adding the color index number of the number 0 layer to the color index matrix. Step 3, set the layer number j of the layer that being processing. The sum total of the layers m is the same as the sum total of colors in the camouflage pattern. Step 4, calculate the parallax of the layer Dx in the new serial image, According to the serial image number i and the layer number j. The parallax of the layer Dx can be expressed as: Dx ¼ ði n=2Þ
ðj mÞ
Dxmin Step 5, shift the layer according to the parallax Dx. Then judge if the layers of different depth have being finished or not. If not, reset the layer number to process the next layer. If yes, then merge the layers. 3.3

Layer Merging Module

The layer merging module is used to merge the image layers according to the sheltery relation between each layer and generate the stereoscopic camouflage pattern serial image. The layer merging module includes the following steps: Step 1, merge the depth number. Merge the depth matrix and use the layer of bigger index number cover the layer of smaller index number. Step 2, turn the depth index number into color index number. According to the location of the number 0 layer, the depth index matrix minus the number of the color index number of the number 0 layer. Step 3, save serial image as true color image [6], according to the color index matrix and color map. Judge the serial image number is finished or not. If not, reset the serial image number and generate the next serial image. If yes, end the algorithm. The generated serial image samples of stereoscopic camouflage pattern are shown in Fig. 5.

Fig. 5. The generated serial image samples of stereoscopic camouflage pattern

4 Sample Analysis and Measurement The stereoscopic camouflage pattern samples have been made, by coding the serial images and printing the coded image, then pasting a lenticular raster on it [7]. The parameters of the samples were shown in Table 1.

388

Q. Lei et al.

The experimental scheme was shown in Fig. 6 and the devices was shown in Fig. 7. Observers look at the stereoscopic camouflage sample with naked eye at a distance of 2 m and adjust the locations both of the front board and the back board. When the observer gets the same depth feeling of the boards and the stereoscopic camouflage pattern, then measures the distance between the front and back board. This distance represents the depth between the camouflage layers.

Table 1. Parameters of stereoscopic camouflage pattern samples Sample number Width of lenticular/mm Thickness of lenticular/mm Line number of raster/l/dpi Minimum parallax/mm Number of layers Theoretical depth/cm

1 1.69 3.08 15 0.05 5 4.968

Fig. 6. The experimental scheme

2 1.69 3.08 15 0.1 5 9.935

3 1.02 4.22 25 0.1 5 21.6861

4 0.79 3.26 32 0.1 5 21.449

5 0.51 2.12 50 0.1 5 21.787

6 1.69 3.08 15 0.05 6 5.961

Fig. 7. The experimental devices

Three observers measured the depth of the stereoscopic camouflage pattern in the illumination condition of 300–550lx. The results of the stereoscopic camouflage pattern depth measurement were shown in Table 2. By comparing Tables 1 and 2, it shows that the results of the stereoscopic camouflage pattern measurement fit the theoretical depth very well. Combined the parameters in Table 1, by comparing sample 1 and sample 2, it shows that when the other parameters are the same, the depth between the first and the last layer of the stereoscopic camouflage pattern grows bigger with the growing of the parallax between the serial images. By comparing sample 1 and sample 6, it shows that the parameters of the lenticular raster and the parallax of them are the same, the depth between the first and the

Stereoscopic Digital Camouflage Pattern Generation Algorithm

389

last layer of grows bigger with the growing of the number of the layers in the stereoscopic camouflage pattern. The results fit the Parallax Design Principles very well.

Table 2. The results of stereoscopic camouflage pattern depth measurement Sample number Observer 1/cm Observer 2/cm Observer 3/cm Average depth/cm

1 5.18 4.78 6.23 5.397

2 10.82 10.25 11.11 10.727

3 20.10 19.85 21.22 20.390

4 19.41 19.53 22.02 20.320

5 18.16 20.01 22.17 20.113

6 5.14 5.33 6.07 5.513

5 Conclusion Stereoscopic digital camouflage pattern is a new kind of camouflage pattern which combined the digital camouflage pattern and the stereoscopic display. The Stereoscopic digital camouflage pattern generation algorithm can be divided into 3 modules. They are Layer Segmentation Module, Parallax Adjust Module and Layer Merging Module. The function of the Layer Segmentation Module is to split the spots into different layers based on colors, and allocate the layers into different depths. The parallax adjustment module adjusts the parallax of the spots in serial images according to the depth order of the layers. The layer merging module is used to merge the layers according to the sheltery relation between each layer and generate the stereoscopic camouflage pattern serial image. The results of the sample depth measurement show that the stereoscopic digital camouflage pattern can produce three-dimensional visual effects. With the action of the layers in different depth, the stereoscopic digital camouflage pattern can disrupt the 3D appearance of the military target.

References 1. Birkemark, C.M.: CAMEVA: a methodology for estimation of target detectability. Opitcal Eng. 40(9), 1835–1843 (2001) 2. Zhang, J.C.: Pattern Painting Camouflage Technology, pp. 47–128. China Textile Press, Beijing (2002) 3. Jianghua, H.: Camouflage Technology, pp. 15–17. National Defense Industry Press, Beijing (2012) 4. Geng, G.H., Zhou, M.-Q.: Analyzing the quality of some common used algorithms of color quantization. Mini Micro Syst. 25(11) (2004) 5. Yu, J., Wang, C., Hu, Z.-Y.: Design on digital disruptive pattern for fixed targets. Comput. Digital Eng. 4, 134–137 (2011) 6. Gonzalez, R.C.: Digital Image Processing, pp. 106–108. Publishing House of Electronic Industry, Peking (2007) 7. Tian, X.: Modern Stereoscopic Printing, pp. 26–27. Wuhan University Publishing House, Wuhan (2007)

Uncertain Region Identification for Stereoscopic Foreground Cutout Taotao Yang, Shuaicheng Liu(B) , Chao Sun, Zhengning Wang, and Bing Zeng University of Electronic Science and Technology of China, Chengdu, China [email protected]

Abstract. This paper presents a method that automatically segments the foreground objects for stereoscopic images. Given a stereo pair, a disparity map can be estimated, which encodes the depth information. Objects that stay close to the camera are considered as foreground while regions with larger depths are deemed as background. Although the raw disparity map is usually noisy, incomplete, and inaccurate, it facilitates an automatic generation of trimaps for both views, where the images are partitioned into three regions: definite foreground, definite background, and uncertain region. Our job is now reduced to labelling of pixels in the uncertain region, in which the number of unknown pixels has been decreased largely. We propose to use an MRF based energy minimization for labelling the unknown pixels, which involves both local and global color probabilities within and across views. Results are evaluated by objective metrics on a ground truth stereo segmentation dataset, which validates the effectiveness of our proposed method. Keywords: Image segmentation

1

· Stereoscopic image · Trimap

Introduction

Recent advances in stereoscopic cameras and displays have facilitated the development of stereoscopic techniques, and some of them have been explored in the research community, including stereoscopic cloning [1,2], warping [3,4], panorama [5], and retargeting [6]. Image segmentation [7, 8], dating back to a couple of decades, has always been a popular field due to its wide application range. In this work, we study the problem of stereoscopic image segmentation [9]. Given two stereo images as input, a raw disparity map between two images can be estimated [10]. The disparity map encodes the depth information. By observations, foreground objects usually stay close to the camera while the background stays relatively far away. Although the raw disparity map is usually noisy, Electronic supplementary material The online version of this chapter (https:// doi.org/10.1007/978-3-319-71598-8 35) contains supplementary material, which is available to authorized users. c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 390–399, 2017. https://doi.org/10.1007/978-3-319-71598-8_35

Uncertain Region Identification for Stereoscopic Foreground Cutout

391

incomplete, and inaccurate, it facilitates an automatic generation of a trimap [11] for both views, where the images are partitioned into three regions: foreground, background, and uncertain region. With this trimap, we only need to focus on pixels in the uncertain region, where the number of unknown pixel labels has been decreased largely, thus possibly leading to an improvement in terms of the efficiency as well as accuracy. In this work, we propose that the unknown pixel labels are decided through an MRF based energy minimization, which takes into consideration both local and global color probabilities within and across views. Compared with traditional monocular image segmentation methods, segmentation based on a stereo pair not only produces more accurate results but also considers the view consistency. Previous works often require an accurate dense disparity estimation for the joint segmentations of two views. For instance, Kim et al. computed disparities from several sets of stereo images to improve the performance of segmentation using the snake algorithm [12]. Ahn et al. combined color and stereo for human silhouettes segmentation [13]. In these works, the performance is highly dependent on the quality of disparities. However, disparity estimation is often prone to errors in the presence of discontinuous depth variations and occlusions at object boundaries. Even with an accurate estimation, some occluded regions of an image have no correspondences at the other image. Besides, according to Middlebury benchmark [14], estimation of dense disparity at the full resolution is computationally expensive. In this work, we only demand coarse disparities for the trimap generation. The segmentation consistency is enhanced by local and global probabilities within and between two views.

Fig. 1. The pipeline of our method. (a) and (b) are stereo images. (c)Disparities between two views are estimated by SGM [10]. (d) Superpixels are estimated from the left view for regularization of disparities. (e) Trimap of the left view is obtained based on (c) and (d). (f) Trimap of the right view is generated by propagating the left trimap using regularized disparities. The final results (g) and (h) are obtained by the MRF minimization.

392

T. Yang et al.

Another class of the methods deals with the stereo segmentation by adding user interactions. For instance, Tasli et al. proposed to add user scribbles of background and foreground at one image and formulate the segmentation as an MRF based energy minimization [15]. Price et al. proposed an interactive method, StereoCut, which allows user scribbles at both sides of the images and interactively refines the segmentation for improvements [9]. Although these methods can achieve good results, the requirement of user marks is somewhat tedious and inefficient, especially when batch processing is demanded. Meanwhile, Bleyer et al. proposed to solve the object segmentation and stereo matching jointly [16], because these two problems are correlated and can promote each other. However, when only the segmentation is desired, it is unnecessary to obtain the stereo matching, which needs extra computational resources. In image matting [11], a trimap is provided as the user input to specify fuzzy regions for soft image segmentation. We follow this idea to generate a trimap automatically according to the raw disparity map that is estimated from the stereo pair. Different from the matting where a decimal value between 0 and 1 is calculated, we label the pixels as background, foreground, or uncertain region. At the segmentation stage, instead of segmenting the whole image, we concentrate on the uncertain region, which not only improves the efficiency but also increases the accuracy. Our method is also related to image co-segmentation [17,18], in which the input consists of two (or more) different photos that share similar foreground objects. The co-occurrence of the similar objects provides additional foreground probabilities that facilitate the joint segmentation. While co-segmentation methods could be used to segment objects in a stereo pair, they do not incorporate the physical information between two views to improve the performance (e.g., view consistency).

2

Our Method

Figure 1 shows our system pipeline where (a) and (b) are the left and right views to form a stereoscopic pair. First, we compute the disparity map between two views based on the method of [10]. Note that disparities do not need to be very accurate, e.g., they may contain errors at object boundaries and are incomplete at some regions (as shown in Fig. 1(c)). Second, we compute image superpixels of the left view according to [19], as shown in Fig. 1(d). Third, we use the superpixels to regularize the initial disparity map so that noise is suppressed and missing regions are filled. Note that the fused result is not shown in the pipeline, but will be discussed in detail in Sect. 2.1. According to the fused disparities, we use a threshold to divide them into foreground and background. Such an initial segmentation is very inaccurate with many errors at object boundaries. Nevertheless, it facilitates the estimation of a trimap to classify the image into three regions: definite foreground, definite background, and uncertain region, referring to Fig. 1(e). After obtaining the trimap of the left view, we can create the trimap of the right view by propagating from the left using disparities,

Uncertain Region Identification for Stereoscopic Foreground Cutout

393

see Fig. 1(f). How to generate trimaps will be discussed in Sect. 2.2. Based on the trimaps, we propose to run an MRF minimization only for pixels in the uncertain region. The final result is shown in Fig. 1(g) and (h). Without loss of generality, we have chosen the left view as the basis in the pipeline.

Fig. 2. (a) The left view and its superpixels. (b) The right view. (c) The raw disparity. (d) The regularized disparity according to superpixels.

2.1

Disparity Regularization

Figure 2 shows an example of disparity regularization, where (a) and (b) are the left and right views, respectively. We run the superpixel segmentation [19] on the left view (shown at the left corner of Fig. 2(a)). Figure 2(c) shows the initial disparity map calculated using the method of [10] (implemented in OpenCV). The disparity values are averaged for all pixels contained in a same superpixel. As a result, each superpixel has an unique disparity value as shown in Fig. 2(d). Notably, we have skipped the “holes” during the average, but assign them with the averaged disparity according to their containing superpixels. There are various methods for generating superpixels (also regarded as oversegmentation or over-smoothing), such as tubor pixels [20] that impose the equal area constraint to produce superpixels with a similar size. Our work does not require the superpixels to be equal sized. Instead, we prefer a set of unbalanced sizes to over-smooth the disparities, e.g., some background regions in Fig. 2(a) are segmented into large superpixels. Although these regularized disparities no long reflect the real disparities due to the averaging, we would like to emphasize that our goal here is not to obtain the accurate disparities, but to use them for the trimap generation. 2.2

Trimap Genaration

We partition the regularized disparity map (Fig. 2(d)) into foreground and background according to a threshold. Figure 3(a) shows the partitioned foreground. As aforementioned, our assumption is that the foregrounds are the objects that stay close to the camera while the backgrounds are those far away from the

394

T. Yang et al.

camera. The threshold is thus set to the middle of the maximum and minimum disparities. Note that Fig. 3(a) is not the final segmentation result. It can be observed that the segmentation boundaries are quite inaccurate. We then detect the boundary pixels and place a window around the boundary as shown in Fig. 3(b). Pixels covered in the window are considered as the unknown pixels, assembling which gives the uncertain region as shown in Fig. 3(c). The right trimap is obtained by propagating from the left one. Rather than propagating the trimap directly, we propagate the boundary pixels and follow the steps of Fig. 3(b) to generate the right trimap (Fig. 3(d)).

Fig. 3. (a) Initial foreground by segmenting the disparity in Fig. 2(d). (b) Boundary pixels are located, along which a local window is moving around. (c) Left trimap. (d) Right trimap.

Fig. 4. (a) A stereo image pair. (b) Our estimated trimaps where a local window (in red) is defined for pixels in uncertain regions. (c) Our results. (Color figure online)

Uncertain Region Identification for Stereoscopic Foreground Cutout

395

The trimap is used to specify unknown regions for the subsequent refinement. Ideally, the unknown regions should cover the entire object boundary. Any missing coverage at this step will cause non-recoverable errors subsequently. The accuracy of a trimap is assured by the boundary of superpixels as well as the regularized disparities, because it is very likely that the over-segmentation does capture the salient object boundaries [19] and the averaged disparities over superpixels are more robust than individual values. 2.3

Segmentation

We build a graph with each pixel in the unknown region as a vertex. The edges are between neighboring pixels. Notably, we do not put graph edges between unknown pixels in the left and right views due to the inaccuracy of the disparities at the boundary. Then, we optimize the labelling for foreground and background at each vertex by minimizing over the following energy:

Fig. 5. Our results on various examples. Please refer the supplementary files for more results.

396

T. Yang et al.

E=

 i

P (ci , li ) +



S(ci , li ; cj , lj ),

(1)

i,j

where ci and cj are neighboring pixels in unknown regions while li and lj are binary labels on ci and cj . The data term measures the probability of a pixel being foreground (li = 1) or background (li = 0), which consists of a global and local color probabilities Pg (ci , li ) and Pl (ci , li ): P (ci , li ) = Pg (ci , li ) · Pl (ci , li ).

(2)

To get the global probability, we take the definite foreground and definite background from both views. Similar to the approach in [21], we use K-means to cluster the RGB colors (normalized to [0,1]) into k = 64 centers. The clusters of foreground and background are denoted as GkF and GkB , respectively. Then, we compute the minimum distances of a pixel color C(i) of ci to each of the clusters: dF (i) = mink C(i) − GkF  for the foreground and dB (i) = mink C(i) − GkB  for the background. Finally, the global probability is calculated as:  dF (i)/(dF (i) + dB (i)), li = 1 (3) Pg (ci , li ) = dB (i)/(dF (i) + dB (i)), li = 0

Fig. 6. (a) Input image pairs (only the left is shown). (b) The ground truth mask on the left image for SnapCut [22] and LiveCut [23]. (c) Result of SnapCut. (d) Result of LiveCut. (e) Result of StereoCut [9]. (e) Our result.

Uncertain Region Identification for Stereoscopic Foreground Cutout

397

For the local probability, we define a local window at each vertex position (Fig. 4(b)). The local window centered at the pixel ci should cover some definite foreground pixels WFm (i) as well as some definite background pixels WBn (i), where m and n denote the number of foreground and background pixels in the local window. The local window W is composed of two windows WL and WR from two views, as shown in Fig. 4(b). These two windows are paired up according to the regularized disparities. As we do not require the pixel-level accuracy for the window’s positioning, the regularized disparities can well satisfy the purpose. Similarly, we compute the minimum distances of a pixel color C(i) of ci to pixels in WFm (i) and WBn (i): dˆF (i) = minm C(i) − WFm  for the foreground and dˆF (i) = minn C(i) − WBn  for the background. Likewise, we can obtain the local color probability by replacing dF (i) and dB (i) with dˆF (i) and dˆB (i) in Eq. 3. Both Pg (ci , li ) and Pl (ci , li ) are linearly normalized to [0,1]. The smoothness term in Eq. (1) measures the similarity between neighboring pixels, which is set to |li − lj | · 1/(1 + C(i) − C(j), where  ·  is the L2 norm distances in RGB color. The whole energy in Eq. (1) can be minimized via a graph cut [24]. For the input pair shown in Fig. 4(a), the final foreground cutouts (for both left and right views) are given in (c).

3

Experiments

We show several examples in Fig. 5 to demonstrate our method. It is clear that the foreground cutouts in these examples are all very clean for both left and right views. Then, Fig. 6 shows the comparison between our method and some previous methods, including two video-based methods (SnapCut [22] and LiveCut [23]) and two interactive stereo segmentation methods (StereoCut [9] and method

Fig. 7. Comparison with [15]. The first, second, and third columns show the input scribbles, the results of [15], and our results.

398

T. Yang et al.

of [15]). Note that a ground-truth mask (the lamppost) is identified in the left image, which is needed in the video-based methods, whereas SteroCut and our method do not need such a mask. The segmentation is performed in the right view for video-based methods, whereas StereoCut and our method segment both views (only the right view is shown). As can be seen, two video-based methods fail to segment the whole lamppost, whereas our method produces a very comparable result with StereoCut [9]. Finally, Fig. 7 shows the comparison with the method of [15] in which some user scribbles are required to mark the foreground and background on one view (as shown in the first column of Fig. 7). Again, our method, without any user scribbles, has produced a very comparable result.

4

Conclusions

We have presented a method for foreground cutout from stereoscopic images. In our method, a raw disparity map is first estimated between two views and upgraded by making use of superpixels. It is then used to generate trimaps for both views so that images are partitioned into three parts: foreground, background, and uncertain region. Next, pixel-labelling (for segmentation) is conducted only for pixels in the uncertain region, which is solved by a graph-cut on an MRF-based energy function. Besides being fully automatic, our method is fast because the number of pixels involved in the labelling process has been decreased greatly. Moreover, our method has been validated by various examples to produce very comparable results as the existing video-based and user-assisted methods. Acknowledgements. This work has been supported by National Natural Science Foundation of China (61502079 and 61370148).

References 1. Lo, W., Baar, J., Knaus, C., Zwicker, M., Gross, M.: Stereoscopic 3D copy & paste. ACM Trans. Graph. 29(6), 147 (2010) 2. Tong, R., Zhang, Y., Cheng, K.: Stereopasting: interactive composition in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1375–1385 (2013) 3. Du, S., Hu, S., Martin, R.: Changing perspective in stereoscopic images. IEEE Trans. Vis. Comput. Graph. 19(8), 1288–1297 (2013) 4. Niu, Y., Feng, W., Liu, F.: Enabling warping on stereoscopic images. ACM Trans. Graph. 31(6), 183 (2012) 5. Zhang, F., Liu, F.: Casual stereoscopic panorama stitching. In: Proceedings of CVPR (2015) 6. Basha, T., Moses, Y., Avidan, S.: Geometrically consistent stereo seam carving. In: Proceedings of ICCV, pp. 1816–1823 (2011) 7. Pal, N., Pal, S.: A review on image segmentation techniques. Pattern Recogn. 26(9), 1277–1294 (1993) 8. Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)

Uncertain Region Identification for Stereoscopic Foreground Cutout

399

9. Price, B., Cohen, S.: StereoCut: consistent interactive object selection in stereo image pairs. In: Proceedings of ICCV, pp. 1148–1155 (2011) 10. Hirschm¨ uller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008) 11. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008) 12. Kim, S.-H., Choi, J.-H., Kim, H.-B., Jang, J.-W.: A new snake algorithm for object segmentation in stereo images. In: Proceedings of ICME, vol. 1, pp. 13–16 (2004) 13. Ahn, J.-H., Kim, K., Byun, H.: Robust object segmentation using graph cut with object and background seed estimation. In: Proceedings of ICPR, vol. 2, pp. 361– 364 (2006) 14. Scharstein, D., Szeliski, R.: Middleburry stereo (2016). http://vision.middlebury. edu/stereo/ 15. Tasli, H., Alatan, A.: User assisted stereo image segmentation. In: 3DTV-CON, pp. 1–4 (2012) 16. Bleyer, M., Rother, C., Kohli, P., Scharstein, D., Sinha, S.: Object stereojoint stereo matching and object segmentation. In: Proceedings of CVPR, pp. 3081– 3088 (2011) 17. Joulin, A., Bach, F., Ponce, J.: Discriminative clustering for image cosegmentation. In: Proceedings of CVPR, pp. 1943–1950 (2010) 18. Hochbaum, D., Singh, V.: An efficient algorithm for co-segmentation. In: Proceedings of ICCV, pp. 269–276 (2009) 19. Felzenszwalb, P., Huttenlocher, D.: Efficient graph-based image segmentation. Int. J. Comput. Vis. 59(2), 167–181 (2004) 20. C ¸ i˘ gla, C., Alatan, A.: Efficient graph-based image segmentation via speeded-up turbo pixels. In: Proceedings of ICIP, pp. 3013–3016 (2010) 21. Li, Y., Sun, J., Tang, C.-K., Shum, H.-Y.: Lazy snapping. ACM Trans. Graph. 23(3), 303–308 (2004) 22. Bai, X., Wang, J., Simons, D., Sapiro, G.: Video SnapCut: robust video object cutout using localized classifiers. ACM Trans. Graph. 28(3), 70 (2009) 23. Price, B., Morse, B., Cohen, S.: LIVEcut: learning-based interactive video segmentation by evaluation of multiple propagated cues. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 779–786 (2009) 24. Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1124–1137 (2004)

Map-Build Algorithm Based on the Relative Location of Feature Points Cheng Zhao1,2, Fuqiang Zhao1,2, and Bin Kong1 ✉ (

1

)

Institute of Intelligent Machines, Chinese Academy of Sciences, Hefei 230031, China [email protected] 2 University of Science and Technology of China, Hefei 230026, China [email protected]

Abstract. Dynamic building road map is a challenging task in computer vision. In this paper we propose an effective algorithm to build road map dynamically with vehicular camera. We obtain top view of video by Inverse Perspective Mapping (IPM). Then extract feature points from top view and generate feature map. Feature map is the description of feature points’ distribution and provides the information of camera status. By detecting camera status, we apply different degree of restraint geometric transformation to avoid unnecessary time cost and enhance robustness. Experiment demonstrated our algorithm’s effectiveness and robustness. Keywords: Road map · Inverse perspective mapping · Image stitching

1

Introduction

According to the China Statistical Yearbook 2016 [1], only in 2015 year, traffic accidents killed more than 58000 people in china. Vehicle manufacturers and drivers seek a more secure way of driving, such as Automatic Driving System and Advanced Driver Assis‐ tance System (ADAS). Rapid development of Autonomous vehicles and Advanced Driver Assistance System (ADAS) has put higher requirements on the application of computer vision technology, which can provide richer information and more reliable control system. Over the past few decades, mature technical technology, such as GPS and preloaded digital map [2–5], has brought us great convenience. Although static map has played a significant role in the navigation and location fields, since the data of static map isn’t real-time, Dynamical surroundings map can provide more accurate decision-making resources in real time. Under certain scenes, for example, pedestrians are present, dynamical map has advantage. This project was supported by the Pilot Project of Chinese Academy of Sciences (XDA08040109), by the National Natural Science Foundation of China (No. 913203002) and Science and Technology Planning Project of Guangdong Province, China (No. 2016B090910002). © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 400–411, 2017. https://doi.org/10.1007/978-3-319-71598-8_36

Map-Build Algorithm Based on the Relative Location of Feature Points

401

1.1 Related Work Some technology has made considerable achievements. Simultaneous localization and map (SLAM) are a great Algorithm. ORB-SLAM [6–8] is one of the most representative algorithms. ORM-SLAM is feature-based Algorithm, consists of three parts, Tracking, Local Mapping and Loop Closing. In the tracking part, ORB-SLAM extracts ORB features of frame, and select key frames to avoid unnecessary redundancy. In the Local Mapping part, the collection of common ORB features set in two selected frames is used to estimate pose of camera. In turn, camera posture is the basis of calculating spatial location of feature points. Bundle adjustment is used to improve the accuracy of feature points. Collections of feature points construct a sparse three-dimensional space mode. The Loop Closing part is helpful to reduce the accumulative error when same scene is detected. SLAM can reconstruct 3D model dynamically, which is foundation of more secure and more convenient control system. Besides ORB-SLAM, there are other wellknown algorithms, such as LSD-SLAM [9], PTAM [10]. SLAM system indeed has a wide range of applications, a very large number of experiments has verified the effectiveness of SLAM. However, in some special case of application, there are some weakness in typical SLAM algorithm. Firstly, SLAM algo‐ rithm extracts feature points from the whole frame, complex-shape environment can provide more robust feature points. SLAM can’t only extract feature of monotonous flat road. That means extracted feature is too sparse to build road map. Secondly, the calcu‐ late cost of SLAM is expensive. For embedded device, this is a more serious problem, insufficient calculation time result in lost tracking of camera. Once system can’t match the feature of previous key frame, the whole SLAM System get lost. Inverse Perspective Mapping (IPM) is an alternate algorithm to obtain surrounding information. According to the perspective principle, IPM gets top view of video camera. IPM is a mature algorithm, IPM can get plane information with a small amount of calculation cost, but only 2-dimensional plane information is preserved. For road infor‐ mation building, it’s enough, because we need to focus on the road plane sign. IPM is sensitive to shaking of vehicular camera. As a reliable algorithm, IPM has been imple‐ mented in many Automatic Driving Systemses and Advanced Driver Assistance System (ADAS). Such as measuring distance [11, 14], vanishing point estimation [12] and detecting [13, 15] (include lane line, obstacle, pedestrian and terrain, etc.). In this paper we propose a new algorithm. We get top view by Inverse perspective mapping (IPM) in advance. Then we extract feature of the IPM of frame, generated feature map, and based on feature of map’s characteristic we simply basic algorithm. In second chapter, we introduce the way to get top view, in third chapter, we discuss the feature map characteristic, and experiment detail is showed in fourth chapter. In last chapter, we sum up the whole algorithm.

2

Parameter Estimation System

Typically, Inverse perspective mapping (IPM) is used to obtain top view of frame, which get from fixed-posture camera. Top view is a scaled-down of real world plane. We can establish closed-loop negative feedback system, and define evaluation function help to

402

C. Zhao et al.

get geometric transformation. We can select 4 points of known size object in video frames and define relative displacement of responding point in IPM frames to get geometric transformation result. By changing different relative displacement approxi‐ mate solutions. Using evaluation function obtains relative displacement in next time, until the distortion of IPM result is acceptable. Typically, we select points that lie on the edge of regular shape of object of real world. In urban road scene, lane line and traffic sign are eye-catching sign, and follow clear known traffic regulations. In our experi‐ ments, we select lane line and outline of bicycle sign (see Fig. 1). Once those points are selected, we never change it. Different selections of respond points in IPM result generate different top views, in this paper we call those top view result as IPM frames. To check the distortion extent, we should define an evaluation function, evaluation function is introduced in next section.

Fig. 1. IPM result of video frame. Red lane line is parallel. Yellow box’s aspect ratio is 15:9 (Color figure online)

Our algorithm can be to show as follows

The computational complexity isn’t important, since posture of vehicular camera is fixed, projective matrix parameters never change. This algorithm runs only once.

Map-Build Algorithm Based on the Relative Location of Feature Points

403

2.1 Evaluation Function Another question is how to define evaluation function. Evaluation function is used to measure the distortion extent, the way of selection is customized. For example, in our experiment, we combine two factor to evaluation function, first factor is cross angle of two lane line, which should be 0. Line angle can be calculated by Hough transform [17] and trigonometric functions. Second factor is aspect ratio of bicycle’s rectangular box, which should be R. So we define evaluation function as follows ( ( ( ))2 ( ))2 eva = 𝛼 ∗ angle1 − angle2 + � ∗ Rcal − Rreal

3

(1)

Feature Map

3.1 Basic Algorithm Before we discuss our algorithm, we should analysis the basic algorithm. According to basic algorithm image stitching, after we get a reliable transformation and apply it in the original frame to get IPM frame. For every IPM frame we detect SURF (speeded up robust features) feature points and extract SURF descriptor. For every two adjacent IPM frame, we match the respond SURF feature points. Then we find another geometric transformation between adjacent IPM frames. For this case, this geometric transforma‐ tion should be similar or affine transformation. This geometric transformation is esti‐ mated by RANSAC or Squares Algorithms, since more than 3 pair matched feature points exist. At last, we apply those geometric transformation for a series of responding IPM frames one by one, map is reconstructed. Two points should be explained. The reason for extracting SURF feature points instead of other feature points is compromise of calculating cost and accuracy, other feature is also feasible. But selected feature should avoid to generating too much response points that lie the edge of background. Another point is why those geometric transformations should be similar or affine transformation. Different from popular algo‐ rithm like Autostitch [16], All IPM frame should be at same plane, we should concentrate on displacement in same plane and rotation. Accord to our experiment, similar trans‐ formation is good enough, affine transformation lead in the distortion. Basic algorithm is a viable option, but part of this algorithm can be optimized. Based on our experiment, we witness 3 weaknesses for basic algorithm. Firstly, error is mainly attributable to the progressive effect. Every inaccurate geometric transformation effect subsequent work. Secondly, for undulating area or shaking of camera result in inaccurate matching result. Refer to first reason, unreliable matrix estimated by inaccurate matching feature pair worsen performance of map. Thirdly, it’s a great idea in theory that built map from transformation, which estimated at matched feature. However, when vehicle travelled straightly, calculation for every two IPM frame is wasteful of time, since two IPM frame have only a single direction of the displacement difference. Moreover, esti‐ mated transformation increased progressive error. For first problem, progressive error is unavoidable, but progressive error suggest two clues: IPM frame should be as standard as possible, that is duty of IPM stage; another

404

C. Zhao et al.

clue is we should preserve original shape of IPM frame, this is another explanation that similar transformation is better choose than affine transformation. For second and third problem, if we detect the status of vehicular camera in advance, we can simply the basic algorithm by applying different degree of restraint geometric transformation to avoid unnecessary time cost and enhance robustness. When vehicle travelled straightly, esti‐ mated transformation has negligible angle, instead, using fixed orientation similar matrix is effectiveness and robustness enough. When the status of vehicular camera is unstable, the inaccurate matched feature points should be abandon. 3.2 Feature Map and Auxiliary Conception Based on the above analysis, we propose a new algorithm to simply the process by detecting current status of vehicular camera. Before we show our algorithm, we should introduce some auxiliary conception.

Fig. 2. Feature Map and auxiliary conception. (a) The displacement of two matched feature points in adjacent IPM frame. (b) Obtain feature map and the value of auxiliary conception

After we get matched SURF feature point pairs, from two IPM frames, we mark former IPM frame as f1, later IPM frame as f2, matched SURF feature points of f1 as fp1, matched SURF feature points of f2 as fp2, the size of feature points is S. Every feature point in fp1 is marked as fp1i, responding feature point in fp2 is marked as fp2i, ( fp1i and fp2i are matched feature points pair, where i = 1, 2, 3 … S). For every matched feature point pair ( fp1i and fp2i), we define ΔDi is relative displacement of fp2i and fp1i. ΔDi = fp2i − fp1i , where i = 1, 2, 3 … . S

(2)

Map-Build Algorithm Based on the Relative Location of Feature Points

405

We collect all ΔDi, remark those ΔDi at original point of new plane, we define this new plane as feature map (see Fig. 2). The feature map can provide richer information about status of vehicular camera. There are 4 auxiliary conceptions to introduce. Peak point is location of the highest density of points. Effective coverage is circle, its center is peak points and its radius is customized d. Effective proportion is the ratio of the number of ΔDi in effective coverage to the sum of matched feature point pair S. effective density area is irregular shape that included peak points and this area’s density exceeds certain threshold (see Fig. 2). Those conceptions are used to help to simply the basic algorithm. In this paragraph, we discuss the distribution of ΔD. If we make a basic assumptions: IPM frame is a scaled-down version of top view of real world (see Fig. 3). The points of different color are evenly distributed features of IPM frame, the gray box are IPM frame at times ti, red box is IPM frame at time ti+1 when vehicle travel straightly, green box is IPM frame at time ti+1 when vehicle turn,blue box is IPM frame at times ti+1 when status of vehicular camera is unstable. ( ) We define the feature point’s location in real world coordinate is xreal , yreal , For the ( ) IPM frame at time ti+1, the relative location of feature point is xb , yb , the origin is ( ) xbo , ybo . Relative location is the location of feature point in IPM frame coordinate system. For the IPM frame at times ti+1, When vehicle travelled straightly, the relative ( ) ( ) location of feature point is xr , yr , the origin is xro , yro ; When vehicle turn left, the ( ( ) ) relative location of feature point is xg , yg , the origin is xgo , ygo ; When vehicle travel ( ) ( ) straightly, the relative location of feature point is xp , yp , the origin is xpo , ypo .

Fig. 3. Ideal model. (a) Vehicle travel straightly. (b) Vehicle turn left. (c) Camera status is unstable (Color figure online)

When vehicle travelled straightly, we have xreal = xb + xb0 = xr + xro

(3-1)

yreal = yb + yb0 = yr + yro

(3-2)

406

C. Zhao et al.

So the distribution of ΔD should be a point. ( ) ( ) ΔD = xr − xb , yr − yb = xbo − xro , ybo − yro

(4)

When vehicle turned left or right, xreal = xb + xbo = xg cos� − yg sin� + xgo

(5-1)

yreal = yb + ybo = xg sin� + yg cos� + ygo

(5-2)

( ) ( ) xg = sin� yreal − ygo + cos� xreal − xgo

(6-1)

( ) ( ) yg = cos� yreal − ygo − sin� xreal − xgo

(6-2)

So we have

So we can obtain ΔD as follows ( ) ( ) ( ) xg − xb = sin� yreal − ygo + cos� xreal − xgo − xreal − xbo

(7-1)

( ) ( ) ( ) yg − yb = cos� yreal − ygo − sin� xreal − xgo − yreal − ybo

(7-2)

Since yg = yreal − ygo and xg = xreal − xgo xg − xb = sin�yg + cos�xg − xb

(8-1)

yg − yb = cos�yg − sin�xg − yb

(8-2)

So the distribution of ΔD point is stretched. Stretching extent is related to yaw angle � and feature points’ location. When � = 0, the distribution of ΔD reduced to a point, equate to Eq. 4. Analysis of unstable status is impossible, various reason effect the distribution of ΔD. The distribution is unpredictable, but the degree of aggregation is reduced inevitably. If vehicle travelled straightly, ΔDi , where i = 1, 2, 3 … . S are highly concentrated. The effective proportion is close to 1, Effective density area keep in small range. In this case the displacement of peak point is displacement of vehicular camera, peak point location is related to speed of vehicle. If vehicle turned, the effective proportion kept in high level, effective density area expanded. In this case the basic algorithm should be applied. If status of vehicle is unstable, the effective proportion is kept in low level, the matched feature point pairs should be abandon. So our algorithm is as follows.

Map-Build Algorithm Based on the Relative Location of Feature Points

407

3.3 Implement of Algorithm The implement of feature map and auxiliary conception is customized. In our experiment, we use 3 * 3 pitch to cover ΔDi , where i = 1, 2, 3 … S in feature map, feature map became mountain-like area. This area’ peak is peak point, if this area’s peak is connected plateau. The plateau’s geometric center is the peak point. If this area has more than one peak, we use the highest connected cross-section’s geometric center as peak point. Effective proportion is a criterion to measure aggregation degree of feature points. If effective proportion of a certain threshold, the matched feature points pairs should be abandoned.

Fig. 4. Feature map. (a) When the vehicle travel straightly. (b) When the vehicle is on a bend. (c) When the status of vehicle is unstable

Effective density area’s threshold is apposite proportion of value of peak point. We use outline of effective density area to measure angle changing extent roughly. If the effective

408

C. Zhao et al.

density area is close to circle or square, so car travel straightly, or the car is on a bend (see Fig. 4).

4

Experiment

In order to evaluate the performance of our algorithm, a database with 5 videos is tested resource, which consist of 2570 raw images. Experiment vehicle is a vehicle mounted CCD video camera. All images are RGB images with 640 * 480 resolutions. All program is implemented by MATLAB on PC. For the geometric transformation of IPM process. As the Eq. 1, We define β = 25, acceptable error is 0.5 degree angle and 2% loss in aspect ratio, so the thread is 0.5. Other reasonable choose is also acceptable. Loop upper limit is 1000. One of the original frame and its IPM is to show as follows (see Fig. 5). The IPM results are gray scale images with 1061 * 1523 resolutions. The Geometric transformation is as follows. For algorithm 2, in order to obtain the effective proportion, we select 30 pixels as effective coverage’s radius, threshold is 80%. For effective density area, the proportion of value of peak point is 70%. Figure 6 shows effective proportion of 5 video frames. 80% is thread, when effective proportion of feature map is less than 80%, matched feature points is abandon. From video frame, we witness that status of those frame’s video is unstable. Experiment confirmed positive correlation between aggregation extent of ΔD and vehicular camera. Figure 7 shows angle of 5 video frames. The upper part of fig are angle of estimated geometric matrix parameters. The lower part of fig is angle of our algorithm. When the angle of geometric matrix is tiny and fixed direction geometric matrix is used to avoid unnecessary progressive error and enhance robustness. When yaw angle is big enough, namely the vehicle is turn left or right, geometric matrix is estimated from basic algorithm.

Fig. 5. Original frame from video and IPM result

Map-Build Algorithm Based on the Relative Location of Feature Points

409

Fig. 6. Effective proportion

Fig. 7. Angle of rotation

Experiment result showed as follows (see Fig. 8 and Table 1). Figures 8(a) and (b) show the different of video 1, Figs. 8(c) and (d) optimization algorithm shows better performance than basic algorithm obviously. Basic algorithm is great, but many factors effect the performance, under massive input and unstable status of vehicular camera, progressive error lead in unavoidable the distortion. Our algorithm provided more reli‐ able result with lower time cost, structure of road map is reconstructed.

410

C. Zhao et al.

Fig. 8. Experiment result. (a) Basic algorithm and the distortion part of video 1. (b) Our algorithm and improved result of video 1. (c) Basic algorithm and the distortion part of video 2. (b) Our algorithm and improved result of video 2

Table 1. The average time cost of algorithm2 Frame number 284 118 445 752 971

5

Basic algorithm 103.14 s 37.24 s 82.76 s 299.20 s 661.03 s

Our algorithm 91.59 s 30.19 s 90.80 s 263.28 s 591.67 s

Conclusion

In this paper, we propose a fast and robust dynamical map building algorithm based on IPM and feature map. We establish feedback algorithm to obtain geometric transfor‐ mation matrix of IPM. According to IPM frame’s characteristic, that IPM frame is a scaled-down version of top view’-eye of real world. Parallelism and aspect ratio is used to checking the accuracy of geometric transformation. In second stage of algorithm we define feature map and some auxiliary conception to help to classify status of camera or vehicle. When vehicle travelled straight, fixed orientation geometric transformation is used to avoid unnecessary and inaccuracy calculation. Experiment results show that the proposed algorithm can provide reliable and effective result. Future work will include search for relationship between of effective density area and estimated geometric trans‐ formation from matched feature points.

Map-Build Algorithm Based on the Relative Location of Feature Points

411

References 1. China: China Statistical Yearbook 2015 (2016) 2. Nedevschi, S., Popescu, V., Danescu, R., et al.: Accurate ego-vehicle global localization at intersections through alignment of visual data with digital map. IEEE Trans. Intell. Transp. Syst. 14(2), 673–687 (2013) 3. Gruyer, D., Belaroussi, R., Revilloud, M.: Map-aided localization with lateral perception. In: 2014 IEEE Intelligent Vehicles Symposium Proceedings, pp. 674–680. IEEE (2014) 4. McLaughlin, S.B., Hankey, J.M.: Matching GPS records to digital map data: algorithm overview and application. National Surface Transportation Safety Center for Excellence (2015) 5. Lu, M., Wevers, K., Van Der Heijden, R.: Technical feasibility of advanced driver assistance systems (ADAS) for road traffic safety. Transp. Planning Technol. 28(3), 167–187 (2005) 6. Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans. Robot. 31(5), 1147–1163 (2015) 7. Gálvez-López, D., Tardos, J.D.: Bags of binary words for fast place recognition in image sequences. IEEE Trans. Robot. 28(5), 1188–1197 (2012) 8. Mur-Artal, R., Tardos, J.D.: ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras. arXiv preprint arXiv:1610.064752016 9. Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 834– 849. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10605-2_54 10. Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR 2007, pp. 225–234. IEEE (2007) 11. Jiang, G.Y., Choi, T.Y., Hong, S.K., et al.: Lane and obstacle detection based on fast inverse perspective mapping algorithm. In: 2000 IEEE International Conference on Systems, Man, and Cybernetics, vol. 4, pp. 2969–2974. IEEE (2000) 12. Nieto, M., Salgado, L., Jaureguizar, F., et al.: Stabilization of inverse perspective mapping images based on robust vanishing point estimation. In: 2007 IEEE Intelligent Vehicles Symposium, pp. 315–320. IEEE (2007) 13. Jiang, G.Y., Choi, T.Y., Hong, S.K., et al.: Lane and obstacle detection based on fast inverse perspective mapping algorithm. In: 2000 IEEE International Conference on Systems, Man, and Cybernetics, vol. 4, pp. 2969–2974. IEEE (2000) 14. Gupta, A., Merchant, P.S.N.: Automated lane detection by K-means clustering: a machine learning approach. Electron. Imaging 2016(14), 1–6 (2016) 15. Fritsch, J., Kühnl, T., Kummert, F.: Monocular road terrain detection by combining visual and spatial information. IEEE Trans. Intell. Transp. Syst. 15(4), 1586–1596 (2014) 16. Ma, B., Zimmermann, T., Rohde, M., et al.: Use of Autostitch for automatic stitching of microscope images. Micron 38(5), 492–499 (2007) 17. Mukhopadhyay, P., Chaudhuri, B.B.: A survey of Hough Transform. Pattern Recogn. 48(3), 993–1010 (2015) 18. Torr, P.H.S., Zisserman, A.: MLESAC: a new robust estimator with application to estimating image geometry. Comput. Vis. Image Underst. 78(1), 138–156 (2000)

Sparse Acquisition Integral Imaging System Bowen Jia, Shigang Wang ✉ , Wei Wu, Tianshu Li, and Lizhong Zhang (

)

College of Communication Engineering, Jilin University, Changchun, China {jiabw15,lits16,zhanglz16}@mails.jlu.edu.cn, [email protected], [email protected] Abstract. Using 3DS MAX to obtain elemental image (EI) array in the virtual integral imaging system need to put large scale camera array, which is difficult to be applied to practice. To solve this problem we establish a sparse acquisition integral imaging system. In order to improve the accuracy of disparity calculation, a method using color segmentation and integral projection to calculate the average disparity value of each color object between two adjacent images is proposed. Firstly, we need to finish the establishment of virtual scene and microlens array model in 3DS MAX. According to the mapping relationship between EI and sub image (SI), we can obtain the SI by first, then calculate to the EI. The average value of the disparity from different color objects between adjacent images is acquired based on color image segmentation method and integral projection method, and then translate a rectangular window of fixed size in accordance with the average disparities to intercept the rendered output images to get the sub images (SIs). Finally, after stitching and mapping of the SIs we obtain the elemental images (EIs), put the EIs into the display device to display 3-dimen‐ sional (3D) scene. The experimental results show that we can only use 12 * 12 cameras instead of 59 * 41 cameras to obtain EIs, and the 3D display effect is obvious. The error rate of disparity calculation is 0.433% in both horizontal and vertical directions, which is obviously better than other methods with disparity error rate of 2.597% and 4.762%. The sparse acquisition integral imaging system is more accurate and more convenient which can be used for EI content acquisition for large screen 3D displaying. Keywords: Integral imaging · Color segmentation · Integral projection · Window interception · Disparity · Mapping

1

Introduction

Integral imaging technology [1–3] is part of the advanced 3D display technologies currently on the international stage. It is an autostereoscopic and multi-perspective ster‐ eoscopic imaging technique that captures and reproduces a light field by using a 2D array of microlenses. In recent years, with the development of industrial production and audience’s love of watching 3D movies at the cinema, 3D display technology has become a hot research. However, people need to wear 3D glasses while watching 3D movies, the viewing experience is not perfect. 3D movies are based on binocular disparity principle [4], which separate the left and right eye images, and produce a ster‐ eoscopic feeling by the fusion of the brain, watching for a long time will produce visual © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 412–422, 2017. https://doi.org/10.1007/978-3-319-71598-8_37

Sparse Acquisition Integral Imaging System

413

fatigue. The integral imaging technology does not need people to wear any device. It is a real representation of 3D scene without visual fatigue which provides continuous image disparities and true color images. It has good application prospects. The integral imaging system is mainly composed of two parts which are picking up process and displaying process. In the pick up process, each microlens allows an image of the subject as seen from the viewpoint of that lens’s location to be acquired. In the display process, each microlens allows each observing eye to see only the area of the associated micro-image containing the portion of the subject that would have been visible through that space from that eye’s location. There are mainly two kinds of integral imaging systems which are optical integral imaging system and computer virtual integral imaging system. The computer virtual pick up method [5] is the most popular EI content acquisition method. However, direct shooting of EIs need us to place a camera array which contains large number of cameras. The rendering process of direct shooting usually lasts for a long time and it is prone to error. Due to the mapping relationship between EI array and SI array [6, 7], we propose a method which obtains the SI array first and then calculates to the EI array. This method reduces the number of cameras required and achieves sparse acquisition inte‐ gral imaging. In view of the present situation, we establish a camera array as virtual micro‐ lens array to pick up and render output images by 3DS MAX [8]. And then we use color segmentation to separate objects of different colors, after that, the horizontal disparity and vertical disparity of the adjacent images of the same color objects are obtained by integral projection method, and the average disparities in the horizontal and vertical directions of several objects satisfying the condition are selected as the final disparities. Next, we trans‐ late a rectangular window of fixed size according to the disparity values to intercept rendered output images to get SI array, and the EI array is obtained after mapping. Finally, the 3D images are captured by our device on the optical platform, which verifies the sparse acquisition integral imaging system.

2

Theory

2.1 Camera Imaging Principle 3D Studio Max, often referred to as 3DS MAX, is a 3D modeling and rendering software helps us to pick up scenes for design visualization. There are free camera and target camera in 3DS MAX which can be used to simulate the real camera to pick up target objects. Figure 1 is the pick up and display process of traditional integral imaging system.

Fig. 1. Traditional integral imaging system

Fig. 2. Imaging principle of 3DS MAX camera

414

B. Jia et al.

The imaging principle [9] is shown in Fig. 2. Where u is the distance from 3D object to the camera lens, v is the distance between the film and camera lens, L is the edge length of the 3D object, l is the length of image on the camera film, f is the focal length of the camera. The Gauss imaging formulas are shown as follows:

1 1 1 + = u v f

(1)

L u = l v

(2)

2.2 Color Segmentation in HSV Space After getting the color images rendered output by 3DS MAX, we need to convert the RGB data into HSV data, set the threshold range of different colors, and segment the different color images [10–12]. Different color thresholds are set as shown in Table 1. After segmenting images of different colors extracted in HSV space, we need to convert it into the RGB space. Table 1. HSV threshold range (Unit: 1) Range H S V

Color Yellow 30–40 >0 >0

Pink 230–240 >0 >0

Orange 0–10 >0 >0

Green 80–90 >0 >0

Blue 150–160 >0 >0

2.3 Window Interception Based on Integral Projection As shown in Fig. 3, L is the distance between one pixel and its nearest microlens, and F is the distance between microlens and the recording medium, the distance between two adjacent microlenses remains unchanged. The ratio of L to F is a constant according to the triangle theorem. We only need to know the disparities of any two adjacent images, and we will know all the disparities. Occlusion problems will affect the accuracy of the disparity calculation. So we select the disparity values which satisfy the condition to take the average value as the final disparity value. Firstly, we establish coordinate system in which the upper left corner of each rendered image is taken as the coordinate origin, from the origin to the right direction is the x axis, and the y axis is down to the vertical direction as shown in Fig. 4. Set average horizontal disparity of dh, and average vertical disparity of dv. After obtaining the disparities, it is necessary to intercept the images with the size of M1 (pixels) * N1 (pixels) by a rectangular window with the size of M (pixels) * N (pixels) to obtain the SI array [13, 14]. And it is necessary to determine the position coordinate (x(1,1), y(1,1)), which is the upper left corner of the rectangular window in the picture taken by the camera in the first row of the first column, where the SI we intercepted is just in the center of the rectangular window. And then according to the average disparities

Sparse Acquisition Integral Imaging System

Fig. 3. Disparity of adjacent images

415

Fig. 4. Window interception

through integral projection [15], we translate the rectangular window to intercept images to obtain SI array. (x(m,n), y(m,n)) is the coordinate of the upper left corner of rectangular window in the picture taken by the camera in the mth row of the nth column, the formulas are as follows: x(m,n) = x(1,1) − (n − 1)dh

(3)

y(m,n) = y(1,1) − (m − 1)dv

(4)

2.4 Mapping Relations Between EIs and SIs The mapping relationship [16] is illustrated in Figs. 5 and 6. Figure 5 is the mapping relationship between EIs and SIs. In order to achieve sparse acquisition, we use the inverse mapping method to obtain EIs through SIs mapping, the process is shown in Fig. 6. The left side is the SI array, which are composed of m * n SIs, the size of each SI is i (pixels) * j (pixels). The pixels of the same position in each SI are extracted and combined together to form a mapping unit picture, namely EI. In the end, all the EIs are combined together in order to get the EI array.

Fig. 5. Mapping process

Fig. 6. Mapping relationship between EI and SI

According to the mapping relationship, the corresponding locations of one pixel in the image before and after mapping [17] are shown in formula (5). Where Former and Latter represent images before and after mapping, x and y represent the pixel position of the image before mapping, x = 0, 1, 2, … (m ∗ i − 1), y = 0, 1, 2, … (n ∗ j − 1), % is

416

B. Jia et al.

the symbol of remainder operation,⌊ ⌋ is the symbol of round down function. EI array is composed of i * j EIs, the size of each EI is m (pixels) * n (pixels). ⌊ ⌋) ⌊ ⌋ y x ,y%j × n + Former(x, y) = Latter x % i × m + i j (

(5)

2.5 Display Principle The values of m and n depend on the relationship between the resolution and size of the display screen and the pitch of micro lens array. If the screen resolution is A (pixels) * B (pixels), the size of display screen is C (mm) * D (mm), the pitch of micro lens array is d (mm), then we can calculate to the values of m and n: A C = m d

(6)

B D = n d

(7)

2.6 Parallax Error Rate The disparity error rate E is defined as the ratio of the average disparity W0 to the exact disparity value W, The formula is as follows: |W − W0 | | E= | W

3

(8)

Experimental Results and Discussion

3.1 Els Generated by Virtual Microlens Array The display screen is the 5.5 inches (in) OPPOr9 phone screen with a resolution of 1280 (pixels) * 1080 (pixels). The size of which is 110.7 mm * 92.7 mm. The size of microlens array is 55 mm * 55 mm, the horizontal and vertical pitches of microlens array are 1.03 mm. According to the formulas (6) and (7), the resolution of the rendered output images should be 12 (pixels) * 12 (pixels). So we need to place 12 rows and 12 columns of free cameras to pick up the virtual scene. Set the free camera lens focal length f = 50 mm, camera pitch p = 10 in, viewing angle θ = 39.598°. Use the teapot and ball model in 3DS MAX as the target objects. The radius of the teapot is 12 in, the radii of the yellow and green balls are 3 in, the radius of the blue ball is 3.5 in, and the radius of the orange ball is 5 in. Place the free camera array at a distance of 210 in from the object. Different views of 3D scene are shown as follows (Fig. 7):

Sparse Acquisition Integral Imaging System

(a) Top view

(b) Front view

(c) Left view

417

(d) Perspective view

Fig. 7. Different views of 3D scene

Set the size of the rendered image as 600 (pixels) * 600 (pixels), put them into the HSV space for color segmentation, and then import the pictures to the RGB space to display, the pictures are as follows:

(a)Color image(b)Pink image(c)Green image(d)Blue image(e)Orange image(f)Yellow image

Fig. 8. Color segmentation (Color figure online)

Figure 8(a) is the color image with the size of 600 (pixels) * 600 (pixels) rendered by the 66th (the 6th row of the 6th column) camera of 3DS MAX. Figure 8(b), (c), (d), (e) and (f) are the pink, green, blue, orange and yellow portions after color segmentation. We select the 66th and the 67th (the 6th row of the 7th column) camera images for different color segmentation, and then do the horizontal and vertical integral projection to obtain horizontal disparity, part of the integral projection of adjacent different color images are shown in Fig. 9. Figure 9(a) is the integral projection of a green ball and Fig. 9(b) is the integral projection of the whole of all objects.

(a) Green: 66,67

(b) Total: 66,67

Fig. 9. Integral projection of camera66 and camera67 (Color figure online)

Similarly, we select the 66th and the 78th (the 7th row of the 6th column) camera images for different color segmentation, and do the integral projection to obtain vertical disparity, the results are shown in Fig. 10.

418

B. Jia et al.

(a) Green: 66,78

(b) Total: 66,78

Fig. 10. Integral projection of camera66 and camera78 (Color figure online)

In order to obtain the disparity comparison standard, we take the target object replaced by a white ball with a radius of 0.05 in, other conditions are not changed, use the camera array to shoot white ball, render the pictures with the same size of 600 (pixels) * 600 (pixels) to do integral projection. We select the 66th and the 67th camera images to do integral projection to obtain horizontal disparity, and select the 66th and the 78th camera images to do integral projection to obtain vertical disparity (Fig. 11).

(a) Standard: 66,67

(b) Standard: 66,78

Fig. 11. Integral projection of disparity standard

According to the integral projections of different color objects we can obtain the location coordinates in the picture and the adjacent picture of the corresponding color objects. By subtracting the coordinates of the two objects, we can get the disparities of the corresponding color objects. The results are shown in the following Table 2: Table 2. Comparison of disparity of different objects (Unit: pixel) Disparity Horizontal Vertical

Color Yellow1 Yellow2 Pink 34.5 33.5 37 36 36 38.5

Blue 39 39

Green 38.5 38.5

Orange 37.5 37.5

Total 37.5 37.5

Standard 38.5 38.5

Sparse Acquisition Integral Imaging System

419

Where Yellow 1 is the yellow ball on the left side and Yellow 2 is the yellow ball on the right side. Pink, Blue, Green, Orange are respectively the corresponding color objects. The General represents the whole of all objects. Since the horizontal pitch of the camera is the same as the vertical pitch, so the horizontal disparity of the object should be the same as the vertical disparity. Due to the horizontal disparity value and the vertical disparity value of the corresponding objects of Yellow 1, Yellow 2 and Pink are not equal, we need to remove this part of the data. So we obtain the average disparity of 38.3333 pixels based on the average disparity of the corresponding Blue, Green and Orange object images. According to the formula (8), the error rate of the disparity calculation is 0.433% both in the horizontal and the vertical direction. The disparity error rate of the whole of all objects is 2.597%. And the disparity error rate of the literature method [18] is 4.762%. The disparity error rate of the method used in this paper is significantly reduced. When the size of the rendered image is 100 (pixels) * 100 (pixels), the disparity value is 1/6 of the rendered image with the size of 600 (pixels) * 600 (pixels), and both of the average horizontal and vertical disparities are 6.38888 pixels. Set the coordinate of the upper left corner of the rectangular window in the picture rendered by the camera of the first row of the first column to (55, 63). Translate the rectangular window with a size of 59 (pixels) * 41 (pixels) from left to right and from top to bottom every 6.38888 pixels to intercept different images. For images that are less than 59 (pixels) * 41 (pixels) after the window interception, we use black pixels to fill the background. Finally, the inter‐ ceptions of the pictures in accordance with the order of the cameras after image splicing are shown in Fig. 12. According to the mapping relationship between EI and the SI, the EI array is obtained as shown in Fig. 13:

Fig. 12. Sub image array

Fig. 13. Elemental image array

3.2 Display by Plat Lens Array In the display process, the focal length of the microlens array is 5 mm, and the distance from the microlens array to the display screen is exactly 3 mm which is just the thickness of the microlens array. According to Gauss imaging formula (1), we can calculate the distance between the center depth plane of the image and the microlens array is 7.5 mm, the image is a virtual image which locates inside the display screen.

420

B. Jia et al.

In order to verify the 3D display of the sparse acquisition integral imaging system, we build a device on the optical platform. Firstly, the adjustable lifting platform is fixed on the optical platform to adjust the height from up to down. Then, the manual translation table is fixed on the adjustable lifting platform to adjust from left to right. The slide rail is fixed on the manual translation stage to adjust from forward to backward. Next, the microlens array is placed on the OPPOr9 mobile phone screen. We need to ensure that the edges of the microlens array and the mobile phone screen are parallel. And put them all together on the optical platform. The other mobile phone iphone6 (4.7 inches, reso‐ lution 1334 (pixels) * 750 (pixels)) is placed on the slide rail stage, then we turn on the camera of iphone6, slowly adjust the scale of the device to make iphone6 move from left to right, from forward to backward to pick up different view of images. Different perspective views of the device are shown as Fig. 14:

(a) Micro lens array (b) Top view of device(c) Right view of device (d) Front view of device

Fig. 14. 3D display optical platform

The different viewpoints of images taken from the device are shown in Fig. 15:

(a)(Row4,Column5) (b)(Row4,Column6)

(c)(Row4,Column9)

(d)(Row6,Column3) (e)(Row6,Column7) (f)(Row6,Column8)

(g)(Row9,Column4) (h)(Row9,Column7)

(i)(Row9,Column9)

Fig. 15. 3D reproduction scenes from different perspectives

All the images in Fig. 15 can be found in Fig. 12. For example, Fig. 15(a) is similar to the SI in the fourth row of the fifth column in Fig. 12.

Sparse Acquisition Integral Imaging System

4

421

Conclusion

In this paper, a sparse acquisition integral imaging system based on color segmentation and integral projection is proposed. We use color segmentation in the HSV space to get different color objects, and use the integral projection method to calculate the disparity values, and then find the average value of the disparities of different objects to reduce the error rate of horizontal/vertical disparity to 0.433%, which is better than the other methods with disparity error rates of 2.597% and 4.762%. The integral projection method commonly used in face recognition and license plate recognition is applied to the integral imaging to obtain the target object position. We do not need to know other parameters such as distance of the object in the acquisition process, we can obtain the disparities only through the information in the pictures. The SI array is obtained by translating the rectangular window with a size of 59 (pixels) * 41 (pixel) according to the disparities. And the EI array is obtained according to the mapping relationship between EI and SI, put the EI array into the phone and place the microlens array on the phone screen to get a good 3D display effect. We can only use 12 * 12 cameras instead of 59 * 41 cameras to obtain the EI array to achieve sparse acquisition. This method makes it easier and more accurate to reproduce the 3D image. Acknowledgement. This work was supported by the of National Natural Science Foundation of China (61631009).

References 1. Wang, Q.H., Deng, H.: 3D pickup and display method of integral imaging. Chin. J. Liq. Cryst. Displays 29(2), 153–158 (2014) 2. Yuan, X.C., Xu, Y.P., Yang, Y., et al.: Design parameters of elemental images formed by camera array for crosstalk reduction in integral imaging. Opt. Precis. Eng. 19(9), 2050–2056 (2011) 3. Jiao, X.X., Zhao, X., Yang, Y., et al.: Pick-up system for three-dimensional integral imaging with camera array. Opt. Precis. Eng. 20(8), 1653–1660 (2012) 4. Cao, X.: Technological bottleneck of virtual reality. Sci. Technol. Rev. 34(15), 94–103 (2015) 5. Kuang, Y., Jiang, J.: Study of virtual scene interaction based on VRML and 3DS Max. Appl. Mech. Mater. 713–715, 2345–2347 (2015) 6. Park, J.H., Jung, S., Choi, H., et al.: Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification. Appl. Opt 43(25), 4882–4895 (2004) 7. Park, J.H., Kim, J., Lee, B.: Three-dimensional optical correlator using a sub-image array. Opt. Express 13(13), 5116–5126 (2005) 8. Perry, T.S.: Autodesk 3D studio max 2017. Animation 30(6), 56 (2016) 9. Jiao, T.T., Wang, Q.H., Li, D.H., et al.: Computer-generated integral imaging based on 3DS MAX. Chin. J. Liq. Cryst. Displays 23(5), 622–624 (2008) 10. Ali, N.M., Rashid, M., Alang, N.K., Mustafah, Y.M.: Performance comparison between RGB and HSV color segmentations for road signs detection. Appl. Mech. Mater. 393, 550–555 (2013) 11. Pujol, F.A., Pujol, M., Jimeno-Morenilla, A., et al.: Face detection based on skin color segmentation using fuzzy entropy. Entropy 19(1), 26 (2017)

422

B. Jia et al.

12. Zhu, Y.Z., Meng, Q.H., Pu, J.X.: Traffic light auto-recognition based on HSV color space and shape feature. Video Eng. 39(5), 150–154 (2015) 13. Guo, M., Si, Y.J., Wang, S.G., et al.: Elemental image generation combing discrete viewpoint pickup with adaptive window interception. J. Jilin Univ. (Eng. Technol. edn.) 46(5), 1681– 1687 (2016) 14. Lyu, Y.Z., Wang, S.G., Zhang, D.T.: Elemental image array generation and sparse viewpoint pickup in integral imaging. J. Jilin Univ. (Eng. Technol. edn.) 43(S1), 1–5 (2013) 15. Yang, F., Zhang, H., Pan, G.F.: Eye location based on adaptive image segmentation and curve blending. Opt. Precis. Eng. 21(12), 3255–3262 (2013) 16. Wang, Y., Piao, Y.: Computational reconstruction for integral imaging with sampled elemental images. Acta Optica Sinica 34(5), 70–75 (2014) 17. Di, B.H.: The Improvement of Digital Contents Creation Technology in Integral Imaging. Jilin University (2014) 18. Lyu, Y.Z.: Research on Generation, Coding and Display of Elemental Image Array in Integral Imaging System. Jilin University (2014)

Marker-Less 3D Human Motion Capture in Real-Time Using Particle Swarm Optimization with GPU-Accelerated Fitness Function Bogdan Kwolek1(B) and Boguslaw Rymut2 1

2

AGH University of Science and Technology, 30 Mickiewicza Av., 30-059 Krakow, Poland [email protected] Rzeszow University of Technology, 12 Powst. Warszawy, 35-959 Rzesz´ ow, Poland

Abstract. In model-based 3D motion tracking the most computationally demanding operation is evaluation of the objective function, which expresses similarity between the projected 3D model and image observations. In this work, marker-less tracking of full body has been realized in a multi-camera system using Particle Swarm Optimization. In order to accelerate the calculation of the fitness function the rendering of the 3D model in the requested poses has been realized using OpenGL. The experimental results show that the calculation of the fitness score with CUDA-OpenGL is up to 40 times faster in comparison to calculation it on a multi-core CPU using OpenGL-based model rendering. Thanks to CUDA-OpenGL acceleration of calculation of the fitness function the reconstruction of the full body motion can be achieved in real-time.

1

Introduction

Recovery of 3D human motion from images is an important problem in computer vision with many potential applications in areas such as interactive entertainment industry, sport or rehabilitation, interfaces for human-computer interaction, surveillance or augmented reality [14]. In general, the aim of 3D human motion recovery is to estimate the 3D joint locations of a human body from visual data. Marker-less motion tracking is one of the most challenging problems in computer vision being at the same time one of the most computationally demanding tasks. In recent years there has been a rapid progress in marker-less human pose recovery. Despite all these advances, 3D human motion reconstruction remains basically unsolved, particularly for unconstrained movement in dynamic and cluttered environments [16]. The challenge is not only to attain sufficient tracking accuracy, but also to develop solutions for real-time tracking [10, 22]. The existing approaches for 3D motion reconstruction can either be described as part-based bottom-up approaches or model-based generative top-down methods. The appearance based approaches rely on image features and a regressor responsible for determining the corresponding pose within a set of predefined c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 423–435, 2017. https://doi.org/10.1007/978-3-319-71598-8_38

424

B. Kwolek and B. Rymut

poses. Model based approaches define a parametric 3D model and search for the optimal solution in the model’s continuous parameter space. The 3D pose reconstruction is achieved by local optimization around the previous pose estimate or by Bayesian filtering. Bottom–up approaches require big amount of training data to capture the large appearance variation that arises due to the highly articulated structure of the human pose. Top–down approaches provide a more accurate estimation of the human pose. Such approaches typically utilize a 3D articulated human body model. They render the body model and then compare the rendered images with the acquired images to calculate the fitness score. The key challenge is to cope with high-dimensionality of the search space as a result of large number of degrees of freedom (DOF). Hierarchical [22] or global-local approaches divide the searching for the best fitness into multiple stages, where a subset of the parameters is optimized, whereas the rest of them are fixed. In recent years, many methods have been proposed for model-based motion tracking. Most work is based on particle filtering (PF) and its variants [2]. However, classical PF can be inefficient in high-dimensional state spaces since a large number of particles is needed to represent the posterior. More recently, stochastic global optimization methods such as Particle Swarm Optimization (PSO) have been gaining popularity in motion tracking [11, 14]. In PSO, unlike in PF, the particles interact with one another, and as they interact, a collective behavior arises, which leads to emergence of global and collective search capabilities allowing the particles to gravitate towards the global extremum. To date, relatively little work has focused on accelerating 3D model based motion tracking, and particularly on accelerating the objective function being the most computationally intensive operation. As shown in [20], the computation time per frame of the most methods is from a few to several seconds. As recently demonstrated, considerable gains in number of frames per second in object tracking can be obtained through GPU-based accelerating the evaluation of the objective function or accelerating the evaluation of particle weights [4,9]. Indeed, thanks to substantial shortening of time needed for evaluation of the objective function on a GPU, and particularly accelerating the rendering that can account more than 75% of total tracking time [3], the 3D motion reconstruction can be done in real-time with a frequency of several frames per second [10, 22]. This work is motivated by the need for acceleration of marker-less 3D motion reconstruction to achieve the tracking in real-time at high frame-rates. Our goal is to accelerate the 3D motion tracking through mixing graphics and compute. We present computation times of OpenGL-accelerated calculation of the fitness function on a multi-core CPU and a GPU. Computation times of the main ingredients of the fitness function are shown as well. We show that CUDAOpenGL interoperability allows us to reconstruct the 3D motion of the full body in a four camera setup with more than 12 Hz. We demonstrate accuracy of the human motion tracking on freely available datasets.

2

CUDA-OpenGL Interoperability

General-purpose computation on the GPU (GPGPU) is a term denoting the employment of the GPU for calculations other than rendering. CUDA is scalable

Marker-Less 3D Human Motion Capture in Real-Time

425

parallel programming model and a parallel programming platform for GPUs. It enables efficient use of the massive parallelism of NVidia GPUs. The CUDA architecture is built around a scalable array of multithreaded Streaming Multiprocessors (SMs). In this model, the host program launches a sequence of kernels, which are parallel portions of an application. A CUDA kernel is executed by an array of threads. All threads run the same code. The threads are organized into a grid of thread blocks. When a CUDA program on the host CPU invokes a kernel grid, the blocks of the grid are enumerated and distributed to multiprocessors. The threads of a thread block run concurrently on one multiprocessor, and multiple thread blocks can execute concurrently on one multiprocessor. As one thread blocks terminates, new blocks are launched on the available multiprocessors. All threads within a single thread block are allowed to synchronize with each other via barriers and have access to a highspeed, per-block shared memory, which allows an inter-thread communication. Threads from different blocks in the same grid can coordinate only via operations in a global memory visible to all threads. In Kepler GK110 microarchitecture, each SMx contains 192 CUDA cores (CCs), and at any one time they can execute up to six wraps of 32 threads. GK110 also supports up to 64 resident warps per SM. In CUDA memory model, the global, constant, and texture memory spaces are persistent across kernel launches by the same application. CUDA offers OpenGL interoperability features that allow to map OpenGL textures and buffers to be used inside a CUDA kernel without copying the whole content. The CUDA-OpenGL memory operations are described in [19].

3

Approach to Real-Time 3D Motion Tracking

The 3D model consists of truncated cones and each truncated cone is described by the center of top circle A, center of base circle B, top radius r1, and bottom radius r2. Figure 1 illustrates extraction of trapezoid representing the truncated cone in a given camera view. Having in disposal the 3D camera location C, 3D coordinates of the bottom circle center A and 3D coordinates of the top circle center B, and, we can determine the plane passing through the points A, B, C. Since the vectors AB and AC are in the same plane, their cross product being perpendicular to the plane, is the normal. The normal is used to calculate the angular orientation of the trapezoid. The projection of the trapezoid onto 2D image is performed on the basis of the camera model. The trapezoid image is generated by projecting the corners and then rasterizing the triangles composing the trapezoid. Though projecting every truncated cones composing of the 3D model we obtain the rendered image, which represents the 3D model in a given pose. In relevant literature the discussed technique is called billboarding [1]. In this work we also employed a more detailed 3D surface mesh model consisting of a surface representing the person skin and an underlying skeleton. Its surface has been deformed using a rigid skinning technique, in which the skeleton acts as a deformer transferring its motion to the skin, and where each skin vertex has assigned one joint as a driver.

426

B. Kwolek and B. Rymut

Fig. 1. Trapezoid representing truncated cone.

The pose of the model is described by position and orientation of the pelvis in the global coordinate system and the relative angles between the limbs. The geometrical transformations of each body part are represented by a local transform matrix and a global transform matrix. The local transform matrixes describe geometrical transforms of model parts in their local coordinate systems, whereas the global transform matrixes represent geometrical transformations in the global coordinate system [13]. The beginnings of the local coordinate systems are in joints, whereas the beginning of the global coordinate system is the center of scene. Transformation of each body part is performed on the basis of rotation matrices describing global and local transformation of the body part [13, 21]. The matrices are calculated on the basis of the joint Euler rotation angles stored in a state vector. In this paper, we formulate the 3D human pose tracking problem as a discrete optimization problem based on image matching in a calibrated multicamera system and solve this problem using PSO [8]. Each particle of the swarm represents a hypothesized pose. The goal of the PSO is to find the best matching between the projected 3D model on the basis of the camera models and the images acquired by the calibrated and synchronized cameras. The evaluation of the system performance has been carried out on publicly available LeeWalk [2] and HumanEva I [17] image sequences. Both image sequences contain the parameters of the Tsai camera model [18] as well as ground-truth data. The LeeWalk image dataset consists of images of size 640 × 480 from four monochrome cameras, whereas HumanEva I dataset contains images of size 640 × 480 from 3 color cameras and images of the same size from 4 monochrome cameras. Both datasets were recorded with 60 fps. Particle swarm optimization is a meta-heuristic that has been applied successfully to many optimization problems, including 3D human body pose tracking. In this meta-heuristic each individual is called particle and moves through a multidimensional space in order to find the best fitness score. The PSO algorithm seeks for a best fitness by iteratively trying to improve a candidate solution with regard to a given measure of quality. It optimizes a fitness function by maintaining a population of candidate solutions and moving these particles around in the search-space according to simple mathematical formulae over the particle’s position and velocity. Each particle’s movement is influenced by its local best known position, and is also guided toward the best known positions in the search-space. The best local and global positions are updated on the basis of the fitness function. The ordinary PSO algorithm can be used for human pose estimation in a single frame.

Marker-Less 3D Human Motion Capture in Real-Time

427

The 3D motion tracking over a sequence of frames can by attained by incorporating the temporal continuity information into the ordinary PSO. Thus, it can be achieved by a series of static PSO-based optimizations, followed by rediversification of the particles to predict the body poses in the next frame. The re-diversification of the particles can be achieved on the basis of normal distribution concentrated around the best pose found by the swarm. Figure 2 illustrates how the synchronous PSO has been decomposed for a parallel execution on the GPU. The algorithm is executed in the six steps: swarm initialization, update of particle velocity, update of particle position, update of particle best position, update of swarm best position and fitness function evaluation. Each stage of the algorithm has been parallelized and then executed on the GPU. The most time consuming part of the motion tracking is rendering of the 3D model and matching of such a projected model with image observations.

Fig. 2. Decomposition of synchronous PSO on GPU.

In this work, in order to speed-up the calculation of the fitness function the rendering of the 3D model is accelerated by OpenGL. The matching of the projected 3D models with the image observations is implemented in CUDA, whereas the transformation of the model to the requested poses as well as the rendering of the 3D model in the predicted poses is realized by OpenGL, see Fig. 3. The discussed framework is general and can be used in particle swarm optimizationbased [14] or particle filtering-based [2] 3D pose inferring. The processing performance of CUDA-OpenGL based fitness function has been compared with performance of fitness function, which was determined on the basis of software CUDA-based rendering of the 3D model.

Fig. 3. Mixing graphics and compute to speed-up human body tracking.

428

4

B. Kwolek and B. Rymut

Evaluation of the Fitness Function Using OpenGL API

At the beginning of this Section we present the objective function. Then, we discuss main steps in evaluation of the fitness score. Afterwards, we explain how OpenGL is used to accelerate the rendering of the 3D model. Then, we discuss the evaluation of the fitness score given the 3D models rendered by OpenGL. 4.1

Objective Function

The vision systems used in this work consist of two pairs of calibrated cameras, which are roughly oriented perpendicularly to each other. The objective function has been calculated in the following manner: F (x) =

2 

(fi (x))wi

(1)

i=1

where wi denotes a smoothing coefficient. The function f1 (x) reflects degree of overlap between the projected 3D model into 2D images and the extracted silhouettes on the images acquired by the cameras. It has been calculated as: 4 4 oc oc f1 (x) = β 4c=1 + (1 − β) c=1 4 ˆc ˆc c=1 o c=1 r

(2)

where oc expresses the overlap degree between the silhouette of the projected 3D model and the silhouette extracted in the image from the camera c, oˆc stands for the number of pixels belonging to the rendered silhouette for the camera c, rˆc denotes the area of the silhouette extracted in the image from the camera c, whereas the β coefficient has been set to 0.5 to equally weight the overlap degree from the rasterized silhouette to the image silhouette and from the image silhouette to the rasterized silhouette. The function f2 (x) expressing the distance map-based fitness between the edges has been calculated as follows: 4

f2 (x) = c=1 4

dc

c=1 ec

(3)

where ec denotes the number of edge pixels in the rendered image for the camera c, whereas dc reflects the sum of the distances from the edges extracted on the image from the camera c to the edges of the rendered model. The distances were calculated using the edge distance maps determined on the camera images. The fitness score expressed by (1) is determined through matching the features on a pair of images. The first image contains the rendered silhouette and edges of the model, whereas the second one contains the silhouette and the distance to the edges, which have been extracted on the images acquired by the calibrated cameras. The first image is a subimage of the frame-buffer, which contains images corresponding to all particles. As aforementioned, information about the silhouette and the edge is stored in a single byte, where the first bit represents the presence of the silhouette, whereas the last one represents the

Marker-Less 3D Human Motion Capture in Real-Time

429

occurrence of the edge of the projected model. In the second image the information is also stored in a single byte, where the first bit expresses the occurrence of the silhouette, whereas the seven remaining bits encode the distance of the pixels to the nearest edge. Given the extracted person silhouettes on the images acquired by the cameras a region of interest (ROI) is extracted. Each ROI is composed by enlarging the rectangle surrounding the extracted person about ten pixels both horizontally and vertically. Then, in order to permit 128-bit words transactions the left-upper coordinates of ROIs are shifted to be a multiple of four pixels. Finally, the width of every ROI is modified to be a multiple of four pixels. 4.2

Evaluation of Fitness Using Mixing Graphics and Compute

The evaluation of the fitness score is realized in three main steps: – calculation of the global transformation matrixes – 3D model rendering using OpenGL API – Calculation of the fitness function value In the first and third step, depending on the choice, CUDA or OpenGL or CPU is used to determine the global transformation matrixes needed by the OpenGL and to evaluate the fitness function given the rendered images by the OpenGL. 4.3

OpenGL-Based Rendering of the 3D Model

The rendering result is stored as the color attachment in the frame-buffer consisting of 32-bits RGBA pixel values, where each component is represented by single byte. This means that in a single RGBA pixel we can store the pixel values of the rendered model onto images of four cameras or the values of four models in different hypothetical poses. Having on regard that in the second case the rendered models are rather in similar poses in comparison to pose estimated in the last frame, we selected this method to store the all rendered models. In consequence, the rendered models, which are stored in four components of an RGBA image, are of similar size and shape. In the rendering of the 3D model, three programmable pipelines were executed: (i) vertices calculation, (ii) silhouette rendering and (iii) outline (edge) rendering. The pipelines were programmed by shader programs written in the GLSL language. The Vertex calculation pipeline uses geometry instancing [15] for creating instances of model vertices. The number of created instances is equal to number of particles multiplied by number of cameras used in the tracking. Each generated model instance is transformed using the world/global transformation matrices of a corresponding particle, which is stored in SSBO [15]. After calculating the model transformation, all vertices of an instance are projected into image coordinates using Tsai camera model. To render more than one model image on

430

B. Kwolek and B. Rymut

a frame-buffer each vertice of the model instance is shifted in different framebuffer region, using a method commonly known as image tiling [12]. Finally the computed vertices are stored in vertex feedback buffer object [15]. The vertices stored in this buffer are then used in the silhouette rendering and edge rendering pipelines. Vertex calculation pipeline uses vertex shader program and geometry shader program for creating instances, projecting model onto image and tiling model images (subimages) into frame-buffer. The silhouette rendering pipeline uses vertices stored in the vertex feedback buffer object (VFBO) to render particle models using triangles primitive on a frame-buffer. In this pipeline the element array buffer object (EABO) containing indexes of vertices stored in the VFBO is used to pull vertices in proper order and to render model appearance using triangles primitive. Silhouette rendering pipeline uses dummy vertex shader program and fragment shader program to pass the input vertices to hardware rasterization stage. Outline (edge) rendering pipeline is almost identical to silhouette rendering pipeline. The only notable difference is the use of adjusted triangle primitive as an input and geometry shader program. Geometry shader program processes adjusted each triangle primitive to detect edge between adjusted triangles by simplified streamlined method [6]. The geometry shader emits new line primitive when an edge between two triangles is detected. The emitted line primitive vertices are then passed to the next rendering stage and drawn on frame-buffer. Hence, the number of the rendered subimages is equal to the number of the particles and the size of each subimage is equal to the size of the camera image. The painting of the silhouettes is realized using color blending. The information about the silhouette and the edge is stored in a single byte, where the first bit represents the presence of the silhouette, whereas the last one represents the occurrence of the edge of the projected model. After the OpenGL synchronization, the frame-buffer is mapped to CUDA memory and then used by CUDA in computation of the fitness score. 4.4

Fitness Evaluation Given the 3D Models Rendered by OpenGL

The value of the fitness score (1) is determined in two kernels, where the first one calculates components oc , oˆc of function (2) and components dc , ec of function (3), whereas the second kernel uses the values of the components to determine the value of function (1). The main computational load is connected with calculation of the components oc , oˆc , dc and ec . Moreover, the size of ROIs changes as the person undergoing tracking moves on the scene, which in turn can lead to unequal computational burden. The first kernel is far more computationally demanding than the second one.

5

Experiments

The experiments were conducted on a PC computer equipped with Intel Xeon X5690 3.46 GHz CPU (6 cores), with 8 GB RAM, and NVidia GTX 780 Ti

Marker-Less 3D Human Motion Capture in Real-Time

431

Table 1. Execution times [ms] of the main components of the fitness function. #particles CPU [ms] World transformation matrixes

Sending world transf. matrixes to OpenGL

OpenGL rendering

Acquire rendering result from OpenGL

Mapping rendering result (memory mapping)

Evaluate fitness value

Overall time

Compute time (without acquisition, mapping and transmission)

CUDA [ms]

CPUOpenGL [ms]

CPU SSE- CUDAOpenGL OpenGL [ms] [ms]

96

0.21

0.85

0.21

0.17

0.83

304

0.52

0.92

0.52

0.45

0.77

512

0.91

0.93

0.91

0.76

0.82

96

-

-

0.26

0.25

0.26

304

-

-

0.31

0.30

0.28

512

-

-

0.34

0.31

0.30

96

-

-

1.48

1.48

1.40

304

-

-

2.81

2.74

2.71

512

-

-

3.92

3.89

3.81

96

-

-

63.37

62.88

-

304

-

-

201.47

212.91

-

512

-

-

309.53

318.27

-

96

-

-

-

-

0.40

304

-

-

-

-

0.34

512

-

-

-

-

0.38

96

26.13

13.87

14.86

6.72

1.50

304

92.03

22.44

50.06

18.96

2.22

512

176.32

36.98

79.23

30.40

2.92

96

26.34

14.72

80.18

71.50

4.40

304

92.55

23.36

255.17

235.36

6.30

512

177.23

37.91

393.93

353.63

8.23

96

26.34

14.72

16.55

8.37

3.73

304

92.55

23.36

53.39

22.15

5.70

512

177.23

37.91

84.06

35.05

7.55

graphics card consisting of 15 multiprocessors and 192 cores per multiprocessor. The card is equipped with 3072 MB RAM and 64 KB on-chip memory per multiprocessor. In the Kepler GK110 architecture, the on-chip memory can be configured as 48 KB of shared memory with 16 KB of L1 cache, or as 16 KB of shared memory with 48 KB of L1 cache. Table 1 presents execution times needed for determining the objective function values in the single iteration of PSO. The presented times were obtained by averaging times for determining the objective functions values during 3D pose

432

B. Kwolek and B. Rymut

estimation in all 450 frames of the LeeWalk sequence. In the discussed test the PSO executed 10 iterations and consisted of 96, 304 and 512 particles, respectively. As we already mentioned, the images in this four-camera test sequence are 480 pixels high and 640 pixels wide. The bottom part of Table 1 presents the overall processing time together with the compute time. The software rendering algorithms, which are responsible for rendering of triangles and edges, perform point inclusion tests for simple polygons [5]. The visible surfaces are determined using painter’s algorithm [7]. The processing on the CPU has been realized on four cores using OpenMP. As we can notice, the processing times obtained on the GPU are far shorter than processing times achieved on the CPU. As we can observe, the compute time on the CPU can be considerably reduced through the use of the SSE instructions. By comparing the overall times and compute times presented in the table, we can observe that CPU-OpenGL and CPU SSE-OpenGL have considerable overheads for data acquisition, mapping and transmission. Having on regard that the 3D model rendering is the most time consuming operation in model-based 3D motion reconstruction, the use of OpenGL leads to far more larger number of processed frames per second (fps). Thanks to effective utilization of the rendering power of OpenGL to render the 3D models in the predicted poses the motion tracking can be done in real-time at high frame-rates. The use of OpenGL-based 3D model rendering makes shorter the tracking time in comparison to software (CUDA) rendering, see also Fig. 4. It is also worth noting that for the small number of the particles, say up to 300, the tracking time of the algorithm with OpenGL-based rendering does not change considerably with the growing particle population.

Fig. 4. 3D Motion tracking time for OpenGL and soft (CUDA) rendering.

The fitness functions discussed above were utilized in a PSO-based algorithm for marker-less motion tracking. Its accuracy has been evaluated on the mentioned above LeeWalk test sequence. Figure 5 illustrates the tracking errors versus frame number. The tracking accuracy on LeeWalk image sequence is comparable to accuracy reported in [22].

Marker-Less 3D Human Motion Capture in Real-Time

433

Fig. 5. Tracking error [mm] vs. frame number on LeeWalk dataset.

Figure 6 illustrates the tracking accuracy that was obtained on LeeWalk image sequence. The larger the overlap between the projected model and the person silhouette is, the better is the 3D human pose estimate.

Fig. 6. 3D human body tracking on LeeWalk sequence.

Table 2 presents the tracking accuracies, which were achieved on LeeWalk and HumanEva I image sequences. The above mentioned results were achieved in ten independent runs of the PSO-based motion tracker with unlike initializations. The results presented in Table 3 show the average times that are needed to estimate the pose in a single frame using the PSO. It presents the average times Table 2. Average tracking errors [mm] on LeeWalk and HumaEva I. Average tracking error using PSO (10 it.) 96 part. 304 part. 512 part. LeeWalk

47.9 ± 12.5 43.3 ± 10.3 40.0 ± 11.7

HumanEva I 55.8 ± 21.3 53.5 ± 18.4 52.3 ± 19.1

434

B. Kwolek and B. Rymut

that were obtained on LeeWalk image sequence. As we can observe, the method relying on truncated cones and trapezoid has shorter times in comparison to mesh-based method. Due to noisy images the mesh-based method did not give noticeably better tracking accuracies. Table 3. Tracking times [ms] for different 3D models. Model type #vert. Number of particles 96 304 Trapezoid Mesh

6

60

75.7 ± 5.9

90.9 ± 7.2

250 500 1000 4000

79.1 ± 7.2 86.7 ± 7.0 90.7 ± 6.0 131.6 ± 7.6

103.5 ± 5.3 114.3 ± 4.9 128.8 ± 5.0 237.4 ± 5.8

Conclusions

In this work we demonstrated a framework for marker-less 3D human motion tracking in real-time using PSO with GPU-accelerated fitness function. We demonstrated that OpenGL-based rendering of the 3D model in marker-less human motion tracking allows us to considerably shorten the computation time of the objective function, which is the most computationally demanding operation. We evaluated the interoperability between CUDA and OpenGL, which permitted mixed compute and rendering acceleration. Thanks to such an acceleration the full-body tracking can be realized in real-time. Acknowledgment. This work was supported by Polish Ministry of Science and Higher Education under grant No. U-711/DS (B. Rymut) and Polish National Science Center (NCN) under research grant 2014/15/B/ST6/02808 (B. Kwolek).

References 1. Akenine-M¨ oller, T., Haines, E., Hoffman, N.: Real-Time Rendering, 3rd edn. A. Peters Ltd., Natick (2008) 2. Balan, A.O., Sigal, L., Black, M.J.: A quantitative evaluation of video-based 3D person tracking. In: International Conference on Computer Communications and Networks, pp. 349–356 (2005) 3. Cano, A., Yeguas-Bolivar, E., Munoz-Salinas, R., Medina-Carnicer, R., Ventura, S.: Parallelization strategies for markerless human motion capture. J. Real-Time Image Process. (2015) 4. Concha, D., Cabido, R., Pantrigo, J.J., Montemayor, A.: Performance evaluation of a 3D multi-view-based particle filter for visual object tracking using GPUs and multicore CPUs. J. Real-Time Image Process. (2015)

Marker-Less 3D Human Motion Capture in Real-Time

435

5. Feito, F., Torres, J.C., Urea-L´ opez, L.A.: Orientation, simplicity and inclusion test for planar polygons. Comput. Graph. 19(4), 596–600 (1995) 6. Hajagos, B., Sz´ecsi, L., Cs´ebfalvi, B.: Fast silhouette and crease edge synthesis with geometry shaders. In: Proceedings of the Spring Conference on Computer, pp. 71–76 (2012) 7. Hughes, J., Van Dam, A., McGuire, M., Sklar, D., Foley, J., Feiner, S., Akeley, K.: Computer Graphics: Principles and Practice. Addison-Wesley, Upper Saddle River (2013) 8. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of IEEE International Conference on Neural Networks, pp. 1942–1948. IEEE Press, Piscataway (1995) 9. Krzeszowski, T., Kwolek, B., Wojciechowski, K.: GPU-accelerated tracking of the motion of 3D articulated figure. In: Bolc, L., Tadeusiewicz, R., Chmielewski, L.J., Wojciechowski, K. (eds.) ICCVG 2010. LNCS, vol. 6374, pp. 155–162. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15910-7 18 10. Kwolek, B., Krzeszowski, T., Gagalowicz, A., Wojciechowski, K., Josinski, H.: Realtime multi-view human motion tracking using Particle Swarm Optimization with resampling. In: Perales, F.J., Fisher, R.B., Moeslund, T.B. (eds.) AMDO 2012. LNCS, vol. 7378, pp. 92–101. Springer, Heidelberg (2012). https://doi.org/10.1007/ 978-3-642-31567-1 9 11. Kwolek, B., Krzeszowski, T., Wojciechowski, K.: Swarm intelligence based searching schemes for articulated 3D body motion tracking. In: Blanc-Talon, J., Kleihorst, R., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2011. LNCS, vol. 6915, pp. 115–126. Springer, Heidelberg (2011). https://doi.org/10.1007/9783-642-23687-7 11 12. McReynolds, T., Blythe, D.: Advanced Graphics Programming Using OpenGL. Morgan Kaufmann Publishers Inc., San Francisco (2005) 13. Parent, R.: Advanced algorithms. In: Parent, R. (ed.) Computer Animation, pp. 173–270. Morgan Kaufmann, San Francisco (2002) 14. Saini, S., Rambli, D.R., Zakaria, M., Sulaiman, S.: A review on particle swarm optimization algorithm and its variants to human motion tracking. Math. Problems Eng. (2014) 15. Segal, M., Akeley, K.: The OpenGL Graphics System. A Specification, Version 4.3. Khronos Group (2013) 16. Shaheen, M., Gall, J., Strzodka, R., Van Gool, L., Seidel, H.P.: A comparison of 3D model-based tracking approaches for human motion capture in uncontrolled environments. In: Workshop on Appl. of Computer Vision (WACV), pp. 1–8 (2009) 17. Sigal, L., Black, M.J.: HumanEva: Synchronized video and motion capture dataset for evaluation of articulated human motion. Technical report CS-06-08, Brown University, Department of Computer Science (2006) 18. Song, L., Wu, W., Guo, J., Li, X.: Survey on camera calibration technique. In: International Conference on Intelligent Human-Machine Systems and Cybernetics, vol. 2, pp. 389–392 (2013) 19. Stam, J.: What every CUDA programmer should know about OpenGL. In: GPU Technology Conference (2009) 20. Yao, A., Gall, J., Gool, L.: Coupled action recognition and pose estimation from multiple views. Int. J. Comput. Vision 100(1), 16–37 (2012) 21. Zatsiorsky, V.: Kinematics of Human Motion. Human Kinetics (1998) 22. Zhang, Z., Seah, H.S., Quah, C.K., Sun, J.: GPU-accelerated real-time tracking of full-body motion with multi-layer search. IEEE Trans. Multimedia 15(1), 106–119 (2013)

Warping and Blending Enhancement for 3D View Synthesis Based on Grid Deformation Ningning Hu, Yao Zhao, and Huihui Bai(&) Institute of Information Science, Beijing Jiaotong University, Beijing 100044, China {yzhao,hhbai}@bjtu.edu.cn

Abstract. This paper proposes an efficient view synthesis scheme based on image warping, which uses grid mesh deformation to guide the mapping process. Firstly as the first contribution we use moving least squares algorithm to get the initial warping position of the reference image. And then as the second contribution a novel grid line constraint is added to the energy equation predefined in a typical image domain warping algorithm which is proposed by Disney Research. Finally, as the third contribution we propose an novel image blending method based on correlation matching to directly solve the stretch problem emerged in image border of the final synthesis result. Experimental results show that our proposed method can get a better visual quality just in image space only, which is a significant advantage compared to the state-of-art view synthesis method who needs not only the corresponding depth maps but also the additional depth information and camera intrinsic and extrinsic parameters. Keywords: Image Domain Warping
Depth image based rendering Moving least squares
Image blending

1 Introduction Multimedia technology has undergone an unprecedented evolution in the new century especially in the content display field. Nowadays stereo 3D (S3D) is becoming a mainstream consumer production in both cinema and home with its immersive 3D experience [1]. However, the indispensible glasses serve as an unbridgeable gap restricting its further popularization in daily life. Multiview auto stereoscopic display (MAD) supports the motion parallax viewing in a limited range with the superiority of glasses free [2]. Considering the existing device capabilities it is unrealistic to directly capture, store and transmit the huge data required by multiple views. Based on the novel 3D format multi view plus depth (MVD), C. Fehn proposed depth-image-based rendering (DIBR) as a view synthesis method, which can generate N-views from M-views (M < N) [3]. The accurate final projection location stems from the mapping process from pixel to world to image coordinate system using camera in-extrinsic parameters and depth maps. But the inaccurate estimation of depth map would lower the synthesis quality.

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 436–444, 2017. https://doi.org/10.1007/978-3-319-71598-8_39

Warping and Blending Enhancement for 3D View Synthesis

437

What’s more this discrete projection method can easily generate occlusion and disocclusion regions. To solve the problem exposed by DIBR, Disney Research developed a special view synthesis scheme called Image-Domain-Warping (IDW) [4] based on nonlinear disparity mapping algorithm [5], which conducts mapping process in image space directly. The original image is covered by a regular grid mesh GðV; E; QÞ with vertices V, edges E, quads Q [6]. It employs a global optimization to obtain a warping function which protects the spatial structure of saliency scenes by deforming homogeneous areas, which is widely used in image retargeting [8]. Because of the continuity mapping there are no holes in IDW as DIBR shows. Nonetheless, Lucas-Kanade (L-K) [11] applied to conducting stereo matching for feature points is a time consuming process. And there is an obvious stretch effect on image border of the synthesis view if the disparity is too large. Inspired by IDW, our paper proposed a simpler warping scheme which aims to reduce the computational complexity in the energy equation and deal with the stretch effect on the image border. SIFT [12] feature points is only used in stereo matching, where moving least squares is used to utilize every feature points for every grid vertices weighted to make full use of SIFT information. We also introduce a grid-line constraint [8] instead of extracting additional vertical edge-points. Finally, a novel blending algorithm is proposed to solve the stretch effect on the left or right image border. This novel view synthesis method enables to have a suitable visual quality with high efficiency.

2 Proposed Method To start with our illustration, it is necessary to give a brief explanation of the basic IDW theory. It is worth nothing that they cover the original image with a regular mesh G to reduce the solution space to grid vertices V in case of a too large system of linear equations. Differently from DIBR who uses dense depth estimation, IDW takes advantage of the sparse feature points P extracted from the reference image. These points are used to assign the supposed location in the synthesis image. Saliency map S is also used to guide the deformation to the non-saliency regions. An energy equation EðwÞ is defined that consists of three constraints: data term Ed , spatial smooth term Es and temporal smooth term Et . w represents the warped function for each grid vertex. Each term is weighted with a parameter: EðwÞ ¼ kd Ed ðwÞ þ ks Es ðwÞ þ kt Et ðwÞ

ð1Þ

By minimizing the energy function the warp defined at the regular grid vertices is computed out. For the non-grid positions, the warp is obtained by bilinear interpolation algorithm. Finally, a synthesis image is rendered using the calculated warp function. In this paper, based on IDW described above, we propose an improving view synthesis scheme as Fig. 1 shows. After extracting the feature points and saliency map from the input images, we use moving least squares (MLS) to get the warp position for each grid vertex in order to make full use of the feature points. We also add a grid line constraint to keep the grid line from over-bending. Iterative optimization is applied to

438

N. Hu et al.

obtain the final results. Image blending is dispensable to solve the stretch effect in the image border. Each step is stated in the following sections.

Fig. 1. Overview of the proposed method

2.1

Image Warping

MLS is a classical algorithm which is widely used in image information [13, 14]. A set of control points with its positions after deformation are previously assigned and then the mission is going to find out the exact location for other points of the image. In our scheme, as left and right views as input images our goal is to obtain the suitable w l and w r for the two reference images, respectively (hereinafter we use w as a unified statement). First sparse SIFT point set P whose outliers excluded by RANSAC [15] is obtained location info indicates the disparity between corresponding point pairs. We use this point set Pðpl ; pr Þ as our control points in MLS algorithm. For a view located in the middle of the two input views, its disparity with the left or right view is: d¼

pr

pl 2

ð2Þ

So the deformation location Qðql ; qr Þ can be calculated as: ql ¼ pl þ d

ð3Þ

qr ¼ pr

ð4Þ

d

Warping and Blending Enhancement for 3D View Synthesis

439

It is because our warp w is defined at grid vertices that we need to propagate the information of the feature points to the mesh vertices to achieve the final warp function. According to MLS, we need to solve for the best w for every grid vertex which guarantees to warp every p to q. The Ed constraints can be formulized as: Ed ðwÞ ¼

V X P X

fij jjwi ðpj Þ

qj jj2

ð5Þ

i¼1 j¼1

where pj and qj are the original and final feature points. fij is the weight factor for every pj distributing to vi : fij ¼

pj

1

ð6Þ

2 vi

Particularly, we apply the image deformation based MLS algorithm to get the initial warp function instead of putting it to the energy equation with other constraints. This decision not only makes full use of the efficient MLS, but also simplifies the large scale linear equations. The rigorous mathematical deduction is showed in [14]. In [2, 4, 16], besides SIFT points they introduce vertical edge-points which behave image structure well. L-K can estimate these disparities exactly but time-consuming. To protect the image spatial structure without an over time-consuming, we introduce a new grid line energy term to prevent the grid lines from serious bending, since the salient objects may occupy multiple connected quads [8]. It can retain the edge orientations well. Figure 2 gives a comparative result. Figure 2(a) suffer edge deformation serious, especially in the holder of the second ball. Meanwhile Fig. 2(b) keeps the holder from getting a bending deformation by adding the grid line constraint. We formulate the grid line constraint term as El ðwÞ ¼

X E

Sij wðvi Þ lij ¼

wðvj Þ v0i vi

v0j vj

lij ðvi

2 vj Þ

ð7Þ

ð8Þ

After the construction of the energy equation, the energy EðwÞ is minimized by solving the linear equations iteratively. The iterative process would not finish until the vertex movements compared to the previous one are smaller than 0.5. 2.2

Image Blending

Due to the different fields of the adjacent camera positions, stretch effect can occur on the right or left border if only left or right image is used for warping. This paper introduces a novel blending method which can employ the border information in the one synthesis image to cover the stretch region in the other synthesis image. The basic idea of the blending algorithm is illustrated in Fig. 3.

440

N. Hu et al.

(a)

(b) Fig. 2. Comparison of whether to add grid line constraint; (a) Without grid line constraint; (b) Grid line constraint added.

Fig. 3. Image blending process

Warping and Blending Enhancement for 3D View Synthesis

(a)

441

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Fig. 4. The Balloons results. The first three are from the DIBR method: (a) Left mapping results; (b) Right mapping results; (c) Overlay (a) with (b) in pixel level; The following two are from our proposed method: (d) Left warping result; (e) Right warping result; (f) Ground truth; (g) Partial enlarged region of (c); (h) Partial enlarged region of (d); (i) Partial enlarged region of (e). (Color figure online)

442

N. Hu et al.

(g)

(a)

(b)

(c)

(d)

(e)

(f)

(h)

(i)

Fig. 5. The champagne tower results. The first three are from the DIBR method: (a) Left mapping results; (b) Right mapping results; (c) Overlay (a) with (b) in pixel level; The following two are from our proposed method: (d) Left warping result; (e) Right warping result; (f) Ground truth; (g) Partial enlarged region of (c); (h) Partial enlarged region of (d); (i) Partial enlarged region of (e). (Color figure online)

Warping and Blending Enhancement for 3D View Synthesis

443

First two blocks are cut in the matched and matching images respective based on the relevant disparity relations. And a ceil C with the maximal gradient value is found out in block B. As a template C is slid in B0 , we need to find a region C 0 with the maximal correction value with C. The correlation level Corrs is calculated as:

Corrs ¼

n n P P

Cði; jÞ  C 0 ði; jÞ

i¼1 j¼1 n n P P

i¼1 j¼1

Cði; jÞ

2

n n P P

i¼1 j¼1

C 0 ði; jÞ

2

!1=2

ð9Þ

After finding the best matching block, we can stitch the block to the original synthesis results according to the location relations between the two blocks and the images.

3 Experimental Results In the experiment, we show two video sequences, one with the resolution of 1024  768, and the other 1280  960. The relevant mesh resolutions are 30  40 and 40  50, which are much smaller than IDW’s 180  100. In the blending process, we assign the cell’s size as 10  10. We accomplish the scheme according to our comprehension since we couldn’t find the IDW’s implementation. To evaluate the performance we use the traditional DIBR method as our comparison. The synthesized effects are shown in Figs. 4 and 5. From the figures it can be seen that the traditional DIBR would suffer blurring effect because of the overlay of the left and right synthesis views. However, our results have achieved better visual quality relying on the optimized energy equation, especially in some parts denoted by red rectangles.

4 Conclusion In this paper, we have proposed a simple but efficient method to synthesis middle view from S3D inputs based on IDW. The first contribution is to obtain the initial warping position by image deformation based on MLS theory. As the second one, we introduce a grid line energy term to our energy equation. Finally, we apply a novel image blending algorithm to solve the border stretch deformation. Our experimental results demonstrate that our proposed method can generate a synthesis image meeting human visual comfort. In the future, we could make a further research on IDW and would introduce depth map to our method for depth map has a strong representation for the spatial structure of the image. Acknowledgements. This work was jointly sponsored by the National Key Research and Development of China (No. 2016YFB0800404), the National Natural Science Foundation of China (No. 61210006, 61672087).

444

N. Hu et al.

References 1. Stefanosti, N., Lang, M., Smolic, A.: Image quality vs rate optimized coding of warps for view synthesis in 3D video applications. In: ICIP, Orlando, USA, pp. 1289–1292. IEEE, September 2012 2. Farre, M., Wang, O., Lang, M., Stefanoski, N., Hornung, A., Smolic, A.: Automatic content creation for multiview autostereoscopic displays using image domain warping. In: Jones, C. D., Smith, A.B., Roberts, E.F. (eds.) ICME, Barcelona, Spain, pp. 1–6. IEEE, July 2011 3. Fehn, C.: Depth-image-based Rendering (DIBR), compression and transformation for a new approach on 3D-TV. In: Stereoscopic Displays and Virtual Reality Systems XI, SPIE, San Jose, Canada, pp. 93–104, May 2004 4. Stefanoski, N., Wang, O., Lang, M., Greisen, P., Heinzle, S., Smolic, A.: Automatic view synthesis by image-domain-warping. IEEE TIP 22(9), 3320–3340 (2013) 5. Lang, M., Hornumg, A., Wang, O., Poulakos, S., Smolic, A., Gross, M.: Nonlinear disparity mapping for stereoscopic 3D. ACM TOG 29(4), 75 (2010). ACM, Seoul, South Korea 6. Wang, H., Zhang, X.P., Xiong, H.K.: Spatial-temporal coherence for 3D view synthesis with curve–based disparity warping. In: VICP, Valletta, Malta, pp. 177–180. IEEE, December 2014 7. Wolf, L., Guttmann, M., Cohen-Or, D.: Non-homogeneous content-driven video retargeting. In: ICCV, Rio de Janeiro, Brazil, pp. 1–6. IEEE, October 2007 8. Wang, Y., Tai, C., Sorkine, O., Lee, T.: Optimized Scale-and Stretch for image resizing. ACM TOG 27(5), 118 (2008). ACM, Singapore 9. Rubinstein, M., Gutierrez, D., Sorkine, O., Shamir, A.: A comparative study of image retargeting ACM TOG 29(6), 160 (2010). ACM, Seoul, South Korea 10. Zhang, G.X., Cheng, M.M., Hu, S.M., Martin, R.R.: A shape-preserving approach to image resizing. Comput. Graph. Forum 28(7), 1897–1906 (2009). Blackwell Publishing Ltd. 11. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI, Vancouver, Canada, pp. 285–289. Morgan Kaufmann Publishers Inc., August 1981 12. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004). Kluwer Academic Publishers 13. Levin, D.: The approximation power of moving least squares. Math. Comput. Am. Math. Soc, 64(224), 1517–1531 (1998) 14. Schaefer, S., McPhail, T., Warren, J.: Image deformation using moving least squares. ACM TOG 25(3), 533–540 (2006) 15. Fischler, M.A., Bollles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). ACM 16. Wang, O., Lang, M., Stefanoski, N.: Image domain warping for stereoscopic 3D applications. In: Emerging Technologies for 3D Video Creation, Coding, Transmission and Rendering, pp. 207–230, June 2013

A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm Based on Ship-Shaped Model Xin Wang, Chunyu Lin(&), Yi Gao, Yaru Li, Shikui Wei, and Yao Zhao Institute of Information Science, Beijing Jiaotong University, No. 3 Shangyuancun, Haidian District, Beijing, People’s Republic of China [email protected]

Abstract. 3D panoramic driving system provides real-time monitoring of a vehicle’s surrounding. It allows drivers to drive more safely without vision blind area. This system captures images of the surroundings by the cameras mounted on the vehicle, maps and stitches the collected images as textures to a 3D model. In this process, the key tasks are to construct a 3D surface model and to design an efficient and accurate texture mapping algorithm. This paper presents a ship-shaped 3D surface model with less distortion and better visual effects. Based on the ship-shaped model, a texture mapping algorithm is proposed, which can obtain the mapping relation between the corrected image and the 3D surface model indirectly by setting up a “virtual imaging plane”. The texture mapping algorithm is accurate and runs fast. Finally, this paper uses an improved weighted average image fusion algorithm to eliminate the splicing traces. Experiments show that the proposed algorithm based on ship-shaped model has better 3D panoramic effect. Keywords: Driver assistance systems Panoramic imaging
Image fusion


Ship-shaped model

1 Introduction Classical 2D panoramic driver assistance system provides a bird’s-eye view. This system uses six fisheye cameras to synthesize a seamless bird’s-eye view image of a vehicle’s surrounding [1]. An improved bird’s-eye view vision system using four fish-eye cameras is proposed in [2]. But this bird’s-eye view vision system has some problems, such as the distortions of non-ground-level objects [1] and limited view scope. 3D panoramic imaging technology can make up the above deficiencies of 2D panoramic system. Many papers have proposed this 3D panoramic imaging technology, which models the surrounding of a vehicle, and maps the corrected images collected by the cameras as textures to the 3D model to generate 3D panoramic image [3–5]. They all propose different models and texture mapping algorithms, but none of them is perfect. For example, there are unnecessary distortion in the bowl model [3] proposed by Fujitsu and the texture mapping algorithm proposed in [5] is only suitable for the model similar to sphere. © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 445–457, 2017. https://doi.org/10.1007/978-3-319-71598-8_40

446

X. Wang et al.

Through the study of panorama modeling, image texture mapping and image fusion algorithm, this paper presents a vehicle-mounted multi-camera 3D panoramic imaging algorithm based on a ship-shaped model. In this algorithm, we establishes a more reasonable 3D surface model, proposes a new algorithm to obtain the texture mapping relation between the corrected images and 3D surface model, and adopts an simple and effective fusion algorithm to eliminate the splicing traces. The algorithm proposed in this paper has following advantages. Firstly, the ship-shaped model has less distortion and better visual effect. Secondly, the texture mapping algorithm runs fast. Thirdly, image stitching area is seamless.

2 Process Overview In general, in order to capture images around a vehicle, four wide-angle cameras facing four different directions should be installed on the vehicle. Because of the distortion of the wide-angle cameras, calibration is needed to obtain intrinsic parameters and extrinsic parameters [6]. With the intrinsic and extrinsic parameters, the images captured by the wide-angle cameras can be corrected as undistorted images, which are called as corrected images in this paper. Because the wide-angle cameras have no sensors to record the depth of field, it is necessary to establish a virtual 3D surface model to simulate the three-dimensional driving environment. After the images captured by the four cameras is corrected, they are projected onto the 3D surface model according to the texture mapping relationship, and the 3D panoramic image is generated through image fusion. Therefore, the process of 3D panoramic imaging technology is roughly shown in Fig. 1.

Fig. 1. The process of 3D panoramic imaging technology

This paper will focus on 3D surface modeling, texture mapping algorithm, as well as image fusion. Hence, we assume that image acquisition and calibration are already ready.

A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm

447

3 3D Surface Modeling 3D surface modeling is to build a 3D surface to which the corrected images as texture is mapped to generate 3D panoramic image. Fujitsu first proposed a bowl model [3], while four different models are compared in [4]. A mesh three-dimensional model is presented in [5]. Through the research of the city road environment, this paper presents a ship-shaped model with low distortion and good visual effect. 3.1

Ship-Shaped Surface Model

According to the actual driving environment, this paper presents a 3D surface model like a ship, as shown in Fig. 2. The bottom of the model is elliptical and it extends upward at a specified inclination.

Fig. 2. Ship-shaped surface model

The general equation of the model is given as follows. Let (x, y, z) be a point on the surface model, then the equation for the bottom of the ship model is given in (1). x2 y2 þ 1 a2 b2

and z ¼ 0

ð1Þ

Where a and b are the values of the semi-major axis and semi-minor axis of the ellipse, whose values are generally based on the length and width of the vehicle, the aspect ratio of the control panel in the driving system, and the actual road conditions. The equation of the slope of the ship model is given in (2). x2 ð rð z Þ þ 1 Þ 2 a 2

þ

y2 ð rð z Þ þ 1 Þ 2 b 2

¼1

ð2Þ

The equation rðzÞ defines the slope of the model, which should be set elaborately to achieve the best visual effects. For example, suppose f ðaÞ ¼ 4a2 is the slope equation, which is the curve function of the section of 3D model on the xoz surface, as shown in Fig. 3. The advantages of the model are as follows. There is a curvature between the bottom plane and the slope, so that the transition between the bottom and the slope is smooth. With elliptical structure increasing the projected area of the images captured by the left and right cameras on the model, it reduces the distortion and is more in line

448

X. Wang et al.

with the human vision experience, especially when displaying the 3D object on left or right side of the car, it looks more real.

Fig. 3. The sectional view of ship-shaped model

4 The Texture Mapping Algorithm Texture mapping maps the source image as texture onto a surface in 3D object space [7]. The main goal of the texture mapping is to determine the correspondence relation between the spatial coordinates of three-dimensional model and two-dimensional texture plane coordinates. A texture mapping algorithm based on the principle of equal-area is proposed in [5] ,which is applicable to the model similar with sphere. We present a texture mapping algorithm to obtain the mapping relation between the corrected images and the 3D surface model by setting up a virtual imaging plane. The algorithm is applicable to any shape of 3D surface model, including the ship model. Firstly, the pixel mapping relationship between the 3D surface model and the virtual imaging plane is set up. Then, the pixel mapping relationship between the virtual imaging plane and the corrected image is derived. Finally, the texture mapping relationship of the 3D surface model and the corrected images is obtained by combining the above two steps. 4.1

The Mapping Relation Between the 3D Model and the Virtual Imaging Plane

As shown in Fig. 4, a virtual imaging plane is set between the camera and the 3D model. For any point P ðx; y; zÞ on the 3D model, the corresponding point P0 ðx0 ; y0 ; z0 Þ can be derived from the perspective projection model [8]. Perspective projection is the process where the three-dimensional objects within the field of view of the camera are projected to the camera imaging plane. In general, the shape of the perspective projection model is a pyramid with the camera as the vertex, and the pyramid is cut by two planes. The plane near the camera is called near plane, and the far is called far plane. Only the objects between the two planes is visible to the camera. In the so-called perspective projection, the three-dimensional objects between the two planes are projected to the near plane. This projection relationship can be expressed by a four-dimensional matrix, called a perspective projection matrix. A commonly used perspective projection matrix is shown in (3).

A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm

449

Fig. 4. Perspective model of 3D model

2

1

6 tanð Þ
g 6 0 T¼6 6 4 0 0

3

0

0

0

1 tanðh2Þ

0

7 0 7 7 7 2fn 5 f n 0

h 2

0 0

f þn f n

1

ð3Þ

Where h is the maximum angle of the camera’s field of view, η is the aspect ratio of the projection plane (i.e., near plane), f is the distance between the far plane and the camera, and n is the distance between the near plane and the camera. Taking an arbitrary camera on the car as example, a virtual imaging plane is set between the camera and the 3D model. The process of projecting the pixels from the 3D model to the virtual imaging plane can be regarded as a perspective projection. For any point on the 3D model, the corresponding point on the virtual imaging plane can be obtained by the perspective projection matrix. Taking the front camera as example, establish three-dimensional coordinate system with the camera as the origin, as shown in Fig. 4. Assuming that the maximum angle of the camera’s field of view is h0 , the distance between the virtual imaging plane and the camera is n0 , the aspect ratio of the virtual imaging plane is l0 , the distance between the far plane and the camera is f0 , point Pðx; y; zÞ is on the 3D model, and the corresponding point P0 ðx0 ; y0 ; z0 Þ is on the virtual imaging plane, then the correspondence between P and P0 is given in (4). 2 1
h0 3 x0 6 tan 2
g0 0 6 y0 7 6 4 0 5¼6 6 z 4 0 w0 0 2

0

0

1 tan

h0 2

0 0




0 f0 þ n0 f0 n0

1

0

3

2 3 7 x 0 7 y7 76 74 5 2f0 n0 5 z 1 f0 n0 0

ð4Þ

Similarly, other three cameras also has the corresponding relationship as given in (4). Therefore, Eq. (4) can be extended as Eq. (5).

450

X. Wang et al.

Fig. 5. Coordinate system Ocenter

2 6 6 4

x0i y0i z0i w0i

3

2

6 7 6 7¼6 5 6 4

tan

1
hi 2
gi

0 0 0

0

0

1 tan

hi 2

0 0




0 fi þ ni fi ni

1

0

3 2 3 7 xi 0 7 yi 7 76 74 5 2fi ni 5 zi 1 fi ni 0

ð5Þ

where i = 0, 1, 2, 3, is respectively denotes the front camera, rear camera, left camera
and right camera. x0i ; y0i ; z0i and ðxi ; yi ; zi Þ are the positions in the coordinate system with the camera corresponding to i as coordinate origin. Equation (5) corresponds to four different coordinate systems. In order to unify the coordinate system, as shown in Fig. 5, we convert the points in the coordinate system denoted by Ocenter into those in the corresponding camera coordinate system. The process of converting the points in the coordinate system Ocenter to the corresponding point in the corresponding camera coordinate system is to make the points rotate boi along the z axis, then rotate boi along the x axis, and finally move along the ! translation vector denoted by Fi ¼ ðXi ; Yi ; Zi Þ. aoi is the angle between the camera and o the horizontal plane, bi is the angle between the camera and x axis, ðXi ; Yi ; Zi Þ is the coordinate in the coordinate system Ocenter , aoi and ðXi ; Yi ; Zi Þ are related to the position and orientation of the camera on the car, the values of boi are 0o , 180o , 270o , and 90o . The values of i is 0, 1, 2, 3 and they respectively correspond to the front camera, rear camera, left camera and right camera. According to computer graphics knowledge [8], the

above rotation relations can be o o expressed by the rotation matrices Rz bi and RX ai , and the translation relation can 2 3 sin boi 0 0 cos boi
6 sin boi cos boi 0 0 7 7, be expressed by the translation matrix TðXi ;Yi ;Zi Þ . Rz boi is 6 4 0 0 1 05 0 0 0 1 2 2 3 3 1 0 0 Xi 1 0 0 0 6 7
6 0 cos aoi sin aoi 0 7 7, and TðX ;Y ;Z Þ is 6 0 1 0 Yi 7. Combining RX aoi is 6 o i i i 4 0 0 1 Zi 5 40 sin ai cos aoi 0 5 0 0 0 1 0 0 0 1 Eq. (5) with the above rotation and translation matrices, the mapping relation between

A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm

451

Fig. 6. Perspective transformation

the pixel points on the 3D surface model and the pixel points on the virtual imaging plane is given in (6). 2 1
3 hi x0i 6 tan 2
gi 0 6y 7 6

6 0i 7 = RZ bo RX ao T ðXi ; Yi ; Zi Þ6 0 i i 4z 5 6 i 4 0 w0i 0 2

4.2

0

0

1 tan

hi 2

0 0




0 fi þ ni f i ni

1

0

3

2 3 7 xi 0 7 yi 7 76 74 zi 5 ð6Þ 2fi ni 5 1 fi ni 0

The Mapping Relation Between the Corrected Image and the Virtual Imaging Plane

The corrected image and the image on the virtual imaging plane can be viewed as images of the camera in two different planes as shown in Fig. 6, which is also known as perspective transformation [9]. In general, the pixel mapping relation of the perspective transformation can be expressed by the following formula (7) ( x ¼ aa1131uuþþaa1232vvþþa113 y ¼ aa2131uuþþaa2232vvþþa123

ð7Þ

where ðu; vÞ is the pixel coordinate in the original imaging plane and ðx; yÞ is the pixel coordinate in the new imaging plane. It can be seen from the Eq. (7) that the pixel mapping relationship of the perspective transformations contains eight unknown parameters. Therefore, four points on the original imaging plane and their corresponding points on the new imaging plane are required to calculate the unknown parameters. As shown in Fig. 7, we make the position of the calibration boards fixed. Therefore, the position on the 3D model of the 16 feature points, which are on the calibration boards, can be determined when designing the 3D model. Assuming that the

452

X. Wang et al.

Fig. 7. Calibration board layout

coordinates of the feature point P1 on the 3D model are ðx1 ; y1 ;
z1 Þ, the coordinates of P1 on the virtual imaging plane of the front camera are x01 ; y01 by Eq. (6). The pixel coordinates of P1 in the corrected image
of the front camera can be obtained by feature point detection [10], which is x001 ; y001 . The same procedure can be easily adapted to obtain the coordinates of these feature points which are P2 , P7 and P8 both on the virtual imaging plane and in the corrected image. Using these four points (P1 , P2 , P7 and P8 ), the pixel mapping relationship between the front corrected image and its virtual imaging plane can be obtained. The same procedure applies to the other three cameras. 4.3

Texture Mapping Between the Corrected Images and the 3D Model

Through the above two steps, the texture mapping relation between the 3D model and the corrected images can be obtained. When generating 3D panoramic image in real time, the delay will be large if calculating each pixel value on the 3D model through the mapping relation. In order to reduce delay, as shown in Fig. 8, the coordinates of the pixels on the 3D model and it in the corrected image can be stored in the memory in one-to-one corresponding order when the system initialized. When the 3D panoramic image is generated in real time, the corresponding texture coordinates of 3D model can be read from the memory where the texture coordinates are stored, and then the texture on the corrected images is mapped to the 3D surface model to generate a 3D panoramic image.

A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm

453

Fig. 8. The process of texture mapping

5 Image Fusion If the panoramic image generated by texture mapping has no fusion processing, there will be a “ghosting” phenomenon in the overlapping areas, as shown in Fig. 9. Therefore, it is necessary to fuse the overlapping areas of the panoramic image.

Fig. 9. The panoramic image before image fusion

In this paper, we improve the weighted average method [11] so that the weight of the texture of the corrected images is proportional to the distance from the texture coordinate to the mapping boundary of the corrected image. This method is simple and has good fusion effect. In the example of the overlap between the front and the right, as shown in Fig. 10, the overlapping area has two boundary lines L and M, L is the mapping boundary of the front corrected image, M is the mapping boundary of the right corrected image. Whalf is half of the car width, and Lhalf is half of the car length. Here, the distance from the texture coordinate to the mapping boundary can be expressed by its angle relative to

454

X. Wang et al.

Fig. 10. Image fusion diagram

the mapping boundary. Therefore, according to the fusion algorithm, the texture of the point A (x, y, z) is CA , which can be obtained by Eq. (8). CA ¼ wfront ð/Þ
Cfront þ wright ð1

/Þ
Cright

ð8Þ

Where Cfront and Cright represent the texture of the corrected images, wfront and wright are wright ð1 /Þ ¼ 1 Wfront ð/Þ, and fusion weight, wfront ð/Þ ¼ a ho hr ,   j yj whalf a ¼ arctan jxj lhalf .

6 Experimental Results and Discussion Using the fusion algorithm described in Sect. 5, the panoramic image obtained by the processing flow described in Sect. 2 is shown in Fig. 11.

Fig. 11. Panoramic image

6.1

Effect of Image Fusion

Figure 12(a) is part of the fusion area that did not use the fusion algorithm, and Fig. 12 (b) is part of the area that uses the fusion algorithm. It can be seen from the figure that the fusion algorithm adopted in this paper basically eliminates the seam of the stitching position.

A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm

a)

455

b)

Fig. 12. Comparison of the effect before and after using fusion algorithm

6.2

Comparison of the Effect of Different Models

At present, the earliest proposed 3D surface model is the bowl-shaped model proposed by Fujitsu, which is shown in [12]. Compared with the bowl-shaped model, the ship-shaped model proposed in this paper has the advantages of less distortion and better visual effect because of considering the driving environment and the visual experience. At first, the ship-shaped structure of the 3D model proposed in this paper can eliminate unnecessary distortion. Paying attention to the lane line of the highway in Fig. 13, the lane line becomes a broken line in Fig. 13(a), while it is almost a straight line in Fig. 13(b).

a)

b)

Fig. 13. Comparison of the distortion between bowl-shaped model and ship-shaped model: (a) the image of the bowl-shaped model and (b) the image of the ship-shaped model.

In addition, the ship-shaped model can simulate the driving environment better and has better visual effects. As shown in Fig. 14, the car on the road is projected to the bottom plane of the bowl model, making people feel that the other vehicle and the vehicle model are not on the same pavement. Considering the road environment, the objects on both sides of the road are projected onto the 3D surface of the ship-shaped model. Therefore, the ship model can better display 3D objects on both sides of the road, which reduces the driver’s misjudgment.

456

X. Wang et al.

a)

b)

Fig. 14. Comparison of the visual effect between bowl-shaped model and ship-shaped model: (a) the image of the bowl-shaped model and (b) the image of the ship-shaped model.

6.3

Performance of Texture Mapping

The above panoramic imaging algorithm is implemented by using the Linux operating system based on kernel version 3.10 and the hardware platform of the i.mx6Q series cortex-A9 architecture 4-core processor. On the experimental platform, the frame rate of the panoramic imaging algorithm is tested in both fixed viewpoint and moving viewpoint respectively, and both frame rates meet the minimum requirements, as is shown in Fig. 15. The experimental results show that the texture mapping algorithm proposed in this paper has high performance and can meet the requirements of real-time display of vehicle display equipment.

Fig. 15. Results of texture mapping performance test

7 Conclusion This paper presents a 3D panoramic imaging algorithm. The contributions mainly includes three aspects: 3D surface modeling, texture mapping algorithm and image fusion. In the aspect of 3D modeling, this paper proposes a ship-shaped model that has

A Vehicle-Mounted Multi-camera 3D Panoramic Imaging Algorithm

457

low distortion and better visual effect. In the aspect of texture mapping algorithm, this paper presents a method to calculate the texture mapping relation by setting up a virtual image plane between the camera and the 3D model. Because the mapping relation are stored in memory after initial run, the algorithm runs fast and meets the requirements of real-time imaging. In the aspect of image fusion, this paper adopts a special weighted average image fusion algorithm, which effectively eliminates the splicing traces of the fusion region. The experimental results prove the feasibility and effectiveness of the proposed algorithm. Acknowledgment. This work was supported in part by National Natural Science Foundation of China (No. 61402034, Nos. 61210006 and 61501379), and the Fundamental Research Funds for the Central Universities (2017JBZ108).

References 1. Liu, Y.-C., Lin, K.-Y., Chen, Y.-S.: Bird’s-eye view vision system for vehicle surrounding monitoring. In: Sommer, G., Klette, R. (eds.) RobVis 2008. LNCS, vol. 4931, pp. 207–218. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78157-8_16 2. Yu, M., Ma, G.: 360 surround view system with parking guidance. SAE Int. J. Commercial Veh. 7(2014-01-0157), 19–24 (2014) 3. Huebner, K., Natroshvili, K., Quast, J., et al.: Vehicle Surround View System: U.S. Patent Application 13/446,613[P], 13 April 2012 4. Yeh, Y.-T., Peng, C.-K., Chen, K.-W., Chen, Y.-S., Hung, Y.-P.: Driver assistance system providing an intuitive perspective view of vehicle surrounding. In: Jawahar, C.V., Shan, S. (eds.) ACCV 2014. LNCS, vol. 9009, pp. 403–417. Springer, Cham (2015). https://doi.org/ 10.1007/978-3-319-16631-5_30 5. Liu, Z., Zhang, C., Huang, D.: 3D model and texture mapping algorithm for vehicle-mounted panorama system. Comput. Eng. Design 38(1), 172–176 (2017). (in Chinese) 6. Zhang Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000) 7. Heckbert, P.S.: Survey of texture mapping. IEEE Comput. Graphics Appl. 6(11), 56–67 (1986) 8. Angel, E.: Interactive computer graphics: a top-down approach with OpenGL, with OpenGL primer package, 2/E. In: DBLP (1997) 9. Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV library. O’Reilly Media, Inc., Upper Saddle River (2008) 10. Jiang, D., Yi, J.: Comparison and study of classic feature point detection algorithm. In: International Conference on Computer Science and Service System, pp. 2307–2309. IEEE Computer Society (2012) 11. Rani, K., Sharma, R.: Study of different image fusion algorithm. Int. J. Emerg. Technol. Adv. Eng. 3(5), 288–291 (2013) 12. Wrap-Around Video Imaging Technology - Fujitsu United States. http://www.fujitsu.com/ cn/about/resources/videos/fss-video/360.html. Accessed 28 May 2017

A Quality Evaluation Scheme to 3D Printing Objects Using Stereovision Measurement Li-fang Wu ✉ , Xiao-hua Guo, Li-dong Zhao, and Meng Jian (

)

Beijing University of Technology, Beijing 100124, China [email protected], [email protected]

Abstract. The paper presents a comprehensive evaluation method on shape consistency by using three-dimensional scanning, reverse engineering and postprocessing. The complete evaluation scheme includes data collection, model alignment and the quality consistency evaluation. Firstly, the point cloud data is obtained by using 3D scanning. Secondly, the printed object and the model are aligned, and we also get the visual point-to-point deviation. Thirdly, some param‐ eters of surface roughness are defined to evaluate comprehensive quality for personalized 3D printed object. Two printed objects from FDM and DLP printer are utilized to test the proposed scheme. The experimental results show that DLP printing is more precise than FDM printing, and it is consistent with the common sense. And it also demonstrates that the efficiency of the proposed scheme to some extent. Keywords: Product inspection · Quality evaluation · 3D printing · Reverse engineering

1

Introduction

With the explosion development of 3D printing technology, it is rapidly applied to a lot of areas such as aerospace, mechanical engineering, civil engineering, architecture, art design, medicine, geographic information systems and so on [1, 2]. Higher requirements are raised for the quality of 3D printed objects. Effective quality inspection technology and evaluation scheme for 3D printed objects are urgently needed. However the popular methods are visual observation qualitatively by now. And there is not a quantitative method and standard. On the other hand, due to the personality of 3D printed products, it is not reasonable to adopt the same quality inspection technology used for products of mass manufacturing [3–5]. In industry manufacturing area, surface roughness is mostly used to evaluate the local geometry of products from variant mechanical manufacturing means, such as boring, milling and so on [6]. In the 21st Century, the product quality assessment level has been improved much in manufacturing industry. In 1929, Schmalz proposed the This paper is supported by the Beijing Municipal Education Commission Science and Technology Innovation Project (JB002012201601). © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 458–467, 2017. https://doi.org/10.1007/978-3-319-71598-8_41

A Quality Evaluation Scheme to 3D Printing Objects

459

concept of the key parameter Hmax and measurement benchmark. And then, E. J. Abbott and Taylor-Hobson, a British company, developed the measuring instrument to evaluate the surface roughness of the work piece. These methods have been proposed for products from variant mechanical processing technologies. The features of these products are in global consistency [7]. In comparison, the 3D printed objects have not such character‐ istic, as a result, these existing methods are not suitable for 3D printed objects. In fact, quality inspection for 3D printed objects includes the shape consistency, the surface roughness, the interior compactness, the resistance to different types of exerted forces, stiffness and so on. In this paper, we focus on the shape consistency inspection. Because the shape is visible, we propose a method based on stereovision. Firstly, we obtain the point cloud data of the printed object using the stereovision. Then the point cloud data and the model are aligned. And the shape deviation could be computed. Finally some parameters of the shape deviation are computed referring to the Chinese national standard (GB/T 1031-2009). The contributions of the paper include as follows: (1) we propose a framework to evaluate the quality of the 3D printing objects. (2) we present the quality evaluation for 3D printing objects combined with Chinese national standard.

2

The Proposed Scheme

The proposed framework is shown in Fig. 1. Obtaining the point cloud data

Aligning the point cloud data with the 3D model

Calculating the shape deviation

Calculate the surface roughness

Fig. 1. Framework of the proposed scheme

Each 3D printed object is printed on the basis of 3D model, therefore, the key issue of quality inspection is to evaluate the consistency between printed object and its corresponding 3D model [8]. These existing methods include comparative method, impression method, stylus method, light cutting method, real-time holographic method and so on. Although these methods can detect the product surface convexity, they are not capable of giving the accurate deviation between the printed objects and

460

L. Wu et al.

the 3D models [9]. Based on the above reasons, the quality inspection of 3D printed object is mainly composed of four parts. They are obtaining the point cloud data, aligning 3D scanned data and the model, calculating the shape deviation, computing the values of some parameters for roughness (Fig. 2).

Fig. 2. Illustration of the 3D scanner

2.1 Obtaining the Point Cloud Data In this paper, the 3D scanner is composed two cameras, a rotation platform and control software. From images of the two cameras, the 3D coordinates of matched points could be computed. Using the rotation platform, images of the object on different views could be captured automatically. The point cloud data of the object could be obtained completely. The object and the corresponding point cloud data is illustrated in Fig. 3.

(a) The printed object

(b) The point cloud data

Fig. 3. The object and corresponding point cloud data

2.2 Aligning the Point Cloud Data with the 3D Model In order to evaluate the consistency between the printed object and the 3D model, it is crucial to align them firstly. Before the alignment, it is important to reduce properly corresponding point cloud data to meet software processing requirements for some larger printed object. Data reduction could be implemented by sampling the scanned point cloud data. The sampling methods generally comprise unified sampling, curvature sampling, grid sampling and random sampling.

A Quality Evaluation Scheme to 3D Printing Objects

461

Unified sampling is characterized by uniform sampling of all points. The character‐ istic of curvature sampling is that a small number of points are kept in the flat area, but a great number of points are kept for high curvature of region, so that the detail could be reserved better. Grid sampling is to create an even point set to reduce the number of ordered points, ignoring curvature and original density. Random sampling is randomly removed a certain percentage of points from the disordered points. According to the surface characteristics of model, we could select different sampling styles in practical application. In this paper, the unified sampling method and curvature sampling method are adopted. The model alignment method could be selected in accordance with different constraint strategies of alignment. The general strategies include the optimal fitting alignment, the feature based alignment, RPS alignment and so on [10]. The optimal fitting alignment could result in the minimum deviation globally. The feature based alignment could result in the minimum error of some specific shapes such as circle, cylinder, rectangle and so on. The RPS alignment is to reach the agreement between its coordinate system with reference coordinate system. As is well known, the 3D printed objects have no unified feature and arbitrary shape, therefore the optimal fitting align‐ ment method is utilized in this paper. After the alignment, the printed object and the 3D model data are placed in a unified coordinate system. 2.3 Calculating the Shape Deviation In the unified coordinate system, the deviation is calculated point by point. Assuming the point on the 3D model is represented as p1(x1,y1,z1) and the corresponding point on the printed object is represented as p2(x2,y2,z2). The deviation d could be computed by Eq. (1). If d is zero, it means that the coordinates of two points are identical, and there is not shape deviation for these points.

|d| =

√

(x2 − x1 )2 + (y2 − y1 )2 + (z2 − z1 )2

(1)

The positive and negative of deviation d are determined by relative positions of the two points. The deviation for the printed object in Fig. 3 is shown in Fig. 4. Furthermore, we could obtain the deviation histogram as shown in Fig. 5. Finally, some parameters of the deviations could be defined and calculated by the mechanic inspection standards [11]. There are six specifications related to the shape. They are straightness, flatness, roundness, cylindricality, line type and surface profile. Taking into account the diversity of printed objects, we need to evaluate the quality of entire printed objects. Therefore, we proposed the evaluation parameters to evaluate the shape consistency between the object and the model. They are the maximum deviation, minimum deviation, average deviation and variance of deviation. It is not difficult to understand that the smaller the evaluation value is, the smaller the deviation is.

462

L. Wu et al.

Fig. 4. Illustration of deviation for the printed object in Fig. 3

Fig. 5. Histogram of the printed object’s deviation

2.4 Calculate the Surface Roughness There are macroscopic and microscopic features in object shape quality. The macro‐ scopic feature could be represented by shape deviation, and the microscopic feature should be expressed by surface roughness. The Chinese national standard, GB/T 1031-2009 Product Geometry Technical Specification (GPS) for Surface Structure Profile Method and Surface roughness parameters and values, specifies that the surface roughness of a object could be expressed by calculating surface profile characteristics of the object. The descriptions of specific parameters are as follows [12]. Ra is arithmetic mean deviation. It could be calculated by averaging the absolute deviations in the measuring range. In general, the bigger the value of Ra is, the greater the surface roughness is. For example, automatic gas cutting and saw or disc sawing are relatively rough manufacturing technologies, in consequence, the value of Ra is gener‐ ally 12.5–50 um. In comparison, the value of Ra is always 0.1–0.4 um for the finer turning cylindrical (precision car, diamond) processing technology.

A Quality Evaluation Scheme to 3D Printing Objects

463

The second parameter Rz is maximum deviation by averaging five maximum devi‐ ations in the measuring range. Rz represents the maximum height of microscopic rough‐ ness for profile. The larger the value of Rz is, the less evenness the surface is. It is easy to find that the above criterion is used to evaluate the surface roughness of machined objects. And these objects are globally consistent and the surface roughness of object is approximately the same in any position of the machined object, in conse‐ quence, the surface roughness of entire object can be represented well by the charac‐ teristic parameters in a local area. In contrast, local region could not represent the global object in the 3D printed object because of random molding deviation on different layers in molding process. Referring to the standard of GB/T 1031-2009, for personality of 3D printed objects, roughness parameters of the objects are calculated to obtain the quality evaluation based on vision in the paper. The surface roughness parameters of printed object are evaluated synthetically on the basis of arithmetic mean deviation Ra between printed object and model, maximum deviation Rz and standard deviation SD of Ra represented in Eq. (2). n

Ra =

1 ∑| | y n i=1 | i |

(2)

Where, yi indicates the deviation of corresponding points from printed object and model [13]. Standard deviation SD is the standard deviation of deviation, which is also a key parameter on estimating dispersion of deviation. There would be a comprehensive assessment for surface roughness by adopting three parameters.

3

Experiments

In order to verify the feasibility of the proposed evaluation method, the objects are printed using digital light processing (DLP) printer and fused deposition modeling (FDM) printer respectively. Figure 6 shows the model and the corresponding printed objects. As we can

(a)

(b)

(c)

Fig. 6. Examples of different types of printings; (a) model 1; (b) The printed object using DLP; (c) The printed object using FDM

464

L. Wu et al.

see the surface of the object from FDM printer occasionally has some bulges while that from DLP printer is relatively smooth. 3.1 Qualitative Evaluation Figure 7 shows shape deviation distribution of two objects in Fig. 6. And the deviation histograms for two printed objects in Fig. 7 are shown in Fig. 8. It can be seen that deviation distribution from printed object on DLP is more concentrated, nevertheless, it is more dispersed for the one on FDM. For the object from DLP, the deviations of about 19% points are smaller than 0.05 mm, and those of about 44% points are smaller than 0.15 mm. While for the object from FDM, they are 9% and 27% respectively. Furthermore, the deviations in the range of [−0.8 mm, 0.3 mm] incude more than 2% points in the object from DLP, while the deviation range is [−1.1 mm, −0.5 mm] in that from FDM.

(a) From DLP

(b) From FDM

Fig. 7. Comparison of deviation from two 3D printed objects

Based on the above analysis, it could be concluded that precision of DLP printer is higher than the FDM printer. It is also consistent with the public knowledge.

A Quality Evaluation Scheme to 3D Printing Objects

465

(a) From DLP

(b) From FDM Fig. 8. The deviation distribution for objects from DLP and FDM

3.2 Quantitative Evaluation In these section, the maximum deviation, minimum deviation, mean deviation, variance, the roughness quantitative parameters Ra, Rz and standard deviation SD of Ra to evaluate surface roughness are computed respectively. Three objects are printed from the models in Figs. 6(a) and 9 respectively. The results are shown in Table 1.

466

L. Wu et al.

(a) model 2

(b) model 3

Fig. 9. Two models

Table 1. Quantitative evaluation of the printed objects From FDM model1 model2 Over printed (%) 27.92 31.65 Under printed (%) 72.08 68.35 Mean deviation (mm) −0.51 −0.27 Variance of deviation 0.82 0.38 0.81 0.50 Ra (mm) Rz (mm) 5.08 3.80

model3 22.83 77.17 −0.27 0.53 0.56 6.93

Ave 27.47 72.53 0.35 0.58 0.62 5.27

From DLP mode1 mode2 34.18 28.97 65.82 71.03 −0.27 −0.05 0.45 0.05 0.51 0.13 4.62 3.70

mode3 35.51 64.49 −0.24 0.15 0.56 6.42

Ave 32.89 67.11 −0.19 0.22 0.40 4.91

SD of Ra

0.73

0.75

0.67

0.38

0.43

0.90

0.62

0.23

It can be seen that surface deviation and roughness of the objects from DLP are smaller. Through qualitative and quantitative comparison, it is obvious that the precision of DLP is higher and the consistency of objects is better than FDM. Compared with machining, FDM printing is equivalent to boring or planning technology, the surface deviation is clearly visible. However DLP printing is much fine and comparable with milling technology.

4

Conclusions

In this paper, we propose a quality inspection scheme for 3D printed objects. It presents an evaluation method based on vision for personality of 3D printed object. It is composed of obtaining the point cloud data, aligning point data with model, analyzing shape devi‐ ation qualitatively and calculating surface roughness quantitatively. The experiment results show that DLP printing is more precise than FDM printing. It is consistent with the known knowledge. It also shows that the efficiency of the proposed scheme. we could improve the precision of evaluation by adding the reference deviation of scanned model in the future work.

A Quality Evaluation Scheme to 3D Printing Objects

467

References 1. Berman, B.: 3-D printing: the new industrial revolution. Bus. Horiz. 55(2), 155–162 (2012) 2. Li, X.L., Ma, J.X., Li, P., et al.: 3D printing technology and its application trend. Process Autom. Instrum. 35(1), 1–5 (2014) 3. Zhou, L., Yang, Q., Zhang, G., et al.: Additive manufacturing technologies of porous metal implants. China Foundry 11(4), 322–331 (2014) 4. Kruth, J.P., Leu, M.C., Nakagawa, T.: Progress in additive manufacturing and rapid prototyping. CIRP Ann. Manuf. Technol. 47(2), 525–540 (1998) 5. Melchels, F.P.W., Domingos, M.A.N., Klein, T.J., et al.: Additive manufacturing of tissues and organs. Prog. Polym. Sci. 37(8), 1079–1104 (2012) 6. Pan, X.B.: Study on key techniques of surface roughness measurement. Zhejiang University, Hangzhou (2011). (in Chinese) 7. Zhou, H., Liu, L.L., Zhao, X.P.: Evaluation parameters of new national standard for surface roughness and its application. Mech. Eng. 8, 83–85 (2003). (in Chinese) 8. Motavall, I., Bidanda, B.: A part image reconstruction system foreverse engineering of design modification. J. Manuf. Syst. 10(5), 35–39 (1991) 9. Rao, X.X., Liu, H.S., Zhong, C.H.: Quality inspection for punching parts based on 3D alignment. Mach. Design Res. 23(2), 90–94 (2007) 10. Li, B.: Quality Inspection for Stamping Die Based on 3D Alignment. Nanchang University, Nanchang (2014). (in Chinese) 11. Zhou, S.: Research on Visualization Evaluation System of Shape Error. Nanchang Hangkong University, Nanchang (2014). (in Chinese) 12. Zhou, Q.F.: Geometrical Product Specification (GPS) - surface texture: profile methodSurface roughness parameters and their values. Mach. Ind. Stand. Qual. 442(3), 30–32+37 (2010). (in Chinese) 13. Gan, X.C., Zhang, Y., Liu, N., Shi, Z.D., et al.: Surface roughness parameters Rz, Rmax, Rt, R3z, RPc and other measurements. China Metrol. 154(9), 75–77 (2008). (in Chinese)

Representation, Analysis and Applications of Large-Scale 3D Multimedia Data

Secure Image Denoising over Two Clouds Xianjun Hu, Weiming Zhang(B) , Honggang Hu, and Nenghai Yu Key Laboratory of Electromagnetic Space Information, CAS, University of Science and Technology of China, Hefei, China [email protected], {zhangwm,hghu2005,ynh}@ustc.edu.cn

Abstract. Multimedia processing with cloud is prevalent now, which the cloud server can provide abundant resources to processing various multimedia processing tasks. However, some privacy issues must be considered in cloud computing. For a secret image, the image content should be kept secret while conducting the multimedia processing in the cloud. Multimedia processing in the encrypted domain is essential to protect the privacy in cloud computing. Hu et al. proposed a novel framework to perform complex image processing algorithms in encrypted images with two cryptosystems: additive homomorphic encryption and privacy preserving transform. The additive homomorphic cryptosystem used in their scheme causes huge ciphertext expansion and greatly increases the cloud’s computation. In this paper, we modified their framework to a twocloud scheme, and also implemented the random nonlocal means denoising algorithm. The complexity analysis and simulation results demonstrate that our new scheme is more efficient than Hu’s under the same denoising performance. Keywords: Secure image denoising · Image sharing Random nonlocal means · Double-cipher

1

Introduction

Multimedia processing in the cloud has been widely used in recent years, such as photo-editing app Prisma1 , and video and photo editing app Artisto2 . The cloud servers can offer high computation and large storage resources; client can outsource local large data and complex computing tasks to the cloud servers to save the local resource. However, cloud server is a third party, and it may not be trusted. The outsourced sensitive multimedia content may be leaked, which will lead to security and privacy issues. For outsourced storage, the simplest way to overcome these issues is to use traditional symmetric cryptography, such as

1 2

This work was supported in part by the Natural Science Foundation of China under Grant U1636201, 61572452, 61522210, and 61632013, and the Fundamental Research Funds for the Central Universities in China (WK2101020005). http://prisma-ai.com/. https://artisto.my.com/.

c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 471–482, 2017. https://doi.org/10.1007/978-3-319-71598-8_42

472

X. Hu et al.

3DES or AES, to encrypt the outsourced sensitive multimedia content. While for outsourced multimedia processing, secure multimedia processing is still a huge challenging problem. Signal processing in the encrypted domain is desired in cloud computing [2]. Modern cryptography provides some vital encryption schemes, such as homomorphic encryption [10, 11, 17,18, 20,29, 30, 32], secret sharing [1,5, 19, 34], and secure multiparty computation [3, 4, 16, 21, 22, 37], to handle multimedia processing in the encrypted domain. The concept of homomorphic encryption is first proposed by Rivest et al. [32] as privacy homomorphism. Since then, nearly 30 years, only partial homomorphism has been achieved, such as Elgamal cryptosystem [18] can perform multiplicative homomorphism, and Paillier cryptosystem [30] can perform additive homomorphism. A breakthrough of fully homomorphic encryption was achieved by Gentry in 2009 [20]. After that, full homomorphic encryption is constantly being improved [10, 11,17, 29]. Even though for practical application, homomorphic encryption is inefficient, signal processing in the encrypted domain based on homomorphic encryption is still a hot research direction. Encrypted domain discrete cosine transform and discrete Fourier transform based on Paillier cryptosystem were implemented by Bianchi et al. [6, 8]. And then encrypted domain discrete wavelet transform and Walsh-Hadamard transform based on Paillier cryptosystem were implemented by Zheng et al. [38–40]. A privacy-preserving face recognition system based on fully homomorphic cryptosystem was presented in [36], and meanwhile, fully homomorphic encryption was applied to genetic data testing [15]. Secret sharing scheme was independently proposed by Blakley [9] and Shamir [34]. The Shamir’s secret sharing scheme is the most frequently used, which supports additive homomorphism [5]. Some secure signal processing schemes based on secret sharing were proposed. A privacy protect wavelet denoising with secret sharing was presented in [33]. However, after every multiplication operation, each party needs to communicate with each other to renormalizing the threshold. In [27], Lathey et al. proposed to perform image enhancement in the encrypted domain with multiple independent cloud servers, and the novelty of their work is that it can deal with arithmetic division operation for nonterminating quotients. In [28], secure cloud-based rendering framework based on multiple cloud centers was presented, and to overcome the computation of real number operation in the encrypted domain, secret sharing scheme without modulus was adopted. Secure multiparty computation was proposed by Yao [37], which can be used as a general method to perform encrypted domain computation [4, 21]. The BGW protocol is a good example [4]. General multiparty computation based on linear secret sharing scheme was proposed [16]. In [31], a scheme for wavelet denoising was proposed, which is based on Lattice cryptography. However, maybe it is not efficient to deal with nonlocal means image denoising algorithm. In [41], Zheng et al. proposed to perform privacy-preserving image denoising using external cloud databases, and their scheme is based on two cloud servers, which one is the image database for storage encrypted image patches, and the other cloud

Secure Image Denoising over Two Clouds

473

server is to generate the garbled circuits and send them to image cloud database to perform comparison operations. For a large image, the communication load between these two cloud servers is considerably huge. Image denoising in the encrypted domain is a concrete research in secure multimedia processing. In [23, 24], Hu et al. proposed a double-cipher scheme to perform nonlocal image denoising. Two encryption schemes, partial homomorphic encryption and privacy-preserving transform were adopted in their scheme. The bottleneck in their scheme is the efficiency of partial homomorphic encryption, which causes cipher expansion and the cloud server performing large computation. In this paper, we presented a new scheme with two non-colluding servers, and the new scheme is more concise and efficient. It can achieve the same denoised performance, while the communication load between cloud servers and client, and the computation complexity in cloud servers side and client side are better than Hu’s scheme. The rest of this paper is organized as follows. In Sect. 2, we introduce Hu’s double-cipher scheme in detail. A comprehensive introduction of our new scheme will be given in Sect. 3. We analyze the computation complexity and communication load about our scheme in Sect. 4. In Sect. 5, we give some discussion about our proposed scheme. Finally, Sect. 6 concludes this paper.

2

Double-Cipher Image Denoising

In this section, we describe the details of double-cipher scheme. Hu et al. proposed the double-cipher scheme in [23, 24]. Monte Carlo nonlocal means image denoising algorithm [14] was adopted as an example to perform nonlinear operation in the encrypted domain. In their framework, the cloud server will get two different cipher images encrypted by two different encryption schemes: Paillier encryption [30] and privacy-preserving Johnson-Lindenstrauss (JL) transform [26] from the same image. The cloud server performed mean filter on the cipher image encrypted by Paillier encryption, while performed nonlocal search on the other cipher image generated by privacy-preserving transform. Here, we firstly present a full description of Hu’s double-cipher scheme, and more details can be read in [24]. We can summarize the double-cipher scheme as three main algorithms: image encryption in the client side, secure image denoising in the cloud, and image decryption in the client side. 2.1

Image Encryption

Binarization attack presented in [24] shows that the cloud server can recover the cipher image through the strong correlation between adjacent image pixels, because spatial close image pixels tend to have similar or even identical pixel value. Therefore, to enhance the security, image scrambling was used to perform decorrelation before image encryption. Because of two encryption schemes, an n-pixel image I was performed two different image scrambling, block image

474

X. Hu et al.

scramble and pixel image scramble, with the same pseudorandom permutation sequence, respectively. For block image scramble, the image I was first split with each pixel as the center in overlapping n image blocks with size l × l. Then each block was made into a vector as an n × l2 matrix α. Here with the pseudorandom permutation sequence, matrix α was performed row scrambling to output a block scrambled ¯ While for pixel image scramble, the image I was scrambling by the image I. ˜ same pseudorandom permutation sequence to output a pixel scrambled image I. ˜ and this makes The indices of rows in I¯ corresponds to the indices of pixels in I, sure the encrypted image can be denoised. A privacy-preserving Johnson-Lindenstrauss (JL) transform was proposed by Kenthapadi et al. [26] based on Johnson-Lindenstrauss theorem [25], which can preserve Euclidean distance, and Hu et al. used this privacy preserving JL transform on image encryption, which was performed in Algorithm 1. After Algorithm 1 performed, an n × k matrix E JL can be generated as ciphertext, where k < l2 . Here we should mention that the size of E JL is about k times larger than that of the original image I. The block size l was chosen as 5, and the projected dimension k was 9 ∼ 18 in [24]. For the second cipher image E P ail , the client encrypted the pixel scrambled image I˜ pixel by pixel with Paillier encryption. After encryption, the client uploaded the two cipher images to the cloud server. Algorithm 1. JL Transform-based Private Projection ¯ projected dimension k; Noise parameter ζ. Input: n × l2 matrix I; Output: The projected n × k matrix E JL . 1. Generate a l2 × k N (0, 1/k) Gaussian distribution matrix P ; 2. Generate an n × k N (0, ζ 2 ) Gaussian distribution noise matrix ∆; ¯ + ∆. 3. E JL = IP

2.2

Secure Image Denoising

Image denoising can be described in a matrix-vector form as: y = wI

(1)

where y, I , and w are the matrix-vector form of noisy image, original image, and the weight of the filter, respectively. The filter matrix w is computed from a nonlocal means kernel function Kij [12, 13], representing the similarity between i-th and j-th image block: Kij = e

−||y(Ni )−y(Nj )||2 h2

,

(2)

where Ni is an image block centered at i, and h denotes the smoothing factor.

Secure Image Denoising over Two Clouds

475

In the encrypted domain, the kernel function Kij can be calculated by JL transformed data matrix E JL , so Kij can be replaced as: ˜ ij = e K

−||E JL (i)−E JL (j)||2 −2kζ h2

,

(3)

where E JL (i) denotes the i-th row of matrix E JL . ˜ can be described as follows: Therefore, the estimated image y ˜ = wI ˜ , ˜ = D−1 Kz y

(4)

where D is a diagonal matrix denoting a normalization factor. The cloud server can perform encrypted the image denoising algorithm with ˜ on the cipher image E P ail [I ]. The denoised encrypted image the weight matrix w is presented as follows: ˜ . (5) E P ail [I ′ ] = (E P ail [I ])w Calculating the weight matrix w by the classic nonlocal means algorithm [12] is extraordinary time-consuming, because the computation complexity is about O(n2 ), and n is the number of image pixel. Monte Carlo Non-Local Means (MCNLM) [14] is a random sampling algorithm, and for each image pixel, it only selects a small number of image blocks to calculate the weight matrix, which was implemented in the encrypted domain to speed up the classic nonlocal means denoising algorithm in [24]. 2.3

Image Decryption

After image denoising in the cloud server, the cloud server sent back the encrypted denoised image, and the client decrypted the cipher image E P ail [I′ ] pixel by pixel with Paillier decryption. At last, pixel inverse scramble was performed, and the client got the denoised image I′ .

3

Secure Image Denoising over Two Clouds

Paillier encryption is an additive homomorphic encryption, which brings large ciphertext expansion and causes heavy communication load between the cloud server and the client, and also the calculation of the modular multiplication and modular exponentiation in the cloud server is remarkably time-consuming. Therefore, to reduce this ciphertext expansion and avoid the modular operations in the encrypted domain, we modified their scheme to a new one with two cloud servers. In our new scheme, the cloud servers only need to perform normal addition and multiplication in the cipher images, and the computation complexity is much lower than previous one. In this section, we present the details of our proposed scheme. In our scheme, we need two cloud servers to perform MCNLM, and the framework of our scheme is presented in Fig. 1. From this framework, we can see that the client also uses

476

X. Hu et al.

two different encryption scheme to encrypt the image. The client uses JL transform to get the cipher image E JL , and uses the other encryption scheme (This encryption scheme will be described later.) to divide the image into two shares E S1 , E S2 . Then the client uploads E JL , E S1 to cloud server 1 (CS 1), and uploads E JL , E S2 to cloud server 2 (CS 2) as step 1 showed in the Fig. 1. As described above, MCNLM is a randomized algorithm, for solving the synchronization problem, CS 1 computes the sample indices, and sent the indices to CS 2 as step 2 showed. With the same sample indices, the two cloud servers can calculate the weight matrix with E JL , and perform the linear denoising on E S1 and E S2 , respectively. After each cloud server completes the denoising algorithm, they ′ ′ sends back their denoised image shares E S1 , E S2 to the client as step 3 showed. The client will get two denoised image shares, and the denoised image will be reconstructed.

Fig. 1. Framework of two-cloud based secure image denoising.

Our new scheme is based on Hu’s double-cipher scheme, and some procedures are the same, in order to simplify the description of our new scheme, we omit the same part and focus on the different part. 3.1

Image Sharing

For an n-pixel image I, and each pixel value is 8-bit, to encrypt this image, the client first generates a matrix E S1 with n elements, and each element is randomly chosen from a uniform distribution. Then the client encrypts the image I as: E S2 = I + E S1 . The cipher image shares E S2 , E S1 are additive homomorphism, which can be used to replace the cipher image generated by Paillier encryption. 3.2

Image Sampling

MCNLM is a randomized algorithm, and the weight of each image pixel is computed from a subset of the image, if the two cloud servers independently compute

Secure Image Denoising over Two Clouds

477

the weight matrix, it will cause the two weight matrices different, and the following denoising fails. In order to solve the synchronization problem, we let one of the cloud servers perform random sampling, and sends the sampling indices to the other cloud server. Two sampling patterns were described in [14], which is uniform sampling and optimal sampling. Each pixel block is sampling based on a fixed probability in the uniform sampling pattern, while an optimization problem need to be solved in the optimal sampling pattern.

4

Complexity Analysis

In this section, the complexity of our proposed scheme will be analyzed. The complexity of the scheme includes communication complexity and computation complexity. We also compare our scheme with Hu’s scheme. 4.1

Communication Complexity

First, we analyze the communication complexity of our proposed scheme. The cipher image E JL should be uploaded to each cloud server, while the cipher image shares E S1 , E S2 should be upload to CS 1 and CS 2, respectively. And ′ ′ also the denoised encrypted image shares E S1 and E S2 should be sent back to the client. Therefore, for an n-pixel 8-bit image, the projected dimension k of JL transform is chosen as 9 ∼ 18, and there are two independent cloud servers. Thus the upload communication data is slightly more than 2×n×(k+1) bytes, and the download communication data is slightly more than 2 × n bytes. While in Hu’s scheme for 1024-bit encryption key, the upload communication data is about n × (k + 256) bytes, and the download communication data is about 8n bytes by using ciphertext compression [7]. For k = 12 in our scheme, that is one-tenth the upload communication data of Hu’s scheme, and a quarter the download communication data of Hu’s scheme. The communication data between cloud servers and the client is significantly decreased. In our new scheme, CS 1 should send the sampling indices to CS 2, for sampling ratio is ρ, this communication data is ρn log(n) bits, while in Hu’s scheme, this is not required. For the sampling ratio is very small, most of the sampling indices are 0, while the sampling ratio is very big, most of the sampling indices are 1. The sampling indices can be compressed effectively. In Hu’s scheme, the sampling ratio set to 0.01 is enough. We list the communication complexity in Table 1. Table 1. Communication Complexity Hu’s scheme

Our scheme

Upload

n × (k + 256) 2n × (k + 1)

Download

8n

Cloud-to-cloud None

2n ρn log(n)/8

478

4.2

X. Hu et al.

Computation Complexity

The computation complexity in our scheme includes the client side and the cloud side. In the client side, client needs to perform image scramble, JL Transform, image sharing, and image reconstruction. Image sharing in our scheme is very concise, which can be efficiently computed. While in Hu’s scheme, the client side needs to perform Paillier encryption, which is more complicated than image sharing. In the cloud side, the cloud server should the perform modular operations in Hu’s scheme, while in our scheme, each cloud server only needs to perform the normal operations as in the plain image. Image decryption in Hu’s scheme is also complicated operation. On the client side, the difference between our scheme and Hu’s scheme is image sharing and Paillier encryption, therefore, we only compare these two parts in our simulation. A simulation was given on an Intel i5 CPU at 2.5 GHz computer running Ubuntu 32-bit v13.04. Time cost of different parts for a 256 × 256 image is listed in Table 2, and we simulated Hu’s double-cipher scheme by their fast algorithm implementation. We can see that our scheme in the client side is much faster than Hu’s scheme. The Paillier encryption is more complicated than image sharing, which brings more calculation and time-consuming. So, our new scheme is more practical. Table 2. Time cost in client side Paillier Encryption Paillier Decryption Hu’s scheme 1.0 Image sharing Our scheme 0.1

4.1 Image reconstruct 0.1

On the cloud server side, in Hu’s double-cipher scheme, the complicated modular multiplication and modular exponentiation need to be performed, while in our new scheme, the cloud servers only need to perform the normal addition and multiplication as in the plain image. The computation time of our proposed scheme approximately equals the plain MCNLM algorithm on the cloud server side.

5

Discussion

In this section, we give some discussion about our proposed scheme. Security. In our new scheme, we adopted a very concise image encryption, image sharing, to replace Paillier additive homomorphic encryption. So in our scheme, we assume that the two cloud servers are non-colluding, and they are honest-but-curious. If we consider a malicious model, we need more complicated secure multiparty computation protocol, and this also can be implemented in

Secure Image Denoising over Two Clouds

479

our framework, which will increase much communication traffic and computation complexity for obtaining higher security. In our scheme, CS 1 received a random matrix generated by the client, and this matrix is independent of the input image itself. Therefore, CS 1 can get nothing about input image from its image sharing. CS 2 gets an image matrix hiding by adding CS 1’s random matrix. This image splitting method guarantees the security of the image content against cloud servers. The random matrix will be changed every time in the client side to encrypt the image. Some optimizations. In our scheme, one cloud server needs to perform the image sampling and the other server waits for the sampling indices. A optimal scheme can be given in Fig. 2. The two cloud servers each select half of the image to perform image sampling and denoising, After completing its own denoising, the two cloud servers send their respective indices to the other party, and it can reduce the waiting time.

Fig. 2. An improvement of two-cloud based our secure image denoising

Our proposed scheme is based on two cloud servers, and we can also change our scheme to a multi-cloud framework based on secret sharing to resist colluding of cloud servers as showed in Fig. 3. Then the communication load between cloud servers and the client, cloud server to cloud server will increase with the number of the cloud servers. If we consider about other deterministic image denoising algorithm [35] in our framework, then the communication load between cloud servers can be omitted, and it will be more efficient. As showed in Figs. 1 and 2 and complexity analysis in Sect. 4, our framework abandons the extraordinary complicated Paillier cryptosystem, the communication load between cloud servers and the client, and the computation cost in the cloud server are significantly decreased. Our new scheme can achieve the same image denoising performance as Hu’s scheme.

480

X. Hu et al.

Fig. 3. A variant of our secure image denoising

6

Conclusion and Future Work

In this paper, we modified Hu’s double-cipher scheme into a two cloud servers scheme, and gave some optimizations. In our scheme, the cloud servers can perform encrypted image denoising as same as in the plain image, and our proposed scheme almost does not increase the amount of calculation for each cloud server. The main drawback of our proposed scheme is probably that we should rent two non-colluding cloud serves, and the client should communicate with each cloud server. But we reduced the cipher expansion effectively, and the total communication load is still lower than Hu’s scheme. The client side’s computational complexity is significant reduction. The cloud servers don’t need to perform complex modular operations in the encryption domain. Efficient implementation of the multimedia nonlinear operation in the encrypted domain sill remains as a difficult problem. Working on more image processing algorithms in the encrypted domain are our future research direction.

References 1. Two verifiable multi secret sharing schemes based on nonhomogeneous linear recursion and LFSR public-key cryptosystem. Inf. Sci. 294, 31–40 (2015). Innovative Applications of Artificial Neural Networks in Engineering 2. Aguilar-Melchor, C., Fau, S., Fontaine, C., Gogniat, G., Sirdey, R.: Recent advances in homomorphic encryption: a possible future for signal processing in the encrypted domain. IEEE Signal Process. Mag. 30(2), 108–117 (2013) 3. Barak, B., Sahai, A.: How to play almost any mental game over the net - concurrent composition via super-polynomial simulation. In: 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2005), pp. 543–552, October 2005 4. Ben-Or, M., Goldwasser, S., Wigderson, A.: Completeness theorems for noncryptographic fault-tolerant distributed computation. In: Proceedings of the Twentieth Annual ACM Symposium on Theory of Computing, STOC 1988, pp. 1–10 (1988)

Secure Image Denoising over Two Clouds

481

5. Benaloh, J.C.: Secret sharing homomorphisms: keeping shares of a secret secret (Extended abstract). In: Odlyzko, A.M. (ed.) CRYPTO 1986. LNCS, vol. 263, pp. 251–260. Springer, Heidelberg (1987). https://doi.org/10.1007/3-540-47721-7 19 6. Bianchi, T., Piva, A., Barni, M.: On the implementation of the discrete fourier transform in the encrypted domain. IEEE Trans. Inf. Forensics Secur. 4(1), 86–97 (2009) 7. Bianchi, T., Piva, A., Barni, M.: Composite signal representation for fast and storage-efficient processing of encrypted signals. IEEE Trans. Inf. Forensics Secur. 5(1), 180–187 (2010) 8. Bianchi, T., Piva, A., Barni, M.: Encrypted domain DCT based on homomorphic cryptosystems. EURASIP J. Inf. Secur. 2009(1), 716357 (2009) 9. Blakley, G.R.: Safeguarding cryptographic keys. In: Proceedings of the National Computer Conference 1979, vol. 48, pp. 313–317 (1979) 10. Brakerski, Z., Vaikuntanathan, V.: Efficient fully homomorphic encryption from (standard) LWE. In: 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, pp. 97–106, October 2011 11. Brakerski, Z.: Fully homomorphic encryption without modulus switching from classical GapSVP. In: Safavi-Naini, R., Canetti, R. (eds.) CRYPTO 2012. LNCS, vol. 7417, pp. 868–886. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3642-32009-5 50 12. Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2005) 13. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 2, pp. 60–65. IEEE (2005) 14. Chan, S.H., Zickler, T., Lu, Y.M.: Monte carlo non-local means: random sampling for large-scale image filtering. IEEE Trans. Image Process. 23(8), 3711–3725 (2014) 15. Check, H.E.: Cloud cover protects gene data. Nature 519(7544), 400 (2015) 16. Cramer, R., Damg˚ ard, I., Maurer, U.: General secure multi-party computation from any linear secret-sharing scheme. In: Preneel, B. (ed.) EUROCRYPT 2000. LNCS, vol. 1807, pp. 316–334. Springer, Heidelberg (2000). https://doi.org/10. 1007/3-540-45539-6 22 17. van Dijk, M., Gentry, C., Halevi, S., Vaikuntanathan, V.: Fully homomorphic encryption over the integers. In: Gilbert, H. (ed.) EUROCRYPT 2010. LNCS, vol. 6110, pp. 24–43. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3642-13190-5 2 18. Elgamal, T.: A public key cryptosystem and a signature scheme based on discrete logarithms. IEEE Trans. Inf. Theory 31(4), 469–472 (1985) 19. Feldman, P.: A practical scheme for non-interactive verifiable secret sharing. In: 28th Annual Symposium on Foundations of Computer Science (SFCS 1987), pp. 427–438, October 1987 20. Gentry, C.: A fully homomorphic encryption scheme. Ph.D. thesis, Stanford University (2009) 21. Goldreich, O., Micali, S., Wigderson, A.: How to play any mental game. In: Proceedings of the Nineteenth Annual ACM Symposium on Theory of Computing, STOC 1987, pp. 218–229 (1987) 22. Hirt, M., Nielsen, J.B.: Robust multiparty computation with linear communication complexity. In: Dwork, C. (ed.) CRYPTO 2006. LNCS, vol. 4117, pp. 463–482. Springer, Heidelberg (2006). https://doi.org/10.1007/11818175 28

482

X. Hu et al.

23. Hu, X., Zhang, W., Hu, H., Yu, N.: Non-local denoising in encrypted images. In: Hsu, R.C.-H., Wang, S. (eds.) IOV 2014. LNCS, vol. 8662, pp. 386–395. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11167-4 38 24. Hu, X., Zhang, W., Li, K., Hu, H., Yu, N.: Secure nonlocal denoising in outsourced images. ACM Trans. Multimedia Comput. Commun. Appl. 12(3), 40:1– 40:23 (2016) 25. Johnson, W.B., Lindenstrauss, J.: Extensions of lipschitz mappings into a hilbert space. Contemp. Math. 26(189–206), 1 (1984) 26. Kenthapadi, K., Korolova, A., Mironov, I., Mishra, N.: Privacy via the johnsonlindenstrauss transform. arXiv preprint arXiv:1204.2606 (2012) 27. Lathey, A., Atrey, P.K.: Image enhancement in encrypted domain over cloud. ACM Trans. Multimedia Comput. Commun. Appl. 11(3), 38:1–38:24 (2015) 28. Mohanty, M., Atrey, P., Ooi, W.T.: Secure cloud-based medical data visualization. In: Proceedings of the 20th ACM International Conference on Multimedia, MM 2012, pp. 1105–1108 (2012) 29. Nuida, K., Kurosawa, K.: (Batch) Fully homomorphic encryption over integers for non-binary message spaces. In: Oswald, E., Fischlin, M. (eds.) EUROCRYPT 2015. LNCS, vol. 9056, pp. 537–555. Springer, Heidelberg (2015). https://doi.org/ 10.1007/978-3-662-46800-5 21 30. Paillier, P.: Public-key cryptosystems based on composite degree residuosity classes. In: Stern, J. (ed.) EUROCRYPT 1999. LNCS, vol. 1592, pp. 223–238. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-48910-X 16 31. Pedrouzo-Ulloa, A., Troncoso-Pastoriza, J.R., Prez-Gonzlez, F.: Image denoising in the encrypted domain. In: 2016 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–6, December 2016 32. Rivest, R.L., Adleman, L., Dertouzos, M.L.: On data banks and privacy homomorphisms. Found. Secur. Comput. 4(11), 169–180 (1978) 33. SaghaianNejadEsfahani, S.M., Luo, Y., c. S. Cheung, S.: Privacy protected image denoising with secret shares. In: 2012 19th IEEE International Conference on Image Processing, pp. 253–256, September 2012 34. Shamir, A.: How to share a secret. Commun. ACM 22(11), 612–613 (1979) 35. Talebi, H., Milanfar, P.: Global image denoising. IEEE Trans. Image Process. 23(2), 755–768 (2014) 36. Troncoso-Pastoriza, J.R., Prez-Gonzlez, F.: Fully homomorphic faces. In: 2012 19th IEEE International Conference on Image Processing, pp. 2657–2660, September 2012 37. Yao, A.C.: Protocols for secure computations. In: 23rd Annual Symposium on Foundations of Computer Science (SFCS 1982), pp. 160–164, November 1982 38. Zheng, P., Huang, J.: Discrete wavelet transform and data expansion reduction in homomorphic encrypted domain. IEEE Trans. Image Process. 22(6), 2455–2468 (2013) 39. Zheng, P., Huang, J.: Implementation of the discrete wavelet transform and multiresolution analysis in the encrypted domain. In: Proceedings of the 19th ACM International Conference on Multimedia, MM 2011, pp. 413–422 (2011) 40. Zheng, P., Huang, J.: Walsh-hadamard transform in the homomorphic encrypted domain and its application in image watermarking. In: Kirchner, M., Ghosal, D. (eds.) IH 2012. LNCS, vol. 7692, pp. 240–254. Springer, Heidelberg (2013). https:// doi.org/10.1007/978-3-642-36373-3 16 41. Zheng, Y., Cui, H., Wang, C., Zhou, J.: Privacy-preserving image denoising from external cloud databases. IEEE Trans. Inf. Forensics Secur. 12(6), 1285–1298 (2017)

Aesthetic Quality Assessment of Photos with Faces Weining Wang1, Jiexiong Huang1, Xiangmin Xu1 ✉ , Quanzeng You2, and Jiebo Luo2 (

)

1 South China University of Technology, Guangzhou, China {wnwang,xmxu}@scut.edu.cn, [email protected] 2 University of Rochester, Rochester, NY 14627, USA {qyou,jluo}@cs.rochester.edu

Abstract. Aesthetic quality assessment of photos is a challenging task regarding the complexity of various photos and subjectivity of human’s aesthetic percep‐ tion. Recent research has suggested that photos with different contents have different aesthetic characters. However, these different aesthetic characters are not considered in previous work of aesthetic assessment. Meanwhile, photos with human faces have become increasingly popular and constitute an important part of social photo collections. In this work, we analyze the characters of this partic‐ ular category of photos, human faces, and study the impact of them on aesthetic quality estimation. This study could have many potential applications, including selection of high aesthetic face photos, face photo editing and so on. To solve this problem, we design new handcrafted features and fine-tuned a new deep Convo‐ lutional Neural Network (CNN) for features. Next, we build decision fusion model to employ all the proposed features for aesthetic estimation. In addition, we analyze the effectiveness of different groups of features in a face photo clas‐ sification task to better understand their differences. Experimental results show that our proposed features are effective and the classifier outperforms several upto-date approaches in aesthetic quality assessment. Keywords: Aesthetic quality assessment · Faces Deep Convolutional Neural Network · Decision fusion

1

Introduction

Aesthetic quality assessment is a subjective task, which is related to human’s perception of visual stimuli. It can improve user experiences and provide better quality of service in many applications, including image retrieval, photo editing, photography and so on. Automated aesthetic quality assessment tries to automatically predict human’s aesthetic perception. It is a challenging task regarding the complexity of various photos and subjectivity of human’s aesthetic perception. Recent work on automated aesthetic quality assessment has been focusing on designing robust machine learning algorithms to distinguish the aesthetic quality of photos. Aesthetic features are critical and directly impact the performance of current machine learning models on this task. Researchers have spent huge efforts in designing novel and descriptive aesthetic features, inspired from painting, photography, art, and so on [1–3, 9–11, 17]. Meanwhile, they also try to provide additional solutions including generic © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 483–495, 2017. https://doi.org/10.1007/978-3-319-71598-8_43

484

W. Wang et al.

features [4] and geo-content information [5]. More recently, deep learning method has shown significantly improved results on a wide range of computer vision tasks [6–8, 15]. It has also been applied to automated aesthetic assessment. Lu et al. [8] designed a double-column Convolutional Neural Networks (CNNs) for aesthetic quality categori‐ zation. Dong et al. [7] used a trained deep learning network to extract features of photos and then applied the Support Vector Machine (SVM) to estimate the aesthetic quality of photos. However, these studies have been using the CNNs fine-tuned from general image classification tasks and studying the aesthetic quality of general images. Mean‐ while, Tang’s research [10] showed that photos with different contents have different aesthetic characters. More than that, psychology research in perception also confirms that certain kinds of contents will be more attractive than others to our eyes, either because we have learned to expect more information from them or because they appeal to our emotions or desires [11]. Therefore, it is expected that by designing different aesthetic features for different kinds of photos, we can achieve better performance in accessing photo aesthetic quality. In this paper, our research focuses on photos with human faces. They constitute an important part of social photo collections. Among our 500 randomly selected photos from online social network, the percentage of photos with faces is 45.6%, which is also the largest category. Indeed, this coincides with the finding that pictures with faces can attract more likes, comments and attention from online social network users1. Therefore, the study of aesthetic quality on this particular category of photos can still have many potential applications. Indeed, there have been several previous studies focusing on aesthetics of photos with faces. Li et al. [11] built a dataset of photos with faces, where each photo has been assigned aesthetic scores by multiple annotators. They also evaluated the performance of several different categories of aesthetic features, including pose features, location features and composition features. However, the dataset is too small to validate the effectiveness and robustness of different groups features. Tang et al. [10] built another dataset with 7 different categories, where “human” is one of them. Each photo has been assigned a label of “high quality” or “low quality”. Particularly, they incorporated face specific features for photos with faces. However, to the best of our knowledge, no research is conducted on employing deep learning to study aesthetic quality of photos with faces. One of the main difficulties is the lack of a large scale labeled datasets. To solve the above mentioned problems, we make the following contributions: 1. Based on large-scale AVA dataset [12], we collect a dataset of photos with faces AVA_Face. It consists of 20,320 photos, which is large enough to fully evaluate different features. 2. We design 78 facial aesthetic features, including facial composition features, EyesChambers features, shadow features, expression features and saliency features. 3. We apply CNN to learn aesthetic features due to the availability of a relative large scale dataset.

1

http://www.news.gatech.edu/2014/03/20/face-it-instagram-pictures-faces-are-more-popular.

Aesthetic Quality Assessment of Photos with Faces

485

4. We fuse the features from CNN and the other two kinds of handcrafted aesthetic features using decision fusion. This is different from previous researches, where the main strategy is early fusion. The overall framework of our method is shown in Fig. 1. The first component is feature extraction. It includes both the hand crafted features as well as features from a fine-tuned CNN. Next, we use decision fusion to aggregate the prediction result from the different features and to evaluate the aesthetic quality. The paper is organized as follows. The newly designed facial aesthetic features are introduced in Sect. 2. In Sect. 3, we apply a new CNN model to learn aesthetic features. In Sect. 4, we introduce Decision Fusion. The datasets and experiments are presented in Sect. 5, then we conclude the paper in Sect. 6.

Fig. 1. Overview of the proposed work.

2

Facial Aesthetic Features

In photos with faces, people pay much more attention to the face regions. Intuitively, features related to face are closely related the aesthetics of photos. Therefore, we design 78 dimensions of facial aesthetic features. (1) For multiple face regions in the photo, 47 facial composition features and 5 facial saliency features are extracted. (2) For the largest face region, 12 Eyes-Chambers features, 9 facial shadow features and 5 facial expression features are extracted.

Fig. 2. Examples of the detected key points of faces.

486

W. Wang et al.

Face++ Research Toolkit v0.1 [19] is used to detect face and their key points in this paper. Figure 2 shows some examples of the detected face key points, which are going to help us design and extract facial aesthetic features. 2.1 Facial Composition Features We define two kinds of facial composition features as follows. Facial distribution features: the number of faces f1, the area of the largest face f2, the average area of all faces f3, the standard derivation of areas of faces f4, the proportion of the area of faces f5, the maximum distance between faces f6, the minimum distance between faces f7, the average distance of faces f8, and the compactness of face f9. Minimal Spanning Tree algorithm [16] is used to calculate the distances between faces. In calculating compactness of face, we first draw a smallest rectangle region, which can contain all faces, and then calculate the diagonal length of this rectangle. Next, we obtain the compactness of face by dividing the average distance to the computed diagonal length. Facial basic features: The facial features consist of color and texture consistency features. In all face regions, the mean value and standard derivation of hue f10 ∼ f11, saturation f12 ∼ f13, and value f14 ∼ f15 are extracted as the color consistency features. Then, we use Gabor wavelet with 4 scales � � {0, 3, 6, 9} and 4 directions � � {0◦ , 30◦ , 60◦ , 90◦ } to extract facial texture features. For each pair of scale and direc‐ tion, we compute their averages as f16 ∼ f31 and standard derivations as f32 ∼ f47. Both of the averages and the derivations are known as the texture consistency features. 2.2 Facial Saliency Features Salient regions are important in aesthetics of photos [2, 10]. In photos with faces, faces, which are the main saliency regions, attract the most attention. Therefore, we detect face regions and extract facial clarity and complexity as facial saliency features. We calculate facial clarity features f48 as follows [10]: ) ( | | Cface = {(x, y)||Fface (x, y)| > � max Fface (x, y) }∕Num | |

(1)

where Num is the number of pixels belong to the face region, Fface is the Fourier transform of face regions, and � is the threshold (set to 0.2 in this paper). Then, we define f49 as the ratio of the clarity of face regions to the entire photo. Rface = Cface ∕Cphoto

(2)

To extract facial complexity features in face regions, we use the number of pixels to describe it. We segment the super pixels of a total photo and get the number of super pixels of face regions Nface and the number of super pixels of the background Nunface.

Aesthetic Quality Assessment of Photos with Faces

487

We define Iface and Iunface are the sets of pixels of face regions and the other regions, then we calculate the facial complexity features f50 ∼ f52 as follows [10].

‖ ‖ Complexity1 = Nface ∕‖Iface ‖ ‖ ‖

‖ ‖ Complexity2 = Nunface ∕‖Iunface ‖ ‖ ‖ ‖ ‖ Complexity3 = Nface ∕‖Iunface ‖ ‖ ‖

(3) (4) (5)

2.3 Eyes-Chambers Features Researchers have tried to design aesthetic features inspired by the locations of different facial parts [13]. Comparably, we design Eyes-Chambers features considered the aesthetic influence. Five Eyes and Three Chambers are traditional criterions for beautiful faces in Chinese Physiognomy. Five Eyes requires that the width of our two eyes, the width between two eyes and distances between two eyes and the contours of face are all about 1/5 of the face width. Three Chambers requires that the length of nose, the distance between nose and jaw and the distance between nose and hair line are all about 1/3 of the face length. It revealed that the width and distance between organs are important determinants of beautiful faces. We calculate the width of two eyes f53 ∼ f54, the portion of them to the width of face f55 ∼ f56, the distance between two eyes and the portion of it to the width of face f57 ∼ f58, and the ratio of the width of two eyes f59 as the Eyes features. Similarly, we calculate length of nose f60, the distance between nose to jaw f61, the ratio of them f62 and the portion of them to the height of face f63 ∼ f64 as the Chambers features. 2.4 Facial Shadow Features The study from [14] indicates that the contrasts between brightness and darkness are able to highlight major regions of photos and thus is correlated with aesthetic qualities. Figure 3 shows several examples, where shadows are correlated with the quality of photos. In general, we can estimate the bright regions and dark regions of faces and use them as shadow computational templates to qualify the shadow values as the facial shadow features. The main steps are shown in Fig. 4. As shown in Fig. 4(c), the two sub regions with the same color are one pair of shadow computational templates. In total, we have 9 pairs of shadow computational templates. For each pair, we calculate the ratio of brightness of two sub region as the shadow features. In such a way, we receive 9 facial shadow features f65 ∼ f73. Equation (6) shows how to compute the shadow feature for a given pair of sub regions.

488

W. Wang et al.

TLk =

∑

v(i)

∑

i∈Tk2 v(i) ∕ { { } }|| || || || || v(i)|i ∈ Tk1 || || v(i)|i ∈ Tk2 || || || || || i∈Tk1

k ∈ {1, 2, … , 9}

(6)

where Tk1 and Tk2 are the two sub regions in the kth pair of shadow computational template, i is the pixel of photos, v(i) is the value of the pixel i in HSV.

(a) high quality

(b) high quality

(c) low quality

(d) low quality

Fig. 3. Difference of shadow in photos with faces.

Fig. 4. Shadow computational templates.

Fig. 5. Examples of facial expression features.

2.5 Facial Expression Features Facial expressions of a given photo can affect viewer’s subjective evaluation of its aesthetic quality [23]. In general, active and positive facial expressions are expected to receive higher aesthetic responses from the viewers. As shown in Fig. 5, facial expres‐ sions are closely related to the detected facial key points. In particular, EyeShape1, EyeShape2, MouthShape1, MouthShape2 are calculated as the facial expression features f74 ∼ f77. The mean of EyeShape1 and EyeShape2 is also included as f78. See Eqs. (7), (8), (9) and (10) for detailed computation of these features. ( ) ED P1 , P2 EyeShape1 = ( ) ED P3 , P4

(7)

Aesthetic Quality Assessment of Photos with Faces

489

( ) ED P1 , P5 EyeShape2 = ( ) ED P5 , P2

(8)

( ) ED P′1 , P′2 MouthShape1 = ( ) ED P′3 , P′5

(9)

( ) ED P′1 , P′5 MouthShape2 = ( ) ED P′5 , P′2

(10)

where ED(.,.) is the Euclidean distance, P stands for the set of key points around eyes and P′ stands for the set of key points around mouth (see Fig. 5 (a) for more details).

3

Aesthetic Quality Assessment Using Deep Learning Method

Recently, deep learning method achieved significant performance improvement in many computer vision tasks [6–8, 15, 22]. From its success in ImageNet Challenge [15, 18], Convolutional Neural Network has been widely used to solve other challenging computer vision tasks. In particular, pre-trained CNN models on ImageNet have been broadly employed for image classification, which has shown promising results on many related vision problems. In this work, we fine-tune a new CNN model on the pre-trained ResNet, which performed best on the ILSVRC2015 classification task [15]. We choose ResNet-50-layer model as the basic structure and replace the number of neurons in fully connected layer as 2, which is the number of the aesthetic quality categories. As we know, aesthetic quality assessment of photos need us to combine both the global and local information of photos. Therefore, in our new CNN model, we innovatively pool the feature maps of the second, third and fourth convolution layer (conv2_x, conv3_x, conv4_x) to combine with the feature maps of fifth convolution group layer (conv5_x) for learning. Figure 6 shows the proposed deep CNN architecture for aesthetic quality assessment. As shown in Fig. 6, though photos will be resized into the same size when they are input into CNN, it may cause a photo loss its composition information if its width is different to its height. To solve it, we pad each photo to make its width is equal to its height. In this paper, we fine-tune our model by the pre-trained ResNet-50-layer model on Caffe [24]. When fine-tuning the network, the learning rate of the convolution layers is 0.001 and for the fully connected layer, the learning rate is 0.005. The learning rates will decrease 90% after 7 epochs. The outputs of softmax will be considered as features learned by CNN in decision fusion which will be introduced in Sect. 4. The deep CNN performs very well in aesthetic assessment which will be discussed in detail in Sect. 5.

490

W. Wang et al.

Fig. 6. The construction of CNN in this paper.

4

Decision Fusion

Comprehensive features have been employed for evaluation of photo aesthetic quality [2]. There are 86 comprehensive aesthetic features including 56 low-level features, 11 rule-based features, 6 information theory features and 13 visual attention features [2]. The results from [2] suggest the effectiveness of these features in aesthetic evaluation of photos. Therefore, we extract these features in our experiments as well. As we introduce above, CNN performs well in image classification. However, it is usually applied as a black box and ignore the basic information of photos to for computer vision tasks. After extracting facial aesthetic features and comprehensive features, how to combine the advantages of handcrafted features and CNN method to enhance the performance of aesthetic assessment is worth considering. In this paper, we use decision fusion method [20] to fuse them (see Fig. 1). First, we learn different SVM classifiers using two different groups of aesthetic handcrafted features. Each SVM is a binary classifier, which classify the photos into high or low aesthetic quality group. For CNN architecture, a softmax classifier is used to classify photos into high or low aesthetic quality as well. Then, the decision values of each photo from the three classifiers, which are considered as features, are used inputs for decision fusion. Specifically, decision fusion learns another binary SVM classifier to produce the final result. The results show that decision fusion greatly improves the performance of aesthetic assessment.

5

Datasets and Experiments

5.1 Datasets There are two widely used datasets of photos with faces including the Li’s human dataset [11] and the human category of CUHKPQ [10]. Li’s human dataset [11] consists of 500 photos along with their aesthetic scores. CUHKPQ [10] is a database consists of 17,690 photos with less noise, which has 7 different categories. In “human” category, there are 3,148 photos with a label of high quality or low quality.

Aesthetic Quality Assessment of Photos with Faces

491

AVA [12] is a large scale database, which consists of 255,529 images with aesthetic scores. Based on AVA dataset, we collect a new dataset AVA_Face2, which consists of photos with faces. This dataset has 20,320 photos, where each photo has at least one human face. Following Dong’s [7] method, we use the following two ways to obtain the binary labels for each photo: AVA_Face1: The photos with scores higher than 5 will get labels of “good”, and others will get labels of “bad”. Eventually, there are 15,017 photos in “good” category and 5,303 photos in “bad” category. AVA_Face2: The photos rating in top 10% are in “good” category and the down 10% are in “bad” category. Therefore, there are both 2,032 photos in two categories. 5.2 Experiments 5.2.1 Experiments on the Li’s Dataset In [11], Li et al. test their features by aesthetic classification and aesthetic regression. In classification, they calculate the accuracy within one Cross-Category Error (CCE) while they calculate the residual sum-of-squares error (RES) in regression. In our experiments, we extract our proposed aesthetic features on Li’s dataset. Since the dataset is too small to fine-tune CNN model, we use the best model trained on AVA_Face1 to test the photos for the decision values. Then, we fuse these features using decision fusion and then train and test the photos as the way of Li’s experiment on Li’s experiment [11] in both classification and regression. Last, we calculate the accuracy within one CCE and the RES, which shown in Table 1. In Table 1, we can see that in classification, our features perform better and increase the accuracy within one CCE a lot. These two results show that our proposed features perform well in aesthetic evalu‐ ation of photos with faces. Table 1. The results on Li’s dataset. Accuracy within one CCE RES

Li’s method [11] 68% 2.38

Our method 93.2% 2.27

5.2.2 Experiments on Human Category of CUHKPQ Dataset In paper [10], Tang et al. randomly choose half photos of CUHKPQ to be training set and the others to be testing set, and then repeat this random partition 10 times. On human category of CUHKPQ, we extract all the three groups of features and test the features as the way of Tang’s method [10]. When using CNN method, we randomly choose half photos as training set to train the model and test the others to get their decision values. Then we swap the training set and testing set to get the decision values of all photos. We compare with two recent studies on this dataset. The first study is from Tang et al. [10] proposed facial aesthetic features and global features in assessment. 2

The IDs of the photos in AVA_Face can be downloaded at https://github.com/huangjx07/ avaface_id.

492

W. Wang et al.

The second one is Guo et al. [21], where semantic LLC features are included. The results are in Table 2. In comparison, we can see that our proposed aesthetic features all perform better than Guo’s features and Tang’s features. The accuracy of our method is higher than another two proposed methods, and in Fig. 7(a), the ROC (receiver operating charac‐ teristic) curve of our approach achieves the best performance as well. All results indicate the superiority of the proposed framework than existing approaches. Table 2. The results on CUHKPQ dataset. Guo [21] Tang [10] Our method

Comprehensive features 92.19% (no result) 94.19%

Facial features (no result) 95.21 95.58%

Features from CNN —— —— 96.60%

Total 95.01% 97.40% 97.94%

Fig. 7. The ROC curves of different approaches in (a) CUHKPQ (b) AVA_Face1 (c) AVA_Face2.

5.2.3 Experiments on the AVA_Face Dataset In this paper, we collect labels of photos in AVA_Face in two ways so that we get AVA_Face1 and AVA_Face2 dataset. When using CNN method, we choose 80% photos of AVA_Face1 dataset as training dataset, 5% as validation dataset and 15% as testing dataset. In such a way, we are able to fine-tune our deep aesthetic model. We also finetune an original ResNet-50-layer [15] in this way as a baseline to prove the effectiveness for combining the feature maps of four convolution layers in our CNN model. The classification accuracies in Table 3 shows that our CNN model performs better after improving. Table 3. The classification accuracies in AVA_Face1 by two different models. AVA_Face1

ResNet-50-layer Our CNN model 79.10% 79.80%

In AVA_Face1 dataset, we extract two kinds of handcrafted features of photos in testing set and fuse the features of testing set using the CNN method by decision fusion. In AVA_Face2 dataset, all photos are tested using the best model fine-tuned by AVA_Face1. Then these features will be fused with the two kinds of handcrafted

Aesthetic Quality Assessment of Photos with Faces

493

features. To test two datasets more correctly, we using 5-cross validation in experiments. The classification accuracies of our approaches are shown in Table 4. For comparison, we also extract Tang’s features [10] and LLC features [21]. The approach in paper [7] is another baseline. The results are shown in Table 4. We can see that our approach obtains the best performance in both of the evaluated datasets. The deep learning method [7] also receives better performance than Tang’s method [10], which is close to the performance of using our fine-tuned CNN features along. However, the fusion of comprehensive features and facial features improve the overall performance. In Fig. 7(b) and (c), the ROC curve of our approach validates the effectiveness of the proposed framework. Table 4. The classification accuracies on AVA photos with faces dataset. LLC features [21] AVA_Face1 73.08% AVA_Face2 63.12%

Tang [10] 74.28% 79.22%

DCNN [7] Our method 76.90% 80.81% 85.12% 86.99%

5.2.4 Analysis of Different Groups of Features In this subsection, we evaluate the effectiveness of different groups of features on face classification using three different datasets, CUHKPQ, AVA_Face1 and AVA_Face2. The results of experiments are summarized in Table 5. We can see that, in CUHKPQ dataset, the facial aesthetic features perform well in classifying photos with faces. When we combine the facial aesthetic features and the comprehensive features, their classification accuracy are 95.58% and 94.19%. Compa‐ rably, the accuracy of features from CNN is 96.60%. The accuracies of two kinds of handcrafted features are similar or even better than that of the features from CNN. The results suggest that facial aesthetic and comprehensive features are effective on classi‐ fying photos with faces. In particular, high quality dataset (CUHKPQ) can produce robust features and thus leads to better classification results. However, in AVA_Face1 and AVA_Face2 dataset, the facial aesthetic features and the comprehensive features perform worse than the features from CNN. This could be due to the size and the noises of the two datasets. On the other hand, CNNs are capable of reducing the noise level and provide more robust features. Overall, in different dataset of photos with faces, three kinds of aesthetic features have their own advantages. Also, we find that when they fused by decision fusion, we can get the best results. Table 5. The classification accuracies of different feature groups on datasets Facial aesthetic features Comprehensive features Features from CNN Decision Fusion

CUHKPQ 95.58% 94.19% 96.60% 97.94%

AVA_Face1 73.69% 75.65% 79.80% 80.81%

AVA_Face2 76.77% 81.20% 84.40% 86.99%

494

6

W. Wang et al.

Conclusion

In this paper, we propose a framework for automatically assessing the aesthetic quality of photos with faces. Well-designed facial aesthetic features and features learned from CNN are extracted for this task. Then we fuse these features with comprehensive features by decision fusion and obtain the best performance compared with selected baselines on several datasets. This is different from previous researches, where the main strategy is early fusion. In addition, we also collect a new dataset of photos with faces based on AVA dataset, which is a large-scale dataset of photos with faces and has a wider appli‐ cation. Experiments show that our method lead to promising results. The study of aesthetic quality on photos with faces can still have many potential applications. Acknowledgements. This work was supported by the Natural Science Foundation of Guangdong Province #2015A030313212, National Natural Science Foundation of China #U1636218, the State Scholarship Found of China #201506155081, the Science and Technology Planning project of Guangdong Province #2014B010111006 and Guangzhou Key Lab of Body Data Science #201605030011.

References 1. Wang, W., Zhao, W., et al.: An efficient image aesthetic analysis system using Hadoop. Signal Process. Image Commun. 39(108), 499–508 (2015) 2. Wang, W., Cai, D., et al.: Synthesized computational aesthetic evaluation of photos. Neurocomputing 172, 244–252 (2016) 3. Datta, R., Joshi, D., Li, J., Wang, J.Z.: Studying aesthetics in photographic images using a computational approach. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part III. LNCS, vol. 3953, pp. 288–301. Springer, Heidelberg (2006). https://doi.org/10.1007/11744078_23 4. Marchesotti, L., Perronnin, F., et al.: Assessing the aesthetic quality of photographs using generic image descriptors. In: IEEE International Conference on Computer Vision, pp. 1784– 1791 (2011) 5. Yin, W., Mei, T., Chen, C.: Assessing photo quality with geo-context and crowd sourced photos. In: IEEE Visual Communications and Image Processing (VCIP), pp. 1–6 (2012) 6. Dong, Z., Tian, X.: Multi-level photo quality assessment with multi-view features. Neurocomputing 168, 308–319 (2015) 7. Dong, Z., Shen, X., Li, H., Tian, X.: Photo quality assessment with DCNN that understands image well. In: He, X., Luo, S., Tao, D., Xu, C., Yang, J., Hasan, M.A. (eds.) MMM 2015, Part II. LNCS, vol. 8936, pp. 524–535. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-14442-9_57 8. Lu, X., Lin, Z., Jin, H., Yang, J., Wang, J.: Rating pictorial aesthetics using deep learning. IEEE Trans. Multimedia 17(11), 2021–2034 (2015) 9. Wang, W., Yi, J., et al.: Computational Aesthetic of Image Classification and Evaluation. J. Comput. Aided Des. Comput. Graph. 26(7), 1075–1083 (2014) 10. Tang, X., Luo, W., Wang, X.: Content-based photo quality assessment. IEEE Trans. Multimedia 15(8), 1930–1943 (2013) 11. Li, C., Gallagher, A., Loui, A.C., et al.: Aesthetic quality assessment of consumer photos with faces. In: IEEE International Conference on Image Processing (ICIP), pp. 3221–3224 (2010)

Aesthetic Quality Assessment of Photos with Faces

495

12. Murray, N., Marchesotti, L., Perronnin, F.: AVA: A large-scale database for aesthetic visual analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2408–2415 (2012) 13. Duan, H.: Research on Facial Attractiveness Analysis Based on Machine Learning. Beijing Jiaotong University (2011) 14. Jin, X., Zhao, M., Chen, X., Zhao, Q., Zhu, S.-C.: Learning artistic lighting template from portrait photographs. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 101–114. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_8 15. He, K. et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) 16. Seth, P., Vijaya, R.: An Optimal minimum spanning tree algorithm. J. ACM 49(1), 16–34 (2002) 17. Wang, W., Cai, D., Xu, X., Liew, A.: Visual saliency detection based on region descriptors and prior knowledge. Signal Process. Image Commun. 29, 424–433 (2014) 18. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009) 19. Megvii Inc. Face ++ Research Toolkit, December 2013. www.faceplusplus.com 20. Huan, R., Pan, Y.: Decision fusion strategies for SAR image target recognition. IET Radar Sonar Navig. 5(7), 747–755 (2011) 21. Guo, L., et al.: Image esthetic assessment using both hand-crafting and semantic features. Neurocomputing 143, 14–26 (2014) 22. Wang, W. et al.: Image aesthetic classification using parallel deep convolution neural networks. Acta Automatica Sinica 42(6) (2016) 23. Xue, S., Tang, H., Tretter, D., Lin, Q.: Feature design for aesthetic inference on photos with faces. In: IEEE International Conference on Image Processing, pp. 2689–2693 (2013) 24. Yangqing, J., et al.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia (2014)

Sensitive Information Detection on Cyber-Space Mingbao Lin, Xianming Lin(B) , Yunhang Shen, and Rongrong Ji Fujian Key Laboratory of Sensing and Computing for Smart City, School of Information Science and Engineering, Xiamen University, Xiamen 361005, China [email protected]

Abstract. The fast development of big data brings not only abundant information to extensive Internet users, but also new problems and challenges to cyber-security. Under the cover of Internet big data, many lawbreakers disseminate violence and religious extremism through the Internet, resulting in network space pollution and having a harmful effect on social stability. In this paper, we propose two algorithms, i.e., iterative based semi-supervised deep learning model and humming melody based search model, to detect abnormal visual and audio objects respectively. Experiments on different datasets also show the effectiveness of our algorithms. Keywords: Object detect · Query by humming Cyber-space security · Internet big data

1

· Deep learning

Introduction

With the advent of big data era [5, 12], the exploding information and data become increasingly aggravating. According to the statistical data by IDC, in the near future, there will be about 18EBs of storage capacity in China. The joint-report by IDC and EMC points that there will be 40000EBs globally in around 2020. Such enormous Internet data brings not only abundant information to extensive Internet users, but also new problems and challenges to Cyber-security. Under the cover of Internet big data, many lawbreakers disseminate violence and religious extremism through the Internet. Such videos or audios are usually implanted in seemingly common data, under which it’s much complicated to figure out whether it is a normal case or not. Recent years, many videos and audios in referring to violence and extreme religious beliefs have been uploaded to the Internet. These illegal data contributes a lot to the propaganda of violent events and extreme religious thoughts. How to find these illegal hidden videos or audios over mass data and get rid of them to manipulate the healthy development of Cyber-space has become a core problem to be solved immediately. There are two types of sensitive data to be detected on the Cyber-space: one is visual objects detection, the other is audio contents detection. c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 496–504, 2017. https://doi.org/10.1007/978-3-319-71598-8_44

Sensitive Information Detection on Cyber-Space

497

Object detection [2, 26] has been a hot topic in the field of computer vision and image processing. A lot of works about specific target detection have been done at home and abroad, e.g., pedestrian detection [6, 8,18, 20], vehicle detection [4, 19], face detection [1, 3, 25], etc. By analyzing their work, we can find that early works focused on artificial definition based visual features detection. It is difficult to gain the semantic features because artificial definition based visual features have highly to do with low-level visual features. For example, Dalal and Triggs [6] raised gray gradient histogram features, which are applied to pedestrian detection. Ahonen et al. [1] presented LBP features, which are used to detect human faces. Due to the lack of interpretation of image semantics, these methods have a disappointing generalization. Recently, deep neural network has been widely applied to the domain of object detection. Not only can it learn feature descriptors automatically from object images, but also it can give a full description from low-level visual features to high-level semantics. Hence, deep learning has become popular in object detection and achieved a series of success, e.g., Tian et al. [4] transferred datasets of scene segmentation to pedestrian detection through combining the deep learning with transfer learning and gain a good achievement. Chen et al. [4] parallelized the deep convolutional network which has been applied to vehicle detection on satellite images. In [1], a deep convolutional network was proposed to detect human face with 2.9% recall improvement on FDDB datasets. Audio retrieval has become a main direction of multi-media retrieval since the 1990s [9, 14]. Based on the used data features, existing techniques are simply divided into three major categories: physical waveform retrieval [14, 15], spectrum retrieval [10] and melody feature retrieval [9,11, 17, 23, 24]. Physical waveform retrieval is time domain signal based. In [15], a prototype of audio retrieval system is designed through splitting audio data into frames with 13 physical waveform related information extracted as a feature vector and Mahalanobis distance used as a similarity metric. Spectrum retrieval is frequency domain signal based. Foote [7] extracted audio data’s MFCC features and then got histogram features, which were applied to audio retrieval. In [10], a feature descriptor method based on global block spectrum has been proposed, which can present the whole spectrum information but lack anti-noise capacities. Melody feature retrieval is based on voice frequency. In 1995, Ghias et al. [9] first suggested humming melody clips to be used as music retrieval, setting a foundation of humming retrieval. McNab et al. [17] extended Ghias’ idea of pitch contour and proposed to find out the continuous pitch to split notes with the help of related core technologies, like approximate melody matching or pitch tracking. Roger Jang and Gao [11], Wu et al. [24], Wang et al. [23] contributed a lot to voice frequency based melody feature retrieval successively. In a word, with the prosperities of Internet big data, Cyber-space security are facing an increasingly serious challenge. Here are the organizations of this paper. The different bricks -sensitive visual object detection and sensitive audio information detection- are presented in Sects. 2 and 3, with proposed methods and experiments included. Conclusion are described in Sect. 4.

498

2

M. Lin et al.

Iterative Semi-supervised Deep Learning Based Sensitive Visual Object Detection

Usually, sensitive visual information contains some particular illegal things, e.g., designated icons. Hence, to some extent, visual detection can be transformed into specific object detection. One big obstacle of specific object detection is to grab labeled data, which is kinda a waste of human resources. What’s more, human-labeled data contains noise, affecting the performance. In real life, usually we can only get data with few labeled and most unlabeled. To solve the lack of labeled data, we proposed an iterative semi-supervised deep learning based sensitive visual object detection. This algorithm can make full use of the supervised information and will focus on more and more specific objects and reinforce them as iterations. 2.1

Iterative Semi-supervised Deep Neural Network Model

Given a set of N labeled vectors D = {(x1 , y1 ), ..., (xi , yi ), ..., (xN , yN )},

(1)

among which, xi is the ith data and yi is its corresponding label, the learning process adjusts the set D each iteration, after which, new set is applied to update the neural network model. First, extract M image blocks with sliding window from each training data in D. A total of N × M blocks are gained, denoted as R R = {r11 , ....rij , ...rN M }

(2)

Here, rij denotes the jth block from ith training data in D. Then, classify blocks R in the neural network model learned by D and we get a new set named P , with each element a triplet P = {(r11 , t11 , s11 ), ..., (rij , tij , sij ), ..., (rN M , tN M , sN M )}

(3)

Here, rij stands for the element in R, namely, the jth block from ith training data. And tij , sij are its corresponding class and score, resulting from the neural network learned by set D. sij is a confidence coefficient of rij belonging to class tij . Based on this, we can construct a new set D′ , which can be used to update the neural network model. D′ = {(rij , tij )|(rij , tij , sij ) ∈ P, tij = yi , sij > τ }

(4)

This shows that the new set consists of the block that its predicted class agree with the label of its original training data and its predicted confidence coefficient exceeds a particular threshold τ . We show a single version of our iterative model in Fig. 1 and the full algorithm is described in Algorithm 1.

Sensitive Information Detection on Cyber-Space

499

Fig. 1. An example of proposed iterative model. First, sensitive training set is collected. Then, apply this training set to training a neural network model and extract region proposals. Third, classify extracted region proposals with trained model. Lastly, rebuild the training set.

Algorithm 1. Iterative semi-supervised deep learning based Sensitive visual object detection Input: pre-trained deep learning model M 0 and initial dataset D0 Output: reinforced deep learning model Step1: initialize No. of iterations i ← 1 Step2: consist of the following sub-steps Step2.1: i ← i + 1 Step2.2: tune model M i−1 with dataset Di−1 and get a new updated model M i Step2.3: according to (2), gain image blocks set Ri Step2.4: classify Ri with model M i , and get P i according to (3) Step2.5: get Di based on (4) Step3: if iteration terminates, turn to Step4, else Step2 Step4: Output latest model M i

2.2

Experiment and Analysis

To verify the effectiveness of proposed algorithm, we compare on Flickr-32 LOGO dataset our algorithm with RCNN. This dataset contains 32 different LOGO and is split into three groups: training set, validation set and test set. Training set consists of 320 images with 10 per class. Validation set and test set consist 960 images with 30 per class, respectively. Also, we use ILSVRC2012 to pretrain CNN neural network model M 0 . Selective Search Algorithm [21] is used as region proposals. For the consideration of fairness, we remove the last softmax layer and add a linear SVM. One thing should be noticed that the proposed method is kinda like RCNN. RCNN belongs to supervised algorithm, which needs the position label of LOGO, but the proposed method doesn’t need.

500

M. Lin et al.

All the experiments were complemented with python and conducted on a Dell workstation with 2 Intel E5 processors, 64G memories, 4G Navidia Quadro GPU and 8 T hard disk. Figure 2 shows how our proposed algorithm updates the dataset. As we can see, the logo object becomes a focus as iterations with a stronger confidence coefficient.

Fig. 2. An example of proposed iterative based algorithm. As iteration goes (from (a) to (d), from (e) to (h)), object becomes clear.

We conduct experiments on Flickr dataset, and the results are compared with the art-of-state RCNN algorithm. We use mAP as an evaluation criterion. The results are shown on Table 1. The first shows the evaluation of accuracy of R-CNN and second shows ours. In third line and fourth line, position regression are added to RCNN and proposed algorithm, denoted as R-CNN-BB and OUR-BB respectively. We should take care that the CNN network in RCNN uses 200 thousand training images for fine tune. But for our model, only 320 images are used for first fine ture and in the 12nd iteration, we acquire up to 4 thousand images. What’s more, as we can see from the table, our proposed method significantly improves over RCNN, with 0.14% improvement comparing R-CNN-BB with OUR-BB. Compared with RCNN, our proposed method - Iterative semi-supervised deep neural network model shows advances. Three general advantages are summarized as following: First, our method can find the most stable and important inner-class features. If an image is discrete point, only a few training data can be derivated. Second, our method has low demand on training data that there is no need to know the position of logo in the image. The training data in the next round is complemented by the confidence coefficient, while RCNN model needs strong supervised information, where positive data is defined by value of IoU (above 0.5).

Sensitive Information Detection on Cyber-Space

501

Table 1. Experiment results on different logo classes comparing proposed method with RCNN Class

Starbucks

Heineken Tsingtao Guiness Corona Adidas Google Pepsi Apple DHL

R-CNN

99.51

75.79

80.58

OUR

77.72

86.83

51.30

HP

61.28

57.19 75.77 36.94 48.33

99.51

73.57

81.17

73.39

88.90

56.79

66.40

58.80 67.89 37.95 52.78

R-CNN-BB 99.51

74.90

83.45

79.11

90.91

53.38

68.39

61.08 76.12 45.00 47.90

OUR-BB

99.28

71.03

85.00

81.86

91.63

53.47

77.14

Class

Rittersport Carlsberg Paulaner Fosters Nvidia

Singha Fedex

Becks Aldi

R-CNN

87.16

49.59

98.33

76.59 88.47 84.49 88.11

OUR

86.11

86.76

68.55

80.38

70.11

68.48 73.22 45.95 56.02 Ford

UPS

50.11

94.82

90.44

64.45

84.60

71.25

76.55 89.56 85.09 85.98

R-CNN-BB 88.52

52.61

98.33

90.33

71.16

80.61

71.17

76.72 89.78 85.27 87.99

OUR-BB

88.03

59.46

95.19

90.19

67.52

81.83

76.12

Class

Stellaartois Erdinger

R-CNN

81.50

OUR

80.29

72.73 90.17 85.58 86.93

Cocacola Ferrari

Chimay Texaco Milka

Shell

52.92

67.02

88.94

64.81

81.82

72.73 89.92 82.07 74.07

58.66

Esso

bmw

mAP

50.54

66.73

89.74

66.41

80.68

54.08

72.49 90.15 79.60 73.96

R-CNN-BB 80.18

70.65

72.22

90.32

64.93

81.82

62.21

72.73 97.93 82.46 76.49

OUR-BB

61.24

67.43

90.91

68.04

79.80

61.58

79.72 89.66 78.05 76.63

79.72

Third, 33 softmax-layers were used in RCNN while ours only use 32 channels in softmax output. We focus on classifying different classes.

3

Sensitive Audio Information Detection on the Internet

Audio data is also a kind of inter-media for illegal information, through which, lawbreakers spread violence and religious extremism, like religious music, oath slogan and so on. Even identical audio context can have disparate voice properties for different individuals in various scenes. However, the melody information that music has, is identical even though individuals have unlike voice properties. 3.1

Humming Based Sensitive Audio Information Detection

The essence of Query by humming [10, 11,13, 16, 17, 22–24] is to detect a specific context of voice by utilizing these unchangeable melody information. In this paper, we put forward a new audio detection method which is based on melody feature. In this proposed method, Empirical Mode Decomposition(EMD) is introduced, with Dynamic Time Warping(DTW) combined. The whole framework is shown as Fig. 3. The whole system can be loosely translated into three parts. First part focuses on dataset construction, in which various sensitive audio information is collected. And then, note feature and pitch feature are collected for each audio. Second part conducts pitch feature extraction of query data, after which feature transformation is applied to extract note feature. Third part is matching stage. Top N nearest neighborhoods with minimum EMD distance of note feature, are selected as candidates. Then, DTW is applied to these candidates to match distance of pitch feature. We re-rank candidates by linear weighting.

502

M. Lin et al.

Fig. 3. A whole framework of sensitive audio detection.

3.2

Experiment and Analysis

To verify feasibility of our framework, we conduct simulation experiment on MIREX competition dataset, where a total of 2,048 songs exist, including 48 target humming songs and others belonging to noise data. Also, 4,431 humming songs are used as queries. Partial searching results are shown in Fig. 4. As we can see, the vast majority of humming query can find its corresponding source songs with a 93% retrieval rate.

Fig. 4. A whole framework of sensitive audio detection.

Sensitive Information Detection on Cyber-Space

4

503

Conclusion

Abnormal sensitive information on the Internet lies in various multimedia, like text, video or audio. As far as text type, existing algorithms can figure it out with efficient results and instantaneity. For video or audio, though enough works are insisting on them, they are still un-solved, which remains an open problem. In this paper, we propose two algorithms, i.e., iterative based semi-supervised deep learning model and Humming melody based search model, to detect abnormal visual and audio objects respectively. And experiments show the feasibility of our proposed methods.

References 1. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006) 2. Cao, L., Luo, F., Chen, L., Sheng, Y., Wang, H., Wang, C., Ji, R.: Weakly supervised vehicle detection in satellite images via multi-instance discriminative learning. Pattern Recogn. 64, 417–424 (2017) 3. Chen, D., Ren, S., Wei, Y., Cao, X., Sun, J.: Joint cascade face detection and alignment. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 109–122. Springer, Cham (2014). https://doi.org/10.1007/ 978-3-319-10599-4 8 4. Chen, X., Xiang, S., Liu, C.-L., Pan, C.-H.: Vehicle detection in satellite images by parallel deep convolutional neural networks. In: 2013 2nd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 181–185. IEEE (2013) 5. Cheng, X.Q., Jin, X., Wang, Y., Guo, J., Zhang, T., Li, G.: Survey on big data system and analytic technology. J. Softw. 25(9), 1889–1908 (2014) 6. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 1, pp. 886–893. IEEE (2005) 7. Foote, J.T.: Content-based retrieval of music and audio. In: Voice, Video, and Data Communications, pp. 138–147. International Society for Optics and Photonics (1997) 8. Geronimo, D., Lopez, A., Sappa, A.D., Graf, T.: Survey of pedestrian detection for advanced driver assistance systems. IEEE Trans. Pattern Anal. Mach. Intell. 32(7), 1239–1258 (2010) 9. Ghias, A., Logan, J., Chamberlin, D., Smith, B.C.: Query by humming: musical information retrieval in an audio database. In: Proceedings of the Third ACM International Conference on Multimedia, pp. 231–236. ACM (1995) 10. Haitsma, J., Kalker, T.: A highly robust audio fingerprinting system. In: ISMIR, vol. 2002, pp. 107–115 (2002) 11. Roger Jang, J.-S., Gao, M.-Y.: A query-by-singing system based on dynamic programming. In: Proceedings of International Workshop on Intelligent System Resolutions (8th Bellman Continuum), Hsinchu, pp. 85–89. Citeseer (2000) 12. Ji, R., Liu, W., Xie, X., Chen, Y., Luo, J.: Mobile social multimedia analytics in the big data era: An introduction to the special issue. ACM Trans. Intell. Syst. Technol. (TIST) 8(3), 34 (2017)

504

M. Lin et al.

13. Jiang, H., Xu, B., et al.: Query by humming via multiscale transportation distance in random query occurrence context. In: 2008 IEEE International Conference on Multimedia and Expo (2008) 14. Jiang, X., Ping, Y.: Research and implementation of lucene-based retrieval system of audio and video resources. Jisuanji Yingyong yu Ruanjian 28(11), 245–248 (2011) 15. Liu, C.-C., Chang, P.-F.: An efficient audio fingerprint design for MP3 music. In: Proceedings of the 9th International Conference on Advances in Mobile Computing and Multimedia, pp. 190–193. ACM (2011) 16. Liu, H., Ji, R., Wu, Y., Liu, W.: Towards optimal binary code learning via ordinal embedding. In: AAAI, pp. 1258–1265 (2016) 17. McNab, R.J., Smith, L.A., Witten, I.H., Henderson, C.L., Cunningham, S.J.: Towards the digital music library: Tune retrieval from acoustic input. In: Proceedings of the first ACM International Conference on Digital Libraries, pp. 11–18. ACM (1996) 18. Su, S.-Z., Li, S.-Z., Chen, S.-Y., Cai, G.-R., Wu, Y.-D.: A survey on pedestrian detection. Dianzi Xuebao (Acta Electronica Sinica) 40(4), 814–820 (2012) 19. Sun, Z., Bebis, G., Miller, R.: On-road vehicle detection: A review. IEEE Trans. Pattern Anal. Mach. Intell. 28(5), 694–711 (2006) 20. Tian, Y., Luo, P., Wang, X., Tang, X.: Pedestrian detection aided by deep learning semantic tasks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5079–5087 (2015) 21. Uijlings, J.R.R., Van De Sande, K.E.A., Gevers, T., Smeulders, A.W.M.: Selective search for object recognition. Int. J. Comput. Vision 104(2), 154–171 (2013) 22. Wang, L., Huang, S., Hu, S., Liang, J., Xu, B.: An effective and efficient method for query by humming system based on multi-similarity measurement fusion. In: International Conference on Audio, Language and Image Processing, ICALIP 2008, pp. 471–475. IEEE (2008) 23. Wang, Q., Guo, Z., Li, B., Liu, G., Guo, J.: Tempo variation based multilayer filters for query by humming. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp. 3034–3037. IEEE (2012) 24. Wu, X., Li, M., Liu, J., Yang, J., Yan, Y.: A top-down approach to melody match in pitch contour for query by humming. In: Proceedings of 5th International Symposium on Chinese Spoken Language Processing, Singapore (2006) 25. Yang, S., Luo, P., Loy, C.-C., Tang, X.: From facial parts responses to face detection: a deep learning approach. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3676–3684 (2015) 26. Zhong, B., Yuan, X., Ji, R., Yan, Y., Cui, Z., Hong, X., Chen, Y., Wang, T., Chen, D., Jiaxin, Y.: Structured partial least squares for simultaneous object tracking and segmentation. Neurocomputing 133, 317–327 (2014)

Optimization Algorithm Toward Deep Features Based Camera Pose Estimation Han Chen, Feng Guo(B) , Ying Lin, and Rongrong Ji Department of Cognitive Science, School of Information Science and Engineering, Xiamen University, Xiamen 361005, China [email protected]

Abstract. Deep convolutional neural networks are proved to be end-toend localization method which tolerates large baselines. However, it relies on the training data and furthermore, camera pose estimation results are not robustness and smooth enough. It all boils down to without backend optimization (e.g. local bundle adjustment). This paper proposes a deep features based SLAM optimization method as well as improves the pose estimation precision by the constraint function. The contribution of our paper is two-fold: (1) We present constraint function based on similarity for fast 2D-3D points mapping and a new optimization approach that estimates camera exterior parameter using multiple feature fusion. (2) For the problem of instability in Cnn based SLAM, a multiple features ensemble bundle adjustment optimization algorithm is presented. Most existing localization approaches simply approximate pose confidence based on reference point distance. Unlike previous work, we employ reconstruction data as a reference, then, the visible 3D points and its related key-points from off-line data sets by random forests are mapped, and a multiple feature fusion is used to measure the assessment score by an constraint function. The above method is used to optimize deep features based SLAM. Experimental results demonstrate the robustness analysis of our algorithm in handling various challenges for estimation of camera pose. Keywords: Estimation of pose estimation Mapping of 2D-3D points

1

· Multiple feature fusion

Introduction

Camera re-localization has been an important task in computer vision. Imagebased re-localization approach is an robust and efficient pipeline in this task. Two common techniques for imaged-based camera pose estimation are image retrieval and 3D scene reconstruction based methods. However, image retrieval error [1,2, 5–9] is larger than the GPS positions sensor. Moreover, the estimation precision is also depend on dataset. Zamir and Shah [5] present a multiple nearest neighbor feature matching framework for geo-locating. This method is restricted if the query feature and dataset features are mismatched. c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 505–514, 2017. https://doi.org/10.1007/978-3-319-71598-8_45

506

H. Chen et al.

To achieve a much better localization result, the reconstruction information utilized to provide valued priori spatial relationships [3, 4,11–13]. However, the existing image-based methods only focus on localization but neglect the followup optimization. To the localization optimization, recent SLAM algorithms are seldom considering the assessment of evaluation localization [17, 18], some of these approaches, such as FastSLAM [20], estimate noise by sensor to measure the estimation pose error. However, these methods can’t represent the model’s efficiency and do nothing to optimization. ORB-SLAM [23] is a novel monocular SLAM system that uses orb-feature and complex optimization. Unlike to us, it’s not used deep feature. The Scene Coordinate Regression Forests [25] use depth images to create scene coordinate labels and map each point from camera coordinates to global scene coordinates. Then, a regression forest is trained to regress these labels and estimates camera pose. However, this algorithm is limited to RGB-D images and its weakness that it only used to indoor scenes. Alex Kendall et al. [10] present a visual re-localization system that uses a Bayesian convolutional neural network to regress camera pose. They trade off reuse of reconstruction information and prefer to real-time performance. Nevertheless, Our proposed method does not utilize any sensor noise data and assess the localization model by point-based similarity through priori reconstruction. In this paper, we first demonstrate how random forests [26] can be used to assess the reconstruction without additional ground-truth, than, we show how can I optimize the deep network SLAM. Our main contribution is a new framework to assess camera exterior parameter by using multiple feature fusion and key-point selection algorithm by random forests and an efficient mapping method to generate relationship of 2D and 3D point. The second contribution is optimize deep network SLAM by above framework.

2

Random Forests for 2D-3D Mapping

We construct random forests with the purpose of finding the relationship between query image and reconstructed data. We novely use the mapping framework to assess the pose evaluated by different localization algorithm. We adopt random forests algorithm owing to its advantages that the lower complexity and robustness enable real time efficient assessment. In essence, the randomized trees are only a mapping tool to retrieve candidate triangle relationship from 2D to 3D. Training of Randomized Tree. A random forest is an ensemble of I decision trees, each consisting of internal and leaf nodes. To evaluate a randomized tree enable to calculate the similarity of 2D pixel point, we start at the root node and descend to a leaf by repeatedly revising the decision function: split(p; δn ) = [fn (p) > δn ]

(1)

where n expresses the index of a node in the randomized tree, p is a key point that passes a non-leaf node, [.] is the 0–1 indicator, δ is a threshold and f () is decision function:

Optimization Algorithm Toward Deep Features

f (P ) = a1 Dshape (p1 , p2 ) + a2 Dtexture (p1 , p2 ) +a3 Dcolor (p1 , p2 )

507

(2)

We define a and D() below in Sect. 3. If split(p; δn ) evaluates to 0, the path branches to the left sub-tree, otherwise it branches to the right. p1 and p2 are point of a pixel pair around the key point. In the training step, reconstruction data contain relevant corresponding 3D and 2D point features. We divide them into training feature and verified features. We show training framework in the Algorithm 1. To be noted that we use the FALoG [21] to faster detection is similar as Youji Feng et al. [13]. The target function Q is used to make division of training and verified features has the same trend as far as possible. Where Θ is the number that verified features divided into the different path of its related training feature. Pverif ication is the set of verified features. λ is a trade off between the similarity of same branches and the diversity of different branches. The target function is based on the ratio that denotes the same decision of training and verified features and measures the partition similarity between them. Q=

Θ |Pverif ication |

[

λ il,jr||ir,jl fi (p) − fj (p)2  + fi (p) − fj (p)2 ]

(3)

i,jl||i,jr

Algorithm 1. Training of Randomized Tree Input: A random sample of collection features P that contains training features and verified features; Output: A trained randomized tree; 1: Initialize a set of random decision function S = {fi (P )}n i=1 2: while remain non-leaf nodes or none of feature samples are survived do 3: Apply each decision fi (P ) to the training features and verified features to divide them into two subsets Pl and Pr . 4: Count the error ep of the verified features surviving the path, i.e. a verified feature is divided the same sub-tree as its corresponding training feature. 5: Calculate the target function: Q 6: Choose the best decision f (P ) which minimizes target function as the test of the node:Ibest =arg minS (f S) (Q) 7: Then, according to best decision function, the rest survival feature samples are divided into two child nodes. 8: end while 9: return Randomized tree I;

508

3

H. Chen et al.

Feature Fusion Algorithm

Although the relationship of 2D to 3D is easy to be obtained by above mapping. It is troublesome for us to judge whether camera pose estimation is good or bad that absence of image pose label. We design an assessment algorithm without manual labeling that utilities the information of point cloud (reconstruction data used in Sect. 2.) and visual geometry to assess the pose estimation. The camera pose is calculated by whatever camera pose prediction algorithm. We assume that  the pose estimation result is the camera exterior R −RCw converted by the x, y, z, (position coordinates) and parameter 0 1 w, p, q, r (4 quaternions). We assume the intrinsic matrix K is accessed by EXIF⎡tags and without ⎤ radial distortion. Then, combining intrinsic parameter f mx 0 px 0 K = ⎣ 0 f my py 0⎦, we can get the transformation matrix by image coordi0 0 1 0 nate and world coordinate, and the transformation is: ⎡ ⎤ ⎡ ⎤ X wx ⎢Y ⎥ ⎥ ⎣wy ⎦ = KR[I| − Cw ] ⎢ (4) ⎣Z ⎦ w image 1 world

2D points and 3D point clouds are related by this transformation matrix. As an evaluating image which has been evaluated the camera matrix, we extract the FALoG query features as set P. For a feature point m(x,y), the approximate nearest feature of a query feature can be found by above mapping algorithm. Showed in the Fig. 1, We search the nearest dataset feature in an established random forests by testing the query feature in each decision node, and finally the query feature arrives at leaf node. The final mapped feature m′ of the forest at point m is simply the most account of node category. Corresponding point

Fig. 1. The framework of our assessment algorithm

Optimization Algorithm Toward Deep Features

509

m′ has its related 3D point. For each 3D point, the database will have at least one corresponding image, each point can be projected into related image. We combined with tree difference pixels-wised feature to compose error function, and use this error function to iterative optimize the pose matrix. We consider tree pixel-wised characteristics that are able to express the variance of different points, respectively are color-based, shape-based and texturebased. For each feature, it can be separately has an error term. Finally, we assess the pose estimation by the tree error item. 1. Color-Based. Color feature [27] imposes the surface properties of the object in an image. In this paper, we adopt the fusion method that fuses single pixel and region pixels feature for maximum expression of the relationship between single and region color features. So we have reached to measure the assessment of pose evaluation by color feature. We simply use color distance function to distinguish between two single pixel color variance: (|R(x, y) − R(x′ , y ′ )| ′ + dpoint (p, p ) = 256 |G(x, y) − G(x′ , y ′ )| |B(x, y) − B(x′ , y ′ )| + (5) 256 256 where p(x, y) and p′ (x′ , y ′ ) are two target point, R(.), G(.) and B(.) are respective RGB values. For region color feature, we assume the target point included in a region that has 5*5 pixels in this paper, showed in Fig. 2.(a). For each RGB channel, center of the region is the target point. We extract gradient values G(x, y) in the horizontal and vertical direction of the target point (x, y). +dv ) which dh is the difference between left two pixels G(x, y) = ln(1 + dh512 average value of target point and average of two pixels on the right. dv by the same way of dh , the orange color showed in Fig. 2(a). For the rest of the four 2 ∗ 2 pixel block(blue color showed in Fig. 2(a)), we down-sample them by average. Then, the color-based diversity is: 

n  i ′ ′ δdregion (p, p ) Dcolor = dpoint (p, p ) + i=1

+η dG (p, p′ )2

(6)

where δǫ{0, 1}, if the diversity of two block down-sample value below dpoint (p, p′ ), δ is set to 0, otherwise it is 1. dregion (p, p′ ) is adopted the same calculation with dpoint (p, p′ ). dG (p, p′ ) is the difference of gradient value and η is regular factor. 2. Shape-Based and Texture-Based. In order to extract shape features more quickly and efficiently for a pixel point, as the center in the patch of 32 ∗ 32 block, we extract shape context

510

H. Chen et al.

feature [22], and the computation of shape context for 25 sample point around. Showed in Fig. 2(b), we distribute the space of the patch with the logarithmic coordinates that is divided into 12∗3 = 36 parts. Finally, the histogram is used to express shape feature vector. To express the shape feature difference, we binary transform the diversity of two point shape histogram, that, Dshape = n item i , where n = 36 and itemǫ{0, 1}, itemi = 1 means the column i=1 value of i patch is larger than average of histogram. For texture, a point in the center of a circle that radius is R, is divided into eight equal Angle area. We calculate average image intensity in each area, if the average is greater than the center pixel value, then the value of this area is set to 1, to the contrary, it’s set to be 0. The eight binary sequences are converted to decimal number to express the texture feature. To the end, the Dtexture is be set by Hamming distance.

Fig. 2. (a) Diversity of 5*5 pixel region for shape-based feature. (b) Shape-Based feature (Color figure online)

Finally, our assessment function E is a combination of the above tree pixelwised feature: E(pi , pj ) = a1 Dshape (pi , pj ) + a2 Dtexture (pi , pj ) +   a3 Dcolor (pi , pj )2

(7)

where ai ǫ(0, 1) denodes the weight of different items. Dcolor is to be treat as a regular term in this fusion diversity function. Dshape and Dtexture are jointed together. We train the parameter in this function by k iteration to make the assessment of evaluation pose to the best. Assessment function describes the pose evaluation error can measure in a way that utilities diversity of appearance between query 2D feature and its re-projection point. Therefore, our approach can successfully assess estimated pose with a known scene.

4

Back-End Optimization Toward Deep Features

Localization error will be reflected in the predicted pose between different keyframes and reconstruction error. The accumulation of keyframe error lead to the global error Growing which limits the precision of the overall system.

Optimization Algorithm Toward Deep Features

511

Deep features based SLAM stores the 2D-3D mapping and keyframes by opening a new thread. We choose keyframes that the E(pi , pj ) < ∂ to the BA (bundle adjustment) [24] keyframes set. Local BA optimizes covisible keyframes which all points can be seen. Observing points contribute to constraint function and the final pose is the ensemble of optimization result and deep network estimation. Global BA is the specific case of local BA and all BA keyframes set and points are optimized, except the origin keyframe. The system will execute local BA between 30 frames and global BA between 300 frames.

5

Experiment

In this section we demonstrate a variety of strategies to obtain assessment of pose estimation results can resolve the problem that absence of ground-truth. We compare different localization with us. Experiment also prove join BA with our constraint function has indeed improve pose estimation. The algorithm can be effect available in the situation of merely has to be evaluated images. 5.1

Dataset

Off-the-shelf state-of-the-art supervised learning camera pose estimation algorithms are specialize in accuracy of pose evaluation, but they overlooked assess pose estimation when absence of camera pose ground truth data. For this paper, We use video of each scene to cut frames as training data and leverage structure from motion to label camera pose to image data automatically. All scenes were recorded by MEIZU MX4 and cut at 720*576 resolution images without location and EXIF information. Test images are crawled from Sina Weibo that without pose labels. Table 1. Comparison of our approach and other state-of-the-art methods in different datasets. Scene’s name

Error (failed frames with a threshold (indoor: 5 cm and 5◦ , outdoor: 2 m and 10◦ )/testing frames) Features fusion

Nearest neighbour [5]

SCoRe [25]

Posenet ORB[10] SLAM [23]

Ours (without BA)

Ours (BA)

Chess [25]

72.6%

81.2%

91.8%

93.2%

95.4%

95.0%

96.2%

Fire [25]

65.7%

69.6%

81.6%

89.4%

90.3%

90.1%

90.6%

Heads [25]

41.7%

47.7%

53.7%

63.0%

65.5%

63.5%

64.9%

Office [25]

63.6%

72.3%

80.2%

86.6%

88.9%

88.3%

89.5%

Song’en Building

57.3%

64.3%

N/A

83.3%

87.4%

87.0%

88.4%

St Mary’s Church [10] 56.7%

62.3%

N/A

82.1%

85.7%

85.1%

85.5%

Shop Facade [10]

63.4%

67.3%

N/A

87.2%

88.8%

88.1%

89.7%

King’s College [10]

64.5%

66.3%

N/A

85.6%

87.3%

87.1%

88.0%

512

H. Chen et al.

5.2

Performance

In Table 1, we show the percentages of proportion of ‘correct’ test frames in eight datasets. We set two threshold(indoor and outdoor respectively) to measure a test image is evaluated a positive pose or not. Obviously, single feature(colorbased, shape-based and texture based) is harder to assess pose estimation because its boundedness. Posenet [10] and ORB-SLAM [23] have done well in assessment task, what’s more, SCoRe Forest [25] which uses depth is failed on outdoor scenes. However, our fusion feature is robust and accuracy than other approach. For the time cost, our method takes approximately 100ms per frame on a modern CPU core.

6

Conclusions

In this paper, we have proposed a novel optimization algorithm for camera pose estimation in real-time. We have shown how to establish 2D-3D points mapping by random forests. We propose a new method that assesses camera exterior parameter by using multiple feature fusion. This allows us to improve pose evaluation confidence in the case that used deep feature. For the problem of instability in deep features based SLAM, we add constraint function based back-end optimization to improve pose estimation. To sum up, our experiments show that the proposed algorithm is robust and accuracy than other state-of-the-art methods. Acknowledgment. This work is supported by the National Key R&D Program of China, No. 2016YFB1001503.

References 1. Robertson, D.P., Cipolla, R.: An image-based system for urban navigation. In: Proceedings British Machine Vision Conference, BMVC 2004, Kingston, UK, 7–9 September 2004. https://doi.org/10.5244/C.18.84 2. Zhang, W., Kosecka, J.: Image based localization in urban environments. In: 3rd International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT 2006), Chapel Hill, North Carolina, USA, 14–16 June 2006. https://doi. org/10.1109/3DPVT.2006.80 3. Irschara, A., Zach, C., Frahm, J.M., Bischof, H.: From structure-from-motion point clouds to fast location recognition (2009) 4. Dong, Z., Zhang, G., Jia, J., Bao, H.: Keyframe-based real-time camera tracking. In: IEEE 12th International Conference on Computer Vision, ICCV 2009, Kyoto, Japan, 27 September–4 October 2009. https://doi.org/10.1109/ICCV.2009. 5459273 5. Zamir, A.R., Shah, M.: Image geo-localization based on multiple nearest neighbor feature matching using generalized graphs. IEEE Trans. Pattern Anal. Mach. Intell. (2014) 6. Cao, S., Snavely, N.: Graph-based discriminative learning for location recognition. Int. J. Comput. Vis. (2015). https://doi.org/10.1007/s11263-014-0774-9

Optimization Algorithm Toward Deep Features

513

7. Chumm, O., Philbin, J., Sivic, J., Isard, M., Zisserman, A.: Total recall: automatic query expansion with a generative feature model for object retrieval. In: IEEE 11th International Conference on Computer Vision, ICCV 2007, Rio de Janeiro, Brazil, 14–20 October 2007. https://doi.org/10.1109/ICCV.2007.4408891 8. Wang, J., Zha, H., Cipolla, R.: Coarse-to-fine vision-based localization by indexing scale-invariant features. IEEE Trans. Syst. Man Cybern., Part (2006). https://doi. org/10.1109/TSMCB.2005.859085 9. Kendall, A., Cipolla, R.: Object retrieval with large vocabularies and fast spatial matching. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), Minneapolis, Minnesota, 18–23 June 2007. https://doi. org/10.1109/CVPR.2007.383172 10. Kendall, A., Cipolla, R.: Modelling uncertainty in deep learning for camera relocalization. In: IEEE International Conference on Robotics and Automation, ICRA 2016, Stockholm, Sweden, 16–21 May 2016. https://doi.org/10.1109/ICRA.2016. 7487679 11. Li, Y., Snavely, N., Huttenlocher, D.P.: Location recognition using prioritized feature matching. In: IEEE European Conference on Computer Vision, Heraklion, Crete, Greece, 5–11 September 2010. https://doi.org/10.1007/978-3-642-155529 57 12. Sattler, T., Weyand, T., Leibe, B., Kobbelt, L.: Image retrieval for image-based localization revisited. In: British Machine Vision Conference, BMVC 2012, Surrey, UK, 3–7 September 2012. https://doi.org/10.5244/C.26.76 13. Feng, Y., Fan, L., Wu, Y.: Fast localization in large-scale environments using supervised indexing of binary features. IEEE Trans. Image Process. (2016). https://doi. org/10.1109/TIP.2015.2500030 14. Arya, S., Mount, D.M., Netanyahu, N.S., Silverman, R., Wu, A.Y.: An optimal algorithm for approximate nearest neighbor searching fixed dimensions. ACM (2012). https://doi.org/10.1145/293347.293348 15. Harltey, A., Zisserman, A.: Multiple View Geometry in Computer Vision, 2 edn 16. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. ACM (2011). https://doi.org/10.1145/358669.358692 17. Klein, G., Murray, D.W.: Parallel tracking and mapping for small AR workspaces. In: International Symposium on Mixed and Augmented Reality, ISMAR 2007, Nara, Japan, 13–16 November 2007. https://doi.org/10.1109/ISMAR.2007. 4538852 18. Engel, J., Sch¨ ops, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Proceedings ECCV 2014–13th European Conference, Zurich, Switzerland, 6–12 September 2014, Part II (2014). https://doi.org/10.1007/978-3-31910605-2 54 19. Li, Y., Snavely, N., Dan, H., Fua, P.: Worldwide pose estimation using 3D point clouds 20. Parsons, S.: Probabilistic robotics by Sebastian Thrun, Wolfram Burgard and Dieter Fox. Knowledge Eng. Review (2006). https://doi.org/10.1017/ S0269888906210993 21. Wang, Z., Fan, B., Wu, F.: FRIF: fast robust invariant feature. In: British Machine Vision Conference, BMVC 2013, Bristol, UK, 9–13 September 2013. https://doi. org/10.5244/C.27.16 22. Belongie, S.J., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell (2002). https://doi.org/ 10.1109/34.993558

514

H. Chen et al.

23. Mur-Artal, R., Montiel, J.M.M., Tard´ os, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM System. IEEE Trans. Robot. (2015) 24. Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Multicore bundle adjustment. Comput. Vis. Pattern Recogn. (2011) 25. Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., Fitzgibbon, A.: Scene coordinate regression forests for camera relocalization in RGB-D images. In: Computer Vision and Pattern Recognition (2013) 26. Cutler, A., Cutler, D.R., Stevens, J.R.: Random forests. In: Machine Learning (2012) 27. Smeulders, A.W.M., Bagdanov, A.D., Gevers, T., Gijsenij, A., Van De Weijer, J., Geusebroek, J.: Color Feature Detection (2011)

Security

A Robust 3D Video Watermarking Scheme Based on Multi-modal Visual Redundancy Congxin Cheng1,2 , Wei Ma1,2(B) , Yuchen Yang1,2 , Shiyang Zhang1,2 , and Mana Zheng1,2 1

2

Faculty of Information Technology, Beijing University of Technology, Beijing, China [email protected] Beijing Key Laboratory of Trusted Computing, Beijing, China

Abstract. Image watermarking is a popular research topic in signal processing. The paper presents a blind watermarking scheme for 3D videos. Given a 3D video, each frame of both views is divided into blocks. Watermark information is embedded by modulating selected DCT coefficients of each block. The modulation strength is controlled by multi-modal visual redundancies existing in the 3D video. Specifically, we compute an intra-frame Just-noticeable Distortion (JND) value and an interframe reference value for the block to determine the strength. The former reflects the visual redundancies in the image plane. The latter represents the visual redundancies of the block from aspects of motion between sequential frames and disparity between the left and right views. We validate the robustness of the proposed watermarking scheme under various attacks through experiments. More importantly, the visual quality of the 3D videos watermarked by our scheme is proved to be as good as that of the original videos, by a proposed LDQ (Loss of Disparity Quality) criterion specially designed for 3D videos, as well as PSNR of single views. Keywords: Stereo video watermarking · Blind watermarking Robustness · Perceptual redundancy · Disparity

1

Introduction

With the rapid development of 3D technologies, high-definition stereo media is widely used in many areas, such as 3D movies, virtual reality, etc. Such media is invaluable. How to protect them from piracy has become a significant issue [2, 6,9–11,14,15]. Digital watermarking is an effective technique to achieve this task via embedding the owners’ information into the 3D media [2, 10, 17]. There are three main considerations for digital watermarking schemes, transparency, robustness and capacity, respectively [5]. First, the quality of the original images/videos should not be affected after being watermarked. Otherwise, it will decrease the commercial value of the media. Second, watermarks could c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 517–526, 2017. https://doi.org/10.1007/978-3-319-71598-8_46

518

C. Cheng et al.

not be removed intentionally or unintentionally. It means that the watermarking schemes should be robust to various attacks of changing the media, e.g. cropping and scaling. Third, the schemes should be able to embed watermarks carrying enough information, i.e. having high capacity. 3D videos generally have two forms, lightweight depth-image-based rendered videos [4] and high-quality stereoscopic videos with left-right views [16]. The former is generally used in Internet or TV systems. The latter is popular in offline movie market. We are targeting at watermarking high-quality stereoscopic videos, rather than lightweight ones [1], to claim their copyrights in the offline market. Watermarking in these videos is more challenging due to the sharp conflict between high visual quality requirements and capacity. In recent years, there appear many watermarking methods for stereo videos. An intuitive strategy is to apply single view image watermarking schemes [8] to every frames of both views of stereo videos, individually as done in [9]. These methods tend to cause visual quality distortion in the watermarked videos, since consistencies between frames and the two views are corrupted. Wu et al. [15] embedded watermarks by altering the DCT coefficients of the left and right views in opposite directions. Although their method is robust to many attacks, the correspondences between the left and right views are damaged, which leads to visual discomfort. Rana et al. [13] determined the embedding positions in the hosts by considered both depth and motion information. This method keeps the consistencies between frames and those between views. However, it sacrifices capacity for the consistencies. Besides, the embedded positions are fragile to attacks. Mohghegh et al. [7] controlled the embedding strength in order to keep stereo correspondences. This method has small capacity and low robustness against salt & pepper noise attacks. This paper presents a blind watermarking scheme for 3D videos. It explores multi-modal visual redundancies in the 3D videos, thereby having high capacity and robustness while keeping good visual quality in watermarked videos. Given a 3D video, at first, all frames of both left and right views are partitioned into non-overlapped blocks. For each block, we compute its DCT coefficients. In the meanwhile, we compute a depth map and a set of motion vectors for each frame. The depth values and the motion vectors in a block are used to calculate a reference value for the block, which represents the perceptual redundancies existing between sequential frames and the left and right views. The reference value, together with Just-noticeable Distortion (JND) [14] which represents those perceptual redundancies in single frames, are used to control the embedding strength of the watermark in the DCT coefficients. The control can effectively preserve visual quality of watermarked videos by avoiding over embedding. In the detection phase, we simply perform the above blocking and DCT steps, and extract watermarks by comparing the DCT coefficients. Experimental results show that our scheme provides high capacity, obtaining good visual quality of watermarked videos and high robustness against various attacks. On the other hand, state-of-the-art watermarking methods [2, 9] generally use PSNR as criterions to evaluate the visual quality of watermarked videos.

A Robust 3D Video Watermarking Scheme

519

PSNR is originally designed for single view images. The consistency of the left and right views, which is significant for high-quality 3D videos [5], are not considered. In this paper, we propose a new measurement, called Loss of Disparity Quality (LDQ). It defines the visual quality of 3D videos from the view of depth perception, which is essential to the visual experiences of 3D videos. LDQ is used together with PSNR to evaluate the visual quality of 3D watermarked videos in this paper. There are two main contributions in this paper. Firstly, a watermarking scheme for 3D videos is proposed and validated. It is demonstrated to have better performances than state-of-the-art methods. Secondly, to the best of our knowledge, we are the first to give an evaluation method specifically designed for 3D video watermarking.

2

Proposed Method

In this section, we explain the embedding and extracting processes of the proposed method, respectively. Since the extracting is simply an inverse process of the embedding, we put more effort in describing the details of the embedding stage. 2.1

Embedding

Given a stereo video, the same operations are carried out on every pair of left and right frames. First, we divide a pair of left and right frames into non-overlapping blocks of size of 8 × 8 pixels. Then, DCT coefficients of each block are computed. Next, watermark embedding is performed for each pair of blocks at the same positions of the left and right frames. Each bit of a watermark, which takes the form of a binary image in this paper, is repeatedly embedded in five selected pairs of middle-frequency DCT coefficients in the two blocks in order to descend the probability of losing information during malicious attacks. The embedding strength is sophisticatedly controlled by various human visual perception factors in viewing stereo videos. We choose five middle-frequency DCT coefficients (indicated in blue in the grid of Fig. 1) in both of the two blocks to embed a single bit of watermark for robustness. Given a selected position (i, j) in the left/right view block, we record its average value g(i,j) , g(i,j) = (C(i,j) + C(i+1,j) + C(i,j+1) + C(i+1,j+1) )/4

(1)

where C(i,j) is the DCT coefficient at (i, j). We define r l + ω2 g(i,j) Gl(i,j) = ω1 g(i,j) r l Gr(i,j) = ω1 g(i,j) + ω2 g(i,j)

(2)

520

C. Cheng et al.

Fig. 1. Pipeline of the embedding process. (Color figure online)

r l where g(i,j) and g(i,j) are the average values of the left and right view blocks at (i, j), respectively. ω1 + ω2 = 1. ω1 is chosen to be six times larger than ω2 in implementation, so that Gl(i,j) and Gr(i,j) average information from both views r l while taking a larger portion from their origins, i.e. g(i,j) and g(i,j) , respectively. Embedding a bit of the watermark, which is denoted as ω(p,q) , in the selected DCT coefficients is to alter the coefficients by the state of ω(p,q) and the values l of Gl(i,j) and Gr(i,j) . If the bit is 0, C(i,j) is modified to be slightly bigger than l r G(i,j) . C(i,j) is set to be slightly smaller than Gr(i,j) . On the contrary, if the bit l r is 1, C(i,j) and C(i,j) are set to be slightly smaller than Gl(i,j) and bigger than r l r G(i,j) , respectively. If C(i,j) have already satisfied the above polarity or C(i,j) relationship, its modulation will be passed. The above embedding process can be expressed by the following equations. l C(i,j)

r C(i,j)

=



=



l + εl(i,j) ), ω(p,q) = 0 Gl(i,j) + β(p,q) (αJN D(i,j) l + εl(i,j) ), ω(p,q) = 1 Gl(i,j) − β(p,q) (αJN D(i,j) r + εr(i,j) ), ω(p,q) = 0 Gr(i,j) − β(p,q) (αJN D(i,j)

(3)

r + εr(i,j) ), ω(p,q) = 1 Gr(i,j) + β(p,q) (αJN D(i,j)

l r where C(i,j) and C(i,j) denote the final DCT coefficients after modulation, at l r (i, j) of the left and right views, respectively. JN D(i,j) and JN D(i,j) denote the JND values [14] of the left and right views, respectively. α is a parameter controlling the influences of JND. α is empirically selected to be 0.05, which could provide robustness strong enough while ensuring visual quality of the videos after embedding. The modulation range of the DCT coefficients is restrained by the JND values and a minimal value ε. Here, ε(i,j) = 0.1g(i,j) . Moreover, in order to avoid large changes of small coefficients, the embedding process won’t be conducted if the modulation range is more than twice the value of the original DCT coefficient.

A Robust 3D Video Watermarking Scheme

521

Compared to α, fixed for all of the blocks and frames, β(p,q) in (3) is flexibly determined by a reference value R(p,q) at block (p,q),  0.5, R(p,q) = 0 (4) β(p,q) = 1, R(p,q) = 1 R(p,q) is calculated by referring to the motion and depth properties of block (p, q). At first, we compute the motion factors of pixel (i, j) in block (p, q), y x and M(i,j) , respectively. along X axis and Y axis, by [3], and recorded as M(i,j) The motion of pixel (i, j) is given by  2 x 2 + My (5) M(i,j) = M(i,j) (i,j)

In parallel, the depth of pixel (i, j), denoted as D(i,j) , is computed by the SGBM algorithm given in [6]. By combining the motion and depth information, we obtain a weighted average influence value of each block, denoted as M D(p,q) which is given by   i j (µ1 D(i,j) + µ2 M(i,j) ) (6) M D(p,q) = 64 where µ1 and µ2 indicate the percentage of depth and motion. µ1 and µ2 are empirically chosen to be 5/6 and 1/6, respectively, since in our experiments, the depth information computed by [6] is evaluated to be more reliable than the motion part obtained by [3]. The number 64 in the denominator is the number of pixels in the block. In implementation, we record the reference values of all the blocks in each frame in a reference matrix. The reference value R(p,q) of the block (p, q) is given by  1, M D(p,q) ≥ M Davg (7) R(p,q) = 0, M D(p,q) < M Davg Here, M Davg is the average value in the reference matrix. If the influence factor of block (p, q) is beyond the average value of the reference matrix, R(p,q) is set to 1, which suggests a strong embedding strength in this block, and vice versa. This is consistent with that fact that human vision system is less sensitive to objects with large motion or depth. 2.2

Extracting

In the extracting phase, given a watermarked video, we divide each frame into blocks and compute the DCT coefficients of each block, as done in the embedding step. The total differences between coefficients of a pair of corresponding blocks in the left and right views is computed by  ′r ′l (C(i,j) − C(i,j) ) z(p,q) = (8) (i,j)

522

C. Cheng et al.

where p and q are block indices. i and j are the indices of the DCT coefficients in the block. As described in the embedding part, the same bit of the watermark is embedded into five DCT coefficients in one pair of blocks for robustness. To extract a bit of the watermark, we compute the differences between corresponding DCT coefficients in the left and right view blocks. Then we sum the differences at the selected five positions to mitigate influences from attacks. The bit of the watermark w(p,q) hidden in blocks at (p,q) is extracted by z(p,q) ,  1, z(p,q) ≥ 0 (9) w(p,q) = 0, z(p,q) < 0 Note that the modification of the selected coefficients in Eq. 3 is moderately controlled for high visual quality of the videos with watermarks. Therefore, the coefficients in the left block is not necessarily changed to be larger or smaller than those in the right block, as supposed in the extraction. It means that the extracted watermark might lose little information as shown in Fig. 2. Nevertheless, the proposed method has good robustness under various attacks as we can see in the experiments.

3

Experiments

In this section, we analyse the performances of the proposed watermarking scheme in aspects of keeping visual quality and the scheme’s capacity. Beside, we also test its robustness against various attacks. Keeping visual quality is the most important factor for watermarking high quality stereo videos in off-line movie market. It is also the fundamental consideration during the design of our watermarking scheme. Three 3D videos, shown in Fig. 2, are used in the experiments. The videos, whose specifications are listed in Table 1, all have high resolution. Note that we treat the videos as a series of frames with no video compression. A binary image (shown in Fig. 2(j)) is used as a watermark pattern, which could be repetitively embedded. 3.1

Visual Quality Evaluation and Capacity Analysis

We compare the visual quality of watermarked videos generated by our method with those obtained by four state-of-the-art methods, including a visual-modulebased method proposed by Niu et al. [9], a differential watermarking scheme Table 1. Specifications of 3D video sequences Resolution

Frames

Video 1 (Fig. 2(a)) 1920 × 1080 24 Video 2 (Fig. 2(b)) 1920 × 1080 47 Video 3 (Fig. 2(c)) 1920 × 1080 82

A Robust 3D Video Watermarking Scheme

523

Fig. 2. (a), (b) and (c) are original stereo video frames (video 1, 2 and 3 in Table 1); (d), (e) and (f) are watermarked frames; (g), (h) and (i) are watermarks extracted from (d), (e) and (f), respectively; (j) is the original watermark pattern.

proposed by Wu et al. [15], an adaptive stereo watermarking scheme using noncorresponding blocks proposed by Mohaghegh et al. [7] and a 3D video watermarking scheme based on 3D-HEVC encoder proposed by Rana et al. [13]. The visual quality is evaluated by traditional Peak Signal to Noise Ratio (PSNR) for single-view fidelity and a proposed LDQ for inter-view consistency. The watermarked videos produced by the proposed method (shown in Fig. 2(d), (e) and (f)) looks totally the same with the original videos. We use PSNR to quantitatively evaluate the similarity between the watermarked videos and the original ones. The higher the PSNR is, the better the visual quality of the watermarked videos is. We compare the PSNR values of our scheme with those of the four state-of-the-art methods (listed in Table 2). It is pretty clear that the proposed method generates watermarked videos with the best visual quality. PSNR simply quantizes the visual fidelity in single views. For stereo videos, the consistency between left and right views are more important. If the consistency is

524

C. Cheng et al. Table 2. PSNR, LDQ and capacity of the five schemes. Proposed scheme

Niu’s scheme [9]

Wu’s scheme [15]

Rana’s scheme [13]

Mohaghegh’s scheme [7]

PSNR

51.11

50.33

48.93

47.80

46.90

LDQ

0.8171

0.7697

0.7639

0.9631

0.8027

1/2048

1/64

1/4098

1/1024

Capacity 1/64

corrupted, the video with watermarks will cause viewers uncomfortable and even dizzy. Inspired by the stereo consistency constraint used for stereo image segmentation in [12], we introduce a new evaluation on the visual quality of watermarked 3D videos, called Loss of Disparity Quality (LDQ), given by LDQ =

′  min(B(pi ,pj ) , B(p ) i ,pj ) n

′ ) max(B(pi ,pj ) , B(p i ,pj )

/n

(10)

Here, pi and pj denote a pair of corresponding pixels in left and right views, respectively. n is the number of the pairs of corresponding pixels. B(pi ,pj ) and ′ stand for color differences between corresponding pixels pi and pj , before B(p i ,pj ) and after embedding, respectively. B(pi ,pj ) is given by B(pi ,pj ) = e−(

0.5ci −cj 2 0.5 ) 256

(11)

ci and cj are the colors of pi and pj , respectively. LDQ represents the variance of the color differences between stereo corresponding pixels in the original and watermarked videos. Ideally, the variation should be close to zero. The closer to 1 the LDQ value is, the better the disparity quality of the watermarked videos is. Table 2 lists the LDQ values of all the five methods. From the table, we can see that our method obtains high LDQ as well. It’s known that low watermark capacity is one of the factors resulting in good visual quality. Since our method performs best in PSNR and LDQ, its capacity might be low. In order to eliminate this concern, we present the capacities, i.e. the number of bits hidden in one pixel in the host videos, of the five methods in Table 2. From the last row of Table 2, we can see that our method and Wu’s scheme [15] have the highest capacity among the five methods. 3.2

Robustness Analysis

As explained in the method part, since the scheme is designed to be blind for convenience, the extracted watermarks under no attack might have lost parts of information, compared with the original ones (refer to Fig. 2). This will not result in low robustness of the method against attacks. In order to evaluate the robustness of the proposed method against various attacks, the average NC

A Robust 3D Video Watermarking Scheme

525

values of the watermarks extracted by the five methods under various attacks are calculated and given in Table 3. The attacks, including compression (here we use JPEG compression since we treat the video as separate frames), relative mild salt & pepper noises, gaussian filtering and scaling, are common in reusing the high-quality videos. From the table, we can see that the proposed method obtains the highest NC value under the attacks of Salt & Pepper noise with noise power of 0.01, and the second or third highest values under the other attacks. Table 3. NC values of the watermarks extracted by the five schemes under various attacks. Attacks

Attack’s parameters

Proposed Niu’s Wu’s scheme Rana’s scheme scheme scheme [9] [15] [13]

Mohaghegh’s scheme [7]

0.7649

0.8447

0.7672

0.6249

0.7547

Salt & Noise power 0.7275 pepper noise 0.01

0.6950

0.6906

0.6358

0.6387

Noise power 0.6435 0.03

0.6075

0.6324

0.6096

0.6759

Gaussian filtering

Filter size (3, 3)

0.7802

0.7838

0.7737

0.5986

0.7871

Scaling

Resolution 1280 × 720

0.8409

0.4864

0.8477

0.5782

0.7246

Resolution 640 × 480

0.6492

0.5625

0.6449

0.5520

0.6783

JPEG Quality 95 compression

4

Conclusion

In this paper, a watermarking scheme for 3D videos was introduced. The highlight of the scheme is to control the embedding strength of watermarks by exploiting multi-modal visual redundancies in 3D videos. The computation of the visual redundancies integrates cues from intra-frame saliency, motion between sequential frames and disparity between the left and right views. Through experiments, we demonstrate the performances of the scheme in visual quality, capacity and robustness. Experimental results show that the proposed method generates watermarked videos with good visual quality, has large capacity and performs well in resisting various attacks. We also presented an evaluation method on the loss of disparity information of watermarked 3D videos, which fills a gap in evaluating the visual quality of 3D watermarking. There are still limitations in our work. In the future, rather than considering the robustness only in stereoscopic image planes, we will attempted to improve the scheme for robustness against video compression which is commonly used for low storage and fast transmission. Acknowledgement. This research is supported by Scientific Research Project of Beijing Educational Committee (KM201510005015), Beijing Municipal Natural Science Foundation (4152006), National Natural Science Foundation of China (61672068, 61370113), and Seed Funding for International Cooperation of Beijing University of Technology.

526

C. Cheng et al.

References 1. Asikuzzaman, M., Alam, M.J., Lambert, A.J., Pickering, M.R.: A blind watermarking scheme for depth-image-based rendered 3D video using the dual-tree complex wavelet transform. In: IEEE International Conference on Image Processing, pp. 5497–5501 (2014) 2. Chammem, A.: Robust watermarking techniques for stereoscopic video protection. Evry Institut National Des Tlcommunications (2013) 3. Farneb¨ ack, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) SCIA 2003. LNCS, vol. 2749, pp. 363–370. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-45103-X 50 4. Fehn, C.: Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Electronic Imaging, pp. 93–104 (2004) 5. Gupta, V., Barve, A.: A review on image watermarking and its techniques. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 4(1), 92–97 (2014) 6. Hu, F., Zhao, Y.: Comparative research of matching algorithms for stereo vision. J. Comput. Inf. Syst. 9(13), 5457–5465 (2013) 7. Mohaghegh, H., Karimi, N., Soroushmehr, S.M., Samavi, S.M.R., Najarian, K.: Adaptive stereo medical image watermarking using non-corresponding blocks. In: International Conference of the Engineering in Medicine and Biology Society, pp. 4214–4217 (2015) 8. Mousavi, S.M., Naghsh, A., Manaf, A.A., Abu-Bakar, S.A.R.: A robust medical image watermarking against salt and pepper noise for brain MRI images. Multimedia Tools Appl. 76(7), 10313–10342 (2017) 9. Niu, Y., Souidene, W., Beghdadi, A.: A visual sensitivity model based stereo image watermarking scheme. In: 3rd European Workshop on Visual Information Processing, pp. 211–215 (2011) 10. Onural, L., Ozaktas, H.M.: Three-dimensional Television: From Science-fiction to Reality. Springer, Heidelberg (2008) 11. Ou, Z., Chen, L.: A robust watermarking method for stereo-pair images based on unmatched block bitmap. Multimedia Tools Appl. 75(6), 3259–3280 (2016) 12. Price, B.L., Cohen, S.: StereoCut: Consistent interactive object selection in stereo image pairs. In: IEEE International Conference on Computer Vision, pp. 1148–1155 (2011) 13. Rana, S., Sur, A.: Blind 3D video watermarking based on 3D-HEVC encoder using depth. In: Proceedings of Indian Conference on Computer Vision Graphics and Image Processing, pp. 1–8 (2014) 14. Wei, Z., Ngan, K.N.: Spatio-temporal just noticeable distortion profile for grey scale image/video in DCT domain. IEEE Trans. Circ. Syst. Video Technol. 19(3), 337–346 (2009) 15. Wu, C., Yuan, K., Cheng, M., Ding, H.: Differential watermarking scheme of stereo video. In: IEEE 14th International Conference on Communication Technology, pp. 744–748 (2012) 16. Yan, T., Lau, R.W.H., Xu, Y., Huang, L.: Depth mapping for stereoscopic videos. Int. J. Comput. Vision 102(1), 293–307 (2013) 17. Yang, W., Chen, L.: Reversible DCT-based data hiding in stereo images. Multimedia Tools Appl. 74(17), 7181–7193 (2015)

Partial Secret Image Sharing for (n, n) Threshold Based on Image Inpainting Xuehu Yan1(B) , Yuliang Lu1 , Lintao Liu1 , Shen Wang2 , Song Wan1 , Wanmeng Ding1 , and Hanlin Liu1 1

2

Hefei Electronic Engineering Institute, Hefei 230037, China [email protected] School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China

Abstract. Shamir’s polynomial-based secret image sharing (SIS) scheme and visual secret sharing (VSS) also called visual cryptography scheme (VCS), are the primary branches in SIS. In traditional (k, n) threshold secret sharing, a secret image is fully (entirely) generated into n shadow images (shares) distributed to n associated participants. The secret image can be recovered by collecting any k or more shadow images. The previous SIS schemes dealt with the full secret image neglecting the possible situation that only part of the secret image needs protection. However, in some applications, only target part of the secret image may need to be protected while other parts may be not in a full image. In this paper, we consider the partial secret image sharing (PSIS) issue as well as propose a PSIS scheme for (n, n) threshold based on image inpainting and linear congruence (LC). First the target part is manually selected or marked in the color secret image. Second, the target part is automatically removed from the original secret image to obtain the same input cover images (unpainted shadow images). Third, the target secret part is generated into the pixels corresponding to shadow images by LC in the processing of shadow images texture synthesis (inpainting), so as to obtain the shadow images in a visually plausible way. As a result, the full secret image including the target secret part and other parts will be recovered losslessly by adding all the inpainted meaningful shadow images. Experiments are conducted to evaluate the efficiency of the proposed scheme. Keywords: Secret image sharing · Partial secret image sharing Image inpainting · Linear congruence · Color image · Lossless recovery Meaningful shadow images

1

Introduction

Through splitting the secret image into noise-like shadow images (also called shadows or shares), secret image sharing (SIS) distributes a secret image among multiple participants. The secret is recovered by collecting sufficient authorized c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 527–538, 2017. https://doi.org/10.1007/978-3-319-71598-8_47

528

X. Yan et al.

participants(shadow images). SIS can be applied in not only information hiding, but also access control, authentication, watermarking, and transmitting passwords etc. Shamir’s polynomial-based scheme [1] and visual secret sharing (VSS) [2] also called visual cryptography scheme (VCS), are the primary branches in this field. In Shamir’s original polynomial-based (k, n) threshold SIS [1], the secret image is generated into the constant coefficient of a random (k − 1)-degree polynomial to obtain n shadow images distributed to n associated participants. The secret image can be losslessly recovered by collecting any k or more shadow images based on Lagrange interpolation. Following Shamir’s scheme and utilizing all coefficients of the polynomial for embedding secret, Thien and Lin [3] reduced share size 1/k times to the secret image. The advantage of Shamir’s polynomial-based scheme [4–6] is lossless recovery. Shamir’s polynomial-based SIS requires more complicated computations, i.e., Lagrange interpolations, for reconstructing and known order of shares, although the scheme only needs k shares for recovering the distortion-less secret image. In (k, n) threshold VSS [7–13], the generated n shadow images are printed onto transparencies and then distributed to n associated participants. The beauty of VSS is that, the secret image can be revealed by superposing any k or more shadow images and human visual system (HVS) with no cryptographic computation. Less than k shares will reveal nothing about the secret except the image size. Inspired by Naor and Shamir’s VSS work, the associated VSS physical properties and its problems are extensively studied, such as contrast [14], threshold [15], different formats [16], multiple secrets [6], noise-like patterns [11,17–20], pixel expansion [8, 9, 21, 22] and so on [23, 24]. In most of the existing SIS schemes, the full (entire) secret image is directly generated into the shadow images. The previous SIS schemes dealt with the full secret image neglecting the possible situation that only part of the secret image may need protection. However, there are many examples that only part of the secret image may need to be protected while other parts may be not in the same image, such as improving part design, sensitive information in part of a image and so on. One possible scenario is described as follows. On the basis of multiple traditional design modules in a product, a company improves one module design of them, while the other modules continue to use the traditional original design. At this point for the overall design drawing of this product, the improved module is needed to be protected and the other original modules can be public. Due to business privacy and access control, this product design drawing is kept by the company’s n managers. Each manager can display the traditional modules in public display to facilitate the product introduction and other activities. In accordance with business needs, k or more managers together have the right to losslessly recover the full product design drawing including the improved module. In this scenario, we need protect target part of the secret image other than the full secret image as well as need access control with (k, n) threshold.

Partial Secret Image Sharing for (n, n) Threshold Based on Image Inpainting

529

Thus, in some applications we may only need to encrypt part of the secret image rather than the full secret image for some reasons. However, the previous SIS schemes have not considered this issue. In order to deal with the partial secret image sharing (PSIS) issue, in this paper, we will introduce PSIS problem as well as propose a novel PSIS scheme for (n, n) threshold based on image inpainting [25, 26] and linear congruence (LC) [27], where (n, n) threshold is a special case of (k, n) threshold under k = n. The target secret part of the color secret image is first manually selected and then automatically removed from the original color secret image to obtain the same input cover images (unpainted shadow images). Then, the target secret part is generated into the pixels corresponding to shadow images by LC in the processing of shadow images texture synthesis (inpainting), so as to obtain the shadow images in a way that looks reasonable to the human eye. As a result, the full secret image including the target secret part and the other parts will be recovered losslessly by adding all the inpainted meaningful shadow images. Experiments are conducted to evaluate the efficiency of the proposed scheme. The rest of the paper is organized as follows. Section 2 introduces some basic requirements for the proposed scheme. In Sect. 3, the proposed scheme is presented in detail. Section 4 is devoted to experimental results. Finally, Sect. 5 concludes this paper.

2

Preliminaries

In this section, we give the PSIS problem description and some preliminaries as the basis for the proposed method. The original secret image S is shared among original total n shadow images, while the reconstructed secret image S ′ is reconstructed from t (k ≤ t ≤ n, t ∈ Z+ ) shadow images. 2.1

Problem Definition

As show in Fig. 1, for the given secret image S in Fig. 1(a), Fig. 1(b) indicates the same input cover image C obtained by manually selecting and removing the target part from the original secret image S, where the notations of different parts and their edge are presented in Fig. 1(c). The region Ω is the target secret part (object), part Φ illustrates the untouched part, and ∂Ω denotes the edge of the 2 parts. Shadow images covered secret after sharing are denoted as SCi , i = 1, 2, ..., n for (k, n) threshold. The PSIS problem can be described as follows: From the selected target part Ω and the associated cover images C1 , C2 , · · · Cn , the PSIS scheme may generate n meaningful shadow images SCi , i = 1, 2, ..., n distributed to n associated participants, where each shadow image looks like a nature image. When any k or more shadow images are collected, the full secret image including the secret target part can be reconstructed. Whereas even if infinite computational power is available, less than k shadow images will reveal nothing about the secret target part.

530

X. Yan et al.

(a)

(b)

Φ

∂Ω Ω

(c)

Fig. 1. An example of PSIS problem description. (a) The secret image S; (b) the same input cover image C by manually selecting and removing the target part with color green from the original secret image S; (c) notations in the problem definition. (Color figure online)

2.2

Linear Congruence

Equations (1) and (2) are the basic equations for LC secret sharing, by which (n, n) threshold secret sharing can be achieved, where P denotes a number larger than the biggest pixels value, xi and y represent the i-th shared pixel and secret pixel, respectively. In Eq. (1), a one-to-many mapping between y and sum of all the x is established, so the secret value can be recovered losslessly with all the shared values. But there is no direct map relationship between secret value and less than k shared value, thus the method is secure. So Eq. (1) guarantees the feasibility of precise recovery and security for the proposed scheme. At the meanwhile, the condition in Eq. (2) ensures that no duplicate values exist in first n shared pixels. (x1 + x2 + · · · + xn ) mod P = y

(1)

xi = xj , when i = j.

(2)

Partial Secret Image Sharing for (n, n) Threshold Based on Image Inpainting

531

Equation (3) is the basic equation for LC secret recovery. After removing the duplicate shared values, the rest of shared values xi1 , xi2 , · · · , xit take part in the computation for the recovered secret value y ′ using Eq. (3). y ′ = (xi1 + xi2 + · · · + xit ) mod P

(3)

Figure 2 indicates an example by directly applying LC method for (2, 2) threshold, where the input secret image is the same as Fig. 1. We can see that the secret target part can be reconstructed losslessly, while the corresponding the secret target parts of the shadow images are noise-like. In the revealing process of the secret image, it only needs to iterate t pixels and execute t − 1 times addition operation and one time module operation to decode a secret pixel. Obviously, the time complexity is smaller. Hence, LC sharing idea is selected in our scheme as an SIS method.

(a)

(b)

(c)

Fig. 2. Simulation results of directly applying LC method for threshold (2, 2). (a)−(b) two shadow images SC1 , SC2 ; (c) recovered result by SC1 and SC2 .

532

2.3

X. Yan et al.

Image Inpainting

Many image inpainting schemes were proposed in the literature, here Criminisi et al. approach [25, 26] will be applied in our scheme, which is researched widely. We will introduce Criminisi et al. image inpainting approach in detail. The keynote of it is the selection of patch priorities order in region-filling. The patch with the highest priority will be filled preferentially. The priorities will be renewed after every filling until the image is inpainted totally in the same manner. The main inpainting process includes: 1. Select the part Ω to be inpainted, and Ω = S − Φ. 2. Determine the size of template window, denoted as Ψp , using image texture feature, where any p ∈ ∂Ω exhibits center of template window and the size of the window should be larger than the biggest texture element. 3. Compute patch priorities by Eq. (4) which is defined as the product of the confidence term and data term. W (p) = C(p)D(p)

(4)

where C(p) and D(p) denote the confidence term and data term, respectively, defined as follows:  C(q) C(p) =

D(p) =

¯ q∈ψp ∩Ω

|ψp |   ∇Sp⊥ · np  a

(5)

(6)

where |ψp | indicates the area of ψp and a is a normalization factor. ∇Sp⊥ and np denote the isophote direction and the normal vector direction at point p, respectively. The confidence term expresses the amount of reliable information contained in template window. The data term measures the difference between the isophote direction and the normal vector direction. In a word, we conclude that the template window includes more information and the difference between the isophote direction and the normal vector direction is less, the priority of the patch is higher. 4. Find pˆ using Eq. (7) and the most matching block ψqˆ ∈ Φ in the source image using template window according to Eq. (8), where the evaluation standard is the Sum of Squared Differences (SSD). Finally, the most matching block replaces the patch of current window. pˆ = arg max W (p)

(7)

p∈∂Ω

qˆ = arg min d(ψpˆ, ψq ) q∈Φ

(8)

Partial Secret Image Sharing for (n, n) Threshold Based on Image Inpainting

533

5. Renew the confidence terms after each filling process C(q) = C(ˆ p) for any q ∈ ψpˆ ∩ Ω. 6. Repeat steps 3–5 until the image is inpainted completely. Taking as an example, the inpainted image by Criminisi et al. approach is presented in Fig. 3. We can see that another visually plausible image is obtained, thus image inpainting will be applied in the proposed scheme to achieve meaningful shadow images.

Fig. 3. An example of the inpainted image from Fig. 1(b) by Criminisi et al.’s approach

3

The Proposed PSIS Scheme

Here, in order to deal with the PSIS issue, in this paper, we propose a novel PSIS scheme for (n, n) threshold based on image inpainting and LC. We present the proposed PSIS scheme based on the original secret image S and the selected secret target part Ω, resulting in n output meaningful shadow images SC1 , SC2 , · · · SCn . Our generation steps are described in Algorithm 1. In step 2 of our Algorithm, each shadow image has its own order (highest priority) in which the filling proceeds, i.e., pˆi . Aiming to inpaint synchronously, among the n candidate orders, the highest priority is selected again in Step 3 as the applied order for all the n shadow images inpainting. After the most matching block replaces the patch of the current window in Step 4, the secret block of the current window corresponding to the secret target part will be generated into the n shadow images corresponding blocks based on LC sharing in Steps 5–6. Thus, the modified patches of the current window will be the basis for next inpainting processing so that the target secret pixel is encoded into the pixels corresponding to shadow images in the processing of shadow images inpainting. As a result, meaningful shadow images may be achieved in a visually plausible way. The secret recovery of the proposed scheme is the same as LC method according to Eq. (3) by all the n shadow images.

534

X. Yan et al.

Algorithm 1. The proposed PSIS scheme for (n, n) threshold Input: The threshold parameters (n, n), the original secret image S and The selected secret target part Ω. Output: n shadow images SC1 , SC2 , · · · SCn Step 1: Remove the target part Ω from the original secret image S to obtain the same n cover images, denoted as C1 , C2 , · · · Cn . Let SCi = Ci be the input inpainting image. Determine the size of template window, denoted as Ψp∗ . Step 2: For each shadow image, find pˆi using Eq. (7). Step 3: Find i∗ = arg max Wi (ˆ pi ), set pˆi = pˆi∗ , i = 1, 2, · · · n. i∈[1,n]

Step 4: Based on pˆi and Eq. (8), search for the most matching block to gain ψqˆi , where i = 1, 2, · · · n. For each cover image, the most matching block replaces the patch of current window. Step 5: For each position (i, j) ∈ {(i, j)|M1 ≤ i ≤ M2 , N1 ≤ j ≤ N2 }, where (M1 , N1 ) and (M2 , N2 ) denote the coordinates of the processing template window in S, repeat Step 6. Step 6: Least modify SC1 (i, j) , SC2 (i, j) , · · · SCn (i, j) to satisfy Eq. (1). Step 7: Renew the confidence terms after each filling process SC(qi ) = SC(pˆi ) for any qi ∈ ψpˆi ∩ Ω, i = 1, 2, · · · n. Step 8: Repeat steps 2–7 until each cover image is inpainted completely. Step 9: Output n shadow images SC1 , SC2 , · · · SCn .

4

Experimental Results and Analyses

In this section, experiments and analyses are conducted to evaluate the effectiveness of the proposed method. In the experiments, the same example as shown in Fig. 1 will be employed as the input color secret image, with size of 512 × 384. Simulation result by the proposed PSIS scheme is presented in Fig. 4 for (4, 4) threshold, where Fig. 4(a) − (d) show the generated four shadow images SC1 , SC2 , SC3 , SC4 and the recovered results by two or more shadow images are presented in Fig. 4(e) − (o). Here “+” indicates addition and module operation as in LC. The shadow images corresponding to the secret target part are meaningful in a way that looks reasonable to the human eye, although little artifact appears in the shadow images due to the modification of the shadow images blocks for secret block generation based on LC sharing in Steps 5–6 of our Algorithm. We can see that the secret target part can be reconstructed losslessly when all the four shares are collected. The recovering results by less than four shadow images cannot recovery the secret while secret leakage appears especially for the edge of the selected secret target part, which also may be caused by the modification of the shadow images blocks in Steps 5–6 of our Algorithm. In the revealing process of the secret image, it only needs to execute one time addition operation and one time module operation to decode a secret pixel so that the recovery computation complexity is simple. Decreasing the artifacts and threshold extension will be our future work. Additional (3, 3) threshold simulation result is illustrated in Fig. 5, we can conclude similar results as the previous simulation except for a little more artifact in the shadow image, which is caused by LC sharing feature.

Partial Secret Image Sharing for (n, n) Threshold Based on Image Inpainting

(a) SC1

(b) SC2

(c) SC3

(d) SC4

(e) SC{+,1,2}

(f) SC{+,1,3}

(g) SC{+,1,4}

(h) SC{+,2,3}

(i) SC{+,2,4}

(j) SC{+,3,4}

(k) SC{+,1,2,3}

(l) SC{+,1,2,4}

535

(m) SC{+,1,3,4}

(n) SC{+,2,3,4}

(o) SC{+,1,2,3,4}

Fig. 4. Experimental example of the proposed scheme for threshold (4, 4)

Based on the above results we can conclude that: 1. The target part is successfully inpainted into visually plausible shadow image. 2. Each shadow image is meaningful in a way that looks reasonable to the human eye, while Every single shadow image can not disclose the secret image. 3. When t < n shadow images are collected, we cannot recovery the secret although secret leakage appears. 4. When t = n shadow images are recovered by only addition and module operation, the secret image including the secret target part could be recovered losslessly. 5. An acceptable the partial secret image sharing (PSIS) for (n, n) threshold is achieved in our scheme.

536

X. Yan et al.

(a) SC1

(d) SC{+,1,2}

(b) SC2

(e) SC{+,1,3}

(c) SC3

(f) SC{+,2,3}

(g) SC{+,1,2,3}

Fig. 5. Experimental example of the proposed scheme for threshold (3, 3)

5

Conclusion

This paper considered the new partial secret image sharing (PSIS) issue as well as proposed a novel PSIS scheme for (n, n) threshold based on image inpainting and linear congruence (LC) secret sharing method. Experiments showed that the output inpainted shadow images are meaningful, and the full secret image including the target secret part and other parts can be recovered losslessly by addition. Decreasing the artifacts and threshold extension will be our future work. Acknowledgement. This work is supported by the National Natural Science Foundation of China (Grant Number: 61602491).

References 1. Shamir, A.: How to share a secret. Commun. ACM 22(11), 612–613 (1979) 2. Naor, M., Shamir, A.: Visual cryptography. In: De Santis, A. (ed.) EUROCRYPT 1994. LNCS, vol. 950, pp. 1–12. Springer, Heidelberg (1995). https://doi.org/10. 1007/BFb0053419

Partial Secret Image Sharing for (n, n) Threshold Based on Image Inpainting

537

3. Thien, C.C., Lin, J.C.: Secret image sharing. Comput. Graph. 26(5), 765–770 (2002) 4. Lin, S.J., Lin, J.C.: VCPSS: A two-in-one two-decoding-options image sharing method combining visual cryptography (vc) and polynomial-style sharing (pss) approaches. Pattern Recogn. 40(12), 3652–3666 (2007) 5. Yang, C.N., Ciou, C.B.: Image secret sharing method with two-decoding-options: Lossless recovery and previewing capability. Image Vis. Comput. 28(12), 1600– 1610 (2010) 6. Li, P., Ma, P.J., Su, X.H., Yang, C.N.: Improvements of a two-in-one image secret sharing scheme based on gray mixing model. J. Vis. Commun. Image Represent. 23(3), 441–453 (2012) 7. Yan, X., Wang, S., El-Latif, A.A.A., Niu, X.: Visual secret sharing based on random grids with abilities of AND and XOR lossless recovery. Multimedia Tools Appl., 1–22 (2013) 8. Yang, C.N.: New visual secret sharing schemes using probabilistic method. Pattern Recognit. Lett. 25(4), 481–494 (2004) 9. Cimato, S., De Prisco, R., De Santis, A.: Probabilistic visual cryptography schemes. Comput. J. 49(1), 97–107 (2006) 10. Wang, D., Zhang, L., Ma, N., Li, X.: Two secret sharing schemes based on boolean operations. Pattern Recognit. 40(10), 2776–2785 (2007) 11. Wang, Z., Arce, G.R., Di Crescenzo, G.: Halftone visual cryptography via error diffusion. IEEE Trans. Inf. Forensics Security. 4(3), 383–396 (2009) 12. Weir, J., Yan, W.Q.: A comprehensive study of visual cryptography. In: Shi, Y.Q. (ed.) Transactions on Data Hiding and Multimedia Security V. LNCS, vol. 6010, pp. 70–105. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-64214298-7 5 13. Yang, C.N., Sun, L.Z., Yan, X., Kim, C.: Design a new visual cryptography for human-verifiable authentication in accessing a database. J. Real-Time Image Proc. 12(2), 483–494 (2016) 14. Wu, X., Sun, W.: Improving the visual quality of random grid-based visual secret sharing. Sig. Process. 93(5), 977–995 (2013) 15. Yan, X., Wang, S., Niu, X.: Threshold construction from specific cases in visual cryptography without the pixel expansion. Sig. Process. 105, 389–398 (2014) 16. Luo, H., Yu, F., Pan, J.S., Lu, Z.M.: Robust and progressive color image visual secret sharing cooperated with data hiding. In: Eighth International Conference on Intelligent Systems Design and Applications, ISDA 2008, vol. 3, Kaohsiung, Taiwan, IEEE, pp. 431–436. IEEE (2008) 17. Yan, X., Wang, S., Niu, X., Yang, C.N.: Generalized random grids-based threshold visual cryptography with meaningful shares. Sig. Process. 109, 317–333 (2015) 18. Zhou, Z., Arce, G.R., Di Crescenzo, G.: Halftone visual cryptography. IEEE Trans. Image Process. 15(8), 2441–2453 (2006) 19. Liu, F., Wu, C.: Embedded extended visual cryptography schemes. IEEE Trans. Inf. Forensics Secur. 6(2), 307–322 (2011) 20. Yan, X., Wang, S., Niu, X., Yang, C.N.: Halftone visual cryptography with minimum auxiliary black pixels and uniform image quality. Digit. Signal Proc. 38, 53–65 (2015) 21. Guo, T., Liu, F., Wu, C.: Threshold visual secret sharing by random grids with improved contrast. J. Syst. Softw. 86(8), 2094–2109 (2013) 22. Fu, Z., Yu, B.: Visual cryptography and random grids schemes. In: Shi, Y.Q., Kim, H.-J., P´erez-Gonz´ alez, F. (eds.) IWDW 2013. LNCS, vol. 8389, pp. 109–122. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-43886-2 8

538

X. Yan et al.

23. Ateniese, G., Blundo, C., De Santis, A., Stinson, D.R.: Visual cryptography for general access structures. Inf. Comput. 129(2), 86–106 (1996) 24. Li, P., Yang, C.N., Wu, C.C., Kong, Q., Ma, Y.: Essential secret image sharing scheme with different importance of shadows. J. Vis. Commun. Image Represent. 24(7), 1106–1114 (2013) 25. Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplarbased image inpainting. IEEE Trans. Image Process. 13(9), 1200–12 (2004). A Publication of the IEEE Signal Processing Society 26. Shen, W., Song, X., Niu, X.: Hiding traces of image inpainting. Res. J. Appl. Sci. Eng. Technol. 4(23), 4962–4968 (2012) 27. Liu, L., Lu, Y., Yan, X., Wan, S.: A progressive threshold secret image sharing with meaningful shares for gray-scale image. In: 2016 12th International Conference on Mobile Ad-Hoc and Sensor Networks (MSN), pp. 380–385. IEEE (2016)

A New SMVQ-Based Reversible Data Hiding Scheme Using State-Codebook Sorting Juan-ni Liu(&), Quan Zhou, Yan-lang Hu, and Jia-yuan Wei National Key Laboratory of Science and Technology on Space Microwave, Xi’an Institute of Space Radio Technology, Xi’an 710100, Shaanxi, China [email protected]

Abstract. A reversible data hiding scheme makes it possible to extract secret data and recover the cover image without any distortion. This paper presents a novel reversible data hiding scheme based on side-match vector quantization (SMVQ). If the original SMVQ index value of an image block is larger than a threshold, a technique called state-codebook sorting (SCS) is used to create two state-codebooks to re-encode the block with a smaller index. In this way, more indices are distributed around zero, so the embedding scheme only needs a few bits to encode the indices, which produces more extra space for hiding secret data. The experimental results indicate that the proposed scheme is superior to some previous schemes in embedding performance while maintaining the same visual quality as that of VQ compression. Keywords: Reversible data hiding
Vector quantization Side-match vector quantization
State-codebook sorting

1 Introduction With the rapid development of multimedia and communication technologies, massive amounts of multimedia data are transmitted over the Internet, which is convenient for people to exchange and share information without time or space constraints. However, such easy access to the Internet makes illegal distribution of multimedia easier and poses key challenges on security protection of confidential data. To solve this security problem, many researchers have proposed various data protection approaches, including encryption and data hiding techniques. An encryption method just encrypts the secret data into a meaningless form using cryptographic algorithms. The unrecognizable form may raise the suspicion of malicious attackers. On the contrary, data hiding is used to embed the secret data imperceptibly in meaningful cover media. In this way, attackers will not be suspicious of the media that carries the secret data. Generally, data hiding techniques can be classified into three domains, i.e., the spatial domain [1–6], the transform domain [7, 8] and the compression domain [9–17]. In the spatial domain, the cover image is modified directly and undetectably to conceal the secret data. In the transform domain, the secret data is embedded by altering the frequency coefficients of the cover image. However, due to the bandwidth limitation, many image compression schemes, such as vector quantization (VQ) [9], block truncation coding (BTC) [10, 11] and JPEG [12] have been introduced into the © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 539–550, 2017. https://doi.org/10.1007/978-3-319-71598-8_48

540

J. Liu et al.

compression domain to save storage and bandwidth space. VQ is a widely used image compression technique with properties of simple implementation and high compression rate. In this work, we mainly focus on the data hiding in VQ-related [13–17] image compressed codes. This hiding technique aims to provide a way for low bandwidth transmission, as well as a means of covert communication. In 2009, Chang et al. [13] proposed a reversible information embedding method for VQ-compressed image using the joint neighboring coding (JNC) technique. Their method only uses the left or upper neighboring indices to embed secret data and obtains a low hiding capacity. In 2011, Yang et al. [14] proposed an MFCVQ-based reversible data hiding scheme, which use the Huffman-code strategy and 0-centered classification to reduce the bit rate of the output code stream, however the hiding capacity is still low. In the same year, Chang et al. [15] designed a locally adaptive coding scheme in the VQ compression domain. This scheme achieves both good reconstructed image visual quality and a high embedding rate. In 2013, Wang et al. [16] proposed a new VQ-based scheme for reversible data hiding. In this scheme, a technique called adjoining state-codebook mapping (ASCM) is used to improve the side- match vector quantization (SMVQ)-based scheme. Therefore, it can reduce the size of the output code stream by 13% on average when 16,129 bits are embedded in each test image. In 2015, Lin et al. [17] proposed a new reversible data hiding scheme for VQ-compressed images. The search-order coding (SOC) and state-codebook mapping (SCM) are used in the proposed scheme for reducing the size of the VQ index table and for hiding a scalable amount of data. However, the scheme does not work effectively when images have low correlation among neighboring indices. One of the common characteristics of VQ-based data hiding schemes mentioned above is that the secret data are embedded in the compressed code stream. This type of data hiding algorithm focuses on providing larger embedding capacity with lower compressed bit rate. Therefore, considering how to make room to enlarge hiding capacity and how to reduce the index values to further suppress the bit rate are significant subjects of our proposed scheme. Generally, we employ SMVQ to improve the distribution of VQ index table, and adopt a state-codebook sorting (SCS) technique to further reduce the index values of SMVQ index table. So the proposed embedding scheme only needs few bits to encode the indices, thus yielding more space for secret data hiding. Compared with related existing schemes, our proposed method has higher embedding capacity and lower bit rate.

2 Proposed Scheme 2.1

State-Codebook Sorting

The goals of SMVQ-based data hiding scheme are to embed large amount of secret data and achieve a low compressed bit rate. The main factor in achieving good embedding performance is the distribution of SMVQ index table, which is determined by the state-codebook used for encoding an image block. In the final analysis, the state-codebook is the critical element for SMVQ-based data hiding method. Therefore we propose an SCS method to improve the distribution of SMVQ index table.

A New SMVQ-Based Reversible Data Hiding Scheme

541

As is well-known, correlation among neighboring blocks of a natural image is very strong, which is the key feature exploited in SMVQ, so we can use the border pixels of neighboring blocks to predict the current block. However, when the image block is complex, the prediction result becomes worse, which indicates the state-codebook used for encoding is not appropriate. In order to solve this problem, we adopt a strategy that the block whose index value is larger than threshold T needs to be re-encoded using new generated state-codebook. The threshold is chosen on the basis of empirical value. The statistics distribution of indices in Fig. 1 shows that most of the high-frequency SMVQ indices of the eight representative images range from 0 to 7, i.e., most of the indices are smaller than 8 (the integer power of 2). With the higher frequency of used indices, better embedding capacity can be achieved while maintaining low bit rate and the good visual quality of the reconstructed image. Therefore we take 8 as the threshold T to determine whether an SMVQ encoded image block needs to be re-encoded using the new state-codebooks generated by our sorting method. The detailed processes of the state-codebook sorting method are introduced as follows. indices[0,7]

indices[8,255]

100.00% 90.00%

Frequency of index

80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00%

Lena

Peppers Baboon

Boat Goldhill Airplane Sailboat Tiffany Test images

Fig. 1. High-frequency histogram of state-codebook sorted indices.

To encode a block of an image (except the first row and first column blocks that are encoded by VQ), the scheme first compresses it by the conventional SMVQ algorithm, then compares the index value v with T, if it is smaller than T, the index value is reserved, otherwise our scheme sorts the state-codebook by each codeword’s mean value in both ascending and descending order to create two state-codebooks, which are used again to encode the block to get new index values v1 and v2, then compares them with T. If only one of v1 and v2 is smaller than T, the scheme replaces v with that one, or if v1 and v2 are both smaller than T, replaces v with the smaller one of v1 and v2, and otherwise leaves the original index v unchanged. In this way, more indices are distributed around zero. According to the abovementioned SMVQ scheme, a 2-bit indicator is required to distinguish four cases of block encoding results. If a block is encoded using the original

542

J. Liu et al.

SMVQ index value which is smaller than T, the indicator is set as “00”, or if a block is re-encoded by the state-codebook sorted in ascending order, the indicator is set as “01”, or the indicator is set as “10” for the case that the state-codebook is sorted in descending order, or else the indicator is set as “11” to denote that a block’s original index value is larger than T and the state-codebook sorting method cannot effectively reduce the index. In other words, the first three cases obtain the index value that is smaller than T, while only for the fourth case, the index value is equal to or larger than T. Subsequently, the reverse processes of the SCS- based SMVQ are described below. When decoding an SMVQ index v, firstly judge its 2-bit indicator ind, if it is “00” or “11”, which means v is the index value of original state-codebook, so the image block can easily be recovered by table lookup method in original state-codebook. However, if the indicator is “01” or “10”, we need to sort the state-codebook in ascending order or descending order to produce the sorted state-codebook, in which the corresponding image block is found by table lookup. Then these image blocks compose the recovered image. 2.2

Data Embedding Process

In the data embedding process, the secret bits can be easily embedded into the index table IT. Similar to the scheme proposed by Wang et al. [16], our scheme uses a flag of two bits to identify the index encoding types, and embeds secret data in the index whose value is smaller than T. In order to enlarge the embedding capacity while maintaining a low output bit rate, 4 bits secret data is embedded in each embeddable index and another threshold Ta is applied to further divide the index, whose indicator is “11”, into two parts, i.e., P1 2 ½T; Ta 1Š and P2 2 ½Ta ; M 1Š. Since index values are divided into three groups using threshold T and Ta, the index smaller than T (T = 8) can be encoded by 3 bits, and the index with indicator “11” (i.e., P1 and P2) needs another 1-bit indicator to identify which group the index belongs to, so P1 is encoded as

1 ðP1 T Þ2 and P2 is encoded as 0||(P2)2, where “||” denotes the concatenation operation and (x)2 means the binary representation of x. It is worth noting that the length of P1 should be power of 2, i.e., Ta ¼ 2i þ T ði ¼ 1; 2; . . .; log2 M 1Þ into the secret data. Though additional 1-bit flag is required, we can effectively reduce the bit rate of final code stream. Because, on the one hand, more indices are smaller than T, on the other hand, the code length of P1 is much smaller than the bit size if P1 and P2 are encoded as one group. One thing to note is that the indices in the first row and the first column of IT are the seed area, where the index values will be directly transformed into binary form of log2N bits without embedding any secret data. The residual indices of IT will be processed in order from left-to-right and top-to-bottom by the proposed embedding algorithm which is described in detail as follows: Input: Grayscale cover image I, N-sized super codebook C, threshold T and Ta, secret data stream B. Output: Code stream CS in binary form.

A New SMVQ-Based Reversible Data Hiding Scheme

543

Step 1: Encode image I using SCS-based SMVQ technique to obtain the index table IT and the block encoding indicator table indT. Step 2: Read index v which is not in the seed area from the index table IT. Step 3: Read index v’s corresponding indicator ind from encoding indicator table indT. Step 3:1: if ind is “11”, this means that v is equal to or larger than T, and no secret data will be embedded in this index. Step 3:1:1: If v < Ta, v is encoded by q = 11||1||(v − T)2, where the size of (v − T)2 is log2(Ta − T). Step 3:1:2: If v  Ta , v is encoded by q = 11||0||(v)2, where the size of (v)2 is log2M. Step 3:2: Otherwise, i.e., the ind is “00”, “01” or “10”, which means the index v is smaller than T. Step 3:2:1: If B is not empty, fetch 4 bits secret data as s and remove them from B, v is encoded by q = ind||(v)2||s, where the size of(v)2 is 3. Step 3:2:2: If B is empty, v is encoded by q = ind||(v)2, where the size of (v)2 is 3. Step 4: Send q to the code stream CS. Step 5: Repeat Step 2 to Step 4 until all indices in the index table IT are processed. Step 6: Output the code stream CS. After the above steps are processed completely, each index of IT can be encoded to form the code stream CS. The sender then distributes the encoded binary stream CS to the receiver. 2.3

Data Extracting Process

At the receiving side, with the received code stream CS and the size of state-codebook M, the decoder can extract the embedded secret message and reconstruct the original SMVQ indices simultaneously, and after that the cover image can be restored by decoding SMVQ indices with N-sized super codebook C. Initially, the indices in the seed area are recovered directly by converting every log2N bits codes, which are read from CS, to decimal value. The residual indices in non-seed area and secret data are reconstructed easily through distinguishing four cases with the help of indicators. The detailed processes for extracting secret data and restoring the indices in the non-seed area are described as follows: Input: The binary code stream CS, N-sized super codebook C, the size of state-codebook M, threshold T and Ta, the number of unrecovered secret data Nx. Output: The secret data stream B and the original SMVQ-compressed index table IT. Step 1: Set the secret bit stream B empty. Step 2: Read 2 bits indicator from the binary code stream CS as ind.

544

J. Liu et al.

Step 3: if ind is “11”, this means the index value is equal to or larger than T and no secret data is embedded in the index. Then read the next one bit from CS as lab. Step 3:1: If lab is “1”, which means T  v0 \Ta , then read next log2(Ta−T) bits from CS as q, v0 is reconstructed by converting q to decimal value then adding with T. Step3:2: If lab is “0”, which means v0  Ta , then v0 is reconstructed by reading next log2M bits from CS and converting them to decimal value. Step 4: Otherwise, i.e., the ind is “00”, “01” or “10”, which means the index value v0 is smaller than T. And v0 is reconstructed by reading next 3 bits from CS and converting them to decimal value. Step 4:1: If Nx > 0, 4 subsequent bits are extracted from CS as secret data s0 , then concatenate s0 into the secret data stream Bði:e: B ¼ Bks0 Þ and let Nx = Nx − 4. Step 4:2: If Nx = 0, there is no secret data need to be extracted. Step 5: Put v0 in the recovered index table IT. Step 6: Repeat Step2 to Step5 until all the bits of code stream CS are processed. According to the restored SMVQ index table IT and the super codebook C, we can reconstruct the original image by the reverse process of SCS-based SMVQ. In this way, the whole decoding process is realized.

3 Experimental Results and Analysis In this section, the experimental results are presented to demonstrate the performance of the proposed scheme. Figure 2 shows eight test images of size 512  512, each image is divided into 16384 non-overlapping 4  4 blocks in the codebook encoding procedure. The super codebook of sizes 512 with codewords of 16 dimensions were trained by the LBG algorithm. The secret data in the experiment is in binary format, 0 and 1, which are generated by a pseudo-random number generator, and four bits are embedded in each embeddable index. If a high level of security is required, secret data can be encrypted prior to embedding using well-known cryptographic methods as DES or RSA. In these experiments, four criteria are employed to evaluate the performance of our proposed scheme, i.e., visual quality, compressed bit rate, embedding rate and embedding efficiency. The peak signal-to-noise (PSNR) ratio is used to estimate the degree of distortion between the original cover image and the stego-image. The compressed bit rate Rb is used to evaluate the compression performance of an encoding algorithm, which is defined as Rb ¼ jjCSjj=ðH  WÞ, where ||CS|| represents the total length of the output code stream, H and W is the height and width of the cover image. The embedding rate Re, which describes how many secret bits can be embedded into one index of the SMVQ index table, is defined as Re ¼ jjSjj=NIDX , where ||S|| is the number of confidential bits embedded into the SMVQ index table, namely Cp, and NIDX is the number of indices in the index table generated by SMVQ. The last criterion is the

A New SMVQ-Based Reversible Data Hiding Scheme

545

embedding efficiency Ee, which describes the ratio of the size of secret data to the length of output code stream, is defined as Ee ¼ jjSjj=jjCSjj.

(a) Lena

(b) Airplane

(c) Baboon

(d) Sailboat

(e) Goldhill

(f) Boat

(g) Peppers

(h) Tiffany

Fig. 2. Eight grayscale test images of size 512  512.

3.1

Performance of SCS

The motivation for the SCS technique is to increase the probability of getting small indices. Experimental results in Table 1 illustrate the feasibility of reducing the index value by using SCS method. M is the size of state-codebook used to generate SMVQ index table. When M = 512, it means the image quality of SMVQ is as same as that of VQ, and the index value is reduced by 7.18% on average. On the other hand, a smaller size of state-codebook (M = 32) can obviously achieve an average of 64.42% reduction of the index value. Table 1. Image quality and percentage of index value improvement. Image Lena Peppers Baboon Boat Goldhill Airplane Sailboat Tiffany Average

M = 32 PSNR (dB) 30.1 29.6 23.1 28.5 29.2 29.7 27.3 30.6 28.5

M = 512 Percentage (%) PSNR (dB) 65.17 32.2 60.82 31.4 59.72 24.4 67.36 30.2 63.39 30.8 70.79 31.6 61.02 28.6 67.08 32.3 64.42 30.2

Percentage (%) 2.89 3.65 7.44 12.32 0.23 13.49 2.13 15.29 7.18

546

3.2

J. Liu et al.

Parameters Choosing

In order to maintain the same image quality as VQ, we make the size of state-codebook equal to that of super codebook, i.e., M = N = 512. Table 2 is a comparison between our scheme and Wang et al.’s scheme [16]. Evidently, the visual quality of our proposed scheme is same to that of Wang et al.’s scheme and VQ compression. Since there are no distortions in the index encoding and data hiding process and all distortions only occur in the SMVQ-index table generating process, the PSNR value of the stego-image is exactly equal to that of the SMVQ compressed cover images. Table 2. Visual quality results (dB). Image Lena Peppers Baboon Boat Goldhill Airplane Sailboat Tiffany

Proposed 32.2 31.4 24.4 30.2 30.8 31.6 28.6 32.3

Wang et al.’s scheme VQ 32.2 32.2 31.4 31.4 24.4 24.4 30.2 30.2 30.8 30.8 31.6 31.6 28.6 28.6 32.3 32.3

The threshold Ta is another important parameter which affects the compressed bit rate of the output code stream, therefore we also test which threshold can lead to a better result of our proposed scheme. The bit rate results with different Ta are shown in Table 3, where no secret data is embedded in the test images. The bit rate values are in the range of 0.338–0.506 and the best one for each test images is shown in bold. We can easily find that, with the increase of Ta the bit rate decreased first then increased, therefore the better value for Ta would be 40 since more than half of the test images have better compressed bit rate when Ta is 40. The following experiments are done under these conditions. Table 3. The bit rate of proposed scheme with different Ta (bpp). Ta Lena Peppers Baboon Boat Goldhill Airplane Sailboat Tiffany

12 0.402 0.382 0.506 0.393 0.459 0.366 0.455 0.345

16 0.392 0.371 0.494 0.385 0.441 0.361 0.441 0.341

24 0.385 0.366 0.481 0.378 0.427 0.356 0.427 0.338

40 0.383 0.368 0.472 0.374 0.422 0.354 0.418 0.338

72 0.387 0.376 0.470 0.377 0.429 0.356 0.422 0.340

136 0.398 0.387 0.480 0.386 0.447 0.362 0.438 0.345

A New SMVQ-Based Reversible Data Hiding Scheme

3.3

547

Comparison Results with Related Schemes

We firstly considered only the scheme of Wang et al. [16] for performance comparisons of bit rate and embedding efficiency by embedding various sizes of secret data into the SMVQ index table. The bit rate comparison results of the two schemes are shown in Table 4. It is observed that, with the increase of embedding capacity, the encoding length for each index becomes longer, therefore the bit rates of the two schemes become gradually increased. In addition, under the same embedding capacity, our scheme achieves a lower bit rate than that of Wang et al.’s scheme for each test image. And the reduction of bit rate can be 1.7% to 9.1%, the average improvement of bit rate for our scheme is nearly 6.2%. The reason for achieving a lower bit rate is that in our proposed scheme the indices belonging to P1 2 ½T; Ta 1Š, which embeds no secret data, can be encoded using few bits. On the other hand, the number of indices whose values are smaller than T is increased by SCS method, accordingly the number of indices that cannot be used for data embedding is decreased, hence the length of code stream is reduced. Table 4. Bit rate comparison for embedding various sizes of secret data (bpp). Wang et al.’s scheme (SCI = 3) Proposed scheme

Capacity 0 5000 10000 20000 30000 0 5000 10000 20000 30000

Lena 0.417 0.436 0.455 0.493 0.532 0.383 0.402 0.421 0.459 0.498

Peppers 0.405 0.424 0.443 0.481 0.519 0.368 0.387 0.406 0.444 0.482

Baboon 0.482 0.501 0.52 0.558 0.596 0.472 0.491 0.51 0.548 0.586

Boat 0.397 0.416 0.435 0.473 0.511 0.374 0.393 0.412 0.45 0.489

Goldhill 0.458 0.477 0.496 0.534 0.572 0.422 0.441 0.46 0.498 0.536

Airplane 0.382 0.4 0.419 0.456 0.496 0.354 0.372 0.391 0.43 0.466

Sailboat 0.458 0.477 0.496 0.535 0.573 0.418 0.437 0.456 0.495 0.533

Tiffany 0.358 0.377 0.396 0.434 0.472 0.338 0.357 0.376 0.414 0.452

Table 5. Embedding efficiency comparison for embedding various sizes of secret data (%). Wang et al.’s scheme (SCI = 3) Proposed scheme

Capacity 5000 10000 20000 30000 5000 10000 20000 30000

Lena 4.4 8.4 15.5 21.5 4.7 9.1 16.6 23.0

Peppers 4.5 8.6 15.9 22.0 4.9 9.4 17.2 23.7

Baboon 3.8 7.3 13.7 19.2 3.9 7.5 13.9 19.5

Boat 4.6 8.8 16.1 22.4 4.9 9.3 16.9 23.4

Goldhill 4.0 7.7 14.3 20.0 4.3 8.3 15.3 21.4

Airplane 4.8 9.1 16.7 23.1 5.1 9.8 17.7 24.6

Sailboat 4.0 7.7 14.3 20.0 4.4 8.4 15.4 21.5

Tiffany 5.1 9.6 17.6 24.2 5.3 10.2 18.4 25.3

548

J. Liu et al.

Table 5 presents the corresponding embedding efficiency for the comparison experiment. The results show that, the embedding efficiencies are respectively higher than that of Wang et al.’s scheme, therefore a better performance can be achieved by using our SCS-based SMVQ technique. The increased percentage of embedding efficiency ranges from 1.5% to 10.0%, on average 6.3%, which indicates if the sizes of output code stream produced by the two schemes are same, the number of secret bits that can be embedded in our scheme will increase by 6.3% on average compared with Wang et al.’s scheme. We have conducted another experiment to evaluate the performance of the proposed scheme by embedding the maximum numbers of secret bits into the index table. The results in Table 6 show that, the smooth image can get better results, such as image “Tiffany”, which has the highest hiding capacity and embedding efficiency. While the results of complex image become worse to some extent, the embedding rate of image “Baboon” is only 2.00 bpi, which is the lowest one of the test images. However, the embedding rate of proposed scheme is about 2.85 bpi on average, which means almost 2.85 bits secret data can be embedded in one index. Additionally, the average bit rate is 0.560 bpp, and the average embedding efficiency is 31.4%. Table 6. The results for embedding maximum sizes of secret data in different test images. Image Lena Peppers Baboon Boat Goldhill Airplane Sailboat Tiffany Average

Cp (bit) 48040 50096 32788 50356 37868 55320 39636 58884 46624

Re (bpi) 2.93 3.06 2.00 3.07 2.31 3.38 2.42 3.59 2.85

Rb (bpp) 0.566 0.559 0.597 0.566 0.566 0.565 0.569 0.562 0.569

Ee (%) 32.4 34.2 20.9 33.9 25.5 37.4 26.6 39.9 31.4

To demonstrate the superiority of the proposed scheme, three related schemes, i.e., Chang et al.’s scheme [13], Wang et al.’s scheme [16] and Lin et al.’s scheme [17] were finally compared with the proposed scheme. The sizes of codebooks adopted in all schemes are 512 and the suggested optimal settings are used as given in their corresponding algorithms. Table 7 shows the comparison results for embedding 16,129 bits in different test images. For most test images, the proposed scheme achieves the optimal result compared with other schemes. In terms of bit rate, we can see that the proposed scheme generally produces the smallest one. Although Lin et al.’s scheme gets lower bit rate for simple images such as “Airplane” and “Tiffany”, the average bit rate is 1.5% higher than that of the proposed scheme, because the proposed scheme can greatly reduce the bit rate of complex images. Compared with Wang et al.’ scheme and Chang et al.’s scheme, the reduction of bit rate is 6.4% and 18.8% respectively.

A New SMVQ-Based Reversible Data Hiding Scheme

549

On the other hand, the embedding efficiency of our scheme has been increased in comparison with other schemes. Since under the same embedding capacity, the embedding efficiency is only determined by the total length of the output code stream, therefore, a data hiding scheme with low bit rate will achieve a high embedding efficiency. In our scheme, the average embedding efficiency for embedding 16,129 bits is nearly 13.7%, which outperforms all the other schemes: Chang et al.’s scheme (11.1%), Wang et al.’s scheme (12.8%), and Lin et al.’s scheme (13.5%). Table 7. Comparison on bit rate and embedding efficiency of different data hiding schemes for embedding 16,129 bits. Image

Lena Peppers Baboon Boat Goldhill Airplane Sailboat Tiffany Average

Chang et al.’s scheme (m = 6)

Wang et al.’s scheme (SCI = 3)

Lin et al.’s scheme (SOL = 2, SCL = 8) Rb (bpp) Ee (%) Rb (bpp) Ee (%) Rb (bpp) Ee (%) 0.552 11.2 0.479 12.8 0.440 14.0 0.545 11.3 0.466 13.2 0.426 14.4 0.608 10.1 0.552 11.2 0.546 11.3 0.556 11.1 0.471 13.1 0.459 13.4 0.581 10.6 0.519 11.9 0.511 12.0 0.537 11.5 0.445 13.8 0.407 15.1 0.582 10.6 0.520 11.8 0.508 12.1 0.503 12.2 0.419 14.7 0.385 16.0 0.558 11.1 0.484 12.8 0.460 13.5

Proposed scheme

Rb (bpp) 0.445 0.429 0.534 0.436 0.483 0.415 0.480 0.399 0.453

Ee (%) 13.8 14.3 11.5 14.1 12.7 14.8 12.8 15.4 13.7

In short, the above experiments demonstrate that the proposed algorithm has a better performance in both embedding capacity and bit rate. Additionally, in certain test images or especially in complex images, our scheme achieves a lower bit rate than other schemes because large amount of index values are smaller than T due to the SCS-based SMVQ technique.

4 Conclusion We propose a new reversible data hiding method for SMVQ compressed images. By employing the SCS-based SMVQ technique, our scheme achieves a higher embedding capacity with lower bit rate than related schemes. Since the SCS method is used to further reduce the index values of SMVQ index table, the proposed embedding scheme only needs few bits to encode the indices, thus yielding more space for secret data hiding. Though a 2-bit flag is required to distinguish the types of indices, we can effectively reduce the bit rate of the final code stream because large amounts of index values are smaller than T, and the indices belonging to [T, Ta − 1] are used to reduce the code length, with no secret data embedded. Several experiments were conducted with 512 codewords to evaluate the performance of proposed scheme. The comparison

550

J. Liu et al.

results show that the proposed scheme outperforms other schemes both in compression rate and embedding efficiency. Especially, for the complex images “Baboon” and “Boat”, the embedding efficiency is improved by 1.8% and 5.2% compared with Lin et al.’s scheme [17]. Moreover, the proposed scheme maintains the same visual quality as that of VQ compression. Acknowledgement. This work is supported by the National Natural Science Foundation of China (No. 61372175).

References 1. Rudder, A., Kieu, T.D.: A lossless data hiding scheme for VQ indexes based on joint neighboring coding. KSII Trans. Internet Inform. Syst. 9(8), 2984–3004 (2015) 2. Li, X., Zhou, Q.: Histogram modification data hiding using chaotic sequence. In: Wang, W. (ed.) Mechatronics and Automatic Control Systems. LNEE, vol. 237, pp. 875–882. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-01273-5_98 3. Li, X.L., et al.: General framework to histogram-shifting-based reversible data hiding. IEEE Trans. Image Process. 22(6), 2181–2191 (2013) 4. Hong, W., Chen, T.S., Luo, C.W.: Data embedding using pixel value differencing and diamond encoding with multiple-base notational system. J. Syst. Softw. 85(5), 1166–1175 (2012) 5. Hong, W., Chen, T.S.: A novel data embedding method using adaptive pixel pair matching. IEEE Trans. Inf. Forensics Secur. 7(1), 176–184 (2012) 6. Hong, W.: Adaptive image data hiding in edges using patched reference table and pair-wise embedding technique. Inform. Sci. 221(1), 473–489 (2013) 7. Fang, H., Zhou, Q., Li, K.J.: Robust watermarking scheme for multispectral images using discrete wavelet transform and tucker decomposition. J. Comput. 8(11), 2844–2850 (2013) 8. Su, P.C., Chang, Y.C., Wu, C.Y.: Geometrically resilient digital image watermarking by using interest point extraction and extended pilot signals. IEEE Trans. Inf. Forensics Secur. 8(12), 1897–1908 (2013) 9. Wang, W.J., Huang, C.T., Wang, S.J.: VQ applications in steganographic data hiding upon multimedia images. IEEE Syst. J. 5(4), 528–537 (2011) 10. Guo, J.M., Liu, Y.F.: High capacity data hiding for error-diffused block truncation coding. IEEE Trans. Image Process. 21(12), 4808–4818 (2012) 11. Guo, J.M., Tsai, J.J.: Reversible data hiding in low complexity and high quality compression scheme. Digit. Signal Proc. 22(5), 776–785 (2012) 12. Mobasseri, B.G., et al.: Data embedding in JPEG bit stream by code mapping. IEEE Trans. Image Process. 19(4), 958–966 (2010) 13. Chang, C.C., Kieu, T.D., Wu, W.C.: A lossless data embedding technique by joint neighboring coding. Pattern Recogn. 42(7), 1597–1603 (2009) 14. Yang, C.H., et al.: Huffman-code strategies to improve MFCVQ-based reversible data hiding for VQ indexes. J. Syst. Softw. 84(3), 388–396 (2011) 15. Chang, C.C., Nguyen, T.S., Lin, C.C.: A reversible data hiding scheme for VQ indices using locally adaptive coding. J. Vis. Commun. Image Represent. 22(7), 664–672 (2011) 16. Wang, W.J., et al.: Data embedding for vector quantization image processing on the basis of adjoining state-codebook mapping. Inform. Sci. 246(14), 69–82 (2013) 17. Lin, C.C., Liu, X.L., Yuan, S.M.: Reversible data hiding for VQ-compressed images based on search-order coding and state-codebook mapping. Inform. Sci. 293, 314–326 (2015)

An Efficient Privacy-Preserving Classification Method with Condensed Information Xinning Li ✉ and Zhiping Zhou (

)

Engineering Research Center of Internet of Things Technology Applications, Ministry of Education, Jiangnan University, Wuxi 214122, China [email protected]

Abstract. Privacy-preserving is a challenging problem in real-world data clas‐ sification. Among the existing classification methods, the support vector machine (SVM) is a popular approach which has a high generalization ability. However, when datasets are privacy and complexity, the processing capacity of SVM is not satisfactory. In this paper, we propose a new method CI-SVM to achieve efficient privacy-preserving of the SVM. On the premise of ensuring the accuracy of classification, we condense the original dataset by a new method, which trans‐ forms the privacy information to condensed information with little additional calculation. The condensed information carries the class characteristics of the original information and doesn’t expose the detailed original data. The timeconsuming of classification is greatly reduced because of the fewer samples as condensed information. Our experiment results on datasets show that the proposed CI-SVM algorithm has obvious advantages in classification efficiency. Keywords: Privacy-preserving · Condensed information Classification · Support vector machine (SVM)

1

Introduction

Classification is an important method used to analyze mass data in data mining so that better data prediction can be realized [1, 2]. Nevertheless, this case causes to anxiety about privacy and secure considerations. Most classification algorithms rely on the learning of the original training samples, which is easy to expose the training data and lead to the leakage of privacy information, especially for hospital diagnosis results and financial information [3]. However, when we try to protect the original data, the accuracy and time-consuming of classification may be affected. Therefore, it is the main task to maintain a good trade-off between privacy-preserving and classification effective. Support vector machine (SVM) is widely used as a reliable classification algorithm for high dimensional and nonlinear data [4]. In recent years, many researches have been made in privacy-preserving SVM classification. Nevertheless, if the training dataset is large in the realization, the learning speed of SVM will be very slow. Meanwhile, as a single SVM can only be used for two-class classification, researches often combine multiple SVMs to achieve multi-class classification, such as LIBSVM [5], which may lead to excessive numbers of iterations and much more time-consuming in learning stage. © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 551–562, 2017. https://doi.org/10.1007/978-3-319-71598-8_49

552

X. Li and Z. Zhou

In order to preserve privacy dataset and improve the classification efficiency of SVM, researchers proposed some methods to compress the original dataset, so that privacy information may be hidden and the reduced dataset may save the time-consuming of classification. Literature [6] applied a global random reduced kernel composed by local reduced kernels to generate a fast privacy-preserving algorithm. Literature [7] proposed the use of edge detection technology to extract the potential support vectors and remove the samples from the classification boundaries, so as to reduce the training samples to improve the training speed. Literature [8] used RSVM (The reduced SVM) [9] to save time in the data classification stage. Unfortunately, most of the existing schemes are at the expense of cutting the dataset, which may inevitably lead to the loss of original information. Nowadays, researchers put forward the method of clustering the data before classification, the condensed information can be used to hide the original data and improve the classification speed. Literature [10] used the fuzzy clustering method (FCM) to reduce and compress the training data, the clustering centers were applied to the SVM in the training phase. However, in fact, the purpose of us is to realize the compression of the original information but not find the global optimal clustering centers, the iterative process for finding cluster centers in FCM is superfluous under the circumstances, which may lead to unnecessary time-consuming. In this paper, we focus on the privacy-preserving of SVM and design a scheme for training the SVM from the condensed information by clustering. Unlike traditional methods, condensed information SVM (CI-SVM) refers breadth-first search (BFS) algorithm to access all nodes and generate similarity matrix, then the process of global clustering is based on the similarity between samples. CI-SVM enables the data owner to provide condensed data with fuzzy attributes to the public SVM without revealing the actual information since the public SVM may be untrustworthy. On the other hand, as the determination of attribute weights indirectly affects the clustering results, some researchers have taken into account its impact on the clustering results [11]. In this paper we introduce the weighted clustering based on dispersion degree into the similarity matrix, which aim is to ensure more realistic clustering centers. In addition to the condensed data itself, the amount of data can be reduced obviously compared with the original data, which means the classification speed may be greatly improved. The rest of this paper is organized as follows: In Sect. 2, we review SVM and FCMSVM proposed by the previous researchers, discuss the reason for improvement. Section 3 introduces our CI-SVM privacy preserving classifier, which achieves SVM classification from clustering data to protect original information. Then in Sect. 4, we list the experimental results and make a brief analysis. Finally, we conclude the paper in Sect. 5.

2

Related Work

2.1 Support Vector Machines The SVM is based on the theory of structural risk minimization and performs well on training data [12]. Usually a SVM classifier should be given an input pair (xi , yi ) with p features in i-th data point and y ∈ (+1, −1) is the corresponding class label. We often

An Efficient Privacy-Preserving Classification Method

553

use a soft constraint yi (� ⋅ xi + b) ≥ 1 − �i to handle nonlinear separable data, � is the weight vector of the linear SVM and b is the bias value. Therefore, the following primal objective should be minimized:

∑ 1 �i , ‖�‖2 + C 2 i=1 n

min �,b,�

�i ≥ 0, i = 1, 2, … n

(1)

Where C > 0 is the regularization parameter, we usually solve the following dual problem: min �

n ∑

1 ∑∑ ��Q 2 i=1 j=1 i j ij n

�i −

i=1

subject to

n ∑

n

(2)

�i yi = 0, 0 ≤ �i ≤ C

i=1

Where Qij is decided by k positive semi-definite matrix, Qij = yi yj K(xi , yi ) is the kernel function and K(xi , yi ) ≡ �(xi )T �(yi ), the decision function is

f (�) =

n ∑

�i yi K(xi , �) + b

(3)

i=1

We utilize LIBSVM to handle multi-class datasets and the strategy of classification is one-versus-one [5]. It designs a SVM classifier between any two samples, which means k-class samples need to design k(k − 1)∕2 classifiers. For training data from i-th and j-th class:

∑ 1 ij T ij (� ij )t (� ) � + C min ij ij ij � ,b ,� 2 t

(4)

Each classifier obtains the corresponding classification results. There is a voting strategy in multi-class classification: each classification result is regarded as a voting where votes can be accumulated for all samples. In the end a sample is designated to be in a class with the maximum number of votes. 2.2 Review of FCM-SVM In recent years, an effective method to speed up the training of SVM is clustering the original data before classification to release the initial data size [13–15]. Researchers usually use the FCM to condense information so that the meaning of original data can be reserved. FCM is a kind of fuzzy iterative self-organizing data analysis algorithm (ISODATA). It divides the dataset xi (i = 1, 2, … , n) into c fuzzy classes, where each cluster center should be calculated. It transforms the aim clustering to minimize the nonsimilarity index of the value function. Different with the hard clustering, FCM allows

554

X. Li and Z. Zhou

each data point to belong to all clustering centers and uses uij = (0, 1) to measure the M ∑ attribution degree, where uij = 1, (j = 1, 2, … , n), M is the number of clustering i=1

centers. The objective function of FCM is defined as follows: MinJfcm (U, V) =

M n ∑ ∑

uqij dij2

(5)

i=1 j=1

‖ ‖ Where q ∈ [1, +∞) is the weighted index, dij = ‖cm − xj ‖ is the Euclidean distance ‖ ‖ between the clustering center cm and data point xj. Then update as follows: n ∑

vi =

j=1

uqij xj

n ∑ j=1

uij = uqij

∑ M

c=1

(

1 )2∕(q−1)

dij

(6)

dcj

In FCM-SVM, FCM is the iterative process to find the optimal clustering centers, then the generated clustering centers will be used as the reduced dataset to train the SVM. The time complexity of FCM is O(n3 p). However, what we only need is the condensed information to classify in the SVM, the traditional clustering methods like FCM have too many unnecessary iterations in the process of finding clustering centers, which means the time-consuming of clustering is too much and unnecessary.

3

Secure SVM with Condensed Information

In this section, we design a data security solution CI-SVM which makes SVM known to the public. To overcome the security weakness of SVM and the overlong timeconsuming of classification, we design a condensed information scheme and use the clustering centers to replace the essential information between samples, so that public SVM can only obtain the condensed data clustered by the undirected weighted graph.

Fig. 1. CI-SVM privacy-preserving scheme.

An Efficient Privacy-Preserving Classification Method

555

Figure 1 shows the proposed CI-SVM for privacy-preserving. The left-hand expresses the process of data owner training and testing data. The data owner clusters data through our clustering method and sends the condensed information to the public SVM. The right-hand is the CI-SVM in public, it derives the condensed data to classify, then returns classification information of samples to the data owner. 3.1 Feature Weight Calculation In a dataset, the data resources may contain many data features with different attributes, some of the features may cause high contribution rate to the clustering results that named high separable features. In contrast, others’ contribution rate to the clustering results is very little, which are named isolated features or noise features. If we ignore the impact of them, the clustering accuracy may be difficult to improve, which is doomed to low classification accuracy in SVM. To overcome the robustness of the traditional algorithm and reduce the effect of noise data on the clustering results, we adopt variation coefficient weighting method to determine the contribution of each feature, which may increase the weights of high separable features so that the accuracy of clustering may be improved. Set a set of data X = {x1 , x2 , … , xn }, the variation coefficient vx is calculated as follows:

vx =

Sx | | |X | | |

(7)

n 1∑ Where X = X , i = (1, 2, … , n) is described as the mean of data, n i=1 i √ n 1 ∑ Sx = (X − X)2 is the standard deviation. vx represents the dispersion degree n − 1 i=1 i of the feature, which is named variation coefficient here. Accordingly, each feature’s contribution rate is calculated as follows:

wk =

vx p ∑ vk

(8)

k=1

p represents the number of features. Different data sets have different feature dimensions p, we use wk as the symbol for the weight of the k-th feature, where k = (1, 2, … , p).

3.2 Proposed Condensed Information Algorithm Generally, the clustering method is used to concentrate the information of samples, and the sample points generated by clustering are used instead of the essential information. In order to overcome the large time-consuming caused by the excessive number of iter‐ ations in traditional clustering algorithms, we propose a fast clustering method to protect

556

X. Li and Z. Zhou

the privacy information. The main idea of the algorithm is to treat all the sample data as the nodes in the weighted network. The similarity matrix is generated to represent the similarity between them, which is used to distinguish classifications until all nodes are marked to their respective classification. Specific algorithm steps are as follows: (1) Calculate the weights Sij between any two nodes in the weighted network. Taking into account all attribute features, the similarity between the object xi and yi should be calculated as follows: p ∑

sij = √

(xik wk )(xjk wk )

k=1 p ∑

√

(xik wk )2

k=1

p ∑

(9) (xjk wk )2

k=1

Where xik and xjk respectively denotes the attributes of the feature k in node xi and xj.

(2) Establishing the undirected weighted graph means setting up the sample similarity matrix. It can be known from formula (9) that Sij = Sji, Sii = 1. So the similarity matrix is symmetric about the main diagonal. In the algorithm, we only preserve the lower triangular matrix:

Sn×n

⎛ 1 0 ⎜S 1 = ⎜ 21 ⋮ ⋮ ⎜ ⎝ Sn1 Sn2

… … ⋱ …

0⎞ 0⎟ ⋮⎟ ⎟ 1⎠

(10)

(3) Searching the maximum similarity value in the similarity matrix Sab, then the nodes xa1 and xb1 are classified into the first classification b1, Continue searching for all

connection nodes of xa1 and xb1 to find the node xc1 as the following formula:

(

S + Sbc Sc = MAX ac 2

) (11)

Then the node xc1 will be classified into the first classification as b1 = {xa1 , xb1 , xc1 }. xa1, xb1 and xc1 should be marked as visited node, which means they will not be accessed for the next iteration.

(4) Repetitive executing the step (3) until all nodes are marked to their respective classification. Data X = {x1 , x2 , … , xn } will be divided into M classes as Y, where m = (1, 2, … , M):

An Efficient Privacy-Preserving Classification Method

Y = [b1 = {xa1 , xb1 , xc1 }, … , bm = {xam , xbm , xcm } … , bM = {xaM , xbM , xcM }]

557

(12)

Then calculate the cluster center of each class as bm : c ∑

bm =

i=a

xim

3

∈ [b1 , b2 , … , bM ]

(13)

That is to say, bm is the condensed information of each three simples. The schematic diagram of the condensed information is shown in Fig. 2.

(a)Before condensing information

(b)Condensed information

Fig. 2. The result of the condensed information.

It can be seen from Fig. 2(a) that before concentrating on the original information of training samples, attackers can get the training samples and the location of the related field information easily in the learning process. Moreover, they can also get the related support vectors at the end of the training, which makes it easy to cause the leakage of privacy information. Figure 2(b) shows that the original training samples are replaced by the clustering centers. As we all know, the decision function of SVM in the process of classification is generated by the support vector expansion, and the generation of support vector depends on the learning process of the original data. According to the classification criteria of SVM, the learning process is completely visible, so the infor‐ mation of the support vector and some data is exposed. The support vector is different from other data, which contains important information of the sample, so it is easy to cause the leakage of important information. In our scheme we only need to use the new samples made up by clustering centers to train, so that the real support vectors can be hidden to avoid the exposing of privacy information. At the same time, the clustering process of this paper can not completely hide the statistical distribution of datasets, but most of the condensed information produced is not coincident with the original infor‐ mation, to some extent, the distribution of the original data can be hidden. There is a rule to decide the labels of the clustering centers: Setting Lm as the label of the m-th condensed data bm , Lxim i = a, b, c representatives the label of the original

558

X. Li and Z. Zhou

sample in the m-th class. In most cases, the label of bm is decided by the highest frequency label in bm: Lbm = max num(Lxim ). If Lxim is different from each other in bm, which means Lxam ≠ Lxbm ≠ Lxcm. Then Lbm will be decided by the label of nearest simple with the ‖ ‖ clustering center bm : selecting xim that satisfied min‖xim − bm ‖, then Lbm = Lxim. ‖ ‖ The time complexity of the condensed information algorithm is calculated as follows: (1) Computing the weights of p features in n samples, which cost O(np); (2) Calculating the similarity Sij between any two nodes in the weighted network, which cost O(n2 p); (3) Searching the maximum similarity value in the similarity matrix Sab, which cost O(n2 ). Accordingly, the total computation cost is O(np + n2 p + n2 ) ≈ O(n2 p).

4

Experiments and Results Analysis

4.1 Experimental Conditions In this section, we compare the accuracy and time-consuming of our algorithm with several previous privacy-preserving classification algorithms and normal classification algorithms. The experience datasets we choose are from LIBSVM websites and UCI data base, which are real world data. Some of them are diagnosis result, financial data and population statistics that contain sensitive information and need to be protected. Given the fact that our method can also achieve multi-class classification, the datasets we choose contains both of them. All of our experiments are conducted on Matlab R2016b and Windows7 with Intel Core i5-6500 CPU at 2.1 GHz, 4 GB RAM. The basic information of the experimental data is shown in Table 1: Table 1. The dataset Dataset Heart disease Liver disorder Breast cancer Australian Diabetes German credit Iris Wine Thyroid Control

Sample number 270 345 683 690 768 1000 150 178 215 600

Features 13 6 10 14 8 24 4 13 5 60

Class 2 2 2 2 2 2 3 3 3 6

4.2 Accuracy and Time-Consuming In this part, we calculate the weights of features in several datasets. In order to verify the classification accuracy of CI-SVM, we compare it with the LIBSVM in literature [5], FCM-SVM in literature [10], and RSVM in literature [9]. FCM-SVM is a classification

An Efficient Privacy-Preserving Classification Method

559

algorithm which can improve classification speed. RSVM is a privacy-preserving method with random linear transformation, it uses randomly generated vectors to reduce the training data quantity. All of the four classification algorithms apply the RBF kernel function. The parameter C and g are separately optimized by grid search, the search range is C = {2−5 , 2−3 , … , 211 , 213 } and g = {2−15 , 2−13 , … , 21 , 23 }. In each experiment we randomly select 80% as the training sample, to reduce the impact of randomness, we use 300 times fivefold cross-validation. Figures 3 and 4 respectively shows the classification performance of two-class and multi-class. RSVM is almost not used in multi-class datasets because the accuracy is usually unsatisfactory. We can see that for two-class classification, the CI-SVM is consistent with the RSVM in accuracy, and the accuracy is greatly improved compared with LIBSVM and FCM-SVM both in two-class and multi-class classification. However, for muti-class classification, the accuracy of CI-SVM is basically flat with LIBSVM and FCM-LIBSVM, even slightly lower than the latter two. It is because that the class of condensed information depends on the highest frequency class of samples in the collection. Therefore, for two-class datasets, it is easy to figure out the label of condensed information, so the exact clustering in advance can improve the classification accuracy to some extent. But for multi-class datasets, if the frequency of each class is equal in a collection, the label is decided on the class of the sample nearest to clustering center, which may causes inevitable clustering error and affects the accurancy of clas‐ sification.

Accurancy(%)

100% 90% 80% 70% 60%

Australian Diabetes LIBSVM

Breast cancer

German credit

FCM-LIBSVM

RSVM

Liver disorder

Heart disease

CI-SVM

Fig. 3. Comparison of cross-validation accuracy in SVMs for two-class datasets.

In this paper, we measure the training time of the 4 algorithms in the above two-class and multi-class datasets, the aim is to display how much time-consuming can be saved by CI-SVM. The training time contains average parameter research process and the training time of classifier by the selected parameter. Literature [10] used FCM algorithm in the process of clustering. Tables 2 and 3 separately show the time-consuming in training SVMs and clustering.

X. Li and Z. Zhou

Accurancy(%)

560

100% 98% 97% 95% 94% 92% 91% 89% 88% 86% Iris LIBSVM

Wine

Thyroid

FCM-LIBSVM

Control

CI-SVM

Fig. 4. Comparison of cross-validation accuracy in SVMs for multi-class datasets. Table 2. The comparison of time-consuming in training SVMs Dataset Heart disease Liver disorder Breast cancer Australian Diabetes German credit Iris Wine Thyroid Control

Literature [5] (s) 2.13 3.59 7.18 7.19 8.59 11.6 1.69 2.87 5.67 16.2

Literature [10] (s) 0.68 1.25 2.48 2.86 3.15 4.78 0.52 0.91 1.85 5.68

CI-SVM (s) 0.45 0.63 1.78 1.94 2.15 2.53 0.19 0.27 0.42 1.80

Table 3. The comparison of time-consuming in clustering Dataset Heart disease Liver disorder Breast cancer Australian Diabetes German credit Iris Wine Thyroid Control

Literature [10] (s) 0.350 0.746 1.271 1.568 1.725 3.054 0.386 0.583 1.201 4.320

CI-SVM (s) 0.124 0.173 0.571 0.648 0.751 0.916 0.056 0.084 0.105 0.513

In view of RSVM is a different implementation of the SVM compared with the others, it is just as a reference and we don’t try to compare the training time with it. It can be seen from the above tables that: no matter for two-class or multi-class datasets,

An Efficient Privacy-Preserving Classification Method

561

the time-consuming of CI-SVM is generally less than the other two methods, especially to low dimensional datasets. Compared with LIBSVM, it is because the number of condensed samples obtained by the new clustering algorithm is reduced to 33%, so the number of support vectors is greatly reduced. As the time-consuming of SVM classifi‐ cation process is directly proportional to the number of support vectors, the time complexity may reach O(Nsv3 ) in the worst case (Nsv is the number of support vectors), so the classification efficiency of CI-SVM is much better than LIBSVM. As for FCMSVM, when it has the same clustering centers with CI-SVM, the time-consuming of classification between them is almost flat. However, in clustering phase, the time complexity of FCM is O(n3 p), while it is O(n2 p) in CI-SVM. The time-consuming of clustering is shown in Table 3, it confirms the high efficiency of our algorithm in the clustering process. In summary, the clustering efficiency of our scheme is significantly higher than the existing methods.

5

Conclusion

In this paper, we propose a classification scheme CI-SVM for privacy preserving by designing a new clustering method before entering into SVM classifier, which can be applied to both the two-class and multi-class datasets. It avoids excessive iterations caused by traditional clustering method and the dataset are reduced while the similarity relationship of original dataset can also be retained. To evaluate and verify the signifi‐ cance of this paper, we conducted experiments and applied the datasets from LIBSVM websites and UCI data base. Moreover, we also carried out a comparative study with LIBSVM and FCM-SVM to display the superiority of the CI-SVM. The compare of time complexity and experiment results indicate that the timeconsuming can be greatly saved while the classification accuracy is guaranteed. What’s more, the stage of clustering incurs little computation time compared with other clus‐ tering methods, which means the extra calculation imposed on data proprietor can also be executed fast. Acknowledgment. This work was supported by the Fundamental Research Funds for the Central Universities (JUSRP51510).

References 1. Gu, B., Sheng, V.S., Tay, K.Y., et al.: Incremental support vector learning for ordinal regression. IEEE Trans. Neural Netw. Learn. Syst. 26(7), 1403–1416 (2015) 2. Paul, S., Magdon-Ismail, M., Drineas, P.: Feature selection for linear SVM with provable guarantees. Pattern Recogn. 60, 205–214 (2016)

562

X. Li and Z. Zhou

3. Kokkinos, Y., Margaritis, K.G.: A distributed privacy-preserving regularization network committee machine of isolated Peer classifiers for P2P data mining. Artif. Intell. Rev. 42(3), 385–402 (2014) 4. Chen, W.J., Shao, Y.H., Hong, N.: Laplacian smooth twin support vector machine for semisupervised classification. Int. J. Mach. Learn. Cybernet. 5(3), 459–468 (2014) 5. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2(3), 27 (2011) 6. Sun, L., Mu, W.S., Qi, B., et al.: A new privacy-preserving proximal support vector machine for classification of vertically partitioned data. Int. J. Mach. Learn. Cybernet. 6(1), 109–118 (2015) 7. Li, B., Wang, Q., Hu, J.: Fast SVM training using edge detection on very large datasets. IEEJ Trans. Electr. Electron. Eng. 8(3), 229–237 (2013) 8. Zhang, Y., Wang, W.J.: An SVM accelerated training approach based on granular distribution. J. Nanjing Univ. (Nat. Sci.), 49(5), 644–649 (2013). (in Chinese). [ , , . [J]. : , 2013, 49(5): 644-649] 9. Lin, K.P., Chang, Y.W., Chen, M.S.: Secure support vector machines outsourcing with random linear transformation. Knowl. Inf. Syst. 44(1), 147–176 (2015) 10. Almasi, O.N., Rouhani, M.: Fast and de-noise support vector machine training method based on fuzzy clustering method for large real world datasets. Turk. J. Electr. Eng. Comput. Sci. 24(1), 219–233 (2016) 11. Peker, M.: A decision support system to improve medical diagnosis using a combination of k-medoids clustering based attribute weighting and SVM. J. Med. Syst. 40(5), 1–16 (2016) 12. Aydogdu, M., Firat, M.: Estimation of failure rate in water distribution network using fuzzy clustering and LS-SVM methods. Water Resour. Manage. 29(5), 1575–1590 (2015) 13. Shao, P., Shi, W., He, P., et al.: Novel approach to unsupervised change detection based on a robust semi-supervised FCM clustering algorithm. Remote Sens. 8(3), 264 (2016) 14. Kisi, O.: Streamflow forecasting and estimation using least square support vector regression and adaptive neuro-fuzzy embedded fuzzy c-means clustering. Water Resour. Manage. 29(14), 5109–5127 (2015) 15. Wang, Z., Zhao, N., Wang, W., et al.: A fault diagnosis approach for gas turbine exhaust gas temperature based on fuzzy c-means clustering and support vector machine. Math. Probl. Eng. 2015, 1–11 (2015)

ㆯᏱ ⋤ᩥ ⓠ 㒌⹡༖ ᇶன⢏ศᕸⓗᨭᣢྥ㔞ᮘຍ㏿帼兒᪉ἲ ༡ி኱ᏛᏛ㔴 ⮬↛⛉Ꮫ∧

Cross-Class and Inter-class Alignment Based Camera Source Identification for Re-compression Images Guowen Zhang, Bo Wang ✉ , and Yabin Li (

)

Dalian University of Technology, Dalian City 116024, Liaoning Province, People’s Republic of China [email protected]

Abstract. With the sophisticated machine learning technology developing the state of art of model based camera source identification has achieved a high level of accuracy in the case of matching identification, which means the feature vectors of training and test sets follow the same statistical distribution. For a more prac‐ tical scenario, identifying the camera source of an image transmitted via social media applications and internet is a much more interesting and challenging work. Undergoing serials of manipulations, re-compression for instance, the feature vectors of training and test sets mismatch, thus decreasing the identification accu‐ racy. In this paper, cross-class and inter-class alignment based algorithms, inspired by transfer learning, are proposed to minimize the distribution difference between the training and the test sets. Experiments on four cameras with five image quality factors indicate that the proposed cross-class, inter-class alignment based algorithms and their combination outperform the existing LBP method, and presents high identification accuracies in re-compression images. Keywords: Cross-class alignment · Inter-class alignment Camera source identification · Re-compression

1

Introduction

As the development of the social platforms and online media brings great convenience to people, digital images, as one kind of multimedia, are becoming much more important in our daily lives. At the same time, various image processing software and applications have been very popular, and people do not need professional skills for image tampering any longer. Image tampering and forging become easy and convenient. Recently, several events of tampering and forgery of digital images have had negative effect on various fields, justice and science for instance. People’s confidence in the news and social integ‐ rity has declined. More seriously, when the tampered image is used as an evidence in the court, there would be lots of troubles in the justice system. Because of the increasing requirements of digital media integrity and authenticity analysis, the digital image forensics technology has been rapidly developed in the last decade [1]. As one of the most important branch of digital images forensics, source camera identification focuses on identifying the device type, device model or device individual which captures the image. Correspondingly, the source camera identification © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 563–574, 2017. https://doi.org/10.1007/978-3-319-71598-8_50

564

G. Zhang et al.

is divided into three categories [2]. Device-based identification, which aims to explore the type of device capturing the image [3], usually focuses on distinguishing the computer generated image, camera image, cell phone image and scanned image. Modelbased identification pays more attention on identifying the device model [4], for instance camera model, which generates the digital image. Camera-based approaches look forward to specifying the camera individual. In this paper, only the camera model iden‐ tification approaches could be investigated and discussed. In recent years, several algorithms for model-based camera source identification have been proposed. Most of these methods follow a framework, which considers the problem of camera model identification as the classification issue, and as a result, it could be solved by machine learning approaches. As a matter of fact, numerous existing methods of source camera identification in the framework have achieved an approxi‐ mately high detection accuracy more than 90%. A typical approach proposed by Xu et al., extracts 354-dimensional features based on local binary patterns to distinguish camera models. Considering 13 brand and 18 camera models, the average accuracy of the LBP method reaches up to 98% [5]. Another outstanding work by Swaminathan et al. constructs an efficient liner model to estimate the interpolation coefficients of color filter array [6]. Using the CFA coefficients as the feature vector, a distinguished classifier is established for camera source identification, and a high average discrimination accu‐ racy of 90% is achieved on a dataset of 19 camera models [4]. Although the decent high detection accuracies, these algorithms set a default scenario that the training and test image samples are considered raw and unprocessed images, in other words, in the labo‐ ratory environment. Considering a more practical scenario, the digital image, captured by a camera or cell phone, is uploaded to the internet via the fashionable social media applications, for instance Twitter and Facebook in the U.S., QQ, WeChat and Microblog in China. Sequentially, the image could be spread and forwarded. When the digital image is adopted as an evidence in the court, the digital image is usually manipulated and processed in the internet. Because of the popularity of the social media applications, this scenario is more practical and significant for the real world. In this paper, the more practical and meaningful scenario is explored and discussed. In the more practical social media application based scenario, the digital images may experience retouching, geometric transformation such as resizing, re-compression and even D/A, A/D transform and so on, because of the limitations of the social media application platforms and communication channels [2]. It is the enhanced operations and manipulations that make the test images, which are used for camera source identi‐ fication, no longer the raw ones from the devices. Furthermore, the statistical charac‐ teristics and feature distributions deviated from that of the training raw digital images. The deviation of the training raw image samples and the test manipulated image samples makes the practical camera source identification a challenging task. Typically, the manipulations in the social media applications and the internet is complicated and manifold. To simplify the model, we focus on the re-compression operation in the image pipeline. In spite of the restrictions of terminal screen resolution, the re-compression image via the social media applications and internet show few visual differences from its raw copy. Thus, JPEG compression is an almost essential operation

Cross-Class and Inter-class Alignment

565

of image processing in social networks, which makes the simplification reasonable and feasible. But as mentioned before, the statistical characteristics and feature distributions vary largely, which means the identification accuracy in the laboratory environment is no longer reliable. Wang et al. analyzes the influence of JEPG compression on the typical camera model identification approaches [7]. The classification accuracies of the existing algorithms decrease rapidly with the decreasing of JPEG compression quality. An intuitionistic solution could construct an online training system, which generates the training image samples and trains the classification model online, according to the re-compression quality factor obtained from the test samples. In this scheme, the computation costs is huge and online training is time-consuming. Obviously, the scheme of online training system could work while it is unrealistic. In this paper, we focus on the camera source identification of re-compression images. Instead of the online training system, algorithms based on cross-class alignment (CCA) and inter-class alignment (ICA), inspired by transfer learning, are proposed. The paper is organized as follows. In Sect. 2, following by the instruction of moti‐ vation of transfer learning, CCA and ICA based methods are proposed, conjunction with a description of the LBP features used in the method. The experiments are demonstrated and discussed in Sect. 3. Finally, the paper is concluded in Sect. 4.

2

Proposed Algorithms

In this section, we first give a brief introduction of the transfer learning, which inspires us to design the algorithms. Subsequently, the cross-class and inter-class alignment based algorithms are introduced in detail. 2.1 Transfer Learning In the practical scenario of camera source identification for a digital image, the most important reason of fast reduction of identification accuracy is the variation of statistical characteristics and feature distributions between training set and the test image samples, which is caused by the re-compression manipulation used in the social media applica‐ tions and internet [7]. The variation makes the existing methods deviate from the assumption that training set and test set are subject to the same distribution. Without the foundation of classification model, the source camera identification methods based on machine learning spontaneously degrade in the performance field. To solve the mismatch of the distribution, an important approach named transfer learning is proposed. It relaxes the two restrictions of the basic assumptions in traditional machine learning: first one is that the statistical characteristics and feature distributions of training data and test data follow the identically distribution, and the other is that there are enough labeled samples to train a good classification model. Transfer learning aims to use the learned knowledge of training data to solve the problems that there are few or even not any labeled data in test data [8]. Transfer learning suggests that if the training data and test samples are inherently correlated, the training progress in the classifier

566

G. Zhang et al.

definitely contributes to the classification of the test samples, even though the distribu‐ tion of training data varies from that of the test samples. Source Task

Target Task

Domain

Domain

Knowledge

Learning system

Fig. 1. Transfer learning.

The ‘domain’ contains the feature space χ and the marginal probability distribution P(x), where the feature space χ denotes eigenvector space. The ‘task’ can be considered as the tag space y and the target prediction function f (•), where f (•) is used to predict the label of the test sample equivalent to P(x|y). In the other word, the ‘task’, including the source task Ts and target task Tt in the camera source identification, means classifi‐ cation, as Fig. 1 illustrated. In this scenario, transfer learning try to solve the problem described as following.

{

Ds ≠ Dt Ts = Tt

(1)

The key of transfer learning means constructing a projection, or a transformation to minimize the distribution difference between the training set and the test set. 2.2 Cross-Class Alignment Based and Inter-class Alignment Based Approaches Inspired by the transfer learning, we try to design a transformation to minimize the distribution deviation. In this case, two independent approaches are proposed. In our work, the re-compression manipulation is the only factor that makes the distributions of source domain and the target domain deviated. Considering the characteristics of JPEG compression, a Gaussian model is used to evaluate the deviation of these two domains. Cross-Class alignment based approach: Based on the assumption that the test image samples are all re-compressed, a cross-class alignment base approach, which means a global alignment between the source domain (training set) and the target domain (test set), is proposed. It supposes that we have a set of ns samples S = {p1, p2, ···, pns} ∈ Rd in the source domain and a set of nt samples T = {q1, q2,······qnt} ∈ Rd in the target domain, where d is the dimension of the feature vector. To evaluate the difference between these two Gaussian model domains, the

Cross-Class and Inter-class Alignment

567

minimization of expectation and the standard deviation are considered, as Eqs. (2) and (3) shows. E(�(sj )) = E(tj )

(2)

�(�(sj )) = �(tj )

(3)

where sj denotes of the j-th feature of training samples, and tj disnotes the j-th feature of the test samples, j = 1, 2,···, d. E(sj), σ(sj) is defined to represent the expectation and the standard deviation of the j-th feature of samples in source domain. The transformation φ(·) for each feature is defined as following [9]. �(sji ) = (sji − E(sj ))

�(tj ) + E(tj ) �(sj )

(4)

where j = 1, 2,···, d, and i = 1, 2,···, ns. Inter-class alignment based approach: The global alignment, cross-class alignment, focuses on the re-compression effect between the training set and the test set. Considering the alignment between the labels, an inter-class alignment base approach is presented. Correspondingly, we aim to mini‐ mize the intra-class expectation and standard deviation, as Eqs. (5) and (6) shows.

E(�(sj ), y) = E(tj , y)

(5)

�(�(sj ), y) = �(tj , y)

(6)

where j = 1, 2···, d. Similar with the cross-class alignment, the transformation φ(·) could be described as following, with a specified label for each class [10].

�(sji ) = (sji − E(sj , yi ))

�(tj , yi ) + E(tj , y) �(sj , yi )

(7)

where j = 1, 2···, d, i = 1, 2···, ns. The problem of the Eq. (7) is that the label of the target domain is unavailable, which means the E(tj,y) and σ(tj,y) are not available. As a result, we have to use the joint estimate p(y|ti) instead of the label y. To obtain the p(y|ti), we directly train a classifier model using the training samples and predict the test samples. Therefore, the approximate E (tj,y) and σ (tj,y) could be computed as following: nt

∑ j 1 t p(y|ti ) p(y|ti ) i=1 i i=1

E(tj , y) ≈ ∑nt

(8)

568

G. Zhang et al.

√ √ √ �(tj , y) ≈ √

nt

∑ j 1 (t − E(tj , y)2 p(y|ti ) ∑nt p(y|ti ) i=1 i i=1

(9)

where j = 1, 2,···, d. Furthermore, the cross-class alignment and inter-class alignment can be combined, to minimize the influence of re-compression. 2.3 LBP Features To verify the proposed alignment based algorithms, efficient features are required in our approaches. LBP [5], CFA [6] and IQM [11] are admitted outstanding feature vectors used for model-based camera source identification. In our work, the LBP features are adopted and also in our future work, the other feature vectors are certified resultful. The LBP features, propose by Xu et al., are designed based on uniform gray-scale invariant local binary patterns [3], which can be described as:

LBPu2 = p,R

∑p−1 p=0

s(gp − gc )2p

(10)

where R is the radius of a circularly symmetric neighborhood used for local binary patterns, and P is the number of samples around the circle. We set R = 1, P = 8. gc and gp represent gray levels of the center pixel and its neighbor pixels, respectively. Which is showed in Fig. 2.

P=8, R=1

U

U

UN

UN

U-uniform, UN-non-uniform

Fig. 2. (Left) Constellation of neighborhood. (Right) Example of ‘uniform’ and ‘non-uniform’ local binary patterns.

Function s is defined as:

s(x) =

{

1, x ≥ 0 0, x < 0

(11)

The differences between the central pixels and the neighborhood pixels are firstly calculated. Subsequently, the differences are binary quantized and coded according to the function s. to form a 8-dimensional histogram with a total of 28 = 256 bins named local binary patterns. Inspired by [12], both of the ‘uniform’ and ‘non-uniform’ local binary pattern are included in [5]. Considering the majority of ‘uniform’ local binary

Cross-Class and Inter-class Alignment

569

pattern in the total patterns, only 58 ‘uniform’ patterns are merged with the ‘nonuniform’ patterns to generate 59 effective patterns. For each color channel, the LBP features are extracted from (i) original image, (ii) its prediction-error counterpart, and (iii) its 1st-level diagonal wavelet subband, resulting in a total of 59 × 3 = 177 features, as Fig. 3 illustrated. Respect to the same processing strategy for red and blue channel because of Bayer CFA, we only extract LBP features from red and green channels to reduce the dimension of the feature vector. Finally, a total of 177 × 2 = 354 features are achieved. Prediction Error

Predictor

Prediction Error

59 D

Prediction Error

59 D

Prediction Error

59 D

One Color Channel(Sp atial Domain)

Wavelet Transform

Prediction Error

Fig. 3. Feature extraction framework for one color channel

3

Experimental Results

3.1 Experimental Setup and Parameters To testify the performance of the proposed algorithms, four camera models from ‘Dresden Image Database’ are used in our experiments. For each camera model, 350 image samples are selected randomly and each image is cropped into 6 non-overlap subimages. Therefore, 2100 images samples are obtained in each camera model, as Table 1 shows. Table 1. Image dataset. Camera model Agfa_DC-830i Kodak_M1063 Pentax_OptioA40 Sony_DSC-W170

Resolution 3264 × 2448 3664 × 2748 4000 × 3000 3648 × 2736

Images number 350 350 350 350

Samples number 2100 2100 2100 2100

Abbr AGF KOD PET SON

In all of our experiments, LibSVM [13] is used as the classifier to train the classifi‐ cation model and classify the test image samples. 1500 images of each camera are randomly selected as the training set, and the remaining 600 images of each camera are treated as the test samples. To simulate the re-compression manipulations, five typical quality of JPEG images are investigated in our experiments, including the original JPEG, which means the initial JPEG quality factors used in the camera, and the standard quality factors of 100, 90, 80 and 70.

570

G. Zhang et al.

3.2 Experimental Result and Analysis The LBP method is used as the baseline in our experiments. Table 3 shows the average identification accuracies for the different training models with various quality images, and the different quality test images. It is easy to draw a conclusion that the classification reaches the highest identification accuracy when the training model matches the test image in the matter of image quality, as the diagonal elements indicate. For instance, with the model trained by the original JPEG images, a high accuracy of 94.04% is achieved for the original test samples. In the mismatching case, such as the re-compres‐ sion quality factor of 100, the average accuracy decrease to 90.63%. While for the JPEG quality factor of 80 and 70, the classifier is considered to be out of work, as the accuracies drop to 47.29% and 33.92% (Table 2). Table 2. Average accuracies of different quality images for the baseline of LBP. Training test Original 100 90 80 70

Original 94.04 89.38 66.04 52.54 44.58

100 90.63 93.54 61.67 45.96 37.58

90 82.04 69.54 93.96 58.38 46.33

80 47.29 41.88 43.46 86.13 63.67

70 33.92 32.46 30.83 59.54 80.50

In our following experiments, we use the raw images as the training set, which is widely considered as the best strategy in the practical scenario. With the CCA based algorithm, the confusion matrixes for re-compression qualities of 100, 90, 80 and 70 are shown in Tables 3, 4, 5 and 6. Table 3. Confusion matrix of CCA method for quality factor of 100. AGF KOD PET SON

AGF 93.00 2.67 1.50 6.83

KOD 3.33 93.33 0.17 5.33

PET 1.17 1.17 97.50 0.83

SON 2.33 3.00 0.83 87.00

Table 4. Confusion matrix of CCA method for quality factor of 90. AGF KOD PET SON

AGF 75.83 8.17 2.00 5.33

KOD 13.17 86.50 1.17 5.00

PET 2.83 0.83 96.83 3.17

SON 8.17 4.67 0.33 87.00

Cross-Class and Inter-class Alignment

571

Table 5. Confusion matrix of CCA method for quality factor of 80. AGF KOD PET SON

AGF 47.00 13.50 9.33 23.50

KOD 25.83 58.50 5.33 20.67

PET 19.17 6.00 76.83 3.00

SON 8.00 22.00 8.50 52.83

Table 6. Confusion matrix of CCA method for quality factor of 70. AGF KOD PET SON

AGF 34.50 10.50 13.50 21.00

KOD 21.50 59.50 11.67 39.33

PET 33.00 7.17 67.33 8.50

SON 11.00 22.83 7.50 31.17

Similarly, Tables 7, 8, 9, 10 and 11 show the details of the experimental results of ICA based algorithm for various JPEG quality factors. Table 7. Confusion matrix of ICA method for quality factor of 100. AGF KOD PET SON

AGF 94.17 2.50 1.17 6.33

KOD 3.00 95.00 1.17 10.50

PET 0.67 1.00 97.50 0.67

SON 2.17 1.50 1.17 82.50

Table 8. Confusion matrix of ICA method for quality factor of 90. AGF KOD PET SON

AGF 69.83 2.00 1.17 1.33

KOD 4.83 85.00 1.33 1.17

PET 5.33 1.00 97.17 5.83

SON 20.00 12.00 0.33 91.67

Table 9. Confusion matrix of ICA method for quality factor of 80. AGF KOD PET SON

AGF 19.50 22.67 2.83 31.33

KOD 16.00 38.50 2.83 23.00

PET 38.67 15.33 88.17 6.83

SON 25.83 23.50 6.17 38.33

572

G. Zhang et al. Table 10. Confusion matrix of ICA method for quality factor of 70. AGF KOD PET SON

AGF 19.67 34.50 16.00 35.83

KOD 2.67 26.83 4.17 12.00

PET 71.83 25.17 78.00 28.33

SON 5.83 13.50 1.83 28.33

Table 11. Confusion matrix of CCA + ICA method for quality factor of 100. AGF KOD PET SON

AGF 93.83 2.33 1.17 6.83

KOD 2.83 94.17 0 4.50

PET 1.00 1.17 97.67 0.67

SON 2.33 2.33 1.17 88.00

For comparison, a combination of CCA and ICA base algorithm is also evaluated with the same image data set and experimental parameters. The confusion matrixes are shown in Tables 11, 12, 13 and 14. Table 12. Confusion matrix of CCA + ICA method for quality factor of 90. AGF KOD PET SON

AGF 67.67 3.00 1.50 1.67

KOD 14.00 92.67 1.33 3.17

PET 4.83 – 97.00 2.33

SON 13.50 4.33 0.17 92.83

Table 13. Confusion matrix of CCA + ICA method for quality factor of 80. AGF KOD PET SON

AGF 27.83 6.83 5.33 6.00

KOD 33.83 39.17 2.67 8.50

PET 22.50 7.67 80.67 2.83

SON 15.83 46.33 11.33 82.67

Table 14. Confusion matrix of CCA + ICA method for quality factor of 70. AGF KOD PET SON

AGF 18.17 7.00 20.50 16.50

KOD 11.33 41.00 12.83 29.00

PET 50.50 11.00 51.50 10.50

SON 20.00 41.00 15.17 44.00

Cross-Class and Inter-class Alignment

573

By investigating all of the confusion matrixes of CCA, ICA and the combination of CCA and ICA, we can find that the classification performance can be promoted in different degree. For instance, the CCA, ICA and combination reaches 92.83%, 92.29% and 94.64% respectively for the image quality factor of 100, compared with the accuracy of 90.63% of the baseline, as Table 15 shows. Meanwhile for the quality factors of 100 and 90, the combination of CCA and ICA obtains the best results of 94.64% and 87.54%. But for the low quality factors of 80 and 70, the CCA based algorithm shows the highest accuracies of 58.79% and 48.13%. A further analysis indicates that the increasing inac‐ curate tags labeled in the ICA based algorithm have negative effects on the combination of CCA and ICA. Table 15. Comparison of the proposed algorithms and the baseline. Quality 100 90 80 70

4

LBP 90.63 80.24 47.29 33.92

CCA 92.83 86.54 58.79 48.13

ICA 92.29 85.91 46.25 37.08

CCA + ICA 94.64 87.54 57.58 38.67

Conclusion

This paper focused on identifying the camera source of image with different JPEG quality re-compression manipulations. Inspired by transfer learning, cross-class align‐ ment and inter-class alignment based algorithms are presented. Experiments indicate that the proposed CCA, ICA and the combination outperform the baseline. In the case of re-compression quality factors of 100 and 90, the average accuracies of 94.64% and 87.54% are achieved by the combination algorithm. Meanwhile for the quality factors of 80 and 70, decent accuracies of 58.79% and 48.13% are shown respectively. Although for the quality factors of 100 and 90, the algorithms have good performance, the accu‐ racues achieved in the case of re-compression quality factors of 80 and 70 illustrate the algorithms should be improved. Acknowledgement. This work is supported by the National Science Foundation of China (No. 61502076) and the Scientific Research Project of Liaoning Provincial Education Department (No. L2015114).

References 1. Piva, A.: An overview on image forensics. ISRN Sig. Process. 2013, 1–22 (2013) 2. Wang, B., Yang, F.: An overview and trends on digital image source forensics. J. Inf. Secur. Res. 2(5), 501–511 (2016) 3. Mckay, C., Swaminathan, A., Gou, H., et al.: Image acquisition forensics: forensic analysis to identify source. In: IEEE International Conference on Acoustics, Speech and Processing, pp. 1657–1660. IEEE (2008)

574

G. Zhang et al.

4. Deng, Z., Gijsenij, A., Zhang, J.: Source camera identification using auto-white balance approximation. In: 2011 IEEE International Conference on Computer Vision (ICCV). IEEE, pp. 57–64 (2011) 5. Xu, G., Shi, Y.Q.: Camera model identification using local binary patterns. In: Proceedings IEEE International Conference on Multimedia and Expo (ICME), pp. 392–397. IEEE (2012) 6. Popescu, A.C., Farid, H.: Exposing digital forgeries in color filter array interpolated images. IEEE Trans. Signal Process. 53(10), 3948–3959 (2005) 7. Bo, W., Yin, J.F., Li, Y.B.: Analysis of the effect form the JPEG compression on source camera model forensics. J. Netw. Inf. Secur. (2016) 8. Zhang, F.: Survey on transfer learning research. J. Softw. 26(1), 26–39 (2015) 9. Arnold, A., Nallapati, R., Cohen, W.W.: A comparative study of methods for transductive transfer learning. In: International Conference on Data Mining Workshops, pp. 77–82 (2007) 10. Li, X., et al.: Generalized transfer component analysis for mismatched JPEG steganalysis. In: IEEE International Conference on Image Processing, pp. 4432–4436 (2014) 11. Avcibas, I.: Steganalysis using image quality metrics. IEEE Trans. Image Process. 12(2), 221–229 (2003) 12. Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002) 13. Chang, C.-C.: LIBSVM: a library for support vector machine. ACM Trans. Intell. Syst. Technol. 2(3), 1–17 (2011)

JPEG Photo Privacy-Preserving Algorithm Based on Sparse Representation and Data Hiding Wenjie Li, Rongrong Ni(B) , and Yao Zhao Institute of Information Science and Beijing Key Laboratory of Advanced Information Science and Network Technology, Beijing Jiaotong University, Beijing 100044, China {rrni,yzhao}@bjtu.edu.cn

Abstract. Wide spread of electronic imaging equipment and Internet makes photo sharing a popular and influential activity. However, privacy leaks caused by photo sharing have raised serious concerns, and effective methods on privacy protection are required. In this paper, a new privacypreserving algorithm for photo sharing, especially for human faces in group photos, is proposed based on sparse representation and data hiding. Our algorithm uses a cartoon image to mask the region that contains privacy information, and performs sparse representation to find a more concise expression for this region. Furthermore, the sparse coefficients and part of the residual errors obtained from sparse representation are encoded and embedded into the photo by means of data hiding with a secret key, which avoids introducing extra storage overhead. In this way, the privacy-protected photo is obtained, and only people with correct key can reverse it to the original version. Experimental results demonstrate that the proposed privacy-preserving algorithm does not increase storage or bandwidth requirements, and meanwhile ensures a good quality of the reconstructed image. Keywords: JPEG Data hiding

1

· Privacy-preserving · Sparse representation

Introduction

With the widespread of mobile phones as well as the popularization of social network applications, sharing photos online is becoming more and more enjoyable, especially group photos taken in specific scenes. However, due to the fact that most of the photo sharing applications do not have an option to protect individual privacy, a negative impact on privacy protection has been caused by this popular activity. Recently, a large number of privacy disclosure cases have aroused the public concern on privacy protection methods. To find a sound solution to the privacy leakage problem, a large body of work has been done on the privacy protection of videos and photos. In terms of video privacy protection, some video surveillance systems protect the privacy c Springer International Publishing AG 2017  Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 575–586, 2017. https://doi.org/10.1007/978-3-319-71598-8_51

576

W. Li et al.

information by implementing access rights management [1] or using scrambling technique to protect privacy in specific regions [2]. As for photo privacy protection, considering the fact that almost all the photos used in photo sharing applications are in JPEG format, many effective methods are proposed based on JPEG photos, including JPEG scrambling [3], JPEG transmorphing [4], P3 (privacy-preserving photo sharing) algorithm [5] and so on. Similarly, scrambling technique also can be used in photo privacy protection by arbitrarily modifying the signs of the quantized discrete cosine transform (DCT) coefficients. This method ensures that the privacy information in sensitive regions can be protected, but it is hard to reach a good visual effect at the same time. To improve that, Lin et al. [4] replaced the sensitive region with a smiley face, and inserted the original privacy information into the application markers, while this approach increased the storage and bandwidth requirements. What’s more, another popular method on photo privacy protection is P3 algorithm, which divided the photo into a public part and a secret part by setting a threshold for quantized DCT coefficients. Only the public part was uploaded to the photo sharing service providers (PSPs). Thus, the secret part that contains most of the privacy information can be protected. However, P3 only works on the whole-image level and it needs an extra cloud storage provider to store the encrypted secret part. In this paper, we propose a novel method on photo privacy-preserving based on sparse representation and data hiding, which ensures that the photo privacy can be protected in an effective and convenient way. Meanwhile, if the correct key is provided, the photo can be recovered to its original version without need to keep a copy of the original. In this algorithm, the sparse representation is used to find a more concise expression for each image signal in the target region, which can benefit the following data hiding step. And the data hiding approach is employed to avoid extra overhead to storage. Based on the proposed algorithm, the privacy information can be protected, and the privacy-protected photo can be recovered without sacrificing the storage, the bandwidth, and the quality of the reconstructed photo. Furthermore, according to this photo privacy-preserving algorithm, we propose an efficient photo sharing architecture that consists of two parts: (1) a client side for performing photo privacy protection and photo reconstruction, and (2) a server for storing privacy-protected photos. The rest of this paper is organized as follows. The proposed image privacypreserving algorithm is presented in detail in Sect. 2. And the experimental results along with corresponding analysis are discussed in Sect. 3. Finally, Sect. 4 will draw the conclusions of this paper.

2

Photo Privacy-Preserving Algorithm

In this section, a detailed introduction of the proposed photo privacy-preserving algorithm is given in two aspects: photo privacy protection and photo reconstruction. Based on the fact that JPEG is the most popular format of the photos in online social networks, the proposed algorithm is mainly applied on photos in JPEG format.

JPEG Photo Privacy-Preserving Algorithm

2.1

577

Photo Privacy Protection

The flowchart of the photo privacy protection process is shown in Fig. 1. The main procedures include face detection and cartoon mask addition, sparse representation and residual errors calculation, residual errors addition, along with arithmetic coding and data hiding.

Fig. 1. Flowchart of photo privacy protection process.

Face Detection and Cartoon Mask Addition. For a photo owner, to perform photo privacy protection, face detection is conducted at the beginning to provide a number of candidate regions that probably need to be selected to protect. The selected regions can also be other regions where there is no human face, which means the target regions can also be chosen by the owner arbitrarily. The boundaries of the selected regions are further adjusted to be aligned with the blocks used in JPEG compression. Note that, for the convenience of introducing our algorithm, the procedures are described in the case that only one region is selected to protect, as is shown in Fig. 1(b), and the privacy protection method for multiple regions is similar. Based on the coordinate of target region, the image with mask Imask , shown as Fig. 1(c), is generated depending on the cartoon mask and its edge information. Here, the use of kinds of cartoon masks can not only protect privacy, but also make photo sharing be more interesting and ensure a good visual effect of the privacy-protected photo. Since the original image Iori (Fig. 1(a)) is assumed in JPEG format, its DCT coefficients can be easily extracted. Therefore, to generate Imask in JPEG format, the most straightforward approach is to change the DCT coefficients of the target region in Iori . In this way, the double compression on the whole image can be avoided, and consequently the distortion of the whole image is reduced. Note that, the DCT coefficients mentioned here and all of the DCT coefficients mentioned below refer to the quantized DCT coefficients. To obtain the new DCT coefficients of the target region, the first step is to generate the region with cartoon mask in pixel domain by doing pixel replacement. Due to the difference between the size of the cartoon mask and target

578

W. Li et al.

region, the cartoon mask along with its edge information are resized to match target region. Then, for the position outside the edge, pixel values of the original image are preserved. And for the position inside the edge, the original image values are replaced by corresponding pixel values of the cartoon mask. The second step is JPEG compression. Denote Q as the quality factor of Iori . The new DCT coefficients of target region can be obtained by doing JPEG compression on the small rectangular region obtained from the first step using the quality factor Q. After that, the leading coefficients are divided into two parts: (1) the coefficients completely independent with the cartoon mask and (2) the coefficients associated with the cartoon mask. As shown in Fig. 2(a), coefficients of the blocks with green borders belong to the first part, and coefficients of other blocks belong to the second part. Note that, the size of these blocks is 8 × 8 by default. Finally, we can change the DCT coefficients of the target region in Iori by the new coefficients. To minimize distortion, only the DCT coefficients of blocks that belong to the second part are used to replace the original DCT coefficients of these blocks. Thus, Imask in JPEG format is generated. Furthermore, the new target region Robj which contains all the privacy information, as is shown in Fig. 1(f), will be processed in the following steps.

Fig. 2. Regional division methods.

In addition, the cartoon mask in PNG format and its edge information along with a dictionary which will be used in next step are stored on the client side. This is reasonable since they are very small. Here, the storage of the edge information can avoid edge detection of the cartoon mask image in every photo privacy protection process. Sparse Representation and Residual Errors Calculation. Considering that the new target region Robj contains too much information, K-means singular value decomposition (K-SVD) algorithm [6,7] is employed to design an overcomplete dictionary D that lead to sparse representation. Since the amount of information in U and V components is very small, their DCT coefficients can be encoded and embedded directly in the arithmetic coding and data hiding step, and this process is omitted in Fig. 1. That is to say, only Y component of Robj is processed in this step.

JPEG Photo Privacy-Preserving Algorithm

579

Given Y component of Robj , the sparse representation is performed on patchlevel with the over-complete dictionary D trained by K-SVD algorithm. Unless stated, the size of the patches in our algorithm is always set to 8 × 8, which is the same as the size of the blocks used in JPEG compression. For each block, the pixel values are firstly vectorized as yi ∈ Rn×1 (i = 1, 2, · · · , S). S denotes the total number of blocks in Robj . n is the length of a single vector, which is equal to 64 in our algorithm. After that, the dictionary D, which contains K prototype signal atoms for columns, is used to calculate sparse representation coefficients xi for corresponding yi . More specifically, based on D, yi can be represented as a sparse linear combination of these atoms min yi − Dxi 22

subject to

xi

xi 0 ≤ L,

(1)

where  · 0 is the l0 norm, which indicates the number of nonzero entries of a vector. And L is used to control the sparse degree of xi . Note that the K-SVD training is an offline procedure, and the well trained dictionary is fixed in our algorithm. Generally, as long as the sparse coefficients and the dictionary D are known, a reconstructed signal y′i corresponding to the original signal yi can be computed. For the convenience of the encoding of xi , we adjust them to integers x′i = round(xi ). Consequently, the reconstructed signal y′i can be calculated by formula 2. (2) y′i = Dround(xi ) = Dx′i , where i = 1, 2, · · · , S, xi ∈ RK×1 , and x′i ∈ ZK×1 . Note that, the reconstructed signal y′i is not strictly equal to the original signal yi , which means that there exists a difference between them. And, by doing subtraction, we can get the residual errors in spatial domain. However, since part of the residual errors will be added directly to the DCT coefficients of a specific area in Imask in the next step, the residual errors between DCT coefficients of y′i and yi should be calculated instead of the residual errors in spatial domain. Therefore, we directly extract the DCT coefficients of yi from Iori , denoted as Cyi . Moreover, by doing pixel value translation (from [0, 255] to [−128, 127]), DCT transformation, and quantization using the same quantization table with Iori , the DCT coefficients of y′i can be obtained, denoted as Cy′i . Thus, the residual errors of DCT coefficients can be obtained by formula (3). ei = Cy′i − Cyi .

(3)

where ei ∈ Zn×1 . Figure 3 shows the process of sparse representation and residual errors calculation. After all these procedures, the sparse coefficients matrix X and the residual error matrix E corresponding to Robj are obtained. The size of matrix E is the same as the size of the selected rectangular region, as is shown in Fig. 1(b), and the values at the positions outside of Robj are set to 0.

580

W. Li et al.

Fig. 3. Sparse representation and residual errors calculation.

Residual Errors Addition. In this step, the residual error matrix E generated from the previous procedure is divided into two parts according to the division method shown in Fig. 2(b). Based on the division method, blocks with green borders are defined as edge blocks and other blocks are defined as internal blocks. Thus, the matrix E can be divided into two parts: E1 and E2 , corresponding to the residual errors in edge blocks and internal blocks. To reduce the amount of information that needs to be embedded, E2 is directly added to the DCT coefficients of the corresponding internal blocks in Imask . (4) Cshow = Cmask + E2 , where Cmask indicates the DCT coefficients of the internal blocks in Imask , and Cshow represents the new DCT coefficients of these blocks. Thus, E2 can be easily extracted in the photo reconstruction process by doing a subtraction. Since E2 is very small and most values in it are 0, the distortion caused by this procedure almost does not affect the visual quality of the photo and the size of the JPEG file is almost unchanged. Finally, we get Ierror , as is shown in Fig. 1(d). Besides, E1 will be processed in the next step. Arithmetic Coding and Data Hiding. Unlike some previous photo privacy protection methods that increased the storage or bandwidth requirements, our algorithm considers the privacy information as secret data and directly embeds it into the photo. The sparse coefficients matrix X, the residual error matrix E1 , the DCT coefficients of U and V components of Robj along with some auxiliary information will be encoded and embedded into the photo in this step. First, arithmetic coding method is used to encode X and E1 . Two short binary bit streams can be generated and further sequentially linked to form a

JPEG Photo Privacy-Preserving Algorithm

581

long bit stream. Here, the length of the first short bit stream that comes from sparse coefficients matrix is used to distinguish which part of the long bit stream is corresponding to X in the decoding phase, and this parameter is denoted as lX . We use 16 bits to encode lX , and subsequently set these bits at the beginning of the long bit stream. Then the long bit stream will be embedded into the DCT coefficients of Y component of Ierror based on F5 steganography [8]. Reversible data hiding method is also available in case you need to recover the original image losslessly in the photo reconstruction process. Additionally, the DCT coefficients of U and V components of Robj will be directly encoded and embedded to the corresponding component of Ierror . An encryption key is used in F5 steganography, so that only the person with correct key is able to extract the secret data hidden in the photo. Note that, two parts of the DCT coefficients of Y component are not available to embed. One is the target region, and its DCT coefficients are used to exactly extract the error matrix E2 in the reconstruction process. The other one is DCT coefficients of the blocks in the first column, which is preserved to embed the auxiliary information.

Fig. 4. Structure of the auxiliary information bit stream. (Color figure online)

Finally, as for the auxiliary information, it mainly contains three parts: (1) the coordinate position of target region, including the coordinate of top left pixel (x, y), and the width and height of target region (w, h), (2) the length of the three bit streams that need to be embedded into Y, U and V components respectively, denoted as lenY , lenU and lenV , and (3) the parameters of F5 steganography in three components, denoted as kY , kU and kV . What’s more, if there are many cartoon mask images available, the sequence number of the used cartoon mask should also be considered as part of the auxiliary information. To obtain the auxiliary information before extracting secret data, Jsteg steganography is used to embed these bits into the DCT coefficients of the blocks in the first column. Figure 4 shows the specific structure of the auxiliary information bit stream. Through all above procedures, the final image Ishow which will be uploaded to the sever can be obtained. Anyone who has access to this sever can obtain Ishow , but only the person with correct key can recover the photo to its original version. Thus, this algorithm is appropriate for photo privacy protection.

582

2.2

W. Li et al.

Photo Reconstruction

After downloading the privacy-protected photo Ishow from the server to the client side, people who has correct key can obtain its original version by reversing the protection procedures described above. The specific process is as follows: (1) Extract the auxiliary information from the blocks in the first column. Thus, the coordinate position of target region, the length of three bit streams along with the parameters of F5 steganography can be obtained at the beginning. (2) Extract the bit streams of secret information from the DCT coefficients of Y, U and V components based on the auxiliary information obtained from step one. Then, by performing arithmetic decoding, the sparse coefficients matrix X, the residual error matrix E1 along with the DCT coefficients of U and V components of the target region can be recovered. Furthermore, the DCT coefficients of the reconstructed edge blocks, denoted as C1 , and the DCT coefficients of the reconstructed internal blocks, denoted as C2 , can be calculated based on the dictionary D and the sparse coefficients matrix X. Note that, the dictionary as well as the cartoon mask image and its edge information are shared by every user on the client side. (3) Resize the cartoon mask to match the target region, and then perform JPEG compression on the mask with the same quality factor as Ishow to obtain its DCT coefficients of Y component. In this way, the DCT coefficients of internal blocks in cartoon mask can be obtained, that is Cmask . (4) Extract error matrix E2 by a simple subtraction E2 = Cshow − Cmask ,

(5)

where Cshow and Cmask indicate the DCT coefficients of Y component in the internal blocks of Ishow and cartoon mask image, respectively. (5) Recover the original DCT coefficients of Y component of target region by  C1 + E1 for edge blocks C= (6) C2 + E2 for internal blocks, where C1 and C2 are the DCT coefficients obtained from step two. For edge blocks, we recover its original DCT coefficients by adding E1 to C1 , and by adding E2 to C2 , the original DCT coefficients of internal blocks can be recovered. Thus, the DCT coefficients of Y component of Robj can be recovered. 6) Replace the DCT coefficients of the target region in Ishow by the recovered DCT coefficients of Y, U and V components. In this way, the image can be recovered to the original version that contains privacy information.

3

Experimental Results and Analysis

In this section, several experiments are conducted to demonstrate the efficacy of the proposed photo privacy-preserving algorithm, including: (1) length comparison between the bit streams from original DCT coefficients and the ones from the proposed method, and (2) performance comparison between JPEG transmorphing method and the proposed method.

JPEG Photo Privacy-Preserving Algorithm

3.1

583

Length Comparison Between Bit Streams

The first experiment is designed to test whether our proposed method can reduce the number of bits required for image signal representation, which can be done simply by comparing the length between bit streams from original DCT coefficients and bit streams from our sparse representation based method. 1000 images under natural scenes from The Images of Groups Dataset [9] are used in the first experiment. These images are collected from Flicker images, with different kinds of scenes and different size, and the number of people in these images is also different. All these images are group photos that multiple people involved, so the size of the face regions is relatively small. For each image, face detection method from Yu [12] is employed to determine the target region. And, to reduce the complexity of the experiment, only the face which is first detected in the image is selected to protect. As for the dictionary used in the experiment, it is trained by K-SVD algorithm. The training samples are the 8 × 8 blocks taken from face regions of 725 images, and these images are from the first part of color FERET database [10, 11]. During the training process, the values of some parameters are set as follows: (1) the number of nonzero entries of sparse coefficients xi is set to 4, which means L is equal to 4, and (2) the size of the final dictionary is 64 ×K, and K is set to 128, and (3) the maximal number of iterations is set to 50. Since the dictionary is well trained, it is fixed in our algorithm, and shared by all the users. Table 1. Length comparison between bit streams. Original DCT Proposed coefficients (bits) method (bits) Average value 8122

5814

Bitrate decrease (%) 28.42

The results are shown in Table 1, and the average length of the bit streams of these 1000 images with two methods are listed in the table. Note that, only the bit streams of Y components are taken into consideration in this experiment. From this table, it can be clearly found that the proposed method has a better performance on shortening the length of bit streams. Additionally, we compute the percentage of the bitrate decrease to characterize the proportion of the bits which can be reduced by applying proposed sparse representation based method. Note that, the bitrate decreases by 28.42% [(8122−5814)/8122], suggesting that 28.42% of bits can be avoided embedding to the image. Figure 5 shows the scatter plot of the length of bit streams with two methods among different images. These images are arranged in ascending order based on the size of target regions, which leads to the upward trend of the length. Obviously, the values of black dots from the original DCT coefficients are almost all bigger than the corresponding values of the red dots from proposed method. As the size of target region increases, the gap between the length of these two

584

W. Li et al.

Fig. 5. Length comparison for the bit streams of 1000 images with two methods. (Color figure online)

methods is becoming more and more apparent, and the advantage of our sparse representation based algorithm is becoming more and more obvious. 3.2

Performance Comparison with JPEG Transmorphing

JPEG transmorphing [4] is an algorithm that can not only protect the photo privacy but also recover the original image with a modified version. This method first replaced the face region with a smiley face to hide the sensitive information, and then inserted this part of sensitive information into the application markers of the JPEG file to ensure that the photo can be recovered when it is needed. In this section, we will compare our proposed method with JPEG transmorphing mainly in two aspects: size of overhead on the JPEG files and quality of reconstructed images. Still the 1000 images from The Image of Groups Dataset [9] are used in the second experiment to measure the performance of these two methods. Experimental results are shown in Table 2, and the results in this table are all average values. The values of PSNR are computed based on the pairs of

JPEG Photo Privacy-Preserving Algorithm

585

Table 2. Performance comparison between JPEG transmorphing and proposed method. PSNR (dB) Bitrate increase (%) JPEG transmorphing 50.5926 Proposed method

49.7378

10.80 0.51

original image and the reconstructed image, and the values in the last column are obtained by comparing the file size of the original images and the privacy-protected images. One can see from the table that the bitrate significantly decreased by switching JPEG transmorphing to the proposed method. In other words, among these 1000 images, the proposed method can keep almost the same quality of reconstructed images as JPEG transmorphing method without adding a large amount of extra storage overhead on the image files. Notice that the large amount of overhead on the image files in JPEG transmorphing method is mainly caused by inserting a sub-image which contains privacy information into the application markers of the JPEG file. Once the size of target region increases, the bitrate of JPEG transmorphing method will also increase. Besides, in the proposed algorithm, once the size of target region increases, the length of the bit stream will also increase, and consequently the quality of the reconstructed image will decrease. In this case, reversible data hiding method can be chosen to replace F5 steganography, which allows users to obtain the reconstructed image in a lossless way. Finally, some examples of photo privacy protection based on our proposed method are shown in Fig. 6, including the case that different cartoon masks are used to protect multiple regions in one photo.

Fig. 6. Some examples based on our proposed method. (a) Original photos. (b) Photos after privacy protection. (c) Reconstructed photos.

586

4

W. Li et al.

Conclusions

This paper presents a photo privacy-preserving algorithm based on sparse representation and data hiding, which can protect the photo privacy as well as reconstruct the photo in a high quality for people who has the correct key. The proposed method conducts sparse representation to reduce the amount of information that requires to be embedded into the image. And data hiding method is employed to avoid adding extra storage overhead on the image files. Experimental results have shown that, compared to the advanced alternatives, our method does not increase the requirements of storage or bandwidth and can keep a pretty high quality for the reconstructed photo. Future work lies in improving the applicability of the algorithm on photos with oversize privacy areas. Acknowledgments. This work was supported in part by the National Key Research and Development of China (2016YFB0800404), National NSF of China (61672090, 61332012), Fundamental Research Funds for the Central Universities (2015JBZ002).

References 1. Senior, A., Pankanti, S., Hampapur, A., Brown, L., Tian, Y.L., Ekin, A., Connell, J., Shu, C., Lu, M.: Enabling video privacy through computer vision. IEEE Secur. Priv. 3(3), 50–57 (2005) 2. Dufaux, F., Ebrahimi, T.: Scrambling for privacy protection in video surveillance systems. IEEE Trans. Circuits Syst. Video Technol. 18(8), 1168–1174 (2008) 3. Lin, Y., Korshunov, P., Ebrahimi, T.: Secure JPEG scrambling enabling privacy in photo sharing. In: 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, vol. 4, pp. 1–6 (2015) 4. Lin, Y., Ebrahimi, T.: Image transmorphing with JPEG. In: IEEE International Conference on Image Processing, pp. 3956–3960 (2015) 5. Ra, M.R., Govindan, R., Ortega, A.: P3: toward privacy-preserving photo sharing. In: 10th USENIX Conference on Networked Systems Design and Implementation, pp. 515–528 (2013) 6. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006) 7. Rubinstein, R., Peleg, T., Elad, M.: Analysis K-SVD: a dictionary-learning algorithm for the analysis sparse model. IEEE Trans. Signal Process. 61(3), 661–677 (2013) 8. Westfeld, A.: F5—A Steganographic Algorithm: High Capacity Despite Better Steganalysis. In: Moskowitz, I.S. (ed.) IH 2001. LNCS, vol. 2137, pp. 289–302. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45496-9 21 9. Gallagher, A., Chen, T.: Understanding images of groups of people. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 256–263 (2009) 10. Phillips, P.J., Wechsler, H., Huang, J., Rauss, P.: The FERET database and evaluation procedure for face recognition algorithms. Image Vis. Comput. 16(5), 295–306 (1998) 11. Phillips, P.J., Moon, H., Rizvi, S.A., Rauss, P.J.: The FERET evaluation methodology for face recognition algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 22, 1090–1104 (2000) 12. Face Detection. https://github.com/ShiqiYu/libfacedetection

Surveillance and Remote Sensing

An Application Independent Logic Framework for Human Activity Recognition Wengang Feng ✉ , Yanhui Xiao, Huawei Tian, Yunqi Tang, and Jianwei Ding (

)

School of Investigation and Anti-Terrorism, People’s Public Security University of China, Beijing 100038, China [email protected]

Abstract. Cameras may be employed to facilitate data collection, to serve as a data source for controlling actuators, or to monitor the status of a process which includes tracking. In order to recognize interesting events across different domains in this study we propose a cross domain framework supported by relevant theory, which will lead to an Open Surveillance concept - a systemic organization of components that will streamline future system development. The main contri‐ bution is the logic reasoning framework together with a new set of context free LLEs which could be utilized across different domains. Currently human action datasets from MSR and a synthetic human interaction dataset are used for experi‐ ments and results demonstrate the effectiveness of our approach. Keywords: Human activity recognition · Logic reasoning framework Markov logic networks

1

Introduction

Societies around the globe have become accustomed to the ubiquitous presence of camera sensors in the private, public and corporate space for purposes of security, monitoring, safety and even provide a natural user interface for human machine inter‐ action. Cameras may be employed to facilitate data collection, to serve as a data source for controlling actuators, or to monitor the status of a process which includes tracking. Thus there is an increasing need for video analytic systems to operate across different domains to recognize interesting events for the purpose of behavior analysis and activity recognition. In order to recognize interesting events across different domains in this study we propose a cross domain framework supported by relevant theory, which will lead to an Open Surveillance concept - a systemic organization of components that will streamline future system development. The framework we propose utilizes Markov logic networks – a combination of probabilistic Markov network models with first-order logic. The proposed framework in this research will pave the way for the establishment of a soft‐ ware library similar to the widely-used OpenCV for computer vision (see Fig. 1). Open Surveillance may be conceptualized as the middleware that connects the computer vision (i.e., domain specific) components with the application that responds to the interpretation of the streaming video. © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 589–600, 2017. https://doi.org/10.1007/978-3-319-71598-8_52

590

W. Feng et al.

Fig. 1. Structure of processing: computer vision functions and definition of logic description is unique for each domain

Existing approaches for behavior analysis and activity recognition are domain specific. No effort has been made to contribute to a framework that functions across domains. A framework is proposed in [1] to recognize behavior in one-to-one basketball on top of arbitrary trajectories from tracking of ball, hands and feet. This framework uses video analysis and mixed probabilistic and logical inference to annotate events that occurred given a semantic description of what generally happens in a scenario. However in order to extend the framework to other domains new low-level events (LLEs) pred‐ icates must be re-defined according to the characteristics of different domains. Thus, a more general and systematic framework is needed and it is essential for building compo‐ nents that function across surveillance domains. Types of interesting events may vary from simple to complex activities and from single agent to multi-agent activities. As in [1] multi-agent activities are challenging due to interactions that lead to large state spaces and complicate the already uncertain low level processing. A spatio-temporal structure is mostly utilized and leveraged to distin‐ guish amongst complex activities. Trajectories are one representation used to capture motion and its spatio-temporal structure. Issues addressed in this study include: (a) How may the patterns of motion be represented with spatio-temporal structure and how can they be decomposed into atomic elements? (b) What reasoning mechanisms must be established to infer high level events (HLEs) from LLEs? (c) Given the noisy nature of extracting LLEs from videos, how may the uncertainty be managed during inference? (d) How may the framework be generalized to function across multiple domains? A complete general set of context-free LLEs is developed, which are represented as spatial, temporal and spatio-and-temporal predicates, such that any activity can be subjected to using the set of LLEs structured inside first-order logic and representing time using an approach based on Allen interval logic. Trajectories are used to represent motion patterns of interesting objects. Due to the uncertainty of motion pattern

An Application Independent Logic Framework

591

representation Markov logic networks are utilized to handle the noisy input trajectories. The set of LLEs is integrated with a Markov logic network. We do not use manual annotations of video to indicate presence of objects but assume that detection of inter‐ esting objects is not a problem and can be handled by existing methods. What’s more the capturing sensor is assumed to be stationary. We tested our approach in two different domains: human gesture and human inter‐ actional activities. For human gesture two datasets are used: Microsoft human actions and Weizmann Dataset. For human interactional activities we used a synthetic one. Every action is distinguished from others with its own specific combinations of spatiotemporal patterns of body parts or agent centroid. Events are normally defined by their interaction with properties of the world and observations of the world. Observations of the world are incorporated into a knowledge base using a set of soft rules. In order to simplify we only focus on the definition of new LLEs and how to design it to be capable of functioning in the cross domain framework. Moreover, we treat both properties and observations as LLEs since they are generated depending on preprocessing results. This work is a major step in assisting in many sectors the development of new video stream monitoring systems. Such systems must rely less on constant attention of human operators. In the following section related work is discussed and the event reasoning framework is explained in more detail in the third section. Experiments and conclusion are discussed in the fourth and last sections, respectively. 1.1 Related Work A survey on human action recognition categorized learning approaches into direct clas‐ sification and temporal state-space models and action detection. Our approach falls into the category of temporal state-space models. More specifically it belongs to generative temporal state-space models. Generative temporal state-space models for action recognition learn a joint distribution over both observations and action labels. In the literature, there is research examining the gener‐ ative temporal state-space models. The three principal approaches incorporate HMMs, grammars, and Markov logic networks. Hidden states of hidden Markov models (HMMs) correspond to different phases (key gestures or sub-actions) of a complete action. They model state transition probabilities and observation probabilities. Oliver et al. [2] used a coupled HMM to model the inter‐ action between two tracks. When a single HMM is trained for each action, the problem of action recognition is formulated as finding the action HMM that leads to highest probability for the observed sequence. HMMs are widely used in human action recog‐ nition [3]. However, HMMs do not handle missing observations well and require large training sets to learn structures that humans can describe easily. Grammars are also categorized as generative models that specify explicitly how action sequences can be observed in order. Context-free grammars (CFG) [5] can define high-level interactions hierarchically using spatial and temporal logic predicates. Context free grammar parsing doesn’t handle the uncertainty of low-level processing

592

W. Feng et al.

thus it is sensitive to low-level failures. While it has been extended to incorporate prob‐ abilistic inference [6], experiences indicate they do not scale well to large datasets and can not deal with multi-agent cases. Markov logic networks (MLNs) [7] are a probabilistic logic, that combine the prob‐ abilistic Markov network model with first-order logic. Morariu and Davis [1] presented a Markov logic network based framework for automatic recognition of complex multiagent events. They automatically detect and track players, their hands and feet, and the ball, generating a set of trajectories which are used in conjunction with spatio-temporal relations to generate event observation. Domain knowledge plays an important role in defining rules, observations, properties and actions of interest. Perse et al. [8] transform trajectories into a sequence of semantically meaningful symbols and compare them with domain experts driven templates in order to analyze team activities in basketball games. However, there is no research towards unifying low-level observation, properties and actions to provide an intermediate layer, on top of which high-level interesting events for different applications under the domain of image/video analysis based surveillance can be developed. In order to acquire a proof of concept (POC) for the proposed cross domain logic framework in our experiment, rules for expressing knowledge are provided manually, though they can be learned through training.

2

Logic Event Reasoning Framework

The proposed logic event reasoning framework uses intuitive knowledge about human action and interaction rules over spatial, temporal and spatio-temporal semantic-free LLEs over trajectories to infer HLEs in a descriptive first-order logic (in Horn clause form). For video analytics related applications popular top-down approaches to extract activities of interests from videos provide spatio-temporal information through trajec‐ tories. Activities can be captured and represented as trajectories and it is then the input of the proposed logic reasoning framework as shown in Fig. 1. The proposed logic reasoning framework uses intuitive knowledge about human action and interaction rules over the input trajectories. Trajectories need to be preprocessed to acquire the universe of discourse. The universe of discourse includes a set of time intervals (T), a set of objects (O), a set of atomic motion trajectory segments (S) and a set of locations of interest (L). Afterwards the semantic-free low level events are grounded from the information acquired from the preprocessed trajectories. Thus grounded databases are ready with grounded predicates for further logic reasoning to recognize high-level events. Some predicates are grounded directly from the sensory data (to be described below). Spatial, spatio-temporal and temporal predi‐ cates will be discussed in detail in the section of the Low-Level Event Grounding. Rules for HLEs must be predefined for each application domain, however learning them from labelled exemplars could also be an alternative. The grounded predicates populate the inference graph of a Markov logic network [9]. Trajectories are decomposed into segments and every segment indicates a possible event interval, from which a larger set of all possible action intervals can be derived. Then probabilistic inference is used to determine which event interval candidate best indicates an interesting high level event.

An Application Independent Logic Framework

593

2.1 Preprocessing In order to extract elements of universal discourse from raw trajectories a preprocessing step is needed. Raw trajectories are decomposed into atomic motion segments. The atomic motion segments are the smallest meaningful sub-trajectories, which indicate no motion changes with respect to speed and movement direction. Every atomic motion segment can either indicate a movement or staying at one location; every atomic motion segment is associated with an event interval. What’s more action intervals of high-level events need to be derived from event intervals extracted from observations. How action intervals are generated will be discussed in more detail in the fourth section. 2.2 Low-Level Event Grounding The domain of discourse over which the low-level motion predicates are defined as follows: T = t1 , t2 , …, a set of time intervals defined in terms of logical units O = o1 , o2 , …, a set of labelled objects; the labels are application domain dependent S = �1 , �2 , …, a set of atomic motion trajectory segments L = �1 , �2 , …, a set of locations The spatial, temporal and spatio-and-temporal predicates over the domain of discourse are defined to be semantic-free LLEs. Spatial and spatial-temporal predicates are described below. Doing so will also illustrate the basic simplicity of the computer vision processing that must be revisited each time a new application domain is introduced. Less than a dozen predicates are needed and may be categorized as spatial and spatial temporal. Spatial-predicates: ( ) near �i , �j , location �i and �j are near to each other ( ) far �i , �j , location �i and �j are far from each other ( ) parellel �i , �j , motion trajectory segment �i and �j are parallel ( ) cross �i , �j , motion trajectory segment �i and �j cross with each other without extension ( ) nonparellel �i , �j , motion trajectory segment �i and �j cross with each other with extension ( ) on �i , �j , location �i is on the motion trajectory segment �j Spatial-temporal predicates: ( ) move oi , �i , ti , object oi moves in motion trajectory segment �i within time interval ti ( ) stopAt oi , �i , ti , object oi stops at location �i within time interval ti

594

W. Feng et al.

Functions over the domain of discourse are as follows: ( ) Di �i , the direction of motion segment �i, the direction of motion segment �i is the ith quantized direction ( ) startLoc �i , the start location of trajectory segment �i ( ) endLoc �i , the end location of trajectory segment �i ( ) len �i , the length of trajectory segment �i ( ) v �i , the average speed of trajectory segment �i ( ) a �i , the acceleration of trajectory segment �i Temporal predicates: 11 temporal relationships in total between these predicates, which are defined over time intervals, are expressed using the following base binary relations and their inverses: before, meets, overlaps, starts, during, finishes, and equals, which are known as Allen’s temporal intervals [10]. As mentioned previously the trajectories of the objects of interest are the first-hand information which describe the motion of the objects. From the raw trajectories we need to derive knowledge in terms of the domain of discourse - time intervals (T), labelled objects (O), atomic motion trajectory segments (S) and locations of interest (L). As shown in Fig. 2 show the knowledge is derived from trajectories is discussed over two domains-human action and human interaction. The left part of Fig. 2 illustrates a person executing a sidekick. A human interation “meet” is shown on the right of the figure.

Fig. 2. Computer vision methods are used to extract objects, motion (segments with direction), and temporal data from video and create corresponding predicates

An Application Independent Logic Framework

595

For sidekick at the left the moving objects is the left foot o1. Trajectories of the left foot are obtained from tracking o1. The trajectories are abstracted into segments �1 and �2 also shown on the two graphs at the left. Proper segmentation of a complex motion is a research problem. These two extracted motion trajectory segments are associated with time inter‐ vals, t1 and t2 respectively. In the language of force dynamics, t1 meets t2. In aggregate, these result in the grounded predicates shown in the ( database ) of Fig. 2.( For human ) action sidekick the spatio-temporal predicates are move o1 , �1 , t1 and move o1 , �2 , t2 are used to describe the movement. Quantitized movement direction ( ) ( ) of two segments �1 and �2 are measured as functions over segments D2 �1 and D4 �1 . Basically previously described predicates give detailed information about the movement of the body part footleft. ( ) ( ) moves in )the second direction move footLeft, �1 , t1 and D2 �1 - the footleft ( ( quantitized ) within the interval t1 as a segment �1. move footLeft, �2 , t2 and D4 �1 - the footleft moves in the fourth direction within the interval t2 as a segment �2. Relations ( )among (time inter‐ ) vals t1, t(2 and )t3 are described by the temporal predicates meets t1 , t2 , starts t1 , t3 and finishes t1 , t3 . The nonparallel ( spatial ) relation of these two segments is grounded as the spatial predicate nonparellel �1 , �2 . Similarly, the human interaction “meet” at the right of Fig. 2 o1 and o2 are two moving agents along a straight path. Trajectories are extracted into motion segments and loca‐ tions of interest. Agents being tracked through time and trajectories are extracted into either segments or locations of interest. Trajectories of agent1 are extracted into �1 and location of interest �1 and for agent2 �2 and location of interest �2 are extracted. The movements of) agent1( and agent2 are as( spatio-temporal predicates ( ) ( described ) ) move o1 , �1 , t1 , stopAt o1 , �1 , t3 , move o2 , �2 , t2 , stopAt o2 , �2 , t4 . Agent1 moves as a motion segment �1 within time interval t1 and stops at a location �1 within time interval t2. Agent2 moves as a motion segment �2 within time interval t3 and stops at a location relation between two locations of interest is �2 within time interval t4. The ( spatial ) grounded as predicate near �1 ,(�2 . Two ) motions segments �1, �2 are parallel and the relation is (described as relations parellel � , � ) ( ) 1 2 (; Temporal ) ( )between( four)time intervals ( ) are (before) t1 , t3 , before t1 , t4 , before t2 , t4 , before t2 , t3 , before t2 , t4 , equal t1 , t2 and t3 , t4 . All action rules consist of three types of predicates (temporal, spatial and spatio‐ temporal predicates), which is described in details in a following subsection. (In reality, the predicates are not grounded at this point. However, it assists in illustration.) The keys to the later stages of processing are twofold: (1) The segments that correctly decompose a motion (such as a human gesture); (2) The choice of predicates that enable the proper interpretation of LLEs. We provide models and examples. For instance, our preliminary testing focused on human gestures. Heuristically, trajectories were segmented at sharp inflection points or at points where speed change abruptly such as “stops” and abstracted as straight lines. Excepting temporal predicates, a single motion predicate was necessary, appropri‐ ately instantiated from the domain of discourse described earlier.

596

W. Feng et al.

2.3 High-Level Event Representation With grounded database rules describing high-level activities are needed for inference engine MLNs to calculate probability of each state of the world. In other words the probability of high-level events given the observation stored in the grounded databases needs to be inferred by MLNs. Basically the rules for a high-level event can be described as a simple Horn clause representation reasoning over semantic-free low-level events. Due to space limitation a subset of rule examples for human action and interactions are selected and illustrated as follows: 2.4 MLNs Let x be the state of the world, i.e., the truth value of all LLEs and HLEs. In general, we wish to know the probability of each state x ∈ X of the system, which can be expressed in a Markov network as

( ) ∑ 1 1∏ � (x) = exp wi fi (x) P(X = x) = Z j j Z i where Z is the partition function �j (x) and fi (x) are real-valued potential and feature functions in the state. The basic inference can be stated in terms of finding the most probable state x given some evidence y ⊆ x, and this is formally defined as arg max P(x|y ) = arg max x

x

∑

wi fi (x)

i

To solve the arg max we choose to use Markov logic networks. MLNs are a language that combine first-order logic and Markov networks. On the logic side, formulas have soft constraints such that a world that violates a formula is less probable than one that satisfies the formula. On the statistical side, complex models can be represented compactly. They can also express any probability distribution over the state of possible worlds X. Compared to previous developments in knowledge-based model construction and statistical relational learning, MLNs are less restricted since they are supported by a comprehensive set of learning and inference algorithms. Due to their generality, MLNs can integrate logical and statistical approaches from different fields within one frame‐ work. Refer again to Fig. 2. The grounding database is populated with predicates. However, this is for illustration only. As presently formulated, we populate it with tuples from the relation R(T, O, S, L). We refer to the grounding database together with the logic rules, Fi, as the knowledge base (KB). Prior to inference, the Markov network must be instan‐ tiated using the KB and weights. At first a set of constants representing objects in the domain of interest is extracted from the grounding database. Secondly with the MLN’s formulas, a set of vertices (predicates) for the logic network is generated by replacing the variables in each formula with the set of constants extracted. If any two vertices (predicates) contribute to the grounding of the same formula, an edge is established

An Application Independent Logic Framework

597

between these two vertices. Therefore each full or partially grounded formula Fi is graphically represented as a clique in the Markov network. As shown in Fig. 2 computer vision methods are used to extract objects, motion (segments with direction), and temporal data from video and create corresponding predicates. ( ) To state more declaratively, an MLN graph L consists of Fi , wi pairs where Fi is a formula in first-order logic and wi is a real number interpreted as the weight of Fi. While not a precise relation, a weight wi may be interpreted as indicating that Fi is ewi times likely to have a true grounding for an arbitrary world than not have one. The number of true groundings of Fi in state x, ni (x, y), is used as the feature function fi (x) in the formulae above. Any MLN defines a Markov network ML,C [⋅], and thus finding the most probable state in MLN can be expressed in a Markov network manner as arg max P(x|y ) = arg max x

3

x

∑

wi ni (x, y)

i

Experiments and Results

We have conducted experiments in two domains (human action recognition and human interaction recognition) to prove the principle of the proposed architecture. Naturally, we could not develop an entire system tailored to a domain such as security in a public place. We took shortcuts: (1) For human action recognition in order to simulate the computer vision aspect, we used Kinect R; this system identified objects leaving us to extract the tracks, the intervals, and segment the motion; For Weizmann dataset we semiautomatically extracted trajectories of a human body part. For each video, we first manually marked the body parts (head, hands and feet) to track, and then run the TLD tracker to generate trajectories for them. (2) For the human interaction recognition a synthetic dataset is generated as in [2]; (3) To accomplish the inference we used the more general Alchemy [11] package. 3.1 Motion Segmentation and Action Interval Generation The MSR Microsoft Research action dataset provides 3-dimensional trajectories capturing motion in 3D real-world settings. We simplified to utilize (x, y) coordinates of the trajectories and four quantized directions in 2D are used. Trajectories are downsampled so that motion direction changes can be calculated from only two neighboring points over the motion trajectory. Afterwards trajectories are decomposed into segments such that for every segment the motion direction doesn’t change abruptly. We fit every segment into a circle. When the radius is large enough the segment can be fitted into a line; however when the radius is small enough it is actually a curve or even a full circle and the curve/circle is segmented further into several sub-segments. The angle subtended at the center of a circle by start and end points on the circle determines the number of sub-segments (the maximal subtended angle is 45°). We fit a line over each segment and calculate the motion direction associated with it.

598

W. Feng et al.

Every point over the trajectory has the fourth dimension, t, to indicate the time stamp when it is captured. Thus it is possible to generate action interval candidates. Both start and end time stamps are considered as moments - start moments and end moments. Action interval is defined as [momentstart and momentend]. momentstart is one of start moments and momentstart is one of end moments. For each action video clip a set of action interval candidates can be generated by choosing momentstart and momentend among moments from segments. What’s more a subset of action interval candidates is the interval length to be within a specific range [further generated by constraining ] durationmin , durationmax 1∕fps as in Fig. 3. Similarly for human interaction the trajectories of synthetic agents mimicking human behavior in a virtual environment are decomposed into segments due to abrupt changes of motion direction and speed. When agents stop at one location the segments are considered as locations of interest instead of movement segments.

Fig. 3. Action interval generation

As shown in Fig. 3 there are five motion segments �1 − �5 and the starting moments and ending moments constitutes the moments m1 − m8. Among them start moments are m1, m2, m3, m5 and m6 the rest are end moments. With the interval length being constrained, final action interval candidates are �1 − �9. 3.2 Results and Analysis For the Microsoft human action dataset the motions targeted were sidekick, draw an X with hand motions, and clap. 76 videos were used. We chose a subset of Weizmann dataset of two people performing 7 natural actions: bend, jack, jump, walk, wave one hand, wave two hands and skip. For human interaction recognition the interaction

An Application Independent Logic Framework

599

targeted were Meet, ChangeDirection, ChangeDirectionMeet, MeetWalkTogether and MeetWalkSeparate. Ten synthetic events where four agents are involved were used. Rules are provided manually for each motion even though it can be learned from training examples. Please refer to Appendix A for rules defined for actions - LeftSideKick, DrawX, HandClapping, Meet, ChangeDirectionMeet, WalkTogether and WalkSepa‐ rate. The result for the Microsoft action dataset is shown in Table 1. Some videos contained only a single action while others contained more than one instance of the same action. Table 1. Experiments based on a skeletal version of the narrow-angle view architecture for Human Action Recognition Actions DrawX HandClap LeftSideKick

TP 45 53 22

FP 4 0 0

FN 1 5 0

Precision 0.92 1.0 1.0

Recall 0.98 0.91 1.0

The result for Weizmann dataset and for human interactional activities is shown in Tables 2 and 3 respectively. Table 2. Experiments for Weizmann dataset Actions Bend Jack Jump Walk Wave1 Wave2 Skip pJump

TP 2 4 4 2 3 3 6 6

FP 0 1 0 1 1 0 1 0

FN 0 0 1 3 1 3 1 0

Precision 1.0 0.8 1.0 0.67 0.75 1.0 0.86 1.0

Recall 1.0 1.0 0.8 0.4 0.75 0.5 0.86 1.0

Table 3. Experiments based on a skeletal version of the narrow-angle view architecture for Human Interaction Recognition Actions Meet ChangeDirection ChangeDirectionMeet Approach MeetWalkSeparate

TP 26 7 7 7 11

FP 0 0 0 0 0

FN 0 1 1 0 1

Precision 1.0 1.0 1.0 1.0 1.0

Recall 1.0 0.87 0.87 1.0 0.92

600

4

W. Feng et al.

Conclusion

A reasoning framework which combines first-order-logic with Markov logic networks is presented in order to recognize both simple and complex activities. Semantic-free predicates are defined thus low-level events (LLEs) and high-level events (HLEs) of interest across different domains can be described by encoding those LLEs and HLEs and temporal logic (Allen’s interval logic) in a first-order-logic presentation. The main contribution is the logic reasoning framework together with a new set of context free LLEs which could be utilized across different domains. Currently human action datasets from MSR and a synthetic human interaction dataset are used for experi‐ ments and results demonstrate the effectiveness of our approach. In the future we will validate our proposed framework to more domains such as intelligent traffic surveillance and design a real-time mechanism for human activity recognition across different domains. Acknowledgement. This paper is supported by the National Natural Science Foundation of China (Grant No. 61501467, Grant No. 61402484, Grant No. 61503387 and Grant No. 61503387), and Fundamental Research Funds for the Central Universities of China (2016JKF01203).

References 1. Morariu, V.I., Davis, L.S.: Multi-agent event recognition in structured scenarios. In: IEEE Computer Vision and Pattern Recognition (Colorado Springs CO), pp. 3289–3296 (2011) 2. Oliver, M.N., Rosario, B., Pentland, P.: A Bayesian computer vision system for modeling human interactions. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 831–843 (2000) 3. Lu, W.-L., Little, J.J.: Simultaneous tracking and action recognition using the PCA-HOG descriptor, p. 6 (2006) 4. Ogale, Abhijit S., Karapurkar, A., Aloimonos, Y.: View-invariant modeling and recognition of human actions using grammars. In: Vidal, R., Heyden, A., Ma, Y. (eds.) WDV 2005-2006. LNCS, vol. 4358, pp. 115–126. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3540-70932-9_9 5. Ogale, A.S., Karapurkar, A., Aloimonos, Y.: View-invariant modeling and recognition of human actions using grammars, pp. 115–126 (2006) 6. Ivanov, Y.A., Bobick, A.F.: Recognition of visual activities and interactions by stochastic parsing. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 852–872 (2000) 7. Domingos, P., Kok, S., Poon, H., Richardson, M., Singla, P.: Unifying logical and statistical AI. In: Proceedings of the Twenty-First National Conference on Artificial Intelligence, Boston, MA, pp. 2–7 (2006) 8. Perše, M., Kristan, M., Kovačič, S., Vučković, G., Perše, J.: A trajectory-based analysis of coordinated team activity in a basketball game, pp. 612–621, May 2009 9. Richardson, M., Domingos, P.: Markov logic networks. Mach. Learn. 62(1–2), 107–136 (2006) 10. Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983) 11. Kok, S., Singla, P., Richardson, M., Domingos, P.: The alchemy system for statistical relational ai. Technical report, University of Washington (2005)

An Altitude Based Landslide and Debris Flow Detection Method for a Single Mountain Remote Sensing Image Tingting Sheng and Qiang Chen ✉ (

)

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China [email protected]

Abstract. The altitude information of single remote sensing image may aid in detecting the natural disaster, such as landslide or debris flow. Accordingly, in this paper, an approach based on altitude is proposed to detect landslide and debris flow for a single mountain remote sensing image. Firstly, we extract the features of landslide and debris flow areas and introduce slow feature analysis (SFA) to improve the feature distinguishability. Then, machine learning and a training model are used to detect suspected landslide and debris flow areas. By using the altitude information calculated by dark channel prior, we analyze the altitude distribution of suspected areas to judge whether landslide and debris flow occur in these regions. The experimental results of multiple mountain remote sensing images with landslide or debris flow demonstrate that the proposed algorithm can accurately detect landslide debris flow areas in a single mountain remote sensing image. Keywords: Landslide and debris flow detection · Altitude information Slow feature analysis · Remote sensing image

1

Introduction

Landslide and debris flow are two kinds of typical geological disasters, because of the most widely distribution and the most serious results. Therefore, it is necessary to detect landslide and debris flow. With rapid development of sensors and improvement of the resolution of remote sensing images, remote sensing is becoming more and more concerned by geological hazard researchers, and it is used as a kind of landslide and debris flow investigation and detection means. To the end of the 20th century, the interpretation of stereoscopic aerial photographs was still the most common method for landslide and debris flow mapping and detection [1]. In remote sensing images, the interpretation of landslide and debris flow was mainly based on relief, hue, topography and other features [2–4]. In recent years, people have begun to use software platforms (such as Photoshop, ArcGIS, and CorelDraw) and combine Digital Elevation Models (DEMs) to implement humancomputer interaction interpretation [5–7], which can enhance images and make spatial features of landslide and debris flow more intuitive. However, these methods are not as simple as the method directly analyzing real relative altitude of suspected regions. © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 601–610, 2017. https://doi.org/10.1007/978-3-319-71598-8_53

602

T. Sheng and Q. Chen

Sheng and Chen [8] proposed an approach based on dark channel prior for altitude extraction of single remote sensing image, and the obtained altitude information can be applied to detect landslide and debris flow. Thus, in this paper, a new method based on altitude information is proposed for landslide and debris flow detection. The main contributions of our work are: (1) the relative altitude information generated from single image was utilized for landslide and debris flow detection; (2) slow feature analysis (SFA) was used to improve the feature distinguishability.

2

Landslide and Debris Flow Detection Combined with Altitude

Figure 1 shows the block diagram of our algorithm including suspected landslide and debris flow detection and refinement based on altitude information. Firstly, effective features (including RGB and image texture processed by SFA) are extracted from training data, then we use support vector machine (SVM) to detect the suspected debris flow and landslide regions. Secondly, altitude information is extracted based on dark channel prior for single mountain remote sensing image. Finally, we refine our results by analyzing the altitude information near suspected areas.

Fig. 1. Block diagram of our algorithm (Color figure online)

2.1 Suspected Landslide and Debris Flow Areas Detection (1) Image features extraction After observing a large number of landslide and debris flow areas, we choose the image features as follows: The first part: RGB. The landslide and debris flow areas are always covered with a lot of mud mixture, which is showing a yellow to brown color state. Thus, RGB is a

An Altitude Based Landslide and Debris Flow Detection

603

significant difference between landslide and debris flow areas and other typical surface features, such as green forests, blue-green rivers, and gray construction areas. The second part: Image texture. Image texture represents the spatial arrangement information of color or intensities in an image or selected regions of an image. The cooccurrence matrix captures numerical features of a texture using spatial relations of similar gray tones. The contrast, correlation, energy and homogeneity computed from the co-occurrence matrix are used to classify textures. (2) Texture feature pre-processing with Slow feature analysis (SFA) Slow feature analysis (SFA) is an unsupervised learning algorithm for extracting slowly varying features from a quickly varying input signal [9]. The learned functions tend to be invariant to frequent transformations of the input and the extracted slowly-varying signals that can be interpreted as generative sources of the observed input data. These properties make SFA suitable for many data processing applications. SFA is a one-shot algorithm, and is guaranteed to find the optimal solution (within the considered function space) in a single step. In this paper, the slow feature analysis was used to improve the distinguishability of the extracted texture features. There are two types of data used in the experiment: land‐ slide or debris flow areas, other areas (including green vegetation areas, rivers and build‐ ings), and each type has about 3000 samples. The results are shown in Figs. 2 and 3,

Fig. 2. Texture features before SFA (Color figure online)

604

T. Sheng and Q. Chen

where the horizontal axis represents the number of pixels and the vertical axis represents the value of features. Figure 2 shows contrast, correlation, energy and homogeneity curves for original data. The red curve represents the feature values of landslide or debris flow areas, and the blue curve shows the features of other areas. Figure 3 shows the texture feature values processed by slow feature analysis.

Fig. 3. Texture features after SFA (Color figure online)

By observing the original data (Fig. 2), it can be found that the two categories of the samples always have a large overlap, which is not conducive for classification. However, the distinction of the slowest feature (Fig. 3(a)) obtained after slow feature analysis is the best, indicating that slow feature analysis can improve the distinguishability. Thus, the slowest feature is selected as the input for the following model training based on support vector machine (SVM). (3) Landslide and debris flow detection based on SVM Based on the above features, we use SVM [10] to detect suspected landslide and debris flow areas. Small areas (less than 100 pixels) are excluded as false objects. Then we used morphological closing using the structuring element 3 × 3 to remove the gap, and obtain the final suspected landslide and debris flow areas, as shown in Fig. 4(b). The white regions represent the suspected landslide or debris flow areas in Fig. 4(a). Figure 4 indicates that there are false landslides or debris flow areas in the primary

An Altitude Based Landslide and Debris Flow Detection

605

detection result, such as the right parts because their image features are similar with those of the landslide or debris flow areas. To remove the false areas, the altitude infor‐ mation near the detected areas will be utilized.

Fig. 4. Suspected landslide or debris flow areas detection result (b) for a remote sensing image (a).

2.2 Landslide and Debris Flow Areas Refinement Based on Altitude To improve the detection accuracy, the altitude information generated based on dark channel prior [8] is then used for landslide and debris flow refinement. The main steps are listed as follows: (1) Extract the altitude information near suspected landslide or debris flow areas. We are interested in the periphery of these suspected landslide or debris flow regions, which can be obtained by morphological expansion. It should be noted that the periphery is not the bigger the better, because the farther the distance, the altitude of the periphery is likely to change, which means that it cannot be used to approx‐ imate the altitude of the suspected landslide or debris flow areas. By comparing experimental results, this paper determined the outward expansion of three pixels with disk-shaped structuring element, as shown in Fig. 5(a). (2) Remove false landslide or debris flow areas based on altitude distribution. Figure 5(b) shows the histogram of the altitude distribution in the peripheral areas (Fig. 5(a)). Since the altitude of the landslide or debris flow peripheral areas has a descending trend from high to low, they have a more dispersed altitude distribution than false areas. The standard deviation (Std) of the altitude histogram is used to describe the degree of dispersion, and remove false landslide or debris flow areas. From Fig. 5, we can observe that the standard deviation (0.0296) of the false land‐ slide or debris flow area (region 2) is obviously lower than those of the real landslide or debris flow area (regions 1). Thus, we can set a proper threshold, such as 0.11, of the standard deviation to remove the false landslide or debris flow areas.

3

Experimental Results and Analysis

Due to the limitations of data sources, a landslide image, four debris flow images of Zhouqu and five debris flow images of other areas were collected. Figure 6 shows the

606

T. Sheng and Q. Chen

Fig. 5. Suspected landslide or debris flow areas and the corresponding distribution of the peripheral altitude values in Fig. 4.

detection result of the landslide image. Figures 7 and 8 show the detection results of two debris flow images of Zhouqu and other area, respectively. For quantitative analysis, three general quantitative evaluation indices are used, including precision, recall and accuracy. The greater the indices, the better the detection effect. These three indices are defined as follows: TP Precision is the fraction of retrieved instances that are relevant: P = . Recall TP + FP TP . Accuracy describes is the fraction of relevant instances that are retrieved: R = TP + FN TP + TN . where TP, TN, the ability to detect correctly of a classifier: A = TP + FN + FP + TN FP and FN represent true positive, true negative, false positive and false negative, respectively. Figures 6, 7 and 8 show the detection results of three images with landslide or debris flow, where (a) is the original images with landslide or debris flow, (b) is the suspected landslide or debris flow areas detected by support vector machine, (c) is the altitude map extracted from original images, (d) shows the manually calibrated landslide or debris flow areas that are taken as ground truth (GT), (e) is the final landslide or debris flow areas detected by our algorithm, (f) shows the deviation between our results and GT (red and blue mean the false and missing areas, respectively). From Figs. 6, 7 and 8, it can be found that the majority of false and missed areas appear in the edge of regions, and it has a certain relationship with GT. Table 1 shows the accuracy indices of landslide or

An Altitude Based Landslide and Debris Flow Detection

607

debris flow detection with and without altitude information. Our algorithm can correctly detect the areas with landslide or debris flow in a single mountain remote sensing image. Meanwhile, Fig. 9 shows the comparison of two methods based on three indices, which also verifies that the landslide or debris flow detection with altitude is better than that without altitude.

Fig. 6. Detection of a landslide image (Color figure online)

608

T. Sheng and Q. Chen

Fig. 7. Detection of a debris flow image (Color figure online)

Fig. 8. Detection of another debris flow image (Color figure online)

An Altitude Based Landslide and Debris Flow Detection Table 1. Evaluation of the landslide or debris flow detection of ten images Evaluation index Methods

Precision

Without altitude Landslide 0.8271 Debris flow 0.3680 1 (Zhouqu) Debris flow 0.8635 2 (Zhouqu) Debris flow 0.3779 3 (Zhouqu) Debris flow 0.7611 4 (Zhouqu) Debris flow 0.5455 5 (other area) Debris flow 0.8640 6 (other area) Debris flow 0.8632 7(other area) Debris flow 0.5981 8 (other area) Debris flow 0.6187 9 (other area) Mean 0.6687

Recall

Accuracy

With altitude 0.8275 0.9696

Without altitude 0.8436 0.5864

With altitude 0.8461 0.4832

Without altitude 0.9467 0.8419

With altitude 0.9471 0.9408

0.9559

0.6495

0.6822

0.8482

0.8830

0.7540

0.4394

0.4696

0.8908

0.9418

0.7852

0.7404

0.7634

0.8095

0.8276

0.5447

0.8528

0.9102

0.8847

0.8857

0.7900

0.8373

0.8431

0.8561

0.8138

0.8609

0.9690

0.9727

0.9431

0.9441

0.7652

0.9422

0.9422

0.9432

0.9715

0.7321

0.9384

0.9392

0.9206

0.9498

0.7985

0.7799

0.7852

0.8885

0.9105

Fig. 9. Comparison of three indices

609

610

4

T. Sheng and Q. Chen

Conclusions

We propose a novel landslide and debris flow detection approach by combining altitude information from single mountain remote sensing image. Firstly, the effective image features of landslide and debris flow areas are extracted, and then the texture features are pre-processed by using SFA to obtain the slowest feature as input for SVM training. Secondly, the suspected landslide or debris flow areas are generated with SVM, and the altitude map is calculated by dark channel prior. Finally, we utilize the altitude infor‐ mation near the suspected landslide or debris flow areas to refine the primary detection results. The experimental results demonstrate that our method can accurately detect landslide or debris flow areas in a single mountain remote sensing image.

References 1. Graciela, M., Lorenz, H., Radu, G.: Remote sensing of landslides: an analysis of the potential contribution to geospatial systems for hazard assessment in mountainous environments. Remote Sens. Environ. 98(2), 284–303 (2005) 2. Tralli, D., Blom, R., Zlotnicki, V., Donnellan, A., Evans, D.L.: Satellite remote sensing of earthquake, volcano, flood, landslide and coastal inundation hazards. ISPRS J. Photogrammetry and Remote Sens. 59(4), 185–198 (2005) 3. Lu, H., Nakashima, S., Li, Y., Yang, S., Serikawa, S.: A fast debris flow disasters areas detection method of earthquake images in remote sensing system. Disaster Adv. 5(4), 796– 799 (2012) 4. Wang, G., Cheng, G., Zhu, J., et al.: The effect of topography and satellite attitude parameters on early detection of regional landslides based on InSAR. In: Canadian Geothnical Conference (2016) 5. Liu, W., Yamazaki, F.: Detection of landslides due to the 2013 Thypoon Wipha from highresolution airborne SAR images 2015. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 4244–4247 (2015) 6. Barlow, J., Martin, Y., Franklin, S.: Detecting translational landslide scars using segmentation of landsat ETM + and DEM data in the northern cascade mountains, British Columbia. Can. J. Remote. Sens. 29(4), 510–517 (2003) 7. Li, C.: Detection of landslide with multi-resolution DEM. faculty of construction and environment, The Hong Kong Polytechnic University (2014) 8. Sheng, T., Chen, Q.: A dark channel prior based altitude extraction method for single mountain remote sensing image. IEEE Geosci. Remote Sens. Lett. 14(1), 132–136 (2017) 9. Wiskott, L., Sejnowski, T.: Slow feature analysis: Unsupervised learning of invariances. Neural Comput. 14(4), 715–770 (2002) 10. Gu, B., Sheng, V.S., Wang, Z., Ho, D., Osman, S., Li, S.: Incremental learning for v-support vector regression. Neural Netw. 67, 140–150 (2015)

Improved Fully Convolutional Network for the Detection of Built-up Areas in High Resolution SAR Images Ding-Li Gao1,2, Rong Zhang1,2 ✉ , and Di-Xiu Xue1,2 (

)

1

2

Department of Electronic Engineering and Information Science, USTC, Hefei 230027, China {xy6287,xuedixiu}@mail.ustc.edu.cn, [email protected] Key Laboratory of Electromagnetic Space Information, Chinese Academy of Sciences, Hefei 230027, China

Abstract. High resolution synthetic aperture radar (SAR) images have been widely used in urban mapping and planning, and built-up areas in high resolution SAR images are the key point to the urban planning. Because of the high dynamics and multiplicative noise in high resolution SAR images, it is always difficult to detect built-up areas. To address this matter, we put forward an Improved Fully Convolutional Network (FCN) to detect built-up areas in high resolution SAR images. Our improved FCN model adopt a context network in order to expand the receptive fields of feature maps, and it is because that contextual fields of feature maps which are demonstrated plays a critical role in semantic segmenta‐ tion performance. Besides, transfer learning is applied to improve the perform‐ ance of our model because of the limited high resolution SAR images. Experiment results on the TerraSAR-X high resolution images of Beijing areas outperform the traditional methods, Convolutional Neural Networks (CNN) method and original FCN method. Keywords: High resolutions · SAR images · Built-up areas Improved Fully Convolution Networks

1

Introduction

High resolution synthetic aperture radar (SAR) is the only imaging system that can generate high resolution imagery anytime, even in inclement weather or darkness. Thus, SAR images are widely used to observe the land, e.g. disaster management, land cover overlapping, etc. With great impact on human’s life, especially in urban mapping and planning, the detection of built-up areas is more frequently applied to make rational use of the resource in urban areas. Besides, SAR images processing is commonly recognized as a hard task because of the high dynamics and multiplicative noise. As for detecting built-up areas in high resolution SAR images, the feature extraction is the key point to get a great detecting performance. There are many methods proposed to extract different features to detect built-up areas in high resolution SAR images, for instance optical texture features [1] and Labeled Co-occurrence Matrix (LCM [2]) improved from Gray Level Co-occurrence Matrix (GLCM [2]). But using these features © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 611–620, 2017. https://doi.org/10.1007/978-3-319-71598-8_54

612

D.-L. Gao et al.

to detect built-up areas in high resolution SAR images cannot perform well because of the strong speckle noise. In addition, the multiscale CNN method proposed in [13] can extract robust multiscale and hierarchical features directly form images against the strong speckle noise. But the multiscale CNN method [13] cannot form an end-to-end trainable deep network to detect built-up areas in high resolution SAR images, the conversion process of image block labels to pixel labels will limit the detecting perform‐ ance. Meanwhile, the limited size of image blocks result in the limited size of contextual field of feature maps, adversely affecting the detecting performance. In this paper, we utilize the FCN model to form an end-to-end trainable network to detect built-up areas in high resolution SAR images, and the FCN model has evolved to be one of the most important architectures in semantic segmentation after it is success‐ fully adopted in [5]. While using the FCN model to detect built-up areas in high reso‐ lution SAR images, it is a pixel-to-pixel detecting process and we can directly get the label map of high resolution SAR images so that avoid the conversion process of image block labels to pixel labels, and the limited size of contextual field caused by the image blocks, leading to a better detecting performance. Generally, it is difficult to train a strong FCN directly, thus, simply adapts the archi‐ tecture of pre-trained classification network (e.g. VGGNet (VGG-16 [3]) trained on ImageNet [4] etc.) is always a good choice. Although this way can perform well in semantic segmentation, for different resolution data, it has unavoidable architecture defects to limit the performance in semantic segmentation. First and most, the pre-trained CNN is trained with low resolution images (e.g. 224 × 224 pixels), whereas our input SAR images are in high resolution. The simple adaption techniques adopted in FCN model cannot effectively address the domain gap which lead to a less optimized segmen‐ tation performance of FCN [11]. And it is because the feature maps which are used for classification in FCN model have limited contextual fields, causing the inconsistence in predictions for local ambiguous regions. For this problem, we put forward an Improved Fully Convolutional Network to improve the detecting performance, we introduce a context network to replace the fully connected (fc) layers in the original FCN model so that expanding the receptive fields of corresponding feature maps, reducing such a domain gap so that get a better perform‐ ance in semantic segmentation. The context network is stacked with several convolu‐ tional blocks (conv blocks), and we assemble Convolution, Batch Normalization, Recti‐ fied Linear Units (ReLU) as a conv block. Experimental results show that our improved FCN model performances better than traditional methods, the multiscale CNN method [13] and the original FCN method.

2

Fully Convolutional Network

Deep learning model can learn different hierarchies of features, among features, highlevel features is building upon low-level features. Take the classic deep learning model CNN for demonstration, the CNN model can learn different hierarchies of semantic information in images, when trained with regularization, the CNN model can perform well in visual object recognition and image classification. Based on CNN

Improved Fully Convolutional Network

613

model, Long et al. [5] extend the classic deep classification architecture (CNN) by replacing the fully connected layers with fractional convolutional layers to learn perpixel labels end-to-end. Besides, Long et al. [5] describe the fractional convolutional layer as useful learning filters to upsample so that mapping or connecting the coarse outputs to the dense pixel space. Comparing with the CNN model to get semantic information in images, FCN model can form an end-to-end trainable network to learn per-pixel labels rather than per-block labels. Thus, the prediction process of per-pixel labels avoid the conversion of image block labels to image pixel labels so that improve the performance of semantic perform‐ ance. Besides, there is not size limitation in input images when using FCN model while the input images should have the same size if using CNN model. In this paper, we employ the FCN model to detect built-up areas in high resolution SAR images. Empirically, it is hard to train a strong FCN directly. The usual method is to adapt the architecture of pre-trained classification network trained on ImageNet [4]. And for us, we choose the VGG-16 [3] model as our basic model, then we use our dataset of high resolution SAR images to conduct fine-tune process. Figure 1 illustrates the architecture of FCN model (the black vertical line denotes the convolution layer). Each convolutional layer is followed by ReLU activation func‐ tion. Such a high-dimensional model is prone to overfit on the relatively small high resolution SAR images dataset. Under consideration, we take great care to mitigate overfitting through dropout and regularization during training. We plug in a batch normalization layer after every three convolutional layers from the fifth convolutional layer in the FCN model. Besides improving the training speed of the experiment, it can also mitigate overfitting which works similiarly as a regularization way. In addition, we add a dropout layer after the last convolutional layer in the FCN model which can also have an effect on mitigating overfitting.

Fig. 1. Architecture of the FCN model

3

Improved Fully Convolutional Network

The pre-trained CNN in FCN model is optimized for the classification of low-resolution images, leading to a less competitive performance when used for classification or semantic segmentation in high resolution SAR images. It is because that the simple adaptation techniques adopted in FCN model cannot effectively address the domain gap, and the feature maps which used for classification in FCN model have limited contextual fields. Take the well-known FCN model (VGG-16 [3]) as instance, the feature map used for prediction (fc7, c.f. Table 1) don’t have the matching contextual view with high resolution SAR images, the size of corresponding contextual view is only 404 × 404 pixels, and it is not large enough to make sure that nearly all elements in feature maps

614

D.-L. Gao et al.

are accessible to full-image context of high resolution. Meanwhile, the role of contextual filed plays in the final semantic segmentation performance is considered pretty important on the basis of previous literatures [6, 7]. In order to close such a domain gap, we adopt a context network in our FCN model to improve the contextual fields as for the feature maps which used for classification part right after the pre-trained CNN model. Table 1. Receptive fields of feature maps in FCN model (VGG-16 [3]) Layer Receptive field(px) Conv3_1 24 × 24 Pool4 100 × 100

Conv1_1 3×3

Conv1_2 5×5

Pool1 6×6

Conv2_1 10 × 10

Conv2_2 14 × 14

Pool2 16 × 16

Conv3_2 32 × 32 Conv5_1 132 × 132

Conv3_3 40 × 40 Conv5_2 164 × 164

Pool3 44 × 44 Conv5_3 196 × 196

Conv4_1 60 × 60 Pool5 212 × 212

Conv4_2 76 × 76 Fc6 404 × 404

Conv4_3 92 × 92 Fc7 404 × 404

3.1 Context Network According to the principles mentioned above, we attach a context network on top of the pre-trained CNN in order to improve the final performance of detecting built-up areas in high resolution SAR images. And the main effect which context network brought is to enlarge the receptive fields of the feature maps matching with the high resolution SAR images. Besides, our improved FCN model discards the last two fc layers (fc6, fc7) which are specific to image classification [8]. And replace it with our context networks. Through this way, we can efficiently reduce the size of our improved FCN model and gain a larger contextual field for high resolution SAR images. As for the context network, the components consist of the following three structures: 1. Convolution: Conv can efficiently aggregate neighborhood context. From a mathe‐ ∑j=i−m matical point of view y(i) = j=i+m k(i)x(i) where k means the Conv kernel with size 2m + 1, x and y are input and output signals. And it illustrates that signal y can be interpreted as a contextualized x. Meanwhile, Conv has been broadly and success‐ fully devised and used in many literatures [6, 9, 15] to perform context aggregation. 2. Batch Normalization: Batch Normalization [10] is important to accelerate training process and improve the semantic segmentation performance. In the context network, we plug in the batch normalization layer right after the convolution layer. 3. ReLU: ReLU is the activation function we plug in the ReLU layer right after the batch normalization layer. We assemble convolution, batch normalization, ReLU mentioned above as a conv block, and we constitute our context network by stacking multiple conv blocks.

Improved Fully Convolutional Network

615

3.2 Receptive Field We know that the receptive field of convolution layers is related to the size of convo‐ lutional kernel and the stride size of former layers. When calculating the receptive filed of each layers, first, we can get the stride size of each layer, it can be calculated as (1),

stride(t) = stride(1) × stride(2) × … × stride(t − 1)

(1)

stride(t) means the stride size of each layer lt. The way to calculate the size of receptive field is from the deep layer to shallow layer. And it can be calculated as (2) ′ Vreceptivefield =

(( ) ) Vreceptivefield − 1 × stride + Sizeconv

(2)

′ In (2), Vreceptivefield means the receptive field size of the layer lt which we want, and Vreceptivefield means the receptive field size of the layer lt mapped to the previous layer lt−1, stride. means the stride size of each layer, Sizeconv means the size of convolutional kernel. From the calculating process in (2), we can get the size of receptive field which we want layer by layer. According to the calculating process above, we can conveniently enlarge the size of receptive field when stacking multiple conv blocks to form context network with proper number of conv blocks used so that improve the detecting performance efficiently. And the size of receptive field we want will be the key rule to design the proper context network (determine the number of conv blocks used to be stacked with.)

3.3 Architecture of the Improved FCN Model Figure 2 illustrates the overall framework of the improved FCN model employed in this paper. Comparing with the original FCN model, the improved FCN model replaces the fc6, fc7 layer in original FCN model as the context network which is stacked with several conv blocks. As for the high resolution SAR images, the designed context network in the improved FCN model can enlarge the receptive filed to a proper size corresponding to the input images.

Fig. 2. Architecture of the context network, which is composed with a stack of M conv blocks. After each conv block (5 × 5 conv kernel in above example), the receptive fields of feature maps are expanded. Note that the spatial dimensionality of these feature maps keeps unchanged [11].

616

4

D.-L. Gao et al.

Transfer Learning

In this paper, we also explore the benefits of transfer learning in training our FCN model and the improved FCN model with limited high resolution SAR images. In transfer learning, the method adapted to initialize our FCN model and the improved FCN model is sharing the learned weights from the source model. Specifically, the weights selected from convolution and upsampling layers in the source model will be copied or transferred to the corresponding layers in our own model, and the remaining layers of our model are then randomly initialized and trained using the high resolution SAR images dataset via supervised fine-tuning. For instance, we initialize our FCN model and the improved FCN model with the learned weights of selected convolution layers from the VGG-16 [3] model. When training a large target model on the small target dataset, transfer learning can significantly decrease the probability of the occurrence of severe overfitting. Specifi‐ cally, the benefits that transfer learning offered are listed as follows [14]: 1. Domain adaption –It will be allowed that the target task is different but related to the source task in transfer learning. 2. The difference between the distribution of source and target datasets are allowed. 3. It will be achievable to gain better convergence and accuracy performance when trained with limited training data. In addition, when using transfer learning to do the semantic segmentation task, we can do some changes in FCN model for different resolution dataset in order to get a proper size of receptive field. Because it is known that the size of contextual field of feature maps plays a great role in the performance of semantic segmentation.

5

Experiments and Results

In our experiment, we use the high resolution TerraSAR-X SAR images collected on November 25, 2011 of Beijing areas to verify our method. The SAR images have range resolution of 2.3 m and azimuth resolution of 3.3 m. Building types in images include Dot villa district, residential quarter buildings, squatter settlement, etc. And 7900 samples were extracted in eastern areas of Beijing according to the high resolution optical maps to train the FCN model and the improved FCN model. As for the context network in the improved FCN model, notice that the off-centering pixels require significantly larger receptive fields than centering pixels, in order to get the same actual contextual views, the context network is expected to be designed to expand the receptive field twice larger than the input image [11]. And the hidden dimen‐ sion of Conv blocks is fixed to 512. In the training process, we initialized the new parameters engaged from the context work by Gaussian distribution with variance 0.01. And for our FCN model and the improved FCN model. The deconvolution operation [5] is used to perform upsampling operation. Both two models mentioned above are trained with SGD, and the value of momentum is set to 0.9. At the beginning of the training process, the learning rate is initialized to be 0.001, the value will decreased by 10 times

Improved Fully Convolutional Network

617

after 10 epochs (total epochs is 25). Besides, the FCN model and the improved FCN model are trained with the provided segmentation masks, the size of images for the phase of train and validation is 500 × 500 pixels. And the test set is a SAR image of (2500 × 4000) pixels from the northern areas of Beijing, the test image is separate from the train and validation dataset. As for the batch normalization layer, the statistics (mean and variance) is updated after the network is converged. In our experiment, we use detection rate, false alarm rate and accuracy of classifi‐ cation to evaluate our result [12]. GLCM is the detecting result when using GLCM texture features consulting [1, 13], and LCM is the result of [2, 13], Multiscale CNN is the result of [13]. FCN, Improved FCN represent the result using our FCN model and improved FCN model discussed in this paper. From the comparing of results mentioned in Table 2, we can find that our improved model performs better to detect built-up areas in high resolution SAR images. Table 2. Performance comparisons of different methods (corresponding to different models) Method

Detection rate

False alarm rate

GLCM LCM Multiscale CNN FCN Improved FCN

84.38% 89.39% 92.14% 92.20% 92.53%

15.82% 23.40% 10.71% 9.16% 8.63%

Accuracy of classification 88.78% 86.16% 92.86% 92.48% 93.05%

In addition, we design several different context networks with different architectures. And all the context networks designed have the same size of receptive field. Specifically, all the improved FCN model with different context networks have a significantly large receptive field than the origin FCN model, and the size of receptive fields make sure that nearly all elements in feature maps are accessible to full-images context. Table 3. Performance comparisons of improved FCN model with different setup of context networks. The format k × k (m) denotes that the context network is stacked with m consecutive k × k convolution blocks Context network 3 × 3(12) 5 × 5(6) 7 × 7(4) FCN

Receptive Field (px) 980 × 980 980 × 980 980 × 980 404 × 404

Detection rate False alarm rate 92.45% 8.72% 92.53% 8.63% 92.38% 8.59% 92.20% 9.16%

Accuracy of classification 92.95% 93.05% 92.84% 92.48%

The performance in Table 3 clearly shows the context network can improve the performance to detect built-up areas in high resolution SAR images. From Table 3, the format k × k (m) denotes that the context network is stacked with m consecutive k × k convolution blocks. All context networks have the same size receptive field which is carefully designed to make sure that all elements in feature maps are

618

D.-L. Gao et al.

accessible to full –image context. And their detecting performances are very similar, almost the same which illustrates the size of receptive field rather than the architec‐ ture of context network plays a critical role in the final performance when detecting built-up areas in high resolution SAR images. In addition, the new parameters play a great role in filling the domain gap during the fine-tuning of improved FCN. Comparing the magnitude of parameters of improved FCN and the original FCN model, the magnitude of parameters of improved FCN is larger than the original FCN model. It illustrates that the detecting performance improve‐ ment of built-up areas in high resolution SAR images is not simply originate from engaging more parameters. Besides, we conduct a controlled experiment to validate that the larger receptive field rather than the larger magnitude of parameters to improve the detecting performance built-up areas in high resolution SAR images. In order to make the comparsion of context networks with different magnitude of parameters, we design two different context networks, the only differences between them is the hidden dimension. The hidden dimension of context network (*) is increased from 512 to 1024. Thus, the context network with mark (*) involves larger magnitude of parameters. And the result in Table 4 illustrates that the larger magnitude of parameters don’t significantly improve the performance when comparing with the original context network (hidden dimension is 512). Therefore, it can be inferred that the magnitude of parameters doesn’t play a great role in improving the detecting performance of built-up areas in high resolution SAR images. Table 4. Performance comparisons of context networks with different magnitudes of parameters. Context network

Detection rate

False alarm rate

5*5(6) 5*5(6)(*)

92.53% 92.46%

8.63% 8.57%

Accuracy of classification 93.05% 93.13%

Figure 3 shows our detection result on the high resolution SAR images, Fig. 3(a) shows us the origin high resolution SAR image, it illustrates that the size and the type of buildings in this area are complex. Both large span of built-up areas and scattered building appeared in this areas. Figures 3(b) shows the label of built-up areas which is manually labeled according to optical maps, and the regions in red are the built-up areas. Figure 3(c) shows the detecting result with FCN model, the regions in blue are the builtup areas. Figure 3(d) shows the detecting result of built-up areas with the improved FCN model. According to the detecting result (c), (d), we can find that the improved FCN model performs better than the FCN model.

Improved Fully Convolutional Network

619

Fig. 3. Experimental results (a) SAR Image of northern areas of Beijing. (b) Manually Labeled Image. (c) Detection Result of FCN model. (d) Detection Result of improved FCN model.

6

Conclusion

In this paper, we proposed an improved FCN model to solve the problem of built-up areas detection in high resolution SAR images. By the approach of introducing a context network to expand the receptive fields of feature maps in the FCN model, we improve the performance of detecting built-up areas in high resolution SAR images. Experi‐ mental results on TerraSAR-X SAR images get a detection rate of 92.53%, false alarm rate of 8.63%, accuracy of classification of 93.05%. It indicates that the improved FCN model is effective to detect built-up areas in high resolution SAR images. Acknowledgement. This work was supported in part by the National Nature Science Foundation of China (No. 61331020).

References 1. Yang, W., et al.: Supervised land-cover classification of TerraSAR-X imagery over urban areas using extremely randomized clustering forests. In: 2009 Joint Urban Remote Sensing Event (2009). IEEE 2. Li, N., et al.: Labeled co-occurrence matrix for the detection of built-up areas in highresolution SAR images. In: SPIE Remote Sensing. International Society for Optics and Photonics (2013)

620

D.-L. Gao et al.

3. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014) 4. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015) 5. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015). 1, 2, 3, 4, 5, 6, 7, 8 6. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. arXiv preprint arXiv:1606.00915 (2016). 2, 3, 7, 8 7. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated con-volutions. arXiv preprint arXiv:1511.07122 (2015). 2, 3, 7, 8 8. Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Advances in Neural Information Processing Systems, pp. 3320–3328 (2014). 1 9. Lin, G., Shen, C., van dan Hengel, A., Reid, I.: Efficient piecewise training of deep structured models for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016) (2016). 2, 3, 7, 8 10. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML (2015). 5 11. Shuai, B., Liu, T., Wang, G.: Improving fully convolution network for semantic segmentation. arXiv preprint arXiv:1611.08986 (2016) 12. Shufelt, J.A.: Performance evaluation and analysis of monocular building extraction from aerial imagery. IEEE Trans. Pattern Anal. Mach. Intell. 21(4), 311–326 (1999) 13. Li, J., Zhang, R., Li, Y.: Multiscale convolutional neural network for the detection of builtup areas in high-resolution SAR images. In: 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). IEEE (2016) 14. Tran, P.V.: A fully convolutional neural network for cardiac segmentation in short-axis MRI. arXiv preprint arXiv:1604.00494 (2016) 15. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015)

SAR Image Registration with Optimized Feature Descriptor and Reliable Feature Matching Yanzhao Wang1,2(&), Juan Su1, Bichao Zhan2, Bing Li1, and Wei Wu1 1

2

Xi’an High-Tech Research Institute, Xi’an 710025, China [email protected] Beijing Institute of Remote Sensing Equipment, Beijing 100854, China

Abstract. The scale-invariant feature transform (SIFT) algorithm has been widely used in feature-based remote sensing image registration. However, it may be difficult to find sufficient correct matches for SAR image pairs in some cases that exhibit significant intensity difference and geometric distortion. In this letter, a new robust feature descriptor extracted with Sobel operator and improved gradient location orientation hologram (GLOH) feature is introduced to overcome nonlinear difference of image intensity between SAR images. Then, an effective false correspondences removal method by improving the analysis of bivariate histogram is used to refine the initial matches. Finally, a reliable method for affine transformation error analysis of adjacent features is put forward to increase the number of correct matches. The experimental results demonstrate that the proposed method provides better registration performance compared with the standard SIFT algorithm and SAR-SIFT algorithm in terms of number of correct matches, correct match rate and aligning accuracy. Keywords: SAR image registration
Scale-invariant feature transform (SIFT) Gradient location orientation hologram (GLOH)
Bivariate histogram

1 Introduction Image registration is the process of aligning two or more images of the same scene taken by different sensors, at different times, or from different viewpoints, and finding their corresponding geometric relationships [1]. As a coherent imaging system, Synthetic Aperture Radar (SAR) can produce high-resolution images and work under all-time and all-weather conditions [2]. SAR image registration plays an important role in integrated application of SAR data and comprehensive reflection of the geographic information. It is an indispensable procedure for many applications, such as information fusion, change detection and three-dimensional reconstruction. Image registration methods are usually divided into two categories: intensity-based methods and feature-based methods [3]. Intensity-based methods detect no features and the alignment between two images is determined by similarity between pixel intensities. Mutual information and cross correlation are mainly used similarity measures. Due to monotonous textures and complex geometric deformations in SAR images, the © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 621–632, 2017. https://doi.org/10.1007/978-3-319-71598-8_55

622

Y. Wang et al.

intensity-based methods usually suffer from a high degree of computational complexity of global optimization and often cause local extrema when estimating the correspondence between the registered images. Compared with intensity-based methods, feature-based methods extract and match significant features from two images and the correlation between those features is used to determine the alignment. Generally, features extracted include point, edge, and the centroid of a specific area. Feature-based methods are computationally efficient and have strong robustness to radiation difference and noise. Hence, feature-based methods are the mainstream methods for SAR image registration [4]. Among feature-based methods, scale-invariant feature transform (SIFT) [5] is a representative algorithm. It has been widely used in image registration for its invariance to image rotation and scaling and partial invariance to changes in camera viewpoint and illumination [6]. Keypoints detected by SIFT are usually stable and sufficient. Some SIFT-based improvements have been proposed for SAR image registration. Chen et al. [2] proposed a new definition of gradient computation with ROEWA operator and reducing the dimension of feature descriptors, which improved the computational efficiency. Schwind et al. [7] proposed SIFT-OCT, in which the performance of feature detectors is analyzed to improve the robustness of the algorithm. Improved SIFT algorithms based on anisotropic scale space [6, 8] effectively preserve fine details and suppress the speckle noise in SAR images, which improves the correct match rate. All of these improvements are mainly to improve the computational efficiency or reduce the adverse effect of speckle noise. However, the number of correct matches is not sufficient to ensure registration accuracy due to significant nonlinear intensity difference and geometric distortion between SAR images mainly caused by different imaging conditions, such as sensor postures, imaging times, bands, polarizations [9]. Since SIFT does not have complete invariance to intensity and affine changes, performance degradation may happen when SIFT is directly applied to SAR images with poor intensity correlation or large geometric distortion. To overcome this problem, Li et al. [10] proposed R-SIFT that refined the gradient orientation of each pixel, and assigned more main orientations to each key point. In the step of feature matching, Wang et al. [11] analyzed the property of dominant orientation consistency and adopted it to improve the matching stability. In this letter, a novel SAR image registration method based on improved SIFT is proposed. Sobel operator is introduced to redefined the gradient computation, and gradient location orientation hologram (GLOH) is improved to form a new feature descriptor, which overcomes the nonlinear intensity difference between SAR images. Improved bivariate histogram analysis is utilized to eliminate false correspondences from the initial matching results. In addition, a reliable iterative selection method based on affine transformation error analysis of adjacent features is proposed to effectively increase the correct matches. Experiments on various SAR images show the applicability of our method to find sufficient feature matches and obtain accurate registration results.

SAR Image Registration with Optimized Feature Descriptor

623

2 Description of the Proposed Method Traditional SIFT algorithm consists of three major stages. Firstly, Gaussian scale space is constructed by convolving the original image with Gaussian kernel at different scales, and a series of difference of Gaussian (DoG) images are achieved by subtracting adjacent Gaussian images. Extrema of the DoG images are detected as the candidate features. Secondly, dominant orientation of each keypoint is calculated for each keypoint, and a 128-element feature descriptor is constructed based on the gradients in the local image patches aligned by its dominant orientation. Finally, feature points are matched using the nearest neighbor distance ratio (NNDR). More details about SIFT can be found in [5]. Traditional SIFT algorithm usually fails to provide favorable results when directly used to SAR images. Nonlinear intensity difference between the reference image and the same areas in the sensed image results in large gradient differences between correspondences that are expected to be correctly matched, which reduces the robustness and distinctiveness of feature descriptor that is generated by gradient magnitudes and gradient orientations. Meanwhile, local geometric deformation could cause false correspondences that affect the accuracy of transformation parameters. 2.1

New Gradient Computation

The extraction of SIFT feature descriptor depends on gradient histograms around the keypoint location. The premise of correct matching is that descriptors have strong distinctiveness, which means eigenvectors corresponding to different image areas are obviously different while those of the same areas are similar. Significant nonlinear intensity difference leads to different gradient magnitudes and gradient orientations in the same image areas, thus the rotation invariance can’t be ensured and features can’t be accurately matched according to minimum Euclidean distance. In traditional SIFT algorithm, the gradient magnitudes and gradient orientations of keypoints are computed with finite difference operator sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðLðx þ 1; yÞ Lðx 1; yÞÞ2 mðx; yÞ ¼ þ ðLðx; y þ 1Þ Lðx; y 1ÞÞ2

ð1Þ

 ðLðx; y þ 1Þ hðx; yÞ ¼ arctan Lðx þ 1; yÞ

ð2Þ

Lðx; y 1Þ Lðx 1; yÞ



Where mðx; yÞ and hðx; yÞ represent the gradient magnitude and gradient orientation of the keypoint respectively. Finite difference is an efficient operator with good convergence, but fails to provide robustness to nonlinear intensity difference between SAR images. In addition, finite difference could easily be disturbed by speckle noise, which will cause lots of false features to be detected. Sobel [12] is a gradient detection operator commonly used in edge detection. The approximation of the gradient is obtained by convolving the image with a spatial convolution template. Sobel could slightly be affected by intensity difference and it can

624

Y. Wang et al.

smooth speckle noise to a certain extent. So the proposed method adopts Sobel for a computation of the gradient magnitudes and gradient orientations of the keypoints. And the newly defined gradients will be used in the process of orientation assignment and descriptor extraction. Firstly, the gradient magnitude of the Gaussian scale-space image is calculate by Sobel as R1r ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðR1x;r Þ2 þ ðR1y;r Þ2

ð3Þ

Where r is the scale parameter of Gaussian scale space. R1x;r and R1y;r denote the horizontal and vertical derivatives of the Gaussian scale-space image, respectively. Then, the newly defined gradient magnitude and gradient orientation are defined as R2r ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðR2x;r Þ2 þ ðR2y;r Þ2

ð4Þ

  . G2r ¼ arctan R2y;r R2x;r

ð5Þ

Where R2x;r and R2y;r denote the horizontal and vertical derivatives of gradient magnitude image R1r in Gaussian scale space, respectively. All the derivatives denote edge detection along horizontal and vertical directions, which can be calculated by convoluting original image A with horizontal template Bx and vertical template By 2

1 Rx ¼ Bx  A ¼ 4 2 1

2.2

0 0 0

3 þ1 þ 2 5 A; þ1

2

1 Ry ¼ By  A ¼ 4 0 þ1

2 0 þ2

3 1 0 5 A þ1

ð6Þ

Descriptor Extraction with Improved GLOH

In traditional SIFT algorithm, the invariance to image rotation and scaling are achieved by main orientation rotation and descriptor normalization. However, the affine invariance of the descriptor is not completely achieved, which will cause many unstable keypoints and weaken the adaptability to local deformation in SAR images. Gradient location orientation hologram (GLOH) feature [13] is an extension of the SIFT descriptor designed to improve its distinctiveness and robustness. The main idea is to replace the grid pattern of SIFT descriptor with affine-like log-polar concentric circles, and then reduce the dimension of the newly formed descriptor with principal component analysis (PCA) algorithm. GLOH has been proved to have nice affine invariance [13], so we improve it for a more practical version and adopted it to extract a new robust feature descriptor. Main improvements of GLOH and steps for new descriptor extraction are as follows.

SAR Image Registration with Optimized Feature Descriptor

625

(1) Radius adaptively set according to image scale In standard GLOH feature, the radiuses of three concentric circles are empirically set to 6, 11 and 15. Since the size of the image continuously changes with the increase of scale parameter r, it is essential to adaptively change the circular neighborhood to an appropriate size. Inspired by [14], we set the maximum radius R1 of the bins to 12 r, while the second largest radius and that of the center circle are set to 0.73 R1 and 0.25 R1 , respectively. (2) Dimensionality reduction with gradient orientations re-quantified Reducing the dimension of feature descriptor with PCA requires lots of sample training in advance, and the process of dimensionality reduction is time consuming. So we quantize the gradient orientations in 8 bins instead of 16 in all 17 location bins, and the dimension of the descriptor eigenvector is reduced from 272 to 136. Facts have proved that the efficiency of the algorithm is greatly improved with the robustness and effectiveness of the mew descriptor well maintained.

Fig. 1. SIFT descriptor extraction

Fig. 2. Improved GLOH descriptor

Note that the gradients computed by (4) and (5) are used in the process of orientation assignment and descriptor extraction. Figures 1 and 2 denote SIFT feature descriptor and our improved GLOH descriptor with 8 gradient orientations in each location bin. The difference between these two descriptors can be intuitively observed and compared. 2.3

Removal of False Matches Using Optimized Bivariate Histogram

When there are repeated patterns in the image, many keypoints in the reference image are matched to the same one in the sensed image using SIFT matching strategy (distance ratio). Therefore, we use dual-matching strategy [6] to find initial correspondences,

626

Y. Wang et al.

which could improve the rate of correct matching. Fast sample consensus (FSC) algorithm [15] is adopted to refine the matching results and calculate the values of transformation model parameters. FSC is an optimization for the classical random sample consensus (RANSAC) algorithm [16], which could obtain more correct matches with a less number of iterations. But like RANSAC, if the number of outlier is higher in the input matching set, then the FSC algorithm fails to provide sufficient correct matches within a specific number of iterations. Even after the identification of matching candidates using dual-matching strategy, SAR images could still produce unreliable tie points which will lead to many false matches. Due to the geometric deformation and randomly distributed speckle noise in SAR images, most of the false matches are randomly distributed, while the “correct set” of matching keypoints corresponds to a denser region of the 2-D representation of the horizontal (Dx) and vertical (Dy) distances between the matches. Based on this principle, we have improved the bivariate histogram [17] to further exclude false matches before using them as an input set for FSC. The bivariate histogram is formed based on the scatter plot of a set of horizontal (Dx) and vertical (Dy) distances between the matches. The number of bins B in the bivariate histogram is computed as 1 þ 3:322 log10 P, where P is the number of matched features. The correct matches could produce a dominant peak in a particular bin of the histogram, and features corresponding to that bin are the refined matches. False matches are iteratively excluded when matching candidates belong to a histogram bin with an absolute frequency less than 10% of the maximum absolute frequency. Note that in [17], the iterative procedure stops when the registration accuracy measure RMSE is blow a threshold e or the maximum number of iterations Q is achieved. However, we find that the procedure usually suffers from a high degree of computational complexity, which could reduce the overall efficiency of feature matching. Fortunately, experimental results show that a maximum number of eight iterations (Q0  8) are sufficient for the procedure to converge to a subset of correct matches. So the number of iterations is set as the termination condition of the procedure in our improved bivariate histogram analysis. 2.4

Increasing the Number of Correct Matches

Affine transformation is selected as a transformation model in our method. Let the four parameters of the affine transformation be represented in a vector form w ¼ ½s; h; tx ; ty ŠT , where s is the scale; h is the rotation; tx and ty represent the translation parameters, respectively. The values of the transformation parameters w can be preliminarily calculated by FSC after removal of false matches with optimized bivariate histogram. A lot of methods have been proposed to reduce the negative effect of false matches on matching accuracy, but few methods aim at increasing correct matches. In fact, more correct matching points will obtain more accurate transformation parameters, and further a more accurate registration result. In traditional SIFT algorithm, only the nearest neighbor of one feature point is considered as the best matching candidate of it, but complex geometric differences between SAR images may cause a slight displacing of some feature points from their

SAR Image Registration with Optimized Feature Descriptor

627

true positions. In this case, many correct matches may not be found since their correct correspondence points are not their nearest neighbors. So a correct correspondence cannot be found by considering only the first nearest neighbor. In this letter, we propose a novel method, called transformation error analysis of adjacent features, to increase the number of correct matches. We consider that the correct correspondence of one keypoint may be one of its first four nearest neighbors. These nearest neighbors are iteratively judged with the affine transformation parameters w obtained above in the order of the distance of these two points, from the nearest to the farthest. Main steps of the process are as follows.    1  pn represent the feature points detected from p ; . . .; ^ Step 1: Let p1 ;


pm and ^ i the reference and the sensed image, respectively.  1p represents  an element  1 imagem  from set p ;


p . Find four nearest neighbors ^pi ; ^p2i ; ^p3i ; ^p4i of pi form  1  ^ pn . These neighbors are arrayed on the distance between the SIFT feap ; . . .; ^  1 2 3 4 pi and pi , ^ p1i is the nearest neighbor of pi , and ^p4i is the farthest pi ; ^ pi ; ^ pi ; ^ tures of ^ one.     Step 2: Calculate the transformation errors e1 ; e2 ; e3 and e4 of pi ; ^p1i , pi ; ^p2i  i 4  i 3 p ;^ pi , respectively, using the values of transformation parameters w. pi and p ; ^    

pi is denoted as e pi ; ^pi ; w ¼ kð^xi ; ^yi Þ The transformation error of pi ; ^ Tððxi ; yi Þ; wÞk:. Step 3: In order of e1 to e4 , judge one after another if one of them is less than the error threshold e (1 pixel). If one is less than e, consider its corresponding feature and pi as a correct match, and the process of judgment stops, otherwise the next transformation error will be checked. If all of these four transformation errors (e1 ; e2 ; e3 and e4 ) fail to satisfy the 1 pixel transformation error criterion, it is decided that the feature pi has no correct match.   Step 4: Check all elements in set p1 ;


pm if they have a corresponding correct  1  pn according to the procedure described from Step 1 to Step 3, match in ^ p ; . . .; ^ and the final feature matches could be confirmed. Note that the final feature matches obtained above will be calculated for new affine transformation parameters, which may greatly improve the registration accuracy.

3 Experimental Results and Analysis To verify the adaptability to intensity difference and the overall registration performance of the proposed method, two pairs of SAR images with significant intensity difference are tested. These image pairs are shown in Figs. 3 and 4 respectively. The first pair is two multi-band images from the same region. One image with a size of 500  500 form C-band taken by Radarsat-2 is selected as the reference image, and the other one with a size of 375  375 from X-band taken by TerraSAR is selected as the sensed image. There are obvious scale difference and geometric distortion between these two images. The second pair is two 512  512 multi-polarization images with the reference image obtained from the HV mode, and the sensed image of the same scene

628

Y. Wang et al.

obtained from the VV mode. To increase the difficulty of the test, the sensed image is simulated with a rotation of 30 .

Fig. 3. The first image pair

3.1

Fig. 4. The second image pair

Intensity Difference Adaptability Verification

Much in the spirit of a Hough-like voting scheme, a procedure for registration of remotely sensed images named Mode Seeking (MS) was proposed [18]. The basic idea of MS is that the histograms of the scale ratios, main orientation difference, horizontal shifts, and vertical shifts of the correct matches should exhibit a single mode (i.e. the proportion of one column in the histogram is much larger than those of other columns). In this section, the standard SIFT algorithm and our method are used to extract feature points from the first image pair, respectively. Then features are coarsely matched with distance ratio ðdratio ¼ 0:8Þ, and the histograms of the scale ratios Sref =Ssensed , main orientation difference Dh, horizontal shifts DX, and vertical shifts DY of the matched feature points are calculated (Figs. 5 and 6). The adaptability to intensity difference of the improved gradient computation and new descriptor extraction is verified based on modes of the histograms.

Fig. 5. Histograms obtained by SIFT

SAR Image Registration with Optimized Feature Descriptor

629

It can be seen that the four histograms obtained by the standard SIFT can’t exhibit a single mode, which, due to the intensity difference between SAR images. Such intensity difference will result in different main orientation difference of correspondences that are expected to be correctly matched. The rotation invariance of feature descriptor can’t be ensured and in this case, features could hardly be correctly matched according to the minimum Euclidean distance between the descriptors that are formed by gradient orientations and gradient magnitudes. In contrast, all histograms obtained by our proposed method exhibit a single mode. The results show that the improved gradient computation and descriptor extraction enhance the robustness of the algorithm to nonlinear intensity difference, and improve the rate of correct matching. 20

proportion /%

proportion /%

15

10

5

0 0

1 S

ref

2

3

5 0 -400

-200

sensed

0 200 ∆θ /degree

400

(b) main orientation difference

(a) scale ratios

20

proportion /%

proportion /%

10

/S

20 15 10 5 0 -500

15

0 ∆X /pixels

(c) horizontal shifts

500

15 10 5 0 -400

-200

0 200 ∆Y /pixels

400

(d) vertical shifts

Fig. 6. Histograms obtained by proposed method

3.2

SAR Image Registration Performance Verification

To evaluate the registration performance of the proposed method, we experimentally validate it on two SAR image pairs (Figs. 3 and 4). Registration comparisons between the proposed method and the traditional SIFT and SAR-SIFT [14] are implemented to demonstrate the superiority of our method in registration performance. SAR-SIFT algorithm is specifically designed for speckle noise in SAR images, which can suppress the influence of noise on feature detection and descriptor extraction. In our experiments, the threshold for dual-matching is set to 0.8. The number of correct matches, the correct match rate ðCMRÞ and the root mean squared error ðRMSEÞ are adopted to quantitatively evaluate the registration performances. Figures 7 and 8 show feature matches of the three methods and registration results of our proposed method, and the quantitative evaluation results for each method are listed in Table 1.

630

Y. Wang et al.

Fig. 7. Registration result of image pair 1

It can be seen that, of the three methods used for comparison, our proposed method gives the best performance. Compared with the standard SIFT, our approach not only obtains more number of correct matches but also has a much higher matching accuracy. Due to the use of Sobel operator and the improved GLOH, the robustness to intensity difference and the affine invariance of the new feature descriptor outperform that of the standard SIFT. The consistency of the scale, orientation and location of the correspondences is enhanced, which attributes to obtaining more feature matches. Meanwhile, bivariate histogram is improved for a more practical use to eliminate the mismatches, and more features are checked for an accurate correspondence, which improves the correct match rate and the final registration precision.

Fig. 8. Registration result of image pair 2

Although SAR-SIFT obtains more correct matches than the standard SIFT in both experiments and even more than those of our approach in image pair 2, it would lead to relatively little CMR and large RMSE, which is an unsatisfactory result. The main

SAR Image Registration with Optimized Feature Descriptor

631

reason is that SAR-SIFT uses ROEWA operator to calculate the gradient orientation of the keypiont, which is mainly applied to images with more speckle noise. That is to say, it would have limited effect on images with nonlinear intensity difference.

Table 1. Quantitative comparison of the proposed method with SIFT and SAR-SIFT. SAR images Method

CMR RMSE

Dataset1

0.79 0.89 0.93 0.82 0.90 0.94

Dataset2

Number of correct matches SIFT 13 SAR-SIFT 24 Proposed method 31 SIFT 11 SAR-SIFT 23 Proposed method 20

3.09 1.06 0.91 2.83 0.97 0.85

In addition, our approach inherits the superiority of the standard SIFT. An accurate registration result can still be achieved under conditions of scale and rotation changes.

4 Conclusion In this letter, a novel SAR registration method based on improved SIFT is proposed. Sobel operator is used for gradient computation of the keypoints and GLOH is improved for feature descriptor extraction, which makes the algorithm more robust to complex nonlinear intensity difference between the images. Bivariate histogram is improved for eliminating mismatches after dual-matching, improving the robustness of the algorithm to geometric changes. In addition, a reliable method based on transformation error analysis of adjacent features is proposed to increase the number of correct matches. Experimental results on SAR images show that our method reveals better performance than the state-of-the-art methods in terms of registration accuracy in some cases.

References 1. Zitova, B., Flusser, J.: Image registration methods: a survey. Image Vis. Comput. 21(11), 977–1000 (2003) 2. Chen, Y., Zhao, H.C., Chen, S., et al.: Image matching algorithm based on SIFT for missile-borne SAR. Syst. Eng. Electron. 38(6), 1276–1280 (2016) 3. Salvi, J., Matabosch, C., Fofi, D., et al.: A review of recent range image registration methods with accuracy evaluation. Image Vis. Comput. 25(5), 578–596 (2007) 4. Yang, L., Tian, Z., Zhao, W., et al.: Robust image registration using adaptive coherent point drift method. J. Appl. Remote Sens. 10(2), 025014 (2016) 5. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

632

Y. Wang et al.

6. Wang, S.H., You, H.J., Fu, K.: BFSIFT: a novel method to find feature matches for SAR image registration. IEEE Geosci. Remote Sens. Lett. 9(4), 649–653 (2012) 7. Schwind, P., Suri, S., Reinartz, P., et al.: Applicability of the SIFT operator to geometric SAR image registration. Int. J. Remote Sens. 31(8), 1959–1980 (2010) 8. Fan, J.W., Wu, Y., Wang, F., et al.: SAR image registration using phase congruency and nonlinear diffusion-based SIFT. IEEE Geosci. Remote Sens. Lett. 12(3), 562–566 (2015) 9. Chen, T.Z., Li, Y.: A high performance edge point feature match method of SAR images. Acta Autom. Sinica 39(12), 2051–2063 (2013) 10. Li, Q., Wang, G., Liu, J., et al.: Robust scale-invariant feature matching for remote sensing image registration. IEEE Geosci. Remote Sens. Lett. 6(2), 287–291 (2009) 11. Wang, F., You, H.J., Fu, X.Y.: Adapted anisotropic gaussian SIFT matching strategy for SAR registration. IEEE Geosci. Remote Sens. Lett. 12(1), 160–164 (2015) 12. Yang, Q., Xiao, X.: A new registration method base on improved Sobel and SIFT algorithms. In: Proceeding of the 3rd International Conference on Computer and Electronic Engineering, p. 528 (2010) 13. Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1615–1630 (2005) 14. Dellinger, F., Delon, J., Gousseau, Y., et al.: SAR-SIFT: a SIFT-like algorithm for SAR images. IEEE Trans. Geosci. Remote Sens. 53(1), 453–466 (2015) 15. Wu, Y., Ma, W.P., Gong, M.G., et al.: A novel point-matching algorithm based on fast sample consensus for image registration. IEEE Geosci. Remote Sens. Lett. 12(1), 43–47 (2015) 16. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981) 17. Gonalves, H., Real, L.C., Gonalves, J.A.: Automatic image registration through image segmentation and SIFT. IEEE Trans. Geosci. Remote Sens. 49(7), 2589–2600 (2011) 18. Kupfer, B., Netanyahu, N.S., Shimshoni, I.: An efficient SIFT-based mode-seeking algorithm for sub-pixel registration of remotely sensed images. IEEE Geosci. Remote Sens. Lett. 12(2), 379–383 (2015)

An Improved Feature Selection Method for Target Discrimination in SAR Images Yanyan Li1 ✉ (

)

and Aihua Cai2 ✉ (

)

1

Research Institute of Electronic Science and Technology, University of Electronic Science and Technology of China, Chengdu, China [email protected] 2 China Electronics Technology Group Corporation, Beijing, China

Abstract. Due to synthetic aperture radar (SAR) imaging principals, at partic‐ ular azimuth or depression angle, targets and clutters may become very hard to distinguish, to solve this problem, many complicated features have been developed, this is not only a tough work, but little improvement of discrimi‐ nation accuracy is obtained. In this paper, an improved target discrimination method is proposed, one-class quadratic discriminator (OCQD) has been used, compared with traditional method using Bayes discriminator, when number of features is limited, our new method has higher target classification correction than old methods, considering that target classification correction is more important than clutter classification correction, our proposed method has a good performance on target discrimination. First, discrimination scheme based on genetic algorithm (GA) is introduced. Second, feature extraction algorithms of SAR images have been introduced. Third, an improved feature selection method based on GA has been proposed, in which the OCQD has been used and a new fitness function has been designed. Finally, the theory of OCQD algo‐ rithm is explained. According to the experiment result based on moving and stationary target acquirement and recognition (MSTAR) database, our new method reduces target undetected rate by 1.5% compared to the state-of-the-art methods in target discrimination, besides, the efficiency of feature selection based on GA has been improved by 77%. Keywords: Synthetic Aperture Radar (SAR) · Target discrimination Feature extraction · Feature selection · Genetic Algorithm (GA)

1

Introduction

Over past two decades, as the speed of synthetic aperture radar (SAR) imaging grow rapidly, knowledge based real-time interpretation of SAR image became harder and harder, development of SAR-ATR (automatic target recognition) system has become a hot area. A typical synthetic aperture radar (SAR) automatic target recognition (ATR) system consists of three stages: detection, discrimination and recognition [1–3]. • Detection stage: locate regions of interest (ROIs), ROI chips may contain targets are sent to the second stage. © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 633–645, 2017. https://doi.org/10.1007/978-3-319-71598-8_56

634

Y. Li and A. Cai

• Discrimination stage: As show in Fig. 1, the purpose is to discriminate targets from clutters, to reduce cost of recognition, removing as many natural clutters and manmade clutters as possible, ROIs contain targets are sent to the third stage.

(a)

(b)

(c)

Fig. 1. ROIs. (a) natural clutter; (b) man-made clutter; (c) target

• Recognition stage: fetch detailed information from target chips. Target discrimination is the second stage in SAR-ATR system, its performance has a great influence on the whole performance of the system [4, 5]. Feature based scheme is widely used in SAR target discrimination [6, 7], which consists of training stage and testing stage, as show in Fig. 2. Feature Selection Based on GA

Feature Extraction Extracting 1st Feature

The GA

feedback

Training Images Extracting nst Feature

The Optimal Feature Subset

The OCQD

(a)

Test Images

Extracting the Corresponding Optimal Feature Set

The Optimal The OCQD Feature Set

(b) Fig. 2. Discrimination scheme based on GA

The Result of Discrimination

An Improved Feature Selection Method for Target Discrimination

635

1.1 Training Stage Some images are used as training set, while others as testing set. At first, some features are extracted from training images, then Genetic Algorithm (GA) is used to select the optimal combination of these features (one-class quadratic discriminator is used to judge whether the feature subset is good or not), as show in Fig. 2(a). 1.2 Testing Stage As show in Fig. 2(b), corresponding optimal feature set we obtained at training stage are extracted from testing images, then one-class quadratic discriminator (OCQD) is used to discriminate targets from clutters.

2

Extraction of Features

Seven features including three pixel level features: area feature, peak power ratio feature and fractal dimension feature, and four spatial edge property features, are extracted to distinguish targets from clutters.

(

)

2.1 Area Feature f 1 = A

Compute the threshold by using CFAR algorithm, area feature A is the number of pixels above this threshold. 2.2 PPR Feature

( ) f 2 = PPR

Peak power ratio (PPR) is the ratio of a% brightest pixels’ power of the image to the total power of the image.

(

)

2.3 Fractal Dimension Feature f 3 = FD Fractal dimension feature defined as [8]: FD =

log N1 − log N2 log d2 − log d1

(1)

N1 is the minimal number of 1 × 1 boxes that can cover the brightest pixels in the image, N2 is the minimal number of 2 × 2 boxes that can cover the brightest pixels in the image. d1 = 1 and d2 = 2 represents the side length of box.

636

Y. Li and A. Cai

(

)

2.4 Spatial Edge Property Features f � − f � [9] Assume the N × M image I changes with a parameter t, then the image is I(t), the pixel strength is Iij(t). Do logarithmic operation on image I, every pixel deducts the mid-value of the image Ī to get standard image, denoted as: xij = Iij − Ī

(2)

Using sigmoidal function below to shift standard images:

s(x;t) =

1 , −∞ < t < +∞ 1 + exp(−c(x − t))

(3)

In the expression, c is a positive real number. Then we get a bunch of new images w(t), each value of pixel in w(t) expresses as: ( ) wij (t) = s xij ;t

(4)

A measurement on dispersion degree of image pixel quality to image centroid is:

r(t) =

√( )2 ( )2 i − ic (t) + j − jc (t) w (t) j=1 ij ∑ N ∑M w (t) i=1 j=1 ij

∑N ∑M i=1

(5)

In this formula, ic and jc are center-of-mass coordinates of images. Obtain the best threshold t0 when r(t) changes the most:

{ } t0 = argmax ||r′ (t)||

(6)

As t0 is obtained, four spatial edge property features can be calculated as: The average value of pixel mass in the image: f4 = log

∑N ∑M i=1

j=1

( ) wij t0

(7)

MN

The spatial scatter extension feature of pixels:

( ) f 5 = r t0

(8)

( ) f6 = r′ t0

(9)

( ) f7 = r′′′ t0

(10)

Inflexion point:

Acceleration:

An Improved Feature Selection Method for Target Discrimination

3

637

Improved Feature Selection Method Based on GA

GA is an intelligent algorithm that can search best combination of features, it is widely used in object detection [8, 10–12]. As show in Fig. 3, feature selection based on GA mainly contains five problems: chromosome encoding; initialization of population; design of fitness function; selection of genetic operators; Updating population.

Initialization Population

Yes

Mutation Rate higher than 0.09 or Generation is higher than 50? Computing Fitness of each individual Best Fitness remains unchanged for 3 generations? Yes Increasing Mutation Rate by 0.02

No

Selection, Crossover, Mutation Updating the population Resetting Mutation Rate as 0.01 Return the Optimal Individual Fig. 3. Feature selection based on genetic algorithm

3.1 Chromosome Encoding A chromosome contains seven binary genes, stands for seven corresponding features, if a gene is 1, then corresponding feature is selected, otherwise, the feature is not selected. Figure 4 is an example of how a chromosome is encoded.

638

Y. Li and A. Cai Feature Index

1

2

3

4

5

6

7

Chromosome

1

0

1

0

0

1

1

Fig. 4. Feature subsequence

3.2 Initializing Population To ensure the diversity of genes, we randomly generate 100 chromosomes as initial population. 3.3 Design of Fitness Function Every chromosome has a fitness value calculated by fitness function, the function should reveal our purpose, which is the combination of selected features has lowest classifica‐ tion error rate and computation complexity. We define fitness function as: F(f ) = −(k log10 (l) + nf log10 (nc ) − q ∗ dmax )

(11)

In the function, k is the number of selected features, l is the number of candidate features; nf is the number of classification errors that classify clutters into target class, we use OCQD to compute nf , nc is the number of clutters in clutter class; dmax is the maximum distance between targets and target class, q is a positive real number, we set q as 0.03 in our experiment. 3.4 Genetic Operators Selection We choose three algorithms to search the best individual in population. Selection algo‐ rithm find top 20 individuals (unselected individuals set ones) and deliver them to the cross algorithm; set cross rate as 0.8, when it occurs, adjacent individuals change their genes randomly; set initial mutation rate as 0.01, the rate increases by 0.02 as fitness value of best individual does not improve, when mutation occurs, choose and reverse a gene in the individual randomly. 3.5 Updating Population We update the population after three operators if the largest fitness value of new popu‐ lation is larger than the largest fitness value of the old population, repeating the loop until the mutation rate is higher than 0.09. Then we obtain selected features, as well as mean vector, covariance matrix and dmax of training target class.

An Improved Feature Selection Method for Target Discrimination

4

639

One Class Quadratic Discriminator

Unlike Bayes discriminator [8, 10], we only describe the similarity degree of a ROI to target class, when the degree is under a specific threshold, the ROI is considered as a target, otherwise as a clutter. Let the target class be C1, clutter class be C2, and the feature subsequence obtained by training data of Ci (i = 1, 2) be fi1, fi2 , … , fiMT (MT is the number of training images of Ci). The mean and covariance matrix of target class C1 can be calculated as following: 1 ∑M T f j=1 1j MT

(12)

)( )T 1 ∑M T ( f 1j − �1 f 1j − �1 j=1 MT

(13)

�1 = �1 =

Let the extracted feature vector of a training image be � . The quadratic distance of � to class C1 is:

( )T ( ) di = � − �1 � −1 � − �1 1

(14)

4.1 Application of OCQD in Training Stage Firstly, computing d1 which is the quadratic distance that target image to C1, set the maximal d1 as dmax, secondly, we calculate quadratic distance d2 that clutter image to C1, nf is the number of clutter images that quadratic distance less than dmax, which can express as: nf = |{i|d2 < dmax , 1 ≤ i ≤ m}|

(15)

Where m is the number of clutter images in training database. According to Eq. (11), fitness value of particular feature subsequence can be obtained. 4.2 Application of OCQD in Testing Stage Mean vector, covariance matrix and dmax of class C1 corresponding to optimal feature subsequence can be obtained at training stage, quadratic distance of every testing image to class C1 can be easily calculated, we consider an image whose quadratic distance smaller than dmax as target, while larger than dmax as clutter.

640

5

Y. Li and A. Cai

Experiments

5.1 Introduction of MSTAR Database Resolution of MSTAR database is 0.3 × 0.3 m, HH polarization mode, X-band, under 15 degree depression angle and the distance is about 5 km. Target class contains 8 kind of vehicles, including 2S1, T62, T72, ZSU tanks; BRDM2, BTR60, ZIL armored vehicles; D7 bulldozer. Each type of target distributed in the entire 360 degree azimuth coverage, and image size is 128 × 128 pixels, we choose 75 targets covering 360 degree azimuth from each type of target class (total target number is 600), randomly choosing 200 target chips as training data and 400 chips as testing data. Clutter class contains 100 clutters, the total clutter coverage is about 10 km2, including rural areas and urban areas, using CFAR detector based on Rayleigh distribution [13, 14], set false alarm rate as 0.001, we obtain 789 clutter chips in the size of 128 × 128 pixels, randomly choosing 292 chips as training data and 497 chips as testing data. 5.2 Experiment Results Experiment Condition Experiment condition is Core2 CPU, 2G memory, with the use of Matlab7.10.0. The average time of extracting all features is less than 0.5 s for a 128 × 128 image. Feature Selection Based on GA At training stage, we extract seven features for each training chips, run GA 10 times by using new method and traditional method [8, 10], respectively. As show in Tables 1 and 2, we obtain the best combination of seven features. The average efficiency of feature selec‐ tion based on GA is improved by 77%. Table 1. Feature selection based on GA using bayes discriminator Loop 1 2 3 4 5 6 7 8 9 10 Average

Optimal subset 1100101 1100101 1100101 1100101 1100101 1100101 1100101 1100101 1100101 1100101 –

Fitness value −8.3734 −8.3734 −8.3734 −8.3734 −8.3734 −8.3734 −8.3734 −8.3734 −8.3734 −8.3734 –

Time (s) 28.8303 29.3761 27.1241 27.3363 27.3347 27.0968 28.2719 28.9134 24.8204 26.5234 27.56274

An Improved Feature Selection Method for Target Discrimination Table 2. Feature selection based on GA using OCQD discriminator Loop 1 2 3 4 5 6 7 8 9 10 Average

Optimal subset 1000011 1000011 1000011 1000011 1000011 1000011 1000011 1000011 1000011 1000011 –

Fitness value −1.7288 −1.7288 −1.7288 −1.7288 −1.7288 −1.7288 −1.7288 −1.7288 −1.7288 −1.7288 –

Time (s) 17.551 14.9377 16.2526 16.2625 15.0132 14.8707 15.3847 15.137 15.5217 14.7814 15.57125

Fig. 5. Quadratic distance distribution

641

642

Y. Li and A. Cai

Quadratic Distance Distribution Graphs As optimal feature subsequence is obtained, OCQD is used to calculate the quadratic distance of each image to class C1, to see classification result clearly, define new quad‐ ratic distance as Eq. (16), distribution of dnew is shown in Fig. 5. In distribution graphs, targets above 0-axis and clutters below 0-axis are considered as correct classification.

dnew = dmax − di

(16)

As show in Fig. 5, (a) is optimal subsequence used on training database, all images are classified correctly; (b) is optimal subsequence used on testing database, all targets are classified correctly, while 4 clutters are classified incorrectly; (c) is seven features used on training database, all images are classified correctly; (d) is seven features used on testing database, 1 target and 2 clutters are classified incorrectly. From the distribution graphs we can see, the proposed method have a good perform‐ ance on feature selection and target discrimination (target classification correction is more important than clutter classification correction). Comparison of Discrimination Performance At testing stage, optimal subsequence of features is extracted. Comparison with tradi‐ tional method [8, 10] and proposed method on discrimination rate (DR) (DR means the rate that classify images to the right class) is showed in Fig. 6. From the bar graph we can see:

Bar Graph of Discrimination Rate 100.0%

Detection Rate

99.5%

Old Method (seven features)

99.0%

Old Method (optimal features)

98.5% 98.0%

Our Method (seven features)

97.5% 97.0%

Our Method (optimal features)

96.5% 96.0% Target Clutter Total DR Target Clutter Total DR DR (testing DR DR (training DR (training (training data) (testing (testing data) data) data) data) data)

Data Type Fig. 6. Discrimination rate (Our Method vs. Old Method)

An Improved Feature Selection Method for Target Discrimination

643

1. Using optimal feature subsequence selected by GA either by new method or by traditional method to discriminate, we can achieve higher target discrimination rate than using all seven features. 2. Compared to traditional method, our method has higher target DR (100% to 99.5% on training data, 100% to 98.5% on testing data) and total DR (100% to 99.78% on training data, 99.55% to 99% on testing data). 3. Proposed method has lower clutter DR on testing data (99.4% to 99.2%).

(a)

(b)

(c)

(d)

Fig. 7. Target discrimination results on embedded SAR images (Color figure online)

644

Y. Li and A. Cai

Visualized Discrimination Result As the imagery of clutter images and targets are under the same condition, we choose 8 types of targets randomly and embed them into clutter images, smooth edge of targets and clutter image, set false alarm rate as 0.001, CFAR detector based on Rayleigh distribution [13, 14] is used to achieve chips of target and clutter. Discrimination performance of proposed method and old method are similar until clutter-like target is chosen. As show in Fig. 7, (a) is embedded SAR image; (b) is the result of CFAR detector, 8 target chips and 29 clutter chips are obtained (object in red rectangle is target, object in green rectangle is clutter); (c) is the discrimination result of old method, 7 targets are correct classified, 1 target is not detected, all of clutters are classified correctly; (d) is the discrimination result of proposed method, as we can see, 8 targets are classified correctly, while 2 clutters are classified incorrectly.

6

Conclusion

In this paper, an improved method on feature selection based on GA is proposed, oneclass quadratic discriminator (OCQD) is used in this method. By using limited number of regular features, we can achieve better performance than traditional methods. Our experiment is based on MSTAR database. At first, 600 target chips and 789 clutter chips in the size of 128 × 128 pixels are obtained by using CFAR detector, then 200 target chips and 292 clutter chips are chosen as training data randomly, the rest as testing data. Compared to traditional methods [8, 10], our method has higher target discrimination rate (100% to 99.5% and 100% to 98.5% on training data and testing data, respectively), target discrimination experiment on embedded SAR images also show this character‐ istic, meanwhile, proposed method improves efficiency of feature selection based on GA by about 77%. But new method shows slight inadequate capacity of clutter discrim‐ ination (99.4% by old method and 99.2% by proposed method on testing data). Consid‐ ering that cost brought about by target classification error is more serious than that brought about by clutter classification error, our new method has better performance than traditional methods on target discrimination.

References 1. Dudgeon, D.E., Lacoss, R.T.: An overview of automatic target recognition. Linc. Lab. J. 6(1), 3–10 (1993) 2. Bhanu, B., Dudgeon, D.E., Zelnio, E.G., Rosenfeld, A., Casasent, D., Reed, I.S.: Guest editorial introduction to the special issue on automatic target detection and recognition. IEEE Trans. Image Process. 6(1), 1–6 (1997) 3. Oliver, C.J., Quegan, S.: Understanding Synthetic Aperture Radar Images. Artech House, Boston (1998) 4. Novak, L.M., Owirka, G.J., Brower, W.S., Weaver, A.L.: The automatic target-recognition system in SAIP. Linc. Lab. J. 10(2), 187–202 (1997) 5. Kreithen, D.E., Novak, L.M.: Discriminating targets from clutter. Linc. Lab. J. 6(1), 25–51 (1993)

An Improved Feature Selection Method for Target Discrimination

645

6. Blacknell, D.: Adaptive design and characterisation of features for SAR ATR. In: Proceedings of the SPIE Conference on Algorithms SAR Imagery IV, vol. 4382, pp. 252–263 (2001) 7. Krawiec, K., Bhanu, B.: Visual learning by coevolutionary feature synthesis. IEEE Trans. Syst. Man Cybern. B Cybern. 35(3), 409–425 (2005) 8. Bhanu, B., Lin, Y.: Genetic algorithm based feature selection for target detection in SAR images. Image Vis. Comput. 21, 591–608 (2003) 9. Verbout, S.M., Novak, L.M.: New image features for discriminating targets from clutter. In: Proceedings of SPIE Conference on Algorithms SAR Imagery IV, vol. 3395, pp. 120–137 (1998) 10. Gao, G.: An improved scheme for target discrimination in high-resolution SAR images. IEEE Trans. Geosci. Remote Sens. 49(1), 277–294 (2011) 11. Lin, Y., Bhanu, B.: Evolutionary feature synthesis for object recognition. IEEE Trans. Syst. Man Cybern. C: Appl. Rev. 35(2), 156–171 (2005) 12. Bhanu, B., Lin, Y.: Object detection in multi-modal images using genetic programming. Appl. Soft Comput. 4(2), 175–201 (2004) 13. Gao, G., Kuang, G., Zhang, Q., Li, D.: Fast detecting and locating groups of targets in highresolution SAR images. Pattern Recogn. 40(4), 1378–1384 (2007) 14. Greco, M.S., Gini, F.: Statistical analysis of high-resolution SAR ground clutter data. IEEE Trans. Geosci. Remote Sens. 45(3), 566–575 (2007)

The Detection of Built-up Areas in High-Resolution SAR Images Based on Deep Neural Networks Yunfei Wu1,2, Rong Zhang1,2 ✉ , and Yue Li1,2 (

1

)

Department of Electronic Engineering and Information Science, USTC, Hefei 230027, China {wuyunfei,lyue}@mail.ustc.edu.cn, [email protected] 2 Key Laboratory of Electromagnetic Space Information, Chinese Academy of Sciences, Hefei 230027, China

Abstract. The detection of built-up areas is an important task for high-resolution Synthetic Aperture Radar (SAR) applications, such as urban planning and envi‐ ronment evaluation. In this paper, we proposed a deep neural network based on convolutional neural networks for the detection of built-up areas in SAR images. Since lables of neighboring pixels have strong correlation in SAR images, infor‐ mations on labels of neighboring pixels could help making better prediction. In addition, built-up areas in SAR images possess various scales, multiscale repre‐ sentations is critical for the detection of built-up areas. Based on above observa‐ tions, we introduce the structured prediction into our network, where a network classifies multiple pixels simultaneously. Meanwhile, we attempt to adopt multilevel features in our network. Experiments on TerraSAR-X high resolution SAR images over Beijing show that our method outperforms traditional methods and CNNs methods. Keywords: High-resolution SAR images · Detection of built-up areas · Structured prediction · Multi-level · Deep neural networks

1

Introduction

In recent years, the urban area is developing rapidly. Consequently, the monitoring and planning of urban areas become an important research field. Different from optical sensors, Synthetic Aperture Radar (SAR) is independent from sun illumination and weather conditions, which makes the information in SAR very useful for cities. In that case, the utilization of SAR data for the monitoring of urban areas has become the topic of recent discussions. Built-up area is the most obvious sign of urban areas. Detection of built-up areas in SAR images promises several applications, such as urban planning, disaster assessment, environmental monitoring. Therefore, the detection of built-up areas is of great importance. Different techniques for built-up areas detection have been presented in literature. Borghys et al. [1] proposed an automatic detection method of built-up areas in highresolution polarimetric SAR images in which most features are based on statistical properties of built-up areas. Yang et al. [2] developed a method for the land-over clas‐ sification of TerraSAR-X imagery over urban areas used texture features. Li et al. [3] © Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 646–655, 2017. https://doi.org/10.1007/978-3-319-71598-8_57

The Detection of Built-up Areas in High-Resolution

647

employed Labeled Co-occurrence Matrix for the detection of built-up areas in highresolution SAR images. Generally speaking, the most challenge problem of the detection of built-up areas in SAR images is feature extraction. The features used in all the afore‐ mentioned works are hand designed with domain knowledge and can significantly impact the classification accuracy. Recently, deep learning, especially convolutional neural networks (CNNs) [4, 5], has achieved much success in visual recognition tasks, for instance, object detection and image classification. Experiments showed that features extracted from CNNs are effec‐ tive and powerful [6, 7]. Lately, CNNs has been applied to the detection of the built-up areas in SAR images [8]. With the help of powerful features extracted by CNNs, Li et al. [8] achived state-of-art result. However, such a method classifies pixels separarely and ignore the strong correlation on labels between neighboring pixels. As we know, pixels belongs to background is more likely adjacent with background pixels than pixels belong to built-up areas in SAR images. We can obtain better result if we could make use of the informations on labels of neighboring pixels. In this paper, we proposed a deep neural network based on CNNs for the detection of built-up areas in SAR images. To make use of the informations on labels of neigh‐ boring pixels, our network is designed to be able to obtain multiple lables for pixles at the same time. In addition, since the built-up areas in SAR images possess various scales, we try to adopt multiscale features in ournetwork. Features extracted from different conv layers possess different receptive field sizes, making full use of them could help to detect built-up areas in various scales. We observed the results getting from all conv layers in our networks, and discovered that they can be complementary for the detection of builtup areas in SAR images. Based on the above observation, we adopt multi-level features in our network. The rest of this paper is organized as follows. In Sect. 2, we describe the method we proposed in detail. Section 3 shows the experiments and results. We present our conclu‐ sion in Sect. 4.

2

Structured Prediction

By automatically learning hierarchies of features from massive training data, CNNs obtained state-of–art results in most visual tasks of natural images, such as classification [9] and object detection [10]. Inspired of the great success made in natural images, several reseachers have attempt to adopt CNNs to proccess SAR data [11, 12]. Since built-up areas in SAR images are rich of structure informations, Li et al. [8] proposed a multiscale CNN model to extract the features of built-up areas to detect the built-up areas in SAR images. By densely predicting patches in SAR images, Li obtained good detection result compared with traditional methods. However, the multiscale CNN model classifies individual pixels separarely. As a result, the strong correlation on labels between neighboring pixels in SAR images would be ignored. It is well known that pixels in background is more likely to be adjacented with background pixels than pixels in built-up areas in SAR images. The information on labels of neighboring pixels could help making better decision.

648

Y. Wu et al.

As pointed by Liskowski et al. [13], we could make use of the information if labels for all pixels are available at the same time. In their work, Liskowski posed blood vessels segementation task as multilable inference problem on a set of binary predictions subject to a joint loss. This is a special case of structured prediction [14]. The structured prediction (SP) networks is designed to obtain information on multiple labels of pixels at the same time (Fig. 1). It can be achived by slight modification on existing deep architectures: we only need to set the number of units in final fully connected layer as m2, which indicate that if the centural m2 pixels of input patch are belong to built-up areas in SAR images. The loss function of SP network employs cross entropy (CE) loss: JCE (̂y, y) = −

∑

(yi log ŷ i + (1 − yi ) log(1 − ŷ i ))

(1)

i

where ŷ i and yi are the prediction and the target for i th output node.

SPnet

Fig. 1. An example for structured prediction: the n*n patch processed by CNN and get m*m labels of central m*m pixels of input patch.

In order to better analyse our method, i.e. the improved SP network, in the following experiment, we explore two kinds of SP networks: the plain SP networks and the fc4 concat fc3

fc3 fc2

fc3_2

fc2

fc3_2

fc1

fc2_2

fc1

fc2_2

fc3_1

conv3

fc1_2

fc3_1

conv3

fc1_2

fc2_1

pool2

pool4

fc2_1

pool2

pool4

fc1_1

conv2

fc1_1

conv2

pool3

pool1

pool3

pool1

conv1

conv1

input

input

Fig. 2. Models of two SP networks: left is the plain SP networks, right is the improved SP networks

The Detection of Built-up Areas in High-Resolution

649

improved SP networks. The models of these two networks are shown in Fig. 2, and the architectures are shown in Table 1. Pointed by Peng et al. [15], large kenel helps obtaining better performance. And Li et al. [8] indicate that large kenel help to reduce the effect caused by strong speckle noise in SAR images. To such consideration, we employ large kernel in convolutional layers in these two kinds of SP networks. Table 1. Architecture of SP networks. Layer Name conv1 pool1 conv2 pool2 conv3 fc1/fc1_1/fc1_2 fc2/fc2_1/fc2_2 fc3/fc3_1/fc3_2/fc4 pool3 pool4

Kenel size*channel 9*9*50 4*4 8*8*100 4*4 3*3*300 1000 1000 100 4*4 2*2

Stride 1 4 1 4 1

4 2

In both two SP networks, Rectification non-linearity was used in used in all convo‐ lutional layers and fully connected layers to accelerate the convergence of stochastic gradient decent. In adidtion, drop out layer is employed in first two fully connected layers. 2.1 The Plain SP Network The plain SP network is a sequential combination of convolutional layers, maxpooling layers and fully connected layers. However, since we introduce large kenel in networks, SP networks would be hard to train and easily encounter the problem of overfitting. In consideration of such circumstances, we add extra supervision to the plain SP networks. As pointed out by [16], extra supervision using hidden layer feature maps leads to reduction in testing error. In prediction stage, the result obtained by extra supervison will be abandoned. 2.2 The Improved SP Network The model of the improved SP network is shown in Fig. 2. Each conv layer in the improved SP network is connected to a stack of fully connected layers. And the results are concatenated and processed by a fully connected layer to obtain final classifier output. The motivation behind this is that we would like to introduce multiscale features to the final classifier. As we know, the neurons in different levels have different receptive field sizes, they can be seen as representations of multiscale. Considering the dynamics

650

Y. Wu et al.

of the sizes of built-up areas in SAR images, we hypothesize that the hierarchical information could help to make better decision. Thus, we combine hierarchical features from all the conv layers. Instead of combining hierarchical features directly, we choose to combine them after they are proccessed by serveral fully connected layers so that features to be concatenated are trained to be more discriminative. Since receptive field sizes in different conv layers in our network are different, our network could learn multiscale features. Such information is helpful for the detection fo built-up areas in SAR images. SAR images are corrupted by speckle noise, which could significantly impact the detection result. And the sizes of built-up areas in SAR images are so dynamic. By introducing multi-level features, our network could have the ability to suppress the effect of speckle noise and obtain good detection result at the same time. We show the intermediate results of the plain SP network in Fig. 3. From left to right, the receptive field sizes decrease. We can see that under the complicated environment condition in SAR images, network with single receptive filed size can not always obtain satisfactory detection result. By embedding the multi-level features into classifier, our network is expected to achive better detection result.

Fig. 3. Example of intermediate results of the plain SP network (a) Original image. (b) Reference. (c) Result of fc3 layer. (d) Result of fc3_2 layer. (e) Result of fc3_1 layer.

The Detection of Built-up Areas in High-Resolution

3

651

Experiment and Result

High-resolution TerraSAR-X SAR images of Beijing collected on November 25, 2011 were selected to verify our method. The SAR image is of range resolution of 2.3 m, and azimuth resolution of 3.3 m. The types of building areas in images includes Dot villa district, residential quarter buildings, squatter settlement and etc. Training: We used caffe [17] to train our networks, and Stochastic Gradient Descent is used for training. The initial learning rate is 0.0001. We use momentum of 0.9 and weight decay of 0.0005. Dataset: In the following experiments, we set the size of input patches as 84*84, and choose the output of our network as 10*10. We selected 90000 patches as train data, and 24000 patches as validation data. The test data is formed by an SAR image of 2500*4000 pixels. Qualitative results: Fig. 4 shows the fragments of detection results obtained by multi‐ scale CNN and two kinds of the SP networks. The first column of Fig. 4 shows the detection result of road area in SAR images. We can see clearly from it that road areas are quite similar with build-up areas in SAR images, and by making use of the

Fig. 4. Fragment of the detection result of SAR images (a) Original image. (b) Reference. (c) Multiscale CNN. (d) Plain SP network. (e) Improved SP network.

652

Y. Wu et al.

information on labels of neighboring pixels, the SP networks behave better than multi‐ scale CNN. The second column and the third column indicate that the SP networks obtain good results in building dense areas and “slender” built-up areas. The last two column of Fig. 4 are the failure examples of the SP networks, but we can see that the SP networks still obtain comparable results in such areas. In general, the improved SP network obtain better result in the examples by introducing multi-level features. We visualize the entire detection result in Fig. 5. From Fig. 5(c), we can see that multiscale CNN model obtained a good performance, most built-up areas have been detected successfully. However, as mentioned above, we can see that multiscale CNN model performs not so satisfactory in road areas and building dense areas, and then cause

Fig. 5. Experiment results (a) SAR image of northern areas of Beijing. (b) Manually labeled image. (c) Detection result of multiscale CNN. (d) Detection result of the plain SP network. (e) Detection result of the improved SP network.

The Detection of Built-up Areas in High-Resolution

653

high false alarm rate in such areas. The performance of the plain SP network can be seen form Fig. 5(d). In general, the pain SP network achived better detection result than multiscale CNN. But since the plain SP network belongs to single scale network, it is not able to deal with the complex size of built-up areas in SAR images. And then the plain network obtain lower detection rate. The detection result of the improved SP network can be seen in Fig. 5(e), it shows that the improved SP network achived best result. Pixel level results: Performance of pixel level accuracy is presented in Detection rate (DR), False alarm rate (FA), Accuracy of classification (Acc) [18], they are defined as: DR =

TP , TP + FN

FA =

FP , TP + FP

Acc =

TP + TN , TP + TN + FP + TP

where TP, TN , FP and FN are respectively the numbers of true positive, true negative, false positive, and false negative decisions. In multiscale CNN, the three performance indicators are based on the output of network: positive decision is made if the network judge the input patch belongs to builtup areas, otherwise, negative decision is made. And in SP networks, the three perform‐ ance indicators are based on the default interpretation of network decisions: positive decision is made if the output of the unit (sigmoid) is greater than 0.5 threshold, other‐ wise, negative decision is made. Pixel level accuracy is shown in Table 2, the detection result of multiscale CNN is result of [8]. We can see that the improved SP network obtain best result on Detection rate and Accuracy of classification. On the False alarm rate, our method is a little higher than the plain SP network, we think that it is because network is hard to optimise when introducing multilevel features. Table 2. Pixel level accuracy. Method

Detection rate

False alarm rate

GLCM LCM [3] CNN42 CNN84

84.38% 89.39% 90.43% 90.38%

15.82% 23.40% 12.77% 17.10%

Accuracy of classification 88.78% 86.16% 90.52% 89.64%

Multiscale CNN [8] Plain SP network Improved SP network

92.14% 91.00% 92.40%

10.71% 9.08% 9.87%

92.86% 93.18% 93.32%

As mentioned above, in SP networks, positive decision is made if the output is greater than 0.5 threshold. However, this threshold is not enough to show the advantage of SP networks. Figure 6 shows the change of result when setting different thresholds in the improved SP network. From Fig. 6, we can find that the curve of Accuracy of classifi‐ cation changes slower near the threshold of 0.5. On the contrary, the Detection rate decrease when threshold increase. It indicates that we can slightly change the threshold to obtain different Detection rate and False alarm rate while keeping Accuracy of

654

Y. Wu et al.

classification in a stable state. For example, we can set the judge threshold smaller than 0.5 to get high performance of detection rate or bigger than 0.5 to get lower false alarm rate (Table 3). To some extent, it means that we can control the detection result by setting different thresholds.

Fig. 6. Experiments results getting from different threshold.

Table 3. Pixel level accuracy. Method

Detection rate

False alarm rate

Multiscale CNN [8] Plain SP network-0.4 Improved SP network-0.4 Plain SP network-0.6 Improved SP network-0.6

92.14% 92.41% 94.04% 89.41% 90.48%

10.71% 10.17% 11.44% 8.03% 8.43%

Accuracy of classification 92.86% 93.19% 93.17% 93.06% 93.27%

In Table 3, the suffix “-0.4” or “-0.6” means we choose 0.4 or 0.6 as the threshold of SP networks. From Table 3, we can find that we could obtain controllable results by setting different thresholds.

4

Conclusion

In this paper, we proposed an improved structured prediction network for the detection of built-up areas in SAR images. By making use of the information on labels of neigh‐ boring pixels and multi-level features, our network achived success in the detection of built-up areas in SAR images. In particularly, we can obtain controllable results by setting different thresholds on the output of the improved SP networks. the experiments carried out on TerraSAR-X SAR image of Beijing confirmed that our method is effective to detect built-up areas in SAR images.

The Detection of Built-up Areas in High-Resolution

655

Acknowledgment. This work was supported in part by the National Nature Science Foundation of China (No.61331020).

References 1. Borghys, D., Perneel, C., Acheroy, M.: Automatic detection of built-up areas in highresolution polarimetric SAR images. Pattern Recogn. Lett. 23(9), 1085–1093 (2002) 2. Yang, W., Zou, T., Dai, D., et al.: Supervised land-cover classification of TerraSAR-X imagery over urban areas using extremely randomized clustering forests. In: Urban Remote Sensing Event, 2009 Joint, pp. 1–6. IEEE (2009) 3. Li, N., Bruzzone, L., Chen, Z., et al.: Labeled co-occurrence matrix for the detection of builtup areas in high-resolution SAR images. In: SPIE Remote Sensing. International Society for Optics and Photonics, p. 88921A-88921A-12 (2013) 4. LeCun, Y., Kavukcuoglu, K., Farabet, C.: Convolutional networks and applications in vision. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 253–256. IEEE (2010) 5. Krizhevsky, A., Sutskever, I., Hinton, G E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012) 6. Karpathy, A., Toderici, G., Shetty, S., et al.: Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725–1732 (2014) 7. Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv: 1408.5882 (2014) 8. Li, J., Zhang, R., Li, Y.: Multiscale convolutional neural network for the detection of builtup areas in high-resolution SAR images. In: 2016 IEEE International on Geoscience and Remote Sensing Symposium (IGARSS), pp 910–913. IEEE (2016) 9. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014) 10. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015) 11. Chen, S., Wang, H., Xu, F., et al.: Target classification using the deep convolutional networks for SAR images. IEEE Trans. Geosci. Remote Sens. 54(8), 4806–4817 (2016) 12. Gong, M., Zhao, J., Liu, J., et al.: Change detection in synthetic aperture radar images based on deep neural networks. IEEE Trans. Neural Netw. Learn. Syst. 27(1), 125–138 (2016) 13. Liskowski, P., Krawiec, K.: Segmenting retinal blood vessels with deep neural networks. IEEE Trans. Med. Imaging 35(11), 2369–2380 (2016) 14. Bakir, G.: Predicting Structured Data. MIT press, Cambridge (2007) 15. Peng, C., Zhang, X., Yu, G., et al.: Large kernel matters–improve semantic segmentation by global convolutional network. arXiv preprint arXiv:1703.02719 (2017) 16. Lee, C Y., Xie, S., Gallagher, P W., et al.: Deeply-supervised nets. AISTATS. 2(3), p. 5 (2015) 17. Jia, Y., Shelhamer, E., Donahue, J., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM (2014) 18. Shufelt, J.A.: Performance evaluation and analysis of monocular building extraction from aerial imagery. IEEE Trans. Pattern Anal. Mach. Intell. 21(4), 311–326 (1999)

SAR Automatic Target Recognition Based on Deep Convolutional Neural Network Ying Xu1, Kaipin Liu1, Zilu Ying1, Lijuan Shang1, Jian Liu1, Yikui Zhai1,2 ✉ , Vincenzo Piuri2, and Fabio Scotti2 (

)

1

School of Information and Engineering, Wuyi University, Jiangmen 529020, China [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 2 Department of Computer Science, Università degli Studi di Milano, 26013 Crema, Italy {vincenzo.piuri,fabio.scotti}@unimi.it

Abstract. In the past years, researchers have shown more and more interests in synthetic aperture radar (SAR) automatic target recognition (ATR), and many methods have been proposed and studied for radar target recognition. Recently, deep learning methods, especially deep convolutional neural networks (CNN) has proven extremely competitive in image and speech recognition tasks. In this paper, a deep CNN model has been proposed for SAR automatic target recogni‐ tion. The proposed deep model named SARnet, has two stage convolutionalpooling layers and two full-connected layers. Due to the demand of requirement of large scale of the data in deep learning, we proposed an augmentation method to get a large scale database for the training of CNN model, by which the CNN model can learn more useful features through the large scale database. Experi‐ mental results on the MSTAR database show the effectiveness of the proposed model and has achieved encouraging results with a correct recognition rate of 95.68%. Keywords: Synthetic Aperture Radar (SAR) · Deep learning Deep Convolutional Neural Networks (CNN) · Data augmentation

1

Introduction

In the past few decades, as SAR can operate regardless of weather conditions, day and night, so it is of high value in military applications [1–3]. As the automatic target recog‐ nition (ATR) is the primary application of SAR, so ATR task has been used more and more frequently in military and commercial applications, such as reconnaissance, navi‐ gation, guidance, remote sensing, and resource exploration etc. More and more researchers have shown interests to SAR automatic target recognition, a great develop‐ ment has developed and many effective algorithms have been proposed. However, it is still a challenging issue due to the complexity of measured information such as speckle noises, variation of azimuth, and poor visibility, so how to interpret the SAR images and recognize the true targets correctly still remains to be further studied and explored.

© Springer International Publishing AG 2017 Y. Zhao et al. (Eds.): ICIG 2017, Part III, LNCS 10668, pp. 656–667, 2017. https://doi.org/10.1007/978-3-319-71598-8_58

SAR Automatic Target Recognition Based on Deep CNN

657

SAR automatic target recognition also has two key procedures, feature extraction and classifier construction, which including many kinds of algorithms, such as principal component analysis (PCA) [4], independent component analysis (ICA), linear discrim‐ inate analysis (LDA), nonnegative matrix factorization (NMF), sparse representation coefficient for feature extraction, and theoretically all classification algorithms can be extended to the application of SAR image target classification, such as support vector machine (SVM) [5], K-nearest neighbor (KNN), mean square error (MSE), hidden Markov models (HMM). All the above methods are applied for SAR automatic target recognition successfully. Deep learning method was proposed by Hinton [6] in 2006, and achieved great success in many fields such as in speech, image processing, medicine, natural language understanding, etc. Deep network implements the mapping of low-level features to highlevel features layer by layer. It learns the relationship between features automatically. Recent years, as one model of deep learning, convolutional neural networks has become more and more popular, because of its powerful learning skill, it has been shown an exciting results in image classification [7, 8], face recognition [9, 10], object detection [11], etc. There are also some deep learning methods used in SAR automatic target recognition, such as CNNs [12], sparse auto-encoder [13], etc., which has achieved impressive results. For CNNs, researchers devote themselves to design their own networks, which include the number of convolutional kernels, layers and other relying parameters, to learn high-level features for the image recognition. As a results, the tasks have benefited from the robust and discriminative representation via CNN models. In this paper, inspired by the Lenet, a convolutional networks has been proposed, which has two stage convolutional-pooling layers and two full-connected layers. To get the best results, and Relu activation function is adopted after the first fully-connected layers. The deep learning algorithm has a high demand of scale of data. However, since the number of SAR images for specific targets are always very limited. Since SAR ATR is very sensitive to rotational features, this paper used a method of data augmentation to get more rotational SAR images. The amount of the training images is 22 times than the original database.

2

Convolutional Neural Networks

2.1 Convolutional Neural Networks The fundamental convolutional neural network is comprised of input layer, convolu‐ tional layer, pooling layer, fully connected layer and an output layer. The deep convo‐ lutional neural networks often have two or more convolutional layers, which also followed by pooling layers. The framework of the deep convolutional neural network is shown in Fig. 1. (1) Convolutional layer: Convolutional neuron layer is the key component of CNN. For image classification task, the input to the convolutional layer always being treated as 2-dimension (gray image) matrices or 3-dimension (color image)

658

Y. Xu et al.

matrices, because of the local connected and the parameter sharing, the operation of the convolutional layer can be represented as below: w(x, y) ∗ f (x, y) =

a b ∑ ∑

w(s, t)f (x − s, y − t)

(1)

s=−a t=−b

where w(x, y) is the filter parameter of one layer, which named convolutional kernel, f (x, y) is the neighborhood of a image. After the convolutional operation, a feature map can be obtained, which represents one kind of feature of the input image.

Input image

Convolutional layer

Pooling layer

Convolutional layer

Pooling layer

Convolutional layer

Pooling layer

Feature representation

classifier

output

Fig. 1. Framework of the deep convolutional neural network

(2) Pooling layer: In the pooling layer, each feature map is treated separately and the operation of pooling plays an important role in CNN for feature dimension reduc‐ tion. In order to reduce the number of output neurons in the convolutional layer, pooling algorithms, which are used frequently, include max-pooling and averagepooling. And the kernel size always being used is 2 × 2. (a) Max-pooling. For a random 2 × 2 neighborhood, the output of the max-pooling operation is the maximum pixel value of the neighborhood. (b) Average-pooling. For random 2 × 2 neighborhood, the output of the averagepooling operation is the average value of the neighborhood. (3) Activation function: In the theory of neural network, the activation function is a important part to obtain the predict value, Sigmoid or Tanh activation function is often used as the activation function. Sigmoid or Tanh activation function is a nonlinear activation for neural networks and often leads to robust optimization during DNN training. But it may suffers from a vanishing gradient when lower layers have gradients of nearly 0 because higher layer units are nearly saturate at −1 or 1. The vanishing gradient may lead to a slow convergence or a poor local optima for convolution neural networks. To overcome vanishing gradient, the Rectified linear unit (ReLU) [14] offers a sparse representation. The Relu is more biologically plausible than widely used logistic sigmoid or hyper‐ bolic tangent. And the rectified linear function is shown below: f (x) = max(0, x)

(2)

SAR Automatic Target Recognition Based on Deep CNN

659

The formula (2) can show that the unit is not active if the value is 0 or below 0, which can make the feature representation sparse. In this paper, Relu is used as the activation function, and the functional graph is shown in Fig. 2.

Fig. 2. The graph of Relu function

2.2 The Framework of SARnet For the SARnet mentioned before, the paper used the traditional convolutional neural networks as shown in Fig. 1, the framework of SARnet include two stage convolutionalpooling layers and two full-connected layers. After each convolutional-pooling layer and the first full connected layer, a relu function followed, the loss used in this paper is softmax loss. The detail parameter setting is presented in Table 1. In Table 1, Conv is short for convolutional, FC is short for the fully connected. Table 1. The architecture of the SARnet Layer type Input layer Conv layer1 Pooling-layer1 Relu1 Conv layer2 Pooling layer2 Relu2 FC layer1 FC layer2

Input size 49 × 49 49 × 49 43 × 43

Kernel size – 7×7 MAX/2 × 2

Kernel number – 35 –

Output size – 43 × 43

22 × 22 16 × 16

7×7 MAX/2 × 2

70 –

16 × 16

8×8 700

– –

700 3

700 3

22 × 22

8×8

As shown in Table 1, the SARnet has two convolutional layers, whose kernel size are both 7 × 7, two pooling layers, which operate the max-pooling, and the pooling kernel size are both 2 × 2, the experiment of choosing the number of kernel number and kernel size are presented in Sect. 4.

660

3

Y. Xu et al.

Image Pre-processing

3.1 The Extracting of ROI (Region of Interest) of SAR Image The SAR image has a lot of background noise, which will be negative to the recognition rate. Also, the target in the image may occupy a little part of the image and in the center of the image. So it is necessary to extract the region of interest (ROI) that is the object could be recognized for the task of ATR. To extract ROI region, a simple image interception pre-processing procedure is applied to the original SAR image. As the target in the center, firstly, we locate the centroid of an image, the formula is as follows: (xc , yc ) = (

m10 m01 , ) m00 m00

(3)

Where, the p + q-order origin moment is represented as follows mpq =

∑∑ x

xp yq f (x, y)

(4)

y

In (3), the m10 and m01 is 1-order origin moment, m00 is the 0-0-order origin moment, In (4), (x, y) denotes a pixel’s coordinate of an image, f(x, y) denotes the pixel value. Once the (xc, yc) is located, a L × L image could be extracted from the original SAR image with the centroid (xc, yc). Figure 3(a) is a original image from the MSTAR database and the Fig. 3(b) and (c) shows the results of ROI images extracted from the original SAR images. Figure 3(b) is the ROI image, which L = 64, and the Fig. 3(c) is the ROI image with L = 49.

(a) Original Image

(b) ROI Image L=64

(c) ROI Image L=49

Fig. 3. ROI images

3.2 Data Augmentation In machine learning, training data is a important factor of the whole processing, it can decide the kind of features that learned from the original images. So a large scale training database is necessary for the machine learning, especially for the deep learning. For CNNs, if we want to get an impressive results, the researchers often chose a large scale database to learn more high-level and useful features.

SAR Automatic Target Recognition Based on Deep CNN

661

Since SAR images are sensitive to the depression angles and azimuth angles, which will decide the SAR image. Also, it is clear that an object would not change its class if the object is rotated, or a random noise added to the image, but the amplitude of each pixel of the image will change, or the view of the image will change. As a result, the variety of the feature will gain through this way. This is to say more images in one class can be obtained, which result in a larger scale of data set. With this motivation, this paper formulate the methods of SAR image augmentation. The methods as following, and the results are shown in Figs. 4 and 5.

(a) Original Image

(b) Rotated Image

(c) ROI Original

(d) ROI Rotated

Fig. 4. Results of method 1

(a) Original Image

(b) Rotated Image

(c) ROI Original

(d) ROI Noised

Fig. 5. Results of method 2

Method 1: Rotate the object: Rotate the original SAR image clockwise or anticlock‐ wise with an angles; Extract the ROI with an expected L, where the L is the resolution of the ROI image. Method 2: Add a random noise to the image: For each image, we generate a random integer from −10 to 10, then add the random integer to the image; Extract the ROI with an expected L, where the L is the resolution of the ROI image. We use the image of MSTAR data set as a example. In Figs. 4(a) and 5(a) are original image, and Fig. 5(c) is the ROI image with 49 of the original image. In Fig. 4, the original image is rotated with 15 angles anticlockwise, firstly, and then extract the ROI image from the rotated image. Through Fig. 4, we can observe that the direction of the object is changed obviously by rotating a random angle, In Fig. 5, we add a random integer to the image, here the random integer is 9 in Fig. 5. Two resulted images which obtained by the method we proposed in Figs. 4 and 5, are definitely different from the original image visually. The result of experiment demonstrate the effectiveness of the methods we proposed. For the MSTAR database, we use the methods mentioned above to obtain more training data. Firstly, we rotate the every original image with integral angles from [1, 10], clock‐ wise and anticlockwise respectively. Secondly, we add a random integer from [−10, 10]

662

Y. Xu et al.

to the original image. Thus, the final train database after data augmentation is 22 times compared with the original training database.

4

Experimental Results and Analysis

4.1 Experimental Environment The training procedure of SARnet is implemented by the Caffe, which is the deep learning framework, the training environment is GPU GTX980. The testing procedures in Matlab-R2014a environments are conducted on Window 7 with the Inter CPU and 32G RAM. The criteria we applied to evaluate the performance of the model is Rank-1 recog‐ nition rate, it is defined as, where is the number of samples, which is recognized correctly, and is the total number of the testing samples. 4.2 MSTAR Database In the following experiments, the experimental data we used is MSTAR database, which is provided by the Sandia National Laboratory (SNR) SAR sensor platform operating at X-band. The collection was jointly sponsored by Defense Advanced Research Projects Agency (DARPA) and Air force Research Laboratory as part of the Moving and Stationary Target Acquisition and recognition (MSTAR) program [15]. The database is divided into training set and testing set, both two sets contain multiple types of ground military target, including T72 (Main Battle Tanks), BMP2 (Armored Personal Carriers), BTR70 (Armored Personal Carriers) etc. The mentioned three targets’ optical images and their SAR images are illustrated in Fig. 6, and the configuration of each class is shown in Table 2.

(a) Target T72 Optical Image

(b) Target BTR70 Optical Image

(d) Target T72 SAR Image

(e) Target BTR70 SAR Image

(c) Target BMP2 Optical Image

(f) Target BMP2 SAR Image

Fig. 6. Optical images and SAR images of three classes

SAR Automatic Target Recognition Based on Deep CNN

663

Table 2. Training set and testing set for the three classes of targets Training set Target class T72_SN132

Sample number 232

BMP2_SNC21

233

BTR70_SNC71

233

Testing set Target class T72_SN132 T72_SN812 T72_SNS7 BMP2_SN9563 BMP2_SN9566 BMP2_SNC21 BTR70_SNC71

Sample number 196 195 191 195 196 196 196

The size of the original SAR image is the variation of the azimuth angle of every target in MSTAR database is from 0 to 360°. The database collected at 17° angles are used for training, and collected at 15° depressing angles are used for testing. 4.3 Training Methodology To train the convolution network SARnet, we randomly select 20% images from each class as the validation set, for the validation set, there is no need to do the data augmen‐ tation, and for the rest images in each class of the training set, we use the methods mentioned above to get more training images. At last, the augmentated data set with 13772 images, which is 22 times than the original training set, is used as the training set during the training procedure. And we choose the size of the input as 49 × 49. The open source deep learning framework Caffe [16] is used for training the model. We use the stochastic gradient descent to update the weights W by a linear combi‐ nation of the negative gradient ∇L(W) and the previous weight update Vt [17], the weights updating formula as shown below: Vt+1 = �Vt − �∇L(Wt )

(5)

Wt+1 = Wt + Vt+1

(6)

Where, � is the learning rate, which for the negative gradient, � is momentum for the weight of the previous update. In this experiment, we set the learning rate as 0.0001 during the whole training processing, and the momentum � is set as 0.9. The method of parameter initialization for both convolution and fully-connected layers is Xavier. The deep model is trained on GTX980 about 6 h until it reaches its convergence. 4.4 Experiment on the Configuration of SARnet As the configuration of the convolutional neural network has many parameters, and the whole model is changed as one of the parameters changed. To get the best results, we trained many models with different configurations, which is inspired by the Lenet that has two convolutional-pooling stage without Relu activation function, and only has the Relu activation function follow behind the first fully-connected layers. Testing results

664

Y. Xu et al.

of the different models are shown in Table 3. We extract the features of the testing set from the models, and SVM is utilized for accuracy. In Table 3, The convolutional layer parameters are denoted as kernel size/kernel number. In Table 3, the size of the pooling layer is 2, and the second fully-connected layer’s output size is 3, which are not shown in Table 3. Table 3. Testing results of different models Model 1 2 3 4

Conv1 4/15 6/25 7/35 8/40

Conv2 4/30 6/50 7/70 8/80

FC1 300 500 700 800

Accuracy 95.16% 95.24% 95.47% 94.43%

We can see the Model 3 has the highest accuracy, as for the SAR image, the target in the image only has small amount of pixels, oppositely, the background of the SAR image occupies the most part of the SAR image, unlike the face images or some natural images, of which the target often occupy the most part. So, it is better to set a relatively larger convolutional to get a better features and better performance. Here we choose Model 4 as the best configuration of the SARnet. 4.5 Experiment on the Influence of the Relu Activation Function Here, we choose the Model 3 as the best configuration of the SARnet, but the convolu‐ tional-pooling stage is not followed by the Relu activation function, so in this experi‐ ment, in order to test the influence of the Relu activation function, we added the Relu activation function to each convolutional-pooling stage, the testing results shown in Table 4. The configuration of the two models’ fully connected stages are same, so the Table 4 doesn’t describe the fully connected stage. In Table 4, C-P is short for the convolutional-pooling stage, and the R is short for the Relu layer. Table 4. Testing results of the influence of the Relu activation function Model C-P1 3 7/35, 2 5 7/35, 2

R1 No Yes

C-P2 7/70, 2 7/70, 2

R2 No Yes

Accuracy 95.47% 95.68%

As shown in Table 4, the influence of the Relu activation function is obvious. As mentioned in section B, the Relu activation function, which following the each convo‐ lutional-pooling stage, can make features of every layers sparse, which can make the features more distinguish, and reduce the interdependency of the parameters of every layers. So we choose the Model 5 as the last model of SARnet. To be more convincing, the paper test the recognition accuracy of the model, which can achieve the best accuracy, with the sub-class of the testing set. The testing results are shown in Table 5. As shown in Table 5, the recognition rate of the BTR70 without sub-class in the testing set achieves 100%, and for T72, 13 targets are predicted to BMP2,

SAR Automatic Target Recognition Based on Deep CNN

665

which achieve 94.5%, and for the BMP2, 27 targets are predicted to T72 and BTR70, which achieve 95.40%. The recognition rate of the sub-class demonstrate the effective‐ ness of the SARnet model, but the accuracy of the T72 and BMP2 still need to be improved. Table 5. Recognition rate of the sub-class

Testing set

T72

BMP2

BTR70

SN132 SN812 SNS7 SN9563 SN9566 SNC21 SNC71

Training set T72 BMP2 183 13 183 12 184 7 3 191 13 181 5 188 0 0

Accuracy (%) BTR70 0 0 0 1 2 3 196

94.50

95.40

100

4.6 Comparison with Other Proposed Algorithms To be more general, a comparison with other proposed algorithms is presented in Table 5. The deep learning algorithms, such as CNNs [12], the other machine learning algorithms, such as JSRC [19] is a novel joint sparse representation based multi-view automatic target recognition method. As Table 6 shown, our CNN model outperforms some proposed method and has reached an encouraging results of the SAR ATR task. Our method focus more on data augmentation and designing of a CNN model to learn the high level features without hand-crafted design, which can obtain an effective recognition result. Table 6. Comparison with other proposed algorithms Method SVM [5] HMM [18] Single-View SRC [19] JSRC [19] CNNs [12] SARnet

5

Accuracy 90.92% 94.00% 92.30% 95.60% 93.00% 95.68%

Conclusion

In this paper, we proposed convolutional neural network, which names SARnet, for the SAR automatic target recognition based on data augmentation. The convolutional neural network can learn the high level features of the SAR images automatically. With a large scale of data by augmentation, the proposed model can reach effective result. Also, this

666

Y. Xu et al.

augmentation method can be adopted in other feature learning method to improve feature extraction effectiveness in the future. Acknowledgment. This work is supported by NNSF (No. 61372193), Guangdong Higher Education Outstanding Young Teachers Training Program Grant (No. SYQ2014001), Characteristic Innovation Project of Guangdong Province (No. 2015KTSCX143, 2015 KTSCX145, 2015KTSCX148), Youth Innovation Talent Project of Guangdong Province (No. 2015KQNCX172, No. 2016KQNCX171), Science and Technology Project of Jiangmen City (No. 201501003001556, No. 201601003002191), and China National Oversea Study Scholarship Foundation.

References 1. Huan, R., Pan, Y.: Decision fusion strategies for SAR image target recognition. IET Radar Sonar Navig. 5(7), 747–755 (2011) 2. Lee, J.-H., Cho, S.-W., Park, S.-H., Kim, K.-T.: Performance analysis of radar target recognition using natural frequency: frequency domain approach. Prog. Electromagn. Res. 132, 315–345 (2012) 3. Varshney, K.R., Cetin, M., Fisher, J.W., Willsky, A.S.: Sparse representation in structured dictionaries with application to synthetic aperture radar. IEEE Trans. Sign. Process. 56(8), 3548–3560 (2008) 4. Chamundeeswari, V.V., Singh, D., Singh, K.: An analysis of texture measures in PCA-based unsupervised classification of SAR images. IEEE Geosci. Remote Sens. Lett. 2(6), 214–218 (2009) 5. Zhao, Q., Principe, J.C.: Support vector machines for SAR automatic target recognition. IEEE Trans. Aerosp. Electron. Syst. 37(2), 643–654 (2001) 6. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006) 7. He, K., Zhang, X., Ren, S., et al.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. Comput. Sci., 1026–1034 (2015) 8. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems, pp. 1097–1105. Curran Associates Inc. (2012) 9. Sun, Y., Chen, Y., Wang, X., Tang, X.: Deep learning face representation by joint identification-verification. In: Proceedings of Advances in Neural Information Processing Systems, vol. 27, pp. 1988–1996 (2014) 10. Sun, Y., Wang, X., Tang, X.: Deep learning face representation from predicting 10000 classes. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1891–1898 (2014) 11. Szegedy, C., Reed, S., Erhan, D., et al.: Scalable, high-quality object detection. Computer Science (2015) 12. Wilmanski, M., Kreucher, C., Lauer, J.: Modern approaches in deep learning for SAR ATR. SPIE Defense+Security, 98430N (2016) 13. Sun, Z., Xue, L., Xu, Y.: Recognition of SAR target based on multilayer auto-encoder and SNN. Int. J. Innov. Comput. Inf. Control 9(11), 4331–4341 (2013) 14. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, vol. 3, pp. 807–814 (2010)

SAR Automatic Target Recognition Based on Deep CNN

667

15. Mossing, J.C., Ross, T.D.: An evaluation of SAR ATR algorithm performance sensitivity to MSTAR extended operating conditions. In: Proceedings of SPIE, vol. 3370, pp. 554–565 (1998) 16. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, pp. 675–678 (2014) 17. Bottou, L.: Stochastic Gradient Tricks. Inbook (2012) 18. Kottke, D.P., Fiore, P.D., Brown, K.L., et al.: A design for HMM based SAR ATR. In: Proceedings of SPIE, Orlando, FL, USA, vol. 3370, pp. 541–551 (1998) 19. Zhang, H.C., Nasrabadi, N.M., Zhang, Y.N., Huang, T.S.: Multi-view automatic target recognition using joint sparse representation. IEEE Trans. Aerosp. Electron. Syst. 48(3), 2481–2497 (2012)

Author Index

An, Ping I-567 Bai, Huihui III-436 Bai, Lian-fa I-362, I-386, I-174 Bai, Xiang I-238 Bai, Xiao III-370 Bao, Qiuxia II-313 Bao, Xinyan II-455 Bi, Du-Yan II-274, II-580 Cai, Aihua III-633 Cao, Fengyin III-287 Cao, Youxia II-119 Chan, Ka-Hou III-299 Chen, Dan I-602 Chen, Fei II-466 Chen, Han III-505 Chen, Jiajie II-86 Chen, Jiansheng I-614, II-552 Chen, Jing I-533 Chen, Jingbo II-552 Chen, Jingjia II-252 Chen, Jingying I-651 Chen, Junli II-395 Chen, Lei I-545, II-262 Chen, Lvran II-38 Chen, Mei I-577 Chen, Qiang III-601 Chen, Qiaosong I-374 Chen, Xiang I-35 Chen, Xiaochun I-35 Chen, Xiaopeng III-27 Chen, Xiaoyu I-386 Chen, Xingyan II-353 Chen, Yanjia II-612 Chen, Yuan II-440 Cheng, Congxin III-517 Cheng, Hengda I-273 Cheng, Yudong II-14 Cheng, Yuehua I-523 Cui, Hao I-340 Cui, Haolong II-455 Cui, Ziguan II-14

Deng, Kailian II-590 Deng, Xin I-374 Ding, Jianwei I-48, III-589 Ding, Ling III-58 Ding, Wanmeng III-527 Ding, Wenshan II-580 Ding, Youdong III-275 Dong, Jiayu II-62 Dong, Lanfang I-149, II-216 Dong, Yan I-443 Dong, Ya-Yun II-274 Dou, Jian II-3

Fan, Zhaoxuan I-262 Fan, Zunlin II-580 Fang, Jianwu II-3, III-254 Fang, Sisi I-545 Fang, Yongchun I-432 Feng, Guanghua I-58 Feng, Haoyang I-351 Feng, Shangsheng I-374 Feng, Wengang I-48, III-589 Feng, Xiaoyi I-420, I-555, I-626, III-231 Fu, Dongmei II-487, III-243 Fu, Haiyan II-206 Fu, Shun I-35 Fu, Xiaolong II-369 Gan, Junying I-211 Gan, Zongliang II-14 Gao, Ding-Li III-611 Gao, Guangshuai I-443 Gao, Lei II-139 Gao, Qishuo III-358 Gao, Qiuli III-149 Gao, Wei I-454 Gao, Xinbo I-397, III-170 Gao, Yi III-445 Gong, Ruobin III-275 Gong, Zhiqiang II-97 Granger, Eric I-555 Guo, Enlai I-362

670

Author Index

Guo, Feng III-505 Guo, Xiao-hua III-458 Guo, Yuchen III-70 Han, Jing I-174, I-362, I-386 Han, Mingyan I-501 Hao, You I-199 He, Dongxu II-552 He, Linyuan II-580 He, Na II-86 He, Tong III-204 He, Yuqing II-505 Heng, Jianyu I-295 Hongbin, Zang I-224 Hongsen, He I-224 Hou, Zhiqiang I-186 Hu, Guang I-501 Hu, Han II-325 Hu, Honggang III-471 Hu, Jiang-hua III-383 Hu, Mengnan I-466 Hu, Ningning III-436 Hu, Wei II-163 Hu, Xianjun III-471 Hu, Xiaowei III-36 Hu, Yan-lang III-539 Hu, Zhanyi I-454 Hua, Wei II-466 Huang, Deliang I-127 Huang, Dongjin III-275 Huang, Jiexiong III-483 Huang, Ningsheng I-533 Huang, Qinghua II-513 Huang, Rui II-395 Huang, Ruochen II-313 Huang, Shijia I-127 Huang, Xin I-477 Huang, Yongfeng II-109 Hui, Zheng I-397 Im, Sio-Kei

III-299

Ji, Rongrong III-496, III-505 Jia, Bowen III-412 Jia, Shijie II-353 Jia, Xiuping III-358 Jia, Xiuyi III-311 Jia, Yunde I-577 Jian, Meng III-458

Jiang, Bo II-119 Jiang, Jun II-602 Jiang, Wanli I-262 Jiang, Xiaoyue I-420, I-626 Jiang, Yinghua I-1 Jiang, Yizi II-513 Jiang, Zhiguo I-664 Jiang-hua, Hu II-545 Jin, Zefenfen I-186 Ke, Bo II-38 Ke, Yihao II-325 Khatibi, Siamak III-134 Kong, Bin III-400 Kong, Xiangwei II-206 Kwolek, Bogdan III-423 Lan, Tian III-231 Le, Chao I-489 Lei, Peng III-370 Lei, Qin II-545, III-383 Leng, Zhen I-533 Li, Baoquan I-432 Li, Bijun III-58 Li, Bing I-602, III-621 Li, Chao I-118, II-385 Li, Chunlei I-443 Li, Cong I-614 Li, Fei II-612 Li, Haisen II-174 Li, Hejuan III-275 Li, Henan II-299 Li, Houqiang II-151 Li, Hua I-162, I-199 Li, Hui I-675 Li, Jianxiang I-512 Li, Jie III-170 Li, Kai I-567 Li, Lexin I-374 Li, Maowen I-70 Li, Miao I-545, II-262 Li, Min III-93 Li, Mingyang III-325 Li, Qiang II-206 Li, Quan-He II-274 Li, Shirui I-162, I-199 Li, Shuai II-430 Li, Tao II-3, III-254 Li, Tianpeng I-614

Author Index

Li, Tianshu III-412 Li, Wang III-194 Li, Wei-Hong I-250 Li, Wenjie III-575 Li, Wentao III-36 Li, Xiaofeng I-351 Li, Xingyue I-639 Li, Xinning III-551 Li, Yabin III-563 Li, Yanyan III-633 Li, Yaru III-445 Li, Yue III-646 Lian, Lina II-62 Liang, Dong I-23 Liao, Pin III-158 Liao, Qingmin III-264 Liao, Wenhe III-311 Lim, Samsung III-358 Lin, Chunyu III-445 Lin, Dajun II-38 Lin, Li III-287 Lin, Mingbao III-496 Lin, Songnan I-533 Lin, Tianwei I-262 Lin, Wenqian II-385 Lin, Xianming III-496 Lin, Ying III-505 Lin, Zijian II-38 Liu, Bin I-306, I-316, I-328, I-687 Liu, Chaodie I-443 Liu, Cheng-Lin I-408, I-590 Liu, Feng II-14 Liu, Hanchao I-149 Liu, Hanlin III-527 Liu, Hao II-590 Liu, Jian I-211, III-656 Liu, Jijun III-3 Liu, Jing I-420 Liu, Juan-ni III-539 Liu, Kaipin III-656 Liu, Kun II-580 Liu, Leyuan I-651 Liu, Ligang I-1 Liu, Lintao III-527 Liu, Longzhong II-513 Liu, Mingyang I-512 Liu, Ning I-127 Liu, Peixin I-351 Liu, Qiankun I-316 Liu, Qiao III-113

Liu, Qing II-405 Liu, Shaojun III-264 Liu, Shiguang II-560 Liu, Shuaicheng I-501, III-204, III-390 Liu, Tingting III-311 Liu, Wenyu I-238 Liu, Xiaoman III-3 Liu, Yang II-487, II-528 Liu, Yiguang I-295 Liu, Yongkang I-466 Liu, Yu I-420 Liu, Yuanyuan II-229 Liu, Yue I-512 Liu, Yun III-370 Liu, Zhi I-139 Liu, Zhijin I-567 Liu, Zhoufeng I-443 Long, Jiangtao I-58 Lu, Hong-bing II-528 Lu, Yuliang III-527 Luo, Bin II-119 Luo, Jiebo III-483 Luo, Shang I-284 Luo, Ziquan II-86 Lv, Chao II-430 Lv, Pin III-182 Ma, Dongdong III-264 Ma, Feilong III-102 Ma, Haosen II-487 Ma, Hongqiang II-430 Ma, Huimin I-489, II-139, II-417 Ma, Leiming II-440 Ma, Lizhuang III-70 Ma, Shanshan I-501 Ma, Shi-ping II-274, II-430, II-580 Ma, Siwei II-286 Ma, Wei III-517 Ma, Yupeng I-626 Ma, Zhuoqi III-170 Mao, Qi II-286 Mao, Yafang I-250 Meng, Xinrui I-454 Miao, Huanghui I-23 Min, Weidong I-340 Min, Xin I-118 Mo, Hanlin I-199 Mu, Xufu I-533 Mu, Zhichun II-405 Mughees, Atif III-347

671

672

Author Index

Ngai, Edith II-325 Ni, Rongrong III-575 Pan, Bin III-325 Pan, Jing II-505 Pan, Shijie III-231 Pang, Qingyu II-570 Pang, Xiaoli II-455 Pang, Xiaoyan I-118 Pang, Yanwei II-505 Peng, Hu I-35 Peng, Jinye I-555, I-626, III-231 Peng, Junkai III-182 Peng, Lingbing III-204 Peng, Yuxin I-477 Piuri, Vincenzo I-211, III-656 Qi, Hongyu II-163 Qi, Zhe-ting III-47 Qiao, Kan I-118 Qin, Manjun I-664 Qiu, Yuwei II-139 Rehman, Sadaqat ur II-109 Ren, Jingru I-139 Ren, Zhihang III-204 Rymut, Boguslaw III-423 Scotti, Fabio I-211, III-656 Shan, Jiaxin II-97 Shan, Wei I-139 Shang, Lijuan III-656 Shen, Liquan I-567 Shen, Yunhang III-496 Sheng, Tingting III-601 Shi, Jing II-129 Shi, Li III-311 Shi, Tingting II-513 Shi, Xiangfu II-24, II-74 Shi, Yanan III-243 Shi, Yuying III-113 Shi, Zhenwei III-325 Shuai, Yuan II-197 Song, Jiarong III-15 Song, Shide I-545 Song, Yonghong II-252 Su, Juan I-602, III-621 Su, Xu I-224 Sun, Chao III-390

Sun, Sun, Sun, Sun, Sun, Sun, Sun, Sun,

Guangling I-139 Jie II-151 Jinqiu II-174 Nongliang II-496 Shouqian I-118, II-385 Weidong II-570 Xiaofan II-590 Xueqi I-85

Tamura, Shinichi II-476 Tan, Bi-Ren I-408 Tan, Weimin I-97 Tan, Zhentao I-328 Tang, Jin II-119 Tang, Jun I-23 Tang, Wen III-275 Tang, Wenyi I-306 Tang, Wenzhong III-370 Tang, Xiaoan III-194 Tang, Yunqi I-48, III-589 Tao, He-Meng III-124 Tao, Linmi III-102, III-347 Tian, Huawei I-48, III-589 Tian, Qiang II-528 Wan, Shouhong I-639 Wan, Song III-527 Wan, Weitao I-614 Wang, Aobo II-299 Wang, Bin I-284, III-287 Wang, Bo III-563 Wang, Chen II-602, III-370 Wang, Chengyi II-552 Wang, Di III-254 Wang, Guijin III-36 Wang, Hanzi I-11 Wang, Jian II-505 Wang, Jin I-374 Wang, Jing II-129 Wang, Jinke II-476 Wang, Li II-216 Wang, Mu II-353 Wang, Na III-158 Wang, Nannan III-170 Wang, Nian I-23 Wang, Rong I-466 Wang, Runhua I-432 Wang, Ruyan II-336 Wang, Shanshe II-286

Author Index

Wang, Shen III-527 Wang, Shigang II-229, II-299, III-412 Wang, Shiping II-580 Wang, Weining II-513, III-483 Wang, Xiaodan II-369 Wang, Xiaotong III-214 Wang, Xihan I-555 Wang, Xin I-186, III-445 Wang, Xing I-11 Wang, Xinggang I-238 Wang, Xiumei I-397 Wang, Yanzhao III-621 Wang, Yidong I-489 Wang, Yongfang II-197 Wang, Yongzhuang II-51 Wang, Zhengbing I-523 Wang, Zhengning I-501, III-390 Wang, Zhengsheng I-523 Wei, Jia-yuan III-539 Wei, Shikui III-445 Wei, Xiaohui II-455 Wei, Xin II-313 Wei-dong, Xu II-545 Wen, Wei III-134 Wu, Ancong I-250 Wu, Chunhong III-243 Wu, Dapeng II-336 Wu, Di II-325 Wu, Dongpeng II-580 Wu, Fen III-287 Wu, Hefeng I-127 Wu, Jian II-197 Wu, Kun I-85 Wu, Li-fang III-458 Wu, Lin I-58 Wu, Liyang II-580 Wu, Na II-262 Wu, Wei I-602, III-412, III-621 Wu, Xi III-325 Wu, Xiaofeng I-284 Wu, Xiao-Jun I-675 Wu, Xinxiao I-577 Wu, Yi-Chao I-408 Wu, Yunfei III-646 Xia, Wu III-102 Xia, Zhaoqiang I-555, I-626, III-231 Xiang, Zhang II-545 Xiao, Changlin I-162 Xiao, Chuang-Bai III-124

Xiao, Fan I-11 Xiao, Guobao I-11 Xiao, Jinsheng III-58 Xiao, Yanhui I-48, III-589 Xie, Fengying I-664 Xie, Hongwen III-93 Xie, Shaobiao III-325 Xie, Tingting II-187, III-27 Xie, Zhifeng III-70 Xin, Peng II-430 Xing, Yuhang III-93 Xiong, Hongkai II-51 Xu, Changqiao II-353 Xu, Chun-yu III-383 Xu, Dong-Bin III-124 Xu, Guanlei III-214 Xu, Guili I-523 Xu, Leilei II-163 Xu, Pei I-374 Xu, Tao I-262 Xu, Ting-Bing I-590 Xu, Wei I-512 Xu, Wei-dong III-383 Xu, Xiangmin II-513, III-483 Xu, Xiaogang III-214 Xu, Xiao-pan II-528 Xu, Ying I-211, III-656 Xu, Yuelei II-430 Xu, Zhenyu I-295 Xu, Zhongwai III-93 Xue, Aijun II-369 Xue, Danna II-174 Xue, Dixiu III-337 Xue, Di-Xiu III-611 Xue, Jianru II-3, III-254 Xue, Shan II-129 Xue, Ya-dong III-47 Yan, Bo I-97 Yan, Jingwen II-187, III-27 Yan, Xuehu III-527 Yan, Yan I-11 Yan, Yongluan I-238 Yang, Chuxi II-229, II-299 Yang, Guangfei I-23 Yang, Hang I-85 Yang, Huazhong III-36 Yang, Ming I-262 Yang, Peipei I-590 Yang, Su I-108

673

674

Author Index

Yang, Taotao III-390 Yang, Wenfei I-687 Yang, Xin I-238 Yang, Yan II-336 Yang, Yuchen III-517 Yang, Zeng-yue II-528 Yang, Zhong III-15 Yang, Zhongliang II-109 Yao, Leiyue I-340 Yao, Mingyu I-577 Ye, Changkun II-417 Ye, Daoming III-337 Ye, Guoqiao II-325 Ye, Xulun II-24, II-74 Ye, Zhaoda I-477 Yilmaz, Alper I-162 Yin, Bangjie I-639 Yin, Fei I-408 Yin, Xuanwu III-36 Yin, Yong III-82 Ying, Zilu I-211, III-656 Yingyue, Zhou I-224 You, Quanzeng III-483 You, Shaodi II-417 Yu, Chen II-405 Yu, Chunlei I-70 Yu, Jing II-570, III-124 Yu, Nenghai I-306, I-316, I-328, I-687, III-471 Yu, Wangsheng I-186 Yu, Yang II-97 Yuan, Jing II-528 Yuan, Li II-405 Yuan, Yuan I-545, II-262 Yuan, Zhenguo II-187 Yue, Anzhi II-552 Zeng, Bing III-204, III-390 Zeng, Junying I-211 Ze-wei, Duan II-545 Zhai, Congcong II-239 Zhai, Yikui I-211, III-656 Zhan, Bichao III-621 Zhang, Anqin I-108 Zhang, Changdong III-311 Zhang, Changjiang II-440 Zhang, Guo-peng II-395, II-528 Zhang, Guowen III-563 Zhang, Hongchao II-560 Zhang, Huying III-58 Zhang, Jian I-545

Zhang, Jianxin I-1 Zhang, Jingjing II-602 Zhang, Jiulong I-108 Zhang, Junda III-194 Zhang, Kai II-417 Zhang, Kaige I-273 Zhang, Lijun II-602 Zhang, Liming I-284 Zhang, Lin I-420 Zhang, Lizhong III-412 Zhang, Long II-24, II-74 Zhang, Ning III-325 Zhang, Qin III-82 Zhang, Rong III-337, III-611, III-646 Zhang, Sen III-47 Zhang, Shiyang III-517 Zhang, Tao III-149 Zhang, Tianyi III-15 Zhang, Wei I-174 Zhang, Weiming III-471 Zhang, Weishan I-108 Zhang, Wenjun III-70 Zhang, Wenyao III-158 Zhang, Xi II-528 Zhang, Xiaoqin II-417 Zhang, Xinfeng I-108 Zhang, Xinsheng II-590 Zhang, Xiuwei II-612 Zhang, Xuebo I-432 Zhang, Xulei II-430 Zhang, Xu-Yao I-590 Zhang, Yanbin I-35 Zhang, Yanning II-174, II-612 Zhang, Yi I-174, I-362, I-386, II-405 Zhang, Yuanlin II-252 Zhang, Yu-Jin II-109 Zhang, Yunfei II-239 Zhang, Zengshuo I-70 Zhao, Baojun I-70 Zhao, Cheng III-400 Zhao, Fuqiang III-400 Zhao, Houlong II-496 Zhao, Huimin II-187, III-27 Zhao, Jie III-134 Zhao, Jieyu II-24, II-74 Zhao, Li-dong III-458 Zhao, Wen III-158 Zhao, Xu I-262 Zhao, Yan II-229, II-299 Zhao, Yao III-436, III-445, III-575

Author Index

Zhao, Yi I-651 Zhao, Zhenbing II-163 Zheng, Changwen III-182 Zheng, Huicheng II-38, II-62, II-86 Zheng, Jin I-11 Zheng, Ling I-374 Zheng, Mana III-517 Zheng, Wei-Shi I-250 Zheng, Yunan I-295 Zhong, Ping II-97 Zhong, Yang III-254 Zhou, Huabing I-651 Zhou, Jian III-58 Zhou, Jun III-370 Zhou, Lijia III-214 Zhou, Ning I-351

Zhou, Quan III-539 Zhou, Wengang II-151 Zhou, Xingyu III-243 Zhou, Yipeng II-325 Zhou, Zhiping III-551 Zhu, Hong II-129 Zhu, Ming I-85 Zhu, Mingning II-430 Zhu, Rui II-86 Zhu, Shuyuan III-204 Zhu, Yonggui III-113 Zhu, Yu II-174 Zhu, Yun II-197 Zou, Chang I-639 Zou, Junni II-239 Zu, Hongliang II-476

675