Deformable Registration Techniques for Thoracic CT Images: An Insight into Medical Image Registration [1st ed.] 9789811058363, 9789811058370

Citation preview

Ali Imam Abidi · S. K. Singh

Deformable Registration Techniques for Thoracic CT Images An Insight into Medical Image Registration

Deformable Registration Techniques for Thoracic CT Images

Ali Imam Abidi S. K. Singh •

Deformable Registration Techniques for Thoracic CT Images An Insight into Medical Image Registration

123

Ali Imam Abidi Department of Computer Science and Engineering School of Engineering and Technology Sharda University Greater Noida, Uttar Pradesh, India

S. K. Singh Department of Computer Science and Engineering Indian Institute of Technology, BHU Varanasi, Uttar Pradesh, India

ISBN 978-981-10-5836-3 ISBN 978-981-10-5837-0 https://doi.org/10.1007/978-981-10-5837-0

(eBook)

© Springer Nature Singapore Pte Ltd. 2020 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Contents

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

1 1 7 10 11 12

2 Theoretical Background . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Morphological Classification of Images . . . . . . . 2.2.1 Rigid Images . . . . . . . . . . . . . . . . . . . . . 2.2.2 Deformable Images . . . . . . . . . . . . . . . . 2.3 Geometric Deformation Models: A Survey . . . . . 2.4 Classification of Registration Methodology Used 2.4.1 Feature-Based Registration . . . . . . . . . . . 2.4.2 Intensity-Based Registration . . . . . . . . . . 2.5 Feature Detection/Description Methods . . . . . . . 2.6 Database Employed . . . . . . . . . . . . . . . . . . . . . 2.7 Accuracy and Similarity Measures Used . . . . . . 2.7.1 Target Registration Error . . . . . . . . . . . . 2.7.2 Signal-to-Noise Ratio (SNR) . . . . . . . . . 2.7.3 Peak Signal-to-Noise Ratio (PSNR) . . . . 2.7.4 Structural Similarity Index (SSIM) . . . . . 2.7.5 Normalized Cross-Correlation (NCC) . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

15 15 16 16 19 23 27 27 30 32 34 36 36 38 39 39 40

3 A Moving Least Square Based Framework for Thoracic CT Image Registration . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

41 42 42 46

1 Introduction . . . . . . . . . . . . . 1.1 Background . . . . . . . . . . 1.2 Motivation . . . . . . . . . . . 1.3 Objective of the Book . . . 1.4 Contributions . . . . . . . . . 1.5 Organization of the Book

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

v

vi

Contents

3.3.1 Preparation . . . . . . . . . 3.3.2 Proposed Methodology 3.4 Results and Discussion . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

46 46 50 52

4 A Path Tracing and Deformity Estimation Methodology for Registration of Thoracic CT Image Sequences . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Proposed Methodology . . . . . . . . . . . . . . . . . 4.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

55 56 58 61 61 62 66 76

5 Deformable Thoracic CT Images Sequence Registration Using Strain Energy Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Proposed Methodology . . . . . . . . . . . . . . . . . . . . . . 5.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

77 78 79 81 81 82 85 89

6 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Scope for Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

91 91 92

Appendix A: Geometrical Deformation Models for Elastic Images . . . . .

93

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

Appendix B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Appendix C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Appendix D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Appendix E . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

About the Authors

Dr. Ali Imam Abidi is an Assistant Professor at the Department of Computer Science & Engineering, School of Engineering & Technology, Sharda University. He received his Ph.D. from the Department of Computer Science and Engineering, Indian Institute of Technology, Banaras Hindu University (IIT-BHU), Varanasi, India. His primary areas of research include (but is not limited to) deformable image registration, image feature data analysis etc. as well as the study and applications of behavioural design concepts. He has published numerous research articles, letters, papers in conference proceedings and book chapters. He has been a constant invited reviewer for reputed journals like MTAP, MBEC, NAS Letters etc Prof. S. K. Singh holds a B.Tech. in Computer Engineering, M.Tech. in Computer Applications and a Ph.D. in Computer Science and Engineering. Currently, he is a Professor at the Department of Computer Science and Engineering, Indian Institute of Technology, Banaras Hindu University (IIT-BHU), Varanasi, India, and is also a Certified Novell Engineer Administrator. He is a member of LIMSTE, the Institute of Electrical and Electronics Engineers (IEEE), the International Association of Engineers (IAENG) and the International Center of Sustainable Excellence (ISCE). His research areas include biometrics, computer vision, image processing, video processing, pattern recognition and artificial intelligence. He has published over 50 articles in national and international journals, as well as book chapters and conference papers.

vii

Chapter 1

Introduction

1.1 Background Image Registration is defined as the process of establishing correspondences between two images. It is the process of aligning images so that corresponding features can easily be related and the best structural superimposition can be achieved. The term is also used to mean aligning images with a computer model or aligning features in an image with locations in physical space. The images might be acquired with different sensors (e.g., sensitive to different parts of the electromagnetic spectrum) or the same sensor at different times. The present differences between images are introduced due to different imaging conditions. Image registration is a crucial step in all image analysis tasks in which the final information is gained from the combination of various data sources like in image fusion, change detection, and multichannel image restoration. Typically, registration is required in remote sensing (multispectral classification, environmental monitoring, change detection, image mosaicing, weather forecasting, creating super-resolution images, integrating information into geographic information systems [GIS]), in medicine (combining computer tomography [CT], and NMR data to obtain more complete information about the patient, monitoring tumor growth, treatment verification, comparison of the patient’s data with anatomical atlases), in cartography (map updating), and in computer vision (target localization, automatic quality control), to name a few. In general, its applications can be divided into four main groups depending upon the manner of image acquisition: Acquisition from different viewpoints (multi-view analysis): In this category images of the same scene are acquired from different viewpoints/angles. The aim is to gain a larger two-dimensional view or a three-dimensional representation of the scanned/acquired scene. Some examples are computer vision-shape recovery (shape from stereo), Remote sensing-mosaicing of images of the surveyed area, etc., some examples are shown in Figs. 1.1 and 1.2.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0_1

1

2

1 Introduction

Fig. 1.1 Multi-view capture of the same scene and its registered image showing the difference between two viewpoints. (Used from the original “head and lamp” stereo scene pair along with the transition released by University of Tsukuba in 1997 comprises the Tsubuka stereo dataset [1800 stereo pairs] http://www.cvlab.cs.tsukuba.ac.jp/dataset/tsukubastereo.php)

Images acquired at different timestamps (multi-temporal image analysis): In this categorization, images of the same scene are acquired at different times from the same viewing angle and same acquisition apparatus, often on regular basis and possibly under different conditions. The aim is to find and evaluate changes in the scene which seem to have happened between the consecutive image acquisitions. Some of the examples are remote sensing-monitoring of global land usage, landscape planning, computer vision-automatic change detection for security monitoring, motion tracking. Medical imaging based monitoring of the healing therapy, monitoring of the tumor evolution, etc. Some examples are shown using Figs. 1.3, 1.4 and 1.5. Images acquired using different sensors (multi-modal analysis): When images of the same scene are acquired by different sensors while muting other variables such as time gap, viewing angle, etc., and analyzed, it falls into the multi-modal analysis category. The aim is to integrate the information obtained from different source streams

1.1 Background

3

Fig. 1.2 Another sample that can be taken as an example for multi-view image registration; first two are the aerial shots of a single scene at different angles, third is the registered image. (A digital aerial photograph [geometrically uncorrected] to a digital orthophoto [supplied by the Massachusetts Geographic Information System (MassGIS), has been orthorectified] covering the same area centered on the business district of West Concord, Massachusetts, USA https://in.mathworks. com/discovery/image-registration.html)

Fig. 1.3 Registration of two images with temporal change (lesser number of trailers parked) (http:// old.vision.ece.ucsb.edu/registration/demo/ex3.shtml)

to gain more complex and detailed scene representation. Examples of applications: Remote sensing-fusion of information from sensors with different characteristics like panchromatic images, offering better spatial resolution, color/multispectral images with better spectral resolution, or radar images independent of cloud cover and solar illumination. Medical imaging-combination of sensors recording the anatomical body structure like magnetic resonance image (MRI), ultrasound, or CT with sensors monitoring functional and metabolic body activities like positron emission tomography (PET), single-photon emission computed tomography (SPECT), or magnetic resonance spectroscopy (MRS). Results can be applied, for instance, in radiotherapy and nuclear medicine. Some examples are shown in Figs. 1.6, 1.7 and 1.8.

4

1 Introduction

Fig. 1.4 Two years temporal progress of deforestation can be observed in Amazonian Rainforests (http://old.vision.ece.ucsb.edu/registration/demo/ex4.shtml)

Scene to model registration: When images of a scene are registered with a computer-generated model of that scene. This model can be a visual representation of the scene, for instance, maps or digital elevation models (DEM) in GIS, another scene with similar content (e.g., another patient), “average” specimen, etc. The aim is to localize the acquired image in the scene/model and/or to compare them. Some examples are remote sensing-registration of aerial or satellite data into maps or other GIS layers. Creating Panoramas from the end to end stitching of acquired images. In computer vision, target template matching with real-time images, automatic quality inspections, etc. are good examples. In case of medical imaging, a comparison of the patient’s image with digital anatomical atlases, specimen classification is an application. An example of image stitching is depicted in Fig. 1.9. Though there can be more, usually there are two images involved in the process of image registration. One of them is called the moving image denoted by M or source image and is denoted by S, this image is to be registered against a fixed image indicated as F or a target image denoted by T. An example has been shown in Fig. 1.10. Image registration can also be explained in terms of forward transform mapping, i.e., mapping of points from the physical space of the fixed image into the physical space of the moving image. This is shown in Fig. 1.11. This implies that the transform will accept as input points from the fixed image and it will compute the coordinates of the analogous points in the moving image. What tends to create confusion is the fact that when the transform shifts a point on the positive X direction, the visual effect of this mapping, once the moving image is re-sampled, is equivalent to manually shifting the moving image along the negative X direction. In the same way, when the transform applies a clock-wise rotation to the fixed image points, the visual effect of this mapping once the moving image has been re-sampled is equivalent to manually rotating the moving image counter-clock-wise. The reason why this direction of mapping has been chosen for implementation of registration framework widely is that this is the direction that better fits the fact that the moving image is expected to be re-sampled using the grid of the fixed image.

1.1 Background

5

Fig. 1.5 Brain tumor (cancerous) size reduction over a time of 8 months for a baby diagnosed at 8 months of age (http://www.inquisitr.com/423292/babys-brain-tumor-gone-after-dadputs-marijuana-on-pacifier-video/)

The nature of the re-sampling process is such that an algorithm must go through every pixel of the fixed image and compute the intensity that should be assigned to this pixel from the mapping of the moving image. This computation involves taking the integral coordinates of the pixel in the image grid, usually called the “(i, j)” coordinates, mapping them into the physical space of the fixed image (transform T1 in Fig. 1.11), mapping those physical coordinates into the physical space of the moving image (transform to be optimized), then mapping the physical coordinates of the moving image into the integral coordinates of the discrete grid of the moving image (transform T2 in the figure), where the value of the pixel intensity will be computed by interpolation.

6

1 Introduction

Fig. 1.6 MRI and CT image representation of the same axial section showing soft tissues and bony structures, respectively. The third is fusion of these both images finding the most optimal transformation (http://docplayer.net/21680192-Image-registration-and-fusion-professormichael-brady-frs-freng-department-of-engineering-science-oxford-university.html)

CT

PET

Fig. 1.7 CT image, PET image, and fusion of both coronal slices of the thoracic section to find the best view possible (http://docplayer.net/21680192-Image-registration-and-fusion-professormichael-brady-frs-freng-department-of-engineering-science-oxford-university.html)

1.1 Background

7

Fig. 1.8 PET image, MR image volume, and fusion of both coronal slices of the human brain to find the best representation possible (http://docplayer.net/21680192-Image-registration-and-fusionprofessor-michael-brady-frs-freng-department-of-engineering-science-oxford-university.html)

Registration of a moving image IM (x, y) to a fixed image IF (x, y) both of dimension D, is the problem of finding a displacement field u(x) that makes IM (x + u(x)) spatially aligned to IF (x). The obtained transformation is defined as T (x) = x + u(x) If the underlying transformation model allows local deformations, i.e., nonlinear displacement fields u(x), then it is called Deformable Image Registration (DIR).

1.2 Motivation Many exciting potential applications of deformable image registration (DIR) have been found in diagnostic medical imaging and radiation oncology. Automated propagation of physician-drawn contours to multiple image volumes, functional imaging,

8

1 Introduction

Fig. 1.9 End to end stitching of two images at pre-designated points to create a Panorama. Reproduced from Zitova et al. (2003)

Fig. 1.10 Source (fixed), target (moving), and the registered image (CT and MRI are combined in a single image) (http://kevin-keraudren.blogspot.in/2014/12/medical-image-analysis-ipythontutorials.html)

1.2 Motivation

9

Fig. 1.11 Different coordinate systems involved in the image registration process. Note that the transform being optimized is the one mapping from the physical space of the fixed image into the physical space of the moving image. Reproduced from https://www.orfeo-toolbox.org/ SoftwareGuide/SoftwareGuidech9.html#x33-1350009.3

and 4D dose accumulation in thoracic radiotherapy are just a few examples. However, before such applications can be successfully and safely implemented, it is required that the DIR spatial accuracy performance be rigorously and objectively assessed. Objective evaluation of DIR is an active area of research. A framework for DIR evaluation is an essential utility for algorithm optimization, performing comparisons between algorithms, models, and implementations, acceptance testing prior to clinical implementation, and quality assurance of DIR on a routine basis. Based on many previously reported frameworks for DIR evaluation based on manual identification of large sets of prominent image features between image volume pairs it has been demonstrated that considerable misrepresentation of DIR spatial accuracy performance characteristics can result from analyses based on inadequate landmark sample size and distribution. Also, it has been shown that large feature point samples though slow down the registration process but more importantly facilitate thorough characterization of spatial accuracy performance in terms of clinically relevant parameters, such as relative spatial location or displacement magnitude. Analyses based on landmark samples

10

1 Introduction

that are not sufficient in size, or that are biased towards structures that are generally easier to identify risk misrepresentation of the actual spatial accuracy performance of an algorithm [dir-lab/motivation]. These considerations make objective comparison of published DIR spatial accuracies difficult to interpret and potentially misleading. Therefore, it seemed only appropriate and justified to investigate and make comparisons between various reported/published DIR models with the ones proposed in this work on the basis of common image database and error analysis metrics. Similar motivational issues are listed below point wise: • Some of the hardest problems in deformable image registration are problems where large anatomical differences occur between image acquisitions. • These would be large deformations due to images acquired in prone and supine positions and (dis)appearing structures between image acquisitions. • To find the transformation that aligns the source best with the target image irrespective of the modality(ies) employed. • By best alignment, the idea comes down to obtaining the optimum solution noninvasively in a reasonable time. • The tuning of the developed methods to specific problems (i.e., how to best combine different objectives such as similarity measure and transformation effort). This is one of the reasons why, despite significant progress, clinical implementation of such techniques has proven to be difficult. Due to uneven stress on a particular deformable image modality from the field of radiotherapy, there are invariably scattered databases available from different modalities and of different anatomical parts of the body, which hinders a proper study on different modality investigations with the aim of common disease diagnosis.

1.3 Objective of the Book The objective of the proposed work in this book is to accurately register thoracic image pairs and image sequences for ten subjects at hand over a complete breathing cycle from full inhale to full exhale positions for all three anatomical positions, i.e., Axial, Coronal, and Sagittal. This is desired to be achieved by applying geometrical transformation based image registration methodologies on deformable image pairs as well as image sequences for all the test subjects through all three APs. The objective of this work has been listed out in points below for clarity and precision as follows: • To develop a deformable Registration model for Thoracic CT image sequences across multiple subjects (10 subject data has been taken as a sample). • This model would help in accounting for the large anatomical differences happening in a subject over time.

1.3 Objective of the Book

11

• The deformation could be assessed and could be used as a prerequisite (correcting z-error) for radiotherapies, etc. • The same deformation can be used to put up a comparative study between multiple subjects into assess the possible presence and extent of chronic pulmonary disorders etc. (a part of this study).

1.4 Contributions Physical modeling based on geometric transforms can provide accurate results for medical image deformations, but its application in the field of deformable image registration is limited due to the difficulties in determining precise boundary conditions of the image mesh. In this work, three specific and different methodologies have been proposed to assess the deformation happening in thoracic region of ten subjects during a course of breathing, i.e., from full inhale to full exhale with minimum image registration error. The methods have been proposed in a simplified way to gain better and more realistic deformed images without any a priori knowledge on the boundary conditions. The three methods which are also our primary contribution to the work are as follows: • A comprehensive literature review and comparative study of various classical as well as state-of-the-art methods for deformable image registration under varying acquisition conditions. Further, design of new and efficient algorithms for registration of thoracic CT images. • A common feature point set correspondence based application of the concept of least squares to assess the transformation required in the full inhale (moving image) with respect to the fixed image for them to get the best alignment. In turn, registering the moving image onto the fixed image as a result. The proposed method is compared against contemporary methods on the basis of an accuracy metric called the target registration error to establish relevance. • A novel deformable image sequence registration methodology; a common feature point cloud is described between frames of the sequence. The position of those feature landmark points in every frame help in the assessment of the deformation. The changing position of those common landmark points in all frames determine the flow (optical motion) and thus the deformation in the image, this helps in transformation required for the moving image with respect to the reference image (fixed image). There has to be a reference frame out of the sequence pre-decided before determining the optical flow. Assessment of the flow would give the clear movement of points across frames, it is depicting using trails/path the points leave behind while deformation. This would give a clearer picture into the motion of a human thorax during breathing from full inhale to full exhale phase noninvasively. Also, we can estimate the displacement of each of these landmark points having obtained their optical flow (velocity) with respect to the runtime of image sequence in question.

12

1 Introduction

• An automatic deformable thoracic ct image registration using strain energy minimization. Medical images are broadly categorized as elastic or deformable images; in this work, the deformations in the elastic images are modeled after already established theories of elasticity by Navier–Cauchy, etc. An image pair is considered as a system, in case of perfect alignment of moving image with the fixed image the energy of this system is supposed to be at an optimum minimum. Strain energy minimization is the transformation applied to the moving image for that optimum alignment. Strain energy of the system at hand is minimized iteratively and checked using minimum intensity difference metric as validation to break the loop. Without using points or features, large image stacks can be registered using this method faster than point/feature-based methods.

1.5 Organization of the Book This book consists of six chapters. An outline of the chapters is as follows: In this chapter: Here a brief introduction of the topic starting with basic background information has been presented, followed by the motivation and objectives of the book. Finally, the chapter concludes with a detailed list of contributions this work provides in the field of deformable image registration. Chapter 2: Concepts and theoretical background behind the topic of image registration as a whole and then deformable image registration specifically have been presented in this chapter. Starting with a basic introduction, it explains morphological classification of images and relevance of this information in the proposed works. Based on the morphological properties of images, a survey of deformable image registration methods based on geometric image deformation models has been presented. Small preludes of registration methodologies proposed in the book have been provided. Feature detectors/description methods used are explained with proper justification for their use. Then the image database used in the proposed methods is discussed, database dimensions, acquisition details, ethical issues that come along with medical image acquisition, etc. Lastly, accuracy/similarity metrics employed to assess the proposed registration methodologies. Chapter 3: This chapter presents a moving least squares approach for deformable image registration of thoracic ct images using common landmark point sets. It starts with a small introduction to the topic, followed by a background study, preparation for, and description of the implementation of the methodology. This is followed by results and discussion and finally conclusion. Chapter 4: A point correspondence path tracing and deformity estimation methodology for registration of thoracic ct image sequences from full inhale to full exhale positions has been proposed in this chapter. It starts with a small introduction to the topic, followed by a background study, preparation for and description of the implementation of the methodology. This is followed by results and discussion and finally conclusion.

1.5 Organization of the Book

13

Chapter 5: An automatic deformable thoracic ct image sequence registration using strain energy minimization has been coined in this chapter. It starts with a small introduction to the topic, followed by a background study, preparation for and description of the implementation of the methodology. This is followed by results and discussion and finally conclusion. Chapter 6: It carries conclusions and summarizes the main findings of this work. This chapter also proposes some possible future perspectives of the research work conducted so far.

Chapter 2

Theoretical Background

Before getting on with the proposed methods, it seems elementary to mention and explain some concepts which are to be used extensively in the coming chapters in detail. This chapter presents the building blocks which are necessary to understand the concepts presented and discussed in the forthcoming chapters. Here we elaborate on the images classifications based on which they are up for different registration methodologies depending on their morphological properties. This would lead to a discussion on geometrical transformation models inspired from physical models. A standalone description of the registration algorithms used in the methods proposed the feature detection/description method(s) used, the database employed, and registration accuracy and similarity measures would be discussed. This chapter would help in setting just the right tone for upcoming content.

2.1 Introduction The need for establishing this distinction between rigid and deformable images before talking about transformations in both these kinds of images is imperative. The application and use of registration algorithms to these images vary and differ both on the basis of viability (time and space complexity) and the nature of transformations that suit best for the image(s) it is being applied for. In the coming chapters, there will be detailed propositions of registration methodologies for different image sets. To keep the discussion of these algorithms relevant, it was necessary to detail the morphological properties of these varieties of images (with real-life examples). The geometric transformation models proposed in the past for deformable images are discussed; they take inspiration from physical models such as Navier–Cauchy’s theories of dynamics in elastic and near-elastic objects. It is quite interesting to see the similarities between these elasticity models and deformable images of various human body organs.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0_2

15

16

2 Theoretical Background

2.2 Morphological Classification of Images Image registration methods, in general, have been applied to a variety of kinds of images. These images have been classified into two types, namely rigid and deformable images depending on their temporal behavior. These methods and their application depend on the morphological properties of the images it is being used for. For instance, an image registration technique that provides optimal results for rigid images both in terms of registration error and run time/space complexity might fail on all aspects when applied to deformable images. Similarly, an IR algorithm giving optimal results for deformable images in aforementioned terms might be a waste of run time/space complexity and resources when applied to rigid images, as the same results could have been obtained using smaller resources had rigid image-specific IR algorithm been used.

2.2.1 Rigid Images Rigid images are most commonly those of structures with rigid morphological properties, e.g., bones, buildings, geographical structures, etc. Images that do not exhibit morphological changes such as warps etc. over a period of time can be classified as rigid images. Rigid images can be modeled after real-life real objects with the least elasticity. To understand the behavior of rigid images, imagine an image as a collection of innumerable small minuscule points, the image will be a rigid image if there is no (ideally) relative motion between the points under deformations as can be seen in Fig. 2.1. Figure 2.2, is an example from the “Möbius Transformations Revealed” page illustrating the rigid motion of a Riemann sphere through different transformations. The graticule of lines that we see on the plane and on the sphere form an image of a square in the complex plane. It is a projection of the sphere on the two-dimensional plane. The first row depicts the translation transformation; Fig. 2.2a, c are the last initial and final stages of the transformation while (b) is an intermediary stage. These transformations can be formulated in terms of matrices, so as to make its theoretical understanding, development, and applications easy. The elegance of formulating these transformations in terms of matrices is that several of them can be combined, simply by multiplying the matrices together to form a single matrix. This means that repeated re-sampling of data can be avoided when re-orienting an image. As for translation (Fig. 2.2a–c); suppose a point x in an image is to be translated by q units, then the transformation would be simply y = x +q

2.2 Morphological Classification of Images

17

Fig. 2.1 Rigid image morphological behavior under deformation. Reproduced from https://www. slicer.org/wiki/Slicer3:Registration

which in matrix terms would be considered as ⎤ ⎡ 1 y1 ⎢ y2 ⎥ ⎢ 0 ⎢ ⎥=⎢ ⎣ y3 ⎦ ⎣ 0 1 0 ⎡

0 1 0 0

⎤⎡ ⎤ 0 q1 x1 ⎥ ⎢ 0 q2 ⎥⎢ x2 ⎥ ⎥ 1 q3 ⎦⎣ x3 ⎦ 0 1 1

(2.1)

Looking into rotation (Fig. 2.2d–f), consider a point at coordinate (x 1 , x 2 ) on an image. A rotation of this point to new coordinates (y1 , y2 ), by θ radians around the origin, can be generated by the transformation: y1 = cos(θ )x1 + sin(θ )x2 y2 = cos(θ )x2 − sin(θ )x1 This can be loosely related to the affine transformation in non-rigid images. For the three-dimensional cases, there are three orthogonal planes that an object can be rotated in. These planes of rotation are normally expressed as being around the axes. A rotation of q1 radians about the first (x) axis is normally called pitch and is performed by ⎤ ⎡ ⎤⎡ ⎤ 1 0 0 0 y1 x1 ⎢ y2 ⎥ ⎢ 0 cos(q1 ) sin(q1 ) 0 ⎥⎢ x2 ⎥ ⎢ ⎥=⎢ ⎥⎢ ⎥ ⎣ y3 ⎦ ⎣ 0 − sin(q1 ) cos(q1 ) 0 ⎦⎣ x3 ⎦ 1 1 0 0 0 1 ⎡

(2.2)

18

2 Theoretical Background

Fig. 2.2 Rigid image motions as shown in Mobius Transformations using a Riemann Sphere; a–c is the translation motion, d–f is the rotation motion, g–i is the zoom or dilation, j–o is the inversion and p–r is a combination of them all. Reproduced from http://visualizingmath.tumblr.com/post/ 116773525226/spacepetals-mobius-transformation-as-rigid

Rotations are combined by multiplying this matrix together with the other planar matrices in the appropriate order. The order of operations is important. The transformations described so far will generate purely rigid-body mappings. Zooms/dilations (Fig. 2.2g–i) are needed to change the size of an image, or to work with images whose pixel sizes are not isotropic, or differ between images. These represent scaling along the orthogonal axes, and can be represented via

2.2 Morphological Classification of Images

⎤ ⎡ q1 y1 ⎢ y2 ⎥ ⎢ 0 ⎢ ⎥=⎢ ⎣ y3 ⎦ ⎣ 0 1 0 ⎡

19

0 q2 0 0

0 0 q3 0

⎤⎡ ⎤ 0 x1 ⎢ x2 ⎥ 0⎥ ⎥⎢ ⎥ 0 ⎦⎣ x3 ⎦ 1 1

(2.3)

Rigid image transformation has a variety of obvious applications in registering images of solid and still structures etc., acquired from different viewing angles. It is also applicable in case of medical images of anatomical parts that do not deform significantly with overtime duration. Such anatomical parts maybe bones, human brain structure, etc. The shape of a human brain changes very little with head movement, so rigid image transformations can be used to model different head positions of the same subject. Matching of two brain images whether it’s an MR or CT image is performed by finding the rotations and translations that optimize some mutual function of the images. Rigid image registration techniques have employed to monitor changes in the brain in individual subjects who underwent serial MRI examinations. This approach allows disease progression and response to treatment to be monitored with great sensitivity. It fits naturally with the non-invasive nature of MRI.

2.2.2 Deformable Images Deformable images are those of structures, shape, and size of which can be modeled after tangible physically deformable models. Images that exhibit morphological changes such as warp, shape changes, etc. when subjected to transformations and/or external forces over a period of time are categorized as deformable/nonrigid/elastic images. These images may or may not return to an original state with time. Deformable images can be modeled after real-life objects with elastic morphological properties. To understand the behavior of deformable images, imagine an image as a collection of innumerable small minuscule points, the image will be a deformable image if there is the independent relative motion of the constituent points under deformation as can be seen in Fig. 2.3. This essentially means that however the constituent points of a deformable image be connected/related to each other, under an external deformation, they might tend to lose that connection/association. A common example of deformable images is a temporal image sequence of an amoeba in motion. This is shown as visualization in Fig. 2.4 how amoeba, an inherently shapeless microorganism moves and assimilates its food (assimilation process is not a part of the image) using its pseudopods (false limbs). There are many transformations both geometric and physical which are applicable only to deformable images and provide the best emulations of real-life deformations. These are radial basis functions (thin plate or surface splines, multi-quadrics, etc.), physical continuum models (viscous fluids), and large deformation models (diffeomorphisms).

20

2 Theoretical Background

Fig. 2.3 Deformable image morphological behavior under deformation. Reproduced from https:// www.slicer.org/wiki/Slicer3:Registration

Fig. 2.4 Amoeboid motion realized as a deformable image motion. Reproduced from Akron School presentation art https://www.akronschools.org

2.2 Morphological Classification of Images

21

A quadratic transformation model is defined by second-order polynomials: ⎤ ⎡ ⎤⎡ 2 ⎤ x a00 . . . a09 x ⎢ y  ⎥ ⎢ a10 . . . a19 ⎥⎢ y 2 ⎥ ⎥ ⎢ ⎥⎢ ⎥ T (x, y, z) = ⎢ ⎣ z  ⎦ = ⎣ a20 . . . a29 ⎦⎣ . . . ⎦ 1 0...1 1 ⎡

(2.4)

Radial basis functions however like splines be it thin plate or b-splines use a linear combination of basic functions θi to describe the deformation field instead of using a polynomial as a linear combination of higher order terms. ⎤ ⎤ ⎡ ⎤⎡ a00 . . . a0n θ1 (x, y, z) x ⎥ ⎢ y  ⎥ ⎢ a10 . . . a1n ⎥⎢ ... ⎥ ⎥ ⎢ ⎥⎢ T (x, y, z) = ⎢ ⎣ z  ⎦ = ⎣ a20 . . . a2n ⎦⎣ θn (x, y, z) ⎦ 1 1 0...1 ⎡

(2.5)

A common choice is to represent the deformation field using a set of (orthonormal) basis functions such as Fourier (trigonometric) basis functions or wavelet basis functions. In the case of trigonometric basis functions, this corresponds to a spectral representation of the deformation field where each basis function describes a particular frequency of the deformation. The term spline originally referred to the use of long flexible strips of wood or metal to model the surfaces of ships and planes. These splines were bent by attaching different weights along their length. A similar concept is used to model spatial transformations. Many registration techniques using splines are based on the assumption that a set of corresponding points or landmarks can be identified in the source and target images. This is analogous to the use of point landmarks for rigid or affine registration using the Procrustes method. Thin plate splines are part of a family of splines that are based on radial basis functions. Radial basis function splines can be defined as a linear combination of n radial basis functions θ (s). T (x, y, z) = a1 + a2 x + a3 y + a4 z +

n 



b j θ ϕ j − (x, y, z)

(2.6)

j=1

As for the B-splines: u(x, y, z) =

3  3 3  

θl (u)θm (v)θn (w)ϕi+l, j+m,k+n

(2.7)

l=0 m=0 n=0

where i = x/δ−1, j = y/δ−1, j = y/δ−1, k = z/δ−1, u = x/δ−x/δ, v = y/δ−y/δ, w = z/δ−z/δ and θl represents the lth function of the B-splines.

22

2 Theoretical Background

In case of elastic deformations [Navier’s elastic PDE (partial differential equation)]: μ∇ 2 u(x, y, z) + (λ + μ)∇(∇ · u(x, y, z)) + f (x, y, z) = 0

(2.8)

Here u describes the displacement field, f is the external force acting on the elastic body, ∇ denotes the gradient operator, and ∇ 2 denotes the Laplace operator. The parameters μ and λ are Lamé’s elasticity constants which describe the behavior of the elastic body. There are a large number of applications for deformable image registration. Since almost all anatomical parts or organs of the human body are deformable structures, they come across as a most common application for deformable image registration, for example, lung motion during breathing is shown in Fig. 2.5. Areas of considerable interest for deformable image registration are the applications in which the geometry during image acquisition is unknown or distorted, and include correction for scaling, gantry tilt, and magnetic field inhomogeneity. Other areas of deformable image registration can be classified into either the registration of deformable structures of the same individual (intra-subject registration) or the registration across individuals (inter-subject registration). Due to the different nature of these image registration tasks, the algorithms developed to solve them have quite different characteristics.

Fig. 2.5 Motion of lungs during breathing. Reproduced from https://www.dir-lab.com/FRAMES2. gif

2.3 Geometric Deformation Models: A Survey

23

2.3 Geometric Deformation Models: A Survey An image registration algorithm can be divided into three main components, a deformation model, an objective function, and an optimization method. A registration algorithm’s result naturally depends on the deformation model and the objective function. The registration result’s dependency on an optimization strategy follows from the fact that image registration is inherently an ill-posed problem according to Hadamard’s definition of well-posed problems (Hadamard 2014). For example in a rigid setting, let us consider a scenario where two images of a disk (white background, gray foreground) are registered. Despite the fact that the number of parameters is only 6, the problem is ill-posed. The problem has no unique solution since a translation that aligns the centers of the disks followed by any rotation results in a meaningful solution (Sotiras et al. 2013). However, since the subject of this work is deformable images that too body organs, in these situations, in general, no closed-form solutions exist to estimate the registration parameters. In this setting, the search methods reach only a local minimum in the parameter space. The approach that one should take depends on the anatomical properties of the organ (for example, the heart and liver do not adhere to the same degree of deformation), the nature of observations to be registered (same modality versus multi-modal fusion), the clinical setting in which registration is to be used (e.g., offline interpretation versus computer-assisted surgery). The primary interest of this work lies in deformable registration hence problems with relatively higher-degree-of-freedom settings have been discussed particularly. The main scope of this work is focused on applications that seek to establish spatial correspondences between medical images and thus the organ state of which the images are recorded with respect to time. The scope of this work has been extended to cover applications where the interest is to recover the apparent motion of objects between sequences of successive images (optical flow estimation) (Fleet and Weiss 2006; Baker et al. 2011). Deformable registration and optical flow estimation are closely related problems. Both problems aim to establish correspondences between images. In the deformable registration case, spatial correspondences are sought, while in the optical flow case, spatial correspondences, that are associated with different time points, are looked for. Given data with a good temporal resolution, one may assume that the magnitude of the motion is limited and that image intensity is preserved in time, optical flow estimation can be regarded as a small deformation mono-modal deformable registration problem. The parameters that registration estimates through the optimization strategy correspond to the degrees of freedom of the deformation model (these are variational approaches in general attempt to determine a function, not just a set of parameters). There is a great variation in this number, from six in the case of global rigid transformations to millions when nonparametric dense transformations are considered. Increasing the dimensionality of the state space almost always results in enriching the descriptive power of the model. This model enrichment also brings along an increase in the model’s complexity which, in turn, results in a more challenging and

24

2 Theoretical Background

computationally demanding inference. Furthermore, the choice of the deformation model implies an assumption regarding the nature of the deformation to be recovered. The geometric transformation models inspired by physical models suggested by Modersitzki in 2004 (Modersitzki 2004) and currently being employed can be separated into three/five basic categories, i.e., elastic body models, viscous fluid flow models, diffusion models (curvature registration and flows of diffeomorphisms). Elastic body models can be further subdivided into Linear and Nonlinear models. In the case of linear models, images under deformation are modeled as an elastic body. The Navier–Cauchy Partial Differential Equation (PDE) describes this deformation. μ∇ 2 u + (μ + λ)∇(∇ · u) + F = 0 where F is the force field that drives the registration based on an image matching criterion, μ refers to the rigidity that quantifies the stiffness of the material, and λ is Lamés first coefficient. The image grid was modeled after an elastic membrane that is deformed under the influence of internal and external competing forces until a state of equilibrium is reached. The external force influences deformation in the image to achieve matching and the internal force exercises the elastic properties of the material (Broit 1981). This approach was extended in a hierarchical fashion by Bajcsy and Kovacic where the coarsest scale solution was up-sampled and was used to initialize the finer one when linear registration was used at a lowest resolution (Bajcsy and Kovaˇciˇc 1989). Linear elastic models have also been found useful when registering brain images based on sparse correspondences. They were used for the first time by Davatzikos (1997) based on geometric characteristics to establish mapping between the cortical surfaces. Modeling the images as inhomogeneous elastic objects led to the estimation of a global transformation function. Spatially-varying elasticity parameters were used to emulate the fact that certain structures tend to deform more than others. An important drawback of image registration, in general, is that if deformed image is used as input to an inverse process of the previously used transformation (forward), the output obtained will not be the same as the original input image for the forward transformation. The idea of parallel estimation of both forward and backward transformations, while compensating for inconsistent transformations by adding a constraint to the objective function was introduced later. Linear elasticity was used as a regularization constraint and Fourier series’ were used to parameterize the transformation (Christensen and Johnson 2001). A unidirectional approach was also introduced by Leow et al. that coupled the forward and backward transformations and provided an inverse consistent transformation by construction, thus diminishing the idea of a constraint addition to penalizing the inconsistency error (Leow et al. 2005). An important drawback of linear elastic models is their inability to cope with larger deformations. Nonlinear elastic models were proposed so as to account for large deformations. These models ensure the preservation of topology of deformable images emulating hyper-elastic materials and their properties. The use of the Finite

2.3 Geometric Deformation Models: A Survey

25

Element method provided a solution for the nonlinear equations and local linearization (Rabbitt et al. 1995). Two of the modeling processes for deformation were proposed, they were based on the concept of St. Venant-Kirchhoff elasticity energy (Pennec et al. 2005; Yanovsky et al. 2008). Viscous Fluid flow models: Image under deformation is modeled as a viscous fluid; these models do not assume small deformations hence can cope with the larger ones (Christensen et al. 1996). This transformation is governed by the Navier–Stokes equation that is simplified by assuming a very low Reynold’s number flow  μ f ∇ 2 v + μ f + λ f ∇(∇ · v) + F = 0 where v is the velocity field, while μf and λf are the viscosity coefficients. Christensen et al. extended their earlier work to recover transformations for brain anatomy; fluid transformation preceded by elastic registration step was used to refine the result obtained (Christensen et al. 1997). The processes in use until then had an important drawback in the form of computational inefficiency. To circumvent this shortcoming a new fast algorithm based on a convolution filter in scale-space was proposed (Bro-Nielsen and Gramkow 1996). Fluid deformation models were used in an atlas-enhanced registration setting (Wang and Staib 2000) while the same models were used to tackle multi-modal registration (D’Agostino et al. 2003). More recently, an inverse consistent variant of fluid registration to register diffusion tensor images was proposed (Chiang et al. 2008). Diffusion models: The deformation, in this case, is modeled by the diffusion equation u + F = 0 Thirion, inspired by Maxwell’s Demons (Thomson 1874), proposed to perform image matching as a diffusion process, his work, in turn, inspired most of the work done in image registration using diffusion models (Thirion 1998). The most suitable version for medical image analysis involved selecting all image elements as Demons, calculating demon forces by considering the optical flow constraint, assuming a nonparametric deformation model that was regularized by applying a Gaussian filter after each iteration, and a tri-linear interpolation scheme. The use of Demons was able to provide dense correspondences but lacked sound theoretical justification (Sotiras et al. 2013). However, this did not stop it from being an immediate success and soon enough a fast algorithm based on Demons (Thirion 1998) for image registration was proposed by Fischer and Modersitzki (Fischer and Modersitzki 2002) which provided theoretical insights into its workings. Vercauteren et al. (2007) adopted the alternate optimization framework that Cachier et al. (2003) proposed, to relate symmetric Demons forces with the efficient second-order minimization (ESM) (Malis 2004). In this methodology, an auxiliary variable was used to separate the matching and regularization terms. ESM optimization was used to perform matching by minimizing the data term whereas regularization was achieved by Gaussian smoothing.

26

2 Theoretical Background

A variation of Thirion’s Demon Algorithm was proposed by Vercauteren et al. endowed with the diffeomorphic property (Vercauteren et al. 2007). In this approach, opposite to classical Demons approaches, an update field is estimated in all the iterations of the algorithm. A compositional rule is used between the previous estimate and the exponential map of the update field to estimate the running transformation. This exponential map is calculated by using the composition of displacement fields and the “scaling and squaring” method (Higham 2005; Moler and Van Loan 2003). Diffeomorphism of the mapping is ensured by the exponentiation of the displacement field. As an application of the model, Stefanescu et al. proposed a way of performing adaptive smoothing by taking into account the knowledge regarding the elasticity of tissues (Stefanescu et al. 2004). The Demons algorithm has found use not only in the study of scalar images but its application has been extended to multi-channel images (Peyrat et al. 2008), diffusion tensor ones (Yeo et al. 2009), as well as different geometries (Yeo et al. 2010). Peyrat et al. used multi-channel Demons to register the time-series of cardiac images by enforcing trajectory constraints. Each time instance was considered as a different channel while the estimated transformation between successive channels was considered as constraint (Peyrat et al. 2008). Yeo et al. (2010) derived Demons forces from the squared difference between each element of the Log-Euclidean transformed tensors while taking into account the reorientation introduced by the transformation. Curvature Registration: These image registration methodologies don’t necessarily need an extra affine linear pre-registration step since the regularization scheme associated with it does not affect the affine linear transformations. This constraint has been used by Fischer and Modersitzki in (2003, 2004). Despite several attempts to solve the underlying transformation function using the Gâteaux derivatives with Neumann boundary conditions, Henn (2006) pointed out that the resulting underlying function space still penalized the affine linear displacements. Henn, further proposed including second-order terms as boundary conditions in the energy and applying a semi-implicit time discretization scheme to solve the full curvature registration problem. Beuthien and associates (Beuthien et al. 2010), proposed another way to solve the curvature-based registration problem based on the approach presented in (BroNielsen and Gramkow 1996) for the viscous fluid registration scenario. Instead of devising a numerical scheme to solve the PDE that resulted from the curvature registration equilibrium equation, recursive convolutions with an appropriate Green’s function were used. Flows of Diffeomorphisms have also been one of the propositions for deformation modeling. In this case, the deformation is modeled by considering its velocity over time according to the Lagrange transport equation (Christensen et al. 1996; Dupuis et al. 1998; Trouvé 1998). This framework is also known as large deformation diffeomorphic metric mapping (LDDMM). It allows for the definition of a distance between images or sets of points (Joshi and Miller 2000; Marsland and Twining 2004). The mathematical rigor of the LDDMM framework comes at an important cost. The fact that the velocity field has to be integrated over time results in high computational and memory demands. Moreover, the gradient descent scheme that is usually employed to solve the optimization problem of the geodesic path estimation

2.3 Geometric Deformation Models: A Survey

27

converges slowly (Ashburner and Friston 2011). More efficient optimization techniques for the LDDMM have been investigated in Ashburner and Friston (2011), Marsland and McLachlan (2007), Cotter and Holm 2006). For a tabular comparison of these methods, Tables A.1–A.5 from Appendix A can be referenced.

2.4 Classification of Registration Methodology Used There are a plethora of registration algorithms being proposed by researchers and scientists being used in a multitude of applications. The classification taken up in this work (Fig. 2.6) is the most basic classification of image registration algorithms roughly based on the work of Barbara Zitova and Jan Flusser (2003) (Fig. 2.7).

2.4.1 Feature-Based Registration This approach for image registration is based on detection and extraction of salient structures, i.e., features in the images. Significant regions of interest in an image and lines (region boundaries, coastlines, roads, rivers) or points (region corners, line intersections, points on curves with high curvature) are understood as features ready to be used for image registration. They should be distinct, spread all over the image, and efficiently detectable in both images. Each feature point has a certain set of attributes, for the features to be distinct; there should be at least one attribute of two even similar features. There should be an unbiased equally weighted uniform spread of the feature points all across the image region to be registered. These features are usually detected for being present in both source and the target images simultaneously and they are expected to be stable in time to stay at fixed positions in the target image during the whole experiment while the source image feature points may be moving. The comparability of feature sets in the source and target images is assured by the invariance and accuracy of the feature detector and by the overlap criterion. Fig. 2.6 Image registration methodology classification

28

2 Theoretical Background

Fig. 2.7 The iterative strain energy minimization process

In other words, the number of common elements of the detected sets of features should be sufficiently high, regardless of the change of image geometry, radiometric conditions, presence of additive noise, and of changes in the scanned scene. The “remarkableness” of the features is implied by their definition. Feature-based registration methods are concerned with finding the transformation that minimizes the distances between features, extracted from the pre-interventional image, or a model, and corresponding 2D features. The extraction of these geometrical features greatly reduces the amount of data, which in turn makes such

2.4 Classification of Registration Methodology Used

29

registrations fast (Markelj et al. 2012). The core algorithms of feature detectors in most cases follow the definitions of the “point” as line intersection, the centroid of closed-boundary region, or local modulus maxima of the wavelet transform. Corners form specific class of features because “to-be-a-corner” property is hard to define mathematically (intuitively, corners are understood as points of high curvature on the region boundaries). Feature-based matching methods are typically applied when the local structural information is more significant than the information carried by the image intensities. They allow registering images of completely different natures (like aerial photograph and map) and can handle complex between-image distortions. The common drawback of the feature-based methods is that the respective features might be hard to detect and/or unstable in time. The crucial point of all feature-based matching methods is to have discriminative and robust feature descriptors that are invariant to all assumed differences between the images.

2.4.1.1

Moving Least Squares

Moving least squares is a method of reconstructing continuous functions from a set of unorganized point samples via the calculation of a weighted least squares measure biased towards the region around the point at which the reconstructed value is requested. In computer graphics, the moving least squares method is useful for reconstructing a surface from a set of points. Often it is used to create a 3D surface from a point cloud through either down-sampling or up-sampling. Moving Least Squares (MLS) methods are linear systems of equations for the global least squares, and the weighted, local least squares approximation of function values from scattered data. By scattered data, it should be understood as an arbitrary set of points in Rd which carry scalar quantities (i.e., a scalar field in d dimensional parameter space). This scattered point cloud is the feature point set common in both target and source image pairs. The point cloud of source image is interpolated to the points in the target image using MLS, this registers surfaces in source to the target image. The MLS method was proposed by Lancaster and Salkauskas (1981) for smoothing and interpolating data. The idea was to start with a weighted least squares formulation for an arbitrary fixed point in Rd and then move this point over the entire parameter domain, where a weighted least squares fit is computed and evaluated for each point individually. It can be shown that the global function f (x), obtained from a set of local functions:  m θ (||x − xi ||)|| f x (xi ) − f i ||2 (2.9) f (x) = f x (x), min fx ε

d

i

So instead of constructing a global approximation, it constructs and evaluates a local polynomial fit continuously over an entire domain , resulting in the MLS fit function. Moving least squares will be explained in detail with examples pertaining to this book in coming chapters.

30

2 Theoretical Background

2.4.1.2

Optical Flow Motion

Optical flow or optic flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene (Warren and Strelow 1985). It is a dense field of displacement vectors that defines the translation of each pixel in a region. It is computed using the brightness constraint, which assumes the brightness constancy of corresponding pixels in consecutive frames. Optical flow motion estimation is commonly used as a tool in motion-based segmentation and point tracking applications. Popular techniques for computing dense optical flow include methods by Horn and Schunck (1981), Lucas and Kanade (1981), Black and Anandan (1996), and Szeliski and Coughlan (1997). It is a well-known registration technique that is equivalent to the equation of motion for incompressible flow in physics. The concept of optical flow was originally introduced in computer vision in order to recover the relative motion of an object and the viewer in between two successive frames of a temporal image sequence. Its fundamental assumption is that the image brightness of a particular point stays constant, i.e., I (x, y, z, t) = I (x + δx, y + δy, z + δz, t + δt)

(2.10)

After a bit of mathematical interpolations, it basically comes down to: I + ∇ I · u = 0

(2.11)

where ΔI the temporal difference between the images, ∇I is the spatial gradient of the image, and u describes the motion between the two images. In general, additional smoothness constraints are imposed on the motion field u in order to obtain a reliable estimate of the optical flow. It also helps in tracking common feature points across the sequence of images. Temporal image sequence registration and deformity estimation using optical flow motion are explained in detail in upcoming chapters.

2.4.2 Intensity-Based Registration Intensity-based registration methods compare intensity patterns between images. The moving image is subjected to transformations such that the resulting transformed image exhibits minimum intensity differences with the fixed image. It has recently become the most widely used registration basis for several important applications. In this context, the term intensity is invariably used to refer to the scalar values in image pixels or voxels. The physical meaning of the pixel or voxel value depends on the modalities being registered and is very often not a direct measure of optical power (the strict definition of intensity).

2.4 Classification of Registration Methodology Used

31

Intensity-based registration involves calculating a transformation between two images using the pixel or voxel values alone. In its purest form, the registration transformation is determined by iteratively optimizing some “similarity measure” calculated from all pixel or voxel values. For deformable image registration, a major attraction of intensity-based algorithms is that the amount of preprocessing or userinteraction required is much less than for point-based methods. As a consequence, these methods are relatively easy to automate. Intensity-based registration algorithms can be used for a wide variety of applications: registering images with the same dimensionality, or different dimensionality; both rigid transformations and registration incorporating deformation; and both inter-modality and intra-modality images.

2.4.2.1

Strain Energy Minimization

This registration technique gives the best results for deformable/non-rigid/elastic images. There is a potential energy associated with an elastic system at a time. Since, the images involved in the study are of a human body organ, they can be categorized as non-rigid or deformable images and the energy principles of elastic systems are applicable to this set of images. Potential energy of an elastic twodimensional system at static equilibrium is supposed to be pure strain energy. The potential energy function although consists of tensile stress, shear modulus, shear strain and both of the Lame’s constants. This energy function is reduced to just strain energy variables and is equated to zero for static equilibrium conditions. This complete process is known as strain energy minimization. It is a transformation between the source and the target image pair. The strain energy function of the source image is minimized comparing intensity values iteratively to such a stage that no lesser intensity difference is found between the registered image and the target image. Obviously, the intensity differences are never achieved to zero nor does the strain energy of the source-target image pair system ever reach zero in real-life conditions. It is fully automatic in its mode of operation and helps in faster and more accurate image registration in comparison to pure point-based registration methods. This factor gives this method an upper hand when it comes to real-life medical image registration problems. The intensity-based energy minimization methodology seems more practical, stable, and cost-efficient for deformable images in comparison to landmark-based or segmentation based methodologies for similar purposes. The method is simpler and faster than its contemporaries because the energy function is worked upon directly without solving large matrix system assemblies.

32

2 Theoretical Background

2.5 Feature Detection/Description Methods Feature detection/description is an important precursor for implementing the featurebased registration techniques for both deformable and rigid images. Feature detection is the process where we automatically examine an image to extract features that are unique to the objects in the image, in such a manner that we are able to detect an object based on its features in different images. This detection should ideally be possible when the image shows the object with different transformations, mainly scale and rotation, or when parts of the object are occluded (Pedersen 2011). The feature keypoints obtained in the process have certain attributes on the basis of which their distinctiveness etc. is determined. These attributes are coordinate position pt(x, y), the angle (orientation), the magnitude (response, strength), size (diameter), octave (pyramid octave in which keypoint is detected), and the object_id. The SURF (speeded up robust features) algorithm has been explained here using details courtesy (Kang et al. 2015). It consists of two major parts: (1) detector and (2) descriptor. The detector uses a basic Hessian matrix approximation and an integral image, which significantly reduces computation time. The procedure it uses comprises four steps: (1) integral image; (2) Hessian matrix-based interest points; (3) scale-space representation and (4) interest point localization. First, in order to speed up local feature extraction, an integral image I (x) (as shown in Fig. 2.8) is adapted to the SURF algorithm. The entry of an integral image I (x) at a position x = (x, y)T can be represented as the summation of all the pixels in the input image I within a rectangular region generated by the origin and x as follows: I (x) =

j≤y i≤x   i=0 j=0

Fig. 2.8 Integral image calculation. Reproduced from Bay et al. (2008)

I (i. j)

(2.12)

2.5 Feature Detection/Description Methods

33

Once the integral image has been computed, it takes three additions to calculate the sum of the intensities over any upright, rectangular area. Therefore, computation is independent of the size of the rectangle. In Step 2, the Hessian matrix, H(x, σ ), is used to determine the interest points. The Hessian matrix, H(x, σ ) in x at scale σ is defined as follows:   L x x (x, σ ) L x x (x, σ ) H (x, σ ) = (2.13) L yx (x, σ ) L yy (x, σ ) where L xx (x, σ ) is the convolution of the Gaussian second-order derivative (∂ 2 /(∂x 2 ))g(σ ) with image I at a given point x, and similarly for L xy (x, σ ) and L yy (x, σ ). To reduce the computational cost, SURF uses the following approximation for H(x, σ ):  Happr =

Dx x Dx y D yx D yy

 (2.14)

Blob-like structures are then detected at the location where the determinant d(H appr ) is maximum using:  2 d Happr = Dx x D yy − w Dx y

(2.15)

where the relative weight w is used to balance the expression for d(H appr ), which is needed for energy conservation between the Gaussian and the approximated Gaussian kernels. In Step 3, the scale-space representation step, Gaussian approximation filters are adapted to each level of filter size in the scale-space to extract interest points from images. This scale-space representation concept has also been applied to the SIFT (Scale-invariant feature transform) algorithm. However, the SIFT algorithm iteratively reduces the image size, whereas the SURF algorithm uses the integral images, allowing up-scaling of the filter at a constant cost. As a result, the SURF algorithm is computationally more efficient and conserves more high-frequency components with no aliasing. In Step 4, the interest point localization step, interest point detection is performed using the non-maximum suppression (NMS) over three neighborhood scales (3 × 3×3 neighborhood pixels). The points that have the maxima of the determinant of the Hessian matrix are then regarded as the feature points by NMS. In the descriptor, in order to assign invariability to the interest points, every interest point sought by the detector has to carry its own indicator. When deformations such as viewpoint angle changes, scale changes, increasing blur, image rotation, image blur, compression, and illumination changes occur, the interest point descriptors can be employed to look for correspondences between the original image and the transformed image. In SURF, the procedure used by the descriptor comprises of two steps: (1) orientation assignment and (2) descriptor based on the sum of Haar wavelet responses.

34

2 Theoretical Background

In Step 1, the orientation assignment step, image orientation is especially used to identify invariability of the interest point with respect to image rotation. The orientation is computed by detecting the dominant vector of the summation of the Gaussian weighted Haar wavelet responses under sliding window split circle region by π /3. Because the horizontal and vertical responses of the Haar wavelet include both the strength and directional property of interest points, image orientation efficiently represents the essentials of the image point with respect to image rotation. In Step 2, the descriptor based on the sum of Haar wavelet response step, to discover the descriptor of interest point, the orientation selected in the orientation assignment step, and the square region around the interest point are needed. Each of the square regions is split into smaller 4 × 4 sub-regions. For each sub-region, the horizontal Haar wavelet response d x and the vertical Haar wavelet response d y are computed at 5 × 5 regularly spaced sample points. The d x and

 d y from

are

each

sub-region then utilized to form the 4D description vector v = dx , d y , |dx |, d y , which is called the descriptor. This method is more robust than that of SIFT. Though there are many combined and standalone feature detectors/descriptors are available in open-source environment, SURF (Speeded Up Robust Feature) feature detector/descriptor has been employed for the same in this work. SURF is a unique scale- and rotation-invariant detector and descriptor, outperforming contemporary methods with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster. Focus is on scale and in-plane rotation-invariant detection and descriptions. These seem to offer a good compromise between feature complexity and robustness to commonly occurring photometric deformations in thoracic images. Skewing, anisotropic scaling, and perspective effects are assumed to be second-order effects that are covered to some degree by the overall robustness of the descriptor. For guaranteed invariance to any scale changes the input thoracic images are analyzed at different scales. The detected interest points are provided with rotation and scale-invariant descriptor. These basic advantages that SURF provides relating to speed and relative accuracy in face of rotation, illumination changes, and several other distortions led to its use for setting up a common landmark point cloud set between the source and target image pair to assist and speed up the application of feature-based image registration algorithms. Looking at the fact that these operations were being performed on CT images of real subjects (not fabricated phantoms etc.), the deformations in images were voluntary and spontaneous SURF came out to be the best choice of feature detector/descriptor for the job in terms of accuracy and speed.

2.6 Database Employed The dataset used comprised of a total (3 × 10) × 10, i.e., 300 thoracic CT images across ten subjects. The dataset was obtained from the publicly available database, http://www.dir-lab.com, with proper downloading permissions from the concerned administrator. All images were anonymized and all procedures followed were in

2.6 Database Employed

35

accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Declaration of Helsinki 1975, as revised in 2008 (5). Informed consent was obtained from all patients for being included in the study. All patients or legal representatives signed informed consent. The images lie between CT phases 0–6, i.e., end-inspiration to end-expiration in timestamp range t 00 → t 06 . The image dimensions lie between 396 × 396 and 432 × 400 pixels. There were six frames from a temporal thoracic image sequence each for every Anatomical Plane (AP), i.e., Axial (supine), Coronal, and Sagittal for all the ten subjects acquired simultaneously with a gap of 0.1 s starting from time t = 0.1–0.6 s. The three anatomical planes are explained through graphical representations in Figs. 2.9(a, b, c) and 2.10. All images were identified as I NAP (x, y, t) where   + N , t ∈ R |1 ≤ N ≤ 10; 0.1 ≤ t ≤ 0.6 , (x, y) are the x and y coordinates in the Cartesian plane and AP signifies the three anatomical planes of view, i.e., Axial (a), Coronal (c), and Sagittal (s). Suppose the third frame from coronal AP for subject “case 9”, would be notified as I9c (x, y, 0.3). A view of the image database is shown in Tables 2.1 and 2.2 for representational purposes.

Fig. 2.9 a Axial, b Coronal, and c Sagittal anatomical positions. Taken from the employed database

36

2 Theoretical Background

Fig. 2.10 A three-dimensional cubical representation of the APs

2.7 Accuracy and Similarity Measures Used Choices of the quality of alignment as the measure of success follow directly from our definition of registration, which is the determination of a transformation that aligns points in one view of an object with corresponding points in another view of an object. Most of the work and much of the literature on the subject of registration inevitably focuses on the quest for registration methods that produce a better alignment for some combination of modalities. The success of the registration, which we are relating monotonically to the quality of the alignment, has been estimated in published work by visual inspection, by comparison with a gold standard, or by means of some selfconsistency measure. Although the great majority of studies of registration quality have been carried out for rigid-body registration algorithms, the same concepts are also applicable for non-rigid registration. The measurement of registration success will be some statistical estimate of some geometrical measure of alignment error. Many such measurements have been used to measure the quality of registration, but not all are of equal value (Silva et al. 2007). An understanding of their meanings is crucial to understanding and evaluating claims of registration accuracy.

2.7.1 Target Registration Error It is a common geometrical measure to assess alignment errors. It can be understood as the displacement between two corresponding points after registration, i.e., after one of the points has been subjected to the registering transformation. The word “target” in the name of this error measure is meant to suggest that the error is being measured at an anatomical position that is the target of some intervention or diagnosis. Such errors would be expected to be more meaningful than errors measured at points with no intrinsic clinical significance. Suppose p represents a point in the first image of a

2.7 Accuracy and Similarity Measures Used

37

Table 2.1 CT images at t = 0.1 and t = 0.6 s

ANATOMICAL PLANES (T & S Images) Subjects

Axial

Coronal

Sagittal

1

2

3

4

5

6

7

8

9

10

pair to be registered, and q a point in the second image. A registration method applied to this pair leads to a transformation T that, without loss of generality, registers the first image to the second. The difference between the two vectors representing the transformed point and the corresponding point gives the target registration error (T RE ). Thus, TRE = T ( p) − q However, it’s the magnitude of the target registration error (T RE ) that is usually reported and documented.

38

2 Theoretical Background

Table 2.2 Working database through all anatomical planes from t = 0.1–0.6 s Axial

Coronal

Sagittal

1 2 3 4 5 6 7 8 9 10

2.7.2 Signal-to-Noise Ratio (SNR) The signal-to-noise ratio (SNR) is used in imaging as a physical measure of the sensitivity of a (digital or film) imaging system. It has been used as a metric to demonstrate enhanced similarities in a pair of images later on in comparison to the pair in its former state. The initial image pair is the source and target image pair and the later one is transformed source and target image pair. Transformed source image is the source image we get post the registration process. An increased SNR value for the post-registration image pair helps in indicating better performance by the registration algorithm. 

a−1 b−1

SNR = 10 · log10 a−1 0 b−1 0

0

0

[t(x, y)]2



[t(x, y) − s  (x, y)]2

where, s (x, y) is the transformed image post-registration and t(x, y) is the target image in question.

2.7 Accuracy and Similarity Measures Used

39

2.7.3 Peak Signal-to-Noise Ratio (PSNR) Peak signal-to-noise ratio, often abbreviated PSNR, is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. PSNR is most commonly used to measure the quality of reconstruction of lossy compression. In terms of assessing an image registration algorithm, it has been used as an image similarity metric as well. It would compare the transformed source image s (x, y) against the target image t(x, y) and get a value; this value is compared against the same for original source-target image pair. Increased value in PSNR for the transformed-target image pair would indicate a better transformation and thus a better image registration process.  PSNR = 10 · log10

1 a·b

max(t(x, y))2

a−1 b−1 2  0 0 [t(x, y) − s (x, y)]



2.7.4 Structural Similarity Index (SSIM) The structural similarity index is a method for measuring the similarity between two images. The SSIM index is a full reference metric; in other words, the measurement or prediction of image quality is based on an initial uncompressed or distortion-free image as a reference. SSIM is designed to improve on traditional methods such as peak signal-to-noise ratio (PSNR) and mean squared error (MSE), which have proven to be inconsistent with human visual perception. The difference with respect to other techniques mentioned previously such as SNR or PSNR is that these approaches estimate absolute errors; on the other hand, SSIM is a perception-based model that considers image degradation as perceived change in structural information, while also incorporating important perceptual phenomena, including both luminance masking and contrast masking terms. Structural information is the idea that the pixels have strong inter-dependencies especially when they are spatially close. SSIM has been used as an increasing factor for the transformed-target image pair in comparison to the original source-target image pair; indicating a better image transformation process and thus a better image registration process.   2μx μ y + C1 2σx y + C2   SSIM(x, y) = 2 μx + μ2y + C1 σx2 + σ y2 + C2 where μx , μy are the mean intensities of the respective signals; σ the respective standard deviation and C 1 , C 2 constants.

40

2 Theoretical Background

In practice, however, it is usually required to have a single overall quality measure of the entire image. In those scenarios, a mean SSIM (MSSIM) index is used to evaluate the overall image quality. For image dimensions [a, b]: a·b 1  MSSIM(x, y) = SSIM(xi , yi ) a · b i=0

2.7.5 Normalized Cross-Correlation (NCC) Normalized cross-correlation (NCC) has been commonly used as a metric to evaluate the degree of similarity (or dissimilarity) between two compared images. The main advantage of the normalized cross-correlation over the ordinary cross-correlation is that it is less sensitive to linear changes in the amplitude of illumination in the two compared images. Furthermore, the normalized cross-correlation is confined in the range between –1 and 1. The setting of the detection threshold value is much simpler than the cross-correlation. The normalized cross-correlation does not have a minimal frequency domain expression. It cannot be directly computed using the more efficient FFT (Fast Fourier Transform) in the spectral domain. Its computation time increases dramatically as the window size of the template gets corpulent.

b  i=1 j=1 t(i, j)s (i,

a b 2 i=1 j=1 [t(i, j)]

a NCC =

j)

where t(i, j), s  (i, j)) are the target and transformed source image frames, respectively, both dimensions[a, b].

Chapter 3

A Moving Least Square Based Framework for Thoracic CT Image Registration

This chapter proposes an automatic registration process for tracing of the deformity path of the thoracic region based on feature detector cum descriptor Speeded Up Robust Feature (SURF) and Moving Least Squares (MLS). Corresponding control point pairs or landmarks in image pairs or groups can be used to define the deformation with respect to time, point of view, or modality. The manual definition of the number of control points in an image such that it is enough to define all kinds of deformations in that respective image is a tedious task. Hence, an automatic definition of control points has been the way forward taken in the proposed work. The control point cloud on the images is defined by its feature set which is obtained by using the SURF detector/descriptor, which serves as an input for MLS algorithm to trace the deformations of the image (thoracic image in this case). The proposed strategy begins with having a pair of images of the same dimensions to be registered, obtained at different timestamps of a temporal image sequence. The corresponding anatomical landmark points are identified in both these images. These landmark points are inputs to a point cloud-assisted continuous surface regeneration algorithm. This technique analytically solves a number of least squares problems to find the local elastic transformations. Applying these local transformations on the image pair creates the deformations throughout the considered sequence of images. The relative deformations between the image pair are accounted for by the surface reconstruction algorithm and are adjusted using respective control point sets for both images thus registering one with respect to another. In our experiments, Target Registration Error (E TR ) is used as the quantitative measure for the evaluation of performance. The E TR obtained for the dataset was found to be considerably lower than more traditional and prevalent transforms such as affine, thin-plate splines, or finite element-based approach. Therefore, the MLS-based method was found to be more suitable for the real-time applications of Image-Guided Interventions (IGI) demanding higher speed and more accurate image registrations.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0_3

41

42

3 A Moving Least Square Based Framework for Thoracic CT …

3.1 Introduction Deformable images have been a constant focus of study and research in the field of image registration. There have been extensive intra- and inter-patient studies and projects in this particular direction which have resulted in a variety of registration algorithms. Deformable registration is considered as an ill-posed problem because there is generally no unique solution to a registration problem. Usually, it is formulated as an optimization problem. In the case of the study of thoracic images from inhale to exhale phases (or vice versa), finding a finally registered image is a problem for which many algorithms and methods have been proposed, tested, and compared over time. Apart from the peripheral deformations happening in the thoracic region, there are also local deformations of the internal organs (within the periphery) going on throughout the breathing process. The definitions of the deformity patterns governing those motions are quite unclear yet and much research hasn’t happened towards this direction. It is a problem of bigger proportions for oncology researchers and radiologists alike to define and describe the inner deformations in their studies. This study’s clinical relevance cannot be stressed upon more. Since overall respiratory motion is related to lung function, it has a diagnostic value to itself. Any organ motion pertaining to breathing can lead to image artefact and position uncertainties during image-guided clinical interventions. A particular case for such image-guided interventions (IGI) can be the radiotherapy planning of thoracic and abdominal tumors; the respiratory motion causes important uncertainties and is a significant source of error (Keall et al. 2006). Image registration has recently started playing an important role in this scenario; it helps in the estimation of any motion caused due to breathing during acquisition and the description of the temporal change in position and shape of the structures of interest by establishing the correspondence between images acquired at different phases of the breathing cycle (Ehrhardt et al. 2011). The present study intends to shed light on the deformity paths of local deformations in the thoracic periphery which can be helpful as a prerequisite for radiation therapy (based on their dosimetric evaluations), tumor growth progression with time and also towards making deformable subject-specific motion models more precise and accurate.

3.2 Background The background study of this chapter initially includes a study of a few most prominent proposed algorithms in the direction of the study of the moving least squares and its applications. Then the proposed methods relating to image registration of thoracic CT images are discussed. The propositions are categorically discussed keeping in mind their acute relevance and their year of occurrence. Propositions occurring at a later instant in the timeline are given higher priority in discussion in comparison to

3.2 Background

43

earlier works to establish a better context. These methods are compared in a tabular format in Table B.1 in Appendix B. The concept of least square methodologies such as weighted least squares and moving least squares was first proposed in 1981 by Lancaster and Salkauskas (1981). They presented “an analysis of least squares methods for smoothing and interpolating scattered data/points”. They proposed a non-interpolating least squares method as an alternate representation of the local approximation based on the choice of weight functions. This became the basis of more recent moving least squares, giving it its characteristic dynamic weight function choice option to project smoother surfaces for all data points coming into consideration in real-time. In particular, they proved theorems concerning the smoothness of interpolants and the description of MLS processes as projection methods. The differences between interpolating and noninterpolating MLS method as projection methods were pointed and singled out. The effects of the choice of weight functions and asymptotic behavior of such single variable and multivariate functions have been studied in detail. In the earlier discussed method, interpolating and non-interpolating nonlinear least square methods have been discussed as projection of given scattered data/points. These scattered points were not necessarily representations/emulations of real-world objects around us in multiple dimensions. One of the most prominent works in the direction of emulating/projecting real-world objects from scattered point sets/data using moving least squares came from Alexa et al. (2003). This work stressed upon the use of point sets to represent shapes. It set its goal in defining surfaces from a set of points close to an original surface, this is approximated using MLS. A projection procedure has been defined which would project any point near the point set onto the surface. Then, the MLS surface is defined as the points projecting onto themselves. The smoothness conjecture is motivated and the respective projection is computed. The proposed model was tested on “the Stanford bunny” (a computer graphics 3D test model developed by Greg Turk and Marc Levoy in 1994) along with other models. The proposed approach showed smoother silhouettes and more accurate highlights in comparison to more traditional methods like Splatting and Gouraud-shaded mesh models. The problem with both Splatting and Gouraud-shaded mesh was found to be that these models were not sampled densely enough exhibiting relatively inaccurate highlights. A parameter h connected to feature size is used such that features with radius smaller than h are smoothed out. Actual timings and memory requirements for the projection procedure depended heavily on the feature size h. As long as h was small with respect to the total diameter of the model, the use of the main memory of the projection procedure was found to be negligible. For the Stanford bunny dataset, total of 36,000 points of the bunny was projected on to a surface definition in roughly 10–30 s. It was found to be possible to provide a point set representation that conforms to a specified tolerance and the use of a point set (without connectivity) as a representation of shapes. While the earlier works included using MLS to project known multivariate functions and established computer graphics 3D test models like the Stanford bunny and the Aphrodite statue. This work by Schaeffer et al. (2006) implemented the concept of MLS to define deformations in rigid images of real-world objects. They proposed

44

3 A Moving Least Square Based Framework for Thoracic CT …

an image deformation method based on moving least squares using various classes of linear functions including affine, similarity, and rigid transformations. These deformations were realistic and gave the user an impression of manipulating real-world objects. Image deformations were built based on collections of points with which the user could control the deformation. A deformation function was constructed satisfying the three properties of Interpolation, Smoothness and Identity using MLS. The proposed method was applied for affine, Similarity, Rigid and Elastic deformations on a set of images. It was found to perform deformations faster than contemporary methods. Deformations were constructed such as to minimize the amount of local scaling and shear and restricting the classes of transformations used in moving least squares to similarity and rigid-body transformations. This method, using MLS completely avoided the use of input image triangulation unlike the method proposed by Igarashi et al. (2005), thus producing globally smooth deformations. It was showed how solutions could be computed directly from the closed-form deformation using similarity transformations thereby bypassing the nonlinear minimization (contrary to Igarashi et al. 2005). The method is generalized enough to accommodate different distance metrics dependent on the topology of the shape rather than the simple, Euclidean distance used as weight factor. The methods discussed till now deal with known multivariate function projections into point sets or data/points projections of established 3D models or definition of everyday objects’ rigid image deformities using MLS. Castillo et al. (2009) suggested a framework for deformable image registration of two images using MLS for corresponding sets of feature landmark point pairs in both images and evaluation of its spatial accuracy. They used an in-house developed Matlab® based software interface called APRIL (Assisted Point Registration of Internal Landmarks) to facilitate manual selection of landmark feature pairs between image volumes. This point set of the pair when subjected to MLS registered the source landmark point set to the corresponding target point set. The image registration error was calculated in terms of fiducial error or spatial errors. The uncertainty of spatial error USE estimates was found to be inversely proportional to the square root of the number of landmark point pairs and directly proportional to the standard deviation of spatial errors, i.e., USE ∝ 1/(Lpp)1/2 and USE ∝ SDSE. Cumulative distribution functions (CDFs) were generated from the corresponding set of error measurements for each case. To simulate the spatial error information derived from validation point sets of different sizes, uniform samples of the individual CDFs were obtained for sample sizes ranging from 10 to 5000. For each sample size, 100,000 independent sample sets were obtained. At each sample size increment, an independent calculation of the mean spatial error was performed for each of the 100,000 error samples. The feasibility of generating large (>1100) validation landmark sets has been demonstrated on five component phase pairs from clinically acquired treatment planning 4D CT data. The results demonstrate that large landmark point sets provide an effective means for objective evaluation of DIR with a narrow uncertainty range, and suggest a practical strategy for qualitative analysis of DIR spatial accuracy on a routine clinical basis. No proposition on the estimation of deformity between the registered image pairs was made though.

3.2 Background

45

The EMPIRE10 challenge conducted by Murphy et al. (2011) was a study of Evaluation of Registration Methods on Thoracic CT. EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison and evaluation of non-rigid registration algorithms and techniques which are applied to a database of intra-patient thoracic CT image pairs. Evaluation of non-rigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms have been virtually impossible in the past. The result of this study comprised of a comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms were applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters were chosen by researchers’ expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. Some methods up for comparison were: Asclepios1, Asclepios2, CMS, DIKU, DROP, elastix, IMI Lubeck Diffeomorph, Lyon FFD, MGH, Nifty Reggers, OFDP, picsl exp, picsl gsyn, Robust TreeReg Leuven, Spline MIRIT Leuven, etc. All methods were fully automatic with the exception of MGH. It was found that generic registration algorithms performed better than data specific methods. It might still be the case that combines aspects of both could improve performance even further, particularly on more difficult scan pairs. The EMPIRE10 challenge enabled detailed, independent, and fair evaluation of non-rigid registration algorithms. Edward Castillo et al. (2014) proposed a moving least squares approach for computing spatially accurate transformations that satisfy strict physiologic constraints. It involved computation of physiologically realistic spatial transformation from a sparse point cloud of displacement estimates using MLS and any combination of upper bound, lower bound, or equality constraints placed on the Jacobian. MLS defined a spatial transformation from a sparse point cloud of estimated displacements and provided simple analytic derivative estimates for all voxel locations; given displacement estimates from the automated block. Five publicly available (cases 6–10 from www.dir-lab.com) inhale/exhale thoracic CT image pairs each with 300 landmarks (for DIR validation) were registered by first obtaining a sparse point cloud of displacement estimates via block matching. Two MLS transformations were then computed, one with no Jacobian constraints and the other with strict contraction Jacobian constraints. Both MLS fields achieved similarly low average millimeter error on all five cases (between 1.16 and 1.26). However, the constrained MLS yielded a strict contraction (all Jacobian values between 0 and 1) while the unconstrained MLS resulted in regions of expansion (Jacobian values larger than 1) despite registering from inhale to exhale. The proposed MLS approach was found capable of producing Jacobian constrained transformations without any degradation in spatial accuracy. Though applied to block match estimates, the approach can be employed in conjunction with displacement estimates from any DIR algorithm.

46

3 A Moving Least Square Based Framework for Thoracic CT …

3.3 Method 3.3.1 Preparation The dataset used comprised of a total (3 × 10) × 10, i.e., 300 CT images across 10 subjects ranging from 396 × 396 to 432 × 400 pixels. There were 10 frames each for every anatomical plane, i.e., Axial (supine), Coronal, and Sagittal for all the 10 subjects acquired simultaneously with a gap of 0.1 s starting from time t = 0 to 1 s, thus 4DCT image dataset. All images were identified as I NA P (x, y, t)  called the + where N , t ∈ R |1 ≤ N ≤ 10, 0.1 ≤ t ≤ 1 , (x, y) are the x and y coordinates in the Cartesian plane and AP signifies the three anatomical planes of view, i.e., Axial (a), Coronal (c), and Sagittal (s). So, the sixth subject’s Coronal CT image acquired at t = 0.6 s will be identified as I 6c (x, y, 0.6). Sample of images used from all viewpoints and all subjects at timestamps 0.1 and 0.6 s are summarized in Table 3.1.

3.3.2 Proposed Methodology The methodology is as such that two images at different timestamps are taken; one with the earlier timestamp, i.e., t = 0.1 s is considered as base/reference image and the one with the later timestamp, i.e., t = 0.6 s deformed image (real-time image). These are fed into the SURF feature detector and a common corresponding feature set in the form of a Cartesian point cloud is obtained. This feature set then serves as input to the MLS algorithm as control point cloud corresponding to both reference and the deformed image. The MLS algorithm then traces the deformation in the real-time image with respect to the corresponding base image. The overall process can be referred to in Fig. 3.1. The proposed model uses the Speeded Up Robust Feature detector (SURF) (Bay et al. 2006, 2008) to obtain a feature set of the deformed image as well as the reference image. It detects and describes the feature set irrespective of any scaling and/or rotation in the corresponding images. SURF gives better results than previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster than any other state of the art feature detector (Pang et al. 2012; Yoon et al. 2009). This was achieved by relying on integral images for image convolutions; by building on the strengths of the leading existing detectors and descriptors (specifically, using a Hessian matrix-based measure for the detector, and a distribution-based descriptor); and by simplifying these methods to the essential. This leads to a combination of novel detection, description, and matching steps (Bay et al. 2008) as can be seen in Fig. 3.2. An implementation of the algorithm over the inhale and exhale frames for the first subject at t = 0.1 and 0.6 s, respectively, i.e., I1A (x, y, (0.1, 0.6)) is shown in Figs. 3.3, 3.4 and 3.5. MLS has been successfully applied to surface reconstruction from points/point clouds and other point set surface definitions (Alexa et al. 2003; Schaefer et al.

3.3 Method

47

Table 3.1 All three anatomical viewpoints for all the 10 subjects at time t = 0.1 and 0.6 s

ANATOMICAL PLANES (at t=0.1 & 0.6 sec) Axial

Coronal

Sagittal

1 2 3 4 5 6 7 8 9 10

2006). Given a set of control point pairs on the source and the target images, the MLS technique determines the transformation Tv (x) that best minimizes the least square error expression: 

|Tv ( pi ) − qi |2

(3.1)

i

where pi and qi are the ith source and target control point pair, respectively, obtained by SURF algorithm. Since a single affine transformation is obtained as a result of the above transformation, there is no control over the scaling/shearing of the image. ‘weighing function wi to the least square error function which results in a different transformation function for each point of evaluation of the image.

48

3 A Moving Least Square Based Framework for Thoracic CT …

Fig. 3.1 The proposed model

Fig. 3.2 The working model for SURF



wi |Tv ( pi ) − qi |2

(3.2)

i

where the weighing function wi is of the form:  wi = 1  pi − v2α

(3.3)

where v is the point of evaluation in Eqs. (3.1)–(3.3), α is a parameter of the weighing function. This parameter changes values depending on the changing point of evaluation thus changing the weighing function (Eq. 3.3) and in turn changing the transformation for each point of the image. It performs better than its contemporaries while tracing deformations that are realistic and guides the user in manipulation of real-world objects. It also allows the user to specify the deformations using either sets of points or line segments, the later useful for controlling curves and profiles

3.3 Method

49

Fig. 3.3 Corresponding feature points at their respective positions in the inhale (left) and the exhale (right) frame recorded at t = 0.1 and 0.6 s, respectively

Fig. 3.4 Corresponding feature points matched in the inhale (left) and the exhale (right) frame recorded at t = 0.1 and 0.6 s, respectively

present in the image. For each of these techniques, it provides simple closed-form solutions that yield fast deformations, which can be performed in real-time as is shown in Fig. 3.5 as a result of using point set features from the inhale and the exhale frames in Figs. 3.3 and 3.4. The combined implementation of the SURF feature detector and MLS algorithm as a two-stage process attributes this methodology with faster processing speeds in feature detection/description and efficient tracking of the interest points (obtained from SURF) during their transition through frames/slices.

50

3 A Moving Least Square Based Framework for Thoracic CT …

Fig. 3.5 Registered axial image for the first subject from inhale frame at t = 0.1 s to exhale frame at t = 0.6 s

As we can see in Fig. 3.3, an implementation of the SURF algorithm gave out matching features with their respective coordinate values in the inhale (t = 0.1 s) and the exhale (t = 0.6 s) frames for the axial AP CTs of subject “case 1”. A similar process was employed for all subjects throughout all the three APs. This provided us with the respective coordinate point values for the mutual interest points between them. These frames along with the interest point values are used by the MLS algorithm to provide a registered image for every set of AP for a subject. Result of this process being the registered images for every subject through all three APs for all 10 subjects. Along with the registered images as output, the average translation of the interest points is also obtained in x and y Cartesian directions in “pixel” units for all 10 subjects under observation from all three APs. The average translation is basically the collective deviation for all the interest points from an initial stage to the final stage in terms of Euclidean distance in pixel units in their own respective cases.

3.4 Results and Discussion These average translations were compiled for all the 10 subjects with respect to the common denominator, i.e., the number of frames/slices. Since the number of frames taken into account was 6; there were total five translation gaps between them. The translation data was comparatively classified for all 10 subjects into different graphical representations in terms of it being in x or y direction and the AP it belongs to. A widespread consensus now exists that it would be useful to use prior knowledge of respiratory organ motion and its variability to improve radiotherapy planning and

3.4 Results and Discussion

51

treatment delivery (Blackall et al. 2006). The estimated deformation for a particular subject, when computed using the proposed methodology, can be compared and analyzed against a corresponding standard atlas to assess the extent of the abnormality. Figure 3.6a, b show the average x and y translation values for all subjects in axial AP. Looking into “case 10” y translation plot, the average deformations estimated over inter-frame durations of a single inhale–exhale process (stated above) were 0.278 ± 0.11 pixels with a maximum value of 0.408 pixels while transitioning from frame 4 to 5. The variations in the deformations exhibited by the subject “case 10” were significantly larger than the considered population average of 0.039 ± 0.13 pixels. Similarly Figs. 3.7a, b and 3.8a, b show the average x, y translation for all interest points in the coronal and sagittal APs, respectively. The maximum translation obtained for a case/subject during transition from one slice to another signifies the maximum variations in breathing pattern than the considered population and atlas data in general, which in this study happens to be subject “case 10”. Similarly, the minimum deviation obtained from either of the APs points out slice/frame transitions with no apparent anatomical deformation in the thoracic periphery of the subjects.

Fig. 3.6 a Average x-translations for axial AP; b Average y-translations for axial AP

Fig. 3.7 a Average x-translations for coronal AP; b Average y-translations for coronal AP

52

3 A Moving Least Square Based Framework for Thoracic CT …

Fig. 3.8 a Average x-translations for sagittal AP; b average y-translations for sagittal AP

3.5 Conclusion A methodology has been presented showing how a feature point set generated by SURF can be used through MLS for deformable image transformations in medical images such as the thoracic pectus excavatum exhale and inhale frames used in this work. The accuracy of the deformable registration performed using the proposed methodology is assessed in terms of the target registration error (E TR ) and is compared with two other prevalent methods for all three APs (refer Table 3.2) for the same database. The error values (pixel values converted to mm using the resolution of images) for the proposed methodology are considerably lower than the other methods for almost all subjects considered. The use of SURF can be explained by its better performance in terms of smaller time complexities (stated earlier) and common feature points than the existing state of the art feature detector and descriptors. MLS, however, was the best available choice for Deformable Image Registration (DIR) based on landmark points. Although the proposed methodology provides a fast and accurate way of DIR for medical images and thus an account of deformity in the thoracic periphery, there is much scope for improvement in the overall process. One way this can be achieved in the future is by modifying the SURF and/or MLS procedures themselves involved in the process, i.e., bringing newer versions of the existing methods better suited with the application. Another way is to improve and enhance the quality as well as the quantity of the database used. Also, the aforementioned procedure can provide better results if applied for different human anatomy altogether.

0.52

0.845

0.94

0.589

0.637

0.673

0.655

0.725

3

4

5

6

7

8

9

10

method

0.344

2

a Proposed

0.428

1

1.842

1.96

1.594

1.182

1.168

1.551

1.344

0.987

0.932

0.883

0.908

0.843

1.041

0.953

0.904

0.931

0.704

0.464

0.42

0.451

Sagittal

Axial

APs

Coronal

SIFT+MLS (scale invariant feature transform)

Subject (case No.)

0.795

0.735

0.673

0.647

0.699

0.86

0.785

0.61

0.464

0.498

Axial

1.882

2.19

1.494

1.432

1.368

1.541

1.404

1.217

1.052

1.123

Coronal

0.998

0.933

1.131

0.983

0.994

0.771

0.824

0.514

0.51

0.541

Sagittal

SURF+TPS (Thin-Plate Splines)

Table 3.2 Target registration error comparison through all APs for all subjects

0.385

0.215

0.183

0.147

0.209

0.22

0.165

0.13

0.054

0.128

Axial

SURF+MLSa

0.972

1.34

0.564

0.512

0.428

0.501

0.424

0.367

0.382

0.353

Coronal

0.458

0.353

0.591

0.463

0.454

0.081

0.214

0.034

0.09

0.121

Sagittal

3.5 Conclusion 53

Chapter 4

A Path Tracing and Deformity Estimation Methodology for Registration of Thoracic CT Image Sequences

The chapter presents a methodology that involves an automatic registration process for tracing deformity paths of the thoracic region between full inhale and exhale positions based on Hessian-matrix-based feature detector and Haar wavelets based descriptor along with Optical Flow Motion (OFM) estimation based point tracker technique. The proposed work presents a unique and innovative arrangement of methods to compute the average deformation of the thoracic region from all anatomical positions. Often clinical studies on image-based respiratory systems either suffer roadblocks or yield inconsistent results due to artifacts from a variety of subjects’ erratic breathing patterns. This leads to a loss of resources and time to ultimately get inconclusive and potentially wrong results. This work can be seen as an automatic way of computing average thoracic deformation for a set of diverse subjects. In an image sequence, corresponding control point pairs or landmarks can be used to define the internal deformations with respect to time, point of view, or modality. Defining enough number of control points in a thoracic image temporal sequence to describe the deformations happening in it is a tedious task. This inspired the use of the automatic definition of control points in the proposed work. The credibility and performance of the above-proposed method are demonstrated by its exemplary experimental results. The proposed methodology registers consecutive, equidistant frames in a temporal, thoracic CT image sequence starting from full inhale to full exhale positions by assessing the transformations happening in the image over time for a group of test subjects through all three Anatomical Positions (APs). The sequences are such that the first and last frames of a sequence are the most deformed frames with respect to each other, i.e., the first frame is the full inhale frame and the last one is the full exhale frame. The number of these frames depends upon the time gap considered between acquiring of these frames. Larger the time gap, lesser would be the number of frames in the sequence from start to finish. These frames are compared and contrasted against each other to find a common landmark set of points between them. They would be the same points in different frames at different (maybe) coordinate locations. These points serve as input to an OFM estimation based point tracking algorithm which would track a point from the first temporal frame up until the last © Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0_4

55

56

4 A Path Tracing and Deformity Estimation Methodology …

frame in the sequence it appears in, (few points were not found to carry through all the frames from start to finish (as we later found out in the process)). This process is completed for remaining points in the common landmark point set throughout the sequence. One of the frames is considered as a “base” or “reference” and the rest of the frames are registered with respect to the reference/base. In the proposed method, the full inhale frame which is the first one has been considered as the reference frame as a default for all subjects through all APs. The distances each point cover between temporally ordered frames of the sequence from its starting frame to the last frame it is observed in is estimated using its coordinate values in those, respectively, ordered frames. Average translation values are calculated for the point set for all test subjects through all three APs. These point set average translations summed up over the complete sequence give an estimation of the deformations happened from first to last frame of the temporal image sequence. Each point in the common landmark point cloud has been traced through frames and optical flow determined for it.

4.1 Introduction Accounting for organ motion in image-based lung cancer radiation treatment is considered as an important challenge in medical imaging (Goitein 2004). Lung deformations have been a constant focus of studies for the verification of medical imaging equipment and for medical training purposes for a long period of time and still, physiologically speaking, very little is understood about the respiratory movement (Stevo et al. 2009). The movement of the lung is passive; a result of the movement of other parts of the body, such as the diaphragm and the thoracic cage, and it is not possible to observe the lung directly, as it would collapse if the thoracic cage is opened. The clinical relevance of this research is diverse. Respiratory motion is related to the function of the lung and therefore a diagnostic value in itself (Ehrhardt et al. 2011). Furthermore, breathing-induced organ motion potentially leads to image artifacts and to position uncertainty in image-guided procedures. Particularly in radiotherapy planning of thoracic and abdominal tumors, respiratory motion causes important uncertainties and is a significant source of error (Keall et al. 2006). Therefore, there has been a large and continuing growth in studies and applications of 4D CT images for motion measurement, radiotherapy treatment planning, as well as functional investigations (Reinhardt et al. 2008). “A non-invasive method to describe lung deformations was proposed using NURBS surfaces based on imaging data from CT scans of actual patients” (Stevo et al. 2009; Tsui et al. 2000). V. B. Zordan and associates created an anatomical-inspired, physically-based model of human torso for the visual simulation of respiration (Zordan et al. 2004). It has been shown that breathing motion is not a robust and 100% reproducible process (Nehmeh et al. 2004; Vedam et al. 2004) and now there is a widespread common consent that it would be useful to use prior knowledge of respiratory organ motion and its “variability to improve radiotherapy planning and treatment delivery” (Blackall et al. 2006).

4.1 Introduction

57

The framework that has been acquired in this article is that the constituents of a thoracic image sequence with starting frame as the full inhale and ending frame as the full exhale are compared to find a set of common feature points, only distinction in them being different coordinate values (maybe) and that they exist in different temporal frames. These common feature points are collectively called as the corresponding feature set. This feature set then serves as input to an OFM estimation algorithm as a control point cloud corresponding to the thoracic image sequence. The estimation algorithm then traces the deformation in the thoracic image sequence right through initial to the final frame. The role of image registration techniques is increasing in these applications. Image registration enables the estimation of the breathing-induced motion and the description of the temporal change in position and shape of the structures of interest by establishing the correspondence between images acquired at different phases of the breathing cycle. A variety of image registration approaches have been used for respiratory motion estimation in recent years (Sarrut et al. 2006). Image Registration is the alignment/overlaying of two or more images so that the best superimposition can be achieved. These images can be of the same subject at different points in time, from different viewpoints or by different sensors. This way the contents from all the images in question can be integrated to provide richer information. It helps in understanding and thus reducing the differences that occurred due to variable imaging conditions. Most common applications of image registration include remote sensing (integrating information for GIS), combining data obtained from a variety of imaging modalities (combining a CT and an MRI view of the same patient) to get more information about the disease at once, cartography, image restoration, etc. An image registration method targets to find the optimal transformation that aligns the images in the best way possible. If the underlying transformation model allows local deformations, i.e., nonlinear fields’ u(x), then it is called Deformable Image Registration (DIR) (Muenzing et al. 2014). Image registration has been categorized into two kinds based on the type of image it is being applied for. The two kinds of images are Rigid Images and Deformable Images. Rigid images are those of structures with rigid morphological properties, e.g., bones, buildings, geographical structures, etc. Deformable images are those of structures shape and size of which can be modeled after tangible physically deformable models (Sotiras et al. 2013). Rigid image registration although is an important aspect of registration it is not the topic of discussion in this article. Since the discussion is about medical image registration and almost all anatomical parts or organs of the human body are deformable structures, the concentration here is on DIR (Oliviera and Tavares 2012). One of the three basic categories of physical models (Modersitzki 2004) conceptually utilized in this article is the diffusion models. Thirion, inspired by Maxwell’s Demons (Thomson 1874), proposed to perform image matching as a diffusion process, his work, in turn, inspired most of the work done in image registration using diffusion models (Thirion 1998). J. M. Peyrat and associates used multi-channel Demons to register time-series of cardiac images by enforcing trajectory constraints (Peyrat et al. 2008). Each time instance was considered as a different channel while

58

4 A Path Tracing and Deformity Estimation Methodology …

the estimated transformation between successive channels was considered as a constraint. Yeo et al. (2010) derived Demons forces from the squared difference between each element of the Log-Euclidean transformed tensors while taking into account the reorientation introduced by the transformation. A safer and more accurate evaluation of the respiratory movement will help in the selection of the appropriate medicine, the determination of the effectiveness of a treatment, to reduce the number of cases of a clinical trial, observe the progress of rehabilitation treatments, among other possible applications. The present work uses a novel and never-tried-before automatic approach for deformity estimation in a temporal sequence of thoracic CT images.

4.2 Background The background study of this chapter includes a detailed discussion of prominent works and algorithms studied in the process of proposition of this method. The algorithms and earlier proposed methods under discussion are categorically ordered keeping in mind their acute relevance and year of occurrence/proposition/publication to the scientific community. Propositions occurring at a later instant in timeline are given higher priority in discussion in comparison to earlier works to establish a better context. These methods are compared in a tabular format in Table C.1 in Appendix C. Sarrut et al. (2006) proposed to “simulate an artificial four-dimensional (4-D) CT image of the thorax during breathing”. It was performed by deformable registration of two CT scans acquired at inhale and exhale breath-hold. Breath-hold images were acquired with the ABC (Active Breathing Coordinator) system. Dense deformable registrations were performed. The method was a minimization of the sum of squared differences (SSD) using an approximated second-order gradient. Gaussian and linear elastic vector field regularizations were compared. A new preprocessing step, called a priori lung density modification (APLDM), was proposed to take into account lung density changes due to inspiration. It consisted of modulating the lung densities in one image according to the densities in the other, in order to make them comparable. Simulated 4-D images were then built by vector field interpolation and image re-sampling of the two initial CT images. A variation in the lung density was taken into account to generate intermediate artificial CT images. The Jacobian of the deformation was used to compute voxel values in Hounsfield units. The accuracy of the deformable registration was assessed by the spatial correspondence of anatomic landmarks located by experts. APLDM produced statistically significantly better results than the reference method (registration without APLDM preprocessing). The mean ± standard deviation of distances between automatically found landmark positions and landmarks set by experts were 2.7 ± 1.1 mm with APLDM, and 6.3 ± 3.8 mm without. Inter-expert variability was 2.3 ± 1.2 mm. The differences between Gaussian and linear elastic regularizations were not statistically significant. In the second experiment using 4-D images, the mean difference between automatic and manual

4.2 Background

59

landmark positions for intermediate CT images was 2.6 ± 2.0 mm. The generation of 4-D CT images by deformable registration of inhale and exhale CT images was found to be feasible. This might lower the dose needed for 4-D CT acquisitions or might even help to correct 4-D acquisition artifacts. Such a 4-D CT model could be used to propagate contours, to compute a 4-D dose map, or to simulate CT acquisitions with an irregular breathing signal. It could serve as a basis for 4-D radiation therapy planning. Despite these encouraging and fruitful projections, further work was found to be needed to make the simulation more realistic by taking into account hysteresis and more complex voxel trajectories. Stevo et al. (2009) proposed a method for registration of temporal sequences of coronal and sagittal images obtained from magnetic resonance (MR). They suggested that for each image in coronal and sagittal MRI sequences, the information contained in the intersection segment would be determined, and the matching would be done to determine the best sagittal images for each coronal image and vice versa. The final MR image registration would be the determination of the best images in a sequence that fits a chosen image in another sequence. One of the registration approaches used was determining the distance between the images by comparing pixel by pixel and combining these differences in a single value. The other one was Fourier transform based since Fourier description of an edge is also used for template matching. The resulting pairs from both algorithms were different. It was noticed that both pairs have a satisfactory visual registration. The temporal sequence of images represented discrete instants in time, and such an almost perfect fitting is considered very rare. The temporal registration algorithm based on a pixel-by-pixel comparison and Fourier transform showed several satisfactory results, however, it was found not possible to overcome the temporal low rate of image acquisition. One of the future works according to the authors could be the definition of a new registration algorithm combining pixel comparison and time segmentation. Castillo et al. (2010) suggested a four-dimensional deformable image registration (4D DIR) algorithm, referred to as 4D local trajectory modeling (4DLTM) and it was applied to thoracic 4D computed tomography (4DCT) image sets. The proposed method exploited the incremental continuity present in 4DCT component images to calculate a dense set of parameterized voxel trajectories through space as functions of time. The spatial accuracy of the 4DLTM algorithm was compared with an alternative registration approach in which component phase to phase (CPP) DIR is utilized to determine the full displacement between maximum inhale and exhale images. Cubic polynomials were found to provide sufficient flexibility and spatial accuracy for describing the point trajectories through the expiratory phases. The resulting average spatial error between the maximum phases was 1.25 mm for the 4DLTM and 1.44 mm for the CPP. The 4DLTM method was found to capture the long-range motion between 4DCT extremes with higher spatial accuracy (lesser spatial error). Ehrhardt et al. (2011) proposed statistical modeling for 4D respiratory lung motion using diffeomorphic image registration. It was an approach to generate a mean motion model of the lung based on thoracic 4D computed tomography (CT) data of different patients to extend the motion modeling capabilities. The modeling process consisted

60

4 A Path Tracing and Deformity Estimation Methodology …

of three steps: an intra-subject registration to generate subject-specific motion models, the generation of an average shape and intensity atlas of the lung as anatomical reference frame, and the registration of the subject-specific motion models to the atlas in order to build a statistical 4D mean motion model (4D-MMM). In all steps, a symmetric diffeomorphic nonlinear intensity-based registration method was employed. The model was evaluated by applying it for estimating respiratory motion of ten lung cancer patients. The prediction was evaluated with respect to landmark and tumor motion, and the quantitative analysis resulted in a mean target registration error (T RE ) of 3.3 ± 1.6 mm. With regard to lung tumor motion, it was shown that prediction accuracy is independent of tumor size and motion amplitude in the considered data set. The statistical respiratory motion model was found to be capable of providing valuable prior knowledge in many fields of applications. Authors also presented two examples of possible applications of the proposed method in radiation therapy and image-guided diagnosis. Sato et al. (2011) proposed a method for registration of temporal sequences of coronal and sagittal MR images through respiratory patterns. This work discussed the determination of the breathing patterns in a time sequence of images obtained from magnetic resonance (MR) and their use in the temporal registration of coronal and sagittal images. A time sequence of this intersection segment of orthogonal coronal and sagittal sequences were stacked, defining a two-dimension spatio-temporal (2DST) image. An interval-Hough transform algorithm was used to search for synchronized movements with the respiratory function. A greedy active contour algorithm would adjust small discrepancies originated by asynchronous movements in the respiratory patterns. The results of the proposed method in the form of synchronized sequences were compared with the pixel-by-pixel comparison method. The proposed method increased the number of registered pairs representing composed images and allowed an easy check of the breathing phase. Xiong et al. (2012) proposed a method for tracking the motion trajectories of junction structures in 4D CT images of the lung. It was hailed as a novel method to detect a large collection of natural junction structures in the lung and use them as reliable markers to track lung motion. The image intensities within a small region of interest surrounding the center were selected as its signature. Under the assumption of the cyclic motion, the trajectory was described by a closed B-spline curve and search for the control points by maximizing a metric of combined correlation coefficients. Local extremas were suppressed by improving the initial conditions using random walks from pair-wise optimizations. Several descriptors were also introduced to analyze the motion trajectories. The method was applied to 13 real 4D CT images. More than 700 junctions in each case were detected with an average positive predictive value of greater than 90%. The average tracking error between automated and manual tracking was in the sub-voxel category and smaller than the published results using the same set of data. Zhang et al. (2013) proposed a method for modeling respiratory motion to reduce motion artifacts in 4D CT images. A patient-specific respiratory motion model was proposed, based on principal component analysis (PCA) of motion vectors obtained from deformable image registration, with the main goal of reducing image artifacts

4.2 Background

61

caused by irregular motion during 4D CT acquisition. Displacement vector fields relative to a reference phase were calculated using an in-house deformable image registration method. The authors then used PCA to decompose each of the displacement vector fields into linear combinations of principal motion bases. These projections were parameterized using a spline model to allow the reconstruction of the displacement vector fields at any given phase in a respiratory cycle. Finally, the displacement vector fields were used to deform the reference CT image to synthesize CT images at the selected phase with much reduced image artifacts. The initial large discrepancies across the landmark pairs were significantly reduced after deformable registration, and the accuracy was similar to or better than that reported by state-ofthe-art methods. The motion model was used to reduce irregular motion artifacts in the 4D CT images of three lung cancer patients. Visual assessment indicated that the proposed approach could reduce severe image artifacts. The proposed approach was found able to mitigate shape distortions of anatomy caused by irregular breathing motion during 4D CT acquisition. Fuerst et al. (2014) proposed a patient-specific biomechanical model for the prediction of lung motion from 4-d ct images. It was an approach to predict the deformation of the lungs and surrounding organs during respiration. It was basically, a computational model of the respiratory system, which comprised of an anatomical model extracted from computed tomography (CT) images at end-expiration (EE), and a biomechanical model of the respiratory physiology, including the material behavior and interactions between organs. The method was then tested on five public datasets. Results showed that the model was able to predict the respiratory motion with an average landmark error of 3.40 ± 1.0 mm over the entire respiratory cycle. The estimated 3-D lung motion may be constituted as an advanced 3-D surrogate for more accurate medical image reconstruction and patient respiratory analysis.

4.3 Method 4.3.1 Preparation The dataset used comprised of a total (3 × 6) × 10, i.e., 180 thoracic CT images across 10 subjects. There were six frames from a temporal thoracic image sequence each for every Anatomical Plane (AP), i.e., Axial (supine), Coronal, and Sagittal for all the 10 subjects acquired simultaneously with a gap of 0.1 s starting t = 0 to 0.6 s. All images were identified as I NAP (x, y, t) where   from time + N , t ∈ R |1 ≤ N ≤ 10, 0.1 ≤ t ≤ 1 and (x, y) are the coordinates in the Cartesian plane, t being the timestamp at which the particular frame/image was recorded, N would be the number assigned to the test subject and AP signifies the three anatomical planes of view, i.e., Axial (a), Coronal (c) and Sagittal (s). So, the sixth subject’s Coronal CT image acquired at t = 0.3 s would be identified as I6c (x, y, 0.3). Samples

62

4 A Path Tracing and Deformity Estimation Methodology …

Table 4.1 Working database through all anatomical planes from t = 0.1 to 0.6 s Axial

Coronal

Sagittal

1 2 3 4 5 6 7 8 9 10

of images used from all viewpoints and all subjects from timestamps 0.1–0.6 s are summarized in Table 4.1.

4.3.2 Proposed Methodology The procedure acquired is as such that a temporal thoracic image sequence from time t = 0.1 to 0.6 s is taken such that first frame of the sequence is the full inhale frame and the last frame is full exhale frame. This paper uses the Speeded up Robust Feature detector (SURF) (Bay et al. 2006, 2008) to obtain a feature set comprising of common feature points throughout the image sequence. It detects and describes the feature set irrespective of any scaling and/or rotation in the corresponding images. SURF provides better approximations in comparison to previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster than any other state-of-the-art feature detector. These feature sets are then fed into the OFM estimation algorithm to identify the deformation path throughout the temporal sequence, be it peripheral or local. Optical flow has been successfully applied to motion estimation of points/point clouds and other point set surface definitions over a temporal sequence (Sun et al. 2014). It performs better than its contemporaries while tracing deformations that are realistic and guides the user in manipulation of real-world objects. It also allows the user to specify the deformations using either sets of points or line segments, the later useful for controlling curves and profiles present in the image. For each of these techniques, it provides simple closed-form solutions that yield fast deformations,

4.3 Method

63

Fig. 4.1 The proposed framework structure

which can be performed in real-time. The proposed methodology aims to track and estimate the deformations by tracking the transition of the interest points through the sequence from full inhale to full exhale frame. The overall process can be referred to in Fig. 4.1. A novel scale- and rotation-invariant detector and descriptor, has been coined as Speeded Up Robust Features (SURF) by Bay et al. (2006, 2008). It provides better approximations in comparison to previously proposed schemes with respect to repeatability, distinctiveness, and robustness, yet can be computed and compared much faster. Focus is on scale and in-plane rotation-invariant detection and descriptions. These seem to offer a good compromise between feature complexity and robustness to commonly occurring photometric deformations in thoracic images. Skewing, anisotropic scaling and perspective effects are assumed to be second-order effects that are covered to some degree by the overall robustness of the descriptor. For guaranteed invariance to any scale changes the input thoracic images are analyzed at different scales. The detected interest points are provided with rotation and scale-invariant descriptor. The detector is based on Hessian-matrix-based on its good performance in accuracy (Bay et al. 2008). Blob-like structures are detected at locations with maximum determinant. In comparison to the Hessian–Laplace detector (Mikolajczyk and Schmid 2001) Hessian determinant is used for scale selection (Lindeberg 1998). Given a point a = (x, y) in an image I NA P , the Hessian matrix H (a, σ ) at scale σ is defined as follows   L x x (a, σ ) L x y (a, σ ) (4.1) H (a, σ ) = L yx (a, σ ) L yy (a, σ ) 



) where L x x (a, σ ) is the convolution of the Gaussian second derivative ∂g(σ with the ∂a 2 AP image I N at point a, similarly for L x y (a, σ ) and L yy (a, σ ). Though Gaussians are optimal for scale-space analysis (Koenderink 1984), they have to be made discrete and cropped in practice. This results in a loss in repeatability of the detector for thoracic CT image rotations around odd multiples of π /4.

64

4 A Path Tracing and Deformity Estimation Methodology …

Fig. 4.2 The working model of SURF

The SURF method consists of multiple stages to obtain relevant feature points from a sequence of thoracic images. The single SURF stages are (as shown in Fig. 4.2): 1. An integral image is constructed for each frame of the input thoracic image sequence, it allows for fast computation of box type convolution filters (Viola and Jones 2001). This enables very few memory accesses and hence results in drastic improvement in computational time (Cornelis and Gool 2008), which is especially crucial when we are dealing with a sequence of images. An integral image I NA P  (a) at a location a = (x, y)T represents the sum of all pixels in the input image within a rectangular region formed by the origin and a. I NA P (a) =

j≤y i≤x  

I NA P (i, j)

(4.2)

i=0 j=0

2. Candidate feature points are searched by the creation of a Hessian scale-space pyramid (SURF detector). Approximation of the Hessian as a combination of box filters allows fast filtering. High contrast feature points are selected. 3. Feature vector is calculated (SURF descriptor) based on its characteristic direction to provide rotation invariance. Feature vector is normalized for immunity to changes in lighting conditions. 4. Matching of descriptor vectors between the thoracic image sequence frames using distance measures such as Mahalanobis distance and Euclidean distances, etc. Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene (Warren and Strelow 1985). In recent times, the term optical flow has been co-opted by computer vision experts to incorporate related techniques from image processing and control of navigation, such as motion detection, object segmentation, time-to-contact information, focus of expansion calculations, luminance and motion compensated encoding and stereo disparity measurement (Beauchemin and Barron 1995). Sequences of ordered thoracic images allow the estimation of motion as either instantaneous image velocities or discrete image displacements (Aires et al. 2008). Barron et al. provided a performance analysis of a number of optical flow techniques. It emphasizes the accuracy and density of measurements (1994). Suppose we have a continuous thoracic image frame I NA P ; f (x, y, t) refers to the gray-level of (x, y) at time t. It represents a dynamic thoracic image as a function of position and time. Few assumptions also work in hindsight:

4.3 Method

65

Fig. 4.3 Flow of a common feature point (x, y) through a sequentially temporal thoracic image sequence with N frames, arrows indicate the changing velocity vector v

• The detected feature point moves but does not actually change intensity. • Feature point at location (x, y) in frame i is the feature point at (x + x, y + y) in frame i + 1 (detailed in Fig. 4.3). For making computation simpler and quicker the real-world three-dimensional (3-D + time) objects are transferred to a (2-D + time) case. Then the thoracic image can be described by the 2-D dynamic brightness function of I (x, y, t). Provided that in the neighborhood of the feature point, change of brightness intensity does not happen in the motion field, the following expression can be used: I (x, y, t) = I (x + δx, y + δy, t + δt)

(4.3)

Taylor series is used for the right-hand side of the above equation, to obtain I (x + δx, y + δy, t + δt) = I (x, y, t) +

∂I ∂I ∂I x + y + t ∂x ∂y ∂t

+ Higher order terms

(4.4)

From Eqs. 4.3 and 4.4; neglecting the higher order terms, ∂I ∂I ∂I x + y + t = 0 ∂x ∂y ∂t

(4.5)

Dividing the terms in Eq. 4.5 by t on both sides (to get the equation in terms of x, y component velocity) ∂ I x ∂ I y ∂I =0 + + t t ∂x ∂y ∂t

(4.6)

where x t = Vx , y t = Vy ; thus, ∂I ∂I ∂I Vx + Vy + =0 ∂x ∂y ∂t

(4.7)

where Vx and Vy are the x and y components of velocity or optical flow of I (x, y, t); ∂I ∂I , and ∂∂tI being the spatio-temporal derivatives of I (x, y, t). ∂x ∂y

66

4 A Path Tracing and Deformity Estimation Methodology …

Ix · vx + I y · v y = −It

(4.8)

Vector representation being ∇ I · v == −It

(4.9)

where ∇ I is the spatial gradient of brightness intensity and v is the optical flow (velocity vector) of the previously detected feature points, It being the time derivative of the brightness intensity.

4.4 Results and Discussion The feature detector/descriptor implemented on the temporal image sequence gave out matching feature points among the six continuous frames of the thoracic continuous temporal image sequence (0.1 ≤ t ≤ 0.6) where t is the timestamp of frames in the sequence for all Anatomical Positions (AP) with average translation values. The average translation between inter-frame durations for all common points “P” from the initial to final frame: P P P Pi=1 d1 Pi=1 d2 Pi=1 d N −1 d1avg = , d2avg = · · · d N −1avg = , P P P Figures 4.4, 4.5 and 4.6 indicate the image registration process from the sequence for all test subjects through all three APs. Though the proposed method was applied to all the subject data at hand, for representation purposes, subject “case 5” sagittal AP data has been extensively used (as can be seen in Figs. 4.7, 4.8, 4.9, 4.10 and 4.11. The temporal sequence starting

Fig. 4.4 Image sequence frames and the registered image for all subjects-Axial

4.4 Results and Discussion

67

Fig. 4.5 Image sequence frames and the registered image for all subjects-Coronal

Fig. 4.6 Image sequence frames and the registered image for all subjects-Sagittal

from t = 0 to t = 0.6 s is considered with a gap of 0.1 s between two consecutive frames in the sequence. So, frame 1 is the one acquired at t = 0.1 and frame 6 is the one corresponding to t = 0.6 s. The feature points are color-coded with respect to the indices and IDs assigned to them throughout the process. The trails/tracks they leave after motion also exhibit the same color combination as assigned to respective feature points. There were 242 such feature points for the “case 5” sagittal AP image sequence, the attributes of which are shown in the Table E.1 (Appendix-E). Each of them had a track associated with them; these tracks have been labeled as “Track_‘point no’” where the value of “point no.” ranges from 0 to 241. Other attributes associated with each track included “Track_Duration” which indicated the time in seconds for that respective track to finish and the point to reach its ultimate frame. “Track_Start” is the stating time of every feature point trail/track; the value is “0” for all points, first frame being the reference frame for registration. “Track_Stop” is the end time in

68

4 A Path Tracing and Deformity Estimation Methodology …

Fig. 4.7 The test image temporal sequence (accordingly labeled). Subject “case 5” Sagittal AP

Fig. 4.8 The color-coded feature points and their colored trails showing the distinct paths for Sagittal AP “case 5”, frames are labeled in order of their temporal sequence

4.4 Results and Discussion

69

Fig. 4.9 The registered image for the corresponding temporal sequence for subject “case 5” Sagittal AP

Fig. 4.10 Color-coded optic flow for subject “case 5” sagittal AP with flow orientation scheme

70

4 A Path Tracing and Deformity Estimation Methodology …

Fig. 4.11 The overall image sequence optic flow with magnitude scale

seconds for respective tracks; values may be different for different feature points. “Track_Displacement” is the net displacement from the point of origin for a feature point over the sequence. “Track_(X, Y ) Locations” are the points of origin of the respective tracks. “Track_Min/Max/Mean Speeds” are the minimum, maximum, and mean speeds of the feature point trail/track for each point through the sequence. The displacement/translation obtained is inherently in pixel units. With the knowledge of PPI (pixel per inch) value of the respective images in question, the displacements can be converted into more tangible units. These average translations for all such feature points for all test subjects through all three APs are shown in Tables 4.2, 4.3 and 4.4. Their corresponding line plots for all 10 subjects are shown as Figs. 4.12, 4.13 and 4.14 for easier comparative analysis over the complete breathing pattern. A corresponding false color registered image representation is shown as Fig. 4.9. Optical flow representation of the image sequence with respect to the registered image along with a flow orientation scheme is shown in Fig. 4.10. The optical flow at any point in the image can be decoded using the flow orientation scheme coding pinwheel given alongside. There was a rather large strip of single color found in the optical flow representation, which is synonymous with the false color representation in Fig. 4.9. That is the location with maximum displacement/translation in the sequence and also of maximum deformation with respect to the reference frame. Where Fig. 4.10 indicated the optical flow orientation, magnitude of the optical flow is an important aspect that can be ignored when observing an image sequence over time. Figure 4.11 represents the optical flow magnitude spread over the complete

Case 1

0.047

0.074

0.121

0.236

0.165

Slices

1

2

3

4

5

0.087

0.041

0.090

0.054

0.000

Case 2

AXIAL average translation (pixels)

0.235

0.077

0.078

0.212

0.050

Case 3

0.054

0.120

0.260

0.263

0.128

Case 4

0.229

0.335

0.217

0.220

0.103

Case 5

Table 4.2 Average translations (in pixels) for all test subjects through Axial AP

0.277

0.224

0.232

0.192

0.122

Case 6

0.175

0.123

0.160

0.173

0.105

Case 7

0.227

0.236

0.197

0.176

0.081

Case 8

0.346

0.273

0.154

0.157

0.148

Case 9

0.415

0.550

0.491

0.235

0.235

Case 10

4.4 Results and Discussion 71

Case 1

0.049

0.257

0.544

0.555

0.361

Slices

1

2

3

4

5

0.381

0.396

0.451

0.445

0.241

Case 2

CORONAL average translation (pixels)

0.529

0.443

0.490

0.284

0.090

Case 3

0.495

0.414

0.436

0.441

0.337

Case 4

0.682

0.617

0.574

0.545

0.090

Case 5

Table 4.3 Average translations (in pixels) for all test subjects through Coronal AP

0.532

0.458

0.434

0.444

0.272

Case 6

0.503

0.522

0.547

0.574

0.413

Case 7

0.645

0.700

0.600

0.563

0.316

Case 8

1.432

1.508

1.541

1.515

0.705

Case 9

0.586

0.707

2.594

0.587

0.389

Case 10

72 4 A Path Tracing and Deformity Estimation Methodology …

Case 1

0.056

0.067

0.229

0.120

0.131

Slices

1

2

3

4

5

0.042

0.125

0.144

0.038

0.102

Case 2

SAGITTAL average translation (pixels)

0.027

0.041

0.036

0.031

0.033

Case 3

0.237

0.228

0.184

0.225

0.198

Case 4

0.079

0.092

0.131

0.081

0.022

Case 5

Table 4.4 Average translations (in pixels) for all test subjects through Sagittal AP

0.545

0.483

0.511

0.515

0.218

Case 6

0.504

0.574

0.504

0.451

0.283

Case 7

0.659

0.666

0.639

0.603

0.387

Case 8

0.326

0.374

0.336

0.410

0.318

Case 9

0.505

0.521

0.476

0.439

0.348

Case 10

4.4 Results and Discussion 73

74

4 A Path Tracing and Deformity Estimation Methodology …

Fig. 4.12 Average displacement for all subjects in Axial AP

Fig. 4.13 Average displacement for all subjects in Coronal AP

4.4 Results and Discussion

75

Fig. 4.14 Average displacement for all subjects in Sagittal AP

sequence with the first frame as reference. As can be seen from the magnitude scale provided alongside, the bigger red arrows indicate areas with higher magnitude of flow and larger deformations, while the blue and black arrows indicate areas lower optical flow magnitude and smaller deformations in respective locations. As we can see in Fig. 4.12, the axial translations were recorded highest for subject “case 10” and the lowest corresponding values were for “case 2”. The average value for “case 10” was recorded at 0.3851 pixels, which was way above the population average of 0.184 shown by a line across the plot. In case of coronal AP as can be seen in Fig. 4.13, the biggest deformations throughout the sequence are exhibited by the subjects “case 10” and “case 9” at 2.594 and 1.54 pixels, respectively. The population average, in this case, being 0.5847 marked by a straight line in the corresponding plot. Though apart from “case 10” only “case 9” exhibited bigger deviations than the average value, the change in deformation with respect to inter-frame durations was more or less constant; on the other hand “case 10” exhibited enormous shift from the average value while transitioning from 3rd frame to 4th frame. Looking at Fig. 4.14 for the sagittal AP, all subjects though a bit above and below the average maintain an almost constant rate of change in the deformations and do not exactly exhibit any erratic patterns through the observed full inhale to exhale process. After having a comprehensive look at all subjects’ deformation pattern data through axial, coronal, and sagittal APs collectively, it was inferred that subject “case 10” singled out as the only one with maximum deformation. This analysis indicates anomalous breathing patterns from the aforementioned subject among the considered consensus average.

76

4 A Path Tracing and Deformity Estimation Methodology …

4.5 Conclusion A framework has been presented showing how to use a feature point set generated using a Hessian-matrix-based feature detector and Haar wavelets based descriptor such as SURF through a motion estimation technique such as OFM tracking for deformable image transformations in medical images such as the thoracic “pectus excavatum” (Haller et al. 1987; Kim et al. 2010) full exhale and full inhale used in this work. This conclusion is of high clinical relevance from a diagnostic point of view as well; the artifacts and position uncertainties due to uneven breathing patterns which hamper the image-guided clinical interventions can be corrected to a point where there influence on the actual data and the diagnostics based on them is brought down to the least. This work can be looked upon as an automatic way of deformable image registration for high contrast medical images using landmark (control) points. Although the proposed methodology provides a fast and accurate way of DIR for medical images and thus an account of deformity in the thoracic periphery, there is much scope for improvement in the overall process. One way this can be achieved in the future is by modifying the SURF and/or the Motion estimation procedure involved in the process. Another way is to improve and enhance the quality as well as the quantity of the database used. Also, the aforementioned procedure can provide better results if applied for different human anatomy altogether. However diligently and accurately it may have been done, there might still be some scope of improvement and betterment in the methodology and also in its presentation. The search and pursuit of better methods for deformable medical image registration are still on.

Chapter 5

Deformable Thoracic CT Images Sequence Registration Using Strain Energy Minimization

The idea of deformable image registration (DIR) has been explored for a thoracic CT (computed tomography) image database of ten subjects. Thoracic CT image acquisition for clinical interventions requires a well-defined procedure that has already been underlined on the basis of field expertise and past experiences. Despite strict adherence to the procedure, the acquired images are prone to distortions and artefacts. This might happen due to organ motion during the breathing process (at times even in breath-hold procedures), slight (even involuntary) movements or acquisition variations in supine and prone positions, etc. An intensity differences based energy minimization method has been proposed. The moving image is transformed in the process such that it gets maximum alignment with the fixed image. This is achieved by energy minimization of the moving image in an iterative process. It is a simpler and more practical method for thoracic CT image registration than the prevalent approaches. This has been shown by lower mean registration errors for the patient data; the errors were as such axial: 0.283 ± 0.08, coronal: 0.784 ± 0.32 and sagittal: 0.66 ± 0.2 pixels. This registration of moving image onto the fixed image in the sequence will help in minimizing the adverse effects of the otherwise present discrepancies, phase errors, and discontinuity artifacts that might have crept in during the acquisition. The proposed method begins with a pair of images of the same dimensions; these images are part of an image sequence and have a considerable temporal difference between them. The image sequence has been acquired as a part of the breathing process. Of the two, image appearing earlier in the temporal timeline is considered as the target image, and the one appearing later is considered as the source image. Both images represent the extremes of a breathing cycle such that the first image is full inhale and the last image is full exhale. Both images have their own specific energy signatures. Both these images have to be registered against each other. For the registration process, no direct comparisons between images are done; instead, the source image is independently transformed in such a way that the transformed image has minimum intensity difference with the target image. It is an iterative process (as can be referred to in Fig. 5.2), at each stage of which transformed versions of source image are compared to the target image for intensity difference of zero or less than © Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0_5

77

78

5 Deformable Thoracic CT Images Sequence Registration …

a third decimal place value. If none of the two conditions are met, the transformed image goes into further transformation and the process continues until the source image is transformed to a level that it satisfies previously laid conditions. In our experiments, SNR (signal to noise ratio), PSNR (peak SNR), mean SSIM (Structural SIMilarity) index and NCC (Normalized Cross-Correlation) have been used to estimate and establish increased similarity between the later transformed–target image pair in comparison to previous source–target image pair. Mean Registration Error E T −R is used as the quantitative measure for the evaluation of performance. The E T −R obtained for the dataset was found to be considerably lower than more traditional and prevalent transforms such as affine and b-splines based approaches.

5.1 Introduction Organ motion pertaining to breathing can lead to image artefacts and position uncertainties during image-guided clinical interventions. A particular case for such imageguided interventions (IGI) can be the radiotherapy planning of thoracic and abdominal tumors; the respiratory motion causes important uncertainties and is a significant source of error (Keall et al. 2006). During a process of image acquisition, slight movement from the subject can translate into potential discrepancies in the acquired image sequence. Images in such an acquired sequence more than often end up out of sync and prove to be not of much use for both medical applications and/or research purposes. A non-invasive method to describe lung deformations was proposed using NURBS surfaces based on imaging data from CT scans of actual patients (Tsui et al. 2000). Image registration has recently started playing an important role in this scenario; it helps in the estimation of any motion caused due to breathing during acquisition and the description of the temporal change in position and shape of the structures of interest by establishing the correspondence between images acquired at different phases of the breathing cycle (Ehrhardt et al. 2011). Image Registration is the alignment/overlaying of two or more images so that the best superimposition can be achieved. These images can be of the same subject at different points in time, from different viewpoints or by different sensors. This way the contents from all the images in question can be integrated to provide richer information. It helps in understanding and thus reducing the differences that occurred due to variable imaging conditions. Most common applications of Image Registration include remote sensing (integrating information for GIS), combining data obtained from a variety of imaging modalities (combining a CT and an MRI view of the same patient) to get more information about the disease at once, cartography, image restoration etc. An image registration method targets to find the optimal transformation that aligns the images in the best way possible. Image registration methods can be broadly classified into three basic classes, landmark (or point) based registration (Mcgregor 1998; Rohr et al. 2001; Bookstein and Green 1993), segmentation based registration (Sull and Ahuja 1995; Feldmar and Ayache 1996; Jain et al. 1996) and the image intensity-based registration (Szeliski and Coughlan 1994; Kybic and

5.1 Introduction

79

Unser 2003) depending on them being more cost-efficient, fast and flexible over the others with respect to the image family it is being used to register and the application of the registration process. It is further categorized into two kinds based on the type of image it is being applied for. The two kinds of images are Rigid Images and Deformable Images. Rigid images are those of structures with rigid morphological properties, e.g., bones, buildings, geographical structures, etc. If the underlying transformation model allows local deformations, i.e., nonlinear fields’ u(x), then it is called Deformable Image Registration (DIR) (Muenzing et al. 2014). Deformable images are those of structures shape and size of which can be modeled after tangible physically deformable models (Sotiras et al. 2013). Rigid image registration although is an important aspect of registration it is not the topic of discussion in this article. Since the discussion is about Medical Image Registration and almost all anatomical parts or organs of the human body are deformable structures, the concentration here is on DIR (Oliviera and Tavares 2012). The proposed methodology is based on intensity-based registration. It is fully automatic in its mode of operation and helps in faster and more accurate image registration in comparison to pure landmark-based registration methods. This factor gives our method an upper hand when it comes to real-life medical image registration problems. The intensity-based energy minimization methodology seems more practical, stable, and cost-efficient for deformable images in comparison to landmark-based or segmentation based methodologies for similar purposes. The method is simpler and faster than its contemporaries because the energy function is worked upon directly without solving large matrix system assemblies.

5.2 Background The background study of this chapter initially includes a study of few most prominent proposed algorithms in the direction of study of the energy minimization based nonlinear elastic image registration and its applications. Then the proposed methods relating to image registration of thoracic CT images are discussed. The propositions are categorically discussed keeping in mind their acute relevance and their year of occurrence. Propositions occurring at a later instant in timeline are given higher priority in terms of detailed discussion in comparison to earlier works to establish a better context. These methods are compared in a tabular format in Table D.1 in Appendix D. Pennec and associates (Pennec et al. 2005) suggested a statistical regularization framework for nonlinear registration based on the concept of Riemannian Elasticity. In the proposed method, elastic energy has been interpreted as the distance of the Green-St. Venant strain tensor to the identity, which in turn reflects the deviation of the local deformation from a rigid transformation. By changing the usually employed Euclidean metric for a more suitable Riemannian one, a consistent statistical framework has been defined to quantify the amount of deformation. These statistics were then used as parameters in a Mahalanobis distance to measure the statistical deviation

80

5 Deformable Thoracic CT Images Sequence Registration …

from the observed variability, giving a new regularization criterion that is called the statistical Riemannian elasticity. It was found that this new criterion is able to handle anisotropic deformations and is inverse consistent. Preliminary results and observations showed that it can be quite easily implemented in a non-rigid registration algorithm. Bao Zhang and associates (Zhang et al. 2011) proposed a three-dimensional elastic image registration methodology based on strain energy minimization with its application to prostate magnetic resonance imaging. The registration algorithm was also applied on ten sets of human prostate data, each with two typical deformation states (one with 0 cc of air and the other with 40–60 cc of air inflated in the endorectal coil balloon). There were a total of 200–400 landmarks used to derive the transformation depending on the size of each prostate. They described it as a novel 3-D elastic registration procedure that is based on the minimization of a physically motivated strain energy function that requires the identification of similar features (points, curves, or surfaces) in the source and target images. The Gauss–Seidel method was used in the numerical implementation of the registration algorithm. The registration procedure was validated on synthetic digital images, MR images from a prostate phantom, and MR images obtained on patients. Registration errors were assessed by averaging the displacement of a fiducial landmark in the target to its corresponding point in the registered image. The registration error on patient data was 1.8 ± 0.7 pixels. Registration also improved image similarity (normalized cross-correlation) from 0.72 ± 0.10 to 0.96 ± 0.03 on patient data. Registration results on prostate data in vivo demonstrated that the registration procedure could be used to significantly improve both the accuracy of localized therapies such as brachytherapy or external beam therapy and can be valuable in the longitudinal follow-up of patients after therapy. Ronald W. K. So and associates (So et al. 2011) proposed a technique for nonrigid image registration of brain magnetic resonance images using graph cuts. A graph-cut-based method was proposed for non-rigid medical image registration on brain magnetic resonance images. In this proposal, the non-rigid medical image registration problem has been reformulated as a discrete labeling problem. They modeled the non-rigid registration as a multi-labeling problem by Markov random field. The image registration problem was therefore modeled by two energy terms based on intensity similarity and smoothness of the displacement field. The MRF energy was minimized using graph cuts algorithm via α-expansions. The registration results of the proposed method were compared with two state-of-the-art medical image registration approaches: free-form deformation based method and demons method. In addition, the registration results were also compared with that of the linear programming based image registration method. The proposed method was found to be more robust against different challenging non-rigid registration cases with consistently higher registration accuracy than those three methods and gives realistic recovered deformation fields. Andrew R. Dykstra and associates (Dykstra et al. 2012) proposed a method that coregisters high-resolution preoperative MRI with postoperative computerized tomography (CT) for the purpose of individualized functional mapping of both normal and pathological (e.g., interictal discharges and seizures) brain activity. The proposed method accurately (within 3 mm, on average) localizes electrodes with respect to

5.2 Background

81

an individual’s neuroanatomy. Furthermore, they outlined a principled procedure for either volumetric or surface-based group analyses. The method was demonstrated in five patients’ data with medically-intractable epilepsy undergoing invasive monitoring of the seizure focus prior to its surgical removal. Accuracy of the method was found within 3 mm of average. The straightforward application of this procedure to all types of intracranial electrodes, robustness to deformations in both skull and brain, and the ability to compare electrode locations across groups of patients make this procedure an important tool for basic scientists as well as clinicians. H. P. Heinrich and associates (Heinrich et al. 2013) proposed an MRF-Based Deformable Registration and Ventilation Estimation of Lung CT. In the proposed method three major challenges associated with lung ct registration viz. large motion of small features, sliding motions between organs, and changing image contrast due to compression are addressed and potentially higher quality of discrete approaches is preserved. First, an image-derived minimum spanning tree is used as a simplified graph structure, which coped well with the complex sliding motion and allowed to find the global optimum very efficiently. Second, a stochastic sampling approach for the similarity cost between images is introduced within a symmetric, diffeomorphic B-spline transformation model with diffusion regularization. The complexity is reduced by orders of magnitude and enables the minimization of much larger label spaces. In addition to the geometric transform labels, hyper-labels are introduced, which represent local intensity variations in this task, and allow for the direct estimation of lung ventilation. The improvements are validated in accuracy and performance on exhale-inhale CT volume pairs using a large number of expert landmarks. The three challenges posed in the beginning are met. Keita Nakagomi and associates (Nakagomi et al. 2013) proposed a segmentation based registration methodology that uses multi-shape graph cuts with neighbor prior constraints for lung segmentation from a chest CT volume. A novel graph cut algorithm has been proposed that can take into account multi-shape constraints with neighbor prior constraints, and reports on a lung segmentation process from a three-dimensional computed tomography (CT) image based on this algorithm. It is a novel segmentation algorithm that improves lung segmentation for cases in which the lung has a unique shape and pathologies such as pleural effusion by incorporating multiple shapes and prior information on neighbor structures in a graph cut framework. The efficacy of the proposed algorithm is demonstrated by comparing it to the conventional one using a synthetic image and clinical thoracic CT volumes.

5.3 Method 5.3.1 Preparation The dataset used comprised of a total (3 × 10) × 10, i.e., 300 thoracic CT images across 10 subjects. All images were anonymized and all procedures followed were in accordance with the ethical standards of the responsible committee on human

82

5 Deformable Thoracic CT Images Sequence Registration …

experimentation (institutional and national) and with the Declaration of Helsinki 1975, as revised in 2008 (5). Informed consent was obtained from all patients for being included in the study. All patients or legal representatives signed informed consent. The images lie between CT phases 0–5, i.e., end-inspiration to end-expiration in timestamp range t00 → t05. The image dimensions lie between 396 × 396 and 432 × 400 pixels. There were 6 frames from a temporal thoracic image sequence each for every Anatomical Plane (AP), i.e., Axial (supine), Coronal, and Sagittal for all the 10 subjects acquired simultaneously with a gap of 0.1 s starting AP from time t = 0.1 to 0.6 s. All images  were identified as I N (x, y, t) where  + N , t ∈ R |1 ≤ N ≤ 10, 0.1 ≤ t ≤ 0.6 , (x, y) are the x and y coordinates in the Cartesian plane and AP signifies the three anatomical planes of view, i.e., Axial (a), Coronal (c), and Sagittal (s). Suppose the 3rd frame from coronal AP for subject “case 9”, would be identified as I9c (x, y, 0.3). A view of the image database is shown in Table 5.1 for representational purposes.

5.3.2 Proposed Methodology What we have is a temporal sequence of images starting from time t = 0.1 to t = 0.6 s. It starts from the end-inspiration phase and continues up to the end-expiration phase of the breathing cycle. The last image of the aforementioned sequence being diametrically most deformed with respect to the first image. We have proposed a method to register these two images with respect to each other. The two images are the target (T ) and the source images (S) at t = 0.1 and t = 0.6 s, respectively. These images belong to the same domain  and are related through a transformation   TR . This transformation is such that the resulting transformed image S  has the minimum energy distribution difference in terms of a similarity measure with the target image T, this has been shown in Fig. 5.1. In simpler terms it can be stated as: “a transformation sought such that the transformed image has minimum intensity difference with the target image”. There is potential energy associated with an elastic system at a time. Since the images involved in the study are of a human body organ, they can be categorized as non-rigid or deformable images and the energy principles of elastic systems are applicable to this set of images. Potential energy of an elastic two-dimensional system at static equilibrium is pure strain energy; it can be defined as (Ugural and Fenster 2003): ¨     (5.1) 1/2 λe2 + 2μ εx2 + ε2y + μγx2y d U= 

where  is the image dimension, λ is the tensile stress (engineering constant), μ is the shear modulus, together they are called the Lame’ constants; εx and ε y are normal

5.3 Method

83

Table 5.1 All three anatomical viewpoints for all the 10 subjects at time t = 0.1 and 0.6 s Anatomical planes (T and S images) Axial

Coronal

Sagittal

1

2

3

4

5

6

7

8

(continued)

84

5 Deformable Thoracic CT Images Sequence Registration …

Table 5.1 (continued) Anatomical planes (T and S images) Axial

Coronal

Sagittal

9

10

Fig. 5.1 Overview of the proposed methodology

strains in the x and y directions, respectively, γx y is the shear strain in the x-y plane pointing towards the y direction and “e” is the unit change in image dimensions. The Poisson’s ratio value for Lung tissue averages close to 0.46 (Al-Mayah et al. 2008; Brock et al. 2005; Sundaram and Gee 2005; Zhang et al. 2004). In Eq. (5.1), the first term λe2 can be ignored since it is two orders lower than the rest of the terms. This makes the energy expression independent of tensile stress λ: ¨ U=

    1/2 2μ εx2 + ε2y + μγx2y d



This can be further simplified to: ¨ U =μ 



  εx2 + ε2y + 1/2γx2y d

(5.2)

5.3 Method

85

Suppose u, v are the displacements in x and y directions, respectively. Normal extension in the direction “a” (a = x, y); shear strain γab in strain εa is defined as original length the plane a-b would be the sum of angle of shear (for smaller degrees of shear). Thus, , similarly γx y = ∂u + ∂∂vx . Exacting these values to Eq. (5.2): εx = ∂∂ux and ε y = ∂v ∂y ∂y

 ¨  2 2  ¨  ∂u ∂u ∂v 2 ∂v 1 U =μ + + dxdy + μ dxdy ∂x ∂y 2 ∂y ∂x

(5.3)

So, the expression for energy function “U” in Eq. (5.1) has been reduced to strictly a strain energy function in Eq. (5.3), Eq. (5.3) hence can be rewritten for Ustrain as: Ustrain

 ¨  2 2  ¨  ∂u ∂u ∂v ∂v 2 =μ + dxdy + 1/2μ dxdy (5.4) + ∂x ∂y ∂y ∂x

The strain energy “Ustrain ” minimization requires that over the image boundary conditions between the source and the target images: δUstrain = 0

(5.5)

Such that the minimization constraint can  be expressed in terms of intensity difference between the transformed image S  and the target image (T ) over the image dimensions’ () as: (5.6) (I S  − IT )d = 0 

It is an iterative process; as we can see in Fig. 5.2, during the iteration, each time a transformed image is obtained, it is compared against the fixed image and an intensity difference mapping and value is calculated. These intensity differences are checked at each step. If very little or negligible change (say up to third decimal place) is observed, the iteration is stopped and the finally transformed image is considered as the required registered image. In case of progressively changing intensity differences for consecutive iterations, the iteration is continued until the stopping factor comes into play.

5.4 Results and Discussion Iterative energy minimization using intensity differences across the image boundaries yields a transformed image (S  ) which was pitted against the actual target image (T ) at different stages of the iteration to assess the level of transformation. Out of the ten subjects’ data at hand, the coronal AP of subject “case 3” has been chosen to elaborate

86

5 Deformable Thoracic CT Images Sequence Registration …

Fig. 5.2 Flowchart of the iterative process in the registration procedure

and demonstrate the proposed technique with results. The transformed image (S  ) after the complete registration process showed an increase of 51.64% SNR (signalto-noise ratio) value with respect to the target image (T ) in comparison to the source image (S) with respect to the target image. The change in PSNR (peak SNR) value was recorded at 41.64% in S  -T in comparison to S-T pair. A new metric called the SSIM (Structural Similarity) index has been used (Wang et al. 2004). It has been used to estimate and measure the similarity between two images. It has been used as a deciding metric which would give a percentage similarity between the two images in question, i.e., the fixed and the moving image and the fixed-transformed image pair. The mean SSIM index for the S-T pair was calculated at 0.4975, the same index for the S  -T pair came at 0.735. Along with similarity measures such as SNR, PSNR, and m-SSIM, NCC (normalized cross-correlation) has been used to demonstrate as to how close the transformed image (S  ) has come to the target image (T ) as a result of the registration process. The NCC value for S-T pair was estimated at 0.8817, for the S  -T pair it was calculated at a higher value of 0.9749 which further helps in

5.4 Results and Discussion

87

establishing the closeness of the transformed image to the target source and hence, the proposed methodology as an efficient deformable image registration approach. The earlier discussed iterative process and how it results in the finally registered image has been shown in Fig. 5.3. Figure 5.3a, b are the fixed and moving images, respectively, they are also the diametrically opposite images of a breathing cycle (i.e., full inhale and full exhale) in a respiration process. Figure 5.3c is the intensity difference mapping (IDM) of (a) and (b) before the iteration starts. Transformation is applied to the moving image and the transformed image is obtained. An IDM and corresponding value are calculated for the newly transformed moving image and the fixed image. Changes in IDM and value for current and previous stage is observed, if the change is zero or negligible in comparison to the intensity difference value at either of the two stages of the iteration, the iteration is stopped there and last transformed image is the registered image. Figure 5.3d is the transformed image at the 7th iteration, Fig. 5.3e is its IDM with respect to the fixed image. In this particular instance of subject “case 3”, it took 174 iterations to obtain the finally registered image which is Fig.5.3l, m is the final IDM indicating minimal difference of the registered image with respect to the fixed image indicating a seamless and smooth registration process. Figure 5.3f, g are the transformed and IDM (with the fixed image) images at 20th iteration; Fig. 5.3h, i are then transformed and IDM (with the fixed image) at the 55th iteration; similarly, Fig. 5.3j, k are the same at the 130th iteration. Figure 5.3n, o are the deformation vector and deformation field representations, respectively, for the finally registered image. Figure 5.4 shows the energy minimization process for subject “case 3” coronal AP, the iterative process continues until a finally registered image is obtained at 174th iteration (that is where the minimization process stops). The initial descent was observed as fast with respect to iterations until 110th iteration, after which the minimization process progresses with diminutive changes in intensity differences. It finally picks up at 124th iteration until to finally finish the process at 174th. The proposed technique was practically implemented on all the subject data at hand, i.e., three anatomical positions across ten subjects. After obtaining the finally registered images for a complete dataset, they were pitted against the fixed images of their own sequence’s respective sub-datasets. Similarity metrics such as SNR, pSNR, mean-SMIM index, and NCC were calculated and compared for each S-T and RT pairs for improvements (if any) which might suggest closeness of the registered image towards the fixed image. The observations are collected in Table 5.2, they are average values over the complete dataset through all APs; all similarity metrics clearly seem to improve from S-T to R-T image pair for all subjects. Where there are significant changes in the case of coronal and sagittal APs, respective changes are not as notable in axial AP’s data, this can be explained by usually comparatively smaller deformations in the “anterior-posterior” direction. As can be seen in Fig. 5.5, the mean registration errors (E_(T-R)) obtained for all the subjects involved in the test have been plotted through all three APs. Without the scope of any significant deformations comparable with coronal and sagittal APs, lowest mean registration errors were recorded for axial APs after using all the tested transforms. The proposed method yielded least mean E_(T-R) (for all APs) while

88

5 Deformable Thoracic CT Images Sequence Registration …

Fig. 5.3 The iterative process graphical results on “case 3” coronal AP

5.4 Results and Discussion

89

Fig. 5.4 Energy minimization versus iterations for “case 3” coronal AP

Table 5.2 SNR, pSNR, m-SSIM, NCC for all subjects under study from all APs; S-T is the source-target pair, R-T is registered-target pair for proposed method Similarity estimation of S-T and R-T using various metrics for all subjects Axial

Coronal

Sagittal

S-T

R-T

S-T

R-T

S-T

R-T

SNR (dB)

16.23 ± 1.48

16.29 ± 1.96

12.51 ± 1.37

16.29 ± 1.62

12.62 ± 1.3

16.13 ± 1.6

PSNR (dB)

20.52 ± 1.14

20.58 ± 1.62

15.35 ± 1.36

19.13 ± 1.6

16.33 ± 1.5

19.83 ± 1.8

m-SSIM index

0.744 ± 0.05

0.742 ± 0.04

0.49 ± 0.08

0.58 ± 0.12

0.57 ± 0.13

0.64 ± 0.14

NCC

0.964 ± 0.01

0.969 ± 0.01

0.85 ± 0.03

0.93 ± 0.02

0.89 ± 0.03

0.95 ± 0.03

followed by b-spline and affine transforms in order. Not relying on landmark-based features to establish correspondences instead of applying purely intensity difference based energy minimization can be attributed to these results.

5.5 Conclusion A novel, practically more feasible and accurate deformable image registration methodology for thoracic image sequences has been proposed. It could be a boon for real-life applications such as image acquisition for radiotherapy planning of

90

5 Deformable Thoracic CT Images Sequence Registration …

Fig. 5.5 Mean registration error (pixels) for all 10 subjects through all APs

thoracic lesions, dosimetric evaluation, tumor growth progression (with time), and determination of subject-specific deformable motion models. An effort has been made to model elastic image deformations after real-life 2D elastic object deformations such that all the constituents of that object are constantly in spontaneous motion and are not at equilibrium. Motion of 2D elastic objects due to internal forces has been used as an inspiration to determine deformations in thoracic CT images. Results from our study showed an average target registration error of less than 1 pixel over the entire thoracic ct image volume. Such an accurate registration of thoracic ct images obtained in the deformed state can be useful in treatment planning and also for longitudinal evaluation of progression/regression in patients with lung cancer. Although the utility of this method has been shown for ct image volume, the method can be applied to images of any other imaging modalities as well.

Chapter 6

Conclusion and Future Work

Deformable image registration is a challenging problem due to various types of possible deformations and high chance of false registration. In particular, registration of CT image stacks/sequences is a very difficult task because of the sheer number of landmark feature points involved in the registration process. DIR techniques able to account for displacement and deformation of organs in a series of medical images acquired in connection with fractions of radiotherapy are a key component in the efforts to improve the treatment guided by image data. The conclusions of the work of this book and suggestions for future research are presented in this chapter.

6.1 Concluding Remarks The study was set out to explore new and accurate deformable image registration techniques for thoracic CT image pairs and image sequences. The investigations were set up on a three dimensional CT image database of 10 subjects. For each subject, there were 10 images in temporal sequence through all three anatomical positions, i.e., axial, coronal, and sagittal, out of which first six were temporally aligned with a gap of 0.1 s from full inhale to full exhale position. The objective was to register the image pair and sequences accurately from the above-mentioned data (or any other modality image) by applying geometrical transformation based registration algorithms. Three such registration algorithms were proposed, both standalone and composite algorithms. One of the objectives of the algorithms was to determine an image registration model for a variety of breathing motion data from many subjects. It is known that different individuals have different breathing frequencies depending on many factors like their respective lifestyles, genetic or hereditary diseases, etc. The study was conducted to develop algorithms to adjust and normalize these variations, thus providing a common denominator upon which more accurate analyses can be made in the future, both from medical imaging and clinical research perspectives. There hasn’t been a consolidated method to assess the deformations happening in the thoracic region during the process of breathing. The proposed © Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0_6

91

92

6 Conclusion and Future Work

method helps in assessing this deformation in the form of average displacement of all common landmark points in that image sequence from full inhale to exhale positions. This has been implemented on all test subjects and has been demonstrated for one subject in further detail. Also, the displacing points on the image leave clear and color-coordinated paths that reflect the exact motion of those points through frames of the image sequence. This would help in assessing and analyzing individual motion separately at every point of the medical image if required. This would be highly beneficial in detecting abnormal behavior in organs when compared to normal established baselines. Accuracy of these algorithms was determined using metrics like Target registration error, image similarity metrics, etc. Lower values of target registration error for applied algorithms in comparison to those prevalent indicated higher deformable image registration accuracies. Likewise, similarity metrics indicating higher percentage of correspondence between the transformed image and target image (post-registration) in comparison to the initial similarity between source and target image indicate better registration than the usually employed methods to achieve the same objective.

6.2 Scope for Future Work The proposed methods proved to be accurate and fulfilling the objectives keeping in mind which the work was started, they can be seen as the stepping stones to more accurate and fast techniques to achieve deformable image registration in the future. The proposed methods seem to exhaust the scope of this book, there are a few modifications in already existing methods and few new ideas that are in order to be taken up in the future to enhance and push the boundaries of image processing and medical imaging in particular. One of the primary modifications would be soft computing based feature point marking system. The idea is to use an automatic/semiautomatic learning-based relevant landmark point marking system. Organ based information from both medical and image processing perspectives will be used as a pre-requisite for the learning procedure to enable the landmark point marker to highlight only relevant areas instead of either manually plotting points or using an automatic method which marks landmark points randomly (based on presumptions other than the medical kind). This would help in highlighting those areas of the medical image which actually do move rather than those which do not most of the time thus making better use of the resources and making the whole process faster and more relevant. The image registration resulting from these relevant common landmark point cloud would be less erroneous and more dynamic according to the organ of which the medical images are being registered. Deformable image registration has been playing a pivotal role in correcting the “human error” aspect of medical image acquisition irrespective of the image modality it is being used for and has been a major contributor in clinical research based on these images for similar reasons. The methods proposed in this work will become a small part of an already vast cluster of similar algorithms, all working in tandem towards a common objective: fast, accurate, and efficient image-based clinical intervention as when required.

Appendix A

Geometrical Deformation Models for Elastic Images

See Tables A.1, A.2, A.3, A.4 and A.5.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0

93

Year

2004

2005

Author

M. Droske et al.

A. Leow et al.

Table A.1 Elastic body models

Inverse consistent mapping in 3D deformable image registration: Its construction and statistical properties

A variational approach to non-rigid morphological image registration

Title

About

A new approach to inverse consistent image registration. A uni-directional algorithm is developed using symmetric cost functionals and regularizers

A novel variational method to non-rigid registration of multi-modal data

Method

Instead of enforcing inverse consistency using an additional penalty that penalizes inconsistency error, the new algorithm directly models the backward mapping by inverting the forward mapping. The resulting minimization problem could then be solved uni-directionally involving only the forward mapping, without optimizing in the backward direction

A suitable deformation was determined via the minimization of a morphological, i.e., contrast invariant, matching functional along with appropriate regularization energy

Findings

The algorithm was evaluated by applying it to the serial MRI scans of a clinical case of semantic dementia. The statistical distributions of the local volume change (Jacobian) maps were examined by considering the Kullback–Liebler distances on the material density functions

It was found suitable for the registration of multimodal data, as confirmed by some numerical results

(continued)

Statistically significant differences were detected between consistent versus inconsistent matching when permutation tests were performed on the resulting deformation maps

Remarks

94 Appendix A: Geometrical Deformation Models for Elastic Images

Year

2005

2007

Author

X. Pennec et al.

A. D. Leow et al.

Table A.1 (continued)

Statistical properties of Jacobian maps and the realization of unbiased large-deformation nonlinear image registration

Riemannian elasticity: A statistical regularization framework for nonlinear registration

Title

About

A method has been proposed to provide rigorous mathematical analyses of the Jacobian maps, and use them to motivate a new numerical method to construct unbiased nonlinear image registration

The elastic energy has been interpreted as the distance of the Green-St Venant strain tensor to the identity, which reflects the deviation of the local deformation from a rigid transformation

Method

It is established that that logarithmic transformation is crucial for analyzing Jacobian values representing morphometric differences. Statistical distributions of log-Jacobian maps are examined by defining the Kullback–Leibler (KL) distance on material density functions arising in continuum-mechanical models

By changing the Euclidean metric for a more suitable Riemannian one, a consistent statistical framework is defined to quantify the amount of deformation. These statistics are then used as parameters in a Mahalanobis distance to measure the statistical deviation from the observed variability, giving a new regularization criterion that we called the statistical Riemannian elasticity

Findings

Symmetrization of image registration statistically reduces skewness in the log-Jacobian map

It was found that the new criterion is able to handle anisotropic deformations and is inverse-consistent

Remarks

(continued)

Preliminary results showed that it can be quite easily implemented in a non-rigid registration algorithm

Appendix A: Geometrical Deformation Models for Elastic Images 95

Year

2008

2011

Author

I. Yanovsky et al.

C. L. Guyader and L. A. Vese

Table A.1 (continued)

A combined segmentation and registration framework with a nonlinear elasticity smoother

Unbiased volumetric registration via nonlinear elastic regularization

Title

About

A new non-parametric combined segmentation and registration method

A new nonlinear image registration model which is based on nonlinear elastic regularization and unbiased registration

Method

The modeling is twofold: first, registration is jointly performed with segmentation since guided by the segmentation process; it means that the algorithm produces both a smooth mapping between the two shapes and the segmentation of the object contained in the reference image. Second, the use of a nonlinear elasticity-type regularizer allows large deformations to occur, which makes the model comparable in this point with the viscous fluid registration method

The nonlinear elastic and the unbiased regularization terms are simplified using the change of variables by introducing an unknown that approximates the Jacobian matrix of the displacement field. This reduces the minimization to involve linear differential equations. The new model is written in a unified variational form and is minimized using gradient descent

Findings

The shapes to be matched were viewed as Ciarlet–Geymonat materials. Existence of minimizers of the introduced functional was proved and an approximated problem based on the Saint Venant–Kirchhoff stored energy for the numerical implementation and solved by an augmented Lagrangian technique

The new unbiased nonlinear elasticity model was found to be computationally more efficient and easier to implement than the unbiased fluid registration. The unbiased large-deformation nonlinear elasticity method was tested using volumetric serial magnetic resonance images and showed advantages for medical imaging applications Several applications are proposed here to demonstrate the potential of this method to both segmentation of one single image and to registration between two images

Remarks

96 Appendix A: Geometrical Deformation Models for Elastic Images

Year

2005

2007

Author

W. R. Crum et al.

N. D. Cahill et al.

Fourier methods for nonparametric image registration

Anisotropic multi-scale fluid registration: Evaluation in magnetic resonance breast imaging

Title

Table A.2 Viscous fluid flow model About

It was shown that Fourier methods can be employed to quickly solve the linear PDE systems for every combination of standard regularizers (diffusion, curvature, elastic, and fluid) and boundary conditions (Dirichlet, Neumann, and periodic)

A multi-resolution fluid registration algorithm that improves on previous works on multiple levels of free form deformation (FFD)

Method

Faster techniques based on Fourier methods, multigrid methods, and additive operator splitting; exist for solving the linear PDE systems for specific combinations of regularizers, and boundary conditions were applied on a mammography image set

Directly solving the Navier–Stokes equation at the resolution of the images; accommodating image sampling anisotropy using semi-coarsening and implicit smoothing in a full multi-grid (FMG) solver, and exploiting the inherent multi-resolution nature of FMG to implement a multi-scale approach

Findings

Fourier methods can be employed to quickly solve the linear PDE systems for every combination of standard regularizers

Evaluation was on five magnetic resonance (MR) breast images subject to six biomechanical deformation fields over 11 multi-resolution schemes. Quantitative assessment was by tissue overlaps and target registration errors and by registering using the known correspondences rather than image features to validate the fluid model

Remarks

(continued)

The results showed that fluid registration of 3D breast MR images to sub-voxel accuracy is possible in minutes on a 1.6 GHz Linux-based Athlon processor with coarse solutions obtainable in a few tens of seconds. Accuracy and computation time are comparable to FFD techniques validated for this application

Appendix A: Geometrical Deformation Models for Elastic Images 97

Year

2008

Author

M.-C. Chiang et al.

Table A.2 (continued)

Fluid registration of diffusion tensor images using information theory

Title

About This work presented an information-theoretic cost metric, symmetrized Kullback–Leibler (sKL) divergence, or J-divergence, to fluid registration of diffusion tensor images

Method Three-dimensional DTI data from 34 subjects were fluidly registered to an optimized target image. The flow was regularized with a large-deformation diffeomorphic mapping based on the kinematics of a Navier–Stokes fluid. A driving force was developed to minimize the J-divergence between the deforming source and target diffusion functions while reorienting the flowing tensors to preserve fiber topography

Findings It was showed that the sKL-divergence based on full diffusion PDFs is adaptable to higher-order diffusion models, such as high angular resolution diffusion imaging (HARDI). The sKL-divergence was sensitive to subtle differences between two diffusivity profiles, showing promise for nonlinear registration applications and multi subject statistical analysis of HARDI data

Remarks

98 Appendix A: Geometrical Deformation Models for Elastic Images

Year

2008

2009

Author

J. M. Peyrat et al.

B. T. T. Yeo et al.

DT-REFinD: Diffusion tensor registration with exact finite-strain differential

Registration of 4D time-series of cardiac images with multichannel diffeomorphic demons

Title

Table A.3 Diffusion model About

The DT-REFinD algorithm for the diffeomorphic nonlinear registration of diffusion tensor images

A generic framework for intersubject non-linear registration of 4D time-series images

Method

Results were borrowed from the pose estimation literature in computer vision to derive an analytical gradient of the registration objective function. By utilizing the closed-form gradient and the velocity field representation of one parameter subgroups of diffeomorphisms, the resulting registration algorithm came to be diffeomorphic and fast. The algorithm was contrasted and compared with a traditional FS alternative that ignores the reorientation in the gradient computation

Spatio-temporal registration is defined by mapping trajectories of physical points as opposed to spatial registration that solely aims at mapping homologous points. The trajectories were determined which had to be registered in each sequence using a motion tracking algorithm based on the Diffeomorphic Demons algorithm. Simultaneously pairwise registrations were performed of corresponding time-points with the constraint to map the same physical points over time

Findings

It was shown that the exact gradient leads to significantly better registration at the cost of computation time. Alignment quality was assessed with a battery of metrics including tensor overlap, fractional anisotropy, inverse consistency, and closeness to synthetic warps

It was shown that this trajectory registration can be formulated as a multichannel registration of 3D images

Remarks

(continued)

The improvements persist even when a different reorientation scheme, preservation of principal directions, was used to apply the final deformations

This framework is applied to the inter-subject non-linear registration of 4D cardiac CT sequences

Appendix A: Geometrical Deformation Models for Elastic Images 99

Year

2010

2010

Author

M. Modat et al.

B. T. T. Yeo et al.

Table A.3 (continued)

Spherical demons: Fast diffeomorphic landmark-free surface registration

Diffeomorphic demons using normalized mutual information, evaluation on multimodal brain MR images

Title

About

The spherical Demons algorithm for registering two spherical images

A diffeomorphic demons implementation using the analytical gradient of Normalised Mutual Information (NMI) in a conjugate gradient optimizer

Method

Exploiting spherical vector spline interpolation theory, it was shown that a large class of regularizers for the modified Demons objective function can be efficiently approximated on the sphere using iterative smoothing. Based on one parameter subgroups of diffeomorphisms, the resulting registration is diffeomorphic and fast. The Spherical Demons algorithm can also be modified to register a given spherical image to a probabilistic atlas

Hailed as the first reported qualitative and quantitative assessment of the demons for inter-modal registration

Findings

Two variants of the algorithm corresponding to warping the atlas or warping the subject were demonstrated. Registration of a cortical surface mesh to an atlas mesh, both with more than 160 k nodes requires less than 5 min when warping the atlas and less than 3 min when warping the subject on a Xeon 3.2 GHz single processor machine. This is comparable to the fastest nondiffeomorphic landmark-free surface registration algorithm

Experiments to spatially normalize real MR images, and to recover simulated deformation fields, demonstrated similar accuracy from NMI-demons and classical demons when the latter may be used, and similar accuracy for NMI-demons on T1w–T1w and T1w–T2w registration Technique was validated in two different applications that use registration to transfer segmentation labels onto a new image (1) parcellation of in vivo cortical surfaces and (2) Brodmann area localization in ex vivo cortical surfaces

Remarks

100 Appendix A: Geometrical Deformation Models for Elastic Images

Year

2009

2010

Author

B. Glocker et al.

B. Beuthien et al.

Recursive Green’s function registration

Approximated curvature penalty in non-rigid registration using pairwise MRFs

Title

Table A.4 Curvature registration About

It has been tried to minimize a joint functional that is comprised of a similarity measure and a regularizer in order to obtain a reasonable displacement field that transforms one image to the other

An approximated curvature penalty using second-order derivatives defined on the MRF pairwise potentials is proposed

Method

A generalized and efficient numerical scheme for solving such system of PDEs simply by applying 1-dimensional recursive filtering to the right-hand side of the system based on the Green’s function of the differential operator that corresponds to the chosen regularizer

Labeling of discrete Markov Random Fields (MRFs) for solving the problem of non-rigid image registration. Smoothness is achieved by penalizing the derivatives of the displacement field

Findings

The associated Green’s function for the diffusive and curvature regularizers was presented and it was shown that how one may efficiently implement the whole process by using recursive filter approximation

It was demonstrated that the approximated term has similar properties as higher-order approaches (invariance to linear transformations), while the computational efficiency of pairwise models remained preserved

Remarks

Appendix A: Geometrical Deformation Models for Elastic Images 101

Year

2009

Author

M. Hernandez et al.

Registration of anatomical images using paths of diffeomorphisms parameterized with stationary vector field flows

Title

Table A.5 Flows of diffeomorphisms About Proposed paradigm for diffeomorphic registration is the Large Deformation Diffeomorphic Metric Mapping (LDDMM). In this framework, transformations are characterized as endpoints of paths parameterized by time-varying flows of vector fields defined on the tangent space of a Riemannian manifold of diffeomorphisms and computed from the solution of the non-stationary transport equation associated to these flows

Method Optimization in LDDMM is performed on the space of non-stationary vector field flows resulting in a time and memory consuming algorithm. The stationary parameterization is included for diffeomorphic registration in the LDDMM framework. The variational problem related to this registration scenario is formulated and associated Euler–Lagrange equations are derived

Findings The performance of the non-stationary versus the stationary parameterizations in real and simulated 3D-MRI brain datasets is evaluated. Compared to the non-stationary parameterization, proposed method provides similar results in terms of image matching and local differences between the diffeomorphic transformations while drastically reducing memory and time requirements

Remarks

(continued)

102 Appendix A: Geometrical Deformation Models for Elastic Images

Year

2009

Author

M. D. Craene et al.

Table A.5 (continued)

Large diffeomorphic FFD registration for motion and strain quantification from 3D-US sequences

Title

About A new registration method for the in vivo quantification of cardiac deformation from a sequence of possibly noisy images

Method In the proposed method, referred to as Large Diffeomorphic Free Form Deformation (LDFFD), the displacement field at each time step is computed from a smooth non-stationary velocity field, thus imposing a coupling between the transformations at successive time steps. Main contribution is to extend this framework to the estimation of motion and deformation in an image sequence. Similarity is captured for the entire image sequence using an extension of the pairwise mutual information metric. The LDFFD algorithm is applied here to recover longitudinal strain curves from healthy and Left-Bundle Branch Block (LBBB) subjects

Findings Strain curves for the healthy subjects were in accordance with the literature. For the LBBB patient, strain quantified before and after Cardiac Resynchronization Therapy showed a clear improvement of cardiac function in this subject, in accordance with clinical observations

Remarks

(continued)

Appendix A: Geometrical Deformation Models for Elastic Images 103

Year

2011

2011

Author

J. Ashburner et al.

L. Risser et al.

Table A.5 (continued)

Simultaneous multi-scale registration using large deformation diffeomorphic metric mapping

Diffeomorphic registration using geodesic shooting and Gauss–Newton optimization

Title

About

A practical methodology to integrate prior knowledge about the registered shapes in the regularizing metric

A nonlinear image registration algorithm based on the setting of Large Deformation Diffeomorphic Metric Mapping (LDDMM), but with a more efficient optimization scheme, both in terms of memory required and the number of iterations required in reaching convergence

Method

First presented the notion of characteristic scale at which image features are deformed. Then proposes a methodology to compare anatomical shape variations in a multi-scale fashion, i.e., at several characteristic scales simultaneously. In this context, a strategy was proposed to quantitatively measure the feature differences observed at each characteristic scale separately

Instead of performing a variational optimization on a series of velocity fields, the algorithm is formulated to use a geodesic shooting procedure, so that only an initial velocity is estimated. A Gauss–Newton optimization strategy is used to achieve faster convergence

Findings

Ability of the proposed method is compared to segregate a group of subjects having Alzheimer’s disease and a group of controls with a classical coarse to fine approach, on standard 3D MR longitudinal brain images. It was finally applied to quantify the anatomical development of the human brain from 3D MR longitudinal images of pre-term babies

The algorithm was evaluated using freely available manually labeled datasets and found to compare favorably with other inter-subject registration algorithms evaluated using the same data

Remarks

(continued)

The method registers accurately volumetric images containing feature differences at several scales simultaneously with smooth deformations

–

104 Appendix A: Geometrical Deformation Models for Elastic Images

Year

2011

Author

G. Auzias et al.

Table A.5 (continued)

Diffeomorphic brain registration under exhaustive sulcal constraints

Title

About A global, geometric approach that performs the alignment of the exhaustive sulcal imprints (cortical folding patterns) across individuals

Method The DIffeomorphic Sulcal-based COrtical (DISCO) technique proceeded to the automatic extraction, identification, and simplification of sulcal features from T1-weighted Magnetic Resonance Image (MRI) series. These features are then used as control measures for fully-3-D diffeomorphic deformations

Findings Quantitative and qualitative evaluations showed that DISCO correctly aligns the sulcal folds and gray and white matter volumes across individuals. The comparison with a recent, iconic diffeomorphic approach (DARTEL) highlighted how the absence of explicit cortical landmarks may lead to the misalignment of cortical sulci

Remarks DISCO can also be combined with (DARTEL) to further improve the consistency and accuracy of alignment performances

Appendix A: Geometrical Deformation Models for Elastic Images 105

Appendix B

See Table B.1.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0

107

Year

2003

2006

Author

M. Alexa et al.

S. Schaefer et al.

Image deformation using moving least squares

Computing and rendering point set surfaces

Title

Table B.1 Tabular literature survey, Chap. 3 About

An image deformation method based on Moving Least Squares using various classes of linear functions including affine, similarity, and rigid transformations. These deformations are realistic and give the user the impression of manipulating real-world objects

Use of point sets to represent shapes. Defining surfaces from a set of points close to an original surface is approximated using MLS

Method

Image deformations were built based on collections of points with which the user controls the deformation. A deformation function was constructed satisfying the three properties of Interpolation, Smoothness and Identity using MLS

A projection procedure is defined which projects any point near the point set onto the surface. Then, the MLS surface is defined as the points projecting onto themselves. The smoothness conjecture is motivated and respective projection is computed

Findings

The proposed method was applied for Affine, Similarity, Rigid and Elastic deformations on a set of images. It was found to perform deformations faster than contemporary methods

The proposed model was tested on ‘the Stanford bunny’ along with other models. The proposed approach showed smoother silhouettes and more accurate highlights in comparison to more traditional methods like Splatting and Gouraud-shaded mesh model

Remarks

(continued)

It was showed how solutions could be computed directly from the closed-form deformation using similarity transformations thereby bypassing the non-linear minimization. The method is general enough to accommodate different distance metrics dependent on the topology of the shape rather than the simple, Euclidean distance used as our weight factor

Thus, it is possible to provide a point set representation that conforms to a specified tolerance. The use of a point set (without connectivity) as a representation of shapes

108 Appendix B

Year

1981

2009

2011

Author

P. Lancaster, K. Salkauskas

R. Castillo et al.

K. Murphy et al.

Table B.1 (continued)

Title

Evaluation of Registration Methods on Thoracic CT: The EMPIRE10 Challenge

A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets

Surfaces generated by moving least squares methods

About

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of non-rigid registration techniques

Deformable Image Registration using Moving least squares for corresponding sets of feature landmark point pairs

An analysis of least squares methods for smoothing and interpolating scattered data was presented. In particular, theorems are proved concerning the smoothness of interpolants and the description of MLS processes as projection methods

Method

Methods on comparison: Asclepios1, Asclepios2, CMS, DIKU, DROP, elastix, IMI Lubeck Diffeomorph, Lyon FFD, MGH, Nifty Reggers, OFDP, picsl exp, picsl gsyn, Robust TreeReg Leuven, Spline MIRIT Leuven

APRIL (Matlab based in-house sw UI) for manual selection of landmark feature points. This point set is subjected to MLS, which registers the source landmark point set to the corresponding target point set

A non-interpolating least squares method as an alternate representation of the local approximation based on the choice of weight functions

All methods were fully automatic with the exception of MGH. Generic registration algorithms can perform better than data specific methods. It may still be the case that combines aspects of both could improve performance even further, particularly on more difficult scan pairs

(continued)

The EMPIRE10 challenge enabled detailed, independent, and fair evaluation of non-rigid registration algorithms

No proposition on the estimation of deformity between the registered image pairs

U SE α 1/(L PP )1/2 U SE α SDSE . The uncertainty of spatial error estimates was found to be inversely proportional to the square root of the number of landmark point pairs and directly proportional to the standard deviation of spatial errors

Remarks NA

Findings The differences between interpolating and non-interpolating MLS method as projection methods. The effects of the choice of weight functions and the asymptotic behavior of such single variable and multivariate functions

Appendix B 109

Year

2014

Author

E. Castillo et al.

Table B.1 (continued)

A Moving Least Squares Approach for Computing Spatially Accurate Transformations That Satisfy Strict Physiologic Constraints

Title

About Computation of a physiologically realistic spatial transformation from a sparse point cloud of displacement estimates using MLS and any combination of upper bound, lower bound, or equality constraints placed on the Jacobian

Method MLS defined a spatial transformation from a sparse point cloud of estimated displacements and provided simple analytic derivative estimates for all voxel locations. Given displacement estimates from automated block

Findings Two MLS transformations were computed for five (5) pairs of inhale–exhale thoracic CT images, one with no Jacobian constraints and the other with strict contraction Jacobian constraints. Despite registering from inhale–exhale, the constrained MLS yielded a strict contraction (all Jacobian values between 0 and 1) while the unconstrained MLS resulted in regions of expansion

Remarks The proposed MLS approach was found capable of producing Jacobian constrained transformations without degrading spatial accuracy

110 Appendix B

Appendix C

See Table C.1.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0

111

Year

2006

2009

Author

D. Sarrut et al.

N. Stevo et al.

Registration of Temporal Sequences of Coronal and Sagittal Images Obtained from Magnetic Resonance

Simulation of four-dimensional CT images from deformable registration between inhale and exhale breath-hold CT scans

Title

Table C.1 Tabular literature survey, Chap. 4 About

For each image in coronal and sagittal MRI sequences, the information contained in the intersection segment was determined, and the matching is done to determine the best sagittal images for each coronal image and vice-versa. The registration is the determination of the best images in a sequence that fits a chosen image in another sequence

Simulation of an artificial four-dimensional (4-D) CT image of the thorax during breathing. It is performed by deformable registration of two CT scans acquired at inhale and exhale breath-hold

Method

One of the registration approaches used in determining the distance between the images by comparing pixel by pixel and combining these differences in a single value. The other one is Fourier transform based

Dense deformable registrations were performed. The method was a minimization of the sum of squared differences (SSD) using an approximated second-order gradient

Findings

The resulting pairs from both algorithms were different. It was noticed that both pairs have a satisfactory visual registration. The temporal sequence of images represented discrete instants in time, and such an almost perfect fitting is very rare

Statistically better results than the reference method. The mean (and standard deviation) of distances between automatically found landmark positions and landmarks set by experts were 2.7(1.1) mm with APLDM, and 6.3(3.8) mm. The mean difference between automatic and manual landmark positions for intermediate CT images was 2.6(2.0) mm

Remarks

(continued)

The temporal registration algorithm based on pixel by pixel comparison and Fourier transform showed several satisfactory results, however, it is not possible to overcome the temporal low rate of image acquisition. One of the future works would be the definition of a new registration algorithm combining pixel comparison and time segmentation

The generation of 4-D CT images by deformable registration of inhale and exhale CT images is feasible. This can lower the dose needed for 4-D CT acquisitions or can help to correct 4-D acquisition artifacts. The 4-D CT model can be used to propagate contours, to compute a 4-D dose map, or to simulate CT acquisitions with an irregular breathing signal. It could serve as a basis for 4-D radiation therapy planning

112 Appendix C

Year

2010

2011

Author

E. Castillo et al.

J. Ehrhardt et al.

Table C.1 (continued)

Statistical Modeling of 4D Respiratory Lung Motion Using Diffeomorphic Image Registration

Four-dimensional deformable image registration using trajectory modeling

Title

About

An approach to generate a mean motion model of the lung based on thoracic 4D computed tomography (CT) data of different patients to extend the motion modeling capabilities

A four-dimensional deformable image registration (4D DIR) algorithm, referred to as 4D local trajectory modeling (4DLTM), is presented and applied to thoracic 4D computed tomography (4DCT) image sets

Method

The modeling process consisted of three steps: an intra-subject registration to generate subject-specific motion models, the generation of an average shape and intensity atlas of the lung as anatomical reference frame, and the registration of the subject-specific motion models to the atlas in order to build a statistical 4D mean motion model (4D-MMM). In all steps, a symmetric diffeomorphic nonlinear intensity-based registration method was employed

The method exploits the incremental continuity present in 4DCT component images to calculate a dense set of parameterized voxel trajectories through space as functions of time. The spatial accuracy of the 4DLTM algorithm is compared with an alternative registration approach in which component phase to phase (CPP) DIR is utilized to determine the full displacement between maximum inhale and exhale images

Findings

The model was evaluated by applying it for estimating respiratory motion of ten lung cancer patients. The prediction was evaluated with respect to landmark and tumor motion, and the quantitative analysis resulted in a mean target registration error (TRE) of 3.3 ± 1.6 mm. With regard to lung tumor motion, it was shown that prediction accuracy is independent of tumor size and motion amplitude in the considered data set

Cubic polynomials were found to provide sufficient flexibility and spatial accuracy for describing the point trajectories through the expiratory phases. The resulting average spatial error between the maximum phases was 1.25 mm for the 4DLTM and 1.44 mm for the CPP

Remarks

(continued)

The statistical respiratory motion model is capable of providing valuable prior knowledge in many fields of applications. We present two examples of possible applications in radiation therapy and image-guided diagnosis

The 4DLTM method captures the long-range motion between 4DCT extremes with high spatial accuracy

Appendix C 113

Year

2011

2012

Author

A. K. Sato et al.

G. Xiong et al.

Table C.1 (continued)

Tracking the motion trajectories of junction structures in 4D CT images of the lung

Registration of temporal sequences of coronal and sagittal MR images through respiratory patterns

Title

About

A novel method to detect a large collection of natural junction structures in the lung and use them as reliable markers to track the lung motion

This work discussed the determination of the breathing patterns in time sequence of images obtained from magnetic resonance (MR) and their use in the temporal registration of coronal and sagittal images

Method

The image intensities within a small region of interest surrounding the center are selected as its signature. Under the assumption of the cyclic motion, the trajectory was described by a closed B-spline curve and search for the control points by maximizing a metric of combined correlation coefficients. Local extrema are suppressed by improving the initial conditions using random walks from pair-wise optimizations. Several descriptors are introduced to analyze the motion trajectories

A time sequence of this intersection segment of orthogonal coronal and sagittal sequences was stacked, defining a two-dimension spatio-temporal (2DST) image. An interval Hough transform algorithm searches for synchronized movements with the respiratory function. A greedy active contour algorithm adjusts small discrepancies originated by asynchronous movements in the respiratory patterns

Findings

The method was applied to 13 real 4D CT images. More than 700 junctions in each case are detected with an average positive predictive value of greater than 90%. The average tracking error between automated and manual tracking is sub-voxel and smaller than the published results using the same set of data

The results of the proposed method in the form of synchronized sequences are compared with the pixel-by-pixel comparison method

Remarks

–

(continued)

The proposed method increases the number of registered pairs representing composed images and allows an easy check of the breathing phase

114 Appendix C

Year

2013

2014

Author

Y. Zhang et al.

B. Fuerst et al.

Table C.1 (continued)

Patient-specific Biomechanical Model for the Prediction of Lung Motion From 4-D CT Images

Modeling respiratory motion for reducing motion artifacts in 4D CT images

Title

About

An approach to predict the deformation of the lungs and surrounding organs during respiration

A patient-specific respiratory motion model, based on principal component analysis (PCA) of motion vectors obtained from deformable image registration, with the main goal of reducing image artifacts caused by irregular motion during 4D CT acquisition

Method

A computational model of the respiratory system, which comprises an anatomical model extracted from computed tomography (CT) images at end-expiration (EE), and a biomechanical model of the respiratory physiology, including the material behavior and interactions between organs

Displacement vector fields relative to a reference phase were calculated using an in-house deformable image registration method. The authors then used PCA to decompose each of the displacement vector fields into linear combinations of principal motion bases. These projections were parameterized using a spline model to allow the reconstruction of the displacement vector fields at any given phase in a respiratory cycle. Finally, the displacement vector fields were used to deform the reference CT image to synthesize CT images at the selected phase with much reduced image artifacts

Findings

The method was then tested on five public datasets. Results showed that the model was able to predict the respiratory motion with an average landmark error of 3.40 ± 1.0 mm over the entire respiratory cycle

The initial large discrepancies across the landmark pairs were significantly reduced after deformable registration, and the accuracy was similar to or better than that reported by state-of-the-art methods. The motion model was used to reduce irregular motion artifacts in the 4D CT images of three lung cancer patients. Visual assessment indicated that the proposed approach could reduce severe image artifacts

Remarks

The estimated 3-D lung motion may constitute as an advanced 3-D surrogate for more accurate medical image reconstruction and patient respiratory analysis

The proposed approach can mitigate shape distortions of anatomy caused by irregular breathing motion during 4D CT acquisition

Appendix C 115

Appendix D

See Table D.1.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0

117

Year

2005

2011

Author

X. Pennec et al.

B. Zhang et al.

Three-dimensional elastic image registration based on strain energy minimization: application to prostate magnetic resonance imaging

Riemannian Elasticity: A Statistical Regularization Framework for Non-linear Registration

Title

Table D.1 Tabular literature survey, Chap. 5

A novel 3-D elastic registration procedure that is based on the minimization of a physically motivated strain energy function that requires the identification of similar features (points, curves, or surfaces) in the source and target images

The elastic energy has been interpreted as the distance of the Green-St Venant strain tensor to the identity, which reflects the deviation of the local deformation from a rigid transformation

About

The Gauss–Seidel method was used in the numerical implementation of the registration algorithm. The registration procedure was validated on synthetic digital images, MR images from prostate phantom, and MR images obtained on patients. The registration error, assessed by averaging the displacement of a fiducial landmark in the target to its corresponding point in the registered image

By changing the Euclidean metric for a more suitable Riemannian one, a consistent statistical framework is defined to quantify the amount of deformation. These statistics are then used as parameters in a Mahalanobis distance to measure the statistical deviation from the observed variability, giving a new regularization criterion that we called the statistical Riemannian elasticity

Method

The registration error on patient data was 1.8 ± 0.7 pixels. Registration also improved image similarity (normalized cross-correlation) from 0.72 ± 0.10 to 0.96 ± 0.03 on patient data

It was found that the new criterion is able to handle anisotropic deformations and is inverse-consistent

Findings

(continued)

Registration results on prostate data in vivo demonstrated that the registration procedure could be used to significantly improve both the accuracy of localized therapies such as brachytherapy or external beam therapy and can be valuable in the longitudinal follow-up of patients after therapy

Preliminary results showed that it can be quite easily implemented in a non-rigid registration algorithm

Remarks

118 Appendix D

Year

2011

2012

Author

R. W. K. So et al.

A. R. Dykstra et al.

Table D.1 (continued)

Title

Individualized localization and cortical surface-based registration of intracranial electrodes

Non-rigid image registration of brain magnetic resonance images using graph-cuts

A method that co-registers high-resolution preoperative MRI with postoperative computerized tomography (CT) for the purpose of individualized functional mapping of both normal and pathological (e.g., interictal discharges and seizures) brain activity

A graph-cut based method for non-rigid medical image registration on brain magnetic resonance images. The non-rigid medical image registration problem is reformulated as a discrete labeling problem

About

The method accurately (within 3 mm, on average) localizes electrodes with respect to an individual’s neuroanatomy. Furthermore, we outline a principled procedure for either volumetric or surface-based group analyses

Modeled the non-rigid registration as a multi-labeling problem by Markov random field. The image registration problem is therefore modeled by two energy terms based on intensity similarity and smoothness of the displacement field. The MRF energy is minimized by graph-cuts algorithm via α-expansions

Method

The method was demonstrated in five patients with medically-intractable epilepsy undergoing invasive monitoring of the seizure focus prior to its surgical removal. Accuracy was within 3 mm of average

Compared the registration results of the proposed method with two state-of-the-art medical image registration approaches: free-form deformation based method and demons method. In addition, the registration results were also compared with that of the linear programming based image registration method

Findings

(continued)

The straightforward application of this procedure to all types of intracranial electrodes, robustness to deformations in both skull and brain, and the ability to compare electrode locations across groups of patients make this procedure an important tool for basic scientists as well as clinicians

The proposed method was found to be more robust against different challenging non-rigid registration cases with consistently higher registration accuracy than those three methods and gives realistic recovered deformation fields

Remarks

Appendix D 119

Year

2013

2013

Author

H. P. Heinrich et al.

K. Nakagomi et al.

Table D.1 (continued)

Title

Multi-shape graph cuts with neighbor prior constraints and its application to lung segmentation from a chest CT volume

MRF-Based Deformable Registration and Ventilation Estimation of Lung CT

A novel graph cut algorithm that can take into account multi-shape constraints with neighbor prior constraints, and reports on a lung segmentation process from a three-dimensional computed tomography (CT) image based on this algorithm

Three major challenges associated with lung ct registration viz. large motion of small features, sliding motions between organs, and changing image contrast due to compression are addressed and potentially higher quality of discrete approaches is preserved

About

A novel segmentation algorithm that improves lung segmentation for cases in which the lung has a unique shape and pathologies such as pleural effusion by incorporating multiple shapes and prior information on neighbor structures in a graph cut framework

First, an image-derived minimum spanning tree is used as a simplified graph structure, which coped well with the complex sliding motion and allowed us to find the global optimum very efficiently. Second, a stochastic sampling approach for the similarity cost between images is introduced within a symmetric, diffeomorphic B-spline transformation model with diffusion regularization. The complexity is reduced by orders of magnitude and enables the minimization of much larger label spaces. In addition to the geometric transform labels, hyper-labels are introduced, which represent local intensity variations in this task, and allow for the direct estimation of lung ventilation

Method

The efficacy of the proposed algorithm is demonstrated by comparing it to conventional one using a synthetic image and clinical thoracic CT volumes

The improvements are validated in accuracy and performance on exhale-inhale CT volume pairs using a large number of expert landmarks

Findings

–

The three challenges posed in the beginning are met

Remarks

120 Appendix D

Appendix E

See Table E.1.

© Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0

121

Track_0

Track_1

Track_2

Track_3

Track_4

Track_5

Track_6

Track_7

Track_8

Track_9

Track_10

Track_11

Track_12

Track_13

Track_14

Track_15

Track_16

Track_17

Track_18

Track_19

Track_20

Track_21

Track_22

Track_23

Track_24

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

Track_label

5

5

5

5

4

3

5

5

5

5

5

4

5

5

5

5

5

5

5

5

5

5

5

5

5

Track_duration

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Track_start

5

5

5

5

4

3

5

5

5

5

5

4

5

5

5

5

5

5

5

5

5

5

5

5

5

Track_stop

0.73

6.84

1.1

5.85

3

4.38

3.42

0.26

6.42

2.27

7.7

0.46

0.45

0.55

4.28

0.77

0.27

1.5

6.52

0.24

10.9

2.93

5.09

14.33

0.26

Track_displacement

Table E.1 Track data for subject ‘case 5’ sagittal AP Track_x_location

8.86

137.87

326.97

181.60

203.69

169.36

232.33

273.80

150.67

231.29

324.77

323.07

51.16

271.08

170.26

8.18

270.18

319.22

138.69

264.41

113.78

187.64

160.27

99.35

55.58

Track_y_location

150.87

140.06

146.73

142.24

143.52

145.27

146.68

138.70

137.12

134.51

139.27

133.45

136.91

132.75

125.42

125.81

130.22

122.76

113.65

115.45

113.1

112.85

111.02

110.7

114.13

Track_mean_speed

0.54

2.17

0.56

1.79

1.27

2.15

2.04

0.4

1.79

1.27

2.71

1.41

0.27

0.62

1.44

0.99

0.77

1.21

1.91

0.6

2.32

1.64

1.39

3.2

0.97

Track_max_speed

1.31

6.7

0.98

4.55

2.89

3.85

2.91

0.74

6.24

2.42

9.07

2.39

0.46

1.8

2.59

2.09

1.91

1.92

4.04

1.17

4.59

3.53

2.42

4.87

1.96

Track_min_speed

0.09

0.66

0.06

0.5

0.31

0.56

0.71

0.25

0.3

0.65

0.23

0.66

0.14

0

0.61

0.24

0

0.35

0.18

0.28

0.87

0.66

0.73

1

0.34

Track_median_speed

0.38

1.21

0.56

1.24

1.51

2.03

2.26

0.34

0.82

0.95

1.36

1.9

0.25

0.38

0.97

0.62

0.13

1.22

1.05

0.55

1.42

1.09

1.50

3.69

0.54

0.47

2.55

0.33

1.59

1.21

1.65

0.93

0.2

2.51

0.73

3.63

0.87

0.12

0.69

0.98

0.78

0.97

0.72

1.8

0.35

1.64

1.22

0.68

1.77

(continued)

Track_std_speed 0.73

122 Appendix E

Track_25

Track_26

Track_27

Track_28

Track_29

Track_30

Track_31

Track_32

Track_33

Track_34

Track_35

Track_36

Track_37

Track_38

Track_40

Track_41

Track_42

Track_43

Track_44

Track_45

Track_46

Track_47

Track_48

Track_49

Track_50

26

27

28

29

30

31

32

33

34

35

36

37

38

39

41

42

43

44

45

46

47

48

49

50

51

Track_label

5

5

5

5

5

5

1

5

5

4

4

5

5

5

5

4

4

5

4

5

5

5

5

5

5

Track_duration

Table E.1 (continued)

Track_start

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Track_stop

5

5

5

5

5

5

1

5

5

4

4

5

5

5

5

4

4

5

4

5

5

5

5

5

5

Track_displacement

1.74

0.96

1.4

18.19

1.95

13.68

5.49

8.3

2.58

1.06

8.59

12.9

1.28

0.58

7.33

10.59

13.13

1.63

14

0.9

6.43

7.55

0.21

0.63

13.08

Track_x_location

335.92

295.28

47.88

143.78

293.02

220.44

113.58

182.51

8.75

9.5

159.05

129.53

48.70

333.06

177.98

140.99

332.69

289.19

81.56

330.28

168.78

156.12

286.48

50.12

74.93

Track_y_location

187.98

192.26

195.51

196.58

184.74

185.44

180.31

177.42

187.31

190.77

173.06

166.71

176.52

171.77

165.70

163.05

170.36

162.56

152.41

160.52

154.22

149.82

155.84

155.37

142.74

Track_mean_speed

0.75

0.5

1.8

5.77

0.56

3.35

5.49

2.57

2.75

1.28

2.32

2.76

0.63

0.73

2.2

2.66

3.37

3.99

3.63

2.16

1.84

2.89

0.81

0.29

3.11

Track_max_speed

1.62

0.75

3.68

12.85

1.35

7.23

5.49

5.05

6.64

1.67

4.94

5.21

0.94

1.32

3.84

4.72

7.19

8.81

11.66

2.88

3.06

7.1

1.28

0.48

12.98

Track_min_speed

0.21

0.09

0.12

1.4

0.04

1.13

5.49

0.96

0

0.53

0.72

0.43

0.22

0.02

1.25

1.34

0.27

0.22

0.68

0.29

0.95

0.71

0.24

0.07

0.41

Track_median_speed

0.69

0.58

1.72

3.82

0.45

2.54

5.49

1.59

1.52

1.65

2.55

2.15

0.69

0.74

1.81

3.19

5.26

1.99

1.16

2.77

1.47

2.26

0.74

0.28

0.81

0.54

0.25

1.46

4.71

0.57

2.33

NaN

1.86

3.16

0.66

1.92

2.09

0.27

0.58

1.12

1.62

3.4

3.85

5.36

1.1

0.96

2.61

0.4

0.15

(continued)

Track_std_speed 5.52

Appendix E 123

Track_51

Track_52

Track_53

Track_54

Track_55

Track_56

Track_57

Track_58

Track_59

Track_60

Track_61

Track_62

Track_63

Track_64

Track_65

Track_66

Track_67

Track_68

Track_69

Track_71

Track_72

Track_73

Track_74

Track_75

Track_76

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

72

73

74

75

76

77

Track_label

5

5

5

5

5

5

2

5

5

5

5

5

5

5

5

5

5

5

5

5

2

5

2

5

3

Track_duration

Table E.1 (continued)

Track_start

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Track_stop

5

5

5

5

5

5

2

5

5

5

5

5

5

5

5

5

5

5

5

5

2

5

2

5

3

Track_displacement

8.33

19.24

21.98

2.81

0.31

1.12

10.37

1.79

16.8

2.4

16.63

23.5

12.58

0.52

0.54

3.31

7.98

15.61

6.3

0.94

2.51

0.13

3.34

16.36

8.74

Track_x_location

250.39

148.37

118.8

303.88

9.59

49.74

134.56

339

159.58

9

130.4

72.35

224.74

303.44

8.61

47.03

192.74

131.91

206.94

338.14

220.57

9.4

146.43

337.83

7.59

Track_y_location

243.77

220.57

220.1

229.49

239.52

235.92

216.35

220.73

214.9

226.26

219.84

216.39

206.36

213.91

214.38

214.16

192.88

196.09

197.23

203.95

205.30

204.24

205.46

202.19

183.11

Track_mean_speed

3.82

4.89

4.49

0.96

0.43

0.65

5.45

1.08

3.42

1.78

3.87

4.72

4.88

0.5

0.52

0.86

2.2

3.21

1.97

3.17

2.57

0.63

1.87

4.57

2.91

Track_max_speed

7.39

7.99

7.92

1.87

0.78

1.02

8.47

2.57

7.59

4.21

7.32

7.19

8.03

1.31

1.03

2.99

4.29

7.54

2.93

5.92

3.4

1.18

2.16

9.92

8.74

Track_min_speed

0.08

2.76

0.44

0.67

0.1

0.32

2.43

0.26

0.35

0.12

2.02

0.51

2.2

0.07

0.04

0.08

1.27

1.22

1.18

0.93

1.74

0.21

1.57

0.74

0

Track_median_speed

4.32

3.1

3.9

0.75

0.49

0.67

8.47

0.74

2.23

1.43

3.24

6.67

4.61

0.43

0.5

0.43

1.77

2.2

1.85

2.25

3.4

0.74

2.16

2.5

0

2.66

2.75

3.26

0.51

0.3

0.29

4.27

0.89

3.3

1.56

2.22

3.08

2.23

0.48

0.48

1.21

1.21

2.5

0.69

2.18

1.18

0.43

0.41

3.85

(continued)

Track_std_speed 5.05

124 Appendix E

Track_77

Track_78

Track_79

Track_80

Track_81

Track_82

Track_83

Track_84

Track_85

Track_86

Track_87

Track_88

Track_89

Track_90

Track_92

Track_93

Track_94

Track_95

Track_97

Track_98

Track_101

Track_102

Track_103

Track_104

Track_105

78

79

80

81

82

83

84

85

86

87

88

89

90

91

93

94

95

96

98

99

102

103

104

105

106

Track_label

5

5

5

5

5

5

5

5

5

5

5

5

5

5

2

5

5

5

3

5

5

5

5

5

5

Track_duration

Table E.1 (continued)

Track_start

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Track_stop

5

5

5

5

5

5

5

5

5

5

5

5

5

5

2

5

5

5

3

5

5

5

5

5

5

Track_displacement

22.83

0.78

17.81

1.36

0.91

0.86

1.96

27.14

0.13

4.29

0.52

0.21

20.98

0.33

3.83

0.18

17.12

6.34

4.63

21.98

3.15

2.72

12.2

14.28

0.27

Track_x_location

102.94

126.55

342.38

139.31

19.46

112.24

344.13

93.69

60.22

346.99

16.20

53.34

175.78

12.97

341.28

11.6

188.53

258.7

266.26

122.63

284.23

339.94

219.57

202.3

50

Track_y_location

261.32

7.76

272.59

11.53

11.41

7.41

282.58

280.63

289.99

298.99

302.02

260.25

252.71

270.9

265.75

253.95

245.2

247.68

251.83

245.86

243.57

245.24

238.32

233.74

250.94

Track_mean_speed

4.59

0.48

3.66

0.35

2.01

0.18

1.18

5.43

0.19

1.46

0.21

0.26

4.6

0.24

2.56

0.38

5.38

1.85

2.22

5.03

1.30

1.17

7.39

4.46

0.23

Track_max_speed

14.91

1.52

8.23

0.77

4.57

0.32

2.77

11.19

0.26

3.8

0.54

0.46

12.03

0.5

4.46

1.03

10.93

4.46

5.14

13.55

2.2

3.89

13.8

9.2

0.4

Track_min_speed

0.91

0.06

0.22

0.06

0

0.07

0.47

1.4

0.07

0.27

0.04

0.09

1.53

0.05

0.67

0.04

0.44

0.97

0.45

1.47

0.26

0.13

2.64

0.93

0.12

Track_median_speed

2.99

0.07

1.43

0.12

1.01

0.18

0.69

3.13

0.21

1.08

0.14

0.28

3

0.24

4.46

0.21

3.71

1.15

1.09

1.79

1.22

0.33

6.6

3.46

0.19

5.88

0.65

3.93

0.34

1.95

0.09

0.96

4.55

0.08

1.39

0.20

0.14

4.22

0.17

2.69

0.42

4.52

1.48

2.54

5.21

0.79

1.57

4.11

3.77

(continued)

Track_std_speed 0.12

Appendix E 125

Track_106

Track_107

Track_108

Track_109

Track_110

Track_111

Track_112

Track_113

Track_114

Track_115

Track_116

Track_117

Track_118

Track_119

Track_120

Track_121

Track_122

Track_123

Track_124

Track_126

Track_127

Track_128

Track_129

Track_130

Track_131

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

127

128

129

130

131

132

Track_label

5

5

5

5

5

5

3

5

5

5

5

2

5

5

5

5

5

5

5

5

5

5

5

5

4

Track_duration

Table E.1 (continued)

Track_start

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Track_stop

5

5

5

5

5

5

3

5

5

5

5

2

5

5

5

5

5

5

5

5

5

5

5

5

4

Track_displacement

0.12

26.24

14.75

0.57

1.02

0.43

0.04

5.21

0.86

3.67

1.42

0.42

2.61

1.5

11.88

5.34

1.55

23.73

0.57

0.59

5.9

2.43

0.48

0.12

17.17

Track_x_location

280.12

99.35

349.37

63.53

75.39

17.89

274.51

350.67

148.79

266.22

23.71

18.92

265.42

101.65

269.5

65.33

87.2

88.74

15.07

161.01

267.84

93.57

15.2

58.68

161.35

Track_y_location

41.08

313.94

313.01

323.35

39.33

314.51

29.69

320.89

16.17

13.85

331.56

13.11

12.41

12.63

20.07

334.38

25.72

326.64

23.49

23.23

17.04

17.05

280.86

273.71

266.23

Track_mean_speed

0.15

7.46

3.45

1.32

1.60

0.17

7.72

2.52

0.24

1.72

0.75

0.21

0.7

0.45

4.44

1.08

0.36

5.25

0.14

0.38

2.18

0.78

0.64

0.34

6.38

Track_max_speed

0.27

13.61

5.63

1.97

2.56

0.4

11.6

8.22

0.58

3.29

1.17

0.42

1.78

0.91

7.49

2.11

0.7

12.97

0.28

0.55

3.87

1.25

1.67

0.74

11.75

Track_min_speed

0.06

0.71

1.25

0.31

0.46

0.01

0.23

0.32

0.04

0.61

0.48

0

0.05

0.08

0.27

0.55

0.13

1

0.07

0.1

0

0.12

0.17

0.03

1.18

Track_median_speed

0.14

7.91

4.18

1.42

2.12

0.18

11.33

1.62

0.15

1.78

0.61

0.42

0.82

0.32

4.88

0.95

0.34

4.09

0.11

0.42

2.3

0.66

0.45

0.15

10.52

0.09

4.88

2.01

0.7

1.04

0.15

6.49

3.26

0.21

1.13

0.3

0.29

0.8

0.4

2.61

0.62

0.24

4.54

0.09

0.2

1.39

0.48

0.61

0.33

(continued)

Track_std_speed 5.52

126 Appendix E

Track_132

Track_133

Track_134

Track_135

Track_136

Track_137

Track_138

Track_139

Track_140

Track_141

Track_142

Track_143

Track_144

Track_146

Track_147

Track_148

Track_149

Track_150

Track_151

Track_152

Track_153

Track_154

Track_155

Track_156

Track_157

133

134

135

136

137

138

139

140

141

142

143

144

145

147

148

149

150

151

152

153

154

155

156

157

158

Track_label

5

5

5

5

5

2

5

5

5

5

5

4

5

3

5

5

5

5

5

5

5

5

5

5

5

Track_duration

Table E.1 (continued)

Track_start

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Track_stop

5

5

5

5

5

2

5

5

5

5

5

4

5

3

5

5

5

5

5

5

5

5

5

5

5

Track_displacement

11.77

6.81

5.89

3.66

3.04

2.78

0.16

0.53

1.4

32.22

2.1

1.14

5.17

0.04

6.89

0.84

0.47

2.30

3.78

0.53

28.77

0.91

7.08

13

1.94

Track_x_location

294.65

97.02

295.36

293.54

236.06

67.71

12.29

186.46

20.37

82.45

21.23

196.07

137.49

285.35

11.65

207.48

67.93

354.66

109.2

66.46

109.73

172.18

273.32

348.35

165.9

Track_y_location

71.48

67.06

73.1

69.27

80

350.67

40.84

42.1

355.26

339.28

361.50

48.82

51.54

51.72

59.83

56.88

57.83

367.15

57.81

312.42

304.07

31.78

27.35

307.13

26.92

Track_mean_speed

4.26

1.45

7.42

2.37

0.68

1.4

1.03

0.47

0.49

6.72

0.99

0.51

1.11

0.51

1.45

0.24

2.86

1.67

0.86

0.22

7.14

0.3

6.06

3.93

0.42

Track_max_speed

13.14

2.03

10.36

4.71

1.25

1.94

2.51

0.88

0.73

12.09

1.45

0.95

2.47

0.77

4.64

0.63

6.36

2.62

1.69

0.27

12.20

0.39

11.41

9.25

1.3

Track_min_speed

1.07

0.34

4.47

0.2

0.21

0.87

0.24

0.19

0.05

2.52

0.24

0.18

0.23

0.28

0.18

0.11

0.45

0.63

0.39

0.09

0.71

0.09

0.09

1.48

0.04

Track_median_speed

2.3

1.49

9

3.01

0.75

1.94

0.71

0.44

0.49

5.28

1.43

0.57

0.85

0.47

0.44

0.16

3.61

1.54

0.71

0.24

6.88

0.37

7.25

1.95

0.27

5

0.69

2.73

1.93

0.41

0.76

0.91

0.26

0.26

3.75

0.58

0.33

0.86

0.25

1.89

0.22

2.7

0.89

0.49

0.07

4.23

0.13

5.64

3.31

(continued)

Track_std_speed 0.51

Appendix E 127

Track_159

Track_160

Track_163

Track_164

Track_165

Track_167

Track_168

Track_169

Track_171

Track_172

Track_173

Track_174

Track_175

Track_176

Track_177

Track_178

Track_180

Track_181

Track_182

Track_183

Track_184

Track_185

Track_186

Track_187

Track_188

160

161

164

165

166

168

169

170

172

173

174

175

176

177

178

179

181

182

183

184

185

186

187

188

189

Track_label

5

5

5

5

5

5

3

4

5

5

5

5

4

3

5

5

5

5

5

5

5

5

5

5

5

Track_duration

Table E.1 (continued)

Track_start

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Track_stop

5

5

5

5

5

5

3

4

5

5

5

5

4

3

5

5

5

5

5

5

5

5

5

5

5

Track_displacement

8.72

0.73

0.3

0.91

8.35

9.56

5.51

7.54

0.32

4.02

0.59

15.96

0.27

1.18

4.58

11

13.36

7.67

3.71

0.22

2.7

1.16

6.32

0.34

6.51

Track_x_location

130.02

84.73

94.97

61.99

164.04

176.94

142.34

102.52

189.81

189.59

7.80

210.15

298.56

272.8

284.9

239.78

262.07

336.78

316.59

305.18

151.41

244.25

115.16

64.57

353.73

Track_y_location

392.14

392.17

392.13

80.08

392.14

392.12

392.15

76.17

78.59

392.13

82.33

392.18

79.61

392.13

392.14

392.17

392.14

392.23

392.18

392.1

91.27

93.56

88.62

67.3

342.75

Track_mean_speed

2.06

0.27

0.31

0.49

2.67

4.51

1.84

1.75

1.45

1.03

0.18

3.81

0.82

4.55

3.74

3.29

3.90

2.83

0.74

0.53

1.25

0.4

1.51

0.24

2.65

Track_max_speed

7.91

0.43

0.41

0.75

7.89

13.19

3.04

2.7

2.75

2.78

0.55

10.97

1.57

7.41

8.77

4.64

5.4

4.74

1.02

1.21

1.85

0.63

2.64

0.49

5.06

Track_min_speed

0.02

0.12

0.21

0.29

0.02

0.41

0.62

0.57

0.16

0.09

0.01

0.07

0.20

1.72

0.5

1.45

1.96

0.73

0.24

0.12

0.78

0.21

0.75

0.02

1.58

Track_median_speed

0.77

0.27

0.33

0.48

2.49

3.38

1.86

1.97

1.27

0.59

0.09

1.52

1.17

4.51

2.86

3.8

4.24

3.23

0.93

0.39

1.08

0.41

1.03

0.16

1.81

3.3

0.12

0.09

0.17

3.18

5.92

1.21

1.09

1.05

1.04

0.22

4.71

0.66

2.84

3.28

1.24

1.38

2.01

0.35

0.44

0.49

0.16

0.9

0.22

(continued)

Track_std_speed 1.46

128 Appendix E

Track_189

Track_190

Track_191

Track_192

Track_193

Track_194

Track_195

Track_196

Track_197

Track_198

Track_199

Track_200

Track_201

Track_202

Track_203

Track_204

Track_205

Track_207

Track_208

Track_210

Track_211

Track_212

Track_213

Track_215

Track_218

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

208

209

211

212

213

214

216

219

Track_label

3

3

4

4

4

4

4

4

4

4

4

4

3

5

5

2

5

5

5

5

5

5

5

5

5

Track_duration

Table E.1 (continued)

Track_start

2

2

1

1

1

1

1

1

1

1

1

1

1

0

0

0

0

0

0

0

0

0

0

0

0

Track_stop

5

5

5

5

5

5

5

5

5

5

5

5

4

5

5

2

5

5

5

5

5

5

5

5

5

Track_displacement

11.87

8.8

0.74

12.8

3.16

11.82

4.18

2

0.55

1.26

2.97

0.43

5.8

0.12

1.46

5.54

1.61

9.24

1.21

6.08

0.06

0.24

0.27

1.28

3.7

Track_x_location

164.1

344.93

62.44

154.66

18.39

108.74

234.02

47.88

218.61

294.09

126.64

285.31

293.54

24.42

177.49

195.37

59.02

307.4

7.21

314.51

33.04

66.68

56.67

312.97

116.09

Track_y_location

255.73

285.62

299.05

205.48

14.75

170.77

239.71

221.05

57.98

70.43

39.73

51.76

392.14

369.15

95.71

93.01

97.7

95.57

105.67

111.68

391.97

392.14

392.1

107.75

392.15

Track_mean_speed

4.81

5.22

0.64

3.55

0.93

3.87

4.06

1.07

0.26

6.72

0.82

0.32

2.62

0.37

0.99

3

0.58

2.11

0.89

4.06

0.03

0.33

0.41

1.54

4.33

Track_max_speed

8.87

11.75

1.03

5.94

3.16

7.38

5.35

2.53

0.38

11.67

1.7

0.47

6.82

0.88

1.32

5.59

0.96

9.39

1.03

7.11

0.05

0.45

0.86

3.59

11.46

Track_min_speed

1.68

0.47

0.31

1.86

0

1.65

2.31

0.36

0.15

2.3

0.1

0.14

0.23

0.1

0.39

0.42

0.32

0.06

0.66

0

0.01

0.17

0.04

0

1.22

Track_median_speed

3.87

3.43

0.63

3.58

0.27

4.7

4.93

0.75

0.3

10.32

0.79

0.38

0.8

0.29

1.06

5.59

0.58

0.38

0.97

3.95

0.02

0.31

0.39

0.86

2.65

3.69

5.85

0.3

1.74

1.5

2.74

1.37

0.99

0.1

4.97

0.66

0.15

3.65

0.3

0.37

3.66

0.25

4.07

0.16

2.77

0.02

0.11

0.36

1.75

(continued)

Track_std_speed 4.1

Appendix E 129

Track_219

Track_221

Track_222

Track_223

Track_225

Track_226

Track_227

Track_228

Track_229

Track_230

Track_231

Track_232

Track_233

Track_234

Track_235

Track_236

Track_239

Track_240

Track_241

220

222

223

224

226

227

228

229

230

231

232

233

234

235

236

237

240

241

242

Track_label

2

2

2

2

2

2

2

2

2

2

3

3

2

3

3

3

3

3

3

Track_duration

Table E.1 (continued)

Track_start

3

3

3

3

3

3

3

3

3

3

2

2

2

2

2

2

2

2

2

Track_stop

5

5

5

5

5

5

5

5

5

5

5

5

4

5

5

5

5

5

5

Track_displacement

22.34

1.42

10.5

0.61

2.08

6.57

1.4

9.26

5.94

18.36

13.64

5.94

0.37

4.2

5.41

1.35

1.57

1

1.45

Track_x_location

352.25

354.17

205.37

330.39

323.06

188.51

289.31

129.44

143.72

111.38

147.34

153.42

293.99

245.61

312.13

137.98

154.9

75.01

75.43

Track_y_location

334.21

355.49

392.17

160.87

133.87

181.98

60.99

278.8

270.03

294.26

188.86

178.74

187.58

217.09

105.92

121.22

392.14

40.21

39.32

Track_mean_speed

11.17

0.71

5.25

2.09

1.26

3.61

0.71

5.28

2.97

9.58

5.57

4.01

0.37

3.52

2.09

1.02

1.23

1.69

0.94

Track_max_speed

14.38

1.18

10.09

2.16

2.28

3.93

1.24

8.57

3.09

13.99

8.2

7.16

0.55

6.23

5.29

1.61

1.95

3.03

2.12

Track_min_speed

7.97

0.24

0.41

2.03

0.25

3.29

0.18

1.99

2.85

5.17

1.43

1.48

0.19

0.47

0.48

0.65

0.68

0.67

0.23

Track_median_speed

14.38

1.18

10.09

2.16

2.28

3.93

1.24

8.57

3.09

13.99

7.07

3.38

0.55

3.86

0.51

0.80

1.06

1.37

0.46

Track_std_speed

4.53

0.67

6.84

0.09

1.44

0.45

0.75

4.66

0.17

6.24

3.63

2.89

0.25

2.9

2.77

0.52

0.66

1.21

1.03

130 Appendix E

References

Aires KR, Santana AM, Medeiros AA (2008) Optical flow using color information: preliminary results. In: Proceedings of the 2008 ACM symposium on applied computing, Mar 2008. ACM, pp 1607–1611 Alexa M, Behr J, Cohen-Or D, Fleishman S, Levin D, Silva CT (2003) Computing and rendering point set surfaces. IEEE Trans Vis Comput Graph 9(1):3–15 Al-Mayah A, Moseley J, Brock KK (2008) Contact surface and material nonlinearity modeling of human lungs. Phys Med Biol 53(1):305 Ashburner J, Friston KJ (2011) Diffeomorphic registration using geodesic shooting and GaussNewton optimisation. NeuroImage 55(3):954–967 Bajcsy R, Kovaˇciˇc S (1989) Multiresolution elastic matching. Comput Vis Graph Image Process 46(1):1–21 Baker S, Scharstein D, Lewis JP, Roth S, Black MJ, Szeliski R (2011) A database and evaluation methodology for optical flow. Int J Comput Vis 92(1):1–31 Barron JL, Fleet DJ, Beauchemin SS (1994) Performance of optical flow techniques. Int J Comput Vis 12(1):43–77 Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Computer vision— ECCV 2006, vol 3951. Springer Berlin Heidelberg, Zurich, pp 404–417 Bay H, Ess A, Tuytelaars T, Van Gool L (2008) SURF: speeded up robust features. Comput Vis Image Understand (CVIU) 110:346–359 Beauchemin SS, Barron JL (1995) The computation of optical flow. ACM Comput Surv (CSUR) 27(3):433–466 Beuthien B, Kamen A, Fischer B (2010) Recursive Green’s function registration. In: Medical image computing and computer-assisted intervention–MICCAI 2010. Springer Berlin Heidelberg, pp 546–553 Black MJ, Anandan P (1996) The robust estimation of multiple motions: parametric and piecewisesmooth flow fields. Comput Vis Image Understand 63(1):75–104 Blackall JM, Ahmad S, Miquel ME, McClelland JR, Landau DB, Hawkes DJ (2006) MRIbased measurements of respiratory motion variability and assessment of imaging strategies for radiotherapy planning. Phys Med Biol 51:4147–4169 Bookstein FL, Green WD (1993) A feature space for edgels in images with landmarks. J Math Imaging Vis 3(3):231–261 Bro-Nielsen M, Gramkow C (1996) Fast fluid registration of medical images. In: Visualization in biomedical computing. Springer Berlin Heidelberg, pp 265–276 Brock KK, Sharpe MB, Dawson LA, Kim SM, Jaffray DA (2005) Accuracy of finite element model-based multi-organ deformable image registration. Med Phys 32(6):1647–1659 Broit C (1981) Optimal registration of deformed images © Springer Nature Singapore Pte Ltd. 2020 A. I. Abidi and S. K. Singh, Deformable Registration Techniques for Thoracic CT Images, https://doi.org/10.1007/978-981-10-5837-0

131

132

References

Cachier P, Bardinet E, Dormont D, Pennec X, Ayache N (2003) Iconic feature based nonrigid registration: the PASHA algorithm. Comput Vis Image Understand 89(2):272–298 Castillo R, Castillo E, Guerra R, Johnson VE, McPhail T, Garg AK, Guerrero T (2009) A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets. Phys Med Biol 54(7):1849 Castillo E, Castillo R, Martinez J, Shenoy M, Guerrero T (2010) Four-dimensional deformable image registration using trajectory modeling. Phys Med Biol 55(1):305 Castillo E, Castillo R, Fuentes D, Guerrero T (2014) SU-EJ-197: a moving least squares approach for computing spatially accurate transformations that satisfy strict physiologic constraints. Med Phys 41(6):202 Chiang MC, Leow AD, Klunder AD, Dutton RA, Barysheva M, Rose SE, McMahon KL, de Zubicaray GI, Toga AW, Thompson PM (2008) Fluid registration of diffusion tensor images using information theory. IEEE Trans Med Imaging 27(4):442–456 Christensen GE, Johnson HJ (2001) Consistent image registration. IEEE Trans Med Imaging 20(7):568–582 Christensen GE, Rabbitt RD, Miller MI (1996) Deformable templates using large deformation kinematics. IEEE Trans Image Process 5(10):1435–1447 Christensen GE, Joshi SC, Miller MI (1997) Volumetric transformation of brain anatomy. IEEE Trans Med Imaging 16(6):864–877 Cornelis N, Gool LV (2008) Fast scale invariant feature detection and matching on programmable graphics hardware. In: CVPRW’08. IEEE computer society conference on computer vision and pattern recognition workshops, June 2008. IEEE, pp 1–8. Cotter CJ, Holm DD (2006) Singular solutions, momentum maps and computational anatomy. arXiv preprint nlin/0605020 D’Agostino E, Maes F, Vandermeulen D, Suetens P (2003) A viscous fluid model for multimodal non-rigid image registration using mutual information. Med Image Anal 7(4):565–575 Davatzikos C (1997) Spatial transformation and registration of brain images using elastically deformable models. Comput Vis Image Understand 66(2):207–222 Dupuis P, Grenander U, Miller MI (1998) Variational problems on flows of diffeomorphisms for image matching. Q Appl Math 56(3):587 Dykstra AR, Chan AM, Quinn BT, Zepeda R, Keller CJ, Cormier J, Madsen JR, Eskandar EN, Cash SS (2012) Individualized localization and cortical surface-based registration of intracranial electrodes. NeuroImage 59(4):3563–3570 Ehrhardt J, Werner R, Schmidt-Richberg A, Handels H (2011) Statistical modeling of 4D respiratory lung motion using diffeomorphic image registration. IEEE Trans Med Imaging. https://doi.org/ 10.1109/TMI.2010.2076299 Feldmar J, Ayache N (1996) Rigid, affine and locally affine registration of free-form surfaces. Int J Comput Vis 18(2):99–119 Fischer B, Modersitzki J (2002) Fast diffusion registration. Contemp Math 313:117–128 Fischer B, Modersitzki J (2003) Curvature based image registration. J Math Imaging Vis 18(1):81–85 Fischer B, Modersitzki J (2004) A unified approach to fast image registration and a new curvature based registration technique. Linear Algebra Appl 380:107–124 Fleet D, Weiss Y (2006) Optical flow estimation. In: Handbook of mathematical models in computer vision. Springer US, pp 237–257 Fuerst B, Mansi T, Carnis F, Salzle M, Zhang J, Declerck J, Kamen A (2015) Patient-specific biomechanical model for the prediction of lung motion from 4-D CT images. IEEE Trans Med Imaging 34(2):599–607 Goitein M (2004) Organ and tumor motion: an overview. Sem Radiat Oncol 14:2–9 Hadamard J (2014) Lectures on Cauchy’s problem in linear partial differential equations. Courier Corporation Haller JA, Kramer SS, Lietman SA (1987) Use of CT scans in selection of patients for pectusexcavatum surgery: a preliminary report. J Pediatr Surg 22(10):904–906

References

133

Heinrich HP, Jenkinson M, Brady M, Schnabel J (2013) MRF-based deformable registration and ventilation estimation of lung CT. IEEE Trans Med Imaging 32(7):1239–1248 Henn S (2006) A full curvature based algorithm for image registration. J Math Imaging Vis 24(2):195–208 Higham NJ (2005) The scaling and squaring method for the matrix exponential revisited. SIAM J Matrix Anal Appl 26(4):1179–1193 Horn BK, Schunck BG (1981) Determining optical flow. In: 1981 technical symposium east, Nov 1981. International Society for Optics and Photonics, pp 319–331 http://www.dir-lab.com/Motivation.html Igarashi T, Moscovich T, Hughes JF (2005) As rigid-as-possible shape manipulation. ACM Trans Graph 24(3):1134–1141 Jain AK, Zhong Y, Lakshmanan S (1996) Object matching using deformable templates. IEEE Trans Pattern Anal Mach Intell 18(3):267–278 Joshi SC, Miller MI (2000) Landmark matching via large deformation diffeomorphisms. IEEE Trans Image Process 9(8):1357–1370 Kang TK, Choi IH, Lim MT (2015) MDGHM-SURF: a robust local image descriptor based on modified discrete Gaussian-Hermite moment. Pattern Recogn 48(3):670–684 Keall PJ et al (2006) The management of respiratory motion in radiation oncology report of AAPM task group 76. Med Phys 33:3874–3900 Kim HC, Park HJ, Nam KW, Kim SM, Choi EJ, Jin S, Kim MG (2010) Fully automatic initialization method for quantitative assessment of chest-wall deformity in funnel chest patients. Med Biol Eng Comput 48(6):589–595 Koenderink JJ (1984) The structure of images. Biol Cybern 50(5):363–370 Kybic J, Unser M (2003) Fast parametric elastic image registration. IEEE Trans Image Process 12(11):1427–1442 Lancaster P, Salkauskas K (1981) Surfaces generated by moving least squares methods. Math Comput 37(155):141–158 Leow A, Huang SC, Geng A, Becker J, Davis S, Toga A, Thompson P (2005) Inverse consistent mapping in 3D deformable image registration: its construction and statistical properties. In: Information processing in medical imaging, July 2005. Springer Berlin Heidelberg, pp 493–503 Lindeberg T (1998) Feature detection with automatic scale selection. Int J Comput Vis 30(2):79–116 Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: IJCAI, vol 81, Aug 1981, pp 674–679 Malis E (2004) Improving vision-based control using efficient second-order minimization techniques. In: 2004 IEEE international conference on robotics and automation. Proceedings. ICRA’04, vol 2, Apr 2004. IEEE, pp 1843–1848 Markelj P, Tomaževiˇc D, Likar B, Pernuš F (2012) A review of 3D/2D registration methods for image-guided interventions. Med Image Anal 16(3):642–661 Marsland S, Twining CJ (2004) Constructing diffeomorphic representations for the groupwise analysis of nonrigid registrations of medical images. IEEE Trans Med Imaging 23(8):1006–1020 Marsland S, McLachlan R (2007) A Hamiltonian particle method for diffeomorphic image registration. In: Information processing in medical imaging, July 2007. Springer Berlin Heidelberg, pp 396–407 Mcgregor B (1998) Automatic registration of images of pigmented skin lesions. Pattern Recogn 31(6):805–817 Mikolajczyk K, Schmid C (2001) Indexing based on scale invariant interest points. In: Eighth IEEE international conference on computer vision, 2001. ICCV 2001. Proceedings, vol 1, pp 525–531 Modersitzki J (2004) Numerical methods for image registration. Oxford University Press Moler C, Van Loan C (2003) Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later. SIAM Rev 45(1):3–49 Muenzing SEA, Van Ginnekan B, Viergever MA, Pluim JPW (2014) DIRBoost-an algorithm for boosting deformable image registration: application to lung CT intra-subject registration. Med Image Anal 18:449–459

134

References

Murphy K, Van Ginneken B, Reinhardt JM, Kabus S, Ding K, Deng X, Vercauteren T (2011) Evaluation of registration methods on thoracic CT: the EMPIRE10 challenge. IEEE Trans Med Imaging 30(11):1901–1920 Nakagomi K, Shimizu A, Kobatake H, Yakami M, Fujimoto K, Togashi K (2013) Multi-shape graph cuts with neighbor prior constraints and its application to lung segmentation from a chest CT volume. Med Image Anal 17(1):62–77 Nehmeh SA, Erdi YE, Pan T et al (2004) Quantitation of respiratory motion during 4D-PET/CT acquisition. Med Phys 31(6):1333–1338 Oliviera FPM, Tavares JMRS (2012) Medical image registration: a review. Comput Methods Biomech Biomed Eng 17:73–93 Pang Y et al (2012) Fully affine invariant SURF for image matching. Neurocomputing 85:6–10. ISSN 0925-2312 Pedersen JT (2011) SURF: feature detection & description, Q4 2011 Pennec X, Stefanescu R, Arsigny V, Fillard P, Ayache N (2005) Riemannian elasticity: a statistical regularization framework for non-linear registration. In: Medical image computing and computerassisted intervention–MICCAI 2005. Springer Berlin Heidelberg, pp 943–950 Peyrat JM, Delingette H, Sermesant M, Pennec X, Xu C, Ayache N (2008) Registration of 4D timeseries of cardiac images with multichannel diffeomorphic demons. In: Medical image computing and computer-assisted intervention–MICCAI 2008. Springer Berlin Heidelberg, pp 972–979 Rabbitt RD, Weiss JA, Christensen GE, Miller MI (1995) Mapping of hyperelastic deformable templates using the finite element method. In: SPIE’s 1995 international symposium on optical science, engineering, and instrumentation, Aug 1995. International Society for Optics and Photonics, pp 252–265 Reinhardt JM, Ding K, Cao K, Christensen GE, Hoffman EA, Bodas SV (2008) Registration-based estimates of local lung tissue expansion compared to xenon CT measures of specific ventilation. Med Image Anal 12(6):752–763 Rohr K, Stiehl HS, Sprengel R, Buzug TM, Weese J, Kuhn MH (2001) Landmark-based elastic registration using approximating thin-plate splines. IEEE Trans Med Imaging 20(6):526–534 Sarrut D, Boldea V, Miguet S, Ginestet C (2006) Simulation of four-dimensional CT images from deformable registration between inhale and exhale breath-hold CT scans. Med Phys 33(3):605– 617 Sato AK, Stevo NA, Tavares RS, Tsuzuki MSG, Kadota E, Gotoh T, Kagei S, Iwasawa T (2011) Registration of temporal sequences of coronal and sagittal MR images through respiratory patterns. Biomed Signal Process Control 6(1):34–47 Schaefer S, McPhail T, Warren J (2006) Image deformation using moving least squares. ACM Trans Graph (TOG) 25(3):533–540 Silva EA, Panetta K, Agaian SS (2007) Quantifying image similarity using measure of enhancement by entropy. In: Defense and security symposium, Apr 2007. International Society for Optics and Photonics, p 65790U So RW, Tang TW, Chung AC (2011) Non-rigid image registration of brain magnetic resonance images using graph-cuts. Pattern Recogn 44(10):2450–2467 Sotiras A, Davatzikos C, Paragios N (2013) Deformable medical image registration: a survey. IEEE Trans Med Imaging 32:1153–1190 Stefanescu R, Pennec X, Ayache N (2004) Grid powered nonlinear image registration with locally adaptive regularization. Med Image Anal 8(3):325–342 Stevo N, Campos R, Tavares RS, Tsuzuki MDSG, Gotoh T, Kagei S, Iwasawa T (2009) Registration of temporal sequences of coronal and sagittal images obtained from magnetic resonance. In: Proceedings of XX international congress of mechanical engineering Sull S, Ahuja N (1995) Integrated matching and segmentation of multiple features in two views. Comput Vis Image Understand 62(3):279–297 Sun D, Roth S, Black MJ (2014) A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int J Comput Vis 106(2):115–137

References

135

Sundaram TA, Gee JC (2005) Towards a model of lung biomechanics: pulmonary kinematics via registration of serial lung images. Med Image Anal 9(6):524–537 Szeliski R, Coughlan J (1994) Hierarchical spline-based image registration. In: 1994 IEEE computer society conference on computer vision and pattern recognition, 1994. Proceedings CVPR’94, June 1994. IEEE, pp 194–201 Szeliski R, Coughlan J (1997) Spline-based image registration. Int J Comput Vis 22(3):199–218 Thirion JP (1998) Image matching as a diffusion process: an analogy with Maxwell’s demons. Med Image Anal 2(3):243–260 Thomson W (1874) The kinetic theory of the dissipation of energy. Proc R Soc Edinburgh 8:325–334 Trouvé A (1998) Diffeomorphisms groups and pattern matching in image analysis. Int J Comput Vis 28(3):213–221 Tsui BM, Segars WP, Lalush DS (2000) Effects of upward creep and respiratory motion in myocardial SPECT. IEEE Trans Nucl Sci 47:1192–1195 Ugural AC, Fenster SK (2003) Advanced strength and applied elasticity. Pearson Education Vedam SS, Keall PJ, Docef A, Todor DA, Kini VR, Mohan R (2004) Predicting respiratory motion for four-dimensional radio-therapy. Med Phys 31(8):2274–2283 Vercauteren T, Pennec X, Malis E, Perchant A, Ayache N (2007) Insight into efficient image registration techniques and the demons algorithm. In: Information processing in medical imaging, July 2007. Springer Berlin Heidelberg, pp 495–506 Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, 2001. CVPR 2001, vol 1. IEEE, p I-511 Wang Y, Staib LH (2000) Physical model-based non-rigid registration incorporating statistical shape information. Med Image Anal 4(1):7–20 Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612 Warren DH, Strelow ER (1985) Sensory substitution in blind children and neonates. In: Electronic spatial sensing for the blind. Springer Netherlands, pp 273–298 Xiong G, Chen C, Chen J, Xie Y, Xing L (2012) Tracking the motion trajectories of junction structures in 4D CT images of the lung. Phys Med Biol 57(15):4905 Yanovsky I, Le Guyader C, Leow A, Toga A, Thompson P, Vese L (2008) Unbiased volumetric registration via nonlinear elastic regularization. In: 2nd MICCAI workshop on mathematical foundations of computational anatomy, Oct 2008 Yeo BT, Vercauteren T, Fillard P, Peyrat JM, Pennec X, Golland P, Clatz O (2009) DT-REFinD: diffusion tensor registration with exact finite-strain differential. IEEE Trans Med Imaging 28(12):1914–1928 Yeo BT, Sabuncu MR, Vercauteren T, Ayache N, Fischl B, Golland P (2010) Spherical demons: fast diffeomorphic landmark-free surface registration. IEEE Trans Med Imaging 29(3):650–668 Yoon H, Chung H, Hahn H (2009) SURF algorithm with color and global characteristics. In: ICCAS-SICE 2009, pp 183–187 Zhang T, Orton NP, Mackie TR, Paliwal BR (2004) Technical note: a novel boundary condition using contact elements for finite element based deformable image registration. Med Phys 31(9):2412– 2415 Zhang B, Arola DD, Roys S, Gullapalli RP (2011) Three-dimensional elastic image registration based on strain energy minimization: application to prostate magnetic resonance imaging. J Digit Imaging 24(4):573–585 Zhang Y, Yang J, Zhang L, Balter PA, Dong L (2013) Modeling respiratory motion for reducing motion artifacts in 4D CT images. Med Phys 40(4):041716 Zitova B, Flusser J (2003) Image registration methods: a survey. Image Vis Comput 21(11):977– 1000 Zordan VB, Celly B, Chiu B, DiLorenzo PC (2004) Breathe easy: model and control of simulated respiration for animation. In: Proceedings of the 2004 ACM SIGGRAPH/eurographics symposium on computer animation, Grenoble, France