Robot Vision

Citation preview

I

Robot Vision

Robot Vision

Edited by

Aleš Ude

In-Tech

intechweb.org

Published by In-Teh In-Teh Olajnica 19/2, 32000 Vukovar, Croatia Abstracting and non-proit use of the material is permitted with credit to the source. Statements and opinions expressed in the chapters are these of the individual contributors and not necessarily those of the editors or publisher. No responsibility is accepted for the accuracy of information contained in the published articles. Publisher assumes no responsibility liability for any damage or injury to persons or property arising out of the use of any materials, instructions, methods or ideas contained inside. After this work has been published by the In-Teh, authors have the right to republish it, in whole or part, in any publication of which they are an author or editor, and the make other personal use of the work. © 2010 In-teh www.intechweb.org Additional copies can be obtained from: [email protected] First published March 2010 Printed in India Technical Editor: Martina Peric Cover designed by Dino Smrekar Robot Vision, Edited by Aleš Ude p. cm. ISBN 978-953-307-077-3

V

Preface The purpose of robot vision is to enable robots to perceive the external world in order to perform a large range of tasks such as navigation, visual servoing for object tracking and manipulation, object recognition and categorization, surveillance, and higher-level decisionmaking. Among different perceptual modalities, vision is arguably the most important one. It is therefore an essential building block of a cognitive robot. Most of the initial research in robot vision has been industrially oriented and while this research is still ongoing, current works are more focused on enabling the robots to autonomously operate in natural environments that cannot be fully modeled and controlled. A long-term goal is to open new applications to robotics such as robotic home assistants, which can only come into existence if the robots are equipped with signiicant cognitive capabilities. In pursuit of this goal, current research in robot vision beneits from studies in human vision, which is still by far the most powerful existing vision system. It also emphasizes the role of active vision, which in case of humanoid robots does not limit itself to active eyes only any more, but rather employs the whole body of the humanoid robot to support visual perception. By combining these paradigms with modern advances in computer vision, especially with many of the recently developed statistical approaches, powerful new robot vision systems can be built. This book presents a snapshot of the wide variety of work in robot vision that is currently going on in different parts of the world. March 2010

Aleš Ude

VII

Contents Preface 1. Design and fabrication of soft zoom lens applied in robot vision

V 001

Wei-Cheng Lin, Chao-Chang A. Chen, Kuo-Cheng Huang and Yi-Shin Wang

2. Methods for Reliable Robot Vision with a Dioptric System

013

E. Martínez and A.P. del Pobil

3. An Approach for Optimal Design of Robot Vision Systems

021

Kanglin Xu

4. Visual Motion Analysis for 3D Robot Navigation in Dynamic Environments

037

Chunrong Yuan and Hanspeter A. Mallot

5. A Visual Navigation Strategy Based on Inverse Perspective Transformation

061

Francisco Bonin-Font, Alberto Ortiz and Gabriel Oliver

6. Vision-based Navigation Using an Associative Memory

085

Mateus Mendes

7. Vision Based Robotic Navigation: Application to Orthopedic Surgery

111

P. Gamage, S. Q. Xie, P. Delmas and W. L. Xu

8. Navigation and Control of Mobile Robot Using Sensor Fusion

129

Yong Liu

9. Visual Navigation for Mobile Robots

143

Nils Axel Andersen, Jens Christian Andersen, Enis Bayramoğlu and Ole Ravn

10. Interactive object learning and recognition with multiclass support vector machines

169

Aleš Ude

11. Recognizing Human Gait Types

183

Preben Fihl and Thomas B. Moeslund

12. Environment Recognition System for Biped Robot Walking Using Vision Based Sensor Fusion Tae-Koo Kang, Hee-Jun Song and Gwi-Tae Park

209

VIII

13. Non Contact 2D and 3D Shape Recognition by Vision System for Robotic Prehension

231

Bikash Bepari, Ranjit Ray and Subhasis Bhaumik

14. Image Stabilization in Active Robot Vision

261

Angelos Amanatiadis, Antonios Gasteratos, Stelios Papadakis and Vassilis Kaburlasos

15. Real-time Stereo Vision Applications

275

Christos Georgoulas, Georgios Ch. Sirakoulis and Ioannis Andreadis

16. Robot vision using 3D TOF systems

293

Stephan Hussmann and Torsten Edeler

17. Calibration of Non-SVP Hyperbolic Catadioptric Robotic Vision Systems

307

Bernardo Cunha, José Azevedo and Nuno Lau

18. Computational Modeling, Visualization, and Control of 2-D and 3-D Grasping under Rolling Contacts

325

Suguru Arimoto, Morio Yoshida and Masahiro Sekimoto1

19. Towards Real Time Data Reduction and Feature Abstraction for Robotics Vision

345

Rafael B. Gomes, Renato Q. Gardiman, Luiz E. C. Leite, Bruno M. Carvalho and Luiz M. G. Gonçalves

20. LSCIC Pre coder for Image and Video Compression

363

Muhammad Kamran, Shi Feng and Wang YiZhuo

21. The robotic visual information processing system based on wavelet transformation and photoelectric hybrid

373

DAI Shi-jie and HUANG-He

22. Direct visual servoing of planar manipulators using moments of planar targets

403

Eusebio Bugarin and Rafael Kelly

23. Industrial robot manipulator guarding using artiicial vision

429

Fevery Brecht, Wyns Bart, Boullart Luc Llata García José Ramón and Torre Ferrero Carlos

24. Remote Robot Vision Control of a Flexible Manufacturing Cell

455

Silvia Anton, Florin Daniel Anton and Theodor Borangiu

25. Network-based Vision Guidance of Robot for Remote Quality Control

479

Yongjin (James) Kwon, Richard Chiou, Bill Tseng and Teresa Wu

26. Robot Vision in Industrial Assembly and Quality Control Processes

501

Niko Herakovic

27. Testing Stereoscopic Vision in Robot Teleguide

535

Salvatore Livatino, Giovanni Muscato and Christina Koeffel

28. Embedded System for Biometric Identiication Ahmad Nasir Che Rosli

557

IX

29. Multi-Task Active-Vision in Robotics

583

J. Cabrera, D. Hernandez, A. Dominguez and E. Fernandez

30. An Approach to Perception Enhancement in Robotized Surgery using Computer Vision Agustín A. Navarro, Albert Hernansanz, Joan Aranda and Alícia Casals

597

Design and fabrication of soft zoom lens applied in robot vision



1

1 X

Design and fabrication of soft zoom lens applied in robot vision Wei-Cheng Lina, Chao-Chang A. Chenb, Kuo-Cheng Huanga and Yi-Shin Wangb

a Instrument

Technology Research Center, National Applied Research Laboratories Taiwan b National Taiwan University of Science and Technology Taiwan

1. Introduction The design theorem of traditional zoom lens uses mechanical motion to precisely adjust the separations between individual or groups of lenses; therefore, it is more complicated and requires multiple optical elements. Conventionally, the types of zoom lens can be divided into optical zoom and digital zoom. With the demands of the opto-mechanical elements, the zoom ability of lens system was dominated by the optical and mechanical design technique. In recent years, zoom lens is applied in compact imaging devices, which are popularly used in Notebook, PDA, mobile phone, and etc. The minimization of the zoom lens with excellent zoom ability and high imaging quality at the same time becomes a key subject of related efficient zoom lens design. In this decade, some novel technologies for zoom lens have been presented and they can simply be classified into the following three types: (1)Electro-wetting liquid lens: Electro-wetting liquid lens is the earliest liquid lens which contains two immiscible liquids having equal density but different refractive index, one is an electrically conductive liquid (water) and the other is a drop of oil (silicon oil), contained in a short tube with transparent end caps. When a voltage is applied, because of electrostatic property, the shape of interface between oil and water will be changed and then the focal length will be altered. Among the electro-wetting liquid lenses, the most famous one is Varioptic liquid lens. (2) MEMS process liquid lens: This type of liquid lens usually contains micro channel, liquid chamber and PDMS membrane which can be made in MEMS process, and then micro-pump or actuator is applied to pump liquid in/out the liquid chamber. In this way, the shape of liquid lens will change as plano-concave or plano-convex lens, even in bi-concave, bi-convex, meniscus convex and meniscus concave. It can also combine one above liquid lens to enlarge the field of view (FOV) and the zoom ratio of the system.

2

Robot Vision

(3)Liquid crystals lens: Liquid crystals are excellent electro-optic materials with electrical and optical anisotropies. Its optical properties can easily be varied by external electric field. When the voltage is applied, the liquid crystals molecules will reorient by the inhomogeneous electric field which generates a centro-symmetric refractive index distribution. Then, the focal length will be altered with varying voltage. (4)Solid tunable lens: One non-liquid tunable lens with PDMS was increasing temperature by heating the silicon conducting ring. PDMS lens is deformed by mismatching of bi-thermal expansion coefficients and stiffness between PDMS and silicon. Alternatively the zoom lens can be fabricated by soft or flexible materials, such as PDMS. The shape of soft lens can be changed by external force, this force may be mechanical force like contact force or magnetic force; therefore the focal length will be altered without any fluid pump. All presented above have a common character that they process optical zoom by changing lens shape and without any transmission mechanism. The earliest deformed lens can be traced to the theory of “Adaptive optics” that is a technology used to improve the performance of optical systems and is most used in astronomy to reduce the effects of atmospheric distortion. The adaptive optics usually used MEMS process to fabricate the flexible membranes and combine the actuator to make the lens with the motion of tilt and shift. However, there has not yet a zoom lens with PDMS that can be controlled by the fluid pressure for curvature change. In this chapter, PDMS is used for fabricating the lens of SZL and EFL of this lens system will be changed using pneumatic pressure.

2. PDMS properties PDMS, Dow Corning SYLGARD 184, is used because of its good light transmittance, a twopart elastomer (base and curing agent), is liquid in room temperature and it is cured by mixing base and curing agent at 25°C for 24 hour. Heating will shorten the curing time of PDMS, 100 °C for 1hour, 150°C for only 15 minutes. PDMS also has low glass transition temperature (-123 °C), good chemical and thermal stability, and contain flexibility and elasticity from -50 to 200°C. Since different curing parameter cause different Young's Modulus and refractive index, this section introduces the material properties of PDMS, including Young's Modulus, reflection index (n) and Abbe number (ν) in different curing parameter. 2.1 PDMS mechanical property (Young's Modulus) Young's Modulus is the important mechanic property in the analysis of the maximum displacement and deformed shape of PDMS lens. In order to find the relationship of curing parameter and Young's Modulus, tensile test is used, master curing parameters is curing time, curing temperature and mixing ratio. The geometric specification of test sample is define as standard ASTM D412 98a , the process parameter separate as 10:1 and 15:1, cured at 100 °C for 30, 45, 60 minutes and 150 °C for 15, 20, 25 minutes. As the result, in the same mixed ratio, higher curing temperature and long

Design and fabrication of soft zoom lens applied in robot vision

3

curing time cause larger Young’s Modulus; in the same curing parameter, in mixed ratio 10:1 has larger Young’s Modulus than 15:1. The mixed ratio 15:1 is soft then 10:1 but is weaker, so the fabrication of PDMS lens will use the ratio 10:1. 2.2 PDMS optical property (Refractive index and Abbe number) Refractive index (n) and Abbe number (ν) is the essential optic parameter of material optical properties in optical design. Spectroscopic Ellipsometer is used to inspect the n in wavelength 587nm, then discuss the nPDMS of mixed ratio 10:1 and 15:1 cured at 100°C for 30, 40, and 60min. The ν is defined as follows: Vd 

nd  1 nF  nC

(1)

nF, nC, and nd is the refractive index at 486.1、656.3 and 587.6nm. The nd and νd of PDMS is calculated according to the data by using least square method. As the result, curing parameter is not the key point that influences the optical properties of PDMS material; mixed ratio has more influence on optical properties. 10:1 has larger n and ν than 15:1. In this research, the n 1.395 and ν 50 is used. Table.1 shows the comparison with PDMS and the other most used optical materials. Materials BK7 PMMA COC PC PS Refractive index (n) 1.517 1.492 1.533 1.585 1.590 Abbe number (vd) 64.17 57.442 56.23 29.91 30.87 Table. 1. Optical property of PDMS and common optical material

PDMS 1.395 50

3. Methodology This section introduces the design process of pneumatic SZL system; this system is divided into SZL unit and pneumatic control unit. As the PDMS material properties is obtained, then the optical design, optical mechanism, PDMS lens mold and mold inserts is consequently designed and experimentally fabricated. After assembly of the PDMS lens and optic mechanism, the SZL system is presented and investigated for its optical performance of desired EFL. 3.1 Design of pneumatic soft zoom lens unit Components of the SZL unit are shown in Fig.1. The SZL unit includes BK7 lens and PDMS lens, the PDMS lens were designed with flat and spherical lens. The EFL is 33.56mm when using PDMS spherical lens and the EFL is 32.55mm when using PDMS flat lens. In order to simply verify this idea, optical performance of the SZL system is not the main purpose, thus diffractive optical elements and correct lens is a good choice to modify the aberration for better optical performance in this system. Mechanisms of SZL unit will be devised as barrel, O-ring, cell, retainer and spacer.

4

Robot Vision

Fig. 1. Components of the soft zoom lens system. 3.2 SZL system The SZL system assembly procedure is following these steps as show in Fig.2.: (1) PDMS lens is put into the barrel, (2) the spacer which constrain the distance between PDMS lens and BK7 lens is packed, (3) O-ring is fixed which can avoid air to escape, (4) BK7 lens is loaded and mounted by retainer, (5) BK7 protect lens is equipped with at both end of the SZL, and finally, the SZL system is combined with the SZL unit and pneumatic control unit with a pump (NITTO Inc.), a regulator, a pressure gauge and also a switch.

Fig. 2. Flow chart of SZL assembly process.

Design and fabrication of soft zoom lens applied in robot vision

5

Fig. 3. Soft zoom lens system contain pneumatic supply device for control unit. Fig.4. shows the zoom process of SZL system for variant applied pressures. The principle of SZL system is by the way of pneumatic pressure to adapt its shape and curvature, the gas input is supplied with NITTO pump and max pressure is up to 0.03 MPa. The regulator and pressure gauge control the magnitude of applied pressure, and there are two switches at both end of the SZL, one is input valve and the other is output valve.

Fig. 4. Zoom process of the SZL during the pneumatic pressure applied.

6

Robot Vision

3.3 PDMS lens mold and PDMS soft lens processing The lens mold contains two pair of mold inserts, one pair is for PDMS spherical lens and another is for PDMS flat lens. Each pair has the upper and the lower mold inserts. The material of mold insert is copper (Moldmax XL) and is fabricated by ultra-precision diamond turning machine. Fig.4. shows the fabrication process. Since the PDMS lens is fabricated by casting, the parting line between the upper and the lower mold inserts need have an air trap for over welling the extra amount PDMS. At the corner of the mold are four guiding pins to orientate the mold plates. The fabrication processes is shown at Fig.5. The fabrication process of PDMS lens is mixing, stirring, degassing and heating for curing. At first, base A and curing agent B is mixed and stirred with 10:1 by weight, then degas with a vacuum pump. After preparing the PDMS, cast PDMS into the mold cavity slowly and carefully, then close the mold and put into oven for curing. As the result, at curing temperature 100°C, PDMS lens can be formed after 60 min. Fig.6. is the PDMS lens of SZL unit. There are spherical lens and flat lens for experimental tests and they are fabricated at the same molding procedure.

(a) PDMS cast into mold cavity

(b) Mold cavity full of PDMS

(c) Curing in oven

(d) After curing

(a) PDMS flat lens

(b) PDMS sphere lens

Fig. 5. Fabrication processes of PDMS lens.

Fig. 6. PDMS lens of soft zoom lens system.

Design and fabrication of soft zoom lens applied in robot vision

7

3.4 Deformation analysis of PDMS lens In the deformation analysis of PDMS lens, the measured Young’s Modulus and Poisson’s Ratio of PDMS are inputted to the software and then set the meshed type, boundary condition and loading. The boundary condition is to constrain the contact surface between the PDMS lens and mechanism like barrel and spacer, and the load is the applied pressure of pneumatic supply device. Fig.7. shows the analysis procedure. As the result of the deformation analysis, comparing to the SZL with PDMS flat and spherical lens, the flat lens has larger deformation with the same curing parameter of both PDMS lens. Fig.8. shows the relationship of the maximum displacement versus the Young’s Modulus of PDMS flat lens and spherical lens.

Fig. 7. Flow chart of PDMS lens deformation analysis. PDMS flat lens V.S spherical lens

Max. displacement(mm

6 5 4 3 2 1 0

0.01

0.02 Pressure(MPa)

0.03

flat E = 1 MPa

flat E = 1.2 MPa

flat E = 1.4 MPa

spherical E = 1MPa

spherical E = 1.2MPa

spherical E = 1.4MPa

Fig. 8. Relationship of PDMS lens Young’s Modulus and maximum displacement.

8

Robot Vision

4. Results and Discussion The EFL of soft zoom lens system is inspected by an opto-centric instrument (Trioptics OptiCentric) in a transmission mode show as Fig.9. The lens system with PDMS spherical lens are cured at 100°C for 60, 70 min and separately inspected at five conditions as 0, 0.005, 0.01, 0.015 and 0.02 MPa. The EFL of soft zoom lens with PDMS lens cured at 100°C for 60 min changes from 33.440 to 39.717 mm or increasing 18.77% during the applied pressure from 0 to 0.02 MPa. The EFL of soft zoom lens with PDMS lens cured at 100°C for 70 min changes from 33.877 to 39.189 mm or increasing 15.68%. PDMS lens cured at 150°C for 45 min changes from 33.254 to 37.533 mm or increasing 12.87%. The longer curing time or larger curing temperature affects the stiffness of elastomer and the Young’s modulus increases for less deformable by the external force.

Fig. 9. The effective focal length measurement by using Trioptics. For the PDMS flat lens cured at 100°C for 60 min, the EFL of this SZL system changes from 32.554 to 34.177mm or increasing 4.99%. Fig.10. is the relationship of applied pressure and EFL of soft zoom lens system. Fig.11. shows the relationship of applied pressure and system zoom ratio. The EFL of the SZL system with flat PDMS lens seems not changing conspicuously as that with PDMS spherical lens. The variation of thickness of the flat lens is not so obvious than that of the spherical lens due to the deformation induced by the pneumatic pressure. The repeatability of the soft zoom lens is also inspected for the SZL with PDMS spherical lens and is measured for 100 times. Result is shown in Fig.12. 34.500 EFL[mm]

EFL[mm]

34.000 33.500 33.000 32.500

40.000 38.000 36.000 34.000 32.000 30.000 0.000

32.000 0.000

0.005

0.010

0.015

Pressure[MPa]

(a) 100 °CPDMS flat lens

0.020

0.005

0.010

0.015

0.020

Pressure[MPa] 100℃ 60min

100℃ 70min

(b) 100 °CPDMS spherical lens

Fig. 10. The relationship of applied pneumatic pressure and effective focal length with PDMS lens cured at 100°C.

Design and fabrication of soft zoom lens applied in robot vision

20.000 Zoom ratio[%]

10.000 Zoom ratio[%]

9

7.500 5.000 2.500

15.000 10.000 5.000 0.000

0.000 0.000

0.005

0.010

0.015

0.000

0.020

0.005

0.010

0.015

0.020

Pressure[MPa]

Pressure[MPa]

100℃ 60min

100℃ 60min

(a) PDMS flat lens

100℃ 70min

(b) PDMS spherical lens

Fig. 11. The relationship of applied pneumatic pressure and zoom ratio with different shape of PDMS lens. V a r ia b le : V a r 1 , D is t r ib u t io n : N o r m a l

V a r ia b le : V a r 1 , D is t r ib u t io n : N o r m a l

1 0 0

8 0

9 0

7 0

8 0

N o . o f o b s e r va tions

No . of o b se rva tio ns

6 0

7 0 6 0 5 0 4 0 3 0

5 0 4 0 3 0 2 0

2 0 1 0

1 0

0

0 3 6 .4 3 5

3 6 .4 6 5 3 6 .4 5 0

3 6 .4 9 5 3 6 .4 8 0

3 6 .5 2 5 3 6 .5 1 0

3 6 .5 5 5 3 6 .5 4 0

3 6 .5 8 5

3 6 .9 2 5

3 6 .9 7 5 3 6 .9 5 0

3 6 .5 7 0

3 7 .0 2 5 3 7 .0 0 0

3 7 .0 7 5 3 7 .0 5 0

3 7 .1 2 5 3 7 .1 0 0

3 7 .1 7 5 3 7 .1 5 0

C a t e g o r y ( u p p e r li m it s )

C a t e g o r y ( u p p e r lim i t s )

(a) EFL repeatability at 0.16 kg/cm2

Fig. 12. The repeatability of the SZL system.

(b) EFL repeatability at 0. 2 kg/cm2

In order to comprehend the variation of EFL with zoom effect of the developed SZL system, a CCD imaging experiment was performed for verification. A mechanical adapter was design to assemble the SZL unit onto the CCD camera. Fig.13. shows the experimental setup of this imaging experiment. In this experiment, the ISO 12233 pattern was used and the object length from the pattern to the SZL unit was fixed to observe the acquired image from the CCD camera for the pneumatic pressure applied from 0 to 0.02 MPa. The captured image during the pressure applied for each applied pressure is from clear to indistinct as show in Fig.14. It is obviously verified the zoom effect of this developed SZL system with the fabricated PDMS lens.

CCD

SZL

PC pump ISO 12233 pattern regulator, pressure gauge

Fig. 13. The experimental setup of the soft zoom lens system imaging.

10

Robot Vision

0MPa

0.004MPa

0.008MPa

0.01MPa

0.012MPa

0MPa

0.004MPa

0.008MPa

0.01MPa

0.012MPa

Fig. 14. The image of soft zoom lens system from the image test.

1.75

7.0

1.5

6.0

EFL variation[mm

EFL variation[mm

As a summary of experimental results and discussion, EFL is direct proportion to the magnitude of the applied pressure during the EFL inspection. When the pressure is applied, the EFL of the system and shape of PDMS lens change. The major EFL variation occurs in the beginning of applied pressure. For example, when the pressure is applied from 0 to 0.005MPa the EFL variation of PDMS lens cured at 100°C for 60min is 3.577mm, then when the applied pressure is up to 0.02MPa the variation of EFL is 6.277mm, the variation between 0.015 to 0.02MPa is only 0.749mm. Fig.15. shows the relationship of EFL variation and pressure. Comparing to the SZL with PDMS flat and spherical lens, the flat lens has larger deformation with the same curing parameter, but the zoom ratio does not as good as spherical lens. According to the experimental result, the thickness, curing parameter and the geometry shape of the PDMS lens can influence zoom ability. Therefore, the imaging experiment was performed by the SZL system with spherical PDMS lens and the obtained image has been verified for its feasibility of zoom effect.

1.25 1 0.75 0.5 0.25 0

0.000

0.005

0.010

0.015

Pressure[MPa] 100℃ 60min

(a) 100°C PDMS flat lens

0.020

5.0 4.0 3.0 2.0 1.0 0.0 0.000

0.005

0.010

0.015

0.020

Pressure[MPa] 100℃ 60min

100℃ 70min

(b) 100°C PDMS spherical lens

Fig. 15. The relationship of applied pneumatic pressure and variation of effective focal length.

5. Conclusion Based on the experimental results, the novel design of this SZL system has been proved effectively for its zoom effects for image. The EFL of SZL system with PDMS lens cured at 100°C for 60 min changes from 33.440 to 39.717 mm or increasing 18.77% for the applied pressure from 0 to 0.02 MPa. The curing temperature and time significantly affects the stiffness of the PDMS lens and causes different results of EFL. Experimental results also show that zoom effects of the developed SZL system are significantly affected by the shape,

Design and fabrication of soft zoom lens applied in robot vision

11

thickness of PDMS soft lens. The SZL system has been verified with the function of zoom ability. In future, the deformed shape of PDMS lens after applied pressure needs to be analyzed and then the altered lens’s shape can be obtained to be fit and calculated. It can be imported to the optical software to check the EFL variation after applying the pressure and also compare with the inspection data. Furthermore, it can find the optimized design by the shape analysis of PDMS lens when the pneumatic pressure is applied, then the correct lens is obtained to modify the optical image property. On the operation principle, we should find other actuator instead of fluid pump to change the shape of PDMS lens and EFL of SZL with external pressure for zoom lens devices. Further research will work on the integration of the SZL system with imaging system for mobile devices or robot vision applications.

6. Future work As the result, the pneumatic control soft zoom lens can simply proof our idea that the effective focal length of zoom lens will be altered through the deformation varied by changing the pneumatic pressure. As we know, the deformed lens whether in pneumatic system or MEMS micro-pump system is not convenient to apply in compact imaging devices. In order to improve this defect, we provide another type of soft zoom lens whose effective focal length can be tuned without external force. Fig.16. shows one of the new idea, flexible material are used to fabricate the hollow ball lens then filled with the transparent liquid or gel. The structure like human eyes and its shape can be altered by an adjustable ring, thus the focal length will be changed.

Fig. 16. The structure of artificial eyes. Another one of the improvement of pneumatic control soft zoom lens is showed in Fig.17. This zoom lens can combine one and above soft lens unit, each unit has at least one soft lens at both end and filled with transparent liquid or gel then seal up. The zoom lens with two and above soft lens unit can be a successive soft zoom lens and the motion is like a piston when the adjustable mechanism is moved, the fluid that filled in the lens will be towed then the lens unit will be present as a concave or convex lens. Due to different combination, the focal length is changed from the original type. Those two type soft zoom lens will be integrated in imaging system which can applied in robot vision.

12

Robot Vision

Fig. 17. The structure of successive soft zoom lens.

7. References Berge, Bruno, Peseux, Jerome, (1999). WO 99/18456, USPTO Patent Full-Text and Image Database , 1999 Berge B. ,Varioptic, Lyon, (2005). Liquid lens technology : Principle of electrowetting based lenses and applications to image, 2005 Chao Chang-Chen, Kuo-Cheng, Huang, Wei-Cheng Lin, (2007) US 7209297, USPTO Patent Full-Text and Image Database, 2007 Dow Corning, (1998). Production information-Sylgard® 184 Silicone Elastomer, 1998 David A. Chang-Yen, Richard K. Eich, and Bruce K. Gale, (2005). A monolithic PDMS waveguide system fabricated using soft-lithography techniques," Journal of lightwave technology, Vol. 23, NO. 6, June (2005) Feenstra, Bokke J., (2003). WO 03/069380, USPTO Patent Full-Text and Image Database, 2003 Huang, R. C. and Anand L., (2005). Non-linear mechanical behavior of the elastomer polydimethylsiloxane (PDMS) used in the manufacture of microfluidic devices, 2005 Jeong, Ki-Hun, Liu Gang L., (2004). Nikolas Chronis and Luke P. Lee, Turnable microdoublet lens array, Optics Express 2494, Vol. 12, No. 11, 2004 Peter M. Moran, Saman Dharmatilleke, Aik Hau Khaw, Kok Wei Tan, Mei Lin Chan, and Isabel Rodriguez, (2006). Fluidic lenses with variable focal length, Appl. Phys. Lett. 88, 2006 Rajan G. S., G. S. Sur, J. E. Mark, D. W. Schaefer, G. Beaucage, A. (2003). Preparation and application of some unusually transparent poly (dimethylsiloxane) nanocomposites, J. Polym. Sci., B, vol.41, 1897–1901 R.A. Gunasekaran, M. Agarwal, A. Singh, P. Dubasi,P. Coane, K. Varahramyan, (2005). Design and fabrication of fluid controlled dynamic optical lens system, Optics and Lasers in Engineering 43, 2005

Methods for Reliable Robot Vision with a Dioptric System



13

2 X

Methods for Reliable Robot Vision with a Dioptric System E. Martínez and A.P. del Pobil

Robotic Intelligence Lab., Jaume-I University, Castellón, Spain Interaction Science Dept., Sungkyunkwan University, Seoul, S. Korea 1. Introduction There is a growing interest in Robotics research on building robots which behave and even look like human beings. Thus, from industrial robots, which act in a restricted, controlled, well-known environment, today’s robot development is conducted as to emulate alive beings in their natural environment, that is, a real environment which might be dynamic and unknown. In the case of mimicking human being behaviors, a key issue is how to perform manipulation tasks, such as picking up or carrying objects. However, as this kind of actions implies an interaction with the environment, they should be performed in such a way that the safety of all elements present in the robot workspace at each time is guaranteed, especially when they are human beings. Although some devices have been developed to avoid collisions, such as, for instance, cages, laser fencing or visual acoustic signals, they considerably restrict the system autonomy and flexibility. Thus, with the aim of avoiding those constrains, a robot-embedded sensor might be suitable for our goal. Among the available ones, cameras are a good alternative since they are an important source of information. On the one hand, they allow a robot system to identify interest objects, that is, objects it must interact with. On the other hand, in a human-populated, everyday environment, from visual input, it is possible to build an environment representation from which a collision-free path might be generated. Nevertheless, it is not straightforward to successfully deal with this safety issue by using traditional cameras due to its limited field of view. That constrain could not be removed by combining several images captured by rotating a camera or strategically positioning a set of them, since it is necessary to establish any feature correspondence between many images at any time. This processing entails a high computational cost which makes them fail for real-time tasks. Despite combining mirrors with conventional imaging systems, known as catadioptric sensors (Svoboda et al., 1998; Wei et al., 1998; Baker & Nayar, 1999) might be an effective solution, these devices unfortunately exhibit a dead area in the centre of the image that can be an important drawback in some applications. For that reason, a dioptric system is proposed. Dioptric systems, also called fisheye cameras, are systems which combine a fisheye lens with a conventional camera (Baker & Nayar, 1998; Wood, 2006). Thus, a conventional lens is changed by one of these lenses that has a short focal length that allows cameras to see objects in an hemisphere. Although fisheye devices present several advantages over catadioptric sensors

14

Robot Vision

such as no presence of dead areas in the captured images, a unique model for this kind of cameras does not exist unlike central catadioptric ones (Geyer & Daniilidis, 2000). So, with the aim of designing a dependable, autonomous, manipulation robot system, a fast, robust vision system is presented that covers the full robot workspace. Two different stages have been considered:  moving object detection  target tracking First of all, a new robust adaptive background model has been designed. It allows the system to adapt to different unexpected changes in the scene such as sudden illumination changes, blinking of computer screens, shadows or changes induced by camera motion or sensor noise. Then, a tracking process takes place. Finally, the estimation of the distance between the system and the detected objects is computed by using an additional method. In this case, information about the 3D localization of the detected objects with respect to the system was obtained from a dioptric stereo system. Thus, the structure of this paper is as follows: the new robust adaptive background model is described in Section 2, while in Section 3 the tracking process is introduced. An epipolar geometry study of a dioptric stereo system is presented in Section 4. Some experimental results are presented in Section 5, and discussed in Section 6.

2. Moving Object Detection: A New Background Maintance Approach As it was presented in (Cervera et al., 2008), an adaptive background modelling combined with a global illumination change detection method is used to proper detect any moving object in a robot workspace and its surrounding area. That approach can be summarized as follows:  In a first phase, a background model is built. This model associates a statistical distribution, defined by its mean color value and its variance, to each pixel of the image. It is important to note that the implemented method allows to obtain the initial background model without any restrictions of bootstrapping  In a second phase, two different processing stages take place: o First, each image is processed at pixel level, in which the background model is used to classify pixels as foreground or background depending on whether they fit in with the built model or not o Second, the raw classification based on the background model is improved at frame level Moreover, when a global change in illumination occurs, it is detected at frame level and the background model is properly adapted. Thus, when a human or another moving object enters in a room where the robot is, it is detected by means of the background model at pixel level. It is possible because each pixel belonging to the moving object has an intensity value which does not fit into the background model. Then, the obtained binary image is refined by using a combination of subtraction techniques at frame level. Moreover, two consecutive morphological operations are applied to erase isolated points or lines caused by the dynamic factors mentioned above. The next step is to update the statistical model with the values of the pixels classified as background in order to adapt it to some small changes that do not represent targets.

Methods for Reliable Robot Vision with a Dioptric System

15

At the same time, a process for sudden illumination change detection is performed at frame level. This step is necessary because the model is based on intensity values and a change in illumination produces a variation of them. A new adaptive background model is built when an event of this type occurs, because if it was not done, the application would detect background pixels as if they were moving objects.

3. Tracking Process Once targets are detected by using a background maintenance model, the next step is to track each target. A widely used approach in Computer Vision to deal with this problem is the Kalman Filter (Kalman & Bucy, 1961; Bar-Shalom & Li, 1998; Haykin, 2001; Zarchan & Mussof, 2005; Grewal et al., 2008). It is an efficient recursive filter that has two distinct phases: Predict and Update. The predict phase uses the state estimate from the previous timestep to produce an estimate of the state at the current timestep. This predicted state estimate is also known as the a priori state estimate because, although it is an estimate of the state at the current timestep, it does not include observation information from the current timestep. In the update phase, the current a priori prediction is combined with current observation information to refine the state estimate. This improved estimate is termed the a posteriori state estimate. With regard to the current observation information, it is obtained by means of an image correspondence approach. In that sense, one of most well-known methods is the Scale Invariant Feature Transform (SIFT) approach (Lowe, 1999; Lowe, 2004) which shares many features with neuron responses in primate vision. Basically, it is a 4stage filtering approach that provides a feature description of an object. That feature array allows a system to locate a target in an image containing many other objects. Thus, after calculating feature vectors, known as SIFT keys, a nearest-neighbor approach is used to identify possible objects in an image. Moreover, that array of features is not affected by many of the complications experienced in other methods such as object scaling and/or rotation. However, some disadvantages made us discard SIFT for our purpose:  It uses a varying number of features to describe an image and sometimes it might be not enough  Detecting substantial levels of occlusion requires a large number of SIFT keys what can result in a high computational cost  Large collections of keys can be space-consuming when many targets have to be tracked  It was designed for perspective cameras, not for fisheye ones All these approaches have been developed for perspective cameras. Although some research has been carried out to adapt them to omnidirectional devices (Fiala, 2005; Tamimi et al., 2006), a common solution is to apply a transformation to the omnidirectional image in order to obtain a panoramic one and to be able to use a traditional approach (Cielniak et al., 2003; Liu et al., 2003; Potúcek, 2003; Zhu et. al (2004); Mauthner et al. (2006); Puig et al., 2008). However, this might give rise to a high computational cost and/or mismatching errors. For all those reasons, we propose a new method which is composed of three different steps (see Fig. 1): 1. The minimum bounding rectangle that encloses each detected target is computed 2. Each area described by a minimum rectangle identifying a target, is transformed into a perspective image. For that, the following transformation is used:

16

Robot Vision

3.

(1)  = c/Rmed (2) Rmed = (Rmin + Rmax)/2 (3) c1 = c0 + (Rmin + r) sin  (4) r1 = r0 + (Rmin + r) cos  where (c0, r0) represent the image center coordinates in terms of column and row respectively, while (c1, r1) are the coordinates in the new perspective image;  is the angle between Y-axis and the ray from the image center to the considered pixel; and, Rmin and Rmax are, respectively, the minimum and maximum radius of a torus which encloses the area to be transformed. These radii are obtained from the four corners of the minimum rectangle that encloses the target to be transformed. However, there is a special situation to be taken into account. It is produced when the center of the image is in the minimum rectangle. This situation, once it is detected, is solved by setting Rmin to 0. On the other hand, note that only the detected targets are transformed into its cylindrical panoramic image, not the whole image. It allows the system to reduce the computational cost and time consumption. Moreover, the orientation problem is also solved, since all the resulting cylindrical panoramic images have the same orientation. In this way, it is easier to compare two different images of the same interest object The cylindrical panoramic images obtained for the detected targets are compared with the ones obtained in the previous frame. A similarity likelihood criterion is used for matching different images of the same interest object. Moreover, in order to reduce computational cost, images to be compared must have a centroid distance lower than a threshold. This distance is measured in the fisheye image and is described by means of a circle of possible situations of the target having as center its current position and as radius the maximum distance allowed. In that way, the occlusion problem is solved, since all parts of the cylindrical panoramic images are being analyzed.

(a) Original Image

(b) Image from Segmentation Process

Fig. 1. Tracking Process

(c) Minimum rectangle that encloses the target

(d) Corresponding cylindrical panoramic image

(e) Matching process

This method is valid for matching targets in a monocular video sequence. Thus, when a dioptric stereo system is provided, the processing is changed. When an interest object firstly appears in an image, it is necessary to establish its corresponding image in the other camera of the stereo pair. For that, a disparity estimation approach is used. That estimation is carried out until the interest object is in the field of view of both stereo cameras. In that moment, the same identifier is assigned to both images of the interest object. Then, matching is done as a monocular system in a parallel way, but as both images have the same identifier it is possible to estimate its distance from the system whenever it is necessary. As it is performed in a parallel way and processing is not time-consuming, a real-time performance is obtained.

Methods for Reliable Robot Vision with a Dioptric System

17

4. Experimental Results For the experiments carried out, a mobile manipulator was used which incorporates a visual system composed of 2 fisheye cameras mounted on the robot base, pointing upwards to the ceiling, to guarantee the safety in its whole workspace. Fig. 2 depicts our experimental setup, which consists of a mobile Nomadic XR4000 base, a Mitsubishi PA10 arm, and two fisheye cameras (Sony SSC-DC330P 1/3-inch color cameras with fish-eye vari-focal lenses Fujinon YV2.2x1.4A-2, which provide 185-degree field of view). Images to be processed were acquired in 24-bit RGB color space with a 640x480 resolution.

Fisheye Vari-focal Lens

24-bit RGB, 640x480 images

Fig. 2. Experimental Set-up Different experiments were carried out obtaining a good performance in all of them. Some of the experimental results are depicted in Fig. 3. where different parts of the whole process are shown. The first column represents the image captured by a fisheye camera, then the binary image generated by the proposed approach appears in the next column. The three remaining columns represent several phases of the tracking process, that is, the generation of the minimum bounding rectangle and the cylindrical panoramic images. Note that in all cases, both the detection approach and the proposed tracking process were successful in their purpose, although some occlusions, rotation and/or scaling had occurred.

5. Conclusions & Future Work We have presented a dioptric system for reliable robot vision by focusing on the tracking process. Dioptric cameras have the clear advantage of covering the whole workspace without resulting in a time consuming application, but there is little previous work about this kind of devices. Consequently, we had to implement novel techniques to achieve our goal. Thus, on the one hand, a process to detect moving objects within the observed scene was designed. The proposed adaptive background modeling approach combines moving object detection with global illumination change identification. It is composed of two different phases, which consider several factors which may affect the detection process, so that constraints in illumination conditions do not exist, and neither is it necessary to wait for some time for collecting enough data before starting to process.

18

Fig. 3. Some Experimental Results

Robot Vision

Methods for Reliable Robot Vision with a Dioptric System

19

On the other hand, a new matching process has been proposed. Basically, it obtains a panoramic cylindrical image for each detected target from an image in which identified targets where enclosed in the minimum bounding rectangle. In this way, the rotation problem has disappeared. Next, each panoramic cylindrical image is compared with all the ones obtained in the previous frame whenever they are in its proximity. As all previous panoramic cylindrical images are used, the translation problem is eliminated. Finally, a similarity likelihood criterion is used for matching target images at two different times. With this method the occlusion problem is also solved. As time consumption is a critical issue in robotics, when a dioptric system is used, this processing is performed in a parallel way such that correspondence between two different panoramic cylindrical images of the same target taken at the same time for both cameras is established by means of a disparity map. Therefore, a complete real-time surveillance system has been presented.

6. Acknowledges This research was partly supported by WCU (World Class University) program through the National Research Foundation of Korea funded by the Ministry of Education, Science and Technology (Grant No. R31-2008-000-10062-0), by the European Commission’s Seventh Framework Programme FP7/2007-2013 under grant agreement 217077 (EYESHOTS project), by Ministerio de Ciencia e Innovación (DPI-2008-06636), by Generalitat Valenciana (PROMETEO/2009/052) and by Fundació Caixa Castelló-Bancaixa (P1-1B2008-51).

7. References Baker, S. & Nayar, S.K. (1998). A Theory of Catadioptric Image Formation. Proceedings of International Conference on Computer Vision (ICCV), pp. 35–42, 1998, Bombay, India Baker, S. & Nayar, S.K. (1999). A Theory of Single-Viewpoint Catadioptric Image Formation. International Journal on Computer Vision, Vol. 2, No. 2, (1999), page numbers (175196) Bar-Shalom, Y. and Li X-R (1998). Estimation and Tracking: Principles, Techniques and Software, YBS Publishing, ISBN: 096483121X Cervera, E.; Garcia-Aracil, N.; Martínez, E.; Nomdedeu, L. & del Pobil, A. Safety for a Robot Arm Moving Amidst Humans by Using Panoramic Vision. Proceedings of IEEE International Conference on Robotics & Automation (ICRA), pp. 2183–2188, ISBN: 9784244-1646-2, May 2008, Pasadena, California Cielniak, G.; Miladinovic, M.; Hammarin, D. & Göranson, L. (2003). Appearance-based Tracking of Persons with an Omnidirectional Vision Sensor. Proceedings of the 4th IEEE Workshop on Omnidirectional Vision (OmniVis), 2003, Madison, Wisconsin, USA Fiala, M. (2005). Structure from Motion using SIFT Features and the PH Transform with Panoramic Imagery. Proceedings of the 2nd Canadian Conference on Computer and Robot Vision (CRV), pp. 506–513, 2005 Geyer, C. & Daniilidis, K. (2000). A Unifying Theory for Central Panoramic Systems and Practical Applications. Proceedings of 6th European Conference on Computer Vision (ECCV), pp. 445–461, ISBN: 3-540-67686-4, June 2000, Dublin, Ireland Grewal, M., Andrews, S. and Angus, P. (2008) Kalman Filtering Theory and Practice Using MATLAB, Wiley-IEEE Press, ISBN: 978-0-470-17366-4

20

Robot Vision

Haykin, S. (2001). Kalman Filtering and Neural Networks. John Wiley & Sons, Inc. ISBN: 978-0471-36998-1 Kalman, R.E. and Bucy, R.S. (1961). New Results in Linear Filtering and Prediction Theory. Journal of Basic Engineering (1961) Liu, H.; Wenkai, P.I. & Zha, H. (2003) Motion Detection for Multiple Moving Targets by Using an Omnidirectional Camera. Proceedings of IEEE International Conference on Robotics, Intelligent Systems and Signal Processing, pp. 422–426, October 2003, Chansgsha, China Lowe, D.G. (1999). Object recognition from local scale-invariant features, Proceedings of International Conference on Computer Vision, pp. 1150–1157, September 1999, Corfu, Greece Lowe, D.G. (2004). Distinctive image features from Scale-Invariant Keypoints. International Journal of Computer Vision, Vol. 60, No. 2, (2004), page numbers (91-110) Mauthner, T.; Fraundorfer, F. & Bischof, H. (2006). Region Matching for Omnidirectional Images Using Virtual Camera Planes. Proceedings of Computer Vision Winter Workshop, February 2006, Czech Republic Murillo, A.C.; Sagüés, C.; Guerrero, J.J.; Goedemé, T.; Tuytelaars, T. and Van Gool, L. (2006). From Omnidirectional Images to Hierarchical Localization. Robotics and Autonomous Systems, Vol. 55, No. 5, (December 2006), page numbers (372 – 382) Potúcek, I. (2003). Person Tracking using Omnidirectional View. Proceedings of the 9th International Student EEICT Conference and Competition, pp. 603–607, ISBN: 80-2142370, 2003, Brno, Czech Republic Puig, L.; Guerrero, J. & Sturm, P. (2008). Matching of Omnidirectional and Perspectiva Images using the Hybrid Fundamental Matrix. Proceedings of the Workshop on Omnidirectional Vision, Camera Networks and Non-Classical Cameras, October 2008, Marseille, France Svoboda, T.; Pajdla, T. & Hlavác, V. (1998). Epipolar Geometry for Panoramic Cameras. Proceedings of the 5th European Conference on Computer Vision (ECCV), pp. 218–231, ISBN: 3-540-64569-1, June 1998, Freiburg, Germany Tamimi, H.; Andreasson, H.; Treptow, A.; Duckett, T. & Zell, A. (2006). Localization of Mobile Robots with Omnidirectional Vision using Particle Filter and Iterative SIFT. Robotics and Autonomous Systems, Vol. 54 (2006), page numbers (758-765) Wei, S.C.; Yagi, Y. & Yachida, M. (1998). Building Local Floor Map by Use of Ultrasonic and Omni-directional Vision Sensor. Proceedings of IEEE International Conference on Robotics & Automation (ICRA), pp. 2548–2553, ISBN: 0-7803-4300-X, May 1998, Leuven, Belgium Wood, R.W. (1906). Fish-eye Views. Philosophical Magazine, Vol. 12, No. 6, (1906), page numbers (159–162) Zarchan, P. and Mussof, H. (2005) Fundamentals of Kalman Filtering: A Practical Approach, American Institute of Aeronautics & Ast (AIAA), ISBN: 978-1563474552 Zhu, Z.; Karuppiah, D.R.; Riseman, E.M. & Hanson, A.R. (2004). Keeping Smart, Omnidirectional Eyes on You [Adaptive Panoramic StereoVision]. IEEE Robotics and Automation Magazine – Special Issue on Panoramic Robotics, Vol. 11, No. 4, (December 2004), page numbers (69–78), ISSN: 1070-9932

An Approach for Optimal Design of Robot Vision Systems

21

3 0 An Approach for Optimal Design of Robot Vision Systems Kanglin Xu New Mexico Tech USA

1. Introduction The accuracy of a robot manipulator’s position in an application environment is dependent on the manufacturing accuracy and the control accuracy. Unfortunately, there always exist both manufacturing error and control error. Calibration is an approach to identifying the accurate geometry of the robot. In general, robots must be calibrated to improve their accuracy. A calibrated robot has a higher absolute positioning accuracy. However, calibration involves robot kinematic modeling, pose measurement, parameter identification and accuracy compensation. These calibrations are hard work and time consuming. For an active vision system, a robot device for controlling the motion of cameras based on visual information, the kinematic calibrations are even more difficult. As a result, even though calibration is fundamental, most existing active vision systems are not accurately calibrated (Shih et al., 1998). To address this problem, many researchers select self-calibration techniques. In this article, we apply a more active approach, that is, we reduce the kinematic errors at the design stage instead of at the calibration stage. Furthermore, we combine the model described in this article with a costtolerance model to implement an optimal design for active vision systems so that they can be used more widely in enterprise. We begin to build the model using the relation between two connecting joint coordinates defined by a DH homogeneous transformation.We then use the differential relationship between these two connecting joint coordinates to extend the model so that it relates the kinematic parameter errors of each link to the pose error of the last link. Given this model, we can implement an algorithm for estimating depth using stereo cameras, extending the model to handle an active stereo vision system. Based on these two models, we have developed a set of C++ class libraries. Using this set of libraries, we can estimate robot pose errors or depth estimation errors based on kinematic errors. Furthermore, we can apply these libraries to find the key factors that affect accuracy. As a result, more reasonable minimum tolerances or manufacturing requirements can be defined so that the manufacturing cost is reduced while retaining relatively high accuracy. Besides providing an approach to find the key factors and best settings of key parameters, we demonstrate how to use a cost-tolerance model to evaluate the settings. In this way, we can implement optimal design for manufacturing(DFM) in enterprises. Because our models are derived from the Denavit-Hartenberg transformation matrix, differential changes for the transformation matrix and link parameters, and the fundamental algorithm for estimating depth using stereo cameras, they are suitable for any manipulator or stereo active vision system. The remainder of this article is organized as follows. Section 2 derives the model for analyzing the effect of parameter errors on robot

22

Robot Vision

poses. Section 3 introduces the extended kinematic error model for an active vision system. It should be noted that this extended model is the main contribution of our article and that we integrate the robot differential kinematics into an active vision system. Section 4 provides more detailed steps describing how to use our approach. Section 5 discusses some issues related to the design of active vision systems and DFM. Section 6 presents a case study for a real active vision system and cost evaluation using a cost-tolerance model. Finally, Section 7 offers concluding remarks.

2. Kinematic Differential Model Derived from DH Transformation Matrix A serial link manipulator consists of a sequence of links connected together by actuated joints (Paul, 1981). The kinematical relationship between any two successive actuated joints is defined by the DH (Denavit-Hartenberg) homogeneous transformation matrix. The DH homogeneous transformation matrix is dependent on the four link parameters, that is, θi , αi , ri , and di . For the generic robot forward kinematics, only one of these four parameters is variable. If joint i is rotational, the θi is the joint variable and di , αi , and ri are constants. If joint i is translational, the di is the joint variable and θ, αi , and ri are constants. Since there always exist errors for these four parameters, we also need a differential relationship between any two successive actuated joints. This relationship is defined by matrix dAi which is dependent on dθi , dαi , dri , and ddi as well as θi , αi , ri , and di . Given the relationship between two successive joints Ai and differential relationship between two successive joints dAi , we can derive an equation to calculate the accurate position and orientation of the end-effector with respect to the world coordinate system for a manipulator with N degrees of freedom(N-DOF). In this section, we will first derive the differential changes between two successive frames in subsection 2.1. We then give the error model for a manipulator of N degrees of freedom with respect to the world coordinate system in subsection 2.2. 2.1 The Error Relation between Two Frames

For an N-DOF manipulator described by the Denavit-Hartenberg definition, the homogeneous transformation matrix Ai which relates the (i-1)th joint to ith joint is (Paul, 1981)   cθi −sθi cαi sθi sαi ri cθi  sθi cθi cαi −cθi sαi ri sθi   Ai =  (1)  0 sαi cαi di  0 0 0 1 where s and c refer to sine and cosine functions, and θi , αi , ri , and di are link parameters. Given the individual transformation matrix Ai , the end of an N-DOF manipulator can be represented as T N = A1 A2 · · · A N −1 A N (2) We will also use the following definitions. We define Ui = Ai Ai+1 · · · A N with UN +1 = I, and a homogeneous matrix   ni oi ai pi Ai = (3) 0 0 0 1 where ni , oi , ai and pi are 3 × 1 vectors. Given the ith actual coordinate frame Ai and the ith nominal frame A0i , we can obtain an additive differential transformation dAi dAi = Ai − A0i .

(4)

An Approach for Optimal Design of Robot Vision Systems

23

If we represent the ith additive differential transformation dAi as the ith differential transformation δAi right multiplying the transformation Ai , we can write dAi = Ai δAi .

(5)

In this case, the changes are with respect to coordinate frame Ai . Assuming the link parameters are continuous and differentiable we can represent dAi in another way, that is ∂Ai ∂Ai ∂Ai ∂Ai dAi = dθ + dα + dr + dd . (6) ∂θi i ∂αi i ∂ri i ∂di i Comparing (5) with (6), we obtain δAi

= +

 ∂Ai ∂Ai dθ + dα ∂θi i ∂αi i  ∂Ai ∂Ai dr + dd . ∂ri i ∂di i

A −1

For the homogeneous matrix, the inverse matrix of Ai is  nit − pi · ni t  − pi · oi o i Ai−1 =   at − pi · ai i 01×3 1

(7)

   

(8)

By differentiating all the elements of equation (1) with respect to θi , αi , ri and di respectively, we obtain   −sθi −cθi cαi cθi sαi −ri sθi  cθi ∂Ai −sθi cαi sθi cαi ri cθi   (9) =   0 0 0 0 ∂θi 0 0 0 0   0 sθi sαi sθi cαi 0  0 −cθi sαi −cθi cαi 0  ∂Ai  (10) =  0 cαi −sαi 0  ∂αi 0 0 0 0   0 0 0 cθi  0 0 0 sθi  ∂Ai   (11) = 0 0 0 0  ∂ri 0 0 0 0   0 0 0 0  0 0 0 0  ∂Ai  (12) =  0 0 0 1  ∂di 0 0 0 0 Substituting equations (8), (9), (10), (11) and (12) into (7), we obtain  0 −cαi dθi sαi dθi dri  cαi dθi 0 −dαi u  δAi =  −sαi dθi dαi 0 v 0 0 0 0

   

(13)

24

Robot Vision

where u = ri cαi dθi + sαi ddi and v = ri sαi dθi + cαi ddi . Since dAi = Ai δAi , therefore (Paul, 1981)     dxi ddi  dyi  =  di cα dθi + sαi dri  (14) dzi −di sα dθi + cαi dri     δx i dαi  δyi  =  sα dθi  (15) δzi cα dθi  t where (dxi dyi dzi )t is the differential translation vector and δxi δyi δzi is the differential rotation vector with respect to frame Ai . Let di = (dxi dyi dzi )t and δi = (δxi δyi δzi )t . The differential vectors di and δi can be represented as a linear combination of the parameter changes, which are di = k1i dθi + k2i ddi + k3i dri

(16)

δi = k2i dθi + k3i dαi

(17)

and

where k1i = (0 ri cαi − ri sαi )t , k2i = (0 sαi cαi )t and k3i = (1 0 0)t . 2.2 Kinematic Differential Model of a Manipulator

Wu (Wu, 1984) has developed a kinematic error model of an N-DOF robot manipulator based on the differential changes dAi and the error matrix δAi due to four small kinematic errors at an individual joint coordinate frame. Let d N and δ N denote the three translation errors and rotation errors of the end of a manipulator respectively where d N = (dx N dy N dz N ) and δ N = (δxN δyN δzN ). From (Wu, 1984), we obtain 

dN δN



=



+



M1 M2



M3 0



dθ +



dd +



M2 0



dr

M4 M3



dα

(18)

where dθ dr dd dα

= = = =

(dθ1 dθ2 . . . dθ N )t (dr1 dr2 . . . dr N )t (dd1 dd2 . . . dd N )t (dα1 dα2 . . . dα N )t

and M1 , M2 , M3 and M4 are all 3 × N matrices whose components are the function of N joint variables. The ith column of M1 , M2 , M3 and M4 can be expressed as follows: M1i

=

 nun+1 · k1i + (piu+1 × niu+1 ) · k2i  o u · k1 + ( p u × o u ) · k2  i i i +1 i +1 i +1 aiu+1 · k1i + (piu+1 × aiu+1 ) · k2i 

An Approach for Optimal Design of Robot Vision Systems

 nun+1 · k2i 2 u  o  i +1 · k i aiu+1 · k2i

=



M3i

=



=



M2i

M4i

25

 nun+1 · k3i 3 u  o  i +1 · k i aiu+1 · k3i

 (piu+1 × niu+1 ) · k3i  ( p u × o u ) · k3  i +1 i +1 i (piu+1 × aiu+1 ) · k3i

where niu+1 , oiu+1 , aiu+1 and piu+1 are four 3 × 1 vectors of matrix Ui+1 which is defined as Ui = Ai Ai+1 · · · A N with U N +1 = I. Using the above equations, the manipulator’s differential changes with respect to the base can be represented as   dn do da dp dT N = (19) 0 0 0 1 where dn

=

o1u δzN − a1u δyN

do

=

−n1u δzN + a1u δxN

da

=

n1u δyN − o1u δxN

dp

=

n1u dx N + o1u dy N + a1u dz N

and n1u , o1u , a1u are four 3 × 1 vectors of matrix U1 . Finally, the real position and orientation at the end of the manipulator can be calculated by TR N = T N + dT N

(20)

where T N = A1 A2 · · · A N .

3. Extended Model for an Active Visual System An active vision system, which has become an increasingly important research topic, is a robot device for controlling the motion of cameras based on visual information. The primary advantage of directed vision is its ability to use camera redirection to look at widely separated areas of interest at fairly high resolution instead of using a single sensor or array of cameras to cover the entire visual field with uniform resolution. It is able to interact with the environment actively by altering its viewpoint rather than observing it passively. Like an end effector, a camera can also be connected by a fixed homogeneous transformation to the last link. In addition, the structure and mechanism are similar to those of robots. Since an active visual system can kinematically be handled like a manipulator of N degrees of freedom, we can use the derived solutions in the last section directly. In this section, we will first introduce the camera coordinate system corresponding to the standard coordinate system definition for the approaches used in the computer vision literature and describe a generic algorithm for location estimation with stereo cameras. We will then integrate them with the kinematic error model of a manipulator.

26

Robot Vision

P = (x,y,z)

Z Y

Object

Image plane

(u,v)

X

f

Fig. 1. The camera coordinate system whose x- and y-axes form a basis for the image plane, whose z-axis is perpendicular to the image plane (along the optical axis), and whose origin is located at distance f behind the image plane, where f is the focal length of the camera lens.

3.1 Pose Estimation with Stereo Cameras

We assign the camera coordinate system with x- and y-axes forming a basis for the image plane, the z-axis perpendicular to the image plane (along the optical axis), and with its origin located at distance f behind the image plane, where f is the focal length of the camera lens. This is illustrated in Fig. 1. A point, c p = ( x, y, z)t whose coordinates are expressed with respect to the camera coordinate frame C, will project onto the image plane with coordinates pi = (u, v)t given by     f u x π ( x, y, z) = = (21) v y z If the coordinates of x p are expressed relative to coordinate frame X, we must first perform the coordinate transformation c p = c T x x p. (22) Let a Tc1 represent the pose of the first camera relative to the arbitrary reference coordinate frame A. By inverting this transformation, we can obtain c1 T a , where  c1  r11 c1 r12 c1 r13 c1 t x c1 r c1 t  c1 r c1 r  y  c1 22 23 21 Ta =  (23)  c1 r31 c1 r32 c1 r33 c1 tz  0 0 0 1 For convenience, let c1

R1

c1

R2

c1

R1

= = =

(c1 r11 (c1 r21 (c1 r31

c1

r12

c1

c1

r22

c1

r23 )

(25)

c1

r32

c1

r33 )

(26)

r13 )

(24)

An Approach for Optimal Design of Robot Vision Systems

27

(a)

(b)

Fig. 2. The TRICLOPS Active Vision System has four axes. They are pan axis, tilt axis, left vergence axis and right vergence axis. The pan axis can rotate around a vertical axis through the center of the base. The tilt axis can rotate around a horizontal line that intersects the base rotation axis. The left and right vergence axes intersect and are perpendicular to the tilt axis. These two pictures come from Wavering, Schneiderman, and Fiala (Wavering et al.). We would like to thank to the paper’s authors.

Given the image coordinates, Hutchinson, Hager and Corke (Hutchinson et al., 1996) have presented an approach to find the object location with respect to the frame A using the following equations: A1 · a p = b1 (27) where A1

b1

=

=

c1 R



f1 f1



u1 v1

− u1 c1 R − v 2 1 1

c1 t

− f1 c1 t − f z 1 z

c1 R c1 R

c1 t c1 t

x y

Given a second camera at location a Xc2 , we can compute we have an over-determined system for finding a p     A1 b1 a p= b2 A2

3

3



(28)

 c2 X

(29) a,

A2 and a p similarly. Finally

(30)

where A1 and b1 are defined by (28) and (29) while A2 and b2 are defined as follows:   f 2 c2 R1 − u2 c2 R3 A2 = f 2 c2 R2 − v2 c2 R3

b2

=



u2 v2

c2 t

− f2 c2 t − f z 2 z

c2 t c2 t

x y



.

(31)

(32)

28

Robot Vision

3.2 Pose Estimation Based on an Active Vision System

As mentioned before, assuming that the camera frames are assigned as shown in Fig. 1 and that the projective geometry of the camera is modeled by perspective projection, a point c p = (c x c y c z)t , whose coordinates are expressed with respect to the camera coordinate frame will project onto the image plane with coordinates (u v)t