Sensing Technologies for Field and In-House Crop Production: Technology Review and Case Studies (Smart Agriculture, 7) 9819979269, 9789819979264

Citation preview

Smart Agriculture 7

Man Zhang · Han Li · Wenyi Sheng · Ruicheng Qiu · Zhao Zhang   Editors

Sensing Technologies for Field and In-House Crop Production Technology Review and Case Studies

Smart Agriculture Volume 7

Series Editors Zhao Zhang, Key Laboratory of Smart Agriculture System Integration Research, Ministry of Education, China Agricultural University, Beijing, China Yiannis Ampatzidis, UF/IFAS Southwest Florida Research and Education Center, University of Florida, Immokalee, FL, USA Paulo Flores, Department of Agricultural and Biosystems Engineering, North Dakota State University, Fargo, ND, USA Yuanjie Wang, Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing, China

The book series Smart Agriculture presents progress of smart agricultural technologies, which includes, but not limited to, specialty crop harvest robotics, UAV technologies for row crops, innovative IoT applications in plant factories, and big data for optimizing production process. It includes both theoretical study and practical applications, with emphasis on systematic studies. AI technologies in agricultural productions will be emphasized, consisting of innovative algorithms and new application domains. Additionally, new crops are emerging, such as hemp in U.S., and covered as well. This book series would cover regions worldwide, such as U.S., Canada, China, Japan, Korea, and Brazil. The book series Smart Agriculture aims to provide an academic platform for interdisciplinary researchers to provide their state-of-the-art technologies related to smart agriculture. Researchers of different academic backgrounds are encouraged to contribute to the book, such as agriculture engineers, breeders, horticulturist, agronomist, and plant pathologists. The series would target a very broad audience – all having a professional related to agriculture production. It also could be used as textbooks for graduate students.

Man Zhang · Han Li · Wenyi Sheng · Ruicheng Qiu · Zhao Zhang Editors

Sensing Technologies for Field and In-House Crop Production Technology Review and Case Studies

Editors Man Zhang Key Laboratory of Smart Agriculture Systems, Ministry of Education China Agricultural University Beijing, China

Han Li Key Laboratory of Smart Agriculture Systems, Ministry of Education China Agricultural University Beijing, China

Wenyi Sheng Key Laboratory of Smart Agriculture Systems, Ministry of Education China Agricultural University Beijing, China

Ruicheng Qiu Key Laboratory of Smart Agriculture Systems, Ministry of Education China Agricultural University Beijing, China

Zhao Zhang Key Laboratory of Smart Agriculture Systems, Ministry of Education China Agricultural University Beijing, China

ISSN 2731-3476 ISSN 2731-3484 (electronic) Smart Agriculture ISBN 978-981-99-7926-4 ISBN 978-981-99-7927-1 (eBook) https://doi.org/10.1007/978-981-99-7927-1 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore Paper in this product is recyclable.

Preface

Agricultural production encompasses both field and in-house cultivation. Achieving optimal agricultural practices demands a wealth of information for effective management, including monitoring crop growth conditions and soil nutrient levels, tasks predominantly carried out by manual labor. Escalating labor costs, a diminishing workforce, subjective human judgment, and inefficiencies have imposed significant challenges on agricultural production. Consequently, the need for the development of novel sensing technologies for both field and in-house crop production has come into sharp focus, and it is this research area that serves as the central theme of this book. Soil nutrient conditions play a pivotal role in crop production, yet existing methods involve manual soil sampling and laboratory testing, rendering them offline and inefficient. The opening chapter delves into emerging techniques, such as visible light, near-infrared, and laser-induced breakdown spectroscopy, for real-time soil nutrient monitoring. Meanwhile, in the context of crop growth assessment, multispectral imaging and robotic systems are steadily supplanting human labor, automating the evaluation process—a subject comprehensively explored in the second, third, and fourth chapters. The initial four chapters serve as an encompassing overview of state-of-the-art technologies for field and in-house crop production. The subsequent four chapters delve into specialized studies. For instance, Chap. 5 introduces an innovative deep learning-based method called “SeedlingsNet” for automatic wheat seedling density detection—an invaluable resource for wheat production. Addressing the lodging issue, Chap. 6 focuses on machine vision-based approaches, offering an in-depth exploration of this critical topic. Chapters 6 and 7 both concentrate on crop field production, while the final two chapters pivot toward post-harvest technologies, specifically pertaining to automatic seed germination testing. Chapter 7 details the development of a system that employs sensing technologies to optimize machine working parameters. Finally, Chap. 8 delves into the intricacies of an automated seedling detection system utilizing machine vision.

v

vi

Preface

The editors of this book, Drs. Man Zhang, Han Li, Wenyi Sheng, Ruicheng Qiu, and Zhao Zhang, are internationally recognized experts in the field of sensing technologies for field and in-house crop production. Their collaboration has brought together a group of accomplished and knowledgeable authors who have contributed to the eight chapters of this book. These chapters comprehensively cover essential sensing technologies, presented in a systematic and logical arrangement. As a result, this book offers a timely and comprehensive source of information for readers interested in delving into the vital subject of sensing technologies for field and in-house crop production. Beijing, China

Man Zhang Zhao Zhang

Contents

1 A Review of Three-Dimensional Multispectral Imaging in Plant Phenotyping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tao Xia, Wenyi Sheng, Runze Song, Han Li, and Man Zhang

1

2 Recent Advances in Soil Nutrient Monitoring: A Review . . . . . . . . . . . Qianying Yuan, Wenyi Sheng, Zhao Zhang, Han Li, and Man Zhang

19

3 Plant Phenotyping Robot Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuandan Yang, Han Li, Man Zhang, and Ruicheng Qiu

39

4 Autonomous Crop Image Acquisition System Based on ROS System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yida Li, Han Li, Liuyang Wang, and Man Zhang

53

5 SeedingsNet: Field Wheat Seedling Density Detection Based on Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yunxia Li, Zuochen Jiang, Zhao Zhang, Han Li, and Man Zhang

77

6 Wheat Lodging Detection Using Smart Vision-Based Method . . . . . . . Afshin Azizi, Tianjing Yi, Zhao Zhang, Han Li, and Man Zhang

89

7 Design, Construction, and Experiment-Based Key Parameter Determination of Auto Maize Seed Placement System . . . . . . . . . . . . . . 103 Xinyu Wei and Zhao Zhang 8 Development and Test of an Auto Seedling Detection System . . . . . . . 121 Jiaying Liu and Zhao Zhang

vii

Chapter 1

A Review of Three-Dimensional Multispectral Imaging in Plant Phenotyping Tao Xia, Wenyi Sheng, Runze Song, Han Li, and Man Zhang

Abstract Plant phenotype refers to the external characteristics and traits which can be influenced by the interaction between plant genotype and environment, thus reflecting the adaptability and productivity of plants to the environment. As the two critical technical means in plant phenotyping, three-dimensional imaging could provide accurate information about the plant shape, structure, and spatial distribution non-destructively through laser scanning or camera shooting, while multispectral imaging captures plant images in multiple spectral bands to analyze various characteristics of plants. By combining three-dimensional and multispectral imaging, researchers can better understand plant growth and development. In this review, we discussed various techniques and instruments used for three-dimensional multispectral imaging. Specifically, we focused on the instrumentations, their system configuration, and their operational workflows. Furthermore, this survey also explored the key parameters associated with these sophisticated instruments, including spectral bands, computable plant phenotyping parameters, applicable range, and the diverse spectrum of objects they are capable of detecting. In addition, we engaged with the challenges that lie ahead and the promising avenues for future research in the realm of three-dimensional multispectral imaging. This review underscores its remarkable potential in advancing the field of plant phenotyping, thereby revolutionizing crop breeding and management practices. Keywords Plant phenotyping · Three-dimensional and multispectral imaging · Plant growth and development

T. Xia · W. Sheng (B) · R. Song · H. Li · M. Zhang College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China e-mail: [email protected] Key Laboratory of Smart Agriculture System Integration, Ministry of Education, China Agricultural University, Beijing 100083, China © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 M. Zhang et al. (eds.), Sensing Technologies for Field and In-House Crop Production, Smart Agriculture 7, https://doi.org/10.1007/978-981-99-7927-1_1

1

2

T. Xia et al.

1.1 Introduction Plant phenotype generally refers to plants’ external characteristics and traits [13]. Nowadays, the concept of plant phenotype has expanded beyond the physical state of plants [8, 9] and includes fields such as biochemistry [2, 12]. In the past few decades, phenotyping has become an essential tool for characterizing many plants’ physiological processes, functions, and structures [27]. The commonly studied plant phenotypic traits encompass structural, physiological, and component content traits. Structural traits comprise parameters such as plant height, leaf length, leaf width, leaf area, leaf color, flower number, root length, root width, root biomass, and fruit size. For example, the leaves of coniferous trees are slender, which can reduce water evaporation and help the plants survive in a dry environment. In contrast, the leaves of broad-leaved trees are more expansive, which can absorb more sunlight and help the plants survive in a sunny environment. Physiological traits a variety of characteristics, including photosynthetic efficiency, respiration, transpiration, stomatal conductance, stress resistance, and growth rate. For example, photosynthetic efficiency affects the growth and development of plants. High photosynthetic efficiency can help plants utilize light energy more effectively, improving their growth rate and yield. Component content traits include pigment content, sugar content, protein content, etc. For instance, a high pigment concentration may boost photosynthetic effectiveness even further. Plant pigments can absorb light of different wavelengths and convert it into chemical energy for photosynthesis. The commonly studied plant phenotypic traits are shown in Fig. 1.1.

Fig. 1.1 The commonly studied plant phenotypic traits

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

3

Plant phenotypes result from the interaction between genotype and environment [23, 30, 32, 38]. Plant phenotyping is the process of measuring plants. Standard imaging methods for plant phenotyping includes red–green–blue (RGB), three-dimensional laser, and fluorescence imaging [30]. The data produced from RGB photographs is frequently utilized, and RGB imaging is the method for plant grading that is the most intuitive. Laser scanning in plant measurements typically uses laser triangulation technology to measure the geometric shape of the plant, which is considered the most advanced technology for representing accuracy. Contrary to techniques that rely on reflected light; fluorescence imaging picks up light released by certain fluorescent compounds or chemical elements that change a portion of the received light into re-emitted fluorescence. This imaging technology is ideal for plant stress detection, especially for symptoms before environmental stress or microbial-induced disease. With the rapid development of technology, two critical technical means have emerged in plant phenotype identification. Three-dimensional imaging can provide accurate information about plant shape, structure, and spatial distribution non-destructively through techniques such as laser scanning and camera imaging [22]. The three-dimensional imaging method for describing spatial information is widely used in plant phenotype analysis [40]. Multispectral imaging captures plant images in multiple spectral bands to analyze plant characteristics, such as chlorophyll content, water stress, and nutrient status (Wang et al. 2001). Combining three-dimensional and multispectral imaging allows the responses to be obtained simultaneously, and the resulting three-dimensional multispectral point cloud data is promising multimodal data [34]. Academics may use this information to acquire a more thorough plant growth and development knowledge. Plant phenotype can also be measured by considering the entire plant (whole-plant phenotype) or its organs (component phenotype). Jimenez et al. (2000) proposed a fruit harvesting system that detects fruit based on the plant’s color and morphological characteristics, in which laser scanning technology was one of the earliest methods applied to plant phenotype analysis. Paulus et al. [28, 29] used histogram analysis of three-dimensional point cloud data based on surface features to perform organ segmentation of wheat, grape, and barley plants. Deep learning techniques are the primary image analysis approaches in plant phenotypic assessment. Scholars have employed these techniques for various tasks, including calculating plant stem counts and widths [1], leaf numbers in rose plants [10], and flower detection in cotton [48]. Numerous investigations at the microscale of plant tissues have been conducted recently using increasingly sophisticated imaging techniques, such as computed tomography (CT) and magnetic resonance imaging (MRI), significantly advancing the digitalization of plant phenotypic morphology and structure [52]. This abstract demonstrates the combination of 3D imaging and multispectral imaging in plant phenotyping research, along with the key parameters associated with the equipment, such as spectral bands, computable plant phenotype parameters, and detected objects. It also highlights the functionalities achieved by the phenotyping platform formed by these technologies. Furthermore, it touches upon existing

4

T. Xia et al.

limitations and provides future recommendations. This provides further reference information for exploring plant phenotyping research and establishing detection platforms. The remaining portions of this paper are arranged as follows: in Sect. 1.2, we will provide a summary of three-dimensional imaging techniques and further elaborate on laser triangulation, Time-of-Flight, and stereo vision. In Sect. 1.3, we will discuss multispectral imaging and its three implementation methods. We will further discuss other application areas and the connections and insights brought about by other spectral bands in three-dimensional multispectral imaging systems. In Sect. 1.4, we will introduce some of the existing commercially available three-dimensional multispectral imaging systems. In Sect. 1.5, we will present some perspectives and discuss certain limitations of the proposed approach.

1.2 Three-Dimensional and Multispectral Imaging 1.2.1 Three-Dimensional Imaging Three-dimensional imaging technology can provide three-dimensional information about the observed object, which can be viewed in corresponding software. Compared with traditional 2D imaging methods, three-dimensional imaging can provide more intuitive physical information about object volume, height, position, and structure [17]. In several disciplines, including urban planning [36], cultural heritage preservation [43], and forest measurement [22], three-dimensional imaging technology is extensively employed. Three-dimensional imaging technology is divided into contact three-dimensional imaging technology and non-contact three-dimensional imaging technology. Contact three-dimensional imaging technology can directly affect the imaging results when in contact with the measured object, especially when the contact surface of the measured object is soft. This will lead to inaccurate imaging data since direct contact with the measured item will alter the phenotype [6]. The primary technology used at the moment is non-contact, using three-dimensional image technology to address demands beyond touch [16, 45]. This technology does not require direct contact with the measured object. Instead, the three-dimensional information of an object is obtained indirectly through various methods. Noncontact three-dimensional imaging technology includes active three-dimensional and passive three-dimensional imaging technology. Active three-dimensional imaging technology typically uses lasers or other energy sources to actively emit light or energy to the target object, then generates a three-dimensional model by measuring the reflection or scattering of the light or energy on the object’s surface. Passive imaging technology generates a three-dimensional model by measuring the light or energy emitted by the target object. Active three-dimensional imaging technology includes laser triangulation [39], time-of-flight [21], structured light

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

5

Fig. 1.2 A summary of three-dimensional imaging techniques

[33], and Moiré fringe methods. For example, structured light emits specific structured light with characteristic points, and the structured light projection on the object surface can be captured successively to calculate the three-dimensional information of the object surface. Linear laser is a type of structured light. Passive three-dimensional imaging technology includes stereo vision, texture-based shape recovery, and shadow-based shape recovery. The classification of three-dimensional imaging technology is shown in Fig. 1.2. We will briefly overview three frequently used methods in the following section: laser triangulation, time-of-flight, and stereo vision.

1.2.1.1

Laser Triangulation

Laser triangulation often uses a line laser projected onto the object to be measured. The camera captures the line laser’s surface projection. It is possible to determine the depth of the object’s surface using the triangle connection. The relative movement between the line laser and the measured object may also gather three-dimensional information about the object’s surface [41]. As shown in Fig. 1.3, the oblique projection structure commonly used in the laser triangulation method is illustrated. The laser emits a beam of light, and the reflected laser’s color that returns after being emitted corresponds to the reflective surface it bounces off (e.g., red laser reflection corresponds to the red platform). The blue laser reflected from the object’s height after being reflected from the blue platform enters the sensor surface through the camera lens. If there is a change in height, such as the blue platform increasing

6

T. Xia et al.

Fig. 1.3 A commonly used oblique projection structure in laser triangulation. The height d to be detected is related to the distance difference d' sensed on the sensor surface (digitized from [Laser light reflected from different distances strikes the sensor in different locations]. What is Laser Triangulation? https://www. movimed.com/knowledge base/what-is-laser-triangula tion/)

in height by d to reach the same height as the red platform, the red laser reflected from the object will enter the sensor surface through the camera lens, and the sensor surface will produce a corresponding distance difference d ' . The greater the height d, the greater the distance difference d ' .

1.2.1.2

Time-Of-Flight

Time-of-Flight (TOF) calculates the distance between the object and the sensor by measuring the time it takes for the light to travel from the sensor to the object and back. During TOF sensor operation, the sensor needs to emit and receive light. The TOF sensor emits a laser, which is reflected to the TOF sensor after being reflected by the plant that needs to be measured for depth information. The separation between the measuring item and the TOF sensor, the distance between the measured item and the calculated object is calculated by multiplying the speed of light by the time it takes for the laser to be sent and received. Many popular TOF depth cameras are on the market, such as Depth Sense sensor (Sony, Tokyo, Japan), Swiss Ranger 400 (MESA Imaging, Zurich, Switzerland), and Kinect v2 sensor (Microsoft, Redmond, Washington, USA). The RGB camera, infrared camera, and infrared projector are the three cameras that makeup Kinect v2 sensor, shown in Fig. 1.4.

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

7

Fig. 1.4 The Kinect v2 s binocular vision sensor

1.2.1.3

Stereo Vision

Eyes are located on both sides of the head and observe the same object from different angles, resulting in slightly different images, known as disparity. The greater the disparity, the closer the object is to the eyes; the smaller the disparity, the farther the object is from the eyes. Stereo vision refers to using the disparity difference between the two eyes of humans to generate a sense of depth and achieve three-dimensional space perception. Stereo vision generally uses binocular stereo vision technology. The primary binocular stereo vision technology method is to use two cameras to capture images from slightly different left and right perspectives, then pairing these two images to create a sense of depth. The principle of binocular stereo vision is shown in Fig. 1.5. Ol and Or are the optical centers of the left and right cameras, respectively. D is a point in space. M l and M r are the image points of point M in the two cameras, respectively. Ol , M l and D are on the same line. Or , M r and D are on the same line. Each point in three-dimensional space and the imaging plane has a projection line, such as the ray Ol M l and the ray Or M r in the figure. All points on the ray Ol M l are projected into the image point M l , such as the space points D and D' . Therefore, only the image points in a single image cannot uniquely determine the three-dimensional coordinates of the space point. However, if two cameras are used to simultaneously photograph the same space point D and obtain the image points M l and M r of point D on the image planes of the left and right cameras, respectively, the position of the space point the intersection of the lines Ol M l and OrMr can uniquely determine D. A binocular camera that uses binocular stereo vision technology consists of two cameras with lenses at a certain angle to each other. The larger the angle between the cameras, the better the final three-dimensional imaging effect. However, if the angle exceeds a specific range, the overlapping area of the images will decrease, affecting the accuracy of image matching. Therefore, when designing a binocular camera, choosing an appropriate angle is vital. Well-known products that use binocular stereo vision technology include the ZED 2 K Stereo Camera (STEREO LABS, San Francisco, CA, USA), as shown in Fig. 1.6, with a depth range of 0.5–20 m and a maximum resolution of 4416 × 1242.

8

T. Xia et al.

Fig. 1.5 A schematic diagram depicting the fundamental principle of the binocular stereo vision technique

Fig. 1.6 The ZED 2 K stereo camera

1.2.2 Multispectral Imaging Multispectral imaging is a technique used to capture three-dimensional image information, including one-dimensional spectral information projected onto an object and the corresponding two-dimensional spatial information of the object [42]. Multispectral imaging typically uses two or more spectral bands to project light onto the measured object, with LED light sources being the most commonly used technology. Compared to traditional multispectral imaging systems, LED light sources can simplify the entire system and improve image stability [3]. Multispectral imaging data can be acquired in three ways, as shown in Fig. 1.7. The light from a specific band of the light source projected onto the object is received by a collector. Based on how the collector receives the light, multispectral imaging is classified into three types: point, line, and area scanning. Point scanning records spectral information for a single point each time, line scanning records a row or column of point data at a time, and area scanning records the spectral information for a surface one band at a time, finally obtaining all the spectral information for that surface. Multispectral imaging has many applications in fields such as plant phenotype detection [42], military [46], and high-precision color reproduction [49].

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

9

Fig. 1.7 Three types of Multispectral Imaging (MSI) methods, based on how the collector receives the light from a specific band of the light source projected onto the object being measured: a point scanning, b line scanning, and c area scanning

1.3 Three-Dimensional Multispectral Imaging Systems in Plant Phenotyping 1.3.1 System Structure Three-dimensional multispectral imaging is an interdisciplinary technology, with its core essence in the fusion of multispectral information and three-dimensional data. Currently, most research involves acquiring three-dimensional point clouds using diverse three-dimensional imaging technologies and capturing multispectral information through multispectral cameras. These two datasets are subsequently fused by projecting the multispectral information onto the corresponding three-dimensional point cloud [37]. Liu et al. [19] studied the characteristics of multispectral point clouds and registered three-dimensional point clouds created by different sensors. They found that the points created from the ultraviolet spectrum (UV), blue, green, red, and NIR images in the multispectral point cloud had complementary properties (see Fig. 1.8).The measured item will not reflect all the light in the same band since various light wavelengths are reflected by the same leaf differently. Consequently, the initial picture of a single band does not show the whole image of the analyzed item. In Fig. 1.8, the original image of band A is represented by green patterns, red patterns represent the original image of band B, and the original image of band C is represented by blue patterns. After being registered in the same coordinate system, the original images of each band complement each other, and the resulting image is closer to the object being measured. To obtain the characteristics of the plant’s three-dimensional

10

T. Xia et al.

Fig. 1.8 Illustration of the complementary nature of multiple bands. The original image of band A is represented by a green pattern, the original image of band B is represented by a red pattern, and the original image of band C is represented by a blue pattern

point cloud, CM-140GE-UV monochrome cameras, GMUV42528C lenses, and five filter films corresponding to 365 nm (UV), 465 nm (blue), 528 nm (green), 630 nm (red), and 748 nm (NIR) were used to create a three-dimensional point cloud in each spectral band of UV, blue, green, red, and NIR. The multispectral point cloud was generated by mapping the picture intensity values for each spectral band to the point cloud after all the point clouds had been lined up on the same three-dimensional coordinate system. As illustrated in Fig. 1.9, Sun et al. [35] created a high-integration, multispectral, three-dimensional, non-destructive detecting system for greenhouse tomato plants. The SOC710 hyperspectral imaging device, Kinect sensor, motorized turntable, and tripod were some parts utilized in the multispectral three-dimensional reconstruction system for greenhouse tomatoes. The same imaging chamber was utilized to simultaneously acquire RGB-D and multispectral pictures of each plant at four angles of view (AOV) with a 90° angle interval using the Surface Optics SOC710 series hyperspectral imaging device and the Kinect camera. Sun et al. [35] proposed a heterogeneous sensor image registration technology based on Fourier transform to register the SOC710 multispectral reflectance to the Kinect depth image coordinate system. It was then suggested to rebuild the multispectral three-dimensional point cloud model of tomato plants using a three-dimensional multi-view RGB-D image reconstruction approach based on Kinect sensor position estimation and self-calibration. Plant indices were calculated based on the multispectral point cloud model of the plant canopy.

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

11

Fig. 1.9 Greenhouse tomato plant multispectral three-Dimensional reconstruction system: 1. SOC710 hyperspectral imaging instrument; 2. Kinect sensor; 3. Tripod; 4. Tomato plant; 5. Electric turntable

Jurado et al. [14] used a professional drone (model: DJI Matrice 210), a highresolution digital camera (model: Sony Alpha 7 RIII) and a multispectral sensor (model: Parrot Sequoia) to carry out canopy phenotyping. The full-frame RGB camera took photos with 48 million pixels, while the multispectral sensor captured the reflectance of four spectra: near-infrared (NIR) at 770–810 nm, red at 640– 680 nm, green at 530–570 nm, and red edge (REG) at 730–740 nm. The initial three-dimensional model of the study area was obtained by using a high-resolution camera on a unmanned aerial vehicle (UAV) to shoot multiple overlapping RGB images and then using the structure from motion (SfM) algorithm to model the three-dimensional model. In addition, a drone-based multispectral sensor was used to capture the reflectance of some narrow bands (green, NIR, red and red edge), and then the reflectance map generated by each multispectral image was mapped to the three-dimensional model to enrich the RGB point cloud of the high-detail geometry of the olive tree. The reflectance seen in the plant canopy serves as a representation of the final three-dimensional model of the olive tree. As indicated in Table 1.1, a number of vegetation indices (NDVI, RVI, GRVI, and NDRE) were also calculated using these reflectance data. In the mapping process, that is, using the input RGB and multispectral images to reconstruct the olive plantation in three dimensions, the two point clouds (RGB and multispectral) are aligned and set in the same coordinate system after correcting the position and orientation of the high-resolution model by some tie points. Each multispectral picture’s reflectance map is computed first; then, the multispectral image is mapped. The three-dimensional model is then improved by weighting the reflectance data following the multispectral camera’s viewpoint. By creating a three-dimensional model using the SfM technique, Matese et al. [25] validated that regions with high NDVI values correlate to regions with high canopy height using a drone and a multispectral camera.

12

T. Xia et al.

Table 1.1 Calculation of multiple vegetation indices based on NIR, RED, GREEN, and REG reflectance values Vegetation indices

Calculation

NDVI (Normalized Differential Vegetation Index)

(NIR-RED)/(NIR+RED)

GRVI (Green Normalized Difference Vegetation Index)

NIR/GREEN

RVI (Red Normalized Difference Vegetation Index)

NIR/RED

NDRE (Normalized Difference Red Edge Index)

(NIR-REG)/(NIR+REG)

Shaharom et al. [31] effectively produced a high-density point cloud in the data of the red, red edge, and green bands by employing a multispectral camera to shoot a hole, processing all the acquired data using the SfM concept. Clamens et al. [4] developed a real-time technique to acquire, process, register, and fuse multispectral pictures into a three-dimensional image. The matching sensors employed were the CMS-V multispectral camera and the Kinect V2. The Kinect V2 sensor could acquire the depth and RGB information of the object (for obtaining three-dimensional information), and the depth and RGB images captured by the Kinect sensor were already factory-calibrated. Therefore, calibrated depth and RGB information could be obtained directly from the Kinect sensor. The multispectral camera required specific acquisition, pre-processing, and intrinsic calibration before using its data. The multi-sensor system, which is seen in Fig. 1.10, has a Kinect sensor (C0), a depth camera (C1), and an RGB camera (C2), both of which were factory-calibrated (shown by transformation T1). Therefore, the multispectral image could be registered onto the three-dimensional point cloud generated by the Kinect depth camera. The most critical step in the process of transformation T2 from T1 was that after collecting, calibrating, and pre-processing the images, the multispectral camera obtained nine image channels. The red image channel of the multispectral camera and the red image channel of the Kinect sensor’s RGB image were used for external calibration. Mansouri et al. [24] used a monochrome CCD camera C, a standard photographic lens, seven interference filters B, and a liquid crystal display (LCD) projector with a motorized wheel A in front of the camera/lens system. The wheel had eight holes, seven containing the filters, and one was empty for capturing unfiltered images. Combining the multispectral camera with the LCD projector, they scanned a threedimensional artistic object with the rotating electric wheel A. During geometric reconstruction, which is the three-dimensional reconstruction process, the camera and the LCD projector were placed at the same height and a similar distance from the object. The LCD projector emitted a vertical line to scan the object. Without a filter, the radiating pixels were visible through the gap. In terms of three-dimensional points, each pixel describes one. Since the junction point of the emission plane and the line of sight were at this location, and the camera’s optical center, this intersection point could be determined. The three-dimensional locations of various components of the emission pattern on the object are determined through geometric reconstruction using triangulation. In spectral reconstruction, the reflection under the seven

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

13

Fig. 1.10 A multi-sensor system that consists of the Kinect V2 (C0), including the depth camera (C1) and the RGB camera (C2) and the multispectral camera (C3). The registration from the depth camera to the RGB camera and from the RGB camera to the multispectral camera is represented by T1 and T2, respectively

interference filters was finally used to obtain the reflectance spectrum rate through a neural network. The reflectance spectrum rate is the ratio of the reflected flux of a specific band to the incident flux of the same band. Calculating the reflected light from the incoming light requires knowledge of the reflectance spectrum rate. Finally, the reflectance spectrum rate was associated with each three-dimensional reconstruction point to simulate the appearance of a three-dimensional object under any virtual light source. Photoacoustic (PA) imaging scans formed cross-sectional images at a specific depth, and the slices imaged at different depths were stacked together to form a three-dimensional image [47]. Yoon et al. [50] produced three-dimensional multispectral photoacoustic imaging by collecting three PA pictures corresponding to three optical wavelengths (756, 797, and 866 nm). Es Sebar et al. [7] utilized a camera with different bandpass filters to obtain spectral images corresponding to the transmitted bands. During the imaging process, the camera did not move while the object was photographed from different angles to obtain data. Using the free program Meshroom, they created a multispectral, three-dimensional representation of the item. This also motivated us to use the techniques above for imaging plant phenotypes.

1.3.2 Factors Affecting Imaging Under laboratory conditions, when using an artificial light source for close-range measurement, the plant’s spectral reflectance is affected by the three-dimensional

14

T. Xia et al.

structure due to the inverse square law of the Lambert cosine law. Obtaining the threedimensional point cloud of the plant can help us obtain accurate spectral reflectance. Another method to avoid the influence of a close-range light source requires using certain facilities to achieve the diffuse reflection condition of the artificial light source [20]. Liu et al. proposed a simple diffuse reflection panel setting, a simplified version of a diffuse reflection dome where the light is fully diffused and can be used for plant phenotyping measurements. Xie et al. [44] used RGB-D and multispectral cameras to generate high-precision plant three-dimensional multispectral point clouds at close range. Still, the registration was affected by the illumination effects. They developed a multimodal image registration method to register the images. Hyperspectral imaging technology, like multispectral imaging technology, is used to capture plant spectral information. Hyperspectral imaging technology has the advantages of having multiple bands, narrow spectral ranges, and large amounts of data. The spectral range of hyperspectral imaging technology is around 400– 2500 nm, and several hundred bands can be used with a spectral resolution of several nanometers [51]. We can obtain plant phenotype information at higher spatial and temporal resolution using hyperspectral imaging technology. However, when using hyperspectral imaging, many bands may not contribute significantly to the analysis of plant phenotypes, and analyzing too many bands may increase workload. Mehl et al. [26] analyzed hyperspectral imaging data using continuous spectral bands, while multispectral imaging technology analyzes discrete spectra at several wavelengths. After determining the analysis direction of the object being imaged, hyperspectral imaging technology can be used to obtain the bands needed to construct a multispectral camera. Workload may be significantly reduced by obtaining certain bands using hyperspectral imaging technology and then utilizing multispectral imaging technology to gather and analyze data corresponding to those bands. Hyperspectral images often have a low spatial resolution but a wide spectral range, while multispectral images have a high spatial resolution but fewer spectral bands. Dian et al. [5] combined hyperspectral and multispectral images to take advantage of both, achieving high spatial resolution and a wide spectral range. Four fusion techniques for hyperspectral and multispectral pictures are additionally presented.

1.4 Commercial Three-Dimensional Multispectral Imaging Systems Currently, there are many consumer-level products for three-dimensional and multispectral imaging sensors, but there are not many mature businesses for threedimensional multispectral imaging technology. Because 2D images are projections of the three-dimensional world, there are some tasks and situations that are easier to solve in three-dimensional models than in 2D images [18], Consequently, three-dimensional multispectral imaging technology offers more problem-solving

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

15

capabilities than multispectral imaging technology and has enormous commercial potential. A few three-dimensional and multispectral imaging devices are commercially available for plant phenotyping, such as the PlantEye (Phenospex, Heerlen, Netherlands), which obtains three-dimensional point cloud information of plant phenotypes through line laser scanning [15], as shown in Fig. 1.11. PlantEye can obtain information such as leaf area, plant height, leaf tilt angle, color index, leaf projection area, etc. At the same time, PlantEye’s multispectral light source comes from the LED installed in the scanner, which has four spectral bands: red, green, blue, and NIR. PlantEye can obtain RGB and NIR images and various vegetation indexes such as NDVI on the three-dimensional image. PlantEye operates in any lighting situation because it employs clever algorithms to remove ambient light altogether. PhenoWatch (Agripheno, Shanghai, China) uses laser radar for three-dimensional imaging and integrates RGB, multispectral, and hyperspectral imaging. It can perform single-plant recognition, separation of plant structures and parameters such as leaf area and tilt angle. The MSDC-RGBN-1-A (Spectral Devices Inc, Telford, PA, USA) from Spectral Devices Inc. It is additionally able to acquire RGB threedimensional photos and different vegetation metrics. The Netherlands National Plant Phenotyping Facility (NPEC) is equipped to measure the phenotypes of field plants, assess the phenotypes of roots, and conduct high-throughput three-dimensional reconstruction on plants. The current cost of a large-scale phenotyping platform that can maturely implement three-dimensional multispectral imaging is several hundred

Fig. 1.11 The morphological and physiological data measured by the PlantEye, a commercial three-dimensional and multispectral imaging device

16

T. Xia et al.

thousand yuan or more. Consequently, the high cost, large amount of data, and subsequent equipment maintenance are challenges that need to be addressed.

1.5 Conclusion and Outlook From the earlier description, it can be understood that there are now numerous devices available for three-dimensional multispectral imaging. These devices combine the advantages of both three-dimensional imaging and multispectral imaging. Threedimensional imaging technology can quickly and accurately obtain the threedimensional morphology information of plants, including parameters such as plant height, width, length, curvature, etc., improving plant phenotype research’s accuracy and reliability. Multispectral imaging technology can simultaneously obtain multispectral information of plants, including parameters such as reflectance and absorption of various bands. These parameters can be used to evaluate plants’ physiological and biochemical status, such as chlorophyll content, leaf water content, leaf thickness, etc., thereby helping us better understand the growth and development patterns of plants and strategies for adaptation to the environment. Three-Dimensional multispectral imaging technology can efficiently screen and evaluate many plants, helping us select the most adaptive and productive plant varieties. Nowadays, three-dimensional imaging and multispectral sensors combine to create three-dimensional multispectral imaging, and the complete hardware system is very complicated. It’s essential to consider making the system more straightforward and less expensive while keeping high-precision readings. An intriguing study area at the moment is how to leverage the three-dimensional point cloud of plants in close-range measurements to assist us in acquiring more precise spectrum reflectance under artificial light sources.

References 1. Baweja HS, Parhar T, Mirbod O, Nuske S (2018) Stalknet: a deep learning pipeline for highthroughput measurement of plant stalk count and stalk width. In: Field and service robotics: results of the 11th international conference. Springer International Publishing, pp 271–284 2. Baylin SB, Gazdar AF, Minna JD, Bernal SD, Shaper JH (1982) A unique cell-surface protein phenotype distinguishes human small-cell from non-small-cell lung cancer. Proc Natl Acad Sci 79(15):4650–4654 3. Chu JJ, Cui GH, Liu YW, Xu T, Ruan XK, Cai QB, Tan YH (2018) A method for measuring surface color based on spectral tunable led light source and multispectral imaging technology. Acta Optica Sinica 08:421–429 4. Clamens T, Alexakis G, Duverne R, Seulin R, Fauvet E, Fofi D (2021) Real-time multispectral image processing and registration on 3d point cloud for vineyard analysis. In: VISIGRAPP (4: VISAPP), pp 388–398 5. Dian R, Li S, Sun B, Guo A (2021) Recent advances and new guidelines on hyperspectral and multispectral image fusion. Inform Fusion 69:40–51

1 A Review of Three-Dimensional Multispectral Imaging in Plant …

17

6. Ding SW, Zhang XH, Yu QF, Yang X (2017) Overview of non-contact 3d reconstruction measurement methods. Laser Optoelectron Progress 07:27–41 7. Es Sebar L, Lombardo L, Buscaglia P, Cavaleri T, Lo Giudice A, Re A, Grassini S (2023) 3D multispectral imaging for cultural heritage preservation: the case study of a wooden sculpture of the Museo Egizio di Torino. Heritage 6(3):2783–2795 8. Fiorani F, Schurr U (2013) Future scenarios for plant phenotyping. Annu Rev Plant Biol 64:267–291 9. Geissler T, Wessjohann LA (2011) A whole-plant microtiter plate assay for drought stress tolerance-inducing effects. J Plant Growth Regul 30:504–511 10. Giuffrida MV, Doerner P, Tsaftaris SA (2018) Pheno-deep counter: a unified and versatile deep learning architecture for leaf counting. Plant J 96(4):880–890 11. Jiménez AR, Ceres R, Pons JL (2000) A vision system based on a laser range-finder applied to robotic fruit harvesting. Mach Vis Appl 11:321–329 12. Jimenez-Marin D, Dessauer HC (1973) Protein phenotype variation in laboratory populations of Rattus norvegicus. Comp Biochem Physiol Part B: Comp Biochem 46(3):487–488 13. Johannsen W (1911) The genotype conception of heredity. Am Nat 45(531):129–159 14. Jurado JM, Ortega L, Cubillas JJ, Feito FR (2020) Multispectral mapping on 3D models and multi-temporal monitoring for individual characterization of olive trees. Remote Sens 12(7):1106 15. Kjaer KH, Ottosen CO (2015) 3D laser triangulation for plant phenotyping in challenging environments. Sensors 15(6):13533–13547 16. Li JL, Xin QQ, Tian L, Zhu W (2014) A review of online measurement method for large forgings. J New Ind 01:59–64 17. Liu H, Bruning B, Garnett T, Berger B (2020) Hyperspectral imaging and 3D technologies for plant phenotyping: from satellite to close-range sensing. Comput Electron Agric 175:105621 18. Liu H, Lee SH, Chahl JS (2017) A multispectral 3-D vision system for invertebrate detection on crops. IEEE Sens J 17(22):7502–7515 19. Liu H, Lee SH, Chahl JS (2018) Registration of multispectral 3D points for plant inspection. Precision Agric 19:513–536 20. Liu Y, Pears N, Rosin PL, Huber P (eds) (2020) 3D imaging, analysis and applications. Springer, Berlin/Heidelberg, Germany, pp 109–166 21. Lu CQ, Song YZ, Wu YP, Yang MF (2018) 3D information acquisition and error analysis based on TOF computational imaging. Infrared Laser Eng 10:160–166 22. Luo X, Feng ZK, Deng XR, Hao XY, Chen XX (2007) Application of 3D laser scanning imaging system in forest measuring. J Beijing For Univ S2:82–87 23. Mahner M, Kary M (1997) What exactly are genomes, genotypes and phenotypes? and what about phenomes? J Theor Biol 186(1):55–63 24. Mansouri A, Lathuilière A, Marzani FS, Voisin Y, Gouton P (2007) Toward a 3D multispectral scanner: an application to multimedia. IEEE Multimed 14(1):40–47 25. Matese A, Di Gennaro SF, Berton A (2017) Assessment of a canopy height model (CHM) in a vineyard using UAV-based multispectral imaging. Int J Remote Sens 38(8–10):2150–2160 26. Mehl PM, Chao K, Kim M, Chen YR (2002) Detection of defects on selected apple cultivars using hyperspectral and multispectral image analysis. Appl Eng Agric 18(2):219 27. Pan YH (2015) Analysis of concepts and categories of plant phenome and phenomics. Acta Agron Sin 02:175–186 28. Paulus S, Dupuis J, Mahlein AK, Kuhlmann H (2013) Surface feature based classification of plant organs from 3D laser scanned point clouds for plant phenotyping. BMC Bioinform 14(1):1–12 29. Paulus S, Dupuis J, Riedel S, Kuhlmann H (2014) Automated analysis of barley organs using 3D laser scanning: an approach for high throughput phenotyping. Sensors 14(7):12670–12686 30. Samal A, Choudhury SD (eds) (2020) Intelligent image analysis for plant phenotyping. CRC Press 31. Shaharom MFM, Abd Mukti SN, Raja Maharjan G, Tahar KN (2023) Multispectral’s threedimensional model based on SIFT feature extraction. Int J Geoinform 19(5)

18

T. Xia et al.

32. Strohman RC (1995) Linear genetics, non-linear epigenetics: complementary approaches to understanding complex diseases. Integr Physiol Behav Sci 30:273–282 33. Su XY, Zhang QC, Chen WJ (2014) Three-dimensinal imaging based on structured illumination. Chin J Lasers 02:9–18 34. Sun D, Robbins K, Morales N, Shu Q, Cen H (2022) Advances in optical phenotyping of cereal crops. Trends Plant Sci 27(2):191–208 35. Sun G, Wang X, Sun Y, Ding Y, Lu W (2019) Measurement method based on multispectral three-dimensional imaging for the chlorophyll contents of greenhouse tomato plants. Sensors 19(15):3345 36. Tang Z, Zhang XC, Cao KB (2010) Research on urban planning with three-dimensional technology based on skyline. Bull Surv Mapp 05:0–12+41 37. Tudor PM, Christy M (2011) Rapid high-fidelity visualisation of multispectral 3D mapping. In: Laser radar technology and applications XVI, vol 8037. SPIE, pp 159–166 38. Varki A, Wills C, Perlmutter D, Woodruff D, Gage F, Moore J, Bullock T (1998) Great ape phenome project? Science 282(5387):239–239 39. Wan J, Huang YQ (2006) Study on laser triangulation method measurement. J Sanming Univ 04:364–364 40. Wang J, Zhang Y, Gu R (2020) Research status and prospects on plant canopy structure measurement using visual sensors based on three-dimensional reconstruction. Agriculture 10(10):462 41. Wang XJ, Gao J, Wang L (2004) Survey on the laser triangulation. Chin J Sci Instrum S2:601– 604+608 42. Wang ZS, Jia YP, Zhang J, Wang RH (2021) Multispectral imaging and its applications in plant science research. Chin Bull Botany 04:500–508 43. Wu YH, Zhou MQ (2009) Application of 3d scanning technique in heritage protection. Comput Technol Dev 09:173–176 44. Xie P, Du R, Ma Z, Cen H (2023) Generating 3d multispectral point clouds of plants with fusion of snapshot spectral and RGB-D images. Plant Phenom 5:0040 45. Xiong LL, Qian D, Li HZ (2013) Overview of three-dimensional reconstruction based on real-time. J Commun Univ China (Sci Technol) 06:38–43 46. Xu H, Wang XJ (2007) Applications of multispectral/hyperspectral imaging technologies in military. Infrared Laser Eng 01:13–17 47. Xu M, Wang LV (2006) Photoacoustic imaging in biomedicine. Rev Sci Instrum 77(4) 48. Xu R, Li C, Paterson AH, Jiang Y, Sun S, Robertson JS (2018) Aerial images and convolutional neural network for cotton bloom detection. Front Plant Sci 8:2235 49. Yang WP, Xu N, Jian DJ, Li YB, Lu Q, Sun YN, Luo X, Luo YD (2009) Application and development of multispectral imaging technology in color reproduction. J Yunnan Natl Univ (Nat Sci Edn) 03:191–197 50. Yoon C, Lee C, Shin K, Kim C (2022) Motion compensation for 3d multispectral handheld photoacoustic imaging. Biosensors 12(12):1092 51. Zhang B (2016) Advancement of hyperspectral image processing and information extraction. J Remote Sens 5:1062–1090 52. Zhao CJ, Lu SL, Guo XY, Du JJ, Wen WL, Miao T (2015) Advances in research of digital plant: 3d digitization of plant morphological structure. Sci Agric Sinica 17:3415–3428

Chapter 2

Recent Advances in Soil Nutrient Monitoring: A Review Qianying Yuan, Wenyi Sheng, Zhao Zhang, Han Li, and Man Zhang

Abstract Soil nutrient monitoring assumes a critical role in advancing modern agriculture. Precise evaluation of soil nutrient levels lays the foundation for creating systematic and informed fertilization strategies. This, in turn, amplifies crop growth and yield, all the while optimizing soil well-being. This review provides a comprehensive survey of prevalent soil nutrient analysis techniques, with a specific emphasis on laboratory methodologies, spectroscopy approaches, and electrochemical methods. The merits and demerits of these techniques are meticulously examined and deliberated upon, along with prospects for future advancements. Among the array of methods discussed, spectroscopy methods take the spotlight, encompassing visible light, near-infrared (NIR), and laser-induced breakdown spectroscopy. Additionally, the incorporation of ion-selective membranes for electrochemical evaluation of nitrogen, phosphorus, and potassium also finds a notable place. In the grand scheme, the exploration and application of soil nutrient testing bear profound significance in augmenting agricultural productivity, guaranteeing food security, and realizing sustainable agricultural progress. Keywords Soil nutrient NPK · Laboratory · Near-infrared spectroscopy · Ion selective membranes · Electrodes

2.1 Introduction Soil stands as the foundational cornerstone of agricultural production, playing a pivotal role in orchestrating material and energy cycles within agricultural systems. Its significance extends further as it underpins the survival and proliferation of diverse Q. Yuan · W. Sheng (B) · Z. Zhang · H. Li · M. Zhang College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China e-mail: [email protected] Key Laboratory of Smart Agriculture System Integration, Ministry of Education, China Agricultural University, Beijing 100083, China © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 M. Zhang et al. (eds.), Sensing Technologies for Field and In-House Crop Production, Smart Agriculture 7, https://doi.org/10.1007/978-981-99-7927-1_2

19

20

Q. Yuan et al.

life forms on our planet. Research underscores the fact that a staggering majority— more than 80% of heat, over 75% of protein, and a substantial portion of dietary fiber consumed by humans—originates directly from soil [43]. In this context, the rational exploitation and preservation of soil resources emerge as non-negotiable imperatives for both agricultural advancement and the maintenance of a harmonious ecological equilibrium. At the heart of evaluating soil resource quality lies soil fertility—a compass indicating a soil’s ability to furnish and regulate nutrients, water, air, and warmth for crop sustenance. It encapsulates the foundational trait and inherent nature of the soil itself. The Farm to Fork Strategy, a visionary endeavor in Europe, sets its sights on fostering scientific breakthroughs and integrating innovative technologies (including cutting-edge machinery, genetically modified organisms, and information and communication technologies tailored for efficient and sustainable agriculture). Simultaneously, it seeks to cultivate an elevated consciousness and demand for sustainable food practices. In this epoch of agricultural sensing advancements, farmers have garnered an acute awareness of the pivotal role technological strides play in achieving elevated crop yields. By meticulously tracking soil nutrients and pesticide levels, they can precisely calibrate the application of fertilizers and pesticides, optimizing their dosages for specific plots of land. The outcome is amplified crop productivity, reduced expenditures, and a curtailed ecological footprint stemming from chemical pesticide use [20]. Soil is the most fundamental production input in agriculture, serving as the central component for material and energy cycling in agricultural processes. It forms the basis for the survival and proliferation of various life forms on Earth. Studies have demonstrated that more than 80% of the heat, over 75% of the protein, and a significant portion of dietary fiber consumed by humans directly originate from the soil [43]. Rational utilization and conservation of soil resources are indispensable for agricultural development and maintaining a healthy ecological balance. The primary indicator of soil resource quality is soil fertility, which refers to the soil’s capacity to supply and regulate nutrients, water, air, and heat for crop growth. It constitutes the fundamental attribute and intrinsic characteristic of the soil. The Farm to Fork Strategy in Europe aims to foster scientific discoveries and implement innovative technologies (such as advanced machinery, genetically modified plants or animals, and information and communication technologies for efficient and sustainable agriculture) while raising awareness and demand for sustainable food practices. With the advancement of agricultural sensing technology, farmers have become increasingly cognizant of the significance of technological progress in achieving higher crop yields. Precise monitoring of soil nutrients and pesticides enables targeted application of varying doses of fertilizers and pesticides to farmland, thereby increasing crop yield, reducing costs, and mitigating the environmental impact of chemical pesticides [20]. The Food and Agriculture Organization of the United Nations (FAO) defines soil fertility as the “ability of the soil to sustain plant growth by providing necessary plant nutrients.“ Among the various nutrients in the soil, nitrogen, phosphorus, and potassium are elements with higher demand and uptake by crops, requiring supplementation through fertilization. Soil nitrogen is a major component of crop

2 Recent Advances in Soil Nutrient Monitoring: A Review

21

proteins and plays a crucial role in the growth of crop stems, roots, and fruit development. Soil phosphorus can increase crop yield, improve crop quality, and enhance crop resistance to cold and drought. Soil potassium promotes crop photosynthesis and the transportation of photosynthetic products, stimulating protein synthesis, thereby increasing the protein content in crops. To meet the nutritional needs of crops during their growth, people began using chemical fertilizers to supplement soil nutrients, greatly improving soil fertility, and subsequently promoting the development of modern agriculture. However, the indiscriminate use of chemical fertilizers can reduce their efficiency and result in economic losses and environmental pollution, leading to eutrophication of surface water. Therefore, obtaining accurate and rapid information on soil nutrient content and applying fertilizers in a balanced and appropriate manner is of significant importance for the sustainable development of agriculture. This review provides a summary and overview of soil nutrient monitoring methods in recent years, focusing on laboratory, spectroscopy, and electrochemical methods. We will primarily discuss the detection of major essential nutrients of N (nitrogen), P (phosphorus), and K (potassium), analyzing their accuracy, while also addressing some of the challenges currently faced in soil nutrient testing.

2.2 Basic Concepts of Soil Nutrients Soil nutrients refer primarily to the essential elements provided by the soil, which are necessary for the growth of plants. They serve as a critical material foundation for soil fertility and are among the factors in the soil that can be easily controlled and regulated, nutrients required for soil growth as shown in Table 2.1. Nutrients present in the soil can be categorized into essential elements and beneficial elements, both of which play vital roles in supporting the complete growth cycle of plants [16]. Essential elements include both organic and inorganic elements. Organic elements consist of carbon (C), hydrogen (H), and oxygen (O), mainly absorbed by plants through their leaves from the atmosphere. On the other hand, inorganic elements are taken up by plants from the soil through their roots. These inorganic elements include nitrogen (N), phosphorus (P), potassium (K), sulfur (S), calcium (Ca), and magnesium (Mg). Among them, N, P, and K are considered macro-nutrients, and their deficiency can lead to significant yield losses, while S, Ca, and Mg are secondary elements that seldom restrict plant growth. Trace elements encompass iron (Fe), manganese (Mn), molybdenum (Mo), zinc (Zn), boron (B), chlorine (Cl), and nickel (Ni). Trace elements are predominantly naturally present in soil at low concentrations. However, excessively high concentrations of trace elements in the soil can result in plant toxicity. Additionally, there are beneficial elements, such as sodium (Na), cobalt (Cb), and silicon (Si), which are required in smaller quantities by crops [30].

22

Q. Yuan et al.

Table 2.1 Nutrients required for soil growth [30] Beneficial plant elements

Essential plant elements Carbon (C)

Nitrogen (N)

Boron (B)

Cobalt (Cb)

Hydrogen (H)

Phosphorous (P)

Chlorine (Cl)

Silicon (Si)

Oxygen (O)

Potassium (K)

Iron (Fe)

Sodium (Na)

Calcium (Ca)

Manganese (Mn)

Magnesium (Mg)

Molybdenum (Mo)

Sulfur (S)

Nickel (Ni) Zinc (Zn)

2.3 Soil Nutrient Measurement Methods Currently, soil nutrient measurement methods can be categorized into laboratory analysis, spectroscopy analysis, and electrochemical analysis. Laboratory analysis is expensive, time-consuming, and requires sophisticated equipment, but it provides precise measurements. Spectroscopy analysis offers non-destructive, rapid, and efficient detection of soil nutrient content, enabling the determination of soil properties. Electrochemical analysis allows for on-site testing and requires fewer nutrients, facilitating real-time analysis.

2.3.1 Laboratory Analysis Methods In the laboratory, various automatic analyzers are used to measure constant nutrients (N, P, K), which mainly involves two steps: sampling and quantitative nutrient extraction. During the sampling process, samples can be collected at different depths according to the nutrient uptake by different crops, allowing for more targeted acquisition of nutrient information from the soil where the crops are grown. The collected soil samples are thoroughly dried, finely ground, and sieved to obtain a homogeneous mixture of soil samples. Chemical separation of the samples is performed using extractants, and finally, the soil nutrient content is analyzed and evaluated. Currently, several analysis methods are employed, including colorimetric methods, atomic absorption spectroscopy, and chromatography [16]. The colorimetric method is one of the standard laboratory soil testing methods. It is based on the principle that specific chemical substances in the sample react with reagents to produce colored compounds. The concentration of chemical components is indirectly determined by measuring the intensity of the generated color. NO3− -N is determined using the cadmium reduction and diazo dye method to form nitrite. NH4− -N is measured using the Nessler reagent; P is determined using the stannous chloride method; K is obtained by precipitation with tetraphenylboron, and then a colored complex is formed through reactions with specific substances. The color

2 Recent Advances in Soil Nutrient Monitoring: A Review

23

intensity is subsequently interpreted on a pre-prepared color chart. The color readings are then correlated with the actual values (estimated values) of each parameter within a certain range. In the colorimetric method for soil nutrient determination, it is essential to strictly control the dosage of reagents, reaction time, and reaction conditions to ensure the accuracy and reliability of the measurement results. Additionally, environmental factors at the testing site, such as the intensity of ambient light, should also be taken into account as they can affect the readings and lead to result errors [11]. In recent years, continuous advancements in digital technology have enabled the integration of mobile colorimetric soil nutrient testing with digital application systems. Moonrungsee et al. [26] introduced a portable colorimetric analyzer for determining soil P content based on a smartphone’s camera. This method can provide accurate P content values similar to those obtained by laboratory analytical instruments. It offers the convenience of on-site testing, affordability, and ease of operation. However, the color of images captured by smartphone cameras can be significantly affected by ambient light conditions. To address this issue, they designed a lightproof box with a simple LED flash to ensure consistent illumination during image capture. Liu et al. [21] developed a sandwich-structured chip-level soil nutrient colorimeter based on the Beer-Lambert law and MEMS technology, as depicted in Fig. 2.1. The detection limits for N, P, and K were reported to be 83.6, 143, and 40.9 ppm, respectively. The performance of this device surpassed that of commercial detectors, as demonstrated in the interference test shown in Fig. 2.2. The compact design of the system resulted in a small size of only 6 cm × 4 cm × 6 cm, making it highly portable. Consequently, further research can be conducted on colorimetric-based soil nutrient detection to develop a cost-effective and portable sensor. Currently, conducting a single nutrient content measurement in the laboratory costs between 8 to 25 US dollars. The process is expensive and relatively complex, but it enables accurate measurement of soil nutrient content and is often used as the standard for other nutrient sensing measurements [16]. In comparison to colorimetric

Fig. 2.1 a Sandwich structure of a soil nutrient colorimeter with light source and microchannel photodetector and b demonstration of the inlet connected to the peristaltic pump and tested under an applied voltage [21]

24

Q. Yuan et al.

Fig. 2.2 Anti-interference testing results of the colorimeter (black line) compared with commercial equipment (red line) under a dark conditions and b normal light conditions (approximately 300 lx). The blue line represents the reference [21]

methods, spectroscopy-based approaches can offer more precise and rapid analysis of soil nutrients. They leverage the characteristics of soil samples in absorbing, scattering, or transmitting specific wavelengths of light to infer the nutrient content in the soil.

2.3.2 Spectroscopy Methods The spectroscopy method involves the interaction between optical diffuse reflection, incident light, and surface substances of the soil within specific wavelength ranges. The different physicochemical properties of substances in the soil lead to variations in the characteristics of reflected light. Through this, the nutrient content in the soil can be determined, making it a rapid method for measuring soil properties [23]. Compared to electrochemical methods, this optical approach offers non-destructive measurements and does not require soil sample collection, providing several advantages. As a result, many researchers have explored its potential in the development of soil nutrient measurements. The only limitation of this method is the need for soil mapping and the generation of extensive databases. Mohamed et al. [25] collected 100 soil surface samples at a depth of 25 cm and employed chemical analysis and reflectance spectroscopy within the wavelength range of 350–2500 nm. They determined the spectral response positions for soil nutrients (N, P, K), as illustrated in Fig. 2.3.

2 Recent Advances in Soil Nutrient Monitoring: A Review

25

Fig. 2.3 Spectral response of soil nutrients (N, P, K) with each color representing a wavelength range (blue, green, red, NIR, short-wave infrared SWIR1, SWIR2) [25]

2.3.2.1

Near-infrared (NIR) Spectroscopy

NIR refers to electromagnetic waves with wavelengths in the range of 780–2500 nm, which is the region beyond the visible light that humans can perceive. Due to its efficiency, non-destructive and non-polluting nature, and ease of operation in detecting soil nutrient information, NIR spectroscopy has been widely used in soil nutrient testing and agricultural product inspection. Zhou et al. [50] developed a portable total nitrogen (TN) concentration detector based on NIR spectroscopy. They employed a modular concept and Extreme Learning Machine (ELM) algorithm for designing and modeling the detector, respectively. The ELM estimation model for TN concentration achieved a determination coefficient (R2 ) of 0.90, with a validation R2 of 0.82. The detector demonstrated stable performance and high accuracy. Yunus et al. [47] used the DT calibration method to predict the maximum correlation coefficient (R) for N, P, and K, resulting in predicted R values of 0.86 for N, and 0.90 for P and K, respectively. The results indicated that NIR Spectroscopy (NIRS) can serve as a rapid and synchronous alternative method for predicting soil quality indicators. Tan et al. [41] proposed a novel approach for detecting soil N content using a combination of NIR spectroscopy and Random Forest regression (RF). This method enables low-cost, environmentally friendly, and fast detection of soil N content. Comparing with SVM and BP models, the RF model showed better precision and predictive performance, as shown in Table 2.2. Du et al. developed a portable N detector based on NNIR reflectance spectroscopy. They used a small and compact Fourier-transform infrared (FTIR) coupled spectrometer and software for data acquisition and analysis. The wavelength range for soil N determination was found to be 1500–1850 nm, and the nitrogen-containing group wavelength was 2000–2400 nm. The detection coefficient was 0.934, and the root

26

Q. Yuan et al.

Table 2.2 Comparison of soil N content predictions using three different models by Tan et al. [41]

Number

Model

RMSE

R2

1

SVM (linear)

0.156

0.780

2

BP (Levenberg-Marquardt)

0.111

0.876

3

Random forest

0.116

0.921

mean square error (RMSE) was 1.923, indicating good quality for N determination using diffuse reflectance spectroscopy. Ahmad Jani et al. [1] used the microNIR 1700 ES handheld spectrometer for spectral collection, enabling on-site prediction of TN, total organic carbon, and soil pH in soil. However, they also mentioned the difficulties faced in sampling, preprocessing, etc. Additionally, for areas with insufficient power supply, this portable instrument becomes impractical. Preprocessing the acquired spectral data can improve detection accuracy to varying degrees. Using a portable sensing device with a photodiode array detector (NIR), absorbance data within the wavelength range of 1000–2500 nm was obtained to acquire NIR spectroscopy data for soil samples. Gaussian filtering (GF) correction method exhibited good performance, with an R2 of 0.63 for N and 0.48 for Mg (magnesium) [27]. Peng et al. [29] investigated various preprocessing methods for soil nutrient analysis at the detection level using AvaField spectrometer with NIR spectroscopy. They found that Genetic Algorithm-Back Propagation Neural Network optimization accurately detected TN, TP, and total potassium in the soil. Additionally, they discovered that Partial Least Squares Regression (PLSR) is the most commonly used and effective multivariate analysis technique, but not an efficient nutrient component detection method. Singh et al. [39] measured 17 different soil parameters in 507 soil samples from different regions. They concluded that the performance of the Cubist model was superior to that of Support Vector Regression (SVR) and PLSR. For the mixed dataset (0–90 cm), most soil properties estimated by the Cubist model exhibited high accuracy (LCC ≥ 0.7), except for pH, EC, available potassium (AK), and boron. Additionally, the LCC values were low for CEC and Zn using PLSR ( 0.97. The error rate between measured nitrate and reference nitrate was 7

7

pH

NO3 −

54.1

62.6

2–12

2.3 × 10−5 to −6 × 10−2 1 × 10−5 2.0–11.0

9

>7

2 Recent Advances in Soil Nutrient Monitoring: A Review

35

Paczosa-Bator et al. [28] proposed the use of superhydrophobic polymer/carbon nanocomposites as a novel solid contact material for solid-state ISEs. The K-selective electrode exhibited good Nernstian response within the range of 106.5–101 M KCl, with a slope of 59.10 mV/dec. Due to the large capacitance of the solid contact and its superhydrophobic nature, the electrode demonstrated excellent long-term potential stability. Yoon et al. [46] reported the potential performance of ion-toelectron transducer based on reduced graphene oxide (RGO) in all-solid-state K ion sensors. The K ISE based on RGO showed high sensitivity (53.34 mV/log[K]), low detection limit −4.24 log[K], 0.06 mM), and the collected data could be transmitted to a mobile application through a Bluetooth module. Electrochemical sensors offer high sensitivity and good selectivity, but their prices tend to be higher due to the requirement for expensive equipment, materials, and specialized knowledge in their production. Additionally, their durability is generally lower, as many electrochemical sensors may not withstand long-term in-situ measurements when buried in soil [31].

2.4 Summary and Outlook Currently, with the continuous growth of the population, crop yield has become a matter of concern. The adverse environmental effects of excessive fertilizer use are also receiving increasing attention. Therefore, timely detection of soil nutrients and the rational application of fertilizers are crucial. Traditional laboratory measurements require sampling, transportation, and storage, which are time-consuming and not suitable for large-scale implementation. Visible and near-infrared spectroscopy methods can quickly quantify soil properties, but they are susceptible to environmental influences, leading to potential errors in measurements. Therefore, achieving accurate on-site measurements remains a challenge. Electrochemical methods offer the advantage of direct soil measurement and portability, making them promising for real-time analysis. However, due to the need to detect multiple soil nutrient elements and the potential interactions between ions, the discovery of more clearly selective ISMs is needed to ensure accuracy. Various sensing methods are continuously evolving, moving towards low-cost, high sensitivity, portability, and long lifespan, thereby improving the economic and environmental sustainability and increasing the adoption of agricultural technologies. Unmanned aerial vehicles (e.g., drones) and remote sensing technologies are also gradually being applied to soil nutrient detection, enabling efficient large-scale soil nutrient assessments. The Internet of Things (IoT) is also evolving, integrating IoT with sensors to form multiple sensor network nodes capable of detecting a wide range of soil nutrient elements, thus creating a more intelligent sensing network to support agriculture better. Combining technologies such as artificial intelligence, big data, and machine learning can establish intelligent decision support systems that integrate soil nutrient data, crop growth data, and other information to help farmers develop scientifically sound fertilization plans. For smart agriculture to be widely adopted

36

Q. Yuan et al.

by farmers, it needs to be cost-effective and easy to operate, thus maximizing crop yield and quality.

References 1. Ahmad Jani HF, Meder R, Hamid HA, Razali SM, Yusoff KHM (2021) Near infrared spectroscopy of plantation forest soil nutrients in Sabah, Malaysia, and the potential for microsite assessment. J Near Infrared Spectrosc 29:148–157. https://doi.org/10.1177/096703352110 07673 2. Ali Md A, Mondal K, Wang Y, Jiang H, Mahal NK, Castellano MJ, Sharma A, Dong L (2017) In situ integration of graphene foam–titanium nitride based bio-scaffolds and microfluidic structures for soil nutrient sensors. Lab Chip 17:274–285. https://doi.org/10.1039/C6LC01 266C 3. Ali Md A, Wang X, Chen Y, Jiao Y, Mahal NK, Moru S, Castellano MJ, Schnable JC, Schnable PS, Dong L (2019) Continuous monitoring of soil nitrate using a miniature sensor with poly(3-octyl-thiophene) and molybdenum disulfide nanocomposite. ACS Appl Mater Interfaces 11:29195–29206. https://doi.org/10.1021/acsami.9b07120 4. Ali MdA, Dong L, Dhau J, Khosla A, Kaushik A (2020) Perspective—electrochemical sensors for soil quality assessment. J Electrochem Soc 167:037550. https://doi.org/10.1149/1945-7111/ ab69fe 5. Aliah BSN, Kodaira M, Shibusawa S (2013) Potential of Visible-Near Infrared Spectroscopy for mapping of multiple soil properties using real-time soil sensor. In: Kondo N (ed) Presented at the SPIE SeTBio, Yokohama, Japan, p 888107. https://doi.org/10.1117/12.2031009 6. Artigas J, Beltran A, Jiménez C, Baldi A, Mas R, Dom´ınguez C, Alonso J (2001) Application of ion sensitive field effect transistor based sensors to soil analysis. Comput Electron Agric 31, 281–293. https://doi.org/10.1016/S0168-1699(00)00187-3 7. Bakker E, Pretsch E, Bühlmann P (2000) Selectivity of potentiometric ion sensors. Anal Chem 72:1127–1133. https://doi.org/10.1021/ac991146n 8. Bricklemyer RS, Brown DJ, Barefield JE, Clegg SM (2011) Intact soil core total, inorganic, and organic carbon measurement using laser-induced breakdown spectroscopy. Soil Sci Soc Am J 75:1006–1018. https://doi.org/10.2136/sssaj2009.0244 9. Chen M, Zhang M, Wang X, Yang Q, Wang M, Liu G, Yao L (2020) An all-solid-state nitrate ion-selective electrode with nanohybrids composite films for in-situ soil nutrient monitoring. Sensors 20:2270. https://doi.org/10.3390/s20082270 10. Criscuolo F, Taurino I, Stradolini F, Carrara S, De Micheli G (2018) Highly-stable Li+ ionselective electrodes based on noble metal nanostructured layers as solid-contacts. Anal Chim Acta 1027:22–32. https://doi.org/10.1016/j.aca.2018.04.062 11. Dimkpa C, Bindraban P, McLean JE, Gatere L, Singh U, Hellums D (2017) Methods for rapid testing of plant and soil nutrients. In: lichtfouse E (ed) Sustainable agriculture reviews, sustainable agriculture reviews. Springer International Publishing, Cham, pp 1–43. https://doi. org/10.1007/978-3-319-58679-3_1 12. Gholizade A, Soom MAM, Saberioon MM (2013) Visible and near infrared reflectance spectroscopy to determine chemical properties of paddy soils 13. Han D, Joe YJ, Ryu J-S, Unno T, Kim G, Yamamoto M, Park K, Hur H-G, Lee J-H, Nam S-I (2018) Application of laser-induced breakdown spectroscopy to Arctic sediments in the Chukchi Sea. Spectrochim Acta Part B 146:84–92 14. He Y, Liu X, Lv Y, Liu F, Peng J, Shen T, Zhao Y, Tang Y, Luo S (2018) Quantitative analysis of nutrient elements in soil using single and double-pulse laser-induced breakdown spectroscopy. Sensors 18:1526. https://doi.org/10.3390/s18051526 15. Juvé V, Portelli R, Boueri M, Baudelet M, Yu J (2008) Space-resolved analysis of trace elements in fresh vegetables using ultraviolet nanosecond laser-induced breakdown spectroscopy. Spectrochim Acta Part B 63:1047–1053

2 Recent Advances in Soil Nutrient Monitoring: A Review

37

16. Kashyap B, Kumar R (2021) Sensing methodologies in agriculture for soil moisture and nutrient monitoring. IEEE Access 9:14095–14121. https://doi.org/10.1109/ACCESS.2021.3052478 17. Kim E-A (2018) Changes in the mineral element compositions of soil colloidal matter caused by a controlled freeze-thaw event 18. Kim, H-J, Hummel J, Birrell S (2006) Evaluation of nitrate and potassium ion-selective membranes for soil macronutrient sensing. Trans ASABE 49:597–606. https://doi.org/10. 13031/2013.20476 19. Kim H-J, Sudduth KA, Hummel JW (2009) Soil macronutrient sensing for precision agriculture. J Environ Monit 11:1810. https://doi.org/10.1039/b906634a 20. Lee WS, Alchanatis V, Yang C, Hirafuji M, Moshou D, Li C (2010) Sensing technologies for precision specialty crop production. Comput Electron Agric 74:2–33. https://doi.org/10.1016/ j.compag.2010.08.005 21. Liu R-T, Tao L-Q, Liu B, Tian X-G, Mohammad M, Yang Y, Ren T-L (2016) A miniaturized on-chip colorimeter for detecting NPK elements. Sensors 16:1234. https://doi.org/10.3390/s16 081234 22. Martin MZ, Wullschleger SD, Garten CT, Palumbo AV (2003) Laser-induced breakdown spectroscopy for the environmental determination of total carbon and nitrogen in soils. Appl Opt 42:2072. https://doi.org/10.1364/AO.42.002072 23. McCarty GW, Reeves JB (2006) Comparison of near infrared and mid infrared diffuse reflectance spectroscopy for field-scale measurement of soil fertility parameters. Soil Sci 171(2):94–102. https://doi.org/10.1097/01.ss.0000187377.84391.54 24. Miziolek AW, Palleschi V, Schechter I (2006) Laser induced breakdown spectroscopy (LIBS): fundamentals and applications. Cambridge University Press, Cambridge, UK 25. Mohamed ES, Baroudy AAE, El-beshbeshy T, Emam M, Belal AA, Elfadaly A, Aldosari AA, Abdelraouf AM, Lasaponara R (2020) Vis-NIR spectroscopy and satellite landsat-8 oli data to map soil nutrients in arid conditions: a case study of the northwest coast of Egypt. Remote Sens 12:3716. https://doi.org/10.3390/rs12223716 26. Moonrungsee N, Pencharee S, Jakmunee J (2015) Colorimetric analyzer based on mobile phone camera for determination of available phosphorus in soil. Talanta 136:204–209. https://doi.org/ 10.1016/j.talanta.2015.01.024 27. Munawar AA, Yunus Y, Devianti D, Satriyo P (2021) Agriculture environment monitoring: rapid soil fertility evaluation by means of near infrared spectroscopy. IOP Conf Ser: Earth Environ Sci 644:012036. https://doi.org/10.1088/1755-1315/644/1/012036 28. Paczosa-Bator B (2015) Ion-selective electrodes with superhydrophobic polymer/carbon nanocomposites as solid contact. Carbon 95:879–887. https://doi.org/10.1016/j.carbon.2015. 09.006 29. Peng Y, Zhao L, Hu Y, Wang G, Wang L, Liu Z (2019) Prediction of soil nutrient contents using visible and near-infrared reflectance spectroscopy. IJGI 8:437. https://doi.org/10.3390/ ijgi8100437 30. Pilon-Smits EA, Quinn CF, Tapken W, Malagoli M, Schiavon M (2009) Physiological functions of beneficial elements. Curr Opin Plant Biol 12:267–274. https://doi.org/10.1016/j.pbi.2009. 04.009 31. Postolache S, Sebastião P, Viegas V, Postolache O, Cercas F (2022) IoT-based systems for soil nutrients assessment in horticulture. Sensors 23:403. https://doi.org/10.3390/s23010403 32. Radziemski LJ, Cremers DA (2006) Handbook of laser induced breakdown spectroscopy. Wiley, New York, NY, USA 33. Ren GX, Wei ZQ, Fan PP, Wang XY (2019) Visible/near infrared spectroscopy method applied research in wetland soil nutrients rapid test. IOP Conf Ser: Earth Environ Sci 344:012123. https://doi.org/10.1088/1755-1315/344/1/012123 34. Rodríguez-Pérez JR, Marcelo V, Pereira-Obaya D, García-Fernández M, Sanz-Ablanedo E (2021) Estimating soil properties and nutrients by visible and infrared diffuse reflectance spectroscopy to characterize vineyards. Agronomy 11:1895. https://doi.org/10.3390/agronomy1 1101895

38

Q. Yuan et al.

35. Sacko BD, Sanogo S, Konare H, Ba A, Diakite T (2018) Capability of visible-near infrared spectroscopy in estimating soils carbon, potassium and phosphorus. OPJ 08:123–134. https:// doi.org/10.4236/opj.2018.85012 36. Sedaghat S, Jeong S, Zareei A, Peana S, Glassmaker N, Rahimi R (2019) Development of a nickel oxide/oxyhydroxide-modified printed carbon electrode as an all solid-state sensor for potentiometric phosphate detection. New J Chem 43:18619–18628. https://doi.org/10.1039/ C9NJ04502C 37. Senesi GS (2014) Laser-induced breakdown spectroscopy (LIBS) applied to terrestrial and extraterrestrial analogue geomaterials with emphasis to minerals and rocks. Earth Sci Rev 139:231–267. https://doi.org/10.1016/j.earscirev.2014.09.008 38. Senesi GS, Senesi N (2016) Laser-induced breakdown spectroscopy (LIBS) to measure quantitatively soil carbon with emphasis on soil organic carbon. A review. Analytica Chimica Acta 938:7–17. https://doi.org/10.1016/j.aca.2016.07.039 39. Singh K, Majeed I, Panigrahi N, Vasava HB, Fidelis C, Karunaratne S, Bapiwai P, Yinil D, Sanderson T, Snoeck D, Das BS, Minasny B, Field DJ (2019) Near infrared diffuse reflectance spectroscopy for rapid and comprehensive soil condition assessment in smallholder cacao farming systems of Papua New Guinea. CATENA 183:104185. https://doi.org/10.1016/j.cat ena.2019.104185 40. Takahashi T, Thornton B (2017) Quantitative methods for compensation of matrix effects and self-absorption in Laser Induced Breakdown Spectroscopy signals of solids. Spectrochim Acta, Part B 138:31–42. https://doi.org/10.1016/j.sab.2017.09.010 41. Tan B, You W, Tian S, Xiao T, Wang M, Zheng B, Luo L (2022) Soil nitrogen content detection based on near-infrared spectroscopy. Sensors 22:8013. https://doi.org/10.3390/s22208013 42. Wang T, He M, Shen T, Liu F, He Y, Liu X, Qiu Z (2018) Multi-element analysis of heavy metal content in soils using laser-induced breakdown spectroscopy: a case study in eastern China. Spectrochim Acta Part B 149:300–312 43. Wilding LP, Lin H (2006) Advancing the frontiers of soil science towards a geoscience. Geoderma 131:257–274. https://doi.org/10.1016/j.geoderma.2005.03.028 44. Xu X, Du C, Ma F, Shen Y, Zhou J (2019) Fast and simultaneous determination of soil properties using laser-induced breakdown spectroscopy (LIBS): a case study of typical farmland soils in China. Soil Syst 3:66. https://doi.org/10.3390/soilsystems3040066 45. Yan XT, Donaldson KM, Davidson CM, Gao Y, Wu H, Houston AM, Kisdi A (2018) Effects of sample pretreatment and particle size on the determination of nitrogen in soil by portable LIBS and potential use on robotic-borne remote Martian and agricultural soil analysis systems. RSC Adv 8:36886–36894. https://doi.org/10.1039/C8RA07065B 46. Yoon JH, Park HJ, Park SH et al (2020) Electrochemical characterization of reduced graphene oxide as an ion-to-electron transducer and application of screen-printed all-solidstate potassium ion sensors. Carbon Lett 30:73–80. https://doi.org/10.1007/s42823-019-000 72-6 47. Yunus Y, Devianti, Satriyo P, Munawar AA (2019) Rapid prediction of soil quality indices using near infrared spectroscopy. IOP Conf Ser: Earth Environ Sci 365:012043. https://doi. org/10.1088/1755-1315/365/1/012043 48. Zaytsev SM, Krylov IN, Popov AM, Zorov NB, Labutin TA (2018) Accuracy enhancement of a multivariate calibration for lead determination in soils by laser induced breakdown spectroscopy. Spectrochim Acta, Part B 140:65–72. https://doi.org/10.1016/j.sab.2017.12.005 49. Zhang G, Song H, Liu Y, Zhao Z, Li S, Ren Z (2018) Optimization of experimental parameters about laser induced breakdown and measurement of soil elements. Optik 165:87–93. https:// doi.org/10.1016/j.ijleo.2018.03.125 50. Zhou P, Li M, Yang W et al (2021) Development and performance tests of an on-the-go detector of soil total nitrogen concentration based on near-infrared spectroscopy. Precision Agric 22:1479–1500. https://doi.org/10.1007/s11119-021-09792-0

Chapter 3

Plant Phenotyping Robot Platform Yuandan Yang, Han Li, Man Zhang, and Ruicheng Qiu

3.1 Overview of Crop Phenotyping Robots According to the United Nations World Population Prospects 2022 report, the world population reached 8 billion on November 15, 2022, and will exceed 10 billion in this century [1]. With the world’s population growing and environmental changes taking place, agriculture will face challenges such as climate hazards, loss of fertile land, and water scarcity. As a result, plant breeding scholars are working hard to create highyield crop varieties that can adapt to future climate conditions and resist new pests and diseases. Advancements in DNA sequencing and genotyping technologies have significantly improved marker-assisted selection and genome prediction-guided plant breeding. The success of these approaches relies on extensive genetic marker data. Therefore, it is crucial to acquire high-quality plant phenotype datasets to investigate genetic traits, including growth, yield, and fitness aspects [2]. Traditionally, the quantification of crop traits has relied on labor-intensive, time-consuming, and costly artificially destructive sampling methods. This manual approach involves several steps that requires human intervention, thereby introducing the potential for human error. Typically, plants are cut at fixed time points or specific phenological stages to measure their phenotypic characteristics, which restricts the ability to continuously monitor individual crop traits throughout the growing season. The challenging and damaging process of crop phenotyping has caused breeders to prioritize yield as the most essential trait. However, yield is often considered as a weakly inherited phenotype in crop breeding, making it challenging to achieve significant improvements through selection for a single trait [3]. Y. Yang · H. Li (B) · M. Zhang · R. Qiu Key Laboratory of Smart Agriculture System Integration, Ministry of Education, Beijing 100083, China e-mail: [email protected] Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture and Rural Affairs, China Agricultural University, Beijing, China © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 M. Zhang et al. (eds.), Sensing Technologies for Field and In-House Crop Production, Smart Agriculture 7, https://doi.org/10.1007/978-981-99-7927-1_3

39

40

Y. Yang et al.

Therefore, in addition to crop yield, there is an urgent demand for high-throughput and non-destructive measurements of multiple crop traits across diverse environmental conditions. Currently, there are four main types of high-throughput automated measurement devices: conveyor belt platforms, gantry platforms, UAV (Unmanned Aerial Vehicle) platforms, and autonomous mobile robot platforms. These technological advancements enable efficient and accurate assessment of crop traits without damaging the plants and enable comprehensive monitoring throughout various growth stages and environmental circumstances. The conveyor belt-type platform offers numerous advantages, such as high precision, immunity to external interference, and robust repeatability. However, this platform may be less efficient when deployed outdoors and could cause inadvertent damage to crops prone to breaking or lodging. Furthermore, its ability to evaluate crop traits under genuine outdoor conditions is limited. On the other hand, the gantry platform presents the benefit of accommodating multiple sensors, facilitating the precise acquisition of phenotype information with remarkable efficiency. However, installing and operating this platform outdoors in the long-term leads to increased costs for maintenance. Furthermore, its application is restricted to long-term fixed sample plots, posing challenges for migration to alternative locations. The reflective nature of the “gantry” support structure may cause sunlight reflection and shadows that could affect plant growth and sensor imaging quality. The UAV phenotyping platform has garnered considerable favor among field crop researchers owing to its remarkable high throughput and efficiency. However, it is essential to acknowledge that further enhancements are necessary to improve its accuracy in detecting individual crop phenotypes. In recent years, the mobile robot platform has emerged as a highly promising research direction for high-throughput phenotypic detection. It has numerous advantages, such as flexibility, versatility, minimal impact on crops, and enhanced plant phenotype detection accuracy. These features make it an appealing and practical option for researchers in the field of crop phenotyping. Autonomous mobile robots, equipped with intelligent capabilities for autonomous navigation and obstacle avoidance, have been extensively utilized in the industrial domain [4]. However, autonomous robots face challenges in agriculture due to rugged operating terrains, unpredictable and changing environmental conditions. Robot systems in agricultural environment face higher navigation requirements due to these factors. Nonetheless, autonomous mobile robots have increasingly played a vital role in modern agriculture and are considered integral components of precision agriculture or digital agriculture [5]. The rapid development of the robot industry, coupled with reduced hardware costs and the market’s demand for efficient agricultural operations, has further accelerated the progress of agricultural robots [6]. Regarding automatic crop phenotype detection, the integration of advanced sensor technology and data processing algorithms allows robot systems to efficiently gather essential morphological and physiological phenotype information of crops, such as plant height, stem diameter, leaf width, length, and angle. As a result, autonomous robotic systems, along with advanced imaging and sensing capabilities,

3 Plant Phenotyping Robot Platform

41

are regarded as indispensable components of high-throughput plant phenotyping. This integration significantly enhances the capacity, speed, coverage, repeatability, and cost-effectiveness of plant trait measurements.

3.2 Design Requirements of Crop Phenotyping Robot Platform The phenotyping robot platform serves as a specialized sensor carrier, enabling the quickly acquisition of crop phenotype information through a comprehensive detection system. Various precision instruments, including color cameras, Light Detection and Ranging (LiDAR) sensors, depth cameras, spectral sensors, spectral cameras, thermal imagers, and fluorescence sensors, are commonly employed for crop phenotype monitoring [11]. Due to the demanding nature of this application, the a stable environment is paramount. Therefore, the phenotyping robot requires exceptional balance and shock absorption capabilities to maintain a stable environment. Furthermore, due to the diverse range of crop types and variations in plant heights, the phenotyping robot platform must be capable of adjusting its height, accommodate different plant heights and variations in crop types. In order to design a crop phenotype monitoring robot, it is essential to meet the following essential criteria [12]: (1) Optimal Movement Performance: The robot needs ample power, adjustable driving speed within specific ranges, and balance capabilities for smooth operation indoors and in the field, ensuring a secure and stable setting for sensor functionality. (2) Sufficient Bearing Capacity: The robot should be capable of securely carrying an array of sensors, ensuring their dependable installation. (3) Broad Applicability: The robot’s design should enable its use across diverse crop planting arrangements and various plant heights. (4) Enhanced Data Collection Efficiency: The robot should offer the capacity to monitor phenotype information from multiple rows of crops simultaneously, thus enabling efficient data collection. In essence, the phenotyping robot platform plays a crucial role in facilitating advanced crop phenotype monitoring through its sensor integration, movement capabilities, and adaptability to varying crop conditions.

3.3 Research Status of Crop Phenotyping Robots The automation of the crop phenotype measurement process can be achieved through various approaches, including conveyor belt-type, gantry crane-type, unmannedtype, and vehicle-type methods [13]. These methods can be broadly categorized as either mobile sensor-based or mobile crop-based approaches. In essence, the

42

Y. Yang et al.

phenotypic characteristics of crops, such as plant height, stem diameter, and leaf area, require sensors to capture data from multiple angles of the plant. Moreover, in scenarios involving repeated measurements of crop traits across larger populations, data collection at multiple time points during the growth season is essential. In these cases, phenotyping robotic systems emerge as highly effective solutions, offering the requisite speed and precision for such phenotyping tasks. Robotic platforms designed for crop phenotype measurement can be classified into two primary categories [14]: those tailored for indoor or controlled environments (such as greenhouses or laboratories) and those optimized for outdoor settings (such as fields). Each category presents distinct challenges and considerations.

3.3.1 Robot Platform for Indoor Crop Phenotype In the realm of indoor phenotypic platform design, two main approaches are commonly pursued. The first method involves placing crops in fixed positions while robots navigate around the facility to engage with the plants. The second method focuses on transporting crops to predetermined positions using conveyor belts or other automated mechanisms operated by robots [15]. Both strategies share the common goal of enabling sensors to gather comprehensive crop phenotype data from various perspectives. Depending on the type of phenotypic information required, the acquisition process can be further categorized into contact and non-contact methods. Non-contact sensors are typically employed for capturing most phenotypic information, including plant height, stem diameter, and leaf area. RGB cameras and depth sensors (such as Time-of-Flight cameras or 3D laser scanners) are often utilized to capture plant images or point clouds, which are then used to estimate the morphological traits of the plants. For instance, Chaudhury et al. [16] employed a gantry robot system comprising a 7-DOF manipulator equipped with a 3D laser scanner to measure the surface area and volume of Arabidopsis and barley. Wu et al. [17] introduced an automated multi-robot system, each armed with a depth camera to collect point cloud data of crops. Lu et al. [18] employed a robotic arm equipped with a ToF camera to perform 3D reconstruction of stationary corn plants and extract phenotypic traits such as plant height and stem diameter from the resulting 3D model. Peng et al. [19] achieved the extraction of parameters like plant height, stem diameter, and leaf area of individual tomato seedlings in a greenhouse using an autonomous mobile robot equipped with an Aubo-i5 robotic arm. In addition to crop characterization traits, intrinsic biochemical characteristics like chlorophyll concentration, water content, and ion concentration hold significant value for crop breeding. Such data necessitate contact sensors [20]. When direct interaction between sensors and crops is required, robot arms can be equipped with graspers and appropriate sensors to mimic human manipulation of crops and acquire corresponding phenotypic information from target crop organs [21]. For example, Bao et al. [22] devised a robot arm equipped with RGB sensors, hyperspectral sensors,

3 Plant Phenotyping Robot Platform

43

Table 3.1 Robotic platforms for successfully measuring indoor crop phenotypic information Researcher

Species

Plant trait

Software system

Alenyà Ribas et al. [23]

Anthurium, green flowers

Leaf chlorophyll content

ROS 1

Lu et al. [18]

maize

Plant height, leaf length

Qt

Chaudhury et al. [16]

Arabidopsis, barley

Surface area, volume

not recorded

Atefi et al. (2019)

Corn, sorghum

water content, temperature, etc

MATLAB

Peng et al. [35]

tomato

Plant height, stem diameter, etc

ROS 1

thermal sensors, TOF camera, and a fluorometer to measure the physiological parameters of crops. The robotic arm measured crop reflection spectra, temperature, and fluorescence by imaging leaves and positioning the probe millimeters away from the leaf surface. Alenya et al. [23] integrated a near-infrared SPAD single-photon detector onto a robot arm, enabling direct measurement of chlorophyll concentration in flower petals and green fruit leaves. Atefi et al. [24] developed a robot system with a TOF camera, a four-degree-of-freedom robot manipulator, and a custom fixture. This system controlled the robot arm to grip points on crop leaves using image processing and deep learning. Simultaneously, fiber optic cables, thermistors, and linear potentiometer sensors were employed to gather leaf hyperspectral reflectance, leaf temperature, and stem diameter data, facilitating the creation of predictive models for leaf chlorophyll content, water content, and nutrient concentrations. In summary, indoor phenotyping platforms encompass diverse methods, including mobile sensor-based and mobile crop-based approaches. These methodologies enable data collection from various angles, and non-contact and contact sensors are leveraged to extract morphological and biochemical traits of crops. Successful implementation of these techniques yields valuable insights for crop research and breeding. Table 3.1 summarizes indoor robotic systems that have successfully measured crop phenotypic information in recent years.

3.3.2 Field Crop Phenotyping Robot Platform The high-throughput collection of phenotypic information from field crops holds significant research value. While cultivating and quantifying crop traits indoors is relatively straightforward, substantial differences exist between indoor and fieldgrown crops [25]. Phenotypic data obtained from field conditions offers a more direct and reliable foundation for crop enhancement efforts. Field robotic systems typically employ crop rows for navigation, but variables like temperature, sunlight, wind, and uneven terrain present considerable challenges for navigation and data collection in outdoor settings. As a result, outdoor phenotyping robots must possess

44

Y. Yang et al.

a range of capabilities to function effectively in such environments. These include adaptation to field conditions, autonomous navigation, and obstacle avoidance, plant recognition and parameter measurement, and a high level of robustness and safety. To address the complexities of diverse field terrains and navigation hurdles, Zhang et al. [26] developed a high-precision path-tracking approach using a vehicle robot capable of managing unknown wheel-terrain interactions. This system counted corn plants via target detection method. Shafiekhani et al. [27] introduced Vinobot, a semiself-navigating robot that autonomously moves based on crop behavior reference. This robot can autonomously navigate after being manually aligned with the crop row. Vinobot features a 6-DOF robotic arm and a 3D imaging sensor mounted on a mobile platform, allowing the acquisition of point cloud data for 3D reconstruction of corn and sorghum plants to obtain phenotypic information. The substantial variations in illumination and temperature within field environments pose challenges in sensor selection and data processing for crop phenotyping. Abel et al. [28] incorporated a spectrometer into the robotic arm of the Robotanist to capture spectral reflectance measurements of sorghum leaves and stems. They utilized the RANSAC (Random Sample Consensus) method for leaf and stem detection and employed machine learning to estimate chlorophyll, water, and starch content. Baweja et al. [29] utilized a high-resolution stereo imager on the Robotanist platform to capture dense image data of a sorghum experimental field. Accurate stem diameter and plant height were then extracted through semantic segmentation. Choudhuri et al. [30] proposed two methods for estimating the stem width of crops using a small mobile robot: one using RGB data and the other using RGB + LIDAR data. Bao et al. [31] estimated phenotypic parameters of sorghum plants such as plant height, stem diameter, and volume through stereoscopic imaging using vision sensors mounted on tractor platforms. Qiu et al. [32] installed a laser radar on the AgriRover01 mobile robot platform, employing laser scanning to measure the height of corn plants across a large range. Sun Na et al. [33] also utilizing the AgriRover-01 platform, optimized feature extraction algorithms to successfully calculated maize row spacing and plant height data. Fan et al. [34] independently developed an ultra-narrow, high-throughput crop phenotyping data acquisition platform. This platform obtained corn point clouds using an RGB-D camera, enabling measurements of parameters such as stem diameter, leaf width, leaf length, leaf angle, and leaf rotation angle. In summary, the outdoor collection of field crop phenotypic information involves innovative approaches and sophisticated robotic systems. These systems tackle challenges posed by diverse outdoor conditions, navigation complexities, and sensor selection, providing valuable insights for advancing agricultural research and crop breeding. Table 3.2 summarizes recent outdoor robotic systems that have successfully measured phenotypic information about crops in the field.

3 Plant Phenotyping Robot Platform

45

Table 3.2 Robot platforms that successfully measured crop phenotype information in the field Researcher

Species

Plant trait

Software system

Shafiekhani et al. (2017) (Vinobot)

Corn, sorghum

Plant height

ROS 1

Abel et al. (2018) (Robotanist)

sorghum

Starch and water content in stems and chlorophyll content in leaves

ROS 1

Baweja et al. [29] (Robotanist)

sorghum

Stem number, stem thickness ROS 1

Choudhuri et al. [30]

sorghum

Stem diameter

Python

Bao et al. [22]

sorghum

Plant height, stem diameter, stem surface area, volume

Not recorded

Qiu et al. [32] (AgriRover-01)

Corn

plant height

ROS 1

Sun et al. [33] (AgriRover-01)

Corn

Crop row spacing, plant height

ROS 1

Zhang et al. [26]

Corn

Plant count

Not recorded

Fan et al. [34]

Corn

Stem diameter, leaf width, leaf length, leaf Angle, leaf rotation Angle

ROS 1

3.4 A Case Study of Tomato Plant Phenotype Inspection Robot 3.4.1 Hardware Components As per the design requirements outlined in Sect. 1.1.2, the robot platform’s structure is depicted in Fig. 3.2. The core industrial computer chosen for the system is the 8F377VGGA-TD embedded industrial mainboard. This mainboard operates on the Ubuntu 18.04 operating system, utilizing ROS Melodic for comprehensive system control. The navigation’s underlying controller, powered by an STM32 core, is developed to parse ROS instructions and transmit them to the motor driver. Positioned at the robot’s forefront, the LiDAR interacts directly with the industrial computer. The IMU is positioned at the center of the robot chassis. For maximum motion capabilities, the robot arm is mounted both above and at the rear of the chassis. The AUBO-i5 6-DOF manipulator is driven and controlled via the manipulator controller, maintaining interaction with the industrial computer. A Kinectv2 depth camera is affixed to the robotic arm’s end through a 3D-printed flange, facilitating direct interaction with the IPC for data exchange. To address potential interface limitations between the industrial computer and multiple devices, an Ethernet switch is integrated, enabling communication between the laser radar, mechanical arm, and the industrial computer. The navigation function and phenotype data acquisition function can be conveniently operated through the display screen connected to the industrial computer.

46

Y. Yang et al.

(a) Conveyor belt phenotyping platform [7]

(c) UAV phenotyping platform

[9]

(b) Gantry phenotyping platform[8]

(d) Mobile robot phenotyping platform[10]

Fig. 3.1 Various phenotyping platforms

The physical representation of the robot platform is illustrated in Fig. 3.3. The platform boasts ample power, adjustable and controlled driving speeds, exceptional obstacle clearance, and balance capabilities. It is capable of accommodating a diverse array of sensors on the robotic arm, rendering it adaptable to crops with varying planting row spacing and plant heights.

3.4.2 Software Design The software system designed for automated phenotypic data collection by robots in greenhouses encompasses two primary functions: automatic navigation and multiangle tomato point cloud acquisition facilitated by the robotic arm. Navigation relies on a pre-established two-dimensional grid map generated through SLAM (Simultaneous Localization and Mapping). The operational flow of the robotic automated phenotypic data collection process is depicted in Fig. 3.4. The procedure initiates by determining the starting and ending points for the phenotypic measurement operation based on the tomato plant’s position within the greenhouse. Subsequently, the robot generates its operational path within the map based on the established start and end points. To traverse multiple crop rows within

3 Plant Phenotyping Robot Platform

47

Fig. 3.2 Block diagram of robot hardware platform structure

Fig. 3.3 Robot hardware platform 1. Depth camera 2. Robot arm 3. Network switch 4. Industrial computer 5.IMU 6. Motor 7. Bottom controller 8. Lidar 9. Robot arm controller 10. Display screen

48

Y. Yang et al.

Fig. 3.4 Operation flow chart of the phenotypic measurement robot

(a) Identification and overview (b) Side view point cloud point cloud acquisition acquisition Fig. 3.5 Acquisition scenario of phenotypic measurement robot

(c) Side view point cloud acquisition

3 Plant Phenotyping Robot Platform

49

the greenhouse, the option of setting multiple end points is available. The inspection robot progresses sequentially to multiple target points. Concurrently, the robot arm transitions to its default crop detection position. Upon the depth camera located at the robot arm end entrance into the suitable area for phenotypic data collection, navigation pauses. The depth camera determines the crop position relative to its own, computing the robot arm’s collection point position and orientation based on this information. The robotic arm then proceeds to the calculated pose, positioning itself at the collection point. The depth camera captures both crop images and point clouds during this stage. Depending on the crop’s size and data requirements, the number of acquisition locations can be adjusted. Subsequent to collecting individual tomato phenotype data, navigation recommences until the robot reaches the designated endpoint, effectively concluding the automated phenotypic data acquisition task. This well-orchestrated process ensures an accurate and comprehensive collection of phenotypic information from the tomato plants within the greenhouse.

3.4.3 Extraction of Phenotype Information The extraction of crop phenotypic parameter information from 3D point clouds typically involves four key steps: multi-view point cloud registration, segmentation of individual plants, segmentation of plant organs, and measurement of phenotypic parameters. The method presented by Peng Cheng et al. [35] is described as an example: Multi-view Point Cloud Registration: The previously collected crop point cloud information was initially registered. This involves saving the pose of the robotic arm’s end effector during point cloud collection for coarse registration. Subsequently, point cloud registration was achieved using the Iterative Closest Point (ICP) method, resulting in a more complete tomato point cloud. Segmentation of Individual Tomato Plants and Organs: Firstly, a single crop plant point cloud was extracted from the fully registered point cloud by pass-through filtering. From the fully registered point cloud, the goal is to segment individual tomato plants and their corresponding stems and leaves. Given the complex 3D morphological structure of tomato plants and their variations, a fixed approach for segmentation is challenging. To address this, the complete tomato point cloud after registration was skeletonized. The main stem and petioles were segmented using the skeleton’s connection points. Tomato leaves, consisting of petioles and oddnumbered pinnate compound leaves, were segmented using region growing and clustering techniques. Phenotypic Parameter Measurement: The segmented leaf point clouds were used to extract phenotypic parameters such as plant height, stem diameter, leaf inclination angle, and leaf area. The plant height parameter was determined by the difference between the maximum and minimum z-axis coordinates of the plant point cloud. The stem diameter measurement involves selecting a fixed physiological position

50

Y. Yang et al.

Fig. 3.6 Process of obtaining crop phenotypic parameters

on the stem, and a stem fragment is extracted for measurement. Plant stem width was calculated from the maximum size of the bounding box in the x–y plane. The segmented leaves were triangulated using the greedy projection triangulation algorithm, and the leaf area is determined by summing the areas of individual triangular facets. The extraction steps of tomato plant phenotypic parameters above are shown in Fig. 3.6. Validation of Phenotypic Parameters: The accuracy of the extracted phenotypic parameters was validated against ground truth values. Plant height and stem diameter were measured manually. Leaf area was determined by capturing leaf images with known area markers. The true leaf area was then calculated using the number of leaf pixels, pixel size of the marker, and number of marker pixels. The relationship between the extraction results of specific parameters and the truth value was shown in Fig. 3.7. Figure 3.7 shows that the proposed method successfully extracted phenotypic information from 3D point clouds. The results of the phenotype parameter acquisition were compared against true values. Strong correlations were found between the extracted and measured values of plant height and leaf area. The coefficient of determination (R2) for plant height was 0.97, and the root mean square error (RMSE) was 1.40 cm. For leaf area, R2 was 0.87, and RMSE was 37.56 cm2. However, the measurement error of stem diameter parameter was larger (R2 = 0.53, RMSE = 1.52 mm), mainly attributed to the thin stem of tomato seedlings and the sensor’s inherent accuracy limitations.

3 Plant Phenotyping Robot Platform

51

Fig. 3.7 Results of crop phenotypic parameters

References 1. Zhiwei L (2022) United nations: world population reaches 8 billion [N]. People’s Daily 014 2. Chawade A, Ham JV, Blomquist H et al (2019) High-throughput field-phenotyping tools for plant breeding and precision agriculture[J]. Agronomy 9(5):258 3. Furbank RT, Tester M (2011) Phenomics—technologies to relieve the phenotyping bottleneck[J]. Trends Plant Sci 16(12):635–644 4. Shiyun Li, Chenghong Xu (2022) Research on the impact of industrial robots on regional industrial structure in China [J]. Areal Res Devel 41(01):6-12 5. Pandey P, Hemanth ND, Young SN (2021) Frontier: autonomy in detection, actuation, and planning for robotic weeding systems[J]. Trans ASABE 2:64 6. Li C, Shi W, Ji Z, Liu Y (2022) Analysis on research progress of Agricultural robot at home and abroad [J]. Southern Agricult Machin 53(05):156–158 (in Chinese) 7. Geng Bai, Yufeng Ge, Sarah Blecha, et al. (2015) Phenotyping transgenic wheat in a greenhouse through multispectral and thermal imaging[J] 8. Virlet N , Sabermanesh K , Sadeghitehran P et al. (2016) Field scanalyzer: an automated robotic field phenotyping platform for detailed crop monitoring[J].Functional Plant Biology, 44 9. Scott C , Torsten M , Amy C ,et al. (2014) Pheno-Copter: a low-altitude, autonomous remote-sensing robotic helicopter for high-throughput field-based phenotyping[J]. Agronomy 4(2):279–301 10. Murman JN (2019) Flex-Ro: A robotic high throughput field phenotyping system[J] 11. Ruicheng Qiu (2019) Research on field crop phenotype measurement based on vehicle platform [D]. China Agricultural University 12. Huali Yuan (2019) Structure design and implementation of crop phenotype monitoring robot [D]. Nanjing Agricultural University 13. Atefi A , Ge Y , Pitla S , et al. (2021) Robotic technologies for high-throughput plant phenotyping: contemporary reviews and future perspectives[J]. Front Plant Sci 14. Wang, Hailong, Peng,et al. (2015) Sensors, Vol. 15, Pages 11889–11927: Fruit Quality Evaluation Using Spectroscopy Technology: A Review[J] 15. Uchiyama H, Sakurai S, Mishima M, et al. (2017) An easy-to-setup 3D phenotyping platform for KOMATSUNA dataset[C]// 2017 IEEE International Conference on Computer Vision Workshop (ICCVW). IEEE 16. Chaudhury A , Ward C , Talasaz A , et al. (2017) Machine vision system for 3D plant phenotyping[J]. IEEE/ACM Transactions on Computational Biology & Bioinformatics, 99 17. Wu C, Zeng R, Pan J et al. (2019) Plant phenotyping by deep-learning-based planner for multi-robots[J]. IEEE Robot Autom Lett 99:1–1 18. Lu H, Tang L, Whitham SA, Mei Y (2017) A robotic platform for corn seedling morphological traits characterization[J]. Sensors 17(9):2082

52

Y. Yang et al.

19. Cheng Peng (2022) Design and development of greenhouse tomato phenotype measurement robot system [D]. China Agricultural University 20. Biskup B, Scharr H, Fischbach A et al (2009) Diel growth cycle of isolated leaf discs analyzed with a novel, high-throughput three-dimensional imaging method is identical to that of intact leaves[J]. Plant Physiol 149(3):1452–1461 21. Baranska SM (2007) Identification and quantification of valuable plant substances by IR and Raman spectroscopy[J]. Vibrat Spectr 22. Bao Y, Zarecor S, Shah D, et al. (2019) Assessing plant performance in the Enviratron[J]. Plant Methods 2019(c):1–14 23. Alenyà Ribas G, Dellen B, Foix Salmerón S et al. (2012) Robotic leaf probing via segmentation of range data into surface patches[J]. Recercat Home 24. Atefi A, Ge Y, Pitla SK et al. (2020) Robotic detection and Grasp of Maize and Sorghum: stem measurement with contact[J]. Robotics (3) 25. Sébastien Dandrifosse, Bouvry A, Leemans V et al. (2020) Imaging wheat canopy through stereo vision: overcoming the challenges of the laboratory to field transition for morphological features extraction[J]. Front Plant Sci 11:1 26. Zhang B (2020) High precision control and deep learning-based corn stand counting algorithms for agricultural robot[J]. Autonomous Robots 44(7) 27. Shafiekhani A, Fritschi FB, Desouza GN (2018) Vinobot and Vinoculer: from real to simulated platforms[C]// International Society for Optics and Photonics 28. Abel J (2019) in-field robotic leaf grasping and automated crop spectroscopy[J] 29. Baweja HS, Parhar T, Mirbod O et al. (2018) StalkNet: A Deep Learning Pipeline for HighThroughput Measurement of Plant Stalk Count and Stalk Width[J] 30. Choudhuri A, Chowdhary G (2018) Crop stem width estimation in highly cluttered field environment, in Proc.2018. Comput[J] Plant Phenotyping, 6–13 31. Bao Y, Tang L, Breitzman MW et al (2019) Field-based robotic phenotyping of sorghum plant architecture using stereo vision[J]. J Field Robot 36(2):397–415 32. Qiu Q, Sun N, Bai H et al (2019) Field-based high-throughput phenotyping for maize plant using 3D LiDAR point cloud generated with a “phenomobile"[J]. Frontiers in Plant Ence 10:554 33. Na Sun (2019) Research on high-throughput phenotypic information collection system of field maize [D]. Agricultural University of Hebei 34. Zhengqiang Fan (2022) Study on high-throughput phenotype parameter acquisition of field maize based on robot mobile platform [D]. Northwest A&F University 35. Peng Cheng, Li Shuai, Miao Yanlong et al. (2022) Stem and leaf segmentation and phenotypic extraction of tomato plants based on three-dimensional point cloud [J]. Trans Chinese Soc Agricul Eng 38(09):187-194

Chapter 4

Autonomous Crop Image Acquisition System Based on ROS System Yida Li, Han Li, Liuyang Wang, and Man Zhang

Abstract The research on monitoring methods for potato water stress plays an important role in improving the growth quality and yield of potatoes. The crop water stress index (CWSI) can be calculated based on thermal infrared images and visible light images, which allows for the determination of crop water stress levels. In order to achieve automated, high-throughput, non-destructive continuous acquisition of visible light, thermal infrared images, and raw temperature data, this study integrated and developed an indoor crop inspection system with binocular cameras and thermal infrared cameras. According to different functional requirements, the hardware components are designed with modularization, including the main control module, motion control module, image acquisition module, and real-time monitoring module. The main control module realizes control over the image acquisition module for capturing images, transmission and storage of images and temperature data, and data interaction with the motion control module. Multiple nodes are designed based on the Robot Operating System (ROS) architecture to facilitate communication among different module nodes using a publish/subscribe pattern, including the motion control node, thermal infrared camera node, and binocular camera node. Feasibility tests conducted in the laboratory for one day demonstrate that the motion module control accuracy meets the requirements of image acquisition. The modules and node control programs work together to achieve image acquisition and local storage. Through stable testing and application in a greenhouse for 18 days, a total of 648 inspections were conducted, with each inspection taking 3 min. This resulted in 11,664 visible light images, 5,832 thermal infrared images, and 5,832 sets of raw temperature data. The tests confirm the stable operation of the system and normal acquisition of images and temperature data, demonstrating the system’s capability for automated, high-throughput, non-destructive data acquisition. Furthermore, off-line processing of the data obtained by the inspection system extracts three-dimensional Y. Li · M. Zhang Key Laboratory of Smart Agriculture System Integration, Ministry of Education, China Agricultural University, Beijing 100083, China H. Li (B) · L. Wang Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture and Rural Affairs, China Agricultural University, Beijing 100083, China e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 M. Zhang et al. (eds.), Sensing Technologies for Field and In-House Crop Production, Smart Agriculture 7, https://doi.org/10.1007/978-981-99-7927-1_4

53

54

Y. Li et al.

point cloud data of the potato canopy, which contains color information, temperature information, and the CWSI index. This indicates that the system provides an effective technical and equipment support for obtaining crop phenotype information. Keywords Image acquisition · Crop phenotype · Water stress · Robot operating system · Potato

4.1 Overview of the Indoor Crop Inspection System 4.1.1 Research Status of Indoor Crop Inspection Systems The agricultural land is rapidly decreasing, the aging population is increasing, and the adverse global climate conditions are worsening, leading to an increasingly prominent food problem [1]. The cultivation of high-yielding crop varieties is an effective approach to address this issue. In recent years, there has been rapid technological development in the field of plant genome information analysis, which plays a crucial role in high-yield breeding. However, the current methods for obtaining crop phenotype information mostly rely on manual measurements, which are inefficient and limited in sample size. These limitations make it challenging to perform phenotype detection on a large scale of crop populations. This limitation restricts the timeliness of data acquisition and the development of breeding techniques [2–6]. Automated, high-throughput, and non-destructive monitoring of the Crop Water Stress Index (CWSI) for potato crops can contribute to drought-resistant breeding research. The calculation of CWSI can be achieved through the processing of visible light images and thermal infrared images of crops [7]. Therefore, an automated image acquisition system can provide an automated, high-throughput, and non-destructive method for obtaining moisture stress data for potato crops [8], significantly improving the efficiency of phenotype information acquisition [9–11]. The automated image acquisition system is mainly based on images captured by RGB cameras, multispectral cameras, hyperspectral cameras, thermal infrared cameras, etc., and is used for the three-dimensional reconstruction of crop canopies [12–14], the acquisition of morphological characteristics and various nutritional traits of crop harvesting organs [15], and the detection of biotic and abiotic stress-related states and traits of crops [16, 17]. Two-dimensional images contain rich crop phenotype information [18] and can obtain morphological parameters of important plant organs. However, incomplete crop information acquisition may occur due to leaf occlusion [9]. By utilizing the image acquisition system to obtain multiple sets of two-dimensional images and performing three-dimensional reconstruction techniques, the physical morphology of crops such as plant height, stem diameter, and leaf area can be obtained, which has become a research hotspot [12–14]. Currently, the measurement efficiency of ground-based CWSI in field environments is low, and automatic rapid detection has not been achieved, which hinders

4 Autonomous Crop Image Acquisition System Based on ROS System

55

the study of the spatiotemporal variation patterns of CWSI in field environments, the variation patterns of CWSI during different growth stages and diurnal variations, as well as the optimal detection position of CWSI. Indoor phenotype studies can conduct various precise simulated grading experiments on crops under complex experimental conditions by controlling the growth environment, enabling quantitative and qualitative analysis of crops [19, 20]. Therefore, to study the spatiotemporal variation patterns of CWSI in potato crops, it is necessary to develop an indoor crop inspection system. Many domestic and foreign research institutions and companies have developed specific systems to meet the requirements of various phenotype acquisitions. Existing commercialized automated image acquisition platforms include Scanalyzer HTS, HyperAlxpert, and Scanalyzer 3D developed by Lemna Tec in Germany [21], large conveyor-type indoor phenotyping platforms such as WPScan Conveyor developed by WPS in the Netherlands, and devices such as PlantEye F600 and FieldScanD developed by Phenospex in the Netherlands. The advantages of these platforms are the integration of multiple sensors to obtain a large amount of high-precision plant phenotype data and extract specific parameters, but they are expensive. Domestic researchers have also conducted research on automated image acquisition platforms. Guo Qinghua et al. developed a medium-sized crop three-dimensional phenotyping measurement platform that integrates various types of sensors with a laser radar sensor as the core. It can take photos of any number of crops within the platform range in a downward manner by setting the working mode, and obtain various imaging data [22]. Zhang et al. developed a 3D (Three Dimensions) robot system for crop phenotype acquisition by integrating multiple sensors at specific positions and automatically capturing spectral images of crops at specific time intervals by triggering the system [23]. Wu et al. developed a phenotype platform, MVS-Pheno V2, to address the challenge of complex three-dimensional reconstruction of small-sized crops, providing a feasible solution for obtaining phenotype information of individual lowstem crops [13]. However, the above-mentioned platforms are not suitable for the automatic acquisition of CWSI in potato crops. Therefore, there is an urgent need to develop a low-cost, domestically produced, high-throughput crop image acquisition system specifically designed for extracting CWSI parameters in potatoes. This paper aims to develop an indoor crop inspection system based on the Robot Operating System (ROS) architecture and programmed in C ++ . The system integrates stereo cameras and thermal infrared cameras, enabling automated inspection and scheduled photography of multiple plants. It can capture two sets of visible light images, one set of thermal infrared images, and raw temperature data, providing technical support for high-throughput, non-destructive, and automated extraction of CWSI parameters in potato crops.

4.1.2 Indoor Crop Inspection System Overall Design Due to the need for potato crop water stress research, it is necessary to calculate the average temperature of the plant canopy. The required canopy temperature data

56

Y. Li et al.

is extensive, graded, and collected over multiple time periods throughout the day. However, there are several unfavorable factors in the greenhouse environment, such as limited operating space, high humidity in the morning and evening, and inconvenient wiring. To meet the data processing requirements and address these challenges, a small-scale, high-throughput, non-destructive, and automated inspection system for 3D image acquisition was designed. The platform primarily consists of four components: the main control module, motion control module, image acquisition module, and real-time monitoring module. The physical design of the indoor crop inspection system is shown in Fig. 4.1. The main control module utilizes the Jetson Nano (NVIDIA, USA) as the main controller, running the Ubuntu 18.04 operating system with ROS. The motion control module consists of three stepper motor drivers, an aluminum alloy structural framework, a sliding table module, and an STM32 microcontroller (STMicroelectronics, Italy). The image acquisition module integrates a stereo camera and a thermal infrared camera. The real-time monitoring module includes a host computer and various software specifically developed for high-throughput crop phenotype acquisition. The system is built on the ROS software architecture and creates three nodes: the motion control node, thermal infrared camera node, and stereo camera node. Through the publish/subscribe communication pattern, these nodes receive and publish messages on specific topics, ultimately achieving automatic crop image acquisition in the system.

Fig. 4.1 Indoor crop inspection system overall design of the physical map a Jetson Nano b Aluminum alloy structural framework c (1) Sliding table module (2) Closed-loop drive integrated motor (3) Motion control development board d (1) Thermal infrared camera (2) Stereo RGB camera

4 Autonomous Crop Image Acquisition System Based on ROS System

57

4.2 System Hardware Design The hardware design framework of the indoor crop inspection system is shown in Fig. 4.2. The hardware connections for implementing the inspection and photography functions mainly involve the use of the Jetson Nano main controller, which connects to the stereo RGB camera via a USB serial port. The thermal infrared camera and the STM32 microcontroller are connected to the Jetson Nano main controller via an RJ45 Ethernet port and a switch. The STM32 microcontroller communicates with the three integrated motor stepper drive units through RS232 serial communication. The PC connects to the main control module via an RJ45 Ethernet port and the switch, allowing real-time monitoring and control of the motion module, image acquisition module, and main control module through software. The inspection system is programmed to set fixed positions for capturing images of the potato plant canopy. The visible light, thermal infrared images, and raw temperature data are stored in a designated local folder, providing a large amount of reliable image and temperature data for subsequent research on potato water stress.

24V-5V

Master controller (Jetson Nano) RJ45 RJ45

PC

Network switch 24V-5V

RJ45 TCP/IP USB3.0 UVC

RJ45

W5500 Ethernet module Spi Gpio Motion controller STM32

Image collection module

binocular camera

Thermal infrared camera

RS232 Modbus

Motor 1

RS232 Modbus

Motor 2

RS232 Modbus

Motor 3 DC/DC module

Motion control module

24V-12V

ROS platform publish / subscribe messages

Fig. 4.2 Hardware design framework of indoor crop inspection system

58

Y. Li et al.

4.2.1 Main Control Module The main control module is responsible for controlling the image capture module, transmitting and saving image and temperature data, ensuring accurate and fast data communication with the motion control module, and enabling real-time monitoring and control of the entire system through a host computer. Among the mainstream embedded microprocessors, Raspberry Pi and Jetson Nano are commonly used. To meet the performance requirements of the system, Jetson Nano, which has the advantage of a graphics processing unit (GPU), is chosen as the main control microprocessor. It has dimensions of 100 mm × 80 mm × 29 mm, a quad-core ARM A57 CPU running at 1.43 GHz, a 128-core Maxwell GPU, 4 GB of 64-bit LPDDR4 memory with a bandwidth of 25.6 GB/s, four USB ports, one Ethernet port, and two HDMI ports. Please refer to Fig. 4.1a for a physical illustration. The Jetson Nano development board is equipped with Ubuntu 18.04 operating system and runs on ROS (Robot Operating System). Communication and control of other modules within the ROS architecture, as well as communication between different modules, are implemented. The Jetson Nano can be powered in three ways: via MicroUSB (5 V/2A), DC power (5 V/4A), or pin power (5 V/3A). Considering the additional modules such as the USB stereo camera, the MaxN (10W) power mode is selected, and the DC power interface is used. The 24 V power supply from the distribution box is converted to 5 V for the Jetson Nano using a 24 V to 5 V DC/ DC module. Additionally, a jumper cap is placed on J48 to ensure a stable operating voltage for the Jetson Nano during system operation. The Jetson Nano is enclosed in a suitable frame designed and 3D printed using Solidworks software and a 3D printer, as shown in Fig. 4.3. It is installed above the Z-axis of the sliding module for easy connection with the data cable of the image capture module and for streamlined wiring.

( a ) Shell rendering Fig. 4.3 Master module package

( b ) Physical package diagram

4 Autonomous Crop Image Acquisition System Based on ROS System

59

During the process of inspection image collection, it is necessary to perform visual operations on the system. Jetson Nano can be connected to a display screen via an HDMI cable, or connected to a PC using Nomachine software for command control and viewing of various image data collected and saved locally.

4.2.2 Motion Control Module The research on potato water stress requires setting different moisture gradients for the crops. It is important to ensure that the indoor crop inspection system can accurately capture canopy images of multiple plants simultaneously. Adequate workspace, precise point-to-point movement, and accurate and fast data communication are the requirements for the motion control module. The system selects an aluminum alloy structural frame as the supporting framework, as shown in Fig. 4.1b. In order to capture complete visible light and thermal infrared images of a single potato canopy, the aluminum alloy structural frame is designed as a rectangular prism composed of 14 profiles, with dimensions of 1.8 m (length), 1.8 m (width), and 1.5 m (height). Two of the profiles are fixed above the frame to connect the sliding platform module, and four directional wheels are installed at the four corners of the platform for easy movement. The system achieves spatial point-to-point movement of the platform through a three-dimensional sliding platform module (CCM Remote Technology Co., Ltd., China), as shown in Fig. 4.1c (1). The sliding platform module consists of CCMW4010 modules, motor mounting flanges, couplings, footings, photoelectric switches, and other components. The horizontal dimensions of the two axes are 1.5 m, and the vertical dimension is 1 m, with a maximum payload of 10 kg. Three CCMW40-10 modules are connected together by sliders, and the slider on the vertical axis is used to install the main control module and image acquisition module. The motors drive the sliding platform module through motor mounting flanges and couplings. The sliding platform module is fixed on the aluminum alloy structural frame with footings. The photoelectric switch operates at a voltage of 24 V and can detect objects as small as 0.8 mm × 1.8 mm (opaque). Adjusting the position of the photoelectric switch helps determine the appropriate platform zero point and provides limit protection for the module. The system uses three 120W closed-loop drive control integrated machines (Beijing Wains Technology Co., Ltd., China) as the power devices. The motors are connected to the sliding platform module through couplings and motor mounting flanges, as shown in Fig. 4.1c (2). Each motor consists of a 57 stepper motor and a VSMD124_045T driver. The driver is based on the Modbus RTU protocol and supports incremental encoders. The motor size is 76 mm × 57 mm × 57 mm, with a rated power of 120W and a maximum torque of 19 kg·cm. The driver size is 57 mm × 57 mm × 21 mm, with a peak current of 4.5A. It operates at an input voltage of 24 V. The speed control accuracy is ± 1 rpm, the position control accuracy is ± 1Pulse, and the torque control accuracy is ± 3%. The motor parameters, such as current, speed,

60

Y. Li et al. Closed-loop drive control machine

Jetson Nano

WIZnet W5500

RJ45 TCP/IP

RS232 STM32F103R Modbus CT6

Closed-loop drive control machine

Closed-loop drive control machine

power circuit

Fig. 4.4 Core circuit diagram

acceleration/deceleration, start/stop, can be controlled using the RS232 serial bus and Modbus RTU standard bus protocol. Increasing the motor resolution by setting the driver subdivision to 1/64, the motor operation is controlled by PWM pulse waves to minimize the impact of slider movement on image acquisition quality. The module uses a development board with STM32F103RCT6 chip as the motion controller. The core circuit diagram of the development board is shown in Fig. 4.4. The motion controller works with the main control module to control the closed-loop drive control integrated machines and achieve automatic timed point-to-point movement in space. The STM32 microcontroller uses the WIZnet W5500 as the Ethernet controller for real-time communication and instruction reception with Jetson Nano. The MAX232 chip is used for communication and instruction operation with the closed-loop drive control integrated machines. The power input and voltage conversion module provides reliable and stable power supply for the motion controller. The physical layout of the development board, with dimensions of 140 mm × 60 mm × 20 mm, is shown in Fig. 4.1c (3). It is housed in a metal distribution box located in the lower right corner of the main diagram. Indoor experiments and greenhouse data collection have demonstrated that STM32, as a motion controller, can meet the requirements of the system’s motion control with high cost-effectiveness.

4.2.3 Image Acquisition Module The calculation of Potato Crop Water Stress Index (CWSI) requires temperature data from the plant canopy. Temperature data can be obtained in a non-destructive and high-throughput manner by installing a thermal infrared camera in the system. However, thermal images obtained solely from the infrared camera require manual

4 Autonomous Crop Image Acquisition System Based on ROS System

61

processing to extract canopy temperature information. To address this, a stereo camera is integrated into the system. The obtained thermal infrared images and visible light images can automatically and accurately generate the three-dimensional fused point cloud dataset of the potato canopy, incorporating color, depth, and temperature information through the subsequent algorithm. This significantly improves the efficiency of water stress research and provides technical support for reconstructing the temperature point cloud of the entire crop. The image acquisition module utilizes an IPT384 network-type thermal imager (produced by High-Gain Intelligence, China) to capture thermal infrared data of the crops, as shown in Fig. 4.1d (1). It operates at a voltage of 12 V, with dimensions of 88 mm × 60 mm × 60 mm. The temperature measurement range is -20°C to 150°C, with a measurement distance of 0.5 m to 25 m. The image resolution is 384 × 288 pixels, and the measurement resolution is 0.1°C. The imager supports both electric and automatic focusing. A PXYZ-S-H65-060 USB3.0 stereo camera (produced by PixelLeap, China) is used to capture visible light data of the crops, as shown in Fig. 4.1d (2). It operates at a voltage of 5 V, with dimensions of 101 mm × 28 mm × 31 mm. The image resolution is 1280 × 720 pixels, and it features manual focusing. The camera communicates via the UVC protocol and is compatible with various systems and hardware platforms, capable of synchronously outputting left and right visible light images. A suitable frame integrating both cameras is designed using Solidworks software and 3D printing technology, as shown in Fig. 4.5. The integrated module is mounted on the Z-axis of the 3D slide module, ensuring that both cameras capture canopy images in the vertical direction of the crops. The thermal infrared camera operates at 12 V, while the stereo camera operates at 5 V. Therefore, power is supplied to these cameras by connecting them to a 24 V power source via DC/DC modules for voltage conversion (24 V to 12 V and 24 V to 5 V). To simplify the wiring, the integrated control module and the cameras are mounted vertically on the Z-axis of the 3D slide module. The thermal infrared camera is connected to the Jetson Nano control module via a switch for communication, while the stereo camera communicates with the Jetson Nano via USB3.0.

4.2.4 Real-Time Monitoring Module Using the C ++ programming language on the ROS platform architecture, the control communication between nodes in the system is implemented through the publish/ subscribe pattern. This includes data transmission and reception between the main control module and the motion control module, as well as image acquisition, storage, and monitoring from the image acquisition module. With the developed motion control software, the position movement of the slide module can be controlled in real-time, and the spatial coordinates of the desired inspection points can be calibrated. The developed ROS multi-node control software enables functionalities such as setting preset points, querying camera status, and obtaining real-time data.

62

Y. Li et al.

Fig. 4.5 Image acquisition module package

4.2.4.1

System Communication Control

The indoor crop inspection system achieves stable communication within the greenhouse through wired connections between the computer, controller, and operating terminal. Wired communication consists of two parts: serial communication and network communication, including: (1) USB 3.0 serial communication using the UVC protocol for wired communication between the dual-camera and Jetson Nano. (2) RS232 serial communication using the MODBUS RTU protocol for the STM32 motion controller to control the operation status and provide real-time feedback of the closed-loop drive control system. (3) RJ45 network communication using a switch to enable wired communication between the thermal infrared camera and Jetson Nano, as well as control functions for capturing, storing images, and temperature. (4) Data transmission and reception between STM32 and Jetson Nano through TCP/ IP protocol using W5500 and a switch with RJ45 network interface. (5) Jetson Nano and the remote display control screen use RJ45 network interface and a switch to establish a connection within the same local area network for remote monitoring and control. Software was developed under the ROS architecture, integrating network information, motion control nodes, thermal infrared camera nodes, and dual-camera nodes. The motion control node in the software mainly handles the deletion and retrieval of preset positions, real-time display and retrieval of motor position coordinates. The thermal infrared camera node is responsible for acquiring thermal infrared camera data and monitoring its operational status. The dual-camera node is responsible for specifying image capture and storage files and monitoring the operational status of the dual-camera. Under the ROS architecture, the motion control node, thermal infrared camera node, and dual-camera node act as publishers, sending various information as

4 Autonomous Crop Image Acquisition System Based on ROS System

Thermal Infrared Camera Node

Thermal infrared data folder update message

Motion control node

Location coordinate update information

Location coordinate update information

63

Binocular camera node

Binocular data folder update message

Fig. 4.6 ROS node publish/subscribe diagram

messages to corresponding topics. Subscribers receive the messages and execute the corresponding programs. Specifically, the motion control node publishes the feedback coordinate information of the three axes to the designated topic, while the thermal infrared camera node and dual-camera node receive messages to perform data acquisition and storage tasks. The thermal infrared camera node and dual-camera node publish messages containing image data retrieval results to the designated topic, and the motion control node subscribes to the topic to receive messages and control the motor accordingly. The ROS node publish/subscribe relationship is illustrated in Fig. 4.6.

4.2.4.2

Software Design and Application

Visualization, real-time performance, and automation are crucial for the efficient operation of the indoor crop inspection system. The developed motion control software, ROS multi-node control software, the demo provided by the thermal infrared camera, and the Nomachine remote desktop tool are utilized to achieve functions such as preset coordinate positioning, camera imaging monitoring, motor and camera parameter adjustments, selection of data storage folders, manual acquisition of various images and temperature data, and remote monitoring and control of system operation. Accurate positioning of preset coordinates is the first step in achieving automated high-throughput data acquisition in the system. With the help of the self-developed motion control software and the demo provided by the thermal infrared camera, the motor can be controlled to move at a user-defined speed and direction while observing real-time thermal infrared images. The current position feedback data of the three axes in the motion control software is recorded when the demo image contains only one complete potato plant. This data is used as the preset coordinate, and the two-dimensional array in the motion control node (spatial coordinates of nine target points) is modified to ensure accurate movement to the corresponding coordinates during automated data acquisition by the image capture module. The system’s automated image acquisition process flowchart is shown in Fig. 4.7. First, the program starts, and the timing trigger interval is set according to actual requirements. The STM32 motion controller communicates with Jetson Nano and the closed-loop drive control system for data transmission and determines if it is at the origin. If it is not at the origin, an instruction is sent to the closed-loop drive control system via the MODBUS RTU protocol to move to the origin and reset the

64

Y. Li et al. Start

Open jetson nano to communicate with the image acquisition module to collect thermal infrared, visible light images and temperature matrices.

Timing : fixed time STM32 control motor inspection

YES Open the network port communication between jetson nano and STM32, the serial port communication between STM32 and stepper motor, and return to the origin.

NO

Does storage file add new data?

YES Re-enable, coordinate zeroing

Move according to the set coordinates

NO

Is it the last set point?

YES NO Back to zero Is it at the set point?

End

Fig. 4.7 Image acquisition system workflow diagram

coordinates. Jetson Nano sends the preset coordinate information written in the ROS system to STM32, which receives real-time feedback of the coordinates from the closed-loop drive control system and communicates with Jetson Nano to determine if the preset coordinate has been reached. After reaching the preset coordinate, Jetson Nano controls the image capture module to acquire thermal infrared images, temperature matrices, and visible light images, and performs local storage operations. The image capture node creates a data storage folder and determines whether to publish the data acquisition completion information after judging if new images have been added. Finally, it is determined if it is the last preset coordinate. If it is not, data acquisition continues for the next coordinate point. Otherwise, a cycle of data acquisition is completed.

4 Autonomous Crop Image Acquisition System Based on ROS System

65

Fig. 4.8 Calibration point image acquisition and storage results

4.3 Indoor Crop Inspection System Testing 4.3.1 Feasibility Testing of the System To demonstrate the feasibility and convenience of the system’s software and hardware design for indoor image acquisition research, a scaled-down platform framework was constructed in the laboratory. Nine markers were placed below the platform, and the motion control software and thermal infrared camera provided by the official demo were used to control the camera’s movement above the markers. The motion control software recorded the coordinates of the nine markers and transmitted the information to the motion control node. On the PC side, the Nomachine software was used to operate the command line node, and the system followed the workflow shown in Fig. 4.6 to complete multiple rounds of image acquisition for inspection. The captured image results are shown in Fig. 4.8. The markers were positioned within the imaging area of the thermal infrared camera, demonstrating that the system can effectively acquire crop image data and reliably save it to specific folders. The system operated without any issues during an automatic data collection on January 15, 2022, proving its capability for highthroughput and automated acquisition of water stress data for potato crops.

4.3.2 System Stability Testing and Application The greenhouse experiment was conducted from April 24th to May 1st, 2022 (7 days) and from November 24th to December 4th, 2022 (11 days) in the No. 9 greenhouse of the Xiaotangshan National Precision Agriculture Research Base in Changping District, Beijing. The system utilized a timed triggering mode for each round of inspection and data collection. Based on the different temperatures in the morning, noon, and evening, the system was set to collect data at specific intervals. From 8:30 AM to 10:00 AM, data was collected every 30 min. From 10:00 AM to 12:00 PM, data was collected every 20 min. From 12:00 PM to 3:00 PM, data was collected every 10 min. From 3:00 PM to 5:00 PM, data was collected every 20 min. From

66

Y. Li et al.

( a ) Binocular left image

( b ) Binocular right image

( c ) Thermal infrared image

Fig. 4.9 Original image

5:00 PM to 6:00 PM, data was collected every 30 min. Each round of inspection and data collection took 3 min, resulting in a total of 648 sets of data, 11,664 visible light images, 5,832 thermal infrared images, and 5,832 sets of raw temperature data. During the 18-day greenhouse application period, no abnormal situations occurred, indicating that the system has reliable stability. The collection and storage of a large amount of data demonstrate the system’s capability for automated high-throughput data acquisition. The collected raw image data is shown in Fig. 4.9.

4.4 Extraction of Three-Dimensional Distribution of Potato Plant CWSI 4.4.1 Image Data Processing The visible light and thermal infrared image data obtained from the indoor inspection platform need to undergo multimodal image registration. Compared to feature-based registration, camera-based registration based on photogrammetry exhibits faster and more accurate performance. The camera-based registration principle utilizes the relative positional relationship between cameras for registration. The programming implementation is done using C ++ in conjunction with the OpenCV 4.5.0 (Open Source Computer Vision Library) library. Firstly, calibration of the two cameras is necessary as a prerequisite for image processing in potato plant water stress research. The calibration experiment utilizes the chessboard calibration method based on planar calibration board proposed by Professor Zhang Zhengyou. A chessboard grid with each square size of 30 × 30 mm is used in the experiment. The calibration board is placed directly below the camera, and 30 sets of visible light and thermal infrared images are captured by photographing the calibration board at different angles. The visible light images are then resized and cropped using bicubic interpolation to match the resolution of the thermal infrared images. The processed images are shown in Fig. 4.10.

4 Autonomous Crop Image Acquisition System Based on ROS System

( a ) Scaling the checkerboard in the visible light image after clipping

67

( b ) Checkerboard in thermal infrared image

Fig. 4.10 Checkerboard images in different cameras

The resized and cropped visible light images and thermal infrared images are imported into Matlab 2018a with the Stereo Camera Calibrator toolbox. Calibration is performed for the left camera-right camera pair and the left camera-thermal infrared camera pair. The calibration interface is shown in Figs. 4.11 and 4.12. The average errors for the two calibrations are 0.11 and 0.18 pixels, respectively. This provides the intrinsic parameters (including focal length, principal point, distortion coefficients, etc.) for the two stereo cameras and the thermal infrared camera, as well as the extrinsic parameters (including rotation matrix and translation matrix) between the two visible light cameras and the thermal infrared camera. These parameters are essential for subsequent crop canopy image processing.

Fig. 4.11 Calibration between left and right cameras of binocular camera

68

Y. Li et al.

Fig. 4.12 Calibration between left camera and thermal infrared camera of binocular camera

Registering thermal and visible light images involves finding the geometric transformation between the two. It requires correctly identifying multiple corresponding point pairs in the two images and then calculating the optimal homography transformation. The specific workflow is illustrated in Fig. 4.13. First, the intrinsic and extrinsic parameters of the left and right cameras of the stereo camera are loaded, and stereo rectification is applied to eliminate distortion and align the left and right visible light images. Next, in order to quickly select the optimal set of points, these point pairs are chosen as much as possible on the potato crops. The Laplacian algorithm is used to sharpen the left and right visible light images, enhancing the contours/edges and gradient changes in the images [24]. Then, the sharpened images are converted from the RGB color space to the HSV color space. The HSV model can partially avoid the thresholding problem caused by the high discreteness and high correlation of the RGB model and effectively utilize the color space for segmentation [25]. The upper and lower limits of the hue (H) channel (HL, HU), saturation (S) channel (SL, SU), and value (V) channel (VL, VU) are set to obtain mask parameters. The target image, i.e., the green channel image, is extracted by performing a logical “AND” operation [26]. Then, Speeded Up Robust Features (SURF) feature detection is performed on the two target images, and their feature descriptors are established. The detected feature points are matched using a fast nearest neighbor search algorithm, and the matching results are displayed on the input image. Some mismatches in the matching results may have a negative impact on the registration, and they are filtered out using epipolar geometry constraints. Epipolar geometry is a geometry in stereovision that imposes constraints between image points.

4 Autonomous Crop Image Acquisition System Based on ROS System

Fig. 4.13 The workflow of coarse registration method for visible and thermal infrared images

69

70

Y. Li et al.

Filtered feature point pairs are extracted, and the intrinsic and extrinsic parameters of the left and right cameras of the stereo camera are loaded. The world coordinates corresponding to these point pairs are estimated using the principle of triangulation. Equation (4.1) is used to perform back-projection, projecting these world coordinate points onto the thermal image. ] R T × Coordinatesworld =K× T 0 1 [

S × Coordinatesthermal

(4.1)

In the formula, S represents the non-zero scale factor, which is the z-component of the world coordinates computed from the feature point pairs. Coordinatesthermal represent the homogeneous coordinates of the back-projected points on the thermal image. K is the internal parameter matrix of the calibrated thermal infrared camera, R is the 3D rotation matrix of the calibrated thermal infrared camera relative to the left camera of the stereo camera, and T is the translation matrix of the calibrated thermal infrared camera relative to the left camera of the stereo camera. They represent the homogeneous coordinates of the world coordinates computed from the feature point pairs. The surface structure of the target crop canopy is highly complex, with significant depth variations. Due to calibration errors, direct registration results in large errors. Therefore, the Structural Similarity Index (SSIM) is introduced for similarity measurement. The calculation formula for SSIM is as follows: ( (2 ux uy + c1 ) 2σxy + c2 ) )( ) SSIM(x, y) = ( (4.2) u2x + u2y + c1 σx2 + σy2 + c2 In the equation, x and y represent two input images. ux 、uy denote the mean grayscale values of images x and y, respectively. σx2 and σy2 represent the variances of pixel grayscale values in images x and y, respectively. σxy represents the covariance between images x and y. To avoid division by zero, c1 and c2 are small constants. In this study, they are used to extract structural information from the transformed thermal infrared and visible light images for similarity measurement. A higher value indicates a higher structural similarity between the registered images. Assuming there is a small positional deviation (Δx , Δy )between the computed back-projected coordinates in the thermal image and the true corresponding coordinates in the thermal infrared image, this deviation has an initial value of (0, 0). The feature points in the visible light image and the corresponding points in the thermal infrared image, which have been compensated for the positional deviation, are used as input for the RANSAC (Random Sample Consensus) algorithm. The goal is to compute the homography matrix that best describes the transformation relationship between them and apply the homography transformation to the thermal image.

4 Autonomous Crop Image Acquisition System Based on ROS System

71

Repeat the previous three steps by increasing the value of (Δx , Δy ). Since the SSIM (Structural Similarity Index) has the property of having a unique maximum value, the transformation that produces the highest SSIM value is selected as the optimal homography transformation. Finally, the feature points in the left image of the visible light and the back-projected points in the thermal image are used as input, and the RANSAC algorithm is employed to compute the homography transformation that best describes the relationship between these point pairs. The criterion for the RANSAC algorithm is to maximize the number of inliers, which means that the best homography transformation selected will match the maximum number of point pairs between the visible light and thermal images. The computed result will be the optimal solution in terms of least squares error. The obtained transformation matrix is then used to transform the thermal image. The feature point matching after processing the visible light image is shown in Fig. 4.14, and the registration results of the visible light and thermal infrared images are shown in Fig. 4.15.

Fig. 4.14 Comparison of feature points after visible light image processing

Y. Li et al.

image registration

Homography transformation of thermal image

thermal infrared imagery

visible image

72

Fig. 4.15 Registration results of visible light and thermal infrared images

4.4.2 Extraction of Crop Canopy CWSI 3D Information Acquisition Method Based on Binocular Stereo Vision: Low Cost and Simple Equipment, an Important Direction in the Field of 3D Reconstruction. This platform utilizes binocular stereo vision and multimodal image fusion technology to extract 3D Crop Water Stress Index (CWSI) for potato crops. The functionality of this module is implemented using C ++ in conjunction with OpenCV 4.5.0 and PCL 1.12.0 (Point Cloud Library). Stereo matching, which obtains depth information based on image data, is a popular research topic in binocular stereo vision. By finding corresponding points in two images and calculating the disparity, the depth information of the corresponding 3D point in space can be obtained. The depth value can be calculated based on the principle of similar triangles. D=

B· f d

(4.3)

In the equation, D represents the depth value in millimeters. B is the baseline length of the binocular camera, and f is the focal length of the camera in pixel units. The difference between the horizontal coordinates of the corresponding pixels in the left and right images, denoted as d = xl − xr , is called disparity in pixel units.

4 Autonomous Crop Image Acquisition System Based on ROS System

73

First, the rectified left and right visible light images are taken as inputs, and the Semi Global Block Matching (SGBM) algorithm is used to compute the disparity map. SGBM is a variant of the Semi Global Matching (SGM) algorithm, where the core step is to select matching primitives to construct cost aggregation functions based on multiple directions and scanlines, and find the optimal solution for the cost aggregation functions. Next, due to occlusions or uneven lighting, some disparity values in the disparity map are unreliable. Median filtering is applied to remove isolated noise caused by misalignments. After removing incorrect matches, the discarded pixels create invalid value holes. A multi-level mean filtering method is used to iteratively fill the holes. The multi-level mean filtering algorithm fills the holes multiple times by changing the filter window size and utilizing the integral image of the disparity map. Then, Eq. (4.3) is used to calculate the depth value of a point in space based on the principle of similar triangles. Equation (4.4) is used to calculate the three-dimensional coordinates of the point, combining with other information. ⎧ ⎨

Z=D 0 ·D X = x−x f ⎩ y−y0 Y = f ·D

(4.4)

In the equation, (x, y) represents the pixel coordinates in the image, (x0 , y0 ) represents the principal point pixel coordinates, and X, Y, Z represent the coordinates of the potato crop point cloud. Finally, based on the homography transformation relationship between visible light and thermal images in multimodal image registration, a new point cloud data type is defined using the Point Cloud Library (PCL) to integrate the three-dimensional coordinates, RGB color values, and temperature information of the potato crop canopy. By combining environmental and reference plane temperature information, the three-dimensional Crop Water Stress Index (CWSI) is calculated using formula (4.5). Thus, a three-dimensional point cloud data of the potato crop canopy is generated, which includes both color information, temperature information, and CWSI index, as shown in Fig. 4.16. C W S I = (Tcrop − Twet )/(Tdry − Twet ))

(4.5)

In the equation, Tcrop represents the temperature (°C) of the crop canopy or crop leaves, Tdry represents the temperature (°C) of the leaves when the stomata are closed and no transpiration occurs, and Twet represents the temperature (°C) of the leaves when the stomata are fully open and the leaves are in a state of maximum transpiration.

74

Y. Li et al.

( a ) Clipping visible light image

( b ) Thermal infrared image

( c ) Fusion point cloud of left and right images of binocular camera

( d ) Visible thermal infrared fusion point cloud

Fig. 4.16 The processed image data

4.5 Conclusion For the study of potato water stress, an integrated indoor crop inspection system was developed, which consists of a main control module, a motion control module, an image acquisition module, and a real-time monitoring module. Based on the ROS system, multiple functional nodes were designed, including motion control nodes, thermal infrared camera nodes, and stereo camera nodes, enabling automatic, non-destructive, and high-throughput acquisition of images of multiple potato crop canopies. A 1-day timed image acquisition test was conducted in the laboratory environment, validating the feasibility of the system in terms of normal image acquisition and storage. An 18-day practical application was carried out in a greenhouse environment, resulting in the acquisition of 11,664 visible light images, 5,832 thermal infrared images, and 5,832 sets of raw temperature data, with no exceptional circumstances indicating the system’s stability. Batch storage and processing of the obtained data were performed, and a fused point cloud of visible light and thermal infrared images was generated, verifying the reliability of the collected data quality and laying the foundation for subsequent generation of CWSI point clouds. The main innovation of this system lies in its ability to achieve long-term, highthroughput data acquisition of multiple potato crops in an automated manner. It provides a large amount of reliable images and temperature data for the fusion of three-dimensional information and temperature information of potato crops, as well as water stress research. Currently, image processing is performed offline, but in the future, image processing nodes can be added to the system to automatically extract and save phenotype parameters such as CWSI.

4 Autonomous Crop Image Acquisition System Based on ROS System

75

References 1. Cheng Shengkui, Li Yunyun, Liu xiaojie, et al. (2018) Thoughts on food security in China in the new period [J]. J Nat Resour, 33(06):911–926 2. Huichun Z, Hongping Z, Jiaqiang Z et al (2020) Research Progress and prospect in plant phenotyping platform and image analysis technology [J]. Trans Chin Soc Agric Mach 51(03):1– 17 3. Gerland P, Raftery AE, Sevcikova H, et al. (2014) World population stabilization unlikely this century[J]. Sci (New York, N.Y.), 346(6206):234–237 4. Huichun Z, Guosu W, Liming B et al (2019) Visible Camera-based 3D phenotype measurement system and Time-series visual growth model of plant [J]. Trans Chin Soc Agric Mach 50(10):197–207 5. Zhang Derong (2019) Study on rapid detection of phenotypic character parameters of plants [D]. Zhejiang University 6. Wanneng Y, Hui F, Xuehai Z et al (2020) Crop phenomics and High-Throughput phenotyping: past decades, current challenges, and future perspectives [J]. Mol Plant 13(2):187–214 7. Wang Liuyang, Miao Yanlong, Han Yuxiao, et al. (2023) Extraction of 3D distribution of potato plant CWSI based on thermal infrared image and binocular stereovision system[J]. Front Plant Sci, 13 8. Xu Lingxiang, Chen Jiawei, Ding Guohui, et al. (2020) Indoor phenotyping platforms and associated trait measurement:Progress and prospects [J]. Smart Agric, 2(01):23–42 9. Man C, Hongbo Y, Zhenjiang C et al (2020) Review of Field-based information acquisition and analysis of High-throughput phenotyping [J]. Trans Chin Soc Agric Eng 51(S1):314–324 10. Joshi S, Thoday KE, Daetwyler HD, et al. (2021) High-throughput phenotyping to dissect genotypic differences in safflower for drought tolerance.[J]. PloS one, 16(7) 11. Dissanayake R, Kahrood HV, Dimech AM, et al. (2020) Development and application of ImageBased High-Throughput phenotyping methodology for salt tolerance in lentils [J]. Agron, 10(12) 12. Wang Yinghua, Hu Songtao, Ren He, et al. (2022) 3DPhenoMVS: A Low-Cost 3D tomato phenotyping pipeline using 3D reconstruction point cloud based on multiview images [J]. Agron, 12(8) 13. Wu Sheng, Wen Weiliang, Wang Yongjian, et al. (2020) MVS-Pheno: A portable and LowCost phenotyping platform for maize shoots using multiview stereo 3D reconstruction [J]. Plant Phenomics, 2020 14. Wu Sheng, Wen Weiliang, Gou Wenbo, et al. (2022) A miniaturized phenotyping platform for individual plants using multi-view stereo 3D reconstruction [J]. Front Plant Sci, 13 15. Gao Tian, Zhu Feiyu, Paul P, et al. (2021) Novel 3D imaging systems for High-Throughput phenotyping of plants [J]. Remote Sens, 13(11) 16. Caporaso N, Whitworth MB, Fisk ID (2018) Near-Infrared spectroscopy and hyperspectral imaging for non-destructive quality assessment of cereal grains[J]. Appl Spectrosc Rev 53(8):667–687 17. Guo Zilong, Yang Wanneng, Chang Yu, et al. (2018) Genome-Wide association studies of image traits reveal genetic architecture of drought resistance in rice [J]. Mol Plant, 11(6):789-805 18. Langstroff A, Heuermann MC, Stahl A, et al. (2021) Opportunities and limits of controlledenvironment plant phenotyping for climate response traits [J]. TAG. Theor Appl Genet Theor Und Angew Genet, 135(1):1–16 19. Chen Dijun, Neumann K, Friedel S, et al. (2014) Dissecting the phenotypic components of crop plant growth and drought responses based on high-throughput image analysis.[J]. Plant Cell, 26(12):4636–4655 20. Pieruschka R, Schurr U (2019) Plant phenotyping: past, present, and future [J]. Plant Phenomics, 2019 21. He Yong, Li Xiyao, Yang Guofeng, et al. (2022) Research progress and prospect of indoor highthroughput germplasm phenotyping platforms [J]. Trans Chin Soc Agric Eng, 38(17):127-141

76

Y. Li et al.

22. Qinghua G, Fangfang W, Shuxin P et al (2016) Crop 3D: a platform based on LiDAR for 3D high-throughput crop phenotyping. Sci Sin Vitae 46:1210–1221 23. Chongyuan Z, Honghong G, Jianfeng Z et al (2016) 3D robotic system development for Highthroughput crop phenotyping [J]. IFAC PapersOnLine 49(16):242–247 24. Tian Xiangdong, Wang Jian, Du Dongfeng, et al. (2020) Medical imaging and diagnosis of subpatellar vertebrae based on improved Laplacian image enhancement algorithm [J]. Comput Methods Programs Biomed, 187:105082 25. Hamuda E, Mc Ginley B, Glavin M, Jones E (2017) Automatic crop detection under field conditions using the HSV colour space and morphological operations [J]. Comput Electron Agric 133:97–107 26. Li Li, Chen Shiwang, Yang Chengfei, et al. (2020) Prediction of plant transpiration from environmental parameters and relative leaf area index using the random forest regression algorithm [J]. J Clean Prod. 261:121136

Chapter 5

SeedingsNet: Field Wheat Seedling Density Detection Based on Deep Learning Yunxia Li, Zuochen Jiang, Zhao Zhang, Han Li, and Man Zhang

Abstract Wheat is one of the important food crops. Plant density directly affects the yield and quality of wheat, and estimation of plant density can be achieved through counting wheat seedlings. However, the wheat seedlings in the field have smaller and thinner leaves, smaller spacing between densely planted plants, and cross obstructed leaves, which makes it difficult to count wheat seedlings based on object detection. This study collected RGB canopy images of wheat during the second leaf stage in the field, and divided the wheat seedling density into three levels: low, medium, and high. Based on classification methods, the estimation of wheat seedling plant density was achieved. The results showed that the detection model of wheat seedling plant density was constructed using three lightweight Classful networks, ConvNeXt, VanillaNet, and MobileVit. The detection model based on MobileVit had the best performance on the training set, with an accuracy of 0.99, a loss of 0.31, and a detection accuracy of 0.62 on the test set. In view of the impact of image length–width ratio on the performance of the model, this paper conducts research. The results show that the image aspect ratio has an impact on the performance of the model. When the aspect ratio of the input image is closer to 1:1, the model has Y. Li · Z. Zhang (B) · H. Li · M. Zhang Key Laboratory of Smart Agriculture System Integration, Ministry of Education, Beijing 100083, China e-mail: [email protected] Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture and Rural Affairs, Beijing 100083, China Y. Li e-mail: [email protected] H. Li e-mail: [email protected] M. Zhang e-mail: [email protected] Z. Jiang College of Mechanical and Electrical Engineering, Zhongguo Jiliang University, Hangzhou, Zhejiang, China e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023 M. Zhang et al. (eds.), Sensing Technologies for Field and In-House Crop Production, Smart Agriculture 7, https://doi.org/10.1007/978-981-99-7927-1_5

77

78

Y. Li et al.

good performance. In summary, building a wheat seedling plant density detection model based on MobileVit can achieve grade estimation of field wheat seedling plant density, providing data support for refined management of wheat fields. Keywords Wheat · Seedling density detection · Lightweight convolutional neural network · Length–width ratio

5.1 Introduction Wheat is one of the three major food crops in the world. About 230 million hectares of land worldwide are used for wheat cultivation, and wheat production is about 6.8 tons. Wheat plant count in the field is a basic task, closely related to wheat planting density. The planting density is one of the important factors in agricultural production, which has a great impact on crop yield and quality [1, 2, 9]. In the cultivation management, more reasonable planting density can be selected through wheat seedling counting to improve the spatial distribution uniformity of wheat population, enhance the tillering ability of wheat, and realize the stability and improvement of yield and quality. In addition, research on planting density plays an important role in early breeding decisions to improve yield [12]. Therefore, it is necessary to estimate wheat seedling density accurately. The traditional crop planting density estimation is to randomly select a number of plots in the field, count the number of plants in these plots, and then take an average to represent the crop planting density of the plot. This method is time-consuming, labor-intensive, and weak generalization. To solve this problem, computer vision technology has become a powerful tool. Researchers have been attempted to estimate the number of seedlings by analyzing field images. Utilizing RGB images to count field crops has become a hot topic, which is mainly based on traditional image processing [13], shallow machine learning [8] and deep learning technology [10]. Of the above three methods, the traditional image processing methods are easy to be interfered by light, noise, weed background and other factors. The expression ability of shallow features such as color, shape and texture extracted by shallow machine learning methods is limited, and the representation of high-dimensional features is limited due to the non-optimality of artificial feature selection. In recent years, deep convolutional neural networks (CNNS) have shown strong performance in agricultural images object detection [14]. Many algorithms based on deep learning models have been successfully applied to the detection and counting of various crops. Deep learning can automatically learn different features from images or data sets to identify and locate individual objects of interest [6]. Fan et al. [3] combined the full convolutional network (FCN) based on unmanned aerial vehicle (UAV) RGB images to count sugar beet, corn and strawberry. The error was