Application and Research Advances on Deep Learning in Apple’s Industry Chain

doi:10.13304/j.nykjdb.2021.0688

Abstract

Abstract:

China is a big apple producer， with wide apple planting areas and many varieties. Combining deep learning with machine vision technology and applying it to the entire industrial chain of apple planting and production are important means and direction for the technological upgrading of the apple industry. This article focused on 3 key stages of fruit tree planting， harvesting and postharvest inspection in the apple industry chain， and systematically combed the related application and research progress of deep learning technology， which mainly included leaf disease and insect pest identification， planting monitoring， target detection of harvesting robot and non-destructive grading testing of apple after harvest. Based on the analysis and comparison of the technical differences and commonalities between different technologies， the difficulties and challenges that deep learning faced in apple industry chain were discussed.

Key words: deep learning, apple, machine vision, convolutional neural network

摘要：

我国是苹果生产大国，苹果种植面积广、品种多。将深度学习与机器视觉技术相结合并运用于苹果种植和生产的全产业链中是苹果产业技术升级的重要手段和方向。聚焦苹果产业链中的果树种植、收获采摘和产后检测3个关键阶段，系统性梳理深度学习技术的相关应用与研究进展，其中主要涉及叶部病虫害识别、种植监测、采摘机器人的目标识别和苹果无损分级检测等研究领域，在分析对比不同技术之间的差异与共性的基础上，探讨深度学习在苹果产业链中所面临的困难与挑战。

关键词: 深度学习, 苹果, 机器视觉, 卷积神经网络

CLC Number:

S126

Hao HUANG, Shengqiao XIE, Du CHEN, Heng WANG. Application and Research Advances on Deep Learning in Apple’s Industry Chain[J]. Journal of Agricultural Science and Technology, 2022, 24(10): 79-89.

黄昊, 谢圣桥, 陈度, 王恒. 深度学习在苹果产业链中的应用与研究进展[J]. 中国农业科技导报, 2022, 24(10): 79-89.

Figures/Tables 5

References 54

1	HINTON G E, SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks [J]. Science,2006,313:504-507.
2	Schmidhuber J. Deep learning in neural networks: an overview [J]. Neural Networks, 2015,61:85-117.
3	LECUN Y, BENGIO Y, HINTON G. Deep learning [J]. Nature, 2015, 521(7553): 436-444.
4	DARGAN S, KUMAR M, AYYAGARI M R, et al.. A survey of deep learning and its applications: a new paradigm to machine learning [J]. Arch. Comput. Methods Eng., 2020, 27(4): 1071-1092.
5	SHRESTHA A, MAHMOOD A. Review of deep learning algorithms and architectures [J]. IEEE Access, 2019,7:53040-53065.
6	ALOM M Z, TAHA T M, YAKOPCIC C, et al.. A state-of-the-art survey on deep learning theory and architectures [J]. Electronics, 2019, 8(3):292-357.
7	周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(6):1229-1251.
	ZHOU F Y, JIN L P, DONG J. Review of convolution neural network [J]. Chin. J. Computers, 2017,40(6):1229-1251.
8	邸洁,曲建华.基于Tiny-YOLO的苹果叶部病害检测[J]. 山东师范大学学报(自然科学版), 2020,35(1): 78-83.
	DI J, QU J H. A detection method for apple leaf diseases based on Tiny-YOLO [J]. J. Shandong Norm. Univ. (Nat. Sci.), 2020,35(1): 78-83.
9	FRANCIS M, DEISY C. Disease detection and classification in agricultural plants using convolutional neural networks—avisual understanding [C] // Proceedings of 6th International Conference on Signal Processing and Integrated Networks (SPIN). Nndia Amity, 2019: 1063-1068.
10	ZHENG Z, PAN S, ZHANG Y. Fruit tree disease recognition based on convolutional neural networks [C] // Proceedings of International Conferences on Ubiquitous Computing & Communications; Data Science and Computational Intelligence; Smart Computing, Networking and Services. China Shenyang, 2019:118-122.
11	AGARWAL M, KALIYAR R K, SINGAL G, et al.. FCNN-LDA: A faster convolution neural network model for Leaf disease identification on apple’s leaf dataset [C] // Proceedings of 12th International Conference on Information & Communication Technology and System (ICTS). Indonesia Surabaya,2019:246-251.
12	SON C H. Leaf spot attention network for apple leaf disease identification [C] // Proceedigns of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. USA Seattle, IEEE,2020:229-237.
13	NAGARAJU Y, VENKATESH, SWETHA S, et al.. Apple and grape leaf diseases classification using transfer learning via fine-tuned classifier [C] // Proceedigns of 2020 IEEE International Conference on Machine Learning and Applied Network Technologies (ICMLANT). India Hyderabad,IEEE,2020:1-6.
14	鲍文霞,吴刚,胡根生,等.基于改进卷积神经网络的苹果叶部病害识别 [J]. 安徽大学学报(自然科学版), 2021,45(1):53-59.
	BAO WENXIA, WU GANG, HU GENSHENG, et al.. Apple leaf disease recognition based on improved convolutional neural network [J]. J. Anhui Univ. (Nat. Sci.), 2021,45(1):53-59.
15	WANG X A, TANG J, WHITTY M. Side-view apple flower mapping using edge-based fully convolutional networks for variable rate chemical thinning [J/OL]. Computers Electron. Agric.,2020,178: 105673 [2021-07-01]. .
16	BHATTARAI U, BHUSAL S, MAJEEDY, et al.. Automatic blossom detection in apple trees using deep learning [J]. IFAC-Papers,2020,53(2):15810-15815.
17	MAZZIA V, KHALIQ A, SALVETTI F, et al.. Real-time apple detection system using embedded systems with hardware accelerators: an edge AI application [J]. IEEE Access,2020,8:9102-9114.
18	WANG D, LI C, SONG H, et al.. Deep learning approach for apple edge detection to remotely monitor apple growth in orchards [J]. IEEE Access,2020,8: 26911-26925.
19	CHEN Z, TING D, NEWBURY R, et al. Semantic segmentation for partially occluded apple trees based on deep learning [J]. Computers Electron. Agric.,2021,181:105952-105958.
20	LIU C, HAN J, CHEN B, et al.. A novel identification method for apple (Malus domestica Borkh.) cultivars based on a deep convolutional neural network with leaf image input [J]. Symmetry, 2020,12(2):217-235.
21	LI J, XIE S, CHEN Z, et al.. A shallow convolutional neural network for apple classification [J]. IEEE Access,2020,8:111683-111692.
22	张力超,马蓉,张垚鑫.改进的LeNet-5模型在苹果图像识别中的应用 [J] .计算机工程与设计, 2018, 39(11):3570-3575.
	ZHANG L C, MA R, ZHANG Y X. Application of improved LeNet-5 model in apple image recognition [J]. Computer Eng. Design, 2018, 39(11):3570-3575.
23	王丹丹,何东健.基于R-FCN深度卷积神经网络的机器人疏果前苹果目标的识别 [J].农业工程学报,2019,35(3):156-163.
	WANG D D, HE D J. Recognition of apple targets before fruits thinning by robot based on R-FCN deep convolution neural network [J]. Trans. Chin. Soc. Agric. Eng.,2019,35(3):156-163.
24	冯娟,曾立华,刘刚,等.融合多源图像信息的果实识别方法 [J] .农业机械学报,2014,45(2):73-80.
	FENG J, ZENG L H, LIU G, et al.. Fruit Recognition algorithm based on multi-source Images Fusion [J]. Trans. Chin. Soc.Agric. Mach.,2014,45(2):73-80.
25	宋怀波,何东健,潘景朋.基于凸壳理论的遮挡苹果目标识别与定位方法 [J].农业工程学报,2012, 28(22):174-180.
	SONG H B, HE DO J, PAN J P. Recognition and localization methods of occluded apples based on convex hull theory [J]. Trans. Chin. Soc. Agric. Eng.,2012, 28(22):174-180.
26	刘晓洋,赵德安,陈玉,等.夜间低照度条件下苹果采摘机器人的图像识别 [J].华中科技大学学报(自然科学版),2015,43(S1):525-528.
	LIU X Y, ZHAO D A, CHEN Y, et al.. Image recognition of apple harvesting robot in artificial light source of low illumination at night [J]. J. Huazhong Univ. Sci. Technol. (Nat. Sci.),2015,43(S1):525-528.
27	孙飒爽.自然环境下绿色苹果目标的识别与分割方法研究 [D].杨凌:西北农林科技大学,2019.
	SUN S S. Recognition and segmentation of green apple targets in natural environments [D]. Yangling:Northwest A&F University,2019.
28	张亚静,李民赞,乔军,等.一种基于图像特征和神经网络的苹果图像分割算法 [J] .光学学报,2008, 28(11):2104-2108.
	ZHANG Y J, LI M Z, QIAO J, et al.. Segmentation algorithm for apple recognition using image features and artificial neural network [J]. Acta Optica Sin.,2008, 28(11):2104-2108.
29	王津京,赵德安,姬伟,等.基于BP神经网络的苹果图像分割算法 [J] .农机化研究,2008(11):19-21.
	WANG J J， ZHAO D A， JI W， et al.. Segmentation of apple image by BP neural network using in apple harvesting robot ［J］. J. Agric. Mechanization Res.， 2008（11）：19-21
30	GENÉ-MOLA J, VILAPLANA V, ROSELL-POLO J R, et al.. Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities [J]. Computers Electron. Agric.,2019,162:689-698.
31	LFABC D, YM D, XIN Z D, et al.. Faster R-CNN-based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting [J]. Biosys. Eng., 2020, 197:245-256.
32	CHU P, LI Z, LAMMERS K, et al.. Deep learning-based apple detection using a suppression mask R-CNN [J]. Pattern Recognition Lett.,2021,147:206-211.
33	周晓彦,王珂,李凌燕.基于深度学习的目标检测算法综述 [J].电子测量技术, 2017,40(11):89-93.
	ZHOU X Y, WANG K, LI L Y. Review of object detection based on deep learning [J]. Electron. Measure. Technol., 2017,40(11):89-93.
34	KUZNETSOVA A, MALEVA T, SOLOVIEV V. Using YOLOv3 algorithm with pre- and post-processing for apple detection in fruit-harvesting robot [J]. Agronomy, 2020,10(7):1016-1034.
35	武星,齐泽宇,王龙军,等.基于轻量化YOLOv3卷积神经网络的苹果检测方法[J].农业机械学报,2020,51(8):17-25.
	WU X, QI Z Y, WANG L J, et al.. Apple detection method based on light-YOLOv3 convolutional neural network [J]. Trans. Chin. Soc. Agric. Mach., 2020,51(8):17-25.
36	彭红星,黄博,邵园园,等.自然环境下多类水果采摘目标识别的通用改进SSD模型[J].农业工程学报,2018,34(16):155-162.
	PENG H X, HUANG B, SHAO Y Y, et al.. General improved SSD model for picking object recognition of multiple fruits in natural environment [J]. Trans. Chin. Soc. Agric. Eng., 2018,34(16):155-162.
37	中华人民共和国国家质量监督检验检疫总局,中国国家标准化管理委员会. 鲜苹果: [S].北京:中国标准出版社,2008.
38	何进荣,石延新,刘斌,等.基于深度学习的富士苹果外部品质分级方法 [J]. 农业机械学报,2021, 4(25):1-12.
	HE J R, SHI Y X, LIU B, et al.. Extrenal quality granding method of Fuji apple based on deep learning [J]. Trans. Chin. Soc. Agric. Mach.,2021, 4(25):1-12.
39	薛勇,王立扬,张瑜,等.基于GoogLeNet深度迁移学习的苹果缺陷检测方法[J].农业机械学报,2020, 51(7):30-35.
	XUE Y, WANG L Y, ZHANG Y, et al.. Defect detection method of apples based on GoogLeNet deep transfer learning [J]. Trans. Chin. Soc. Agric. Mach.,2020, 51(7):30-35.
40	FAN S, LI J, ZHANG Y, et al.. Online detection of defective apples using computer vision system combined with deep learning methods [J]. J. Food Eng.,2020,286:110102-1101011.
41	OHALI Y A. Computer vision based datad fruit grading system: Design and implementation [J]. J. King Saud Univ. Computer Inform. Sci.,2011,23(1):29-36.
42	SOFU M M, ER O, KAYACAN M C, et al.. Design of an automatic apple sorting system using machine vision [J]. Computers Electron. Agric.,2016, 127:395-405.
43	王立扬,张瑜,沈群,等.基于改进型LeNet-5的苹果自动分级方法 [J]. 中国农机化学报,2020,41(7): 105-110.
	WANG LIYANG, ZHANG YU, SHEN QUN, et al.. Automatic detection and grading method of apples based on improved LeNet-5 [J]. J. Chin. Agric. Mechan., 2020,41(7):105-110.
44	石瑞瑶. 基于机器视觉的苹果在线分级系统平台的研究 [D]. 沈阳:沈阳农业大学,2018.
	SHI RUIYAO. Research on apple online grading system platform based on machine vision [D]. Shenyang: Shenyang Agricultural University,2018.
45	凌强. 基于机器视觉的苹果品质分级技术的研发[D].哈尔滨:黑龙江大学,2019.
	LING QIANG. Research and development of apple quality grading technology based on machine vision [D]. Harbin: Heilongjiang University,2019
46	朱启兵,黄敏,赵桂林.基于邻域粗糙集和高光谱散射图像的苹果粉质化检测[J].农业机械学报, 2011,42(10):154-157.
	ZHU Q B, HUANG M, ZHAO G L. Apple mealiness detection based on neighborhood rough set and hyperspectral scattering image [J]. Trans. Chin. Soc. Agric. Mach., 2011,42(10):154-157.
47	曹玉栋,祁伟彦,李娴,等.苹果无损检测和品质分级技术研究进展及展望[J].智慧农业,2019, 1(3):29-45.
	CAO Y D, QI W Y, LI X, et al.. Research progress and prospect on non-destructive detection and quality grading technology of apple [J]. Smart Agric.,2019, 1(3):29-45.
48	LASHGARI M, IMANMEHR A, TAVAKOLI H. Fusion of acoustic sensing and deep learning techniques for apple mealiness detection [J]. J. Food Sci. Technol.,2020,57(6):2233-2240.
49	BAI Y, XIONG Y, HUANG J, et al.. Accurate prediction of soluble solid content of apples from multiple geographical regions by combining deep learning with spectral fingerprint features [J]. Postharvest Biol. Technol.,2019,156:110943-110952.
50	徐焕良,孙云晓,曹雪莲,等.基于光子传输模拟的苹果品质检测研究[J].农业机械学报,2021,52(8): 1-13.
	XU H L, SUN Y X, CAO X L, et al.. Research on apple quality detection based on photon transmission simulation [J]. Trans. Chin. Soc. Agric. Mach., 2021,52(8): 1-13.
51	MARCUS G, DAVIS E. Rebooting AI: Building Artificial Intelligence We Can Trust [M] .Vintage,2019.
52	马岽奡,唐娉,赵理君,等.深度学习图像数据增广方法研究综述[J].中国图象图形学报,2021, 26(3): 487-502.
	MA D G, TANG P, ZHAO L J, et al.. Review of data augmentation for image in deep learning [J]. J. Image Graphics, 2021, 26(3): 487-502.
53	PARK W, KIM D, LU Y, et al.. Relational knowledge distillation [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. USA California, IEEE, 2019: 3967-3976.
54	YOUNG S, WANG Z, TAUBMAN D, et al.. Transform quantization for CNN compression [J]. Proceedigns of IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE, 2021:5700-5714.

品种 Variety	指标/Index
品种 Variety	果实硬度 Hardness/（N·cm^-2）	可溶性固形物 Soluble solid content/%
富士系 Fuji	≥7	≥13
嘎拉系 Gala	≥6.5	≥12
藤木1号 Fujiki	≥5.5	≥11
元帅系 Marshal	≥6.8	≥11.5
华夏 Huaxia	≥6.0	≥11.5
粉红女士 Pink lady	≥7.5	≥13
澳洲青苹 Granny Smith	≥7.0	≥12
乔纳金 Jonagold	≥6.5	≥13
秦冠 Qinguan	≥7.0	≥13
国光 Guoguang	≥7.0	≥13
华冠 Huaguan	≥6.5	≥13
红将军 Red general	≥6.5	≥13

品种 Variety	指标/Index
品种 Variety	果实硬度 Hardness/（N·cm^-2）	可溶性固形物 Soluble solid content/%
富士系 Fuji	≥7	≥13
嘎拉系 Gala	≥6.5	≥12
藤木1号 Fujiki	≥5.5	≥11
元帅系 Marshal	≥6.8	≥11.5
华夏 Huaxia	≥6.0	≥11.5
粉红女士 Pink lady	≥7.5	≥13
澳洲青苹 Granny Smith	≥7.0	≥12
乔纳金 Jonagold	≥6.5	≥13
秦冠 Qinguan	≥7.0	≥13
国光 Guoguang	≥7.0	≥13
华冠 Huaguan	≥6.5	≥13
红将军 Red general	≥6.5	≥13

[1]	Yue ZHAO, Yong WEI, Huiyong SHAN, Zhimin MU, ZHANGJianxin, Haiyun WU, Hui ZHAO, Jianlong HU. Wheat Ear Detection Method Based on Deep Learning [J]. Journal of Agricultural Science and Technology, 2022, 24(9): 96-105.
[2]	Haitao LIU, Xin HAN, Yubin LAN, Lili YI, Baoju WANG, Lihua CUI. Precise Recognition Method of Cotton Top Buds Based on YOLOv4 Network [J]. Journal of Agricultural Science and Technology, 2022, 24(8): 99-108.
[3]	Kui FANG, Cheng LI, Xiao HE, Yineng CHEN. Research on Multi-angle Identification of Grape Leaf Disease Based on 3D Reconstruction [J]. Journal of Agricultural Science and Technology, 2022, 24(7): 86-96.
[4]	Jianwei WU, Jie HUANG, Xiaofei XIONG, Han GAO, Xiangyang QIN. Research and Application of Intelligent Recognition Method of Peach Tree Diseases Based on AI [J]. Journal of Agricultural Science and Technology, 2022, 24(5): 111-118.
[5]	Yinyan GAO, Yi SUN, Baochun LI. Estimating of Wheat Ears Number in Field Based on RGB Images Using Unmanned Aerial Vehicle [J]. Journal of Agricultural Science and Technology, 2022, 24(3): 103-110.
[6]	ZHOU Huiru, WU Boming. Advances in Research on Deep Learning for Crop Disease Image Recognition [J]. Journal of Agricultural Science and Technology, 2021, 23(5): 61-68.
[7]	LYU Chunyang, LIU Shengping, GUO Xiuming, XIAO Shunfu, LIU Dazhong, YANG Feifei, LI Luhua. Detection of Honeybee Based on SSD Model [J]. Journal of Agricultural Science and Technology, 2021, 23(5): 98-107.
[8]	CHEN Jun, ZHANG Qi, YANG Mengyu, YUAN Zhenyang. Impacts of Orchard Herbage-mulching on Photosynthetic Characteristics and Leaf Quality of Apples in Arid Desert Area [J]. Journal of Agricultural Science and Technology, 2021, 23(5): 160-167.
[9]	YAO Yingfang, LIU Feng, ZHANG Haidong, LI Chao, ZHOU Haijun, WANG Man. Research on Octagon Color and Fruit Shape Recognition Based on Machine Vision [J]. Journal of Agricultural Science and Technology, 2021, 23(11): 110-120.
[10]	LI Tai, LU Shijun, HUANG Jiazhang, CHEN Lei, FAN Xieyu. Research Progress of Apple Quality Evaluation Standards [J]. Journal of Agricultural Science and Technology, 2021, 23(11): 121-130.
[11]	YANG Zhen1,2, YANG Xin1*, YANG Xiaobin1, WANG Pengfei1, LI Jianping1, LIU Hongjie1, LI Xuejun1. Parameter Optimization Design of Apple Seedling Stubble Cutter Based on Virtual Orthogonal Test [J]. Journal of Agricultural Science and Technology, 2021, 23(1): 98-106.
[12]	XU Beibei1, WANG Wensheng1,2*, GUO Leifeng1, CHEN Guipeng3. A Review and Future Prospects on Cattle Recognition Based on Non-contact Identification [J]. Journal of Agricultural Science and Technology, 2020, 22(7): 79-89.
[13]	WANG Mengmeng1, DANG Hongzhong2*, LI Gangtie1, FENG Jinchao2, YAN Jingqiuzi1, HU Yang1, LI Xing1, YANG Chao1. Sap Flux Density Characteristics of Apple Orchards and Their Relationship with Environmental Factors in Gully Region of Loess Plateau [J]. Journal of Agricultural Science and Technology, 2020, 22(7): 140-147.
[14]	ZHANG Jie1,2, MA Yajun1, HE Zhibin1, GAO Fangfang1, ZHANG Shaohua1, WANG Chaoran1, ZHAO Danchen1. Application of Microbial Fertilizer Instead of Fertilizer in Apple Planting [J]. Journal of Agricultural Science and Technology, 2019, 21(7): 128-135.
[15]	DONG Wei1, QIAN Rong1, ZHANG Jie 2*, ZHANG Liping1, CHEN Hongbo2, ZHANG Meng1, ZHU Jingbo1, BU Yingqiao3. Vegetable Lepidopteran Pest Auto Recognition and Detection Counting Based on Deep Learning [J]. Journal of Agricultural Science and Technology, 2019, 21(12): 76-84.