Creation of hybrid systems for analyzing hyperspectral data and RGB images for early diagnosis of grain crops

Abstract

The article is devoted to the development and validation of a hybrid system for early diagnostics of stress conditions in cereal crops based on the fusion of hyperspectral data and RGB images. The aim was to quantitatively assess the contribution of each modality and to demonstrate that their integration improves the accuracy and robustness of detecting pre-visual stress symptoms. Materials and methods: field experiments conducted in 2023-2024 on winter wheat (Triticum aestivum L., variety Moskovskaya 40 ) under three treatments – nitrogen deficiency, septoria (Septoria tritici), and combined stress – using remote sensing with a DJI Matrice 300 RTK UAV. Data collection included a Headwall Nano-Hyperspec push-broom sensor (400-1000 nm, 2.2 nm, ~270 bands, ~5 cm/pixel) and a Zenmuse P1 RGB camera (45 MP, ~1.5 cm/pixel); radiometric/atmospheric correction of HSI, color calibration of RGB, and orthotransformation were performed. Vegetation indices (NDVI, PRI, ChlI, etc.) were extracted, and PCA/MNF were applied for dimensionality reduction; color (HSV, Lab) and texture (GLCM) features were derived from RGB. SVM (RBF), Random Forest, and 3D-CNN were compared; early (feature-level) and late (decision-level) fusion approaches were investigated. Results: individual HSI indices outperformed RGB features at early stages; ChlI achieved 84.11% accuracy versus ≤70% for the best RGB features. For unimodal models, HSI-3D-CNN reached 92.46%, outperforming SVM/RF but at a higher computational cost; RGB models lagged by 16-18 pp. The hybrid architecture with early feature fusion and CNN achieved 96.33% overall accuracy and F1=0.959, exceeding the best HSI model by 3.87 pp and the RGB model by 20.52 pp. Validation against ground-based biochemical measurements revealed strong correlations: r=0.917 (nitrogen deficiency), r=0.884 (septoria), r=0.945 (control), confirming the physiological relevance of the predictions.