Model | WageningenUR

Machine Learning Model

Currently, to predict upcoming weather events, farmers rely on wireless sensors placed in different parts of the orchard. Nevertheless, the statistical algorithms used to process the data offer accuracy within approximately one kilometre [1]. However, frost events can be local, with temperature varying within a few hundred meters [2]. Therefore, to predict the effects of frost in an orchard, farmers need a method that includes local measurements. Since flowering induction depends on temperature, local temperature predictions are necessary to estimate if trees in that orchard will flower.

The aim of PseuPomona is to counteract early flowering of fruit trees by interfering with plant’s internal flowering mechanisms. Therefore, we designed a tool to predict levels of FLOWERING LOCUS T (FT), the main protein responsible for flowering induction (Figure 1).

Figure 1: A flow chart showing the structure of the machine learning model. Round nodes represent data sets and square nodes represent models. We used data for model training (blue), data generation (black) or predictions (red).

The model consists of two neural networks – one predicting temperature [3] and the other predicting a plant’s florigen levels corresponding to that temperature [4]. The neural networks first had to be trained to capture the relationship between input and output – in our case, present and future temperature, as well as the correlation between the temperature and florigen levels. Meteorological data were obtained from a weather station at Wageningen University & Research, and plant florigen content was simulated by a modified plant framework model [5].

Our model can forecast temperature time series and plants’ FT content based solely on temperature data. This means that farmers require access only to a temperature sensor to be able to see when they should apply PseuPomona to avoid early flowering of their trees.

We created a machine learning model that consists of two neural networks – a long short-term memory (LSTM) neural network [3] for predicting the plant model’s output from temperature and a multilayer perceptron (MLP) neural network [4] for temperature forecasting. The source code for both models, as well as the modified plant model, can be found in this GitLab repository. To train the LSTM neural network, we had to simulate florigen production in plants at different temperatures. For this, we used the plant framework of Arabidopsis thaliana [5].

The original Arabidopsis thaliana framework model [5] simulates growth (the time it takes for a plant to flower) of A. thaliana based on defined day and night temperature, CO₂ concentration and light intensity, as well as day length. In this simulation, the model includes ordinary differential equations (ODEs) describing expression of several genes controlling flowering. One of them describes change of concentration of FLOWERING LOCUS T mRNA (FTm), which codes for a florigen protein. We changed this ODE to make FTm concentration temperature dependent. This was done by including the temperature coefficient (Q10) (Equation 1) value [6] of FTm production, where T₁ and T₂ are two different temperatures and k(T₁) and k(T₂) are reaction rates at those temperatures.

$katex$Q_{10} = \dfrac{k(T_2)}{k(T_1)}^{\dfrac{\phantom{bb}10}{T_2-T_1}}$katex$

Equation 1: An equation describing temperature coefficient (Q10).

We modified the ODE describing FTm concentration change by making the production of FTm Q10-dependent, while FTm degradation remained uninfluenced by temperature (Equation 2).

$katex$\displaystyle \dfrac{d(FTm)}{\phantom{bbb}dt} = Q_{10} \cdot{} FTm_{production} - FTm_{degradation}$katex$

Equation 2: An equation describing change of FTm concentration over time (d[FTm]/dt), where FTm production is dependent on its Q10 value.

The result is a model with temperature-regulated FTm production, which is able to replicate experimental data found in literature [7] (Figure 2).

Figure 2: A: Relative levels of FT measured experimentally in constant 16 °C and 23 °C [7]. B: Relative FTm levels simulated by the modified plant framework model in constant 16 °C and 23 °C.

We also modified the model to use real life temperature time series to allow for simulating FTm levels from real data. Daily FTm level time series and daily accumulated FTm concentration were added as outputs of the model to be later used for neural network training. Temperature data was retrieved from the Veenkampen weather station repository.

We obtained temperature measurements from the Veenkampen weather station at Wageningen University & Research. The weather station records many atmospheric parameters, including temperature, every half an hour. We downloaded all available daily weather records from 1st of January 2015 to 31st of December 2022. Since data from some days were missing, a total of around 2000 daily temperature time series were collected.

The modified plant framework model takes real daily temperature time series and simulates plant growth and daily FTm concentration change, producing a FTm level time series and accumulated FTm level (dailyFTm) for each day. The modified plant model can be found in this GitLab repository.

We ran the modified plant framework model using real temperature time series. The temperature values and related accumulated FTm levels (dailyFTm) and the time it would take a plant to flower in that temperature (flowering time) were used to train the LSTM neural network.

We constructed an LSTM neural network to predict outputs of the modified plant model from current temperature. The aim of the model was to forecast dailyFTm and flowering time values, as well as daily minimum temperature. We defined temperature time series as input and dailyFTm, flowering time and minimum temperature values as output.

The structure was taken from a previously published deep neural network predicting GFP expression in single cells [3] (Table 1). It was constructed in Python with the TensorFlow library [6]. The code can be found in this GitLab repository.

Layer	Layer type	Number of nodes	Activation function
1	LSTM	64	Leaky ReLu
2	LSTM	16	Leaky ReLu
Output	Dense	3	-

Table 1: Structure of our LSTM model with activation functions of hidden layers.

Data generated by the modified plant model was split into training, validation, and test sets, with the ratio of 70:20:10 respectively. We altered the data in different ways to achieve the highest correlation coefficient between real and predicted values in training, validation, and test sets. Transforming meant standardizing the output values by log10 transformation of dailyFTm and flowering time values and dividing the minimum temperature by 10. This brought the data to similar orders of magnitude, making their correlation clearer [8]. Trimming meant removing data with one of the three output values outside of 1.5 times their interquartile range [9]. We trimmed and transformed the data in different order and combinations and compared correlation coefficients of different methods (Table 2).

Model	dailyFTm train	floweringTime train	mintemp train	dailyFTm val	floweringTime val	mintemp val	dailyFTm test	floweringTime test	mintemp test
Trans_trimm	0.990722	0.986454	0.963777	0.992890	0.992149	0.982335	0.995584	0.995540	0.991955
Trimm_trans	0.741044***	0.798482***	0.845862***	0.741408***	0.808965***	0.855569***	0.727059***	0.801541**	0.847575**
Trimmed	0.349831**	0.368234**	0.315805**	0.384096**	0.349320**	0.315187*	0.441584**	0.366433**	0.37270*
Transformed	0.037512***	0.003147***	0.454247**	0.058699***	-0.010887***	0.220853***	0.063569***	-0.052085***	0.665332*
Unchanged	-0.029234***	-0.255780**	0.189468***	-0.002834***	-0.228909**	0.082170***	-0.042134***	-0.257102**	0.141064**

Table 2: Average correlation coefficient of real and predicted plant model output and minimum temperature values. The average is calculated from ten runs for each variable (dailyFTm – simulated daily FTm accumulation, floweringTime – simulated time to flowering, mintemp – daily minimum temperature) for each data alteration method (Unchanged, Transformed – output values standardised, Trimmed – outliers removed, Trimm_trans – first outliers removed, then output values standardised, Trans_trimm – first output values standardised, then outliers removed) with prediction on training, validation and test data. The table includes p-value *below 0.05, **below 0.01 and ***below 0.0001 from Welch’s test compared to the result of Trimm_trans for each variable.

The neural network was trained ten times for each data transformation technique.
Mean correlation coefficient for training, validation and test set was calculated for each data split.
Mean correlation coefficient values were compared to determine the best data manipulation approach.
Correlation coefficients from each method were compared using Welch’s test to determine how statistically significant the differences in accuracy are.

Transforming and then trimming the data proved to be the most effective. This meant we could test the model’s ability to predict future outputs from present temperature series by using the same data alteration method. To do that, we compared each input value to output values from a day in the future. Then, we repeated the training and testing steps for 1-, 3- and 5-day shifts (Table 3).

Time shift (days)	dailyFTm train	floweringTime train	mintemp train	dailyFTm val	floweringTime val	mintemp val	dailyFTm test	floweringTime test	mintemp test
1	0.314747***	0.130853***	0.212256**	0.221159***	0.062133***	0.202358***	0.569692**	0.035144**	0.162873**
3	0.112230***	-0.176435***	0.145648***	0.134052***	-0.115537***	0.059648***	0.013817***	-0.017328***	0.189414***
5	0.617298**	0.416171**	0.434278***	0.475320***	0.335660***	0.323558***	0.287688***	0.180590***	0.226028***

Table 3: Average correlation coefficient of real and predicted output values. The average is calculated from ten runs for each variable (dailyFTm – simulated daily FTm accumulation, floweringTime – simulated time to flowering, mintemp – daily minimum temperature) for each time shift with prediction on training (train), validation (val) and test data sets. The table includes p-value *below 0.05, **below 0.01 and ***below 0.0001 from Welch’s test compared to the value from each set obtained with the best present prediction method (Trans_trimm).

The results indicated that, even though the prediction of present values was very accurate, future predictions were too inaccurate. This meant that we had to take a different approach to make future predictions.

The LSTM neural network performed well when predicting current daily FT level and flowering time. However, it struggled to predict future values. Therefore, we decided to use a previously published MLP model [8] to predict future temperature.

We used a previously constructed MLP neural network [10] to predict future half-hourly temperature series of temperature values from the current temperature. This MLP model had been shown by the authors to be more accurate at predicting future temperature values than the LSTM model that we constructed.

The whole structure of the neural network was taken from Jallal et al. (2019). We constructed it using the TensorFlow library [6] in Python (Table 4).

Layer number	Layer type	Number of nodes	Activation function
1	Dense	8	tanh
2	Dense	5	tanh
Output	Dense	1	-

However, the authors coded the original model in MATLAB. This caused the originally used Levenberg-Marquardt optimizer to be unavailable, so we used ADAM instead [12]. The code can be found in this GitLab repository.

The authors showed that the original model performs best when shifting the temperature values 4 hours in the future. We tested how the neural network performed when trying to predict temperature time series 1, 3 and 5 days ahead. Then, we compared correlation coefficients between real and predicted values for each time shift (Table 5).

Time shift (days)	Train	Val	Test
1	0.902327	0.762921	0.836295
3	0.810092***	0.452257***	0.366915***
5	0.794803***	0.712413***	-0.053343***

Table 5: Average correlation coefficient of real and predicted temperature values. The average is calculated from 10 runs for each time shift, with prediction on training (train), validation (val) and test data. The table includes p-value *below 0.05, **below 0.01 and ***below 0.0001 from Welch’s test compared to the 1-day shift values.

The neural network was trained ten times for times shift.
Mean correlation coefficient for training, validation and test set was calculated for each time shift.
Correlation coefficients from each method were compared to the 1-day shift values using Welch’s test to determine how statistically significant the differences in accuracy.

We discovered that this MLP model performs better in temperature predictions up to three days in the future than the LSTM model. A significant difference between correlation coefficients when using training and validation sets and the training set suggests that the model is overfitting – this issue has not been yet considered.

Because of its superior performance in temperature prediction, the MLP model is used to predict future temperatures from current measurements carried out in an orchard. This future temperature is then fed to the LSTM model, which we changed to predict future levels of FT in the plant, as well as days required for flowering. We tested the final model predicting future FTm levels in the plant from current temperature time series by calculating correlation coefficients between real and predicted values from training, validation and test sets. The correlation coefficient values were mean values which we calculated from training the neural network 10 times. We carried out the tests with 1-, 3- and 5-day future predictions (Table 6).

Time shift (days)	dailyFTm train	floweringTime train	dailyFTm val	floweringTime val	dailyFTm test	floweringTime test
1	0.600813	0.656140**	0.572494	0.680161**	0.618459	0.724209**
3	0.521414**	0.365461*	0.427223*	0.303132*	0.806855**	0.352567
5	0.495639	0.605488	0.496568	0.621982*	0.586582	0.729452**

Table 6: Average correlation coefficient of real and predicted plant model output values. The average is calculated from 10 runs for each time shift, with prediction on training (train), validation (val) and test data. The table includes p-value *below 0.05, **below 0.01 and ***below 0.0001 from Welch’s test. Values related to each output were compared to their equivalent from Table 3.

We showed that this combined model can predict the concentration of FTm in plants up to three days into the future. Although the agent-based model shows that full root colonisation takes at least four days to occur, antiflorigen secretion will starts before this is obtained and we already know that not much antiflorigen production is needed to prevent flowering. Therefore we hypothesise that application three days before a frost would be sufficient for PseuPomona to fulfil its task at delaying flowering.

[1] Prabha, T., & Hoogenboom, G. (2008). Evaluation of the Weather Research and Forecasting model for two frost events. Computers and Electronics in Agriculture, 64(2), 234–247. https://doi.org/10.1016/J.COMPAG.2008.05.019.

[2] Snyder, R. L., & de Melo-Abreu, J. P. (2005). Frost Protection: fundamentals, practice, and economics. In FAO (Ed.), Environment and natural resources series (Vol. 1).

[3] Klumpe, H. E., Lugagne, J. B., Khalil, A. S., & Dunlop, M. J. (2023). Deep Neural Networks for Predicting Single-Cell Responses and Probability Landscapes. ACS Synthetic Biology, 12. https://doi.org/10.1021/ACSSYNBIO.3C00203/SUPPL_FILE/SB3C00203_SI_001.PDF.

[4] Jallal, M. A., Chabaa, S., El Yassini, A., Zeroual, A., & Ibnyaich, S. (2019). Air temperature forecasting using artificial neural networks with delayed exogenous input. 2019 International Conference on Wireless Technologies, Embedded and Intelligent Systems, WITS 2019, 1–6. https://doi.org/10.1109/WITS.2019.8723699.

[5] Chew, Y. H., Seaton, D. D., Mengin, V., Flis, A., Mugford, S. T., George, G. M., Moulin, M., Hume, A., Zeeman, S. C., Fitzpatrick, T. B., Smith, A. M., Stitt, M., & Millar, A. J. (2022). The Arabidopsis Framework Model version 2 predicts the organism-level effects of circadian clock gene mis-regulation. In Silico Plants, 4(2), 1–15. https://doi.org/10.1093/insilicoplants/diac010.

[6] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., … Research, G. (n.d.). TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. Retrieved October 7, 2023, from www.tensorflow.org.

[7] Mundim, K. C., Baraldi, S., Machado, H. G., & Vieira, F. M. C. (2020). Temperature coefficient (Q10) and its applications in biological systems: Beyond the Arrhenius theory. Ecological Modelling, 431, 109127. https://doi.org/10.1016/J.ECOLMODEL.2020.109127.

[8] Blázquez, M. A., Ahn, J. H., & Weigel, D. (2003). A thermosensory pathway controlling flowering time in Arabidopsis thaliana. Nature Genetics, 33(2), 168–171. https://doi.org/10.1038/NG1085.

[9] How to use Data Scaling Improve Deep Learning Model Stability and Performance - MachineLearningMastery.com. Retrieved October 4, 2023, from https://machinelearningmastery.com/how-to-improve-neural-network-stability-and-modeling-performance-with-data-scaling/?__cf_chl_tk=fmp8VbfyjWzirMgqJUaYtvj7O3_cIcUVMrzX7WZAGyg-1696345286-0-gaNycGzNFHs.

[10] Why “1.5” in IQR Method of Outlier Detection? | by Shivam Chaudhary | Towards Data Science. Retrieved October 4, 2023, from https://towardsdatascience.com/why-1-5-in-iqr-method-of-outlier-detection-5d07fdc82097.