< Back to previous page

Project

Design and implementation of a real-time gloss free SWIR hyperspectral imaging based fruit sorting system

Quality assessment and process monitoring are essential for today's fruit industry sector and the world's economy. From picking fruit in orchards, to transport and handling practices, to storage and packaging, each step will influence the quality of the end-product when presented to the consumers. The appearance, consisting of the shape, size, colour or absence of any damage are essential criteria relevant to consumers, and influence their will on buying. It is therefore essential to provide a fast, consistent quality assessment of each fruit to match the expectations of the market. To remain competitive to this demand at low cost, fast and efficiently, non-destructive automated quality sorting lines are needed. Among the different defects affecting fruit quality, bruises are one of the most problematic industrial losses. The detection of bruises in fruit such as apples during handling is therefore required. The browning process of bruises results in progressive apple tissue softening and colour changes. As this natural process takes time before it becomes visible, there is a gap of a few days between the mechanical damage causing the bruise and the consequent visible brown spots, which lower the price of apples.  It is therefore important to detect bruises on each apple as early as possible after damaging to limit consequent economical losses.

Light is the fastest known information carrier. In ambient conditions, it is also harmless for fruit or the surrounding workers, and a cheap technology. Among the different non-destructive and non-invasive techniques, the usage of light and the analysis of the information it can carry is therefore the most promising path. By shining light onto apples, and observing the absorbed, reflected and scattered light, bruises may be detected non-destructively at high speed. Among the most recent technologies reported, hyperspectral imaging (HSI), being the combination of the machine vision and spectroscopy fields, is showing promising paths. More particularly, the short-wave infra-red (SWIR) range has been demonstrated to promote successful detection of bruises in apples at early stages. However, there are still limiting factors when using SWIR HSI prior its success in industry. Among them, the most predominant are high noise levels arising from the detectors, non-uniform illumination, specular reflections and real-time HSI data handling.  This research aims to tackle those problems be first describing and modelling the different components consisting of a SWIR HSI sorting system being the illumination and the imager, and further optimizes their configuration for better image quality. Those building blocks are further put together combined with improved data handling more robust and efficient usage of SWIR HSI in industry. This research is split into 10 chapters.

Chapter 1 covers the current practices in image based fruit sorting, with a stronger emphasis on hyperspectral imaging. It further compares visible and near infra-red (Vis-NIR) to SWIR HSI and where are the additional challenges when using SWIR over Vis-NIR. The chapter then describes the relevance of apples in industry and why early bruise detection. The chapter ends with the outline of this thesis.

Chapter 2 describes the state-of-the-art in light, its interaction with matter, with a focus on polarisation and vibrational spectroscopy. The browning bio-chemical process of apples is further described. The algorithms used to process light spectral information, also referred to multivariate data analysis which are applied within this research are then described.   

Chapter 3 is focused on characterising the noise and sensitivity of a SWIR hyperspectral imager, to quantify the signal to noise ratio (SNR). To quantify the pixel-to-pixel variation or the detector’s response, a radiometric calibration method is proposed which dynamically removes the detector noise. This approach removed 6% further noise compared to conventional sequential noise sampling methods.  The average detector noise or dark current evolution through time is then shown, which was noted to vary non-linearly, with a sub-linear trend one hour after start-up to stabilize after 3 hours, with up to 12% of the imager’s dynamic range. Contrast of each spectral image is also described using a novel custom-made checkerboard calibration rig. The following showed a ratio per wavelength up to 15000 versus 1 raw values with 100-1700 nm. The checkerboard also enabled accurate spatial calibration using a thin lens model.     

Chapter 4 describes how to measure, model analytically and using non-sequential ray-tracing software the spectral and angular distribution of halogen tungsten (HT) spots, considered as the standard in SWIR HSI illumination. The far field angular distribution was modelled with a Gaussian distribution with an R² of 0.99, while the spectra using a Plank based 5th order polynomial with and R² of 0.98. The modelled spectra enabled to convert photometric measurements into radiometric units, and estimate the energy contribution of the spots in the SWIR spectral range, with up to 63% of the total spectral power. Further, near-field spot distribution is measured and modelled within the ray-tracing software, comparing irradiance distributions when using or not diffusers. It was shown that the irradiance patterns could be reproduced with a peak relative error of 12% when using diffusers, while up to 30% without.

Conventional illumination distribution and light beam shaping are non-linear problems, which often are solved using iterative methods such as simplex or simulated annealing (SA) optimization algorithms, which can result in sub-optimal solutions or time consuming searches. Chapter 5 introduces novel constrained non-linear global optimization algorithms which can handle more efficiently such problems while simultaneously offering information on the sensitivity of a configuration near its optimum. A design is proposed using 4 HT spots placed around a flat target, using the source models from chapter 4. The two proposed optimization methods are referred to as Design of Computer Experiments with Design Augmentation (DACEDA) and Design of Computer Experiments with Simplex post-optimization (DACES). A 2 variables analytical version of the problem using isotropic source models enabled to compare DACES and DACEDA’s modelled design space with an overall average relative error of 2%, with a peak up to 10% at the corners of the design space. The SNR of ray-traced near-field sources modelled in chapter 4 is quantified using the Rose model, setting the stop criterion of the proposed optimization algorithms. The simulated irradiance distribution uniformity is then optimized for a 2 and 5 variables case studies with DACES, DACEDA, simplex and SA. In the 2D case, it was shown that DACES performed best after 30 simulations while in the 5D case, DACEDA performed best after 65 simulations. Both algorithms were further used in a case study with DACEDA for tolerance analysis, and DACES for optimization of a configuration for apples, which was used within the remaining of the thesis.   

Among the main challenges when using SWIR HSI for fruit quality inspection, are the glossy regions observed from their arbitrary deformed toroidal shape and waxy surface. Therefore, this research further aimed at reducing the influence of those specular reflections, both numerically and optically.

In chapter 6, a first proposed approach is to use chemometrics tools combined with image analysis to reduce or remove those artefacts, using a multiclass classifier or a stepwise approach. The proposed method using iterative steps to remove progressively automatically unwanted regions resulted in 6% higher prediction accuracy than a multi-class partial least-squares discriminant analysis (PLS-DA) classifier. Appropriate wavelength selection using interval PLS-DA enabled to improve further by 4%. Furthermore, the stepwise algorithm enabled to detect for multiple cultivars up to 80% six hours after bruising.

Chapter 7 uses the multi-class PLS-DA classifier from chapter 6 on a real-time case study of one cultivar, and compare pixel based calibration models to conventionally used region based ones. Pixel based models, encountering for variations described in chapter3, improved prediction accuracy at the pixel level up to 2%. With a cultivar based model built for 2 hours after damage, using area normalization as spectral pre-processing and image post-processing, a pixel-based prediction of accuracy of 95.6% was obtained, while up to 98% at the sample level. The following was demonstrated on a real-time SWIR HIS sorting system at a rate of 200 ms per apple at a scanning speed of 0.3 m/s.

To further improve those results, chapter 8 aims at quantifying the degree of glossiness for apples as a function of the light geometrical path, also referred to as surface scatter properties or bi-directional scatter distribution function (BSDF). It was shown that apples have a Gaussian gloss trend around the specular angle, and are Lambertian outside the glossy region.

 

 

 

 

Moreover, polarization properties of apples are then investigated in chapter 9, in the aim to remove optically gloss arising from apples using a cross-polarized imaging system. It was shown that gloss could be removed for multiple cultivars using cross-polarization, and that the resulting scattered reflected light was Lambertian, thus improving the image uniformity and bruise-sound contrast region.

Finally, the combination of the conclusions drawn from the different chapters and future research perspectives are given in chapter 10. It can be concluded that near-field ray-traced sources with diffusers are the best choice for SWIR illumination, which can be optimized using DACES or DACEDA for improved uniformity. It was shown in this research that real-time SWIR HSI is possible, and using broad spectral has significant added value. It was shown that using the knowledge of polarization and surface scatter properties of apples, linear cross-polarized imaging configuration is the most efficient solution to remove gloss of fruit in images. Alternatives are also possible, by numerically removing gloss using area-normalization and PLS-DA, with image post-processing, which are cheaper, but more time consuming.

Date:12 Dec 2011 →  16 Jun 2017
Keywords:SWIR, hyperspectral imaging, gloss free, cross-polarisation, real-time food sorting, early bruise detection
Disciplines:Other chemical sciences, Nutrition and dietetics, Agricultural animal production, Food sciences and (bio)technology, Analytical chemistry, Macromolecular and materials chemistry, Agriculture, land and farm management, Biotechnology for agriculture, forestry, fisheries and allied sciences, Fisheries sciences
Project type:PhD project