< Back to previous page

Project

Neuromorphic Sensor Fusion and Continual Learning for Drone Navigation and Radar Sensing

Today, there is an ever increasing demand for embedding advanced intelligence
capabilities in energy- and area-constrained battery-powered devices such as
advanced wearables, on-body health monitoring systems, AR / VR headsets and
small robots such as drones. Therefore, there has been a surge in research effort
focusing on the use of Deep Neural Networks (DNNs) and their acceleration in
specialized hardware. However, these DNN solutions are still lagging far behindthe remarkable efficacy of biological brains, such as, for example, in the honey
bee, which is capable of a large range of complex tasks while consuming only
10 μW of power. Hence, the study of event-driven Spiking Neural Networks(SNNs) and local Hebbian plasticity rules (such as Spike-Timing-Dependent
Plastity or STDP) has attracted great attention in recent years, as a more
bio-plausible, neuromorphic model of computation, attempting to replicate in a
more faithful manner the biological neural mechanisms found in the brain. This
emerging AI field, termed Spiking Neuromorphic Computing, is seeking to take
direct inspiration from biology in order to enable a highly accurate execution of
complex AI tasks under the tightest of energy and chip area budgets, where
conventional DNNs would need significantly more energy and area.
Currently, however, a landslide adoption of SNNs and local STDP for online
learning has been jeopardized by a lack of high-accuracy, large-scale real-
world demonstrations on tasks other than traditional pattern recognition using
small-scale databases, and by the limited number of applications with sensory
modalities other than image-type data. To help address these current issues, this
thesis focuses on the algorithmic aspect of SNNs and their practical application
for both radar processing and the navigation of small drones via radar-imaging
sensor fusion, and online learning approaches using STDP. As such, significant
contributions are made to both the theoretical understanding of SNNs and
STDP, as well as to innovative neuromorphic application paradigms.
In this regard, this thesis provides the following contributions. Firstly, the novel
use of SNNs with radar signals is studied with an in-depth comparison between
SNN and DNN performance. It is shown that SNN implementations cannot
simply be substituted for more heavy DNN implementations. In particular, the
crucial importance of pre-processing for signal-to-spike encoding is demonstrated
for the radar gesture application by experimenting with more than 10 different
encoding approaches. It is shown that the impact of radar pre-processing on thetask accuracy is far more important when using SNNs compared to using DNNs.
By using an appropriate pre-processing, it is shown that a 4-bit weight SNN
can reach the same accuracy (93%) compared to a DNN using full-precision 

weights.
Secondly, the experience gathered in radar pre-processing is used to set up a first-of-its-kind drone platform embarking a retina-inspired event-based camera and
a 79 GHz radar sensor for studying drone navigation tasks such as Simultaneous
Localization and Mapping (SLAM), and people detection in an indoor warehousecontext, using a custom radar detector running in the on-board radar MCU.
Thirdly, this thesis provides a novel theoretical framework for the study of SNN-
STDP learning systems. As opposed to prior works that explore SNN-STDP
from a bottom-up perspective, by setting up randomly-connected networks
following direct biological inspiration, this thesis explores SNN-STDP from a
top-down perspective, by first mathematically formulating the optimization
problem to be solved (based on joint Dictionary Learning and Basis Pursuit) and
then deriving the underlying SNN-STDP substrate architecture that provably
solves this optimization problem through its neural and weight dynamics. It is
shown that the proposed top-down approach significantly outperforms the prior
bottom-up works in terms of accuracy for common event camera benchmark
datasets.
Fourthly, after the validation of the proposed SNN-STDP theory using
benchmark tasks, this thesis proposes what is, to the best of our knowledge,
the first SLAM system for drones, fusing event camera and radar data using an
SNN-STDP architecture that continuously learns to adapt itself to its unknown
environment in an unsupervised manner, and without the need for any offline
pre-training phase. Instead, the drone starts flying in an unknown indoor
warehouse that was not captured in the SNN during any prior offline training
phase; during the flight, the SNN extracts features from its sensory data bylearning the structure of the environment on the fly using STDP. Remarkably,
it is shown that the proposed system outperforms offline SNN training in terms
of SLAM accuracy.
Fifthly, the applicability of the proposed SNN-STDP design framework to other
large-scale real-world tasks is explored by studying the continual learning of
people detection by drones, using the proposed top-down SNN-STDP theory
under a convolutional network setting. Experiments are conducted for comparing
the performance of the proposed continual learning system against conventional
CNN models trained offline. Here, it is shown that the proposed SNN-STDP
system outperforms the use of a same-size CNN by reaching a gain of more
than 19% on the peak F1 score (used as a conventional measure of detection
performance).
Finally, in order to pave the way towards the use of continual-learning SNN-
STDP for the control of dynamical agents, this thesis explores how continual
Hebbian learning networks can be used within an Active Inference (AIF) scheme
in order to learn agent control from sensory observations. Experimental studies
are conducted using the Mountain Car environment from the OpenAI gym
suite, to study the effect of the various Hebbian network parameters on the task
performance. It is shown that the proposed Hebbian AIF approach outperforms
the use of Q-learning (as popularly used to solve Mountain Car), while not
requiring any replay buffer, in contrast to typical reinforcement learning systems.
In this way, this thesis significantly contributes to the algorithmic aspect of
SNNs and STDP learning techniques, helping towards a wider adoption of
neuromorphic computing for extreme-edge AI applications.

Date:25 Sep 2020 →  Today
Keywords:AI, Radar, Drones
Disciplines:Neuromorphic computing
Project type:PhD project