A multimodal AI approach for intuitively instructable autonomous systems

Subtitle:a case study of an autonomous off-highway vehicle

In current production shop floors, a fleet of production machines and AGVs form a full manufacturing system with a high degree of automation. These current manufacturing systems need to deal with high variability of products and production tasks. Every task, however, requires a proper reconfiguration & control that is often done manually requiring complex settings and long configuration time. With AI techniques the reconfiguration of these systems to deal with a new task can be made more intuitive. In this paper we present the upgrade of an autonomous system, used for manufacturing assets handling and transportation, with AI features that make it easy to reconfigure in order to deal with high variability of assets and missions. Visual and spoken information is used to instruct and guide the autonomous vehicle using an AI multimodal framework where first, spoken language, with different local dialects, is translated to digital instructions, that can be associated to visual information to form control instructions to the autonomous vehicle. Different AI models, respectively for spoken language understanding, visual perception, vision based navigation are associated through a multimodal AI framework to intuitively control the AGV to perform a specific task. Beside the challenges related to the integration of these models in the AGV platform, other challenges related to dealing with variabilities of dialects, objects, surroundings and ambient conditions are partly tackled in this research.

Publication year:2022

Keywords:P2 Proceeding

Accessibility:Closed