Towards an autonomous flying cameraman

This PhD research project focuses on the development of a robotic flying camera man. Within the Cametron project we aim at making the process of filming an event completely automatic. A human camera camera crew will be replaced by automated static PTZ cameras and UAV-drones sporting a camera and microphone. The movie director will be replaced by a virtual director so that an event, for example a music performance or a sports game, can be captured in real time with minimal human intervention. This project focuses on the virtual camera man part of Cametron, in which an unmanned aerial vehicle (UAV), sporting a PTZ camera and microphone, is robotized such that it totally autonomously can gather video of an event, based on high abstraction level instructions from the virtual director. These instructions are no more detailed than framing a certain (set of) actor(s) in a certain cinematographic shot composition (long shot, shoulder shot, mid shot, close-up, extreme close-up, etc.). The main challenges that have to be conquered in this project are real-time actor detection and tracking, topological localization, image-based visual servoing and motion planning. Indeed, first task is the detection of persons in the UAV's camera. Because of the very broad definition of ‘actor’ in this project (e.g. lecturer, rock star, cyclist), this task requires fast general person detectors and trackers to be developed to keep the right object centered in the image while objects and/or cams move. Tracking trough time can be done by the well-known “tracking-by-detection”-framework, but for this specific application extreme conditions apply, such as unconstrained varying viewpoints and the very limited amount of on-board computing power. Therefore real time person and action tracking algorithms should be implemented on specific hardware that fits on the units. Knowing its viewpoint is crucial for each camera unit. Yet, for the present application, it is not necessary to know the exact metric location. The quality of the source data matters more than exact localization. In order to record data according to the orders of the director, a unit needs to know its approximate, relative position w.r.t. other units and the action. Therefore, a qualitative, topological layout of the camera positions should suffice. A topological model of the (dynamic) environment will be developed in which each flying camera can localize itself. The last challenge in the project is visual servoing. We also want the drones to fly very smooth so that the captured images aren't blurry or shaking, therefor a good control algorithm will be designed for the drones. Moreover, because the UAVs have 6 degrees of freedom, and the PTZ unit mounted on the adds another 3 DOFs, this control system is certainly not trivial. Fortunately, because in this application, the image produced by the on-board camera is most important, rather than the exact position of the UAV, we can use the image-based visual servoing (IBVS) paradigm. The first goal we want to reach is capturing a lesson or presentation with static PTZ cameras that automatically pan, tilt and zoom to follow the speaker. Afterward, we will expand this to track and capture a single person with a single camera mounted on a UAV-drone and the final goal is to capture an event with several flying units collaborating together.

Jaar van publicatie:2018

Toegankelijkheid:Closed