Competition Overview
Multiple studies have proven that many UAV bugs can be potentially detected before field tests if proper simulation-based testing is in place. This suggests the need for further research on setting up simulation environments that test UAVs' behavior in diverse, complex, and realistic scenarios.
However, the engineering complexity of UAVs and their test environments, and the difficulty of setting up realistic-enough simulation environments that can capture the same bugs as physical tests represent relevant obstacles.
In the UAV Testing Competition, we aim to provide software testing researchers with a simple platform to facilitate their onboarding in the UAV domain. Using the provided platform and case study, the goal is to use search-based techniques to generate challenging test cases for autonomous vision-based UAV navigation systems.
- The Software Under Test is PX4-Avoidance, a vision-based autonomous obstacle avoidance system developed on top of PX4-Autopilot.
- We create challenging scenarios for PX4-Avoidance by placing static obstacles on the UAV's path.
- The ultimate goal is to find some specific obstacle configurations (size, position) that could lead to a crash or unsafe flight by the autopilot, as seen in the image below.
In the tool competition, each participant presents a robust test generator capable of generating a diverse set of tests. The primary objective is to find potential vulnerabilities within the PX4 obstacle avoidance system. This involves manipulating obstacle sizes and placements within the test environment, with the ultimate goal of either causing the UAV to crash or significantly diverting it from its intended path.
The goals of the tool competition are as below:
- The objective is to develop a test generator capable of creating diverse and effective tests to uncover vulnerabilities within the PX4 avoidance system.
- The generated test will be for a predefined UAV firmware, model, and mission.
- The generated test will create a challenging environment by manipulating object sizes and placements to cause either UAV crashes or significant deviations in its flight path.
The effectiveness of these generated tests will be measured based on the number of failed cases and the diversity of test scenarios. The goal is to identify potential system weaknesses comprehensively.
-
PX4 : PX4 is an open-source autopilot software stack primarily used for controlling unmanned aerial vehicles(UAVs). It provides a flexible and customizable platform for designing and controlling the drones, including capabilities for navigation, stabilization, and mission planning. PX4 is compatible with various hardware platforms and is widely used in both academic and commercial drone applications. It supports a range of UAV types, from small quadcopters to fixed-wing aircraft and even VTOL (Vertical Take-Off and Landing) vehicles. Developers and researchers often use PX4 as a foundation for creating and testing new drone capabilities and applications.
-
PX4 Avoidance : PX4 Avoidance is a software module in the PX4 Autopilot ecosystem that provides obstacle detection and avoidance capabilities. PX4 Avoidance uses various sensors and algorithms to help UAVs navigate and avoid obstacles in their environment. It allows UAVs to detect obstacles such as buildings, trees, and other objects in the path and adjust their flight path to avoid collisions and navigate safely around these obstacles. Overall, PX4 Avoidance is a critical component for ensuring the safe and reliable operation of UAVs in complex and dynamic environments.
-
PX4 Flight Logs: PX4 flight logs are comprehensive records of a drone's operational data and telemetry during its flights. These logs include detailed information such as GPS coordinates, altitude, motor RPM, sensor data, and flight modes. They are invaluable for troubleshooting, performance analysis, and debugging, as they allow developers and operators to examine precisely what happened during a flight, identify potential issues, and fine-tune the drone's behavior and systems for optimal performance and safety. These logs are stored in a standardized format (.ulg), making them compatible with various analysis and visualization tools for in-depth technical examination. Here is a sample flight log.
-
Gazebo : Gazebo is an open-source 3D robot simulator that provides a realistic and physics-based simulation environment for testing and validating UAVs and robotic systems. PX4 often utilizes Gazebo as a simulation platform to create virtual environments where developers and researchers can test UAVs without the need for physical hardware. This allows for various scenarios, including flight testing, obstacle avoidance, and mission planning, to be tested in a safe and controlled virtual environment. Gazebo simulates the physical properties and dynamics of the UAV and its surroundings, including sensors, wind, and terrain. It is a valuable tool for both software and hardware development, as it enables testing and debugging of UAV control algorithms and systems before deploying them to actual UAV hardware.
Aerialist (unmanned AERIAL vehIcle teST bench) is a novel test bench for UAV software that automates all the necessary UAV testing steps: setting up the test environment, building and running the UAV firmware code, configuring the simulator with the simulated world properties, connecting the simulated UAV to the firmware and applying proper UAV configurations at startup, scheduling and executing runtime commands, monitoring the UAV at runtime for any issues, and extracting the flight log file after the test completion.
With Aerialist, we aim to provide the competition participants with an easy platform to automate tests on the simulated UAVs, allowing them to do experiments required to overcome the UAV simulation-based testing challenges. The Test Generators submitted to the competition are required to build on top of Aerialist to simplify the evaluation process. Check Aeialist's Documentation for more details on the usage.