Reinforcement learning-based energy management for sustainable hybrid fixed-wing UAVs

Environmental sustainability has become a priority across all sectors, including aerospace. Developing a sustainable energy system for Unmanned Aerial Vehicles (UAVs) is crucial to minimize ecological impact while maintaining operational efficiency. This requires integrating multiple energy sources and balancing power generation, storage, and consumption. Learning-based methodologies offer considerable potential to optimize UAV energy management due to their robustness and adaptability. This paper presents an investigation into AI-driven energy management strategy for a fixed-wing UAV equipped with a hybrid energy system comprising fuel cells (FC), photovoltaics (PV), and batteries. Reinforcement Learning (RL), specifically the Soft Actor-Critic (SAC) algorithm, is employed to optimize operation of the hybrid energy system components to supply the power required for straight flight while keeping the battery within safe limits. The system components are modeled and integrated into a training environment using OpenAI Gym and Stable-Baselines3. To capture realistic conditions, turbulence is modeled and used to compute the power demand during flight. The investigation examines four reward structures and evaluates their influence on training outcomes, as well as the impact of turbulence intensity on policy learning. Agents are evaluated based on training rewards and hybrid system performance. Our results show that SAC can meet UAV power demands while keeping the battery within safe limits, and that a reward structure prioritizing power tracking and battery safety, combined with moderate turbulence during training, yields the best performance. These findings provide practical guidelines for choosing reward structures and disturbance levels when training RL-based UAV energy-management controllers.