In our research, we investigate the application of reinforcement learning techniques for unmanned aerial vehicle (UAV) maritime search path planning. The core contribution of this work is the development of a Proximal Policy Optimization (PPO) based framework that enables UAV agents to autonomously plan efficient search trajectories over uncertain maritime environments. Throughout this study, we emphasize the transformative potential of drone technology in enhancing maritime search and rescue operations.
Maritime emergency rescue constitutes a critical component of ensuring safety in ocean activities. Traditional search and rescue operations primarily rely on manned helicopters and rescue vessels. However, these conventional approaches face significant limitations in terms of response time, operational cost, and coverage capability. The emergence of advanced drone technology offers a compelling alternative that can fundamentally transform maritime search missions. UAVs possess unique advantages including flexible deployment, reduced operational costs, rapid response capabilities, and the ability to cover vast oceanic areas efficiently.
Our proposed methodology addresses the fundamental challenge of target location uncertainty in maritime environments. Unlike terrestrial search scenarios where terrain obstacles are primary concerns, maritime search is characterized by dynamic target drift caused by ocean currents, wind, and wave actions. This inherent uncertainty necessitates sophisticated path planning algorithms that can adapt to evolving probability distributions of target locations.
Maritime Search Problem Formulation
We formulate the maritime search problem as a sequential decision-making process where a UAV agent must optimally allocate search effort across a discretized search region. The search area is partitioned into an M × N grid, where each cell represents a unit area that the UAV can search. For each grid cell indexed by coordinates (m, n), we define a containment probability Pmn that represents the likelihood of the target being located within that cell.
The probability distribution is derived from drift prediction data provided by maritime search environment support platforms. These predictions account for environmental factors including wind speed, ocean currents, wave height, and visibility conditions. Table 1 summarizes the key parameters that influence the sweep width of fixed-wing UAVs during maritime search operations.
| Search Target | Altitude 150 m (m) | Altitude 300 m (m) | Altitude 600 m (m) |
|---|---|---|---|
| Person in Water | 370 | 370 | — |
| 4-Person Life Raft | 7,600 | 8,000 | 8,000 |
| Vessel < 5 m | 6,100 | 6,800 | 7,600 |
| Vessel 12 m | 35,700 | 35,700 | 39,800 |
The effective sweep width is further modified by environmental conditions as specified in Tables 2 and 3. When weather conditions are favorable, we set the UAV sweep width to 300 m for typical search operations.
| Wind Speed (km/h) | Wave Height (m) | Person in Water | Life Raft |
|---|---|---|---|
| 0-28 | 0-1 | 1.0 | 1.0 |
| 28-46 | 1-1.5 | 0.5 | 0.9 |
| >46 | >1.5 | 0.25 | 0.6 |
| Visibility (km) | 6 | 9 | 19 | 28 | >37 |
|---|---|---|---|---|---|
| Coefficient | 0.4 | 0.6 | 0.8 | 0.9 | 1.0 |
UAV Agent Model and State Space Design
In our framework, we model the UAV as a point-mass agent operating in a two-dimensional plane. This simplification is justified because maritime search environments lack terrain elevation variations, allowing us to focus exclusively on the path planning optimization problem. The UAV agent maintains a constant cruising speed and a fixed sweep width throughout the mission, and we assume sufficient endurance to cover the designated search area.
The state space is constructed by combining the probability matrix with real-time search status information. We maintain a visitation matrix visitedmn(t) that tracks whether each grid cell has been searched, is currently being searched, or remains unexplored at time step t. The state representation is defined as:
The state vector incorporates the following components as shown in Table 4.
| Component | Data Type | Dimension | Description |
|---|---|---|---|
| Grid Probability Matrix | Float | M × N | Normalized containment probabilities |
| Visitation Status | Boolean | M × N | Whether cells have been searched |
| Agent Position | Integer | 2 | Current coordinates (x, y) |
| Previous Action | Integer | 1 | Action taken at t-1 |
| Current Cell Probability | Float | 1 | Probability of current location |
| Current Cell Status | Boolean | 1 | Whether current cell is searched |
Action Space Definition
The action space determines the movement decisions available to the UAV agent at each time step. In a grid-based environment, the agent could theoretically move to any of the eight neighboring cells. However, based on the International Aeronautical and Maritime Search and Rescue (IAMSAR) Manual guidelines, we restrict the action space to four cardinal directions: up, down, left, and right. This restriction ensures continuous coverage of the search area and aligns the grid cell size with the UAV’s sweep width.
Formally, the action at time step t is defined as:
$$a_t \in \{1, 2, 3, 4\}$$
where the mapping is: 1 = up, 2 = right, 3 = down, 4 = left.
Reward Function Design
The design of an appropriate reward function is crucial for guiding the reinforcement learning agent toward optimal search behavior. We base our reward function on Koopman’s optimal search theory and incorporate mechanisms to address the sparse reward problem commonly encountered in reinforcement learning applications.
The immediate reward for searching a grid cell at time t is defined as:
$$r_{mn}(t) = \begin{cases} P_{mn} \cdot w \cdot g^{t}, & \text{if } \text{visited}_{mn}(t) = 0 \\ r_{\text{punish}}, & \text{if } \text{visited}_{mn}(t) = -1 \end{cases}$$
where Pmn represents the prior probability of cell (m, n), w is the reward weight parameter that amplifies the contribution of high-probability regions, g ∈ [0, 1] is the decay factor that reduces reward value over time to reflect decreasing search effectiveness, and rpunish is the penalty for revisiting previously searched cells.
To maintain training stability, we normalize the immediate rewards:
$$\tilde{r}_t = \frac{r_t – r_{\text{min}}}{r_{\text{max}} – r_{\text{min}}}$$
The cumulative discounted reward for each episode is computed using the temporal difference method:
$$R_t = r_t + \gamma \cdot r_{t+1} \cdot (1 – d_t), \quad \gamma \in [0, 1]$$
where γ is the discount factor that balances immediate versus future rewards, and dt is the termination flag (0 for non-terminal, 1 for terminal states).
At the end of each episode, we incorporate a path efficiency reward to encourage comprehensive coverage:
$$r_e = \begin{cases} \frac{E}{L} \cdot B, & \text{if } \frac{E}{L} \geq 0.6 \\ 0, & \text{otherwise} \end{cases}$$
where E is the number of unique cells explored, L is the episode length, and B is the base reward value. The 60% threshold ensures that efficiency rewards are only granted when the agent achieves meaningful coverage of the search area.
The total reward for each episode is:
$$G_t = \sum_{0}^{L} R_t + r_e$$
Proximal Policy Optimization Algorithm
We adopt the Proximal Policy Optimization (PPO) algorithm for training our UAV search agent. PPO represents a significant advancement over traditional policy gradient methods by incorporating a trust region constraint that stabilizes training while maintaining implementation simplicity. The algorithm comprises four main phases: initialization, data sampling, advantage estimation, and policy optimization.
Network Architecture
Our implementation employs an Actor-Critic architecture with fully connected neural networks. The Actor network outputs a probability distribution over actions, while the Critic network estimates the state value function. Both networks use ReLU activation functions and the Softmax function for probability normalization.
The policy πθ(at|st) represents the probability of taking action at given state st, parameterized by θ. The Critic network Vφ(st) estimates the expected cumulative reward from state st, parameterized by φ.
Data Sampling Phase
During data sampling, the agent interacts with the environment by sampling actions from the current policy:
$$a_t \sim \pi_{\theta}(a_t|s_t), \quad \pi_{\theta}(a_t|s_t) = \text{Softmax}(f_{\theta}(s_t))$$
The log probability of each action is computed for subsequent policy gradient calculations:
$$\log \pi_{\theta}(a_t|s_t) = \log \frac{e^{f_{a_t}(s_t)}}{\sum_a e^{f_a(s_t)}}$$
Advantage Estimation
We employ Generalized Advantage Estimation (GAE) to compute advantage values that balance bias and variance. GAE computes the exponentially weighted average of multi-step temporal difference errors:
$$\delta_t = r_t + \gamma \cdot V_{\phi}(s_{t+1}) \cdot (1 – d_t) – V_{\phi}(s_t)$$
$$A^{\text{GAE}(\gamma, \lambda)}_t = \sum_{k=0}^{\infty} (\gamma \lambda)^k \delta_{t+k}$$
where λ ∈ [0, 1] is the GAE parameter that interpolates between Monte Carlo estimation (λ = 1) and temporal difference learning (λ = 0). The return is then computed as:
$$R_t = A^{\text{GAE}}_t + V_{\phi}(s_t)$$
Policy Optimization
The core innovation of PPO lies in its clipped surrogate objective function, which constrains policy updates to prevent destructive large updates. The importance sampling ratio is:
$$r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$$
The clipped objective function is:
$$L^{\text{CLIP}}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) \cdot A^{\text{GAE}}_t, \text{clip}(r_t(\theta), 1 – \varepsilon, 1 + \varepsilon) \cdot A^{\text{GAE}}_t \right) \right]$$
where ε is the clipping range that limits the policy update magnitude. We also compute the value function loss using mean squared error:
$$L^{\text{VF}}(\phi) = \mathbb{E}_t \left[ \left( V_{\phi}(s_t) – R_t \right)^2 \right]$$
To encourage exploration, we add an entropy bonus:
$$H(\pi_{\theta}) = -\sum \pi_{\theta}(a|s) \log \pi_{\theta}(a|s)$$
The total loss function combines these components:
$$L^{\text{TOTAL}} = L^{\text{CLIP}}(\theta) + c_1 L^{\text{VF}}(\phi) – c_2 H(\pi_{\theta})$$
where c1 and c2 are weighting coefficients for the value function and entropy terms, respectively.
Hyperparameter Optimization
Through extensive experimentation, we optimized the key hyperparameters for our maritime search scenario. Table 5 presents the final hyperparameter configuration.
| Hyperparameter | Value | Description |
|---|---|---|
| Discount Factor (γ) | 0.995 | Balances immediate vs. future rewards |
| Clip Range (ε) | 0.2 | Limits policy update magnitude |
| Learning Rate (α) | 6 × 10-5 | Controls parameter update speed |
| Update Epochs | 4 | Number of optimization passes per batch |
| Total Episodes | 20,000 | Total training episodes |
| Episode Length (L) | 120 | Steps per episode |
Our hyperparameter analysis revealed several important insights about the training dynamics. The discount factor γ = 0.995 provides an optimal balance between considering long-term search efficiency and maintaining training stability. Lower discount factors (e.g., γ = 0.9) caused training oscillations, while higher values extended convergence time without significant performance gains.
The learning rate α = 6 × 10-5 achieved the best trade-off between convergence speed and final policy quality. Higher learning rates (e.g., 8 × 10-5) led to faster initial improvement but resulted in suboptimal final policies, while lower rates (e.g., 3 × 10-5) required excessive training episodes without reaching optimal performance.
The clip range ε = 0.2 provides adequate flexibility for policy improvement while maintaining the trust region constraint that prevents catastrophic policy updates. This configuration ensures stable and efficient learning throughout the training process.
Simulation Results and Performance Analysis
We validated our proposed method through comprehensive simulations based on a realistic maritime accident scenario. The simulation parameters assumed an incident in the East China Sea under favorable weather conditions, with drift prediction data obtained from the national maritime search environment support platform.
The training convergence analysis is presented through reward and loss curves. Our results demonstrate that the algorithm achieves stable convergence after approximately 10,000 episodes, with the reward function steadily improving and the loss function decreasing toward zero.
Search Efficiency Comparison
We evaluated the search efficiency of our proposed method against two baseline approaches: the standard parallel track search method recommended by IAMSAR Manual and a genetic algorithm (GA) based approach. The search efficiency metric is defined as:
$$E_f = \frac{\sum_{i=1}^{L} p_i}{L}$$
where L is the total number of search steps and pi represents the prior probability of each visited cell. This metric quantifies how effectively the algorithm prioritizes high-probability regions during the search mission.
Table 6 presents the comparative search efficiency results across different search step intervals.
| Steps | PPO Algorithm | Genetic Algorithm | Parallel Track Method |
|---|---|---|---|
| 20 | 0.4679 | 0.3546 | 0.0492 |
| 40 | 0.4929 | 0.4493 | 0.0878 |
| 60 | 0.5075 | 0.4086 | 0.1568 |
| 80 | 0.4988 | 0.4015 | 0.1756 |
| 100 | 0.4838 | 0.3919 | 0.2112 |
| 120 | 0.4512 | 0.3941 | 0.2383 |
The results demonstrate that our PPO-based method achieves significantly higher search efficiency, particularly during the early stages of the search mission. The search efficiency initially increases as the agent focuses on high-probability regions, reaching a peak at approximately 60 steps, before gradually declining as the remaining unexplored regions contain lower probability values.
In contrast, the parallel track method shows monotonically increasing efficiency, indicating that this approach does not prioritize high-probability regions early in the search. The genetic algorithm exhibits intermediate performance but remains inferior to our proposed method throughout the search mission.
Cumulative Probability Analysis
We further evaluated the cumulative probability achieved by each method, which represents the total probability mass covered during the search. Table 7 presents the cumulative probability values at different step intervals.
| Steps | PPO Algorithm | Genetic Algorithm | Parallel Track Method |
|---|---|---|---|
| 20 | 0.125 | 0.095 | 0.013 |
| 40 | 0.265 | 0.215 | 0.048 |
| 60 | 0.410 | 0.325 | 0.108 |
| 80 | 0.538 | 0.428 | 0.190 |
| 100 | 0.652 | 0.521 | 0.295 |
| 120 | 0.738 | 0.608 | 0.415 |
The cumulative probability results confirm that our proposed method consistently outperforms both baseline approaches. After 120 search steps, the PPO-based method achieves a cumulative probability of 0.738, compared to 0.608 for the genetic algorithm and 0.415 for the parallel track method. This represents a 21.4% improvement over the genetic algorithm and a 77.8% improvement over the conventional parallel track method.
Robustness Analysis
To evaluate the robustness of our algorithm, we conducted experiments under different probability map configurations, including variations in search area size and probability distribution patterns. We computed the normalized search efficiency and statistical metrics to assess algorithm stability.
The normalized search efficiency for different configurations is computed as:
$$E_{f,k} = \frac{\sum_{i=1}^{L} p_i}{\sum_{i=1}^{M \cdot N} p_i \cdot L} \cdot \eta$$
Table 8 presents the statistical analysis results across 50 simulation runs.
| Metric | Value |
|---|---|
| Sample Size | 50 |
| Mean Efficiency | 0.3336 |
| Standard Deviation | 0.0412 |
| Coefficient of Variation (CV) | 12.35% |
| Interquartile Range (IQR) | 0.0493 |
The coefficient of variation CV = 12.35% is well below the 15% threshold, indicating excellent stability across different probability map configurations. This demonstrates that our method maintains consistent performance regardless of the specific spatial distribution of target probabilities. The low IQR value further confirms the concentration of performance metrics around the mean.
Analysis of Search Path Characteristics
Our path planning results reveal distinctive characteristics that differentiate the PPO-based approach from traditional methods. The reinforcement learning agent develops an intelligent search strategy that naturally prioritizes high-probability regions during the early phase of the mission.
In the initial search phase (steps 1-40), the agent focuses on the highest probability concentration regions, achieving rapid accumulation of search probability. This behavior aligns with the optimal search theory principle of allocating search effort proportional to probability density.
During the intermediate phase (steps 41-80), the agent expands the search coverage to secondary probability regions while maintaining efficient transitions between high-value areas. The path demonstrates smooth trajectories that minimize redundant coverage and maximize the discovery of new high-probability cells.
In the final phase (steps 81-120), the agent systematically covers the remaining lower-probability regions, ensuring comprehensive coverage of the entire search area. The algorithm demonstrates the ability to balance between exploitation of known high-probability regions and exploration of unexplored areas.
The path efficiency analysis reveals that the PPO-based method achieves significantly higher search efficiency compared to alternative approaches. The efficiency peaks at approximately 60 steps, indicating that the algorithm effectively identifies and prioritizes the most valuable search regions early in the mission.
Impact of Drone Technology on Maritime Search Operations
The advancement of drone technology has fundamentally revolutionized maritime search and rescue operations. Our research demonstrates how modern drone technology can be leveraged to significantly improve search efficiency in challenging maritime environments. The integration of reinforcement learning with drone technology enables autonomous decision-making capabilities that were previously unattainable with conventional search methods.
The key advantages of incorporating drone technology into maritime search operations include:
1. Rapid Deployment: Modern drone technology allows for immediate launch and response, significantly reducing the time between incident occurrence and search initiation.
2. Wide Coverage: Fixed-wing UAVs equipped with advanced drone technology can cover extensive search areas in minimal time, leveraging their endurance and speed advantages.
3. Cost Efficiency: Compared to manned aircraft operations, drone technology offers substantially lower operational costs while maintaining high search effectiveness.
4. Autonomous Operation: The combination of drone technology with reinforcement learning algorithms enables autonomous search path planning without requiring continuous human intervention.
5. Multi-Sensor Integration: Modern drone technology supports integration of various sensor payloads including electro-optical cameras, infrared sensors, and synthetic aperture radar, enhancing detection capabilities across diverse conditions.
6. Real-Time Adaptation: Drone technology facilitates real-time data processing and adaptive path planning, allowing the search strategy to evolve based on newly acquired information.
7. Safety Enhancement: Utilizing drone technology for initial search operations reduces risk to human personnel and enables operations in hazardous weather conditions.
The simulation results confirm that reinforcement learning enhanced drone technology can achieve substantially higher search efficiency compared to conventional methods, particularly in scenarios with significant target location uncertainty.
Algorithm Convergence Analysis
We conducted comprehensive convergence analysis to understand the learning dynamics of our PPO-based approach. The reward progression exhibits three distinct phases:
Phase 1 (Episodes 1-5,000): Rapid improvement phase where the agent learns basic search behaviors, including avoiding previously searched cells and moving toward high-probability regions. During this phase, the average episode reward increases from approximately -0.2 to 0.4.
Phase 2 (Episodes 5,001-12,000): Refinement phase where the agent optimizes its search strategy, learning to balance between exploring new regions and exploiting known high-probability areas. The reward continues to improve, reaching approximately 0.7 by the end of this phase.
Phase 3 (Episodes 12,001-20,000): Convergence phase where the policy stabilizes and marginal improvements become smaller. The reward asymptotically approaches the optimal value of approximately 0.8.
The loss function demonstrates corresponding behavior, decreasing from initial values around 0.6 to near-zero values at convergence, indicating successful policy optimization. The entropy term maintains a moderate level throughout training, ensuring sufficient exploration even in the later stages.
Comparative Analysis with Alternative Reinforcement Learning Approaches
While our primary comparison focuses on traditional search methods, we also recognize the landscape of alternative reinforcement learning algorithms for drone technology applications. Approaches such as Deep Q-Networks (DQN) and Rainbow have been applied to similar problems, but PPO offers distinct advantages for the maritime search scenario.
DQN methods, while effective for discrete action spaces with limited state complexity, often suffer from sample inefficiency and overestimation bias. The Rainbow algorithm, which combines multiple DQN improvements, introduces additional complexity that can make it difficult to tune for specific applications.
PPO addresses these limitations through its clipped objective function, which provides stable and reliable training across diverse problem domains. The algorithm’s ability to process continuous state spaces efficiently makes it particularly well-suited for the high-dimensional probability matrices encountered in maritime search planning.
Furthermore, PPO’s natural extensibility to multi-agent settings through MAPPO provides a clear pathway for scaling our approach to multi-UAV cooperative search missions. This scalability is crucial for real-world maritime search operations where multiple UAVs can collaborate to cover extensive search areas.
Practical Implementation Considerations
Our research identifies several practical considerations for implementing reinforcement learning-based drone technology in maritime search operations:
1. Computational Requirements: The training phase requires substantial computational resources, with our implementation utilizing GPU acceleration for efficient neural network training. However, the inference phase is computationally lightweight and can be executed on onboard UAV computers.
2. Real-Time Adaptation: The trained policy can be deployed for real-time path planning, with action selection requiring less than 10 ms of computation time per decision step. This enables responsive adaptation to changing environmental conditions.
3. Transfer Learning: Our experiments demonstrate that trained policies can be effectively transferred to new search scenarios with similar probability distribution characteristics, reducing the need for retraining in each mission.
4. Integration with Existing Systems: The proposed method can be integrated with existing maritime search planning systems, serving as an intelligent decision support tool for search coordinators.
5. Safety Constraints: The reinforcement learning framework can incorporate operational constraints such as no-fly zones, weather avoidance, and communication range limitations through appropriate reward shaping.
Limitations and Future Directions
While our research demonstrates significant improvements in maritime search efficiency, several limitations warrant further investigation. The current framework assumes a static probability distribution, whereas real maritime environments involve continuous target drift that requires dynamic probability updates. Extending the algorithm to handle time-varying probability distributions represents an important direction for future work.
The simplified UAV dynamics model, while adequate for demonstrating the core reinforcement learning approach, could be enhanced to incorporate more realistic flight characteristics including turning radius constraints, altitude-dependent sweep width variations, and fuel consumption considerations.
Multi-UAV coordination represents another promising research direction. The PPO framework can be naturally extended to multi-agent settings through MAPPO, enabling cooperative search strategies where multiple UAVs coordinate their movements to maximize collective search efficiency. This extension would address the practical requirement for large-scale maritime search operations.
Environmental uncertainty quantification could be improved by incorporating probabilistic weather forecasts and ocean current predictions directly into the state representation. This would enable the algorithm to adapt its search strategy based on predicted changes in target drift patterns.
Conclusion
In this research, we have developed and validated a reinforcement learning-based approach for UAV maritime search path planning. Our method leverages the Proximal Policy Optimization algorithm to train UAV agents that can autonomously plan efficient search trajectories in uncertain maritime environments.
The key contributions of our work include:
1. A comprehensive framework for formulating maritime search as a reinforcement learning problem, including state space design, action space definition, and reward function engineering that incorporates fundamental search theory principles.
2. Detailed hyperparameter optimization that identifies optimal configuration parameters for the maritime search scenario, providing practical guidance for implementing PPO in similar applications.
3. Quantitative validation demonstrating that our method achieves 21.4% higher cumulative probability compared to genetic algorithm-based approaches and 77.8% improvement over conventional parallel track search methods.
4. Robustness verification across multiple probability map configurations, with a coefficient of variation of 12.35% indicating stable and reliable performance.
5. Analysis of search path characteristics revealing that the reinforcement learning agent naturally develops an intelligent prioritization strategy, focusing on high-probability regions during the early search phase for maximum efficiency.
The advancement of drone technology has opened new possibilities for maritime search and rescue operations. Our research demonstrates how reinforcement learning can enhance drone technology capabilities, enabling autonomous intelligent search path planning that significantly outperforms traditional approaches. As drone technology continues to evolve, the integration of advanced artificial intelligence methods will further enhance the effectiveness of maritime search operations, ultimately contributing to improved safety and rescue outcomes in maritime environments.
The results of our study have direct practical implications for maritime search planners and UAV operators. The proposed method can be integrated into existing search planning systems to provide intelligent decision support, enabling more efficient allocation of search resources and improving the probability of successful target detection. Future work will focus on extending the framework to multi-UAV cooperative search scenarios and incorporating dynamic environmental factors for enhanced real-world applicability.
Acknowledgments and Data Availability
The drift prediction data utilized in this research was obtained from the national maritime search environment support platform. The simulation framework was implemented using PyTorch for neural network operations and custom environment modules for maritime search simulation. The complete code framework and trained models are available for research purposes upon reasonable request.
Our research provides a foundation for the continued development of intelligent drone technology for maritime applications. We believe that the combination of reinforcement learning and drone technology will play an increasingly important role in enhancing maritime safety and rescue capabilities in the years ahead.