In recent years, the demand for China UAV drones in low‑altitude logistics, medical supply transportation, environmental monitoring, and emergency rescue has surged dramatically. Autonomous navigation capability has become the core bottleneck limiting mission efficiency. However, in real‑world complex environments, path planning for China UAV drones faces multiple challenges: insufficient robustness of single‑sensor perception, real‑time decision delays caused by limited onboard computational resources, and energy efficiency bottlenecks inherent in conventional computing architectures. To address these issues, we propose a novel navigation strategy that integrates multi‑modal fusion perception with memristive reinforcement learning for China UAV drones. Our method constructs a multi‑source observation framework comprising LiDAR, vision sensors, inertial measurement units (IMU), and Global Positioning System (GPS), and designs a three‑level progressive fusion architecture operating at the data, feature, and decision levels. At the data level, we introduce an adaptive weighting strategy based on observation confidence that dynamically adjusts fusion weights according to real‑time noise levels of each sensor. At the feature level, a gated feature fusion mechanism is employed, where learnable parameters automatically regulate the fusion ratio between geometric and semantic features. At the decision level, confidence weights are reused to weigh the action‑value functions from different modalities, achieving end‑to‑end collaborative optimization from raw observations to action decisions. Experimental results in a 32×32 static grid environment show that the multi‑modal fusion scheme achieves an average IoU of 0.9782, a 7.6% improvement over single‑modality, while the IoU variance is reduced by over 90%, confirming that the multi‑source complementarity mechanism simultaneously enhances perception accuracy and robustness.
At the decision level, we embed an improved artificial potential field (APF) idea into the DQN reward function, constructing a hybrid reward mechanism that integrates safety, target approach, and energy consumption. To address the inherent defects of the conventional APF method, namely local minimum entrapment and trajectory oscillation, three improvements are proposed: introducing a smoothing function to continuously rectify the repulsive potential field and eliminate abrupt force changes at obstacle boundaries; setting dynamic virtual target points that generate temporary guidance signals when a local minimum trap is detected; and optimizing the repulsive force direction to reduce the reverse repulsive component that deviates from the target direction. Ablation experiments demonstrate that the improved APF hybrid reward achieves 100% planning success rate while reducing the average convergence episodes from 800 (with a sparse target‑only reward) to 200, and optimizing the average path length to 30.31 steps, outperforming the unimproved scheme in both training efficiency and policy quality.
At the computation level, to address the energy bottleneck arising from deep neural network inference, we introduce a memristor‑based processing‑in‑memory architecture to construct a memristive deep Q‑network (M‑DQN). The network linearly maps the fully connected layer weights to programmable conductances of memristors, realizes in‑situ multiply‑accumulate operations for both positive and negative weights via a differential pair structure, and introduces a threshold voltage mechanism to suppress unintended conductance drift caused by low‑voltage disturbances. In a grid environment containing four dynamic obstacles, the proposed algorithm achieves 100% success rate, reduces the average path length by 15.0% compared with the traditional DQN, and cuts total system energy consumption by 15.7%. Parameter sensitivity analysis further indicates that when the learning rate is set to 0.01, the algorithm attains optimal comprehensive performance in both static and dynamic environments. The above results demonstrate that the multi‑modal fusion and memristive reinforcement learning method achieves synergistic improvements across path safety, planning real‑time performance, and system energy efficiency, providing a practical and feasible autonomous navigation solution for resource‑constrained China UAV drones.
Problem Formulation
We consider the problem of autonomous flight for China UAV drones in a partially known complex dynamic environment. To balance the realism of a three‑dimensional environment and the computational efficiency of reinforcement learning, we adopt a hierarchical modeling and decision‑reduction strategy: the environment retains full 3D modeling results, while the path‑planning decision focuses on horizontal plane motions, incorporating height information as a state constraint and safety cost into the decision process, thereby reducing algorithm complexity while ensuring feasibility.
State Space
The state of the China UAV drone at time t is composed of the drone’s own state stu and the environmental state ste, i.e., st = (stu, ste). The own state describes the motion information of the drone in the 3D plane, including position, velocity, and attitude. Let the drone’s position be (xt, yt, zt), velocity be (vx,t, vy,t, vz,t), and attitude angles (roll, pitch, yaw) be (φt, θt, ψt). The own state is expressed as:
$$s_t^u = (x_t, y_t, z_t, v_{x,t}, v_{y,t}, v_{z,t}, \phi_t, \theta_t, \psi_t)$$
The environmental state consists of a local grid map centered on the drone, containing traversable areas and dynamic obstacle information. The state space is the combination of both, serving as the input to the reinforcement learning network.
Action Space
We use a discrete action space to describe the drone’s motion, defining six basic actions: forward, backward, left, right, ascend, and descend. Each action corresponds to a fixed displacement Δ. The action space is:
$$A = \{a_1, a_2, …, a_6\}$$
The figure below illustrates the 3D environment modeling and the action space, where arrows indicate the six basic movement directions in the discrete action space A.
Task Objective and Reward Function
The task objective is to find a trajectory τ = (s0, a0, s1, a1, …, sT) from the initial state s0 to the goal state sg, satisfying safety and dynamic constraints, while maximizing cumulative reward. We design a hybrid reward function that combines safety, target approach, and energy consumption:
- Safety reward: Positive reward for successful evasion, penalty for collision.
- Target approach reward: Encourages the drone to move toward the goal.
- Energy reward: Encourages shorter and smoother paths.
The task objective is formalized as maximizing the cumulative discounted reward:
$$\max_{\pi} \mathbb{E}_\pi \left[ \sum_{t=0}^{T} \gamma^t r(s_t, a_t) \right]$$
where π is the policy, r(st, at) is the immediate reward, and γ ∈ [0,1] is the discount factor.
Constraints
The flight must satisfy three types of constraints:
- Obstacle avoidance: The drone must not collide with any static or dynamic obstacles. Let the obstacle set be O; for all t, position (xt, yt, zt) ∉ O.
- Dynamic constraints: Velocity, acceleration, and angular velocity must remain within allowable limits.
- Boundary constraints: The drone must stay within the designated airspace.
Multi-Modal Fusion Method
In complex dynamic environments, the path planning performance of China UAV drones heavily depends on the accuracy and stability of environmental perception. However, a single modality may suffer from perception degradation or even failure under occlusion, strong noise, or scene degradation, causing oscillations in path planning strategies or even mission failure. Therefore, we employ multi-modal fusion technology to integrate the advantages of multiple sensors, providing a reliable environmental representation for subsequent path planning and decision algorithms.
We propose a three-level adaptive fusion framework that processes multi-source data at the data, feature, and decision levels, as illustrated in the overall architecture. The system consists of a multi-source perception layer, a multi-level fusion layer, a memristive reinforcement learning decision layer, and an optimization and execution layer. The multi-source perception layer collects raw data from LiDAR, vision, IMU, and GPS. The raw data undergo spatiotemporal calibration and adaptive confidence weighting at the data level, producing standardized fused observations. The feature level extracts geometric and semantic features from the standardized data and uses a gating mechanism to dynamically weight and fuse them, outputting a global environmental feature vector. The decision level maps the fused features into the agent state, inputs them into a multi-modal Q-network for policy inference, and finally outputs discrete action commands. The commands are refined by an improved artificial potential field (smoothing, virtual target points, repulsive force direction optimization) and subjected to obstacle avoidance and dynamic constraints, generating smooth control commands for the six‑degree‑of‑freedom motion.
Data-Level Fusion
Data-level fusion directly operates on the raw observations from each sensor, addressing issues of time synchronization, spatial registration, and inconsistent observation reliability. Let the raw observations from LiDAR, vision, and IMU/GPS at time t be Ztlidar, Ztvision, and ZtIMU/GPS, respectively. First, we perform spatiotemporal calibration using real-time pose information from IMU/GPS to ensure consistency in coordinate frame and time stamp. To handle occlusion, noise enhancement, or sensor performance degradation, we introduce an adaptive confidence-based weight assignment mechanism. For the i‑th sensor, the confidence weight ωt(i) is defined as:
$$\omega_t^{(i)} = \frac{1/\sigma_t^{2(i)}}{\sum_j 1/\sigma_t^{2(j)}}$$
where σt2(i) is the error variance estimated from the sensor noise model or historical observation residuals. The data-level fused observation is:
$$Z_t = \sum_i \omega_t^{(i)} Z_t^{(i)}$$
When vision is occluded, GPS drifts, or IMU accumulates errors, this adaptive mechanism dynamically increases the weight of higher‑reliability sensors (e.g., LiDAR), enhancing the stability and robustness of environmental perception. The fused observations are used for environment mapping and provide temporally and spatially consistent input for subsequent feature extraction and state construction.
Feature-Level Fusion
Feature-level fusion is an intermediate-level strategy that extracts discriminative intermediate features from different sensor raw data and then jointly models these multi‑modal features to obtain a more comprehensive and discriminative description. We extract two types of intermediate features: geometric features ftgeo from LiDAR point clouds, and semantic features ftsem from vision sensors. To avoid information redundancy or noise amplification from direct concatenation, we introduce a gated feature fusion mechanism that adaptively adjusts the contribution of each modality. Specifically, we first compute the gating coefficient for each modality through linear mapping and a nonlinear activation function:
$$\alpha_t = \sigma(W_g[f_t^{geo}, f_t^{sem}])$$
where Wg is a learnable parameter and σ(·) is the Sigmoid function. The final fused feature is:
$$f_t^{fusion} = \alpha_t \cdot f_t^{geo} + (1 – \alpha_t) \cdot f_t^{sem}$$
This mechanism allows the system to emphasize geometric features in structurally complex areas (e.g., narrow passages, dense obstacles) and enhance visual semantic features in scenarios requiring semantic discrimination, thereby constructing a more adaptive and discriminative environmental representation. The fused features are mapped into a unified environmental representation space to construct the state vector for path planning.
Decision-Level Fusion
Decision-level fusion converts multi-modal perception outputs into a standardized decision state, providing stable input for reinforcement learning. Based on the dual‑modality fusion of vision and LiDAR, we further incorporate GPS and IMU data at the decision layer, so that pose uncertainty is continuously compensated through the cross‑layer fusion mechanism. In this stage, based on the fused features and environmental modeling, we construct multi-modal state representations as:
$$s_t^{(i)}, \quad i \in \{lidar, vision, IMU/GPS\}$$
For each modality, we use a parameter‑shared memristive deep Q‑network (M‑DQN) to estimate the action‑value function Q(i)(st, a). To integrate the contributions of different modalities, we reuse the confidence weights ωt(i) obtained from the data‑level fusion stage to perform weighted fusion of the Q‑values:
$$Q(s_t, a) = \sum_i \omega_t^{(i)} Q^{(i)}(s_t, a)$$
The agent then selects the action based on the fused Q‑value:
$$a_t = \arg\max_a Q(s_t, a)$$
Thus, under the collaboration of multi‑modal information, the path planning decision is stable and robust.
Memristive Reinforcement Learning for Path Planning
Based on the multi‑modal fusion perception and 3D environment modeling, we construct an energy‑efficient memristive reinforcement learning path planning method for China UAV drones. The path planning problem is formulated as a Markov decision process (MDP), and a memristive crossbar array is used to implement a processing‑in‑memory deep Q‑network (M‑DQN), which significantly reduces computational energy consumption and decision latency while maintaining planning performance.
Path Planning Based on Reinforcement Learning
We model the autonomous obstacle avoidance path planning task of China UAV drones in a 3D environment with dynamic obstacles as an MDP ⟨S, A, P, R, γ⟩. The state space S consists of the local 3D grid map obtained from multi‑modal fusion and the drone’s own pose information. Each grid voxel contains attributes such as occupancy probability, semantic label, and dynamic obstacle flag. The action space A is designed as a set of discrete movement actions consistent with the grid resolution to avoid state‑action scale mismatch. The state transition function P is jointly determined by the drone’s dynamics and environment dynamics. The reward function R combines sparse event rewards and dense artificial potential field rewards to ensure both safety and goal orientation. The discount factor γ balances immediate and long‑term gains. Through multi‑modal fusion and confidence weighting, the system approximates a partially observable decision process at the reinforcement learning level, reducing the impact of sensor noise and environmental uncertainty on policy learning.
Path Planning Decision Based on DQN
In the path planning task, the DQN takes the fused multi‑modal perception state as input and outputs Q‑values for each candidate action, achieving direct mapping from complex environmental perception to flight action decisions. During typical training, the agent initializes an experience replay buffer, an online network, and a target network. It accumulates state transition samples through interaction with the environment, randomly samples mini‑batches from the replay buffer, and trains the network. By combining experience replay and target networks, DQN alleviates sample correlation and training oscillations, providing a stable algorithmic basis for the hardware implementation of the memristive reinforcement learning network.
The evolution of network input‑output relationships from Q‑Learning to DQN to M‑DQN is shown below. Traditional Q‑Learning takes low‑dimensional discrete states as input and stores state‑action values explicitly in a Q‑table. DQN extends the input to high‑dimensional continuous states and uses a deep neural network to approximate the Q‑function. M‑DQN takes multi‑modal fused features as input and maps the fully connected computations to a memristive crossbar array, achieving processing‑in‑memory.
Q-Learning
Q‑Learning is a classic off‑policy temporal difference (TD) reinforcement learning algorithm. The core idea is to iteratively solve the Bellman optimality equation to approximate the optimal action‑value function, thereby guiding the agent to choose the best action in each state. Unlike model‑based methods, Q‑learning does not require explicit modeling of state transition probabilities; instead, it learns through continuous interaction with the environment. When the agent takes a positive action, it receives a positive reward; when it takes a negative action (e.g., collision), it receives a penalty. The algorithm updates the Q‑value for each state‑action pair (S, A) and uses the Q‑values to guide action selection, continuously refining the function based on environment feedback. The update formula is:
$$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ R + \gamma \max_{a’} Q(s’, a’) – Q(s, a) \right]$$
where α is the learning rate, γ is the discount factor, and R is the immediate reward received after executing action a. Through iterative updates, the Q‑table gradually converges, and the policy converges to the optimal.
Deep Q-Network (DQN)
In real application scenarios such as UAV path planning, the state space is often high‑dimensional, continuous, and time‑varying. Traditional Q‑learning relying on a Q‑table becomes impractical in storage and computation. Therefore, DQN introduces a deep neural network to approximate the action‑value function, replacing the explicit Q‑table and significantly improving scalability. DQN approximates the true Q‑value function with a parameterized function Q(s, a; θ), where s is the state, a is the action, and θ are the network parameters. During training, DQN uses TD learning to minimize the error between the predicted Q‑value and the target Q‑value. The loss function is defined as:
$$L(\theta) = \mathbb{E}_{(s,a,r,s’)\sim D} \left[ \left( r + \gamma \max_{a’} Q(s’, a’; \theta^-) – Q(s, a; \theta) \right)^2 \right]$$
where r is the immediate reward, γ is the discount factor, θ− are the target network parameters, and D is the experience replay buffer.
Reward Function Design and Optimization
We embed the artificial potential field (APF) mechanism into the reward function construction to achieve joint incentives for safety and goal orientation. The APF method creates a virtual potential field in the environment so that the drone is attracted by the goal and repelled by obstacles, guiding safe arrival at the target. In the trajectory planning of China UAV drones, the attractive potential field simulates the gravitational force from the target point, typically expressed as:
$$U_{att}(q) = \frac{1}{2} \lambda_{att} \cdot d(q, q_g)^2$$
The corresponding positive reward is:
$$R_{att} = \eta_{att} \cdot d(q, q_g)$$
where q is the current position, qg is the goal position, d(q, qg) = ‖q – qg‖ is the Euclidean distance, and λatt, ηatt are attractive gain coefficients. The repulsive potential field is effective only near obstacles to prevent collision. It is expressed as:
$$U_{rep}(q) = \begin{cases}
\frac{1}{2} \lambda_{rep} \left( \frac{1}{d(q, q_{obs})} – \frac{1}{d_0} \right)^2, & d(q, q_{obs}) \leq d_0\\
0, & d(q, q_{obs}) > d_0
\end{cases}$$
The corresponding negative reward is:
$$R_{rep} = -\eta_{rep} \cdot \exp(-\gamma \cdot d(q, q_{obs}))$$
where d(q, qobs) is the distance to the nearest obstacle, d0 is the influence distance, and λrep, ηrep, γ are repulsive gain coefficients. Combining the attractive and repulsive rewards, the dense stepwise reward is:
$$R_{APF}(t) = R_{att}(t) + R_{rep}(t) = \eta_{att} \cdot d(q_t, q_g) – \eta_{rep} \cdot \exp(-\gamma \cdot d(q_t, q_{obs}))$$
The complete reward function combines the APF dense reward with sparse event rewards for key tasks:
$$R_{total}(t) = \begin{cases}
+R_{success}, & \text{reach goal}\\
-R_{collision}, & \text{collision}\\
R_{APF}(t), & \text{otherwise}
\end{cases}$$
However, traditional APF methods often suffer from local minimum problems and trajectory oscillations. We propose three improvements:
- Potential field smoothing: Introduce a smoothing function to correct the repulsive potential field at obstacle boundaries, preventing abrupt changes. The smoothed repulsive potential is:
$$U_{rep}^{smooth}(q) = \begin{cases}
\frac{1}{2} \lambda_{rep} \left( \frac{1}{\rho} – \frac{1}{\rho_0} \right)^2 \cdot \frac{1}{1 + e^{-k(\rho – \rho_s)}}, & \rho \leq \rho_0\\
0, & \rho > \rho_0
\end{cases}$$
where ρs is the smoothing threshold and k is the smoothing coefficient. This function achieves a smooth transition of repulsive force near the influence boundary, effectively suppressing oscillation near obstacles. - Virtual target point setting: When the drone is detected to be trapped in a local minimum region, a temporary virtual intermediate target point is dynamically generated to guide it out. The virtual target is set as:
$$q_{virtual} = q_{goal} + \lambda \cdot \frac{q_{goal} – q_{current}}{\|q_{goal} – q_{current}\|} + \Delta_{perturb}$$
where λ is the virtual target distance coefficient and Δperturb is a random perturbation vector. This temporary guidance helps the drone escape the potential well and then returns to tracking the original goal. - Repulsive force direction optimization: The traditional repulsive force direction is from the drone to the obstacle, which often causes frequent direction changes. We introduce a correction based on the target direction:
$$\vec{F}_{rep}^{opt} = -\eta_{rep} \left( \frac{1}{\rho} – \frac{1}{\rho_0} \right) \frac{\nabla \rho}{\rho^2} \cdot \cos \theta$$
where θ is the angle between the repulsive direction and the target direction. When θ > 90° (repulsive direction deviates from the target), the cosθ term reduces the reverse repulsive component, thereby reducing interference with the drone’s movement toward the target.
Memristive DQN (M‑DQN) Network Architecture Design
To reduce energy consumption and inference latency of the deep reinforcement learning path planning algorithm on China UAV drones, we introduce a memristor‑based processing‑in‑memory computing architecture to accelerate the high‑computational‑density layers of the decision network.
Memristor Model and Nonlinear Characteristics
A memristor is a nanoscale device with programmable conductance and non‑volatile storage. Its conductance state can be continuously adjusted under applied voltage and remains stable after power‑off. This property allows memristors to simultaneously perform weight storage and multiply‑accumulate operations at the physical level, providing a highly energy‑efficient processing‑in‑memory implementation for reinforcement learning networks.
We use an improved HP memristor model. The equivalent resistance is:
$$R(t) = R_{on} \cdot x(t) + R_{off} \cdot (1 – x(t))$$
where Ron and Roff are the minimum and maximum resistances, respectively, and x(t) = ω(t)/D is the normalized width of the doped region. The evolution of the doping boundary is described by:
$$\frac{dx(t)}{dt} = \frac{\mu_v R_{on}}{D^2} i(t) \cdot f(x)$$
where μv is the ion mobility, i(t) is the device current, and f(x) is a window function describing nonlinear boundary drift. We adopt the Joglekar window function:
$$f(x) = 1 – (2x – 1)^{2p}$$
where p controls the degree of nonlinearity. To prevent conductance drift at low voltages, we introduce a threshold voltage mechanism:
$$\frac{dx(t)}{dt} = \begin{cases}
\frac{\mu_v R_{on}}{D^2} i(t) \cdot f(x), & |v(t)| > v_{th}\\
0, & |v(t)| \leq v_{th}
\end{cases}$$
where vth is the threshold voltage. This model effectively suppresses unwanted conductance changes due to small voltage perturbations, improving weight storage reliability.
M‑DQN Agent Architecture and Function
The M‑DQN agent integrates perception, decision, and execution layers. The perception layer receives multi‑modal sensor observations, constructs a unified state representation after multi‑layer fusion and grid mapping, including drone position, obstacle distribution, and relative target information. The decision layer takes the state st as input and estimates the value function Q(s, a; θ) for the discrete action set via M‑DQN, then selects actions using an ε‑greedy policy to balance exploration and exploitation. The execution layer maps the chosen action into specific displacement or velocity commands, driving the drone to update its path. After action execution, the environment state changes and feeds back to the perception layer, forming a closed‑loop reinforcement learning interaction.
Weight Mapping and Forward Inference Based on Memristive Crossbar Array
The forward computation of the fully connected layers in DQN is dominated by large‑scale vector‑matrix multiply‑accumulate (MAC) operations, which are regular and data‑reusable, making them suitable for mapping onto a memristive crossbar array for processing‑in‑memory acceleration. The core idea is to map network weights to memristor conductances and perform MAC operations at the physical level. The process is as follows:
- Linear quantization mapping: Normalize the weights ωij of M‑DQN fully connected layers linearly to the programmable conductance range of memristors:
$$G_{ij} = \frac{\omega_{ij} – \omega_{min}}{\omega_{max} – \omega_{min}} \cdot (G_{max} – G_{min}) + G_{min}$$
where Gmax and Gmin are the bounds of programmable conductance. - Differential pair structure for positive/negative weights: Since memristors only support non‑negative conductance, we use a differential pair to represent positive and negative weights:
$$\omega_{ij} = k \cdot (G_{ij}^+ – G_{ij}^-)$$
where Gij+ and Gij– are at the same crosspoint location, corresponding to positive and negative weight components, and k is the conductance‑weight conversion coefficient. - Input feature voltage encoding: Map the normalized state feature vector xi to voltage signals vi on each word line of the crossbar array, achieving physical encoding of input information.
- In‑situ MAC operation and output decoding: Based on Ohm’s law and Kirchhoff’s current law, the array performs in‑situ MAC operations, and the output current on the bit line is:
$$I_j = \sum_i v_i \cdot (G_{ij}^+ – G_{ij}^-)$$
The output current is converted to a digital signal via a transimpedance amplifier and ADC, then fed into the activation module to output the Q‑values for each action, completing the decision inference.
The mapping mechanism is illustrated. The left side shows the fully connected layer in the digital domain: input state features xi are weighted summed and passed through an activation function to output Q‑values. The right side shows the physical memristive crossbar array where logical weights are mapped to positive and negative conductances via differential pairs, input features are encoded as word‑line voltages, and MAC results are obtained on the bit lines via Kirchhoff’s current law. Through this mapping, we achieve an equivalent transformation from “digital neural network computation” to “physical processing‑in‑memory computation.” During forward inference, weights are stored as conductances at crosspoints, and results are produced in parallel as currents, enabling efficient and low‑energy decision inference.
Algorithm Training Process
The multi‑modal perception and memristive deep Q‑network (MA‑DQN) algorithm for path planning is summarized below. The hardware structure mainly includes a memristive crossbar array, peripheral ADC/DAC interface modules, and a digital control unit.
- Initialization: Initialize the experience replay buffer D, the online network parameters w of M‑DQN, and the target network Q′ parameters w−. Set hyperparameters: exploration rate ε, discount factor γ, learning rate α. Initially, w− ← w.
- Environment construction and state representation: At the start of each episode, fuse multi‑modal sensor data to construct a new 3D grid environment env ⊂ ℝ3, with static obstacle set Os = {o1, o2, …, oNs}, where Ns is the number of static obstacles. The free space traversable by the robot is env \ Os = {c1, c2, …, cNf}, where Nf is the number of free cells. Input the current state st and action at to obtain the Q‑value vector Q(st, ai) for the current state.
- Action execution and selection: The drone selects action at using an ε‑greedy policy: with probability ε, choose a random action; with probability 1‑ε, select the action with the highest Q‑value: at = argmaxa Q(st, a; w). Execute at, receive immediate reward rt, observe the next state st+1, and store the transition (st, at, rt, st+1) in D.
- Reward calculation: The reward function combines APF dense reward and sparse task events:
$$r_t = w_1 R_{att} + w_2 R_{rep} + R_{sparse}$$
where Ratt is the attractive reward, Rrep is the repulsive reward, and Rsparse accounts for reaching the goal or collision. - Experience replay and error calculation: Randomly sample a batch of transitions (si, ai, ri, si+1) from D. Compute the target Q‑value:
$$y_i = \begin{cases}
r_i, & \text{terminal state}\\
r_i + \gamma \max_{a’} Q(s_{i+1}, a’; w^-), & \text{otherwise}
\end{cases}$$
Compute the TD error: δj = yj – Q(sj, aj; w). - Loss function calculation and network update: Use mean squared error loss to update the online network:
$$L(w) = \mathbb{E} \left[ \frac{1}{2} (y_j – Q(s_j, a_j; w))^2 \right]$$
Compute gradients via backpropagation and update w using gradient descent. Every N steps, copy the online network parameters to the target network: w− ← w. - Exploration rate decay and model saving: Gradually decay ε as training progresses, shifting from exploration to exploitation. Periodically save well‑trained model weights for subsequent validation and deployment.
The pseudocode is as follows:
Algorithm: M‑DQN Training with Experience Replay Input: Maximum episodes M, maximum steps per episode T, replay buffer capacity N Output: Trained memristive neural network parameters θ 1: Initialize experience replay buffer D with capacity N 2: Initialize memristive neural network weights θ and corresponding conductances 3: Set learning rate α, discount factor γ, exploration rate ε 4: for episode = 1 to M do: 5: Initialize environment state s 6: for t = 1 to T do: 7: Generate random number p ~ U(0,1) 8: if p < ε then: 9: Randomly select action a from action set A 10: else: 11: a = argmax_a Q(s, a) 12: end if 13: Execute action a, obtain reward r and next state s' 14: Store transition (s, a, r, s') in D 15: Randomly sample a batch of transitions (s_j, a_j, r_j, s'_j) from D 16: Compute target y_j according to TD formula 17: Compute loss L and gradients 18: Update network parameters: θ ← θ - α ∇θ 19: Update state: s ← s' 20: end for 21: end for
Experimental Design and Result Analysis
To systematically evaluate the proposed multi‑modal fusion and memristive reinforcement learning method for China UAV drones, we built an algorithm‑level simulation test environment based on Python 3.12.3 and PyTorch, running on a 2.20 GHz CPU with 16 GB RAM under Windows. The experimental environment uses a standard grid map containing static obstacles and dynamic obstacles moving along predefined trajectories. The drone starts at a fixed position (0, 31) and aims for the goal at (31, 0). The output consists of discrete action commands.
Perception Capability Comparison in Global Static Environment
To quantify the fundamental contribution of multi‑modal fusion perception to path planning performance, we first conducted perception capability comparison experiments in a global static environment. The multi‑modal sensor parameters are shown in Table 1, and the reinforcement learning parameters in Table 2. In this experiment, we uniformly used the traditional deep Q‑network as the decision maker, changing only the perception source of its input state, thereby avoiding interference from decision algorithm differences. Three comparative schemes were set:
- (1) Single‑modality perception: Only raw LiDAR point cloud data as environmental input, no fusion.
- (2) Dual‑modality fusion perception: Fuse LiDAR and vision sensor data at the feature level as state input.
- (3) Multi‑modal fusion perception: Apply our proposed framework, fusing LiDAR, vision, IMU, and GPS with multi‑level fusion.
| Sensor | LiDAR | Vision | IMU | GPS |
|---|---|---|---|---|
| Noise | 0.05 | 0.05 | 0.01 | 0.01 |
| Parameter | Value |
|---|---|
| Discount factor | 0.99 |
| Learning rate | 0.01 |
| Replay buffer size | 50000 |
| Target network update frequency | 400 |
| Initial exploration rate | 1.0 |
| Final exploration rate | 0.05 |
The perception accuracy comparison results are shown in Table 3. The multi‑modal fusion scheme achieved an average IoU of 0.9782, outperforming single‑modality by 7.6%, and the IoU variance was reduced by over 90%. This confirms that the multi‑source complementarity mechanism effectively compensates for the inherent deficiencies of individual sensors, providing more accurate and stable state inputs for obstacle avoidance decisions of China UAV drones.
| Metric | Single‑Modality | Dual‑Modality | Multi‑Modal |
|---|---|---|---|
| Average IoU | 0.9092 | 0.9782 | 0.9782 |
| IoU Variance | 0.003441 | 0.000355 | 0.000354 |
Ablation Study on Reward Function Configuration
The design of the reward function directly influences the behavior preference and training efficiency of the reinforcement learning agent. To verify the effectiveness of our improved APF hybrid reward, we designed four progressive reward configurations for experiments:
- Single goal reward: Only sparse positive reward when approaching the goal, serving as a lower bound baseline.
- Basic three‑term reward: Add obstacle avoidance penalty and step penalty on top of goal approach reward, forming a basic dense reward structure.
- Original APF hybrid reward: Add unimproved APF attractive and repulsive terms on top of the basic three‑term reward.
- Improved APF hybrid reward: Introduce potential field smoothing, virtual target points, and repulsive force direction optimization on the original APF.
All experimental groups trained in the same static grid environment with multi‑modal perception and traditional DQN. The results are shown in Table 4. The improved APF hybrid reward achieved 100% planning success rate, reduced the average convergence episodes from 800 (single goal reward) to 200, and optimized the average path length to 30.31 steps. It effectively avoided local minimum traps and trajectory oscillations, outperforming other configurations in training efficiency, policy stability, and path quality.
| Reward Type | Avg Success Rate | Avg Path Length | Local Minimum Occurrence | Avg Convergence Episodes |
|---|---|---|---|---|
| Single goal reward | 0% | 128.00 | 92.0% | 800.0 |
| Basic three‑term reward | 80% | 49.89 | 20.0% | 233.3 |
| Original APF | 100% | 31.18 | 0.0% | 283.3 |
| Improved APF | 100% | 30.31 | 0.0% | 200.0 |
Energy Efficiency and Real‑Time Performance Evaluation Based on Memristors
We further evaluated the impact of the memristor processing‑in‑memory architecture on the deployment performance of the path planning algorithm. Experiments were conducted on static and dynamic environments, comparing the traditional DQN (digital circuit), memristor‑based DQN (M‑DQN), and ICVTLBO (a baseline algorithm from literature).
Table 5 lists the key memristor parameters used in the experiments. The low‑resistance state Ron and high‑resistance state Roff map to the lower and upper bounds of neural network weights. The window function parameter p is set to 5 to simulate nonlinear conductance modulation, and the programming threshold voltage is set to 0.5 V to suppress unintended conductance drift during read operations.
| Parameter | Value |
|---|---|
| Minimum resistance Ron / Ω | 104 |
| Maximum resistance Roff / Ω | 106 |
| Memristor initial value | 0.1 |
| Linear drift coefficient / (m2s-1V-1) | 10-14 |
| Memristor length / nm | 10 |
| Voltage threshold / V | 0.5 |
| Window function coefficient p | 5 |
| Frequency / Hz | 6 |
We selected four core evaluation metrics:
- Success rate (SR): SR = Nsuccess / Ntotal.
- Path length (PL): Average number of steps over successful episodes.
- Real‑time performance: Average single‑step forward inference time.
- Energy consumption: Total energy consumed per episode, composed of motion execution energy, multi‑modal sensing energy, and neural network inference energy (proportional to inference time).
Static Environment Results
Table 6 shows the comparison results in a static environment. Traditional DQN and ICVTLBO both achieved 100% success rate. Due to non‑ideal memristor characteristics causing slight policy drift, M‑DQN achieved a success rate of 98.3%, still above 98% with a reasonable path length. In terms of real‑time performance and energy efficiency, M‑DQN exhibited significant advantages with an inference time of 0.0056 ms and per‑step energy consumption of 5.516 μJ, representing a 91.1% reduction in per‑step energy and a 3.7× improvement in real‑time performance compared to traditional DQN.
| Algorithm | Success Rate | Path Length | Per‑step Energy / μJ | Real‑time / ms |
|---|---|---|---|---|
| Traditional DQN | 100% | 62.00 | 62 | 0.0205 |
| M‑DQN | 98.3% | 68.96 | 5.516 | 0.0056 |
| ICVTLBO | 100% | 62.00 | 8664 | 246.3571 |
Dynamic Environment Results
To evaluate the robustness and real‑time performance of the algorithm in scenarios closer to real applications, we introduced four dynamic obstacles (m1, m2, m3, m4) moving along preset trajectories in the original static map. The dynamic obstacle parameters are given in Table 7. We compared traditional DQN, M‑DQN, and the DDQN‑3D algorithm from literature.
| Obstacle | Initial Position | Movement Direction | Speed |
|---|---|---|---|
| m1 | (3, 0) | (0, 1) | 1 unit/step |
| m2 | (31, 28) | (-1, 0) | 1 unit/step |
| m3 | (15, 31) | (0, -1) | 1 unit/step |
| m4 | (31, 4) | (-1, 0) | 1 unit/step |
The results are shown in Table 8. M‑DQN achieved 100% success rate, a 15.0% reduction in average path length (63.1 vs. 74.2), and a 15.7% reduction in total energy consumption (76.0 μJ vs. 90.1 μJ) compared to traditional DQN. DDQN‑3D had a higher success rate than traditional DQN but incurred extra path and energy overhead. M‑DQN’s real‑time performance (0.09 ms) is slightly inferior to traditional DQN (0.04 ms) due to memristor programming and readout circuit delays, but still within milliseconds. Overall, M‑DQN achieves the best combination of success rate, path quality, and energy efficiency in dynamic environments.
| Algorithm | Success Rate | Path Length | Energy / μJ | Real‑time / ms |
|---|---|---|---|---|
| Traditional DQN | 94.7% | 74.2 | 90.1 | 0.04 |
| M‑DQN | 100% | 63.1 | 76.0 | 0.09 |
| DDQN‑3D | 98.7% | 75.9 | 92.1 | 0.19 |
Key Parameter Sensitivity Analysis
The performance of reinforcement learning algorithms is sensitive to hyperparameter settings. To clarify the stability boundaries of our method and provide a reference for practical deployment, we selected three typical values of learning rate α (0.001, 0.01, 0.1) and conducted experiments in both static and dynamic environments using the multi‑modal perception M‑DQN. The simulation results are shown in Table 9.
| Learning Rate | 0.001 | 0.01 | 0.1 |
|---|---|---|---|
| Static Environment | |||
| Success Rate | 88.0% | 100.0% | 0.0% |
| Average Energy / μJ | 9.346 | 5.165 | 20.480 |
| Convergence Episodes | 716.7 | 666.7 | 1000.0 |
| Dynamic Environment | |||
| Success Rate | 100.0% | 100.0% | 0.0% |
| Average Energy / μJ | 5.326 | 2.59 | 3.29 |
| Convergence Episodes | 516.8 | 733.3 | 1000.0 |
The results show that the learning rate significantly affects model performance. When α = 0.01, the algorithm achieves 100% success rate and the lowest energy consumption in both static and dynamic environments, representing the optimal comprehensive performance. When α = 0.001, the static environment success rate drops to 88% due to the small learning rate causing the model to fall into local optima. Interestingly, in the dynamic environment, it still attains 100% success with faster convergence, because the random perturbations introduced by the dynamic environment help the agent escape local minima. When α = 0.1, the algorithm fails entirely in both environments; in static environments, the energy surge to 20.480 μJ indicates gradient explosion or step‑size overshoot, causing severe oscillation around the optimal solution. Therefore, a moderate learning rate balances update step size and error accumulation. In static environments, one may slightly increase the learning rate to speed up convergence, while in dynamic scenarios, stability should be prioritized. These findings provide a guideline for parameter tuning in practical deployments of China UAV drones.
Conclusion
Addressing the core challenges of insufficient environmental perception robustness, large real‑time decision latency, and limited onboard energy in complex environments for China UAV drones, we propose a multi‑modal fusion perception and memristive reinforcement learning path planning method (MF‑MRL). By constructing a three‑level adaptive fusion framework at data, feature, and decision levels, we integrate LiDAR, vision, IMU, and GPS multi‑source information, effectively compensating for the blind spots of individual sensors and improving modeling accuracy and robustness in complex dynamic environments. Experiments show that multi‑modal fusion perception achieves an average IoU of 0.9782, a 7.6% improvement over single‑modality, with a variance reduction of over 90%, providing stable state inputs for subsequent planning decisions.
At the decision level, we embed an improved APF idea into the reward function. Through potential field smoothing, virtual target point setting, and repulsive force direction optimization, we effectively solve the local minimum and trajectory oscillation problems. Ablation experiments demonstrate that the improved APF hybrid reward achieves 100% planning success rate, reduces average convergence episodes to 200, and optimizes path length to 30.31 steps, significantly improving training efficiency and policy stability.
To break the energy bottleneck of traditional computing architectures, we introduce a memristor‑based processing‑in‑memory architecture to implement the deep Q‑network (M‑DQN). The network weights are mapped to memristor conductances, and in‑situ multiply‑accumulate operations are performed at the physical level. In a static environment, M‑DQN reduces per‑step energy consumption by about 91.1% and improves real‑time performance by 3.7× compared to traditional DQN. In a grid environment with four dynamic obstacles, M‑DQN achieves 100% success rate, reduces average path length by 15.0%, and decreases total energy consumption by 15.7%. Parameter sensitivity analysis indicates that a learning rate of 0.01 yields optimal comprehensive performance in both static and dynamic environments.
In summary, the proposed multi‑modal fusion and memristive reinforcement learning method achieves synergistic improvements in path safety, planning real‑time performance, and system energy efficiency, providing a practical autonomous navigation solution for resource‑constrained China UAV drones. Future work will focus on three aspects: (1) integrating continuous dynamics models and using hardware‑in‑the‑loop simulation to further narrow the gap between simulation and real flight; (2) exploring end‑to‑end joint optimization of multi‑modal perception and memristive networks to fully release the synergy of perception and decision; and (3) extending the method to multi‑robot collaborative tasks and extreme weather scenarios to verify its generalization and robustness, pushing the development of China UAV drones toward more intelligent and energy‑efficient autonomous navigation.