The evolution of Unmanned Aerial Vehicles (UAVs) has ushered in an era where coordinated multi-agent systems are not just advantageous but essential for complex missions. Drone formation flying represents a paradigm shift from single-vehicle operations, enabling a group of UAVs to function as a cohesive unit. This coordination dramatically enhances mission efficiency, robustness, and capability in applications ranging from wide-area surveillance and search-and-rescue to precision agriculture and atmospheric sampling. The core challenge in realizing effective drone formation flight lies in maintaining precise relative positioning among the member vehicles. Traditional methods relying on inter-vehicle communication or ground-based tracking are susceptible to latency, jamming, and infrastructure dependence.
Vision-based sensing emerges as a compelling alternative, offering a passive, rich-information, and cost-effective solution. By processing video streams, a UAV can directly observe its neighbors, extracting relative bearing and, with additional constraints, range information. This capability is fundamental for drone formation control algorithms, which require accurate state feedback to maintain desired geometric patterns. The problem thus translates to robust visual target tracking in dynamic aerial environments. This article, from the perspective of system implementation, details the development and integration of a feature-based visual tracking system on a Digital Signal Processor (DSP) platform, specifically designed to provide reliable displacement data for drone formation keeping.
Theoretical Foundation: Feature Tracking with the KLT Algorithm
Visual target tracking in the context of drone formation flight must account for rapid motion, changing perspectives, and potential variations in lighting. Among various tracking paradigms, feature-based methods offer a good balance between computational efficiency and robustness. The Kanade-Lucas-Tomasi (KLT) feature tracker is a seminal algorithm in this domain. Its principle is based on the invariance of appearance for a small image patch, or feature window, between consecutive video frames, assuming brightness constancy and small, coherent motion.
The core mathematical formulation seeks the displacement vector \(\mathbf{d} = [\delta_x, \delta_y]^T\) that minimizes the sum of squared differences (SSD) of pixel intensities within a feature window \(\mathcal{W}\). For a feature point centered at \(\mathbf{x} = [x, y]^T\) in image \(J\) (at time \(t\)), we find its corresponding location in image \(I\) (at time \(t+1\)) such that:
$$
\epsilon = \sum_{\mathbf{x} \in \mathcal{W}} [I(\mathbf{x} – \mathbf{d}) – J(\mathbf{x})]^2 w(\mathbf{x})
$$
Here, \(w(\mathbf{x})\) is a weighting function, often a Gaussian kernel emphasizing central pixels. For small \(\mathbf{d}\), we can approximate \(I(\mathbf{x} – \mathbf{d})\) using a first-order Taylor expansion around \(\mathbf{x}\):
$$
I(\mathbf{x} – \mathbf{d}) \approx I(\mathbf{x}) – \mathbf{g}^T \mathbf{d}
$$
where \(\mathbf{g} = [I_x, I_y]^T\) is the spatial gradient vector of image \(I\) at \(\mathbf{x}\). Substituting this approximation leads to:
$$
\epsilon \approx \sum_{\mathcal{W}} [I(\mathbf{x}) – J(\mathbf{x}) – \mathbf{g}^T \mathbf{d}]^2 w(\mathbf{x}) = \sum_{\mathcal{W}} [\mathbf{g}^T \mathbf{d} – \Delta I]^2 w(\mathbf{x})
$$
with \(\Delta I = I(\mathbf{x}) – J(\mathbf{x})\). Minimizing \(\epsilon\) with respect to \(\mathbf{d}\) by setting the derivative to zero yields the classic optical flow equation in a weighted least squares form:
$$
\mathbf{G} \mathbf{d} = \mathbf{e}
$$
where \(\mathbf{G} = \sum_{\mathcal{W}} w(\mathbf{x}) \mathbf{g} \mathbf{g}^T\) is a \(2 \times 2\) gradient covariance matrix, and \(\mathbf{e} = \sum_{\mathcal{W}} w(\mathbf{x}) \Delta I \mathbf{g}\). The displacement is solved as \(\mathbf{d} = \mathbf{G}^{-1} \mathbf{e}\), provided \(\mathbf{G}\) is invertible. This condition is crucial for feature point selection.
System Architecture for Onboard Processing
Implementing real-time visual tracking for drone formation flight necessitates an embedded system with significant computational power for image processing. A DSP-based architecture is ideal due to its optimized design for repetitive, mathematically intensive operations. The system is built around the TMS320DM642 DSP, a processor highly optimized for video and image processing applications.
The hardware ecosystem consists of several key components: the DM642 development board acts as the computational core; a pan-tilt unit (PTU) equipped with a camera serves as the sensing and actuation module; and an RS-485 interface facilitates communication between the DSP and the PTU for control commands. The camera captures analog video, which is then digitized by a video decoder on the board before being transferred to the DSP’s memory via Direct Memory Access (DMA). The processed output, such as tracking markers, can be sent to a video encoder for display. The software pipeline on the DSP handles the complete tracking workflow: image acquisition, preprocessing (e.g., grayscale conversion, noise reduction), feature point management, KLT tracking, outlier rejection, and finally, calculation of target displacement and bounding box updates.
Methodological Details: A Robust Tracking Pipeline
While the KLT algorithm provides the fundamental tracking mechanism, a practical system for drone formation applications requires several enhancing stages to ensure robustness against outliers, feature degradation, and target scale changes.
1. Feature Point Selection and Management
Not all pixels are suitable for tracking. Good features are located at corners or regions with high texture variation in both directions. This is mathematically ensured by examining the gradient covariance matrix \(\mathbf{G}\). A good feature point requires \(\mathbf{G}\) to be well-conditioned, meaning both of its eigenvalues \(\lambda_1\) and \(\lambda_2\) are large. We select points where the minimum eigenvalue exceeds a threshold \(\lambda_{min}\):
$$
\min(\lambda_1, \lambda_2) > \lambda_{min}
$$
Typically, \(\lambda_{min}\) is set to a small positive value (e.g., 1.0). A feature window size of \(7 \times 7\) or \(9 \times 9\) pixels offers a good trade-off between distinctiveness and computational load. For higher resolution images, a pyramidal implementation is used, where tracking starts at a coarse, down-sampled level and is refined at finer levels, enabling the capture of larger displacements.
Features are not permanent. Due to occlusion, motion blur, or rotation, tracked features can degrade or be lost. Therefore, the system continuously monitors the quality and quantity of tracked features. A feature is considered lost if the residual error \(\epsilon\) after solving for \(\mathbf{d}\) becomes too high. When the number of reliable features for a target falls below a threshold (e.g., 5), or after a fixed number of frames (e.g., N=30), a re-initialization step is triggered within an expanded search region around the last known target position to detect new, high-quality features.
2. Outlier Rejection with RANSAC and Clustering
Raw feature tracks inevitably contain outliers—incorrect matches due to noise, repetitive textures, or independent motion in the background. Feeding these outliers directly into the target state estimate would cause drift and failure. A two-stage filtering process is employed.
First, the Random Sample Consensus (RANSAC) algorithm is applied to the set of matched feature point pairs \(\{ (\mathbf{p}_i, \mathbf{q}_i) \}\) from consecutive frames. RANSAC randomly selects a minimal subset of matches (e.g., 3 pairs for an affine motion model) to hypothesize a geometric transformation \(\mathbf{H}\) (often a simple translational or affine model suffices for adjacent frames in drone formation tracking). It then computes how many other matches are consistent with this model (inliers). This process is repeated iteratively, and the model with the largest number of inliers is selected. Matches classified as outliers are discarded. The geometric consistency for a point pair is judged by a distance threshold \(\tau\):
$$
\|\mathbf{q}_i – \mathbf{H} \mathbf{p}_i\|^2 < \tau \quad \text{and} \quad \|\mathbf{p}_i – \mathbf{H}^{-1} \mathbf{q}_i\|^2 < \tau
$$
Second, a spatial clustering step is performed. Even after RANSAC, inlier features might belong to different objects (e.g., part of the target drone and part of the background). Assuming the target is a compact entity, its features should be spatially clustered. We compute the centroid \(\mathbf{c}\) of all remaining feature points. Features whose Euclidean distance from the centroid exceeds a dynamic threshold (e.g., a multiple of the current estimated target radius) are considered outliers and removed. The final target position for the frame is taken as the centroid of this filtered, clustered set of feature points, and the target bounding box is adjusted to encompass them.
3. From Pixel Displacement to Formation Control Input
The output of the visual tracker is a sequence of target image plane coordinates \((\hat{x}_t, \hat{y}_t)\) and an associated bounding box size \((w_t, h_t)\). The core measurement for immediate drone formation control is the pixel displacement from the image center or a reference point. Let the image center be \((c_x, c_y)\). The error signal for a pan-tilt controller would be:
$$
e_x = \hat{x}_t – c_x, \quad e_y = \hat{y}_t – c_y
$$
This error can be directly used to command the PTU, keeping the neighboring drone centered in the field of view—a basic visual servoing task. For true drone formation keeping, where relative 3D position must be maintained, additional information is needed. If the focal length \(f\) of the camera is known and the physical size \(S\) (e.g., wingspan) of the tracked drone is approximately known, an estimate of the relative range \(Z\) can be derived from the apparent size \(s\) in pixels (\(s = \max(w_t, h_t)\)):
$$
Z \approx \frac{f \cdot S}{s}
$$
The bearing angles \(\theta_x, \theta_y\) are related to the pixel offsets: \(\theta_x \approx \arctan(e_x / f)\). Combining rough range and bearing provides a 3D relative position estimate. However, this monocular cue is scale-ambiguous if the physical size \(S\) is not known precisely; thus, sensor fusion with other onboard sensors (e.g., inertial, airspeed) or state estimation filters (Kalman Filters) is ultimately required for robust drone formation navigation.
| Stage | Key Function | Mathematical/Algorithmic Tool | Purpose in Formation Tracking |
|---|---|---|---|
| Preprocessing | Image Simplification & Enhancement | Grayscale conversion, Gaussian filtering | Reduce data complexity, suppress noise |
| Feature Selection | Identify Trackable Points | Minimum eigenvalue analysis of \(\mathbf{G}\) matrix | Find reliable anchor points on target drone |
| KLT Tracking | Estimate Inter-frame Motion | SSD minimization, Lucas-Kanade equations | Compute displacement of each feature point |
| Outlier Rejection (RANSAC) | Remove Incorrect Matches | Random sampling, geometric model consensus | Filter out matches inconsistent with dominant drone motion |
| Spatial Clustering | Isolate Target Features | Distance thresholding from centroid | Separate target drone features from background features |
| State Estimation | Compute Target State | Centroid calculation, bounding box update | Produce single target position and size for control system |
| Feature Management | Maintain Feature Quality | Tracking error check, periodic re-initialization | Ensure a sufficient number of good features over time |
Experimental Validation and Results
The described system was implemented and tested in scenarios simulating drone formation conditions. The camera captured video at a resolution of 720×576 pixels. For processing efficiency, images were sub-sampled to 360×288. A feature window of 7×7 pixels was used, and a maximum of 20 strong feature points were initially selected on the target drone (a scale model used for testing). The system successfully tracked the target drone under conditions of translation and moderate rotation.
The integration of RANSAC and clustering proved critical. In raw KLT output, especially when the target moved against a cluttered background, several feature points would incorrectly latch onto the background. The RANSAC stage effectively identified and removed these outliers based on the motion model inconsistency. Subsequently, the clustering step removed any remaining inliers that were geometrically distant from the main cluster, ensuring that the estimated centroid corresponded solely to the target drone. This pipeline maintained a stable bounding box around the target even when the number of individual feature points fluctuated.
The primary output—the target’s centroid in pixel coordinates—was logged over time. The trajectory showed smooth and consistent following of the target’s motion. The system demonstrated a frame rate sufficient for real-time reaction, a key requirement for closed-loop drone formation control. The displacement data \((\delta_x, \delta_y)\) between frames, calculated from the feature point displacements, provides the essential velocity cue in the image plane, which can be integrated with other states in a navigation filter.
Conclusion and Future Perspectives
This work demonstrates the feasibility and effectiveness of a DSP-based, feature-driven visual tracking system for the problem of drone formation relative localization. By leveraging the computationally efficient KLT algorithm and fortifying it with robust outlier rejection mechanisms like RANSAC and spatial clustering, the system achieves reliable tracking of a cooperative target under realistic conditions. The output provides direct image-plane error signals for servo control and foundational displacement information that can be fused with other sensor data to estimate full relative state.
The path forward for vision-based drone formation flight involves addressing several challenges. First, scaling to multiple neighbors requires multi-target tracking and data association logic. Second, dealing with more aggressive maneuvers and occlusions will necessitate more sophisticated motion prediction and re-acquisition strategies, possibly using learned target appearance models. Third, tight integration with an Extended Kalman Filter (EKF) or a Nonlinear Model Predictive Controller (NMPC) is essential to translate image measurements into stable 3D formation control commands, compensating for the scale ambiguity inherent in monocular vision. Finally, exploring stereo or multi-camera setups on the DSP platform could provide direct 3D information, further enhancing the robustness and accuracy of the drone formation sensing suite. The integration of vision as a primary sensor promises to enable autonomous, resilient, and close-proximity drone formation flying in GPS-denied or communication-hostile environments.