The inspection and maintenance of power transmission lines using Unmanned Aerial Vehicles (UAVs) is a critical measure for ensuring grid safety and stability. Specialized operational skills training for this purpose is therefore of paramount importance. In practical field operations, drone inspection tasks are often executed by a team of two or more personnel working in coordination. Traditional training methods typically involve an expert or instructor first delivering theoretical lectures, followed by hands-on practical sessions. However, relying on verbal descriptions of precautions, responsibilities, and coordination points often leaves personnel with an insufficiently profound understanding of the collaborative process and its essential techniques. Conducting practical training on actual lines immediately after theoretical instruction can lead to poor operational coordination, posing significant safety risks. To address these challenges, we have developed an immersive simulation training platform by integrating Virtual Reality (VR), human-computer interaction, and networking technologies. This platform allows multiple trainees, assuming different operational roles, to participate in and experience how to collaboratively complete inspection and maintenance procedures for line equipment, thereby effectively resolving the issues prevalent in current drone inspection training.
Currently, simulation training is widely applied across various industries such as military and education for operational drills. Collaborative operation simulation has been utilized in fields including aerospace, assembly, and air combat simulation. Results indicate that this approach not only yields considerable economic benefits but also significantly enhances training standards. Within the domain of power line maintenance training, simulation was initially applied by using 3D visualization technology to recreate operational scenarios and standardize procedures, thereby improving training effectiveness through intuitive explanations of work methods, crew composition, tool requirements, steps, and safety measures. In recent years, with advancements in VR technology, devices such as VR headsets, controllers, and sensors have been gradually incorporated into line maintenance training. This integration enables interactive and immersive experiences of the maintenance process, substantially enhancing training outcomes and making skill training based on virtual scenarios a reality. However, existing systems are largely confined to single-user, single-process simulation and lack support for multi-user collaborative operations.
Collaborative simulation training is built upon VR technology. Within a computer-generated 3D virtual environment that encompasses both the work scene and the actions of personnel, different users interact in real-time via their respective terminals. They drive avatar models within the shared virtual environment to simulate the entire operational process. Implementing virtual collaborative operations necessitates solving two key problems: first, controlling concurrent operational behaviors from multiple users to prevent conflicts; second, ensuring consistency of the simulated scene across different client terminals. Focusing on these issues, this article elaborates on the methodology for constructing a multi-user collaborative operation model for power line maintenance. By designing the system composition, architecture, and network topology for multi-user collaborative operations, and utilizing foundational development platforms such as Unity 3D, 3DMax, VC++, and database management systems, we have realized a multi-user collaborative simulation system for line maintenance operations. This system also enables monitoring and management of the operational process, successfully addressing the problems of multi-user concurrent operation conflict control and virtual scene consistency.
Modeling Multi-User Collaborative Operations
Design of Collaborative Operation Workflow
In a multi-user collaborative simulation for line maintenance, each operator uses VR interaction devices to enter the same virtual work scene and perform simulated operational tasks. The process involves numerous concurrent operations that may lead to conflicts. Therefore, controlling multi-user concurrent behaviors to prevent conflicts is the primary challenge in realizing virtual collaborative training for line maintenance. Given the complexity of collaborative line maintenance processes and the variety of tools and equipment involved, we employ a token mechanism combined with a dynamic authority allocation method to orchestrate the operational workflow of a task. By proactively assigning operation tokens, the system guides the procedure at each stage, preventing conflicts during multi-user operations.
Based on the actual operational requirements of line maintenance tasks, each operation process can be categorized into either parallel operation sequences or multi-user synchronous operation sequences. Each trainee role is defined by four attributes: sequence number, position, authority level, and status. The position is assigned according to real operational requirements, and the status can be either “Waiting” or “Operating.” The operational content and responsible roles for each step are designed based on actual job specifications. At the beginning of each step, operational authority is dynamically assigned. A completion flag is set at the end of each step. Upon receiving this flag, the operator releases their authority, which is then reallocated before the next step begins. For instance, consider a maintenance task involving K personnel and consisting of M operational steps. After the simulation task starts, the system first assigns different authority levels (1 to K, where 1 is highest) to different roles based on actual requirements. The role with the highest authority (e.g., level 1) is responsible for executing the specific actions for that step, while others assist. If the highest-authority role encounters an anomaly (e.g., network disconnection), the system automatically reassigns the operation token to the waiting personnel with the next highest authority. This dynamic allocation prevents concurrency conflicts and ensures smooth process execution and system stability.
Parallel Operation Process
In line maintenance collaboration, there are scenarios where multiple personnel independently perform their respective tasks within the same timeframe. We define this as a parallel operation process. The workflow is illustrated in the figure below. The system first segregates the personnel into N independent serial operation groups. Within any single group, the system allocates authority to all participants based on real operational rules. The member with the highest priority operates in a single-user serial mode. If this member experiences an anomaly, the system actively selects a substitute from the waiting members to continue. All members release their operational authority upon completing a step and receive newly assigned authority upon entering the next phase. Due to strict operational standards in line maintenance, if one of two parallel groups has not finished its assigned task, it must wait for the other to complete before both can proceed to the next type of operational step.
Multi-User Synchronous Operation Process
Certain drone inspection tasks, such as UAV flight control or gimbal operation, require synchronous action from multiple operators to ensure correctness and effectiveness. The system must manage authority for such tasks. The workflow for multi-user synchronous operation is shown in the subsequent figure. First, two personnel are selected based on descending authority order. The system then issues a collaboration command to both. The operation commences only after both acknowledge. During execution, the system continuously monitors for anomalies like disconnections. If such an event occurs, the system promptly issues the operation command to the next personnel in the authority hierarchy, ensuring task completion. Operators release their authority after finishing their part of a step before proceeding to the next.
Simulation Task Modeling
Hierarchical Task Structure
To simulate the multi-user collaborative process of UAV power line inspection, it is essential to comprehensively and clearly describe all possible operational tasks and authentically replicate each specific action. Decomposing the simulation task hierarchically and modeling/encapsulating line maintenance operations allows for more flexible control of virtual humans and UAVs to execute designated actions, enabling accurate real-time control of the inspection simulation. Therefore, task decomposition is key to operational process simulation. This paper combines task decomposition with human activity simulation, designing an operation task decomposition model centered on human activities. The specific operational process for a personnel completing a task is depicted in the figure below.
Since simulating each specific action, based on human activity during the operation, is foundational to simulating the entire process, task decomposition should break down the complete process to the level of all constituent motions and action information, accurately expressing these details. Based on process-oriented and hierarchical design principles, we posit that complex operational tasks consist of several subtasks, each composed of multiple independent and easily describable basic actions. According to action type and task abstraction level, we structure line maintenance operations into three layers from top to bottom: the business-related Maintenance Task Layer, the smaller-goal-oriented Work Unit Layer, and the task-agnostic Basic Action Layer. The decomposition model is illustrated in the following figure.
| Layer | Description | Example |
|---|---|---|
| Maintenance Task Layer | The final objective, accomplished by a sequence of work units. | UAV takeoff, hover, photograph a specific insulator. |
| Work Unit Layer | Describes the operations a virtual operator must perform to complete a sub-task; a subdivision of the maintenance task process, formed by combining basic actions. | Walk to launch point, perform pre-flight check, pick up controller. |
| Basic Action Layer | Parameterized, generic actions independent of specific tasks, possessing universal semantics. | Walk, bend, turn, pick up/release tool. |
This three-layer hierarchy not only completely describes any operational task but also provides informational support for evaluating and analyzing the operational process.
Establishing the Action Library
Based on the hierarchical task decomposition, when an operator interacts with an object in the virtual scene, basic actions from the library are called and combined to form an operational behavior, completing a work unit. Multiple work units ultimately achieve the maintenance task objective. Therefore, the basic action library must be disjoint, complete, and reusable, encompassing a comprehensive set of repeatable basic actions describing all required real-world operations. To facilitate the modeling and simulation of various line maintenance operations, we defined a set of basic actions as standards for describing operational behaviors. These actions are invoked via specific commands, for example:
1. Worker Gesture(gesture1, gesture2): Adjusts the virtual human’s current posture to the desired hand operation (gesture1) and appropriate body stance (gesture2).
2. Target Location(location): Moves the virtual human to the specified work position.
3. Use Tool(name, gesture1, gesture2, equipment): The virtual human acquires a tool from the scene and operates on a specific equipment object.
4. Operation(equipment_area, action): Executes a specific inspection action on a designated equipment area.
5. Release(equipment, gesture1, gesture2): After completing the operation, the virtual human releases the equipment and restores hand and body to initial or specified states.
We employ a combination of keyframe animation and Inverse Kinematics (IK) computation to express the basic actions of the virtual human. Keyframe methods are used for relatively fixed motions. A basic animation library composed of keyframe animations is established, and operational behaviors are realized by calling these animations. For actions difficult to represent with fixed animations—such as grasping objects of different sizes, where hand aperture and posture vary—IK methods achieve realistic interaction. For instance, by obtaining the initial and target hand positions and orientations, the system calculates the required number of time steps to transition between them, simulating non-fixed actions.
3D Simulation Resource Modeling
Model Definition
To enable dynamic interaction and display between virtual operators and objects in the scene, some 3D objects within the line maintenance simulation environment are non-static. To facilitate animation control and improve interaction efficiency during simulation, we employ feature-based modeling to create predefined object descriptions. These define the interactive features and attributes of non-static 3D objects, describing not only geometric characteristics but also functional features and executable tasks. By pre-storing information related to object interaction and providing parameters for the basic actions in the library, virtual operators can quickly access relevant information and basic actions to perform specified interactive tasks. Integrating information such as interaction object part, interaction location, and equipment state changes enables simulation of any possible interactive behavior between virtual humans and objects. Based on the operational characteristics of line maintenance tasks, we categorize interaction features as follows:
| Category | Feature | Specific Content |
|---|---|---|
| Device Properties | Parts | Describes the geometric shape, hierarchy, position, and physical properties (mass, center of gravity) of each component of the device. |
| Motion | Defines the device’s movement and any other state changes (e.g., color, material). | |
| Interaction Information | Location | Facilitates interaction between the virtual human and the device; defines the device’s interactive part and the virtual human’s position. |
| Posture | Describes the action the virtual human should perform, primarily defining hand operations and body posture. | |
| State Change | Variables | Records the device’s state and its changes after each interactive operation. |
| Human Operation | Operation | Describes preset operations for each virtual role via a simple instruction set (e.g., positioning a role appropriately during equipment operation to avoid collisions). |
3D Modeling
To construct a realistic virtual operating environment for line maintenance, fundamental resource models are required, including 3D models of line equipment, defect models, human models, and the work environment. We use parametric modeling software 3DMax to create the 3D models involved in the virtual line maintenance process. The modeling workflow is shown in the figure below. Finally, the published model files are imported into the VR development platform Unity3D to build the line maintenance simulation scene.
System Design
System Composition
The functionality of the UAV Line Inspection Simulation Training System is divided into Trainee Clients and an Instructor Console. The trainee clients are primarily responsible for the multi-user collaborative operation of the line maintenance process. Multiple trainees, using different hardware devices, can participate in the same maintenance project, interacting via controllers and head-mounted displays for an immersive experience. The instructor monitors, records, and manages the simultaneous training sessions via the console. Currently, network-based multi-computer collaboration primarily employs either distributed or centralized architectures. Distributed collaboration involves network nodes receiving information, processing key data, and sending results to a server or other nodes for arbitration. This method offers reasonable load distribution but has complex control algorithms, suitable for large-scale distributed tasks. Centralized collaboration relies on a server to uniformly receive input from all nodes, perform calculations, and send results back. This approach simplifies conflict handling and control but places higher demands on the server, making it suitable for systems with fewer nodes. Collaborative operation training for line maintenance is typically organized within a local area network (LAN) by a training center, with 2-4 participants per session. Therefore, a centralized collaborative architecture aligns well with the practical needs of line maintenance simulation training, as illustrated in the figure below.
The system consists of one simulation server, multiple trainee client computers, one instructor computer, one stereoscopic projection monitoring system, and a network switch. Trainees wear VR input devices (e.g., controllers, position trackers) and feedback devices (e.g., stereo headsets) for collaborative simulation training. The client computers capture and process data from the VR input devices. Based on calculation results from the simulation server, they generate a first-person perspective 3D work scene for that trainee and drive force feedback devices to provide tactile operational feedback. The simulation server receives operational data from all trainees, executes simulation computations (including 3D virtual scene updates, collision detection, grasping simulation, human/object dynamics, and collaborative processing), and sends the results back to the clients and the instructor console. It also enables process recording, replay, and multi-view observation. The instructor can send control commands for training projects via the instructor console and observe/monitor each trainee’s operation process and collaborative performance through the stereoscopic projection system.
System Architecture
Addressing the functional requirements of the multi-user collaborative simulation training system for UAV power line inspection, we designed a system architecture based on data modeling, comprising Presentation, Logic, Data, and Support layers, as shown in the figure below.
| Layer | Components & Purpose |
|---|---|
| Presentation Layer | Provides human-machine interfaces via different hardware terminals (trainee VR gear, instructor monitors) for users to interact with and display data generated during collaborative operations. |
| Logic Layer (Core) | Processes business logic, system management, and configuration. Includes modules for simulation training, assessment, authority setting, training monitoring, and information management. The collaborative simulation module decomposes operations into the three-layer task model and drives the virtual scene based on the designed collaborative workflow. |
| Data Layer | Encapsulates resources involved in the simulation: 3D Model Library (scene, character, tool, action models), Equipment Information Database, Test Question Database, User Information Database, etc. |
| Support Layer | Provides foundational software development platforms (Unity 3D, 3DMax, VC++, DBMS) and network protocols (TCP/IP, etc.) for scene rendering, program control, networking, and data transmission. |
Network Structure
The line maintenance collaborative simulation system is a multi-computer, multi-display collaborative training environment. During collaborative operations, each operator interacts with the virtual scene via VR devices. For the collaborative process to proceed smoothly, the scene viewed locally by each operator must be consistent with that seen by others. The simulation system must ensure virtual scene consistency, including scene elements, object positions/states, and inter-object constraints. Frequent data exchange between scenes, roles, scripts, and communications with the data server is necessary, making data synchronization among clients crucial for collaborative task realization. To ensure real-time and reliable data synchronization, we designed the collaborative simulation network topology using data synchronization techniques, as illustrated below.
A public data server is established as a synchronization center for virtual scene element data. Each trainee client accesses the public information stored on this server in a Client/Server (C/S) mode. Before system runtime, each client downloads a copy of the virtual scene element data. By modifying this shared database, each client can update its local virtual scene. During operation, the system does not transmit the entire virtual scene data over the network. Instead, it only transmits data concerning object positions, orientations, and interrelationships. Each client receives this data and updates its local scene, virtual human, and feedback device information accordingly. The collaborative operation simulation flowchart is shown in the subsequent figure. By minimizing network bandwidth usage, real-time data interaction is ensured, achieving consistency of the virtual simulation scene across all trainee clients during training. The token allocation logic for conflict prevention in a step involving $K$ users can be formalized. Let $A_i(t)$ represent the authority level of user $i$ at time $t$, and $S(t)$ denote the system state. The user granted the token $T$ for a specific action is determined by:
$$ T(t) = \arg\max_{i \in \{1,…,K\}, \text{ status}(i)=\text{‘Ready’}} A_i(t) $$
subject to operational constraints $C(S(t))$. If user $T(t)$ fails, the token is reassigned:
$$ T'(t) = \arg\max_{i \in \{1,…,K\} \setminus \{T(t)\}, \text{ status}(i)=\text{‘Ready’}} A_i(t) $$
This mechanism ensures seamless continuation of the drone training procedure.
System Implementation
Visual Scene Simulation
Constructing a virtual line maintenance scene is foundational for multi-user collaborative drone training. We first built a 3D dynamic simulation platform for line maintenance scenarios using the Unity 3D VR development platform. Leveraging the 3D model library and this platform, dynamic 3D work scenes can be quickly and easily assembled through custom scripting and file import methods. An example of the platform interface is shown in the figure below.
Functional Realization
The system first uses the 3D dynamic simulation platform to recreate the actual line maintenance environment in 3D. Human-computer interaction is then achieved via VR input devices. Multiple trainees, using different clients, can enter the same work environment, assume different roles, and perform collaborative operations according to the system’s predefined workflow to complete each step. Taking the inspection of a 500 kV single-circuit tension tower as an example, one operator controls the remote controller (Pilot), while another operates the ground control station (Mission Commander). The control setup and display perspectives are identical to actual inspection operations, and training can also be conducted using accompanying VR headsets.
The Pilot is responsible for maneuvering the UAV to a designated position and maintaining a stable hover. The Mission Commander adjusts the gimbal to align the UAV camera with the inspection target and captures photos compliant with operational specifications. After collaboratively completing one task, both personnel move to the next inspection point. This sequence follows the multi-user synchronous operation process. Apart from the assigned tasks, trainees are not restricted from performing other exploratory operations within the simulation, enhancing the realism of the drone training.
By connecting all trainee clients, the instructor console, and the server via a LAN, a collaborative simulation training system is formed. Multiple trainees can form a team according to actual line maintenance requirements, assuming different roles, and conduct collaborative operation training for the same line inspection project within the shared virtual environment. Utilizing the virtual work environment, each trainee experiences the process from a first-person perspective. Following actual job requirements, they freely simulate operations through mutual coordination and can communicate naturally within the real physical space. This approach strengthens the training of multi-user collaborative operational capabilities and essential techniques by providing an authentic sense of job conditions and responsibilities for each role.
During the collaborative operation, the instructor can observe the entire process from a freely controllable third-person perspective via the instructor console, as shown in the final figure. This view can be displayed on a projection screen, facilitating observation and technical exchange among other personnel. This capability is vital for debriefing and assessment in effective drone training programs.
Conclusion
1. We designed workflows for parallel and multi-user synchronous operations in line maintenance using a token mechanism and dynamic authority allocation. By decomposing line maintenance tasks hierarchically and describing features of non-static 3D objects through feature-based modeling, we constructed a multi-user collaborative operation model. This model, supported by a line maintenance resource library and a basic action library, effectively resolves conflicts in multi-user operations and ensures system stability. The hierarchical task structure can be summarized by the relation:
$$ \text{Maintenance Task} = \bigcup_{j=1}^{N} \text{WorkUnit}_j, \quad \text{WorkUnit}_j = \bigcup_{k=1}^{M} \text{BasicAction}_k $$
where actions are drawn from a finite, predefined set.
2. A UAV line inspection simulation training system was constructed by interconnecting trainee clients, an instructor console, and a server via a LAN. Employing data synchronization techniques in the network topology design ensured real-time and reliable data synchronization among clients, achieving consistency of the virtual simulation scene across all operator stations during drone training sessions.
3. Utilizing foundational development platforms such as Unity 3D, 3DMax, VC++, and database software, we implemented the line maintenance collaborative simulation training system. Multiple trainees can participate in the same maintenance project via different clients for collaborative operations. Through head-mounted displays and interaction devices, trainees gain an immersive, interactive experience of the operational process. The instructor console enables monitoring, recording, and management of multi-user training sessions. Results demonstrate that the system operates stably and reliably. It provides an advanced and effective means for collaborative training in line maintenance operations and can be used for specialized skill drone training and pre-operational simulation drills, significantly enhancing the safety and efficacy of personnel preparation for real-world UAV inspection tasks.