A human-machine XR collaborative precise operation method aimed at complex spatial tasks in coal mines

Human-machine collaboration is one of the key technologies for robots to accurately complete complex operational tasks in coal mines. To address the challenges in human-machine collaboration for complex tasks, such as robots’ insufficient ability to perceive task and environmental information, difficulty in communication between humans and machines, and inadequate autonomous planning and control capabilities of robots, a method of human-machine extended reality (XR) collaborative precision operation is proposed. This includes a virtual-real space description method based on multi-agent theory of ‘human-machine-task’, a spatial task programming and measurement method based on ‘human-AR’ virtual-real collaboration, and a VR-based ‘machine-human’ virtual-real collaborative precise calculation and planning method. The virtual-real space description method based on ‘human-machine-task’ multi-agent theory utilizes constructed agents to map actual objects, acquiring the necessary target and environmental information for chassis path planning and robotic arm trajectory planning, thereby enhancing the robot’s perceptual understanding of task and environment information. The spatial task programming and measurement method based on ‘human-AR’ virtual-real collaboration leverages human subjective perception and decision-making abilities to program and tag objects, converting coordinates from a reference coordinate system to the robot coordinate system, which improves communication capabilities between humans and robots. The VR-based ‘machine-human’ virtual-real collaborative precise calculation and planning method autonomously plans the chassis path and robotic arm trajectory based on target and environmental information as well as programming tags, displaying the results in AR, which enhances the robot’s motion planning capabilities, while allowing humans to modify the planning results through AR, thereby improving the robot’s autonomous planning and control capabilities. A simulated coal mine operational environment experimental site was set up to conduct virtual-real spatial programming and measurement experiments, online precise calculation and planning experiments of robots, and human-machine collaboration experiments for the welding scenario of coal mining machine drums. The results indicate that the perception accuracy of the spatial task programming and measurement method based on ‘human-AR’ follows a normal distribution with a mean error of –0.21 mm, a standard deviation of 1.59 mm, and a maximum error of 4.1 mm; the VR-based ‘machine-human’ virtual-real collaborative precise calculation and planning method achieves autonomous robot planning and control, with operational accuracy also following a normal distribution, a mean error of 0.245 58 mm, a standard deviation of 1.678 06 mm, and a maximum error of 5.446 45 mm. The application of human-machine XR integration precision operation methods to execute human-machine collaboration tasks in complex coal mine environments allows robots to assist in completing perception of tasks and environmental information without being affected by harsh environmental conditions. The robots autonomously plan the chassis path and robotic arm trajectory based on the perception results and perform operational tasks, achieving operational accuracy that can reach the level of normal non-coal mine scenarios, thus exceeding the operational accuracy of human-machine collaboration in traditional coal mine settings.