Proceedings of the 31st International Workshop on Intelligent Computing in Engineering

Abstract

Drone-based visual inspection has emerged as a crucial manner for infrastructure inspection, owing to its mobility and potential for automated perception. However, from the perspective of human-agent interaction, the predominant modes in this field currently involve either full-process human intervention or end-to-end execution based on deep learning. The former constrains the level of automation, while the latter overlooks considerations of interactivity and controllability. To improve the level of automation and interactivity of the inspection process, this research explores the integration of Large Language Models (LLMs) into the visual inspection of infrastructure to utilize the capabilities of LLMs to understand human intentions and generate control commands. Specifically, high-level function libraries for drone and sensor control, as well as comprehensive and standardized prompts, are developed to fulfill the objective. The effectiveness of the method is demonstrated in both simulated and laboratory environments.