Abstract: In the foundational research field of mine gas control, a large quantity of coal particles is widely employed in various experiments. Due to the significant impact of particle size distribution on experimental results, the precise and rapid determination of the particle size distribution of coal samples is of paramount research significance. Accurate particle instance segmentation is the key premise of image method to detect particle size distribution. Coal particles possess characteristics such as small volume, diverse shapes, numerous quantities, and a single color, which make them appear in images with small features, irregular edges, and adhesion between particles. These factors make it challenging for deep models to accurately capture and describe the edge shapes of particles. Moreover, the segmentation of dense small targets requires higher computational complexity, and to maintain the algorithm's engineering practicality, the lightweight of the network must be considered. All these factors contribute to the increased difficulty in the design of segmentation models. Additionally, since deep learning is driven by data, the lack of data sets is also a challenge that limits the use of deep learning algorithms. To address the challenges mentioned above, a lightweight coal particle group instance segmentation method based on YOLOv8 is proposed. Firstly, a fast image acquisition system based on flexible vibrating tray to separate particles is independently developed to quickly obtain coal particle pictures in batches. Then, an efficient particle image annotation method based on the SAM model is proposed, which pre-annotates the images and enhances data annotation efficiency. Consequently, a coal particle group dataset with abundant sample diversity is constructed, thus mitigating the scarcity issue in existing datasets for coal particle group. Secondly, based on the YOLOv8n-seg Backbone network, cross-stage local dynamic snake convolution is introduced in the backbone part to enhance the feature extraction capability, thereby improving the ability of the model to capture edge information of small irregular targets, and solving the problem that it is difficult to extract features from dense irregular small targets such as coal particles. Additionally, a high-performance feature fusion path based on adaptive weights is designed in the feature fusion network layer, enhancing the model's representational power. This addresses the challenge of identifying edges caused by irregular edges of coal particles and the adhesion between particles. Finally, by cropping the feature extraction layer responsible for large-size targets, the complexity and parameter count of the model are reduced, thus addressing the issue of computational complexity of segmentation models and their limited applicability. Experimental results show that compared with YOLOv8n-seg, the proposed DP-YOLOv8 improves achieved a 0.8% improvement in mAP@0.5:0.95, while the Params decreases from 3.264×10⁶ to 1.476×10⁶, and the FLOPs decreases from 12.1×10⁹ to 11.4×10⁹. The results show that the lightweight coal particle group instance segmentation method based on YOLOv8 achieves a better trade-off between model segmentation accuracy and lightweight.