Abstract: To address the problems of low detection accuracy, missed detection, and false detection in coal gangue detection systems for underground coal mines caused by low illumination, high noise, motion blur, and other harsh environmental factors, a real-time coal gangue image detection model, BRS-YOLOv10n, is proposed. The overall architecture of the BRS-YOLOv10n object detection framework is divided into four core functional modules: the input preprocessing module, feature extraction backbone network, multi-scale feature fusion network, and detection prediction head. First, to reduce the computational burden of the Transformer, the BiFormer vision Transformer is used to replace the first two C2f modules in the backbone network. Second, a RepLKNet large-kernel convolution is introduced into the last C2f module of the backbone network. The receptive field is expanded through re-parameterized large-kernel convolution, allowing shape information to be more fully utilized and improving the accuracy and stability of feature extraction. Then, the Selective Kernel Networks (SKNets) dynamic attention mechanism is added after each C2f module in the backbone network. By adaptively adjusting the receptive field size, the ability of the model to capture both global and local image information is enhanced. Finally, the Inner-CIoU loss function is introduced to replace the original CIoU loss function. Auxiliary bounding boxes are introduced to accelerate bounding box regression and effectively promote model convergence. Ablation experiments and comparative experiments are designed to verify the comprehensive detection performance of BRS-YOLOv10n. The experimental results show that the BRS-YOLOv10n model exhibits significant advantages under simulated underground coal mine conditions involving low illumination, high noise, and motion blur. The mean average precision reaches 92.3%, which is 3.5% higher than that of the original YOLOv10n, and the average detection speed reaches 62.6 frames/s, meeting the requirements of real-time coal gangue detection. Compared with YOLOv5n, YOLOv7-tiny, YOLOv8n, YOLOv8s, YOLOv9-tiny, YOLOv10n, Faster R-CNN, SSD, CenterNet, and DETR, the BRS-YOLOv10n model achieves significant improvements in detection accuracy and mean average precision for coal gangue detection. The proposed method enables accurate real-time detection and localization of coal gangue in harsh environments and provides technical support for research on intelligent coal gangue sorting.