Abstract: It is well known that the deep hole drilling in soft coal seams containing high pressure gas,hard and soft compound coal seams and outburst coal seams is a worldwide problem,which has become a bottleneck restricting the gas control effect in some high gas and outburst coal mines. In order to reveal the instability and failure mechanism of boreholes drilled in above-mentioned coal seams,based on the Hoek-Brown failure criterion,geological strength index(GSI),and the dynamic characteristics of porosity and permeability of coal containing gas,the plastic flow factor is introduced into the elastoplastic constitutive equations,so that the coal body strain is contained in the gas seepage field equation and the gas pressure is contained in the deformation field equation of coal and rock,and then the fluidsolid coupling dynamic model is established. The model is implemented by using COMSOL software. The distribution law of plastic zones around the boreholes with different coal structures under different side pressure coefficients (SPC)is discussed,and the instability and failure types of borehole in coal seam are classified. Research results indicate that the distribution ranges of the plastic zones around the boreholes decrease gradually with the increase of the GSI value under different SPC,and there is a typical power function relation between the radius of plastic zones and GSI values.However,the shapes of plastic zones vary according to different SPC and GSI values. When GSI value is smaller,the shapes are approximately shaped like fans,ellipses,and the letter "X" under different SPCs. Otherwise,the shapes are approximately shaped like circles,the letter "V",and pans. The failure types of borehole in coal seam can be divided into stress-controlled failure,structural plane-controlled failure,and coal and gas blowout in soft coal seams,which respectively occur in hard coal seams whose GSI values vary from 60 to 100,coal mass with moderate strength,whose GSI values vary from 30 to 60,and soft coal seams,whose GSI values vary from 0 to 30. The main controlling factors of above-mentioned failure are stress,mechanical properties of structural surface,mechanical properties of coal body and gas internal energy.