Abstract: To investigate the material instability to the rock burst phenomenon,a series of uniaxial compression tests on outburst-prone coal specimens under end-strain-restriction were conducted along with a real-time video recording. Firstly,the failure characteristics were analyzed in the laboratory,and the obtained results were compared with the nu-merical simulations of the coal’s rupture potential. A rock burst criterion was developed based on the analysis of the samples’ rupture surface as well as the double linear constitutive relationship. The experimental results showed that the rupture mechanism of coal specimens during the experimental tests displayed a stage-by-stage development provi-ding some predictive information such as internal sounds. When the stress inside the coal reached 60% of its(uniaxial compression) strength,a small amount of coal started to be ejected. At 78% of its total strength,a significant portion of the coal specimen blew up and when the internal stress reached a value close to the total strength, the coal were pushed out gradually,without ejection. The rupture mechanism of coal in these end-strain-restriction experiments could be defined as tensile-shear mixing modus,displaying a scooped-triangle shaped surface. These characteristics formed gradually and predictively throughout the process,implying that the experimental results have a satisfying level of sig-nificance. It has also been found that the decrement in the height-to-diameter ratio could enhance the coupling stress and reduce the coal fracture surface. Using the finite element method,it can be observed that the rock around the side-wall is prone to form the scooped-triangle local deformation bands that evolve into a rupture surface during rock bursts. Based on the morphology of the rupture surface and the theory of strain-gradient plasticity,an instability criterion was established on the point of static balance and energy balance,using a constitutive model of strain softening.