Abstract: Layered rock mass has some obvious anisotropic mechanical properties and the rock system is of high non- linear and complexity. The cusp catastrophic model of the bending failure of layered rock mass (rock beam) based on the geological mechanical model and the potential energy function is proposed,that is used in the study on outside con- trol condition when catastrophe happens to system state,and mainly expatiate on how nonlinear system tends towards system character catastrophe from continuous gradual change state. The mechanism of releasing energy in catastrophe process was analyzed by using this theory. According to the function of stationary curved surface and branch curve,the necessary-sufficient mechanical conditions of system instability were derived. The method and the instability criterion were developed to analyze the stability of layered rock mass. The instability criterion of rock beam reveals that the sta- bility of layered rock mass is controlled by the external force (P,q),the modulus of the elasticity (E),the severe of rock mass (γc ),the interlayer shear stress (pt ) and the geometry (l,h). Combined with the mine examples,the result of stability analysis of roadway surrounding rock by applying the cusp catastrophe criterion of instability is in accord- ance with the actual situation. It shows that the criterion is scientific,effective and reliable. Meanwhile,the evaluation model provides a reference for the stability evaluation of surrounding rock and production.