A coupling calculation method using dynamics and lubrication for a rotor and floating bush bearing system has been developed to predict self-excited oscillations and unbalance oscillations using a flexible multibody dynamics technique with hydrodynamic lubrication theory. Based on experimental results using a floating bush bearing, an accurate and numerically efficient lubrication model is proposed to calculate the hydrodynamic force, the bush rotating speed and the temperature of the oil film. In the multibody dynamics analysis, the rotor was modeled using the finite element method to include bending elasticity and the gyroscopic effect. The bearing bush was modeled by a rigid body. The coupling analysis between the rotordynamics and the oil film force of the bearing was performed by an in-house flexible multibody dynamics solver. Subsequently, the detailed mechanism of the rotor oscillation and the influence of some design parameters were investigated by numerical simulations and experiments.