Usage of self-excitation as an excitation method for a cantilever probe in atomic force microscopy (AFM) has been proposed to improve the low quality factor Q in liquid environments. To realize non-contact mode AFM, it is necessary to reduce the amplitude of the self-excited cantilever probe. For this study, the self-excited oscillation of the cantilever probe is generated by the angular velocity feedback. In addition, the small steady state amplitude is achieved using nonlinear feedback proportional to the squared deflection angle and the angular velocity. Regarding the microcantilever probe as a microcantilever beam, we present the equation of motion, which incorporates the geometrical nonlinear effect. The averaged equation is derived by applying the method of multiple scales and the bifurcation diagram is described theoretically. Results clarify that increasing the nonlinear feedback gain can reduce the cantilever-probe amplitude. Using an AFM that we produced, we demonstrate the nonlinear dynamics of a “van der Pol” type of self-excited cantilever. The steady state amplitude of the self-excited oscillation was 8 nm.