Background modeling is the key technology for moving target detection implemented with the background difference method. The conventional background modeling methods cost large amount of calculation time, in order to improve the speed and precision of the model, this paper presents an improved Frame Difference Background Modeling. The frame difference background modeling was initialized by Selective Averaging method, on the basis of this, Moving objects is detected by background difference method. This kind of modeling method separated the moving targets and got the average of the background area, reduced the influence of moving target to background. The experimental results show that the method needs less prior video sequences and the background of the extracted is more accurately, the effect of the moving target detection is better.