With high accuracy it is difficult to classify rigid and non-rigid objects. In this paper, a few set of small videos has been taken and different features are extracted from them based on their appearance. Here, rigid and non-rigid objects are restricted to vehicles and human beings. Primarily, moving objects are detected by using background subtraction, which is widely used to identify foreground objects in video surveillance. LPGMM model is considered which can work even on dynamic background. Feature extraction based on appearance are carried out using various methodology which are mainly focused on blob analysis viz., SURF (Speeded-Up Robust Features), edge detection and HOG (Histogram of Oriented Gradients). SVM (Support Vector Machine) supervised learning model is used for the binary classification of rigid and non-rigid objects.