Recent Developments in computer software and related hardware technology have provided a value added services to the user. In everyday life, physical gesture is a powerful means of communication. The Project introduces an application using computer vision Hand Gesture Recognition. A camera records a live video stream, from which a snapshot is taken with the help of interface. The system is trained for each type of count hand gesture (using color pointer) at least once(R, G, B color pointer). After that a test gesture is given to it and the system tries to recognize it. A research was carried out on a number of Algorithms that could best differentiate a hand gesture (using color pointer). It was found that the diagonal sum algorithm gave the highest accuracy rate. In the Pre-processing phase, a self-development algorithm removes the background of each training gesture. After that the image is converted into a binary image and the sums of all diagonal elements of the picture are taken. This sum helps us in differentiating and classifying different colors of color pointer. Previous system has used data glove or markers for input in the system. I have no such Constraints for using the system. The user can give hand gesture in view of the camera naturally. A completely robust hand gesture recognition (using color pointer) system is still under heavy research and development the implemented system as an extendible foundation for future work.