This paper provides a simple and efficient method to detect human faces in videos and recognize persons within the video according to a preset database of known persons. The proposed system consists of three main steps. The first step is skin-like regions detection in CIE-Luv color space. The second step is face detection based on skin-like regions, contour detection and geometrical properties such as face shape. The third step is face verification, in which, each face is compared with a gallery of known faces and the location of the best matched one is returned. Face verification step uses variance formula and skin to non-skin percentage in each facial feature to compare the test face and the faces images in the known database. The proposed system was tested on many different videos with different number of persons in the video. The detected faces are compared with a preset database of known images. Experimental results show that the proposed system is efficient enough to detect faces in different lighting conditions, head pose and face expressions. The results of verification step show that the proposed system is able to retrieve faces from a database with good accuracy in a reasonable computational time compared with classical method, variance estimation method and facial extraction method.