ĐSTANBUL TECHNICAL UNIVERSITY INSTITUTE OF SCIENCE AND TECHNOLOGY
M.Sc. Thesis by Kıvılcım HELHEL
Department : Computer Engineering Programme : Computer Engineering
OCTOBER 2009
ĐSTANBUL TECHNICAL UNIVERSITY INSTITUTE OF SCIENCE AND TECHNOLOGY
M.Sc. Thesis by Kıvılcım HELHEL (504061519)
Date of submission : 07 September 2009 Date of defence examination: 06 October 2009
Supervisor (Chairman) : Prof. Dr.Muhittin GÖKMEN(ITU) Members of the Examining Committee : Prof. Dr. A.Coşkun SÖNMEZ (YTU)
Assis. Prof. Dr. M. Ersel KAMAŞAK (ITU)
OCTOBER 2009
EKĐM 2009
ĐSTANBUL TEKNĐK ÜNĐVERSĐTESĐ FEN BĐLĐMLERĐ ENSTĐTÜSÜ
YÜKSEK LĐSANS TEZĐ Kıvılcım HELHEL
(504061519)
Tezin Enstitüye Verildiği Tarih : 07 Eylül 2009 Tezin Savunulduğu Tarih : 06 Ekim 2009
Tez Danışmanı : Prof. Dr. Muhittin GÖKMEN (ĐTÜ) Diğer Jüri Üyeleri : Prof. Dr. A.Coşkun SÖNMEZ (YTÜ)
Yrd. Doç. Dr. M. Ersel KAMAŞAK(ĐTÜ) HARMONĐK GÖRÜNTÜLERĐ KULLANARAK YÜZ TANIMA
FOREWORD
I would like to thank to my supervisor, Prof. Muhittin Gökmen for his guidance throughout my research process and during the preperation of this thesis.
Special thanks to my family, for their patience and continuous support during my master study and during the preparation of this work.
I would also like to thank to TÜBĐTAK (The Scientific and Technological Research Council of Turkey) for supporting me during my master study under the Grant “National Scholarship Programme for Master Science Students”.
And finally, I would like to thank Bi rk an Tu n ç for valuable discussions and for sharing his k no wl ed ge with me.
October 2009 Kıvılcım HELHEL
TABLE OF CONTENTS
Page
ABBREVIATIONS ... ix
LIST OF TABLES ... xi
LIST OF FIGURES ...xiii
SUMMARY ... xv
ÖZET ... xvii
1. INTRODUCTION ... 1
1.1 Face Recognition Systems ... 3
1.2 Face Recognition in Subspaces ... 4
1.3 Challenges in Face Recognition ... 5
1.3.1 Illumination variation ... 5
1.3.2 Pose variation ... 7
1.3.3 The occlusion ... 8
1.3.4 Time delay ... 9
1.3.5 Face expression variation ... 10
1.4 Illumination ... 11
1.5 Harmonic Images ... 12
1.6 Face Recognition From a Single Intensity Image ... 15
1.6.1 Holistic methods ... 15
1.6.2 Local methods ... 17
1.6.3 Hybrid methods ... 19
1.7 Organization of Thesis ... 20
2. LAMBERTIAN REFLACTANCE AND MODELING ILLUMINATION VARIATION WITH SPHERICAL HARMONICS ... 21
2.1 Reflectance and Ligting ... 22
2.2 Spherical Harmonic Representation and Properties of Convolution ... 23
2.3 Approximating the Reflectance Function ... 21
2.4 Generating Harmonic Reflectances ... 21
2.5 Representing Images in Terms of Reflectances... 28
2.6 Recognition ... 30
3. SPHERICAL HARMONICS BASED FACE RECOGNITION USING ONE TRAINIG IMAGE PER PERSON ... 31
3.1 Introduction ... 31
3.2 Getting Harmonic Images ... 32
3.3 Recovery of Basis Images from 2D Statistics for Fixed Pose ... 33
3.3.1 Statistical models of basis images ... 33
3.3.2 Recovering the basis images ... 34
3.3.2.1 Estimating harmonic coefficients ... 34
3.3.2.2 Estimating the error term ... 35
3.3.2.3 Computing the basis image ... 35
viii
4. FACE RECOGNITION USING HARMONIC IMAGES ... 39
4.1 Preprocessing ... 39
4.1.1 Surface reconstruction ... 40
4.1.2 Getting surface normals ... 40
4.1.3 Getting albedos ... 42
4.2 Statistical Models of Harmonic Images ... 43
4.3 Recovery of Basis Images from 2D Statistics for Fixed Pose ... 44
4.4 Recognition ... 47
5. EXPERIMANTS AND RESULTS ... 49
5.1 Basis Image Recovery ... 51
5.2 Recognition ... 52
6. CONCLUSION AND FEATURE WORK ... 53
REFERENCES ... 55
ABBREVIATIONS
PCA : Principal Component Analysis LCA : Linear Component Analysis 2DPCA : Two dimensional PCA LFA : Local Feature Analysis LDA : Linear Discriminant Analysis EBGM : Elastic Bunch Graph Matching HMM : Hidden Markov Model
SVM : Support Vector Machine
PPLS : Parametric Piecewise Linear Subspace LPCMAP : Linear PCMAP
KCCA : Kernel Canonical Correlation analysis GDA : Generalized Discriminant Analysis LBP : Local Binary Patterns
SHBMM : Spherical Harmonic Based Morphable Model ICA : Independent Component Analysis
LIST OF TABLES
Page Table 1.1: Face recognition application categories and examples...2 Table 1.2: Comparison of sensitiveness of local and global features to variations ... 20 Table 2.1: Energy captured by then’th harmonic for the Lambertian kernel... 26 Table 5.1: The seperation of test images according to YaleB database ... 50 Table 5.2: Recognition results ... 52
LIST OF FIGURES
Page Figure 1.1 : Block diagram of a generic face recognition system. ... 4 Figure 1.2 : Same face under different lighting conditions. ... 6 Figure 1.3 : Normalization examples. ... 7 Figure 2.1 : Representatin of the first 11 coefficients, the relative energy captured by each coefficients and accumulated energy, respectively ... 27 Figure 4.1 : The intensity image and the aligned and crooped face image. ... 40 Figure 4.2 : Image filters for obtaining surface normals. ... 41 Figure 4.3 : The intensity image of one subject and the reconstructed surface of this
subject ... 41 Figure 4.4 : The intensity images and albedos of three subject. ... 42 Figure 4.5 : The rendered basis images of one subject. ... 43 Figure 4.6 : The mean images of the statistical models computed from the rendered
basis images ... 44 Figure 4.7 : Algorithm of estimaitng the statistical properties of error term and
harmonic coefficients ... 46 Figure 4.8 : Pseudo-code recovering the basis image from a given new image ... 47 Figure 5.1 : One bootstrap image that is illuminated from 64 different illumination
directions ... 50 Figure 5.2 : The recovered basis images compared to the rendered ones for an
individual subject in the bootstrap set ... 51 Figure 5.3 : The basis images recovered from images of same subject under various
single directional illumination ... 51 Figure 5.4 : The recovered basis images using equation (3.5) ... 52
FACE RECOGNITION USING HARMONIC IMAGES SUMMARY
Face recognition is a task that people do consistently without any effort. The human brain causes the visual information of human face as a biometric identifier, because its apperance is highly informative and discriminative.
Face recognition using biometric information is a challenging function of the brain. Developing algorithms that try to act like brain does in face recognition is the one of the most important aim of the computer vision, pattern recognition and image analysis fields.
The face recognition problem can be explained as a given input image or video sequence, identify or verify one or more individuals in the input using a database that contains face images of known individuals. Because of development in technology and great increase in requirements of applications that use face recognition, the number of studies on face recognition increases rapidy in recent years.
Harmonic images based face recognition approaches are important in literature, because these applications greately facilitate the modelling of generic illumination. They are provide solutions to the face recognition problem under varying lighting conditions, especially outdoor environments.
In this thesis, a brief review of harmonic images (spherical harmonic images) and also an approach that uses the combination of statistical model and harmonic images are presented. This approach uses only one training image per subject. This is an important property for face recognition applications because most of the approaches dealing with illumination variation use more than one image. This approach and a similar approach that have some different parts from this- are investigated and experimental results of second one are presented. The experiments have been done using a database that includes synthetic images illuminated such as in YaleB database. According to the lighting direction, we seperated the images into four subsets that we use in both testing and training. Subset4 has the extreme lighting. For subset1 and subset2 the recognition rate is above 95%, for subset3, it is between 90% and 95% and for subset4 it is between 80% and 88%.
HARMONĐK GÖRÜNTÜLERĐ KULLANARAK YÜZ TANIMA ÖZET
Yüz tanıma insanların hiçbir çaba sarfetmeden gerçekleştirdikleri bir görevdir. Đnsan beyni yüze ait görsel bilgiyi, biyometrik tanımlayıcı olarak kullanır. Çünkü, insan yüzü oldukça bilgi verici ve ayırt edici bir özelliktir.
Biyometrik bilgileri kullanarak yüz tanıma, beynin zorlayıcı bir fonksiyonudur. Yüz tanıma konusunda beyin gibi davranmayı deneyen algoritmalar geliştirme bilgisayarlı görü, örüntü tanıma ve görüntü analizinin en önemli amaçlarından biridir.
Yüz tanıma problemi, girdi olarak verilen bir yüz görüntüsü ya da yüz görüntülerini içeren bir video kaydından, bilinen kişilerin yüz görüntülerini içeren bir veritabanını kullanarak, bir veya daha fazla bireyi tanımlamak veya doğrulamak olarak tanımlanabilir. Teknolojideki gelişmeler ve yüz tanımayı kullanan uygulamalara duyulan ihtiyacın artması nedeniyle, bu alanda yapılan araştırmalar son yıllarda hızla artmıştır.
Harmonik görüntü (küresel harmonik görüntü) tabanlı yüz tanıma yaklaşımları literatürde çok önemlidir. Çünkü bu yaklaşımlar genel aydınlanmaları modellemeyi oldukça kolaylaştırmaktadır. Değişik ışıklandırma durumları altında özellikle dış ortamda, yüz tanıma probleminin çözülmesini sağlamaktadır.
Bu tezde, kısaca harmonik görüntüler (küresel harmonik görüntüler), ayrıca istatiksel modeller ve harmonik görüntülerin bileşimini kullanan bir yaklaşım tanımlanmıştır. Bu yaklaşım, her bir kişi için sadece bir eğitim görüntüsü kullanır ve bu önemli bir özelliktir, aydınlanma değişimleri ile mücadele eden birçok yaklaşım bu amaçla birden fazla görüntü kullanmaktadır. Bu yaklaşım ve buna çok benzeyen, farklı bölümler içeren bir yaklaşım incelenmiş, araştırılmış ve ikincisine ait deneysel sonuçlar açıklanmıştır. Deneyler YaleB veritabanına benzer şekilde aydınlatılmış sentetik görüntüler içeren bir veritabanı kullanılarak gerçekleştirilmiştir. Test sırasında kullanılan görüntüler aydınlanma yönüne göre alt gruplara ayrılmıştır. Altgrup4 en aşırı aydınlanmayı içeren kısımdır. Altgrup 1 ve altgrup2 için tanıma oranı %95’in üzerinde, altgrup3 için %90-%95 arasında ve altgrup4 için %80 ile %85 arasındadır.
1. INTRODUCTION
Biometrics attract recognition technology because they can be used in any application which needs security and control access. In addition they eliminate risks with less advanced technologies, based on what a person has or knows, not who a person is. Biometrics means behavioural characteristics. Fingerprints and iris are the most common biometrics. There are many other behavioural characteristics that become to use in recognition, such as voice, signature and face which have been studied in a few decades.
Face recognition becames one of the most important biometric technologies. Because both iris recognition and fingerprint recognition may not be appropriate for people who do not want to colloborate. In addition iris recognition is expensive to implement. On the other hand, face recognition has many benefits over other biometric technologies. Its reliability and acceptance has a good harmony and it provides security and privacy. It can be performed easily in daily life. The nature of the face structure is very informative and discriminative. So, it is easily usable in recognition without bothering people. It has received the attention of the researchers from computer vision, image processing, computational neuroscience and pattern recognition because it is a highly successful application of these fields. The reasons of the popularity of face recognition applications in last decades are avilability of possible technologies and wide range of government, commercial and law enforced applications [1]. Face recognition applications are categorized as FaceID, access control, security, surveillance, smart cards, law enforcement, face databases, multimedia management, human computer interaction in [1]. Table 1.1 shows these categories and the scenarios of the example applications [1].
2
Table 1.1: Face recognition application categories and examples
Category Example application scenarios
Access Control Facility access, ATMs, computer access, computer program access, computer network access, online examination access
Security Terrorist alert, secure flight boarding systems, stadium audiance scanning, computer and computer application security, internet security
Surveillance Advanced video survelliance, nuclear plant surveillance, portal control
Smart cards Stored value security, user authentication
Law Enforcement Crime stopping and susspect alert, suspect tracking and investigation, criminal face retrieval and recognition
Multimedia Management
Face-based search, face-based video segmentation and summarization, event detection
Human Computer Interaction
Interactive gaming, proactive computing
Face ID Driver licenses, immigration, national id, passport, voter registration
1.1 Face Recognition Systems
Face recognition problem is recognizing one or more people in a given input image or video sequence. It uses a database that includes face images of known people. Recognition from input image and recognition from video sequence may vary according to their recognition criteria, algorithms of segmentation and quality of the input(s).
Face recognition systems generally divided into three types. 1. Verification
2. Identification 3. Watch-list
Verification is a one to one match that compares the face image whose identity is claimed with template face images. The second type, identification, compares an input face image with all image templates in a face database to find out the identity of the input face. Lastly, watch-list compares an input face image with all the face images in the database, too, but it also computes a score for each comparison. Then the scores are sorted numerically from highest score to the smallest one. An alarm is raised when a similarity score is higher than a given threshold. To sum up, watch-list matches an input face to a list of suspects one-to- few matches.
A system that solves the face recognition problem usually consists of four parts. 1. Face detection
2. Face alignment 3. Feature extraction 4. Feature recognition
The face areas are segmented from background by using face detection. Face detection also provides close estimates of location and scale of each detected face. Face alignment is used for a more correct localization and normalized faces. Face alignment locates facial components such as mouth, eyes and nose. It uses geometrical transforms and normalizes the face image about properties such as size and pose. After the face image normalized geometrically, face extraction is applied to obtain effective information. The information is used for defining the
4
differentiation faces of different people. This information includes important features of face image and named as feature vector. Face recognition is the verification and identification in the final part. The feature vector of the query face is matched to faces in the database so that the identity of the face is output if there is a match with enough confidence. If there is no match, the face is indicated as unknown face. Figure 1.1 shows the sketch of these steps [1].
Figure 1.1: Block diagram of a generic face recognition system
1.2 Face Recognition in Subspaces
For face recognition, subspace analysis techniques are useful because the face images reside in a subset of input image space. Suppose that, an image of 80*80 has 6400 pixels that includes patterns. These patterns can be presented with
6400
256 possible combinations of these pixels. Face can be one of the patterns in the input image and reside in a subset of the image space called as face subspace.
Face detection/ tracking Đmage/video Face Alignment (Normalization) Face Location, Size&Pose Feature Extraction Feature Matching Database of known individuals Aligned face Feature vector Identification/verifi cation
The most popular subspace methods are Eigenfaces and Fisherfaces. Eigenfaces is based on principal component analysis (PCA) and Fisherfaces is based on linear discriminant analysis(LDA). In this thesis, another subspace method that is based on QR decomposition is used.
1.3 Challenges in Face Recognition
There are so many sytems that are able to succeed in face recognition and many of them are able to achieve face recognition rates better than 80% [1]. Despite, there are still some points that can cause problems in face recognition processes. Because of recognition and examination process to find which class that face image is belong to, face recognition is a complex recognition problem. If images are not acquired under controlled circumstances, some problems that affect the recognition performance occur especially in systems that use only one image for training. The key problems are the sensibility of the classifiers to variations of circumstances. The most important ones are illumination and pose variations. There has been a huge effort to deal with these problems. Facial expressions, occlusions and the age variations are other problems that affect face recognition algorithms negatively. The number of studies that have been done in these problems are less with respect to researches on illumination and pose variations. The principle of dealing with illumination and pose variations is to develop robust classifiers.
1.3.1 Illumination variation
Lighting is not the same during a day or in every place. It changes time to time and from place to place. Because of face has a 3D structure, a direct lighting source causes strong shadows on face which reduce facial features, the part of the face that is exposed to light direction become highlighted and the other part of the face is reduced under the shadow such as in Figure 1.2. Illumination causes a wide variation even between the face images of the same class as explained as following
“the variations between the images of the same face due to illumination and viewing direction are almost always larger than the image variation due to change in face identity” [2].
6
(a) (b) (c)
Figure 1.2: Same face under different lighting conditions. In (a) the face is illuminated by frontal light. In (b) the face is illuminated from left side. In (c) the face is illuminated from right-side.
To eliminate the effects of ordinary illumination variation in face recognition fundemantally a bidirectional method is applied. In this method, after applying a normalization process to input images, a robust face extraction method is developed to deal with illumination variation. Mean value normalization, histogram equalization and illumination correction are the most popular normalization methods[1]. Linear stretching is an example of an intensity normalization operation. In linear stretching, a linear transformation is used and the original range of the query image is changed into a specified full image. Histogram equalization increases the contrast of the input image without affecting the global contrast. Using this way, the strength of the resulting image is distributed on the histogram and this decreases the strong illumination. Illumination correction is a least-squares method, strong illumination is decreased in the difference image. In this method firstly, I’(x,y) the best fitting plane is obtained as follows,
'
( , )
I x y = × + × + , a x b y c (1.1)
where the values of a, b , c can be estimated using a least-squares method, than the illumination is corrected in the resulting difference image as follows,
'' '
( , ) ( , ) ( , )
I x y =I x y −I x y , (1.2)
In Figure 1.3 the effects of histogram equalization, linear stretching and mean-value normalization methods can be seen.
Dealing with illumination variation is a control topic in computer vision. Therefore, there are lots of suggestions for illumination invariant face recognition. The main aim of this thesis is to eliminate the effect of the illumination variations in face
recognition. So, the studies that try to deal with illumination variations will be explained in more detail in Section 1.3.
(a) (b)
(c) (d)
Figure 1.3: Normalization examples (a) Original image from YaleB (b) Linearly stretched image (c) Histogram equalized image (d) Zero-mean normalized image
1.3.2 Pose variation
The pose of the testing and training images are usually not the same in face recognition sytems. For example, the training image may be frontal while the testing image may be taken from a camera placed in the room, viewing the person from an angular position. Pose changes cause projective deformations and self-occlusions. Therefore, they affect the identification process negatively and challenged the researchers working on the face recognition systems. In face recognition under arbitrary pose variations, the problem of developing algorithms to recognize a face in a different viewpoint is being investigated. There are some methods which deal with up to a certain head rotation. However in some cases such as when security cameras create viewing angles that are out of this range, these methods do not deal with this problem.
Linear subspaces have an important place in face recognition under arbitrary pose variations and lots of works have been done that extend linear subspaces to solve this
8
problem. A method was presented in [3] for face recognition with 3D pose variations. It was developed by Okada et al. . In this method, each known people in the training set is represented by using parametric linear subspace model. They stored the parametric linear subspace model representations of each people in which, each model can be fit to input resulting faces of known individuals whose head pose is aligned toinput face. They have worked on two different linear models: the linear PCMAP (LPCMAP) and the Parametric Piecewise Linear Subspace (PPLS) models. The LPCMAP is a parametric linear subspace model that combines the linear subspaces that are spanned by matrices where matrices connect projections coefficients of training samples on the subspaces and their corresponding 3D head angles. The PPLS is an extended version of the LPCMAP by means of the piecewise linear subspaces, which is a set of linear models. Each model supplies continuous analysis and synthesis mapping and makes it possible to generalize about unknown poses by interpolation. It is showed that the recognition system is strong against large 3D head pose variations. The PPLS system performs better than the LPCMAP because it makes the data size fill much less space.
The light-field, that is a 5D function of position and orientation, is used by Gross et al. in [4], in order to achieve a better robust and stable face recognition in dealing with the pose variation problem. The light-field means the radiance of light in free space. In this method, a set of eigen light-fields is obtained and the PCA is applied to a collection of light-fields of faces of different subjects. However, the average light field could also be estimated and deducted from all the light fields. After eigen coefficients can be estimated, input face images are vectorized in light-field vectors. Then, the vectorized face images are used for training and testing the system, which performs the standard eigenfaces algorithm.
1.3.3 The occlusion
Besides illumination and pose variations in face images, occlusion in the face images causes an important change in the face images. The occlusion on the upper-side of a face decreases the performance of the face recognition more than the occlusion on the lower-side of the face. Local approaches that divide a face into different parts and use a voting space to find match are the most popular ways to solve the problem of partially occluded objects.
The voting tecnique may make mistakes in classification, because it does not understand how good a local match is. To solve this problem, in [5] Martinez et al. has been introduced a method. The method divides each face image into k different local parts, and models each of face by using a Gaussian distribution. Gaussian distribution is for explaining the localization error problem. The mean feature vector and the covariance matrix for each local subspace are drown out and the probability of a given match can be connected with the sum of all distances. This approach is successful in only occluded faces.
Kurita et al. developed a method using neural networks that rebuilds the occluded part and discovers the occluded areas in the input image and this method was presented in [6]. The network is trained on non-occluded images in normal conditions, and during the testing, the occluded areas can be replaced with the recalled pixels and the original face can be rebuilt. They showed that classification performance does not change even when 20-30% of the face images is occluded. In [7], Sahbi and Boujemaa has presented a complete face recognition algorithm based on feature extraction in difficult conditions. Firstly, the features are extracted, then they build a binary image that is subdivided into areas and describe a shape variation between faces. They model the statistical difference of each feature in the face model concerning its corresponding matched features in each candidate face of each training set. A matching class, which expresses the probable difference of this feature concerning the training images, is obtained. For small occlusions and rotation effects, the matching process do well and the recognition rate is high and unchangeable.
1.3.4 Time delay
Another problem in face recognition is the time delay, because the face changes over time in a non-linear way. If the time lapse is not very small between getting training and testing images, the performance of most recognition techniques is decreasing. Time delay problem is usually harder to solve than the others. It is still an unexplored and interesting aspect to improve the face recognition systems so as to make them deal with the changes in age. Only a few works have been done about this problem. There are a few methods that periodically upgrade and retrain the system to deal with this problem. Because of the identification performs frequently, this is not an
10
appropriate and practical solution. Another method is simulating the age of the subject to make system more robust about this type of variation.
In [8], Lanitis et al. have proposed a method that try to cope with this problem. The age functions that is obtained using a set of parameters that describe every image in the face database are used in this method. The best age function is choosen for each subject according to these parameters. Different subject-based age functions allow to consider about external factors which cause to the age variation and this is the most important advantage of this approach. The mean age of the subjects has been simulated in both the experiments and then the recognition task has been performed. 1.3.5 Face expression variation
Expression variations also causes an important amount of change in facial appearance. The developed algorithms are quite robust to facial expressions except from extreme expressions, for example screaming.
In [9], Donato investigated several methods for classifying twelve facial actions. He showed that Gabor filters based and Independent Component Analysis (ICA) methods performed the best among other methods such as Local Feature Analysis (LFA), LDA and Local PCA.
In [10], Tian et al. presented an Automatic Face Analysis system to analyze facial expressions based on both permanent features (eye-brows, eyes, mouth) and transient facial features in a nearly frontal-view face image sequence. They reported recognition rates of 96.4 percent for upper face action units and 96.7 percent for lower face units.
In [11], a study investigating the effects of facial asymmetry in face recognition under varying expression is presented by Liu et al.. They have showed that quantified facial asymmetry improved face recognition significantly when combined with conventional methods such as Fisherfaces and Eigenfaces
In [12], facial expression recognition by Kernel Canonical Correlation Analysis (KCCA) is proposed by Abate et al.. They manually locate 34 facial expressions in each image and using Gabor filters they transform these points into a labeled graph. Then, for each training image, they formed a six-dimensional semantic vector describing basic expressions. Learning the correlation between the semantic vector
and labeled graph vector is achieved by KCCA. They presented better results than conventional approaches like LDA and GDA.
1.4 Illumination
Illumination changes influence the performance of face recognition methods negatively as explained shortly in Section 1.2.1. To solve the illumination problem a pretty large number of methods are investigated. The fundemantal methods can be grouped as: The shape from shading approaches, regenerative methods and representation methods. First and second derivative of the grey level images, 2D Gabor filters and edge maps are exemples of these aproaches respectively. All these techniques, however, failed to deal with the problem by themselves. Therefore some new methods are developed which extend these methods as follows.
In [13], with a new approach the edge map technique was improved by Gao et al., called Line Edge Map. In Line Edge Map face contours were determined and joint in segments. Then, these segments were organized in lines. The Houssdorf distance was changed a bit, to manage the new feature vectors. Besides, a new prefiltering technique was described for screening the whole set of subjects before performing the real testing operation. This method has been tried out in several different poses and illumination. Although this approach outperforms some other methods, for instance linear subspaces or Eigenfaces [14], the Fisherfaces [14] are stil superior as they can maximize the between-subject variability and minimize the within-subject differences.
In [15], Sim et al. presented a model and exampler-based approach for recognition. The main idea was to synthesize many examplers, that are used in the learning stage of a face recognition system. To realize this, they developed a statistical shape-from-shading model to recover the face shape from a single image, and to synthesize the same face under new illumination. The aim of using this is to build a fast classifier that was not possible before because of lack of training data. They used three classifiers: the first classifier performs nearest-neighbour search directly, the second classifier searches for the nearest exampler in the global PCA subspace and the third classifier computes individual PCA subspaces from the examplars. The accuracies of the first two classifires are not good enough and the accuracy of the third one is 95% by using YaleB and CMU-PIE databases.
12
In [16], Georghiodes et al. suggested an illumination cone model for face recognition under varying lighting. In the image space, under arbitrary lighting, the set of images of an object forms a convex cone and this fact was used in this model. It was also explained how the cone was made from as few as three images which are all under small lighting changes. The distances are calculated between the test image and each illumination cone, and the face which has the shortest distance is thought as the true identity.
In [17], Romdhani et al. recovered the shape and texture parameters of a 3D morphable model in an analysis by synthesis manner. This method compute a statistical model during a bootstrap phase which encapsulate texture and shape parameters.
An illumination invariant method which was developed from near-inframed (NIF) images is proposed by Li et al. in [18]. They used Local binary patterns (LBP) and took out features from the near-inframed images. NIF imaging contributed to the monotonic grey level transform to which LBP provided a good solution. Also, they used ada-boasted LBP features and improved a face matching engine, and presented and LDA-like scheme to further select discriminative LBP.
Harmonic images based face recognition methods are methods that are successful under arbitrary unknown illumination conditions and most of them uses only one training image per person. In Section 1.5 the algorithms that use spherical harmonic images for face recognition will be presented.
1.5 Harmonic Images
Harmonic images (spherical harmonic images) model is an important development on light modeling. Many studies have done after the studies of Basri et al. [19] and Ramamoorthi et al. [20].
The first study on turning incoming light into reflection was done by D’Zmura [21]. He showed that describing the reflection in terms of spherical harmonics.With this representation, after truncating high-order components, the reflection process can be written as a linear transformation, so the low-order components of the lighting can be recovered by inverting the transformation. He used this analysis to explore ambiguity in lighting [10].
The most important work in face recognition by using spherical harmonics is done by Basri & Jacobs [19] based on [21]. They showed that, nine or fewer basis images were enough to approximate the images of a convex Lambertion object with high accuracy. In addition, this method was successful to deal with the attached shadows. Basri & Jacobs used a kernel which represents Lambert’s reflectance to model the reflectance functions in analogous with the convolution of each lighting function. They proved that the set of images of a convex Lambertian object could be approximated by a nine-dimensional subspace that include nine harmonic images. They explained that the first nine harmonics includes 99.2% of the energy of kernel and clarified how to get the nine-dimensional subspace from a model of an object which contains albedo and 3D structure. This method will be explained in more deail in Section 2.
The following methods that will be presented, have developed from the study of Basri & Jacobs in [19]. Zhang et al. focused on face recognition with spherical hormonics in [22]. The approach that belong to them will presented as follows: They suggested a method based on spherical harmonics that needs only one training image per subject. This method requires no 3D shape information, because it builds the statistical model based on a collection of 2D basis images. They showed that they can estimate the spherical harmonics basis images from just one image , by using the learned statistics, taken under arbitrary illumination conditions if the pose is invariant. They use YaleB database in experiments and the accuracy was changed between 97.2% and 100% with respect to illuminant angle. This method will be discussed later in more detail.
In[23], Zhang et al., represented two methods for face recognition under arbitrary lighting by using spherical harmonics and compared them. The first method is same in [22] and the second method constructs the statistical models directly in 3D spaces by using both the spherical harmonic illumination representation and a 3D morphable model of human faces. Using the statistical model the basis images are obtained. The recognition method in [19] is used for both two methods. With respect to previous works done in this field, recognition rate is good. But when the poses of training and testing images are very different, the recogniton rates are not as good as under constant pose.
14
In [24], Zhang et al. proposed a 3D Spherical Harmonic Basis Morphable Model (SHBMM). They obtained this model by integrating spherical harmonics into the morphable model framework and demonstrate that any face under arbitrary lighting can be simply represented by three low-dimensional vectors: shape parameters, spherical harmonic basis parameters and illumination coefficients. Given an image of a face, they explicitly recovered the shape information and estimate the spherical harmonic basis of the face. In this method, testing and training images can have different skin albedos and poses. In addition their model detects the shadow errors from the image difference between the testing image and the rendered image. They can remove the cast shadows from the input image and add cast shadows to the image to generate more realistic images. They experimented their method by using CMU-PIE and USF databases. They achieved high recognition rates for images under a wide range of illumination conditions.
In [25], Rara et al. represented a face recognition method using statistical model that combines shape (2D and height maps), appearence(albedo) and spherical harmonic projection information. The method takes a 2D frontal face image under arbitrary illumination as an input and outputs the estimated 3D shape and appearence. Face identification is performed using the shape and albedo coefficients. For frontal images the recognition results are perfect on Extended Yale B database.
In [26], Qing et al. presented an illumination normalization approach by relighting face images with a canonical illumination based on the harmonic images model. Using the harmonic image model in [23], they presented a model-based approach to identify faces robustly under generic illumination with the face relighting technology. Face recognition is achieved by matching the canonical form of the test image with canonical form of the training images. The experimental results that obtained using YaleB are little worse than that of the illumination cone [16] and harmonic examplers [22]. After relighting under arbitrary illumination direction, the error rates are decreased. As a result, relighting strategy improve the performance of the face recognition approach.
In [27], Yue et al. presented a method that extended the spherical harmonics to encode the pose information into the harmonic representation. They showed that the basis images of a rotated test image is a linear combination of the basis images at the frontal pose. They used the learning method that is the same in [22] to recover the
harmonic basis images from only one image. For a rotated testing image under arbitrary illumination condition, a correspondance is applied between this testing image and the training images. Then the frontal pose image was warped from the testing image and the face was identified by using a linear construction based on basis images that is the closest to the testing image. They experienced the method on Vetter’s 3D face model database. They used synthetic images. The recognition rate was nearly perfect, except extreme illumination conditions.
1.6 Face Recognition From a Single Intensity Image
In face recognition field, apperarence-based methods are the dominant ones since 1990’s. Performance of these methods is depending on the number of training samples for each subject. Most of current face recognition methods use at least two samples of the same subject. On the other hand, in real-world applications, the number of samples per person is smaller than the assumed number of samples. In many applications such as driver licence one training sample per subject is used in the database. Therefore, the techniques that are using one training sample per person are studied in last decades. The advantages of one training sample per person in face recognition systems are collecting the samples easily, decreasing the store cost of samples and decreasing the computational cost of the recognition system [29]. Those are desired by the real-world applications.
Face recognition methods from intensity images have three categories: holistic methods, local methods and hybrid methods [29]. This algorithms are categorized depending on the sort of features used by different methods.
1.6.1 Holistic methods
Holistic methods use the whole image as input for identifying a face and concatenate the gray-values of all pixels in the face. In these methods, each face image is represented as a simple high-dimensional vector. Holistic methods keep all the detailed texture and shape information, which is useful to recognize the differences between faces. Under the condition of one sample per person, this representation causes two adverse results. A trade-off between high dimensions of image data and small samples more serious is appeared, when holistic methods are used. The most important problem about these methods is how to deal with very small sample
16
problem. Also, the within-class variation cannot be calculated directly because there are only one vector for each class.
There are two ways to deal with these problems. One of these ways is to squeeze the information as much as possible from the simple face image. It can be done in the high-dimensional face space or in the dimensionality-reduced eigenspace, as extensions of the standard PCA technique. The other way is to build new views or different representations for each image and to include prior information. By this way, the real training set can be made much more bigger.
PCA and LCA are two examples of holistic methods. Much success has been fulfilled by holistic methods, however only one feature vector is used in representing each face image. This representation is sensitive to large appearance because of expression, illumination, pose and partial occlusion.
If only one sample per person is given, the generalization performance of PCA is not enough. But PCA is applicable for any sample size based on the viewpoint of computation. To get higher performance, the method have been extended by Wu et al. in [30]. They presented a method named (PC)2Ato strenghten the information of face space. They obtained the projections of image that reflect the distirbution of the facial features. These features are more useful for face recognition than the features obtained from PCA. Then from the projected images, a new image is synthesized and it is combined with the real image. The aim of this act is to obtain a strengthen procedure. After this process, unimportant features are faded out and the important features become more salient.
A potential solution to reliably estimate the covariance matrix under the small sample size condition, two-dimensional PCA (2DPCA) has recently proposed by Yang et al. in [31]. This method uses 2D image matrices rather than 1D vectors for covariance matrix estimation. So, for small sample size problem, it is computationally cheap and suitable. The good results of 2DPCA has been observed on several well-known image databases.
In[32], Jung et al. develeoped a system to handle the expression, illumination and pose problems. They try to synthesize multiple new face images which imitate the corrupted images for recognition.The error rate of this method is 1.32%. This method improve the similarity between the corrupted images and the training images.
In [33], Penev et al. proposed Local Feature Analysis (LFA) approach. The aim of this approach is to cope with the limitations of PCA such as lack of providing local features and production of global non-topographic linear filters, in this method, a dense set of local feed forward receptive fields defined at each point of the receptor grid, whose outputs were as decorrelated as possible, were derived. The residual correlations contained by these outputs were further used in the sparsification of the output. The final representation was a local sparse-distributed representation in which only a small number of outputs are active for any given input.
1.6.2 Local methods
Local methods use the local facial features as a difference from holistic methods for recognition. Deciding how to include global information into local face models is a difficult subject in these methods.
Local methods are more suitable than holistic methods for dealing with the one sample problem. There are three reasons for this feature of local methods [29]:
1. The original face is represented by a set of low-dimensional local feature vectors, not by one simple full high-dimensional vector.
2. Local methods supply additional flexibility to notice a face based on its parts.
3. Different facial features can rise the variety of the classifiers and it is useful for face identification.
The local feature-based method and the local appearance-based methods are the categories of local methods [2]. Local feature-based methods determine local features firstly, and then extracts features on the located feature points. Usually only one simple image is used to obtain geometrical measures such as width of head or distance between eyes. These measurements are stored in the database to be used later. Most of earlier face recognition methods are belong to this category. Gabor wavelet decomposition and Elastic Graph Matching are examples of this group of methods. Local appearance-based methods simply divide the face image into sub-regions based on which local features are directly taken out. Local region partition, feature extraction, feature selection and classification are the steps of local appearance methods. Local regions are defined in local region partition step. In local
18
feature extraction step, it has to be decided how to represent the information of them, which is important for the performance of a recognition system. Grey-value features and different types of derived features, such as Gobor wavelets are frequently used features. Additional feature selection stage is used for effectiveness and efficiency when a lot of features are created in the previous step. PCA and LDA are common feature selection methods and the most common way for identification is combining classifiers.
In[34], Brunelli et al. presented a method that uses a set of templates to detect the eye position in a new image by looking for the maximum absolute values of the normalized correlation coefficient of these templates at each point in the text image. There are many methods that have similar aim and operations.[35-37]
In [38], Manjunath et al. presented a method for feature detection and representation of face. The method is based on Gabor wavelet decomposition in [39]. Location information and feature information for each detected point is saved and model the relationship between the feature points a topological graph has been constructed. After that, the face recognition problem can be handled as graph matching problem [40] by Wiskott et al.. The recognition rates of this method is exceeds 86%. The disadvantage of this method is not to allow the modifications in using databases because it uses topology graph.
In [41], Lades et al. presented a deformable topology graph matching method, that is known as Elastic Bunch Graph Matching(EBGM) and become a successful method in the literature. Similar to [38], a topology graph is constructed based on Gabor filters. The Gabor features are robust against illumination change, distortion and scaling[39].
In[5], Martinez et al. presented a local probabilistic approach to recognize partially occluded and expression variant face from a single sample per face. The face subspaces are obtained using local patches at the same position of each face image, than they transformed into eigenspaces. In the identification part, testing image is divided into subregions and projected to eigenspaces. To measure the similarity of a given image to the training images, a probabilistic calculation is used. Under synthetically generated images the results shows that the local probabilistic approach does not reduce accuracy even when 1/6 of face is occluded.
In [42], Nefian et al. developed a Hidden Markov Model (HMM) based method. In their method, the observation vectors of the HMM system were extracted by Karhunen-Loeve transform. They presented an 86% recognition rate on ORL database. They also proposed a novel HMM-based face detection method in the paper.
1.6.3 Hybrid methods
Hybrid methods combines the former two methods. They use both holistic and local features, which makes the performance of a recognition system better than individual local or holistic methods. The most important factors affecting the performance of hybrid methods are how to determine which features will be combined and how to combine them in order to keep their advantages and prevent their disadvantages at the same time. Local features and the global features have different properties and can offer complementary information about the classification task. This method is not appropriate for recognition that uses only one training images. Therefore, hybrid methods might be a good way to reduce the complexity of classifiers and make their generalization capability greater.
In [8] Lanitis et al. proposed a flexible appearance based method for face recognition that used shape and intensity information. The statistical shape model is trained on 2D images using PCA. In classification, between-class variations of the shape model are differentiated from the within-class variations of the shape model by discriminant analysis. Local gray-level models were also built on the shape model to cope with the local appearance changes such as local occlusions. A global shape-free representation was also obtained by using mean shape and PCA. Finally, these three representations, shape parameters, shape-free parameters and local features were used together to compute a Mahalanobis distance.
In [43], Huang et al. proposed a component-based detection and recognition system. Component-based methods seperate face into several components such as mouth, eyes, nose that are connected by a flexible geometrical model. By using components, the changes of the head pose would affect the components positions and this would be coped with the flexibility of the geometric model. A disadvantage of the system is the need of a high number of training images containing different pose and illumination variations. In the classification phase, SVM classifier was used. On a
20
set of six subjects; training on 3 images and testing on 200 images, the hybrid system presented a recognition rate of 90 percent.
In [12], Abate et al. represented that the local and global features are separately sensitive to different variation factors. For instance, local features may be affected more by illumination changes and global features may be influenced more by expression changes. In Table 1.2 properties of the features are summarized [12]. Table 1.2: Comparison of the sensitiveness of local and global features to
variations
Variation factors Local Features Holistic Features Small variations Not sensitive Sensitive Large variations Sensitive Very sensitive
Illumination Very sensitive Sensitive Expressions Not sensitive Sensitive
Pose Sensitive Very sensitive
Noise Very sensitive Sensitive Occlusion Not sensitive Very sensitive
The studies on this part is still a few because the methods are not suitable for one sample problem [12].
1.7 Organization of Thesis
In Section 2, general information about the lambertian reflectance and the spherical harmonics is given. In Section 3, face recognition method that uses spherical harmonics and a single training image per person is explained in detail. In Section 4 the method that we worked on is totally presented. In Section 5, experimental results are presented and in Section 6 conclusion and feature work are given.
2. LAMBERTIAN REFLECTANCE AND MODELLING ILLUMINATION VARIATION WITH SPHERICAL HARMONICS
Spherical harmonics are used to cope with illumination variation by modeling the illumination. Most of algorithms that try to solve this problem, are not suffice for modeling illumination variation in uncontrolled conditions like outdoors. The obvious empirical method for analysing illumination variation is recording a number of images of an object under light sources from all different directions. This corresponds to moving a light source in a sphere around the object while the camera and the pose are fixed. Because the light moves in a linear way, the image under multiple light sources can be showed as the linear combination of source images that are obtained under individual light sources. So, instead of using images, the linear combinations or basis images that explain the illumination variation can be used. Usage of at least four basis images is enough for modeling the variability due to the illumination, generally nine basis images are used.
The works on face recognition methods that use spherical harmonic images gain a speed after the works of Basri and Jacobs [19] and Ramamoorthi and Hanrahan [20], because they showed that the set of images of a convex Lambertian object which is obtained under arbitrary illumination conditions can be approximated by a nine-dimensional or fewer subspace. The subspace includes nine basis images or fewer images of the object called as harmonic images. The method in [19] is appropriate for dealing with the attached shadows because the attached shadows can be accounted for with analytically derived linear subspaces.
In Section 2.1, a background information is given about reflectance and lighting, in Section 2.2 the previous works done for attached shadows are discussed, then the face recognition method based on spherical harmonics is explained step by step that developed by Basri et al. [19].
22 2.1 Reflectance and Lighting
There are common assumptions in handling illumination in computer vision and computer graphics. The first one is assumption of distant illumination to represent lighting as a function of direction. The second one is to neglect cast shadows from one part of the object on another. In the following sections, these assumptions are taken into account.
While explaining reflectance and lighting we assumed only distant light that shines on each point in the scene that have the same intensity (the surface of the object reflects the light according to Lambert’s law). According to Lambert’s law [24] objects reflect the light uniformly.
For the illumination from a point light that has a single direction on a Lambertian object, a model is considered. In this model, illumination can be represented by the unit vector ul and intensity l . If there are multiple light sources, the intensity of light can be described as a funcion of direction ( )l ul . In this case, the intensity of light does not depend on the position in the scene and it can be assumed as a non-negative function of the surface of a sphere.
Building accurate face models from reflecting light is a complex task. When the light entering the skin at one point, it may exit another point, so the skin is not homogenous. This situation is not an easily solvable problem. So, the simpler models are more applicable and efficient for face recognition by considering the assumptions [1].
In this model the only variable is albedo of the object. The albedo describes the fraction of the light reflected at that point. According to Lambert’ law and the given information up to now, the intensity (radiosity, reflected light) is obtained as follows:
( ) max( . , 0)l l r
i=l u
ρ
u v , (2.1)where l corresponds to a light ray of intensity, ul corersponds to direction of the light, ρ is the albedo of a surface point and
v
r is the normal direction.By ignoring albedo and assuming the light direction is constant, the reflected light is a function of surface normal only. If light reaches a point from multiple directions,
the light reflected by the point is obtained by taking integral over the contribution for each direction. 2 ( )r ( . ) ( )l r l l S r v =
∫
k u v l u du , (2.2) where ( . )k u vl r =max( . , 0)u vl r and2
S
∫
indicates the integration of the surface of a sphere.2.2 Spherical Harmonic Representation and Properties of Convolution Kernel Until Basri et al. and parallely Ramamoorthi et al. analyzed the properties of illumination cone in terms of spherical harmonics, the presence of attached shadows have been unexplained. Then they showed that when the attached shadows are clarified, the image of convex, Lambertian object can be approximated, using nine basis images with high accuracy. Firstly, the illumination cone and a few works presented then the spherical harmonic representation is explained.
The equation (2.2) is the key to producing linear lighting models that is dealing with the attached shadows. This equation describes how lighting is analogues to a convolution on the surface of the sphere. In this equation, reflectance is determined for each surface normal,vr, by integrating the coming light from all directions weighted by kernel k. For every vr, the kernel is rotated version of the same function and named as half-cosine function.
Since the convolutions are used, they can be analyzed in frequency domain. Fourier basis and harmonic basis are similar. Since unit vectors corresponding to the surface normals the appropriate signal-processing tools are spherical harmonics (on a sphere). These are similar to Fourier series in 2d (on a circle). These tools are examined below:
The spherical harmonics function denoted by hnm. hnm is the n’th order spherical harmonic and represented such as in equation (2.3). These functions represent a set of functions that form on orthonormal basis for the set of all functions on the surface of the sphere. The first part of the equation is normalization factor.
24 (2 1)( )! ( , ) (cos ) 4 ( )! im nm nm n n m h P e n m
π
Φ + − Θ Φ = Θ + , (2.3)where n =0,1,..., n− ≤m≤ and n Pnm is associated Legendre function defined as
2 / 2
(1
)
( )
(
1)
2
!
m n m n n nm n n mz
d
P
z
z
n dz
+ +−
=
−
. (2.4)The spherical harmonics form an orthonormal basis, any piecewise, continues function on the surface of the sphere can be written as a linear combinaton of an infinite series of harmonics. Therefore, kernel k and the light l can be represented as linear combinations of the surface harmonics. Equation (2.5) is the harmonic expansion of light and equation (2.6) is the harmonic expansion of the kernel.
0 n nm nm n m n l l h ∞ = =− =
∑ ∑
(2.5) 0 0 ( ) n n n k u k h ∞ = =∑
(2.6) Funck-Hecke theorem shows that convolution in the function domain is equivalent to multiplication in harmonic domain. Therefore, it can be shown that the possible reflectance of a sphere all lie near a low-dimensional linear subspace of the space of all function defined on the sphere.0 4 * ( ) 2 1 n n nm nm n m n r k l k l h n
π
∞ = =− = = +∑ ∑
(2.7) The Funck-Hecke theorem explained that when the reflectance function, r , the amplitude of the light l , at every order nis scaled by a factor that depends only on the convolution kernel k . This is used to determine which frequencies dominate r . The harmonic expansions of the Lamabertion kernel k is as follows:, n=0 2 , n=1 3 ( 2)! (2 1) ( 1) 1 , n 2, 2 2 ( 1)!( 1)! 2 2 0 n n k n n n even n n
π
π
π
= − + − + ≥ − + , n 2, odd ≤ (2.8)The first few coefficients are,
0 1 2 4 6 8 3 5 7 5 0.8862 k = 1.0233 k 0.4954 2 3 8 13 17 0.1108 k 0.0499 k 0.0285 6 128 256 ( 0) k k k k k
π
π
π
π
π
π
= ≈ ≈ = ≈ = − ≈ − = ≈ = ≈ − = = = (2.9)The energy captured by every harmonic term is calculated by the square of its respective coefficient divided by the total squared energy of the transformed function. The total squared energy in the half-cosine function is given by
2 /2
2 2
0 0 0
2
( ) sin 2 cos sin
3 k d d d π π π π π Θ Θ Θ Φ = Θ Θ Θ =
∫ ∫
∫
(2.10)The relative energy captured by each of the first several coefficients is shown in Table 2.1 [19]. The kernel function, the half-cosine function can be written as
1 1 5
( ) cos( ) cos(2 )
4 2 16
26
Table 2.1: Energy captured by the n’th harmonic for the Lambertian kernel. The middle row shows the cumulative energy up to n. The last row shows the energy that might be affected by the non negative light. The middle row also shows the quality of the approximation of reflectance function.
N 0 1 2 4 6 8
Energy 37.5 50 11.72 0.59 0.12 0.04
Cumulative Energy 37.5 87.5 99.22 99.81 99.93 99.97
Lower bound 37.5 75 97.96 99.48 99.80 99.90
2.3 Approximating the Reflectance Function
The lambertian kernel has most of its energy in the low-ordered terms, so it acts as a low-pass filter. As a result, any reflected function under any light can be approximated using only low-order spherical harmonic reflectances. It is, assumed that, lighting generated by a point light source at the z direction and it is a delta function. The reflectance of a sphere under illumination by a point source is calculated by a convolution of the delta function with the kernel. Appoximation of reflectance due to this point light source corresponds to 99.22 of the energy. The approximation is obtained by using the linear combination of first three harmonics. By using first order harmonics 87.5% of the energy, by using fourth order harmonic 99.81% of the energy is captured. The quality of the approximation improves with the addition of the fourth order term and deteriorates when a first order approximaiton is used. The graphical representation of energy that is captured by harmonic coefficients is in Figure 2.1 [19].
Figure 2.1: Representatoin of the first 11coefficients, the relative energy captured by each coefficients and accumulated energy, respectively.
If the direction of the point light source is different than z, the reflectance that obtained would be the same, but shifted in phase. Shifting in phase is distributed the energy over the harmonics of the same order, but overall enery, on each harmonics of the n’th order is not changed.
2.4 Generating Harmonic Reflectances
Basri et al. showed that constructing a basis to the space can be done analytically. These basis are low ordered hamonics and can be obtained by using Funck-Hecke theorem. In many cases The basis vector for the nm components of reflectances to indicate the reflectance produced by a corresponding basis vector describing the lighting h is wanted. This makes relating the reflectances and lighting easy. These nm reflectences called harmonic reflectances and to get these reflectances the half-cosine kernel is convolved with simple harmonics. The reflectance harmonics are calculated as in equation (2.9):
nm nm n nm
r = ∗k h =c h , (2.12)
where h is the harmonic light and k is the kernel. To determine nm c Funck-Hecke n theorem is used and finally r is given by nm
4 2 1 nm n nm r k h n π = + . (2.13)
