REP2008451: Face Relighting for Recognition
Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
Dept. of CISE, University of Florida
Abstract. In this paper, we present a novel generative model for illumination of
faces with cast shadows and specularities using a single example image. First,
we present a novel method for the estimation of the ABRDF field of a face us
ing Bspline modulated antisymmetric spherical harmonic functions which does
not make the Lambertian assumption but yet uses only nine 2D images (which
were shown in the literature to be the sufficient number of images for a convex
Lambertian surface). Then, using information theoretic nonrigid registration and
intensity mapping techniques, we propose a novel algorithm for transferring this
field from one face to another using just one 2D image of the target face as input.
We present photorealistic results of synthesis of face images under novel illumi
nation conditions (including extreme illumination) and demonstrate the effective
ness of the proposed framework in improving performance of various benchmark
face recognition techniques on the Extended Yale B face database.
1 Introduction
Due to important applications like face recognition ([1], [2], [3], [4], [5], [6], [7], [8])
and facial relighting ([9], [10], [3], [4], [11]) synthesis of facial images under novel illu
mination conditions has attracted immense interest, both in the field of computer vision
and computer graphics. This problem can be defined as follows: Given a few example
images of a face, generate images of that face under novel illumination conditions.
A particularly challenging incarnation of this problem is when only one example
image is available, which is the most common and realistic scenario in the all important
application of face recognition. The attractiveness of this problem stems from the fact
that if multiple images under novel illumination can be generated from a single example
image, they can be used to enhance recognition performance of any learning based
face recognition method. However, it must be noted that this scenario is different (and
more difficult) than the typical graphics relighting problem ([9]) where generally no
limitation on number of example images is considered.
The literature is replete with various proposals to solve this problem, but each of
these work only under certain assumption (e.g. convexLambertian assumption), re
quire specific kind of data (e.g. 3D face scans) and/or manual intervention. Thus, it
is important to compare these methods in the light of these assumptions and require
ments and not just by the claimed results. The method that we propose in this paper
produces results which are better or comparable to those of the existing methods, even
though it works under an extremely emaciated set of requirements. It is a completely
automatic method which works with a single 2D image, does not require any 3D infor
mation, seamlessly handles cast shadows and specularities (i.e. does not make convex
2 Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
Fig. 1. First two rows explain our choice of antisymmetric basis and the last row shows that using
symmetric basis leads to unnatural brightness while antisymmetric basis does not.
Lambertian assumption) and does not requires any specially acquired information (i.e.
works well with i. \, ini benchmark databases like Extended Yale B [2]).
The convexLambertian assumption is inaccurate as human faces are neither ex
actly Lambertian nor exactly convex. It is common to see cast shadows (e.g. perinasal
region, due to nonconvexity) and specularities (e.g. oily forehead and nose tip, due
to nonLambertianess) on facial images and any method which fails to explain these
is clearly limited in its applicability. On the question of using 3D information, it was
recently pointed out by Lee et al. [12] that though the cost of acquiring 3D geome
try is decreasing, most of the existing benchmark face databases consist of a single
or multiple 2D face images (e.g Yale B [6,2]) and hence it is more pragmatic (if not
more accurate) to use only 2D information as input to systems dealing with facial il
lumination problems. Furthermore, recent systems that do use 3D information directly
(based on the morphable model suggested by Blanz and Vetter [13]), require manual
intervention at various stages which is clearly undesirable. At the same time we must
point out that techniques which require specially acquired 2D information or exorbitant
REP2008451: Face Relighting for Recognition
amount of it, are also not attractive. In the light of these possible limiting assumptions
and requirements, next, we survey the existing techniques.
Basri and Jacobs [1], using the 3D shape and albedo information of faces, proposed
a technique for approximating the illumination of a face with a 9 dimensional subspace.
They assume faces to be convex Lambertian object. Based on their results and using
a bootstrap set of 3D facial scans, Zhang and Samaras [3] proposed a technique for
estimation of the 9 illumination subspace basis images from a single 2D image of a
new face. More recently, Zhang et al. [4] put forth a spherical harmonic basis morphable
model which was used to generate novel illumination and pose images from a single 2D
image. Herein they require 3D facial scans to learn the morphable model. Both of the
last two methods work with the convexLambertian assumption. Lee et al. [12] proposed
a technique which does not make the Lambertian assumption but requires specially
acquired 2D information along with the 3D face scans. Based on the work presented in
[1], Lee et al. [2] empirically found a set of universal illumination directions, images
under which can be directly used as a basis for the 9 dimensional illumination subspace.
Again, recognition using this method requires images obtained under the very specific
illumination conditions.
Zhao and Chellappa [8] used shape from shading to generate 3D surfaces of faces
from single example images which were then used to render images under novel illu
mination and pose. Sim and Kanade [7] used statistical shape from shading to generate
novel images which they then used to train a face classification system. Georghiades
et al. [6] used photometric stereo to reconstruct 3D face geometry and the albedo map
from seven frontal images under varying illumination which was then used to render
images under novel illumination conditions. They use explicit ray tracing to handle cast
shadows because they make the Lambertian assumption. These systems which compute
3D information from 2D images either use shape from shading which is prone to errors
[7], make assumptions which are not always applicable (e.g. symmetry [8]), or require
multiple images and computational intensive ray tracing to do so [6].
RiklinRaviv and Shashua [14] in their seminal paper proposed a technique called
quotient images, which worked with illumination invariant signatures of images. Wang
et al. [15] extended their technique to selfquotient and generalized quotient images
and applied their methods to face recognition. Stochek [16] unified quotient images
with image morphing to generate novel illumination and pose images. All of these four
methods work with the Lambertian assumption and hence do not handle cast shadows
and specularities robustly. Debevec et al. [9] proposed a method that captures the face
reflectance field but require a dense sampling of the illumination directions (a lot of
training images) to work.
Our research presented here builds on existing mathematical tools like BSplines
regularization, Spherical Harmonics basis and nonrigid Registration, to produce novel
set of relighting and recognition results. Our contribution may be summarized as fol
lows:
1. Using bicubic BSpline modulated antisymmetric spherical harmonics, we present
a novel framework for capturing ABRDF field of faces using just 9 images.
4 Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
(a) (b)
Fig. 2. Fig 2(a) shows images under novel illumination directions synthesized from the estimated
ABRDF field. In Fig 2(b) the blue circles represent the illumination direction used for learning the
ABRDF field and the solid squares represent the novel illumination directions used to generate
the images in Fig 2(a) (spatial ordering of the squares and the images is the same).
2. Using information theoretic registration and a novel intensity mapping method, we
present a novel algorithm for transferring ABRDF field from a reference face to a
target face using just 1 image of the target face.
3. Putting the above two together, we present a novel system, which doesn't make
any restrictive assumptions, but still can generate images of a face under novel
illumination conditions from just 1 2D image, once a reference ABRDF field has
been estimated.
The rest of the paper is organized as follows. In Sec. 2 we describe the framework
for capturing the ABRDF field. In Sec. 3 we put forth the ABRDF field transfer algo
rithm. In Sec. 4 experimental results are presented, and in Sec. 5 we conclude with a
summary of our contribution.
2 ABRDF Field Estimation
Our goal is to generate images of a face under various illumination conditions using a
single example 2D image. We propose to achieve this by acquiring a reference ABRDF
field once and then transferring it to new faces using their single images. The ABRDF
represents the response of the object at a point to light in each direction, in the presence
of the rest of the scene, not merely the surface reflectivity. Hence by acquiring the
ABRDF field of an object, cast shadows, which are image artifacts manifested by the
presence of scene objects obstructing the light from reaching otherwise visible scene
regions, can be easily captured. Note that since we want to analyze the effects of the
illumination direction change, we would assume ABRDF to be a function of just the
illumination direction (by fixing the viewing direction), though sometimes it is defined
REP2008451: Face Relighting for Recognition
Fig. 3. Closeup of the boxed region shows that multiple bumps arising from cast shadows and
specularities have been neatly estimated by us. This is not possible with a Lambertian model.
to be a function of both illumination and viewing direction. In this section we describe
the first part of this process reference ABRDF field estimation, using novel bicubic
BSpline modulate antisymmetric spherical harmonics.
2.1 Surface Spherical Harmonic Basis
The surface spherical harmonic basis, the analog to the Fourier basis for cartesian sig
nals, provide a natural orthonormal basis for functions defined on a sphere. In gen
eral, the spherical harmonic basis are defined for complexvalued functions but as the
ABRDF is a real valued function, we choose the real valued spherical harmonic basis
to represent the ABRDF functions. We denote the spherical harmonic basis functions
as f'm (order: 1, degree: m), with 1 = 0, 1, 2, ... and 1 < m < 1:
21+ 1 (1 m n)
(47r (1+m)! '
where Pi in are the associated Legendre functions and m (0, y) is defined as
) 2cos m m > 0,
(0,4)= 1 0, (2)
/2 sin Im m < 0.
Note that even orders of the spherical harmonics basis functions are antipodally
symmetric while odd orders are antisymmetric. We now claim that perceptually speak
ing, given a limited number of data samples, the ABRDFs are best approximated using
only antipodally antisymmetric components of the spherical harmonic bases. To recog
nize this there are two crucial questions that must be examined, first, would using just
even or odd ordered bases drastically limit our approximation of ABRDF and second,
which of the two symmetric or antisymmetric bases are better.
6 Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
Our Method Ground Truth Lambertian
Fig. 4. Cast shadows (below the nose) & specularities (on the nose) are seamlessly handled by us.
The answer to the first question is no, because even though ABRDF is a function
defined on a sphere, we are only interested in its behavior on the frontal hemisphere.
Hence, if we ignore the function's behavior at very extreme angles (0 and 180 ), once
the ABRDF has been modeled accurately on the frontal hemisphere, the rear hemi
sphere can be filled in appropriately to make the function either symmetric or antisym
metric. To visualize this, polar plots in the first column of Fig. 1 show a typical ABRDF
function defined on a semicircle. The second column shows the same function being ap
proximated by a antipodally symmetric (row 1) and an antipodally antisymmetric (row
2) function and hence by not using both type of components, we do not loose much
in the approximation power. Note that for visualization, we have scaled the problem
down to 2D and the blue circle represents the zero value in these polar plots. A more
important reason that keeps us from using the complete set of bases is that for a fixed
number of given example images, using just symmetric or antisymmetric components
allows us to go to higher orders which are necessary to approximate discontinuities like
cast shadows and specularities in the image.
To answer the second question, we have to look at the function behavior at the ex
treme angles (0 and 180 ). In reality, most facial ABRDF functions have a positive
value near one of the extreme angles (as they face the light source) and a very small
(w 0) value near the other extreme angle (as they go into attached shadows). Hence,
the function in column 1 of Fig. 1 is very close to physical ABRDFs. Now clearly the
function behavior at 0 and 180 is neither antipodally symmetric nor antisymmetric
and hence using just one of the two would lead to errors in approximation at these ex
treme angles. Now the error caused by symmetric approximation is perceptually very
noticeable as it gives the function a positive value where is should be 0 (Fig. 1 (last
column, first row) and see regions marked by arrows in Fig. 1 last row, they are unnatu
rally bright) while the error caused by antisymmetric approximation is not perceptually
noticeable as it gives the function negative value where it should be 0, which can be
easily set to 0 as it is known that ABRDF is never negative (Fig. l(last column, second
row) and see Fig. l(last row), last two images). Nonnegativity is achieved similarly in
the Lambertian model using the max function [1]. Errors at the nonzero end of the
function are not perceptually noticeable, as can be seen from the last row of Fig. 1.
2.2 Bicubic BSpline Modulated Spherical Harmonic
For a fixed pose, each pixel location has an associated ABRDF and across the whole
face, we have a field of such ABRDFs. To model such a field of spherical functions (S2 x
R2 R), we propose to use modulated spherical harmonics by combining spherical
REP2008451: Face Relighting for Recognition
Reference Target CCRE MI
Fig. 5. Registration Results: 1st column contains the reference image, 2nd column contains the
target image, 3rd & 4th columns contain deformed faces produced by CCRE and MI respectively.
harmonic basis within a single pixel and Bsplines basis across the field. The Bspline
basis, Ni,k, where
Ni, 1 if < t < +(3)
o0 otherwise, (3)
and
Ni, k(t) Nik (t) ti Nil,kl(t) ti (4)
acts as a weight to the spherical harmonic basis. Here Ni,h(t) is the spline basis of
degree k + 1 with associated knots (t , + 1,...,t,+ 1). Hence the expression for the
modulated spherical harmonics is given by
21+ 1 (1 m)N
M( ) 1 n)! i,4(XJl)Nj,4(x2)P ( ) (0, )q (0, 0), (5)
(47 (1 + mn)!
with , (0, y) and P im as defined before, x (Xi, X2) are the spline control points, i
and j are the basis indices. We have chosen the bicubic spline as it is one of the most
commonly used in literature and more importantly, it provides enough smoothness for
the ABRDF field so that the discontinuities present in the field due to cast shadows are
appropriately approximated (this can be seen in the results presented).
There are three distinct advantages of using our novel bicubic Bspline modulated
spherical harmonics for ABRDF field estimation. First, the builtin smoothness provides
a degree of robustness against noise which is very common when dealing with image
data. Second, it allows us to use neighborhood information while estimating ABRDF at
each pixel location. Finally, it provides us a continuous representation of the spherical
harmonic coefficient field which we will exploit during the ABRDF transfer defined in
the next section.
2.3 ABRDF Field Estimation
If the ABRDF field is available for a face, images of the face under novel illumina
tion directions can be rendered by simply sampling the ABRDF at each location in
8 Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
appropriate directions. But in a realistic setting only few images of a face (sample of
ABRDF field) are given, and hence the problem at hand is of ABRDF field estimation
from these few samples. Motivated by the reasoning outlined in the previous two sub
sections, we propose to use the bicubic BSpline modulated antisymmetric spherical
harmonic functions for this task.
Using S, (0, y), the given data samples (intensity values) in (0, y) direction at loca
tion x, the ABRDF field can be estimated by minimizing the following error function:
IE(w,. ) (0, E x, )) se,4)\\1 (6)
S,0 1, T m=l i,j
where the first term in the summation is the representation of the ABRDF function
using modulated antisymmetric spherical harmonic functions. T is the set of odd natural
numbers and wijim are the unknown coefficients of the ABRDF field that we seek.
Here, the spline control grid is overlayed on data grid (pixels) and the inner summation
on i and j is over the bicubic BSpline basis domain. We minimize this objective
function using the nonlinear conjugate gradient method initialized with a unit vector,
for which the derivative of error function with respect to wijim can be computed in the
analytic form as,
"'m l 1: (=1 E E w r ;ij im (x) (0, S (0,
wijlm ,, 1CT lTm= 1 i,j
N,4(X )N,4(2)T (9, ). (7)
We found that with order 3 and order 5 approximation, this model was able to yield
sufficiently good synthesis results, with order 3 performing slightly better than order 5.
We believe that this is because the order 5 approximation overfits the data. In an order
3 (value of 1) modulated antisymmetric spherical harmonic approximation, values of
the unknown coefficients can be recovered with just 9 images under different illumina
tion conditions. Estimation is better if the given 9 images somewhat uniformly sample
the illumination directions and improves if more images are present. As ABRDF is a
positive function, any negative values produced by the model are set to 0 (as also done
by the max function in the Lambertian model [1]).
We must point out that illumination subspace method [1, 17] and techniques based
on it [3, 4, 2, 13] use spherical harmonics to represent components of the Lambert's law
while we do not even assume the applicability of Lambert's law. We are able to per
form better and capture the nonLambertian features like cast shadows and speculari
ties because we use our bicubic BSpline modulated antisymmetric spherical harmonic
functions to model the ABRDF field.
We now present the results produced by the proposed novel ABRDF estimation
technique with 9 samples. In Fig. 2(a), we present the novel images synthesized from
the learnt ABRDF field which clearly demonstrate that photorealistic images can be
generated by our model. Note the sharpness of the cast shadows in the last row. The
presented technique is capable of both extrapolating and interpolating illumination di
rections from the sample images provided to it (Fig. 2(b)). In Fig. 3 (left) we present
REP2008451: Face Relighting for Recognition
Fig. 6. Image under novel illumination directions synthesized from a single example image. The
illumination direction azimuth varies from 120 to 120 along the X axis (in steps of 20) and
elevation varies from 60 to +60 along the Y axis (in steps of 10).
the estimated ABRDF field overlayed on a face and in Fig. 3 (right), our model can
be seen to capture multiple bumps with varying sharpness to account for shadows and
specularities. Our model's ability to capture cast shadows and specularities in images is
clearly demonstrated in Fig. 4.
3 ABRDF Field Transfer
Here we describe the second part of our novel method, which deals with transferring the
ABRDF field from one face (reference) to another (target) and thus generating images
under various novel illuminations for the target face using just one exemplar image. We
start by making the observation that the basic shapes of features are more or less the
same on all faces (also noted by RiklinRaviv and Shashua in [14]) and thus the optical
artifacts, e.g. cast and attached shadows, created by these features are also similar on
all faces. This means that the nature of the ABRDFs on various faces is also similar and
10 Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
Fig. 7. The first column has the reference faces, the second column has the single example image
of the target face, the next four columns have the images of the target face rendered under four
novel illumination directions (right, top, bottom, left)
hence, we should be able to derive the ABRDF field of the target face using a given
reference ABRDF field.
With this in mind, we proceed by first estimating the nonrigid warping field be
tween the reference and the target face images. This can be formalized as the estima
tion of a nonrigid coordinate transformation T that minimizes the following objective
function,
E2(T)= MIfT(I,,(T(x)), ( .(8)
aXl
Here M is a general matching criterion which depends on the registration method. ,,ef
and Itarget are the reference and target images respectively, while x is the location
on image domain I. As the images to be registered may have different illumination,
techniques which work with brightness constancy assumption (eg. [18], [13]) would
fail here. Hence we propose to use information theoretic match measure (defines M)
based methods as they are insensitive to illumination change. We experimented with
Mutual Information (MI) [19] and CrossCumulative Residual Entropy (CCRE) [20]
based registration and found that CCRE, which is more robust to noise, performs better
than MI. In Fig. 5 we present results produced by MI and CCRE for visual comparison.
Once the deformation field has been recovered, we use it to warp the source image's
ABRDF field coefficients to displace the ABRDFs into appropriate locations for the
target image. In section 2, we mentioned that by using modulated spherical harmonic
functions, we can obtain a continuous representation of the coefficient field, which we
write here explicitly as
wlm(x) = NiX,4(XN,4(X2)Wijlm (9)
ij
The ABRDF field coefficients for the target image l,, (x) can be computed using eq.
9 (this saves us from computing interpolated values again) as im (x) wim(T(x)),
where T is the deformation field recovered by minimization of eq. (8). Using wim(x)
the ABRDF field can be readily computed using the spherical harmonic basis. As can
be noted from Fig. 5, though the locations of the ABRDF have been changed to match
REP2008451: Face Relighting for Recognition
Fig. 8. The 1st column has the reference face, the 2nd column has the single example image of the
target face, the next 4 columns have the images of the target face rendered under 4 novel illumi
nation directions (right, top, bottom, left). The 1st row demonstrate applicability of our technique
to novel poses while the 2nd row demonstrate its robustness while dealing with occlusion.
the target face image, they are still the source image's ABRDF and thus the images
obtained by sampling them would appear like the source image (as in column 3 & 4 of
Fig. 5).
We propose to fix this discrepancy by using a simple, yet very effective intensity
mapping technique. Unlike the existing methods (e.g [21]) which seek a single transfor
mation that maps all intensities from one image to another, we propose to use a separate
transformation for each pixel. Since we have the geometric transformation between the
reference and the target image, we define the intensity mapping quotient Q (x) for each
location x as
Q(x) = Itaet(x)/Iref(T(x)). (10)
Now as images are known to be noisy and the division operation accentuates that
noise, we propose that the image intensity mapping quotient field be smoothed using
a Gaussian kernel G,. Using this, the intensity value at location x of an image of the
target face under novel illumination direction (0, y) can be computed as
1
Itaret (0, 9, x) Wi E (x)f1m (0, ) G, (Q(x)), (11)
cT m=l
where the argument (0, y, x) indicates that the apparent BRDF at location x is being
queried in direction (0, 9).
Before presenting the experimental results we must point out that this intensity map
ping quotient (eq. 10) is not the same as the Quotient image proposed by RiklinRaviv
and Shashua [14] as they make explicit Lambertian assumption and define their quotient
image to be ratio of the albedos which is clearly not the case here.
4 Experiments and Results
In Fig. 6 we present a set of images generated under novel illumination conditions of
the target face (2nd row 2nd column in Fig. 5) using just 1 image. It can be noted that
the specularity of the nose tip and cast shadows have been captured to produce photo
realistic results. Next, in Fig. 7 we present novel images of the same subject using three
different reference faces. Discounting minor artifacts, it can be noted that these images
12 Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
40 T d with Rf
35 Transferred with Ref 9
30 Estimated from 64 example s
S25
S20
E 15
10
5
0
(1,2) 3 4 All
Database subsets (Lighting gets harsher from subset 1 to 4)
Fig. 9. The dark bar represents error when the ABRDF field is transferred using a reference
ABRDF field generated using 9 image, the gray bar is for the case when the reference field was
generated using 64 images and the white bar represents the error in the images generated by the
estimation method presented in Sec. 2 with all 64 images (this can be looked as a baseline case
for the other two bars).
are perceptually similar. This shows that most race matched images can act as a good
reference.
In the next set of experiments we demonstrate the robustness and versatility of our
technique. First, we demonstrate that we can produce good results even when parts of
the face in the target image are occluded (Figure 8). This is accomplished by setting
the intensity mapping quotient to unity and performing a histogram equalization in the
occluded regions. The results show that our framework can handle larger occlusion than
what was demonstrated recently by Wang et al. [22]. Second, even though we do not
use any 3D information, our technique is capable of generating photorealistic images
of faces in poses different from that of the reference face under novel illumination direc
tions. At this stage, our framework can handle poses that differ up to 12 In Fig. 8 we
look at the quantitative error introduced by our method as a function of the number of
images used for the ABRDF field estimation. We compare the synthesized novel images
to the ground truth images present in the Extended Yale B database. We observe that
the quantitative error increases with the harshness of illumination direction which we
attribute to the lack of accurate texture information for extreme illumination directions.
Finally we present two sets of results for the application of the proposed techniques
to face recognition. First, using a simple Nearest Neighbor classifier we compare the
results of ourABRDF estimation technique using 9 sample images with those of existing
techniques which use multiple (from 4 to 9) images (Table 1). For this experiment
we assume that 9 gallery images with known illumination directions per person are
available (from subset 1 and 2). We compute the ABRDF field of each face using the
technique described in Section 2, generate a number of images under novel illumination
directions (defined on a grid) and then use all of them in our Nearest Neighbor classifier
as gallery images. The results demonstrate that our technique can produce competitive
results even when used with a naive classifier like Nearest Neighbor. To make the results
comparable to the competing methods we used the 10 subjects from the Yale B face
database. Results were averaged over 5 independent runs of the recognition algorithm.
REP2008451: Face Relighting for Recognition
Method IError Rate (%) in Subsets
(1,2) (3) (4)
Correlation 0.0 23.3 73.6
Eigenfaces 0.0 25.8 75.7
Linear Subspaces 0.0 0.0 15.0
Illum. Cones Attached 0.0 0.0 8.6
9 Points of Light (9PL) 0.0 0.0 2.8
Illum. Cones Cast 0.0 0.0 0.0
Harmonic Images Cast 0.0 0.0 2.7
Our Method (Sec. 2) 0.0 0.0 4.1
Table 1. Recognition results on Yale B Face Database of various techniques reported in literature.
All results except ours were summarized from [2]
Our second set of experiments demonstrate how the ABRDF transfer technique,
which works with a single image, can be used to enhance various existing benchmark
face recognition techniques (Table 2). For this, we make use of the fact that the perfor
mance of most recognition systems is improved when a better training set is present.
We present results for Nearest Neighbor (NN), Eigenfaces and Fisherfaces, where we
assume that only a single nearfrontal illumination image of each subject is available
in the gallery set. For ABRDF+NN, ABRDF+Eigenfaces and ABRDF+Fisherfaces, we
use this single image to generate more images (using the ABRDF transfer technique de
scribed in Section 3) and then use all of them to train the classifiers. Experiments were
carried out using 10 randomly selected subjects from the Extended Yale B Database.
Results were computed using 3 different reference faces (other than the 10 selected sub
jects) over 5 independent runs each of the recognition algorithms and then averaged.
5 Conclusion
We introduced a novel bicubic BSpline modulated spherical harmonics based frame
work which can determine the ABRDF field of a face using a single 2D image and
can capture specularities and cast shadows without using any 3D information or man
ual intervention. The background knowledge required by our framework can be de
rived from as few as 9 images as opposed to more extensive requirements of existing
methods. Through various experiments we showed that the synthesis of images under
novel illumination directions results in photorealistic rendering. Further, our two sets
of recognition experiments demonstrated that with 9 sample images we can produce
results comparable to the state of the art, and furthermore, with just 1 image, we can
enhance recognition rates of other recognition techniques.
References
1. Basri, R., Jacobs, D.: Lambertian reflectance and linear subspaces. PAMI 25 (2003) 218233
2. Lee, K., Ho, J., Kriegman, D.J.: Acquiring linear subspaces for face recognition under vari
able lighting. PAMI 27 (2005) 684698
3. Zhang, L., Samaras, D.: Face recognition under variable lighting using harmonic image
exemplars. In: CVPR. (2003) 1925
14 Ritwik Kumar, Angelos Barmpoutis, Arunava Banerjee, Baba C. Vemuri
Method Error Rate (%) in Subsets
(1,2) (3) (4) Total
NN 8.3 72.5 85.0 49.3
ABRDF + NN 2.2 19.2 51.4 26.4
Eigenfaces 6.8 74.2 85.8 49.3
ABRDF+Eigenfaces 3.2 15.8 52.4 21.8
Fisherfaces 6.8 74.2 88.6 50.2
ABRDF+Fisherfaces 1.1 11.7 67.1 24.4
Table 2. Recognition results of various benchmark methods on the Extended Yale Face Database.
4. Zhang, L., Wang, S., Samaras, D.: Face synthesis and recognition from a single image under
arbitrary unknown lighting using a spherical harmonic basis morphable model. In: CVPR.
(2005) 209216
5. Adini, Y, Moses, Y, Ullman, S.: Face recognition: The problem of compensating for
changes in illumination direction. PAMI 19 (1997) 721732
6. Georghiades, A.S., Belhumeur, PN., Kriegman, D.J.: From few to many: Illumination cone
models for face recognition under variable lighting and pose. PAMI 23 (2001) 643660
7. Sim, T., Kanade, T.: Combining models and exemplars for face recognition: An illuminating
example. In: CVPR Workshop on Models versus Exemplars in Comp. Vision 2001. (2001)
8. Zhao, W.Y, Chellappa, R.: Illuminationinsensitive face recognition using symmetric shape
fromshading. In: CVPR. (2000) 286293
9. Debevec, P, Hawkins, T., Tchou, C., Duiker, H., Sarokin, W., Sagar, M.: Acquiring the
reflectance field of a human face. SIGGRAPH (2000) 145156
10. Peers, P., Tamura, N., Matusik, W., Debevec, P.: Postproduction facial performance relight
ing using reflectance transfer. SIGGRAPH (2007) 52
11. Wen, Z., Liu, Z., Huang, T.S.: Face relighting with radiance environment maps. In: CVPR.
(2003) 158165
12. Lee, J., Moghaddam, B., Pfister, H., Machiraju, R.: A bilinear illumination model for robust
face recognition. In: ICCV. (2005) 11771184
13. Blanz, V, Vetter, T.: Face recognition based on fitting a 3d morphable model. PAMI 25
(2003) 10631074
14. Shashua, A., RiklinRaviv, T.: The quotient image: Classbased rerendering and recognition
with varying illuminations. PAMI 23 (2001) 129139
15. Wang, H., Li, S., Wang, Y, Zhang, J.: Self quotient image for face recognition. In: ICIP.
(2004) II: 13971400
16. Stoschek, A.: Imagebased rerendering of faces for continuous pose and illumination direc
tions. In: CVPR. (2000) I: 582587
17. Ramamoorthi, R., Hanrahan, P.: The relationship between radiance and irradiance: Deter
mining the illumination from images of a convex lambertian object". JoSA A 18 (2001)
24482459
18. Szeliski, R., Coughlan, J.: Hierarchical splinebased image registration. In: CVPR. (1994)
194201
19. Viola, P, William M. Wells, I.: Alignment by maximization of mutual information. IJCV 24
(1997) 137154
20. Wang, F., Vemuri, B.C.: Nonrigid multimodal image registration using crosscumulative
residual entropy. IJCV 74 (2007) 201205
21. Guimond, A., Roche, A., Ayache, N., Meunier, J.: Multimodal brain warping using the
demons algorithm and adaptative intensity corrections. IEEE TMI 20 (2001) 5869
22. Wang, Y, Liu, Z., Hua, G., Wen, Z., Zhang, Z.Y, Samaras, D.: Face relighting from a single
image under harsh lighting conditions. In: CVPR. (2007) 18
