Face Recognition in Art Images

Portraits are subject to several complexities such as aesthetic sensibilities of the artist or social standing of the sitter. Moreover, the number of samples available to model these effects is often limited. For robust automated face recognition, it also becomes important to model the characteristics of the artist. From a set of portraiture where the identities of subjects is known, we derive appropriate features that are based on domain knowledge of artistic renderings and learn statistical models for the distributions of the match and non-match scores, which we refer to as the portrait feature space (PFS). The features considered include well-known facial recognition attributes like local features and anthropometric distances. Thereafter, we learn which of the chosen features were emphasized in various works involving (a) same artist depicting same sitter, (b) same sitter but by different artists, (c) same artist but depicting different sitters, and we show that the knowledge of these specific choices can provide valuable information regarding the sitter and/or artist. Further, we use the learned PFS on a number of cases that have been open questions to art historians. They are usually in the form of validating two portraits as belonging to the same person. Using statistical hypothesis tests on the PFS, we provide quantitative measures of similarity for each of these questions. It is, to the best of our knowledge, the first study that applies automated face recognition technologies to the analysis of portraits of multiple subjects in various forms - paintings, death masks, sculptures.

• Sample Publications

  • Deep Learning Based Identity Verification in Renaissance Portraits.

    A. Gupta, N. C. Mithun, C. Rudolph and A. Roy-Chowdhury, IEEE International Conference on Multimedia and Expo (ICME) 2018.

  • Computerized Face Recognition in Renaissance Portrait Art

    R. Srinivasan, C. Rudolph, and A. K. Roy-Chowdhury, Signal Processing Magazine. 2015.

    Supplmental Material

  • Recognizing the Royals - Leveraging Computerized Face Recognition for Identifying Subjects in Ancient Artworks

    R. Srinivasan, A. Roy-Chowdhury, C. Rudolph, J. Kohl, ACM Intl. Conf. on Multimedia, 2013.

  • Quantitative Modeling of Artists Styles in Renaissance Face Portraiture

    R. Srinivasan, A. Roy-Chowdhury, C.Rudolph, J. Kohl, 2nd International Workshop on Historical Document Imaging and Processing, 2013.

Face Tracking and Recognition in Video

In this project, we have studied the problem of robust face tracking and recognition in the presence of large pose and illumination changes. First, we developed a theoretical model for understanding the effects of lighting, motion and shape in the process of image formation, bulding upon multilinear tensor algebra. This has led to the development of robust tracking of facial features and an analysis-by-synthesis framework for face recognition from video sequences.

• Sample Publications

  • A Physics-Based Analysis of Image Appearance Models

    Y. Xu, A. Roy-Chowdhury, IEEE Trans. on Pattern Analysis and Machine Intelligence, August 2011.

    Supplemental Material

  • A Theoretical Analysis of Linear and Multi-linear Models of Image Appearance

    Y. Xu and A. Roy-Chowdhury. IEEE Conf. on Computer Vision and Pattern Recognition, 2008.

  • Learning A Geometry-Integrated Image Appearance Manifold From A Small Training Set

    Y. Xu and A. Roy-Chowdhury. IEEE Conf. on Computer Vision and Pattern Recognition, 2008.

  • Inverse Compositional Estimation of 3D Pose And Lighting in Dynamic Scenes

    Y. Xu and A. Roy-Chowdhury, IEEE Trans. on Pattern Analysis and Machine Intelligence, July 2008.

  • Integrating Illumination, Motion and Shape Models for Robust Face Recognition In Video

    Y.Xu, A. Roy-Chowdhury, and K. Patel. EURASIP Journal on Advances in Signal Processing: Advanced Signal Processing and Pattern Recognition Methods for Biometrics, 2008.

  • Integrating Motion, Illumination and Structure in Video Sequences, With Applications in Illumination-Invariant Tracking

    Y. Xu and A. Roy-Chowdhury. IEEE Trans. on Pattern Analysis and Machine Intelligence, May 2007.

  • Integrating the Effects of Motion, Illumination and Structure in Video Sequences

    Y. Xu and A. Roy-Chowdhury, IEEE Intl. Conf. on Computer Vision, 2005.

  • Super-resolved Facial Texture Under Changing Pose and Illumination

    J. Yu, B. Bhanu, Y. Xu, A. Roy-Chowdhury, IEEE Intl. Conf. on Image Processing, 2007.