Welcome to the Unconstrained Ear Recognition Challenge!
The 1st Unconstrained Ear Recognition Challenge (UERC) that was organized in the scope of the IEEE/IAPR International Joint Conference on Biometrics 2017. The goal of the challenge was to advance the state-of- technology in the field of automatic ear recognition, to provide participants with a challenging research problem and introduce a benchmark dataset and protocol for assessing the latest techniques, models, and algorithms related to automatic ear recognition.
Despite the numerous application possibilities in security, surveillance applications, forensics, criminal investigations or border control, the existing research in ear recognition has seldom gone beyond laboratory settings. This can mostly be attributed to the enormous appearance variability of ear images when captured in unconstrained settings. However, due to recent advances in computer vision, machine learning and artificial intelligence (e.g. with deep learning), many recognition problems are now solvable (to at least some extent) in unconstrained settings and many modalities that were too complex to use in real-life situations are now becoming a viable source of data for person recognition.
The Unconstrained Ear Recognition Challenge (UERC) built on the advances outlined above and address the problem of ear recognition “in the wild”. While many competitions and challenges were organized in the scope of past biometrics-oriented conferences (ICB, BTAS, FG, etc.) for various biometric modalities and numerous problems, ear recognition has not yet been studied within group evaluations making the UERC a unique opportunity with high potential impact.
The challenge was held on an extended version of the Annotated Web Ears (AWE)dataset, containing a total of 9,500 ear images. The images were collected with a semi-automatic procedure that involved web-crawlers and a subsequent manual inspection. Because the AWE images were not gathered in controlled laboratory-like conditions, they better represent the variability in ear appearance than existing datasets of ear images. However, the problem of automatic ear recognition is also significantly harder. A few example images from the extended AWE dataset are shown below.
A more in depth description of the images, acquisition procedure, dataset characteristics and other information on the AWE dataset is available in the Neurocomputing paper.
UERC used three image datasets:
The 3,300 images of the main dataset contain various annotations, such as the level of occlusion, rotation (yaw, roll and pitch angles), presence of accessories, gender and side. This information was also made available during training and can be exploited to build specialized recognition techniques.
The 3,300 images of the main part of the dataset was split into a training set of 1,500 images (belonging to 150 subjects) and a test set of 1,800 images (belonging to 180 subjects). The identities in the training and test set were disjoint. The purpose of the training set is to train recognition models and set any open hyper-parameters, while the test set is reserved for the final evaluation. The test set MUST NOT have been used to learn or fine-tune any parameters of the recognition model. The organizers reserved the right to exclude a team from the competition (and consequently the jointly authored IJCB conference paper) if the final result analysis suggested that the test images were also used for training.
The train and test sets were split:
UERC tested the recognition performance of all submitted algorithms through identification experiments. The participants had to generate a similarity score matrix with comparisons of each image in the probe.txt file to each image in the gallery.txt file and returned the resulting similarity matrix to the organizers for scoring. Thus, each participant was required to generate a 7442x9500 matrix for each submitted system.
The number of approaches that each participant was allowed to submit was not limited. However, only approaches with a least a short description (written by the participants) were included in the IJCB summary paper of UERC 2017. Submissions were possible via a simple web interface.
The submitted similarity matrices were scored by the organizers. Rank-1 recognition rates, complete CMC curves and the Area-under- the CMC (AUC) curve were computed and reported for each submitted algorithm. The AUC was used to rank the participating algorithms. Based on the annotations available with the extended AWE dataset, we also computed results for sub-sets of the data, e.g. focusing only on certain ranges of head rotations, presence/absence of accessories, etc.
We provided all participants with a Matlab starter kit that generated a matrix of comparisons for each sample in the dataset and computed all relevant performance metrics. The starter kit helped participants to quickly start with the research work and generate score matrices compliant with our scoring procedures. For the baseline approaches we made a number of descriptor-based approaches (using SIFTs,POEMs, LBPs, etc.) available to the participants as well as a deep learning approach.
If you have any questions, suggestions or would like to participate in the competition, feel free to contact email@example.com.