-
The Unconstrained Ear Recognition Challenge 2023 - Maximizing Performance and Minimizing Bias (UERC 23)
The 3rd Unconstrained Ear Recognition Challenge (UERC) is organized in the scope of the International Joint Conference on Biometrics 2023. The goal of the challenge is to further advance the state-of-technology in the field of automatic ear recognition, to provide participants with a challenging research problem and introduce a benchmark dataset and protocol for assessing the latest techniques, models, and algorithms related to automatic ear recognition.The results of UERC 2023 will be published in the IJCB conference paper authored jointly by all participants of the challenge.Interim Rankings from May 20th
Model UERC Ranking (lower is better) EarRecV8 16.66% EarRecV8STFI 17.25% KU-EAR 17.86% AIID.ear 18.14% ViTEar 18.15% EEar 20.30% Baseline 29.87% RecogEAR 40.01% For Participants
UERC submissions are open! Submit your approach here.General Info
UERC 2023 is organized as a two-track competition, where each track is focused on one specific goal. Participants are free to enter only a single track or compete in both. For each track, a dataset, evaluation tool written in Python, and baseline models in Python are made available. A detailed description of the two tracks is given below:- Track 1: Fair Ear Recognition. The first UERC 2023 evaluation track collects ear recognition models and scores their performance on ear images captured in unconstrained environments. Here, the performance indicators include both a measure of recognition performance as well as an estimate of the exhibited demographic bias. Both recognition and bias scores contribute to the overall ranking. Participants are free to develop any type of model to maximize performance while minimizing bias. The final submission for this track includes a working solution (source code or compiled binary), which the organizers run to evaluate the performance on a sequestered test data.
- Track 2: Bias Mitigation. The second UERC 2023 track addresses bias mitigation strategies explicitly. Here, a baseline ResNet model (written in Python) is made available to the participants, and the goal is to design bias mitigation schemes that reduce the initial bias of the models without adversely affecting performance. Such schemes may include additional model blocks and network components, normalization layers, knowledge infusion mechanisms, score normalization procedures, image preprocessing approaches, and any other solution capable of reducing bias of the predefined base model. Similarly to the first track, participants have to submit a working solution that the organizers evaluate on the sequestered test data.
Data, Training and Model Development
For the training and model development two main sources of data are available to the UERC 2023 participants:- A novel dataset of around 14,000 newly collected images as well as the training and testing images from UERC 2017 and 2019, and
- over 234,000 images of around 600 subjects from the VGGFace-Ear dataset.
The training set is available for model training and hyperparameter selection. Additionally, part of the training data is split from the training set to allow the participant to estimate their performance (in terms of verification errors as well as bias) during model development.The testing data is sequestered and not made available to the participants. The sequestered test data consists of six groups with all ethnicity-gender combinations from the following base categories:- Ethnicities: Asian, Black and White
- Gender categories: Female and Male
Each ethnicity-gender group in the sequestered test data consists of 10 subjects and approximately 250 images, resulting in a total of 1500 images used for testing and ranking of the submitted approaches.All images used for UERC 2023 are annotated with the three attributes (including the images from VGGFace-Ear dataset):- gender: female (f), male (m),
- ethnicity: caucasian (1), asian (2), south asian (3), black (4), middle eastern (5), hispanic (6), other,
head side: left (l), right (r).
Visually, the images used for UERC 2023 look similar to the images used for UERC 2017 and 2019.Organizers
- Asst. Prof. Žiga Emeršič
University of Ljubljana, Faculty of Computer and Information Science, Slovenia, EU - Prof. Hazım Kemal Ekenel
Istanbul Technical University, Department of Computer Engineering, Turkey - Prof. Guillermo Camara-Chavez
Federal University of Ouro Preto, Brazil - Prof. Peter Peer
University of Ljubljana, Faculty of Computer and Information Science, Slovenia, EU - Prof. Vitomir Štruc
University of Ljubljana, Faculty of Electrical Engineering, Slovenia, EU
Important Dates
- April 18th: Sequestered data available.
- April 21th: First interim ranking.
May 1stMay 23rd: Registration closes, end of the competition.May 15thJune 5th: Summary paper submission.
If you have any questions, suggestions or would like to participate in the future competitions, feel free to contact ziga.emersic@fri.uni-lj.si.
CMC curves of the submitted approaches generated on the testing split of the AWEx dataset (shown in log-arithmic scale). The legend on the right is sorted accordingto the rank-1 performance.
CMC curves of the submitted approaches generated on the complete UERC test dataset involving 3,540 subjects (shown in logarithmic scale). The legend is sortedaccording to the rank-1 scores. U17 denotes approachesfrom UERC 2017.
CMC curves of the scalability experiments on the entire UERC dataset. The rank of the experiments is plotted on a logarithmic scale to better highlight the performance at the lower ranks.
