MeDA: Medical Domain Adaptation Classification Challenge

Mexican International Conference on Artificial Intelligence (MICAI)
Tuesday November 14th,  2023, Mérida Yucatán (Hybrid Event)
http://www.micai.org/2023/  




Organizers

  • Dr. Gilberto Ochoa-Ruiz, CV-inside Lab Leader, Advanced AI Group, ITESM
  • Dr. Andres Mendez-Vazquez, CINVESTAV
  • Dr. Gerardo Rodriguez-Hernandez, ITESM
  • Msc. Ivan Reyes-Amezcua, CINVESTAV
  • Msc. Jorge Gonzalez-Zapata, CINVESTAV

Goal: 

Data limitations often hinder the use of advanced machine learning methods in various medical fields, except for a select few large public datasets. Domain Adaptation Classification offers the capability to draw from similarities among diverse medical imaging datasets, allowing for more efficient learning of new tasks. Yet, the potential of Domain Adaptation Classification in medical imaging is not fully explored. 

With the "MeDA" challenge, we aim to inspire the medical imaging and machine learning sectors to delve deeper into its capabilities in the medical imaging realm, and to craft algorithms capable of managing the significant variability in tasks and data within this sector.


What is the purpose of the challenge?

Participants are tasked with creating an algorithm using the provided MedMNIST dataset that can efficiently learn new tasks. The algorithm's effectiveness is then assessed based on its capability to perform well on a private and previously unknown test task sourced from undisclosed datasets. Link to the dataset: https://medmnist.com/  


Challenge Training Dataset

MedMNIST v2 is a comprehensive collection of standardized biomedical images resembling the MNIST format. It is composed of 12 datasets (2D images), all pre-processed to 28 x 28 (2D) or 28 x 28 x 28 (3D) dimensions, eliminating the need for prior knowledge by users. With a diverse range of tasks and scales, it offers 708,069 2D and 9,998 3D images, covering major biomedical imaging modalities. This dataset can aid in various research and educational projects in biomedical image analysis, computer vision, and machine learning. Several methods, including neural networks and AutoML tools, were benchmarked on MedMNIST v2.


How does the challenge work? 

Challenge general overview: 


Challenge Mechanics

  • We ask participants to make use of the meta-train dataset (MedMNIST v2), which is essentially a compilation of various datasets, encompassing a broad spectrum of imaging modalities such as binary, multi-class, or multi-label classification. 

  • Teams that participate in the MeDA challenge will utilize this data to train a cross-domain learning algorithm. Its effectiveness will be locally gauged by evaluating its capacity to perform well on a left-out classification task using a limited N number of samples. 

  • Potential algorithmic strategies can be inspired by techniques like meta-learning, transfer learning, or self-supervised pre-training.


Testing and ranking

  • Teams must submit a trained model (exported in ONNX format), and this model must be trained only on the provided dataset (MedMNIST v2).

  • The organizers will test the model using a private, unseen 3 testing datasets targeting different  endoscopic classification tasks. Each dataset has been processed so it contains 6 classes. The application area is not revealed to avoid giving an unfair advantage through information leakage.

Metrics used to evaluate the performance of the models are the average of accuracy and F1 score across the 3 testing datasets. Teams will be ranked based on these results.

  •  


Submission procedure

  • A CMT link is provided for the participant teams. The participants should create a team name and do the submission using an anonymous email (i.e, team-name@domain.com ) to avoid any bias in the challenge process. The submission should include the paper and link to code/model (ONNX model). 

   Note: please avoid any institutional or country-based references in the name of your team


    CMT submission site: Track: MeDA Challenge 2023


https://cmt3.research.microsoft.com/User/Login?ReturnUrl=/MICAI2023/


Challenge Timeline

Submission portal opens: 20 september 2023
Registration deadline: 20 october 2023 (create a sumision in CMT)
Submission deadline: 30 october 2023 (final submission in CMT)
Top-performing teams contacted: 7 november 2023
Results announced at MICAI 2023: 14 november 2023

 

Conference Participation Dynamics

  • Teams must send an 4-page paper abstract in IEEE standard format showing their procedure, explaining their method and showing results.

  • The best ranking teams will do a short presentation of their work during the MICAI conference on Tuesday November 14th, 2023

  • The mode of participation will be HYBRID (teams can join via zoom if is not possible for them to attend) 


Prizes

Prizes will be announced during the competition in Merida. The top three methods will receive a full registration grant for MICAI 2024 to publish an extended version of their paper.


Rules

  1. Participating teams may consist of one or more individuals (max 5 individuals). Each individual may only be part of one team. The creation of multiple accounts to circumvent this rule is not permitted.

  2. All valid results will be announced publicly through the leaderboard. The teams of the three top-performing methods will be invited to publish their work in a special issue.

  3. We aim to publish an analysis of the challenge results. The three top-performing teams will be invited to contribute to this publication. Moreover, the challenge organizers may invite additional teams with particularly novel/interesting algorithms to contribute.

  4. Participating teams are free to publish their own algorithms and results separately but may only reference the overall challenge results once the challenge paper is published.

  5. The participants may use the data we provide, i.e. our meta-dataset for training. Additionally, a set of publicly available and commonly used computer vision datasets, may be used, specifically:

    1. ImageNet (ILSVRC 2012), miniImageNet, tieredImageNet

    2. CIFAR 100, CIFAR-FS

    3. MSCOCO

    4. Omniglot

  6. The use of openly available pre-trained neural networks trained exclusively on the above datasets is also permitted. Any pre-trained networks used, as well as the data used for training must be reported in the post-submission report.

  7. The test data will contain previously unseen (not contained in the meta-training data) data scarce scenarios that bear resemblance to the meta-training data, but is distinct from them.

  8. Participating teams are expected to make their methods fully reproducible. This includes the availability of code, any additional data used, as well as instructions on how to replicate the results.


Contact

For further details and questions please contact the organizers

Acknowledgements

This challenge is organized in the context of the  project 322537“ML-INSIDE: Novel Machine Learning Methods  for Image aNalysiS & bIomeDical Engineering in Endoscopy” funded by CONAHCYT through  the SEP-CONAHCYT-ANUIES-ECOS NORD Francia program.


 

Organizing Institutions

Sociedad Mexicana de Inteligencia Artificial iimas iingen Universidad Autónoma de Yucatán - UADY Instituto de Química de la UNAM TecNM/IT de Merida