IntPhys 2019 Challenge
----------------------

The challenge 2019 edition is hosted on `codabench`_. It contains training dev and
test data for the three blocks (O1, O2 and O3). **Ground truth for the test
dataset is not provided** and is kept secret for evaluation.


.. important::

   The codalab URL of the challenge is
   https://www.codabench.org/competitions/3075/.


To participate to the challenge, follow those steps:

* Register on `codabench`_,

* Download the **test dataset** and the **starter kit** from
  :ref:`download`. If you want to train your model on our data,
  download the train dataset as well.
  A dev set, structured as the test dataset with ground truth labels,
  is also available.

* Extract the ``test.*.tar.gz`` archives, it will produce the
  directories tree ``test/{O1, O2, O3}`` (i.e. each block in a
  sub-directory).

* In the starter kit you will find the file ``task.txt``, listing all
  the movies of the test set, for which you must compute a
  plausibility score. In the task file, the paths to movies are
  expressed as paths relative to the ``test`` directory. For example:

  .. code-block:: none

     O1/0001/1
     O1/0001/2
     O1/0001/3
     O1/0001/4
     O1/0002/1
     ...
     O3/1080/3
     O3/1080/4


* **Submission format**

  * Participants must submit one plausibility score :math:`P(x)` per
    movie, in a file named ``answer.txt``, bundled in a ``.zip`` archive.

  * Each score :math:`P(x)` is a probability, so we must have :math:`0
    \leq P(x) \leq 1` for all :math:`x`.

  * The ``answer.txt`` file must have the following format: each line
    contains the movie path (as provided by the ``task.txt`` file)
    along with the plausibility score you computed, in the format
    ``<movie-path> <score>``. For example:

    .. code-block:: none

       O1/0001/1 0.9751
       O1/0001/2 0.0614
       O1/0001/3 0.0397
       O1/0001/4 0.0874
       O1/0002/1 0.8663
       ...
       O3/1080/3 0.1986
       O3/1080/4 0.5458

    An example submission is provided for each task in the challenge's
    starter kit.

  * Once your submission zip file is ready, you can use the script
    ``validate.py`` from the starter kit to confirm your file is in the
    expected format and will not be rejected.

    Exemple of validation output:

    .. code-block:: none

       validating ../build/challenge/submission_example.zip ...
       check zip extension ...
       check valid zip format ...
       check answer.txt is in zip ...
       check answer.txt is the only file in zip ...
       check structure of answer.txt ...
       check entries are valid for in answer.txt ...
       check answers are in [0, 1] in answer.txt ...
       submission is valid, ready to be submitted!

  * Once your submission archive is valid, go to
    https://www.codabench.org/competitions/3075/
    and submit it to codabench.

* Each result submitted by a participant is evaluated on a codabench server and
  the detailed score is available to the participant. A public leaderboard will
  be frequently updated on this webpage, participant who don’t want to appear in
  this leaderboard should email ``contact@cognitive-ml.fr``.


Evaluation
~~~~~~~~~~

* **evaluation metric**: absolute and relative error rates are
  detailed in the :ref:`evaluation` page.

* **computed scores**: both the relative and absolute error rate are
  computed for each movie and the **average score for each block** is
  derived as the the final score. We distinguish 3 conditions:
  *occluded*, *visible* and *all* (i.e. mixing occluded and visible
  movies).

* The evaluation program ``score.py`` is provided in the starter kit.


.. _rules:

Terms and rules
~~~~~~~~~~~~~~~

* **General terms:** This challenge is governed by the `CodaBench Terms
  and Conditions <https://github.com/codalab/codalab-competitions/wiki/Privacy>`_.

* **Registration:** The participants must register to Codabench and
  provide a valid email address. Participation is open to any
  individual, research team or company which agrees the present rules.

* **Submissions:** Participants are limited to one submission per day
  and per task, up to a maximum of 100 submissions per tasks.

* **Citation:** Participants of this challenge should quote the following paper:
  `R. Riochet, M. Ynocente Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard,
  & E. Dupoux (2018). IntPhys: A Benchmark for Visual Intuitive Physics
  Reasoning, ArXiv 1803.07616 <https://arxiv.org/abs/1803.07616>`_

* **Contact:** For any question/issue relevant to all participants,
  please use the `challenge's forum
  <https://competitions.codalab.org/forums/17493/>`_. Else you can
  contact ``contact@cognitive-ml.fr``


Credits
~~~~~~~

This Challenge is organized by the `CoML team <https://cognitive-ml.fr>`_
(EHESS - ENS - CNRS - INRIA). It was funded by the European Research Council
(ERC-2011-AdG-295810 BOOTPHON), the Agence Nationale pour la Recherche
(ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL* ), and a grant from Facebook AI
Research.

.. figure:: ../../data/codalab-logo-fullcolor-positive.png
   :align: left
   :width: 6em
   :alt: Codalab
   :target: https://www.codabench.org/

This challenge is hosted on `codalab`_, an open-source web-based platform for
machine learning competitions.


Contributors:

- Organization: M. Bernard, R. Riochet, E. Dupoux.
- Design of the Blocks: E. Dupoux, R. Riochet, V. Izard.
- Datasets preparation (Unreal Engine/Python): M. Bernard, M. Ynocente
  Castro, E. Simon, M. Métais, V. Daul.
- Codalab/Website: M. Bernard, R. Riochet.
- Human Data: R. Riochet

.. _codabench: https://www.codabench.org