Training Set: learning by observation
Note
The train dataset is available at the Download and resources page.
This describes a training set for training models in an unsupervised fashion from observation only. The video clips are cast in the same environment as the test set (similar objects, occluders, motions, etc), but with completely random events, camera positions, etc. Obviously, it only contains possible events. To help with model training, we do however provide additional metadata about object ground truth positions, depth, etc.
Examples
Here are 4 examples extracted from the train dataset:
Depth and masks
Each video comes with its associated depth field and objects masking:
Depth
The depth field of a scene is encoded as a 16 bits grayscale image, from white (close to the camera) to black (far from the camera). For a given pixel value p, the encoded distance d is expressed in millimeters as d = 2^{16}-1-p. The quantification resolution is one millimeter.
Masks
The objects masking is encoded a 8 bits grayscale image. It provides a per-frame based object segmentation, with the object identifiers randomly changing at each frame. The fact that the masks are changing on each frame is done on purpose. Indeed, one of the problem to solve is precisely how to link the object masks across frames and, with that information, the object permanance problem is almost solved.
For the train scenes, the provided metadata details the masks gray value for each frame. For instance:
{ "masks": { "floor": 61, "object_1": 48, "occluder_1": 14, "occluder_2": 194, "sky": 193 } }
Metadata
Note
The status is provided for train but not for dev nor test datasets.
A detailed status in JSON format is given for each train video. It records all the parameters of the scene (objects location is given in centimeters, rotation is in degrees).
Example of json file (only 2 frames for illustration purposes, real json files contain 100 frames):
{
"header": {
"block_name": "train",
"block_type": "train",
"is_possible": true,
"floor": {
"location": {
"x": -200.0,
"y": -4000.0,
"z": 0.0
},
"rotation": {
"roll": 0.0,
"pitch": 0.0,
"yaw": 0.0
},
"scale": {
"x": 10.0,
"y": 20.0,
"z": 1.0
},
"friction": 0.5,
"restitution": 0.5,
"material": "M_Floor_01"
},
"light": {
"type": "SkyLight"
},
"light_1": {
"type": "SkyLight"
},
"camera": {
"location": {
"x": 0.0,
"y": 0.0,
"z": 180.0422821044922
},
"rotation": {
"roll": 0.0,
"pitch": 8.506651878356934,
"yaw": -4.223939895629883
},
"field_of_view": 90.0,
"aspect_ratio": 1.0,
"projection_mode": 0
}
},
"frames": [
{
"occluder_1": {
"location": {
"x": 405.12677001953125,
"y": -80.42643737792969,
"z": 0.0
},
"rotation": {
"roll": 89.99999237060547,
"pitch": 0.0,
"yaw": 28.398197174072266
},
"scale": {
"x": 0.8100548982620239,
"y": 1.0,
"z": 1.2185389995574951
},
"friction": 0.5,
"restitution": 0.5,
"material": "M_Brick_Cut_Stone",
"speed": 1.1377057419424133,
"moves": [
112,
180
]
},
"occluder_2": {
"location": {
"x": 681.0262451171875,
"y": 369.7522888183594,
"z": 0.0
},
"rotation": {
"roll": 0.0,
"pitch": 0.0,
"yaw": -83.7597885131836
},
"scale": {
"x": 0.8828107714653015,
"y": 1.0,
"z": 1.0996980667114258
},
"friction": 0.5,
"restitution": 0.5,
"material": "M_Bricks_4",
"speed": 4.447713113166794,
"moves": [
186
]
},
"object_1": {
"location": {
"x": 762.6484375,
"y": 48.82008361816406,
"z": 42.0124626159668
},
"rotation": {
"roll": 0.0,
"pitch": 0.0,
"yaw": 113.01049041748047
},
"scale": {
"x": 0.840249240398407,
"y": 0.840249240398407,
"z": 0.840249240398407
},
"friction": 0.5,
"restitution": 0.5,
"material": "Yellow",
"mass": 73.9890365600586,
"initial_force": {
"x": -8527.78515625,
"y": 4952.4140625,
"z": 14996.8095703125
},
"velocity": {
"x": 0.0,
"y": 0.0,
"z": 0.0
},
"shape": "Sphere"
},
"object_2": {
"location": {
"x": 476.0985412597656,
"y": -689.1371459960938,
"z": 56.96540069580078
},
"rotation": {
"roll": 0.0,
"pitch": 0.0,
"yaw": -55.98881530761719
},
"scale": {
"x": 1.139307975769043,
"y": 1.139307975769043,
"z": 1.139307975769043
},
"friction": 0.5,
"restitution": 0.5,
"material": "GreenMaterial",
"mass": 146.7883758544922,
"initial_force": {
"x": 8689.44140625,
"y": 47984.6484375,
"z": 3588.540283203125
},
"velocity": {
"x": 0.0,
"y": 0.0,
"z": 0.0
},
"shape": "Sphere"
},
"masks": {
"floor": 61,
"object_1": 48,
"occluder_1": 14,
"occluder_2": 194,
"sky": 193
}
},
{
"occluder_1": {
"location": {
"x": 405.12677001953125,
"y": -80.42643737792969,
"z": 0.0
},
"rotation": {
"roll": 89.99999237060547,
"pitch": 0.0,
"yaw": 28.398197174072266
},
"scale": {
"x": 0.8100548982620239,
"y": 1.0,
"z": 1.2185389995574951
},
"friction": 0.5,
"restitution": 0.5,
"material": "M_Brick_Cut_Stone",
"speed": 1.1377057419424133,
"moves": [
112,
180
]
},
"occluder_2": {
"location": {
"x": 681.0262451171875,
"y": 369.7522888183594,
"z": 0.0
},
"rotation": {
"roll": 0.0,
"pitch": 0.0,
"yaw": -83.7597885131836
},
"scale": {
"x": 0.8828107714653015,
"y": 1.0,
"z": 1.0996980667114258
},
"friction": 0.5,
"restitution": 0.5,
"material": "M_Bricks_4",
"speed": 4.447713113166794,
"moves": [
186
]
},
"object_1": {
"location": {
"x": 753.1764526367188,
"y": 54.320823669433594,
"z": 57.58119201660156
},
"rotation": {
"roll": 0.0,
"pitch": 0.0,
"yaw": 113.01049041748047
},
"scale": {
"x": 0.840249240398407,
"y": 0.840249240398407,
"z": 0.840249240398407
},
"friction": 0.5,
"restitution": 0.5,
"material": "Yellow",
"mass": 73.9890365600586,
"initial_force": {
"x": -8527.78515625,
"y": 4952.4140625,
"z": 14996.8095703125
},
"velocity": {
"x": -284.1618957519531,
"y": 165.02378845214844,
"z": 467.0665588378906
},
"shape": "Sphere"
},
"object_2": {
"location": {
"x": 485.75006103515625,
"y": -635.8397216796875,
"z": 59.862754821777344
},
"rotation": {
"roll": 0.0,
"pitch": 0.0,
"yaw": -55.98881530761719
},
"scale": {
"x": 1.139307975769043,
"y": 1.139307975769043,
"z": 1.139307975769043
},
"friction": 0.5,
"restitution": 0.5,
"material": "GreenMaterial",
"mass": 146.7883758544922,
"initial_force": {
"x": 8689.44140625,
"y": 47984.6484375,
"z": 3588.540283203125
},
"velocity": {
"x": 289.5486145019531,
"y": 1598.939208984375,
"z": 86.9214859008789
},
"shape": "Sphere"
},
"masks": {
"floor": 213,
"object_1": 215,
"occluder_1": 255,
"occluder_2": 203,
"sky": 144
}
}
]
}