[Evaluation] Possible camera-space/world-space mismatch in the scene-flow metrics of Table 1

[ZhouZhanbo]

First of all, thank you for releasing MotionCrafter and its evaluation pipeline. I found the joint geometry-and-motion formulation very interesting, and the benchmark covering Kubric, Spring, Virtual KITTI 2, Dynamic Replica, and Point Odyssey is especially valuable.

I am developing another model that also predicts dense 3D scene flow. Among the existing benchmarks, I believe the MotionCrafter benchmark is the most suitable one for evaluating my model. Before reporting results for my own method, I first tried to reproduce the MotionCrafter results in Table 1 to verify that my preprocessing, inference, and evaluation pipeline were correct.

During this reproduction, I noticed a possible coordinate-frame mismatch between the scene-flow annotations produced by the released preprocessing scripts and the scene-flow evaluation.

Possible coordinate-frame mismatch

As far as I understand, the preprocessed scene_flow is stored as a camera-space endpoint offset (referred to as CSO below).

For a transition from frame (t) to frame (t+1), the preprocessing scripts effectively compute

[
F_t^{\mathrm{CSO}}

P_{t+1}^{C_{t+1}} - P_t^{C_t},
]

where:

• (P_t^{C_t}) is the point map at frame (t), expressed in camera (t);
• (P_{t+1}^{C_{t+1}}) is the corresponding endpoint, expressed in camera (t+1).

For example, datasets/preprocess/gen_spring_video.py contains:

scene_flow = cam_coords - point_map

Similarly, the Kubric preprocessing constructs the next-frame point map from the tracked coordinates and next-frame depth, followed by:

scene_flow = point_map_deform - point_map

Therefore,

point_map[t] + scene_flow[t]

recovers the corresponding endpoint in the coordinate system of camera (t+1), rather than a world-space endpoint.

However, in the released evaluation behavior, the predicted point map and predicted scene flow are aligned/transformed into the world coordinate system, while the GT scene flow is passed directly to sceneflow_metrics:

sflow = sceneflow_metrics(
    aligned_sflow[:-1],
    gt_sflow[:-1],
    gt_dmask[:-1],
)

This appears to compare a world-space predicted displacement with the original camera-space endpoint offset.

Please correct me if I have misunderstood the intended convention.

Reproduction setup

I evaluated three settings:

1. Reproduce 1

   • INFER_FRAMES=8
   • released scene-flow evaluation behavior
   • predicted point map treated as a world map

2. Reproduce 2

   • INFER_FRAMES=25
   • released scene-flow evaluation behavior
   • predicted point map treated as a world map
   • the inference window is consistent with the default MotionCrafter setting

3. Reproduce 3

   • INFER_FRAMES=25
   • predicted point map treated as a world map
   • GT scene flow converted from CSO to the first-camera world coordinate system before evaluation

For convenience, I added a local Boolean flag:

--no-eval_scene_flow_in_world

to reproduce the released evaluator behavior, and

--eval_scene_flow_in_world

to evaluate the prediction and GT consistently in the world coordinate system.

The following values use the same presentation scale as Table 1: (\mathrm{Rel}_p) and EPE are multiplied by 100, while (\delta^p) and APD are percentages.

Geometry results

Method	Kubric Rel↓	Kubric δ↑	Spring Rel↓	Spring δ↑	VKITTI2 Rel↓	VKITTI2 δ↑	Dynamic Replica Rel↓	Dynamic Replica δ↑	Point Odyssey Rel↓	Point Odyssey δ↑
MotionCrafter, Table 1	3.40	98.73	29.20	77.27	14.60	84.58	4.04	99.00	9.94	94.90
Reproduce 1, 8 frames	6.38	95.58	32.43	75.98	15.63	82.38	4.89	98.77	11.66	92.76
Reproduce 2, 25 frames	6.63	95.20	27.98	78.60	15.33	82.87	4.49	98.99	10.58	93.41
Reproduce 3, 25 frames, corrected flow coordinates	6.63	95.20	27.98	78.60	15.33	82.87	4.49	98.99	10.58	93.41

As expected, changing only the GT scene-flow coordinate conversion does not affect the geometry metrics.

Scene-flow results

Method	Kubric EPE↓	Kubric APD0.05↑	Spring EPE↓	Spring APD0.1↑	VKITTI2 EPE↓	VKITTI2 APD0.3↑	Dynamic Replica EPE↓	Dynamic Replica APD0.05↑	Point Odyssey EPE↓	Point Odyssey APD0.05↑
MotionCrafter, Table 1	4.60*	68.01*	5.61*	90.17*	71.75*	25.90*	0.51	99.72	3.49	80.66
Reproduce 1, 8 frames	7.26	58.98	5.72	86.62	72.53	25.13	0.49	98.77	3.45	80.95
Reproduce 2, 25 frames	7.03	60.74	5.87	86.96	72.59	24.47	0.49	99.82	3.39	81.44
Reproduce 3, 25 frames, corrected flow coordinates	1.78	91.95	2.70	94.39	9.30	94.49	0.35	99.82	1.50	94.17

The 25-frame result using the released behavior is quite close to the Table 1 numbers, particularly on Spring, Virtual KITTI 2, Dynamic Replica, and Point Odyssey.

After converting the GT scene flow into world coordinates, all five datasets improve substantially. The largest change is on Virtual KITTI 2:

EPE:       72.59 -> 9.30
APD_0.3:   24.47 -> 94.49

This large difference seems consistent with a camera/world coordinate mismatch, since Virtual KITTI 2 contains significant camera motion.

My proposed conversion

Given the stored CSO convention, I convert the GT flow using both the current-frame and next-frame camera poses.

Let (T_t^{C\rightarrow W}) be the camera-to-world pose. The world-space scene flow should be

[
F_t^W =
T_{t+1}^{C\rightarrow W}
\left(P_t^{C_t} + F_t^{\mathrm{CSO}}\right)

T_t^{C\rightarrow W}P_t^{C_t}.
]

My implementation is:

def scene_flow_to_world(
    scene_flow,
    pose_c2w,
    point_map,
    device,
):
    # Current point in world coordinates.
    point_map_world = to_world(
        point_map[:-1],
        pose_c2w[:-1],
        device=device,
    )

    # point_map[t] + scene_flow[t] is the corresponding endpoint
    # represented in camera t+1.
    point_map_deformed = point_map[:-1] + scene_flow[:-1]

    # Transform the endpoint using the next-frame camera pose.
    point_map_deformed_world = to_world(
        point_map_deformed,
        pose_c2w[1:],
        device=device,
    )

    return point_map_deformed_world - point_map_world

I then changed the metric computation from:

sflow = sceneflow_metrics(
    aligned_sflow[:-1],
    gt_sflow[:-1],
    gt_dmask[:-1],
)

to:

gt_sflow_world = scene_flow_to_world(
    gt_sflow,
    gt_pose,
    gt_pmap,
    device=device,
)

sflow = sceneflow_metrics(
    aligned_sflow[:-1],
    gt_sflow_world,
    gt_dmask[:-1],
)

Here, gt_pose has first been normalized so that the first camera pose is the identity, consistent with the world-coordinate convention used by the evaluator.

Questions

Could the authors please help clarify the following?

1. Is the preprocessed scene_flow intended to be the CSO quantity

   [
   P_{t+1}^{C_{t+1}} - P_t^{C_t}?
   ]

2. In Table 1, was the world-space prediction compared directly against this untransformed CSO ground truth?

3. Is the conversion above the intended way to evaluate both prediction and GT in a common world coordinate system?

4. If so, would it be possible to clarify the distinction between the released/Table 1 protocol and a coordinate-consistent world-space protocol in the evaluation documentation?

I completely understand that benchmark implementations can contain historical conventions, and I am not suggesting that the published numbers should simply be replaced. My main goal is to ensure that future methods using this benchmark report scene-flow results under a clearly defined and coordinate-consistent protocol.

Thank you again for releasing the code, models, and benchmark. I would greatly appreciate any clarification.

[RuijieZhu94]

Hi @ZhouZhanbo, thanks for pointing out this issue. I think you are right. The evaluation code is updated. I will check whether the number is still correct. BTW, have you used my split to reproduce the results? I am wondering why some geometric metrics are also not that aligned.

[ZhouZhanbo]

Hi @ZhouZhanbo, thanks for pointing out this issue. I think you are right. The evaluation code is updated. I will check whether the number is still correct. BTW, have you used my split to reproduce the results? I am wondering why some geometric metrics are also not that aligned.

Thank you very much for such a quick response.
Yes, I used the splits provided in evaluation/benchmark_datasets_lists. I compared my generated meta_infos.txt files with the provided files for all five datasets, and the sample lists match exactly.
However, I should clarify that I regenerated the benchmark data from the raw datasets rather than using the exact preprocessed benchmark files used for the paper.
The scripts for Kubric, Spring, Virtual KITTI 2, Dynamic Replica, and Point Odyssey are located under datasets/preprocess, whose documentation describes them as preprocessing scripts for training data. Meanwhile, evaluation/preprocess does not appear to contain preprocessing scripts for these five Table 1 datasets. Therefore, I made several minimal adaptations so that the training preprocessing scripts could process the evaluation/test splits and reproduce the provided benchmark sample lists.
For example, I made the Dynamic Replica split configurable so that it could process the test split, and restricted Virtual KITTI 2 preprocessing to the scenes and clone condition used by the released evaluation list. I also configured the corresponding raw-data directories, split names, clip lengths, and output metadata for the other datasets.
Therefore, although my sample split matches the released split exactly, my regenerated HDF5 files may not be bit-identical to the benchmark files used to obtain the paper results. Differences in resizing, normalization, camera conversion, masks, or preprocessing versions could potentially explain the remaining differences in the geometry metrics.
In addition to these evaluation-split adaptations, I found and corrected two apparent issues in the released Kubric preprocessing code:
The loaded input camera poses were overwritten by an empty output list:
camera_poses = np.load(camera_path, ...)[
"camera2world_matrix"
]
...
camera_poses = []
...
camera_pose = camera_poses[idx]
I kept the loaded poses in a separate variable (input_camera_poses) and used that variable when processing each frame.
The field named visibility in the generated tracking file actually stores data["occluded"], so True means occluded rather than visible. The original logic was:
forward_depth_mask = ~visibility
scene_flow[forward_depth_mask] = 0
visible_mask = forward_depth_mask
This zeros the scene flow at visible pixels and then uses those pixels as the deformation evaluation mask. I changed it to:
occluded_mask = visibility
scene_flow[occluded_mask] = 0
visible_mask = ~occluded_mask
The camera-pose issue affects the stored Kubric poses and consequently any conversion of point maps or scene flow into the first-camera/world coordinate system. The visibility issue affects the Kubric scene-flow GT and deformation mask, but should not affect the geometry metrics because point_map and valid_mask are generated independently.
Since the first issue would normally make the released Kubric preprocessing script fail by indexing an empty list, I suspect that the benchmark data used for the paper may have been generated using another version of this script. Could you please confirm whether these two Kubric issues were already corrected in the preprocessing pipeline used for Table 1?
If possible, could you also clarify whether the exact preprocessed benchmark GT files, or the exact preprocessing commands/configurations used for the five Table 1 datasets, will be released? That would help me determine whether the remaining geometry differences come from preprocessing rather than inference or evaluation.

[RuijieZhu94]

Hi @ZhouZhanbo, sorry for that. Currently I don't have the access to directly share the benchmark with you. Yeah the processing code might have some issues. This is not the orignial code because I cleaned it using my agent. Later I will check the processing code and make sure it is reproducible.

[ZhouZhanbo]

Hi @ZhouZhanbo, sorry for that. Currently I don't have the access to directly share the benchmark with you. Yeah the processing code might have some issues. This is not the orignial code because I cleaned it using my agent. Later I will check the processing code and make sure it is reproducible.

Hi @RuijieZhu94, thank you very much for the clarification.

I completely understand that the cleaned preprocessing code may have diverged from the original internal version. Thank you for being willing to check the preprocessing code and make the benchmark reproducible.

Since the scene-flow evaluation code has now been updated, may I ask whether you would also consider re-running the Table 1 evaluation using the corrected evaluation code together with the original/official preprocessed evaluation data used for the paper?

I think this would be very helpful for clarifying two things:

1. Whether the Table 1 scene-flow numbers remain consistent after converting the GT scene flow into the same world coordinate system as the prediction.
2. Whether the geometry differences I observed are mainly caused by preprocessing differences, rather than inference or metric implementation.

At the moment, because I can only regenerate the HDF5 files from the released preprocessing scripts, it is difficult for me to determine whether the remaining discrepancies come from:

• the updated scene-flow coordinate conversion,
• differences between the released cleaned preprocessing code and the original preprocessing pipeline,
• resizing / normalization / mask conventions,
• or possible differences in the exact benchmark files used for the paper.

Therefore, if you could re-run the evaluation on your side with the official evaluation data and the corrected code, or later provide the exact preprocessing commands/configurations for the five Table 1 datasets, that would be extremely helpful for reproducibility.

I really appreciate your quick response and your effort in maintaining the released code and benchmark.

[RuijieZhu94]

Hi @ZhouZhanbo, this is my reproduced results. For more infos, plz contact me via email.

Method	Kubric Rel↓	Kubric δ↑	Spring Rel↓	Spring δ↑	VKITTI2 Rel↓	VKITTI2 δ↑	Dynamic Replica Rel↓	Dynamic Replica δ↑	Point Odyssey Rel↓	Point Odyssey δ↑
MotionCrafter, Table 1	3.40	98.73	29.20	77.27	14.60	84.58	4.04	99.00	9.94	94.90
Our rerun, 25 frames, world-consistent evaluation	3.79	98.42	29.62	78.09	12.78	87.54	4.52	98.94	10.24	93.81

Method	Kubric EPE↓	Kubric APD0.05↑	Spring EPE↓	Spring APD0.1↑	VKITTI2 EPE↓	VKITTI2 APD0.3↑	Dynamic Replica EPE↓	Dynamic Replica APD0.05↑	Point Odyssey EPE↓	Point Odyssey APD0.05↑
MotionCrafter, Table 1	4.60*	68.01*	5.61*	90.17*	71.75*	25.90*	0.51	99.72	3.49	80.66
Our rerun, 25 frames, world-consistent evaluation	1.71	92.78	2.31	96.10	8.86	95.42	0.36	99.75	1.38	94.55