Large gap when reproducing ScanNet Table III results

[Henry-Bi]

I am trying to reproduce the ScanNet results reported in Table III of the paper,
especially the InstantSFM w/o depth setting. However, my reproduced Chamfer
results are much worse than the reported numbers, so I would like to confirm whether
my preprocessing / frontend / config setup is correct.

Environment

• Repo: latest official InstantSFM code
• Dataset: ScanNet scenes used in Table III
• Depth: disabled
• Frontend: COLMAP SIFT features + matching, exported as database.db
• Backend: ins-sfm

Scenes tested

I tested the ScanNet scenes reported in Table III:

scene0001_00
scene0001_01
scene0002_00
scene0002_01
scene0003_00
scene0003_01
scene0003_02

### Paper numbers

According to Table III, InstantSFM w/o depth reports approximately:

scene0001_00: 0.557
scene0001_01: 0.768
scene0002_00: 0.244
scene0002_01: 0.205
scene0003_00: 2.353
scene0003_01: 1.211
scene0003_02: 0.556
Average:      0.842

### My reproduced results

Using COLMAP sequential matching and no depth, my average Chamfer is around 10-11,
much worse than the paper.

I also tried several checks:

1. Using ScanNet intrinsics as COLMAP PINHOLE.
2. Letting COLMAP use default SIMPLE_RADIAL.
3. Disabling view graph calibration.
4. Fixing image filename/order issues by preparing zero-padded image symlinks.
5. Running COLMAP exhaustive matching on scene0001_00.

For example, on scene0001_00, using exhaustive matching:

frontend: COLMAP exhaustive
verified pairs: ~47k
largest connected component: ~0.94-0.95
registered images by InstantSFM: ~350-400
Chamfer: ~9.5-9.8

This is still far from the paper value 0.557.

[Henry-Bi]

Could you please clarify the official reproduction setup for ScanNet Table III?

1. Which exact frontend was used for the reported ScanNet numbers?

   • COLMAP sequential matching?
   • COLMAP exhaustive matching?
   • GLOMAP frontend?
   • Any custom pair generation?

2. What COLMAP camera model was used for ScanNet?

   • PINHOLE with ScanNet intrinsics?
   • SIMPLE_RADIAL?
   • Shared camera or per-image camera?

3. Were image names / image IDs required to follow temporal order for sequential
   matching?

4. Was view graph calibration enabled for the reported w/o depth results?

5. Is there any special preprocessing for ScanNet poses / intrinsics / evaluation
   alignment?

6. Are the Table III numbers evaluated with the scripts under instantsfm/eval, or
   with another evaluation protocol?

Even when the frontend graph seems very dense and connected, InstantSFM registers
only part of the sequence and the Chamfer remains much larger than the paper result.
I would like to know whether this is due to a wrong reproduction setup, a missing
preprocessing step, or a difference in the evaluation protocol.

Any guidance or official command/config for reproducing the ScanNet Table III w/o
depth results would be very helpful.

[cre185]

Hi. For your quetions:

1. We use COLMAP exhaustive matching but not sequential. No custom pairs included.
2. SIMPLE_RADIAL, default configuration for COLMAP/GLOMAP. Cameras may be shared if all the images are captured by same device (or, can assume they have same intrinsics), that would make the result better.
3. No theoretically, but we're doing so. This problem can originate from COLMAP's feature matching process, however as we're using exhaustive there shouldn't be a big difference.
4. Yes.
5. I think this is likely where the main difference arises. According to your desciption, you didn't make any alignment between the optimized results and GT. They need to be aligned or the calculated Chamfer Distance would be meaningless. We use COLMAP's model aligner to align the two pcds before calculating CD.
6. No. We're not using the outdated logic in instantsfm/eval anymore, so I can't guarantee the logics there are correct. However CD are relatively easy to implement, so I don't think the problem is huge here.

[Henry-Bi]

Thank you for the clarification. After adding the alignment step with colmap model_aligner, the ScanNet Chamfer results become much more reasonable.

For the 7 ScanNet scenes, using my previous run, the mean aligned CD is now:

InstantSFM w/o depth: 0.9027
InstantSFM w/o depth no VGC: 0.9047
Paper Table III w/o depth: 0.8420

So the large Chamfer gap was mainly caused by my missing alignment step. Thx for pointing this out.

I have one follow-up question about registered image coverage.

In some runs, InstantSFM seems to register only part of the input sequence. I would like to know whether this is expected behavior for the global pipeline, or whether it indicates that some stage of my reproduction is still not fully aligned with your setup.

Could you please advise which logs or intermediate statistics are most useful for diagnosing registered image coverage?

[cre185]

If the overlap between different continuous segments of images is small, such thing would happen. Generally speaking, there are often a specific part of images that have bad matching points and thus removed from the reconstruction, so it is likely an expected behavior.

[Henry-Bi]

Thanks for the explanation. I did one more check on scene0001_00 using the same COLMAP exhaustive frontend database for both COLMAP mapper and InstantSFM.
After alignment with colmap model_aligner, I got:

Method Registered images Points3D Aligned CD

COLMAP mapper 572 / 973 12626 0.9916

InstantSFM w/o depth 353 / 973 4346 1.4948

InstantSFM w/o depth, no VGC 353 / 973 4342 1.4954

So in this particular reproduction, COLMAP does not completely fail on scene0001_00; it registers more images than InstantSFM and gives a lower aligned CD, although the reconstruction is still fragmented into multiple components.

I noticed that Table IV mentions COLMAP failures on ScanNet due to poor reconstruction quality / too few registered cameras, making alignment and metric computation unreliable.

Could this difference be due to a different failure criterion, COLMAP configuration, or evaluation threshold?

[cre185]

I'm not quite sure, COLMAP is changing and updating as always. However in our experiments, we simply use COLMAP's model aligner as the only criteria for alignment. If model aligner outputs a failure, then we consider the reconstruction as failed and report it in the table. We did not change COLMAP's evaluation criteria or other configurations as well. That's all I can remember.

[Henry-Bi]

Thanks for the detailed explanation. This has completely addressed my concerns.

[Henry-Bi]

Sorry for borthering you again, would you mind sharing the version of the used COLMAP？

I'm not quite sure, COLMAP is changing and updating as always. However in our experiments, we simply use COLMAP's model aligner as the only criteria for alignment. If model aligner outputs a failure, then we consider the reconstruction as failed and report it in the table. We did not change COLMAP's evaluation criteria or other configurations as well. That's all I can remember.