GitHub + Docker

Reproduce benchmark outputs

The reproduction path is GitHub-first and Docker-first. It follows the SOP folders for datasets, truths, algorithms, timing, resource logs, adapters, and EPA outputs.

Repository contract

SOP-aligned GitHub repository structure

The website defines the public contract now so repository cleanup can implement the same paths without changing result semantics.

vio-benchmark/
  docker/
    <system>/
      Dockerfile
  adapters/
    <dataset>/
  datasets/
    <dataset>/<sequence>/
  truths/
    <dataset>/<sequence>.txt
  algorithms/
    <system>/<dataset>/<sequence>.txt
  timing/
    <system>/<dataset>/<sequence>.txt
  resource/
    <system>/<dataset>/<sequence>/bm_logger.csv
  epa_outputs/
    <system>/<dataset>/<sequence>/summary.csv
  scripts/
    run_system.sh
    run_epa.sh
    summarize_results.sh

Docker workflow

Commands that reproduce artifacts

Every public numeric row should be traceable to a Git commit, Docker image, dataset case, and EPA output directory.

Clone the benchmark repository

git clone <public-vio-benchmark-repo-url>
cd vio-benchmark
mkdir -p datasets truths algorithms timing resource epa_outputs adapters docker scripts

Build a system Docker image

docker build -t vio-benchmark/openvins:ubuntu20.04 docker/openvins
docker build -t vio-benchmark/basalt:ubuntu20.04 docker/basalt

Run one system on one sequence

docker run --rm \
  -v "$PWD/datasets:/datasets:ro" \
  -v "$PWD/algorithms:/algorithms" \
  -v "$PWD/timing:/timing" \
  vio-benchmark/<system>:latest \
  run_sequence --dataset euroc_mav --sequence MH_01_easy

Evaluate with EPA in a container

docker run --rm \
  -v "$PWD:/workspace" \
  vio-benchmark/epa:latest \
  epa /workspace/truths/euroc_mav/MH_01_easy.txt \
      /workspace/algorithms/openvins/euroc_mav/MH_01_easy.txt \
      --out-dir /workspace/epa_outputs/openvins/euroc_mav/MH_01_easy

Release rule

No repository, no public number

A result can stay in the dashboard as TBD before the reproduction path is public. It should not become a numeric public value until the command, container, and output artifacts can be audited.

Required metadata

System commit, Docker tag, dataset sequence, config file, hardware platform, and command log.

Required outputs

Pose trajectory in algorithms/, timing file in timing/, EPA summary in epa_outputs/, and staged resource log when available.

Publication gate

Promote only validated numeric rows. Keep all incomplete public values as TBD, N/A, Partial, or Unresolved.