GitHub + Docker

Reproduce benchmark outputs

The public benchmark is centered on a GitHub repository, Docker targets, standard pose/timing/resource outputs, and EPA artifacts. Repository URLs stay TBD until release rather than being fabricated.

Repository contract

Expected GitHub repository structure

The website defines the public contract now so the repository cleanup can implement it consistently.

vio-benchmark/
  docker/
    <system>/
      Dockerfile
  datasets/
    <dataset>/<sequence>/
  groundtruth/
    <dataset>/<sequence>/gt.txt
  results/
    pose/<system>/<dataset>/<sequence>/poses.txt
    time/<system>/<dataset>/<sequence>/run_time.csv
    resource/<system>/<dataset>/<sequence>/monitor_cpu_memory.csv
    epa/<system>/<dataset>/<sequence>/metrics.json
  scripts/
    run_system.sh
    run_epa.sh
    summarize_results.sh

Docker workflow

Commands that reproduce artifacts

Every public numeric row should be traceable to a Git commit, Docker image, dataset case, and EPA output directory.

Clone the benchmark repository

git clone <public-vio-benchmark-repo-url>
cd vio-benchmark
mkdir -p datasets groundtruth results/{pose,time,resource,epa}

Build or pull Docker images

docker build -t vio-benchmark/openvins:latest docker/openvins
docker build -t vio-benchmark/basalt:latest docker/basalt

Run one system on one sequence

docker run --rm \
  -v "$PWD/datasets:/datasets:ro" \
  -v "$PWD/results:/results" \
  vio-benchmark/<system>:latest \
  run_sequence --dataset euroc_mav --sequence V1_01_easy

Evaluate with EPA in a container

docker run --rm \
  -v "$PWD:/workspace" \
  vio-benchmark/epa:latest \
  epa /workspace/groundtruth/euroc_mav/V1_01_easy/gt.txt \
      /workspace/results/pose/openvins/euroc_mav/V1_01_easy/poses.txt

Release rule

No repository, no public number

A result can stay in the dashboard as TBD before the reproduction path is public. It should not become a numeric public value until the command, container, and output artifacts can be audited.

Required metadata

System commit, Docker tag, dataset sequence, config file, hardware platform, and command log.

Required outputs

Pose trajectory, timing CSV, CPU/memory monitor, EPA metrics, EPA summary, and plots when generated.

Publication gate

Promote only validated numeric rows. Keep all incomplete public values as TBD, N/A, Partial, or Unresolved.