← Sinan Lab › Methodology
中文

Measurement methodology v0.1

1. What we measure

One measurement unit is model version × hardware × precision (fp16 / int8 …) × batch = 1 × input resolution × action-chunk length. Metrics: single-step inference latency p50 / p95 (ms), first-action latency, throughput (action chunks per second), peak VRAM (GB), power (optional).

2. How a run is done

  1. Warm up for at least 20 steps before timing; then time at least 200 steps.
  2. Fix input resolution and action-chunk length; record both in config.
  3. Record a hardware fingerprint: GPU model, driver, CUDA, CPU, RAM, OS, power mode.
  4. Report p50 and p95. Never report a single run as “the” latency.

3. Sources and their labels

Every number carries one of three source types, always shown next to it: Literature (from a paper or official doc), Maintainer measured (run by Sinan Lab), Crowd upload (from the CLI). A configuration becomes “verified” only after at least three independent contributors or a maintainer re-run agree.

4. Display rules

  • Fewer than 3 samples: shown with a Preliminary (n<3) badge.
  • Values more than 3σ from the median are quarantined for review and not displayed.
  • Unverified fields are shown as Unverified. We never fill them with defaults.

5. Cost

cost per 1k inferences (CNY) = p50 ms ÷ 1000 × hourly rental (CNY) ÷ 3600 × 1000. The rental price is the card's public on-demand hourly rate from the Sinan Compute price library, with source and date shown. If either input is missing, the cell shows “—” and the reason.

6. What we do not say

We state measurements and conditions. We do not rank models as better or worse; fitness depends on the task, and that judgment is yours.

7. Crowd CLI (planned)

sinan-bench hw                          # hardware fingerprint
sinan-bench run --model gr00t-n1.7 --steps 200 --precision fp16
sinan-bench report --last              # local JSON / Markdown report
sinan-bench upload --last              # metrics + fingerprint only; no images, data or keys

Uploads carry only metrics and the hardware fingerprint. Same fingerprint and configuration are de-duplicated within 24 hours.