Skip to content

Data and recovery

What the data directory is for

This project stores the core results of one run under:

data/runs/<run_id>/

That is not only a directory convention. It gives training, recovery, replay, and analysis one shared source of truth.

Why centralized run storage matters

If checkpoints, game records, and metrics are spread across unrelated directories, three things become harder immediately:

  1. matching a checkpoint to the data that produced it
  2. finding the latest recoverable state
  3. using tools without guessing which files belong together

What usually lives in a run directory

If this is your first time looking at the directory, think of it as the archive bag for one run. The important artifacts for that run should all be there.

Checkpoints

What it is: a training state snapshot that preserves enough information to resume work.

Why it is needed: long-running processes fail or stop. Checkpoints are the recovery input.

Inputs and outputs:

  • input: model weights, optimizer state, scheduler state, step counters, random state
  • output: readable latest.pt, previous.pt, and best.pt files

How to judge a normal result:

  • a newly written checkpoint can be loaded again
  • the three-slot layout stays bounded instead of growing forever

If you only want the shortest takeaway: a checkpoint is not just a backup copy. It is the recovery starting point for continued training.

Game records and index

What it is: self-play game logs plus an index that can locate them later.

Why it is needed: training needs raw game data, and replay or inspection needs a way to find a specific game efficiently.

Inputs and outputs:

  • input: actions, scores, captures, end reasons, configuration linkage
  • output: compressed record shards and games.index.sqlite

How to judge a normal result:

  • the index points back to the right shard and line
  • compressed records can be read, not only written

The index is like a table of contents. It does not store the whole game by itself. It tells you where to find the game.

Metrics

What it is: structured runtime and training statistics.

Why it is needed: without metrics, later analysis turns into guesswork.

Inputs and outputs:

  • input: training step, loss, worker count, hardware policy, throughput, and similar run signals
  • output: structured metrics files suitable for later analysis

How to judge a normal result:

  • metrics are written continuously
  • the recorded fields line up with configuration, checkpoints, and runtime behavior

If you only have a final summary and no ongoing metric trace, it becomes much harder to tell when the run started going wrong.

What recovery means here

Recovery is more than reloading a model file. A meaningful recovery path should restore at least:

  • model weights
  • optimizer and scheduler state
  • training step
  • generated game counters
  • the relevant configuration snapshot reference

If those pieces are missing, the process may look resumed while actually continuing from a different state.

How to judge whether recovery is working

Use these checks:

  1. the process can continue from the latest checkpoint after interruption
  2. metric steps stay continuous after recovery
  3. the recovery path does not silently switch model route or configuration meaning

A practical test looks like this:

  • let training run long enough to write checkpoints and metrics
  • stop the process
  • start it again
  • verify that the step count continues instead of restarting from zero or from a much older point