Run lifecycle¶

This page walks through one chalk run, from the command line to the finished transcript. The first part happens on your machine, the rest inside the sandbox that the runtime starts.

On your machine¶

uv run chalk run --config run_config.json

1. Read the config. --config can be JSON or a path to a JSON file. Chalk looks up the API keys in your shell (see API keys). Then the run config preprocessors from plugins can rewrite the config, in order of their entry point names.
2. Pick the runtime. This is --runtime if you passed it, otherwise your platform's default for the task's hardware. Chalk checks that the runtime is installed and set up before it goes any further. An explicit --runtime is checked before step 1, since it doesn't depend on the task.
3. Split parallel runs. With -n N, run i gets the run id <run_id>-<i>, the transcript file <name>_<i>.json, and the websocket port plus i. --websocket-port sets the first run's port. If one of the ports is already taken, for example by another chalk run, Chalk stops before the build. Give separate chalk run commands that run at the same time different ports. apple-container moves a run to a free port instead.
4. Build the image and tag it chalk. --dev skips this step; see Development loop.
5. Remove leftover containers from earlier runs. Chalk only removes containers whose names start with chalk_run_ followed by the longest prefix all the run ids share, so commands with different run ids don't touch each other's containers.

6. Start one sandbox per run, named chalk_run_<run_id>. Inside it, Chalk runs itself again:

   /root/.venv/bin/chalk run --no-containerized --config <JSON>
   

   The directory that contains transcript_file is mounted into the sandbox, so the transcript and the artifacts end up next to each other on your machine. On firecracker, that directory goes into the VM on a drive and comes back out after the VM shuts down. On modal, the sandbox's /out is copied back into it when the run ends. The websocket port is published to your machine.

7. Show progress. The terminal UI connects to each run's websocket and shows its events as they happen. With --no-ui, Chalk prints every run's output straight to the terminal instead.

8. Report. Chalk prints where each transcript and artifact directory is. With --no-ui, it exits with a non-zero code if any run ended with an error.

Inside the sandbox¶

1. Harden the process. Chalk removes every directory the student can write to from PATH, LD_LIBRARY_PATH, LD_PRELOAD and LD_AUDIT, and restarts itself. It also removes group and world write permission from mount points the runtime left writable, except the workdir and the temp directories.
2. Load the task from the environment package.
3. Start the tool server. An HTTP MCP server starts in a subprocess, on the host and port from mcp_server_config. It runs as root; see Tools.
4. Create the agent and set up the student's firewall. The student can reach localhost and the sandbox's own addresses (see CHALK_STUDENT_NETWORK in Runtimes). It can never reach the websocket port. It can only reach the tool server and the model proxy when a CLI agent needs them. If the firewall rules were applied, Chalk then tries to reach a few outside addresses as the student, and refuses to run if any of them answers. With use_fake_model, the fake model takes the agent's place here.
5. Start the event streams: the terminal output, the websocket, and the backend if backend_uri is set.
6. Set up the task:
   1. Chalk calls task.configure_tools() and registers the task's tools with the tool server. Tools that a CLI agent brings itself are skipped.
   2. If a file already exists at transcript_file, it's deleted.
   3. Chalk records a TaskStartedEvent.
   4. The default student resource limits are applied, and Chalk logs which kind of confinement is in effect.
   5. Chalk calls task.pre_hook(). Whatever it returns goes into the transcript as the metadata of a TaskPreHookCompletedEvent.
   6. Chalk adds the system message from task.system_prompt. It skips this when the property returns None, and for CLI agents, which send their own.
7. Run the steps. For each step:
   1. Chalk records a StepStartedEvent.
   2. The step's instructions go to the agent as a user message, with any extra_config changes applied; see Extra config.
   3. The agent works on the step (see The model loop).
   4. Files listed in extra_config["extra_artifact_paths"] are saved as artifacts.
   5. Chalk calls step.pre_scoring_hook(), then step.judge.evaluate(transcript), which produces a ScoringEvent. If either raises a StudentMisbehaviorError, the step scores 0; see Scoring.
   6. Chalk calls step.post_hook().
   7. Chalk records a StepCompletedEvent. If the judge said not to continue, the remaining steps are skipped and the run counts as failed.
8. Finish. Chalk records a TaskCompletedEvent with status passed or failed. If an exception happens anywhere in the run, Chalk records an ErrorEvent instead and the run ends with status error.
9. Write the transcript to transcript_file, even after an error. Then Chalk gives the transcript and the artifacts to the owner of the directory they were written to, so a container running as root doesn't leave root-owned files on your machine. Finally, the tool server stops.

The hooks are described in Tasks and steps.

sequenceDiagram
    participant CLI as chalk run (sandbox)
    participant MCP as Tool server<br/>(subprocess)
    participant Runner as Runner
    participant Agent as Agent

    CLI->>CLI: Harden, load task
    CLI->>MCP: Start tool server
    CLI->>CLI: Firewall the student
    CLI->>Runner: run()

    Runner->>Runner: task.configure_tools()
    Runner->>MCP: Register task tools
    Runner->>Runner: Default resource limits
    Runner->>Runner: task.pre_hook()
    Runner->>Runner: System message

    loop For each step
        Runner->>Agent: Step instructions
        loop Model loop
            Agent->>Agent: Next model message
            opt Tool calls
                Agent->>MCP: Call tools
                MCP-->>Agent: Results
            end
        end
        Runner->>Runner: step.pre_scoring_hook()
        Runner->>Runner: step.judge.evaluate(transcript)
        Runner->>Runner: step.post_hook()
        alt continue_task is false
            Runner->>Runner: Stop (failed)
        end
    end

    Runner->>Runner: TaskCompletedEvent
    Runner->>Runner: Write transcript file

The model loop¶

The builtin agent calls the model through litellm with the transcript's messages and the task's tools, and streams the response. If the response contains tool calls, Chalk runs them one at a time on the tool server, adds each result to the transcript, and calls the model again. A turn without tool calls ends the step.

Some turns have neither text nor tool calls, because they were cut off at the output-token limit or only contain reasoning. These don't end the step. Chalk asks the model to continue instead, up to three times in a row, and stops with an error on the fourth.

The external agent runs the same loop, but gets each message from the backend instead of calling a model. It waits up to two hours for each message. A CLI agent runs its own program once per step and calls the tools itself.

Chalk checks the limits on turns, time and context between turns.

What the run produces¶

To look at the transcripts in a directory later, run chalk dashboard out/.

Stop before the steps¶

--prepare-only sets up the sandbox the same way a real run does, up to and including the pre-hook and the system message. Then it keeps the sandbox running without starting any steps or writing a transcript. Use it to look around in a prepared sandbox:

uv run chalk run --config run_config.json --prepare-only

The sandbox is ready once /tmp/chalk_prepared_<run_id> exists inside it. Press Ctrl-C to stop it. It doesn't work with -n greater than 1, and it doesn't need a model API key.