ArchitecturePartial

Follow one run from specification to reward

OpenAnvil gives an AI a mobile app specification, lets it build the app, runs deterministic tests, and saves the complete result as reusable training data.

Input

A trial contains the app specification, the visible contract, hidden tests, fixtures, and reward weights.

Execution

A model receives the specification and contract, then builds a bare React Native app through five controlled tools.

Output

OpenAnvil saves the app, transcript, test evidence, screenshots, and deterministic reward for every run.

The whole system in one view

The runtime has one job: turn a trial and a model into a graded mobile app plus a complete evidence package.

One run, two entry paths, one evidence contract

Local and cloud launches converge on the same agent, grading, and artifact lifecycle.

WorkingPartialTarget

Choose trial

Specification and contract

›

Start run

Local CLI or cloud API

›

Agent builds

Five tools in a work directory

›

Submit

Or a safety limit ends the loop

›

Grade

Build, functional, flow, contract

›

Publish

Artifacts and reward

The key design choice

The model never sees the hidden tests. Tests produce the grade, so no language model judge sits in the reward path.

A run, in logical execution order

Each row names the component that owns the step and the state of the current implementation.

Load the trial

Owner: CLI and orchestrator

Working

OpenAnvil resolves the trial folder. The model receives spec.md and the files under contract/. Hidden tests and the reference app remain outside model context.

Choose local or cloud

Owner: Orchestrator factory

Working

Local runs use LocalOrchestrator. Cloud runs use a thin HTTP client that calls API Gateway with the selected trial, platform, model, and safety limits.

Create the run record

Owner: Local disk or API Lambda

Working

Local mode creates a run directory. Cloud mode creates a run identifier, writes the manifest and machine state to S3, then publishes a run_start event.

Boot the worker

Owner: Control plane

Partial

In cloud mode, EventBridge, SNS, and SQS deliver the start event to the control Lambda. It launches an EC2 worker and starts the agent runtime through SSM.

Run the agent loop

Owner: Agent runtime

Working

The model alternates between reasoning and tool calls. Every response, tool call, observation, token count, and generated file is written through the Scribe abstraction.

Finish the build

Owner: Agent loop

Working

A run ends when the model calls submit, reaches the step limit, exceeds the wall clock limit, or repeatedly returns an invalid response.

Boot and run grading

Owner: CLI or grader control path

Partial

Local CLI calls the grader after the loop. Cloud submission is designed to enter the grader queue and boot a platform-specific environment. The cloud boot functions are still stubs.

Publish the result

Owner: Grader, Scribe, and web app

Partial

The target output includes per-subtest evidence, screenshots, a reward, terminal machine state, and the complete agent trajectory. The web app reads the same artifact layout from S3.

Local and cloud runs share a contract

The storage location and launch mechanism change, but the run artifacts should keep the same shape.

Local path

CLI starts the loop directly

The orchestrator creates runs/<run-id>/, starts the model loop, and writes artifacts through LocalScribe. The CLI calls grading after the loop returns.

Cloud path

API and events start an isolated worker

The thin client calls API Gateway at cloud.api_url (default prod in yaml). The API Lambda records the run in S3 and emits an event. The control Lambda launches an EC2 worker through SSM, while S3Scribe preserves the run. Thin CLI now live-follows queued → provisioning → running → agent loop via GET /runs/{id} and /logs polling.

TUI Select UI

openanvil plain now shows a compact select UI for trial, target (default cloud-aws), platform (default both ios,android), provider with key status, model as Friendly (qualified). Arrow-key dropdown via curses when TTY, compact numbered fallback otherwise. Only prompts for missing inference key and OPENANVIL_API_KEY if not in env, with paste-friendly raw reader that bypasses 1024 char limit.

Current integration gap

The local run path currently starts the agent loop without calling the provisioner. The intended six-stage pipeline is not yet one connected function.

The agent loop is small on purpose

OpenAnvil owns the environment and evidence. The model only decides which controlled action to take next.

The agent loop as a closed evidence cycle

Every action returns an observation and writes a durable trace before the next model turn.

WorkingPartialTarget

Model response

Text plus native tool calls

›

Execute tools

read, write, list, bash, submit

›

Record evidence

Transcript and trajectory step

›

Return observation

Tool output becomes context

›

Repeat

Until submit or safety stop

The model can see

The app specification, visible contract, current work directory, and results from its own tool calls.

The model cannot see

Hidden grading tests, the reference implementation, or files outside its assigned work directory.

Safety limits

Maximum steps, a run deadline, per-command timeouts, output truncation, and repeated-format-error termination.

What happens when the agent is done

Submission ends model work. It does not end the OpenAnvil run.

1
Freeze the build output
The final app directory and submission summary become the candidate that grading evaluates.
2
Close the agent evidence
The transcript records the terminal status, total steps, token use, and final submission.
3
Start platform grading
Local mode invokes grading directly. Cloud mode is designed to route submit events through the grader queue.
4
Calculate deterministic reward
Each hidden subtest contributes to one of four weighted grading tiers.
5
Publish artifacts
The app, logs, screenshots, subtests, reward, and machine status become one durable run record.
6
Clean up compute
The intended final step shuts down the simulator, VM, container, or ephemeral worker.

Every run becomes a reusable evidence package

Local disk and S3 use the same conceptual layout so readers do not need to know where the run executed.

<run>/
  manifest.json
  machine.json
  transcript.jsonl
  trajectory/
  app/
  logs/
  grading/
  logs/verifier/
    reward.txt
    grade.json
    per_metric/

Debugging

The transcript, trajectory, logs, and machine state explain where a run stopped and why.

Evaluation

Subtest outcomes and screenshots support deterministic scoring and failure analysis.

Training data

The model trajectory paired with its reward vector can feed later post-training workflows.

Current implementation boundary

This checkpoint prevents the target architecture from being mistaken for shipped behavior.

Working

Trial loading, target selection, the thin cloud client, local agent loop, controlled tools, local and S3 Scribes, run artifacts, and cloud event infrastructure.

Partial

EC2 worker launch, provisioner backends, cloud submission routing, app ingestion, and terminal lifecycle cleanup need end-to-end validation.

Target

Real Android and iOS grader boot, real four-tier execution on both platforms, complete screenshots, and production reward publication.

Understand mobile grading

→

See how Android and iOS environments boot, run tests, and produce reward.

Understand hosting and deployment

→

See how the web app, control plane, storage, and workflow chain fit together.