GradingPartial

Tests decide the reward

OpenAnvil grades the generated app with deterministic hidden tests. Android and iOS use different boot environments, but they produce the same evidence and reward contract.

What starts grading

The build must stop before hidden tests can enter the environment.

Agent submits

Working

Generated app becomes fixed input

›

Submit event

Partial

Cloud control signal

›

Grader queue

Working

Platform and run metadata

›

Control Lambda

Working

Chooses Android or iOS

›

Boot grader

Target

Platform environment starts

Local versus cloud

Local CLI calls the grader after the agent loop. Cloud mode is designed to react to submit or boot_grader events delivered through EventBridge, SNS, and SQS.

Trial metadata

trials/*/trial.toml now requires description under [trial] for CLI menu. Validation via openanvil validate errors if missing, CI checks category + description and runs validate for each trial.

Android and iOS boot differently

The platform boundary affects compute, device startup, installation, and cleanup. It does not change reward semantics.

One trigger branches into two platform environments

Android and iOS boot differently, then converge on the same test tiers and reward contract.

WorkingPartialTarget

Android grader

Target

The target cloud path runs a pinned grader image on a Linux EC2 host with Docker and hardware virtualization.

1Pull the pinned Android grader image from ECR.

2Start the container and boot an Android virtual device.

3Build the generated bare React Native project with Gradle.

4Install the APK through adb and launch the app.

5Run functional, Maestro flow, and contract checks.

6Write evidence, stop the emulator, and remove the container.

Current state

The Docker provisioner can prove parts of image startup, but adb readiness, app installation, Maestro execution, and cloud grader boot are incomplete. _boot_android_grader() is a stub.

iOS grader

Target

The target cloud path runs a pinned macOS Tart VM on an EC2 Mac host controlled through SSM.

1Allocate or reuse the dedicated Mac host within Apple limits.

2Start a pinned Tart VM with Xcode and the offline package pantry.

3Scaffold and build the generated bare React Native project.

4Boot an iOS Simulator, install the app, and launch it.

5Run functional, Maestro flow, and contract checks.

6Write evidence, shut down the simulator, and stop the VM.

Current state

Local Simulator and Tart provisioners contain real setup logic, but the connected cloud grading lifecycle is incomplete. _boot_ios_grader() is a stub.

Four tiers test different failure classes

A single pass or fail hides useful signal. The reward preserves which kind of capability succeeded.

Build

weight 0.20

Does the app compile, install, and boot without crashing?

Functional

weight 0.30

Does the app logic satisfy hidden React Native Testing Library tests?

Flow

weight 0.35

Can Maestro complete expected user journeys on a real simulator or emulator?

Contract

weight 0.15

Did the app obey routes, test IDs, API rules, dependencies, and network limits?

Current state

The grader writes the verifier output shape, but several tier implementations still return placeholder passes. The documentation must not present the resulting score as validated benchmark signal.

How subtests become reward

Every subtest produces a deterministic binary result. Tier means are combined using the trial weights.

Reward composition

The final score preserves four kinds of product correctness, with flow carrying the largest weight.

WorkingPartialTarget

0..1

deterministic reward

Build

compile and boot

20%

Functional

logic through hidden tests

30%

Flow

real user journeys

35%

Contract

interfaces and constraints

15%

reward = 0.20 × build + 0.30 × functional + 0.35 × flow + 0.15 × contract

Each tier score is the mean of its hidden subtests. The final value stays between 0 and 1.

Why no model judge

The same app and tests produce the same answer. Reward does not drift with judge prompts, sampling, or model upgrades.

Why keep the vector

A model may build successfully but fail user flows. Per-tier and per-subtest results preserve that training signal.

What grading publishes

Grading completes only after the evidence and terminal state are durable.

Machine-readable

Reward, grade JSON, per-metric files, subtest outcomes, and terminal machine status.

Human-readable

Scorecards, failure messages, build logs, Maestro output, and final or per-flow screenshots.

Lifecycle

A terminal event updates the app, while the provisioner cleans up the device, VM, container, or worker.

Return to the run lifecycle

→

See where grading fits after the agent loop and before artifact publication.

See the cloud control plane

→

Understand the event bus, queues, compute, storage, app hosting, and deployment workflow.