FAROS: evaluating AI systems
as strategy researchers

A useful research system must do more than propose an investment idea. It must turn the idea into code, test it, interpret a disappointing result and decide what to try next. FAROS evaluates that sequence of work: creating a strategy from scratch and improving an existing one, under the same evaluation budget.

This pilot compares four model–harness systems across forty research trajectories. The recorded histories show models changing hypotheses, abandoning unsuccessful approaches and improving the economics of their strategies. The most informative result is how progress develops: an early lead does not always persist, and a small implementation change can matter more than a more complex prediction model.

These findings come from repeated development backtests, not a live fund or an untouched test set. The opportunity they illustrate is a repeatable, inspectable research process that can be evaluated and improved.

CORAL is the software that runs the research experiment

CORAL is not a trading model. It provides persistent workspaces, called islands, where a researcher keeps strategy code and research notes. It records submitted versions and evaluation results and manages the research session. The model supplies the hypotheses, edits and next-step decisions. The native coding harness—Codex or Claude Code—provides the tools through which it works.

A separate grader turns the submitted signal into a portfolio and computes performance after the declared trading costs. The researcher cannot award itself a score. This separation makes a proposed improvement testable and leaves a record of what changed.

Researchers can explore price trends, volatility, short-selling data and insider activity on a reconstructed equity panel. Public 2021–2022 data supports their analysis. Aggregate feedback on 2023–2024 guides the iterative search. No one supplies a preferred final strategy.

Each of four systems completes five independent runs of each task. Every run receives sixteen submitted strategy evaluations, including failed submissions. That gives forty trajectories and 640 evaluations. Models may reason and edit between tests; equal evaluation budgets do not imply equal tokens, time or cost.

The current experiments keep research histories separate. CORAL can support exchanges between islands, but collaboration is not measured in this release. “Self-improvement” here means better strategies and research decisions within a run, not changes to the model’s weights or demonstrated learning across independent runs.

Results below use the full sixteen-evaluation budget and all five repetitions per capability. All table values are normalized scores. The plots below retain their labeled economic units. Green highlights the highest observed mean; it does not denote a statistically established winner.

Overall equally weights the two normalized capability scores. Each score is 100 × raw effect ÷ a fixed public-control reference gain: 2.89 points for origination and 17.27 points for refinement. Adding models does not change these references. Scores may exceed 100. The scoring rule was adopted after observing the pilot and does not establish equal task difficulty. Scoring and uncertainty details ↗

1. The early leader was not the final leader

After four evaluations, Sonnet’s mean refinement gain was 15.23 percentage points, versus Astra’s 8.42. By evaluation sixteen, Astra reached 23.23 points and Sonnet 18.11. Both estimates use the same five runs at each checkpoint. This is an observed change in ordering, not a significance claim.

Why it matters: a short evaluation measures a different part of research performance. Finding a useful early change and building on repeated feedback need not favor the same system. The learning curves below make that distinction visible.

2. One improvement came from trading the idea less aggressively

In Astra’s first clean refinement run, the researcher had combined price momentum with short-interest information. It then tested a specific hypothesis: daily changes in the combined signal could create costly trading noise. The next submission kept the signal construction and changed only its smoothing weight.

Traded notional fell by 68%. Net profit rose from $93 to $817 on the fixed $10,000 development book. Both higher gross profit and lower costs contributed. The hypothesis was recorded before the test, and the parent–child code comparison confirms the isolated smoothing change.

Why it matters: the researcher did more than search for a stronger predictor. It tested how an existing signal should be expressed over time. That is a concrete example of research on implementation and trading economics.

3. More complex models were not always the useful next step

In Astra’s first origination run, an early short-interest signal produced $171 net. Three subsequent public-trained tree-model submissions lost money after costs in development. The researcher returned to a simpler momentum signal, then combined it with short-interest, short-volume and insider-purchase information. The final submission reached $451 net.

Why it matters: the useful behavior was not a straight line of successful guesses. The history records an unsuccessful branch of research followed by a change of direction. The best-so-far curve hides that detour; the submitted strategies reveal it.

4. Some breakthroughs arrived late—and independent searches overlapped

In one Terra refinement run, the best result after ten evaluations was still a $1,263 loss. The first profitable submission arrived at evaluation 13, at $111 net, after switching to a signal based on short-interest days-to-cover. Stopping at ten tests would have missed that outcome.

Separately, Astra and Luna independently originated signals favoring stocks with lower days-to-cover, a measure of short-selling positions relative to trading volume. They used different formulas but both recorded $171 net. This suggests overlapping discoveries in this task; it does not prove identical portfolios or a general investment effect.

Why it matters: research needs enough room to change direction, but additional researchers do not automatically produce different ideas. Those observations motivate testing longer horizons and, later, whether sharing findings improves search efficiency.

Each curve averages the best strategy retained at a given checkpoint. The shaded region is a pointwise 95% confidence interval across independent research runs. Best-so-far curves cannot decrease by construction: a flat segment may include several unsuccessful experiments. These plots describe gains from additional evaluations, not scaling with tokens or hardware compute.

The compelling part of this pilot is the combination of behaviors: models propose hypotheses, implement them, encounter failures, revise their approach and sometimes improve the economics of an existing idea. The resulting code and research records make that work inspectable. A model comparison can therefore examine how a system researches, not just the profit of its final submission.

The aggregate results also distinguish capabilities. Origination means are closely grouped and their 95% intervals overlap. Refinement shows a wider range of observed means, with Astra highest and Luna more variable. A single score should not obscure those differences. This release does not report multiplicity-adjusted pairwise significance tests.

The next scientific question is whether these behaviors repeat across new tasks and untouched periods, and whether a better research environment makes them more reliable. CORAL supplies a common environment for measuring that progression. Longer-horizon work, collaboration and framework comparisons are natural follow-up experiments, not established results here.

What this pilot does not yet establish

All five repetitions reuse the same historical equity task. The researchers receive repeated development feedback, so better backtests are not evidence of out-of-sample alpha. Reconstructed data, assumed execution costs and variations in native harness versions also limit interpretation. Confidence intervals describe variation between research runs, not future market returns. Fixed public-control normalization stabilizes the scoring scale when models are added; it does not calibrate equal task difficulty.

These limits leave a concrete result to build on: a working environment for comparing autonomous research, with examples of hypotheses, failures and improvements that can be traced to executable strategies.