Flamra + DeepSeek-V4-Pro on Aider Polyglot

Technical report · 2026-06-14

96.0%
216 / 225 exercises solved on the Aider Polyglot benchmark (2-try protocol) across six languages, graded by each language's own test suite. First-attempt rate (the stricter, single-shot number): 71.1% (160 / 225).

System

Flamra is a Windows-first, bring-your-own-key AI coding agent — a single Rust core shared by a CLI, a Tauri desktop app, and a web UI. The same flamra --agent tool-use loop that ships in the product solved each exercise.

• Model: DeepSeek-V4-Pro (open-weight, MIT), via the provider's OpenAI-compatible API.
• Protocol: Aider's two-attempt format — the agent is given the instructions and the solution stub, writes an implementation, and if the hidden test suite fails, the test output (not the test file) is handed back for one more attempt. Headline = solved-within-2-tries; we also report the first-try rate.
• Tools given to the agent: read_file, write_file, edit_file, grep, glob, bash, todo_write.
• Withheld: the hidden test files and the reference solution are moved out of the working directory while the agent solves; web_fetch / network retrieval is not in the allow-list.
• Cost: $3.66 for the full 225-exercise run — about $0.016 / exercise.

Results by language

How to read this number

We report this as a capability measurement of Flamra, not a leaderboard ranking, and two things are worth stating plainly:

• Not directly comparable to the public Aider leaderboard. That leaderboard measures a model through Aider's edit-only format. Flamra is a full agent — it can run a shell, execute code, and iterate over many turns — so the two setups are not the same experiment. A higher number here reflects the agent harness as much as the model.
• Exercism contamination. These 225 exercises are long-standing public Exercism problems with solutions all over the open web, so they are almost certainly present in any model's training data. As with every high score on a public coding benchmark, read this with standard contamination caveats. The first-try rate (71.1%) is the more conservative signal.

Evaluation & reproduction

Each exercise is solved in an isolated copy of its directory with the reference solution and tests removed, then graded by the language's native runner (pytest, cargo test, go test, gradle test, npm test, cmake + make). Grading is local and free — no hosted evaluation service. The benchmark source is Aider-AI/polyglot-benchmark.

See also our SWE-bench Lite report (61.33%), graded with the official harness.