Labbench — an AI research agency, in the cloud
An AI Research Consultancy in the Cloud
Your research department, on retainer.
Point Labbench at your repo. It reads the latest papers, runs real experiments in cloud sandboxes, and ships pull requests — with a report and a podcast.
Scroll to see how it works
The problem
You shouldn't have to quit your job to keep up.
The running joke is that keeping up with AI research is a full-time job. Every week ships results that could change your product — and you're busy shipping the product.
How it works
Hermes runs the engagement.
A president agent orchestrates four stages, end to end. You review the PR.
Watercooler
Ingests your repo into a dossier: what you're building, where it's weak.
Library
Searches the literature across Linkup and Exa; ranks pitches against your dossier.
Lab
Writes real code and runs 30-day benchmarks in Cloudflare sandboxes.
Conference
Opens PRs against your repo, with an HTML report and a podcast.
On retainer. Hermes keeps your research and repo in a three-layer memory across engagements — and between runs it dreams: consolidating what it learned and forging reusable skills, so every engagement starts smarter than the last.
The product
Watch it work, live.
Five views into a running engagement.
Bench
Mission control: every agent, every dollar, live.
Watercooler
The dossier taking shape as your repo is read.
Library
Pitches ranked as papers come in.
Lab
Benchmark curves streaming from the sandboxes.
Conference
The PRs, the report, the podcast.
What we tested
The benchmark: Vending-Bench
Vending-Bench is a long-horizon test: an LLM runs a vending-machine business for many simulated days. Over a long run, agents lose the plot and go broke — the score is Total Assets at the end. Labbench read the research, wrote and tested real code on a fork, and shipped the change that kept the agent solvent: +49% more Total Assets.
Proof
Three hypotheses tested. Two PRs shipped.
The system is real end to end: Labbench read the papers, wrote the code, and ran the benchmarks in cloud sandboxes. The winning hypothesis — a typed business-state briefing from KV memory before every tool-loop request — lifted long-horizon Total Assets by +49% on this run, and Labbench opened the pull request.
Total Assets — this engagement's run
Vending-Bench fork · same in-run baseline · higher is better
Result of this recorded run vs its baseline. Long-horizon benchmarks are high-variance; across repeated runs the winner averages ~+27–38% (n=3). We re-ran to confirm.
The pull requests
typed business-state briefing from KV memory
$634.50 → $949.00 +49% The hypothesis: Before every decision, inject a compact typed summary of the business state (cash, inventory, prices, orders) pulled from the agent's key-value memory — so it never loses track of where things stand over a long run.
Reflexion — persist failure lessons across the run
$634.50 → $732.55 +15% The hypothesis: After a costly mistake, write a short lesson to memory and resurface it when a similar decision comes up again — so the agent stops repeating the same errors.
Built on
Real infrastructure, no mocks.
Technologies we run on