Blacksmith vs Buildkite
Blacksmith runs your existing GitHub Actions jobs on its own faster runners, and you switch by changing one label. Buildkite is a whole CI platform with its own pipelines, and its hosted agents are the machines it runs those jobs on.
The companies and products
Blacksmith
Blacksmith is a CI service: it runs your GitHub Actions jobs on its own faster, cheaper runners instead of GitHub's.
You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. It is built for teams that want quicker, cheaper GitHub Actions, not as a general sandbox you drive by API; a separate Sandboxes product is marked coming soon.
Buildkite
Buildkite is a CI and software delivery platform, and hosted agents are the build machines it runs for you.
You write Buildkite pipelines, and each job runs on a fresh virtual machine that Buildkite starts for that job and destroys when it ends; only cache volumes carry files to the next job. Hosted agents come in Linux and macOS on Apple M4, with Windows in private preview, and you can also run Buildkite's agent on your own machines. It is a CI service, not an agent sandbox, so this page compares the part where they overlap: a machine where an agent or a job does its work.
- Where it runs
- Hosted agents run only in US East Coast data centers
- Compliance
- Maintains a SOC 2 Type II report
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “78th cheapest of 377” means 77 of the other 376 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Blacksmith: price rank | Buildkite: price rank | bochka: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month An agent that builds a whole app needs the same machine for hours with its files intact, but both destroy the machine when the job ends and keep only chosen folders. | 248th of 367, with limits $3,607 a month | 43rd of 367, with limits $11,648 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that lives on its machine all month does not fit a CI job on either, and Buildkite also stops a Linux job after 8 hours; bochka has no session cap on paid plans. | 78th of 366, with limits $5,463 a month | 351st of 366, with limits $17,564 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 281st of 345, with limits $488 a month | 315th of 345, with limits $1,587 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Giving each of your users a sandbox means starting machines from your own backend with a URL they can reach, which bochka does by API and neither CI service offers. | 268th of 361, with limits $1,563 a month | 326th of 361, with limits $4,984 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 276th of 360, with limits $7,488 a month | 316th of 360, with limits $23,881 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 354th of 364, with limits $1,188 a month | 324th of 364, with limits $3,884 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Blacksmith is the easy pick if your CI already lives in GitHub Actions, Buildkite if you want its own pipelines and test tooling, and bochka fits CI you start by API on full VMs with Docker and /dev/kvm. | 78th cheapest of 377 $708 a month | 203rd of 377, with limits $2,318 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to stay up and keep its installed packages, but on both services everything outside sticky disks or cache volumes resets with each job. | 180th of 368, with limits $207 a month | 354th of 368, with limits $715 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Buildkite's desktop access is for a person debugging a macOS job and Blacksmith runners are headless, while bochka streams a 60fps Linux desktop with computer use. | 243rd of 369, with limits $1,088 a month | 70th of 369, with limits $3,500 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 131st of 367, with limits $588 a month | 142nd of 367, with limits $1,772 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 130th of 346, with limits $719 a month | 142nd of 346, with limits $2,354 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 262nd of 338, with limits $14,613 a month | 317th of 338, with limits $46,814 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 64th of 360, with limits $350 a month | 85th of 360, with limits $1,184 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month None of the three fits: all hosted shapes on Blacksmith, Buildkite and bochka are CPU only. | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Both run macOS, Blacksmith on Apple Silicon and Buildkite on Apple M4 with Xcode, while bochka runs Linux only. | 28th of 36, with limits $1,508 a month | 35th of 36, with limits $2,310 a month | Can't run it |
Key features
| Feature | Blacksmith | Buildkite | bochka |
|---|---|---|---|
| What it is | Drop-in GitHub Actions runners; jobs start from your workflows | A CI platform; hosted agents run jobs from Buildkite pipelines | A full VM per agent that you create, stop and start by API, CLI or SDK |
| Isolation | Firecracker microVM per job, destroyed with its state after the job | A virtual machine per job, destroyed with its data; Apple Virtualization framework on macOS | Full VM with its own kernel on dedicated AMD Ryzen 9 9950X hosts |
| How long a machine lives | Until the job ends; Testbox stops after 30 idle minutes by default | Up to 8 hours per job on Linux and 4 on macOS, longer on request | Until you stop it; no session cap on paid plans |
| Files after a run | Only sticky disk folders, up to 5 per job, evicted after 7 days unused | Only cache volume paths, saved after successful jobs, best effort, up to 14 days | The whole disk, saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | No; every machine starts fresh from a job | No; each job gets its own copy of the last committed cache volume | Fork a running sandbox with its disk |
| Docker and nested virtualization | Docker; nested virtualization on x64 Linux runners, not on ARM | Docker on Linux; macOS has the Docker CLI but no daemon; /dev/kvm not documented | Docker, root, systemd and /dev/kvm on every machine |
| Reaching the machine | Opt-in debug SSH, only for the user who triggered the job; no ports or URLs | No exposed ports; a terminal into a running job through Buildkite's UI or bk job ssh | HTTPS URLs for up to 50 ports, token-protected; raw TCP; dedicated IP |
| Desktop and computer use | Headless runners | Browser-based desktop access to debug a running macOS job; no computer-use API | 60fps Linux desktop with Chromium and computer use |
| Coding agents | Its own [code]smith agent from Slack, GitHub or a web UI; Testbox CLI in early beta for local agents | You install Claude Code in your image or step and run it as a pipeline job | Claude Code, Codex and five more preinstalled, with a prompt, steer and events API |
| Keeping API keys out | GitHub Actions secrets and OIDC tokens available inside the job | Anthropic proxy: jobs call it with the job token and Buildkite sets the key | Credentials proxy for model providers and any HTTPS API; switch a credential to another account when credits run out |
| Operating systems | Ubuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple Silicon | Linux (ARM64 on Enterprise), macOS on Apple M4 with Xcode, Windows Server 2022 in private preview | Linux (x64) only |
| Biggest machine | Up to 32 vCPU / 128 GB on Ubuntu x64, pay as you go | Linux up to 8 vCPU / 32 GB on Pro and 64 vCPU / 256 GB on Enterprise | Up to 16 vCPU / 32 GB on the $100 plan and above |
| Running at once | Docs say Blacksmith imposes no concurrency limits | Pro shares a pool of 256 Linux vCPU, up to 128 concurrent jobs | A sandbox count per plan, with starts queued above 90 per minute |
| Regions | US east, US west, Amsterdam and Frankfurt | US East Coast only | EU only (Germany, Finland, France), with EU data residency |
Every feature Battleships tracks (43)
| Feature | Blacksmith | Buildkite | bochka |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | No | Yes |
| Root | Yes | Not published | Yes |
| Systemd | No | No | Yes |
| Desktop GUI | No | Yes | Yes |
| Browser | Yes | Yes | Yes |
| Computer-use API | No | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | No | No | Yes |
| Automatic snapshots | No | Yes | Yes |
| Fork/clone | No | Yes | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | No | Yes | Yes |
| ≥24 h sessions | Yes | No | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | No | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | No | No |
| Egress allowlist | No | No | No |
| Secret proxy | No | Yes | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | No | No | Yes |
| Agent harness API | No | No | Yes |
| MCP server | No | Yes | No |
| GPU | No | No | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | No | Yes | No |
| BYOC | No | No | No |
| Open source | No | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | Yes | No | Yes |
| Audit logs | Not published | Not published | No |
| SSO | No | Yes | No |
| SOC 2 | Yes | Yes | No |
| HIPAA | No | No | No |
| EU data residency | Yes | No | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
We have no comparable CPU measurement for Blacksmith or Buildkite hosted agents: neither was part of the StarSling hpc-sandbox-benchmarks run. On that Node.js benchmark at 4 vCPU / 8 GB, bochka did 26.69 runs per second on dedicated AMD Ryzen 9 9950X hosts. Blacksmith says its runners use fast gaming CPUs, but there is no number measured the same way to set beside it.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Blacksmith and Buildkite have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Blacksmith, a month | Buildkite, a month | bochka, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $36 · Pay-as-you-go · with limits | $166 · Pro · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $207 · Pay-as-you-go · with limits | $715 · Pro · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $2,364 · Pay-as-you-go · with limits | $7,677 · Pro · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $663 · Pay-as-you-go · with limits | $2,204 · Pro · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $1,488 · Pay-as-you-go · with limits | $4,787 · Pro · with limits | $2,000 · $2000/mo · with limits |
Blacksmith: Blacksmith bills per minute of job time with no seat fee: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month. Sticky disks cost $0.50 per GB a month and a static IP $100 a month.
Buildkite: Buildkite Pro costs $30 per active user a month and includes 4,000 Linux vCPU minutes; after that Linux is $0.004 per vCPU minute and macOS $0.02, metered to the second. Enterprise pricing is custom.
Buildkite: Plans start at $20 a month, and the whole fee is usage credit spent per second on running sandboxes. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
None of the agent scenarios fit either service cleanly, because both are CI runners: packages installed outside sticky disks or cache volumes are gone after each job. Buildkite also stops a Linux job after 8 hours, so an agent that never stops needs restarts, and Pro's pool of 256 Linux vCPU makes 300 user sandboxes or 1,000 short jobs at once wait in a queue. Blacksmith's docs say it has no concurrency limit, but every start still has to be a GitHub Actions job, and bochka's $2,000 plan queues starts above 90 per minute for the 1,000-job burst.
| Scenario | Blacksmith | Buildkite | bochka |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Blacksmith when
your CI runs on GitHub Actions and you want faster jobs by changing one label, with no concurrency cap, US or EU regions, and macOS or Windows runners.
- A drop-in for GitHub Actions. If your CI already lives in GitHub Actions, you keep your workflows and switch by changing one runner label.
- No concurrency cap and no seat fee. For big test matrices, Blacksmith's docs say it sets no concurrency limit, and you pay per job minute with no per-user fee.
- US and EU regions. For teams that want runners near them, it runs in Ashburn, Phoenix, Amsterdam and Frankfurt.
Use Buildkite when
you want a full CI platform with its own pipelines and test tooling, self-hosted agents beside hosted ones, macOS on Apple M4, or Enterprise machines up to 64 vCPU.
- A whole CI platform. For teams that want managed CI outside GitHub: pipelines, test tooling, cache volumes, and self-hosted agents next to hosted ones.
- macOS on Apple M4 with Xcode. For iOS and Mac builds, its Mac hosted agents run current Xcode on Apple M4, up to 12 vCPU and 56 GB.
- Very large build machines. For heavy builds on Enterprise, Linux shapes go up to 64 vCPU and 256 GB, with ARM64 shapes too.
Where bochka fits
Blacksmith and Buildkite are both CI services: a machine exists only while a job runs, then it is destroyed. If what you are really running is agents that work for hours or days, start from your own backend and need to be reached from outside, bochka gives each one a full Linux VM that stays. For plain CI on GitHub Actions or Buildkite pipelines, either of them is the better fit.
A machine that keeps its state
On both services the machine is destroyed after each job, and only sticky disk folders or cache volume paths carry over, so an agent reinstalls its tools every run. bochka saves the whole disk every minute, lets you fork a running sandbox and has no session cap.
Reach in: URLs, ports and a desktop
Neither service exposes ports, only debug SSH or a terminal for a person, so you cannot open what an agent built. bochka gives HTTPS URLs on up to 50 ports, raw TCP and a 60fps desktop for computer use.
Agents you drive from outside, keys kept out
On Blacksmith and Buildkite an agent is a job you start from a workflow and read through logs, and only Buildkite proxies a key, for Anthropic alone. bochka ships seven agents with a prompt, steer and events API, and a credentials proxy for any HTTPS API.
Use bochka when what you run is agents, not CI jobs: one full Linux VM per agent that you start by API, that keeps its disk, hosts what it builds, shows a desktop and runs coding agents you drive from outside. Compare it one to one: bochka vs Blacksmith and bochka vs Buildkite.
Questions people ask
Is Blacksmith cheaper than Buildkite?
In every scenario we priced, Blacksmith comes out lower, for example $207 against $715 a month for one machine running all month and $2,364 against $7,677 for 50 machines in parallel. Neither fits these agent scenarios cleanly, and bochka comes to $26 and $317 for the same two.
Which is faster, Blacksmith or Buildkite?
We have no comparable measurement for either: neither was in the same CPU benchmark run. bochka did 26.69 runs per second on that benchmark at 4 vCPU / 8 GB.
Can Blacksmith and Buildkite run Docker?
Yes on Linux. Blacksmith also supports nested virtualization on x64 Linux runners but not on ARM, and Buildkite's macOS agents have the Docker CLI but no daemon. bochka has Docker, root, systemd and /dev/kvm on every machine.
How long can a job run on each?
A Buildkite hosted agent runs up to 8 hours per job on Linux and 4 on macOS, longer on request. A Blacksmith machine lives until its GitHub Actions job ends, and Testbox stops after 30 idle minutes by default. bochka has no session cap on paid plans.
Can I run an AI coding agent on Blacksmith or Buildkite?
On Buildkite you install Claude Code in a pipeline step, and its Anthropic proxy lets it call the API with the job token. Blacksmith has a Testbox CLI in early beta for local agents and its own [code]smith agent; its Sandboxes product is marked coming soon. On bochka, coding agents come preinstalled and you prompt and steer them through an API.
Do Blacksmith and Buildkite run in the EU?
Blacksmith runs in Amsterdam and Frankfurt as well as Ashburn and Phoenix. Buildkite hosted agents run only in US East Coast data centers. bochka runs only in the EU, in Germany, Finland and France.
Try bochka
Install the CLI, then run bochka new.
Product preview