One request, one suitable machine.
No exotic distributed splitting. The scheduler assigns each independent inference request to a single machine that can actually run it — and proves it can before traffic arrives.
The flow
- A developer sends a request to
POST /v1/chat/completionswith a model, messages and an API key. - The scheduler checks the fleet. It tracks each worker’s GPU model, VRAM, which models it can serve, coarse region, latency, reliability history, uptime and queue length.
- One worker is chosen on model compatibility first, then available VRAM, reliability, expected cost and latency.
- The worker runs the job in its sandboxed inference runtime and streams or returns the result. Job states: pending, assigned, running, completed, failed, cancelled — with timeouts and retries on failure.
- Usage is reported — input tokens, output tokens, time to first token, total time — so both sides can reconcile cost.
What we deliberately do not do
We do not split one model across geographically distributed consumer GPUs. The latency and failure modes make that impractical for inference; whole requests on single machines is the design, and it stays the design until evidence says otherwise.
For providers: the agent
The desktop agent (Windows first) detects your GPU and VRAM, measures system resources, runs a local benchmark, reports capacity to the coordinator, and shows utilisation, VRAM, temperature where supported, uptime, completed jobs and earnings once real payments exist. You can pause participation or uninstall at any time.
Safety rules
- Never execute arbitrary code supplied by marketplace customers on contributor machines — sandboxed inference workloads only.
- Signed application updates, strict resource limits, least-privilege permissions, secure authentication, encrypted communications, workload validation, emergency shutdown, explicit user consent.
- The agent never secretly uses your GPU, never hides activity, and never accesses personal files.
- A machine that is merely connected earns nothing — only successfully completed paid work is compensated, once billing exists.
The economics question
The whole idea stands or falls on measured cost per million tokens. Our benchmarking records tokens per second, time to first token, GPU utilisation, VRAM, and estimated electricity and hardware operating cost — then compares against established inference providers. No benchmark result is published until it is actually measured. See status and the roadmap.