Hot failover for self-hosted inference. Zero idle GPUs.

Meet Haru: the GPU orchestration layer we are building under Arkor. Your standby GPUs train LoRA while they wait, and take over serving in seconds when the active region fails.

Alpha · Open source on GitHub

View on GitHub

A standby that earns its keep.

Classic hot standby means paying for GPUs that do nothing. Haru keeps the standby's vLLM servers in level 1 sleep: the process stays alive with model weights parked in CPU RAM, so the VRAM is free. Preemptible LoRA training runs in that freed VRAM until the moment it is needed for serving.

Active domain

Serves traffic

OpenAI-compatible inference across one or many models per GPU, behind a single routing pointer. Heartbeats watch every model, not just the host.

Standby domain

Sleeps and trains

Same models, resident but asleep. The GPUs spend their idle hours on checkpoint/resume LoRA training that Haru preempts without hesitation during failover.

Failover as a verified sequence, not a script.

Every step is a checked state transition that must prove it happened before the next one runs. Crashed steps resume, racing controllers converge, and routing only ever moves atomically.

Step 1

Stop training

The standby's LoRA job gets SIGTERM, a checkpoint grace window, then SIGKILL. Failover never waits for a perfect checkpoint.

Step 2

Verify VRAM

Haru confirms the training GPUs actually released their memory before anything tries to claim it.

Step 3

Wake vLLM

Weights come back from CPU RAM, not disk. Level 1 sleep makes this the fastest possible wake path.

Step 4

Probe every model

A synthetic completion must succeed for every model the fleet routes, not just the ones that happen to respond.

Step 5

Flip the pointer

Routing moves in a single database compare-and-swap. A promotion that fails before this step never moves traffic.

A hardware abstraction layer for GPU fleets.

Haru plays the role a HAL plays in an operating system: a small, stable, provider-neutral surface over messy GPU infrastructure. Fleets, domains, GPU slots, models and placement are all data. Placement goes through SkyPilot, so two failure domains can live in different regions or entirely different clouds, and nothing in Haru hard-codes a vendor. The name also reads as haru (spring: 春) in Japanese.

Own state store

Fleet truth lives in Postgres, advanced only by single-statement compare-and-swap. No distributed locks, no leader election.

Small HTTP API

Promote, demote, reconcile, route intent, and an OpenAI-compatible chat proxy. A product control plane consumes Haru without inheriting its internals.