# Running a Fleet of Firecracker microVMs for eve.dev Agents, Part 4: The Control Plane Source: https://entuit.com/blog/eve-firecracker-fleet-control-plane Published: 2026-07-19 Tags: aws, aws-cdk, firecracker, eve, agents, typescript, lambda, dynamodb, ai-infrastructure Part 4 of the hands-on series. We build the scheduler that turns a deploy request into a running agent — API Gateway and Lambda over a DynamoDB registry of hosts, agents, and placements, a race-safe bin-packing placement algorithm, and EventBridge decoupling the API from the work, all in AWS CDK. --- Three parts in, we have every piece except the one that makes it a platform. [Part 2](/blog/eve-firecracker-fleet-host-fleet) gave us hosts that announce their free capacity into a registry. [Part 3](/blog/eve-firecracker-fleet-agent-images) gave us bootable agent images sitting in S3. What is missing is the thing in the middle: something that takes "deploy `my-agent` version `a1b2c3d`," decides *which host* has room for it, records that decision, and tells that host to boot the microVM. That is the control plane, and it is what turns a pile of machines and images into a system you can deploy to. By the end of Part 4, a single HTTP call — `POST /agents` — will place an agent onto a host and bring it up, and a `GET` will tell you where it landed and whether it is running. This is the part where `deploy my-agent` finally produces a running agent. ## What the Control Plane Owns Keep the responsibilities narrow and the design stays simple. The control plane owns exactly four things: - **The API** — a small HTTP surface: deploy an agent, destroy one, list them, get one's status. - **The registry** — three DynamoDB tables that are the system's source of truth: which hosts exist and how much they have free, which agents are supposed to be running, and where each one is actually placed. - **Placement** — the algorithm that, given an agent's resource request, picks a host with room and reserves that capacity without racing other placements. - **Commanding the host** — telling the chosen host's host-agent (on the port-`7000` management path from Part 1) to boot the microVM from the right image. Critically, the control plane does **not** run agents, touch Firecracker, or move network traffic. It decides and records; the host-agents execute; the routing layer (Part 5) carries traffic. That separation is what lets the whole thing be serverless — there is no long-running scheduler process to babysit, just Lambdas that fire when there is a decision to make. ## The Registry: Three Tables Everything hangs off the data model, so start there. Three tables, each with one job. ```ts // lib/control-plane-stack.ts (registry) import * as cdk from "aws-cdk-lib"; import { Construct } from "constructs"; import * as ec2 from "aws-cdk-lib/aws-ec2"; import * as dynamodb from "aws-cdk-lib/aws-dynamodb"; import * as lambda from "aws-cdk-lib/aws-lambda"; import * as apigw from "aws-cdk-lib/aws-apigatewayv2"; import * as integrations from "aws-cdk-lib/aws-apigatewayv2-integrations"; import * as events from "aws-cdk-lib/aws-events"; import * as targets from "aws-cdk-lib/aws-events-targets"; import { config } from "./config"; export interface ControlPlaneStackProps extends cdk.StackProps { vpc: ec2.IVpc; controlPlaneSg: ec2.ISecurityGroup; } export class ControlPlaneStack extends cdk.Stack { constructor(scope: Construct, id: string, props: ControlPlaneStackProps) { super(scope, id, props); const { vpc, controlPlaneSg } = props; const ddbProps = { billingMode: dynamodb.BillingMode.PAY_PER_REQUEST, removalPolicy: cdk.RemovalPolicy.DESTROY, // tutorial; RETAIN in prod }; // Hosts — the capacity registry the host-agents write to (Part 2). const hosts = new dynamodb.Table(this, "Hosts", { tableName: `${config.fleetName}-hosts`, partitionKey: { name: "hostId", type: dynamodb.AttributeType.STRING }, ...ddbProps, }); // Agents — desired state: what should be running, at what size and version. const agents = new dynamodb.Table(this, "Agents", { tableName: `${config.fleetName}-agents`, partitionKey: { name: "agentId", type: dynamodb.AttributeType.STRING }, ...ddbProps, }); // Placements — the binding: which agent is on which host, right now. const placements = new dynamodb.Table(this, "Placements", { tableName: `${config.fleetName}-placements`, partitionKey: { name: "agentId", type: dynamodb.AttributeType.STRING }, ...ddbProps, }); // Find every agent on a given host — needed to reschedule when a host dies (Part 7). placements.addGlobalSecondaryIndex({ indexName: "byHost", partitionKey: { name: "hostId", type: dynamodb.AttributeType.STRING }, }); // ...event bus, lambdas, and API below } } ``` The `hosts` table is the one Part 2's host-agents already write to — the control plane is its reader. A host row carries its schedulable and free `vcpu`/`mem`, its `lastHeartbeat`, and its private IP (the address the scheduler will call on port 7000). The `agents` table is *desired* state — "this agent, this version, this size, should be running." The `placements` table is *actual* state — the binding of agent to host. Keeping desired and actual in separate tables is what makes reconciliation in Part 7 tractable: the health loop's whole job is making actual match desired. ## EventBridge: Decouple the API From the Work The naive design has the API Lambda do everything synchronously — receive the request, scan hosts, place, command the host, then respond. That makes the caller wait on a microVM boot and gives you nowhere to retry if a host command fails. Instead, we split it with an event bus. The API does the fast part (validate, record desired state, emit an event) and returns immediately; a **scheduler** Lambda does the slow part (place and command) off the event. ```ts const bus = new events.EventBus(this, "FleetBus", { eventBusName: `${config.fleetName}-events`, }); ``` This buys three things. The API stays fast — `POST /agents` returns `202 Accepted` in milliseconds. Failures get automatic retries — a host command that fails is a Lambda invocation EventBridge will retry, not a dropped request. And the same events (`agent.deploy.requested`, `agent.placed`, `agent.running`, `agent.unplaceable`) become the audit trail and the hooks Part 7's health loop subscribes to. The bus is the spine the whole lifecycle hangs off. ## The Two Lambdas There are two functions. The **API handler** is a plain HTTP Lambda — no VPC, it only touches DynamoDB and the bus. The **scheduler** must live *inside the VPC*, in the `controlPlaneSg` from Part 1, because it makes the one call that reaches into private space: the HTTP command to a host-agent on port 7000. ```ts const env = { HOSTS_TABLE: hosts.tableName, AGENTS_TABLE: agents.tableName, PLACEMENTS_TABLE: placements.tableName, BUS_NAME: bus.eventBusName, }; // API handler — fast path. Records desired state, emits an event, returns. const api = new lambda.Function(this, "ApiFn", { runtime: lambda.Runtime.NODEJS_20_X, handler: "api.handler", code: lambda.Code.fromAsset("lambda/api"), environment: env, }); agents.grantReadWriteData(api); placements.grantReadData(api); bus.grantPutEventsTo(api); // Scheduler — slow path. In the VPC so it can reach host-agents on :7000. const scheduler = new lambda.Function(this, "SchedulerFn", { runtime: lambda.Runtime.NODEJS_20_X, handler: "scheduler.handler", code: lambda.Code.fromAsset("lambda/scheduler"), environment: env, timeout: cdk.Duration.seconds(30), vpc, vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }, securityGroups: [controlPlaneSg], }); hosts.grantReadWriteData(scheduler); agents.grantReadWriteData(scheduler); placements.grantReadWriteData(scheduler); bus.grantPutEventsTo(scheduler); // Deploy/destroy events drive the scheduler. new events.Rule(this, "LifecycleRule", { eventBus: bus, eventPattern: { source: ["eve.fleet"], detailType: ["agent.deploy.requested", "agent.destroy.requested"], }, targets: [new targets.LambdaFunction(scheduler)], }); // HTTP API in front of the API handler. const httpApi = new apigw.HttpApi(this, "ControlApi", { apiName: `${config.fleetName}-control`, }); const integ = new integrations.HttpLambdaIntegration("ApiInteg", api); httpApi.addRoutes({ path: "/agents", methods: [apigw.HttpMethod.POST, apigw.HttpMethod.GET], integration: integ, }); httpApi.addRoutes({ path: "/agents/{id}", methods: [apigw.HttpMethod.GET, apigw.HttpMethod.DELETE], integration: integ, }); new cdk.CfnOutput(this, "ControlApiUrl", { value: httpApi.apiEndpoint }); ``` Wire it into the app alongside the earlier stacks, passing the network handles from Part 1: ```ts // bin/app.ts — additions const network = new FleetNetworkStack(app, "EveFleetNetwork", { env }); new HostFleetStack(app, "EveFleetHosts", { env, vpc: network.vpc, hostSg: network.hostSg }); new ArtifactStack(app, "EveFleetArtifacts", { env }); new ControlPlaneStack(app, "EveFleetControlPlane", { env, vpc: network.vpc, controlPlaneSg: network.controlPlaneSg, }); ``` ## The API Handler The API handler is deliberately dumb. It never places anything — it records what *should* be true and lets the scheduler make it so. ```ts // lambda/api/api.ts export async function handler(event) { const method = event.requestContext.http.method; const id = event.pathParameters?.id; if (method === "POST") { // Deploy: record desired state, fire an event, return immediately. const { name, version, vcpu = 2, mem = 1024 } = JSON.parse(event.body); const agentId = `${name}-${version}`; await putAgent({ agentId, name, imageVersion: version, vcpu, mem, desired: "running" }); await emit("agent.deploy.requested", { agentId }); return json(202, { agentId, status: "scheduling" }); } if (method === "DELETE" && id) { // Destroy: flip desired state, fire an event; the scheduler tears it down. await updateAgentDesired(id, "stopped"); await emit("agent.destroy.requested", { agentId: id }); return json(202, { agentId: id, status: "stopping" }); } if (method === "GET" && id) return json(200, await getStatus(id)); // joins agent + placement if (method === "GET") return json(200, await listAgents()); return json(405, { error: "method not allowed" }); } ``` `POST` returns `202` with an `agentId` the moment desired state is written — the actual boot happens asynchronously. `GET /agents/{id}` joins the agent's desired state with its placement row so the caller sees both "supposed to be running" and "running on host i-0abc… since 12:04." ## Placement: Bin-Packing, Race-Safe This is the heart of it. The scheduler receives the `agent.deploy.requested` event, looks up the agent's size, and finds a host. Two design choices matter. First, **bin-packing**: among hosts with enough room, prefer the *tightest* fit — the one that will have the least left over — so agents pack densely onto fewer hosts and the expensive bare-metal boxes stay busy instead of half-empty. Second, and non-negotiable, **race-safety**: two placements happening at once must never both claim the last slice of a host's capacity. ```ts // lambda/scheduler/scheduler.ts export async function handler(event) { const { agentId } = event.detail; if (event["detail-type"] === "agent.destroy.requested") return teardown(agentId); const agent = await getAgent(agentId); // { name, imageVersion, vcpu, mem } // Candidate hosts with room, tightest-fit first (bin-pack for density). const candidates = (await scanHosts()) .filter((h) => h.freeVcpu >= agent.vcpu && h.freeMemMib >= agent.mem) .sort((a, b) => (a.freeVcpu - agent.vcpu) - (b.freeVcpu - agent.vcpu)); for (const host of candidates) { try { await reserveCapacity(host.hostId, agent.vcpu, agent.mem); // atomic; throws if raced await putPlacement({ agentId, hostId: host.hostId, version: agent.imageVersion, status: "booting" }); await commandHost(host, agent); await emit("agent.placed", { agentId, hostId: host.hostId }); return; } catch (e) { if (e.name === "ConditionalCheckFailedException") continue; // lost the race — next host throw e; } } // Nobody had room. Part 7 subscribes to this to trigger a scale-out. await emit("agent.unplaceable", { agentId }); } // The capacity reservation IS the lock. The condition makes oversubscription impossible. async function reserveCapacity(hostId, vcpu, mem) { await ddb.update({ TableName: process.env.HOSTS_TABLE, Key: { hostId }, UpdateExpression: "SET freeVcpu = freeVcpu - :v, freeMemMib = freeMemMib - :m, runningAgents = runningAgents + :one", ConditionExpression: "freeVcpu >= :v AND freeMemMib >= :m", ExpressionAttributeValues: { ":v": vcpu, ":m": mem, ":one": 1 }, }); } ``` That `ConditionExpression` is the whole game. The reservation and the capacity check are a single atomic DynamoDB operation: if another placement got there first and the host no longer has room, the condition fails, the update is rejected, and we simply try the next host. No locks, no scheduler singleton, no oversubscribed hosts — the database enforces correctness under any amount of concurrency. This is what lets placement be a stateless Lambda that can run many copies at once. ## Commanding the Host With capacity reserved and a placement row written, the scheduler tells the host to boot. This is the one call that crosses into private space, which is exactly why the scheduler runs in the VPC under `controlPlaneSg` — Part 1's security group lets that group reach hosts on port 7000, and nothing else can. ```ts async function commandHost(host, agent) { const res = await fetch(`http://${host.privateIp}:7000/microvms`, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ agentId: agent.agentId, image: `agents/${agent.name}/${agent.imageVersion}/rootfs.ext4`, // S3 key from Part 3 vcpu: agent.vcpu, memMib: agent.mem, }), }); if (!res.ok) throw new Error(`host ${host.hostId} rejected boot: ${res.status}`); } ``` The host-agent takes it from here exactly as Parts 2 and 3 described: pull that rootfs key from the artifact bucket (cached locally), write the agent's secrets into the microVM's MMDS, and boot Firecracker. When the microVM is up, the host-agent flips the placement row to `running` and emits `agent.running`. If the boot fails, it frees the capacity it reserved and emits `agent.boot.failed` — and because everything went through EventBridge, that failure is a retriable event, not a silent dead agent. ## Deploy It and Watch an Agent Land ```bash cdk deploy EveFleetControlPlane ``` Grab the `ControlApiUrl` output, then deploy the sample agent you built in Part 3: ```bash curl -sX POST "$CONTROL_API/agents" \ -d '{"name":"my-agent","version":"a1b2c3d","vcpu":2,"mem":1024}' # → 202 {"agentId":"my-agent-a1b2c3d","status":"scheduling"} # A moment later, ask where it landed: curl -s "$CONTROL_API/agents/my-agent-a1b2c3d" # → {"agentId":"my-agent-a1b2c3d","desired":"running","status":"running", # "hostId":"i-0abc123","version":"a1b2c3d"} ``` Behind those two calls: the API wrote desired state and emitted an event; the scheduler bin-packed the agent onto a host and atomically reserved its capacity; the host-agent pulled the image, injected secrets, and booted the microVM; and the placement row flipped to `running`. You now have an agent alive on the fleet, chosen and placed by the platform rather than by you. What you *cannot* do yet is send it a request — it has no address the outside world can reach. That is Part 5. ## Tearing Down The control plane comes down before the hosts and network it coordinates: ```bash cdk destroy EveFleetControlPlane ``` One caveat: destroying the control plane removes the tables, but any microVMs already running on hosts keep running — the host-agents do not stop them just because the scheduler went away. Drain agents first (`DELETE /agents/{id}` for each, or drop the host fleet's desired count to zero from Part 2) before tearing the control plane down, or you will leave orphaned microVMs on hosts with no registry that remembers them. ## What's Next The platform can now make decisions: an HTTP call records intent, EventBridge carries it, a race-safe bin-packing scheduler picks a host and reserves its capacity, and the host-agent boots the microVM — all serverless, all reconcilable. An agent is *running* on the fleet. It just is not *reachable*. In **[Part 5](/blog/eve-firecracker-fleet-routing)** we build the routing layer that fixes that: per-microVM tap devices and NAT on each host so a guest has a real address, an ALB out front, and a per-agent subdomain scheme so a request to `my-agent.fleet.example.com` finds the exact microVM the control plane placed — wherever it landed. That is the step that turns "an agent is running somewhere" into "you can talk to it." If you want the ground-level mechanics behind the boot the host-agent performs here, [Running an eve agent in a Firecracker microVM on AWS](/blog/eve-agent-firecracker-aws-cdk) walks the single-microVM version by hand.