The same email from GitHub again: you have used 100% of your included GitHub Actions minutes. Mid-month. A few private repositories: Android tests, lint, site deploys. At the same time, a Proxmox server in the closet does nothing for most of the day. Free resources sit a meter from my desk, and I wait for the next billing cycle.
The answer looks obvious: run your own runner. But a runner is a machine that runs someone else's code inside my home network. Even if the repositories are mine, the dependencies they download are not. So before I wrote the first line, I set one condition: the tool must not expose anything to the outside.
TL;DR
| Aspect | Idea |
|---|---|
| Problem | CI minute limits on private repositories |
| Resources | free CPU and RAM in the homelab (Proxmox) |
| Unit of work | one job = one fresh LXC container, deleted after the job |
| Network traffic | outbound connections only, zero ports open to the internet |
| Permissions | one GitHub App installed only on selected repositories |
| Credentials | a one-time token per job, nothing permanent on the runner |
| Capacity | one shared limit for all repositories, a FIFO queue |
The tool is called Spawnpool. It is one process in the home network (from here on: the controller) that asks GitHub for jobs and manages containers on Proxmox. This post describes the idea and the flow only, without code.
Why not a regular runner
A permanent runner, registered once and running for months, has two problems. I learned the first one the hard way.
The first is state that moves from job to job. After a few weeks, orphaned Docker volumes used up every inode on the runner's disk. The disk showed 32% usage, and Docker failed to create a file. Every job leaves something behind, and the next job inherits the mess.
The second is a long-lived credential. A permanent runner keeps its runner credentials on disk: the key it uses to log in to GitHub. A job, or a dependency that the job installs, can do something malicious. Then those credentials and the whole machine stay available to it after the job ends.
Autoscalers solve the first problem, but some of them add a third one. They rely on GitHub webhooks, so they need a public address or a tunnel. That is exactly what I did not want.
Three rules
One job, one container
Every job gets a fresh LXC container cloned from a template. The template is an image with the runner and Docker, prepared in advance. The container runs exactly one job, shuts down and is deleted. There is nothing to clean up, because nothing carries over.
Why LXC and not a virtual machine? A container clone starts in a few seconds and is lighter than a VM: no separate kernel and no OS boot. The price is that the container shares the kernel with the host. This is a trade-off I chose on purpose: LXC only for my own private repositories. If I ever let strangers' code in, for example pull requests to a public repository, every job would get a full virtual machine.
Nothing accepts connections from the internet
This is the core of the idea. Spawnpool connects to GitHub itself and asks whether jobs are waiting in the queue. It uses long polling: a request that stays open until GitHub has something to hand over. There are no webhooks and no public address. Spawnpool needs no port forward on the router.
The runner in the container works the same way: it connects to GitHub, takes the job and sends back the results. Every connection that crosses the home network boundary starts from inside the home.
The job containers live in a separate, isolated network that can reach the internet only. The one hole in that isolation leads to the controller, for the one-time token. The controller identifies the container by its address, and the Proxmox firewall (IP and MAC filter) stops a container from spoofing another one's address. The token works for one job. When the runner uses it or the time runs out, the token expires.
One shared limit and a fair queue
The server has one pool of RAM, and there are eight repositories. Each repository can have its own limit of parallel jobs, but above them sits one global limit: the homelab never runs more containers than that in total. A repository can ask for five containers at once, but if the global limit is three, at most three run in the whole homelab.
When there is no free slot, jobs wait in a FIFO queue. First come, first served, with no priorities. A job from a repository that reached its own limit waits without losing its place in the queue. The limit is not a counter that can become wrong: every slot is a reservation for a specific container. A slot becomes free only when Proxmox confirms that the container is gone.
Permissions from one place: a GitHub App
Eight repositories mean eight places to register runners, keep tokens and remember to rotate them. Instead, all permissions come from one source: one GitHub App, installed only on selected repositories.
The controller acts as that app. It signs a short-lived JWT with the private key and gets an installation token from
GitHub that is valid for one hour. With that token, it creates a scale set in every repository: a group of runners that
receives jobs with runs-on: spawnpool. With this token, the controller asks for jobs and requests a JIT (just-in-time)
configuration for every container, valid for one job. The controller uses no personal access tokens (PATs) and no
per-repository registration tokens, and the containers never see the app's key.
The app gets three permissions only. Repository administration is necessary, because without it you cannot manage runners. Metadata read access is required for every app. Actions read access only tells the controller how long a job waited in the queue. The webhook is off, because the app receives nothing.
Adding a repository to the pool is one line in a list in the configuration. Automation that I run from my own computer, with my own account, gives the app access to the repository and then creates the scale set in it. An app cannot add itself to a repository, so the controller does not do this step. Removing the line works the same way in reverse: the scale set disappears, and the app loses access to the repository. So one list says which repositories use the home CI, and there is no second list that can disagree with it.
How it works
Step by step:
- The controller asks GitHub for jobs. It acts as the GitHub App and keeps an open request for every repository.
GitHub answers when a job with
runs-on: spawnpoolappears. The controller checks the limit and puts the job in the queue. - The controller clones the template. When the limit has a free slot, it asks Proxmox for a new container from the template and starts it.
- The container gets a one-time token. After boot, the container asks the controller for its configuration. The controller got it from GitHub earlier, as the app. The container receives a token valid for one job. The token lives only in memory (tmpfs), the file disappears just before the runner starts, and the token stops working once used.
- The runner registers with GitHub and runs the job. This is again an outbound connection. Most jobs run in Docker images, so the template holds only what cannot run that way.
- The container disappears. After the job, the runner shuts down, and the controller deletes the container and frees the slot. The oldest waiting job gets it.
The controller has no database. After a restart, it rebuilds its state from two sources of truth: Proxmox (which containers exist) and GitHub (which jobs wait). It takes over the running containers that carry its tag. A guest in its pool without the tag is what is left after an interrupted clone. It is an orphan, and the controller deletes it.
Inspirations
I am not the first to come up with ephemeral runners on Proxmox. A few projects worth knowing:
- actions/actions-runner-controller: GitHub's official autoscaler and the reference for the whole category, but it needs Kubernetes.
- actions/scaleset: a Go client for the runner scale set API. This mechanism lets you ask GitHub for jobs without webhooks.
- JeffResc/actions-runner-scaleset-proxmox: the same idea on virtual machines, with pools of hot and warm VMs to shorten job start.
- klponce/proxmox-actions-runners: an ARC-style controller where every job gets its own VM instead of a pod.
- Shhinobi/proxmox-jit-runners: an orchestrator for runners in LXC containers with one-time JIT configuration, but built on webhooks.
- cocardoso/gh-runners-manager: ephemeral runners in LXC through the scale set API, with outbound connections only and a web interface. The closest to my approach.
- LukasK32/fleeting-plugin-proxmox: the GitLab counterpart, a plugin for the GitLab Runner autoscaler on Proxmox.
Downsides?
To be fair:
- LXC shares the kernel with the host. An escape from the container gives access to the Proxmox host, and network isolation does not stop that. So this model makes sense only for your own private repositories.
- The app's key is the most valuable secret. It gives administration rights on every repository in the pool, so it lives only on the controller and in the password manager, never in a container.
- Every job starts from zero. Clone, boot and registration cost time that a permanent runner does not lose. There is also no cache between jobs, so heavy tools (for me, JDK and the Android SDK) live in the template.
- The home uplink is the bottleneck. Pushing large Docker images over a home upload takes ages. I left image builds with push on GitHub-hosted runners, and tests, lint and deploys run at home.
- It is one more piece of infrastructure to maintain. The template must be rebuilt with every new runner version, because GitHub stops queueing jobs to runners that are not updated within 30 days of a release. The global limit must match the server's memory, because only the limit protects the server from swapping.
Summary
The idea fits in one sentence: one job gets one fresh container, and every connection leaves the home, never enters it. The rest follows from that. The shared limit protects the server's memory, and the FIFO queue keeps the order. The one-time token means nothing worth stealing stays behind after the job, and one GitHub App holds all permissions in one place.
Minute limits stopped being my problem, and the server in the closet is finally useful.
Have questions? Reach out on LinkedIn.