The SDLC of My Dreams

Experience has taught me¹ that writing a blog post about important parts of your tech setup is worthwhile, even if your future self is the only likely reader.

I’m very fond of the SDLC system I’ve recently built. It offers the high-level ergonomics I’ve always wanted without creating such a complex mental model that inspecting and debugging is difficult.

An overview:

• Prod – The production environment is what you’d expect. One or more containers for each service, with promotion gated by human approval.
• Staging – A fully separate environment, with one container for each service. Staging services talk only to other staging services – e.g. the staging frontend makes its API calls to the corresponding staging backend, not the prod backend. Similarly, staging apps use their own databases. Apps are deployed to staging automatically on any merge to the main branch. Staging services are accessible only when on VPN.
• Preview – This is the fun one: it’s possible to create new, ephemeral “preview” environments on demand through a web UI or a CLI command. Each repo with changes will be built into a fresh image, and the resulting services are also accessible on the VPN. Preview environments get their own databases that are seeded with the complete data from staging.

A full workflow looks like this:

1. For each service you intend to work on, make a new branch. Name all the branches the same (that’s important!).
2. To test your changes:
   • Push the branches to GitHub.
   • Create a preview environment via the web UI or CLI.
   • Test the affected services as appropriate: visiting them in the browser or hitting API endpoints with an HTTP client.
   • Iterate by pushing a fresh version of the app to GitHub. The preview apps are automatically rebuilt and redeployed as needed².
3. When satisfied:
   • Create a PR in each repo.
   • Merge those PRs.
   • Wait for the build to complete and for the new version to be deployed to staging.
   • Test it interactively in the full staging environment.
4. When ready to promote to prod:
   • Promote via the web UI or CLI.

Environments

Why so many environments?

Having some kind of pre-production environment (like “staging” in this case) is widely considered good practice; staging should mirror prod’s configuration, data, and scale closely enough that it’s possible to catch issues there before they get to production, where real users will be impacted.

Preview environments aren’t a necessity, but they sure are handy. It’s great to be able to test code before it gets to main. A lone developer can get away with doing all testing in staging, but that means a lot of code is merged before being confident it works.

Preview Environments

I’ve seen organizations try to solve this in three ways.

The most straightforward is to run the services locally, and sometimes this works reasonably well. It’s usually finicky and hard to onboard new devs though, because there’s a lot of configuration involved to get a Mac/Windows laptop running a bunch of apps and databases that expect to be on Linux machines in the cloud. Realistic data is a problem too; unless you’re going to copy a bunch of data over the network, there’s probably just some kind of “seed” script that adds enough records to the local database that the application is usable.

Another way is to have an always-on “development” environment, much like staging but with some way to deploy code that isn’t on main yet. Devs can push their changes to the development environment and test them there. Each database has an isolated development instance. But this approach is a nightmare once more than two developers are sharing the same environment: you’re all modifying the same apps and data, so it’s hard to rely on – often the whole environment ends up in a broken state because of just one bug in one app. Even if you communicate well and don’t deploy multiple changes to development at once, you still have to sit on your hands while you wait for someone else to finish their testing.

The last option is what I chose: ephemeral preview environments.

Functional Requirements of Preview Environments

Previews are actually trickier than they seem, but let’s start with the easy stuff.

Preview versions of applications need to be running the latest commit of the code on the target branch. They also need realistic, but isolated, data – plenty of good testing data copied into a fresh database. And they must apply any new migrations that have been added to the target branch.

It needs to be possible to preview changes from multiple services at once, because it’s common that a new feature involves both backend and frontend changes. So we need to “wire up” the preview services to each other – telling preview services to talk to other preview services, not the staging or prod instance.

But things get complicated when preview services rely on other services that aren’t in the current preview: for example, I might preview changes to an API that relies on an identity provider, but the IDP itself has no changes so it wouldn’t normally be included in the preview. What should happen then?

We can either create a preview version of the unmodified IDP anyway, or we can point the preview API to the staging IDP instead. I think both approaches are defensible, but I chose the fallback-to-staging approach. It means fewer unnecessary builds and deployments, but it is a bit more complicated to reason about³.

Last, preview environments have to be accessible to the developer but not to the public. I solved this by exposing them only via my Tailscale VPN.

So basically, preview services must:

• run the latest version of the code and apply the new migrations
• have isolated datastores, so we can test without worrying about breaking staging/prod
• talk to other preview services where possible, and otherwise to staging services
• be accessible on VPN only

Machinery

This part is the most technical. You may want to skip it if devops isn’t your thing.

Everything runs in Kubernetes. It’s not the simplest, but it’s incredibly flexible and has a huge support ecosystem.

ArgoCD is a tool for managing Kubernetes and keeping its current state in sync with some “desired” state. I use Image Updater to update that state when there’s a new staging image.

When I merge a change to main in one of my projects:

1. A GitHub webhook triggers a Google Cloud Build job, which builds an image, tags it with the Git SHA, and stores it in Google Artifact Registry.
2. Image Updater polls for new images. When it finds one, it updates Argo’s desired state⁴.
3. Argo tells Kubernetes to use the new image. Kubernetes executes a zero-downtime rollout, taking down the old pods only after the new ones are ready⁵.
4. The new staging version is now running and accessible on the VPN!

Promoting to prod is simple: I built a tool that updates the Argo desired state manually, to the image that’s running in staging. From there on, it’s just like staging: Argo and Kubernetes handle the rest.

To make sure prod services interact with other prod services, and staging services only with staging, I use Kustomize to parameterize the URLs each app needs to know about. Those URLs are then exposed to the running apps as environment variables, so I can run the exact same image in prod and staging but the application itself will know how to reach the right peer services by reading environment variables at runtime.

Preview environments are quite different; my environment manager application does the work itself. Here’s the flow:

1. I run a CLI tool and give it a branch.
2. It sends an HTTP request to my environment manager app.
3. The environment manager app looks through the GitHub repos for all my services (they’re listed in a static config file) to identify which ones have a branch of that name.
4. For each one found, it kicks off a Google Cloud Build.
5. Google Cloud Build tags its images with preview-<sha>⁶ and uploads them to Artifact Registry.
6. My environment manager polls until the build is done and gets the SHA.
7. Now it prepares databases, if needed, using a handy feature of Neon (my Postgres provider): branching. By branching the staging instance of each database used in the preview, it gets an identical copy of staging that’s isolated and ephemeral.
8. With all the images and databases ready, the environment manager now knows the right image tags and can compute the URLs that every preview service needs (some will be preview, some staging, as needed in the fallback-to-staging approach). It copies all database URLs and staging secrets directly into Kubernetes secrets and generates a Kustomize overlay with the image tags and service URLs, then applies the kustomized services with the Kubernetes API.
9. This triggers Kubernetes to roll out the new deployment, just as with staging/prod.
10. The environment manager polls for readiness of the preview services and eventually reports that the environment is ready.

Some Closing Thoughts

I spend a totally indefensible amount of time on developer experience for a hobbyist with zero colleagues and basically one customer (me).

But I’ve always been really drawn to this stuff. I’m an optimizer at heart, and SDLC tends to be one of the highest-friction parts of developing software. I’ve worked at a company with tens of billions in revenue, but devex so painful and half-baked that it became a major motivator to leave.

When it works well, it’s almost invisible; click a button, get your code deployed exactly where it should be, and test it seamlessly. Making that happen requires a lot of developer hours (even with AI!), but even more important is a clear vision of how it should work. It’s only obvious in retrospect.

I was really lucky to work in an engineering department that executed SDLC well, several jobs ago at ReviewTrackers. I learned a lot from that time, but I also identified some things that could be better. And this (still-ongoing) project was an opportunity to see if I could implement my vision. There are surely some rough edges I haven’t found yet, but right now I’m feeling pretty proud of what I’ve built.

This is a project that I would never have embarked on if I didn’t have supreme confidence in the skill of AI coding agents. Not only did they save me countless development hours, but without them I wouldn’t have taken the risk to build something so complex with technologies I don’t work with daily. Claude contributed 100% of the code for this work, and that’s not an exaggeration; I didn’t open an editor at any point.

Could agents have done this without me? Could I have given them a requirements document and let them run? No, not yet – I needed to guide them on ergonomics, and I insisted on being very involved in the architecture. But it feels like they’re very close.

And that’s another reason I decided to build this now: agents are increasingly capable of operating with little or no human feedback, but they are very sensitive to verifiability. Preview environments that can be started from the command line are a perfect way to let agents test their own changes and iterate more effectively.

So I suppose this project has an ironic undertone. My efforts to improve the developer experience are ultimately most valuable as a way to obviate the developer.

You can find the code for my environment manager here, though I may eventually make it private. Sorry.

Subscribe to new posts

Get notified when I publish something new.

Subscribe

Done

close