A few weeks ago I set out to build something beyond “install Argo CD and sync an app.” I wanted a setup that actually mirrored how real platform teams work — separate repos for code and deployment config, proper dev/test/prod promotion, canary rollouts, and none of the shortcuts that make demo projects fall apart the moment you touch them for real.
This post walks through what I built, why it’s structured the way it is, and — just as importantly — the real bugs I hit along the way, because that’s usually where the actual learning happens.
The Starting Point: Why Two Repositories?
My first version had everything in one repo — application source code sitting right next to Terraform, Helm charts, and Argo CD configuration. It worked, but it didn’t feel right, for a simple reason: a developer pushing a one-line code fix should never be one accidental commit away from touching production sync policies or RBAC rules.
So I split it into two:
demo-app— just the application. A small Flask service, a Dockerfile, and a CI workflow. Nothing else.gke-argocd-gitops— everything about how and where that app runs. Terraform, the Helm chart, Argo CD’sApplicationandAppProjectobjects, RBAC, notifications.
Argo CD only ever watches the second repo. It has no idea the first one exists. That separation alone eliminates an entire category of “oops, wrong permissions” incidents.
The Foundation: Terraform
The bootstrap/ folder in the GitOps repo doesn’t create the GKE cluster — that already exists, built by a separate infra project. What it does create:
- Argo CD and Argo Rollouts, installed via Helm
- Three namespaces —
dev,test,prod— each with aResourceQuota(a hard CPU/memory ceiling) and aNetworkPolicy(so adevpod can never reach aprodservice, even by accident) - A Workload Identity Pool, so GitHub Actions can authenticate to Google Cloud without a single downloaded key file anywhere
That last point mattered a lot to me. No JSON key sitting in GitHub Secrets waiting to leak. Just a short-lived, auto-expiring token exchanged fresh on every workflow run.
GitHub Actions: Build Once, Promote Everywhere
This is the principle I was most careful to get right, because it’s easy to get subtly wrong.
The demo-app repo’s CI workflow builds a Docker image exactly once, tags it with the Git commit SHA, scans it with Trivy, and pushes it. Then — and this is the part that took real thought — it commits that new tag into the other repo’s values-dev.yaml.
Cross-repo commits sound simple until you ask “with what credential?” GitHub’s built-in token only has permission inside its own repo. My first instinct was a personal access token. It worked — for a while. Then I remembered: fine-grained PATs are forced to have an expiry, capped at one year. That’s a landmine with a fuse nobody’s watching.
I switched to a GitHub App instead. Its installation token gets minted fresh on every single run, lives for about an hour, and then disappears. Nothing to remember to rotate on a calendar.
From dev, promotion to test and prod happens through a separate, manual workflow — promote.yml. It never rebuilds anything. It verifies the source tag actually exists (catches typos instantly instead of three steps later), retags it, and opens a Pull Request. A human still has to merge that PR before anything moves. For production, it also resolves the image’s content digest and pins it — a tag can technically be repointed at a different image later, but a digest physically cannot.
Argo CD: The Part That Actually Deploys Things
Argo CD’s job is deceptively simple to state: watch a Git repo, make the cluster match it. In practice, understanding how it does that comes down to a handful of components.
The argocd-repo-server clones the repo and renders the Helm chart into plain YAML. The argocd-application-controller is the real brain — it compares that rendered YAML against what’s actually running, and reconciles the difference. Everything else (the API server, Redis cache, notifications controller) supports those two.
Instead of manually creating three Application objects, I used the App of Apps pattern — one root Application that watches a folder, and Argo CD generates the child Applications automatically. Adding a fourth environment later means adding one YAML file, nothing more.
The sync policy is where dev/test and prod genuinely diverge. Dev and test are fully automated — prune: true, selfHeal: true — Argo CD applies changes the moment it sees them and reverts anyone who tries to kubectl edit around it. Prod has no automated block at all. Argo CD just shows a diff and waits. A release manager has to explicitly run argocd app sync after reviewing it.
Production also runs on an Argo Rollouts canary instead of a plain Deployment — traffic shifts in steps (20% → pause → 50% → pause → 100%), with a genuine window at each pause to catch a bad release before it reaches everyone. If something looks wrong mid-rollout, one command aborts it instantly, no new Git commit required.
Branching Strategy: Deliberately Asymmetric
I almost mirrored the same branch names across both repos out of habit, and I’m glad I stopped myself.
demo-app uses plain trunk-based development — one main branch, short-lived feature branches, nothing fancier. It doesn’t need to know about environments at all; that’s not its job.
gke-argocd-gitops uses develop → test → main, mapped directly to dev/test/prod. main here is the most protected branch in the whole setup, exactly the way GitHub already assumes by default for any repo.
The asymmetry is the point. Application developers get a boring, predictable workflow. Environment promotion — a genuinely different concern — lives entirely in the deployment repo, touched only by whoever runs the promotion workflow.
The Bugs That Taught Me the Most
A few things broke in ways that were more instructive than anything that worked on the first try.
AppProject silently blocking a valid resource. My test Application sat OutOfSync for a while with a ServiceAccount the Helm chart clearly created. The error, once I found it: resource :ServiceAccount is not permitted in project. I’d forgotten to whitelist that resource kind in the AppProject‘s namespaceResourceWhitelist. Argo CD doesn’t fail loudly on this — it just quietly refuses to create anything not on the list.
A ResourceQuota rejecting an unrelated pod. A small PreSync hook Job kept failing with must specify limits.cpu. The namespace’s ResourceQuota set a hard ceiling, and once a quota specifies limits at all, every pod in that namespace must declare its own requests/limits explicitly — including a throwaway busybox container I hadn’t thought to configure.
A non-fast-forward push error in the promotion workflow. My first version of promote.yml reused the same branch name across runs. A second run collided with a leftover branch from an earlier attempt. The fix was obvious in hindsight — include the workflow run ID in the branch name, so every run gets a name that can never collide.
Every one of these was a five-minute fix once diagnosed. The value was in seeing why each one happened, not just patching the symptom.
What I’d Tell Someone Starting This Today
Get the boring part right first — the Terraform bootstrap, the Workload Identity setup, the namespace isolation — before touching a single Argo CD Application. Everything downstream depends on that foundation actually being solid.
And don’t skip the “why” behind patterns like build-once-promote-everywhere or the App of Apps structure. They look like extra ceremony for a small demo project, right up until the moment they save you from shipping the wrong thing to production.
Want the full step-by-step setup guide, every Terraform file, and the complete GitHub App configuration walkthrough? Check out the repos:
- gke-argocd-gitops — Terraform, Helm chart, Argo CD config
- demo-app — the application source and its CI workflow
Questions about any specific part of this setup — the Terraform, the GitHub Actions cross-repo bridge, or the Argo Rollouts canary config? Happy to go deeper in the comments.