URL Shortener on Kubernetes (EKS)

A Python/FastAPI URL shortener being built in phases from an empty AWS account, targeting a Kubernetes cluster on AWS EKS provisioned by Terraform, with a GitHub Actions pipeline and CloudWatch observability in later phases. The centerpiece is docs/BUILD_LOG.md, a narrative record kept as the project progresses, not reconstructed afterwards. It documents the reasoning behind each decision and the mistakes as they happened. Phases 0 and 1 are complete (IAM bootstrap, the containerized service, local dev, and secret scanning); Phase 2 lays the Terraform remote-state foundation, and Phase 2.5 pivots the compute layer to Kubernetes on EKS.

Phase 0: IAM before any infrastructure

The first design question: why not just use root access keys for a solo project? Not because "least privilege is good practice," but because root literally cannot be scoped down by IAM policy the way a normal identity can. A leaked root key means total account takeover (billing changes, account closure, deleted CloudTrail logs), not a contained incident. The build log also records the tempting shortcut of attaching AdministratorAccess to the new terraform-deploy user "since we don't know every service we'll touch yet," rejected, since recreating admin on a differently-named identity defeats the point of moving off root. The working policy instead covers only what the current phase needs, extended when terraform apply fails with a specific AccessDenied. The user itself was built with five CLI calls in CloudShell (create-policy, create-user, attach-user-policy, create-access-key).

Two real mistakes followed immediately. First, aws configure on the local machine ended up authenticated as root, via a browser-based root login session, a flow that isn't even valid for a classic IAM user. The fix: a real access key for terraform-deploy under a named profile, with the root session cleared out of default entirely, so anything that forgets to specify a profile fails loudly instead of silently running as root. Second, while fixing that, the actual AccessKeyId/SecretAccessKey pair got pasted into a chat window. It was treated as compromised on the spot, the same principle as a secret hitting Slack or a log, and revoked immediately, with the replacement typed directly into aws configure in a real terminal. Knowing why a mistake is wrong in the abstract and not making it under real conditions turned out to be different skills.

Phase 1: the container, done properly

Why do the health endpoints need to be dedicated routes instead of health-checking /? Because on Kubernetes these become the pod's liveness and readiness probes: an automated control loop deciding whether to restart the container (liveness) and whether to send it traffic (readiness). Couple either to a business route and you get false positives (a static page returns 200 regardless of real health) or false negatives (a slow route triggers needless pod restarts under load). The service keeps /health (liveness) and /ready (readiness, where dependency checks will live) as separate, cheap, dependency-light routes that map directly onto the two probe types.

The Dockerfile started as a deliberately naive single stage (FROM python:3.12, no USER, COPY . .) and was rebuilt multi-stage: a builder stage installs dependencies via pip install --user so only /root/.local is copied into a python:3.12-slim runtime; a non-root appuser with a nologin shell; explicit --chown on both COPY layers; PYTHONUNBUFFERED=1 so container logs aren't buffered and lost; and PATH extended to ~/.local/bin, without which uvicorn isn't found at start. Result: a 230MB image (54.9MB unique content) versus the 1GB+ the naive build produces. Verification went beyond "looks right": 5/5 tests, every endpoint curled, Docker's healthcheck reporting healthy, and docker compose exec app whoami confirming the process actually runs as appuser.

Secret scanning, tested before trusted

The Phase 0 credential exposure led to a gitleaks pre-commit hook: .gitignore only stops new files being tracked; it does nothing for a secret already in history. The hook was tested before being trusted, which surfaced something non-obvious: a bare AKIA...-style key ID isn't flagged (gitleaks' AWS rule requires the real key's checksum structure, and the well-known AKIAIOSFODNN7EXAMPLE placeholder is explicitly allowlisted, so a naive test would have given false confidence). A high-entropy fake secret key was caught by the generic-api-key rule and the hook blocked the commit. The junk test commits were wiped with git update-ref -d HEAD before anything was pushed.

Phase 2: Terraform state, before any stack needs it

Before any real infrastructure, the Terraform state needed a home: a remote backend (an S3 bucket for the state file, a DynamoDB table for locking) so state isn't a local file that no one else, not even future-me on another machine, can see or safely share. The bucket was written with versioning (to recover from a bad apply), default server-side encryption, and a public-access block, each as its own resource, since the inline versioning and encryption blocks on aws_s3_bucket are deprecated in the current AWS provider. It also carries lifecycle { prevent_destroy = true }, so a stray terraform destroy can't wipe the bucket that holds every other stack's state.

The mistake caught here was a placeholder that almost shipped: the bucket was still named mycompany-terraform-state. Two problems, one obvious and one not. S3 bucket names are globally unique across every AWS account, not just mine, so a generic name like that would likely collide and fail with BucketAlreadyExists. The subtler one: the terraform-deploy IAM policy from Phase 0 was scoped to a specific bucket ARN, so a name that didn't match would fail apply with AccessDenied. Two pieces of infrastructure written phases apart still have to agree. Renamed to rtchan-devops-portfolio-tfstate, with the DynamoDB lock table (PAY_PER_REQUEST, LockID as its hash key, the exact schema Terraform's S3 backend expects) added where a bare TODO had been left.

Phase 2.5: the pivot to Kubernetes

The compute layer was originally going to be ECS Fargate; partway in, it became EKS — a deliberate pivot, reasoned out in the build log rather than quietly rewritten. The ECS choice had been sound for the project in isolation (less operational surface, no flat ~$73/month control-plane bill), but it optimized the wrong thing: the point of the project is closing real skill gaps for a job search, and Kubernetes was the gap. A managed EC2 node group was chosen over Fargate profiles for the same reason — it keeps the node management, autoscaling, and IAM surface that make the exercise worth doing. The added cost is stated plainly, not hidden, and the cluster is stood up on demand and torn down between sessions rather than left billing around the clock.