DevOps Engineering Portfolio

Three sanitized case studies from production work at Keel Digital. Each documents the actual constraints and the reasoning behind the architecture that answered them, with before/after diagrams rather than just the finished state. Hostnames, network ranges, and service names are generalized; the architecture and decisions are real. The full write-ups live on GitHub.

Proxmox bare-metal migration

The dev environments ran on 14 DigitalOcean droplets, each billing independently and each with its own public IP. That meant 14 internet-facing machines that needed individual firewall hygiene, patching urgency, and monitoring as if they were production, for what were functionally internal environments.

The replacement is a single OVHcloud bare-metal server running Proxmox VE with two bridges: vmbr0 carries the public interface and connects only to a minimal gateway VM; vmbr1 is an internal RFC1918 bridge where every dev VM lives, with no route out except through the gateway. The gateway masquerades outbound traffic via nftables, and inbound access is deliberately narrow: SSH to the gateway, then onward, with no direct port-forwards to dev VMs by default. Because all egress now passes one chokepoint, logging and restricting what dev environments can reach externally became a single config file instead of 14.

The migration favored rebuild over copy: services already deployed via configuration were rebuilt fresh (which doubled as a test that the environment setup was actually reproducible) while databases were synced ahead of time and re-synced during cutover. New environments ran in parallel with the droplets, and nothing was destroyed until its replacement was confirmed working. Results: ~$1,500 CAD/month eliminated, attack surface down from 14 public IPs to 1, dedicated CPU and NVMe in place of shared-vCPU droplets, and Proxmox snapshot/restore where backups had been per-droplet and ad hoc. The retrospective is candid: the rebuild-vs-migrate classification took longer than the migration itself, and internal DNS should have existed before the first VM moved.

Troubleshooting: the MTU blackhole

Not everything went cleanly. The migrated environment passed every basic connectivity check but failed in use: ping and SSH connected fine, then froze the instant a command returned a large burst of output, and the webserver completed its TCP and TLS handshakes before page loads hung indefinitely. Nothing showed up in logs, because nothing was erroring — packets were simply vanishing. The contradiction, ping works but anything real hangs, pointed at packet size, and DF-bit pings confirmed it: ping -M do -s 1472 (a full 1500-byte frame) hung while smaller payloads went through. Somewhere in OVH's routed path for the dedicated IP, frames over ~1400 bytes were being dropped with no ICMP fragmentation-needed response — a Path MTU Discovery blackhole, where the sender just retransmits full-size segments into the void until the connection times out.

MSS clamping on the gateway (iptables -t mangle ... TCPMSS) didn't resolve it: the FORWARD chain only rewrites routed guest traffic, not the host's own SSH sessions, and clamp-to-pmtu would inherit the same broken PMTUD anyway. The reliable fix was lowering the MTU to 1400 on both the Proxmox host interfaces and the guest NICs, leaving ~100 bytes of headroom under the unknown true path MTU for OVH's encapsulation overhead; the SSH and webserver timeouts cleared immediately and permanently. The takeaway: MTU blackholes are invisible to standard health checks because handshakes and pings use small packets — connects fine, hangs under load is the fingerprint, and ping -M do is the two-minute test that proves it. Read the case study

Self-managed OpenSearch SIEM

A HIPAA environment with FedRAMP in scope, and observability scattered across CloudWatch, Keycloak, and Zabbix with no way to correlate between them. Three constraints shaped the build: logs containing PHI-adjacent metadata couldn't leave the AWS account in transit to a third-party SIEM, putting managed SaaS options off the table without significant legal review; per-GB commercial licensing gets expensive fast at 40+ services of continuous log volume; and detections needed to be owned and tuned in-house, not fit into a vendor's fixed rule taxonomy. Self-managed OpenSearch on AWS answered all three.

Ingestion runs Fluent Bit as a sidecar on every ECS task into a central Logstash cluster, which normalizes fields, enriches at write time (IP geolocation, instance metadata, Keycloak realm context; cheaper once on write than repeatedly at query), and routes log types to index patterns with different retention. ISM policies tier the lifecycle: hot on SSD for 7 days, warm and compressed for 30, then snapshots to encrypted S3 for the one-year retention HIPAA requires. Detections are Sigma rules compiled to OpenSearch DSL with sigma-cli and tracked in git with normal PR review, so the audit trail for "when did we start detecting X" is just git log. Alerts route to Slack for low severity and PagerDuty for high.

The build order mattered: schema and index templates before any data (remapping after indexing means an expensive reindex), sources one at a time in order of compliance value starting with CloudTrail, detections tuned against real data before any dashboards. Auditor requests like "all authentication events for user X over 90 days" went from a manual process to a saved search. The lessons-learned section covers a user_id field collision that forced a mid-stream reindex, and an initial detection set noisy enough that tuning took longer than writing the rules. Read the case study

OPKSSH dev access

Dev servers had three shared Linux accounts (devreadonly, devuser, devadmin), and access meant a sysadmin manually adding your public key to the right authorized_keys file. Static keys never expired, offboarding depended on a checklist being followed on every server, and answering "who has what access" meant matching key fingerprints across machines by hand.

OPKSSH replaces the static key with an SSH certificate cryptographically bound to a Google Workspace identity. A developer runs opkssh login, completes a browser OAuth flow, and receives a time-limited cert carrying their email and Google Group claims. Server-side, an AuthorizedPrincipalsCommand in sshd_config invokes the OPKSSH verifier, which checks the cert against Google's JWKS, confirms the token hasn't expired, and matches group claims against /etc/opkssh/policy.yml, a per-server mapping of Google Groups to the three Linux accounts. No authorized_keys file involved. The account tiers themselves were kept as-is; only what authorizes a person to use each one changed.

The migration ran both auth paths in parallel: a fingerprint audit mapped every existing key to a person (surfacing two keys attributable to nobody current), Google Groups were populated to mirror the existing state, and developers moved over in small batches before authorized_keys was removed entirely. Offboarding was verified directly: a test account was suspended in Workspace and a freshly issued cert for that identity was rejected at OIDC validation. Access changes are now a Google Group edit, a leaked cert is self-revoking at token expiry, and the orphaned-key problem is structurally impossible. Read the case study