Local RAG Assistant

A private, fully local document Q&A assistant. Upload PDFs, notes, or markdown files and ask questions about them — the chat model, the embeddings, and the vector store all run on local hardware via Ollama, FastAPI, and sqlite-vec in Docker Compose. No document content or query ever leaves the machine.

Why local, not another cloud LLM demo

Cloud LLM APIs are the easy path for "chat with your documents," but that means uploading potentially sensitive files to a third party. The harder, more interesting question: how good an experience can you build entirely on your own hardware, with no GPU dependency required? It started there — but a working RAG pipeline alone reads as an AI-engineering demo, not evidence of infrastructure or security judgment. The project was deliberately pushed further in three passes: security hardening, a DevOps pipeline, and a sysadmin-facing deployment story — each one verified against the running stack, not just written and assumed correct.

Security hardening, with a real threat model

Every /api/* route requires an API key checked via a dependency that fails closed — if the key isn't configured, requests get a 500 rather than silently running unauthenticated. Uploads are checked against an extension allowlist, a size cap, and a magic-byte check, so renaming an executable to .pdf doesn't get past validation. Ollama's own API has no auth of its own, so its port was removed from the host entirely — only the app container can reach it over the internal Compose network — and the app itself binds to 127.0.0.1 by default. The container runs as non-root with a read-only root filesystem and every capability dropped, built from a multi-stage Dockerfile so the shipped image carries no pip/uv build tooling at all. The README's threat model also names what a small local model can't fully defend against — prompt injection via ingested documents — rather than pretending the system prompt's instruction to treat retrieved content as data, not commands, is a guarantee instead of a mitigation.

CI/CD — and the bug it caught in itself

CI runs ruff, a pip-audit dependency scan, and a Trivy image scan that fails the build on fixable CRITICAL/HIGH CVEs, with a publish job gated behind all three that pushes to GHCR only from main. Setting it up surfaced a real mistake: the Trivy action was pinned to @0.24.0, but that tag never existed — the actual release is tagged v0.24.0, with a "v" prefix, confirmed with git ls-remote --tags (used specifically because GitHub's REST API was reporting degraded performance at the time, confirmed independently via githubstatus.com, and git's own protocol wasn't affected). The fix went further than swapping the version: the action is now pinned to its immutable commit SHA rather than a mutable tag, since a tag/version mismatch is exactly what broke it in the first place. Dependabot is wired in too, and correctly proposed updating that same SHA pin within a day of merging — a small, concrete confirmation that the approach works the way it's supposed to.

Two deployment paths: Compose or systemd

Docker Compose is the primary path, healthchecked so the app only starts once Ollama reports healthy, with resource limits sized to a 16GB RAM budget. A native systemd unit exists alongside it as a deliberate sysadmin-facing alternative — its sandboxing directives (ProtectSystem=strict, CapabilityBoundingSet=, NoNewPrivileges=true, and more) are the systemd-native equivalent of the container's non-root/read-only/cap-drop posture, since there's no container boundary to lean on there. A backup script rounds out the operational story: it uses SQLite's own online backup API via Python's sqlite3 module — safe to run while the app is concurrently writing, unlike a plain cp — then gzips and prunes backups past a retention window, scheduled via a systemd timer or a plain cron entry depending on the deployment.

Monitoring: plugging into existing Zabbix, not a new stack

A Prometheus-format /metrics endpoint (API-key protected, like the rest of the API) exposes request counts by route, ingest/chat activity counters, and a live-checked gauge for whether Ollama itself is reachable. The Zabbix template that consumes it follows the exact pattern already established in the Keycloak monitoring project below — one HTTP agent item scrapes the metrics endpoint, dependent items extract series via Prometheus pattern preprocessing, and alerts route through the same Google Chat webhook instead of standing up a second one. One trigger deliberately ties two passes together: sustained rate-limit rejections from the security hardening pass now fire a "possible abuse" alert, so a limit nobody was watching becomes a signal someone actually gets paged on.