Overview
A self-hosted lakehouse on a single Contabo VPS (8 vCPU, 32 GB RAM, 400 GB NVMe), running Debian 12. Every service runs in Docker Compose. Caddy handles TLS for the whole stack.
The stack is not a homelab demo — it supports the zoning-compliance scraping pipeline, the QA dashboard, and the RAG chatbot. If a service isn't earning its keep, it gets removed.
Architecture
Five tiers, left to right:
- Ingestion — Python scrapers, Kafka (3 brokers), webhook ingress via Caddy + n8n
- Orchestration — Airflow (12 DAGs), n8n (low-code workflows), Docling (PDF extraction)
- Storage — MinIO (S3-compatible object store), Apache Iceberg (table format), Project Nessie (catalog), Postgres (operational DB)
- Query — Trino (federated SQL), PostgREST (REST API over SQL views), graph layer (GraphRAG)
- Serving — Apache Superset, ChatAEC, RAG chatbot, SlideConsultant
Why self-host?
At the workload scale here (tens of thousands of documents, one analyst, one engineer), managed cloud services would cost more and teach less. Running the full stack locally means:
- Replaying the pipeline from any checkpoint with
kafka-consumer-groups.sh --reset-offsets - Schema evolution on Iceberg tables without downtime or rewrite
- Full observability via Grafana + Loki without a SaaS bill
- Operational discipline: if it breaks at 2 AM, I fix it
Host specs
| Field | Value | |---|---| | Provider | Contabo | | CPU | 8 vCPU | | RAM | 32 GB | | Storage | 400 GB NVMe | | OS | Debian 12 | | Runtime | Docker Compose · 14 stacks | | Ingress | Caddy · automatic TLS | | Observability | Uptime Kuma · Grafana · Loki |