Observability Demo

Note: This is an experimental demo project.

An experimental observability playground demonstrating end-to-end telemetry across Python (Flask), Java (Spring Boot), and Rust (Axum) microservices. It utilizes the OpenTelemetry Collector to pipeline distributed traces, custom metrics, and structured logs into the Grafana LGTM stack (Loki, Grafana, Tempo, Mimir), complete with a PostgreSQL audit layer and synthetic load generation.

Run via Docker Compose or Kubernetes (Kind).

Quick Deploy

Deploy this stack to cloud:

Quick Start

Docker Compose

./launch_docker.sh --run          # full teardown + deploy
./launch_docker.sh --run --keep-data   # keep DB data
./launch_docker.sh --teardown-only     # cleanup

After ~30s:

• Python app: http://localhost:5000
• Java app: http://localhost:8080
• Rust app: http://localhost:8083
• Grafana: http://localhost:3000 (auto-provisioned dashboard)
• Prometheus: http://localhost:9090

Kubernetes (Kind)

./launch_k8s.sh --run              # install deps, create cluster, deploy
./launch_k8s.sh --teardown-only    # delete cluster

Forwarding set up automatically. Access same as Docker Compose.

What This Does

Apps have 5 endpoints each:

• / — greeting page
• /compute/<n> — Fibonacci(n) with 10% error rate (test error handling)
• /auditlog — log system metrics to PostgreSQL
• /auditlog/stats — response time stats (p50/p90/p95/p99)
• /eval?expr=... — math expression evaluator (no external libs, Shunting-Yard algorithm)

Observability:

• Traces via OTLP → Tempo
• Metrics via OTLP → Mimir (with Prometheus scrape from postgres-exporter)
• Logs via OTLP + structured logging → Loki
• PostgreSQL version widget in Grafana dashboard

Canary: Continuous synthetic load (24 TPS docker / 2 TPS k8s)

Stack Overview

Component	Port	Purpose	Version
Python Flask	5000	WSGI app: Fibonacci compute, Postgres audit log + pooling, math expression parser (Shunting-Yard)	1.0.1
Java Spring Boot	8080	Spring MVC app: Fibonacci compute, JDBC audit log, recursive descent expression parser	1.0.1
Rust Axum	8083	Async web server: Fibonacci compute, sqlx async Postgres, tokenizer-based expression evaluator	1.0.1
Grafana	3000	Dashboards + data sources	12.4.2
Prometheus	9090	Metrics scraper	3.11.2
Mimir	9009	Long-term metrics storage	2.17.9
Tempo	3200	Trace storage	2.10.3
Loki	3100	Log storage	3.7.1
OpenTelemetry Collector	4317	OTLP ingestion point	0.149.0
PostgreSQL	5432	Audit log persistence	16
postgres-exporter	9187	PG metrics (version, connections, etc.)	0.19.1
Valkey	6379	Redis fork for storing custom metrics	latest

Testing

See the consolidated test results for recent self-test executions.

# Unit tests (no stack required)
python -m pytest tests/test_evaluator.py tests/test_app.py -v

# Infrastructure tests (requires running stack)
python -m pytest tests/test_infrastructure.py -v

# All tests
python -m pytest tests/ -v

# Shell smoke test (docker compose)
./test.sh

Project Layout

.
├── app.py / java-app/ / rust-app/      Apps + Dockerfile
├── canary/                               Synthetic load generator
├── config/                               YAML configs (otel, tempo, loki, mimir, prometheus)
│   └── dashboards/
│       └── observability-demo-metrics.json
├── k8s/                                  Kubernetes manifests + deploy script
├── tests/                                Test suite
├── docs/                                 Diagrams, screenshots
├── docker-compose.yaml                   Full stack definition
├── launch_docker.sh / launch_k8s.sh      Entry points for deployment
└── pytest.ini / requirements.txt         Python config

Expression Evaluator

Parses math expressions (no eval(), no external libs). Implements Shunting-Yard algorithm:

# Python
curl "http://localhost:5000/eval?expr=2+3*4"
# Java
curl "http://localhost:8080/eval?expr=2+3*4"
# Rust
curl "http://localhost:8083/eval?expr=2+3*4"

Supports: +, -, *, /, ^ (exponent), parentheses, floating point.

PostgreSQL Additions

• Version widget in Grafana dashboard (requires PG_EXPORTER_DISABLE_SETTINGS_METRICS=false)
• Audit table auto-created on app startup
• Connection pooling in all app services
• postgres-exporter scrapes version, connections, cache hit rate, etc.

---

Note: AI was used to learn, build, test, and deploy this project.