ZyzyvaTelemetry

Version 1.0.0 - Shared observability library for the Botify ecosystem.

This library provides a unified wrapper around industry-standard observability tools (Prometheus, Tower, Loki) with ecosystem-specific defaults and conventions.

Features

• Metrics Collection - Via PromEx integration with Prometheus
• Error Tracking - Via Tower with structured JSON logging for Loki
• Health Checks - Standardized health endpoints for all services
• Correlation Tracking - Distributed tracing across services
• Zero Configuration - Ecosystem defaults built-in

Installation

Add to your mix.exs:

def deps do
  [
    {:zyzyva_telemetry, github: "zyzyva/zyzyva_telemetry", tag: "v1.0.0"}
  ]
end

Quick Start

1. Define Your PromEx Module

defmodule MyApp.PromEx do
  use ZyzyvaTelemetry.PromEx,
    otp_app: :my_app,
    service_name: "my_app",
    router: MyAppWeb.Router,
    repos: [MyApp.Repo],
    broadway_pipelines: []  # Add any Broadway pipelines
end

2. Add to Supervision Tree

# lib/my_app/application.ex
def start(_type, _args) do
  children = [
    # ... your other children (Repo, PubSub, etc.)

    {ZyzyvaTelemetry.Supervisor,
     service_name: "my_app",
     promex_module: MyApp.PromEx,
     repo: MyApp.Repo}
  ]

  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

3. Add Routes

# lib/my_app_web/router.ex
pipeline :api do
  plug ZyzyvaTelemetry.Plugs.CorrelationTracker
end

# Metrics endpoint (for Prometheus scraping)
scope "/metrics" do
  forward "/", MyApp.PromEx
end

# Health endpoints
get "/health", ZyzyvaTelemetry.HealthController, :check

What Gets Collected

Automatic Metrics (via PromEx)

• BEAM VM: Memory, processes, schedulers, garbage collection
• Phoenix: Request duration, counts, errors by route
• Ecto: Query duration, pool usage (if repos configured)
• Broadway: Pipeline performance (if configured)
• System: Via node_exporter on the host

Error Tracking

• Automatic exception capture via Tower
• Structured JSON logs written to /var/log/{service_name}/errors.json
• Includes correlation IDs, stack traces, and metadata
• Ready for Promtail/Loki ingestion

Health Status

• Memory usage with thresholds
• Process count monitoring
• Database connectivity (if repo provided)
• Custom health checks
• Exposed at /health endpoint

Usage

Correlation Tracking

Track requests across distributed services:

# In a Phoenix plug or controller
ZyzyvaTelemetry.with_correlation(correlation_id, fn ->
  # All logs and errors within this block
  # will include the correlation_id
  perform_operation()
end)

# Or manually manage correlation
ZyzyvaTelemetry.set_correlation_id("request-123")
# ... do work ...
ZyzyvaTelemetry.get_correlation_id()  # Returns "request-123"

Custom Metrics

Emit custom telemetry events that will be collected by Prometheus:

# Track deployments
ZyzyvaTelemetry.track_deployment("my_app", :success)

# Track errors
ZyzyvaTelemetry.track_error("my_app", "payment_failed")

# Track business operations with timing
:telemetry.span(
  [:ecosystem, :business, :operation],
  %{service_name: "my_app", operation: "process_order"},
  fn ->
    # Your operation here
    result = process_order()
    {result, %{}}
  end
)

Health Checks

Register custom health checks:

# In your application startup
ZyzyvaTelemetry.report_health(:rabbitmq, fn ->
  # Return true if healthy, false otherwise
  check_rabbitmq_connection()
end)

# Get current health status
health = ZyzyvaTelemetry.get_health()
# Returns: %{status: "healthy", service: "my_app", ...}

Marketing Attribution (pageviews, funnel steps, conversions)

ZyzyvaTelemetry.Events + ZyzyvaTelemetry.Plugins.FunnelMetrics give every Zyzyva app the same attribution API. First-touch acquisition (UTM + referrer bucket) rides along automatically via ZyzyvaTelemetry.Acquisition.

1. Configure service name:

# config/config.exs
config :zyzyva_telemetry, service_name: "my_app"

2. Capture first-touch acquisition on every request:

# router.ex
pipeline :browser do
  plug :accepts, ["html"]
  plug :fetch_session
  plug ZyzyvaTelemetry.Plugs.AcquisitionTracker
  # ...
end

3. Enable the PromEx plugin:

defmodule MyApp.PromEx do
  use ZyzyvaTelemetry.PromEx,
    otp_app: :my_app,
    service_name: "my_app",
    router: MyAppWeb.Router,
    repos: [MyApp.Repo],
    additional_plugins: [ZyzyvaTelemetry.Plugins.FunnelMetrics]
end

4. Emit events from your app:

ZyzyvaTelemetry.Events.track_pageview("home")
ZyzyvaTelemetry.Events.track_pageview("blog_post", %{slug: "my-post"})
ZyzyvaTelemetry.Events.track_funnel_step("consultant", "gate_unlocked")
ZyzyvaTelemetry.Events.track_conversion("contact_form")

Prometheus counters exposed (bounded cardinality):

• zyzyva_funnel_pageviews_total{service, page_type, source}
• zyzyva_funnel_steps_total{service, funnel, step, source}
• zyzyva_funnel_conversions_total{service, conversion_type, source}

Loki query patterns (full cardinality — slug, utm_campaign, etc.):

# Pageviews for a blog post across the ecosystem
{job="logs"} | json | event_type="pageview" | page_type="blog_post" | slug="my-post"

# Conversions attributed to a specific campaign
{job="logs"} | json | event_type="conversion" | utm_campaign="launch_apr"

# Funnel trace for a visitor session via correlation_id
{job="logs"} | json | correlation_id="abc-123"

To ship attribution events to Loki, set min_level: :info on the LokiLogger (the default is :warning):

config :zyzyva_telemetry, ZyzyvaTelemetry.LokiLogger,
  loki_url: System.get_env("LOKI_URL"),
  service_name: "my_app",
  min_level: :info

Feedback alerts (Telegram + Loki)

ZyzyvaTelemetry.Feedback.report/1 is the entry point for inbound user feedback / bug reports. Call it from your feedback context after a successful insert — it does two things:

1. Records a structured event_type="feedback" line to Loki (the durable queue a triage agent can read back), plus a Prometheus counter.
2. Fires a best-effort Telegram alert so you see the report arrive live.

# In your feedback context, after the insert succeeds:
ZyzyvaTelemetry.Feedback.report(%{
  type: feedback.type,          # "bug_report" | "feature_request" | ...
  subject: feedback.subject,
  message: feedback.message,
  id: feedback.id,
  user_id: feedback.user_id
})

Configure the bot once (reuses the same Telegram bot as the Grafana alerts):

# config/runtime.exs
config :zyzyva_telemetry, ZyzyvaTelemetry.Telegram,
  bot_token: System.get_env("TELEGRAM_BOT_TOKEN"),
  chat_id: System.get_env("TELEGRAM_CHAT_ID")

Notes:

• Fails safe, not silent. report/1 always returns :ok and never blocks a feedback submit. With no ZyzyvaTelemetry.Telegram config block (e.g. dev) the alert is quietly skipped. If the block is present but bot_token/chat_id don't resolve (wired but misconfigured — env vars unset in prod), it logs a warning that ships to Loki/Grafana, so a dropped alert is visible instead of silent. Use System.get_env/1 (not fetch_env!) so a missing var degrades to this warning rather than crashing boot.

• Loki record needs :info. The feedback log line is emitted at :info, so it only reaches Loki when LokiLogger has min_level: :info (see the attribution note above). The Telegram alert works regardless.

• Routing to its own room later. All alerts share one bot + chat by default. To split feedback into a dedicated Telegram room, point it at a different chat_id — globally in config, or per call:

  ZyzyvaTelemetry.Telegram.notify(text, chat_id: feedback_chat_id)

  A separate bot_token gives full isolation. No code change beyond the new id/token.

Integration with Monitoring Stack

This library is designed to work with the Botify ecosystem monitoring stack:

1. Prometheus scrapes metrics from /metrics endpoint
2. Promtail ships JSON error logs from /var/log/*/errors.json to Loki
3. Grafana provides unified visualization of metrics and logs

See the monitoring-stack repository for infrastructure setup.

Configuration

The library works with minimal configuration, but you can customize:

# Optional: Configure PromEx settings
config :my_app, MyApp.PromEx,
  manual_metrics_start_delay: :no_delay,
  drop_metrics_groups: [],
  grafana: [
    host: "http://localhost:3000",
    auth_token: "your_token"
  ]

# Optional: Configure health check interval
{ZyzyvaTelemetry.Supervisor,
 service_name: "my_app",
 promex_module: MyApp.PromEx,
 repo: MyApp.Repo,
 check_interval: 60_000}  # Check every 60 seconds instead of default 30

Migration from v0.1.0 (SQLite-based)

If upgrading from the SQLite-based v0.1.0:

1. Run cleanup script to remove SQLite artifacts:

   ./cleanup_v1_resources.sh

2. Update supervision tree (see Quick Start above)

3. Update dependencies:

   # Remove
   {:exqlite, "~> 0.33"}
   
   # Add (handled automatically by zyzyva_telemetry)
   {:prom_ex, "~> 1.11"},
   {:tower, "~> 0.6"},
   {:telemetry_metrics, "~> 1.0"},
   {:telemetry_poller, "~> 1.1"}

4. Remove deprecated function calls:

   • log_error/1,2 → Use Tower error tracking instead
   • log_warning/1,2 → Use standard Logger
   • log_exception/3,4 → Exceptions are captured automatically by Tower
   • generate_test_events/0 → No longer needed

5. Update health endpoints:

   # Old
   get "/health", ZyzyvaTelemetry.HealthController, []
   
   # New (same syntax, but different backend)
   get "/health", ZyzyvaTelemetry.HealthController, :check

Advanced Usage

Custom Health Checks

Add sophisticated health checks:

{ZyzyvaTelemetry.Supervisor,
 service_name: "my_app",
 promex_module: MyApp.PromEx,
 repo: MyApp.Repo,
 extra_health_checks: %{
   redis: fn -> check_redis_connection() end,
   queue_depth: fn ->
     depth = MyApp.Queue.depth()
     depth < 1000  # healthy if queue depth < 1000
   end
 }}

Custom PromEx Plugins

Create ecosystem-specific metrics:

defmodule MyApp.CustomPlugin do
  use PromEx.Plugin

  @impl true
  def event_metrics(_opts) do
    [
      counter("my_app.custom.events",
        event_name: [:my_app, :custom, :event],
        description: "Custom event counter",
        tags: [:type]
      )
    ]
  end
end

# In your PromEx module
defmodule MyApp.PromEx do
  use ZyzyvaTelemetry.PromEx,
    otp_app: :my_app,
    service_name: "my_app",
    router: MyAppWeb.Router,
    repos: [MyApp.Repo],
    additional_plugins: [MyApp.CustomPlugin]
end

Architecture

ZyzyvaTelemetry v1.0 wraps industry-standard tools:

• PromEx - Elixir Prometheus client with built-in plugins
• Tower - Error tracking and reporting
• Telemetry - Standard Elixir metrics and instrumentation
• Health Registry - In-memory health check management

The library provides:

• Pre-configured PromEx setup with ecosystem defaults
• Automatic Tower reporter writing JSON logs for Loki
• Correlation ID tracking across distributed services
• Standardized health endpoints

Requirements

• Elixir ~> 1.18
• Phoenix ~> 1.7 (optional, for web endpoints)
• Ecto ~> 3.10 (optional, for database metrics)
• Broadway ~> 1.0 (optional, for pipeline metrics)

License

MIT