LFX V2 Campaign Service

A collection of service endpoints to support Marketing Operations campaign creation and management.

API Endpoints

• /livez: GET — checks that the service is alive (liveness probe). Returns 200 with a text/plain body of OK. Does not depend on database availability.
• /readyz: GET — checks that the service is able to take inbound requests (readiness probe), including a PostgreSQL connectivity check. Returns 200 with a text/plain body of OK when ready, or 503 when not ready.
• /metrics: GET — Prometheus metrics in the text exposition format. Served on the same port as the API, and available in no-database mode and during a cold start (a scrape target that only appears once the database is up is exactly backwards). See Metrics below.

These three endpoints are unauthenticated and are excluded from the generated public API documentation. They are also absent from the Helm chart's HTTPRoute and Heimdall RuleSet, so they are reachable only in-cluster (kubelet probes and the Prometheus scraper), never through the public gateway.

Metrics

GET /metrics serves the Prometheus text exposition format. The endpoint takes no configuration — there is no environment variable to enable it and no separate metrics port. It is independent of the OTEL_* settings below: OTEL_METRICS_EXPORTER defaults to none, so wiring these instruments to the OTLP pipeline would leave /metrics empty in the default deployment.

Service metrics:

Metric	Type	Labels	What it answers
campaign_dispatch_total	counter	platform, outcome	Are campaigns actually landing on each ad platform? outcome is one of success, skipped, failure, panic — panic is separate because it is a bug in this service, not an upstream refusal.
campaign_job_transitions_total	counter	status	Dispatch jobs reaching each state (running, succeeded, partial, failed). A growing gap between running and the terminal states means jobs are getting stuck — the terminal transition is recorded only when the status write actually persisted, so a job stuck by a failed write leaves the gap open rather than closing it.
campaign_upstream_calls_total	counter	platform, operation, outcome	Upstream ad-platform API call volume and error rate.
campaign_upstream_call_duration_seconds	histogram	platform, operation, outcome	Upstream ad-platform latency. Timed only after the pre-platform guards pass, so local refusals do not drag the quantiles toward zero. Bucketed on the service's own call budgets (10ms…45s, with edges on the 20s/30s/45s ceilings) rather than the OTel default, whose ms-scale boundaries would collapse every healthy call into one bucket.
campaign_monitor_unmeasured_rows_total	counter	platform, class	Account-monitor campaign rows returned with fetch_failed=true — why the monitor's totals may understate an account. class is one of auth, throttle, upstream_5xx, decode, no_report (a report-backed account with no finished report yet — expected on a first read), other (a failure the client did not classify) or unknown (a client reported a class outside that set, i.e. classification drift).
campaign_db_pool_*	gauge / counter	none	Database pool health: acquired, idle, total, max and newly-established connections, plus canceled and empty acquires. Exported only when a pool is actually wired — see below.

Plus the standard Go runtime and process collectors.

Label cardinality. No metric here carries a campaign id, brief id, project id, job id, account id or URL as a label value — an unbounded label creates one retained time series per distinct value and is the classic way to take a Prometheus server down. platform is mapped through a closed provider set and anything outside it collapses to unknown; the outcome and monitor failure-class labels are closed enums; and operation passes a shape guard that admits short lower-snake tokens and degrades anything id-shaped or derived to unknown. No metric name, label or help string carries a credential, DSN or token.

Absent pool metrics are meaningful. When no database is wired (no-DB mode, or a cold start before the pool opens) the campaign_db_pool_* series are not exported at all, rather than exported as zeroes. A zero is a measurement: reporting max_connections=0 for a service running without a database is indistinguishable from a pool that has collapsed, and would fire a false exhaustion alert.

Scraping. The chart sets prometheus.io/scrape, prometheus.io/path and prometheus.io/port on the pod template by default, so a discovery-based collector picks the pod up with no further configuration. prometheus.io/port is derived from service.port in the template rather than hardcoded in values.yaml, so the scrape port cannot drift from the port the container listens on. A deployment that overrides podAnnotations replaces the map, so it must re-declare prometheus.io/scrape and prometheus.io/path to keep being scraped.

Environment variables

Configuration priority: CLI flags > environment variables > defaults.

Required (startup)

When any PostgreSQL setting is supplied, the set must be complete or the process exits non-zero. Fully omitting all database settings is allowed for unit tests / metadata-only local runs (no-DB mode; /readyz stays process-ready without a pool). In-cluster they are typically injected from the ExternalSecret-managed Kubernetes secret (lfx-v2-campaign-service-secrets in namespace lfx-v2-campaign-service; keys host, port, username, password, dbname, engine, credential-encryption-key).

• PGHOST (secret key host) — PostgreSQL hostname
• PGUSER (secret key username) — PostgreSQL username
• PGPASSWORD (secret key password) — PostgreSQL password (never logged)
• PGDATABASE (secret key dbname) — PostgreSQL database name
• CREDENTIAL_ENCRYPTION_KEY (secret key credential-encryption-key) — base64-encoded 32-byte AES-256 key used to encrypt ad-platform connection credentials at rest. Required whenever a database is configured. Rollout prerequisite: provision this key on lfx-v2-campaign-service-secrets (via ExternalSecret sync) before deploying the chart revision that references it; otherwise the pod stays in CreateContainerConfigError.

The service composes the DSN in-process from these fields (no DATABASE_URL env var required).

Local / test sample key

For laptop runs against the RDS tunnel (or any local Postgres), you can use this non-production sample (base64 of the 32-byte ASCII string LFX-campaign-local-dev-aes-256!!):

# !!! WARNING !!!
# This is a *TEST/EXAMPLE* key for local/dev use only.
# NEVER use this in production or shared environments!
# -----------------------------------------------------
# (test/dev/example: base64-encoded 'LFX-campaign-local-dev-aes-256!!')
# Example (use printf so the *plaintext* stays exactly 32 bytes — a trailing
# newline from echo would make it 33 and break AES-256 key length):
#   printf '%s' 'LFX-campaign-local-dev-aes-256!!' | base64
export CREDENTIAL_ENCRYPTION_KEY='TEZYLWNhbXBhaWduLWxvY2FsLWRldi1hZXMtMjU2ISE='

Again, do not use this value in shared, staging, or production clusters. Generate a real key for those environments, for example:

openssl rand -base64 32

Optional (with defaults)

• PGPORT (default 5432; secret key port) — PostgreSQL port

• PGENGINE (default empty) — when set, must be postgres or postgresql

• PORT (default 8080) — HTTP listen port (CLI flag -p)

• HOST (default *) — bind interface; * means all interfaces (CLI flag -bind)

• DEBUG (unset) — set to true to enable debug logging (CLI flag -d)

• JWKS_URL — key set every bearer token is verified against (defaults to the in-cluster Heimdall JWKS URL). Set but not an absolute URL fails startup rather than degrading verification.

• JWT_AUDIENCE (default lfx-v2-campaign-service) — required audience

• JWT_ISSUER (default heimdall) — required issuer

• JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL (default unset) — local development only: when non-empty it disables JWT verification and attributes every request to this principal, logging a WARN on every boot that sets it. Two independent guards keep it out of a deployed environment: the chart refuses to render it, and at runtime the pod fails startup rather than degrading whenever it detects it is running in a cluster — so the failure is a crash-loop, not a silently unauthenticated service. Worth knowing before debugging a rollout that will not come up. The runtime guard is a detection, not a proof: it looks for KUBERNETES_SERVICE_HOST and the service-account directory, so a manifest applied outside the chart that both sets this variable and suppresses those two signals (an explicit empty KUBERNETES_SERVICE_HOST plus automountServiceAccountToken: false) is not caught. That combination is three deliberate, visible changes rather than one env line, which is the bar the guard is built to — see internal/infrastructure/config/config.go.

• NATS_URL — NATS server URL (reserved for messaging; defaults to in-cluster NATS URL)

• EVENT_URL_NAT64_PREFIXES (default unset) — comma-separated RFC 6052 §2.2 network-specific NAT64 prefixes this cluster uses, e.g. 2a01:4f8:1::/48. Only /32, /40, /48, /56, /64 and /96 are valid lengths; a malformed value or a wrong length panics at startup, deliberately, so a typo stops the pod rather than decoding at the wrong offset for its lifetime.

  Unset is correct for a cluster with no NAT64, and the well-known 64:ff9b::/96 is decoded regardless. It cannot be discovered in-process: a network-specific prefix is carved from the operator's own global unicast space and is indistinguishable from any other public prefix. On a cluster that uses one, leaving it undeclared is a live SSRF hole — the translator, not this service, makes the IPv4 connection, so an address encoding 169.254.169.254 passes every check here and the fetch reaches the cloud metadata endpoint.

• REDDIT_METRICS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to Reddit Ads metrics reads. Only the exact value true enables them; unset or any other value (including TRUE or a typo) fails closed, and GET .../campaigns/{campaign_id}/metrics answers 400 "not supported for this campaign's platform" for a Reddit campaign. It also gates the Reddit audience and keyword performance reads (GET /projects/{project_id}/reddit-ads/audience and .../reddit-ads/keywords), which answer 400 "not supported" while off. Off by default because no request has yet been made against a live Reddit ad account (LFXV2-3282). The request and response shapes now follow Reddit's official public OpenAPI document, but behaviour a schema cannot express — zero-activity rows, the account's attribution window — is still unconfirmed, and a read returning 200 would look authoritative to every consumer. The chart sets it to "false" (charts/lfx-v2-campaign-service/values.yaml); flip it only after the contract is verified against a live Reddit ad account.

• CAMPAIGN_JOB_RETENTION (default 4320h, i.e. 180 days) — how long a terminal campaign job (succeeded, partial, failed) is kept before the retention sweeper deletes it. A Go duration string.

  campaign_jobs is otherwise append-only: nothing else in the service deletes a row, so without the sweeper the table grows with every brief dispatch forever, and the stuck-job recovery sweep pays for that growth on every pass. One bounded, ordered batch is deleted per pass (see JobRepo.PruneTerminalJobs), so a large backlog drains over several passes rather than in one long transaction. Every replica runs it without leader election, exactly as the stuck-job sweeper does — overlapping passes are harmless because the delete is bounded and idempotent.

  Only terminal jobs are ever eligible, enumerated as an allow-list rather than as "not queued/running". A queued or running row is never deleted at any age: an old non-terminal row is a stuck job, which is precisely the record needed to investigate a dispatch that never finished. (The recovery sweeper fails those out after 15m, at which point they become terminal and their retention window starts from that transition.) Age is measured on updated_at — when the job reached its terminal state — so a job created months ago but completed yesterday counts as recent history.

  The default is deliberately long because these rows are the audit trail of real ad spend. Unset, empty, unparseable (30 days and 7d are not valid Go durations) and non-positive values all fall back to the 180-day default rather than to a short window, so a typo cannot cause early deletion — the rejected value is logged at startup.

• REDDIT_KEYWORD_TARGETING_WRITES_ENABLED (default unset, i.e. OFF) — opts a deployment IN to BOTH Reddit ad-group targeting writes: removing keywords (POST .../campaigns/{campaign_id}/keyword-targeting/removals) and adding negative keywords to excluded_keywords (POST .../campaigns/{campaign_id}/negative-keywords). Enabling it enables both. Only the exact value true enables them; anything else fails closed and both endpoints answer 400 for a Reddit campaign. Off by default because Reddit replaces an ad group's targeting as a whole, so the write sends back every dimension it read, and that round trip has not been exercised against a live ad account. The read (GET .../keyword-targeting) is not gated. The chart sets it to "false".

• REDDIT_CONVERSION_EVENT_ENABLED (default unset, i.e. OFF) — opts a deployment IN to redditConfig.conversionEvent (#549), the pixel event a conversions campaign optimizes toward (sent as optimization_goal on the campaign and the ad group instead of the default PURCHASE). Only the exact value true enables it; anything else fails closed, and a create that sets the field is refused with a 400 before any job exists. Off by default because no live create has yet confirmed Reddit accepts a goal other than PURCHASE on a CONVERSIONS campaign under budget optimization. The chart sets it to "false".

• REDDIT_CUSTOM_AUDIENCES_ENABLED (default unset, i.e. OFF) — opts a deployment IN to Reddit custom audiences (#549): the create inputs redditConfig.customAudienceIds / excludedCustomAudienceIds (the ad group's targeting.custom_audience_ids / excluded_custom_audience_ids) and the picker read GET /projects/{project_id}/reddit-ads/custom-audiences. Only the exact value true enables them; anything else fails closed: a create setting either list is refused with 400 before any job exists, and the read answers 400. Off by default because both shapes come from Reddit's public v3 OpenAPI document and have not been exercised against a live ad account. The audiences live on the ad account every foundation shares, so the create inputs also need global_marketing_ops on the project (403 otherwise) and the read is gated on it at the gateway. The chart sets it to "false".

• REDDIT_TARGETING_RESOLUTION_ENABLED (default unset, i.e. OFF) — opts a deployment IN to resolving redditConfig.subreddits and interests before create (#549): each community is confirmed with GET /targeting/communities and each interest resolved to its id with GET /targeting/interests. An unknown value is a synchronous 400 (the pre-check fails open when the read cannot run) and, at dispatch, a job failure with nothing created (fail closed). Only the exact value true enables it; anything else keeps the create as before. Not role-gated. The chart sets it to "false".

• REDDIT_TARGETING_EXCLUSIONS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to redditConfig.excludedSubreddits and redditConfig.excludedGeoTargets (#549), sent at create time as the ad group's excluded_communities and excluded_geolocations. Only the exact value true enables them; anything else fails closed, and a create naming either is refused with a 400 before any job exists. It names no account resource; it keeps a shape no live create has sent off the shared ad account. The chart sets it to "false" (app.environment.REDDIT_TARGETING_EXCLUSIONS_ENABLED.value); enable it per-environment in the ArgoCD overlay once a live create confirms it.

• TWITTER_CREATE_TARGETING_ENABLED (default unset, i.e. OFF) — opts a deployment IN to twitterConfig.geoTargets, languages, keywords and interestIds (#564), country, language, broad keyword and interest targeting criteria added to the line item an X create makes. Only the exact value true enables them; anything else fails closed, and a create naming any of them is refused with a 400 before any job exists. It names no account resource; it keeps a shape no live create has sent off the shared ad account. The chart sets it to "false" (app.environment.TWITTER_CREATE_TARGETING_ENABLED.value); enable it per-environment in the ArgoCD overlay once a live create confirms it.

• META_CONVERSIONS_METRICS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to conversions in the Meta metrics read (#550): for a campaign this service created with the conversions objective, the Insights read also requests actions and reports the recorded conversion event's action type as conversions (#175 statuses). Only the exact value true enables it; anything else keeps the read as before (conversions unavailable, no actions requested). Read-only, but the event-to-action mapping is unverified live and a wrong one would publish a false zero. The chart sets it to "false".

• TWITTER_CUSTOM_AUDIENCES_ENABLED (default unset, i.e. OFF) — opts a deployment IN to X custom audiences (#564): the create inputs twitterConfig.customAudienceIds / excludedCustomAudienceIds (CUSTOM_AUDIENCE targeting criteria, EQ and NE, on the new line item) and the picker read GET /projects/{project_id}/twitter-ads/custom-audiences. Only the exact value true enables them; anything else fails closed: a create setting either list is refused with 400 before any job exists, and the read answers 400. Off by default because both shapes come from X's published v12 reference and have not been exercised against a live ad account. The audiences live on the ad account every foundation shares, so the create inputs also need global_marketing_ops on the project (403 otherwise) and the read is gated on it at the gateway. The chart sets it to "false".

• MICROSOFT_AD_EXTENSIONS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to microsoftConfig.sitelinks and microsoftConfig.callouts (#563), created at create time as SitelinkAdExtension and CalloutAdExtension entries associated with the new Search campaign. Only the exact value true enables them; anything else fails closed, and a create naming either is refused with a 400 before any job exists. It names no account resource; it keeps a shape no live create has sent off the shared ad account. The chart sets it to "false" (app.environment.MICROSOFT_AD_EXTENSIONS_ENABLED.value); enable it per-environment in the ArgoCD overlay once a live create confirms it.

• MICROSOFT_AUDIENCES_ENABLED (default unset, i.e. OFF) — opts a deployment IN to microsoftConfig.audienceIds, excludedAudienceIds and audienceBidAdjustment (#563): remarketing lists checked with GetAudiencesByIds before anything is created, then attached to the new ad group as observed (bid-only) and excluded audience criteria. Only the exact value true enables them; anything else fails closed, and a create naming any of the three is refused with a 400 before any job exists. The lists live on the account every foundation shares, so they also need global_marketing_ops on the project (403 otherwise). The chart sets it to "false" (app.environment.MICROSOFT_AUDIENCES_ENABLED.value); enable it per-environment in the ArgoCD overlay once a live create confirms it.

• MICROSOFT_CREATE_TARGETING_ENABLED (default unset, i.e. OFF) — opts a deployment IN to microsoftConfig.excludedGeoTargets, proximityTargets and languages (#563): excluded countries (negative location criteria) and radius targets attached to the new campaign before its ad group, and the campaign's ad languages. Only the exact value true enables them; anything else fails closed, and a create naming any of the three is refused with a 400 before any job exists. They name no account resource; the flag keeps a shape no live create has sent off the shared account. The chart sets it to "false" (app.environment.MICROSOFT_CREATE_TARGETING_ENABLED.value).

• GOOGLE_ADS_DEMAND_GEN_AUDIENCE_ENABLED (default unset, i.e. OFF) — opts a deployment IN to googleAdsConfig.demandGenAudience, one existing Google Ads Audience on a Demand Gen ad group (#459). Only the exact value true enables it; anything else fails closed, and a create naming one is refused before the budget. It is a per-deployment opt-in: the Audience is checked against the connected Google Ads account (same customer id). That account is shared by every foundation, and campaign_manager includes foundation executive directors, so create-campaigns also requires global_marketing_ops on the project for any Demand Gen audience: 403 without it, 503 when the check cannot be made (it asks project-service and fga-sync over NATS). The same flag gates each demandGenAdGroups[].audience (#460), the same input on a named ad group, demandGenDemographics.audienceSegments (#498) and demandGenDemographics.customAudiences (#516), existing user lists and custom audiences put in the demographic Audience, and the same two lists in any demandGenAdGroups[].demographics (#518): off, a create naming any is refused before the budget, and each needs global_marketing_ops like the audiences above. The rest of Demand Gen, including ad groups without an audience, is not gated. The chart sets it to "false" (app.environment.GOOGLE_ADS_DEMAND_GEN_AUDIENCE_ENABLED.value); enable both per-environment by setting it to "true" in the ArgoCD overlay.

• GOOGLE_ADS_DISPLAY_AUDIENCES_ENABLED (default unset, i.e. OFF) — opts a deployment IN to the Display audience lists that name an account-owned segment (#462): googleAdsConfig.audienceSegments on the display channel, customAudiences and customAffinities, and since #475 the user-list exclusions excludedAudienceSegments and, on each displayAdGroups entry, its audienceSegments, customAudiences, customAffinities and excludedAudienceSegments. Only the exact value true enables them; anything else fails closed, and a create naming one is refused before the budget, naming the fields to omit. Like GOOGLE_ADS_DEMAND_GEN_AUDIENCE_ENABLED, it is a per-deployment opt-in, each segment is checked against the connected Google Ads account, and create-campaigns requires global_marketing_ops on the project for any of these fields (403 without it, 503 when the check cannot be made). userInterests (Google's own categories, at either level), excludedPlacements (public domains), Search's observation-only audienceSegments and displayAdGroups entries without those lists are not behind the flag. Search's audienceSegments (campaign level and each adGroups entry) still name the shared account's user lists, so they too require global_marketing_ops. The chart sets it to "false" (app.environment.GOOGLE_ADS_DISPLAY_AUDIENCES_ENABLED.value); enable it per-environment by setting it to "true" in the ArgoCD overlay.

• META_CALL_TO_ACTION_ENABLED and META_CONVERSION_EVENT_ENABLED (default unset, i.e. OFF) — opt a deployment IN to metaConfig.callToAction (the button on every Meta ad: SIGN_UP, BUY_TICKETS, … instead of LEARN_MORE) and metaConfig.conversionEvent (the conversions objective's custom_event_type: COMPLETE_REGISTRATION, LEAD, … instead of PURCHASE), #550. Only the exact value true enables each; anything else fails closed, and a create carrying the field is refused with a 400 before any job exists. The chart sets both to "false"; enable them per-environment in the ArgoCD overlay.

• META_AUDIENCE_TARGETING_ENABLED (default unset, i.e. OFF) — opts a deployment IN to metaConfig.targeting (#550): ageMin/ageMax, genders, customAudiences, excludedCustomAudiences and advantageAudience on the Meta ad set. Only the exact value true enables it; anything else fails closed, and a create carrying a targeting block is refused with a 400 before any job exists. Each custom audience is read and must belong to the connected Meta ad account before anything is created, and create-campaigns requires global_marketing_ops on the project for the two audience lists (403 without it, 503 when the check cannot be made). The chart sets it to "false" (app.environment.META_AUDIENCE_TARGETING_ENABLED.value); enable it per-environment by setting it to "true" in the ArgoCD overlay.

• GOOGLE_ADS_PMAX_AUDIENCE_SIGNAL_ENABLED (default unset, i.e. OFF) — opts a deployment IN to googleAdsConfig.performanceMaxAudienceSignal, one existing Google Ads Audience attached to a Performance Max asset group as a signal (#461). Only the exact value true enables it; anything else fails closed, and a create naming one is refused before the budget. Like GOOGLE_ADS_DEMAND_GEN_AUDIENCE_ENABLED, it is a per-deployment opt-in, the Audience is checked against the connected Google Ads account, and create-campaigns requires global_marketing_ops on the project for it. performanceMaxSearchThemes (free text) and the Performance Max sitelinks, callouts and structured snippets are not gated. The chart sets it to "false" (app.environment.GOOGLE_ADS_PMAX_AUDIENCE_SIGNAL_ENABLED.value); enable it per-environment by setting it to "true" in the ArgoCD overlay.

• GOOGLE_ADS_SHARED_NEGATIVE_LISTS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to googleAdsConfig.sharedNegativeKeywordLists, existing shared negative keyword lists attached to a Search campaign (#477). Only the exact value true enables it; anything else fails closed, and a create naming one, on any channel, is refused before the budget. Like GOOGLE_ADS_DEMAND_GEN_AUDIENCE_ENABLED, it is a per-deployment opt-in, each shared list is checked against the connected Google Ads account, and create-campaigns requires global_marketing_ops on the project for any. searchAssets (the caller's own images, logo and business name) is not gated. The chart sets it to "false" (app.environment.GOOGLE_ADS_SHARED_NEGATIVE_LISTS_ENABLED.value); enable it per-environment by setting it to "true" in the ArgoCD overlay.

• GOOGLE_ADS_DEMOGRAPHIC_TARGETING_ENABLED (default unset, i.e. OFF) — opts a deployment IN to the Google Ads demographic inputs not yet exercised by a live (persisted) create: googleAdsConfig.observedDemographics (#497), googleAdsConfig.displayDemographics (#499), googleAdsConfig.demandGenDemographics (#498; its create body was accepted by a validateOnly probe on 2026-11-05, but a persisted create and the read-back its reuse relies on are not yet verified) and each demandGenAdGroups[].demographics (#518, the same object building that ad group its own Audience). Only the exact value true enables it; anything else fails closed, and a create naming one, on any channel, is refused before the budget. It names no account resource; it keeps an unverified shape off the shared account. The chart sets it to "false" (app.environment.GOOGLE_ADS_DEMOGRAPHIC_TARGETING_ENABLED.value); enable it per-environment in the ArgoCD overlay once the probe passes.

• LINKEDIN_ADS_CONVERSIONS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to linkedInConfig.conversions (#552), conversion rule ids associated with the campaign through Campaign Conversions once it exists. Only the exact value true enables it; anything else fails closed, and a create naming it is refused with a 400 before a job exists. The ids also require global_marketing_ops. The chart sets it to "false" (app.environment.LINKEDIN_ADS_CONVERSIONS_ENABLED.value); enable it per-environment in the ArgoCD overlay once the probe passes.

• SURVEY_SEQUENCE_WORKFLOW_ID (default unset) — the HubSpot workflow POST /projects/{project_id}/send-workflows copies when a request names no source_workflow_url. Unset or empty selects the in-code default 1841232183, the LF Research survey-promo workflow, which exists only in the LF portal. The chart sets it to "".

• MICROSOFT_METRICS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to Microsoft Advertising (Bing Ads) metrics reads. Only the exact value true enables them; unset or any other value (including TRUE or a typo) fails closed, and GET .../campaigns/{campaign_id}/metrics answers 400 "not supported for this campaign's platform" for a Microsoft campaign. Off by default because the v13 Reporting contract was implemented from Microsoft's published documentation and has NOT been exercised against a live Microsoft Advertising account. Microsoft's pipeline is also unlike every other platform's — an asynchronous submit/poll/download returning a zipped CSV rather than one JSON GET — so there is more surface to be wrong about. The chart sets it to "false" (charts/lfx-v2-campaign-service/values.yaml); flip it only after the contract is verified against a live Microsoft ad account.

• MICROSOFT_AD_COPY_ENABLED (default unset, i.e. OFF) — opts a deployment IN to microsoftConfig.headlines, descriptions, path1 and path2 (#563), the caller copy of the responsive search ad sent instead of the auto-composed copy. Only the exact value true enables them; anything else fails closed, and a create naming any of them is refused with a 400 before any job exists. It names no account resource; it keeps a shape no live create has sent off the shared ad account. The chart sets it to "false" (app.environment.MICROSOFT_AD_COPY_ENABLED.value); enable it per-environment in the ArgoCD overlay once a live create confirms it.

• MICROSOFT_CREATE_NEGATIVE_KEYWORDS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to microsoftConfig.negativeKeywords (#563), campaign-level negative keywords attached right after a Microsoft Advertising campaign is created, through the same AddNegativeKeywordsToEntities call the add-negative-keywords lever makes. Only the exact value true enables it; anything else fails closed, and a create naming a non-empty list is refused with a 400 before any job exists. It names no account resource; it keeps a create-path call no live create has made off the shared ad account. The chart sets it to "false" (app.environment.MICROSOFT_CREATE_NEGATIVE_KEYWORDS_ENABLED.value); enable it per-environment in the ArgoCD overlay once a live create confirms it.

• TWITTER_METRICS_ENABLED (default unset, i.e. OFF) — opts a deployment IN to BOTH X (Twitter) Ads features built on X's asynchronous stats-jobs API: the account monitor (GET /projects/{project_id}/connection-twitter-ads/account-monitor) and the audience read (GET /projects/{project_id}/twitter-ads/audience). Only the exact value true enables them; unset or any other value fails closed and both endpoints answer 400 "not supported". Off by default because the stats-jobs contract (active_entities, job create, job status, a downloaded gzip results file, segmentation) was implemented from X's published documentation and has NOT been exercised against a live X Ads account. X's per-campaign metrics read (a synchronous endpoint) is unaffected. Their per-account limits are per process, so at runtime only the pod holding a per-account Postgres advisory lease runs X stats jobs — any other pod answers the audience read 503 and skips monitor report submissions. The chart renders the flag at any replica count and only warns in its NOTES.txt above one (#381) — ArgoCD never shows that warning, and at three replicas most X audience reads land on a non-owner pod and 503, so run one replica while the flag is on. The chart sets it to "false" (charts/lfx-v2-campaign-service/values.yaml); flip it only after the contract is verified against a live X ad account.

• LFX_FORCE_SYSTEM_ADS_ACCOUNT (default unset, i.e. OFF) — makes the LF-owned system account (system:linuxfoundation) the PRIMARY credential source when a paid-ads campaign is CREATED, so every new campaign authenticates as the marketing-ops account regardless of any per-project connection (the system row is otherwise only a fallback for projects that have connected no account of their own). Only the exact value true enables it; unset or any other value (including TRUE or a typo) leaves the default resolution.

  It governs creation and account discovery only. Pausing, resuming and reading metrics for a campaign that ALREADY EXISTS keep using the account it was created under, so campaigns created before the flag was turned on stay controllable while it is on — forcing them onto the system account would trip each adapter's account-provenance guard and leave a live, spending campaign the service cannot pause.

  While it is on, an ad account id cannot be SAVED onto a project's connection (400); discovery is resolving the LF credential, so the ids it returns are LF-owned and persisting one would outlive the flag. Clearing a selection stays allowed, as is re-sending the id already stored — the guard refuses a CHANGED id, not a present one. This is what keeps the rollout reversible by flipping the flag back.

  One consequence to plan around: linkedin-ads and reddit-ads declare account_id as REQUIRED on their create payloads, so a create body cannot omit it and those two providers cannot be connected at all while the flag is on. google-ads, meta-ads, twitter-ads (as of LFXV2-3319) and microsoft-ads (as of issue #181) are credentials-first and can still be created. (X's funding_instrument_id is still required; credentials-first defers the ACCOUNT choice only.) The per-endpoint status contract is in docs/api-catalog.md under Platform Connections.

  HubSpot/email is never forced (the path gates on the paid-ads channel kind). A missing or unusable system row fails the dispatch closed rather than falling through to the project connection; a MISSING one is reported as an operator-owned configuration fault (500), not as "connect your project" (404). It shares the exact-match true parse with REDDIT_METRICS_ENABLED but not its lifecycle: this flag is read ONCE, when the credentials source is constructed, so changing it requires a restart, whereas REDDIT_METRICS_ENABLED is re-read from the environment on every metrics read. Enable per-environment via the ArgoCD overlay. See specs/006-force-system-ads-account.

Snowflake (optional, audience building)

Read-only warehouse access, used ONLY to resolve an event's past editions when building an audience (LFXV2-2774). These are OPTIONAL as a GROUP: unless SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER and SNOWFLAKE_PRIVATE_KEY are ALL set, the warehouse is treated as unconfigured and audience building still works — it produces a country-only audience and records the narrower scope in the audience's inclusion summary. A partial or unusable configuration degrades the same way rather than failing startup, because past editions ENRICH an audience rather than making it correct.

• SNOWFLAKE_ACCOUNT — account identifier
• SNOWFLAKE_USER — user for key-pair auth
• SNOWFLAKE_PRIVATE_KEY — PEM private key for key-pair auth
• SNOWFLAKE_WAREHOUSE — optional warehouse; the account default is used when omitted
• SNOWFLAKE_ROLE — optional role; the user's default is used when omitted

LLM / email copy (optional, AI_*)

The LF LiteLLM proxy, used ONLY to generate email copy — subject, preheader, body and CTA — via the POST /projects/{id}/briefs/{id}/email-copy endpoint (LFXV2-2775). Optional as a GROUP: unless BOTH AI_PROXY_URL and AI_API_KEY are set, the endpoint returns 503 (service unavailable). The pod starts successfully with or without these values configured.

• AI_PROXY_URL — LiteLLM proxy base URL; /chat/completions is appended
• AI_API_KEY — the proxy's key, not a Bedrock or Anthropic credential (the proxy holds those), so it cannot be replayed against a model provider directly
• AI_MODEL — optional, not a secret; the model id the proxy routes on. Empty selects llm.DefaultModel.

In-cluster the two secrets come from the same ExternalSecret-managed secret as the rest; AI_MODEL is a plain chart value.

Signed creative-asset URLs for Reddit (optional, CREATIVE_ASSET_*)

Lets a Reddit campaign use an image stored through upload-creative-asset (redditConfig.imageAssetId). Reddit only accepts a public image URL that it fetches, so the service serves the stored bytes at a short-lived HMAC-signed URL (GET|HEAD /_campaigns/creative-assets/{project}/{brief}/{asset}?exp=…&sig=…, anonymous by design) and hands that URL to Reddit (LFXV2-2665). Optional as a PAIR: with either of the first two unset the feature is OFF — the route returns 404 and a Reddit config naming imageAssetId is refused with a 400. A value that is set but unusable stops the pod at startup — each supplied value is validated even while the other half is still missing.

• CREATIVE_ASSET_URL_SIGNING_KEY — secret; standard base64 of at least 32 random bytes (openssl rand -base64 32). In-cluster it comes from creative-asset-url-signing-key on the ExternalSecret-managed secret.
• CREATIVE_ASSET_PUBLIC_BASE_URL — the https origin of the public gateway Reddit can reach (e.g. https://lfx-api.<domain>); no path, query or userinfo.
• CREATIVE_ASSET_URL_TTL — optional Go duration, default 10m, allowed 5m..60m (the floor covers the promoted-post request plus Reddit's fetch and clock skew).

See docs/knowledge/architecture/signed-creative-asset-urls.md for the security model.

Upload memory budget (UPLOAD_MEMORY_BUDGET_BYTES)

Creative-asset uploads draw on ONE per-pod memory budget: each upload's wire side (body buffer, base64 string, decoded bytes; charged before the body is read) and its decoded pixel buffer (charged before the image is decoded) come out of the same budget, so concurrent uploads cannot exceed it in total.

• UPLOAD_MEMORY_BUDGET_BYTES — optional; a byte count or a binary quantity (256Mi, 1Gi). Unset (the default), the budget is 40% of the container memory limit read from the cgroup (v2 memory.max, else v1 memory.limit_in_bytes), or 40% of 512Mi (~205Mi) when no limit is readable. It is never below 192Mi, the cost of one maximum-size upload (a smaller value is raised and logged). A malformed value stops the service at startup.

The wire side may use the budget only up to its size less 80Mi, which is kept so an upload that has already been admitted can always get its pixel charge. When the budget is exhausted an upload waits up to 250ms at each of the two stages (so at most ~500ms), then gets 503 with Retry-After: 1. The budget's size and current use are exported at /metrics as campaign_upload_memory_budget_bytes and campaign_upload_memory_budget_in_use_bytes.

Observability (OTEL_*)

OpenTelemetry is opt-in. Exporters default to none (no collector required for local runs).

• OTEL_SERVICE_NAME (default lfx-v2-campaign-service)
• OTEL_SERVICE_VERSION (default: build version)
• OTEL_EXPORTER_OTLP_PROTOCOL (default grpc) — grpc or http
• OTEL_EXPORTER_OTLP_ENDPOINT — collector endpoint
• OTEL_EXPORTER_OTLP_INSECURE (default false) — insecure when true
• OTEL_TRACES_EXPORTER (default none) — otlp or none
• OTEL_METRICS_EXPORTER (default none) — otlp or none
• OTEL_LOGS_EXPORTER (default none) — otlp or none
• OTEL_PROPAGATORS (default tracecontext,baggage) — comma-separated; jaeger supported
• OTEL_TRACES_SAMPLER (default parentbased_traceidratio when unset) — sampler type (always_on, always_off, traceidratio, parentbased_*, …)
• OTEL_TRACES_SAMPLER_ARG (default 1.0) — sampler argument; for ratio-based samplers, a value in [0.0, 1.0]

Build and run locally (against lfx-v2-dev)

In lfx-v2-dev, Postgres is RDS. The cluster exposes it as an ExternalName Service (lfx/rds-postgres). A plain kubectl port-forward svc/rds-postgres … does not work (ExternalName has no endpoints). Use a short-lived jump pod with socat, then port-forward to that pod.

Credentials live in secret lfx-v2-campaign-service-secrets (namespace lfx-v2-campaign-service), keys: host, port, username, password, dbname, engine.

# 0) Point kubectl at development (example path; adjust if needed)
export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/lfx-v2-dev}"

# 1) Confirm the secret exists (do not print the password)
kubectl -n lfx-v2-campaign-service get secret \
  lfx-v2-campaign-service-secrets

# 2) Read the RDS hostname from the secret (safe: host only)
RDS_HOST="$(kubectl -n lfx-v2-campaign-service get secret \
  lfx-v2-campaign-service-secrets \
  -o jsonpath='{.data.host}' | base64 -d)"
RDS_PORT="$(kubectl -n lfx-v2-campaign-service get secret \
  lfx-v2-campaign-service-secrets \
  -o jsonpath='{.data.port}' | base64 -d)"
RDS_PORT="${RDS_PORT:-5432}"

# Both must be non-empty before creating the jump pod
if [ -z "$RDS_HOST" ] || [ -z "$RDS_PORT" ]; then
  echo "RDS_HOST/RDS_PORT empty — refuse to create broken tunnel" >&2
  exit 1
fi
echo "tunnel target ${RDS_HOST}:${RDS_PORT}"

# 3) Start a jump pod that listens on 5432 and dials RDS.
#    Do NOT use --command (replaces the socat entrypoint →
#    "tcp-listen:…: executable file not found").
#    Do NOT use -it (Gatekeeper blocks interactive TTYs).
#    Do NOT use --rm until you are done (you need the pod alive
#    for port-forward).
#    Delete any prior failed pod first if needed:
#      kubectl -n lfx-v2-campaign-service delete pod pg-tunnel \
#        --ignore-not-found
kubectl -n lfx-v2-campaign-service delete pod pg-tunnel \
  --ignore-not-found
kubectl -n lfx-v2-campaign-service run pg-tunnel \
  --restart=Never --image=alpine/socat -- \
  tcp-listen:5432,fork,reuseaddr \
  "tcp-connect:${RDS_HOST}:${RDS_PORT}"

kubectl -n lfx-v2-campaign-service wait --for=condition=Ready \
  pod/pg-tunnel --timeout=60s

# Confirm args include the real host (not tcp-connect::)
kubectl -n lfx-v2-campaign-service get pod pg-tunnel \
  -o jsonpath='{.spec.containers[0].args}{"\n"}'

In a second terminal (leave this running — stopping it causes connection refused on /readyz):

export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/lfx-v2-dev}"

# 4) Forward laptop:5432 -> jump pod:5432
kubectl -n lfx-v2-campaign-service port-forward \
  pod/pg-tunnel 5432:5432
# Expect:
#   Forwarding from 127.0.0.1:5432 -> 5432
# Later, when the service pings, you may also see:
#   Handling connection for 5432

In a third terminal — build, load creds, run:

export KUBECONFIG="${KUBECONFIG:-$HOME/.kube/lfx-v2-dev}"

# Always use the tunnel endpoint on the laptop, not the RDS FQDN.
# If you export PGHOST from the secret's `host` key, readyz will
# time out (laptop cannot reach private RDS directly).
export PGHOST=127.0.0.1
export PGPORT=5432
export PGUSER="$(kubectl -n lfx-v2-campaign-service get secret \
  lfx-v2-campaign-service-secrets \
  -o jsonpath='{.data.username}' | base64 -d)"
export PGPASSWORD="$(kubectl -n lfx-v2-campaign-service get secret \
  lfx-v2-campaign-service-secrets \
  -o jsonpath='{.data.password}' | base64 -d)"
export PGDATABASE="$(kubectl -n lfx-v2-campaign-service get secret \
  lfx-v2-campaign-service-secrets \
  -o jsonpath='{.data.dbname}' | base64 -d)"
# **Note**: This is a Local-dev sample only (see "Local / test sample key" above).
# NEVER use this in production or shared environments!
export CREDENTIAL_ENCRYPTION_KEY='TEZYLWNhbXBhaWduLWxvY2FsLWRldi1hZXMtMjU2ISE='


# Sanity-check before starting (must be 127.0.0.1, not the RDS FQDN)
echo "PGHOST=$PGHOST PGPORT=$PGPORT PGDATABASE=$PGDATABASE"
# optional: confirm the tunnel accepts TCP
nc -z 127.0.0.1 5432 && echo "tunnel port open"

make build
make run
# On startup, the log line "dependency container initialized" must
# show database=127.0.0.1:5432/<dbname> — if it shows the RDS
# hostname, stop and fix PGHOST.

Smoke-check readiness (expects 200 / OK while the tunnel is up):

curl -sS -w "\nHTTP %{http_code}\n" http://127.0.0.1:8080/readyz
# expect body OK and HTTP 200

curl -sS -w "\nHTTP %{http_code}\n" http://127.0.0.1:8080/livez
# expect body OK and HTTP 200 (even if readyz would be 503)

Cleanup: stop the port-forward (Ctrl-C), then delete the jump pod:

kubectl -n lfx-v2-campaign-service delete pod pg-tunnel \
  --ignore-not-found

Troubleshooting

• tcp-listen:…: executable file not found — used --command. Recreate with run … -- args (no --command).
• Gatekeeper TTY warning / blocked — used -it. Omit -it.
• Pod args show tcp-connect:: — RDS_HOST/RDS_PORT were empty at create. Re-export from secret, delete pod, recreate.
• Startup log shows RDS FQDN as database — PGHOST was taken from secret host. Use export PGHOST=127.0.0.1 and restart.
• connection refused on 127.0.0.1:5432 — port-forward not running. Restart Terminal 2.
• context deadline exceeded with PGHOST=127.0.0.1 — jump pod dialing wrong/empty target, or tunnel stalled. Check pod args; recreate tunnel.

Alternatives

• VPN / direct RDS access — if your laptop can reach the RDS FQDN, skip the jump pod; set PGHOST/PGPORT from the secret host/port keys and make run.
• Local Docker / Homebrew Postgres — no tunnel; use PGHOST=127.0.0.1 with local credentials.
• CloudNativePG ClusterIP Service — kubectl port-forward svc/<cnpg-rw-service> 5432:5432 works without a jump pod.

See also specs/002-db-conn-check/quickstart.md for readiness / liveness validation scenarios.

Run in a local Kubernetes cluster

Prefer make run (above) for day-to-day Go iteration. To exercise the Helm chart — probes, secret refs, and env wiring — build an image and install with the local values override.

make helm-install-local installs into namespace lfx (see HELM_NAMESPACE in the Makefile). The chart still requires secret lfx-v2-campaign-service-secrets (keys: host, port, username, password, dbname) in that same namespace for the required PG* env refs. Without it the pod stays in CreateContainerConfigError.

Pick one of these before installing. Set HELM_NAMESPACE once and pass it to the final make helm-install-local (Makefile assigns HELM_NAMESPACE=lfx with =, so an env prefix does not override).

# Default local release namespace (Option A / C)
HELM_NS=lfx

# Option A — copy the secret from lfx-v2-dev into a local cluster.
# Use distinct source and destination contexts (adjust names to match
# `kubectl config get-contexts`). Rebuild a clean Secret so server-
# managed fields (uid, resourceVersion, …) are not re-applied.
SRC_CONTEXT="${SRC_CONTEXT:-lfx-v2-dev}"
DST_CONTEXT="${DST_CONTEXT:-kind-kind}"

kubectl --context="$SRC_CONTEXT" get secret \
  lfx-v2-campaign-service-secrets \
  -n lfx-v2-campaign-service -o json \
  | jq '{
      apiVersion: .apiVersion,
      kind: .kind,
      type: .type,
      metadata: { name: .metadata.name, namespace: "lfx" },
      data: .data
    }' \
  | kubectl --context="$DST_CONTEXT" apply -f -

# Option B — install into the namespace that already has the secret
HELM_NS=lfx-v2-campaign-service

# Option C — point PG* at a local database in values.local.yaml
# (override PGHOST/PGPORT/PGUSER/PGPASSWORD/PGDATABASE with `value:`
#  entries instead of secretKeyRef; see values.yaml for the keys)

Then:

# 1) Copy the example override (gitignored once renamed)
cp charts/lfx-v2-campaign-service/values.local.example.yaml \
   charts/lfx-v2-campaign-service/values.local.yaml
# Edit values.local.yaml as needed (encryption key sample is included).

# 2) Build the image (pullPolicy: Never in the local values file)
make docker-build

# 3) Load the image into your local cluster if needed (kind example):
#    kind load docker-image \
#      ghcr.io/linuxfoundation/lfx-v2-campaign-service/campaign-service:latest

# 4) Install / upgrade the chart (uses HELM_NS from above)
make helm-install-local HELM_NAMESPACE="$HELM_NS"

values.local.example.yaml documents the copy path and make helm-install-local target. Uninstall with make helm-uninstall.

Development

Common workflow targets (see the Makefile for the full list):

make all           # clean → apigen → fmt → lint → test → build
make clean         # remove bin/ and coverage.out
make apigen        # generate API code from design/ (required before first build)
make fmt           # format Go code (gofmt + simplify)
make check-fmt     # verify formatting (used in CI)
make lint          # run golangci-lint
make test          # run tests with race detector and coverage
make build         # build a local binary
make build-release # build a static release binary for Linux
make run           # build and run locally (needs PG* env; see above)

Verifying a gated metrics read (cmd/verify-metrics)

Before flipping REDDIT_METRICS_ENABLED, MICROSOFT_METRICS_ENABLED or TWITTER_METRICS_ENABLED, run the gated read against a live account with the same database and key settings as make run:

go run ./cmd/verify-metrics -provider reddit-ads -project <slug|lf-system> \
  -campaign <platform campaign id> -window last_7_days [-raw]

It resolves the connection and calls the dispatcher's ReadMetrics exactly as the service does, forces the flag on in its own process only, is read-only (read-only DB session; non-read upstream requests refused before sending) and redacts every secret. google-ads is refused. See docs/knowledge/code/cmd-verify-metrics.md.

MegaLinter (local)

CI runs MegaLinter on pull requests and merge-queue entries (.github/workflows/mega-linter.yaml, Go flavor v9.1.0). Config lives in .mega-linter.yml. To reproduce locally with Docker or OrbStack:

docker pull oxsecurity/megalinter-go:v9.1.0
docker run --rm \
  -e DEFAULT_WORKSPACE=/tmp/lint \
  -e GOTOOLCHAIN=auto \
  -e MEGALINTER_CONFIG=.mega-linter.yml \
  -v "$PWD:/tmp/lint:rw" \
  oxsecurity/megalinter-go:v9.1.0

Reports are written under megalinter-reports/ (gitignored). The first pull is large; a full run often takes several minutes.

For a faster secrets-only check matching the CI gitleaks step:

gitleaks detect --source . --config .gitleaks.toml

Note: if this checkout is a git worktree, the containerized git_diff linter may fail because the worktree .git path is not visible inside the container. That does not affect GitHub Actions (normal actions/checkout).

Knowledge Base (OKF)

docs/knowledge/ is an Open Knowledge Format (OKF) bundle — plain markdown with YAML frontmatter — that gives humans and AI agents a structured map of this repo's architecture, Kubernetes resources, Go packages, and feature specs. Start at docs/knowledge/index.md.

When to update it: after merging a feature PR, changing an API endpoint, adding or modifying a Helm resource, or changing a package's responsibility.

How to update it:

1. Edit the relevant existing concept file under docs/knowledge/**, or add a new one with OKF frontmatter (type, title, description) if no existing concept covers the change. Do not regenerate with go run ./cmd/okfgen — that tool bootstraps new subtrees and will overwrite hand-edited concept files.
2. Add or update the concept's * [Title](url) - description bullet in the relevant index.md.
3. Add a new file docs/knowledge/log/YYYY-MM-DD-<slug>.md (slug = ticket + short description), with a first H1 dated to match the filename followed by **Update** — <what changed and why>. One file per entry — this keeps concurrent PRs from ever editing the same log file.

Validate before pushing:

go run ./cmd/okfvalidate ./docs/knowledge

This is the same check .github/workflows/validate-okf.yml runs in CI.

Agents are expected to do this bookkeeping automatically (see CLAUDE.md); developers making manual changes should follow the same convention.