Draft
Conversation
marcleblanc2
force-pushed
the
prometheus-readonly-root-filesystem
branch
from
a10dc49 to
c5a28ed
Compare
The Prometheus container previously required readOnlyRootFilesystem: false because prom-wrapper writes the Alertmanager configuration file to /sg_config_prometheus/alertmanager.yml at runtime whenever site config observability.alerts or SMTP settings change. Additionally, Alertmanager stores state (silences, notification log) at /alertmanager/. This change enables readOnlyRootFilesystem: true by: 1. Adding an emptyDir volume mounted at /alertmanager for Alertmanager state storage (--storage.path). 2. Setting ALERTMANAGER_CONFIG_PATH=/alertmanager/alertmanager.yml so prom-wrapper writes the Alertmanager config to the writable emptyDir instead of the read-only image layer at /sg_config_prometheus/. The /prometheus TSDB data path was already handled by the existing PVC, and /sg_prometheus_add_ons is already a ConfigMap mount. The baked-in alert rules at /sg_config_prometheus/ are read-only at runtime (only written at image build time), so they remain accessible on the read-only root filesystem. Resolves: https://github.com/sourcegraph/sourcegraph/issues/34012 Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019dbe72-f13d-7233-aa22-cbeb04d03182
marcleblanc2
force-pushed
the
prometheus-readonly-root-filesystem
branch
from
c5a28ed to
b827bfa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Prometheus container has
readOnlyRootFilesystem: false(see sourcegraph/sourcegraph#34012), which is undesirable for security-hardened environments. The reason it was required is that prom-wrapper writes the Alertmanager configuration to/sg_config_prometheus/alertmanager.ymlat runtime wheneverobservability.alertsor SMTP settings change in site config, and Alertmanager stores state at/alertmanager/.Solution
This PR enables
readOnlyRootFilesystem: trueby:Adding an
emptyDirvolume at/alertmanager— provides writable storage for Alertmanager state (--storage.path) and the Alertmanager config file.Setting
ALERTMANAGER_CONFIG_PATH=/alertmanager/alertmanager.yml— redirects prom-wrapper to write the Alertmanager config to the writable emptyDir instead of the read-only image layer at/sg_config_prometheus/.Why this is safe
/prometheus(TSDB data)/sg_prometheus_add_ons(Helm ConfigMap)/sg_config_prometheus(baked-in alert rules + prometheus.yml)/alertmanager(Alertmanager state + config)Changes
values.yaml: SetreadOnlyRootFilesystem: true, removed outdated comment referencing #34012prometheus.Deployment.yaml: AddedALERTMANAGER_CONFIG_PATHenv var,alertmanager-dataemptyDir volume and mountREADME.md: Auto-regenerated by helm-docs