This repository is a test harness for the RamenDR validated pattern.
Default (PATTERN_VARIANT=odf): deploys from the maintained fork
elsapassaro/ramendr-starter-kit (branch ocp-4.22-rhdr-ramen) —
upstream v1.3 plus QE overrides, preview RHDR (rhdr-catalog) on the odf variant, mixed 4-VM fleet
(2 Linux + 2 Windows), HammerDB, ODF.
redeploy.sh pins a fixed commit SHA for the local pattern install; hub Argo CD reconciles
from the fork's remote branch on GitHub (see Upstream pinning below).
Partner variants: set PATTERN_VARIANT to drpartner-s4 or drpartner-minimal. All variants use the same
elsapassaro fork; partner BOMs differ only in variants/<name>/ (preview RHDR is committed in git).
It contains:
- Deployment automation (
scripts/redeploy.sh) that pins the fork to an immutable commit SHA and executes the full deployment flow. - UI and sanity tests using Playwright + Python (
tests/). - RamenDR data validation — HammerDB PostgreSQL TPC-C by default (
dr-validation/README.md,dr-validation/DATABASE-SCHEMA.md).
Tekton pipelines, tasks, PipelineRun examples, and CI image build configuration are maintained in
ramendr-pipelines (internal Red Hat GitLab).
See that repository's README for setup instructions. For credentials shared across TaskRuns,
CI_SHARED_WORKSPACE_GID must match a group granted to every authorized pod through fsGroup or
supplementalGroups. Without this variable, credential files and directories remain owner-only.
scripts/redeploy.sh will:
- Clone the fork
elsapassaro/ramendr-starter-kitat the pinned commit SHA254260328e7e7ec661504b6c540cb02d956d534einto.work/upstream/ramendr-starter-kitand setmain.varianttoPATTERN_VARIANT(odfby default). - Patch upstream
pattern.shto runpodmanwithout a TTY (required for CI — upstream usespodman run -itwhich fails when stdin/stdout are not a terminal). No local file injection into ArgoCD's sync path is needed: all customizations live in the fork. - Provision hub + two spokes on AWS (BYOC spokes).
- Copy your
VALUES_SECRETinto.work/values-secret.yaml, merge fresh spoke kubeconfig paths (ocp-primary_cluster_kubeconfig,ocp-secondary_cluster_kubeconfig), and run upstreampattern.sh make install-byoc(loads secrets to Vault, validates BYOC, deploys pattern). Preview RHDR ImageDigestMirrorSet is applied by GitOpsextraObjects.rhdr-fbc-idmsduring that install. - Wait for ExternalSecrets to create
auto-import-secretandadmin-kubeconfigon the hub; ACM imports the spokes.
BYOC: The fork sets
byoc: true. Your~/values-secret.yamlmay omit spoke kubeconfigs or contain stale paths from a previous deploy —redeploy.shalways refreshes them in.work/values-secret.yaml(gitignored) beforeinstall-byoc. Your source file is never modified.Why a fork? Hub Argo CD fetches values from the remote GitHub repository — local copies placed next to the checkout are invisible to it. Both
redeploy.shand hub Applications should track fork branchocp-4.22-rhdr-ramenso GitOps matches the local pin.
Two different upstream references are in play:
| Consumer | Source | Default |
|---|---|---|
redeploy.sh local checkout |
UPSTREAM_REF commit SHA checked out into .work/upstream/ |
All variants: fork 254260328e7e7ec661504b6c540cb02d956d534e (ocp-4.22-rhdr-ramen) |
| Hub Argo CD Applications | Remote git on GitHub | Fork branch ocp-4.22-rhdr-ramen (main.variant: odf on tip). Partner variants need a fork branch/commit with matching main.variant for stable GitOps. |
To test a different fork commit locally, set UPSTREAM_REPO and UPSTREAM_REF before running
redeploy.sh. For Argo CD to match that commit, push it to the tracked branch or pin
targetRevision on the hub Applications. For PATTERN_VARIANT, GitOps is only stable when
the remote values-global.yaml already has that main.variant (fork + commit, then set
UPSTREAM_REPO).
The deployment script expects tools similar to the original flow:
ocopenshift-installawspodman— must be running when the pattern deploy starts (pattern.shuses a utility container). On macOS, start the VM before a long redeploy or rely onredeploy.shto auto-start it:podman machine startgit- GNU bash 4+ —
redeploy.sh,stabilize-windows-vms.sh, andensure-windows-openssh.shusemapfile(the Bash 3.2 shipped with macOS is too old). On macOS:brew install bash(those scripts re-exec via/opt/homebrew/bin/bashautomatically when present) python3with PyYAML — merges spoke kubeconfig paths into.work/values-secret.yamlbeforeinstall-byoc(redeploy.shauto-installs viapip install --user PyYAMLwhen missing; also listed inrequirements.txt)virtctl— Windows edge VM SSH verification during redeploy (brew install virtctlon macOS whenREQUIRE_WINDOWS_VMS=1)sshpass— Windows Administrator password login viavirtctl sshwhenwindows-adminis inVALUES_SECRET(brew install sshpass/ CI image via EPEL)jq— used by the golden-image Ansible playbook and several redeploy helpersansible-playbook— runsscripts/ansible/odf_fix_dataimportcrons.ymlduring spoke golden-image fix-up (optional; redeploy falls back toocif this step fails)
You will also need AWS credentials configured for the AWS account used for cluster installs and Route53 operations (the script uses the AWS CLI).
During redeploy, redeploy.sh may run scripts/ansible/odf_fix_dataimportcrons.yml on the host
machine where you launch redeploy (connection: local, hosts: localhost). It uses oc with
spoke KUBECONFIG to clean up CDI DataImportCron objects when the virtualization default storage
class differs from the cluster default.
That playbook needs:
- Ansible collection
kubernetes.core— provides thek8s_infoandk8smodules used to list and delete CDI objects - Python package
kubernetes— required by those modules at runtime
Install the collection with ansible-galaxy collection install kubernetes.core. Install the Python
package for the same interpreter Ansible uses for localhost (not necessarily the interpreter
in an activated virtualenv). On Fedora, dnf install python3-kubernetes may target a different
Python than Ansible auto-discovers (e.g. package on 3.14 while Ansible picks 3.12).
redeploy.sh auto-selects the first interpreter that can import kubernetes (default
/usr/bin/python3, then 3.14/3.13/3.12) and sets ANSIBLE_PYTHON_INTERPRETER for the playbook.
Override with export ANSIBLE_PYTHON_INTERPRETER=/usr/bin/python3.14 if needed. If no suitable
interpreter is found, the playbook is skipped and redeploy.sh continues with direct oc
golden-image cleanup instead.
requirements.txt covers UI tests (Playwright/pytest) only; it does not install these Ansible
dependencies.
The redeploy flow requires three openshift-install directories containing install-config.yaml.bak:
~/git/hub-cluster-install/install-config.yaml.bak~/git/ocp-primary-install/install-config.yaml.bak~/git/ocp-secondary-install/install-config.yaml.bak
This repo provides examples with placeholders only under install-config-examples/.
Copy them into your install dirs and replace placeholders:
cp install-config-examples/hub/install-config.yaml.bak.example ~/git/hub-cluster-install/install-config.yaml.bak
cp install-config-examples/ocp-primary/install-config.yaml.bak.example ~/git/ocp-primary-install/install-config.yaml.bak
cp install-config-examples/ocp-secondary/install-config.yaml.bak.example ~/git/ocp-secondary-install/install-config.yaml.bakDo not commit real pullSecret or sshKey values.
Do not commit secrets to this repository.
- Provide
VALUES_SECRET(default:~/values-secret.yaml) locally/through CI secret injection. - Keep kubeconfigs and install dirs out of git (see
.gitignore). - Use the upstream template as a reference: values-secret.yaml.template
(v1.3 /
PATTERN_VARIANT=drpartner-s4also needs the S4 fragmentdr-validation/examples/values-secret-v2-s4.fragment.yaml) - For regional-dr cluster private-key ExternalSecrets, ensure
~/values-secret.yamlincludes hubprivatekeypaths (compare with your team's file via private DM), for example:
- name: privatekey
fields:
- name: ssh-privatekey
path: ~/.ssh/id_ed25519
- name: ssh-publickey
path: ~/.ssh/id_ed25519.pub- For Windows edge VMs (private
quay.io/martjack/*images), addprivatevm-credentials(Quay robotaccessKeyId/secretKey) andwindows-admin(passwordfor localAdministratorSSH). Seedr-validation/examples/values-secret-v2-windows.fragment.yaml. For HammerDB on Windows SQL Server, also addmssql-hammerdb(sa_password,user,password) or exportDR_VALIDATION_MSSQL_*before redeploy. - For local Jira Test Result reporting credentials (
JIRA_BASE_URL/JIRA_EMAIL/JIRA_API_TOKEN, seedocs/jira-test-result-reporting.md), copy.env.exampleto.envand fill in real values..envis gitignored and loaded automatically forpytestruns only (seeconfig/env_file.py); exported environment variables always take priority over.env. This file is unrelated to, and not read by, any Cursor MCP server configuration.
All environment-specific values (additional VM disks, BYOC cluster names, ODF channel pins,
cost-optimized instance profiles) live in the fork's ocp-4.22-rhdr-ramen branch under overrides/ and
values-hub.yaml. To adapt the deployment for a different AWS account or region:
- Fork
elsapassaro/ramendr-starter-kit(or push a new branch on the existing fork). - Edit the relevant
overrides/*.yamlfiles in that branch. - Point
redeploy.shat your fork by settingUPSTREAM_REPOandUPSTREAM_REF:
export UPSTREAM_REPO=https://github.com/<your-org>/ramendr-starter-kit
export UPSTREAM_REF=<commit-sha-or-branch>
./scripts/redeploy.sh --pattern-onlyAll deploys use PATTERN_VARIANT (main.variant in upstream values-global.yaml).
Values live under variants/<name>/. See the
validated patterns variants write-up.
PATTERN_VARIANT |
Upstream default | Purpose |
|---|---|---|
odf (default) |
QE fork ocp-4.22-rhdr-ramen |
Preview RHDR (rhdr-catalog), mixed 4-VM fleet (2 Linux + 2 Windows), HammerDB, ODF |
drpartner-s4 |
Same fork | Partner CSI + S4; preview RHDR from upstream v1.3; DRPolicy 2m-drpolicy; no VMs |
drpartner-minimal |
Same fork | Partner CSI only; preview RHDR from upstream v1.3; no S4 / DRPolicy / VMs |
# Default (odf) — no env var needed:
./scripts/redeploy.sh --pattern-only
# Partner variant — stable GitOps needs a fork that commits main.variant:
export UPSTREAM_REPO=https://github.com/<your-org>/ramendr-starter-kit
export PATTERN_VARIANT=drpartner-s4 # or drpartner-minimal
# drpartner-s4: merge dr-validation/examples/values-secret-v2-s4.fragment.yaml first
./scripts/redeploy.sh --pattern-onlyredeploy.sh always writes main.variant in the local checkout and sets byoc: true in
overrides/values-cluster-names.yaml. Partner BOMs skip Windows VM stabilize, HammerDB bootstrap,
and ODF golden-image fix-up (override with REQUIRE_WINDOWS_VMS / SKIP_*).
GitOps: hub Argo CD reads values-global.yaml from the git remote, not the local patch.
redeploy.sh fails if that git revision still has a different main.variant. The fork at
91b7890… has main.variant: odf committed. Partner deploys need a fork that commits
drpartner-s4 or drpartner-minimal, then UPSTREAM_REPO pointed at that fork.
ACM spoke placement still uses ManagedCluster label clusterGroup=resilient. That label is
independent of main.variant.
Smoke tests skip VM / ODF / MirrorPeer assertions on partner variants and instead check
vp-s4-storage plus 2m-drpolicy (drpartner-s4) or their absence (drpartner-minimal). Export the same
PATTERN_VARIANT when you run pytest. Partner CSI itself is not something the pattern
or these tests can verify.
Two environment variables are required and have no default values:
| Variable | Description |
|---|---|
BASE_DOMAIN |
Base DNS domain for the clusters (must be delegated to Route53 in your account) |
HOSTED_ZONE_ID |
Route53 hosted zone ID for that domain |
Export them before running the script:
export BASE_DOMAIN=your-domain.example.com
export HOSTED_ZONE_ID=Z0123456789ABCDEFGHIJThen from the repo root:
./scripts/redeploy.sh --helpRun either a full redeploy or pattern-only on an existing hub, depending on your workflow.
This repo uses pre-commit to enforce linting and formatting on every commit (Python via ruff, shell via shellcheck, YAML via yamllint, plus general hygiene hooks).
pip install pre-commitOr, using the repo's virtual environment:
python3 -m venv .venv
source .venv/bin/activate
pip install pre-commitRun once after cloning to install the git hook:
pre-commit installAfter this, hooks run automatically on every git commit.
pre-commit run --all-filesIf a hook reformats files, stage the changes and commit again.
The same checks run automatically on every pull request via .github/workflows/pre-commit.yaml.
PRs that fail the checks cannot be merged.
Default mode is HammerDB TPC-C on PostgreSQL (DR_VALIDATION_MODE=hammerdb) on
rhel9-node-001 (one of two Linux edge VMs). The default fleet is four VMs in
gitops-vms: 2 Linux + 1 Windows Server 2022 + 1 Windows Server 2025. Add
privatevm-credentials (Quay robot for quay.io/martjack/* images) and
windows-admin (local Administrator password for Windows SSH verification) to
~/values-secret.yaml before redeploy. A full ./scripts/redeploy.sh run automatically bootstraps PostgreSQL/SQL Server and
builds populated TPC-C tables (customers with IDs, orders, stock, …). Autopilot and the
audit writer stay stopped so Ceph mirroring is not racing a live OLTP firehose.
Smoke tests assert the schema is present; sanity starts writers after DRPC is Healthy,
captures a fresh baseline immediately before Initiate, and stops writers on teardown.
For manual DR, start load first:
./scripts/dr-validation/start-hammerdb-load-incluster.sh
./scripts/dr-validation/status-hammerdb.sh
./scripts/dr-validation/save-db-baseline-snapshot.shstatus-hammerdb.sh (no --schema-only) must pass on every VM so the baseline includes a fresh audit row, not leftover install seed data.
Stop writers afterward with ./scripts/dr-validation/stop-hammerdb-load-incluster.sh.
After DR in the UI, run:
./scripts/dr-validation/post-dr-automation.shDefaults enforce a 2-minute RPO and 20-minute RTO (warn at 15 minutes):
DR_VALIDATION_MAX_RPO_SECONDS=120, RAMENDR_SANITY_MAX_RTO_SECONDS=1200, and
RAMENDR_SANITY_RTO_WARN_SECONDS=900.
See dr-validation/DATABASE-SCHEMA.md for table details.
Set SKIP_DR_VALIDATION=1 only to intentionally skip automatic DR validation.
Set DR_VALIDATION_MODE=timestamp for the legacy per-VM timestamp log mode.
If redeploy was interrupted, re-run automatic bootstrap on an existing environment:
export KUBECONFIG=~/git/hub-cluster-install/auth/kubeconfig
./scripts/redeploy.sh --dr-bootstrap-onlySee dr-validation/README.md and docs/QA-DR-data-validation.md.