Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

ramendr-storage-ui-tests

This repository is a test harness for the RamenDR validated pattern.

Default (PATTERN_VARIANT=odf): deploys from the maintained fork elsapassaro/ramendr-starter-kit (branch ocp-4.22-rhdr-ramen) — upstream v1.3 plus QE overrides, preview RHDR (rhdr-catalog) on the odf variant, mixed 4-VM fleet (2 Linux + 2 Windows), HammerDB, ODF. redeploy.sh pins a fixed commit SHA for the local pattern install; hub Argo CD reconciles from the fork's remote branch on GitHub (see Upstream pinning below).

Partner variants: set PATTERN_VARIANT to drpartner-s4 or drpartner-minimal. All variants use the same elsapassaro fork; partner BOMs differ only in variants/<name>/ (preview RHDR is committed in git).

It contains:

  • Deployment automation (scripts/redeploy.sh) that pins the fork to an immutable commit SHA and executes the full deployment flow.
  • UI and sanity tests using Playwright + Python (tests/).
  • RamenDR data validation — HammerDB PostgreSQL TPC-C by default (dr-validation/README.md, dr-validation/DATABASE-SCHEMA.md).

CI / Tekton

Tekton pipelines, tasks, PipelineRun examples, and CI image build configuration are maintained in ramendr-pipelines (internal Red Hat GitLab). See that repository's README for setup instructions. For credentials shared across TaskRuns, CI_SHARED_WORKSPACE_GID must match a group granted to every authorized pod through fsGroup or supplementalGroups. Without this variable, credential files and directories remain owner-only.

What this repo does

scripts/redeploy.sh will:

  1. Clone the fork elsapassaro/ramendr-starter-kit at the pinned commit SHA 254260328e7e7ec661504b6c540cb02d956d534e into .work/upstream/ramendr-starter-kit and set main.variant to PATTERN_VARIANT (odf by default).
  2. Patch upstream pattern.sh to run podman without a TTY (required for CI — upstream uses podman run -it which fails when stdin/stdout are not a terminal). No local file injection into ArgoCD's sync path is needed: all customizations live in the fork.
  3. Provision hub + two spokes on AWS (BYOC spokes).
  4. Copy your VALUES_SECRET into .work/values-secret.yaml, merge fresh spoke kubeconfig paths (ocp-primary_cluster_kubeconfig, ocp-secondary_cluster_kubeconfig), and run upstream pattern.sh make install-byoc (loads secrets to Vault, validates BYOC, deploys pattern). Preview RHDR ImageDigestMirrorSet is applied by GitOps extraObjects.rhdr-fbc-idms during that install.
  5. Wait for ExternalSecrets to create auto-import-secret and admin-kubeconfig on the hub; ACM imports the spokes.

BYOC: The fork sets byoc: true. Your ~/values-secret.yaml may omit spoke kubeconfigs or contain stale paths from a previous deploy — redeploy.sh always refreshes them in .work/values-secret.yaml (gitignored) before install-byoc. Your source file is never modified.

Why a fork? Hub Argo CD fetches values from the remote GitHub repository — local copies placed next to the checkout are invisible to it. Both redeploy.sh and hub Applications should track fork branch ocp-4.22-rhdr-ramen so GitOps matches the local pin.

Upstream pinning

Two different upstream references are in play:

Consumer Source Default
redeploy.sh local checkout UPSTREAM_REF commit SHA checked out into .work/upstream/ All variants: fork 254260328e7e7ec661504b6c540cb02d956d534e (ocp-4.22-rhdr-ramen)
Hub Argo CD Applications Remote git on GitHub Fork branch ocp-4.22-rhdr-ramen (main.variant: odf on tip). Partner variants need a fork branch/commit with matching main.variant for stable GitOps.

To test a different fork commit locally, set UPSTREAM_REPO and UPSTREAM_REF before running redeploy.sh. For Argo CD to match that commit, push it to the tracked branch or pin targetRevision on the hub Applications. For PATTERN_VARIANT, GitOps is only stable when the remote values-global.yaml already has that main.variant (fork + commit, then set UPSTREAM_REPO).

Prerequisites

The deployment script expects tools similar to the original flow:

  • oc
  • openshift-install
  • aws
  • podman — must be running when the pattern deploy starts (pattern.sh uses a utility container). On macOS, start the VM before a long redeploy or rely on redeploy.sh to auto-start it: podman machine start
  • git
  • GNU bash 4+ — redeploy.sh, stabilize-windows-vms.sh, and ensure-windows-openssh.sh use mapfile (the Bash 3.2 shipped with macOS is too old). On macOS: brew install bash (those scripts re-exec via /opt/homebrew/bin/bash automatically when present)
  • python3 with PyYAML — merges spoke kubeconfig paths into .work/values-secret.yaml before install-byoc (redeploy.sh auto-installs via pip install --user PyYAML when missing; also listed in requirements.txt)
  • virtctl — Windows edge VM SSH verification during redeploy (brew install virtctl on macOS when REQUIRE_WINDOWS_VMS=1)
  • sshpass — Windows Administrator password login via virtctl ssh when windows-admin is in VALUES_SECRET (brew install sshpass / CI image via EPEL)
  • jq — used by the golden-image Ansible playbook and several redeploy helpers
  • ansible-playbook — runs scripts/ansible/odf_fix_dataimportcrons.yml during spoke golden-image fix-up (optional; redeploy falls back to oc if this step fails)

You will also need AWS credentials configured for the AWS account used for cluster installs and Route53 operations (the script uses the AWS CLI).

Ansible golden-image playbook

During redeploy, redeploy.sh may run scripts/ansible/odf_fix_dataimportcrons.yml on the host machine where you launch redeploy (connection: local, hosts: localhost). It uses oc with spoke KUBECONFIG to clean up CDI DataImportCron objects when the virtualization default storage class differs from the cluster default.

That playbook needs:

  • Ansible collection kubernetes.core — provides the k8s_info and k8s modules used to list and delete CDI objects
  • Python package kubernetes — required by those modules at runtime

Install the collection with ansible-galaxy collection install kubernetes.core. Install the Python package for the same interpreter Ansible uses for localhost (not necessarily the interpreter in an activated virtualenv). On Fedora, dnf install python3-kubernetes may target a different Python than Ansible auto-discovers (e.g. package on 3.14 while Ansible picks 3.12).

redeploy.sh auto-selects the first interpreter that can import kubernetes (default /usr/bin/python3, then 3.14/3.13/3.12) and sets ANSIBLE_PYTHON_INTERPRETER for the playbook. Override with export ANSIBLE_PYTHON_INTERPRETER=/usr/bin/python3.14 if needed. If no suitable interpreter is found, the playbook is skipped and redeploy.sh continues with direct oc golden-image cleanup instead.

requirements.txt covers UI tests (Playwright/pytest) only; it does not install these Ansible dependencies.

install-config files (examples)

The redeploy flow requires three openshift-install directories containing install-config.yaml.bak:

  • ~/git/hub-cluster-install/install-config.yaml.bak
  • ~/git/ocp-primary-install/install-config.yaml.bak
  • ~/git/ocp-secondary-install/install-config.yaml.bak

This repo provides examples with placeholders only under install-config-examples/. Copy them into your install dirs and replace placeholders:

cp install-config-examples/hub/install-config.yaml.bak.example ~/git/hub-cluster-install/install-config.yaml.bak
cp install-config-examples/ocp-primary/install-config.yaml.bak.example ~/git/ocp-primary-install/install-config.yaml.bak
cp install-config-examples/ocp-secondary/install-config.yaml.bak.example ~/git/ocp-secondary-install/install-config.yaml.bak

Do not commit real pullSecret or sshKey values.

Secrets policy

Do not commit secrets to this repository.

  • Provide VALUES_SECRET (default: ~/values-secret.yaml) locally/through CI secret injection.
  • Keep kubeconfigs and install dirs out of git (see .gitignore).
  • Use the upstream template as a reference: values-secret.yaml.template (v1.3 / PATTERN_VARIANT=drpartner-s4 also needs the S4 fragment dr-validation/examples/values-secret-v2-s4.fragment.yaml)
  • For regional-dr cluster private-key ExternalSecrets, ensure ~/values-secret.yaml includes hub privatekey paths (compare with your team's file via private DM), for example:
- name: privatekey
  fields:
    - name: ssh-privatekey
      path: ~/.ssh/id_ed25519
    - name: ssh-publickey
      path: ~/.ssh/id_ed25519.pub
  • For Windows edge VMs (private quay.io/martjack/* images), add privatevm-credentials (Quay robot accessKeyId / secretKey) and windows-admin (password for local Administrator SSH). See dr-validation/examples/values-secret-v2-windows.fragment.yaml. For HammerDB on Windows SQL Server, also add mssql-hammerdb (sa_password, user, password) or export DR_VALIDATION_MSSQL_* before redeploy.
  • For local Jira Test Result reporting credentials (JIRA_BASE_URL / JIRA_EMAIL / JIRA_API_TOKEN, see docs/jira-test-result-reporting.md), copy .env.example to .env and fill in real values. .env is gitignored and loaded automatically for pytest runs only (see config/env_file.py); exported environment variables always take priority over .env. This file is unrelated to, and not read by, any Cursor MCP server configuration.

Customizing the deployment

All environment-specific values (additional VM disks, BYOC cluster names, ODF channel pins, cost-optimized instance profiles) live in the fork's ocp-4.22-rhdr-ramen branch under overrides/ and values-hub.yaml. To adapt the deployment for a different AWS account or region:

  1. Fork elsapassaro/ramendr-starter-kit (or push a new branch on the existing fork).
  2. Edit the relevant overrides/*.yaml files in that branch.
  3. Point redeploy.sh at your fork by setting UPSTREAM_REPO and UPSTREAM_REF:
export UPSTREAM_REPO=https://github.com/<your-org>/ramendr-starter-kit
export UPSTREAM_REF=<commit-sha-or-branch>
./scripts/redeploy.sh --pattern-only

Pattern variants

All deploys use PATTERN_VARIANT (main.variant in upstream values-global.yaml). Values live under variants/<name>/. See the validated patterns variants write-up.

PATTERN_VARIANT Upstream default Purpose
odf (default) QE fork ocp-4.22-rhdr-ramen Preview RHDR (rhdr-catalog), mixed 4-VM fleet (2 Linux + 2 Windows), HammerDB, ODF
drpartner-s4 Same fork Partner CSI + S4; preview RHDR from upstream v1.3; DRPolicy 2m-drpolicy; no VMs
drpartner-minimal Same fork Partner CSI only; preview RHDR from upstream v1.3; no S4 / DRPolicy / VMs
# Default (odf) — no env var needed:
./scripts/redeploy.sh --pattern-only

# Partner variant — stable GitOps needs a fork that commits main.variant:
export UPSTREAM_REPO=https://github.com/<your-org>/ramendr-starter-kit
export PATTERN_VARIANT=drpartner-s4   # or drpartner-minimal
# drpartner-s4: merge dr-validation/examples/values-secret-v2-s4.fragment.yaml first
./scripts/redeploy.sh --pattern-only

redeploy.sh always writes main.variant in the local checkout and sets byoc: true in overrides/values-cluster-names.yaml. Partner BOMs skip Windows VM stabilize, HammerDB bootstrap, and ODF golden-image fix-up (override with REQUIRE_WINDOWS_VMS / SKIP_*).

GitOps: hub Argo CD reads values-global.yaml from the git remote, not the local patch. redeploy.sh fails if that git revision still has a different main.variant. The fork at 91b7890… has main.variant: odf committed. Partner deploys need a fork that commits drpartner-s4 or drpartner-minimal, then UPSTREAM_REPO pointed at that fork.

ACM spoke placement still uses ManagedCluster label clusterGroup=resilient. That label is independent of main.variant.

Smoke tests skip VM / ODF / MirrorPeer assertions on partner variants and instead check vp-s4-storage plus 2m-drpolicy (drpartner-s4) or their absence (drpartner-minimal). Export the same PATTERN_VARIANT when you run pytest. Partner CSI itself is not something the pattern or these tests can verify.

Usage

Two environment variables are required and have no default values:

Variable Description
BASE_DOMAIN Base DNS domain for the clusters (must be delegated to Route53 in your account)
HOSTED_ZONE_ID Route53 hosted zone ID for that domain

Export them before running the script:

export BASE_DOMAIN=your-domain.example.com
export HOSTED_ZONE_ID=Z0123456789ABCDEFGHIJ

Then from the repo root:

./scripts/redeploy.sh --help

Run either a full redeploy or pattern-only on an existing hub, depending on your workflow.

Code quality (pre-commit)

This repo uses pre-commit to enforce linting and formatting on every commit (Python via ruff, shell via shellcheck, YAML via yamllint, plus general hygiene hooks).

Install

pip install pre-commit

Or, using the repo's virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install pre-commit

Activate the hooks

Run once after cloning to install the git hook:

pre-commit install

After this, hooks run automatically on every git commit.

Run manually before pushing

pre-commit run --all-files

If a hook reformats files, stage the changes and commit again.

CI enforcement

The same checks run automatically on every pull request via .github/workflows/pre-commit.yaml. PRs that fail the checks cannot be merged.

RamenDR data validation

Default mode is HammerDB TPC-C on PostgreSQL (DR_VALIDATION_MODE=hammerdb) on rhel9-node-001 (one of two Linux edge VMs). The default fleet is four VMs in gitops-vms: 2 Linux + 1 Windows Server 2022 + 1 Windows Server 2025. Add privatevm-credentials (Quay robot for quay.io/martjack/* images) and windows-admin (local Administrator password for Windows SSH verification) to ~/values-secret.yaml before redeploy. A full ./scripts/redeploy.sh run automatically bootstraps PostgreSQL/SQL Server and builds populated TPC-C tables (customers with IDs, orders, stock, …). Autopilot and the audit writer stay stopped so Ceph mirroring is not racing a live OLTP firehose. Smoke tests assert the schema is present; sanity starts writers after DRPC is Healthy, captures a fresh baseline immediately before Initiate, and stops writers on teardown.

For manual DR, start load first:

./scripts/dr-validation/start-hammerdb-load-incluster.sh
./scripts/dr-validation/status-hammerdb.sh
./scripts/dr-validation/save-db-baseline-snapshot.sh

status-hammerdb.sh (no --schema-only) must pass on every VM so the baseline includes a fresh audit row, not leftover install seed data.

Stop writers afterward with ./scripts/dr-validation/stop-hammerdb-load-incluster.sh.

After DR in the UI, run:

./scripts/dr-validation/post-dr-automation.sh

Defaults enforce a 2-minute RPO and 20-minute RTO (warn at 15 minutes): DR_VALIDATION_MAX_RPO_SECONDS=120, RAMENDR_SANITY_MAX_RTO_SECONDS=1200, and RAMENDR_SANITY_RTO_WARN_SECONDS=900.

See dr-validation/DATABASE-SCHEMA.md for table details.

Set SKIP_DR_VALIDATION=1 only to intentionally skip automatic DR validation. Set DR_VALIDATION_MODE=timestamp for the legacy per-VM timestamp log mode.

If redeploy was interrupted, re-run automatic bootstrap on an existing environment:

export KUBECONFIG=~/git/hub-cluster-install/auth/kubeconfig
./scripts/redeploy.sh --dr-bootstrap-only

See dr-validation/README.md and docs/QA-DR-data-validation.md.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages