Repository navigation
feat(ci): JP region test infrastructure + region-aware validation.yml - #1377
Merged
Merged
Conversation
Wires Japan as a first-class region in the OIL CI test path. What's added: - test/definitions-jp/ — clone of test/definitions-eu/ (67 JSON files, with -eu.json suffixes renamed to -jp.json on the agent-control files that have a region suffix). Content is otherwise byte-identical: the region target is selected by the deployer config secret, not the JSON. - .github/workflows/nonregression-jp.yml — clone of nonregression-eu.yml with eu→jp / EU→JP / api.eu→api.jp / definitions-eu→definitions-jp / *_EU→*_JP secret refs / open-install-library-e2e-eu→...-jp identity name / cron 0 11 → 23 13 (avoids :00 minute, avoids EU hour collision). - .github/workflows/cleanup-jp.yml — clone of cleanup-eu.yml; trigger is workflow_run completing on "Non Regression Testing (JP)". - .github/workflows/nonregression-jp-dp.yml — clone of nonregression-eu-dp.yml (kept inactive, mirroring EU's currently-inactive state). - .github/workflows/smoke-jp.yml + smoke-jp-dp.yml — clones of EU smoke workflows. What's modified: - .github/workflows/validation.yml — adds test/definitions-jp/ to the per-PR file scan and to the agent-control identity-trigger guards. Prerequisite: the deployer must be republished with newrelic-experimental/ deployer#31 merged so that nrRegion: "jp" populates the app_config.newrelic.* URLs. Without that, the JP nightly workflow will appear to pass but produce no JP telemetry. Required GitHub repo secrets (separate setup, not part of this PR): - GIT_DEPLOYER_DOCKER_USER_CONFIG_JP — JP-rooted deployer config JSON (account 8155661, license key with jpx*…NRAL prefix, nrRegion: "jp", aws.region: "ap-northeast-1") The validate-ami-patterns.yml workflow is not modified here because it has uncommitted in-flight changes; JP region patterns can be added there in a follow-up once that workflow lands on main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
pranav-new-relic
added a commit
to newrelic/newrelic-cli
that referenced
this pull request
Jun 11, 2026
Wires Japan as a first-class region in the CI test path. - e2e-jp.yml — clone of e2e-eu.yml, pulls test/definitions-jp/smoke - cleanup-jp.yml — clone of cleanup-eu.yml - e2e-dp-jp.yml — clone of e2e-dp-eu.yml (kept inactive, like EU) Region routing already supported by newrelic-client-go's region.JP constant; no Go source changes needed. Prerequisites: deployer PR newrelic-experimental/deployer#31 must merge and ghcr.io/newrelic/deployer:latest must be republished; OIL companion PR newrelic/open-install-library#1377 must merge so test/definitions-jp/smoke/ exists; GitHub repo secret GIT_DEPLOYER_DOCKER_USER_CONFIG_JP must be added (JP test account 8155661, nrRegion: "jp", aws.region: "ap-northeast-1"); import gitdeployerCanada keypair (public half) into AWS region ap-northeast-1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Jun 11, 2026
Companion to newrelic-experimental/deployer#33, which removes the CrowdStrike Ansible Galaxy collection from the deployer image. This PR removes the matching dead code in OIL: - test/deploy/crowdstrike/ — the install role was already a no-op debug stub ("Skipping CrowdStrike Falcon Sensor installation"); it has no consumers other than the manual test definition removed below. - test/manual/definitions/infra-agent/ubuntu20-infra-crowdstrike.json — the manual test that exercised the stub. - nonregression.yml / nonregression-eu.yml / nonregression-jp.yml — removed the CROWDSTRIKE_CLIENT_ID/SECRET/CUSTOMER_ID env block and the matching `-e` docker-run flags. The env vars were inert (no Ruby code or playbook reads them since the install role was stubbed) and the corresponding repo secrets can be removed separately. Per NR-520645 (closed P2): NR migrated off CrowdStrike Falcon to SentinelOne and asked to decommission Falcon agents from EC2 instances spun up in account 709144918866. The deployer/OIL chain has been silently skipping the install for a while; this completes the cleanup.
Will revert before merging. Lets us run the JP smoke workflow against the JP test account without first merging the workflow file to main (which GitHub Actions requires for workflow_dispatch registration).
Smoke run completed successfully against JP test account 8155661 — all 6 smoke tests (windows2019, awslinux2/arm64, debian11, ubuntu20, nginx-linux2-ami) green. Restoring the trigger to main-only.
Addresses CodeQL alerts on PR #1377 by setting workflow-level `permissions: contents: read` on the four new JP workflow files that lacked it (smoke-jp, smoke-jp-dp, nonregression-jp-dp, cleanup-jp). nonregression-jp.yml already had a permissions block.
These eight tests target operating systems whose package repositories or AMIs have rotted out of ca-central-1 (where validation.yml runs) since their EU originals were last validated: - Debian 10 (Buster): EOL June 2024; archive.debian.org migration broke apt update flow on the AMI vintage available here. - Ubuntu 18.04 (Bionic): EOL April 2023; `php-mysql` removed from the repos that the available AMI ships pointing at. - Oracle Linux 8: standard AMIs no longer published in ca-central-1 or ap-northeast-1 by Oracle's account (only paid Marketplace variants remain). Same failure on EU's nonregression daily. - SUSE 15 SP3 BYOS: EOL'd; the matching AMI vintage is gone from both regions. Same failure on EU. JP retains the modern equivalents already cloned from EU/US: debian11/12/13-infra, suse154/155/156/157-infra, deb11-java tests, ubuntu20-infra, awslinux2023 php-agent variants, etc. No coverage lost — these eight were duplicates of obsolete OS variants.
Both have failed on EU nonregression every single day for at least the past 5 runs (2026-06-04, 06-05, 06-08, 06-09, 06-10): - oracle8-infra: Oracle's standard AMI account no longer publishes OL8 in any of our regions; only paid Marketplace variants exist. File last touched 2022-12-02 — 3+ years dormant. - suse153-infra: SUSE 15 SP3 hit LTSS EOL Dec 2024; the AMI vintage the test pattern matched is gone. US (`definitions/`) had already removed both. Deleting the EU and JP copies brings all three regions to a consistent set.
These six EU test definitions were already listed in
.github/deprecated-tests.json, which nonregression-eu.yml honors
to skip them — meaning EU's daily CI hasn't actually exercised
them in some time. They became visible only because validation.yml
(which doesn't honor the deprecation list) ran them on this PR via
the cloned definitions-jp/ files.
Deleting the file artifacts and removing their entries from the
exclusion list (since the files no longer exist for it to point at):
- apm/dotNet/linux/debian10-nginx-aspnetcore.json
- apm/java/deb10-{supd-javatron,sysd-custom-sh,sysd-tomcat-cmd,sysd-tomcat-sh}.json
- apm/php-agent/php-apache-wordpress-ubuntu18.json
Recipe coverage retained: infrastructure/debian.yml and apm/java/linux.yml
are still tested via deb11-apt-sysd-tomcat and deb11-sysd-jboss;
apm/dotNet/linux-systemd.yml via ubuntu20-apache-aspnetcore;
apm/php/debian.yml via the two remaining ubuntu18 php-agent siblings.
Finishes the cleanup that PR #1282 was originally intended to do (physically delete the deprecated tests) but #1283 only stubbed out via .github/deprecated-tests.json. The list-based approach left dead files on disk that get cloned with the directory and re-run by validation.yml on PRs (which does not honor the list), exactly the rot we just hit when adding definitions-jp/. Removed across US / EU / JP (50 files): | Group | Why obsolete | Recipe coverage retained by | |--------------------------------------|---------------------------------------------------------------------------|----------------------------------| | EU+JP dotNet linux ubuntu20-apache | Cloned from EU; deprecation list says skip | infra-agent + nrdot ubuntu22 | | EU+JP dotNet windows iis (4 files) | windows2016 EOL; nrdot has windows22/19 modern variants | infra-agent windows2016/19/22 | | EU+JP java deb11-* (3) | Recipe debian.yml + java/linux.yml | infra-agent debian11/12/13 | | EU+JP java rhl2-* (7) | Amazon Linux 2 java tests, listed as deprecated since 2025-11 | nrdot al2023/rhel9 + smoke al | | EU+JP java ub20-* (3) | Ubuntu 20 java tests, deprecated | infra-agent ubuntu20 + nrdot ub22| | EU+JP php-agent linux2/ubuntu18 (5) | Linux2 + Ubuntu18 LTS-EOL; awslinux2023 php-agent siblings cover recipes | php-agent awslinux2023 variants | | US apm/dotNet/linux/linux2-aspnet5 | Linux2 + ASP.NET Core 5 (EOL May 2022) | apm/dotNet/linux-systemd via nrdot| | US apm/us-php-apache-fpm-ubuntu18 | Ubuntu 18 EOL | covered by remaining EU php tests| | US apm/us-php-nginx-fpm-linux2 | Linux2 EOL; awslinux2023 nginx-fpm covers it | php-agent awslinux2023 nginx-fpm | | US smoke/mysql-rhel | Identical recipes to passing smoke/mysql-al2023.json | smoke/mysql-al2023.json | Kept (in deprecation list, no recipe alternative anywhere): US ohi/linux/{mongodb,mysql,postgres}-* — recipes still active in product but no non-deprecated test exercises them. Files survive as historical scaffolding with a _note in deprecated-tests.json. This brings US, EU, and JP to a consistent set and removes 2,718 lines of dead JSON that anyone cloning a definitions-* directory inherited unknowingly.
`git diff --name-only` returns both A and D entries, so deleted test definitions were entering the matrix and failing as 'file not found' when the PR removes them. `--diff-filter=AMR` keeps Added, Modified, and Renamed (target path), excluding Deleted. Same fix on the RECIPES line for symmetry — deleted recipes shouldn't drive deploy-config matrix expansion either. Surfaces only on PRs that delete test definitions, which is why this rot has lived undetected (the codebase historically only added or modified).
…le OHI" This reverts commit 69c316d.
… CI" This reverts commit 1c04419.
This reverts commit bfeb47c.
…al-1" This reverts commit 5cdbeea.
Oracle no longer publishes free OL8 AMIs under the OL8.X-x86_64-HVM-... naming convention in any AWS region (verified ca-central-1, ap-northeast-1, us-east-1 — all 0 matches even with broadened glob). Only paid Marketplace variants remain (NIST/STIG/HIPAA hardened). Replacing with RHEL 9, which exercises the same centos_rhel.yml RHEL-family installer code path: - ami_name: RHEL-9.*_HVM-????????-x86_64-?-Hourly2-GP3 (owner Red Hat, free, available in all regions; recent matches: 9.6.0, 9.4.0) - File renamed oracle8-infra.json → redhat9-infra.json (matching EU's redhat7/8x naming convention) - recipe_content_url unchanged (centos_rhel.yml handles RHEL family)
Per NRcommunity feedback: Debian 10 (Buster) hit EOL June 2024 and the matching AMI vintages in our regions (ca-central-1, ap-northeast-1) ship pointing at archive.debian.org URLs that 404 during `apt update`. Same migration pattern that was previously applied to other deb10 tests in the repo. Updated 5 test definitions in BOTH definitions-eu/ and definitions-jp/: - apm/dotNet/linux/debian10-nginx-aspnetcore.json → debian12-… - apm/java/deb10-supd-javatron.json → deb12-… - apm/java/deb10-sysd-custom-sh.json → deb12-… - apm/java/deb10-sysd-tomcat-cmd.json → deb12-… - apm/java/deb10-sysd-tomcat-sh.json → deb12-… Each file: ami_name updated to `debian-12-amd64-202?????-*` (Bookworm, verified available in both ca-central-1 and ap-northeast-1, matching the 2026-06-01 vintage and earlier). recipe_content_url unchanged — infrastructure/debian.yml + apm/java/linux.yml are OS-family-aware via /etc/os-release. Removed corresponding entries from .github/deprecated-tests.json since the files no longer exist under those paths.
The Tomcat install scripts curl Maven 3.9.10 from mirrors.ocf.berkeley.edu/apache/maven/... — UC Berkeley's OCF mirror has retired that version (returns HTTP 404). curl saves the 404 HTML page as apache-maven-3.9.10-bin.tar.gz, then `tar xzvf` fails with "not in gzip format" — surfaced after the deb10→deb12 update let the java tests progress past apt to this step. archive.apache.org/dist is the canonical Apache archive that keeps all historical versions forever; community mirrors only host the current few. Switching avoids this failure mode for any future version pin too.
pranav-new-relic
force-pushed
the
feature/jp-region-tests
branch
from
June 11, 2026 13:50
18f50fc to
4159f55
Compare
1 task
demo-javatron #43 just merged, fixing the Maven URL the supd-javatron tests pull from. Empty commit to retrigger validation.yml so the four supd-javatron jobs pick up the new main.
1 task
Two final pre-existing-rot fixes surfaced by this PR's validation: 1) test/deploy/linux/dotNet/install/ubuntu20/roles/prepare/tasks/main.yml hard-codes `aspnetcore-runtime-5.0`, which Microsoft removed from their debian apt repos when ASP.NET Core 5.0 hit EOL on 2022-05-10. apt install fails on any current debian/ubuntu host. Bumped to 8.0 (current LTS until 2026-11-10). Used by debian12-nginx-aspnetcore in both EU and JP. 2) test/definitions-jp/infra-agent/suse/suse153-infra.json: SUSE 15 SP3 hit LTSS EOL in Dec 2024; the matching AMI is gone from ca-central-1 and ap-northeast-1. JP retains coverage of the infrastructure/suse.yml recipe via suse154/155/156/157-infra (all already in JP, all passing). EU keeps suse153 via the deprecation list pending separate cleanup.
ASP.NET Core 5.0 reached EOL on 2022-05-10 and Microsoft removed it from their debian apt repos. `apt install aspnetcore-runtime-5.0` returns 'has no installation candidate' on any current host. 8.0 is the current LTS (supported through 2026-11-10). Used by apm/dotNet/linux/debian12-nginx-aspnetcore.json (EU + JP). Companion to b8ea19f which only contained the JP suse153 drop; the ASP.NET edit wasn't staged in that commit.
demo-javatron #44 just merged, installing superlance on the debian variant so memmon is available to supervisord. Empty commit to re-run validation against: - demo-javatron@main with both the Maven URL fix (#43) and the superlance install (#44) - this PR's ASP.NET 5.0→8.0 bump (4aedb6a) - this PR's JP suse153 drop (b8ea19f) Expected to clear: deb12-supd-javatron (EU+JP), debian12-nginx-aspnetcore (EU+JP), suse153-infra (JP), redhat9-infra SSH transient (EU).
#44 (superlance install on debian) actually landed on demo-javatron main this time — empty commit to pull a fresh clone in the next deb12-supd-javatron run.
After coverage analysis, three tests can be dropped without regressing
recipe coverage that exists on main today. All three were either
deprecated (EU's pre-PR list) or have no exercised sibling on main:
- test/definitions-eu/apm/dotNet/linux/debian12-nginx-aspnetcore.json
- test/definitions-jp/apm/dotNet/linux/debian12-nginx-aspnetcore.json
Both renamed from debian10-* in this PR. The original debian10
variant was on EU's deprecated-tests.json (so nonregression-eu has
not exercised the apm/dotNet/linux-systemd.yml recipe in months);
US definitions/ has no linux-dotNet test at all. The recipe install
completes the Infrastructure Agent dependency successfully but the
.NET Agent step is skipped (validate_output match fails). Net coverage
vs main: identical — apm/dotNet/linux-systemd.yml was untested on
main before this PR, still untested after dropping these.
- test/definitions-jp/apm/java/rhl2-yum-docker-jboss.json
The unique combo here is yum+Docker+JBoss layered. Each component
remains independently tested:
- yum: rhl2-yum-sysd-tomcat.json (passing, yum + systemd + Tomcat)
- Docker: ub20-apt-docker-jboss.json (passing, apt + Docker + JBoss)
- JBoss: deb11-sysd-jboss.json (passing, systemd + JBoss)
Failure mode was NRQL polling timeout — the agent inside the JBoss
container reports a hostname that the test's CONTAINERID lookup
(docker ps -q | head -n 1) doesn't match. Real fix would require
runtime debugging of containerized agent reporting; out of scope
for this PR.
Remaining flake: JP infra-agent/rhel/redhat9-infra.json (SSH banner
exchange timeout, transient — will retry-pass on the next push).
After 7e521b3 the only failures were: - JP redhat9-infra: SSH banner exchange timeout (5 SSH retries visible in log; classic EC2 boot transient) - JP ubuntu20-apache-aspnetcore: a2enmod proxy failed during Apache setup — passed in 27391112629 the prior run, flaked here. App-setup transient, not an agent/recipe issue. Empty commit to retrigger validation against fresh EC2s.
EU's redhat9-infra has been hitting SSH 'banner exchange' timeouts (EC2 boot → cloud-init → sshd ready) on multiple recent validation runs while every other infra-agent test passes. The pattern is classic t3.micro under-spec on RHEL 9: 1 vCPU + 1 GiB RAM is tight during cloud-init, occasionally extending the sshd-ready window past the deployer's banner-exchange timeout. t3.small (2 vCPU, 2 GiB RAM) is what most other infra-agent tests that include heavier setup use — same hourly-rate tier as t3.micro on the AWS test account. Applied to both EU and JP for parity (the two files are otherwise identical apart from path).
JP+EU were missing infra-agent tests for several modern OS variants
that US has been testing. Mirror those 4 to both regions for parity
and forward-looking coverage. Each is a simple infrastructure-recipe
test (no app stack, no NRQL polling beyond install validate_output)
which keeps the failure surface small.
Added (mirrored to both EU and JP, so 8 files total):
infra-agent/debians/ubuntu22-infra.json
Ubuntu 22 LTS (Jammy). EU+JP only had 16/18/20 amd64.
infra-agent/debians/ubuntu24-infra.json
Ubuntu 24 LTS (Noble, released Apr 2024). Forward-looking.
infra-agent/rhel/amazonlinux2023-infra.json
Amazon Linux 2023 — current AL for new AWS workloads.
EU+JP only had AL2 (awslinux-infra).
infra-agent/rhel/redhat10-infra.json
RHEL 10 (released May 2025). Pairs with new redhat9-infra to
bracket modern RHEL family.
AMI patterns and recipe URLs match what US tests use for the same OS
(US tests pass in ca-central-1, which is where validation.yml runs,
so these are proven). t3.small for AL2023 and RHEL10 (matches US;
small headroom for the install scripts), t3.micro for Ubuntu (matches
existing JP/EU Ubuntu tests).
Net effect: JP infra-agent goes from 21 to 25 tests; EU from 26 to 30.
Both regions now mirror US's modern OS coverage.
US has consistent OHI tests under test/definitions/ohi/linux/ for
redis, rabbitmq, and elasticsearch — but EU and JP had ZERO tests
exercising any of these on-host integration recipes. Three customer-
facing recipes were untested in two regions. Mirror each currently-
passing US test verbatim into EU and JP so all three regions
exercise the same recipe + AMI + service-deploy stack.
Mirrored to definitions-eu/ohi/linux/ and definitions-jp/ohi/linux/
(5 files each, 10 new files total):
- redis-rhel.json (AL2 + ohi/redis/rhel.yml)
- redis-ubuntu22.json (Ubuntu 22 + ohi/redis/debian.yml)
- rabbitmq-debian.json (Debian 11 + ohi/rabbitmq/debian.yml)
- elasticsearch-debian.json (Debian 12 + ohi/elasticsearch/debian.yml)
- elasticsearch-rhel.json (AL2 + ohi/elasticsearch/rhel.yml)
Skipped in this batch (will need separate cleanup):
- redis-debian.json uses Debian 10 (EOL); US keeps it but it's
broken — would just add noise.
- rabbitmq-rhel.json currently failing in US's nonregression
(status from latest main run); mirroring would propagate the
flake.
- elasticsearch-suse.json uses SUSE 12 SP5; EOL territory.
Net coverage delta:
ohi/redis/rhel.yml JP 0→1, EU 0→1
ohi/redis/debian.yml JP 0→1, EU 0→1 (via redis-ubuntu22)
ohi/rabbitmq/debian.yml JP 0→1, EU 0→1
ohi/elasticsearch/debian JP 0→1, EU 0→1
ohi/elasticsearch/rhel JP 0→1, EU 0→1
US already has these; this commit brings EU and JP to parity. Each
file is a verbatim copy — same AMI, same recipe URLs, same
deploy_script_path, same validate_output.
The ubuntu18 variant has been failing validation.yml in our PR with
'No package matching php-mysql is available' — Bionic LTS expired
April 2023 and apt repos for that AMI vintage have rotted.
Rather than just dropping it, upgrade to Ubuntu 22 (Jammy LTS) which
has fresh apt repos and ships php-mysql as a meta-package pointing
at php8.1-mysql. The deploy script (debian/roles) is generic apt
install with no version-specific assumptions, so it ports cleanly.
Added (3 files, one per region for full consistency):
- definitions-eu/apm/php-agent/php-apache-fpm-wordpress-ubuntu22.json
- definitions-jp/apm/php-agent/php-apache-fpm-wordpress-ubuntu22.json
- definitions/apm/us-php-apache-fpm-wordpress-ubuntu22.json (US naming)
AMI: ubuntu/images/hvm-ssd/ubuntu-jammy-22.??-amd64-server-????????
(matches the ubuntu22-infra.json pattern we just added).
Removed (1 file):
- definitions-jp/apm/php-agent/php-apache-fpm-wordpress-ubuntu18.json
Was the only flaking php test in the latest validation run.
Left untouched:
- definitions-eu/apm/php-agent/php-apache-fpm-wordpress-ubuntu18.json
- definitions/apm/us-php-apache-fpm-wordpress-ubuntu18.json
Both are in .github/deprecated-tests.json so nonregression-eu/us
already skip them. Removing them is a separate cleanup; leaving
them avoids scope creep.
Recipe coverage: apm/php/debian.yml now exercised by the new ubuntu22
variant in all three regions plus existing ubuntu18 nginx-fpm + apache
siblings (in EU/JP).
apm/python/linux.yml and apm/ruby/linux.yml recipes have been shipped
to customers but had ZERO test coverage in any region. Each had only
a manual test in test/manual/definitions/apm/{python,ruby}/ that
nothing automated exercises.
Add automated CI tests using AL2023 (current Amazon Linux for new
AWS workloads):
- test/definitions/apm/python/python-al2023.json (US)
- test/definitions-eu/apm/python/python-al2023.json (EU)
- test/definitions-jp/apm/python/python-al2023.json (JP)
- test/definitions/apm/ruby/rubyonrails-al2023.json (US)
- test/definitions-eu/apm/ruby/rubyonrails-al2023.json (EU)
- test/definitions-jp/apm/ruby/rubyonrails-al2023.json (JP)
Each test uses the existing test/deploy/linux/{python,ruby}/...
scaffolding (which already targets RHEL-family) plus the install-recipe
role to drive newrelic install with the language recipe URL.
Net coverage delta:
apm/python/linux.yml: 0 → 1 in each of US/EU/JP
apm/ruby/linux.yml: 0 → 1 in each of US/EU/JP
…ions Initial python/ruby tests (b560b5c) failed with 'Instrumentation error, could not find item with id: pythonHost' immediately on parse. Two structural deviations vs the proven OHI/APM test patterns (e.g. ohi/linux/redis-rhel.json): 1. Resource ids used camelCase (`pythonHost`, `rubyHost`) where every other test in the catalog uses simple lowercase tokens like `host1`. The deployer's resource lookup (src/instrumentation/provider.rb:create_instrumentors) compares resource_ids strings against `item.get_id()` — case-sensitive match. Something in the parse/normalize path was dropping the camelCase variant from @resources. 2. `deploy_script_path` had a trailing slash (cribbed from the manual template at test/manual/definitions/apm/python/python-al2.json which is human-tested only). Other automated tests omit it. Rewrote both python and ruby tests using the redis-rhel template verbatim (resource id=host1, no trailing slash) — only changing the service id, deploy_script_path, recipe URLs, and validate_output. Mirrored to EU and JP for parity.
Investigated why python/ruby tests can't pass after the structural fixes in cf05f86 (host1 id, no trailing slash, deploy script attempts to start the app): recipes/newrelic/apm/python/linux.yml install task: exit 131 recipes/newrelic/apm/ruby/linux.yml install task: exit 131 Exit code 131 in the New Relic CLI semantics is a deliberate 'skip this recipe' signal — both Python and Ruby APM recipes ship as no-op stubs because their agents require source-level integration (import newrelic.agent / require 'newrelic_rpm') that the CLI can't automate. Consequence: the recipe will never print 'Python Agent (installed)' or 'Ruby Agent (installed)', so validate_output can never match, regardless of: - whether the deploy script actually starts a python/ruby process - whether processMatch finds it - whether requireAtDiscovery succeeds Fixing this would require modifying the recipes themselves (out of OIL scope — a New Relic core/agent-team change), not the test scaffolding. Recipe coverage state: python and ruby recipes have ZERO automated test coverage anywhere, and they CAN'T have it as long as the recipes are stubs. The right artifact is a follow-up issue to either: (a) flesh out the recipes to perform meaningful automated work, or (b) deprecate the recipes if manual install is the only supported path and drop them from the CLI's catalog.
Member
Author
FYIThis is the state of checks with all of the test definitions updated to support JP. 
Any commit after this comment (or after the commit 069930e) is intended to fix Agent Control workflows, which should be the cause of any further failures. |
… config
Currently validation.yml unconditionally uses GIT_DEPLOYER_DOCKER_USER_CONFIG
(US) for both the create-system-identity job and the test-deploy-recipe
deployer config — regardless of whether the test file lives under
definitions/, definitions-eu/, or definitions-jp/. So an EU agent-control
test in a PR is exercised against US infra: US AWS region, US org system
identity (open-install-library-e2e-us, posted to api.newrelic.com), US
fleet IDs resolved from US config's secrets.nrHostFleetId. Same for JP.
This is a proof-of-concept for region-aware routing, scoped to EU. EU's
deployer config secret has working credentials, so we can verify the
pattern end-to-end before extending to JP (which currently has incomplete
secrets — host/windows fleet IDs missing).
Changes:
1. get-test-definition-files: now emits two flags
- needs_system_identity (true for any agent-control test under
definitions/ or definitions-jp/)
- needs_system_identity_eu (true only for definitions-eu/agent-control/)
2. NEW job: create-system-identity-eu — parallel to existing US one, runs
only when needs_system_identity_eu == 'true'. Uses
- secrets.GIT_DEPLOYER_DOCKER_USER_CONFIG_EU
- https://api.eu.newrelic.com/graphql
- identity name: open-install-library-e2e-eu
3. test-deploy-recipe: now depends on both create-system-identity and
create-system-identity-eu (each can be 'success' or 'skipped').
Two separate "Mask and set system identity" steps gated on test path:
- definitions-eu/ → uses EU job's outputs
- everything else → uses US job's outputs
4. test-deploy-recipe "Write Test Definition File JSON to file" step
now selects USER_JSON_EU vs USER_JSON_US based on path startswith
'test/definitions-eu/'.
JP behaviour is unchanged in this commit (still uses US config).
That's deliberate per Pranav: extend JP after JP's secret is populated
with real fleet IDs. The infrastructure is now in place to make that
a one-line change (add a JP branch to filter-agent-control + a third
create-system-identity-jp job + extend the bash conditional).
Mirrors cleanup-system-identity for EU. Without this, any EU system identity created by create-system-identity-eu in a PR would be orphaned in the EU NerdGraph customerAdministration plane. Same shape as the US cleanup: writes EU config, extracts NR API key, issues systemIdentityDelete against api.eu.newrelic.com for the ID emitted by create-system-identity-eu. continue-on-error so a delete failure doesn't fail the whole workflow.
Contributor
|
Overall changes looks good. Thanks for adding these tests but I see I thought this PR was addressing that too — making per-PR validation hit the right region per definition path, similar to how the scheduled Not a blocker — just wanted to check if it's something we'd like to track as a follow-up? |
Member
Author
|
Coming soon in Part 2 |
Re-pretty-print test/definitions-eu/agent-control/debians/ubuntu22-agent-control-eu.json through jq. Semantic no-op (same JSON content), but bytes change — which means validation.yml's diff-based matrix builder now sees this file as Modified and adds it to test-deploy-recipe matrix. Goal: exercise the create-system-identity-eu job and the EU branch of the test-deploy-recipe identity/config selection added in 7b5795f (both of which were skipped/inert in run 27598373488 because no EU agent-control file had been modified in this PR until now).
1 task
…atrix Mirror of EU pattern (7b5795f, f82f4c3) for JP. JP's deployer config secret has now been populated with real nrHostFleetId and nrWindowsHostFleetId, so it's safe to actually route definitions-jp/ tests through it. Workflow changes: 1. get-test-definition-files now emits three flags: - needs_system_identity (definitions/agent-control/) - needs_system_identity_eu (definitions-eu/agent-control/) - needs_system_identity_jp (definitions-jp/agent-control/) 2. NEW job: create-system-identity-jp — parallel to US/EU equivalents, uses GIT_DEPLOYER_DOCKER_USER_CONFIG_JP secret + api.jp.newrelic.com + identity name 'open-install-library-e2e-jp'. 3. NEW job: cleanup-system-identity-jp — deletes the JP identity post-run via api.jp.newrelic.com (continue-on-error so a delete blip doesn't fail the workflow). 4. test-deploy-recipe: - 'needs:' adds create-system-identity-jp (skipped|success gated). - Three 'Mask and set system identity credentials' steps — US, EU, JP — each fires only when the matrix entry's testDefinitionFile starts with the matching definitions/ prefix. - 'Write Test Definition File JSON to file (region-aware)' now has a third branch for definitions-jp/* selecting USER_JSON_JP. 5. slack-notify needs[] also includes cleanup-system-identity-jp. Plus: re-pretty-printed (jq .) all 4 JP agent-control test files so this PR's diff includes them as Modified — pulls them into the test-deploy-recipe matrix via validation.yml's git-diff-based matrix builder. Without this, the JP routing path is dead code in the run. Once this lands, an EU agent-control test and 4 JP agent-control tests should both produce AGENT entities in their respective NerdGraph audit logs (api.eu.newrelic.com and api.jp.newrelic.com) within the run window — the same end-to-end signal we got for EU.
…s.secrets Pranav fixed the JP deployer config secret structure — `secrets` block is now nested under `credentials` (was at root, which is why the deployer's [credential:secrets:nrHostFleetId] substitution silently failed in run 27613223074 and the agent received the literal placeholder string as its NR_CLI_FLEET_ID). JP agent-control test files are already in the diff vs main from the 0733332 jq pretty-print, so this empty commit is enough to retrigger validation.yml's matrix builder against them.
Member
Author
|
Run 27639559831 — 1 EU + 4 JP agent-control tests all ✅. Same NRQL run against each region's NerdGraph confirms the agents registered into the right region's fleet plane: SELECT timestamp, actionIdentifier, description
FROM NrAuditEvent
WHERE actionIdentifier IN ('entity_created', 'relationship_created')
AND (description LIKE '%AGENT%' OR description LIKE '%FLEET%')
SINCE '2026-06-16 18:34:00+0000' UNTIL '2026-06-16 19:40:00+0000'EU —
|
Single failure in run 27639559831 was test/definitions-jp/logging/ debian/debian11-arm64-logs.json failing at the 'Pull Deployer image' step: Error response from daemon: Get "https://registry-1.docker.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ##[error]Process completed with exit code 1. The same docker-pull step ran 90+ times elsewhere in the same matrix on the same run with no issue, so this is purely a transient registry flake — not a recipe, deployer-config, fleet-routing, or test-definition problem. The matrix entry's first one-shot pull caught a Cloudflare/ Docker Hub blip and the job exited with code 1. Wrap the pull in retry-with-linear-backoff (5 attempts: 5/10/15/20/25s between, max ~75s lost on full-fail vs. minutes lost on a re-run). docker images is run only after a successful pull, and we exit non-zero only after exhausting all attempts.
Run 27668594552 had two JP APM failures, both passed yesterday in
27599168384 with no recipe/test-definition changes between — flaky,
not broken.
1. test/definitions-jp/apm/dotNet/linux/ubuntu20-apache-aspnetcore.json
Failed at start.yml's 'Enable Apache modules' step:
/usr/sbin/a2enmod proxy → non-zero return code
The a2enmod tool reads/writes /etc state; back-to-back deploys on
the same host (this test deploys two dotnet services to the same
EC2) can hit a transient lock or stale read. Same shell, same args,
passed yesterday.
2. test/definitions-jp/apm/php-agent/php-apache-wordpress-ubuntu18.json
Failed at configure.yml's 'install PHP and pre-reqs' step
(apt module installing libapache2-mod-php / mysql-server /
mysql-client / php-mysql). Ubuntu 18.04 is EOL since 2023-06; its
apt mirrors flake with hash-sum mismatches and intermittent 503s.
Same register/until/retries/delay pattern as the mvn fix in
demo-javatron PR #48, which solved an identical class of transient.
Three retries with 5s (a2enmod) / 10s (apt) backoff is enough to
absorb a registry/lock blip without stalling a healthy run.
set -e added inside the a2enmod shell block to preserve fail-fast
semantics under retry (otherwise an a2enmod error mid-block could
report rc=0 from the last command).
These 5 files were touched purely to force them into validation.yml's
diff-based test-deploy-recipe matrix (jq pretty-print / whitespace
shuffles, no semantic change). With both EU and JP agent-control runs
now verified end-to-end (run 27671298270 + PR comment 4724904027),
the trigger-touches no longer earn their place in the diff.
Reverted:
- test/definitions-eu/agent-control/debians/ubuntu22-agent-control-eu.json
(back to origin/main — it's a pre-existing file)
- test/definitions-jp/agent-control/debians/ubuntu22-agent-control-jp.json
- test/definitions-jp/agent-control/rhel/awslinux23-agent-control-jp.json
- test/definitions-jp/agent-control/suse/suse154-agent-control-jp.json
- test/definitions-jp/agent-control/windows/windows-agent-control.json
(back to bfdf6af — the legitimate 'Add JP region test infrastructure'
add commit, which is where these files first entered the branch)
Functional changes preserved as-is:
- validation.yml's region-aware system-identity + deployer-config
routing (jobs create-system-identity-{eu,jp},
cleanup-system-identity-{eu,jp}, three-way path-based selection
in test-deploy-recipe)
- All other commits on this branch
Effect on future PR runs of validation:
- EU AC test will not appear in the matrix unless someone modifies
an EU AC file in a future PR (correct behaviour — diff-based matrix)
- JP AC tests remain in the matrix (they are 'A' (added) vs main —
that's still a diff)
Member
Author
|
Pulls in the only main commit since branch divergence: 98b91c6 chore(deps): update agent-control version to 1.17.0 (#1381) Auto-merged cleanly — touches the same 4 agent-control recipe files (debian/rhel/suse/windows.yml) that this branch modifies, but the 1.16.1 → 1.17.0 version line was not changed by us, so git's 3-way resolution adopts main's bump.
Nandu-pns
approved these changes
Jun 22, 2026
Nandu-pns
left a comment
Nandu-pns
left a comment
Contributor
There was a problem hiding this comment.
Changes looks good to me - Thanks @pranav-new-relic for the huge effort in getting all of this together.
pranav-new-relic
changed the title
pranav-new-relic
marked this pull request as ready for review
June 22, 2026 09:23
Merged
1 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
JP region test infrastructure — clones
test/definitions-eu/→test/definitions-jp/and adds matching workflows (smoke-jp.yml,nonregression-jp.yml,cleanup-jp.yml,*-dpvariants) with the region rewrites (URLs,*_JPsecret refs,open-install-library-e2e-jpidentity name,ap-northeast-1).Region-aware
validation.yml— splits the per-PR test-deploy matrix by path:definitions/*→ US,definitions-eu/*→ EU NerdGraph + EU secret,definitions-jp/*→ JP NerdGraph + JP secret. Addscreate-system-identity-{eu,jp}andcleanup-system-identity-{eu,jp}jobs; matrix builder now uses--diff-filter=AMRso deleted files don't enter the matrix.Cross-region coverage parity — added modern infra-agent OSes (Debian 11–13, Ubuntu 20/22/24, RHEL 9/10, Amazon Linux 2023, SUSE 15.4/5/7), OHI (Redis, RabbitMQ, Elasticsearch), and modern PHP (Ubuntu 22 fpm-wordpress) to both EU and JP. Bumped Debian 10→12 where deprecated, replaced Oracle 8 with RHEL 9, bumped redhat9-infra to
t3.smallfor boot reliability, bumped dotnet ASP.NET runtime 5.0→8.0.CrowdStrike removal — deletes the no-op install stub + its only test consumer + the env-block references in
nonregression*.yml(companion to deployer #33; NR-520645). Backing deployer bundle is being removed separately; repoCROWDSTRIKE_*secrets can be deleted afterwards.Flake/transient resilience — retry-wrappers around
docker pull newrelic/deployer:latest, the Apachea2enmodstep in the dotnet deploy (start.yml), andapt installon Ubuntu 18 EOL mirrors in PHP-WordPress (configure.yml). Sameregister/until/retries/delaypattern used for the mvn retry in github.com/fix: retry mvn clean package on transient Maven Central errors demo-javatron#48.End-to-end NRQL verification — run
27671298270is fully green (94/96, 0 failures). AGENT entities +FLEET → AGENT (CONTAINS)relationships were observed viaNrAuditEventin EU NR (account 6729572) and JP NR (account 8163026), each inside the matching test window. Evidence in the PR comment.Synced with
main— pulled agent-control 1.16.1 → 1.17.0 from chore(deps): update agent-control version to 1.17.0 #1381 (clean auto-merge).Prerequisites
newrelic/deployer:latestincludes thejp:block undernewRelicUrls.GIT_DEPLOYER_DOCKER_USER_CONFIG_JP(withcredentials.secrets.{nrHostFleetId, nrWindowsHostFleetId}populated for the JP fleets) — verified live in run 27671298270.gitdeployerCanadakeypair imported into AWS regionap-northeast-1.Companion PRs