feat(arcsight): add stream connector to sync IOCs to ArcSight ESM active lists (#6717)

[SamuelHassine]

Proposed changes

This PR adds a new STREAM connector stream/arcsight that synchronises IOCs from OpenCTI to an ArcSight ESM Active List in real time.

• Authenticates against the ESM LoginService to obtain an authentication token (re-issued automatically on expiry).
• On create/update: adds the IOC value to the configured Active List (ActiveListService/addEntries); on delete: removes it (ActiveListService/deleteEntries).
• Supported observables: IPv4, IPv6, domain names, URLs and file hashes (MD5/SHA-1/SHA-256), written to a configurable Active List column.
• Built on the modern connectors-sdk settings pattern, with unit tests and connector metadata (manifest, config schema, configuration documentation).

Related issues

Closes #6717

Type of change

• New feature (non-breaking change which adds functionality).

Maintainer review (independent review-and-fix)

Independent senior review of the full connector across several passes, on top of the Copilot threads. Substantive fixes:

• Line endings: the entire connector had been committed with CRLF. In a Linux container CRLF breaks the entrypoint shebang (/bin/sh\r) and the cd command, so the container entrypoint can fail at runtime; it is also inconsistent with the sibling stream connectors. The whole stream/arcsight/ tree is now normalized to LF (content otherwise unchanged - the diff is line-endings only).
• Security: _request never logs str(err). A requests exception string typically embeds the full request URL, and the ESM LoginService login call carries the ArcSight password as a query parameter, so str(err) could leak credentials into the logs. Errors now log only the request path, the status_code and the exception type.
• connector.py: run() validates the live stream id first, so a placeholder/blank CONNECTOR_LIVE_STREAM_ID fails fast at startup instead of only when the first event arrives; the redundant per-message check was removed.
• connector.py: process_message catches only the expected parse errors (json.JSONDecodeError / KeyError / TypeError) and chains via raise ... from err.
• api_client.py: _request retries only on connection/timeout errors, 429 and 5xx; other 4xx (401/403/404) fail fast, which also lets _post_entries trigger an immediate token re-issue on a 401 instead of after three backoff sleeps.
• api_client.py: HTTP responses are closed on the 429/5xx-retry and non-retriable HTTPError paths, and _post_entries closes the entry-management response on success, so a long-running stream connector does not keep connections checked out of the requests Session pool.
• connector.py / docker-compose.yml: check_stream_id() rejects the placeholder case-insensitively and rejects empty/whitespace-only values, and the shipped docker-compose.yml uses the ChangeMe placeholder consistently.
• src/requirements.txt / README.md / __metadata__/connector_manifest.json: the pinned pycti is 7.260615.0, the latest released version, matching both connectors-sdk@master and the pycti shipped by opencti@master; the README minimum and manifest support_version are aligned to the same version.

Decisions (Copilot comment intentionally not applied)

• The Dockerfile apk update && apk upgrade line is kept: it is byte-for-byte identical to stream/sentinelone-intel/Dockerfile and the other stream connectors (infoblox-threat-defense, harfanglab-intel, sumologic-intel); build-time deps are pruned afterwards with apk del, and the upgrade intentionally pulls Alpine base-image security patches. Keeping it avoids a one-off variant inconsistent with the sibling connectors.

Tests

Unit tests cover the settings model (validation, defaults), the connector (fail-fast startup on bad stream id, indicator add/remove dispatch, payload parsing), and the ArcSight client (token issue/re-issue on 401, add/delete entries, STIX value extraction incl. case-insensitive hash keys, 4xx fail-fast vs 429/5xx retry, and log redaction of credentials). 41 unit tests pass; black / isort / flake8 --ignore=E,W clean locally.

Checklist

• My code follows the repository code style (isort, black, flake8, STIX-id pylint).
• I have added unit tests covering the settings, the connector and the client.
• I have updated the documentation (README, connector metadata).
• Commits are signed.

Status

All CI checks are green (tests, lint/format, STIX ID linter, codecov/patch and codecov/project) and there are 0 unresolved review threads (6 resolved). The branch has been rebased onto the latest master as a clean linear history of signed commits (the previous de-stale merge commit has been dropped), so it merges without a merge commit and the local connectors-sdk used by run_test.sh resolves the same pycti==7.260615.0 consistently. mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me (as the author I cannot self-approve).

[codecov]

Codecov Report

❌ Patch coverage is 87.73006% with 20 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
stream/arcsight/src/main.py	0.00%	13 Missing ⚠️
stream/arcsight/src/arcsight_client/api_client.py	92.94%	6 Missing ⚠️
...ream/arcsight/src/arcsight_client/stix_patterns.py	90.90%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (c576089) and HEAD (4a11bf8). Click for more details.

HEAD has 115 uploads less than BASE

Flag	BASE (c576089)	HEAD (4a11bf8)
connectors	119	4

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6718       +/-   ##
===========================================
- Coverage   32.30%    0.35%   -31.95%     
===========================================
  Files        1985     1900       -85     
  Lines      122106   119660     -2446     
===========================================
- Hits        39444      427    -39017     
- Misses      82662   119233    +36571     

Files with missing lines	Coverage Δ
stream/arcsight/src/arcsight_client/__init__.py	100.00% <100.00%> (ø)
stream/arcsight/src/connector/__init__.py	100.00% <100.00%> (ø)
stream/arcsight/src/connector/connector.py	100.00% <100.00%> (ø)
stream/arcsight/src/connector/settings.py	100.00% <100.00%> (ø)
...ream/arcsight/src/arcsight_client/stix_patterns.py	90.90% <90.90%> (ø)
stream/arcsight/src/arcsight_client/api_client.py	92.94% <92.94%> (ø)
stream/arcsight/src/main.py	0.00% <0.00%> (ø)

... and 1119 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds a new stream connector under stream/arcsight to synchronize OpenCTI live-stream indicators (IOC values extracted from STIX patterns) into an ArcSight ESM Active List via the Service Layer REST API, including connector packaging assets, metadata, and tests.

Changes:

• Introduces ArcSight ESM API client with login/token caching and add/delete entry operations for Active Lists.
• Adds stream connector implementation + Pydantic-based settings using connectors-sdk.
• Adds unit tests plus deployment/documentation artifacts (Dockerfile, compose, README, metadata, config schema/docs).

Reviewed changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
stream/arcsight/src/arcsight_client/api_client.py	Implements ArcSight Service Layer REST client (login + Active List entry add/delete).
stream/arcsight/src/arcsight_client/stix_patterns.py	Extracts observable values from supported single-observable STIX patterns.
stream/arcsight/src/arcsight_client/init.py	Exposes ArcSight client package API.
stream/arcsight/src/connector/connector.py	Stream connector wiring: consumes live stream messages and calls ArcSight client.
stream/arcsight/src/connector/settings.py	connectors-sdk settings models for STREAM connector + ArcSight-specific config.
stream/arcsight/src/connector/init.py	Exports connector public API.
stream/arcsight/src/main.py	Runtime entrypoint to build settings/helper and start connector.
stream/arcsight/src/requirements.txt	Connector runtime dependencies.
stream/arcsight/src/config.yml.sample	Sample YAML configuration for local/manual deployment.
stream/arcsight/tests/test_client.py	Unit tests for ArcSight client behavior (token caching, retry, add/delete).
stream/arcsight/tests/test_connector.py	Unit tests for connector message handling and stream listening behavior.
stream/arcsight/tests/test_main.py	Smoke tests for settings/helper/connector instantiation with mocked pycti internals.
stream/arcsight/tests/tests_connector/test_settings.py	Unit tests for settings validation and defaults.
stream/arcsight/tests/tests_connector/init.py	Marks tests package (empty init).
stream/arcsight/tests/conftest.py	Ensures src/ is importable in the test environment.
stream/arcsight/tests/test-requirements.txt	Test dependencies for isolated connector test runs.
stream/arcsight/tests/init.py	Marks tests package (empty init).
stream/arcsight/Dockerfile	Container build for the ArcSight connector.
stream/arcsight/entrypoint.sh	Container entrypoint script.
stream/arcsight/docker-compose.yml	Example docker-compose deployment configuration.
stream/arcsight/.dockerignore	Docker build context exclusions.
stream/arcsight/README.md	Connector documentation (setup, configuration, behavior, supported observables).
stream/arcsight/metadata/connector_manifest.json	Connector manifest metadata for OpenCTI connectors catalog.
stream/arcsight/metadata/connector_config_schema.json	Generated config schema for environment variables.
stream/arcsight/metadata/CONNECTOR_CONFIG_DOC.md	Generated configuration documentation from schema.

[SamuelHassine]

Review-and-fix pass

Independent senior review of the full new connector plus the 2 Copilot threads (both valid; both fixed).

Fixes:

• process_message catches only the expected parse errors (json.JSONDecodeError / KeyError / TypeError) and chains the re-raise via raise ... from err.
• _request() retries only on connection/timeout errors, rate limiting (429) and server errors (5xx); other 4xx responses (401/403/404) fail fast instead of being retried three times. This also lets _post_entries trigger a token re-issue immediately on a 401 rather than after three backoff sleeps (found independently; consistent with the sibling FortiEDR connector). Request failures are logged via meta={...}.
• The settings test now asserts on str(err.value) (the actual exception message) instead of the pytest ExceptionInfo wrapper.

Verified: 31/31 unit tests pass; black / isort / flake8 --select=F clean; all CI checks green (including codecov/patch); 0 unresolved review threads. The commit is GPG-signed.

mergeStateStatus is BLOCKED only because the PR needs one approving review from a maintainer other than me - I am the author, so I cannot self-approve.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 25 changed files in this pull request and generated 3 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full new connector plus the 3 remaining open Copilot threads, fixed in 24a020e.

• Security (the 2 credential-leak threads): _request no longer logs str(err). A requests exception string usually embeds the full request URL, and the ESM LoginService login call carries the ArcSight password as a query parameter, so str(err) could leak the credentials into the connector logs. Errors now log only the request path, the status_code and the exception type. The login request is unchanged (the ArcSight ESM API requires the credentials as query parameters, over TLS), but they no longer reach any log line.
• Correctness (found by my review, not the bot): the shipped docker-compose.yml used CONNECTOR_LIVE_STREAM_ID=CHANGEME (all caps), but check_stream_id() only rejected the exact string ChangeMe, so the compose defaults bypassed the guard and the connector would try to listen to a non-existent stream id. check_stream_id() now rejects the placeholder case-insensitively and rejects blank values, and docker-compose.yml uses the ChangeMe placeholder consistently.
• Consistency: the _load_config_dict test overrides now return -> Self (matching the SDK) instead of dict[str, Any].
• Dockerfile thread (not applied): the apk update && apk upgrade line is byte-for-byte identical to stream/sentinelone-intel and the other stream connectors, prunes build deps afterwards, and intentionally pulls base-image security patches; changing only this connector would create a one-off inconsistency. Justified in the thread.

Verified locally: 40/40 unit tests pass (added redaction + stream-id placeholder tests); black / isort / flake8 --select=F clean. CI: all checks green (tests, lint/format, STIX ID linter, codecov/patch and codecov/project); filigran/cla satisfied (organization member). Review threads: 0 unresolved. Title already matches the Conventional Commits convention; description updated to reflect the final state.

Note: codecov lagged behind GitHub Actions for the latest commit and briefly still referenced the previous commit; it has since reconciled to success for 24a020e (patch coverage above the 80% target).

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 25 changed files in this pull request and generated 1 comment.

[SamuelHassine]

Third review-and-fix pass summary

Independent senior re-review of the full stream/arcsight connector plus the one remaining open Copilot thread.

Code fix (commit bd8d17a):

• connector.py: run() now validates the live stream id (check_stream_id()) before listening, so a placeholder or blank CONNECTOR_LIVE_STREAM_ID fails fast at startup instead of only surfacing once the first event arrives. The redundant per-message check was removed.

Re-reviewed the rest file by file with no further findings: the ArcSight client redacts credentials from logs (it logs only path, status_code and the exception type), fails fast on non-429 4xx with an immediate token re-issue on 401 (while retrying 429 and 5xx), the STIX pattern extraction is correct, and the settings and tests are consistent.

Tests: added test_run_aborts_when_stream_id_missing; connector.py is now 100% covered.

Status: all CI checks are green (codecov/patch and codecov/project, target 80%), black / isort / flake8 are clean locally, 41/41 unit tests pass, and there are 0 unresolved review threads (6 resolved). All commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 25 changed files in this pull request and generated 1 comment.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full stream/arcsight connector plus the one remaining open Copilot thread.

• Code fix (commit 993a49f): fixed the grammar typo in tests/test-requirements.txt ("Main dependencies needs" -> "need"), the last open Copilot thread.
• Independent re-review of the full connector found no further issues: the ArcSight client redacts credentials from logs (only path / status_code / exception type, never str(err), since the login call carries the password as a query parameter), fails fast on non-429 4xx with an immediate token re-issue on 401 while retrying 429 and 5xx, the STIX pattern extraction (incl. case-insensitive hash keys) is correct, run() validates the stream id at startup, and the settings/tests are consistent. The per-event INFO log is bounded to filtered indicator add/remove and is kept consistent with the sibling stream/vectra-ai connector.
• Verification: black / isort / flake8 --ignore=E,W clean and all 41 unit tests pass locally.
• CI: all checks green on 993a49f (tests, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (all Copilot threads replied to and resolved).
• Description refreshed to the final state; title already matches the repo's Conventional Commits convention.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 25 changed files in this pull request and generated 3 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full stream/arcsight connector plus the 3 newly-opened Copilot threads.

Code/test fixes (commit 26ca44f):

• api_client.py: _request now closes the HTTP response on the 429/5xx-retry path and on the non-retriable HTTPError path (the body is never consumed there), and _post_entries closes the entry-management response on success. This stops a long-running stream connector from keeping connections checked out of the requests Session pool until garbage collection.
• tests/tests_connector/test_settings.py: dropped the redundant is True on the isinstance(...) assertion.
• src/requirements.txt: pinned pycti bumped to 7.260615.0 to match the current connectors-sdk; the README minimum and the manifest support_version are aligned to the same version.

CI fix (de-stale):

• The branch was behind master, and run_test.sh installs the local connectors-sdk from the checked-out tree. The stale copy still pinned pycti==7.260609.0, which would conflict with the 7.260615.0 reinstalled from opencti master (connectors-sdk requires pycti==7.260609.0, but 7.260615.0 is installed) on the push-event test run. Merging current master updates the local connectors-sdk pin, so both the push-event and PR-event test runs resolve consistently.

Verification:

• black / isort --profile black / flake8 --ignore=E,W clean and all 41 unit tests pass locally; the new response.close() paths are exercised by the existing retry / fail-fast / _post_entries tests, so patch coverage is preserved. CI is fully green on the new head (both Test stream/arcsight runs, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (all 3 Copilot threads replied to and resolved). Both new commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a Filigran maintainer other than me. As the author I cannot self-approve. Once approved, recommend squash-merge (the branch carries a de-stale merge commit).

[Copilot]

Pull request overview

Copilot reviewed 23 out of 25 changed files in this pull request and generated 5 comments.

[SamuelHassine]

PR review-and-fix + full rebase + pycti

• Independent senior review of the full connector. The one substantive finding (beyond the Copilot threads) was line endings: the entire stream/arcsight/ connector had been committed with CRLF, not just the 5 files Copilot sampled. CRLF in entrypoint.sh is a real runtime bug (the container shebang becomes /bin/sh\r and cd ...\r in the Linux image), and it is inconsistent with the sibling stream connectors. Normalized the whole tree to LF in 4a11bf8 (content unchanged - line-endings only). The rest of the connector reviewed clean: credentials redacted from logs (path / status_code / exception type only), responses closed on all paths, 4xx fail-fast with token re-issue on 401, stream id validated at startup, and correct STIX value extraction.
• Full rebase: rebased onto the latest master (c576089); the previous de-stale merge commit has been dropped, so the branch is now a clean linear history of signed commits.
• pycti: already pinned to 7.260615.0 (the latest release); verified it matches connectors-sdk@master and the pycti shipped by opencti@master, and that requirements.txt, the README minimum and the manifest support_version are aligned.
• Review threads: replied to and resolved all 5 CRLF threads; 0 unresolved.
• Local validation: 41/41 unit tests pass; uv pip check clean; isort --profile black, black --check and flake8 --ignore=E,W clean.
• CI is fully green on the rebased head 4a11bf8 (both Test stream/arcsight runs, Ensure Formatting, Base Linter (flake8), STIX ID Linter, Build and Commit Manifest, codecov/patch and codecov/project).

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a Filigran maintainer other than me. As the author I cannot self-approve.