feat(arcsight-incidents): add external-import connector for ArcSight ESM cases (#6723)

[SamuelHassine]

Proposed changes

This PR adds a new EXTERNAL_IMPORT connector external-import/arcsight-incidents that imports ArcSight ESM cases into OpenCTI as STIX Incidents and Case-Incidents. It is the import side of a bidirectional ArcSight integration (paired with the stream/arcsight connector that pushes IOCs to ESM Active Lists).

• Authenticates against the ESM LoginService, lists case ids via CaseService (capped per run) and fetches each case and its security events.
• Models ArcSight security events as STIX Incidents and the ArcSight case that groups them as a STIX Case-Incident (referencing the event Incidents through object_refs), attributed to an ArcSight author identity and marked with a configurable TLP.
• Built on the modern connectors-sdk settings pattern, with unit tests and connector metadata (manifest, config schema, configuration documentation).

Related issues

Closes #6723

Type of change

• New feature (non-breaking change which adds functionality).

Maintainer review (independent review-and-fix)

Independent senior review of the full connector across several passes, on top of the Copilot threads. Substantive fixes:

• Security: arcsight_client/api_client.py _request never logs str(err). The ESM LoginService login carries the password as a query parameter (and findAllIds / getResourceById carry the auth token), and a requests exception string usually embeds the full request URL, so str(err) could leak those secrets. Errors now log only path + status_code + exception type.
• Security: connector.py top-level error handler no longer logs or forwards str(err) either; it logs the exception type via structured meta and finalizes the work with a sanitized message (ArcSight Incidents connector run failed: <ExceptionType>), so a requests exception URL carrying the password/token cannot leak into the logs or the work message.
• converter_to_stix.py: tlp_level="clear" emits a dedicated TLP:CLEAR custom statement marking (x_opencti_definition="TLP:CLEAR") instead of aliasing stix2.TLP_WHITE; amber+strict likewise emits the OpenCTI-specific marking.
• converter_to_stix.py (deterministic ids): Incident.generate_id / CaseIncident.generate_id are seeded with the event id / case external id in the name (so distinct events or cases that share a display name no longer collapse into one entity), and when the source carries no usable timestamp created/modified fall back to a fixed sentinel instead of "now". The Case-Incident modified now also falls back to the stable created (never "now") when the modified timestamp is missing or unparseable, so a re-imported case keeps a stable id and is not re-sent with a drifting modified each run.
• api_client.py: _request retries only on connection/timeout errors, 429 and 5xx; other 4xx (401/403/404) fail fast. It now also closes the HTTP response on the 429/5xx-retry and non-retriable HTTPError paths, so a periodic run does not keep connections checked out of the requests Session pool.
• connector.py: the initiated work is finalized with in_error=True on the failure path, so a failed run does not leave a dangling in-progress work.
• Docs/tests: config.yml.sample / docker-compose.yml / README.md mark scope / CONNECTOR_SCOPE as required (matching the SDK base settings and the generated schema), the tlp_level / log-level option lists are in sync, the settings test asserts on str(err.value), and a grammar typo in tests/test-requirements.txt is fixed.
• src/requirements.txt / README.md / __metadata__/connector_manifest.json: the pinned pycti is bumped to 7.260615.0 to match the current connectors-sdk, with the README minimum and manifest support_version aligned. The branch was merged with current master to de-stale, so the local connectors-sdk used by run_test.sh pins the same pycti and the test environment resolves consistently.

Decisions (Copilot comment intentionally not applied)

• The Dockerfile apk update && apk upgrade && apk --no-cache add ... line is kept: it is the shared pattern across the sibling connectors (corelight-investigator, fortisiem-incidents, sentinelone-intel, infoblox-threat-defense); the git / build-base build deps are pruned afterwards with apk del, and the libmagic / libffi-dev / libxml2-dev / libxslt-dev packages back pycti's transitive C-extension deps. Changing only this connector would be a one-off and risk breaking the image build; a leaner shared base image is better handled repo-wide.

Tests

Unit tests cover the settings model (validation, defaults), the ArcSight client (auth, case listing/fetch, 4xx fail-fast vs 429/5xx retry, credential-redaction in logs), the converter (TLP:CLEAR marking, severity mapping, deterministic and collision-free incident/case ids, stable fallback timestamps, modified-falls-back-to-created) and the connector (work finalized in_error on failure). 54 unit tests pass; black / isort / flake8 --ignore=E,W clean locally.

Checklist

• My code follows the repository code style (isort, black, flake8, STIX-id pylint).
• I have added unit tests covering the settings, the client, the converter and the connector.
• I have updated the documentation (README, connector metadata).
• Commits are signed.

Status

All CI checks are green (tests, lint/format, STIX ID linter, codecov/patch and codecov/project) and there are 0 unresolved review threads. mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me (as the author I cannot self-approve). The branch carries a de-stale merge commit, so it should be squash-merged.

[codecov]

Codecov Report

❌ Patch coverage is 90.69069% with 31 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...csight-incidents/src/arcsight_client/api_client.py	89.74%	16 Missing ⚠️
external-import/arcsight-incidents/src/main.py	0.00%	13 Missing ⚠️
...port/arcsight-incidents/src/connector/connector.py	96.07%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (3e51b9e) and HEAD (2172839). Click for more details.

HEAD has 115 uploads less than BASE

Flag	BASE (3e51b9e)	HEAD (2172839)
connectors	119	4

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6724       +/-   ##
===========================================
- Coverage   32.30%    0.48%   -31.82%     
===========================================
  Files        1985     1900       -85     
  Lines      122106   119830     -2276     
===========================================
- Hits        39441      586    -38855     
- Misses      82665   119244    +36579     

Files with missing lines	Coverage Δ
...arcsight-incidents/src/arcsight_client/__init__.py	100.00% <100.00%> (ø)
...mport/arcsight-incidents/src/connector/__init__.py	100.00% <100.00%> (ø)
...sight-incidents/src/connector/converter_to_stix.py	100.00% <100.00%> (ø)
...mport/arcsight-incidents/src/connector/settings.py	100.00% <100.00%> (ø)
...port/arcsight-incidents/src/connector/connector.py	96.07% <96.07%> (ø)
external-import/arcsight-incidents/src/main.py	0.00% <0.00%> (ø)
...csight-incidents/src/arcsight_client/api_client.py	89.74% <89.74%> (ø)

... and 1119 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds a new external-import/arcsight-incidents connector that pulls ArcSight ESM cases via the Service Layer REST API and imports them into OpenCTI as STIX 2.1 Incidents, including connector packaging (Docker/compose), metadata, and unit tests.

Changes:

• Introduces ArcSight ESM client logic (login, list case IDs, fetch cases) with retry/backoff behavior.
• Implements case → STIX Incident conversion (author identity, TLP marking, timestamps, severity, external reference) and the connector runtime loop/state handling.
• Adds connector configuration (connectors-sdk settings), generated metadata/config docs, and unit tests (settings/client/converter/connector).

Reviewed changes

Copilot reviewed 24 out of 26 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
external-import/arcsight-incidents/tests/tests_connector/test_settings.py	Settings validation tests for connectors-sdk config model.
external-import/arcsight-incidents/tests/tests_connector/init.py	Test package init.
external-import/arcsight-incidents/tests/test-requirements.txt	Test dependency definition.
external-import/arcsight-incidents/tests/test_main.py	Smoke tests for settings/helper/connector instantiation.
external-import/arcsight-incidents/tests/test_converter.py	Unit tests for case→incident conversion helpers.
external-import/arcsight-incidents/tests/test_connector.py	Unit tests for connector orchestration and helper interactions.
external-import/arcsight-incidents/tests/test_client.py	Unit tests for ArcSight API client behaviors and parsing.
external-import/arcsight-incidents/tests/conftest.py	Test path bootstrap to import connector modules.
external-import/arcsight-incidents/tests/init.py	Test package init.
external-import/arcsight-incidents/src/requirements.txt	Connector runtime dependencies.
external-import/arcsight-incidents/src/main.py	Connector entrypoint wiring settings/helper/connector.
external-import/arcsight-incidents/src/connector/settings.py	connectors-sdk based settings models for this connector.
external-import/arcsight-incidents/src/connector/converter_to_stix.py	ArcSight case → STIX 2.1 Incident conversion.
external-import/arcsight-incidents/src/connector/connector.py	Main connector loop: collect, bundle, send, and persist state.
external-import/arcsight-incidents/src/connector/init.py	Package exports for connector components.
external-import/arcsight-incidents/src/config.yml.sample	Sample YAML configuration for deployment.
external-import/arcsight-incidents/src/arcsight_client/api_client.py	HTTP client for ArcSight LoginService/CaseService endpoints.
external-import/arcsight-incidents/src/arcsight_client/init.py	Package export for ArcSight client.
external-import/arcsight-incidents/README.md	Usage and configuration documentation.
external-import/arcsight-incidents/entrypoint.sh	Container entrypoint.
external-import/arcsight-incidents/Dockerfile	Container build for the connector.
external-import/arcsight-incidents/docker-compose.yml	Example docker-compose deployment.
external-import/arcsight-incidents/.dockerignore	Docker build context exclusions.
external-import/arcsight-incidents/metadata/connector_manifest.json	Connector manifest metadata for registry/manager.
external-import/arcsight-incidents/metadata/connector_config_schema.json	Generated JSON schema for config variables.
external-import/arcsight-incidents/metadata/CONNECTOR_CONFIG_DOC.md	Generated config documentation.

[SamuelHassine]

Review-and-fix pass summary

Independent senior review of the full new connector plus all 5 remaining open Copilot threads.

• Security (my finding, not flagged by the bot): arcsight_client/api_client.py _request no longer logs str(err). The ESM LoginService login carries the password as a query parameter (and findAllIds / getResourceById carry the auth token), and a requests exception string usually embeds the full request URL, so str(err) could leak those secrets into the logs. It now logs only path + status_code + error_type (fixed in c5108b0, with redaction tests).
• The 5 open Copilot threads were already addressed by earlier commits on the branch and are now replied to and resolved: TLP:CLEAR emits a dedicated OpenCTI marking (not TLP_WHITE); the API client uses meta={...} structured logging; and config.yml.sample / docker-compose.yml / README.md all mark scope / CONNECTOR_SCOPE as required.
• Also aligned the _load_config_dict test overrides with the SDK (-> Self).

Verified locally: 45/45 unit tests pass (added 2 redaction tests); black / isort / flake8 --select=F clean. CI: all checks green (tests, lint/format, STIX ID linter, codecov/patch and codecov/project); filigran/cla satisfied (organization member). Review threads: 0 unresolved. Title already matches the Conventional Commits convention; description updated to reflect the final state.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 26 changed files in this pull request and generated 3 comments.

[SamuelHassine]

Second review-and-fix pass summary

Independent senior re-review of the full external-import/arcsight-incidents connector plus the 3 remaining open Copilot threads.

Code fixes (commit 67fe59b):

• Deterministic ids: Incident.generate_id and CaseIncident.generate_id are now seeded with the source timestamp only. _to_iso is split into _parse_timestamp (returns None for a missing or unparseable timestamp) and _format_iso, so a re-imported event or case keeps a stable id instead of creating a duplicate Incident / Case-Incident every run (covers both the missing-timestamp and unparseable-timestamp threads).
• Dangling work: the initiated work is finalized with in_error=True on the failure path.
• Dockerfile: kept as-is with the rationale on the thread (the apk line and its build deps are the shared pattern across the sibling connectors; changing only this one risks a build break) - a leaner shared base image is better handled as a repo-wide change.

Also verified the connector does not have the data-loss pattern from the sibling fortisiem-incidents connector: get_cases lists all case ids each run (no incremental cursor) and last_run is not used to filter.

Tests: added the deterministic id (missing and unparseable timestamp, Incident and Case-Incident), the scalar baseEventIds branch, and the in-error work finalize; the converter module is now 100% covered.

Status: all CI checks are green (codecov/patch and codecov/project, target 80%), black / isort / flake8 are clean locally, 49/49 unit tests pass, and there are 0 unresolved review threads (11 resolved). All commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 26 changed files in this pull request and generated 2 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full external-import/arcsight-incidents connector plus the two remaining open Copilot threads (both on the converter's id/timestamp determinism).

• Code fix (commit 52a42d2): Incident.generate_id / CaseIncident.generate_id are now seeded with the event id / case external id in the name (f"{name} [{event_id}]" and f"{name} [{external_id}]"), so distinct events or cases that share a display name no longer collapse into one entity (the repo pattern used by recorded-future / intel471). When the source carries no usable timestamp, created/modified now fall back to a fixed _FALLBACK_TIMESTAMP sentinel instead of now, so a re-imported event/case keeps a stable id and is not re-sent with drifting timestamps each run.
• Tests: added test_incidents_with_same_name_different_event_id_have_distinct_ids, test_case_incidents_with_same_name_different_external_id_have_distinct_ids, and the stable-fallback-timestamp tests for both incidents and case-incidents (suite 49 -> 53).
• Independent re-review of the rest found no further issues: the credential-redacting client logging, the 4xx fail-fast vs 429/5xx retry, the distinct TLP:CLEAR marking, and the in_error work finalization are all correct and well covered.
• Verification: black / isort / flake8 --ignore=E,W clean and all 53 unit tests pass locally.
• CI: all checks green on 52a42d2 (tests, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (all Copilot threads replied to and resolved).
• Description refreshed to the final state; title already matches the repo's Conventional Commits convention.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 26 changed files in this pull request and generated 2 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full external-import/arcsight-incidents connector plus the 2 newly-opened Copilot threads.

Code/test fixes (commit 6c80ab1):

• converter_to_stix.py: the Case-Incident modified now falls back to the already-stable created value (never "now") when the ArcSight modified timestamp is missing or unparseable, so the deterministic id is no longer re-sent with a drifting modified each run. Covered by test_case_incident_modified_falls_back_to_created_when_unparseable.
• connector.py: the top-level error handler no longer logs or forwards str(err) (the ESM auth flow carries the password/token in query parameters, which a requests exception string can embed); it logs the exception type via structured meta and finalizes the work with a sanitized message. The existing failure test still asserts the work is finalized with in_error=True.
• arcsight_client/api_client.py (independent finding): _request now closes the HTTP response on the 429/5xx-retry and non-retriable HTTPError paths, so a periodic run does not keep connections checked out of the requests Session pool. This is the same hardening the sibling stream/arcsight client received; the bot did not flag it here, but the code path was identical.
• src/requirements.txt: pinned pycti bumped to 7.260615.0 to match the current connectors-sdk; the README minimum and the manifest support_version are aligned.

CI fix (de-stale):

• The branch was behind master, and run_test.sh installs the local connectors-sdk from the checked-out tree. The stale copy still pinned pycti==7.260609.0, which would conflict with the 7.260615.0 reinstalled from opencti master on the push-event test run. Merging current master updates the local connectors-sdk pin, so both test runs resolve consistently.

Verification:

• black / isort --profile black / flake8 --ignore=E,W clean and all 54 unit tests pass locally (53 + the new modified-fallback regression test); the new response.close() paths are exercised by the existing retry/fail-fast tests, so patch coverage is preserved. CI is fully green on the new head (both Test external-import/arcsight-incidents runs, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (both Copilot threads replied to and resolved). Both new commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a Filigran maintainer other than me. As the author I cannot self-approve. Once approved, recommend squash-merge (the branch carries a de-stale merge commit).

[Copilot]

⚠️ Human review recommended

It introduces a full new connector (API client + STIX modeling + deployment artifacts), which warrants a final maintainer review despite the strong unit-test coverage.

Copilot's findings

• Files reviewed: 24/26 changed files
• Comments generated: 0 new

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.