feat(corelight-investigator): create external-import connector for alerts and detections (#6736)

[SamuelHassine]

Proposed changes

New external-import connector for Corelight Investigator (SaaS NDR).

On a schedule the connector:

• queries the Investigator "Detections and Alerts" API (bearer API key), incrementally using connector state (configurable initial look-back window);
• maps each alert / detection to a STIX Incident, with the normalized Investigator severity (1-10) mapped to the OpenCTI severity and the EventType recorded as the incident type;
• extracts the source / destination IP observables referenced by an alert and creates related-to relationships from the Incident to those observables;
• attributes the imported objects to a Corelight Investigator author identity and applies a configurable TLP marking.

Related issues

Closes #6736

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• Unit tests for the client, converter and connector (coverage > 80% on new code)
• Code formatting (black, isort) and linting (flake8, STIX id pylint) pass
• Generated __metadata__ config schema and documentation
• Added/updated the connector README.md, config.yml.sample, docker-compose.yml and connector_manifest.json

Further comments

The exact REST path/parameters of the Investigator Detections and Alerts API are documented in the in-product API reference (require a tenant account), so the query endpoint path is configurable via alerts_path with a documented default, and the response parser accepts the common envelope keys (data / alerts / detections / results). This is the import side of a Corelight integration; the Corelight open sensor consumes intel via Zeek/TAXII (e.g. OpenCTI's native TAXII2 live feed) rather than an outbound IOC API.

Maintainer review (independent review-and-fix)

Independent senior review of the full connector, on top of the automated review threads. Substantive fixes:

• converter_to_stix.py: TLP:CLEAR and TLP:AMBER+STRICT are built as custom statement MarkingDefinitions via a shared _statement_marking helper (matching connectors-sdk TLPMarking.to_stix2_object), so OpenCTI shows the correct TLP label instead of aliasing stix2.TLP_WHITE / TLP_AMBER.
• converter_to_stix.py (deterministic ids): Incident.generate_id is seeded with the alert id in the name (so distinct alerts that share a name do not collapse into one Incident) plus the source timestamp; when an alert has no usable timestamp, created/modified fall back to a fixed sentinel instead of "now", so a re-imported alert keeps a stable id and is not re-sent with drifting timestamps each run.
• connector.py: work_id is initialized before the try and the initiated work is finalized with in_error=True on both error paths; a fetch failure raises (the client raises after retries), so last_run is not advanced on error and the next run retries.
• Docs: scope / CONNECTOR_SCOPE marked mandatory in config.yml.sample and docker-compose.yml (required by the generated schema); the README minimum version is aligned to OpenCTI >= 6.8.12 (matching the manifest support_version).

Tests

Unit tests cover the settings model, the API client (request shaping, retries, error wrapping), the converter (TLP:CLEAR / AMBER+STRICT marking shapes, severity mapping, _parse_timestamp, deterministic and collision-free incident ids, stable fallback timestamps, IP observable extraction + relationships) and the connector (work finalized in_error on both error paths). 42 unit tests pass; black / isort / flake8 clean locally.

Status

All CI checks are green (tests, lint/format, STIX ID linter, codecov/patch and codecov/project) and there are 0 unresolved review threads. mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me (as the author I cannot self-approve).

[Copilot]

Pull request overview

Adds a new external-import connector under external-import/corelight-investigator/ to pull Corelight Investigator “Detections and Alerts” incrementally, convert them into STIX 2.1 Incidents plus related IP observables, and send bundles to OpenCTI with stateful scheduling and configurable TLP.

Changes:

• Implemented a Corelight Investigator HTTP client with basic retry/backoff and response-envelope normalization.
• Implemented STIX conversion + connector runtime (stateful incremental since, bundling, scheduling).
• Added full connector packaging (Docker, compose, sample config, metadata) and unit tests for client/settings/converter/connector.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
external-import/corelight-investigator/src/corelight_investigator_client/api_client.py	Requests-based API client with retries/backoff and payload extraction.
external-import/corelight-investigator/src/corelight_investigator_client/init.py	Exposes client + error types as a small import surface.
external-import/corelight-investigator/src/connector/converter_to_stix.py	Alert/detection → STIX Incident + IP observables + relationships conversion.
external-import/corelight-investigator/src/connector/connector.py	Connector orchestration: stateful “since”, bundling, send, work handling, scheduling.
external-import/corelight-investigator/src/connector/settings.py	Pydantic/connectors-sdk settings models for OpenCTI + connector + Corelight config.
external-import/corelight-investigator/src/connector/init.py	Exposes connector entry classes.
external-import/corelight-investigator/src/main.py	Runtime entrypoint wiring settings → helper → connector.
external-import/corelight-investigator/src/requirements.txt	Connector runtime dependencies.
external-import/corelight-investigator/tests/conftest.py	Pytest fixtures/mocking for OpenCTIConnectorHelper.
external-import/corelight-investigator/tests/test_main.py	Smoke tests for settings/helper/connector instantiation.
external-import/corelight-investigator/tests/test-requirements.txt	Test dependency pinning for this connector.
external-import/corelight-investigator/tests/tests_corelight_investigator_client/test_api_client.py	Unit tests for API client behavior (envelope parsing, params, errors, rate-limit retry).
external-import/corelight-investigator/tests/tests_connector/test_settings.py	Unit tests for settings validation behavior.
external-import/corelight-investigator/tests/tests_connector/test_converter.py	Unit tests for severity mapping, incident creation, observable creation.
external-import/corelight-investigator/tests/tests_connector/test_connector.py	Unit tests for connector orchestration, bundling, and state updates.
external-import/corelight-investigator/Dockerfile	Container build for the connector.
external-import/corelight-investigator/entrypoint.sh	Container entrypoint wrapper.
external-import/corelight-investigator/docker-compose.yml	Example deployment configuration.
external-import/corelight-investigator/config.yml.sample	Manual deployment sample configuration.
external-import/corelight-investigator/README.md	Connector documentation and configuration reference.
external-import/corelight-investigator/metadata/connector_manifest.json	Connector marketplace/manifest metadata.
external-import/corelight-investigator/metadata/connector_config_schema.json	Generated config schema for the connector’s env vars.
external-import/corelight-investigator/metadata/CONNECTOR_CONFIG_DOC.md	Generated connector configuration documentation.

[codecov]

Codecov Report

❌ Patch coverage is 88.53755% with 29 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
external-import/corelight-investigator/src/main.py	0.00%	13 Missing ⚠️
...or/src/corelight_investigator_client/api_client.py	81.96%	11 Missing ⚠️
.../corelight-investigator/src/connector/connector.py	95.08%	3 Missing ⚠️
...ht-investigator/src/connector/converter_to_stix.py	97.89%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (414c97a) and HEAD (0d040c3). Click for more details.

HEAD has 110 uploads less than BASE

Flag	BASE (414c97a)	HEAD (0d040c3)
connectors	114	4

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6737       +/-   ##
===========================================
- Coverage   31.03%    0.42%   -30.62%     
===========================================
  Files        1986     1901       -85     
  Lines      122166   119810     -2356     
===========================================
- Hits        37919      508    -37411     
- Misses      84247   119302    +35055     

Files with missing lines	Coverage Δ
...t/corelight-investigator/src/connector/__init__.py	100.00% <100.00%> (ø)
...t/corelight-investigator/src/connector/settings.py	100.00% <100.00%> (ø)
...ator/src/corelight_investigator_client/__init__.py	100.00% <100.00%> (ø)
...ht-investigator/src/connector/converter_to_stix.py	97.89% <97.89%> (ø)
.../corelight-investigator/src/connector/connector.py	95.08% <95.08%> (ø)
...or/src/corelight_investigator_client/api_client.py	81.96% <81.96%> (ø)
external-import/corelight-investigator/src/main.py	0.00% <0.00%> (ø)

... and 1080 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full new connector plus the two open Copilot threads.

Code fixes (commit 40c75e5):

• converter_to_stix.py: TLP:CLEAR is no longer aliased to stix2.TLP_WHITE. TLP:CLEAR and TLP:AMBER+STRICT are now custom statement MarkingDefinitions built via a shared _statement_marking helper (matching connectors-sdk TLPMarking.to_stix2_object), so OpenCTI shows the correct TLP label instead of TLP:WHITE / TLP:AMBER.
• converter_to_stix.py: Incident.generate_id is seeded with the source timestamp only. With no usable timestamp the STIX created/modified still fall back to a timezone-aware now, but the id seed is None, so a re-imported alert keeps a stable Incident id instead of duplicating on every run.
• connector.py: the initiated work is finalized with in_error=True on the API-error and unexpected-error paths, so a failed run no longer leaves a dangling in-progress work in OpenCTI.
• Minor: _request typed Optional[dict]; removed a stray space in the pydantic version pin.

Tests (commits 40c75e5, 9988e06): added coverage for the distinct TLP:CLEAR marking, the deterministic Incident id, the source-timestamp passthrough, _parse_timestamp, and the in-error work finalize on both error paths.

Status: all CI checks are green (codecov/patch 88.44%, target 80%), black / isort / flake8 are clean locally, and there are 0 unresolved review threads (both Copilot threads resolved). The commits are GPG-signed.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 3 comments.

[SamuelHassine]

Review-and-fix pass summary

Independent senior re-review of the full external-import/corelight-investigator connector plus the 3 remaining open Copilot threads (commit 0d040c3).

• Converter determinism (proactive - same class of bug flagged on the sibling arcsight-incidents connector): Incident.generate_id is now seeded with the alert id in the name (f"{name} [{alert_id}]"), so distinct alerts that share a name no longer collapse into one Incident, and when the alert has no usable timestamp created/modified fall back to a fixed sentinel instead of "now" (no drift / needless updates on re-import). Covered by test_incidents_with_same_name_different_id_have_distinct_ids and test_incident_timestamps_use_stable_fallback_without_timestamp.
• Docs (the 3 open threads): scope / CONNECTOR_SCOPE marked mandatory in config.yml.sample and docker-compose.yml (required by the generated schema), and the README minimum version aligned to OpenCTI >= 6.8.12 to match the manifest support_version.
• Independent re-review confirmed the rest is sound: the client raises on fetch failure after retries (so last_run is not advanced and the run retries), the work item is finalized in_error on both error paths, and the TLP:CLEAR / AMBER+STRICT markings use the connectors-sdk statement-marking shape.
• Verification: black / isort / flake8 clean and all 42 unit tests pass locally.
• CI: all checks green on 0d040c3 (tests, lint/format, STIX ID linter, codecov/patch, codecov/project). filigran/cla satisfied (organization member).
• Review threads: 0 unresolved (all 3 replied to and resolved).
• Description refreshed to the final state; title already matches the repo's Conventional Commits convention.

Remaining (non-CI) blocker: mergeStateStatus is BLOCKED only because reviewDecision is REVIEW_REQUIRED - the PR needs one approving review from a maintainer other than me. As the author I cannot self-approve, so this requires another Filigran maintainer.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.