[Group-IB] Connector Major Update

[Kchekh]

Proposed changes

• Added new collections: compromised/access, compromised/account_group, compromised/bank_card_group, compromised/discord, compromised/masked_card, compromised/messenger, compromised/spd, hi/open_threats, ioc/primary, malware/cnc, malware/config, osi/git_repository, osi/public_leak, and darkweb/forums.

• Removed collections compromised/mule and compromised/imei.

• Added hunting rules for collections.

• Improved data collection for the collections apt/threat_actor, hi/threat_actor, apt/threat, hi/threat, attacks/ddos, malware/malware, osi/vulnerability, attacks/phishing_kit, attacks/phishing_group, and attacks/deface.

• Fully reworked the code architecture and performed a complete refactor.

• Integrated the ciaops library.

• Added the ability to split collection runs across multiple Docker containers.

• Improved the main documentation and the developer documentation.

Related issues

• fix(group-ib): API returns a list when a string is expected #6341

Checklist

• I consider the submitted work as finished
• I have signed my commits using GPG key.
• I tested the code for its functionality using different use cases
• I added/update the relevant documentation (either on github or on notion)
• | Where necessary I refactored code to improve the overall quality

Further comments

[filigran-cla-bot]

Contributor License Agreement

✅ CLA signed 💚

Thank you @Kchekh for signing the Contributor License Agreement! Your pull request can now be reviewed and merged.

We appreciate your contribution to Filigran's open source projects! ❤️

_{This is an automated message from the Filigran CLA Bot.}

[codecov]

Codecov Report

❌ Patch coverage is 0% with 4952 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...roup-ib/src/adapters/stix_adapter_special_mixin.py	0.00%	934 Missing ⚠️
...rt/group-ib/src/adapters/stix_adapter_sdo_mixin.py	0.00%	706 Missing ⚠️
...group-ib/src/adapters/stix_adapter_osi_hi_mixin.py	0.00%	597 Missing ⚠️
...ort/group-ib/src/support/incident_note_markdown.py	0.00%	580 Missing ⚠️
...-ib/src/adapters/stix_adapter_compromised_mixin.py	0.00%	450 Missing ⚠️
...t/group-ib/src/adapters/stix_adapter_core_mixin.py	0.00%	442 Missing ⚠️
external-import/group-ib/src/external_import.py	0.00%	224 Missing ⚠️
external-import/group-ib/src/models/_common.py	0.00%	193 Missing ⚠️
external-import/group-ib/src/models/indicators.py	0.00%	141 Missing ⚠️
external-import/group-ib/src/models/sdo.py	0.00%	107 Missing ⚠️
... and 15 more

❗ There is a different number of reports uploaded between BASE (879d6ea) and HEAD (94396bb). Click for more details.

HEAD has 122 uploads less than BASE

Flag	BASE (879d6ea)	HEAD (94396bb)
connectors	123	1

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6782       +/-   ##
===========================================
- Coverage   33.27%    0.22%   -33.05%     
===========================================
  Files        1993     1923       -70     
  Lines      122690   123709     +1019     
===========================================
- Hits        40829      284    -40545     
- Misses      81861   123425    +41564     

Files with missing lines	Coverage Δ
...mport/group-ib/src/support/portal_external_refs.py	0.00% <ø> (ø)
...rnal-import/group-ib/src/support/text_normalize.py	0.00% <ø> (ø)
external-import/group-ib/src/utils.py	0.00% <ø> (ø)
external-import/group-ib/src/_data/iso3166.py	0.00% <0.00%> (ø)
external-import/group-ib/src/adapters/__init__.py	0.00% <0.00%> (ø)
external-import/group-ib/src/pipeline/__init__.py	0.00% <0.00%> (ø)
external-import/group-ib/src/models/__init__.py	0.00% <0.00%> (ø)
external-import/group-ib/src/main.py	0.00% <0.00%> (ø)
external-import/group-ib/src/adapters/adapter.py	0.00% <0.00%> (ø)
...ternal-import/group-ib/src/support/mitre_mapper.py	0.00% <0.00%> (ø)
... and 18 more

... and 1148 files with indirect coverage changes

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.