Conversation
* Add files via upload * Add files via upload * Refactor regex patterns for observables in ipqs.py * Add files via upload * Add IPQualityScoreMixin for API interactions * Add files via upload * Add files via upload
…models (refs #3712) (#3715) * feat(chatbot): scaffold chatbot_manager app with ChatSession/ChatMessage Initial skeleton for the GSoC 2026 LLM Chatbot integration. Introduces the chatbot_manager Django app under api_app/, following the existing <name>_manager convention used by all other IntelOwl modules. - ChatSession: FK to user, created_at/updated_at timestamps - ChatMessage: FK to session (cascade), role choices (user/assistant), content, indexed timestamp - Admin registration for both models - Empty urls.py / views.py stubs (populated in W2-3) - New settings module intel_owl/settings/chatbot.py exposing OLLAMA_BASE_URL and OLLAMA_MODEL via the project secrets accessor - Initial migration 0001_initial.py Refs #3712 * chore(docker): add Ollama env vars to app template Adds OLLAMA_BASE_URL and OLLAMA_MODEL to docker/env_file_app_template with defaults matching the GSoC proposal (http://ollama:11434, mistral). The CI env file is intentionally left unchanged — the defaults in intel_owl/settings/chatbot.py keep CI green without an ollama service. Refs #3712 --------- Co-authored-by: Berry <berry@intelowl.example.com>
…t flag (#3719) Adds Ollama as an opt-in service that operators can enable with the new --ollama flag on the start script. Per maintainer feedback on #3711, the service is delivered as docker/ollama.override.yml rather than baked into the base compose, so deployments without LLM hardware keep working unchanged. - docker/ollama.override.yml: ollama/ollama:0.5.0 service exposing port 11434 on a named volume (ollama_data), with healthcheck via `ollama list`. - docker/entrypoints/ollama.sh: starts the server, waits for it to be ready, pulls the configured model if not already present (3 retry attempts on failure). Model from OLLAMA_MODEL env var with a pinned mistral:7b-instruct-v0.3-q4_K_M fallback. - start: --ollama flag wired into path_mapping + argument parser, mirroring the --flower / --https patterns. GPU passthrough is intentionally deferred (#3717) — CPU-first per maintainer guidance. Refs #3711 Co-authored-by: Berry <berry@intelowl.example.com>
…t (refs #3720) (#3722) * [GSoC 2026] Add LangChain ReAct agent with job tools and REST endpoint - Add langchain, langchain-ollama, langchain-core to requirements - Add CHATBOT_QUEUE to Celery settings (chatbot dedicated queue) - Implement DjangoChatMessageHistory backed by ChatMessage model - Implement search_jobs, get_job_details, summarize_job tools (ORM-direct, user-scoped to prevent cross-user data access) - Implement build_agent_executor with ChatOllama + create_react_agent, custom ReAct prompt with chat_history injection - Implement ChatSessionViewSet with POST /api/chatbot/sessions/message sync REST endpoint; agent invoked directly (no Celery for sync path) - Add process_chat_message Celery task stub (full impl deferred to W6) - Register chatbot URLs at /api/chatbot/ - Unit tests for all three tools and the message endpoint (LLM mocked) * fix(chatbot): prefetch analyzer config to avoid N+1 queries and normalize tool imports * refactor(chatbot): address review feedback on agent tools and views - return a consistent {errors, ...} JSON envelope from all three tools - fix summarize_job using ReportStatus (analyzer reports are uppercase, the old lowercase/Status comparison flagged every report as failed) - persist chat turns through DjangoChatMessageHistory.add_message - add docstrings/notes on the ReAct flow, the soft_time_limit, and the BaseChatMessageHistory methods for maintainability * refactor(chatbot): use DRF serializers for tool output, fix analyzer report relation - replace hand-built dicts in search_jobs/get_job_details with JobToolSerializer/ JobDetailToolSerializer (+ nested AnalyzerReportToolSerializer), matching IntelOwl style - serialize the assistant ChatMessage directly in the message endpoint response - fix tools querying the non-existent 'analyzer_reports' relation (correct: 'analyzerreports'), which made get_job_details/summarize_job raise FieldError * refactor(chatbot): wrap tool output in a result serializer with to_json() - add ToolResultSerializer base (errors + to_json) and Search/Detail/Summarize result serializers for the {errors, payload} envelope - tools render their output via the serializer instead of manual json.dumps
* feat: add base test class for connector testing framework * refactor: migrate AbuseSubmitter tests to base test class * fix: deepsource errors * fix: deepsource error * fix: suppress deepsource warnings with comments
) Add GET /api/chatbot/sessions/{id}/messages: a detail action on ChatSessionViewSet returning the session messages ordered by timestamp (oldest first), paginated and scoped to the requesting user via self.get_object(). Reuses ChatMessageSerializer. Also drop an internal week-label from a TODO in tasks.py. Refs #3725
…refs #3716) (#3727) * feat(chatbot): add chat message retention policy with cleanup task Add a periodic Celery beat task that deletes ChatSessions whose last activity (most recent message timestamp, falling back to created_at) is older than CHATBOT_MESSAGE_RETENTION_DAYS (env-configurable, default 90). Session deletion cascades to its messages. Refs #3716 * fix(chatbot): drop the langchain-core pin (transitive dep) langchain-core is a sub-dependency of langchain/langchain-ollama; per project convention we don't pin sub-dependencies. The earlier 0.3.59 pin conflicted with langchain-ollama's >=0.3.60 and made the image unbuildable.
) (#3729) * feat(chatbot): add investigation LLM tools (list/tree/summary) Add three tools to the ReAct agent wrapping the Investigation API: list_investigations, get_investigation_tree, summarize_investigation. Scoped via Investigation.objects.visible_for_user (owner + organization-shared). The job tree is assembled in Python to avoid a treebeard per-node N+1 (one get_descendants() query per root, depth/node capped). Lightweight purpose-built serializers in the ToolResultSerializer envelope; the list serializer exposes only DB columns to avoid per-row N+1 from the aggregate properties. Refs #3728 * refactor(chatbot): address review on investigation tools Model the investigation tree with dataclasses instead of dicts (truncated is now an explicit field, converted to a dict only at the envelope boundary), name the result cap as a constant, and move the tool imports to module level (no circular import). Refs #3728
…run_on_failure is False (#3730) * test: add partial failure handling in before_run for connectors * fix(connectors): correct before_run behavior during partial failures
When Netlas has no whois record for an IP, the API returns an empty "items" list. Accessing items[0] raised an uncaught "IndexError: list index out of range". Guard the empty case so the analyzer reports a clean AnalyzerRunException instead of crashing with a traceback. Behaviour is unchanged when results are present. Adds a regression test covering the empty response.
Add a get_data_model(job_id) tool to the ReAct agent, scoped owner-strict (user=user) like the other job tools. Mirrors the public JobSerializer.get_data_model: returns job.data_model.serialize() (polymorphic Domain/IP/File) when present, else {}. Wrapped in the ToolResultSerializer envelope (data_model as a DictField carrying the model serializer output).
Refs #3728
* refactor(connectors): update YETI to API v2 payload and auth schemas * refactor(connectors): YETI observable type handling * fix(yeti): update context date to received_request_time to avoid race condition * test(yeti): add unit tests for YETI connector auth and response handling * YETI: Add IntelOwl as User-Agent to request headers
…#3732) (#3740) * [GSoC 2026] Add analyzer/playbook LLM tools to the chatbot agent (refs #3732) Add two read-only tools to the chatbot ReAct agent: - list_analyzers: lists the enabled observable analyzers, each annotated with a per-user 'runnable' readiness flag (no hard runnable filter, so key-based analyzers stay visible). - recommend_playbook: suggests launchable (starting=True, enabled) playbooks matching an observable's classification, scoped with visible_for_user. Both follow the make_*_tool(user) factory pattern and return the standard {errors, payload} envelope via purpose-built light serializers. Tests cover filtering, enum validation, the result cap, and org/visibility isolation. * [GSoC 2026] Address review: core helper for observable classifications + surface limit clamp (refs #3732) - Add Classification.observable_classifications() as the single source of truth for 'every classification except FILE'; reuse it in aggregate_observable_classification - Surface the limit clamp in the tool 'errors' (list_analyzers + recommend_playbook) so a truncated list is never silent - Tests for both clamp warnings * [GSoC 2026] Address review: extract clamp_limit helper to dedup analyzer/playbook tools (refs #3732)
* Refactor YETI analyzer to implement API v2 auth flow * test(yeti): refactor tests for yeti analyzer
…3756) (#3757) * feat(chatbot): add WebSocket consumer with token-by-token streaming ChatConsumer (Django Channels) joins a per-user group and relays a chat turn streamed token-by-token from a dedicated Celery worker via the channel layer. Implements the process_chat_message task (run-level ReAct callbacks, Final Answer streaming + per-tool status, persistence as source of truth, chat.error on failure). Adds a celery_worker_chatbot service so the chatbot queue has a consumer, and pins the task to it via task_routes. REST sync turn unchanged. Refs #3756 * refactor(chatbot): address review — enum event types, single dataclass style, f-string logs
When BGP Ranking has no ASN history for an IP, the API returns an empty "response" object. Calling .popitem() on that empty dict raised an uncaught "KeyError: 'popitem(): dictionary is empty'" before the intended "ASN not found" AnalyzerRunException could be raised. Guard the empty case so the analyzer reports a clean AnalyzerRunException instead of crashing with a traceback. Behaviour is unchanged when ASN history is present. Adds a regression test covering the empty response.
…avigation (#3765) * feat(chatbot): add React chat panel with WebSocket streaming Wire the chatbot backend to a persistent chat drawer (reactstrap Offcanvas) mounted once in the authenticated layout. The drawer lazily opens a WebSocket on first use and survives open/close across page navigations. Components: ChatPanel (drawer + status badge), ChatMessageList (scroll view with streaming bubble), ChatMessage (user/assistant bubbles, GFM markdown via react-markdown + remark-gfm), ChatComposer (textarea + send button, locked while a turn streams). State: Zustand store (useChatStore) owns messages, streaming text, current tool, connection state and drawer visibility — no WebSocket refs in the store (matches IntelOwl convention). The useChatWebSocket hook owns transport, lazy connect, capped exponential reconnect, session demux, and two watchdog timers (post-ack for dead workers, max-turn for dropped sockets). Integration: Assistant button in AppHeader, ChatPanel in AppMain Layout, WEBSOCKET_CHAT_URI constant, /ws/chat dev proxy entry. Refs #3763 * fix(chatbot): keep chat WS max-turn watchdog armed across mid-turn reconnect onclose cleared both turn watchdogs unconditionally, which defeated the max-turn timer in the exact case it documents: a socket dropping mid-turn. The reconnected socket never receives the in-flight turn's end (group frames are not replayed), so isStreaming stayed true and the composer locked forever. Clear the watchdogs only on terminal closes; keep them armed across a reconnect so the timeout still unlocks the composer. * fix: scope package-lock diff to remark-gfm and fix close button contrast on dark theme * feat(chatbot): implement chat sessions list with backend titles and unlocked navigation This commit introduces the sessions view (PR3) and two major improvements: - Session title from backend: GET /sessions now annotates each session with a 'title' field via a correlated Subquery (the first user message truncated to 40 chars). This eliminates N+1 and removes the need to load the full history on the frontend just to derive a label. - Unlocked navigation during streaming: Users can now switch or create new sessions while a turn is actively streaming. The store handles this by bumping a 'navEpoch' counter, which cleanly abandons the in-flight turn without blocking the UI. The post-ack and max-turn watchdogs are epoch-guarded so they silently expire if the user navigated away mid-turn. * feat(chatbot): reflect assistant availability in the header badge + add an info tooltip * style(chatbot): brighten the header Assistant toggle on hover like the Docs link
…probe) (#3768) * feat(chatbot): add a cached worker+Ollama health endpoint * feat(chatbot): probe assistant health when the drawer opens
…3770) (#3771) * feat(chatbot): add derive_page_context URL->prompt-hint helper * feat(chatbot): add page_context prompt variable (empty default) * feat(chatbot): inject current-page context into the agent prompt * docs(chatbot): cross-reference frontend Routes.jsx in context regexes, add coupling test
…per-tool guidance (#3775)
…ector-testing-framework-2 Refactor Connector Testing framework 2: Migrate MISP tests to base connector test framework
…(refs #3772) (#3773) * feat(chatbot): add context-aware quick-action buttons to the chat panel * feat(chatbot): broaden quick-action chips per review - rename 'What analyzers ran?' -> 'Which plugins ran?' (covers connectors/visualizers) - add 'Evaluate results' job chip - add 'Analyze this investigation' investigation chip Refs #3772
…Socket entry points (#3777) * feat(chatbot): add RATE_LIMITED error detail and retry_after to ErrorEvent * feat(chatbot): add RateLimiter shared between REST and WS entry points * feat(chatbot): add rate-limit settings and Redis cache backend * feat(chatbot): rate-limit the REST POST /api/chatbot/sessions/message endpoint * feat(chatbot): rate-limit the WebSocket chat consumer * test(chatbot): add RateLimiter unit tests (8 cases) * test(chatbot): add REST 429 rate-limit envelope test * test(chatbot): add WS rate-limit error event test * fix(chatbot): import CACHES from settings.cache so the named cache resolves * fix(test): add build_agent_executor mock so rate-limit view test doesn't call real Ollama * fix(chatbot): declare redis dependency in chatbot worker compose service
…-ups) (#3780) * fix(chatbot): validate search_jobs status against api_app.choices.Status and reuse clamp_limit - Validate status parameter against Status.values; unknown values are surfaced in the errors envelope instead of silently returning 0 results (mirrors the list_investigations pattern). - Migrate from bare min(int(limit), 50) to the shared clamp_limit() helper so over-cap requests produce an in-band warning. - Derive the valid-statuses string from Status.values to avoid drift when the enum gains members. Refs #3779 * fix(chatbot): return 503 envelope when the REST chat agent fails - Wrap executor.invoke() in try/except so an Ollama/agent failure returns a clean 503 with ChatErrorDetail.UNAVAILABLE + session_id instead of an unhandled 500. This mirrors the Celery/WS path in tasks.py. Refs #3779 * docs(chatbot): correct stale context_url comment in ChatConsumer The comment described context_url as 'intentionally unused for now (W9)' but it is already captured and forwarded to the chat turn for page-context resolution (derive_page_context in tasks.py). Refs #3779 * refactor(chatbot): reuse clamp_limit in list_investigations Replace the local _MAX_RESULTS constant and min() logic with the shared clamp_limit() helper so all list tools are consistent and silently-truncate-free. Refs #3779
…full-turn (#3794) * [GSoC 2026] test(chatbot): E2E chat WebSocket pipeline + M-1 guardrail + ChatPanel full-turn * [GSoC 2026] test(chatbot): address DeepSource findings (next() guards, drop redundant lambda)
* refactor(tests): remove obsolete connector test files for AbuseSubmitter, MISP, and OpenCTI * refactor(tests): remove unused monkeypatch utilities from connector files
Add a free, key-less ObservableAnalyzer that enriches a CVE observable with real-world exploitation signals instead of CVSS severity alone: - CISA Known Exploited Vulnerabilities (KEV) catalog membership, date added, and known ransomware campaign use. - FIRST.org EPSS score and percentile (probability of exploitation in the wild over the next 30 days). Registers the AnalyzerConfig via a data migration mirroring NVD_CVE, adds it to the FREE_TO_USE_ANALYZERS playbook, and includes a unit test with all external calls mocked. No new dependencies (uses requests) and no API key. Signed-off-by: Devam Shah <devamshah91@gmail.com>
…egacy monkeypatches (#3807)
* fix exiftool failed download * fix repo downloader blocking the build
* fix exiftool failed download * fix repo downloader blocking the build
* Add RDAP analyzer * Add Rdap to the FREE_TO_USE_ANALYZERS playbook
* feat(yeti): implement health check * test(yeti): add health check tests for YETI connector * test(yeti): fix deepsource errors * feat(slack): add health check for slack * feat(slack): add health check tests for Slack connector * feat(opencti): implement health check method for OpenCTI connector * test(opencti): add health check tests for OpenCTI connector * feat(opencti): add ssl_verify and proxies parameters to OpenCTI health_check * feat(misp): implement health check method for MISP connector * test(misp): add health check tests for MISP connector * feat(connectors): add health check comments for MISP, OpenCTI, Slack, and YETI connectors * feat(connectors): add reference links for health check methods in MISP, OpenCTI, Slack, and YETI connectors
…3830) celery_worker_chatbot (defined only in ollama.override.yml) pinned the pulled released image without mounting the source, so the new docker/entrypoints/celery_chatbot.sh - absent from any published image - was not found and the container crashed. Add docker/test.ollama.override.yml (mirroring test.multi-queue.override.yml) to repoint the worker at the :test image and mount the source, and wire it into the start script via the test_ollama mapping and the test_values list.
…nreachable in chatbot, approved by @mlodic)
[GSoC 2026] feat: self-deployed LLM chatbot for natural-language threat intelligence
allow() truncated the remaining window with int(), so in the final second it returned retry_after=0 (client told to retry immediately while still limited) and otherwise under-estimated the wait. Use math.ceil so it is always >= 1 and never under-estimates. Add a regression test freezing time at the window end. This also fixes the flaky test_allow_returns_false_at_limit.
Gsoc 2026/connectors
…low step (#3837) Add docker/ci.ollama.override.yml to repoint celery_worker_chatbot to the built :ci image (mirroring ci.override.yml), and wire --ollama into ci mode in start. The Slow backend-tests step now runs only on the develop->master release PR and includes --ollama, smoke-testing the ollama + chatbot-worker topology; the Fast step becomes its exact complement so every other PR still builds.
Bumps [pyelftools](https://github.com/eliben/pyelftools) from 0.31 to 0.33. - [Changelog](https://github.com/eliben/pyelftools/blob/main/CHANGES) - [Commits](eliben/pyelftools@v0.31...v0.33) --- updated-dependencies: - dependency-name: pyelftools dependency-version: '0.33' dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [google-cloud-webrisk](https://github.com/googleapis/google-cloud-python) from 1.20.0 to 1.22.0. - [Release notes](https://github.com/googleapis/google-cloud-python/releases) - [Changelog](https://github.com/googleapis/google-cloud-python/blob/main/packages/google-cloud-documentai/CHANGELOG.md) - [Commits](googleapis/google-cloud-python@google-cloud-webrisk-v1.20.0...google-cloud-webrisk-v1.22.0) --- updated-dependencies: - dependency-name: google-cloud-webrisk dependency-version: 1.22.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/cache](https://github.com/actions/cache) from 5 to 6. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v5...v6) --- updated-dependencies: - dependency-name: actions/cache dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6.2.0 to 6.3.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v6.2.0...v6.3.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: 6.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps library/nginx from 1.29.2-alpine to 1.31.2-alpine. --- updated-dependencies: - dependency-name: library/nginx dependency-version: 1.31.2-alpine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mlodic <30625432+mlodic@users.noreply.github.com>
Co-authored-by: mlodic <30625432+mlodic@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist for creating a new release
CHANGELOG.mdfor the new version. Tag another maintainer to review the Changelog and wait for their feedback.docker/.envandfrontend/package.json.developand set it as apre-release. Remember to prepend avto the version number.Write the following statement there (change the version number):
masterbranch. Note: Only use "Merge and commit" as the merge strategy and not "Squash and merge". Using "Squash and merge" makes history between branches misaligned.pre-releasetolatest release.