Add multi-LLM support and update dependencies in README and agent.py #10
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,4 +20,8 @@ tasks-archived-sheets/ | |
| tasks-sheets-verified/ | ||
| tasks-sheets/ | ||
|
|
||
| dev/ | ||
| .cursor/rules/jcodemunch.mdc | ||
| uv.lock | ||
|
|
||
| /.vs | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,5 @@ | ||
| <p align="center"> | ||
| <img src="docs/logo.png" alt="open-autoagent" width="160"> | ||
| </p> | ||
|
|
||
| <blockquote> | ||
|
|
@@ -66,6 +64,88 @@ rm -rf jobs; mkdir -p jobs && uv run harbor run -p tasks/ --task-name "<task-nam | |
| rm -rf jobs; mkdir -p jobs && uv run harbor run -p tasks/ -n 100 --agent-import-path agent:AutoAgent -o jobs --job-name latest > run.log 2>&1 | ||
| ``` | ||
|
|
||
| ## Install quickly with an AI harness | ||
|
|
||
| The `setup/` folder installs a skill that lets your AI harness run the full | ||
| Ollama setup for you — clone, `.env`, `uv sync`, Docker base image, Harbor | ||
| smoke test. | ||
|
|
||
| ```bash | ||
| # Linux / macOS / Git Bash | ||
| bash setup/install.sh | ||
|
|
||
| # Windows PowerShell | ||
| .\setup\install.ps1 | ||
| ``` | ||
|
|
||
| Pick your harness when prompted, then trigger it: | ||
|
|
||
| | Harness | Trigger | | ||
| |---|---| | ||
| | Hermes | New session (or `/reset`) → `run open-autoagent-ollama-setup` | | ||
| | Claude Code | In chat: `run open-autoagent-ollama-setup` | | ||
| | Claude Desktop | In chat: `run open-autoagent-ollama-setup` | | ||
| | Cursor | In chat: `run open-autoagent-ollama-setup` | | ||
| | Grok | In chat: `run open-autoagent-ollama-setup` | | ||
| | VS Code + Copilot | `Ctrl+Shift+I` → `#file:.vscode/skills/open-autoagent-ollama-setup/SKILL.md` → `run open-autoagent-ollama-setup` | | ||
| | Visual Studio | `View > GitHub Copilot Chat` → `#file:.github/skills/open-autoagent-ollama-setup/SKILL.md` → `run open-autoagent-ollama-setup` | | ||
|
|
||
| The harness will execute every step and stop if a check fails. | ||
|
|
||
| ## Multi-LLM Support | ||
|
|
||
| The harness supports multiple LLM providers via [LiteLLM](https://github.com/BerriAI/litellm). Configure via environment variables: | ||
|
|
||
| ### Environment Variables | ||
|
|
||
| - `LLM_PROVIDER`: Provider name (`openai`, `anthropic`, `ollama`, `azure`, etc.) | ||
| - `MODEL`: Model name (e.g., `gpt-5`, `claude-3-5-sonnet`, `qwen3.5:35b-a3b-q8_0`) | ||
| - `LLM_BASE_URL`: Optional base URL (required for Ollama, Azure, etc.) | ||
| - `API_KEY`: Provider-specific API key (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) | ||
|
|
||
| ### Using Local Ollama | ||
|
|
||
| ```bash | ||
| cat > .env << 'EOF' | ||
| LLM_PROVIDER=ollama | ||
| MODEL=qwen3.5:35b-a3b-q8_0 | ||
| LLM_BASE_URL=http://host.docker.internal:11434/v1 | ||
| EOF | ||
| ``` | ||
|
|
||
| ### Using OpenAI | ||
|
|
||
| ```bash | ||
| cat > .env << 'EOF' | ||
| LLM_PROVIDER=openai | ||
| MODEL=gpt-5 | ||
| OPENAI_API_KEY=your-api-key | ||
| EOF | ||
| ``` | ||
|
|
||
| ### Using Anthropic | ||
|
|
||
| ```bash | ||
| cat > .env << 'EOF' | ||
| LLM_PROVIDER=anthropic | ||
| MODEL=claude-3-5-sonnet | ||
| ANTHROPIC_API_KEY=your-api-key | ||
| EOF | ||
| ``` | ||
|
|
||
| ### Using Azure | ||
|
|
||
| ```bash | ||
| cat > .env << 'EOF' | ||
| LLM_PROVIDER=azure | ||
| MODEL=your-deployment-name | ||
| AZURE_API_KEY=your-api-key | ||
| AZURE_API_BASE=https://your-resource.openai.azure.com | ||
| EOF | ||
|
Comment on lines
+136
to
+144
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: printf '%s\n' '--- README Azure section ---'
sed -n '95,125p' README.md
printf '%s\n' '--- Azure variable references ---'
rg -n -C 3 'AZURE_API_(KEY|BASE|VERSION)|LLM_PROVIDER|MODEL' --glob '!README.md' .
printf '%s\n' '--- package/dependency references ---'
rg -n -C 2 'litellm|Azure' package.json pyproject.toml requirements.txt setup.py setup.cfg 2>/dev/null || trueRepository: thirdlayerinc/autoagent Length of output: 4688 🌐 Web query:
💡 Result: In LiteLLM, the AZURE_API_VERSION environment variable is a standard requirement for authenticating and routing requests to the Azure OpenAI provider [1][2]. While LiteLLM maintains a default internal API version (e.g., 2025-02-01-preview as of recent versions [3]), it is best practice to explicitly set AZURE_API_VERSION to ensure compatibility with your specific model deployment [1][4]. Key points regarding the configuration: 1. Mandatory Variables: To successfully call Azure models, you generally need to provide AZURE_API_KEY, AZURE_API_BASE, and AZURE_API_VERSION [2][5]. 2. Environment Configuration: You can set these globally as environment variables [1][6][7]: export AZURE_API_KEY="your-key" export AZURE_API_BASE="https://your-resource.openai.azure.com" export AZURE_API_VERSION="2024-02-01" 3. Alternative Configurations: - Within a LiteLLM Proxy configuration (config.yaml), you can specify api_version per model to avoid global conflicts or to support models requiring specific, newer API versions [1][4]. - You can also pass these parameters directly into the completion or embedding function calls in your Python code as keyword arguments [1][8][7]. 4. Troubleshooting: If you encounter 400 Bad Request errors, it is often because the requested model requires a newer API version than the default one being used by LiteLLM [4]. Explicitly defining the version in your configuration or code usually resolves this [4][9]. For checking your current environment configuration, you can use the LiteLLM helper function litellm.validate_environment(model_name) to identify missing required variables [6]. Citations:
🌐 Web query:
💡 Result: The api-version parameter is generally mandatory for Azure REST API requests, including those for Azure OpenAI and other services, as it specifies the version of the API to be used for the request [1][2][3][4]. Azure typically does not support a default api-version; therefore, failing to provide a valid version will often result in a request error [1][2][4]. In practice, users must explicitly specify the api-version in their API calls or configuration, such as in the following examples: For Python (using LiteLLM as an example): response = litellm.completion( model = "azure/", api_base = "", api_version = "2024-08-01-preview", api_key = "", messages = [{"role": "user", "content": "Hello!"}]) For Configuration files (YAML): model_list: - model_name: gpt-4 litellm_params: model: azure/my-gpt4-deployment api_base: https://your-resource.openai.azure.com/ api_version: "2024-08-01-preview" api_key: os.environ/AZURE_API_KEY Some specific Azure endpoints or implementations may offer paths that abstract the need to update version strings monthly (e.g., using "preview" or specific versionless endpoints), but the api-version itself remains a required component of the request structure to ensure correct routing and handling by the Azure service [2]. Citations:
🏁 Script executed: printf '%s\n' '--- agent call path ---'
sed -n '1,115p' agent.py
printf '%s\n' '--- LiteLLM Azure documentation ---'
curl -L --max-time 15 -sS https://docs.litellm.ai/docs/providers/azure/ | rg -n -C 2 'AZURE_API_VERSION|api_version|api-version|default' | head -120
printf '%s\n' '--- LiteLLM Azure source references ---'
curl -L --max-time 15 -sS https://raw.githubusercontent.com/BerriAI/litellm/main/litellm/llms/azure/chat/transformation.py | rg -n -C 3 'api_version|AZURE_API_VERSION|api-version' | head -120Repository: thirdlayerinc/autoagent Length of output: 50380 🏁 Script executed: for ref in v1.60.0 main; do
printf '%s\n' "--- LiteLLM $ref Azure transformation ---"
curl -L --max-time 15 -sS "https://raw.githubusercontent.com/BerriAI/litellm/$ref/litellm/llms/azure/chat/transformation.py" |
rg -n -C 4 'AZURE_API_VERSION|api_version|api-version|default' | head -100
doneRepository: thirdlayerinc/autoagent Length of output: 248 🏁 Script executed: python3 - <<'PY'
import json
import urllib.request
for ref in ("v1.60.0", "main"):
url = f"https://api.github.com/repos/BerriAI/litellm/git/trees/{ref}?recursive=1"
try:
with urllib.request.urlopen(url, timeout=15) as response:
data = json.load(response)
except Exception as exc:
print(ref, "TREE_ERROR", exc)
continue
print(f"--- {ref} Azure-related source paths ---")
for item in data.get("tree", []):
path = item.get("path", "")
if "azure" in path.lower() and path.endswith((".py", ".md")):
print(path)
PYRepository: thirdlayerinc/autoagent Length of output: 466 Document LiteLLM’s Azure configuration documents 🤖 Prompt for AI AgentsSource: MCP tools |
||
| ``` | ||
|
|
||
| The model selection is optional and can be changed dynamically by modifying the environment variables before running the benchmark. | ||
|
|
||
| ## Running the meta-agent | ||
|
|
||
| Point your coding agent at the repo and prompt: | ||
|
|
@@ -153,4 +233,3 @@ You can equip the agent with [Agent Skills for Context Engineering](https://gith | |
| ## License | ||
|
|
||
| MIT | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -26,8 +26,18 @@ | |
| # ============================================================================ | ||
|
|
||
| SYSTEM_PROMPT = "You are an agent that executes tasks" | ||
| MAX_TURNS = 15 | ||
|
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- agent.py changed area ---'
nl -ba agent.py | sed -n '1,130p'
printf '%s\n' '--- repository declarations and callers ---'
rg -n --hidden --glob '!node_modules' --glob '!dist' --glob '!build' \
'MAX_TURNS|run_task|Runner\.run|LLM_BASE_URL|openai-agents|litellm|Ollama' .Repository: thirdlayerinc/autoagent Length of output: 1693 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- agent.py configuration and run_task ---'
sed -n '20,110p' agent.py | cat -n
printf '%s\n' '--- dependency declaration ---'
cat pyproject.toml
printf '%s\n' '--- documented configuration and task contract ---'
sed -n '60,100p' README.md | cat -n
sed -n '35,80p' program.md | cat -n
printf '%s\n' '--- tests and task definitions mentioning turn limits ---'
rg -n --hidden --glob '!node_modules' --glob '!dist' --glob '!build' \
'max_turns|MAX_TURNS|supported task|task|run_task' tests test* docs 2>/dev/null || trueRepository: thirdlayerinc/autoagent Length of output: 8259 🌐 Web query:
💡 Result: In the OpenAI Agents SDK for Python, the Citations:
Make the agent turn limit configurable
🤖 Prompt for AI Agents |
||
|
|
||
|
Comment on lines
+29
to
+30
|
||
| # Multi-LLM configuration via LiteLLM | ||
| # Set environment variables before running: | ||
| # - LLM_PROVIDER: "openai", "anthropic", "ollama", "azure", etc. | ||
| # - MODEL: model name (e.g., "gpt-5", "claude-3-5-sonnet", "qwen3.5:35b-a3b-q8_0") | ||
| # - LLM_BASE_URL: optional base URL (required for Ollama, Azure, etc.) | ||
| # - API_KEY: provider-specific API key (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY) | ||
| import os | ||
| LLM_PROVIDER = os.getenv("LLM_PROVIDER", "openai").lower() | ||
| MODEL = os.getenv("MODEL", "gpt-5") | ||
| LLM_BASE_URL = os.getenv("LLM_BASE_URL") | ||
|
|
||
|
|
||
| def create_tools(environment: BaseEnvironment) -> list[FunctionTool]: | ||
|
|
@@ -53,11 +63,25 @@ async def run_shell(command: str) -> str: | |
| def create_agent(environment: BaseEnvironment) -> Agent: | ||
| """Build the agent. Modify to add handoffs, sub-agents, or agent-as-tool.""" | ||
| tools = create_tools(environment) | ||
|
|
||
| # Build LiteLLM-compatible model string | ||
| if LLM_PROVIDER == "ollama": | ||
| # Ollama uses custom base URL format | ||
| model_string = f"ollama_chat/{MODEL}" if not LLM_BASE_URL else f"ollama_chat/{MODEL}" | ||
| elif LLM_PROVIDER == "azure": | ||
|
Comment on lines
+68
to
+71
|
||
| # Azure uses deployment name format | ||
| model_string = f"azure/{MODEL}" | ||
| elif LLM_PROVIDER == "anthropic": | ||
| model_string = f"anthropic/{MODEL}" | ||
| else: | ||
| # Default to OpenAI format | ||
| model_string = MODEL | ||
|
|
||
| return Agent( | ||
| name="autoagent", | ||
| instructions=SYSTEM_PROMPT, | ||
| tools=tools, | ||
| model=model_string, | ||
|
Comment on lines
+67
to
+84
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: #!/usr/bin/env bash
set -euo pipefail
rg -n 'openai-agents|litellm' pyproject.toml
rg -n 'Litellm(Model|Provider)|model_provider|set_default.*provider|Agent\(' agent.pyRepository: thirdlayerinc/autoagent Length of output: 263 🏁 Script executed: #!/usr/bin/env bash
set -euo pipefail
printf '%s\n' '--- agent.py imports and configuration ---'
sed -n '1,115p' agent.py
printf '%s\n' '--- dependency declarations ---'
sed -n '1,35p' pyproject.toml
printf '%s\n' '--- repository references to provider setup and model construction ---'
rg -n -C 3 'LLM_(PROVIDER|BASE_URL|MODEL)|Litellm(Model|Provider)|ModelProvider|set_default|Agent\(' --glob '*.py' --glob 'pyproject.toml' .Repository: thirdlayerinc/autoagent Length of output: 6851 🌐 Web query:
💡 Result: The OpenAI Agents Python SDK supports integration with LiteLLM to use various LLM providers through the Citations:
Route non-OpenAI models through LiteLLM.
🧰 Tools🪛 Ruff (0.16.2)[warning] 70-70: Useless (RUF034) 🤖 Prompt for AI AgentsSources: MCP tools, Linters/SAST tools |
||
| ) | ||
|
Comment on lines
80
to
85
|
||
|
|
||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,4 +9,5 @@ dependencies = [ | |
| "openpyxl", | ||
| "numpy", | ||
| "harbor", | ||
| "litellm>=1.60.0", | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| .skill-config.json |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "mainRepo": "https://github.com/Oncorporation/open-autoagent", | ||
| "domainRepo": "https://github.com/Oncorporation/secure-torrent-mcp-agent", | ||
| "domainBranch": "domain/secure-torrent", | ||
| "llmProvider": "ollama", | ||
| "model": "qwen3.8:27b-mtp-q8_0", | ||
| "ollamaEndpoint": "http://127.0.0.1:11434", | ||
| "hardware": "AMD Ryzen AI Max+ 395 64GB-64GB", | ||
| "skillMetadata": { | ||
| "name": "open-autoagent-ollama-setup", | ||
| "description": "Set up Oncorporation open-autoagent on a host-native Ollama daemon for Harbor harness experiments.", | ||
| "license": "MIT", | ||
| "version": "1.0.0", | ||
| "hermesTags": ["open-autoagent", "ollama", "harbor", "harness"], | ||
| "hermesCategory": "mcp-install" | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # Setup Workflow | ||
|
|
||
| The setup folder now supports **generic repository setup** via optional configuration. | ||
|
|
||
| ## Workflow | ||
|
|
||
| ### Option 1: Quick Start (Default) | ||
| Use the original hardcoded repo, model, and settings: | ||
|
|
||
| ```bash | ||
| # Linux / macOS | ||
| bash setup/install.sh | ||
|
|
||
| # Windows PowerShell | ||
| .\setup\install.ps1 | ||
| ``` | ||
|
|
||
| Pick your harness, install, done. Defaults: | ||
| - Repo: `https://github.com/Oncorporation/open-autoagent` | ||
| - Model: `qwen3.8:27b-mtp-q8_0` | ||
| - Ollama: `http://127.0.0.1:11434` | ||
|
|
||
| ### Option 2: Custom Configuration | ||
| Use your own repo, model, hardware, LLM provider: | ||
|
|
||
| ```bash | ||
| # 1. Configure | ||
| bash setup/configure.sh # Linux/macOS | ||
| .\setup\configure.ps1 # Windows | ||
|
|
||
| # 2. Install | ||
| bash setup/install.sh # Linux/macOS | ||
| .\setup\install.ps1 # Windows | ||
| ``` | ||
|
|
||
| The installer checks for `.skill-config.json` and uses it to customize the | ||
| SKILL.md before copying. If no config exists, defaults are used. | ||
|
|
||
| ## What Gets Configured | ||
|
|
||
| | Setting | Purpose | Default | | ||
| |---|---|---| | ||
| | Main repo | GitHub/HF/local path to clone | `https://github.com/Oncorporation/open-autoagent` | | ||
| | Domain repo | Optional catalog/context repo | `https://github.com/Oncorporation/secure-torrent-mcp-agent` | | ||
| | Domain branch | Branch name in main repo | `domain/secure-torrent` | | ||
| | LLM provider | `ollama`, `openai`, `anthropic`, `azure` | `ollama` | | ||
| | Model | Model name/tag | `qwen3.8:27b-mtp-q8_0` | | ||
| | Ollama endpoint | Only if provider=ollama | `http://127.0.0.1:11434` | | ||
| | Hardware | Info string (notes only) | `AMD Ryzen AI Max+ 395 64GB-64GB` | | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. **`configure.sh/.ps1`** → prompts → saves to `.skill-config.json` | ||
| 2. **`install.sh/.ps1`** → reads config (or defaults) → processes `SKILL.md.template` → installs customized SKILL.md | ||
| 3. **Harness triggers** → runs the skill with custom repo/model/hardware baked in | ||
|
|
||
| `.skill-config.json` is gitignored so it never gets committed. | ||
|
|
||
| ## Adding a New Harness | ||
|
|
||
| 1. Create `setup/harness/new-harness/` directory | ||
| 2. **Either:** | ||
| - Pre-build: copy `SKILL.md.template` → `new-harness/SKILL.md` (installers will use it as-is) | ||
| - Or: install will auto-generate from template + config | ||
| 3. Update installer menu (both `.sh` and `.ps1`) with harness option and trigger instructions | ||
| 4. Done — installers pick up the new folder automatically | ||
|
|
||
| ## Files | ||
|
|
||
| | File | Purpose | | ||
| |---|---| | ||
| | `configure.sh` / `configure.ps1` | Prompt user for custom config → save to `.skill-config.json` | | ||
| | `install.sh` / `install.ps1` | Read config (or defaults) → process template → install to harness | | ||
| | `SKILL.md.template` | Generic skill template with `{{PLACEHOLDERS}}` | | ||
| | `.skill-config.json` | User config (created by configure, gitignored) | | ||
| | `harness/*/SKILL.md` | Pre-built harness-specific skills (optional) | | ||
| | `open-autoagent-ollama-setup.md` | Legacy canonical skill (fallback only) | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: thirdlayerinc/autoagent
Length of output: 11850
🏁 Script executed:
Repository: thirdlayerinc/autoagent
Length of output: 4350
Load
.envbefore starting the benchmark.The documented commands create
.envbut do not load it.agent.pyreadsLLM_PROVIDER,MODEL, andLLM_BASE_URLfrom the process environment withos.getenv, so the documented settings are not applied. Addset -a && source .env && set +abefore the Harbor command, or document an equivalent export step.🤖 Prompt for AI Agents
Source: MCP tools