Skip to content

🧪 [testing improvement] Add unit tests for provider config accessors - #42

Closed
b3nw wants to merge 13 commits into
devfrom
tests-provider-config-16947276259564390506
Closed

b3nw wants to merge 13 commits into
devfrom
tests-provider-config-16947276259564390506

Conversation

@b3nw

@b3nw b3nw commented Apr 25, 2026

Copy link
Copy Markdown
Owner

🎯 What: The testing gap addressed is the missing unit tests for get_full_provider_config and get_provider_ui_config functions in src/rotator_library/provider_config.py.
📊 Coverage: Test coverage is now complete for both known and unknown providers, verifying data merging and fallbacks.
Result: Enhanced test coverage provides confidence against regressions and validates correct dictionary fallback behaviour.


PR created automatically by Jules for task 16947276259564390506 started by @b3nw

b3nw and others added 10 commits April 24, 2026 19:40
…ardization, and utilities

Core infrastructure improvements:
- Smart 'latest' model alias resolution with cost-based tiebreaking
- Standardized error responses with proper HTTP status codes and error.code field
- ProxyExhaustionError for structured credential exhaustion reporting
- TerminalRequestError for non-rotatable errors (404, model not found)
- Per-provider retry count override via MAX_RETRIES_{PROVIDER} env var
- Retry 429 rate_limit errors with backoff instead of rotating
- Cached token pricing in streaming cost calculation
- Split quota stats into current_period and global/lifetime views
- Log rotation for proxy.log and proxy_debug.log (RotatingFileHandler)
- Include latest virtual models in /v1/models endpoint
- Resolve singleton cache pollution for dynamic providers
- Fork-specific README and .gitignore updates
…ased model filtering, and enhanced X-Initiator heuristic
Test suite designed to catch breakage from branch re-organization without
sending queries to real LLM providers. Covers all critical integration
points that previously broke silently during deployment.

Coverage:
- Anthropic↔OpenAI format translation & streaming
- Error classification (determines retry/rotation behavior)
- Request sanitization (prevents 400s from invalid params)
- Provider-specific request transforms
- Model alias & latest registry parsing
- Usage tracking (windows, quota groups, custom caps)
- Credential discovery, deduplication, env:// URI
- Provider plugin registration & singleton pattern
- Proxy endpoint routing & auth

All tests use synthetic credentials and mocked HTTP.
Runs in ~2.3s. Zero API cost.
…flow

Replaces the old manifest-driven multi-branch replay system with a
simpler linear commit stack. Changes are made via fixup!/autosquash.
Upstream syncs are a single git rebase.

Includes:
- AGENTS.md: entry point for all AI coding agents
- .agent/rules/claude.md: Claude-specific SSH/deployment notes
- .agent/rules/llm-proxy.md: container layout and deployment pipeline
- .agent/skills/upstream-sync/SKILL.md: sync workflow reference
Custom provider for Google Vertex AI Express Mode API keys that uses
x-goog-api-key header authentication against the Vertex AI
OpenAI-compatible endpoint. Supports non-streaming and streaming
chat completions with automatic model discovery.

Models are prefixed as vertex/ (e.g. vertex/gemini-3.1-flash-lite-preview).
Env vars: VERTEX_PROJECT, VERTEX_LOCATION, VERTEX_API_KEY_N
@google-labs-jules

Copy link
Copy Markdown

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces unit tests for provider configuration accessors in tests/test_provider_config.py. The review feedback suggests enhancing the test suite by explicitly verifying the merge priority when combining data sources and refactoring multi-scenario tests using pytest parametrization for better isolation and readability.

Comment on lines +32 to +54
def test_get_full_provider_config_known_provider():
"""Test get_full_provider_config with a known provider."""
mock_scraped_providers = {
"test_provider": {
"api_base": "https://api.test.com",
"models": ["model-a"],
}
}
mock_litellm_providers = {
"test_provider": {
"category": "test_category",
"note": "Test note",
}
}
with patch("rotator_library.provider_config.SCRAPED_PROVIDERS", mock_scraped_providers):
with patch("rotator_library.provider_config.LITELLM_PROVIDERS", mock_litellm_providers):
config = get_full_provider_config("test_provider")

# Should have properties from both
assert config["api_base"] == "https://api.test.com"
assert config["models"] == ["model-a"]
assert config["category"] == "test_category"
assert config["note"] == "Test note"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The test verifies that data from both sources is merged, but it doesn't verify the merge priority (i.e., that UI configuration overrides scraped data). Given that get_full_provider_config uses {**scraped, **ui_config}, it's important to ensure that if a key exists in both, the UI config value is the one preserved. It's also recommended to assert the entire dictionary to ensure no unexpected keys are present.

def test_get_full_provider_config_known_provider():
    """Test get_full_provider_config with a known provider, verifying merge priority."""
    mock_scraped_providers = {
        "test_provider": {
            "api_base": "https://api.test.com",
            "models": ["model-a"],
            "category": "scraped_category",
        }
    }
    mock_litellm_providers = {
        "test_provider": {
            "category": "test_category",
            "note": "Test note",
        }
    }
    with patch("rotator_library.provider_config.SCRAPED_PROVIDERS", mock_scraped_providers):
        with patch("rotator_library.provider_config.LITELLM_PROVIDERS", mock_litellm_providers):
            config = get_full_provider_config("test_provider")
            assert config == {
                "api_base": "https://api.test.com",
                "models": ["model-a"],
                "category": "test_category",
                "note": "Test note",
            }

Comment thread tests/test_provider_config.py Outdated
Comment on lines +68 to +86
def test_get_full_provider_config_partial_data():
"""Test get_full_provider_config when provider is only in scraped or only in UI config."""
# Only in UI config
mock_litellm_providers = {"test_ui_only": {"category": "ui_only_category"}}
mock_scraped_providers = {}

with patch("rotator_library.provider_config.SCRAPED_PROVIDERS", mock_scraped_providers):
with patch("rotator_library.provider_config.LITELLM_PROVIDERS", mock_litellm_providers):
config = get_full_provider_config("test_ui_only")
assert config == {"category": "ui_only_category"}

# Only in scraped config
mock_litellm_providers = {}
mock_scraped_providers = {"test_scraped_only": {"api_base": "https://test.com"}}

with patch("rotator_library.provider_config.SCRAPED_PROVIDERS", mock_scraped_providers):
with patch("rotator_library.provider_config.LITELLM_PROVIDERS", mock_litellm_providers):
config = get_full_provider_config("test_scraped_only")
assert config == {"api_base": "https://test.com", "category": "other"}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This test function covers two distinct scenarios. Using @pytest.mark.parametrize would make the test more concise, readable, and ensure that each case is executed independently.

Suggested change
def test_get_full_provider_config_partial_data():
"""Test get_full_provider_config when provider is only in scraped or only in UI config."""
# Only in UI config
mock_litellm_providers = {"test_ui_only": {"category": "ui_only_category"}}
mock_scraped_providers = {}
with patch("rotator_library.provider_config.SCRAPED_PROVIDERS", mock_scraped_providers):
with patch("rotator_library.provider_config.LITELLM_PROVIDERS", mock_litellm_providers):
config = get_full_provider_config("test_ui_only")
assert config == {"category": "ui_only_category"}
# Only in scraped config
mock_litellm_providers = {}
mock_scraped_providers = {"test_scraped_only": {"api_base": "https://test.com"}}
with patch("rotator_library.provider_config.SCRAPED_PROVIDERS", mock_scraped_providers):
with patch("rotator_library.provider_config.LITELLM_PROVIDERS", mock_litellm_providers):
config = get_full_provider_config("test_scraped_only")
assert config == {"api_base": "https://test.com", "category": "other"}
@pytest.mark.parametrize("provider_key, mock_scraped, mock_ui, expected", [
(
"test_ui_only",
{},
{"test_ui_only": {"category": "ui_only_category"}},
{"category": "ui_only_category"}
),
(
"test_scraped_only",
{"test_scraped_only": {"api_base": "https://test.com"}},
{},
{"api_base": "https://test.com", "category": "other"}
),
])
def test_get_full_provider_config_partial_data(provider_key, mock_scraped, mock_ui, expected):
"""Test get_full_provider_config when provider is only in scraped or only in UI config."""
with patch("rotator_library.provider_config.SCRAPED_PROVIDERS", mock_scraped):
with patch("rotator_library.provider_config.LITELLM_PROVIDERS", mock_ui):
assert get_full_provider_config(provider_key) == expected

@b3nw

b3nw commented Apr 26, 2026

Copy link
Copy Markdown
Owner Author

[Gemini-CLI Review Skill] @jules Reviewing this PR against the checklist and Gemini Code Assist feedback:

  1. [Gemini Code Assist] Merge Priority: Add a test case that explicitly verifies the merge priority when both SCRAPED_PROVIDERS and LITELLM_PROVIDERS contain the same key. The test should confirm which source "wins" for overlapping fields.
  2. [Gemini Code Assist] Parametrization: Refactor the multi-scenario tests (test_get_full_provider_config_partial_data) using @pytest.mark.parametrize for better isolation and cleaner test reports.
  3. [Boundary/Edge] Empty Data Scenarios: Verify behavior when dictionaries are present but empty (which you partially covered) and when they contain malformed entries (e.g., a provider key with a None value) to ensure the merge logic is resilient.

@google-labs-jules

Copy link
Copy Markdown

[Gemini-CLI Review Skill] @jules Reviewing this PR against the checklist and Gemini Code Assist feedback:

  1. [Gemini Code Assist] Merge Priority: Add a test case that explicitly verifies the merge priority when both SCRAPED_PROVIDERS and LITELLM_PROVIDERS contain the same key. The test should confirm which source "wins" for overlapping fields.
  2. [Gemini Code Assist] Parametrization: Refactor the multi-scenario tests (test_get_full_provider_config_partial_data) using @pytest.mark.parametrize for better isolation and cleaner test reports.
  3. [Boundary/Edge] Empty Data Scenarios: Verify behavior when dictionaries are present but empty (which you partially covered) and when they contain malformed entries (e.g., a provider key with a None value) to ensure the merge logic is resilient.

I have implemented the requested changes: I added a test case to explicitly verify the merge priority when both SCRAPED_PROVIDERS and LITELLM_PROVIDERS contain the same key. I also refactored the multi-scenario tests using @pytest.mark.parametrize for better isolation and added a test to verify behavior when dictionaries contain malformed entries, such as a provider key with a None value. These edge cases required updating the logic in get_full_provider_config and get_provider_ui_config to be resilient to None values.

@b3nw
b3nw marked this pull request as ready for review April 26, 2026 03:23
@kilo-code-bot

kilo-code-bot Bot commented Apr 26, 2026

Copy link
Copy Markdown

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 0

Changes Reviewed

  • src/rotator_library/provider_config.py - 2 functions modified to handle None values in provider dictionaries
  • tests/test_provider_config.py - New tests covering known/unknown providers, merge priority, and edge cases

Assessment

The code changes are correct:

  • get_provider_ui_config now explicitly returns {"category": "other"} when the provider key maps to None rather than returning None
  • get_full_provider_config follows the same pattern for both SCRAPED_PROVIDERS and LITELLM_PROVIDERS
  • Merge priority correctly gives UI config precedence over scraped data (test case at lines 89-94)

Tests are comprehensive and test the actual changed behavior.

Files Reviewed (2 files)
  • src/rotator_library/provider_config.py - No issues
  • tests/test_provider_config.py - No issues

Reviewed by minimax-m2.7 · 124,269 tokens

@b3nw

b3nw commented Apr 26, 2026

Copy link
Copy Markdown
Owner Author

[Gemini-CLI Review Skill] Consolidated into the linear commit stack on dev.

@b3nw b3nw closed this Apr 26, 2026
@google-labs-jules

Copy link
Copy Markdown

[Gemini-CLI Review Skill] Consolidated into the linear commit stack on dev.

Acknowledged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant