[CLI] Make API spec validation configurable

Summary

Make the cg check spec command more flexible by allowing users to set compliance thresholds, configure rule strictness, and selectively ignore rules when validating an implementation’s OpenAPI spec against the CommonGrants base spec.

What problem are we solving?

Currently, cg check spec fails if there are any validation errors in an implementation’s OpenAPI spec. This all-or-nothing behavior makes it difficult for teams to adopt the CommonGrants protocol incrementally.

Without more configurability:

Developers cannot silence or ignore certain rules, even if they are known issues or not relevant for their implementation.
The command cannot distinguish between critical errors and minor warnings.
CI/CD pipelines break on partial compliance, making it harder to adopt CommonGrants progressively.

As a result, teams face friction integrating the check into their workflows, which slows down adoption.

What do we propose?

Enhance cg check spec to support more granular configuration:

Compliance threshold: Instead of failing on the first error, calculate a % compliance score (e.g., 95%) and fail only if it falls below a user-configurable threshold (default = 100%).
Configurable strictness: Support distinguishing warnings vs. errors, and allow users to set a strictness level (e.g., fail only on errors, fail on warnings, etc.)
Ignore rules: Allow users to silence specific rules, similar to tools like pylint, ruff, or eslint. Ideally rules could be ignored globally or by path.
Configuration file: Introduce support for a config file (e.g. .cgcheck.yaml or .cgcheck.toml) that defines thresholds, rule strictness, and ignored rules.
CI/CD integration: Ensure exit codes reflect configured thresholds and strictness, making it easy to adopt in pipelines.

Here's an example of a potential config file format:

complianceThreshold: 95
strictness: error  # options: error, warning, strict
ignore: # ignores across all routes
  - "response-schema-conflict"
  - "missing-status-code"

Why does it matter to CommonGrants adopters?

Configurability enables incremental adoption. Instead of having to be 100% compliant from day one, teams can start by tracking compliance, ignoring non-critical rules, and gradually tightening their standards over time. This lowers the barrier to integrating cg check spec into real-world workflows, especially in CI/CD environments.

Effort and dependencies

Moderate effort: ~3 sprints

Acceptance criteria

cg check spec supports a config file for compliance thresholds, rule strictness, and ignored rules
Command reports a compliance % and fails only if below threshold (default 100%)
Users can configure whether warnings count as failures, and can selectively ignore rules
Documentation updated with examples of config-driven usage in local dev and CI/CD environments

Fider

Vote for this feature: Post
Votes: 6

Dependencies

Here are the upstream and downstream dependencies for this issue:

flowchart LR

  %% ─────────────────────────────
  %% Styles
  %% ─────────────────────────────
  classDef default fill:#fff,stroke:#333,stroke-width:1px,color:#000,rx:5,ry:5
  classDef InProgress fill:#e1f3f8,stroke:#07648d,stroke-width:2px,color:#000
  classDef Done fill:#8DE28D,stroke:#204e34,stroke-width:3px,color:#000
  style Canvas fill:transparent,stroke:#171716
  style Legend fill:#F7F7F4,stroke:#171716


  %% ─────────────────────────────
  %% Legend
  %% ─────────────────────────────
  subgraph Legend["Key"]
    direction LR
    k1["Todo"]
    k2["In progress 🛠️ "]:::InProgress
    k3["Done ✔️"]:::Done

    k1 -.-> k2 -.-> k3
  end

  %% ─────────────────────────────
  %% Main canvas
  %% ─────────────────────────────
  subgraph Canvas["Dependencies"]
    direction LR

        HHS/simpler-grants-protocol#331["Make API spec validation configurable"]
        HHS/simpler-grants-protocol#335["Create GH action and badge for CommonGrants compliance"]

    %% ─────────────────────────────
    %% Relationships
    %% ─────────────────────────────
    HHS/simpler-grants-protocol#331 --> HHS/simpler-grants-protocol#335

  end

Context

We need an extensible way to configure the behavior of cg check spec to override or alter how certain validations are applied. Ideally this configuration format would borrow inspiration from other popular linters like eslint, pylint, and ruff and could be easily extended to other CLI commands.

In particular we want to answer the following questions:

File type: What configuration file type(s) are supported?
Format: What format or schema should this config file adhere to?
Discovery: How is the config file passed to or discovered by the CLI?

Decision

We'll support yaml or json config files, that can either be passed using the --config flag to the CLI or automatically discovered if a file exists with the following name in the root of the current directory: cg.config.[yml | yaml | json]

Config files MUST follow the format outlined below, which is designed to closely mirror an updated version of the check spec output, making it easier for CLI users to create config files using the output as a guideline.

`cg check spec` output example

Here's an example of the new proposed output for cg check spec, designed to closely mirror the structure of the overrides section of the config file:

status: Failed
score: 60
issues:
  Extra routes:
    "GET /common-grants/test/":
      level: error
      rule: extra_route
    "POST /common-grants/test/":
      level: error
      rule: extra_route

  Missing routes:
    "GET /common-grants/applications/{appId}":
      level: error
      rule: missing_required_route
    "GET /common-grants/applications/{appId}/status":
      level: error
      rule: missing_optional_route

  Route conflicts:
    "GET /common-grants/opportunities/{oppId}":
      issue_count: 2
      issues:
        - rule:
            level: error
            rule: missing_required_prop
            message: Implementation is missing required property 'data.status.value'.
          where:
            location: response_body
            status: 200
            content_type: application/json
            property: data.status.value

        - rule:
            level: warn
            rule: missing_optional_prop
            message: Implementation is missing optional property 'data.status.description'.
          where:
            location: response_body
            status: 404
            content_type: application/json
            property: status_code

    "POST /common-grants/opportunities/search":
      issue_count: 2
      issues:
        - rule:
            level: error
            rule: type_conflict
            message: Type conflict. Base is 'integer', implementation is 'string' for property 'data.id'.
          where:
            location: response_body
            status: 200
            content_type: application/json
            property: data.id

        - rule:
            level: error
            rule: extra_prop
            message: Implementation has extra property 'status_code' and `additionalProperties` is not enabled.
          where:
            location: response_body
            status: 200
            content_type: application/json
            property: pagination.total_count

        - rule:
            level: error
            rule: extra_prop
            message: Implementation has extra property 'status_code' and `additionalProperties` is not enabled.
          where:
            location: request_body
            property: pagination.total_count

Config example

Here's a sample config file that would adjust how we handle the warnings and errors in the output above, with some optional explanation values documenting each override rule.

# cg.config.yaml
check-spec:
  strictness: standard
  threshold: 100
  overrides:
    extra_routes:
      - path: /common-grants/test/**
        rules:
          extra_route: ignore
        explanation: |
          Ignore extra routes for paths that start with /common-grants/test/

    missing_routes:
      - rules:
          missing_optional_route: ignore
          missing_required_route: warn
        explanation: |
          If a route is missing, ignore if it's optional and raise a warning if it's required

    route_conflicts:
      - method: [GET, POST]
        rules:
          extra_prop: ignore
        where:
          status: "4xx"
          location: response
          property: status_code
        explanation: |
          Ignore extra `status_code` property in 4xx responses

      - method: GET
        path: /common-grants/opportunities/{oppId}
        rules:
          missing_required_prop: ignore
          missing_optional_prop: ignore
          extra_prop: ignore
        where:
          status: 200
          location: response
          property: data.status.**
        explanation: |
          Ignore missing and extra properties on the `data.status` object
          in the response body for GET /common-grants/opportunities/{oppId}

      - path: /common-grants/**
        rules:
          extra_prop: ignore
        where:
          property:
            - pagination.total_count
            - pagination.total_pages
        explanation: |
          Ignore the following extra properties on the `pagination` object
          in both response and request bodies

Config syntax notes

Here's a more detailed breakdown of the override syntax:

method: ["GET", "POST"]    # Optional:Single method or array of methods
path: "/common-grants/**"  # Optional: Route path pattern with glob support, or array of paths
rules:                     # Required: rule_id: ignore | warn | error
  extra_prop: ignore
where:                     # Optional where clause that restricts scope of the override
  location: response_body  # Optional: response_body | request_body | query_param
  status: 200              # Optional: Exact match, array, or 2xx, 4xx
  property: data.**.       # Optional: Glob path to prop, or array of property paths
explanation: |             # Optional: Note for other developers explaining override
  Foo bar baz

Note

Each of the optional properties only restrict the scope of override. For example, method: GET limits an override to routes with a GET method, but omitting the method property defaults to all methods. As a result, the following override would ignore extra_prop globally (i.e. on all endpoints and schemas):

rules:
  extra_prop: ignore

Navigation Menu

Repository navigation

[CLI] Make API spec validation configurable #331

Summary

Acceptance criteria

Fider

Dependencies

Context

Decision

`cg check spec` output example

Config example

Config syntax notes

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[CLI] Make API spec validation configurable #331

Description

Summary

Acceptance criteria

Fider

Dependencies

Activity

widal001 commented on Nov 5, 2025

Context

Decision

cg check spec output example

Config example

Config syntax notes

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`cg check spec` output example