linting schema examples appears to stop after first error

[DavidBiesack]

Given the following schema.yaml file :

$id: 'https://api.apiture.com/schemas/systemIdentityChallenges'
$schema: 'https://json-schema.org/draft/2020-12/schema'
$defs:
  nested:
    title: Nested
    description: Nested within outer.
    type: object
    unevaluatedProperties: false
    required:
      - foo
      - bar
    properties:
      foo:
        type: string
        minLength: 2
        maxLength: 80
      bar:
        type: string
        minLength: 2
        maxLength: 80
    examples:
      - foo: 'foo'
        bar: 'bar'
  outer:
    title: Outer
    type: object
    unevaluatedProperties: false
    required:
      - a
    properties:
      a:
        description: A nested object.
        allOf:
          - $ref: '#/$defs/nested'
      b:
        description: Another nested object.
        allOf:
          - $ref: '#/$defs/nested'
    examples:
      - a:
          foo: 'foo'
          bar: 'bar'
        b:
          foo: 'foo'
          bar: false
          frob: false
        d:
          foo: true
          bar: 123

running

$ jsonschema version
9.2.4
$ jsonschema lint schema.yaml

shows the following:

  Only include instances in the `examples` array that validate against the schema (blaze/valid_examples)
    at schema location "/$defs/outer"
    Invalid example instance at index 0
      The value was expected to consist of a string of 2 to 80 characters
        at instance location "/b/bar"
        at evaluate path "/properties/b/allOf/0/$ref/properties/bar/type"

catches one error (wrong type on (examples[0].b.bar -- boolean should be string)
but does not catch the unevaluated property examples[0].b.frob

if I fix examples[0].b.bar (change false to "bar")
I now see

  Only include instances in the `examples` array that validate against the schema (blaze/valid_examples)
    at schema location "/$defs/outer"
    Invalid example instance at index 0
      The object value was not expected to define the property "frob"
        at instance location "/b/frob"
        at evaluate path "/properties/b/allOf/0/$ref/unevaluatedProperties"
      The object value was not expected to define unevaluated properties
        at instance location "/b"
        at evaluate path "/properties/b/allOf/0/$ref/unevaluatedProperties"

After fixing that (i.e. remove the extra property), running again yields a third error

  Only include instances in the `examples` array that validate against the schema (blaze/valid_examples)
    at schema location "/$defs/outer"
    Invalid example instance at index 0
      The object value was not expected to define the property "d"
        at instance location "/d"
        at evaluate path "/unevaluatedProperties"
      The object value was not expected to define unevaluated properties
        at instance location ""
        at evaluate path "/unevaluatedProperties"

A better experience would be to list all errors, so one is not forced to
make n separate edits/corrections and run the tool n separate times to resolve all problems.

[jviotti]

Hm, this is an interesting one. So what you see is that https://github.com/sourcemeta/blaze will stop evaluation when encountering the first error, and it will return a stack trace up to that point.

This makes sense from a performance point of view, as we want to break as early as possible when we know what the outcome of the validation process will be. But of course, it's not great from an error reporting point of view.

Blaze supports two modes of execution: "exhaustive" and "fast validation". I think we can make the "exhaustive" evaluate every branch on "AND" operations (i.e. make it truly exhaustive) for the sake of better error messages. Let me think about it.

[jviotti]

Here is a simpler and more concrete example:

// schema.json

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "properties": {
    "foo": { "minLength": 2 },
    "bar": { "minLength": 2 }
  }
}

---

// instance.json

{ "foo": "1", "bar": "2" }

---

$ jsonschema validate schema.json instance.json --trace
-> (push) "/properties" (LogicalAnd)
   at ""
   at keyword location "#/properties"
   at vocabulary "https://json-schema.org/draft/2020-12/vocab/applicator"

-> (push) "/properties/bar/minLength" (AssertionStringSizeGreater)
   at "/bar"
   at keyword location "#/properties/bar/minLength"
   at vocabulary "https://json-schema.org/draft/2020-12/vocab/validation"

<- (fail) "/properties/bar/minLength" (AssertionStringSizeGreater)
   at "/bar"
   at keyword location "#/properties/bar/minLength"
   at vocabulary "https://json-schema.org/draft/2020-12/vocab/validation"

<- (fail) "/properties" (LogicalAnd)
   at ""
   at keyword location "#/properties"
   at vocabulary "https://json-schema.org/draft/2020-12/vocab/applicator"

Evaluation stops after the first failure on the first property.

[DavidBiesack]

a command line option (such as -no-short-circuit) on the lint command would be OK (or an option in a ~/.jsonschemarc config/options file so I could set it once and not have to specify it each time I use lint)