CLI/Standard JSON testcase

[djolertrk]

Reduce repetitive Boost test setup by introducing a dedicated TestCase variant for StandardCompiler.

Context: review comment.

cc @cameel

---

Edited by @cameel: Added detailed specification.

Abstract

Create a dedicated TestCase for testing compiler's interface (both CLI and Standard JSON). The case should allow fine-grained assertions against outputs and support rudimentary parametrization to make repetitive tests less verbose.

Motivation

The facilities we currently have for testing the range of compiler's input options and outputs are tedious and too verbose.

We have boost::test-based suites like CommandLineParserTest, CommandLineInterfaceTest or StandardCompiler tests, which allow a lot of flexibility, but are not very readable and don't have a mechanism for automatically updating the expectations. We also have command-line tests - more readable and with an update mechanism but very verbose and without any means to check the output other than printing it all. What we need is a new isoltest-based suite that combines the good parts of both.

Specification

Define a test case based on JSONExpectationTest. The case should have a multi-file structure with a dedicated top-level file listing a series of solc invocations and expectations specified as assertions against compiler's output.

I recommend looking at the examples in parallel with the text to get the idea of how the elements described below would work in practice. The description is detailed to leave as few ambiguities as possible, but the core idea is pretty straightforward.

Input

Files

In the simplest case the input specifies a single solc command. This will be enough only for the most trivial tests that do not pass any files to the compiler. Extra files can be supplied in the multi-source form. In that case files are separated with ==== Source: <path> ==== headers, following our usual convention for multi-source tests.

In addition to sources, the test may also contain special sections that do not represent actual files: ==== Commands ====, ==== Stdin ====, ==== Params ====. These are not meant to be present in compiler's VFS along with other sources. Each of these sections may appear at most once.

Commands

One or more commands must be given.

Unlike in command-line tests, a command must be self-contained. It must start with solc and no input files are added implicitly. One cannot specify arbitrary executable - solc only marks the start of the command for readability. The command is not given to Bash - solc prefix is stripped and the rest is passed directly to compiler's command-line parser.

Empty lines and comments are stripped from the commands. Sequences of whitespace are squashed into a single space. For now there is no quoting mechanism that would allow arguments containing whitespace, but we may consider introducing one in the future.

A command may span multiple lines. It must start on an unindented line and every subsequent indented line is appended to it. No continuation markers like \ are needed.

The test always executes all the commands and each one must satisfy all expectations. I.e. either all the commands must produce the same output or expectations only cover the parts that are the same - unlike command-line tests, JSONExpectationTest does not require the expectations to cover the whole output.

In the multi-file form, commands are introduced by a special header: ==== Commands ====. Additionally, we may consider making the header optional and treating the initial part of the file, before the first header, as the list of commands by default.

Stdin

If a ==== Stdin ==== header is present, its content is used as standard input for every command. This is the equivalent of the stdin file we support in current command-line tests.

Test parameters

A test may use a placeholder that will be filled using predefined parameters. Every command is repeated for every parameter.

Parameters are listed in a ==== Params ==== section, one per line. Empty lines are skipped. Leading and trailing whitespace is stripped.

Every instance of the <PARAM> placeholder in commands, source files, stdin and expectations is replaced by the parameter. This is a simple search and replace. No assumptions are made about the structure of a parameter. When used inside a command, <PARAM> does not introduce quoting: e.g. a b produces two options, not one like "a b" in Bash would.

In the future we may consider allowing multiple parameters for more complex test matrices.

There is no escaping mechanism, so tests cannot contain <PARAM> literally, but we may consider introducing one in the future if it becomes a problem.

If the ==== Params ==== section is present but no <PARAM> placeholder is used anywhere, there should be a warning.

Comments

Files specifying commands and parameters may contain comments. Only single-line # comments are supported.

Comments are stripped from these files before any other processing. In particular, they're stripped from parameters before subsitutions so e.g. a <PARAM> c will not comment-out c when the parameter is b # comment.

Expectations

Settings

The test should support the following settings:

• standardJSON (bool). false by default. When true, the test is assumed to be a Standard JSON test, which means:
  • No ==== Commands ==== section is allowed. A single solc --standard-json - command is implicitly assumed.
  • Standard JSON input must be given on stdin.
  • The output is parsed as JSON and it's possible to define expectations against it.
  • The test fails if the output does not look like Standard JSON. In particular:
    • JSON parsing fails.
    • It has top-level keys other than those allowed in Standard JSON.
    • It does not validate against Standard JSON schema (optional; we would have to define that schema first).
• implicitOptions (string). Default: --pretty-json --json-indent 4. Options that are always implicitly appended at the end of every command.
• stripPrereleaseWarning (bool). true by default. Extra output post-processing that removes the warning that appears in non-release versions of the compiler.
• stripLicenseWarning (bool). true by default: removes the warning about missing SPDX license comment from stderr and parsed Standard JSON output.
• stripVersionWarning (bool). true by default: removes the warning about missing version pragma from stderr and parsed Standard JSON output.

Note that command-line tests have more extensive filtering of the output. For simplicity we can limit ourselves to removing the warnings listed above. Also, no need to try to remove them from raw, unparsed stdout, which adds complexity in the command-line test version. The new test case gives us more tools for hiding undesirable parts of the output.

The test case does not need most of the settings we have in other test cases, like EVM version, optimization or pipeline selection, because these are a part or the command.

Output

When standardJSON setting is disabled, only the following properties are available:

• .exit - exit code of the compiler
  • Zero means success, non-zero means error
  • If not given explicitly, there is an implicit expectation of zero - a command returning with an error should fail the test by default
• .stdout - raw content of stdout
• .stderr - raw content of stderr

When standardJSON setting is enabled, it is possible to assert also a0gainst the parsed JSON output using:

• An absolute path within the whole JSON output. Starts with a dot.
• A path relative to a given contract+output. Uses the parenthesized syntax.

Filters

The set of filters available in JSONExpectationTest should be extended:

• empty (bool): true when length is non-zero
• contains "<string>" (bool): Allows doing partial matches against string properties.
• grep "<string>" (list of string): Returns all lines containing a given string.

We may consider adding even more filters if necessary to convert existing tests. For example wildcard or regex matches.

Examples

Trivial test

solc --version
// ----
// .stderr | empty: true

CLI and grep

==== Commands ====
solc input.sol
    --experimental               # Still an experimental feature
    --debug-info ast-id,ethdebug # ast-id is not enabled automatically when you select ethdebug
    --via-ir                     # ethdebug requires IR
    --ethdebug
==== Source: input.sol ====
contract C {}
// ----
// .stdout | grep "\"mnemonic\":": [
//     "\"mnemonic\": \"PUSH1\",
//     "\"mnemonic\": \"PUSH1\",
//     "\"mnemonic\": \"MSTORE\",
//     "\"mnemonic\": \"PUSH0\",
//     "\"mnemonic\": \"DUP1\",
//     "\"mnemonic\": \"REVERT\"
// ]

Parameters and stdin

solc - --experimental --debug-info ethdebug <PARAM>
==== Stdin ====
contract C { function f() public pure {} }
==== Params ====
# Asm export
--asm-json
--asm-json --ethdebug-program --via-ir
--asm-json --ethdebug-program-runtime --via-ir

# Yul AST
--ir-ast-json
--ir-ast-json --ethdebug-program --via-ir

# Optimized Yul AST
--ir-optimized-ast-json
--ir-optimized-ast-json --ethdebug-program --via-ir
// ----
// .exit: 1
// .stderr: "Error: To use 'ethdebug' with --debug-info you must select also 'ast-id'."

Standard JSON

==== Stdin ====
{
    "language": "Solidity",
    "sources": {"input.sol": {"urls": ["input.sol"]}},
    "settings": {
        "experimental": true,
        "debug": {"debugInfo": ["ethdebug"]},
        <PARAM>
    }
}
==== Source: input.sol ====
contract C { function f() public pure {} }
==== Params ====
"viaIR": false, "outputSelection": {"*": {"*": ["evm.legacyAssembly"]}}
"viaIR": true,  "outputSelection": {"*": {"*": ["evm.legacyAssembly", "evm.bytecode.ethdebug"]}}
"viaIR": true,  "outputSelection": {"*": {"*": ["evm.legacyAssembly", "evm.deployedBytecode.ethdebug"]}}
"viaIR": false, "outputSelection": {"*": {"*": ["irAst"]}}
"viaIR": true,  "outputSelection": {"*": {"*": ["irAst", "evm.bytecode.ethdebug"]}}
"viaIR": false, "outputSelection": {"*": {"*": ["irOptimizedAst"]}}
"viaIR": true,  "outputSelection": {"*": {"*": ["irOptimizedAst", "evm.bytecode.ethdebug"]}}
// ====
// standardJSON: true
// ----
// .exit: 0
// .errors: [{
//     "component": "general",
//     "formattedMessage": "To use 'ethdebug' with settings.debug.debugInfo you must select also 'ast-id'.",
//     "message": "To use 'ethdebug' with settings.debug.debugInfo you must select also 'ast-id'.",
//     "severity": "error",
//     "type": "JSONError"
// }]
// .errors[0].message | contains "ast-id": true
// .contracts: <PATH NOT FOUND>

Parameters everywhere

==== Stdin ====
{
    "language": "Solidity",
    "sources": {"input.sol": {"urls": ["input.sol"]}},
    "settings": {
        "experimental": true,
        "debug": {"debugInfo": ["ast-id", "ethdebug"]},
        "viaIR": false,
        "outputSelection": {"*": {"*": ["evm.<PARAM>.ethdebug", "ir", "irOptimized"]}}
    }
}
==== Source: input.sol ====
contract C { function f() public pure returns (string memory) { return "<PARAM>"; } }
==== Params ====
bytecode
deployedBytecode
// ====
// standardJSON: true
// ----
// (input.sol:C evm.<PARAM>.ethdebug) instructions | empty: false
// (input.sol:C ir)          | contains "/// ethdebug: enabled": true
// (input.sol:C irOptimized) | contains "/// ethdebug: enabled": true

Raw Standard JSON

==== Commands ====
solc --standard-json input.json
==== input.json ====
{"language": "Solidity"}
==== Params ====
// ----
// .stdout | contains "No input sources specified": true

Settings

==== Commands ====
solc input.sol --combined-json hashes
==== Source: input.sol ====
contract C {}
// ====
// standardJSON: false
// implicitOptions:
// stripPrereleaseWarning: false
// stripLicenseWarning: false
// stripVersionWarning: false
// ----
// .stdout: "{\"contracts\":{\"input.sol:C\":{\"hashes\":{}}},\"version\":\"0.8.37\"}"
// .stderr | grep "Warning": [
//     "Warning: This is a pre-release compiler version, please do not use it in production.",
//     "Warning: SPDX license identifier not provided in source file. Before publishing, consider adding a comment containing \"SPDX-License-Identifier: <SPDX-License>\" to each source file. Use \"SPDX-License-Identifier: UNLICENSED\" for non-open-source code. Please see https://spdx.org for more information.",
//     "Warning: Source file does not specify required compiler version!"
// ]

[djolertrk]

More context here: #16961 (comment)

[cameel]

I edited a detailed spec for this into the description to show what exactly I meant. This is an idea on how to build a more manageable mechanism for testing the input and output on top of things we already have: isoltest, multi-file tests, JSONExpectationTest (#16502/#16675). The description is detail-heavy, but at its core it's still fairly minimal due to being able to reuse these.

I think this could actually be good enough to replace not just these tedious boost tests for allowed/disallowed options but also general command-line tests in a lot of cases. The main exceptions are the ones testing CLI parsing (#17024) and path resolution (these require interaction with the actual filesystem).

I'd be curious about @nikola-matic's and @erak's thoughts on this.