How to ignore missing (non-existing) files/tests

[sashko1988]

I have an external system to store test suites. If simplified, it is akin to a testlist file — test nodes are resolved and passed into session.config.args before pytest_collect is executed.

Sometimes, testlists get out of sync with repo content - i.e. files deleted, test classes/test functions removed from modules, but they remain in the testlists.

Pytest raises an error whenever a file, function, or class is not found.

1. --continue-on-collection-errors doesn't help here. The error is still raised, and pytest doesn't execute resolved tests.
2. If the suite contains an non-existing file path, pytest starts pytest_collection and just errors out. try...except inside the hook doesn't even get into the except block. pytest_exception_interact also isn't called.
3. If a file exists, but the test function doesn't, pytest fails before reaching pytest_pycollect_makeitem.

I have an idea for a workaround: check if the file path and class/function exist before starting pytest_collect. But it doesn't feel right because it essentially duplicates what pytest does during the collection phase.

What do you think is the best approach to ignore missing tests?

Reproduce:

Repo with a simple reproduction https://github.com/sashko1988/pytest-how-to-ignore-missing

Example:

Folder structure:

├── README.md
├── test-list-with-missing-file.txt
├── test-list-with-missing-test-function.txt
├── tests
│   ├── __pycache__
│   │   └── test_example.cpython-312-pytest-8.3.4.pyc
│   └── test_example.py
└── tox.ini

tests/test_example.py content

def test_one():
    assert 1 == 1

test-list-with-missing-file.txt content

tests/test_example.py::test_one
tests/test_non_existing.py

Output

❯ tox -e missing-file
missing-file: commands[0]> pytest @test-list-with-missing-file.txt --continue-on-collection-errors
======================= test session starts =======================
platform darwin -- Python 3.12.8, pytest-8.3.4, pluggy-1.5.0
cachedir: .tox/missing-file/.pytest_cache
rootdir: ~/work_stuff/pytest-how-to-ignore-missing-tests
collected 0 items

====================== no tests ran in 0.00s ======================
ERROR: file or directory not found: tests/test_non_existing.py

missing-file: exit 4 (0.13 seconds) ~/work_stuff/pytest-how-to-ignore-missing-tests> pytest @test-list-with-missing-file.txt --continue-on-collection-errors pid=85451
  missing-test-function: FAIL code 4 (0.16=setup[0.03]+cmd[0.14] seconds)
  missing-file: FAIL code 4 (0.13=setup[0.00]+cmd[0.13] seconds)
  evaluation failed :( (0.34 seconds)- [pytest-how-to-ignore-missing-tests](#pytest-how-to-ignore-missing-tests)

[webknjaz]

It'd be nice to have some example repository as a demo, simulating this situation.. I don't understand how you invoke pytest, for example. Can this be reproduced with something like --lf?

[sashko1988]

Updated initial post with more details.
And here's the repo with the needed content: https://github.com/sashko1988/pytest-how-to-ignore-missing

[RonnyPfannschmidt]

This one is tricky

We must fail the cli for missing tests that where explicitly passed

Having a soft version of that ought to be a plugin that would warn/xfail gor missing tests instead of failing instantly

[sashko1988]

Having a soft version out of the box would be great.

1. What's the purpose of --continue-on-collection-errors? It's only used pytest_runtestloop and can't affect collection.

2. Built-in implementation, for example - update the behavior of --continue-on-collection-errors or create a new flag --ignore-missing, and use it in Session.perform_collect() to check if pytest should throw the error or continue and add a warning. (I'm happy to make that change if we decide to add it here)

3. If you were creating a custom plugin, how would you do it? I considered implementing the custom Session object with an overridden perform_collect(), but it can quickly become out of sync with the main pytest.Session.perform_collect(). (that is why having it as part of the core functionality is much easier imo)

[RonnyPfannschmidt]

Currently it's not a common use case and key parts of the behavior are unclear

This is to be explored in a plugin

[sashko1988]

key parts of the behavior are unclear

What parts are unclear or missing? Let's discuss.

[sashko1988]

@RonnyPfannschmidt, Why doesn't --continue-on-collection-errors help in my case? I would expect that it should skip any collection errors (non-existing tests also), which it doesn't do.

  --continue-on-collection-errors
                        Force test execution even if collection errors occur```

[RonnyPfannschmidt]

Errors in the test file are collection errors

Asking to run a test that's gone is a usage error

[sashko1988]

Then why not have a flag that warns the user that some of the supplied tests are missing instead of raising an error?

[sashko1988]

@RonnyPfannschmidt, Regarding plugin route

What I tried, and it doesn't work as expected:

This is the part of my conftest.py:

@pytest.hookimpl(wrapper=True)
def pytest_collection(session):
    # clear session args if they only contain the current working directory,
    # so pytest won't try to traverse it
    clear_session_args(session)

    # resolve tests from files
    resolve_tests_from_files(session)

    # perform collection from session args if they are not empty
    return (yield)

Dumb one:

1. Make a list of valid nodeids by running session.perform_collect([tc.local_nodeid], genitems=False) per resolved entry to filter invalid cases.
2. After making a list of valid nodeids - add them into session.config.args and perform the main collection by return (yield) in the hook. Works sequentially, doesn't work with xdist.
   If running with two nodes, the controller node resolves 0 tests, while the session of both worker nodes (gw0/gw1) resolves all the tests.

Bulk approach:

1. Create a list of valid paths - valid_paths.
2. then generate items (generated_items) from valid_paths with session.perform_collect(list_of_valid_paths)
3. filter them later to save only the tests that were specified in files, stash them in session stash - valid_tests, and all the tests that should be delesected save in tests_to_deselect.
4. when I get to main pytest_modifyitems - run pytest_deselect(tests_to_deselect), and items[:] = items + valid_tests
5. This one is faster but still only works sequentially. When I use xdist, the random controller always errors out on the resolved test number.

I didn't check the behavior in xdist code, but my understanding is the following:

• session.perform_collect triggered from resolve_tests_from_files() - goes all the way to the pytest_collection_finish hook
• Whenever the master node gets to this point, it tries to compare tests collected across workers. And errors out if it finds any difference

Questions:

1. Is there any method I can use instead of triggering session.perform_collect() to validate nodeids?
2. Maybe you know the better approach to this problem cause mine doesn't feel right.

[RonnyPfannschmidt]

I haven't been able to experiment with this

I'm open to making this easier for external plugins

[sashko1988]

I did a filthy thing, but it works.

1. Filter out all non-existing file paths in the resolve_tests_from_files
2. in the pytest_make_collect_report hook, if the collector is Session and its _notfound isn't empty - manually set _notfound to an empty list.
3. report all invalid items.

[RonnyPfannschmidt]

That's a good start

Now we have the horror version and can use that to figure making it something pretty

[webknjaz]

@sashko1988 would you put that into the demo repo too?

[sashko1988]

@webknjaz, give me a couple of days; I will put that in the repo and post an update here!