Add filter to only run integration tests.

[jschwe]

Describe the problem you are trying to solve

Currently cargo test --tests runs both unit and integration tests. Using cargo test --lib runs only the unit-tests of the library, however there is no way to only let the integration tests run, apart from maybe using a naming scheme for the integration tests, which I would like to avoid. It's also possible to write a script which calls cargo test --test <name> for each .rs file in the tests folder, but again, this is something I would like to avoid.

Describe the solution you'd like

I would like to run only the integration tests with e.g. cargo test --integration_tests.

Notes

Looking into libtest the enum TestType seems to already provide the necessary information.

[lvkh1]

I'd like to take this, if no one else has started work on it. I read in the docs that integration tests are generally put in tests, would it be OK to add a flag -i, --integration that just runs the tests in that directory?

[ehuss]

I'm curious if either of you could state your motivation for this? From my perspective, the unit tests are just another testable thing, so I'm uncertain why you would want to specifically exclude them.

As for a design/implementation perspective, I was wondering if maybe it would be better to add globbing to the target names instead of adding new flags? Perhaps something like --test '*' or --test client*?

[jschwe]

The motivation in my case is kind of a "We probably shouldn't be doing it like this" situation. Historically the project I'm working on has unit-tests that are meant to be executed on the host system. I'm now working on integration tests and those target our target platform, so compiling both at the same time doesn't really work.

Of course in the long run our unit-tests should also be able to run on our target platform, but that is currently not the case. TestDesc including information like should_panic is a libtest internal and thus not available on our target. As a quick-fix I'd like to be able to run unit-tests and integration tests separately, so that we can continue to use libtest features for the existing unit-tests until TestDesc is hopefully available outside of libtest one day.

Regarding globbing: I personally would like globbing to be available for --test, but in addition to a specific filter for integration tests, since I would like to avoid adding some prefix to every integration test file.

[ehuss]

I would like to avoid adding some prefix to every integration test file.

With --test '*' you wouldn't need a prefix. That would match all integration tests.

[jschwe]

Ah, okay. The only thing that might be a bit confusing is that --test '*' would have different behavior compared to --tests, which is not that intuitive.

[pronvis]

I need that feature too - in Gitlab yaml I have 2 different stages: tests & integration tests. To avoid running integration tests twice I need flag.
Yep, now I will use --tests '*', but it is crypto a bit :)

[pronvis]

Interesting... This crunch dont work for me because I customize $CARGO_TARGET_DIR (to avoid compiling same artifacts for different projects) and now cargo test --tests looking for tests in directory which is full of builded articats, but no tests.
cargo test --test '*' do not work too.

This is how it looks like:

> cargo test --test '*'
error: no test target named `*`

> cargo test --tests get_video_events
    Finished test [unoptimized + debuginfo] target(s) in 0.15s
     Running {my_home_dir}/rust_build_artifacts/debug/deps/{project_name}-9598706256c32c7d

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 4 filtered out

     Running {my_home_dir}/rust_build_artifacts/debug/deps/{test_file_1}-28b3cb4b0c341f41

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running {my_home_dir}/rust_build_artifacts/debug/deps/{test_file_2}-d0642ae53302a48c

running 0 tests

[jschwe]

@invis87 cargo test --test '*' doesn't work, since this hasn't been implemented yet (That's also why the issue is still open).

[ehuss]

I'm going to close this as fixed by #8752. It now adds the syntax --test '*' which will select all integration tests. It should land on nightly within a few days.

[Integralist]

👋🏻 probably a silly question but re:

With --test '*' you wouldn't need a prefix. That would match all integration tests.

Why does that only match integration tests? I expect the glob to match every test (unit and integration).

[jschwe]

@Integralist Because --test is documented as:

Test the specified integration test. This flag may be specified multiple times and supports common Unix glob patterns.

--tests on the other hand does also include unittests.

[iddm]

Perhaps, we can revisit the issue since the cargo test --test '*' isn't really the best way to do this, compared to the option of having a dedicated command-line option like cargo test --integration?