Cleaning up after all JUnit tests without explicit test suite classes declaration

In Intelij and Eclipse IDEs (and probably some others too) it's possible to run all test classes from a package (or even all test classes in a project) without the need to put each of them explicitly in a test suite class (this is something I want to avoid). Just right click -> run all tests and voilà!

I've got one problem with that approach to testing though. I want to do some cleaning up after all the tests are done, but no matter what I do, nothing seems to work.

At first, I tried using RunListener and its testRunFinished() method, but it is called after every atomic test is done, so not what I want when running many of them.

Then I thought about finalizers and runFinalizersOnExit(true), unfortunatelly, it is deprecated and worked only on one of computers that tests are executed on.

Last thing I tried was to create a "listener" thread, that - given tests execution start and end time differences - would clean up, for instance, after five seconds of test completion. I used code below to test that solution:

import org.junit.Test;

public class Main {
    static {
        System.out.println("In a static block!");

        new Thread(new Runnable() {
            public void run() {
                System.out.println("Starting static thread!");

                try {
                    while (true) {
                        Thread.sleep(1000);
                        System.out.println("Static thread working...");
                    }
                } catch (InterruptedException e) {
                    System.err.println("Static thread interrupted!");
                    e.printStackTrace();
                } catch (Exception e) {
                    System.err.println("Static thread catches exception!");
                    e.printStackTrace();
                } finally {
                    System.err.println("Static thread in finally method.");
                    Thread.currentThread().interrupt();
                }

            }
        }).start();

        System.out.println("Exiting static block!");
    }

    @Test
    public void test() throws Exception {
        System.out.println("Running test!");
        Thread.sleep(3000);
        System.out.println("Stopping test!");
    }
}

With no luck. The thread is killed after the test is done. And even the finally block is never executed...

In a static block!
Exiting static block!
Running test!
Starting static thread!
Static thread working...
Static thread working...
Stopping test!
Static thread working...

Desired behavior would be:

1. right click
2. run all tests
3. TestA is running...
4. TestA done
5. TestB is running...
6. TestB done
7. ... more test classes...
8. cleanup