angular/jasmine: Error: No deferred tasks to be flushed

I'm unit testing one of my directives and I get the following error

Error: No deferred tasks to be flushed

The code I need to test is inside a $timeout

$timeout(function () {
    $window.doStuff();
});

So when I want to check if doStuff was called I do

$timeout.flush();
expect($window.doStuff).toHaveBeenCalled();

However, now I need to test a situation in which I don't expect doStuff to have been called. So I did:

$timeout.flush();
expect($window.doStuff).not.toHaveBeenCalled(); // Notice the not here!!

However, because not $timeout was called, the $timeout.flush() fives me this error.

So the question is, what is the preferred way to test this situation ?