Detach ZF2 event listener during unit tests

I have an event listener set up in the module (it checks user permissions) in the onBootstrap method:

// attach permission listener to route event
$eventManager = $e->getApplication()->getEventManager();
$sm = $e->getApplication()->getServiceManager();
$permissionsListener = $sm->get('MyModule\Listener\Permissions');
$permissionsListener->attach($eventManager);

It works OK in the normal application.

For the unit tests, it would be great if I could disable this permission system to easily check that all pages are working correctly without worrying about setting up a fake admin user.

But I don't know how to do it.

In my test, which extends AbstractHttpControllerTestCase, I have done something like this:

// detach permission listener to route event
$sm = Bootstrap::getServiceManager();
$eventManager = $this->getApplication()->getEventManager();
$permissionsListener = $sm->get('MyModule\Listener\Permissions');
$permissionsListener->detach($eventManager);

But this doesn't disable the permission system. How can this be done?