ClassCleanup method not being called in Visual Studio Unit Test

I have a simple unit test class in Visual Studio, in which a class is attributed with ClassCleanup.

But this method only gets called if I debug the test and is not called when I simply run the test.

Here is how my simple test class looks

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;

namespace UnitTests
{
    [TestClass]
    public class ExampleTests
    {
        private static readonly Guid UniqueDbId = Guid.NewGuid();

        [ClassInitialize]
        public static void Init(TestContext testContext)
        {
            DbSetup.Init(UniqueDbId);
        }

        [ClassCleanup]
        public static void Cleanup()
        {
            DbSetup.Cleanup(UniqueDbId);
        }

        [TestMethod]
        public void DummyTest()
        {
            Assert.IsTrue(true);
        }
    }
}

To prove my point, the method attributed as ClassCleanup calls this method

public static void Cleanup(Guid uniqueDbId)
{
    throw new Exception("This happened");
}

The exception is only thrown when in debug mode.

I've verified this several other ways, such as writing to a log file. I noticed this because the ClassInitialize method creates a database for the tests and the ClassCleanup method should then delete this database but again, it only gets deleted if I run in debug mode.

I believe I've followed the ClassCleanup example on MSDN.

Can anyone explain why the ClassCleanup method is not running when I simply run the tests, i.e., not debugging?