Create a generic class factory to test dynamic class creation/destruction in GoogleTest?

I ran into a bit of a conundrum when I was toying around with Google Test.

I want to unit test a C++ class, that based on events from an external system, will spawn new classes to talk to entities that are coming and going in the system. The event dispatcher and how to create classes for specialized entities is all decided by libraries I cannot control nor change. The design of the library requires me to pass certain parameters to the class constructor on how to connect to the entity, and if that entity does not exist, the constructor will throw. Since these classes will be used depending on their existence and events I want to be able to EXPECT_CALL() on their methods. I obviously need a factory for this that I can Mock or otherwise overload and return Mock classes instead of the real ones.

I already abstracted the event monitor into an IClass so that I can create a Mock of it and feed events in a controlled manner. I created a factory that has series of Construct[x] methods, where each method returns an std::shared_ptr for a new instance of a specific IClass. This works fine, and I can run a complete test suite on the class I want to test in this way. Including proper class destruction.

However, I would like my factory to be more generic, and the best I can come up with is this:

class ClassFactory {
public:
    template<class U, class... Args>
    std::shared_ptr<U> Construct(Args&&... args) {
        return std::make_shared<U>(std::forward<Args>((args)...));
    }
};

The class I want to test though, which would be using the class factory is already compiled into an object file at the time of the test class compilation, so the above templated factory obviously wont work. The template has already been evaluated and no form of template specialization in the test class is going to re-route the Construct calls from the class I want to test.

So my question is: Is it at all possible to create a generic factory for this? And if it is, how would I do it?

Thanks in advance for the help!