How to unit test a builder which uses a static singleton factory in its builder() method?

I am trying to unit test a Selenium page object API, and I have a bunch of builder classes that use a static singleton factory class to instantiate the desired object. I can't for the life of me figure out a way to unit test the build() method in the builder class. How do I mock 'object creation' for my builder?

public class LoadableBuilder<LoadableT extends ILoadable, BeanT extends ILoadableBean>
    implements ILoadableBuilder<LoadableT, BeanT> {

    /**
     * The ILoadableBean object that specifies all the necessary information to construct an instance of the
     * ILoadable that this LoadableBuilder builds
     */
    private final @Getter @Nonnull BeanT state;

    /**
     * The class that of the ILoadable that this LoadableBuilder builds
     */
    private @Getter @Nullable Class<LoadableT> componentClass;

    public AbstractLoadableBuilder(final BeanT state) {
        this.state = state;
    }

    @Override
    public final @Nonnull LoadableBuilder setComponentClass(final Class<LoadableT> componentClass) {
        this.componentClass = componentClass;
        return this;
    }

    @Override
    public final @Nonnull LoadableBuilder setDriver(final WebDriver driver) {
        getState().setDriver(driver);
        return this;
    }

    @Override
    public final @Nonnull LoadableBuilder setLoadTimeoutInSeconds(final @Nonnegative int loadTimeoutInSeconds) {
        getState().setLoadTimeoutInSeconds(loadTimeoutInSeconds);
        return this;
    }

    @Override
    public @Nonnull LoadableT build() {
        return LoadableFactory.getInstance().create(getState(), componentClass);

    }
}

Here is the factory:

public class LoadableFactory {
    private static final class Loader {
        private static final LoadableFactory INSTANCE = new LoadableFactory();
    }

    private LoadableFactory() { }

    public static LoadableFactory getInstance() {
        return Loader.INSTANCE;
    }

    public final<BeanT extends ILoadableBean, LoadableT extends ILoadable> LoadableT create(final BeanT bean, final Class<LoadableT> componentClass) {
        final LoadableT component;

        try {
            final Constructor<LoadableT> ctor = ConstructorUtils.getMatchingAccessibleConstructor(componentClass, bean.getClass());
            component = ctor.newInstance(bean);
        } catch (InstantiationException e) {                
            throw new IllegalArgumentException("Could not instantiate an instance of " + componentClass + because it is abstract or an interface or for some other reason.");
        } catch (InvocationTargetException e) {
            throw new IllegalStateException("Could not instantiate an instance of " + componentClass + " because the constructor threw an exception. Cause: " + e.getCause() +". " + e.getMessage());
        } catch (IllegalAccessException e) {
            throw log.throwing(new IllegalArgumentException("Could not instantiate an instance of " + componentClass + " LoadableFactory does not have access to its class definition");
        }
    }
}