How to access HttpContext inside a unit test in ASP.NET 5 / MVC 6

Lets say I am setting a value on the http context in my middleware. For example HttpContext.User.

How can test the http context in my unit test. Here is an example of what I am trying to do

Middleware

public class MyAuthMiddleware
{
    private readonly RequestDelegate _next;

    public MyAuthMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext context)
    {
        context.User = SetUser(); 
        await next(context);
    }
}

Test

[Fact]
public async Task UserShouldBeAuthenticated()
{
    var server = TestServer.Create((app) => 
    {
        app.UseMiddleware<MyAuthMiddleware>();
    });

    using(server)
    {
        var response = await server.CreateClient().GetAsync("/");
        // After calling the middleware I want to assert that 
        // the user in the HttpContext was set correctly
        // but how can I access the HttpContext here?
    }
}