Test a Django site displays the correct graph - image similarity?

I'm making a simple Django site that graphs a specified data set with matplotlib and displays the resulting graph. I've saved a static copy of the correct graph, and I want to compare this against the image displayed on the page.

home.html

<img id="id_graph_image" src="/graphs/selected_graph.png">

graphs/urls.py

urlpatterns = patterns('',
        # other patterns...
        url(r'^graphs/selected_graph.png$', 'graphs.views.show_selected_graph'),
)

graphs/views.py

def show_selected_graph(request):
    # matplotlib code...
    canvas = FigureCanvasAgg(figure)
    response = HttpResponse(content_type='image/png')
    canvas.print_png(response)
    return response

graphs/tests.py

def test_selected_graph_shows_the_right_graphs(self):
    # request and response...
    graph_on_page = Image.open(StringIO(response.content)).
    expected_graph = Image.open('file://../graphs/static/scenario_1_statistic_1.png')
    # TODO: compare graphs

I'd like to add a Selenium test or a unit test to confirm that the graph view returns the correct image for a given data set. I've tried comparing the two images with PIL and the RMS Difference of the histograms, but any two matplotlib graph images have similar histograms.

Is a "percentage difference" simple? Should I test this in a very different way?

I appreciate any and all help, thanks!