Django Test Runner Not Finding Test Methods

I recently upgraded from Django 1.4 to 1.9 and realized something weird was going on with my tests. Here is the project structure:

project
  manage.py
  app/
    __init__.py
    tests/
      __init__.py
      test_MyTests.py

The test_MyTests.py file looks like this:

from django.test import TestCase

class MyTests(TestCase):
    def test_math(self):
        self.assertEqual(2, 2)

    def test_math_again(self):
        self.assertEqual(3, 3)

The test runner can find all of the tests when I run ./manage.py test app or ./manage.py test app.tests. However when I try running ./manage.py test app.tests.MyTests I get:

  File "/usr/lib/python2.7/unittest/loader.py", line 100, in    loadTestsFromName
parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'MyTests'

If I change the test class name to test_MyTests I can then run ./manage.py test app.tests.test_Mytests and it will find all tests. I was reading the Django docs though and it seems the file name and class name don't have to be the same. In either case that I showed above I still can't run individual tests like this, ./manage.py test app.tests.MyTests.test_math

I would like to be able to run individual tests and test classes, can someone help me here? Thanks.