Could not load file or assembly 'WebDriver' when running a build via TeamCity

I have Selenium unit testing working fine through Visual Studio using the Selenium Web Driver. I also have a CI build set up in TeamCity which is loading the project with my unit tests in it. I'm using Visual Studio Test, VSTest and VS Test 2015 as the config and I'm looking for all assemblies that are tests using the ***Tests.dll value for the test file names parameter.

When I run the build through TeamCity I get under the tests area

System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not load file or assembly 'WebDriver, Version=2.53.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.WRN: Assembly binding logging is turned OFF.

The WebDriver is referenced as a nuget package and I have package restoration as my first step in the build which is working as the build itself compiles.

I've also tried adding that dll directly as a reference in the test file names as well but that didn't seem to make much difference.