Visual Studio Unit Tests running slower on TFS Build

My project has 1000+ unit tests that, in a local machine, all run in less than 10 seconds. But when they run on TFS Build, some tests run significantly slower than others. 3 of them run in about 1-2 minutes, other 4 in 5-30 seconds, and the others in fractions of a second. I've noticed that all those slower tests use fakes, and each one of them is the first to run in it's class. But a lot of the other tests also use fakes (some more intensively) and run in regular time. I would like to know what may be causing that slowdown and how can I fix it.