lundi 30 novembre 2015

Why does Visual Studio use a different output directory for a) normal unit tests and b) data driven unit tests?

At first we only had a couple of data constellations we used to regularly test. Those we tended in the testcode having a couple arrays of data and looping through those to run tests.

Now that we have huge DataSources (namely Excel-Files) that contain 100s of data rows that are to be run through our unit tests we used the VisualStudio.TestTools to configure according DataSources:

<configSections>
  <section name="microsoft.visualstudio.testtools" 
           type="Microsoft.VisualStudio.TestTools.UnitTesting.TestConfigurationSection, Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"/>
</configSections>

<connectionStrings>
  <add name="KradTestConnection" connectionString="Provider=Microsoft.ACE.OLEDB.12.0;Data Source=Beitragstests_KRAD_Tarif2016.xls;Extended Properties=Excel 12.0 Xml" providerName="System.Data.OleDb" />
  <add name="PkwTestConnection" connectionString="Provider=Microsoft.ACE.OLEDB.12.0;Data Source=Beitragstests_PKW_Tarif2016.xls;Extended Properties=Excel 12.0 Xml" providerName="System.Data.OleDb" />
</connectionStrings>

<microsoft.visualstudio.testtools>
  <dataSources>
    <add name="KradTestDataSource" connectionString="KradTestConnection" dataTableName="TestInput" dataAccessMethod="Sequential"/>
    <add name="PkwTestDataSource" connectionString="PkwTestConnection" dataTableName="TestInput" dataAccessMethod="Sequential"/>
  </dataSources>
</microsoft.visualstudio.testtools>

With that we just provided the data files 'Beitragstests_KRAD_Tarif2016.xls' and 'Beitragstests_PKW_Tarif2016.xls' and decorated our testcases with the DeploymentItemAttribute:

[DataSource("PkwTestDataSource")]
[DeploymentItem("Beitragstests_PKW_Tarif2016.xls")]
[TestCategory("DataDriven Berechnungstests")]
[TestMethod]
public void Pkw_HaftplichtMitVollkasko_TestCases() { ... }

So far and only testing those 'data driven unit tests' we had no problems. But our Test-Suite does contain many older testclasses we still require.

If we run data-driven and non-data-driven tests in a seperate test session, everything works out fine and all tests are 'green'.

As soon as we run "ALL" testcases though, many of the non-data-driven tests fail.

Analyzing that behaviour quickly made us aware that - be it mstest or running the tests via resharper (and yes, we disabled 'shadow-copying' in resharper-test-options) - as soon as the data-driven testcases are included in a test session, the tests wont be run in the projects output directory $OutDir that we defined for many projects and solutions (which each deliver a part of the dlls required for our testing suite). Running the data-driven testcases will cause Visual Studio to create a new directory 'TestResults/Deploy_ /Out' located next to the solution file '...sln' instead... in which of course the TestRunner will not find dependent DLLs that are delivered to our $OutDir.

Could anyone please explain to me, why the test engine creates different output directories based on the inclusion of 'data-driven unit tests' into a test session?`

Would anyone have a hint or even a solution about how to get Visual Studio to use the directory $OutDir that we configured in our projects instead of creating a new 'TestResults/Deploy_ /Out' every time?

Thanks in advance!

Aucun commentaire:

Enregistrer un commentaire