samedi 6 février 2016

How to test objects or other single resources / disable parallel testing with uTest

http://ift.tt/1ly2gbC

As said in the documentation uTest runs tests parallel. Test cases inside one test suite can be run sequentially if used

import utest.ExecutionContext.RunNow

but sbt runs different test suites always parallel. http://ift.tt/1NVLO0g

This is usually good for performance but when testing objects or other "single resources" this can cause problems. If same object is tested or used in several test suites, these parallel tests may disturb each other.

Logging is a good example: Logging may use a commom factory object and in some cases this object can contain state like default logging level or data structure for log instances etc. Logging is used in several SW modules / in several test suites and if one of those test suites tests the logging itself then the other test suites may cause random false results in the logging test suite. Database is another example of "single resource".

Is there any way to limit the parallel running of uTest test suites so that the user can decide which test suites are always run sequentially and which test suites can be run parallel?

E.g: Run test suites: com.XXX.yyy.MyTestSuiteA, com.XXX.yyy.MyTestSuiteB and com.XXX.yyy.MyTestSuiteC sequentially (in this order) and the rest parallel (D-Z).

sbt "script" is not a good solution

sbt testOnly com.XXX.yyy.MyTestSuiteA
sbt testOnly com.XXX.yyy.MyTestSuiteB
sbt testOnly com.XXX.yyy.MyTestSuiteC
sbt testOnly com.XXX.yyy.MyTestSuiteD
sbt testOnly com.XXX.yyy.MyTestSuiteE
sbt testOnly com.XXX.yyy.MyTestSuiteF
...

because then nothing is run parallel and I must list all test suites. Something like this might be good:

sbt test -- sequentially com.XXX.yyy.MyTestSuiteA, com.XXX.yyy.MyTestSuiteB, com.XXX.yyy.MyTestSuiteC

meaning that the rest can be run in whatever order. Or

sbt test -- sequentially 

meaning that all test suites are run sequentially

Aucun commentaire:

Enregistrer un commentaire