vendredi 30 janvier 2015

Experiments as unit test?

I've a bunch of python scripts and functions that I need to run for several experiments, with varying input data and output-to-file dumping. For this post an experiment is a triple consisting of input data, a (one or several) method call on the input data and a dump_results procedure.


At the moment I'm abusing unittest to get things done and bundle several experiments as test suites. For instance my file could look like this:



def experiment1():
pass
def experiment2():
pass
...

if __name__ == '__main__':
unittest.TextTestRunner().run(my_experiments_bundle_1)


where my_experiments_bundle_1 is a TestSuite containing all my relevant tests, for instance experiment1 and experiment2.


However, it's a slight abuse of the unittest paradigm as I'm actually not testing the correctness of implemented functionality but rather how the implementation acts on the given input data.


I've found experimentator and expsuite but...


a) I would like to use as few external packages as possible, for easy reproducibility and sharing of the code.


b) it seems to be a bit overkill for my purpose.


Nevertheless, a simple way to store parameters in an xml-like config would be a nice-to-have feature, though optional.


Q: What is a recommended/established and simple way or framework to bundle and manage such experiments? Are there any best practices?


Q: What are possible pitfalls abusing unittest for this purpose?


Thanks


PS: Slightly related is [question 1322414] and [question 14740373], but the former is more about organizing code tests in Java and the latter deals with design of experiments, using pyDOE, which looks to me as aiming particularly at statistics people. (sry, I'm not allowed to post more than two links)


Aucun commentaire:

Enregistrer un commentaire