mardi 3 mai 2016

testing spark jobs with mocks

I have spark job like:

stream.
..do some stuff..
map(HbaseEventProcesser.process)

for now HbaseEventProcesser is an scala Object (singleton). So, there is no problems with serialization.

The problem is in testing that spark job (holdenkarau spark-test lib is used). I want to mock HbaseEventProcessor with some other implementation. I've tried two approaches

  • pass implenetation to spark job (as a constructor argument and than invoke methods inside map). That cause problem with serialization issue
  • use powerMock. Unfortunately, deepcopy operation failed if SharedSparkContext is used.

Is there any other workarounds ?

Aucun commentaire:

Enregistrer un commentaire