I have spark job like:
stream.
..do some stuff..
map(HbaseEventProcesser.process)
for now HbaseEventProcesser is an scala Object (singleton). So, there is no problems with serialization.
The problem is in testing that spark job (holdenkarau spark-test lib is used). I want to mock HbaseEventProcessor with some other implementation. I've tried two approaches
- pass implenetation to spark job (as a constructor argument and than invoke methods inside map). That cause problem with serialization issue
- use powerMock. Unfortunately, deepcopy operation failed if SharedSparkContext is used.
Is there any other workarounds ?
Aucun commentaire:
Enregistrer un commentaire