lundi 28 septembre 2015

Spark (scala) Unit test - Mocking an object member

I have a spark application that has involves 2 scala companion objects as follows.

object actualWorker {
  daoClient

  def update (data, sc) {
    groupedData = sc.getRdd(data).filter. <several_operations>.groupByKey
    groupedData.foreach(x => daoClient.load(x))
  }
}


object SparkDriver {
  getArgs
  sc = getSparkContext
  actualWorker.update(data, sc : sparkContext)

}

The challenge I have is in writing unit-test for this spark application. I am using Mockito and ScalaTest, Junit for these tests.

I am not able to mock the daoClient while writing the unit test.

I have tried the following:

  1. Make ActualWorker a class that can have a uploadClient passed in the Constructor. Create a client and instantiate it in Actual Worker Problem: Task not serializable exception.
  2. Introduce a trait for upload client. But still I need to instantiate a client at some point in the SparkDriver, which I fear will cause the Task Not serializable exception.

Any inputs here will be appreciated.

PS: I am fairly new to Scala and spark

Aucun commentaire:

Enregistrer un commentaire