mardi 6 septembre 2016

Unit Test a Spark Streaming App in scala

I'm working on unit tests for a spark streaming app which gets its data stream from Kafka in Avro format, I intend to test a method within an object:

object OBJ extends logging
{
    def transformationLogic(inputDstream: InputDStream[(String, Object)]): Unit =
    {
        inputDstream.foreachRDD { rdd =>
        val schemaStr = new CachedSchemaRegistryClient("schost").getLatestSchemaMetadata("subject").getSchema
        // more logic after this
    }
}

In my tests, I'm able to generate some Avro messages and pack them into a dstream. Although in order to fully test transformationLogic, I need a way to mock/pass Avro schema for the function to deserialize Avro messages and in unit test I'd like to avoid making calls to schema registry.

I tried scalamock and looked at mockito but no luck yet, any ideas in terms of how to unit test applications like this?

Thanks in advance.

Aucun commentaire:

Enregistrer un commentaire