A Modern Data Infrastructure for Internet of Things

I just wrapped up this webinar.  Thanks for all for attending!

Here are the slides & recording of the webinar.

recording : 

And here are the slides:

Questions During The Webinar

Q1 : Difference between Spark Streaming and Storm

Storm by default is ‘event based processing’ (one-event-at-a-time).  Using Trident on top of Storm, we can do micro-batch processing.

Spark processes events in ‘micro batches’.  For example I can define the ‘batch’ interval to be 5 seconds.  Spark will process what ever number of events captured in that batch (could be none, one, ten or thousand!).   Currently the lowest batch time is about half-a-second (500 ms)

This presentation will give you more details.

Q2 : Does Spark Streaming work with HDFS?

Usually it is a good practice to pair Spark Streaming with HDFS — Spark uses HDFS for checkpointing (to save streaming status periodically)