What streaming technology select for discrete signals processing?

by Serbin   Last Updated May 15, 2019 17:05 PM - source

I will start with an example of the problem. We have 4 different data sources that produce 3 discrete signals input_1, input_2 and input_3, and one analog signal input_4.

enter image description here

I'm looking for the best technology that would provide a flexible and easy (for end user) way to transform that signals to the new ones. On the picture, new signals are out_1 and out_2. Pseudo code for them:

out_1 = input_1 && input_2 && !input_3;
out_2 = out_1 && input_4 > 20;

Number of signals might be quite big. Data sources and sink might be distributed. You might think about this as distributed IoT system, where we should program a logic, like "if TV and lamp is working, and temperature is bigger than 20, turn air condition on". Every signals might come every second/minute/millisecond.

The technologies I'm looking at right now:

  • Kafka as a broker
  • RabbitMQ as a broker
  • Spark
  • Spark Streaming
  • Flink
  • Storm
  • Kafka Streaming
  • Akka
  • Akka Streaming
  • Reactive Streams
  • maybe some more

I'm quite new in this field, and it's difficult to immediately understand all this zoo and choose the right tool.

I already have some kind of a self made solution for this problem, that behaves via actors models (like Akka but not Akka) via RabbitMQ. But it is not so scalable as I want. Probably, I can make a solution based on any of this technologies, but some of this solutions will be just duct tape.

There might be an additional requirements for the system, like: data recalculation for new signal/signal update, late data arrive, data lost for some periods for some signals, etc. But, I hope, this problems might be solved in application programming.

I would be grateful for any reasonable direction in which I need to dig further.



Related Questions


Architecture and framework for incremental pipeline

Updated October 24, 2017 19:05 PM


Architecture for Real Time data processing

Updated November 28, 2016 08:02 AM

how simulate sequential consumers in Kafka

Updated November 24, 2018 08:05 AM

Logs for real-time data unifying

Updated April 06, 2017 09:05 AM