Overview of Apache Storm

Apache Storm is a free and open source distributed real time computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!
Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.


Apache Storm is a free and open source distributed system for real-time computation. Storm is implemented in Clojure & Storm APIs are written in Java by Nathan Marz & team at BackType. Storm is best for distributed processing where unbounded streams of  real time data computation requires. In contrast to batch systems like Hadoop, which often introduce delay of hours, Storm allows us to process online data.Storm is extremely fast, with the ability to process over a million records per second per node on a cluster of modest size

A Storm application is designed as a topology in the shape of DAG(Directed ACyclic Graph) with Spouts and Bolts acting as graph vertices. Edges on the graph are data stream between nodes.

Characteristics of Storm  Fast – benchmarked as processing one million 100 byte messages per second per node  Scalable – with parallel calculations that run across a cluster of machines  Fault-tolerant – when workers die, Storm will automatically restart them. If a node dies, the worker will be restarted on another node.  Reliable – Storm guarantees that each unit of data (tuple) will be processed at least once or exactly once. Messages are only replayed when there are failures.  Easy to operate – standard configurations are suitable for production on day one. Once deployed, Storm is easy to operate. Use cases of Storm         Processing Streams (No need of intermediate Queues)         Continuous Computation         Distributed RPC Spout A Spout is a source of streams in a computation. Spout can read from queue like Kafka , can generate its own stream or read Twitter streaming. Bolt A bolt processes any number of input streams and produces any number of new output streams. Most of the logic of a computation goes into bolts, such as functions, filters, streaming joins, streaming aggregations, talking to databases, and so on. Topology A topology is a network of spouts and bolts, with each edge in the network representing a bolt subscribing to the output stream of some other spout or bolt. A topology is an arbitrarily complex multi-stage stream computation. Topologies run indefinitely when deployed. A topology can have one spout and multiple bolt A topology is arrangement of spout and bolt Understanding Storm Architecture Storm cluster has 3 sets of nodes 1 Nimbus Node 2.Zookeeper Nodes 3.Supervisor Nodes

Nimbus Node
Uploads computations for execution
distributes code across the cluster
launches workers across the cluster
monitors computations and reallocates workers as needed

Zookeeper nodes
coordinates the storm cluster
Zookeeper is a distributed coordination service going to installed on each and every machine
Job tracker


Supervisor nodes
communicates with nimbus through zookeeper, starts and stops workers according to signals from nimbus.
Supervisor actual execution happens at supervisor and only start and stop signals one set by nimbus
Task Tracker


 references: https://storm.apache.org/