-
Matei Zaharia authored
I used the sbt-unidoc plugin (https://github.com/sbt/sbt-unidoc) to create a unified Scaladoc of our public packages, and generate Javadocs as well. One limitation is that I haven't found an easy way to exclude packages in the Javadoc; there is a SBT task that identifies Java sources to run javadoc on, but it's been very difficult to modify it from outside to change what is set in the unidoc package. Some SBT-savvy people should help with this. The Javadoc site also lacks package-level descriptions and things like that, so we may want to look into that. We may decide not to post these right now if it's too limited compared to the Scala one. Example of the built doc site: http://people.csail.mit.edu/matei/spark-unified-docs/ Author: Matei Zaharia <matei@databricks.com> This patch had conflicts when merged, resolved by Committer: Patrick Wendell <pwendell@gmail.com> Closes #457 from mateiz/better-docs and squashes the following commits: a63d4a3 [Matei Zaharia] Skip Java/Scala API docs for Python package 5ea1f43 [Matei Zaharia] Fix links to Java classes in Java guide, fix some JS for scrolling to anchors on page load f05abc0 [Matei Zaharia] Don't include java.lang package names 995e992 [Matei Zaharia] Skip internal packages and class names with $ in JavaDoc a14a93c [Matei Zaharia] typo 76ce64d [Matei Zaharia] Add groups to Javadoc index page, and a first package-info.java ed6f994 [Matei Zaharia] Generate JavaDoc as well, add titles, update doc site to use unified docs acb993d [Matei Zaharia] Add Unidoc plugin for the projects we want Unidoced
Matei Zaharia authoredI used the sbt-unidoc plugin (https://github.com/sbt/sbt-unidoc) to create a unified Scaladoc of our public packages, and generate Javadocs as well. One limitation is that I haven't found an easy way to exclude packages in the Javadoc; there is a SBT task that identifies Java sources to run javadoc on, but it's been very difficult to modify it from outside to change what is set in the unidoc package. Some SBT-savvy people should help with this. The Javadoc site also lacks package-level descriptions and things like that, so we may want to look into that. We may decide not to post these right now if it's too limited compared to the Scala one. Example of the built doc site: http://people.csail.mit.edu/matei/spark-unified-docs/ Author: Matei Zaharia <matei@databricks.com> This patch had conflicts when merged, resolved by Committer: Patrick Wendell <pwendell@gmail.com> Closes #457 from mateiz/better-docs and squashes the following commits: a63d4a3 [Matei Zaharia] Skip Java/Scala API docs for Python package 5ea1f43 [Matei Zaharia] Fix links to Java classes in Java guide, fix some JS for scrolling to anchors on page load f05abc0 [Matei Zaharia] Don't include java.lang package names 995e992 [Matei Zaharia] Skip internal packages and class names with $ in JavaDoc a14a93c [Matei Zaharia] typo 76ce64d [Matei Zaharia] Add groups to Javadoc index page, and a first package-info.java ed6f994 [Matei Zaharia] Generate JavaDoc as well, add titles, update doc site to use unified docs acb993d [Matei Zaharia] Add Unidoc plugin for the projects we want Unidoced
layout: global
title: Spark Streaming Programming Guide
- This will become a table of contents (this text will be scraped). {:toc}
Overview
Spark Streaming is an extension of the core Spark API that allows enables high-throughput,
fault-tolerant stream processing of live data streams. Data can be ingested from many sources
like Kafka, Flume, Twitter, ZeroMQ or plain old TCP sockets and be processed using complex
algorithms expressed with high-level functions like map
, reduce
, join
and window
.
Finally, processed data can be pushed out to filesystems, databases,
and live dashboards. In fact, you can apply Spark's in-built
machine learning algorithms, and
graph processing algorithms on data streams.
Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.
Spark Streaming provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data. DStreams can be created either from input data stream from sources such as Kafka and Flume, or by applying high-level operations on other DStreams. Internally, a DStream is represented as a sequence of RDDs.
This guide shows you how to start writing Spark Streaming programs with DStreams. You can write Spark Streaming programs in Scala or Java, both of which are presented in this guide. You will find tabs throughout this guide that let you choose between Scala and Java code snippets.
A Quick Example
Before we go into the details of how to write your own Spark Streaming program, let's take a quick look at what a simple Spark Streaming program looks like. Let's say we want to count the number of words in text data received from a data server listening on a TCP socket. All you need to do is as follows.
StreamingContext is the main entry point for all streaming functionality.
{% highlight scala %} import org.apache.spark.streaming._ import org.apache.spark.streaming.StreamingContext._ {% endhighlight %}
Then we create a StreamingContext object. Besides Spark's configuration, we specify that any DStream will be processed in 1 second batches.
{% highlight scala %} // Create a StreamingContext with a SparkConf configuration val ssc = new StreamingContext(sparkConf, Seconds(1)) {% endhighlight %}
Using this context, we then create a new DStream by specifying the IP address and port of the data server.
{% highlight scala %} // Create a DStream that will connect to serverIP:serverPort val lines = ssc.socketTextStream(serverIP, serverPort) {% endhighlight %}
This lines
DStream represents the stream of data that will be received from the data
server. Each record in this DStream is a line of text. Next, we want to split the lines by
space into words.
{% highlight scala %} // Split each line into words val words = lines.flatMap(_.split(" ")) {% endhighlight %}
flatMap
is a one-to-many DStream operation that creates a new DStream by
generating multiple new records from each record in the source DStream. In this case,
each line will be split into multiple words and the stream of words is represented as the
words
DStream. Next, we want to count these words.
{% highlight scala %} // Count each word in each batch val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _)
// Print a few of the counts to the console wordCounts.print() {% endhighlight %}
The words
DStream is further mapped (one-to-one transformation) to a DStream of (word, 1)
pairs, which is then reduced to get the frequency of words in each batch of data.
Finally, wordCounts.print()
will print a few of the counts generated every second.
Note that when these lines are executed, Spark Streaming only sets up the computation it will perform when it is started, and no real processing has started yet. To start the processing after all the transformations have been setup, we finally call
{% highlight scala %} ssc.start() // Start the computation ssc.awaitTermination() // Wait for the computation to terminate {% endhighlight %}
The complete code can be found in the Spark Streaming example
NetworkWordCount.
First, we create a JavaStreamingContext object, which is the main entry point for all streaming functionality. Besides Spark's configuration, we specify that any DStream would be processed in 1 second batches.
{% highlight java %} // Create a StreamingContext with a SparkConf configuration JavaStreamingContext jssc = StreamingContext(sparkConf, new Duration(1000)) {% endhighlight %}
Using this context, we then create a new DStream by specifying the IP address and port of the data server.
{% highlight java %} // Create a DStream that will connect to serverIP:serverPort JavaDStream lines = jssc.socketTextStream(serverIP, serverPort); {% endhighlight %}
This lines
DStream represents the stream of data that will be received from the data
server. Each record in this stream is a line of text. Then, we want to split the the lines by
space into words.
{% highlight java %} // Split each line into words JavaDStream words = lines.flatMap( new FlatMapFunction<String, String>() { @Override public Iterable call(String x) { return Lists.newArrayList(x.split(" ")); } }); {% endhighlight %}
flatMap
is a DStream operation that creates a new DStream by
generating multiple new records from each record in the source DStream. In this case,
each line will be split into multiple words and the stream of words is represented as the
words
DStream. Note that we defined the transformation using a
FlatMapFunction object.
As we will discover along the way, there are a number of such convenience classes in the Java API
that help define DStream transformations.
Next, we want to count these words.
{% highlight java %} // Count each word in each batch JavaPairDStream<String, Integer> pairs = words.map( new PairFunction<String, String, Integer>() { @Override public Tuple2<String, Integer> call(String s) throws Exception { return new Tuple2<String, Integer>(s, 1); } }); JavaPairDStream<String, Integer> wordCounts = pairs.reduceByKey( new Function2<Integer, Integer, Integer>() { @Override public Integer call(Integer i1, Integer i2) throws Exception { return i1 + i2; } }); wordCounts.print(); // Print a few of the counts to the console {% endhighlight %}
The words
DStream is further mapped (one-to-one transformation) to a DStream of (word, 1)
pairs, using a PairFunction
object. Then, it is reduced to get the frequency of words in each batch of data,
using a Function2 object.
Finally, wordCounts.print()
will print a few of the counts generated every second.
Note that when these lines are executed, Spark Streaming only sets up the computation it will perform when it is started, and no real processing has started yet. To start the processing after all the transformations have been setup, we finally call
{% highlight java %} jssc.start(); // Start the computation jssc.awaitTermination(); // Wait for the computation to terminate {% endhighlight %}
The complete code can be found in the Spark Streaming example
JavaNetworkWordCount.
If you have already downloaded and built Spark, you can run this example as follows. You will first need to run Netcat (a small utility found in most Unix-like systems) as a data server by using
{% highlight bash %} $ nc -lk 9999 {% endhighlight %}
Then, in a different terminal, you can start the example by using
Then, any lines typed in the terminal running the netcat server will be counted and printed on screen every second. It will look something like this.
{% highlight bash %}
# TERMINAL 1:
# Running Netcat
$ nc -lk 9999 hello world ... {% endhighlight %} |
{% highlight bash %}
TERMINAL 2: RUNNING NetworkWordCount or JavaNetworkWordCount$ ./bin/run-example org.apache.spark.streaming.examples.NetworkWordCount local[2] localhost 9999 ...Time: 1357008430000 ms(hello,1) (world,1) ... {% endhighlight %} |
If you plan to run the Scala code for Spark Streaming-based use cases in the Spark shell, you should start the shell with the SparkConfiguration pre-configured to discard old batches periodically:
{% highlight bash %} $ SPARK_JAVA_OPTS=-Dspark.cleaner.ttl=10000 bin/spark-shell {% endhighlight %}
... and create your StreamingContext by wrapping the existing interactive shell
SparkContext object, sc
:
{% highlight scala %} val ssc = new StreamingContext(sc, Seconds(1)) {% endhighlight %}
When working with the shell, you may also need to send a ^D
to your netcat session
to force the pipeline to print the word counts to the console at the sink.
Basics
Next, we move beyond the simple example and elaborate on the basics of Spark Streaming that you need to know to write your streaming applications.
Linking
To write your own Spark Streaming program, you will have to add the following dependency to your SBT or Maven project:
groupId = org.apache.spark
artifactId = spark-streaming_{{site.SCALA_BINARY_VERSION}}
version = {{site.SPARK_VERSION}}
For ingesting data from sources like Kafka and Flume that are not present in the Spark
Streaming core
API, you will have to add the corresponding
artifact spark-streaming-xyz_{{site.SCALA_BINARY_VERSION}}
to the dependencies. For example,
some of the common ones are as follows.
Source | Artifact |
---|---|
Kafka | spark-streaming-kafka_{{site.SCALA_BINARY_VERSION}} |
Flume | spark-streaming-flume_{{site.SCALA_BINARY_VERSION}} |
spark-streaming-twitter_{{site.SCALA_BINARY_VERSION}} | |
ZeroMQ | spark-streaming-zeromq_{{site.SCALA_BINARY_VERSION}} |
MQTT | spark-streaming-mqtt_{{site.SCALA_BINARY_VERSION}} |
For an up-to-date list, please refer to the Apache repository for the full list of supported sources and artifacts.
Initializing
To initialize a Spark Streaming program in Scala, a
StreamingContext
object has to be created, which is the main entry point of all Spark Streaming functionality.
A StreamingContext
object can be created by using
{% highlight scala %} new StreamingContext(master, appName, batchDuration, [sparkHome], [jars]) {% endhighlight %}
To initialize a Spark Streaming program in Java, a
JavaStreamingContext
object has to be created, which is the main entry point of all Spark Streaming functionality.
A JavaStreamingContext
object can be created by using
{% highlight scala %} new JavaStreamingContext(master, appName, batchInterval, [sparkHome], [jars]) {% endhighlight %}
The master
parameter is a standard Spark cluster URL
and can be "local" for local testing. The appName
is a name of your program,
which will be shown on your cluster's web UI. The batchInterval
is the size of the batches,
as explained earlier. Finally, the last two parameters are needed to deploy your code to a cluster
if running in distributed mode, as described in the
Spark programming guide.
Additionally, the underlying SparkContext can be accessed as
streamingContext.sparkContext
.
The batch interval must be set based on the latency requirements of your application and available cluster resources. See the Performance Tuning section for more details.
DStreams
Discretized Stream or DStream is the basic abstraction provided by Spark Streaming. It represents a continuous stream of data, either the input data stream received from source, or the processed data stream generated by transforming the input stream. Internally, it is represented by a continuous sequence of RDDs, which is Spark's abstraction of an immutable, distributed dataset. Each RDD in a DStream contains data from a certain interval, as shown in the following figure.