-
- Downloads
[SPARK-14936][BUILD][TESTS] FlumePollingStreamSuite is slow
https://issues.apache.org/jira/browse/SPARK-14936 ## What changes were proposed in this pull request? FlumePollingStreamSuite contains two tests which run for a minute each. This seems excessively slow and we should speed it up if possible. In this PR, instead of creating `StreamingContext` directly from `conf`, here an underlying `SparkContext` is created before all and it is used to create each`StreamingContext`. Running time is reduced by avoiding multiple `SparkContext` creations and destroys. ## How was this patch tested? Tested on my local machine running `testOnly *.FlumePollingStreamSuite` Author: Xin Ren <iamshrek@126.com> Closes #12845 from keypointt/SPARK-14936.
Showing
- external/flume/src/main/scala/org/apache/spark/streaming/flume/PollingFlumeTestUtils.scala 2 additions, 2 deletions.../apache/spark/streaming/flume/PollingFlumeTestUtils.scala
- external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala 20 additions, 6 deletions...pache/spark/streaming/flume/FlumePollingStreamSuite.scala
- python/pyspark/streaming/tests.py 1 addition, 1 deletionpython/pyspark/streaming/tests.py
Loading
Please register or sign in to comment