Skip to content
Snippets Groups Projects
  • Chen Chao's avatar
    2f639957
    default task number misleading in several places · 2f639957
    Chen Chao authored
      private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism){
        new HashPartitioner(numPartitions)
      }
    
    it represents that the default task number in Spark Streaming relies on the variable defaultParallelism in SparkContext, which is decided by the config property spark.default.parallelism
    
    the property "spark.default.parallelism" refers to https://github.com/apache/spark/pull/389
    
    Author: Chen Chao <crazyjvm@gmail.com>
    
    Closes #766 from CrazyJvm/patch-7 and squashes the following commits:
    
    0b7efba [Chen Chao] Update streaming-programming-guide.md
    cc5b66c [Chen Chao] default task number misleading in several places
    2f639957
    History
    default task number misleading in several places
    Chen Chao authored
      private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism){
        new HashPartitioner(numPartitions)
      }
    
    it represents that the default task number in Spark Streaming relies on the variable defaultParallelism in SparkContext, which is decided by the config property spark.default.parallelism
    
    the property "spark.default.parallelism" refers to https://github.com/apache/spark/pull/389
    
    Author: Chen Chao <crazyjvm@gmail.com>
    
    Closes #766 from CrazyJvm/patch-7 and squashes the following commits:
    
    0b7efba [Chen Chao] Update streaming-programming-guide.md
    cc5b66c [Chen Chao] default task number misleading in several places