-
- Downloads
Remove GraphX MessageToPartition for compatibility with sort-based shuffle
MessageToPartition was used in `Graph#partitionBy`. Unlike a Tuple2, it marked the key as transient to avoid sending it over the network. However, it was incompatible with sort-based shuffle (SPARK-2045) and represented only a minor optimization: for partitionBy, it improved performance by 6.3% (30.4 s to 28.5 s) and reduced communication by 5.6% (114.2 MB to 107.8 MB). Author: Ankur Dave <ankurdave@gmail.com> Closes #1537 from ankurdave/remove-MessageToPartition and squashes the following commits: f9d0054 [Ankur Dave] Remove MessageToPartition ab71364 [Ankur Dave] Remove unused VertexBroadcastMsg
Showing
- graphx/src/main/scala/org/apache/spark/graphx/GraphKryoRegistrator.scala 0 additions, 2 deletions.../scala/org/apache/spark/graphx/GraphKryoRegistrator.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala 2 additions, 5 deletions...c/main/scala/org/apache/spark/graphx/impl/GraphImpl.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/MessageToPartition.scala 0 additions, 76 deletions...ala/org/apache/spark/graphx/impl/MessageToPartition.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/Serializers.scala 0 additions, 72 deletions...main/scala/org/apache/spark/graphx/impl/Serializers.scala
- graphx/src/test/scala/org/apache/spark/graphx/SerializerSuite.scala 0 additions, 73 deletions.../test/scala/org/apache/spark/graphx/SerializerSuite.scala
Loading
Please register or sign in to comment