-
- Downloads
Replace RoutingTableMessage with pair
RoutingTableMessage was used to construct routing tables to enable joining VertexRDDs with partitioned edges. It stored three elements: the destination vertex ID, the source edge partition, and a byte specifying the position in which the edge partition referenced the vertex to enable join elimination. However, this was incompatible with sort-based shuffle (SPARK-2045). It was also slightly wasteful, because partition IDs are usually much smaller than 2^32, though this was mitigated by a custom serializer that used variable-length encoding. This commit replaces RoutingTableMessage with a pair of (VertexId, Int) where the Int encodes both the source partition ID (in the lower 30 bits) and the position (in the top 2 bits). Author: Ankur Dave <ankurdave@gmail.com> Closes #1553 from ankurdave/remove-RoutingTableMessage and squashes the following commits: 697e17b [Ankur Dave] Replace RoutingTableMessage with pair
Showing
- graphx/src/main/scala/org/apache/spark/graphx/GraphKryoRegistrator.scala 0 additions, 1 deletion.../scala/org/apache/spark/graphx/GraphKryoRegistrator.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/RoutingTablePartition.scala 28 additions, 19 deletions.../org/apache/spark/graphx/impl/RoutingTablePartition.scala
- graphx/src/main/scala/org/apache/spark/graphx/impl/Serializers.scala 7 additions, 9 deletions...main/scala/org/apache/spark/graphx/impl/Serializers.scala
- graphx/src/main/scala/org/apache/spark/graphx/package.scala 1 addition, 1 deletiongraphx/src/main/scala/org/apache/spark/graphx/package.scala
Loading
Please register or sign in to comment