Skip to content
Snippets Groups Projects
  • Gregory Owen's avatar
    c3462c65
    [SPARK-2086] Improve output of toDebugString to make shuffle boundaries more clear · c3462c65
    Gregory Owen authored
    Changes RDD.toDebugString() to show hierarchy and shuffle transformations more clearly
    
    New output:
    
    ```
    (3) FlatMappedValuesRDD[325] at apply at Transformer.scala:22
     |  MappedValuesRDD[324] at apply at Transformer.scala:22
     |  CoGroupedRDD[323] at apply at Transformer.scala:22
     +-(5) MappedRDD[320] at apply at Transformer.scala:22
     |  |  MappedRDD[319] at apply at Transformer.scala:22
     |  |  MappedValuesRDD[318] at apply at Transformer.scala:22
     |  |  MapPartitionsRDD[317] at apply at Transformer.scala:22
     |  |  ShuffledRDD[316] at apply at Transformer.scala:22
     |  +-(10) MappedRDD[315] at apply at Transformer.scala:22
     |     |   ParallelCollectionRDD[314] at apply at Transformer.scala:22
     +-(100) MappedRDD[322] at apply at Transformer.scala:22
         |   ParallelCollectionRDD[321] at apply at Transformer.scala:22
    ```
    
    Author: Gregory Owen <greowen@gmail.com>
    
    Closes #1364 from GregOwen/to-debug-string and squashes the following commits:
    
    08f5c78 [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly
    1603f7b [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly
    c3462c65
    History
    [SPARK-2086] Improve output of toDebugString to make shuffle boundaries more clear
    Gregory Owen authored
    Changes RDD.toDebugString() to show hierarchy and shuffle transformations more clearly
    
    New output:
    
    ```
    (3) FlatMappedValuesRDD[325] at apply at Transformer.scala:22
     |  MappedValuesRDD[324] at apply at Transformer.scala:22
     |  CoGroupedRDD[323] at apply at Transformer.scala:22
     +-(5) MappedRDD[320] at apply at Transformer.scala:22
     |  |  MappedRDD[319] at apply at Transformer.scala:22
     |  |  MappedValuesRDD[318] at apply at Transformer.scala:22
     |  |  MapPartitionsRDD[317] at apply at Transformer.scala:22
     |  |  ShuffledRDD[316] at apply at Transformer.scala:22
     |  +-(10) MappedRDD[315] at apply at Transformer.scala:22
     |     |   ParallelCollectionRDD[314] at apply at Transformer.scala:22
     +-(100) MappedRDD[322] at apply at Transformer.scala:22
         |   ParallelCollectionRDD[321] at apply at Transformer.scala:22
    ```
    
    Author: Gregory Owen <greowen@gmail.com>
    
    Closes #1364 from GregOwen/to-debug-string and squashes the following commits:
    
    08f5c78 [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly
    1603f7b [Gregory Owen] toDebugString: prettier debug printing to show shuffles and joins more clearly