Skip to content
Snippets Groups Projects
  • Sandeep Singh's avatar
    bc167a2a
    [SPARK-928][CORE] Add support for Unsafe-based serializer in Kryo · bc167a2a
    Sandeep Singh authored
    ## What changes were proposed in this pull request?
    Now since we have migrated to Kryo-3.0.0 in https://issues.apache.org/jira/browse/SPARK-11416, we can gives users option to use unsafe SerDer. It can turned by setting `spark.kryo.useUnsafe` to `true`
    
    ## How was this patch tested?
    Ran existing tests
    
    ```
         Benchmark Kryo Unsafe vs safe Serialization: Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
          ------------------------------------------------------------------------------------------------
          basicTypes: Int unsafe:true                    160 /  178         98.5          10.1       1.0X
          basicTypes: Long unsafe:true                   210 /  218         74.9          13.4       0.8X
          basicTypes: Float unsafe:true                  203 /  213         77.5          12.9       0.8X
          basicTypes: Double unsafe:true                 226 /  235         69.5          14.4       0.7X
          Array: Int unsafe:true                        1087 / 1101         14.5          69.1       0.1X
          Array: Long unsafe:true                       2758 / 2844          5.7         175.4       0.1X
          Array: Float unsafe:true                      1511 / 1552         10.4          96.1       0.1X
          Array: Double unsafe:true                     2942 / 2972          5.3         187.0       0.1X
          Map of string->Double unsafe:true             2645 / 2739          5.9         168.2       0.1X
          basicTypes: Int unsafe:false                   211 /  218         74.7          13.4       0.8X
          basicTypes: Long unsafe:false                  247 /  253         63.6          15.7       0.6X
          basicTypes: Float unsafe:false                 211 /  216         74.5          13.4       0.8X
          basicTypes: Double unsafe:false                227 /  233         69.2          14.4       0.7X
          Array: Int unsafe:false                       3012 / 3032          5.2         191.5       0.1X
          Array: Long unsafe:false                      4463 / 4515          3.5         283.8       0.0X
          Array: Float unsafe:false                     2788 / 2868          5.6         177.2       0.1X
          Array: Double unsafe:false                    3558 / 3752          4.4         226.2       0.0X
          Map of string->Double unsafe:false            2806 / 2933          5.6         178.4       0.1X
    ```
    
    Author: Sandeep Singh <sandeep@techaddict.me>
    Author: Sandeep Singh <sandeep@origamilogic.com>
    
    Closes #12913 from techaddict/SPARK-928.
    bc167a2a
    History
    [SPARK-928][CORE] Add support for Unsafe-based serializer in Kryo
    Sandeep Singh authored
    ## What changes were proposed in this pull request?
    Now since we have migrated to Kryo-3.0.0 in https://issues.apache.org/jira/browse/SPARK-11416, we can gives users option to use unsafe SerDer. It can turned by setting `spark.kryo.useUnsafe` to `true`
    
    ## How was this patch tested?
    Ran existing tests
    
    ```
         Benchmark Kryo Unsafe vs safe Serialization: Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
          ------------------------------------------------------------------------------------------------
          basicTypes: Int unsafe:true                    160 /  178         98.5          10.1       1.0X
          basicTypes: Long unsafe:true                   210 /  218         74.9          13.4       0.8X
          basicTypes: Float unsafe:true                  203 /  213         77.5          12.9       0.8X
          basicTypes: Double unsafe:true                 226 /  235         69.5          14.4       0.7X
          Array: Int unsafe:true                        1087 / 1101         14.5          69.1       0.1X
          Array: Long unsafe:true                       2758 / 2844          5.7         175.4       0.1X
          Array: Float unsafe:true                      1511 / 1552         10.4          96.1       0.1X
          Array: Double unsafe:true                     2942 / 2972          5.3         187.0       0.1X
          Map of string->Double unsafe:true             2645 / 2739          5.9         168.2       0.1X
          basicTypes: Int unsafe:false                   211 /  218         74.7          13.4       0.8X
          basicTypes: Long unsafe:false                  247 /  253         63.6          15.7       0.6X
          basicTypes: Float unsafe:false                 211 /  216         74.5          13.4       0.8X
          basicTypes: Double unsafe:false                227 /  233         69.2          14.4       0.7X
          Array: Int unsafe:false                       3012 / 3032          5.2         191.5       0.1X
          Array: Long unsafe:false                      4463 / 4515          3.5         283.8       0.0X
          Array: Float unsafe:false                     2788 / 2868          5.6         177.2       0.1X
          Array: Double unsafe:false                    3558 / 3752          4.4         226.2       0.0X
          Map of string->Double unsafe:false            2806 / 2933          5.6         178.4       0.1X
    ```
    
    Author: Sandeep Singh <sandeep@techaddict.me>
    Author: Sandeep Singh <sandeep@origamilogic.com>
    
    Closes #12913 from techaddict/SPARK-928.