Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    59f84a95
    [SPARK-1687] [PySpark] pickable namedtuple · 59f84a95
    Davies Liu authored
    Add an hook to replace original namedtuple with an pickable one, then namedtuple could be used in RDDs.
    
    PS: pyspark should be import BEFORE "from collections import namedtuple"
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1623 from davies/namedtuple and squashes the following commits:
    
    045dad8 [Davies Liu] remove unrelated code changes
    4132f32 [Davies Liu] address comment
    55b1c1a [Davies Liu] fix tests
    61f86eb [Davies Liu] replace all the reference of namedtuple to new hacked one
    98df6c6 [Davies Liu] Merge branch 'master' of github.com:apache/spark into namedtuple
    f7b1bde [Davies Liu] add hack for CloudPickleSerializer
    0c5c849 [Davies Liu] Merge branch 'master' of github.com:apache/spark into namedtuple
    21991e6 [Davies Liu] hack namedtuple in __main__ module, make it picklable.
    93b03b8 [Davies Liu] pickable namedtuple
    59f84a95
    History
    [SPARK-1687] [PySpark] pickable namedtuple
    Davies Liu authored
    Add an hook to replace original namedtuple with an pickable one, then namedtuple could be used in RDDs.
    
    PS: pyspark should be import BEFORE "from collections import namedtuple"
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1623 from davies/namedtuple and squashes the following commits:
    
    045dad8 [Davies Liu] remove unrelated code changes
    4132f32 [Davies Liu] address comment
    55b1c1a [Davies Liu] fix tests
    61f86eb [Davies Liu] replace all the reference of namedtuple to new hacked one
    98df6c6 [Davies Liu] Merge branch 'master' of github.com:apache/spark into namedtuple
    f7b1bde [Davies Liu] add hack for CloudPickleSerializer
    0c5c849 [Davies Liu] Merge branch 'master' of github.com:apache/spark into namedtuple
    21991e6 [Davies Liu] hack namedtuple in __main__ module, make it picklable.
    93b03b8 [Davies Liu] pickable namedtuple