Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    ba46bbed
    [SPARK-2601] [PySpark] Fix Py4J error when transforming pickleFiles · ba46bbed
    Josh Rosen authored
    Similar to SPARK-1034, the problem was that Py4J didn’t cope well with the fake ClassTags used in the Java API.  It doesn’t look like there’s any reason why PythonRDD needs to take a ClassTag, since it just ignores the type of the previous RDD, so I removed the type parameter and we no longer pass ClassTags from Python.
    
    Author: Josh Rosen <joshrosen@apache.org>
    
    Closes #1605 from JoshRosen/spark-2601 and squashes the following commits:
    
    b68e118 [Josh Rosen] Fix Py4J error when transforming pickleFiles [SPARK-2601]
    ba46bbed
    History
    [SPARK-2601] [PySpark] Fix Py4J error when transforming pickleFiles
    Josh Rosen authored
    Similar to SPARK-1034, the problem was that Py4J didn’t cope well with the fake ClassTags used in the Java API.  It doesn’t look like there’s any reason why PythonRDD needs to take a ClassTag, since it just ignores the type of the previous RDD, so I removed the type parameter and we no longer pass ClassTags from Python.
    
    Author: Josh Rosen <joshrosen@apache.org>
    
    Closes #1605 from JoshRosen/spark-2601 and squashes the following commits:
    
    b68e118 [Josh Rosen] Fix Py4J error when transforming pickleFiles [SPARK-2601]