-
- Downloads
[SPARK-22143][SQL] Fix memory leak in OffHeapColumnVector
## What changes were proposed in this pull request? `WriteableColumnVector` does not close its child column vectors. This can create memory leaks for `OffHeapColumnVector` where we do not clean up the memory allocated by a vectors children. This can be especially bad for string columns (which uses a child byte column vector). ## How was this patch tested? I have updated the existing tests to always use both on-heap and off-heap vectors. Testing and diagnoses was done locally. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #19367 from hvanhovell/SPARK-22143.
Showing
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OffHeapColumnVector.java 1 addition, 0 deletions...e/spark/sql/execution/vectorized/OffHeapColumnVector.java
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java 10 additions, 0 deletions...he/spark/sql/execution/vectorized/OnHeapColumnVector.java
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java 18 additions, 0 deletions.../spark/sql/execution/vectorized/WritableColumnVector.java
- sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorSuite.scala 48 additions, 54 deletions...he/spark/sql/execution/vectorized/ColumnVectorSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala 88 additions, 106 deletions...e/spark/sql/execution/vectorized/ColumnarBatchSuite.scala
Loading
Please register or sign in to comment