-
- Downloads
[SPARK-21357][DSTREAMS] FileInputDStream not remove out of date RDD
## What changes were proposed in this pull request? ```DStreams class FileInputDStream [line 162] protected[streaming] override def clearMetadata(time: Time) { batchTimeToSelectedFiles.synchronized { val oldFiles = batchTimeToSelectedFiles.filter(_._1 < (time - rememberDuration)) batchTimeToSelectedFiles --= oldFiles.keys ``` The above code does not remove the old generatedRDDs. "super.clearMetadata(time)" was added to the beginning of clearMetadata to remove the old generatedRDDs. ## How was this patch tested? At the end of clearMetadata, the testing code (print the number of generatedRDDs) was added to check the old RDDS were removed manually. Author: shaofei007 <1427357147@qq.com> Author: Fei Shao <1427357147@qq.com> Closes #18718 from shaofei007/master.
Please register or sign in to comment