-
- Downloads
[SPARK-20070][SQL] Redact DataSourceScanExec treeString
## What changes were proposed in this pull request? The explain output of `DataSourceScanExec` can contain sensitive information (like Amazon keys). Such information should not end up in logs, or be exposed to non privileged users. This PR addresses this by adding a redaction facility for the `DataSourceScanExec.treeString`. A user can enable this by setting a regex in the `spark.redaction.string.regex` configuration. ## How was this patch tested? Added a unit test to check the output of DataSourceScanExec. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #17397 from hvanhovell/SPARK-20070.
Showing
- core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala 13 additions, 0 deletions...cala/org/apache/spark/internal/config/ConfigBuilder.scala
- core/src/main/scala/org/apache/spark/internal/config/package.scala 10 additions, 2 deletions...main/scala/org/apache/spark/internal/config/package.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 15 additions, 2 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/internal/config/ConfigEntrySuite.scala 15 additions, 4 deletions...a/org/apache/spark/internal/config/ConfigEntrySuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 25 additions, 16 deletions...a/org/apache/spark/sql/execution/DataSourceScanExec.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala 60 additions, 0 deletions...park/sql/execution/DataSourceScanExecRedactionSuite.scala
Loading
Please register or sign in to comment