-
- Downloads
[SPARK-18773][CORE] Make commons-crypto config translation consistent.
This change moves the logic that translates Spark configuration to commons-crypto configuration to the network-common module. It also extends TransportConf and ConfigProvider to provide the necessary interfaces for the translation to work. As part of the change, I removed SystemPropertyConfigProvider, which was mostly used as an "empty config" in unit tests, and adjusted the very few tests that required a specific config. I also changed the config keys for AES encryption to live under the "spark.network." namespace, which is more correct than their previous names under "spark.authenticate.". Tested via existing unit test. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #16200 from vanzin/SPARK-18773.
Showing
- common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java 2 additions, 15 deletions...ain/java/org/apache/spark/network/sasl/aes/AesCipher.java
- common/network-common/src/main/java/org/apache/spark/network/util/ConfigProvider.java 5 additions, 0 deletions...in/java/org/apache/spark/network/util/ConfigProvider.java
- common/network-common/src/main/java/org/apache/spark/network/util/CryptoUtils.java 49 additions, 0 deletions.../main/java/org/apache/spark/network/util/CryptoUtils.java
- common/network-common/src/main/java/org/apache/spark/network/util/MapConfigProvider.java 11 additions, 0 deletions...java/org/apache/spark/network/util/MapConfigProvider.java
- common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java 11 additions, 8 deletions...ain/java/org/apache/spark/network/util/TransportConf.java
- common/network-common/src/test/java/org/apache/spark/network/ChunkFetchIntegrationSuite.java 2 additions, 2 deletions.../org/apache/spark/network/ChunkFetchIntegrationSuite.java
- common/network-common/src/test/java/org/apache/spark/network/RpcIntegrationSuite.java 2 additions, 2 deletions...st/java/org/apache/spark/network/RpcIntegrationSuite.java
- common/network-common/src/test/java/org/apache/spark/network/StreamSuite.java 2 additions, 2 deletions...n/src/test/java/org/apache/spark/network/StreamSuite.java
- common/network-common/src/test/java/org/apache/spark/network/TransportClientFactorySuite.java 7 additions, 2 deletions...org/apache/spark/network/TransportClientFactorySuite.java
- common/network-common/src/test/java/org/apache/spark/network/sasl/SparkSaslSuite.java 25 additions, 14 deletions...st/java/org/apache/spark/network/sasl/SparkSaslSuite.java
- common/network-common/src/test/java/org/apache/spark/network/util/CryptoUtilsSuite.java 51 additions, 0 deletions.../java/org/apache/spark/network/util/CryptoUtilsSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java 2 additions, 2 deletions...a/org/apache/spark/network/sasl/SaslIntegrationSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java 2 additions, 2 deletions...rk/network/shuffle/ExternalShuffleBlockResolverSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleCleanupSuite.java 2 additions, 2 deletions...he/spark/network/shuffle/ExternalShuffleCleanupSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleIntegrationSuite.java 17 additions, 15 deletions...park/network/shuffle/ExternalShuffleIntegrationSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleSecuritySuite.java 2 additions, 2 deletions...e/spark/network/shuffle/ExternalShuffleSecuritySuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RetryingBlockFetcherSuite.java 5 additions, 16 deletions...ache/spark/network/shuffle/RetryingBlockFetcherSuite.java
- common/network-yarn/src/main/java/org/apache/spark/network/yarn/util/HadoopConfigProvider.java 7 additions, 0 deletions.../apache/spark/network/yarn/util/HadoopConfigProvider.java
- core/src/main/scala/org/apache/spark/network/netty/SparkTransportConf.scala 6 additions, 0 deletions...a/org/apache/spark/network/netty/SparkTransportConf.scala
- core/src/main/scala/org/apache/spark/security/CryptoStreamUtils.scala 5 additions, 13 deletions...n/scala/org/apache/spark/security/CryptoStreamUtils.scala
Loading
Please register or sign in to comment