-
- Downloads
SPARK-1480: Clean up use of classloaders
The Spark codebase is a bit fast-and-loose when accessing classloaders and this has caused a few bugs to surface in master. This patch defines some utility methods for accessing classloaders. This makes the intention when accessing a classloader much more explicit in the code and fixes a few cases where the wrong one was chosen. case (a) -> We want the classloader that loaded Spark case (b) -> We want the context class loader, or if not present, we want (a) This patch provides a better fix for SPARK-1403 (https://issues.apache.org/jira/browse/SPARK-1403) than the current work around, which it reverts. It also fixes a previously unreported bug that the `./spark-submit` script did not work for running with `local` master. It didn't work because the executor classloader did not properly delegate to the context class loader (if it is defined) and in local mode the context class loader is set by the `./spark-submit` script. A unit test is added for that case. Author: Patrick Wendell <pwendell@gmail.com> Closes #398 from pwendell/class-loaders and squashes the following commits: b4a1a58 [Patrick Wendell] Minor clean up 14f1272 [Patrick Wendell] SPARK-1480: Clean up use of classloaders
Showing
- core/src/main/scala/org/apache/spark/Logging.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/Logging.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 3 additions, 3 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala 7 additions, 15 deletions...cala/org/apache/spark/executor/MesosExecutorBackend.scala
- core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala 2 additions, 1 deletion...c/main/scala/org/apache/spark/metrics/MetricsConfig.scala
- core/src/main/scala/org/apache/spark/scheduler/ResultTask.scala 0 additions, 1 deletion...rc/main/scala/org/apache/spark/scheduler/ResultTask.scala
- core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala 2 additions, 1 deletion...scala/org/apache/spark/scheduler/SchedulableBuilder.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala 2 additions, 2 deletions...n/scala/org/apache/spark/scheduler/TaskResultGetter.scala
- core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala 2 additions, 1 deletion...in/scala/org/apache/spark/serializer/JavaSerializer.scala
- core/src/main/scala/org/apache/spark/ui/JettyUtils.scala 2 additions, 1 deletioncore/src/main/scala/org/apache/spark/ui/JettyUtils.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 15 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/executor/ExecutorURLClassLoaderSuite.scala 30 additions, 2 deletions...g/apache/spark/executor/ExecutorURLClassLoaderSuite.scala
- repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala 4 additions, 3 deletionsrepl/src/main/scala/org/apache/spark/repl/SparkILoop.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala 3 additions, 1 deletion...in/scala/org/apache/spark/sql/catalyst/util/package.scala
- sql/core/src/main/scala/org/apache/spark/sql/columnar/compression/compressionSchemes.scala 2 additions, 1 deletion...e/spark/sql/columnar/compression/compressionSchemes.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlSerializer.scala 2 additions, 1 deletion...a/org/apache/spark/sql/execution/SparkSqlSerializer.scala
Loading
Please register or sign in to comment