Skip to content
Snippets Groups Projects
Commit 6a7ba1ff authored by Michael Gummelt's avatar Michael Gummelt Committed by Sean Owen
Browse files

Fix dynamic allocation docs to address cached data.

## What changes were proposed in this pull request?

Documentation changes

## How was this patch tested?

No tests

Author: Michael Gummelt <mgummelt@mesosphere.io>

Closes #12664 from mgummelt/fix-dynamic-docs.
parent 2a5c9307
No related branches found
No related tags found
No related merge requests found
...@@ -158,8 +158,9 @@ executors will fetch shuffle files from the service instead of from each other. ...@@ -158,8 +158,9 @@ executors will fetch shuffle files from the service instead of from each other.
shuffle state written by an executor may continue to be served beyond the executor's lifetime. shuffle state written by an executor may continue to be served beyond the executor's lifetime.
In addition to writing shuffle files, executors also cache data either on disk or in memory. In addition to writing shuffle files, executors also cache data either on disk or in memory.
When an executor is removed, however, all cached data will no longer be accessible. There is When an executor is removed, however, all cached data will no longer be accessible. To mitigate this,
currently not yet a solution for this in Spark 1.2. In future releases, the cached data may be by default executors containing cached data are never removed. You can configure this behavior with
`spark.dynamicAllocation.cachedExecutorIdleTimeout`. In future releases, the cached data may be
preserved through an off-heap storage similar in spirit to how shuffle files are preserved through preserved through an off-heap storage similar in spirit to how shuffle files are preserved through
the external shuffle service. the external shuffle service.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment