Skip to content
Snippets Groups Projects
Commit f776bc98 authored by woshilaiceshide's avatar woshilaiceshide Committed by Matei Zaharia
Browse files

[CORE] SPARK-2640: In "local[N]", free cores of the only executor should be...

[CORE] SPARK-2640: In "local[N]", free cores of the only executor should be touched by "spark.task.cpus" for every finish/start-up of tasks.

Make spark's "local[N]" better.
In our company, we use "local[N]" in production. It works exellentlly. It's our best choice.

Author: woshilaiceshide <woshilaiceshide@qq.com>

Closes #1544 from woshilaiceshide/localX and squashes the following commits:

6c85154 [woshilaiceshide] [CORE] SPARK-2640: In "local[N]", free cores of the only executor should be touched by "spark.task.cpus" for every finish/start-up of tasks.
parent 25921110
No related branches found
No related tags found
No related merge requests found
...@@ -57,7 +57,7 @@ private[spark] class LocalActor( ...@@ -57,7 +57,7 @@ private[spark] class LocalActor(
case StatusUpdate(taskId, state, serializedData) => case StatusUpdate(taskId, state, serializedData) =>
scheduler.statusUpdate(taskId, state, serializedData) scheduler.statusUpdate(taskId, state, serializedData)
if (TaskState.isFinished(state)) { if (TaskState.isFinished(state)) {
freeCores += 1 freeCores += scheduler.CPUS_PER_TASK
reviveOffers() reviveOffers()
} }
...@@ -68,7 +68,7 @@ private[spark] class LocalActor( ...@@ -68,7 +68,7 @@ private[spark] class LocalActor(
def reviveOffers() { def reviveOffers() {
val offers = Seq(new WorkerOffer(localExecutorId, localExecutorHostname, freeCores)) val offers = Seq(new WorkerOffer(localExecutorId, localExecutorHostname, freeCores))
for (task <- scheduler.resourceOffers(offers).flatten) { for (task <- scheduler.resourceOffers(offers).flatten) {
freeCores -= 1 freeCores -= scheduler.CPUS_PER_TASK
executor.launchTask(executorBackend, task.taskId, task.name, task.serializedTask) executor.launchTask(executorBackend, task.taskId, task.name, task.serializedTask)
} }
} }
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment