Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    55349f9f
    [SPARK-1740] [PySpark] kill the python worker · 55349f9f
    Davies Liu authored
    Kill only the python worker related to cancelled tasks.
    
    The daemon will start a background thread to monitor all the opened sockets for all workers. If the socket is closed by JVM, this thread will kill the worker.
    
    When an task is cancelled, the socket to worker will be closed, then the worker will be killed by deamon.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1643 from davies/kill and squashes the following commits:
    
    8ffe9f3 [Davies Liu] kill worker by deamon, because runtime.exec() is too heavy
    46ca150 [Davies Liu] address comment
    acd751c [Davies Liu] kill the worker when task is canceled
    55349f9f
    History
    [SPARK-1740] [PySpark] kill the python worker
    Davies Liu authored
    Kill only the python worker related to cancelled tasks.
    
    The daemon will start a background thread to monitor all the opened sockets for all workers. If the socket is closed by JVM, this thread will kill the worker.
    
    When an task is cancelled, the socket to worker will be closed, then the worker will be killed by deamon.
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1643 from davies/kill and squashes the following commits:
    
    8ffe9f3 [Davies Liu] kill worker by deamon, because runtime.exec() is too heavy
    46ca150 [Davies Liu] address comment
    acd751c [Davies Liu] kill the worker when task is canceled