Skip to content
Snippets Groups Projects
  • Andrew Or's avatar
    745e496c
    [Fix #204] Eliminate delay between binding and log checking · 745e496c
    Andrew Or authored
    **Bug**: In the existing history server, there is a `spark.history.updateInterval` seconds delay before application logs show up on the UI.
    
    **Cause**: This is because the following events happen in this order: (1) The background thread that checks for logs starts, but realizes the server has not yet bound and so waits for N seconds, (2) server binds, (3) N seconds later the background thread finds that the server has finally bound to a port, and so finally checks for application logs.
    
    **Fix**: This PR forces the log checking thread to start immediately after binding. It also documents two relevant environment variables that are currently missing.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #441 from andrewor14/history-server-fix and squashes the following commits:
    
    b2eb46e [Andrew Or] Document SPARK_PUBLIC_DNS and SPARK_HISTORY_OPTS for the history server
    e8d1fbc [Andrew Or] Eliminate delay between binding and checking for logs
    745e496c
    History
    [Fix #204] Eliminate delay between binding and log checking
    Andrew Or authored
    **Bug**: In the existing history server, there is a `spark.history.updateInterval` seconds delay before application logs show up on the UI.
    
    **Cause**: This is because the following events happen in this order: (1) The background thread that checks for logs starts, but realizes the server has not yet bound and so waits for N seconds, (2) server binds, (3) N seconds later the background thread finds that the server has finally bound to a port, and so finally checks for application logs.
    
    **Fix**: This PR forces the log checking thread to start immediately after binding. It also documents two relevant environment variables that are currently missing.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #441 from andrewor14/history-server-fix and squashes the following commits:
    
    b2eb46e [Andrew Or] Document SPARK_PUBLIC_DNS and SPARK_HISTORY_OPTS for the history server
    e8d1fbc [Andrew Or] Eliminate delay between binding and checking for logs