Skip to content
Snippets Groups Projects
  • Thomas Graves's avatar
    7edbea41
    SPARK-1189: Add Security to Spark - Akka, Http, ConnectionManager, UI use servlets · 7edbea41
    Thomas Graves authored
    resubmit pull request.  was https://github.com/apache/incubator-spark/pull/332.
    
    Author: Thomas Graves <tgraves@apache.org>
    
    Closes #33 from tgravescs/security-branch-0.9-with-client-rebase and squashes the following commits:
    
    dfe3918 [Thomas Graves] Fix merge conflict since startUserClass now using runAsUser
    05eebed [Thomas Graves] Fix dependency lost in upmerge
    d1040ec [Thomas Graves] Fix up various imports
    05ff5e0 [Thomas Graves] Fix up imports after upmerging to master
    ac046b3 [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase
    13733e1 [Thomas Graves] Pass securityManager and SparkConf around where we can. Switch to use sparkConf for reading config whereever possible. Added ConnectionManagerSuite unit tests.
    4a57acc [Thomas Graves] Change UI createHandler routines to createServlet since they now return servlets
    2f77147 [Thomas Graves] Rework from comments
    50dd9f2 [Thomas Graves] fix header in SecurityManager
    ecbfb65 [Thomas Graves] Fix spacing and formatting
    b514bec [Thomas Graves] Fix reference to config
    ed3d1c1 [Thomas Graves] Add security.md
    6f7ddf3 [Thomas Graves] Convert SaslClient and SaslServer to scala, change spark.authenticate.ui to spark.ui.acls.enable, and fix up various other things from review comments
    2d9e23e [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase_rework
    5721c5a [Thomas Graves] update AkkaUtilsSuite test for the actorSelection changes, fix typos based on comments, and remove extra lines I missed in rebase from AkkaUtils
    f351763 [Thomas Graves] Add Security to Spark - Akka, Http, ConnectionManager, UI to use servlets
    7edbea41
    History
    SPARK-1189: Add Security to Spark - Akka, Http, ConnectionManager, UI use servlets
    Thomas Graves authored
    resubmit pull request.  was https://github.com/apache/incubator-spark/pull/332.
    
    Author: Thomas Graves <tgraves@apache.org>
    
    Closes #33 from tgravescs/security-branch-0.9-with-client-rebase and squashes the following commits:
    
    dfe3918 [Thomas Graves] Fix merge conflict since startUserClass now using runAsUser
    05eebed [Thomas Graves] Fix dependency lost in upmerge
    d1040ec [Thomas Graves] Fix up various imports
    05ff5e0 [Thomas Graves] Fix up imports after upmerging to master
    ac046b3 [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase
    13733e1 [Thomas Graves] Pass securityManager and SparkConf around where we can. Switch to use sparkConf for reading config whereever possible. Added ConnectionManagerSuite unit tests.
    4a57acc [Thomas Graves] Change UI createHandler routines to createServlet since they now return servlets
    2f77147 [Thomas Graves] Rework from comments
    50dd9f2 [Thomas Graves] fix header in SecurityManager
    ecbfb65 [Thomas Graves] Fix spacing and formatting
    b514bec [Thomas Graves] Fix reference to config
    ed3d1c1 [Thomas Graves] Add security.md
    6f7ddf3 [Thomas Graves] Convert SaslClient and SaslServer to scala, change spark.authenticate.ui to spark.ui.acls.enable, and fix up various other things from review comments
    2d9e23e [Thomas Graves] Merge remote-tracking branch 'upstream/master' into security-branch-0.9-with-client-rebase_rework
    5721c5a [Thomas Graves] update AkkaUtilsSuite test for the actorSelection changes, fix typos based on comments, and remove extra lines I missed in rebase from AkkaUtils
    f351763 [Thomas Graves] Add Security to Spark - Akka, Http, ConnectionManager, UI to use servlets
security.md 2.22 KiB
layout: global
title: Spark Security

Spark currently supports authentication via a shared secret. Authentication can be configured to be on via the spark.authenticate configuration parameter. This parameter controls whether the Spark communication protocols do authentication using the shared secret. This authentication is a basic handshake to make sure both sides have the same shared secret and are allowed to communicate. If the shared secret is not identical they will not be allowed to communicate.

The Spark UI can also be secured by using javax servlet filters. A user may want to secure the UI if it has data that other users should not be allowed to see. The javax servlet filter specified by the user can authenticate the user and then once the user is logged in, Spark can compare that user versus the view acls to make sure they are authorized to view the UI. The configs 'spark.ui.acls.enable' and 'spark.ui.view.acls' control the behavior of the acls. Note that the person who started the application always has view access to the UI.

For Spark on Yarn deployments, configuring spark.authenticate to true will automatically handle generating and distributing the shared secret. Each application will use a unique shared secret. The Spark UI uses the standard YARN web application proxy mechanism and will authenticate via any installed Hadoop filters. If an authentication filter is enabled, the acls controls can be used by control which users can via the Spark UI.

For other types of Spark deployments, the spark config spark.authenticate.secret should be configured on each of the nodes. This secret will be used by all the Master/Workers and applications. The UI can be secured using a javax servlet filter installed via spark.ui.filters. If an authentication filter is enabled, the acls controls can be used by control which users can via the Spark UI.

IMPORTANT NOTE: The NettyBlockFetcherIterator is not secured so do not use netty for the shuffle is running with authentication on.

See Spark Configuration for more details on the security configs.

See org.apache.spark.SecurityManager for implementation details about security.