Skip to content
Snippets Groups Projects
Commit e9d268f6 authored by Kent Yao's avatar Kent Yao Committed by Sean Owen
Browse files

[SPARK-20096][SPARK SUBMIT][MINOR] Expose the right queue name not null if set...

[SPARK-20096][SPARK SUBMIT][MINOR] Expose the right queue name not null if set by --conf or configure file

## What changes were proposed in this pull request?

while submit apps with -v or --verbose, we can print the right queue name, but if we set a queue name with `spark.yarn.queue` by --conf or in the spark-default.conf, we just got `null`  for the queue in Parsed arguments.
```
bin/spark-shell -v --conf spark.yarn.queue=thequeue
Using properties file: /home/hadoop/spark-2.1.0-bin-apache-hdp2.7.3/conf/spark-defaults.conf
....
Adding default property: spark.yarn.queue=default
Parsed arguments:
  master                  yarn
  deployMode              client
  ...
  queue                   null
  ....
  verbose                 true
Spark properties used, including those specified through
 --conf and those from the properties file /home/hadoop/spark-2.1.0-bin-apache-hdp2.7.3/conf/spark-defaults.conf:
  spark.yarn.queue -> thequeue
  ....
```
## How was this patch tested?

ut and local verify

Author: Kent Yao <yaooqinn@hotmail.com>

Closes #17430 from yaooqinn/SPARK-20096.
parent 258bff2c
No related branches found
No related tags found
No related merge requests found
......@@ -190,6 +190,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
.orNull
numExecutors = Option(numExecutors)
.getOrElse(sparkProperties.get("spark.executor.instances").orNull)
queue = Option(queue).orElse(sparkProperties.get("spark.yarn.queue")).orNull
keytab = Option(keytab).orElse(sparkProperties.get("spark.yarn.keytab")).orNull
principal = Option(principal).orElse(sparkProperties.get("spark.yarn.principal")).orNull
......
......@@ -148,6 +148,17 @@ class SparkSubmitSuite
appArgs.childArgs should be (Seq("--master", "local", "some", "--weird", "args"))
}
test("print the right queue name") {
val clArgs = Seq(
"--name", "myApp",
"--class", "Foo",
"--conf", "spark.yarn.queue=thequeue",
"userjar.jar")
val appArgs = new SparkSubmitArguments(clArgs)
appArgs.queue should be ("thequeue")
appArgs.toString should include ("thequeue")
}
test("specify deploy mode through configuration") {
val clArgs = Seq(
"--master", "yarn",
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment