Skip to content
Snippets Groups Projects
  • Michael Armbrust's avatar
    236dfac6
    [SPARK-2784][SQL] Deprecate hql() method in favor of a config option, 'spark.sql.dialect' · 236dfac6
    Michael Armbrust authored
    Many users have reported being confused by the distinction between the `sql` and `hql` methods.  Specifically, many users think that `sql(...)` cannot be used to read hive tables.  In this PR I introduce a new configuration option `spark.sql.dialect` that picks which dialect with be used for parsing.  For SQLContext this must be set to `sql`.  In `HiveContext` it defaults to `hiveql` but can also be set to `sql`.
    
    The `hql` and `hiveql` methods continue to act the same but are now marked as deprecated.
    
    **This is a possibly breaking change for some users unless they set the dialect manually, though this is unlikely.**
    
    For example: `hiveContex.sql("SELECT 1")` will now throw a parsing exception by default.
    
    Author: Michael Armbrust <michael@databricks.com>
    
    Closes #1746 from marmbrus/sqlLanguageConf and squashes the following commits:
    
    ad375cc [Michael Armbrust] Merge remote-tracking branch 'apache/master' into sqlLanguageConf
    20c43f8 [Michael Armbrust] override function instead of just setting the value
    7e4ae93 [Michael Armbrust] Deprecate hql() method in favor of a config option, 'spark.sql.dialect'
    236dfac6
    History
    [SPARK-2784][SQL] Deprecate hql() method in favor of a config option, 'spark.sql.dialect'
    Michael Armbrust authored
    Many users have reported being confused by the distinction between the `sql` and `hql` methods.  Specifically, many users think that `sql(...)` cannot be used to read hive tables.  In this PR I introduce a new configuration option `spark.sql.dialect` that picks which dialect with be used for parsing.  For SQLContext this must be set to `sql`.  In `HiveContext` it defaults to `hiveql` but can also be set to `sql`.
    
    The `hql` and `hiveql` methods continue to act the same but are now marked as deprecated.
    
    **This is a possibly breaking change for some users unless they set the dialect manually, though this is unlikely.**
    
    For example: `hiveContex.sql("SELECT 1")` will now throw a parsing exception by default.
    
    Author: Michael Armbrust <michael@databricks.com>
    
    Closes #1746 from marmbrus/sqlLanguageConf and squashes the following commits:
    
    ad375cc [Michael Armbrust] Merge remote-tracking branch 'apache/master' into sqlLanguageConf
    20c43f8 [Michael Armbrust] override function instead of just setting the value
    7e4ae93 [Michael Armbrust] Deprecate hql() method in favor of a config option, 'spark.sql.dialect'
sql.py 56.38 KiB