Skip to content
Snippets Groups Projects
Commit e75e340a authored by Rajesh Balamohan's avatar Rajesh Balamohan Committed by Reynold Xin
Browse files

[SPARK-12925][SQL] Improve HiveInspectors.unwrap for StringObjectIns…

Text is in UTF-8 and converting it via "UTF8String.fromString" incurs decoding and encoding, which turns out to be expensive and redundant.  Profiler snapshot details is attached in the JIRA (ref:https://issues.apache.org/jira/secure/attachment/12783331/SPARK-12925_profiler_cpu_samples.png)

Author: Rajesh Balamohan <rbalamohan@apache.org>

Closes #10848 from rajeshbalamohan/SPARK-12925.
parent 9753835c
No related branches found
No related tags found
No related merge requests found
...@@ -320,7 +320,9 @@ private[hive] trait HiveInspectors { ...@@ -320,7 +320,9 @@ private[hive] trait HiveInspectors {
case hvoi: HiveCharObjectInspector => case hvoi: HiveCharObjectInspector =>
UTF8String.fromString(hvoi.getPrimitiveJavaObject(data).getValue) UTF8String.fromString(hvoi.getPrimitiveJavaObject(data).getValue)
case x: StringObjectInspector if x.preferWritable() => case x: StringObjectInspector if x.preferWritable() =>
UTF8String.fromString(x.getPrimitiveWritableObject(data).toString) // Text is in UTF-8 already. No need to convert again via fromString
val wObj = x.getPrimitiveWritableObject(data)
UTF8String.fromBytes(wObj.getBytes, 0, wObj.getLength)
case x: StringObjectInspector => case x: StringObjectInspector =>
UTF8String.fromString(x.getPrimitiveJavaObject(data)) UTF8String.fromString(x.getPrimitiveJavaObject(data))
case x: IntObjectInspector if x.preferWritable() => x.get(data) case x: IntObjectInspector if x.preferWritable() => x.get(data)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment