-
- Downloads
[SPARK-15762][SQL] Cache Metadata & StructType hashCodes; use singleton Metadata.empty
We should cache `Metadata.hashCode` and use a singleton for `Metadata.empty` because calculating metadata hashCodes appears to be a bottleneck for certain workloads. We should also cache `StructType.hashCode`. In an optimizer stress-test benchmark run by ericl, these `hashCode` calls accounted for roughly 40% of the total CPU time and this bottleneck was completely eliminated by the caching added by this patch. Author: Josh Rosen <joshrosen@databricks.com> Closes #13504 from JoshRosen/metadata-fix.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala 5 additions, 2 deletions.../src/main/scala/org/apache/spark/sql/types/Metadata.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala 2 additions, 1 deletion...rc/main/scala/org/apache/spark/sql/types/StructType.scala
Loading
Please register or sign in to comment