Skip to content
Snippets Groups Projects
Commit e1f68453 authored by tedyu's avatar tedyu Committed by Shixiong Zhu
Browse files

[SPARK-12181] Check Cached unaligned-access capability before using Unsafe

## What changes were proposed in this pull request?

For MemoryMode.OFF_HEAP, Unsafe.getInt etc. are used with no restriction.

However, the Oracle implementation uses these methods only if the class variable unaligned (commented as "Cached unaligned-access capability") is true, which seems to be calculated whether the architecture is i386, x86, amd64, or x86_64.

I think we should perform similar check for the use of Unsafe.

Reference: https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent0.java#L112

## How was this patch tested?

Unit test suite

Author: tedyu <yuzhihong@gmail.com>

Closes #11943 from tedyu/master.
parent 366cac6f
No related branches found
No related tags found
No related merge requests found
......@@ -18,6 +18,7 @@
package org.apache.spark.unsafe;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import sun.misc.Unsafe;
......@@ -37,6 +38,33 @@ public final class Platform {
public static final int DOUBLE_ARRAY_OFFSET;
private static final boolean unaligned;
static {
boolean _unaligned;
// use reflection to access unaligned field
try {
Class<?> bitsClass =
Class.forName("java.nio.Bits", false, ClassLoader.getSystemClassLoader());
Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
unalignedMethod.setAccessible(true);
_unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
} catch (Throwable t) {
// We at least know x86 and x64 support unaligned access.
String arch = System.getProperty("os.arch", "");
//noinspection DynamicRegexReplaceableByCompiledPattern
_unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
}
unaligned = _unaligned;
}
/**
* @return true when running JVM is having sun's Unsafe package available in it and underlying
* system having unaligned-access capability.
*/
public static boolean unaligned() {
return unaligned;
}
public static int getInt(Object object, long offset) {
return _UNSAFE.getInt(object, offset);
}
......
......@@ -23,6 +23,7 @@ import org.apache.spark.SparkConf
import org.apache.spark.internal.Logging
import org.apache.spark.storage.BlockId
import org.apache.spark.storage.memory.MemoryStore
import org.apache.spark.unsafe.Platform
import org.apache.spark.unsafe.array.ByteArrayMethods
import org.apache.spark.unsafe.memory.MemoryAllocator
......@@ -190,6 +191,8 @@ private[spark] abstract class MemoryManager(
if (conf.getBoolean("spark.memory.offHeap.enabled", false)) {
require(conf.getSizeAsBytes("spark.memory.offHeap.size", 0) > 0,
"spark.memory.offHeap.size must be > 0 when spark.memory.offHeap.enabled == true")
require(Platform.unaligned(),
"No support for unaligned Unsafe. Set spark.memory.offHeap.enabled to false.")
MemoryMode.OFF_HEAP
} else {
MemoryMode.ON_HEAP
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment