This article discusses various types of a memory buffer allocation in Java. We will see how to treat any sort of Java buffer uniformly using
sun.misc.Unsafe memory access methods. This article may be especially interesting for ex-C programmers willing to work with the memory on the lowest possible level in Java.
Array allocation limitations
Array size in Java is limited by the fact of using
int as an array index. This means that you can not allocate an array with more than
= 2^31 - 1 ) elements. This doesn’t mean that the longest chunk of memory you can allocate in Java is 2 Gb. You can allocate an array of bigger type instead. For example,
16Gb - 8 bytes, if you have sufficiently high
-Xmx Java setting (usually you should have about 50% more memory in heap – so in order to allocate 16Gb buffer, you will have to specify
-Xmx24G (this is a general rule, actual required heap size may vary).
Unfortunately, you will be limited by your array element type in pure Java. The only useful class for working with arrays is a
ByteBuffer, which offers methods for getting/writing various Java data types in the buffer (see Various methods of binary serialization in Java for more details). The disadvantage of a
ByteBuffer – you are limited with
byte as a source array type, which means a limitation of
2Gb for your buffer.
Treating any arrays as a byte buffer
For a while let’s assume that
2Gb buffers were not sufficient for our needs, but a
16Gb buffer will make us happy. We have allocated a
long, but want to treat this buffer as a byte array. We need to use a best C programmer friend in Java –
sun.misc.Unsafe. This class has 2 sets of methods:
getN( Object, offset ), where
N is a result type for reading a value of given type from the given offset in the object and
putN( Object, offset, value ) for writing a value at a given offset.
Unfortunately, these methods set or get only an individual value. If you want to copy data to/from an array, you will need one more
copyMemory(srcObject, srcOffset, destObject, destOffset, count). It works similar to
System.arraycopy, but copies bytes instead of array elements.
In order to access array data using
sun.misc.Unsafe, you will need 2 components: