'Is there a way to test if the buffer might overflow (go above 2GB) before the write happens in a FlatBufferBuilder?

I am trying to use the FlatBufferBuilder Java API to serialize large amount of data (>100GB) into a sequence of size prefixed flatbuffers.

Is there a way to test if next write of a compiler generated Table object might give an AssertionError tied to growing above 2GB ByteBuffer size limit. There is an int offset() method but it doesn't account for alignment.

Furthermore, It's hard to extend the FlatBufferBuilder class since all the fields are private and only protected member is the finish() method

Does anyone know if its possible to perform this book-keeping to prevent possible buffer overflow ? I know we can't know size of every object beforehand but maybe a good hint can be taken from reusing some parts of this calculation

One possible simple solution could be to add a size cap of little bit less than 1GB possible using offset(). Since minimum alignment is the largest size of any object seen in the buffer, this would ensure it doesn't go above 1GB and total less than 2GB. I'm not sure if it might actually work. This doesn't require making any changes to the builder API.

I tried to do a infinite loop to write a byte value and it exits with an AssertionError tied to 2GB buffer size limit.

I looked at extending the FlatBufferBuilder but most of the members are private.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source