JSON cannot handel numpy's float128 type

Question

I need to dump numpy arrays of dtype float128 to JSON. For that purpose I wrote a custom array encoder, that properly handles arrays by calling tolist() on them. This works perfectly fine for all real valued dtypes, except for float128. See the following example:

import json
import numpy as np


class ArrayEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, np.ndarray):
            out = {'__ndarray__': True,
                   '__dtype__': o.dtype.str,
                   'data': o.tolist()}
            return out
        return json.JSONEncoder.default(self, o)

arr64 = np.array([1, 2, 3], dtype='float64')
arr128 = np.array([1, 3, 4], dtype='float128')

json.dumps(arr64, cls=ArrayEncoder)     # fine
json.dumps(arr128, cls=ArrayEncoder)    # TypeError

TypeError: Object of type float128 is not JSON serializable

The purpose of the encoder is to provide data in a format JSON can handle. In this case the data is converted to a list of plain Python floats, which should not make any trouble. A possible solution would be to change the conversion line in the encoder to

class ArrayEncoder(json.JSONEncoder):
    def default(self, o):
        ...
            'data': o.astype('float64').tolist()
        ...

I am, however, interestend in cause of the problem. Why does JSON raise an error even though it uses an encoder that provides the data in a serializable format?

Answer 1

You have a custom encoder, but it only cares about np.array types - it converts then to a json object and their data as a Python list, and then defaults to normal lists.

The .tolist method you call makes your float128 array elements as standalone Float128 objects, which are not JSON encodable by default, and your encoder explicitly just wants to know about "isinstance(..., np.ndarray)".

You have two options: either check for float128 , so that your encoder is called recursivelly for each value, or convert then on output of np.ndarray either to plain float, or to a string:

class ArrayEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, np.ndarray):
            out = {'__ndarray__': True,
                   '__dtype__': o.dtype.str,
                   'data': o.tolist()} # <-  the other option is to map the results of .tolist here, and convert any float128 to str/float oon this step.
            return out
        elif isinstance(o, np.float128):
            return float(o)
        return json.JSONEncoder.default(self, o)