How to write large JSON data?

Question

I have been trying to write large amount (>800mb) of data to JSON file; I did some fair amount of trial and error to get this code:

def write_to_cube(data):
    with open('test.json') as file1:
        temp_data = json.load(file1)

    temp_data.update(data)

    file1.close()

    with open('test.json', 'w') as f:
        json.dump(temp_data, f)

        f.close()

to run it just call the function write_to_cube({"some_data" = data})

Now the problem with this code is that it's fast for the small amount of data, but the problem comes when test.json file has more than 800mb in it. When I try to update or add data to it, it takes ages.

I know there are external libraries such as simplejson or jsonpickle, I am not pretty sure on how to use them.

Is there any other way to this problem?

Update:

I am not sure how this can be a duplicate, other articles say nothing about writing or updating a large JSON file, rather they say only about parsing.

Is there a memory efficient and fast way to load big json files in python?

Reading rather large json files in Python

None of the above resolve this question a duplicate. They don't say anything about writing or update.

Answer 1

So the problem is that you have a long operation. Here are a few approaches that I usually do:

1. Optimize the operation: This rarely works. I wouldn't recommend any superb library that would parse the json a few seconds faster
2. Change your logic: If the purpose is to save and load data, probably you would like to try something else, like storing your object into a binary file instead.
3. Threads and callback, or deferred objects in some web frameworks. In case of web applications, sometimes, the operation takes longer than a request can wait, we can do the operation in background (some cases are: zipping files, then send the zip to user's email, sending SMS by calling another third party's api...)