JSON file contents in docker container are not reflecting correct changes

Question

I am using fast api to create endpoints. There are 3 basic endpoints:

1. /create to add a record in json file
2. /delete to delete a record in json file
3. /view which displays the record from json file

This json file is called store.json. This functionality is working fine when triggering the uvicorn server locally. But after containerizing it with docker the modification done to this file is not happening correctly. For example while adding records some records are not written in the file.

Below are the code snippets:

/view

@router.get("/")
async def view():
    """
    Fetched the list
    """
    resp = get_records()
    return {"response": resp}


def get_records():
    with open("core/store/store.json") as store_file:
        store = json.load(store_file)
    return store
    

/create

@router.get("/create")
async def create():
    """
    Create the record
    """
    resp = create_records()
    return {"response": resp}


def create_records():
    with open("core/store/store.json") as store_file:
        store = json.load(store_file)
    store.get("students").append({
        "id": random.randint(1, 101),
        "date_created": datetime.now().strftime("%b-%d-%Y at %H:%M")
    })
    with open("core/store/store.json", "w") as outfile:
        json.dump(store, outfile)
    return store.get("students")[-1]
    

/delete

@router.get("/delete{id}")
async def delete(id):
    """
    Delete the record
    """
    resp = del_records(id)
    return {"response": resp}


def del_records(id):
    with open("core/store/store.json") as store_file:
        store = json.load(store_file)
    for info in store.get("students"):
        if info.get("id") == id:
            store.get("students").remove(info)
            break
    with open("core/store/store.json", "w") as outfile:
        json.dump(store, outfile)
    return f"deleted id {id}"

Dockerfile

FROM python:3.8

WORKDIR /app

COPY requirements.txt /app/requirements.txt

RUN python -m pip install --upgrade pip && \
    python -m pip install --no-cache-dir -r /app/requirements.txt

COPY . /app

EXPOSE 4558

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "4558", "--workers", "4"]

Docker version

>> docker --version
Docker version 20.10.12, build e91ed57

Answer 1

You're seeing missing writes since you're running with multiple workers (--workers 4), and your code is prone to race conditions.

Consider the case where you have two workers running e.g. create_records, here described as a primitive concurrency diagram, time running downwards:

Worker 1	Worker 2
receive request	...
open json (0 entries: [])	receive request
add data [123]	open json (0 entries: [])
save json ([123])	add data [456]
...	save json ([456])
receive request	...
open json (1 entries: [456]) ?	...

Only 456 will have ended up in the final file since worker 2 will not know about 123 by the time it's saving the file.

Your options are basically to:

• Lock the file while it's being modified, leading to only one worker being able to manipulate the file at any given time (depending on what you're doing, this could be okay)
• Switch to a more robust storage backend
  • For instance, separate JSON files per record (i.e. your database is a directory of JSON files) can be atomically manipulated.
  • If that's too cumbersome, Sqlite would be a decent next step.