'Request TorchServe service inside a containerized app

I'm creating a text generation app, and I want to use TorchServe to make it scale well

In my DockerFile, I first run a CMD to run TorchServe, then run the app with ENTYPOINT

...
CMD ["torchserve", "--start", "--model-store", "/model_store", "--ts-config", "/config.properties"]
ENTRYPOINT ["python3", "-X", "utf8", "app.py"]

And in my app.py I request the local TorchServe url to get my predictions

TORCH_URL = "http://127.0.0.1:8989/predictions/"

def request_torch(artist_name, input_text):
    """
    Call torchserve service to get inference
    """
    output = requests.post(TORCH_URL+ artist_name,
            data = {'body':input_text})

    decoded_output = output.content.decode('utf-8')
    return decoded_output

Problem is : when deploying my app on gcloud, I get a "connection refused" error

NewConnectionError('<urllib3.connection.HTTPConnection object at 0x3e6cfe0cb7b8>: Failed to establish a new connection: [Errno 111] Connection refused'

Is it possible to call the TorchServe service locally inside such an app ?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source