Any good ideas to fix “connect() failed (111: Connection refused)” error when restarting nodejs server?

Question

Background:

I use a Nginx+NodeJS structure to run the website. The server has quite some traffic like 300 concurrent people online. Visiting all kind of pages. And I use pm2 to manage my node apps.

Problem:

However, when I restart the node server with pm2 restart xxx, in a short duration (like 15 seconds), the users will encounter a 502 error. And accordingly, there is “connect() failed (111: Connection refused)” in the log.

According to one other question on SO.

A 502 Bad Gateway error usually suggests that the proxy (Nginx in NodeJS's case) can't find a destination to route the traffic to.

So I guess the error is occurring because of the moment that a user requests the server while the Node hasn't ready for its business. So my Nginx couldn't "contact" my nodejs and threw a 502 error.

Is there any way to fix this?

Answer 1

If you want to continue serving your users while you restart your Node.js server, you need a second Node.js server. More precisely:

• Before the restart, node myapp.js is running and listening on port A. Nginx routes traffic to port A.
• Now you can start a second Node.js server, probably on a newer version of your app, node mynewapp.js, that listens on port B. While you do that, traffic is still routed to port A.
• Once node mynewapp.js is up and running, you switch Nginx so that it routes traffic to port B.
• Allow a grace period for requests on port A to finish, then you can shut down the node myapp.js process.

Note two potential pitfalls with this approach:

• Long running requests on port A would prevent you from shutting down the "old" Node.js server.
• Requests that leave a state in the Node.js server (in global Javascript variables, say), would lose that state when you switch over to the other Node.js server. But (session) states that you write to a database will survive.