How can I emit Flask-SocketIO requests with callbacks that still work after a user rejoins and their sid changes?

Question

Summarize the Problem

I am using Flask-SocketIO for a project and am basically trying to make it so that users can rejoin a room and "pick up where they left off." To be more specific:

1. The server emits a request to the client, with a callback to process the response and a timeout of 1 second. This is done in a loop so that the request is resent if a user rejoins the room.
2. A user "rejoining" a room is defined as a user joining a room with the same name as a user who has previously been disconnected from that room. The user is given their new SID in this case and the request to the client is sent to the new SID.

What I am seeing is this:

1. If the user joins the room and does everything normally, the callback is processed correctly on the server.

2. It a user rejoins the room while the server is sending requests and then submits a response, everything on the JavaScript side works fine, the server receives an ack but does not actually run the callback that it is supposed to:

   uV7BTVtBXwQ6oopnAAAE: Received packet MESSAGE data 313["#000000"]
   received ack from Ac8wmpy2lK-kTQL7AAAF [/]
   

This question is similar to mine but the solution for them was to update Flask-SocketIO and I am running a version newer than theirs: python flask-socketio server receives message but doesn't trigger event

Show Some Code

I have created a repository with a "minimal" example here: https://github.com/eshapiro42/socketio-example.

In case something happens to that link in the future, here are the relevant bits:

# app.py

from gevent import monkey
monkey.patch_all()

import flask_socketio
from collections import defaultdict
from flask import Flask, request, send_from_directory

from user import User


app = Flask(__name__)
socketio = flask_socketio.SocketIO(app, async_mode="gevent", logger=True, engineio_logger=True)


@app.route("/")
def base():
    return send_from_directory("static", "index.html")

@app.route("/<path:path>")
def home(path):
    return send_from_directory("static", path)

# Global dictionary of users, indexed by room
connected_users = defaultdict(list)
# Global dictionary of disconnected users, indexed by room
disconnected_users = defaultdict(list)


@socketio.on("join room")
def join_room(data):
    sid = request.sid
    username = data["username"]
    room = data["room"]
    flask_socketio.join_room(room)
    # If the user is rejoining, change their sid
    for room, users in disconnected_users.items():
        for user in users:
            if user.name == username:
                socketio.send(f"{username} has rejoined the room.", room=room)
                user.sid = sid
                # Add the user back to the connected users list
                connected_users[room].append(user)
                # Remove the user from the disconnected list
                disconnected_users[room].remove(user)
                return True
    # If the user is new, create a new user
    socketio.send(f"{username} has joined the room.", room=room)
    user = User(username, socketio, room, sid)
    connected_users[room].append(user)
    return True

    
@socketio.on("disconnect")
def disconnect():
    sid = request.sid
    # Find the room and user with this sid
    user_found = False
    for room, users in connected_users.items():
        for user in users:
            if user.sid == sid:
                user_found = True
                break
        if user_found:
            break
    # If a matching user was not found, do nothing
    if not user_found:
        return
    room = user.room
    socketio.send(f"{user.name} has left the room.", room=room)
    # Remove the user from the room
    connected_users[room].remove(user)
    # Add the user to the disconnected list
    disconnected_users[room].append(user)
    flask_socketio.leave_room(room)


@socketio.on("collect colors")
def collect_colors(data):
    room = data["room"]
    for user in connected_users[room]:
        color = user.call("send color", data)
        print(f"{user.name}'s color is {color}.")
    

if __name__ == "__main__":
    socketio.run(app, debug=True)

# user.py

from threading import Event # Monkey patched

class User:
    def __init__(self, name, socketio, room, sid):
        self.name = name
        self.socketio = socketio
        self.room = room
        self._sid = sid

    @property
    def sid(self):
        return self._sid

    @sid.setter
    def sid(self, new_sid):
        self._sid = new_sid

    def call(self, event_name, data):
        """
        Send a request to the player and wait for a response.
        """
        event = Event()
        response = None

        # Create callback to run when a response is received
        def ack(response_data):
            print("WHY DOES THIS NOT RUN AFTER A REJOIN?")
            nonlocal event
            nonlocal response
            response = response_data
            event.set()
      
        # Try in a loop with a one second timeout in case an event gets missed or a network error occurs
        tries = 0
        while True:
            # Send request
            self.socketio.emit(
                event_name,
                data, 
                to=self.sid,
                callback=ack,
            )
            # Wait for response
            if event.wait(1):
                # Response was received
                break
            tries += 1
            if tries % 10 == 0:
                print(f"Still waiting for input after {tries} seconds")

        return response

// static/client.js

var socket = io.connect();

var username = null;
var room = null;
var joined = false;
var colorCallback = null;

function joinedRoom(success) {
    if (success) {
        joined = true;
        $("#joinForm").hide();
        $("#collectColorsButton").show();
        $("#gameRoom").text(`Room: ${room}`);
    }
}

socket.on("connect", () => {
    console.log("You are connected to the server.");
});

socket.on("connect_error", (data) => {
    console.log(`Unable to connect to the server: ${data}.`);
});

socket.on("disconnect", () => {
    console.log("You have been disconnected from the server.");
});

socket.on("message", (data) => {
    console.log(data);
});

socket.on("send color", (data, callback) => {
    $("#collectColorsButton").hide();
    $("#colorForm").show();
    console.log(`Callback set to ${callback}`);
    colorCallback = callback;
});

$("#joinForm").on("submit", (event) => {
    event.preventDefault();
    username = $("#usernameInput").val();
    room = $("#roomInput").val()
    socket.emit("join room", {username: username, room: room}, joinedRoom);
});

$("#colorForm").on("submit", (event) => {
    event.preventDefault();
    var color = $("#colorInput").val();
    $("#colorForm").hide();
    colorCallback(color);
});

$("#collectColorsButton").on("click", () => {
    socket.emit("collect colors", {username: username, room: room});
});

<!-- static/index.html  -->

<!doctype html>

<html lang="en">
    <head>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
        <title>Socket.IO Example</title>
    </head>

    <body>
        <p id="gameRoom"></p>

        <form id="joinForm">
            <input id="usernameInput" type="text" placeholder="Your Name" autocomplete="off" required>
            <input id="roomInput" type="text" placeholder="Room ID" autocomplete="off" required>
            <button id="joinGameSubmitButton" type="submit" btn btn-dark">Join Room</button>
        </form>

        <button id="collectColorsButton" style="display: none;">Collect Colors</button>

        <form id="colorForm" style="display: none;">
            <p>Please select a color.</p>
            <input id="colorInput" type="color" required>
            <button id="colorSubmitButton" type="submit">Send Color</button>
        </form>

        <script src="https://code.jquery.com/jquery-3.3.1.min.js" integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8=" crossorigin="anonymous"></script>
        <script src="https://cdn.socket.io/4.4.1/socket.io.min.js" integrity="sha384-fKnu0iswBIqkjxrhQCTZ7qlLHOFEgNkRmK2vaO/LbTZSXdJfAu6ewRBdwHPhBo/H" crossorigin="anonymous"></script>
        <script src="client.js"></script>
    </body>
</html>

Edit

Steps to Reproduce

1. Start the server python app.py and visit localhost:5000 in your browser.
2. Enter any username and Room ID and click "Join Room."
3. Click "Collect Colors."
4. Select a color and click "Send." The selector should disappear and the server should print out a confirmation.
5. Reload everything.
6. Repeat steps 2 and 3 and copy the Room ID.
7. Exit the page and then navigate back to it.
8. Enter the same username and Room ID as you did in step 6 and click "Join Room."
9. Select a color and click "Send." The selector disappears briefly but then comes back, since the server did not correctly process the response and keeps sending requests instead.

Edit 2

I managed to work around (not solve) the problem by adding more state variables on the server side and implementing a few more events to avoid using callbacks entirely. I would still love to know what was going wrong with the callback-based approach though since using that seems cleaner to me.

Answer 1

The reason why those callbacks do not work is that you are making the emits from a context that is based on the old and disconnected socket.

The callback is associated with the socket identified by request.sid. Associating the callback with a socket allows Flask-SocketIO to install the correct app and request contexts when the callback is invoked.

The way that you coded your color prompt is not great, because you have a long running event handler that continues to run after the client goes aways and reconnects on a different socket. A better design would be for the client to send the selected color in its own event instead of as a callback response to the server.