Redis WATCH MULTI EXEC by one client

Question

I am using NodeJS + Express + Redis on RedisOnGo + node_redis as a client. I expect a lot of concurrency, so trying to test WATCH. This example won't contain Express, just necessary stuff.

var redis = require("redis")
var rc = redis.createClient(config.redis.port, config.redis.host)

rc.auth(config.redis.hash, function(err) {
    if (err) {
        throw err
    }
})

rc.on('ready', function () {
    rc.set("inc",0)
    for(var i=1;i<=10;i++){
        rc.watch("inc")
        rc.get("inc",function(err,data){
            var multi = rc.multi()
            data++ // I do know I can use rc.incr(), this is just for example
            multi.set("inc",data)
            multi.exec(function(err,replies){
                console.log(replies)
            })
        })
    }
})

Expecting result: getting N errors in exec callbacks and finally getting "inc" variable = 10-N.

Unexpected result: getting 0 errors in exec callbacks but finally getting "inc" variable = 1.

Watch doesn't work with my code.

I have found this thread redis and watch + multi allows concurrent users. They say it is because of the only redis client.

Then I found this thread Should I create a new Redis client for each connection?. They say that generating a new client for each transaction "is definitely not recommended". I am lost.

Please also note, that I have to authenticate to Redis server. Thanks in advance!

EDITION 1:

I was able to make it work using local Redis instance (so I do not use client.auth) by creating a new client connection before each WATCH-MULTI-EXEC iteration. Not sure if it is good though, but results now are 100% accurate.

EDITION 2 Made it work if I create a new client connection before each WATCH-MULTI-EXEC iteration and then do client.auth and wait for client.on.

The question still exists, is it OK that I create new client connections for each iteration?

Answer 1

I got your question finally.

If you want to test WATCH for concurrency, I think you need to change your code. as we know. WATCH only monitor changing of value, not getting value operation. so in your current code, all your get command will be executed successfully and get 0, then they will set inc to 1. all the set value are the same (1), so watch won't fail.

In the case, we need to ensure not only write operation is protected, but also read. before you set inc, you need to watch and modify another key which is as a pessimistic lock, and then we can get and change inc. in this way, it will make sure your expectation.

rc.set("inc",0)
for(var i=1;i<=10;i++){
    rc.watch("inc-lock")
    rc.get("inc",function(err,data){
        var multi = rc.multi()
        data++
        multi.incr("inc-lock")
        multi.set("inc",data)
        multi.exec(function(err,replies){
            console.log(replies)
        })
    })
}

I tested it in my PC.

[2013-11-26 18:51:09.389] [INFO] console - [ 1, 'OK' ]

[2013-11-26 18:51:09.390] [INFO] console - [ 2, 'OK' ]

[2013-11-26 18:51:09.390] [INFO] console - [ 3, 'OK' ]

[2013-11-26 18:51:09.390] [INFO] console - [ 4, 'OK' ]

[2013-11-26 18:51:09.391] [INFO] console - [ 5, 'OK' ]

[2013-11-26 18:51:09.391] [INFO] console - [ 6, 'OK' ]

[2013-11-26 18:51:09.392] [INFO] console - [ 7, 'OK' ]

[2013-11-26 18:51:09.392] [INFO] console - [ 8, 'OK' ]

[2013-11-26 18:51:09.393] [INFO] console - [ 9, 'OK' ]

[2013-11-26 18:51:09.393] [INFO] console - [ 10, 'OK' ]

Answer 2

If you want to use transaction/atomic MULTI operations but you want to do so using a shared connection, as far as I know your only option is using LUA.

I use LUA scripting within redis for a number of things, and the thing with LUA is the whole script will execute atomically, which is quite convenient. You have to be aware though that this means if you have a slow LUA script you are making redis slow for everyone using your server.

Also, when using LUA even if you can operate on different keys, be aware if you use more than one key in your script you won't be able to use Redis cluster once it is released. This is due to, when using a cluster, keys will be distributed to different Redis processes so your LUA script might not have access to all of them on a single server.

In any case, the problem with redis cluster would be the same when issuing a MULTI, since MULTI won't be allowed to set different keys on a cluster.

Cheers,

j

Answer 3

I found executeIsolated(callback) via

https://github.com/redis/node-redis/blob/master/docs/isolated-execution.md

Seems like the right solution today.