how to make a thread-safe global counter in python

Question

I'm creating a threading.Timer(2,work) run threads. Inside each work function, upon some condition the global counter must increment without conflict for access of counter variable among the spawned work threads.

I've tried Queue.Queue assigned counter as well as threading.Lock(). Which is a best way to implement thread-safe global increment variable.

Previously someone asked question here: Python threading. How do I lock a thread?

Answer 1

Not sure if you have tried this specific syntax already, but for me this has always worked well:

Define a global lock:

import threading
threadLock = threading.Lock()

and then you have to acquire and release the lock every time you increase your counter in your individual threads:

with threadLock:
    global_counter += 1

Answer 2

One solution is to protect the counter with a multiprocessing.Lock. You could keep it in a class, like so:

from multiprocessing import Process, RawValue, Lock
import time

class Counter(object):
    def __init__(self, value=0):
        # RawValue because we don't need it to create a Lock:
        self.val = RawValue('i', value)
        self.lock = Lock()

    def increment(self):
        with self.lock:
            self.val.value += 1

    def value(self):
        with self.lock:
            return self.val.value

def inc(counter):
    for i in range(1000):
        counter.increment()

if __name__ == '__main__':
    thread_safe_counter = Counter(0)
    procs = [Process(target=inc, args=(thread_safe_counter,)) for i in range(100)]

    for p in procs: p.start()
    for p in procs: p.join()

    print (thread_safe_counter.value())

The above snippet was first taken from Eli Bendersky's blog, here.

Answer 3

If you're using CPython¹, you can do this without explicit locks:

import itertools

class Counter:
    def __init__(self):
        self._incs = itertools.count()
        self._accesses = itertools.count()

    def increment(self):
        next(self._incs)

    def value(self):
        return next(self._incs) - next(self._accesses)

my_global_counter = Counter()

We need two counters: one to count increments and one to count accesses of value(). This is because itertools.count does not provide a way to access the current value, only the next value. So we need to "undo" the increments we incur just by asking for the value.

This is threadsafe because itertools.count.__next__() is atomic in CPython (thanks, GIL!) and we don't persist the difference.

Note that if value() is accessed in parallel, the exact number may not be perfectly stable or strictly monotonically increasing. It could be plus or minus a margin proportional to the number of threads accessing. In theory, self._incs could be updated first in one thread while self._accesses is updated first in another thread. But overall the system will never lose any data due to unguarded writes; it will always settle to the correct value.

¹ Not all Python is CPython, but a lot (most?) is.

² Credit to https://julien.danjou.info/atomic-lock-free-counters-in-python/ for the initial idea to use itertools.count to increment and a second access counter to correct. They stopped just short of removing all locks.