Python asyncio how to give priority to one function during concurrent execution of its subfunctions

Question

Here is a schematic version of my current setup:

import asyncio

async def child(args):
    # await some things

async def parent(args):
    # await some other things
    if some_condition:
        await asyncio.gather(child(args1), child(args2))

async def main():
    await asyncio.gather(parent(args3), parent(args4))

asyncio.run(main())

So in essence I'm running the parent function for different inputs concurrently. Every now and then some condition is met and as a result the corresponding parent process starts the concurrent execution of the child function for different inputs. Since the child function contains an await, it gives up control to the event loop such that other functions within this event loop can run.

My problem is that I only want the child processes to give up control in favor of other child processes of the same parent process (i.e. essentially blocking the other parent processes from running). However, in practice this is not the case and other parent processes are often running before the child processes are finished. This increases the time it takes for the child processes to complete (I actually have more than 2 parent processes, I'm just limiting it here for simplicity).

Is there a way to give priority to the child processes, such that they are completed as fast as possible? I tried a bunch of different asyncio methods but nothing seems to work. Maybe there's an easy way to run the code inside the if statement in a synchronous way, such that it blocks the event loop (while still running the child processes concurrently)?

Thanks in advance!

Answer 1

You can use an asyncio.Lock. With a lock you can simulate a shared resource between coroutines, locking the access to a coroutine if already other owns the lock. In such way, can be implemented that one parent cannot invoke his childs if other parent is awaiting by the execution of his own childs.

import asyncio
from time import time

# lock for parents when want to call childs
lock = asyncio.Lock()

async def child(parent, job):
    print(f"child of {parent} doing {job} at {time()}")
    await asyncio.sleep(3)

async def parent(name, jobs):
    print(f"parent {name} doing stuff")
    await asyncio.sleep(2)
    async with lock:    # only the childs of one parent will run concurrently
        await asyncio.gather(child(name, jobs[0]), child(name, jobs[1]))

async def main():
    await asyncio.gather(
        parent("parent1", ["job1", "job2"]),
        parent("parent2", ["job3", "job4"])
    )

asyncio.get_event_loop().run_until_complete(main())

Output:

parent parent1 doing stuff
parent parent2 doing stuff
child of parent1 doing job1 at 1650394577.7973263
child of parent1 doing job2 at 1650394577.7974186
child of parent2 doing job3 at 1650394580.801368
child of parent2 doing job4 at 1650394580.8014803

Note how the childs of parent2 started to do his jobs 3 seconds after the childs of parent1 ended.

Without the lock you get something like this, every child starts at the same time.

parent parent1 doing stuff
parent parent2 doing stuff
child of parent1 doing job1 at 1650395040.6458762
child of parent1 doing job2 at 1650395040.6460035
child of parent2 doing job3 at 1650395040.6460583
child of parent2 doing job4 at 1650395040.6460981

Hope this answer your quetion.