Can I parallelise a Python function that returns None with Ray?

Question

I have a function that takes in a filename and creates a symlink in a target directory if that filename exists in the "source" directory:

def _symlink_photos(photo_name: str, photos_dir: Path, target_dir: Path) -> None:
    """
    Given a string, checks if it exists in `photos_dir`,
    then creates a symlink in `target_dir`
    """

    photo_path: Path = photos_dir / photo_name
    if photo_path.exists():
        (target_dir / photo_name).symlink_to(photo_path)

This function is meant to be used to iterate over a somewhat big list of filenames (some 70k), like so:

def main() -> None:
    """
    Given a list of photos, checks if they exist in the source directory,
    and symlinks them into the target directory.
    """

    for photo_name in tqdm(photo_list):
        _symlink_photos(photo_name, photo_dir, target_dir)

The brute-force/for-loop approach is taking me too long (tqdm estimates over a day), so I was wondering how to parallelise this job. I knew I could use ray, but in order to use it I would need to write my code like this:

futures = [_symlink_photos.remote(photo_name, photo_dir, target_dir) for photo in photo_list]
result = ray.get(futures)

Yet this does not make sense to me, as my function does not return anything. Can I still use ray? Should I switch to something else, like joblib/threading?

Another alternative I was thinking of is to parallelise the "check if photo_name exists in the source dir" part with ray, i.e. to retrieve the list of photos that actually are in the target directory. Only then I would create the symlinks, hopefully taking less time...

Answer 1

You can use Ray in the way that you described. The ray.get call will simply return a list of None values, which you can ignore.

You can also look up ray.wait which can be used to wait for certain tasks to finish without actually retrieving the task outputs.