Symfony oneup league flysystem file not found

Question

I am uploading files using oneup/flysystem-bundle and vich/uploader-bundle and that works fine.

When trying to delete a file with

 $this->filesystem->delete($path)

it throws error saying that file not found, although the path is correct.

This question suggests that this may be due $this->filesystem using a relative path.

That may be the case, but relative to what?

Initially I used $path as being the full absolute path. Then I tried a few variants of relative path, but nothing worked.

I know I could just use unlink, but I want to understand this.

This how the config file looks like:

oneup_flysystem:
  adapters:
    category_image:
        local:
            directory: "%kernel.project_dir%/public/images/category"
  filesystems:
    category_image_filesystem:
        adapter: category_image
        mount: category_image_filesystem

EDIT: Solution as proposed by Bossman

on config:

    category_image:
        local:
            directory: "%kernel.project_dir%/public/images/category"
            permissions:
                file:
                    public: 0o644
                    private: 0o600
                dir:
                    public: 0o755
                    private: 0o700

on controller:

      $filename = $oldImage->getFilename();
        if ($filename && $this->filesystem->has($filename)) {
            $this->filesystem->delete($filename);
        }