Added new package to PyPI - How to support uninstalling using pip

Question

I just added a new package to PyPI which you can install using:

pip install afori-utils

When i'm trying to uninstall it using:

pip uninstall afori-utils

I get:

Found existing installation: afori-utils 1.0.0
Uninstalling afori-utils-1.0.0:
  Would remove:
    c:\users\dor00\pycharmprojects\square\venv\lib\site-packages\*
  Would not remove (might be manually added):
    c:\users\dor00\pycharmprojects\square\venv\lib\site-packages\~umpy\typing\tests\data\reveal\warnings_and_errors.pyi
    c:\users\dor00\pycharmprojects\square\venv\lib\site-packages\~umpy\typing\tests\test_generic_alias.py
    c:\users\dor00\pycharmprojects\square\venv\lib\site-packages\~umpy\typing\tests\test_isfile.py
    ...
    ...
    ... Lots of other stuff, doesn't seem relevant.

The problem is that the part of "Lots of other stuff" takes a long time to load (looks like it just lists all of my site-packages directory). With other packages (e.g numpy) that I try to uninstall this doesn't happen and there is no "would not remove" part. How can I make my uninstall more like numpy's?

This is my setup.py file:

from setuptools import setup

setup(
    name='afori-utils',
    version='1.0.1',
    description='Some utils by Dor Miron',
    package_dir={'afori_utils': 'afori_utils'},
    packages=['afori_utils'],
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: GNU General Public License (GPL)"
    ],

    install_requires=[
        "numpy>=1.20",
        "matplotlib>=3.4"
    ],
)

Thank you.