Deny access to /lib but allow access to /assets/lib

Question

I'd like to create an nginx rule to deny access (with a 404 so that the attacker doesn't think it exists) to a root directory /lib, but I do not want to include all directories with the word lib such as /assets/lib.

So in other words:

https://www.example.com/lib/* = should be all 404 ...

https://www.example.com/assets/lib/* = should be OK

location ~ /lib {
    deny all;
    return 404;
}

The above works to block access to all directories that have the word "lib" in them, but that also blocks my "/assets/lib".

Thanks!