nginx ignore extension location for subdirectory

Question

I have a nginx block for a wordpress installation, and a subdirectory which redirects to an app I have running. I want .php urls that go to the subdirectory to not be evaluated by fastcgi, since the app has it's own special system for evaluating them. However, php URLs still go to the fastcgi block and return a 404.

Nginx configuration:

server {
        listen 0.0.0.0:443 http2 default_server;
        listen [::]:443 http2 default_server;
        ssl on;
        ssl_certificate /etc/nginx/certificate.pem;
        ssl_certificate_key /etc/nginx/certificate.key;
    location = /favicon.ico { log_not_found off; access_log off; }
    location = /robots.txt { log_not_found off; access_log off; allow all; }
    location ~* \.(css|gif|ico|jpeg|jpg|js|png)$ {
        expires max;
        log_not_found off;
    }
        root /var/www/wordpress;
        index index.html index.php;
    location / {
        try_files $uri $uri/ /index.php$is_args$args;
    }
    location /my_app {
        proxy_pass http://localhost:8081/;
    }
  location ~ \.php$ {
    fastcgi_pass unix:/var/run/php/php-fpm.sock;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
  }
}

I want /index.php to be evaluated by fastcgi. I want /my_app/index.php to be proxy_pass'd to my app.