Very high 'Waiting (TTFB)' time - Opencart 3.0.2.0

Question

I've got an OpenCart website (https://www.haarhuis.shop) which took about 60 seconds to completely load any page of both the customer and admin section. I've managed to bring this down to about 15-20 seconds by increasing the memory_limit in the php.ini file, cleaning up the template files and creating indexes in the MySQL database tables, but 15-20 seconds is still way too much waiting time.

I've used several dev tools to find out what the problem can be but I can't find the solution. 90% of the loading time consists of 'Waiting (TTFB)' time. See the table below:

Event           When        Time        Sum
Redirect        0           0           0
DNS             18          0           0
Connect         18          20          20
TTFB            38          18119       18139
Response        18156       520         18659
DOM             18321       542         19201
Interactive     18793       0           -
Content         18793       36          -

The server has 3 CPU's and 6GB of memory, which should be more than enough. There's a constant usage of about 8% of the memory, and when someone loads the page this will go up to about 40%. The memory_limit is set to 6G, so the problem doesn't lay with the server.

I've debugged the OpenCart source (starting in index.php) and most time (~10 seconds) comes from the last line, start('catalog');(or start('admin'); in the admin directory). The other 5 seconds come from

// VirtualQMOD
require_once('./vqmod/vqmod.php');
VQMod::bootup();

Further debugging shows that the startup/router takes the most time in system/engine/router.php. Now my debugging knowledge stops over here, so is there anyone who can help me out making the website faster?