I'm attempting to scrape a website and these two rules seem to be contradictory in robots.txt User-agent: * Disallow: * Allow: / Does Allow: / mean that I can
I want to use some variables in my robots.txt file in Express. How can I render this file as an EJS template? I already have EJS working for .html files. app.