'What is the smartest way to handle robots.txt in Express?

I'm currently working on an application built with Express (Node.js) and I want to know what is the smartest way to handle different robots.txt for different environments (development, production).

This is what I have right now but I'm not convinced by the solution, I think it is dirty:

app.get '/robots.txt', (req, res) ->
  res.set 'Content-Type', 'text/plain'
  if app.settings.env == 'production'
    res.send 'User-agent: *\nDisallow: /signin\nDisallow: /signup\nDisallow: /signout\nSitemap: /sitemap.xml'
  else
    res.send 'User-agent: *\nDisallow: /'

(NB: it is CoffeeScript)

There should be a better way. How would you do it?

Thank you.

node.jsexpressrobots.txt


Solution 1:[1]

Use a middleware function. This way the robots.txt will be handled before any session, cookieParser, etc:

app.use('/robots.txt', function (req, res, next) {
    res.type('text/plain')
    res.send("User-agent: *\nDisallow: /");
});

With express 4 app.get now gets handled in the order it appears so you can just use that:

app.get('/robots.txt', function (req, res) {
    res.type('text/plain');
    res.send("User-agent: *\nDisallow: /");
});

Solution 2:[2]

1. Create robots.txt with following content :

User-agent: *
Disallow: # your rules here

2. Add it to public/ directory.

3. If not already present in your code, add:

app.use(express.static('public'))

Your robots.txt will be available to any crawler at http://yoursite.com/robots.txt

Solution 3:[3]

Looks like an ok way.

An alternative, if you'd like to be able to edit robots.txt as regular file, and possibly have other files you only want in production or development mode would be to use 2 separate directories, and activate one or the other at startup.

if (app.settings.env === 'production') {
  app.use(express['static'](__dirname + '/production'));
} else {
  app.use(express['static'](__dirname + '/development'));
}

then you add 2 directories with each version of robots.txt.

PROJECT DIR
    development
        robots.txt  <-- dev version
    production
        robots.txt  <-- more permissive prod version

And you can keep adding more files in either directory and keep your code simpler.

(sorry, this is javascript, not coffeescript)

Solution 4:[4]

Here is what I use

router.use('/robots.txt', function (req, res, next) {
  res.type('text/plain')
  res.send(
    `User-agent: *
     Disallow: /admin`);
});

Solution 5:[5]

For choosing the robots.txt depending the environment with a middleware way:

var env = process.env.NODE_ENV || 'development';

if (env === 'development' || env === 'qa') {
  app.use(function (req, res, next) {
    if ('/robots.txt' === req.url) {
      res.type('text/plain');
      res.send('User-agent: *\nDisallow: /');
    } else {
      next();
    }
  });
}

Solution 6:[6]

This is what I did on my index routes. You can just simply write down in your codes what I does given down below.

router.get('/', (req, res) =>
    res.sendFile(__dirname + '/public/sitemap.xml')
)

router.get('/', (req, res) => {
    res.sendFile(__dirname + '/public/robots.txt')
})

Solution 7:[7]

I use robots.txt as a normal file for Prod, and a middleware for other envs.

if(isDev || isStaging){
    app.use('/robots.txt', function (req, res) {
        res.type('text/plain');
        res.send("User-agent: *\nDisallow: /");
    });
}
app.use(express.static(path.join(__dirname, 'public')));

Solution 8:[8]

app.use(express.static('public'))
app.use('/images', express.static('public/images'))
app.use('/videos', express.static('public/videos'))

enter image description here

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 Exadra37
Solution 3 Pascal Belloncle
Solution 4 Anirudh
Solution 5 fernandopasik
Solution 6 Chen Lay
Solution 7 Mahmoud
Solution 8 ????????? ???????

Related Questions