Drupal mobile website/multisite installation and robots.txt

As I’m working on getting a mobile version of this site working, I ran into a problem with having a robots.txt file on a Drupal multisite installation. The root of the problem is that you need to have a robots.txt file like this on your mobile site:

User-agent: *
Disallow: /

That’s to keep the search engines from scanning and storing that content, which will be a duplicate of your main website.

I first tried to use the Drupal RobotsTxt module, but that wouldn’t work because my main website and the mobile website share the same database; something that plugin doesn’t account for.

Solving the problem with Apache rewrite rules

In the end I solved the problem using some more Apache RewriteRule lines:

RewriteCond %{HTTP_HOST} ^alvinalexander\.com$ [NC]
RewriteRule ^robots\.txt$ robots.txt.aa [L]

RewriteCond %{HTTP_HOST} ^m\.alvinalexander\.com$ [NC]
RewriteRule ^robots\.txt$ robots.txt.mobile [L]

I found that solution at this Drupal page. Those lines take care of redirecting requests for the robots.txt file on my two domains. If the request is on my main website they will serve up the robots.txt.aa file, and if the request is on my mobile website, those lines will return the file named robots.txt.mobile. This is needed because the two domains share the same root directory.

Add new comment

Anonymous format

  • Allowed HTML tags: <em> <strong> <cite> <code> <ul type> <ol start type> <li> <pre>
  • Lines and paragraphs break automatically.