Previous | NextZekebits | Blog

Keeping our new website under wraps

Wayne McMaster

8 January 2016

If you are developing your new website on Wordpress it is likely that your developer in allowing you access to the site while it is being developed and populated, in doing so it probably necessary for them to put it on a live publicly accessible domain instead of creating a local development environment or creating a development site.

The most effective way to block search engines from a live website is by password protecting your entire WordPress site on the server level. This means when anyone accesses your website they are asked to provide a username and password even before they reach WordPress. This includes search engines as well. Upon login failure, they are shown 401 error and the bots turn away.

However, that may not be the route chosen by your developer and you are left with a live In that case they must make sure that 1) Robots Access: Failing to Prevent Robots from Exploring the New Website While Under Development

A robots.txt file with Disallow: / (all) for all user agents.

These lines ask robots (web crawlers) not to index your pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect this, some page or random image from your site may get indexed.

If we don't prevent search engines robots from crawling, the version of the website under development can be indexed - with development URLs - , and potentially generate duplicates of the current website. You'll know if it already happened by typing a site:[domain] command in Google, with your pre-production domain and see if there are results. It's not too late to act (disallow in robots.txt, at least).

Robots Access: Failing to Allow Access When the New Website Goes Online

Of course, the new website won't get indexed (oops!).
Silly as it may sound, everybody is so busy at launch that it happens. You wouldn't be the first to forget to update your robots.txt file when the new website goes online.

Leave a comment
Subscribe to Latest Updates
Thank you for subscribing!

Decor14 Finalist

Top 500 Social Media Agency

Latest tweets