SEO Hosting 101 – Don’t Leak Your Staging URLs into Google!

Your web design company has sold you an “advanced” SEO deal with your brand new website and hosting package and everything looks perfect. The site architecture, url handling, migration redirects and everything else SEO is apparently perfect. You go live, safe in the knowledge of a job well done. Months later, nothing. You ranked better before the new site went live!

What went wrong? Sometimes duplicate content problems are not at all easy to find and here’s a classic example. A group of recruitment websites leaking their staging server urls into Google’s, Yahoo’s and Live’s index. In live’s case, the problem is quite severe with more than 6000 indexed urls at the staging site hosting.hotlizard.co.uk.

The moral of the story? There are some highly important checks to carry out and fixes to implement when you’re migrating a website to a different hosting / web design agency. These are tests related to the integrity of your hosting setup rather than the usual “make sure all old urls 301 redirect” stuff.

Here we go:

1) Can bots access the site at the test url? Why not specify the IP range from which access to your test urls will be allowed by the firewall. Most likely, your office and / or your web design agency office will do. That way a crawler will be blocked straight away.

2) Have you leaked some urls already? Check by doing a site:”your-test-url” query in Google. If there’s a problem, detect the incoming user agent on the staging site URLS and redirect appropriately. If your visitor is a search engine crawler, do the right thing. Canonicalise! For example: http://ellisfairbank.hotlizard-hosting.co.uk/transportation should 301 redirect to: http://www.ellisfairbank.co.uk/transportation/

3) Insert a robots.txt file at the root of the url you’ve been testing to disallow all spiders

4) Use a noindex,nofollow in the test platform meta code. Don’t forget to change this when you migrate the live site!

5) Just to be on the safe side, you could register your test domain at Google webmaster tools and totally remove the site from Google’s index. I prefer 301s myself.

One last thing. Make sure every content url in your new site has been changed to the correct live url prior to migration. If that old test url isn’t anywhere on the site, you might be less at risk. ;-)

Comments

  1. g1smd

    I often modify the PHP so that if the wrong domain is called the PHP adds the noindex tag to the page.

    I then don’t have to remember to add or remove anything further.

  2. AK Works

    Great post. Not enough people know or think about this issues; SEOs, web designers alike – all seem to forget about this issue.

    I worked for a real estate website company one time. I was doing SEO for the clients and found out later that their DEV servers were completely indexed by the search engines. They had dupes of all of their clients. It was pretty shameful.

  3. Jim Gaudet

    Good info, thanks.
    g1smd – that is a good idea with the PHP code. I believe I will start implementing that into my websites.

    ~ Jim

  4. g1smd

    I also like to redirect using .htaccess as that fixes the URL to the right domain should anyone copy the URL from their browser, and paste it into the content of some other site as a link.

    However, I always have the PHP option installed too. This acts as a backup in case the .htaccess file goes AWOL on a server move or somesuch. More likely, it helps if there is a URL combination that can access the site, but it is a combination that I am not aware of (usual things are bare IP address, wild-card sub-domains off the main domain, and both sub-domains and folders off the hosting company domain name), as it is easy to miss one if someone else has set the server up.

  5. radoo

    Great article! Very useful.
    Some sites also have problems with the canonical links (with / and without it at the end). This one should be checked also when migrating from a known hosting to an unpredictable one :)

  6. Paul

    Hi Richard, great post, good to see you have a blog these days, I will keep an eye out for new posts!

    That is unfortunate for Hot Lizard that you found those Dev URLs… It makes them look quite bad, especially as they offer an SEO service on those recruitment sites of theirs!

  7. SEO Consultant Joel

    Great post. I have seen this happen many times. Duplicate content is a big no no in the Goog’s Guidelines. Also you want to be careful if you have a lot of good links to URL’s and then change the URL. Eeeekkk!