On a WordPress site of mine, I was perplexed as to why it was not getting indexed in Google or other search engines.
I verified my site in Google Webmaster Tools and tried to submit a sitemap. In the webmaster tools panel, it informed me that my robots.txt file was blocking Google from grabbing my sitemap.
This confused me, as I know I would never knowingly make a robots.txt file to block search engines.
Read on to find out what the problem was, how to fix it, and how it happened in the first place…
I checked out my robots.txt file and saw the following:
As if disallowing all bots wasn’t enough, I saw all the pages of my site had the following meta tag in it.
<meta name='robots' content='noindex,nofollow' />
Okay, so now my SEO was killed on two different levels – search engines can’t index or follow links, if they could somehow first get past the robots.txt.
How did I remedy this crucial mistake? It’s a simple fix.
I navigated to my privacy settings in the WordPress administration panel and selected I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers.
When installing WordPress, make sure you don’t deselect the checkbox that reads: Allow my blog to appear in search engines like Google and Technorati.
You should always verify your site(s) in Google Webmaster Tools to see if there are any problems with indexing. I hope you found this quick WordPress tip informative, because although this is a simple mistake, it’s something to be aware of when SEO’ing WordPress sites.
Looking for web hosting?
Question: What's the best type of web hosting?
Answer: The kind you don't have to worry about.
And that's precisely why I recommend HostGator. I've been using them myself for years and they've been nothing short of superb.
Fast load times, great support, WordPress-friendly, and good prices. Not much more you can ask for.
New customers can get 25% off of any HostGator web hosting package with the coupon: themelab25percentoffNote: I am a HostGator affiliate, but that doesn't make my recommendation any less genuine.