On a WordPress site of mine, I was perplexed as to why it was not getting indexed in Google or other search engines.
I verified my site in Google Webmaster Tools and tried to submit a sitemap. In the webmaster tools panel, it informed me that my robots.txt file was blocking Google from grabbing my sitemap.
This confused me, as I know I would never knowingly make a robots.txt file to block search engines.
Read on to find out what the problem was, how to fix it, and how it happened in the first place…
The Problem
I checked out my robots.txt file and saw the following:
User-agent: *
Disallow: /
As if disallowing all bots wasn’t enough, I saw all the pages of my site had the following meta tag in it.
<meta name='robots' content='noindex,nofollow' />
Okay, so now my SEO was killed on two different levels - search engines can’t index or follow links, if they could somehow first get past the robots.txt.
The Fix
How did I remedy this crucial mistake? It’s a simple fix.
I navigated to my privacy settings in the WordPress administration panel and selected I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers.
Prevention
When installing WordPress, make sure you don’t deselect the checkbox that reads: Allow my blog to appear in search engines like Google and Technorati.
Conclusion
You should always verify your site(s) in Google Webmaster Tools to see if there are any problems with indexing. I hope you found this quick WordPress tip informative, because although this is a simple mistake, it’s something to be aware of when SEO’ing WordPress sites.
Good reminder for those that don’t know about this and those that might have turned it off on purpose but forgot to turn it back on. I do like that WordPress thought to include a method to allow and disallow for folks that want to keep their blog more personal. Still - good reminder
That’d do it.
Yes, I also agree for a good reminder. That can be a helpful thing for all new bloggers.
Hahaha…
Sometimes even a simple thing like that will ruined everything. I did the same mistake last time.
For better coverage, beside submitting your website to Google Webmaster Tool, we MUST also submit our site to Site Explorer(Yahoo) and Webmaster Central(Live Search - MSN).
And other major search engine to consider is ask.com, but I’m not sure if it have webmaster tools feature…
Cool tip!
Some people who make private blogs can keep their blogs private by enabling it, but for bloggers like you and me, we need more hits from Search Engines!!!
WordPress let you do anything!!! Thats what I love with WordPress
Thanks for the tip.
Yes ,i also think that such type of reminder will keep us away from this type of problem because this small mistakes create big problems.
Funny…I had written also written something about robots.txt sometime back - check it out if you want: http://www.myqute.com/2008/06/18/banning-search-engines-construction/
Sorry - link down as I find a way to fix wp!
Yes, I realize some people may prefer to block search engines for various reasons. This reminder was for people like me who may have accidentally blocked them, and then forgot about it.
@Pangeran: Yep, you really can’t forget about other big search engines like Yahoo and Live. I used Google Webmaster Tools as an example since they seem to have a good “reporting” feature if there’s something wrong with your site.
@kelly: Nice article, but there seems to be a problem with your site. There are a few PHP warning errors at the top.
Thanks for the comments everyone.
@Leland, thanks again! Yeah, saw the errors (been there for almost 8days!) and I have not done anything yet to correct the problem. >,< GUILTY! lol
Great point. Sometime pple get too caught up with the whole post frenzy they neglect important settings in WP.