Removing Supplementals in Wordpress or Other Sites
09. Feb 2007 | 21 Comments
I’ve had a few blogs hit supplementals lately even though they’ve got a pretty decent Inbound Link Authority. Generally whenever this happens I’ll query the site in question with Google’s site: operator to see if I can see anything out of the ordinary.
Well chances are that if you’re using wordpress & you’ve not set up your robots.txt correctly then you’ll be getting heaps of pages indexed that you don’t want.
What don’t you want indexed:
- Your Plugins & Content Folder
- Categories
- Galleries & Images
- Feeds & Trackbacks
What you DO want indexed:
- Posts
- Pages
- Sitemap
Shoemoney had a similar problem a few weeks ago (And most people think he’s an SEO?). What happens in that in order to rank a page in the main index Google must attribute a minimum pagerank value to it. If it falls below the threshold it’ll hit supplemental. So you need to make sure that your site has a good distrubution of pagerank or authority. So if most of your links are hitting your homepage & you’re second level of navigation are your categories then you’re losing heaps of link popularity that should be going to your posts instead.
Ony my blogs I’ve implemented the following robots.txt with reasonable success (50% more pages indexed):
User-Agent: Googlebot
Disallow: /index.php
Disallow: /category/
Disallow: /feed/
Disallow: /feedUser-agent: *
Disallow: /wp-
Disallow: /trackback/
There is a good set of reasoning behind doing this on all of your sites. You should almost be telling the engines which pages you want to rank & which pages you don’t. Removing the pages that are worthless to the engines & only serve as a navigation between different parts of the site increase the relevance & internal linking power of other pages.
I have a Joomla content site that has more category pages than static pages. So I exlcluded all the category pages & my traffic has jumped from around 500 uniques/day to 1500 in about 8 days. Gotta love Everflux
Remember if you have unique content on your blog & you’re still suffering from Supplementals you need to get some links to your main page & internals, never stop this cycle of linkbuilding or linkbaiting.
I’ve received about 4 or 5 reviews about Earners Blog that I asked for last week, I’m still looking for a lot more. So keep em coming. There’s $25 & an SEO Audit in it for you.
I also think that the DNS is still settling after the switch to the new hosting, so some of you may still be seeing & commenting on the old version of the site. I’ll try to get the databases synced so we don’t lose some of the comments
In other news check out Eli’s new post about Power Indexing sites with a large number of pages.
21 Comments on "Removing Supplementals in Wordpress or Other Sites"
Ben Cruikshank
10. Feb 2007, 12:39 am
Cool, thanks for the tip. I had noticed recently that I’ve been getting supplemental results so this was great timing. I have implemented this robots.txt with a couple tweaks, so we’ll see how quickly I get out of supplemental.
Ben
http//idleprofit.com
Garrett
10. Feb 2007, 6:06 am
I think I’ve got a fairly similar setup to the one you described (several category pages), but I’m a little confused. Here’s what I got:
Homepage (with links to category pages) > Category pages (with links to individual product pages)
If I disallow my category pages won’t that stop the bot from finding my item pages (since item pages are only linked to from the category pages)? I’ve got too many items to link them from the homepage (thus the need for category pages in the middle).
Thanks.
Clarke
10. Feb 2007, 4:40 pm
Thanks for the great tip. I am going to make sure to put it in place on my blog.
Garrett
11. Feb 2007, 1:24 am
My site isn’t in Wordpress, but I think I can still incorporate/test a lot of the suggestions. Thanks.
Samuel
13. Feb 2007, 6:38 pm
Just a quick question from a not-so-technically-savy guy…..
So I copy the text given above, save it as robots.txt, then which folder do I upload it to?
Thanks in advance
Samuel
13. Feb 2007, 8:18 pm
Thanks!
Samuel
13. Feb 2007, 8:24 pm
If my blog is domain.com/blog, do I upload the robots.txt on domain.com/blog or domain.com?
Samuel
14. Feb 2007, 2:18 pm
Thanks again!
I use feedburner, should I restrict feedburner links from begin indexed?
With regards to:
http://www.earnersblog.com/methods/rss-feeds-for-backlinks
Samuel
15. Feb 2007, 11:19 pm
Since you say that, so can I say that the robots.txt that you gave us will still allow our feedburner links to be indexed?
infonote
18. Feb 2007, 8:35 pm
” User-Agent: Googlebot
Disallow: /index.php
Disallow: /category/
Disallow: /feed/
Disallow: /feed
User-agent: *
Disallow: /wp-
Disallow: /trackback/
”
I cannot understand why you disallowed the index.php?
Can you please explain why?
TIA
TJP @ Investor Trip
22. Mar 2007, 9:22 pm
I made the changes to my robots.txt file, but for some reason my file disappears from my server. I don’t know what’s going on.
How long does it take for Google to read your robots.txt again?
Ant Eksiler
16. Apr 2007, 4:22 am
on my website, I have about 19,300 pages index and most of it is from the galleries which have no real content only images.
ps: my galleries are domain.com/gallery/subgallery/picname.html
should I exclude them from the index? Will this result in higher google traffic? what about google image traffic?
Horoscop
26. Oct 2007, 4:48 am
@Ant Eksiler
@Stuart
If on your’s 19300 pages you put some simple content,some comments and the image page has got his own title and links point to some pages from your site/blog.. your traffic will grow a lot
Cool, thanks for the tip. I had noticed recently that I’ve been getting supplemental results so this was great timing. I have implemented this robots.txt with a couple tweaks, so we’ll see how quickly I get out of supplemental.
Ben
http//idleprofit.com
I think I’ve got a fairly similar setup to the one you described (several category pages), but I’m a little confused. Here’s what I got:
Homepage (with links to category pages) > Category pages (with links to individual product pages)
If I disallow my category pages won’t that stop the bot from finding my item pages (since item pages are only linked to from the category pages)? I’ve got too many items to link them from the homepage (thus the need for category pages in the middle).
Thanks.
Thanks for the great tip. I am going to make sure to put it in place on my blog.
My site isn’t in Wordpress, but I think I can still incorporate/test a lot of the suggestions. Thanks.
Just a quick question from a not-so-technically-savy guy…..
So I copy the text given above, save it as robots.txt, then which folder do I upload it to?
Thanks in advance
Thanks!
If my blog is domain.com/blog, do I upload the robots.txt on domain.com/blog or domain.com?
Thanks again!
I use feedburner, should I restrict feedburner links from begin indexed?
With regards to:
http://www.earnersblog.com/methods/rss-feeds-for-backlinks
Since you say that, so can I say that the robots.txt that you gave us will still allow our feedburner links to be indexed?
” User-Agent: Googlebot
Disallow: /index.php
Disallow: /category/
Disallow: /feed/
Disallow: /feed
User-agent: *
Disallow: /wp-
Disallow: /trackback/
”
I cannot understand why you disallowed the index.php?
Can you please explain why?
TIA
I made the changes to my robots.txt file, but for some reason my file disappears from my server. I don’t know what’s going on.
How long does it take for Google to read your robots.txt again?
on my website, I have about 19,300 pages index and most of it is from the galleries which have no real content only images.
ps: my galleries are domain.com/gallery/subgallery/picname.html
should I exclude them from the index? Will this result in higher google traffic? what about google image traffic?
@Ant Eksiler
@Stuart
If on your’s 19300 pages you put some simple content,some comments and the image page has got his own title and links point to some pages from your site/blog.. your traffic will grow a lot
Add a Comment
Supporters of the Cause
Popular Articles
- Link Building Guide for 2008
- Become a Digg Power User in 48 Hours
- $800 a Day with Yahoo Answers
- Gmail Power Usage
- How to Work from Home - Efficiently
- Dominate Ebay Affiliate Program
- Making Money with Local Niche Directories
- Quick way to Make money with CPA Offers
- Diggproof & Speed up Wordpress
- Make Money with Clickbank & Articles
Recent Posts
- VPS Optimization Guide
- 13 Smokin Tips to Increase Adwords CTR
- Niche Domination - Solid Strategy or One Trick Pony?
- Hack Alert (remv.php) - Upgrade to Wordpress 2.7
- Coupon Affiliates are stealing from you
- $1k+ a month with a Celebrity Website
- Making Money with Internationalised Products
- How to Upgrade to an Adwords My Client Center
- Dealing with Dodgy Merchants & Affliate Networks
- DDOS Attack
Affiliate Tools
Guides / Example Sites
- Link Building Guide
- Discount Vouchers - Pligg
- Credit Cards - Wordpress

Enjoy this post? Get the RSS Feed

















Hi Garrett, there’s loads of other ways to help the spiders find your other pages without Category Pages.
Check out this post on Internal Linking Structures for Wordpress.