This ain't no ordinary Affiliate Marketing Blog - Grab the Feed

I’ve had a few blogs hit supplementals lately even though they’ve got a pretty decent Inbound Link Authority. Generally whenever this happens I’ll query the site in question with Google’s site: operator to see if I can see anything out of the ordinary.

Well chances are that if you’re using wordpress & you’ve not set up your robots.txt correctly then you’ll be getting heaps of pages indexed that you don’t want.

What don’t you want indexed:

  1. Your Plugins & Content Folder
  2. Categories
  3. Galleries & Images
  4. Feeds & Trackbacks

What you DO want indexed:

  1. Posts
  2. Pages
  3. Sitemap

Shoemoney had a similar problem a few weeks ago (And most people think he’s an SEO?). What happens in that in order to rank a page in the main index Google must attribute a minimum pagerank value to it. If it falls below the threshold it’ll hit supplemental. So you need to make sure that your site has a good distrubution of pagerank or authority. So if most of your links are hitting your homepage & you’re second level of navigation are your categories then you’re losing heaps of link popularity that should be going to your posts instead.

Ony my blogs I’ve implemented the following robots.txt with reasonable success (50% more pages indexed):

User-Agent: Googlebot
Disallow: /index.php
Disallow: /category/
Disallow: /feed/
Disallow: /feed

User-agent: *
Disallow: /wp-
Disallow: /trackback/

There is a good set of reasoning behind doing this on all of your sites. You should almost be telling the engines which pages you want to rank & which pages you don’t. Removing the pages that are worthless to the engines & only serve as a navigation between different parts of the site increase the relevance & internal linking power of other pages.

I have a Joomla content site that has more category pages than static pages. So I exlcluded all the category pages & my traffic has jumped from around 500 uniques/day to 1500 in about 8 days. Gotta love Everflux :)

Remember if you have unique content on your blog & you’re still suffering from Supplementals you need to get some links to your main page & internals, never stop this cycle of linkbuilding or linkbaiting.

I’ve received about 4 or 5 reviews about Earners Blog that I asked for last week, I’m still looking for a lot more. So keep em coming. There’s $25 & an SEO Audit in it for you.

I also think that the DNS is still settling after the switch to the new hosting, so some of you may still be seeing & commenting on the old version of the site. I’ll try to get the databases synced so we don’t lose some of the comments :)

In other news check out Eli’s new post about Power Indexing sites with a large number of pages.

Enjoy this post? Get the RSS Feed

Or get updates by email:


Prev Article:

Next Article:


Not making any Money yet? Check out these posts too:

Random Post: Want to earn more? Read a Random Post!

http://www.earnersblog.com/wp-content/plugins/sociofluid/images/digg_48.png http://www.earnersblog.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png http://www.earnersblog.com/wp-content/plugins/sociofluid/images/delicious_48.png http://www.earnersblog.com/wp-content/plugins/sociofluid/images/facebook_48.png http://www.earnersblog.com/wp-content/plugins/sociofluid/images/sphinn_48.png http://www.earnersblog.com/wp-content/plugins/sociofluid/images/twitter_48.png

21 Comments on "Removing Supplementals in WordPress or Other Sites"

Ben Cruikshank 10. Feb 2007, 12:39 am

Cool, thanks for the tip. I had noticed recently that I’ve been getting supplemental results so this was great timing. I have implemented this robots.txt with a couple tweaks, so we’ll see how quickly I get out of supplemental.

Ben
http//idleprofit.com

Garrett 10. Feb 2007, 6:06 am

I think I’ve got a fairly similar setup to the one you described (several category pages), but I’m a little confused. Here’s what I got:

Homepage (with links to category pages) > Category pages (with links to individual product pages)

If I disallow my category pages won’t that stop the bot from finding my item pages (since item pages are only linked to from the category pages)? I’ve got too many items to link them from the homepage (thus the need for category pages in the middle).

Thanks.

Stuart 10. Feb 2007, 12:19 pm

Hi Garrett, there’s loads of other ways to help the spiders find your other pages without Category Pages.

Check out this post on Internal Linking Structures for WordPress.

Clarke 10. Feb 2007, 4:40 pm

Thanks for the great tip. I am going to make sure to put it in place on my blog.

Garrett 11. Feb 2007, 1:24 am

My site isn’t in WordPress, but I think I can still incorporate/test a lot of the suggestions. Thanks.

Samuel 13. Feb 2007, 6:38 pm

Just a quick question from a not-so-technically-savy guy…..

So I copy the text given above, save it as robots.txt, then which folder do I upload it to?

Thanks in advance :)

Stuart 13. Feb 2007, 7:06 pm

Robots.txt should be in the root of your domain.

If your blog is in a sub directory say http://www.yourdomain.com/blog/ then you need to put /blog/ infront of each of the queries in robots.txt

Stu

Samuel 13. Feb 2007, 8:18 pm

Thanks!

Samuel 13. Feb 2007, 8:24 pm

If my blog is domain.com/blog, do I upload the robots.txt on domain.com/blog or domain.com?

Stuart 13. Feb 2007, 9:12 pm

Needs to be in the root domain.com

Then you tell it in the robots.txt to restrict stuff in the /blog/ directory.

Samuel 14. Feb 2007, 2:18 pm

Thanks again!

I use feedburner, should I restrict feedburner links from begin indexed?

With regards to:
http://www.earnersblog.com/methods/rss-feeds-for-backlinks

Stuart 14. Feb 2007, 2:51 pm

No way, use your feedburner page to your advantage. They rank pretty well :)

Samuel 15. Feb 2007, 11:19 pm

Since you say that, so can I say that the robots.txt that you gave us will still allow our feedburner links to be indexed? :)

infonote 18. Feb 2007, 8:35 pm

” User-Agent: Googlebot
Disallow: /index.php
Disallow: /category/
Disallow: /feed/
Disallow: /feed

User-agent: *
Disallow: /wp-
Disallow: /trackback/

I cannot understand why you disallowed the index.php?

Can you please explain why?

TIA

Stuart 18. Feb 2007, 9:34 pm

Well I use a tagging system & it uses the index.php string. Not entirely sure if that would have an adverse effect on the other listings though.

TJP @ Investor Trip 22. Mar 2007, 9:22 pm

I made the changes to my robots.txt file, but for some reason my file disappears from my server. I don’t know what’s going on.

How long does it take for Google to read your robots.txt again?

Stuart 22. Mar 2007, 10:30 pm

Normally within 24 hours :)

Ant Eksiler 16. Apr 2007, 4:22 am

on my website, I have about 19,300 pages index and most of it is from the galleries which have no real content only images.

ps: my galleries are domain.com/gallery/subgallery/picname.html

should I exclude them from the index? Will this result in higher google traffic? what about google image traffic?

Stuart 16. Apr 2007, 10:18 am

It depends, how much traffic do you get from Google Images & is it more beneficial to get that sort of traffic or organic traffic to your main pages?

Excluding the gallery pages will increase rankings for your main pages.

Horoscop 26. Oct 2007, 4:48 am

@Ant Eksiler
@Stuart

If on your’s 19300 pages you put some simple content,some comments and the image page has got his own title and links point to some pages from your site/blog.. your traffic will grow a lot ;)

Supporters of the Cause

Popular Articles


Recent Posts

Affiliate Tools

Guides / Example Sites