NoFollow is Not the Only Way to Block Link Crawling
Posted by reviews on Mar 18, 2008
Other Ways to block search engine spiders from crawling
Actually, NoFollow does not “stop” the search engine spiders (robots) from crawling, but instead Google and others simply do not pass “link juice” to the backlink.
| rel=”nofollow” Action | Yahoo! | MSN Search | Ask.com | |
|---|---|---|---|---|
| Follows the link | Yes | Yes | Not proven | Yes |
| Indexes the “linked to” page | No | Yes | No | Yes |
| Shows the existence of the link | Only for a previously indexed page | Yes | No | Yes |
| In SERPs for anchor text | Only for a previously indexed page | Yes | No | Yes |
However, as Geoland points out so well, there are other ways to block the crawling of links. The other methods are much more limiting than rel="nofollow", because at least NoFollow still allows the crawling and indexing.
Normal methods to check for NoFollow will not show you if a link is truly crawlable (including SEOQuake, right-clicking on a link and looking at properties, etc). In other words, “what you see is not always what you get.”
Robots.txt
One method is to use Robots.txt to control the crawling of spiders. Robots.txt is kept in the top level of the domain.
Examples
Blocks all robots
User-agent: * Disallow: /
Blocks all robots from crawling specific directories
User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /tmp/ Disallow: /administrator/
Meta Elements
The robots attribute is used to control whether or not the search engine spiders are allowed to index a page and whether or not they should follow links from a page. The NOINDEX value prevents a page from being indexed, while the NOFOLLOW prevents links from being crawled.
The robots attribute is supported by all of the major search engines. In addition there are several additional values for the robots meta attribute that are relevant to search engines, such as NOARCHIVE and NOSNIPPET, which are meant to tell search engines what not to do with a web pages content. Meta tags are not the best option to prevent search engines from indexing content of your website. A more reliable and efficient method is the use of the Robots.txt file.
NOINDEX tag tells a search engine not to index a specific page.
NOFOLLOW tag tells a search engine not to follow the links on a specific page.
NOARCHIVE tag tells a search engine not to store a cached copy of your page.
NOSNIPPET tag tells Google not to show a snippet (description) under your a search engine listing, it will also not show a cached link in the search results.
Example
<html> <head> <title>Create Backlinks to My Site</title> <META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”> </head>
Javascript
Any link written in javascript is not crawlable by search engine spiders. It will look like a normal link and will not be lined through (SEOQuake addon) or highlighted (Search Status addon)
Dynamic URLs
Some spiders may also avoid crawling any url that has a “?” in it (dynamically produced) in order to avoid spider traps which may then cause the crawler to download an ‘infinite’ number of URLs from a Web site. Thus it would probably prevent links on these pages from being crawled.
Google Analytics for Tracking Google Base (Froogle) Visitors
Posted by reviews on Dec 3, 2007
For those of us that sell products via an ecommerce site, you probably are familiar with Google Base (formerly Froogle) where you can list your products on Google Product Search. You may wonder if you are getting many visitors from Google Base. You will likely get visitors if your pages are fairly well designed in Google Base and the pricing and attributes are good. The question would be; how does one track Google Base visitors using Google Analytics?
Google has an answer on their Official Google Base Blog, however their advice is odd and a pain in the arse at best, and very bad advice at worst. Their answer is to create another page just for the Google Base visitors, thus creating duplicate content, which contradicts Google’s own webmaster guidelines. Nice.
Regular landing page: http://example.com/page1.html
Unique Base landing page: http://example.com/page2.htmlBy creating two versions of the same page on your website and submitting the unique landing page URLs to Base, Google Analytics can show you exactly how much traffic is being sent to your website from Base. Just make sure to include the unique landing page URL in the link attribute of your bulk upload file.
A much, much better answer was supplied by Scott Horne over at WebProNews. They even provide great graphics to help you. He suggests adding a query string to the end of the url link in Google Base, such as ?ref=base.
It would look like this: http://www.example.com/nifty-product.html?ref=base
You would then log into Google Analytics, go to the specific domain, go to Content, Top Landing Page, and do a search for ref=base.
Very simple!
An Ongoing List of Adsense Alternatives
Posted by reviews on Oct 24, 2007
Webmasters can place Google AdSense JavaScript code on their web pages which would then allow Google AdSense to be displayed as advertisements based upon the content of the web page.
Some webmasters are either not accepted into the Google AdSense program, or do not wish to do business with Google.
For those webmasters, there are many alternatives to Google AdSense.
Click here for a list of alternatives
for the List of Google Alternatives Page of this site