Robots.txt for SEO Ranking

What is "robots.txt"?

For those who don’t know what is robots.txt, it is a text file which instructs search engine robots to crawl a page or not. Any CMS (content management system) based sites are having admin module online and that should not be crawled by Search Engines, using robots.txt you can block that part to not get it crawl.

About /robots.txt

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.It works like this: a robot wants to visits a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *

Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

• robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
• The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

How to create a /robots.txt file

• Where to put it - In the top-level directory of your web server.
• When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.
• For example, for "http://www.example.com/shop/index.html, it will remove the "/shop/index.html", and replace it with "/robots.txt", and will end up with "http://www.example.com/robots.txt".
• So, as a web site owner you need to put it in the right place on your web server for that resulting URL to work. Usually that is the same place where you put your web site's main "index.html" welcome page. Where exactly that is, and how to put the file there, depends on your web server software.
• Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.

What to put in it

• The "/robots.txt" file is a text file, with one or more records. Usually contains a single record looking like this:

User-agent: *

Disallow: /cgi-bin/

Disallow: /tmp/

Disallow: /~joe/

In this example, three directories are excluded.

Note that you need a separate "Disallow" line for every URL prefix you want to exclude -- you cannot say "Disallow: /cgi-bin/ /tmp/" on a single line. 

Also, you may not have blank lines in a record, as they are used to delimit multiple records.

Note also that globbing and regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "User-agent: *bot*", "Disallow: /tmp/*" or "Disallow: *.gif".

What you want to exclude depends on your server. Everything not explicitly disallowed is considered fair game to retrieve. Here follow some examples:

To exclude all robots from the entire server

User-agent: *

Disallow: /

To allow all robots complete access

User-agent: *

Disallow:

(Or just create an empty "/robots.txt" file, or don't use one at all)

To exclude all robots from part of the server

User-agent: *

Disallow: /cgi-bin/

Disallow: /tmp/

Disallow: /junk/

To exclude a single robot

User-agent: BadBot

Disallow: /

To allow a single robot

User-agent: Google

Disallow:

User-agent: *

Disallow: /

To exclude all files except one

This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:

User-agent: *

Disallow: /~joe/stuff/

Alternatively you can explicitly disallow all disallowed pages:

User-agent: *

Disallow: /~joe/junk.html

Disallow: /~joe/foo.html

Disallow: /~joe/bar.html

/robots.txt checker

There are third party tool which allows us to check the /robots.txt :

• Google's robots.txt analysis tool (requires a Google Account)
  
  
  If you really think it is useful, Don't Forget to share.
  For more Content, Tips, Knowledge kindly subscribe to the blog.
  
  
           

Read More

GOOGLE TRICKS

Google Tricks & Tips

In 2016, Statics say that there are more than 1 billion Active user on Google every Month. But they are not using some special features of the Google.
Here let’s see some unknown but amazing Google Features we never know before.

Fun with Google......

1> Barrel Roll

in Google search results, if u search for "do a barrel roll",
before u complete the string, your results page will be spinning!!

2> Google Gravity

Search for "Google gravity" & in suggestions box which appears, click "I’m feeling lucky"....
now you'll see Google homepage & when you'll move your mouse a bit, all elements on page will fall down to taskbar!!!

3> Zerg Rush

Google “zerg rush” and prepare to defend your browser against hordes of the letter ‘o’ in Google’s logo,
which will start to destroy your search results. They’re apparently undefeatable, but you can try to fight
them off by clicking on them and share your high score on Google +.
Just search for "zerg rush" & before u press enter, it has started!!

4> You’ll never find Chuck Norris

Because Chuck Norris jokes never get old, there is a warning hidden in Google search results to ward off those
who dare to attempt to find him. Just search ‘find Chuck Norris’ and hit ‘I’m feeling lucky’ and you’ll see what
I mean!!!

5> Askew

If you run a Google search for the word “askew”, you may think you’ve somehow broken your monitor or telepathically
adjusted your display settings. But really, Google has just shifted a few degrees

6> Goglogo

Ever wanted your own personal version of Google, complete with a customized logo?
You can type “goglogo” into your Google search bar and hit “I’m feeling lucky”...
You'll get Google search with your name!!!

7> Google Sphere

Google sphere transforms your Google Image search page into an interactive spinning orb.
To access it, search for “Google sphere” and click “I’m feeling lucky”.

8> The Loneliest Number

In case you weren’t aware of this, Google’s built-in calculator will confirm it for you:
the loneliest number is one.

9> Quing Ming Festival

If you search for “qingming” (a traditional Chinese festival), Google will decorate your search
results page accordingly, painting the white space with watercolor plants, occasional rippling
water and some children perched atop blinking cows. If you look closely, you may spot a fish or
two leaping from the ‘water’.

10> Current Time

If u type "what time is it" followed by the name of some place, Google will show you the exact time at
that place in search results

Read More