Friends have you ever heard about Robots.txt file?I think most of you heard,today i will give you full detail about this file.This is really very important for you,if you have an website means you are a website owner.
Please read this full article and try to understand its importants.
Robots.txt is a text (not html) file you put on your website to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door .That is why i am saying that if you have a really sensitive data in your website then you should use robots.txt file to prevent data that is displayed by search engines.
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory (i.e. http://web url/robots.txt) and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way.
Also See This:
***Thanks Friends***
 |
| http://www.learning-tutorials.blogspot.com |
Please read this full article and try to understand its importants.
Robots.txt is a text (not html) file you put on your website to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door .That is why i am saying that if you have a really sensitive data in your website then you should use robots.txt file to prevent data that is displayed by search engines.
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory (i.e. http://web url/robots.txt) and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way.
- Structure Of Robots.txt File:
The structure of a robots.txt is pretty simple– it is an endless list of user agents and disallowed files and
directories. Basically, the syntax is as follows:
User-agent:
Disallow:
User-agent are search engines' crawlers and Disallow lists the files and directories to be excluded from indexing.
In addition to User-agent: and Disallow: entries, you can include comment
lines – just put the # sign at the beginning of the line.
For example:
# All user agents are disallowed to see the /temp directory.
User-agent: *
Disallow: /temp/
- Generate And Validate Robots.txt File:
If you want to Validate Your Robots.txt file then please click this link
If you want to Generate Robots.txt file then please click this link
Also See This:
***Thanks Friends***
Nice article and very informative thank you for sharing us.
ReplyDeleteSilver Trading Tips || Gold Trading Tips