What Is A Robot.txt?


What is a Robots.txt file?
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. In short Web site, owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

- To exclude all robot for an entire server:

  user-agent:*
  Disallow:/

- To allow all robots complete access:

  user-agent:*
  Disallow:

- To exclude all robot from part of Server:

   User-agent:*
   Disallow:/cgi-bin/
   Disallow:/tmp/
   Disallow:/junk/

- To exclude Single Robot:

  User-agent:*
  Disallow:user-agent:*
  Disallow:/

- To exclude all file except one:

  User-agent:*
  Disallow:/-joe/stuff/

- Alternatively You can Explicity Disallow pages:

  user-agent:*
  Disallow:/-joe/junk.html
  Disallow:/-joe/foo.html
  Disallow:/-joe/bar.html

Comments

Post a Comment

Popular posts from this blog

What Is Reach ?

Reach is an estimate of how many people in a location could see your ads. How it's calculated: Your estimated reach may differ from census data, because reach calculates thenumber of signed-in users visiting Google sites. Reach also varies due to the length of activity, the number of temporary visitors to a location and the number of Google accounts used per person. How to use it: The reach estimate should only be used to compare location targets. To estimate the number of ad impressions, try the Keyword Planner How it's estimated Reach in AdWords is based on an estimate of the number of people seen on Google properties within the specified geographic area. Therefore, you may find that reach numbers in AdWords may differ significantly from census population data or other sources due to a range of factors, including: Number of accounts used per person Number of temporary visitors to a geographic location Length of time each person spends on a website Number of ...
Read more

Some Of The Top Keywords For Digital Marketing

online marketing social media marketing web marketing marketing online marketing agency digital marketing strategy marketing companies digital media marketing e marketing online advertising website marketing digital advertising internet marketing strategies marketing websites marketing internet top marketing agencies digital marketing business digital online marketing media marketing online marketing business digital marketing strategist role of digital marketing online marketing services internet marketing services digital marketing websites top internet marketing firms PPC search engine internet marketing digital online international marketing digital mark internet marketing services company service marketing
Read more