|
|||||||||||||||
|
Just to share about robot.txt from & help. *What is the robots.txt file*? The file is an ASCII text file that has specific instructions for search http://www.planetmy.com/blog/robottxt/Robots.txt Generator, Create and Maintain your robots.txt files. Advanced Robots.txt Generator Features http://www.basisoft.com/features.htmlOnline tool for syntax verification to robots.txt files, provided by Simon Wilkinson. http://www.sxw.org.uk/computing/robots/check.htmlhttp://authorities.loc.gov/robots.txt
Together, robots.txt and META tags give you the flexibility to express complex access policies relatively easily. A simple example Here is a simple example of a robots.txt file. http://googleblog.blogspot.com/2007/01/controlling-how-search-engines-access.htmlNeed some help. I have a seperate file for key works in a file named robots.txt I have this file in the root directory - same as index.html As in this http://www.freewebsitetemplates.com/forum/f19/robots-txt-9910/User-agent: * Disallow: /PostList.nhn . Disallow: /PostPrint.nhn . Disallow: /NBlogPostPreview.nhn . Disallow: /NBlogHidden.nhn . Disallow: /BlogInfo.nhn http://blog.naver.com/robots.txtTools for creating and analyzing robots.txt files to make sure your robots text file is working http://www.seotoolland.com/robots-txtOfficial blog of the Bing Webmaster Center Team. Just recently a strange problem came across my desk that I thought was worth sharing with you. http://www.bing.com/community/blogs/webmaster/archive/2008/09/12/is-your-robots-txt-file-on-the-clock.aspx# $Id: robots.txt,v 1.41 2010/01/11 18:55:22 shafim Exp $ # # This is a file retrieved by webwalkers a.k.a. spiders that # conform to a defacto standard. http://www.ibm.com/robots.txt# robots.txt for www.sun.com # Requests for updates should be filed through: # http://www.sun.com/contact under Web Site Feedback # Updated: September 5, 2008 http://www.sun.com/robots.txtUser-agent: * Disallow: /user. Disallow: /product/s/ Disallow: /guide/s/ Disallow: /produit/s/ Disallow: /produkt/s/ Disallow: /prodotto/s/ Disallow: /producto/s/ http://www.wikio.fr/robots.txtThe robots.txt file must be placed at the root of your domain (www.yourdomain.com/robots.txt). If you cannot put a robots.txt file up, read our exclusion policy. http://www.archive.org/about/exclude.php# robots.txt for http://www.state.co.us/ User-agent: * Disallow: /test9/ # still in production stage. Disallow: /test6499/ # still in production stage http://www.state.co.us/robots.txtPHP: Parsing robots.txt. If you're writing any kind of script that involves fetching HTML pages or files from another server you really need to make sure that you follow netiquette ... http://www.the-art-of-web.com/php/parse-robots/Article on how to create a robots.txt file. ... Creating a robots .txt file. The robots.txt file contains directives, created by you, that spiders are programmed to obey based on ... http://www.pro-dezign.com/articles/robots_text_file.htmlhttp://virginia.cc.vt.edu/robots.txt
https://adwords.google.com/robots.txt
#Added for Bristol-Myers on Sept 2005. User-agent: vspider . Disallow: / #For all other crawlers. User-agent: * Disallow: /Management/ # don't crawl healthcheck http://www.fda.gov/robots.txtTell web robots what areas of your site are allowed to visit and index, restrict access to your site for BAD bots, tell search engines where your sitemap is located. http://www.websitetoolboxpro.com/robots_txt/Creator and validator of robots.txt files. http://www.clockwatchers.com/robots_tool.html# /robots.txt file for http://disney.go.com/ User-Agent: DCOM FAST Enterprise Crawler. Disallow: /games/html/css/small. Disallow: /games/html/css/large http://disney.go.com/robots.txtArticle with information about using a Robots Text File (robots.txt) on your website. http://www.seopt.com/articles/robots-text-file.htmlAdvertise | Help | Text-only Skin | Yellow Pages | Privacy Policy | Terms & Conditions © Copyright 2010, Lycos, Inc. Lycos is a registered trademark of Lycos, Inc. http://www.hotbot.com/robots.txtUser-agent: * Crawl-delay: 600 . Visit-time: 0100 - 0500 . Request-rate: 1/30 http://apex.oracle.com/robots.txt |
|