A text file placed in the root directory of a Web site that prohibits search engine spiders from indexing all or specific pages of the site. The Robots Exclusion protocol provides a format for designating which directories and files are off limits to the spidering program. See spider.
 | Reproduced with permission from Computer Desktop Encyclopedia. Copyright (c) 1981-2009 The Computer Language Company Inc. All rights reserved. |
Additional Resources
- A1 Sitemap Generator 2.0.4 (Windows)
- Help search engines crawl and index your Web site with HTML, RSS, and XML sitemaps as defined by Google XML sitemaps protocol including video, mobile and news XML sitemaps. Near-endless Web site scanning options. Configure amount of simultaneous connections to use. Supports crawler filters, nofollow, robots.txt, custom connection and read...
- Software downloads 2009-11-21
- iGooMap 1.8 (Mac)
- With iGooMap, you can automatically create XML Sitemaps conforming to the sitemap protocol or import and edit existings sitemaps visually. You can upload the sitemap to your server and notify all search engines supporting sitemaps. Easy handling, just enter the URL of your web site. You can scan...
- Software downloads 2009-10-27
- mySitemap2 2.10 (Mac)
- mySitemap2 allows you to easily create HTML and XML sitemaps Google Sitemaps by scanning websites either locally or already on the internet. Once scanned, mySitemap2 will allow you to view, edit and export the sitemap for inclusion on your website. Creates both HTML and XML sitemaps Scan websites locally or...
- Software downloads 2009-10-26
- WSJ chief criticizes news aggregators - creators carry cost burden
- More important...than net neutrality? You are out of your mind. You probably think you are a journalist as well. What a joke. Do you even proof read your own dribble? It's obvious you don't. What kind of journalist would send out content with grammar and spelling errors? it has...
- Discussion threads 2009-10-23
- Newsmakers: Be careful what you wish for
- Without Google they get no trafficIt does not take a rocket scientist to know that without Google, most of this news sites would probably have little to now traffic at all.So how is Google stealing from them? Since when is sending customers to your door a bad thing?Without the...
- Discussion threads 2009-10-11
- Newsmakers: Be careful what you wish for
- At the World Media Summit, Tom Curley and Rupert Murdoch took turns bashing Google over their klepto antics and ruthless news stealing behavior. They have been stealing their content for too long, and it's about time they paid up. The truth is, they can prevent Google...
- Blog posts 2009-10-11
- RedX Meta Tag Builder 1.7 (Windows)
- A freeware meta-tag generator which allows you to quickly build HTML meta tags for you websites, including (title, description, keywords, robots, categories, author, e-mail, expires, copyright, generator, language, rating, and the optional the refresh tag). This easy to use software allows you to build meta tags and then to save...
- Software downloads 2009-09-22
- Everyone has a secret life but will Google find it?
- Everyone has a secret life but will Google find it?Better still ?"I have even better advice: be very active on the Internet ? anything bad will be diluted into the long tail of you."I should prefer "never do anything bad and publish only high quality items online" ...... especially good...
- Discussion threads 2009-03-04
- RTSoftwares Sitemap Generator 1.0 (Windows)
- Free Sitemap Generator tool to generate HTML/TXT sitemap. Sitemap generated by sitemap generator is robots.txt compatible. Easy-to-use sitemap generator for Web masters. One click sitemap generation with many configurable options. Download your copy of free sitemap generator and create rich html sitemap for your Web site with a simple button...
- Software downloads 2009-01-25
- Chilkat Spider ActiveX 2.2.2 (Windows)
- Crawl a Web site with this free ActiveX spidering component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. Automatically handles HTTPS URLs. Includes safety measures for avoiding large / infinite size Web pages or infinitely growing-in-length URLs. Includes a wind-down features for crawling a maximum number of pages per...
- Software downloads 2008-12-15
- Google opens up your profile to search engines
- Google has been letting people create profiles for a while now -- you can see mine here, and you can create yours here. Until today, your public profile that you created has remained hidden from search engines. Just about a half hour ago, Google added a...
- Blog posts 2008-10-15
- Google PPOB? What is it?
- Google PPOB? What is it?Classified Ads aggregator....Looks like a classified ad aggregator...in russianhttp://google.ru/ppob Google Classifieds BetaRE: Google PPOB? What is it?Garret, this was discussed here (the 1st result for [google ppob robots.txt] in Google):http://blogoscoped.com/archive/2008-10-08-n14.htmlGet with the program dude! ;-)
- Discussion threads 2008-10-09
- Google PPOB? What is it?
- When you are looking for things that Google is about to do, the best place to keep your eye on is Google's robots.txt file -- a document they keep updated with a list of websites they don't want search engines to index. A week ago, they added "/search2001" which...
- Blog posts 2008-10-09
- Google to go back in time?
- Warning, this post is speculative, but I think all signs point to what I am about to describe. Google may be about to launch a search tool that uses the same algorithms used way back in 2001. Why would they do this? I think it's probably the...
- Blog posts 2008-09-30
- Is Cuil killing the Internet?
- Is Cuil killing the Internet?Brute force indexing?I have an inkling Cuil might not be everything it's touted to be. The only reason for brute force indexing is to fill in the gaps where you've come up short in your harvesting/indexing algorithms.RE: Is Cuil killing the Internet?I used Cuil to...
- Discussion threads 2008-09-02
- Linking The Web 2 (Windows)
- Linking The Web is a free utility that regularly monitors all backlinks to your sites. If some of the links don't exist, contain wrong anchor, are covered by a "nofollow" flag or hidden by robots.txt then you will be immediately notified by email. Also the program can keep track of...
- Software downloads 2008-07-14
- New Google Mobile feature lets you search without typing
- Google added a new entry into their robots.txt file -- one way you can keep tabs on what Google doesn't want to see. The new entry that forbids crawlers from seeing http://www.google.com/m/lcb made me naturally curious. The new feature, that from what I can tell is...
- Blog posts 2008-03-22
- SpidEye 1 (Windows)
- SpidEye is like a spider's eye, showing what a Web Crawler would see while navigating the Web. From each page visited it collects and summaries relevant information that are meant to assist Web masters on the design of their Web pages for bots. This freeware provides full HTML navigation, displays...
- Software downloads 2007-12-20
- Google will soon support "Unavailable After" META tag
- Barry Schwartz found an interesting article from highrankings.com that gives us some juicy details about what it takes to get a good rank on Google and what tools we can look for soon. Dan Crow, director of crawl systems at Google, attended the annual SEMNE (Search Engine Marketing New...
- Blog posts 2007-07-14
- Newspapers want Google News' quarter
- Newspapers want Google News' quarterNewspapers want Google News' quarterGoogle short changes the public again. When are people going to wake up and realize Google doesn't run the world. I'm sure these newspapers are going to want back pay as well as current royalties severely affecting Google's bottom line....
- Discussion threads 2007-05-23
Neighboring Terms
Premier Vendor Content Whitepapers, webcasts & resources from our Power Center Sponsors
- New Online Dashboard for IT Leaders
-
Read about top issues IT decision-makers face every day, plus get cost-effective solutions to real-life IT problems.
- Learn more >>
- The best support in the Linux business
-
If Linux is going to power your mission-critical applications, you'd better have the best support known to business. Novell was rated the top provider of Linux technical support.
- Learn more >>
- The best support in the Linux business
-
If Linux is going to power your mission-critical applications, you'd better have the best support known to business. Novell was rated the top provider of Linux technical support.
- Learn more >>
- The more you simplify, the more you save
-
When you transition from your existing Red Hat environment to SUSE Linux Enterprise from Novell, you can recognize dramatic cost savings, perhaps as much 50%
- Learn more >>
Meet Doc
-
-
Here to help you with your Document Management Needs
- Doc is an enigma. Born to a Russian ballerina and a German electrical engineer, he grew up in various locations in the United States. He’s seen the insides of more brands, versions, and generations of printer and printer-related hardware than almost anyone.
- To learn more about this mysterious figure check out his blog on ZDNet and his Workspace on TechRepublic. You’ll be glad you did.
-
Produced by
ZDNet and -
