« Search Engine Marketing Conversation, Part One | Home | Search Engine Marketing For Large Brands - Video »
Don’t Neglect the Robots.txt File
By Mike Murray | May 16, 2007
I sometimes get grief from website owners who wonder why anyone goes
to the trouble of adding a robots.txt file to a website.
After all, it’s designed to tell search engines what not to crawl.
It’s not a special site map listing all of a website’s outstanding
content - page by page.
The reality is that search engines look for a robots.txt file on
websites. Why wouldn’t a website marketer do everything possible to
make a good impression when the search engines stop by?
Think of it as a nice doormat placed in front of a custom home with
fabulous furnishings throughout.
For many websites, a simple robots.txt file will do.
A new report, A Large-Scale Study of Robots.txt underscores
issues related to the robots.txt file, including format, size,
frequency of use and likely errors. Researchers looked at more than
7,000 websites in different industries.
Study authors call for a “better-specified, official standard” because
of ambiguity concerns and errors people make with the robots.txt file.
Here is an excerpt:
We observe 13 cases of misnamed robots.txt files and find 23 files in which a specific name such as“crawler”, “robot”, or “webcrawlers” appear in the User-Agent field. General name in the User-Agent field is an incorrect use of the Robots Exclusion Protocol. We also found 282 robots.txt files with ambiguous rules and 18 files with conflicting rules (e.g. a directory is disallowed first and then allowed or allowed first and then disallowed).
An encouraging sign is that the percentage of websites using a
robots.txt file has increased from 35% to 38.5% in the last 11 months,
according to the study.
Search Engine Land also reported on the robots.txt issue.
For more information, visit the Web Robots Pages, an excellent resource.
Is a robots.txt file a must? I see plenty of websites rank
for excellent search terms even though they lack a robots.txt file. They
also violate HTML standards and fail to use the most effective page titles.
They succeed because something else gives them an edge - site
longevity, content, internal links, inbound links, etc. A keyword in a
long standing domain sure helps too.
But it can’t hurt to make a great first impression on a search engine that stands between you and your next customer.
What does your doormat look like?
Want to know more about search?Visit our search engine marketing forum.
Want to know more about online videos?
Stop by our Internet video marketing forum.
