One of the most telling things you can do in any business is study your competitors. Your competition can reveal weakness in your business and open your eyes to new opportunities in your industry. The same is true in the web world, and with search engine optimization in particular. Why does your competitor consistently rank higher than you for important terms? What programming, content, and link building strategies are they using to help attain those rankings? Answer those questions and you'll be one step closer to out-ranking them and bringing that traffic and those sales to your site.
Who should you study?
A lot of industries will have hundreds of direct competitors, and many more indirect competitors. You can very easily get caught up in "paralysis by analysis" if you try to analyze every possible competitor that you have. The purpose of studying your competition is to better your business, so I prefer to limit my analysis to ten sites or less.
You probably already know a handful of your adversaries. Either they were the incumbent leaders in your industry when you began, or your customers constantly remind you that they have lower prices, or every time you Google a phrase in your industry they come up first in the search results. Those sites are the first ones you should add to your list to research. Fill the rest of the list in by picking sites that consistently rank high for the most searched phrases related to your business. If you haven't already done keyword research, I recommend starting with the SEO Book Keyword Research Tool. Once you understand which terms are searched the most, it will be pretty obvious which sites consistently rank high.
What information should you collect?
There are a plethora of potential metrics that can be used to gauge the competition. The six below can be collected for free in a matter of minutes and do a great job of explaining why a site ranks as well as it does.
Site Age -- the age of a site is generally considered to be one of the top five most influential factors in how high a site ranks. While it's something that is out of your control, sometimes you'll see a site from 1999 consistently outranking a better site from 2006 and it's helpful to understand why. You can see the history of a site using a nifty tool called the Wayback Machine.
Y! Links -- the number of backlinks the domain has in Yahoo. As you probably already know by now, the quantity and quality of links pointing to a site is extremely relevant in determining how high it ranks. Looking at the particular links that your competition has serves as one of the best ways to learn how they market their site. It is also one of the best ways to brainstorm potential link building ideas for your site. For example, if your competitors products are all reviewed on a popular blog in your industry, there's a good chance that same blog would want to review your products as well (giving you free publicity and quality links).
Pages indexed -- this refers to the number of pages listed in a search engines index (also referred to as cached pages). Sites that rank high are generally easy for Google, Yahoo, and MSN to spider and thus have nearly all of their pages included in their indexes. You can check how many pages are indexed by typing in site:www.yoursite.com, or by using a tool like the Indexed Pages tool on SEOmoz. Using sitemaps is a great way to ensure that every page on your site is indexed properly.
Last indexed -- this is the date of the last time Google visited the home page of the site. Ideally your site is indexed every single day, but it usually depends on how frequently you update your content. If you update content regularly, you should be being indexed at least once per week. You can find this date by clicking the "Cached" link next to any Google search result.
Home Page PR -- this refers to Google Page Rank, a 0-10 score that Google gives to assess the value of a web page. It's primarily determined by the quality and quantity of incoming links, and is a quick and dirty way to see how popular a page is. Since it's a universal metric that all webmasters can quickly check, it's often misused to assume how much traffic a site gets or how high it ranks -- neither of which correlate very well with PR. For our purposes though, it's a nice barometer to look at. It can be checked using the Google Toolbar or a site like PRChecker.
Strongest Pages -- SEOmoz has a great strongest pages tool that will list off the most important pages on a domain, based on number of links pointing to it and its current rankings. This gives you an idea of what content on their domain is causing the high rankings. Is it popular articles? Is it product pages? Whatever their strongest pages, you should take note and use those pages as guidelines for potential additions and modifications to your site.
Using the information to your advantage
Pretty quickly you'll start to realize that all of the sites you're analyzing have been around for a few years, are indexed frequently and thoroughly, and have a lot of quality backlinks. So how do you get there? Well there's nothing you can do about site age, but the rest are very much in your control: you can model title tags and page headings after the competition, you can structure your site and internally link the way that they do, you can add sections related to topics that they rank high for, and you can most definitely expand your link building plan by studying what has worked for them. Ultimately, analyzing and understanding your competition will reduce your learning curve and accelerate the growth of your site.