20 September 2007
SEO : Help Search Engines Find All Your Pages
We all like it when the search engine bots come crawling and finding our pages. I, personally, through a little party for the bots and we celebrate with a nice slice of cake and glass of bubbly. But - wait - the bot only found my index page - my site is chock full of Internet goodness - how do I help our poor little bot find all my pages?
Look up there - what’s that - it’s a bird - it’s a plane - no, it’s the mighty sitemap xml file. Yes, the sitemap xml file - the file that tells the search engines about all the pages in your site.
But, how do I create a sitemap xml file? Well, my friend, luckily there are many ways to do it. If you use wordpress, there is a plugin that will take care of it for you. If you have a non-wordpress site there are many websites that can create the sitemap xml for you, such as XML-Sitemaps.com. You simply take the file that they generate for you and save it in the root/main directory of your website. Don’t forget to keep it updated each time you add a new page.
Great - Now I have a sitemap xml file, but how do I let the search engines know about it?
Google and Yahoo offer you a place to submit and monitor your sitemap xml pages for your site.
Google Sitemap Submit : Google Webmaster Tools
Yahoo Sitemap Submit : Yahoo! Site Explorer
But, do I really have to manually add all my sitemap xml files - that seem’s like a lot of work? Of course not, the true geek always knows there is a way around doing work. Another great invention, called the Web Robots file, allows you to tell all the search engines that you have a sitemap xml file without ever having to submit it anywhere.
How do I create a Web Robots file and put a sitemap xml link in there? Simple - Just open your favorite text editor and add something similar to the following…
User-agent: * Disallow: /cgi-bin/ Sitemap: http://geekalicio.us/sitemap.xml
…Then save that file as ‘robots.txt’ (without the quotes) and place it in your website root/main directory. What that will do is tell all those nice little bots not to index anything in your cgi-bin directory, and also, and more importantly, tell them where to find your sitemap xml file.
Remember - we want to make it easy as possible for Google, Yahoo!, Live, and the others to find all your wonderful content. Creating a sitemap xml file, along with a robots.txt file pointing to your sitemap file, won’t guarantee instant indexing, but will certainly help and we could all use all the help we can get. 
