Many visits to websites now are as a result of a search made by a user. In fact you may be reading this very article as a result of using a popular search engine like Google or Yahoo. Traditionally, to make search engines of new content on your site that you might like to make visitors aware of, you had to wait for a visit from a Web Crawler, a program used by search engine companies that visits websites and analyses them to find out sort of information they contain. The problem with this is that you don't know how long you might have to wait for that Web Crawler to next pop round and share your new content with the world. Fortunately, this has now changed, and the big search engines have created a mechanism to update them with up to date content information about your site.
The mechanism they have chosen is called Sitemaps. These are basically an XML document which lists the contents of your website. You can register the locations of these sitemaps with the big search engines and then upload a new Sitemaps format XML document when you want to update the search engine about the contents of your site, after that it is up to the search engine to decide how to use that information.
So our task is broken down into two parts, the first is to create the sitemap, the second is to tell the search engines that it exists. Luckily for us we are using Drupal so we don't have to worry about creating that XML file ourselves, in fact a great module is available that takes care of that for us and will handle the validation work we must do with the search engines later. That module is XML Sitemap. Once you install it into your Drupal installation and activate it, it will handle requests for sitemaps.xml which will appear under the URL for your site. If you access that file directly, you will notice that the XML gets formatted for you into a human readable form.
The next step is to let the search engines know that we have a sitemap available. In one tab of your browser you might like to have the XML Sitemap settings page open (Administer -> XML Sitemap -> Search Engines).
You will see that there are expandable areas for each of the major search engines. In these ares we will put details that the search engines provide us with to verify ownership of the file. This used to mean creating a file with a special name on our website. Fortunately, we need only to tell the XML Sitemap module the name of this validation file and it will create it for us. For Google, Yahoo and Windows Live we must go to their areas for Webmasters and register the details for these sites. This also means registering for accounts on every one of these sites. Ask.com does not seem to require this step.
You will need to register your details for the search engines at each of the following, then you can enter the verification details you are given into the XML Sitemap settings page:
- Google: http://www.google.co.uk/intl/en/webmasters/
- Yahoo: https://siteexplorer.search.yahoo.com/submit
- Windows Live: http://webmaster.live.com/
Note you do not have to create any files, just put the verification infomation into the XML Sitemap settings screen and it will handle this for you. The search engines may need a while to verify your ownership of the site so you may have to revisit them the next day to complete this task.
After validation is complete, you can tell the search engines where your sitemap is. You can also then get XML Sitemap to submit a new sitemap every time a new item is created, or maybe just to do on a cron run if your site has a lot of new content added to it every day.
Hopefully this will mean that search engines become aware of your new content much sooner that would otherwise have been the case.
Re: Tell search engines about new content on your Drupal site
Hi,
Are you using Drupal 6? And, if so, what site map module do you use? The ones I've seen only go up to version 5.
Re: Tell search engines about new content on your Drupal site
Hi James,
I'm using XML Sitemap, which currently is only available for Drupal 5. However, it appears there is a patch and discussion going on about updating this very useful module, have a look at: http://drupal.org/node/157533 I'm sure it will be updated at some point though.
Hope this helps.
Re: Tell search engines about new content on your Drupal site
Ah, that's what I figured. I have been following that discussion, but development is not quick enough. :)
Re: Tell search engines about new content on your Drupal site
I always read that this was a bad idea to submit your sites to search enignes. Is this true?
Re: Tell search engines about new content on your Drupal site
Hi Rhonda,
I was told the same things as well. Submitting to search engines is the last thing that i will do. I was advised to get links from indexed pages and letting the engines crawl to my page. I wonder if other people feel the same way.
thanks
Re: Tell search engines about new content on your Drupal site
@ Rhonda,
Why submit your site to search engine while there are many ways to notify search engine about your site? ( though submitting your site to search engine will not hurt your site). You can try to submit your site to some popular social bookmarking sites if you want (like digg), this will make your site indexed by search engine more faster than solely submit it to serch engine.
Just my $0.02
Re: Tell search engines about new content on your Drupal site
Pointing a few external links to the most important new content on your site is a sure way to get indexed by Google and also have the page rank well.
Re: Tell search engines about new content on your Drupal site
Yesterday I was reading the similar article somewhere else and one thing which confused me is "Robot.txt". Writer there warned that we should have Robot.txt even it can be an empty file. He stated that crawler even can decide not to crawl your site if you will not have that. This is some confusing logic because we already have xml sitemaps. Is it a necessary thing for Drupal as well?
Re: Tell search engines about new content on your Drupal site
A robots.txt file advises a search engine what URLs it should and shouldn't visit. This is used when a search engine visits your site to index it, and Drupal comes with a robots.txt file supplied, which can be customised. Every Further information on robots.txt can be found at: http://www.robotstxt.org/
It would be desirable
It would be desirable something to add, but except “thanks the author” anything on mind does not go, thanks for article.
Re: Tell search engines about new content on your Drupal site
I used to ping my blog after adding new contrent but know it is not so effective anymore. I recommend you only bookmark your new page, add it to Digg, Propeller and other popular social websites and search engine crawlers will find your new content in a day.
Re: Tell search engines about new content on your Drupal site
Well, there are other ways that work quite well too - like social bookmarking, press releases, and though pinging is obsolete to help you get ranked, it will still get you crawled fairly quickly. Google's Wembaster Sitemap generator is quite good though. Definitely recommend it!
Re: Tell search engines about new content on your Drupal site
I use Drupal 5
People say the 6-th is better in use.
Installation fiel and signatures have been changed, as far as I know.