Robots-Nocontent for Page Sections
From my relatively little but significant “web-crawling” experience, one of the major problems is to scavenge meaningful content from the page- which requires that no navigation crap, menus, javascript and adverts should be indexed. Since there is no standard way web-devs design navigation, menus etc. it is impossible to code a parser that works 100% and is a big PITA.
However this piece on Yahoo! Search Blog is welcome news
webmasters can now mark parts of a page with a ‘robots-nocontent’ tag which will indicate to our crawler what parts of a page are unrelated to the main content and are only useful for visitors.
If the trend catches on, and becomes a standard (has to get Google’s support), it would be greatly helpful.
