In another thread, in another tribe, B asked: "Do you know of a good way to parse google search results pages. I am automating some querries."

If I recall correctly, you were developing your DB in Java. I am an almost complete novice to Java, but I have some expertise at doing text mangling operations in Perl. If nothing else, you could use a Perl program to perform a retrieval, extraction and transformation, then pipe the output to your Java app for loading.

In particular, I like the Perl module HTML::TreeBuilder and its counterpart, XML::TreeBuilder. With these two, you can parse a document into an n-ary tree structure, and then there is a set of methods available for traversing these trees. HTML::TreeBuilder isa HTML::Tree, and XML::TreeBuilder isa XML::Tree, so the methods availble to the ::Tree classes are available to objects of the ::TreeBuilder classes, also.

Taking that to its next step, the ::Tree modules have a method called look_down(), which can search for particular tags. You can use this to reduce your context to the smallest unit before trying to extract the useful bits from a given web page.

Of course, there is a drawback to all this, and that is that if the web page layout changes, it all goes to hell.

My understanding is that Google has some sort of a search API, but I don't know if it us usable to the main Google search engine, or if it can only be used to access private subsets (e.g. a company that has installed a Google Search Appliance).