Google’s new toy similar image search looks intriguing, but still it does not allow query by images directly.
True content-based image search is definitely very hard, and so far I haven’t seen any system that (remotely) works.
I do not know exactly how Google’s similar image search is implemented, but it could entirely avoid content-based image search by providing similarity links between images that are already indexed in Google’s database. Thus, the similarity computation could not only be conducted offline, but also leverage contextual information such as texts surrounding images. The similarity links could even be edited by humans if necessary.
I still remember the first encounter that prompted me to feel the need for content-based image search. After submitting my camera-ready paper to SIGGRAPH 2000, Stephen Spencer asked me to secure proper copyrights for all images used in the paper. But there is this famous little green texture 
that I simply could not track down the original ownership (even until today).
If there is a system that would allow me to do so, I will consider it a real success. (I am looking forward to Google’s official deployment of their similar image search system to see if I could unveil the true origin of “my little green friend”).
