Google routinely hires ad quality raters who essentially work from home and help Google identify issues in search results pages. Wired cover this in detail:
Google employs hundreds of people around the world to sit at their home computer and judge results for various queries, marking whether the tweaks return better or worse results than before.
But Google also has a larger army of testers — its billions of users, virtually all of whom are unwittingly participating in its constant quality experiments. Every time engineers want to test a tweak, they run the new algorithm on a tiny percentage of random users, letting the rest of the site’s searchers serve as a massive control group. In other words, just about every time you search on Google, you’re a lab rat. link.