Search Engines: Research Shows the Difficulty of Optimising Search Aids

By September 12, 2014
Keywords : Smart city, SCAMP, Europe

Recent research into the use of spelling suggestions and related search advice for queries on search engines has shown the difficulty of optimising the assistance provided to people searching online for information.

Although nowadays information is widely available on the Web and largely free of charge, finding it quickly can still pose some basic problems.  To get to grips with the issues and assess which search aids work best, Gareth Renaud, a researcher in Computing Science at the University of Glasgow, set out to examine the user-experience when it comes to difficult online searches. He set up three questions: 1) What type of aid (specifically spelling suggestions and/or related search suggestions) are most helpful to users who are having difficulty finding what they want?  2) Whereabouts on the screen should such assistance be displayed?  3) How many suggestions of each type should be offered?  As the type of help most needed can vary from one popular search engine to another, Gareth Renaud sought to eliminate any bias by creating a search engine simulator called Search ConfigurAtor for experiMenting with PuppyIR (SCAMP), which was specifically designed to test search engine approaches and assess user reactions.  

Frustrating the test volunteers so as to devise better user assistance

SCAMP can be configured so as to simulate and test online search mechanisms and the type of assistance provided.  For this experiment, which Renaud calls the ‘synonym experiment’, the simulator was configured to use Bing as the search engine, testing four levels of assistance: no search aids; spelling suggestions; related search suggestions; and both spelling and related search suggestions.  The type of assistance on offer and the consistent way it was displayed on-screen enabled Renaud to isolate user behaviour in response to the type of aid offered and to eliminate any behavioural bias among people who were used to a particular search engine.  The test volunteers were asked to search for four news items on various topical themes, with an added difficulty – denying them the right to use the most obvious search terms so as to challenge their ability to adapt their search.  For example, when asked to find news stories on Scotland’s independence referendum, the volunteers were forbidden to use such words as ‘Scotland’, ‘Scottish’ and ‘independence’.  The results showed that the ‘frustrated’ users struggled to come up with useful synonyms on their own within a set time-limit.  

Search aid in this form proves not significantly helpful

The minimum number of participant interactions with SCAMP was 8, but some test participants interacted up to 53 times before timing out.  The results show that activity varied quite a lot at the beginning but there was a progression towards perusing next pages rather than re-formulating the initial query.  Accordingly, the participants traversed an average of 4.3 web pages for each of the four topics, a much greater number than previously estimated averages for real web searches by Internet users.  As Gareth Renaud had expected, test participants proved more likely to change their initial search query as a result of the related search suggestions.  However, perhaps surprisingly, this was not the case either with spelling suggestions or the combined assistance mode. In these cases, most participants simply ‘soldiered on’ through successive pages thrown up by their original search query before moving on to the next news topic.  Despite all the aid on offer, participants did not make much effort to re-formulate their search: on average there were only 1.5 query re-formulations per topic. Some volunteers actually complained of a surplus of assistance and indicated that the way the suggestions were displayed had ‘cluttered’ the screen, negating any positive contribution to their search results.  Consequently Gareth Renaud states in his paper that the original hypothesis – i.e. that people would be able to find more relevant links if automatic assistance were offered – is not supported by the test outcome.  Overall search performance was not significantly improved by the assisted query re-formulations and there was in the end no significant difference between the related search suggestions and spelling suggestions.  Future investigations should therefore be conducted on a larger scale, include ‘eye tracking’, and also update the model so as to display suggestions optimally and reduce screen ‘clutter’, advises the Glasgow University Computing Science expert.


Legal mentions © L’Atelier BNP Paribas