An App for event sampling in search engine tasks

Given that assigned tasks are completed in a different way to self-selected/naturalistic tasks, it crossed my mind some time ago that an interesting way to gather data on naturalistic tasks might be to ask people to install an ‘event sampling’ app which could be tied into research and some nugget of information users might find interesting.

A nice example of this sort of approach, using apps for event (or time) sampling, is the mappiness app ( which pings users once (or more) a day to gather GPS data and ask the user about their current mood.  The idea is to collect data on times/locations of moods, the user pull is 1) help research 2) see your own personal graph.

If someone interested in search created an app of this type, information could include things (of interest to users and researchers) like:

  • Normally you take x time searching for answers —- this time you spent y time searching for your answer
  • Normally you select m number of links from a search page —- This time you selected n number of links from the search page
  • You said page p had the correct answer – you could have take shorter route q (or clicked x fewer times or something) perhaps either involving a path analysis that includes either an algorithm based ‘shortest path’ or a collaborative filtering based one (‘users who search for x found this result much faster than your search for y’).
  • Something about types of queries made/whatever, this could include (if installed in more than one device, e.g. phone and PC) something about the types of queries made on different devices (and other differences in measures across the two)

If you could also include questionnaires based on some sort of self-efficacy, or search-satisfcation measure I imagine the data would be quite revealing.

Of course, this might lose, for example some of the (uninterupted) nuance in multi-part search tasks, for example those on agoogleaday often include two parts e.g. “how many floors does the building on xy street in New York have?” in which one first must find out what the building is, and then the number of floors.  It also might not be very good for exploratory tasks – although perhaps asking people to ‘bundle’ the searches they’ve made into a particular “need”  they were trying to meet would be interesting.

…writing this I can’t help but suspect such a tool already exists?  Or perhaps there are arguments for just using custom systems to track all searches (rather than event sampling) which I’ve neglected?  I’d welcome thoughts/feedback!

[EDIT 29/04/2013] Just been reminded of HCI Browser

“The HCI Browser is a tool designed to help administer and collect data for studies of web information seeking behavior. Studies involving web search and browsing often involve presenting participants with a set of tasks to do on the Web and then recording information about the web sites visited, searches conducted, windows opened, and other interactions with the browser such as clicks and scrolling events. Sometimes, questionnaires are administered before or after each task to ask participants about their experiences. The HCI browser supports these types of studies and is designed to automatically present tasks, administer questionnaires, and collect interaction data. A major goal of the design is that it will guide participants through the questionnaires and tasks with minimal experimenter intervention.”

It’s not quite what I’m describing because it’s less random…but if it could be triggered periodically then the only thing that would need adding is that trigger mechanism, the rest of the system is built there I think!

Print pagePDF pageEmail page

This Post Has 6 Comments

  1. daveplml says:

    Hi Simon,

    That’s a provocative idea.

    1. I enjoy the possibility that it could provide actionable feedback to people about some kind of search; but one challenge is that not all searches are created equally. For example:

    * When I’m searching for information about the history of American education, my queries are complex, utilize multiple sites and as a result, take a long time. This isn’t a bad thing – it’s just the reality of reading through complex information.

    * If I’m searching for the address of a coffee shop, my search might take 15 seconds (or less).

    So, the unequalness of searches means that the data you’d capture would be fuzzy, unless coded for open-endedness.

    On the other hand, capturing satisfaction, termination (was actionable information found?) or other dimensions might productively enable a better understanding of basic search behavior and fulfillment.

    2. I’d push a bet against the fact that different devices are used for different queries. Phones and mobile devices, for instance, are probably used to query locations more frequently as a percentage of all queries conducted via the device as compared with a computer, which may have a greater share of information queries. But does this tell us anything helpful?

    3. As far as I know, no such automated, open/accessible system exists publicly; but I bet: (a) Large search providers utilize such systems to generate realtime data for their usability groups; (b) Observational studies have replicated some of what you’ve mentioned, but in a non-automated way.

    4. Perhaps it’s also a question of technical complexity: how would an open-source system work as a plugin for Firefox, for example?

    • Simon Knight says:

      Thanks Dave, to reply:

      1) So the issue with just tracking search is exactly as you suggest – not all searches are made equal. And it’s worse than you suggest in fact, because search satisfaction (while commonly used) is problematic in that plenty of users are satisfied with bad/inaccurate/wrong results, or can’t really say why they’re satisfied (which is probably one of the aims of search in education, right?). There are other ways of assessing some of these issues (e.g. ‘lostness’ by looking at a to-and-fro of clicks between SERP and result, or back to the same result event).

      What I like about the idea of event sampling, is that we have a deeper level of analysis at a less intrusive level than if we were asking about EVERY search, and a more naturalistic one than if we observed in class. So we can ask questions about the type of search it was (We could ask users to ‘tag’ their search with predefined kinds), we can ask them how satisfied they were – sure, but we can also ask which result they used and why. Perhaps the trigger for the event sampling would be opening the search page – and on every ‘n’ occurrence they have the option to turn on ‘logging’ and afterwards fill in a short questionnaire. The aim is to get meaningful data, and try to make it fun/interesting/unobtrusive to the user. I don’t know if the mappiness app was particularly successful but I personally really liked the principles underlying it (check it out if you haven’t already! :))

      2) Device specific searches might not tell US anything useful, but it might be an interesting enough hook that users want to see it in their feedback. Also, it does tell advertisers something useful – that’s why they care about this sort of information. Might not tell us anything so interesting about search as such I guess, but there might be scope for ad-hoc collaborative search based on location and app enabling in the future or something? I don’t know…

      3) The Lemur toolbar is a firefox addon which can do some of this stuff, and I have seen some other tools which could be changed to fulfil this suggestion – but no app so far.
      3a) Probably…although they’re doing a lot of big data no-user-interaction stuff where they can infer from patterns of behaviour satisfied users, etc.
      3b) Yup, and generally in more controlled environments – event sampling and critical event sampling is something I’m really interested in from the psychology literature with respect to trying to move out of experimental or controlled observation situations. So, for example, I saw a psychologist present on something to do with putting kids to bed and who was involved, etc. (critical incident) where the participants then had a questionnaire to fill in around the time of the event. Others are doing more random sampling (e.g. mappiness)…I’m sure I’ve seen others but they escape me off the top of my head 🙂

      3. As far as I know, no such automated, open/accessible system exists publicly; but I bet: (a) Large search providers utilize such systems to generate realtime data for their usability groups; (b) Observational studies have replicated some of what you’ve mentioned, but in a non-automated way.

      4. Perhaps it’s also a question of technical complexity: how would an open-source system work as a plugin for Firefox, for example?

  2. daveplml says:


    Hmmmm. mmmmm. mmm. m.

    The ‘mappiness principle’ is basically that widespread, semi-random data collected daily (or greater), stripped of context, can lead to a deeper understanding of the currents and fluctuations in national happiness.

    This seems different – on a number of levels – than embedding a Q/A app into a highly contextualized search process.

    Wondering: How can the ‘mappiness principles’ best be applied to more deeply understand (re)search?


    • Simon Knight says:

      Well, not quite, the mappiness principle is broadly that our measure is happiness, and our variables (or context) are: time, location, activity, whatever we can tell from a photo. So it’s stripped of some context, but by no means all.

      1) What’s the equivalent measure for search, which could apply across varied contexts, types of search (exploratory v. retrieval), collaborative v individual, etc.

      2) What context factors are we actually interested in, and to what extent can we measure those automatically versus needing a short user questionnaire

      And then, which of these things are users going to be most interested in and in what format do they want to see it? (If they want to see it at all – I mean, that’s an important question right, do we think people would download something like this? Would they be interested?)

Leave A Reply

You must be logged in to post a comment.

%d bloggers like this: