I went to a fascinating event last week, which questioned how mainstream usability techniques work and arguing the value of taking evaluation out of the lab and into the field, "demonstrating rich insights into system use that could not have been gained from think-aloud user testing". A nice addition to the debate on the value of lab-based usability testing from the UK Usability Professionals' Association I thought .

The presentation, from Stephanie Wilson of City University and James Fone of FramFab fed back the results of a field study they conducted of shift 'handover' systems used in a busy hospital paediatric unit. The study was carried out using relatively soft tools including observations, interviews and experiments as an in-the-field study, rather than the traditional user lab approach. The background is that according to the NHS National Patient Safety Agency, there are approximately 900,000 safety incidents a year - and with UK doctors having no standardised handover system this issue that is of key concern.

At the heart of the study were two 'artefacts' used in the handover to supposedly provide a 'state of the ward' - a handover sheet and a doctor's book in the form of a diary. A slide picture of the bulging doc's diary and another of the MS Word-created sheet helped drive the point home. What their study revealed was that the handover was not simply a technical transfer of information but also a mix of team building, planning and problem spotting. In other words, just the kind of complexity which is often missed in traditional usability testing which takes place in a relatively controlled environment. The traditional empirical measures of task success and task completion for example really don't help measure medical staff's experience of the critical, dynamic and sensitive healthcare settings.

What they found was that location made a big difference, influencing both the process and the outcome (and that therefore usability testing needs to consider the quality of outcomes, not just the process). Plus that in reality at ward level, not everyone took the local system seriously - Stephanie contrasting the success in traditional usability testing prescribed tasks with the fact that in the wild people generally opt for their own workarounds. While Stephanie and James remained critical of mainstream usability techniques and called for new techniques (James pointing to the time and resource spent in the energy sector in safety systems testing), it was however left to a member of the audience to point out the obvious commercial pressures against post-product launch testing.

Later, reading the online notes of the study, I noticed the researchers had developed a prototype appliance to support shift handover, with the information gathered using tablet PCs and displayed on a large, shared display. Surely a good case for some kind of customised wiki/blog application which gives users a simple lightweight interface to capture that 'messy reality'?