Recently I went for a job where I was asked to show a case study to show how qualitative user testing had helped inform a design that I had worked on; beyond hypothesis-led testing using A/B tests or MVT’s that helped optimise (rather than validate) a design in an agile methodology.
Even though I have used multiple types of user test to help inform my designs in my career to date, arguably from this post, mostly in isolation, I honestly struggled to be able to put a convincing case study together to tell the full story and follow that up with meaningful evidence (with numbers). So I withdrew, even though I knew that I was capable of doing the role.
What I have come to realise that hadn’t struck me before; and that is the value of user testing and now how you really must be able to follow up and elaborate on it during interviews. It’s no longer enough to just have the ability to plan and execute them well to get meaningful results. (And I would strongly recommend anyone reading this to take some time out to be able to do this if you are working in this space).
It is fantastic that we now consider user testing to be embedded in our UX process, and that it is expected especially when we are working in environments where we do have more control over what it is that we can research. This is especially true when we are notworking in environments that are very much focused on delivery; which also indicates a very early stage in terms of UX maturity. Even though I know about them, I have used them, I can plan them, and I know how to execute them; my issue is that as of right now I can’t demonstrate it.
The reason for this is because I had spent approximately 10 years doing UX designwork agency side. The issue working agency side (oh so long ago) was that it wasvery difficult to have conversations with clients about undertaking additional steps in a process that distracts from their ability to have features delivered, and which ultimately raises costs. However, I would also add that there wasn’t a pitch or a tender that didn’t go out without a user test as a line item in them.
It can also be really hard to get clients to do just one. That said, I have had successes with card sorting exercises because the client has ‘been involved’ as part of a workshop (therefore no additional cost to the budgeted amount). I have also had successes with getting the client to ‘watch’ tests as they are happening. Strangely, I’ve even had low-fi user tests omitted from a pitch because an account manager decided that ‘the test’, not the learnings, wasn’t refined enough!
In the last few years I had also used a multitude of techniques to help research problems and help to inform designs; questionnaires, moderated and unmoderated tests, guerrilla-style first-click tests. I have even managed to triangulate between analytics, questionnaires and focus groups to prove that an initiate was not worth pursuing because there was no aptitude for it from users.
As of now, the problem is that I have been responsible for planning, co-ordinating and managing these tests; I have been helping others to undertake and deliver that research that is therefore their own work. But right now I can’t tell a story, because I didn’t take the time to do it at the time when the opportunity was there.
These are the reasons that I cannot as of today tell a meaningful and convincing story when it comes to showing how user testing has guided my design thinking from research to testing. The other part of this may be that with my experience, I am much more likely to be able to understand what should work, and what probably won’t, considering the experience that I have; but that’s up to others to decide. It may also be that my approach tends towards being the right side of wrong first off and then learning more from there. This is also a very agile approach that might not work where an organisation isn’t really agile yet. In this case the need for the story still stands.
As UX professionals, it is really important that you are able to tell a story like this. Take the time to ensure that you can do this. You must be able to tell a story about more than just the numbers; even if the organisation that you are working with or for is focused on conversion rates and total revenue. Where you can really excel is by demonstrating how you have been able to show how the qualitative and quantitive can be combined to tell a bigger story. Bear in mind too that what and where you are working may also have bearing on how to portray this too. Choose from two of the following – do you want it cheap, do you want it fast or do you want it to be of high quality; which one do you leave out to get something delivered?
Frazer Cooper 