The importance of documenting research

Why documenting research is important

Recently, I was attending a workshop when a colleague argued that users had a problem with a navigation pattern we were thinking of using. He thought they had a problem but he had no data to support it. I vaguely remembered seeing participants dealing with it in a usability test. But I could not confirm it either.

I opened my laptop and went down a rabbit hole of local folders, network drives, and Sharepoint stuff to try to find the relevant research. To no avail. Was it that it never happened or that I couldn’t find it (or that it happened but I had failed to document it)? I’ll probably never know.

Finding insight is only part of the problem. Once you’ve found that elusive report, you have to make sense of it again. It takes so much time that I often forget about it and leave the question unanswered. Does this sound familiar?

Good documentation can do more than help answer specific questions, though. Finding relevant insight when you need it is great. But what about keeping track of trends or analysing historical data to get yet more insight from it? Can you do that with your current documentation? I can’t – but I’m working on getting there.

Documented research is reusable

The way I have thought about documentation until recently is, probably, fairly typical: as little as possible unless requested by others. For this reason, most of my documentation is in the form of presentations or PDF reports – apart from my own research notes scattered both in digital formats and notebooks.

For most of my career, I worked at agencies and small startups where spending time creating documentation was frowned upon. But I started to see the need for a change when opportunities for reusing research popped up ever more often at my current, B2B position.

Just before Christmas 2018, I wrote a post on LinkedIn asking my contacts for advice on this matter. At that point, I was still trying to store documentation in a report-like format on Sharepoint (yeah, I know).

Thanks to my colleagues’ comments on that post, I learned about a myriad of tools that others were using to store documentation in a better way. I also realised that I was not alone: many UX researchers seemed to be struggling with documentation. Hence the idea to share what I have learned in this article.

Dilbert by Scott Adams

Types of documentation

So, if creating and storing presentations and reports is not the best solution, how are we supposed to document our work? Reports and Powerpoint are not going to go anywhere, not for me at least, and I suspect that’s the case for the majority of user researchers. But think about it: a report or a presentation is a collection of insight, not an atomic unit of insight. Storing documentation as a collection doesn’t help to re-use or generalize it. Field notes are usually not atomic units, either, but a collection of stuff organized by participant, study, or something else. Even if you use your coded insight, it will still not be organized in a way that allows for easy retrieval.

The solution is to break down your insight into the smallest units that make sense by themselves and store each of those units separately and properly tagged. In this way, you are already generalizing your findings. That opinion that a user gave you about that feature you were testing… that is no longer a piece of feedback about that solution, but a piece of reusable insight. I have to give credit to Tomer Sharon whose article The atomic unit of a research insight introduced me to the proper way of doing this.

Here’s an example:

A few users said that the marketing team’s number one priority is time: if the tool saves them time, they will use it. If it doesn’t save them time, they probably won’t use it, even if it increases their efficacy.
This information is useful, stored as it is. But you can further generalize:
Our customers’ marketing teams prioritize time gains over efficacy.

The tool depends on the use cases

Surely, we agree that storing atomic units of research is better, right? But how do you go about doing it? There are plenty of tools available, a lot more than I thought.

Before we take a closer look at any of them, though, we need to have a good idea of what we want our ideal tool to be able to do. I started by compiling use cases in the form of scenarios, like this:

As a user researcher or product manager, when a new project starts, I want to consult existing insight for anything that could help so I don’t do unnecessary research. I may or may not have anything specific in mind, so this can be a search or a browse. I need to be able to extract and isolate anything I find so I can later review it or share it.

I came up with a few more scenarios like that one. It might take you some time to think of your scenarios. But it is a good exercise that will help you with the next step, which is creating a list of requirements. As an example, you may end up with the following:

• able to store insight in an atomic way
• allows anyone in the <x> teams to add insight
• allows anyone to retrieve the insight without having to install software or learn to use a new tool
• scalable in order to grow with the company

After this exercise, you’ll be in a better position to start reviewing tools. It will also help you when you have to go to your line manager with a budget allocation request for a new tool.

The tool depends on the use cases.

Different data storage tools and their strengths

I categorised the tools into two groups: generic (e.g.: Airtable, Evernote) and specialised services (e.g.: Aurelius, NomNom). The latter consists of tools designed for storing qualitative research documentation. In most cases, they are very prescriptive in terms of organisation and workflow. On the flip side, entering data from studies like usability testing and interviews is straightforward. The specialised services also provide support with analysing, coding, and retrieving insight. If all the users of the application are going to be researchers, this is a good way to go.

However, if you are planning on encouraging multiple teams to enter data into the system and to retrieve insight, specialised services may not be the ideal solution. A product manager or marketer will probably be put off by the alien organisation and terminology used in those tools. And what if you want to store feedback from a form on your website, from support queries, from feature requests…? Those types of data aren’t particularly compatible with the content organisation systems used by most specialised tools.

In my search for the perfect system for our needs, I tried and reviewed a good number of the services available. I’m sharing my findings below, but keep two things in mind: first, this is a partial list of services, although I believe all the well-known names are there. Second, my reviews are not objective. I am taking our requirements into consideration.

Airtable

Airtable is quite like a project management system. You could potentially use it instead of Jira or Trello, for instance. The UI feels like Excel on steroids.

Strengths:

• Powerful and flexible.
• Easy to filter and search for things.
• Encourages you to store insights in small bits.
• Forms to enter data without having to use the app.
• Drag and drop files into existing entries.
• User limit is for Editors/Collaborators – NOT for read-only or for data entry via forms.
• API to integrate with your own tools.

Aurelius

A simple and opinionated tool.

Strengths:

• Very simple to use. No learning curve really.
• Good interfaces to retrieve insight.
• Collections of insight allow you to store/retrieve findings for specific topics across projects.
• CSV export should make it possible to move data to a different app if needed.
  

Just Ask Users

In my view, very specifically targeted at User Research, to organise field research.

Strengths:

• Nice organisation by Product > Project > Study.
• Easy to use.

Dovetail

Super simple (simplistic?) UI and organisational model.

Strengths:

• Simple, easy to learn.
• Many integrations, among them: Slack, Evernote, Jira.
• Graphs with trends for tags and the like.

NomNom

Feature-rich and mature. Integrations and tools to analyse findings.

Strengths:

• Integrations for both getting data into it and sharing.
• Insights/summary sections to help analyse findings.
• Easy tracking of long-term data like support cases and social media.
• Search functionality is pretty impressive. A bit of a learning curve, though.

Glean.ly

In beta, but looks promising. Organises research insight in an original way, based on Atomic Design.

Optimal Reframer

Targeted, it seems to me, at directly gathering insight from field or lab studies, organised by project/study.

Strengths:

• Simple UI, nice to use, and easy to learn. A bit inefficient at times, but very good overall.
• Capture Mode allows for easy data input.

Nvivo

A desktop software package. It was not what I was searching for, so I have to admit I didn’t dedicate too much time to testing it.

Strengths:

• A de facto standard used in academic research.
• A mature product, fully featured.
• Analysis tools.

Evernote

A good, flexible all-rounder.

Strengths:

• Indexing of PDFs and easy attachment of any files. This allows you to add your existing research to the dataset if, like ours, it is in report/presentation form.
• Good tagging and search functions. Bulk assigning of tags.
• Many integrations, e.g. Slack, for retrieving insight.
• Adding data via email is a very good option for non-researchers.
• Easy-to-use and efficient interface. Voice-to-text functionality is quite good.
• Online and offline. Desktop, Web, Mobile.

And the winner is…

I had to think twice whether to tell you which tool we selected. The tool fits our purposes and requirements and it might not apply to your situation. I believe there is something to be learned from our selection, though: we went for Evernote.

If that comes as a surprise, you are not alone. I started my search looking mainly at specialised tools for research documentation. I would like to make mention of Aurelius here, a simple yet efficient and well-thought-out application. You can tell that the developers did things properly and researched with users. I can, in fact, confirm that because the CEO contacted me during my trial to ask for feedback.

Aurelius did not make my list of finalists because we decided that we didn’t want a specialised tool. So we reviewed the above list of tools with our requirements in mind and realized that the decision was between Airtable and Evernote. It came down to these options because:

1. We require an easy interface (and integrations) so anyone can enter data, not just the researchers. Airtable allows data entry via forms and Evernote allows data entry via email or Slack, for instance.
2. Both Airtable and Evernote are mature products. So chances are they won’t disappear tomorrow.
3. I read examples of other well-known companies using both tools in real-life environments. And I liked what I read.

Anyone can learn to use it

We gave both tools a try with a real-life project. What tipped the scale in favour of Evernote was the fact that several people were already familiar with it. Also, Evernote does not introduce any new workflows or alien terminology. So anyone can learn to use the tool in minutes, regardless of experience – something that cannot be said of Airtable.

Evernote has one drawback: it is not easy to share insight for a given project without creating a separate user for every person involved (which can be prohibitively expensive). As far as I know, you can create public links to notes but not to collections of notes (Spaces and Notebooks, as they are called in Evernote). For us, this was not a dealbreaker, we can live without being able to share collections. When the time comes, we will simply create a report or a presentation to share insight with decision makers.