Introduction

As valuable as usability tests are, they can prove a difficult method for some studies. Any usability test is just a snapshot and testing a subject for longer than an hour is very rarely worthwhile. Most crucially, however, usability tests can only give you a very limited insight into the repeated use of a product or service over a longer period of time.

But there is another method, which still isn’t too well-known among many UX experts, that provides a thorough understanding of the user’s long-term use, actions, plans and thought processes: the diary study.

1 How does a diary study work?

The basic idea of the diary study is brilliantly simple: you ask your subjects to record their experiences in a diary. They should be writing down everything they do that relates to the focus of your study.

Here’s an example. For the purposes of developing a website, app or business idea, you want to know how people go about buying a fitted kitchen. That’s something you can’t really find out in a usability lab. Sure, you could test a specific kitchen planning tool or conduct a focus group and talk to participants about their experiences of buying kitchens. But the danger with those methods is that the subjects may not mention a lot of the minor steps they took – simply because they’ve forgotten about them.

With a diary study, you can avoid all of these problems. It provides a detailed insight into the complex series of steps that your subjects actually take and what they think and feel at each one. So here’s how you could set up your study about buying kitchens. You recruit test subjects who are actually planning to buy a fitted kitchen in the near future. You ask them to write down everything they do that helps them to reach their decision. That means every time they visit a kitchen studio or a carpenter, browse kitchen fittings through a shop window, talk to friends about their own kitchens and look around them, access related websites or speak to their partner about it.

At the end of the study, you can read through the diary entries and gain a clear understanding of exactly how the subjects went about their purchase.

Tagebuchstudien ein Leitfaden für die Praxis 1

Diary studies involve subjects writing down their experiences – e.g. of a product, company or using a device.

The benefits of diary studies in a nutshell:

  • They clarify complex decisions.
  • They allow you to test the repeated use of a product or service.
  • They allow you to test subjects for longer periods of time.
  • Users are in a familiar environment or actually in the relevant usage situation.
  • Users are not influenced by the moderator.
  • Details are also recorded.
  • Users decide for themselves what to disclose, so they don’t feel as watched.
  • Users record their actual actions and not just what they remember about them.

2 When should I use a diary study?

The purposes of diary studies actually follow on directly from the benefits listed above. It’s worth considering a diary study whenever you’re interested in testing over a longer time-frame.

The other great thing is that, unlike a usability test, which – as the name suggests – primarily tells you something about the usability of specific applications, diary studies reveal a lot more about the user experience. You can see the exact situations in which the subjects use the applications and find out which competitors you’re up against. But most importantly, you also learn a great deal about the non-digital aspects of the user experience, such as when the user talks to other people about it, or when they go into shops, read catalogues, look at advertisements and so on.

The diary will also pretty much document the entire customer journey. You can find out what touchpoints exist between users/customers and the company/product – straight from the horse’s mouth.

Diary studies also give you a particular insight into:

  • Attitudes
  • Motivations
  • Behaviour
  • Usage situations
  • Habits

This makes diary studies especially useful in the earlier stages of a project. You’ll get to know the users, their thought processes, beliefs and, most importantly, their concerns and problems (pain points) inside out.

Diary studies are also a good option if you want to find out about the long-term use of your product. How often do users open the app? When do they visit the website and which devices do they use to do so? Is anyone else involved? What do users like and dislike about your product? How does their use of the product or their attitudes towards it change over time?

Tagebuchstudien ein Leitfaden für die Praxis 2

It’s difficult to conduct usability tests in everyday situations – that’s where diary studies come in handy.

Examples of when to use diary studies:

  • To test the use of a specific product (e.g. cleaning product, shampoo, mixer, etc.)
  • To find out what people do with dried tomatoes – is there anything we still don’t know?
  • To find out how consumers decide which sandwich spreads to buy – how and when do they buy chocolate or cake?
  • To investigate the everyday use of voice assistants, sound systems, televisions, washing machines, pans, etc.
  • To find out how and why subjects buy shares, take out health insurance, etc.

3 When are diary studies less suitable?

If you’re looking for specific ways to improve individual functions or web pages, a diary study isn’t the right method for you. Neither do you need a diary study to test tasks that can be completed very quickly or in just a few steps.

After all, the drawback of the method is that it is relatively time-consuming and involves a lot of effort, for both the organisers and the participants of the study. Evaluating the results at the end is particularly hard work.

You also shouldn’t opt for a diary study if you’re testing tasks that are quite stressful for the user to complete, because the subjects won’t have time to sit and record their usage experience until after the task is over – or worse still, the subjects may be distracted or restricted during the task by the need to write things down.

Diary studies are also less common in the business-to-business (B2B) sector. The majority of subjects find this method extremely personal, which some consider inappropriate in a business environment.

For cases like these, a contextual inquiry would be a more suitable option.

4 What kinds of diary studies are there?

In principle, diary studies are always conducted in the same way. The main distinguishing factor is when the subjects write their diary entries:

  1. A full record of every event (e.g. subject speaks to their partner about the plans for their kitchen)
  2. An initial snippet (brief notes) with details added later (e.g. subject takes a photo of a friend’s kitchen when they go round to visit. Then in the evening, subject details exactly what they talked about, e.g. via an online form)
  3. Regular prompts (subjects are prompted by an app, e.g. every two hours or every evening, to record what they are currently doing/thinking/eating/reading/listening to – potentially a good option for a broader study on eating habits, for example)
Tagebuchstudien ein Leitfaden für die Praxis 3

Diary studies can also involve the subjects simply taking photos to document their usage situation.

5 How do subjects record their diary entries?

Diary studies also differ in the ways in which subjects record their experiences. The most common methods are:

  • Pen and paper
  • Notes made on a desktop, laptop, smartphone or tablet
  • Text/WhatsApp/Twitter/Slack/email/voicemail
  • Online form/website
  • A specially designed app
  • Photo documentation (usually combined with notes, an app or an online form)
  • Video documentation

In a classic diary study, subjects simply write everything down using a pen and paper. They use an empty notebook, just as they would for a real, personal diary.

You can add a little structure to this method by providing forms for the subject to fill in, e.g. a form for each individual entry. This form could include fields like day, time, location/usage situation, description, feelings etc. Some researchers use diaries designed specifically for the study, containing tasks, background questions or similar prompts. But then the focus isn’t so much on usage situations but more on the subjects’ ideas, attitudes and preferences.

The more subjects participating and the younger the target group, the more suitable a digital method. This would make it easier for you to evaluate the results but could, in some cases, be a bit more difficult for your subjects to get to grips with (depending on age and preference). In particular, you should expect entries recorded on mobile devices to be shorter than if the subjects use a real keyboard or makes notes by hand.

And if the study itself is focussing on the use of an app or smartphone, subjects would have to switch back and forth between the actual application and the app they’re using to make notes.

The digital diary does have a lot of benefits, however, which is why it’s being used increasingly often:

  • You can perform interim analyses.
  • You can ask questions when things are unclear (and the subject is more likely to remember the answer at that point than at the end of the study).
  • You can add new questions or adapt existing ones.
  • You can moderate, motivate and give reminders.
  • If you moderate the study, you’ll get higher quality results because the user will notice that their comments are being read and assessed – or because you can support and encourage those who aren’t recording useful entries.
Tagebuchstudien ein Leitfaden für die Praxis 4

The easier it is for subjects to record their experiences, the better your results will be.

6 What kind of results will I get?

Your results will be more or less structured depending on the method you use. Usually you’ll be dealing with written texts, possibly hand-written. A diary study normally lasts one to two weeks with around 15 subjects taking apart.

With this kind of study, you should therefore expect to have to work through hundreds of pages of text. The work goes by more quickly than initially feared because you’ll soon start to notice repetitions, both by individual subjects and between subjects. So the more results you’ve already read through, the more quickly you’ll make progress.

Tagebuchstudien ein Leitfaden für die Praxis 5

Tagebuchstudien ein Leitfaden für die Praxis 6

The information on the app has already been well structured by the subject. Apps also allow subjects to include photos. In comparison, evaluating hand-written diary entries is much more difficult and time-consuming. However, it’s also easier to encourage subjects to e.g. include drawings.

7 How should I prepare a diary study?

Here’s the best way to plan a diary study:

  1. Establish the aim of your study
  2. Decide on the duration and specific tasks
  3. Define your target group
  4. Recruit your subjects
  5. Brief your subjects
  6. Supervise your subjects, check in with them during the study
  7. Debrief and final questions
  8. Evaluate your results

Here’s a brief explanation of each step:

Establishing the aim of the study

As with any good study, the first thing you need to do is work with your team and stakeholders (i.e. managers, clients, etc.) to determine exactly what you actually want to get out of the study. What interests you in particular? What are your research hypotheses? How will you know whether the study is successful in the end?

Let’s use the same example from earlier: buying a fitted kitchen. We’re interested in this issue because we want to create a website for a carpenter. This carpenter is very successful, but now he wants to expand his business to a new city and is considering offering an online service that allows customers to plan their own kitchens.

This idea isn’t without its risks; the fact is, there are already companies out there offering that very service.

So we want to conduct a diary study to find out what really matters to people when they buy a new kitchen. We want to know where subjects get their information, what they experience in the process, what information they’re lacking and what they’re uncertain about. And of course what we ultimately want to know is: which supplier do they choose in the end and why?

So in marketing terms, the study focuses on:

  • Customer journey
  • Pain points
  • Touchpoints
  • Business opportunities
  • Competition analyses/market conditions (who are the competitors?)

Deciding on the duration and specific tasks

The duration of our example study depends on the activity involved: ideally, we will monitor the subjects from when they initially have the idea of buying a new kitchen, right up to the moment they’ve signed the sales contract, or perhaps beyond even that point. It could also be interesting to see how the subjects feel while they wait for the kitchen to be delivered and when it’s being assembled. Or how they feel on the first few days in their new kitchen.

So we’re talking here about a time-frame of several weeks to several months. Of course, that would make this study an arduous task – the subjects have to remember to write everything down over a long period of time. For our part, we need to send the subjects regular reminders to write their diaries and, by the end, we’ll have a lot of entries to analyse.

That’s why, in practice, people usually try to limit the duration of their studies. So you’d end the study when the subject has bought their new kitchen. Then, once it has been assembled, perhaps you could hold a telephone interview with the owners of the new kitchen to get a little extra information at the end.

So the subjects’ specific task would be to record everything to do with the new kitchen.

In practice, most diary studies are considerably shorter. If it’s a study focusing on a fitness app, cosmetic product or the experience of installing and using an electrical device, a week is normally more than enough time. Most diary studies are conducted within this sort of time-frame. Studies can be shorter than that too, but longer ones aren’t as common because of the amount of time and effort involved.

Defining your target group

In our example with the new kitchen, the target group is very important. After all, everyone has a kitchen and lots of people buy them. But an expensive, carpenter-made kitchen? There aren’t many who want or can afford one of those.

For your diary study, as with any user study, you’ll need to think about the criteria you’ll use to choose subjects. For example, think about:

  • Age, gender, marital status, living conditions
  • Place of residence, housing situation
  • Profession/industry, education (home-owner or tenant)
  • Income
  • Do they own certain devices/products?
  • Preferences, hobbies

Make sure you only set out criteria that are really relevant for your study. After all, the more criteria you have and the more unusual they are, the more difficult it will be to find subjects that meet them.
This stage is much easier if you have already figured out personae for your project, as you will have already put together decisive criteria and can start recruiting subjects. You can find out more about personae in the blog: Complete guide: How to create personas based on data.

Recruiting your subjects

The narrower your target group, the more time you should allow for recruiting subjects. It is important that you make it clear to your subjects from the beginning how long the study is expected to last – they need to be available for the entire duration, otherwise you won’t be able to use their contributions.

A week’s notice is probably enough for a week-long diary study, if you don’t have particularly specific criteria for your subjects. If the study is going to take longer, or if you require somewhat unusual subjects, then it’s better to allow three to four weeks for recruitment.

Outsourcing recruitment often makes the process quicker and cheaper, especially if you need subjects with specific profiles (e.g. to Testing Time).

Briefing your subjects

The brief, that is, the instructions you give to your subjects for the diary study, is crucial for ensuring its success.

Start with the time plan – even if you have already done so during the recruitment process, reiterate to the subjects exactly how long the study will take and how much time each day they need to set aside for it. It’s better for a subject to tell you now that they’re going on a round-the-world trip in the last few days of the study than for you to find out after they’ve left.

It’s often effective to divide the brief into two parts:

  1. Detailed explanation for the beginning
  2. A summary to refer back to

When the subjects get started at the beginning of the study, you should explain their task to them in detail, but as briefly and concisely as possible. Structure your explanation with headings and lists. Provide specific examples, photos or even videos that show the subjects exactly what to do.

The subjects can check their instructions in the summary every time they note something down in their diary. This method ensures that your subjects make their notes in the form that you’d prefer to read them.

Remember that the method for recording diary entries needs to be as user-friendly as possible for the subjects. The easier it is for them to make notes, the better results you’ll get.

It’s worth testing the brief you prepare for your subjects in a pilot study. So before you let the real subjects get started, try out your brief on a small group of test users. If that’s not possible, then test it on friends, colleagues who aren’t involved in the study or anyone else that doesn’t know anything about your project. Let your test users play out at least the first few steps of the diary study using your brief and see how they get on. This will almost always reveal more areas for improvement: the instructions could be formulated more clearly or concisely, the tone of address could be more motivating, etc.

In any case, it’s important to ensure that subjects can reach you as quickly as possible – via email, instant chat, WhatsApp, Slack or telephone – to ask you any questions they may have at the beginning of the project. If you’re testing products for end customers, you should ideally be available around the clock, but at least between 7 am and 10 pm, since many subjects will most likely only have time to do the things they’re supposed to be describing in their diaries before and after work.

In some cases, researchers even conduct the brief in person and invite all of their subjects to a launch event. This approach means you can show your subjects how to record their entries there and then. Alternatively, you could show each subject how to record their entries via a telephone call or a conference call using screen sharing. If you wish to minimise the amount of time and money spent at this stage, you’ll need to provide very well-written explanations for the subjects. In this case, we’d highly recommend using sample entries and illustrations or preparing an explanatory video clip that provides examples of good diary entries.

Supervising and checking in with your subjects

You might think that you can have some time off while the subjects are busy writing their diaries. However, it’s become clear that supervising subjects during the study yields much better results.

You should be available to answer any questions. It also helps if you remind them about their diaries every now and then, because it’s all too easy for them to forget to write their entries. If it doesn’t occur to them until later in the study, they will either leave out key thoughts or events about each entry or record them from memory – which will be more incomplete the longer ago the event happened.

If the diary form you’ve chosen allows you to view the entries during the study, then it’s best to set a reminder in your own calendar: you should be taking a look at your subjects’ entries on a regular basis. If you’re doing this, you’ll notice early on if the subjects have misunderstood your instructions, for example, and you can provide a clearer explanation during the study. You might also discover something fascinating you want to learn more about, which you could explore in more depth through other tasks/briefs later in the study.

Don’t just criticise subjects who forget to write entries or aren’t writing satisfactory ones – make sure you also provide positive feedback for those writing good entries. That way, your subjects know that someone is looking at their work and that it pays to be thorough.

Debrief and final questions

The diary study closes with a more or less detailed debrief, that is, a final discussion, or at least an email or survey sent at the end of the study.

At this point, it helps greatly if you’ve already looked over the subjects’ notes, because then you can clear up any uncertainties and ask any important final questions.

This is also your chance to hear what your subjects think and find out what their conclusion is. You can now quantify subjective satisfaction using metrics, for example.

And finally, thank your subjects for taking part in your study – they’ve given you a very personal insight into their lives over an extended period of time.

The final interview is a brilliant opportunity to get a feel for your subjects’ attitudes, behaviour patterns and user experiences – the very reason you’re conducting diary studies in the first place. I’d therefore be loathe to end the study without holding a personal telephone call, video conference or meeting with every individual subject.

Tagebuchstudien ein Leitfaden für die Praxis 7

The final interview is a great opportunity to find out even more about user behaviour.

Evaluating your results

At the end of the diary study, it’s time to evaluate your results. If the study was conducted over a short time-frame with a small number of subjects and a simple focus, this might not take very long at all. In most cases, however, you will need to sift through many pages of your subjects’ entries. This can be fascinating, but extremely time-consuming.

So, here’s my tip: when you’re planning the study, make sure you have a fairly clear idea of the issues you’re really interested in and how many entries you’ll ask your subjects to write. It’d be a shame if you had to throw away any entries because you didn’t have enough time to read them all. It would have been better in this case to request fewer entries. Because in general, the more entries you request from individual subjects, the lower their overall quality will be. It’s better to have three good entries on the day than 30 shoddy ones.

Look back at the research questions and hypotheses that you defined at the beginning. Which of your assumptions are confirmed by the users’ diary entries? Which are not? Which are debunked by them? Did the subjects’ attitudes or usage situations change over the course of the study? Has your study revealed any additional touchpoints or key people?

If you don’t yet have a customer journey map (CJM), now is the time to create one using your findings from the diary study. If you already have a CJM, then critically examine it to see which points you should perhaps change and where you could expand it.

The way you present the results depends entirely on the intended audience. Always bear in mind that most people prefer a few, concise facts over extensive, in-depth reports that take hours to get through.

Tagebuchstudien ein Leitfaden für die Praxis 9

A simplified overview of the diary study process

You can read a complete case study in this TestingTime blog post: How Zalando performs diary studies – a case study.

8 What kind of software/apps can I use to help me with my diary study?

A lot of market research agencies offer software tools to help researchers conduct diary studies, but the majority of them aren’t designed for UX experts and require time-consuming training. They often don’t fall within the right price bracket, either.

It’s worth taking a look at QDC Studio or dscout Diary.

Tagebuchstudien ein Leitfaden für die Praxis 10
mQuest may be another option – although the screenshots from the iOS app don’t look particularly modern and might be less likely to enthuse subjects.

Some colleagues also use customer support tools such as Intercom.io or survey tools like Surveymonkey or Qualtrics. And of course you can always just use Google Forms. The trick with these tools is set out the forms in such a way that they work well for recording diary entries. If you’re using these, it’s essential to conduct detailed pilot studies with test subjects.

9 How many subjects do I need?

As always, the number of subjects you’ll need for your study depends on how diverse your target group is and how many tasks/usage situations you’re interested in.

But remember that evaluating the results takes a lot of time and effort – so it’s probably better to opt for a manageable number of subjects.

It’s standard practice to use 12 to 20 subjects, but I’ve also met colleagues with 80. I do not envy them when it comes to evaluating the results. When you’re planning the study, don’t forget about the time you’ll need to spend supervising all your subjects. With that many subjects, you won’t get much else done during the project.

Generally speaking, if your subjects have been picked out meticulously, or if they have a good incentive or motivation to take part (e.g. if your study is fun or for a good cause), you won’t need as many of them.

Make sure you allow for subjects backing out of the study before it’s over; it’s inevitable that some of them will, and the longer the study takes and the more time and effort the entries require, the more of them will do so. Subjects don’t necessarily back out just because they’re lazy. They might be unable to continue due to personal reasons, illness or unexpectedly having to travel. You should therefore allow for this possibility by over-recruiting to a certain extent.

By outsourcing recruitment, you can often recruit subjects more quickly and cheaply, especially if you have a particularly specialist target group.

10 What kind of incentives/rewards/payment should I offer subjects?

It’s often effective to stagger your subjects’ payment/incentives. That means they don’t get their entire reward in one go, but instead receive part of it at the beginning of the study, another halfway through and the rest at the end, for example. Paying subjects per diary entry can cause problems, as it might encourage subjects to focus more on producing quantity rather than quality. If you pay your subjects based on the quality of the entries, you could end up upsetting them, for example if you decide their contributions are worse than they thought.

Some colleagues work with a points system: subjects earn points for every activity. A short entry earns one point, a longer entry earns two. For every day that the subject writes at least three diary entries, they earn 10 points. If they stay until the end: 20 points. Each point is worth, say, 20 cents. But this approach is quite elaborate and wouldn’t be so easy for subjects to understand.

Of course, you will need to adapt the incentive to correspond to the subjects’ workload. If they have to write two or three WhatsApp messages a day for a week, EUR 80 for each subject is a reasonable guide.

You shouldn’t offer too high an incentive, otherwise you risk getting subjects who aren’t remotely interested in the focus of the study and just want to get paid.

In general, cash is only ever the second-best solution. If you can link your incentive to the focus of the study, it’s usually even more appealing. Perhaps you could offer a voucher for one of the tested services or allow subjects to keep the product they’ve been testing?

11 Can I combine diary studies with any other methods?

Finding out what the subjects actually do isn’t the sole purpose of diary studies, especially those conducted for market research. Subjects are often given additional tasks to work on too, for example:

  • Upload a photo that reflects your current feelings about the product.
  • If you needed to explain the product to your granny, how would you do it?
  • If your best friend asked you what annoys you about the product, what would you say?
  • Over the course of today, take five photos of places or people that remind you of the product.

Subjects can work on creative tasks like this individually or in groups. If the other subjects see the results, they can respond to them and develop the ideas further. This is a particularly interesting approach for brainstorming.

An easier option for the subjects is polls. You can ask your subjects simple questions such as:

  • Which function is your favourite? A, B or C?
  • What do you think of X? Very good, okay or poor?
  • You could even ask open-ended questions such as: what do you find annoying about Z?

If you’re not conducting any final interviews with your subjects at the end of the diary study, it’s customary to at least provide an online survey for them to complete. Focus groups are also a good option.

Conclusion

As you can see, there’s an extremely wide range of situations that call for diary studies. They don’t need to be any more difficult or time-consuming than usability tests. Even if you don’t yet have any experience with diary studies, you can simply start with a small pen-and-paper or WhatsApp study. You don’t need any kind of specialist technology for that, just a little time and a few subjects.

You will find that diary studies produce extremely useful results, especially for the early phases of a project. Give it a go!