1. What does Machine Learning mean for users?<\/span><\/a> 2. Modeling phase<\/span><\/a> 5. Wrap up<\/a><\/p>\n Let\u2019s make it clear from the get-go that machine learning is not necessarily the most meaningful term to use. To users, Machine Learning translates to automation.<\/strong><\/p>\n For example, instead of spending time looking for new music or movies they like, they allow the system to give them suggestions based on their past choices and therefore the automation of actively searching.<\/p>\n That\u2019s why when we invite people to use ML-based products, we are asking them to automate part of their operations or even their entire roles. And that\u2019s exactly why experience research matters when designing ML-based features.<\/p>\n Without discovering whether users trust automation, find it useful or indeed have a need for it, your product is a wishful shot in the dark.<\/p>\n Involving users will result in a system that is not just smart but also reliable, trustworthy and, most importantly, used.<\/p>\n UX Research (UXR) can help throughout the entire development cycle<\/strong> but especially in three crucial moments in the ML workflow:<\/p>\n 1. Before investing in a product (discovery and ideation) Research directs ML-based technology to answer the core questions before investing in a product:<\/strong><\/p>\n Once you are all on the same page about the answers and users’ needs, you can move on to the ideation phase and brainstorm and select 2-3 ideas to iterate on till you have a winning concept.<\/p>\n It\u2019s already in the ideation phase when the team is considering different solutions, technical feasibility, and potential profitability.<\/p>\n This is when a researcher can assess if the product idea is reasonable to users and pinpoint any probable challenging scenarios before the ML team begins gathering data for building a model.<\/p>\n Researching users\u2019 mental models throughout this stage is crucial to make sure that you conceptualise and understand their would-be system During the <\/span>modelling<\/b> phase<\/span>\u200a\u2014\u200athis is when training a machine learning algorithm happens\u200a<\/span><\/i>\u2014\u200a<\/i><\/b>UXR brings in more data with the goal of closing gaps or holes in the data.\u00a0<\/span><\/p>\n Before and throughout this stage, research can help gain a concrete insight into how the user is currently going through their step-by-step journey and if an ML model can make their tasks more efficient.<\/span><\/p>\n This is where already existing user journeys and product\u2019s learnability are studied extensively through a combination of methods for the relevant industry through contextual inquiries and lab-based <\/span>cognitive walkthroughs<\/span><\/a>. <\/span>By zooming into the existing mental models, UXR can provide insight into how to train the model, its interface design, and its <\/span>learnability<\/span><\/a>.<\/span><\/p>\n When studying the user\u2019s journey, you will notice parts of the process that users enjoy doing because it\u2019s rewarding to be creative and develop output. <\/span>Such parts of a user\u2019s journey make users feel accomplished because user effort is part of the creative process.<\/b> Carefully studying every step of the user’s journey and their mental model enables you to make informed decisions on which processes to automate, how and why.<\/span><\/p>\n This is basically creating an early low-to-no-code prototype that behaves similarly to the ML-based feature or product. For this to happen, you don\u2019t need a fully-fledged prototype. In fact, you can use the<\/span> Wizard of Oz technique<\/span><\/a> to simulate what the system is doing.\u00a0<\/span><\/p>\n In tests, I have been using a combination of an early prototype with Wizard of Oz to capture users\u2019 verbal and behavioural cues on system interaction. When users see how the system reacts, they can form mental models based on their reactions.\u00a0<\/span><\/p>\n You could help your team by asking the following questions while they are thinking about the earliest prototypes:<\/span><\/p>\n Answering these questions will help your team better understand how to model intelligent algorithms. That\u2019s why you want to conduct user research by pinning these questions on top of your plan:\u00a0<\/span><\/p>\n Besides qualitatively testing the prototype, you might consider other approaches in case your team has very large stakes in the product, this is especially for cases when ML is a visible part of the experience journey.<\/span><\/p>\n That\u2019s why measuring users\u2019 attitudes (trust), behaviour (task duration and usability performance) and mental models (how the system should operate) should all equally matter in ML-powered products.<\/span><\/p>\n You have run your research, provided your team with insights and now they are more confident about the design direction. But your work is not done, yet!<\/span><\/p>\n Now\u2019s the time for more research to shine up to ML modelling by reassessing how the trained model meets the users\u2019 expectations.<\/span><\/p>\n Evaluating the ML-based system against user\u2019s mental model: The importance of frequent behavioral studies is to help your team accommodate the ML-model to the user\u2019s mental model. Continuous research is needed because:<\/span><\/p>\n Once the model is in use, it will be easy to draw conclusions based on analytics and other quantitative data, however continuously testing the model in the hands of the users will reveal feedback on how it feels and reveal answers to such questions:<\/span><\/p>\n With insights collected through direct user feedback, product developers can make educated decisions together around what improvements to implement in the next upgrade. <\/span><\/p>\n The agile philosophy has conditioned us to go for lean user research and test an MVP by releasing it into the market as quickly as possible. While embracing the agile method is great, it cannot come at the cost of quality, and as we have demonstrated, this is critical in the ML space.<\/p>\n Research during discovery, ideation, before and post-launch is incredibly valuable. Make sure you help your team by asking the right questions to shape the direction and the design of your ML-powered products.<\/p>\n
\n1.1 Discovery and Ideation<\/span><\/a>
\n1.2 How can research help during the ideation phase?<\/span><\/a><\/p>\n
\n3. Prototyping the algorithms<\/span><\/a>
\n4. Continuous monitoring<\/span><\/a><\/p>\n<\/a>1. What does Machine Learning mean for users?<\/span><\/h2>\n
\n2. While designing and training the machine learning models (modelling)
\n3. While evaluating the implementations for updates<\/p>\n<\/a>1.1 Discovery and Ideation
\n<\/span><\/h3>\n\n
<\/a>1.2 How can research help during the ideation phase?<\/span><\/h3>\n
\nHere are a few points that UXR can help with clarifying:<\/p>\n\n
\nML\u2019s accuracy depends on the type of industry and user needs within that specific domain. Sometimes, the model is designed to predict user preferences for movies or predict a property\u2019s price based on its features. A few irrelevant movie suggestions or accuracy within a few thousand dollars is probably satisfactory to most users. However, an ML model used for cancer prediction or healthcare should be extremely accurate. And should a model provide solutions within a few seconds or instantaneously?UXR can help figure out industry expectations for different domains by interviewing field experts and end-users.<\/li>\n<\/ul>\n<\/a>2. Modeling phase<\/span><\/h2>\n
<\/a>3. Prototyping the algorithms<\/span><\/h2>\n
\n
\n
<\/a>4. Continuous monitoring<\/span><\/h2>\n
\n<\/b>
\nYou want your system to adapt and evolve with the user\u2019s mental model. This way users won\u2019t be forming their own theories about how an ML-powered system works.\u00a0
\n<\/b><\/p>\n\n
\n
<\/a>5. Wrap up<\/span><\/h2>\n