Table of contents
1. From declared to institutionalised user-centricity
1.1 Quality, diversity and availability
1.2 The diversity of requirements accelerates learning
1.3 Concrete measures taken since the creation of the UX Lab
2. On-the-job learnings
2.1 Every study is an experiment
2.2 Systemic thinking beyond individual studies
2.3 Choosing the right test users
2.4 Cyclical vs. situational research process integration
2.5 Buy-in, visibility and consequences of achieved insights
3. Our conclusion after 18 months with the UX Lab
1. From declared to institutionalised user-centricity
As a product developer, catalyst and sometimes also acting as an extended workbench, UXMA supports a wide variety of projects for customers from all industries in parallel, in small to larger project teams at 5 locations in Germany and Greece. For this purpose, we offer assorted services, from software to industrial and UI design to user research, everything under the umbrella term “UX”.
Although some of our projects are isolated undertakings (acting as catalysts in workshops, design sprints or short-term projects), we serve most of our customers on a long-term basis. We have been supporting some of them already over several decades and are thus in a position to continuously improve their products and adapt them to the changing requirements of users.
The special thing about all this is that UXMA itself does not develop products, but rather helps its customers to create successful products. Since the company was founded, our core competence has been user-centricity, which we always ensure with a wide variety of measures, user research among them.
User research has always been an integral part of our projects. In 2020 we institutionalised our explicitly stated user-centricity and founded UXMA’s UX Lab. Its particular goal was to establish a central point to generate and collect user knowledge and skills, as well as to provide tools and templates to all employees undertaking research tasks.
1.1 Quality, diversity and availability
This was a big step for quality assurance. Because until then, although regular exchanges took place within the company and successful concepts were shared in the form of best practices and success stories, all research tasks had been conducted independently and in a decentralised way within each individual project. But there was neither a central entity responsible for implementation, advice and support, nor a uniform quality standard or a central repository for the diverse materials.
Our research activities within the different project teams continue to be different depending on the customer, the project’s status and the framework conditions, just the same as before. But while research is firmly established and cyclically anchored within some projects, isolated but more extensive ad-hoc studies are conducted for other projects.
And there are other projects in turn, where research has been rather cautious so far, and only takes place at critical development points. But the UX Lab results in the activities conducted having more visibility within the company. There is an increased exchange of information about the various methods, their advantages and disadvantages, their costs and efforts, and the successes achieved. This has led to a significant increase in the diversity of the research activities conducted.
And last but not least, the creation of the UX Lab with its experts significantly lowers the entry barrier to doing research. While previously each project team had to plan, offer, conduct and evaluate its research activities by itself, something that was sometimes associated to reservations, uncertainties and a lack of capacities, now efforts and skills can be outsourced to the UX Lab. Its experts can be consulted professionally and freely by the project teams commissioned with activities or fully included in project teams on a temporary basis, as required.
1.2 The diversity of requirements accelerates learning
But it is not only the project teams who benefit from the existence of the UX Lab. Its members also benefit from the enormous range of different subject areas with new content-related and methodological challenges, as well as with the professional and personal exchanges with the projects’ staff, customers and other contacts. Thus, a symbiotic learning, from and with each other, on all levels, arises. In addition to the optimisation of setup and tooling details, there is also an interdisciplinary exchange on research concepts and an active transfer of knowledge between the different competences.
On top, the collection and centralisation of user insights from different areas and industries results in a large technical knowledge transfer potential on all levels. And in some cases, this allows improving the quality of the design even before the actual research begins. This means that evaluations can be started with a lower and more targeted variance.
Thus, from a UX-Lab which was once mainly intended as a quality assurance measure, another, different UX-Lab emerged, acting as a catalyst in which a synergetic value arises as the result of various research undertakings from different industries, going far beyond the insights of each individual undertaking. In addition to the extensive wealth of experiences on user behaviour and perceptions, and the usual methods, set-ups and tools commonly used for these purposes, knowledge about current social and technical trends also grows. The latter has a particularly high value as re-input for the design skills within the company.
On the other hand, the UX Lab is a living, growing system prepared for a high degree of learning due to the constantly changing requirements and to be able to always answer new questions and be up-to-date with the state of the art of the diverse industries, as well as to meet technical and social requirements. In addition to evaluating the actual research content, there is also an evaluation, transfer and further development of the research setup in line with our agile principles.
1.3 Concrete measures taken since the creation of the UX Lab
- Making all research activities accessible, from recording requirements to drafting of offers, planning, implementation and evaluation, for all projects, for all employees;
- Appointment of experts for quantitative and qualitative research, for flexible consultation and support for all project teams;
- Analysis of numerous previously conducted research activities with regard to lessons learned and best practices;
- Definition of action points as the basis for stable and fault-tolerant work processes in all phases of research;
- Creation and cataloguing of templates for offers, project plans, guidelines and questionnaires, protocols, reports, etc.;
- Implementation of a cross-project and cross-customer research repository for the collection, evaluation and synthesis of all collected data and insights;
- Expansion of the portfolio and constant evaluation of various tools for all tasks related to research (prototyping/design, implementation, analysis, archiving) and research methods (online surveys, (un)moderated user tests, etc.);
- Structured division of learnings, insights and fail and success stories in the form of pitches, trainings, workshops and consultation, both internally and externally.
2. On-the-job learnings
What cannot be expressed in one key point, however, are the numerous invaluable findings arising from the interdisciplinary work of our experts since the introduction of the UX Lab. The merging of skills from market research, qualitative user research, various design directions and software engineering, has diversified the range of methods we offer and allowed us to deal more freely with the adaptation and combination of different methods. This has made it easier for us to find a tailored research plan for every question, characterised by an efficient, quick and targeted answer to the problem and able to react to changed requirements anytime.
2.1 Every study is an experiment
Just as we live it in design, in UX research we are also open to conceptually creative approaches resulting from the respective requirements. In our everyday research tasks, we daily experience anew that no test setup can be similar to another one. In addition to the rough categorisation of research activities into methods such as “usability interview”, “online survey” or “unmoderated remote UX test”, numerous other factors also play a role in the success of the study, for example:
- The consistent alignment of the study setup according to the goals or aims of the study (e.g. usability, acceptance, needs assessment, etc.);
- The screening of the right target group;
- The level of prototypes or stimulus materials according to the product’s development status;
- The scope of the test and the possible efficient combination of several research questions in a single setup;
- The correct onboarding depth of the test users to achieve the necessary mindset as the basis for the correct measurement of the relevant parameters.
Our experience shows us that the quality and representational power of the results depend directly on the correct pre-selection, planning and implementation of the setup. Thus, we attach significant importance to the best possible individual setup for each research undertaking. The undertaking’s costs and effort should always be in proportion to the economic benefit of the results of the study, something that often poses particular challenges.
2.2 Systemic thinking beyond individual studies
Particularly in the case of our long-term projects, we do not think from research question to research question, from activity to activity, but across products or even product systems. This is especially important for the efficient integration of research tasks into the product development process.
Viewing the success of a product within its development and use cycle as a whole by means of the close cooperation between design, development and research, helps to ask the right questions at the right time and to quickly share the insights collected with everyone involved. Thus, we can make joint decisions and derive measures that enable the next step in product development and lead to new research activities. An iterative process that must not necessarily be always anchored to fixed cyclical frameworks.
2.3 Choosing the right test users
We have also learned a lot about choosing the right test users. Although in the past we used to recruit test users exclusively through our customers, we now also like to use specialised providers of panels and other services to recruit the desired target group.
We classify our screening of test users roughly into different categories. The choice is mainly influenced by the aim of the study, but also by scheduling and budget-related factors.
Almost no requirements
This applies, for example, to rapid user tests, dry runs or preliminary studies that we carry out internally with employees and that mainly serve one of these two purposes: testing a setup or guideline concerning its comprehensibility and technical process reliability or enabling a rough evaluation of a certain question to save costs and time.
Experts from our own disciplines
We also like to get specific expert opinions, particularly from the areas of design and software, in the form of short expert interviews or surveys. This allows us to generate a large amount of expertise with very little effort.
Neutral test users with low screening requirements
Recruiting test users with low screening requirements (e.g. only gender and age) is particularly meaningful for usability tests in which measured values such as acceptance or need are not relevant, or when a generic product is to be tested.
Neutral test users with a specific screening profile
Many of our studies target the knowledge and opinion of our exact target group(s). In such cases, it is imperative to interview or test exactly the right users who, for example, are already actively using a specific product, are in a particular life situation or have specific expertise. Often it is also a combination of several specific criteria. It has been shown time and again that real user opinions cannot be replaced by internal surveys, and that the results of both differ significantly.
2.4 Cyclical vs. situational research process integration
In our everyday life we use both the cyclical integration of research into the development process (e.g. accompanying sprints) and the purely situational reaction to questions that arise. Both have advantages and disadvantages and, in our experience, both approaches have their raison d’être.
A cyclically repetitive procedure provides great process reliability. Many work steps can be fully or partially automated. A process optimisation usually takes place by repeating the same setup. This makes the research process leaner and more resilient. The reliance on well-established process patterns with the same tools, service providers and procedures with long-term secure planning, offers security to researchers and other stakeholders. An institutionalisation of this kind can be particularly useful in companies where it is challenging to obtain timely approval for research activities.
On the other hand, a long-term routine also has its boundaries. For example, it can be difficult to react to constantly changing requirements in everyday agile life. Capacities can be tied up unnecessarily and costs that are not always required in practice may arise. In addition to active user research, many research questions can also be answered by other means (desk research, analysis of usage data, feedback channels, etc.). Keeping the freedom to decide at short notice which research methods are to be used when and to what extent, can often produce more targeted results. This also helps you to have the ability to react to external influences. The downside is often a higher overhead: new activities have to be argued for, approved, planned, carried out and communicated again and again.
2.5 Buy-in, visibility and consequences of achieved insights
We have made the experience that study results often have a short-term value (e.g. the clarification of a specific question in the development process), and that it is relatively easy to identify those directly affected (e.g. product owners, ideally also the entire development team) and provide them with detailed information on the results. But the results usually also have a long-term value. It does not matter whether the survey involves qualitative in-depth interviews or short quantitative online surveys. Information that could offer added value for other departments or later questions is almost always obtained.
But for the information to reach its recipients, it must be made accessible, visible and usable. The implementation of a Unique Point of Truth, e.g. in the form of a brand guide, wiki or research repository is useful for this purpose. What is important here is to prepare the information in such a way that it gets to all recipients exactly where they are. This concerns, for example, a clearly comprehensible presentation of the experiments carried out, a careful presentation of the methodology, and a summary of the main points. The insights should be prepared in a visually comprehensible manner, something particularly important to prevent misinterpretation. Especially if, for example, text-based key learnings are only skimmed over and not read thoroughly.
Something that is also important to us: As creators and executors of the experiment in question, we have the authority to interpret it. The simple representation of the facts can easily be misinterpreted by outsiders (without a research background) and lead to inconsistent information about the significance of the research. Thus, it is particularly important to provide a clearly understandable interpretation of the results and, in the best case, to present them to the most important stakeholders and also make a clear statement about which conclusions cannot be drawn from the study and which questions could perhaps not be clarified.
3. Our conclusion after 18 months with the UX Lab
The number of our research activities has risen sharply in the last 1.5 years. What began as an institution intended to help us to optimise the existing research activities within our development projects, has continued to evolve. Meanwhile existing and new customers approach us for pure research activities. We have recruited experts who, with their prior knowledge, make a major contribution to the professionalisation of the UX Lab. In addition to the permanent employees of the UX Lab, other project members are also interested in planning and carrying out regular research activities, and they sometimes do so in a completely independent manner.
The range of methods covers both disciplines (qualitatively and quantitatively). In addition to the commonly used methods such as usability tests and online surveys, we always integrate other methods such as unmoderated quantitative UX tests into our everyday project tasks. The recruiting of specifically screened test users has become routine and has proven itself several times in comparison to conducting research with internal test users. Nevertheless, internal rapid user testing is still carried out if required.
Despite the overall positive development, the UX Lab is and will remain a constantly evolving experiment, to be watched and realigned. A well-filled backlog full of internal UX Lab tasks will still occupy us for some time. We are also still trying to establish the right degree of interlinking between the UX Lab and project teams. Whether the UX Lab should be a voluntary offer or a mandatory testing entity for all research activities also has to be determined. And the questions of whether new skills will be needed in the future, whether tools, methods and organisational models will prove their worth, or whether the UX Lab can increasingly be used as a catalyst for mutual learning in the project business, are still open.
We are currently looking back on an exciting and successful creation period, which has not only brought many innovations, many experiments and many firsts with it, but also many experiences. And for the future, we would also like to have new questions, goals and challenges to broaden our horizon.
