Mystery Shopping is a covert participant observation that aims to recreate the customer experience or to measure the service quality in a structured and methodologically controlled way. Depending on where the customer experience or service happens (e.g. in a local store, online, on the telephone), Mystery Shopping unites a wide range of different approaches and methods under its umbrella. With this article, we want to give you an overview over this complex field and guide you through the most important things to consider when planning to conduct a Mystery Shopping study.
The starting point for most Mystery Shopping projects are retailers and shop owners who have an expectation on how their staff should behave in certain situations, how products should be presented on a shelf or what other standards should be met during the purchasing process. Mystery Shopping evaluates to what degree these concepts actually work in practice, by sending test buyers to the shops and allowing them to report their experiences in a standardised way.
One typical case in Mystery Shopping studies is evaluating the staff on how they advise their customers and their service mindedness. Most often, such an evaluation will follow a pre-defined test scenario and test buyers have to be familiarised with the role they are supposed to adopt for the test (e.g. making a complaint or seeking advice). In some cases, test buyers can be selected because they are actually planning to make use of the service, which will lead to a much more natural scenario (i.e. an advice on the credit line, where an established customer relationship with the financial institution is provided).
Another case in Mystery Shopping are holistic compliance checks that include all aspects of the customer experience in a shop, e.g. the availability of certain products, the presentation of products on the shelf, prices, the atmosphere and possibly also the availability and service mindedness of staff, if required.
Traditionally, Mystery Shopping is performed with test buyers who are sent to a local store. However, it’s worth mentioning that remote tests are becoming increasingly important, e.g. Mystery Calls or Mystery Emails to evaluate the service level of support teams or Online Mystery Shopping to assess delivery times of online shops (especially if these depend on third party suppliers) and how returns are processed.
It’s against this background that you will probably not be surprised to hear that every Mystery Shopping project can have very unique requirements and challenges. We’ll cover the most important points to consider in the following text.
The first challenge of Mystery Shopping projects is actually not so much about the method, but about if and how such projects are announced in the business units being tested. Mystery Shopping should remain “a mystery” for those being tested, – hence the name. Sales staff, support teams or shop owners should not be able to distinguish whether the actual customer is a real customer or merely a test buyer evaluating the experience.
The problem is that employees will usually start looking out for test buyers and try to reveal them, as soon as rumours about test purchases start spreading. This is particularly the case, when these tests are officially announced by the management. Unfortunately in turn, this may have an impact on the validity of such studies, as some employees may behave differently than usual. However, if managers feel that test purchases need to be announced, they should ideally only use a very rough time frame (e.g. next quarter).
It should be noted that, at least in some cases, test buyers must reveal their identity directly after the purchase, especially if articles have been embezzled during the checkout for the purpose of the test. This poses an additional challenge and should be reflected when planning a Mystery Shopping project. You may either choose very short field times to finish your tests before rumours have fully spread, or you go with an ongoing study format to insinuate that anyone could be tested anytime.
Whatever your setup is, test buyers need to be recruited carefully and receive a thorough training in advance to perform the tests without arousing the staff’s attention. This is actually the biggest challenge in Mystery Shopping projects.
The most important success factor for Mystery Shopping projects is using the right testers, specifically the process of recruiting and training them. It is key to make sure they appear authentic, regardless whether they are real customers or just trained for that purpose. Test customers need to represent the target group of the tested product or service in a credible way.
There are three fundamental approaches to recruiting test customers for Mystery Shopping projects. You can either
The specific parameters of your project will help you to decide what approach works best for you: will your test buyers have to meet a certain profile (e.g. a high level of prosperity for buying luxury products), will they need a lot of experience with performing these tests (e.g. when testing the customer service) or will they need to be available at large scale all across the country (e.g. when evaluating retailers).
Regardless of your preferred approach, you should put some thoughts into profiling, training and incentivizing your testers properly.
To decide whether a certain candidate is eligible for becoming a tester, a standardised test should be performed at the beginning. It makes sure that all testers have a comparable profile and that, as a result of this, the risk of inconsistent data is brought to a minimum. Mystery Shoppers are typically selected by their demographic profile (e.g. age, gender) and sometimes by their outer appearance and existing branch-related knowledge and experience.
Once you have recruited the tester, you need to prepare them for the tests. Depending on the complexity of your project, this may go far beyond merely providing them with a project briefing. Some projects may even require a workshop with role plays in advance, to make sure the testers are prepared for any eventualities. Remember, the ultimate goal of your testers is to remain unrevealed during the study.
The remuneration of test buyers typically goes beyond granting a small incentive. The time and effort of getting briefed and trained, carrying out the test purchases and giving a de-briefing at the end of the study should be reflected in the compensation. The right amount can vary a lot, depending on the complexity of the study and extent of prior training. However, if the compensation is not considered as being fair, it may harm the test buyers’ motivation during the performance of the test.
Finally, a few words on the right sample size. The sample size of Mystery Shopping projects can vary a lot, but most often these studies work with rather small numbers. The price per interview is typically high in Mystery Shopping projects and, in addition to that, test purchases may involve opportunity costs for the tested company, as the service staff is not able to cater for real clients while being tested. However, there are also projects with large sample sizes (e.g. when recruiting real customers of retail stores via an online panel).
Another recruitment method we would like to recommend is through online panels.
Recruiting mystery shoppers via an online panel is indeed highly cost efficient and allows you to do nationwide tests without having travel expenses. In addition, doing tests with real consumers may lead to more authentic and relevant insights and help you to achieve larger sample sizes.
However, you will also need to set up additional routines for maintaining the quality. In the most likely case, you will not be able to train these shoppers extensively. You may even have some participants in your sample who don’t take their tasks as seriously as others. Try to verify their feedback, e.g. by capturing the geo location or asking for (a photo of) the receipt.
An additional challenge is that employees of the tested shop may become aware of your project, once you start recruiting on a large scale using an online panel. This may conflict with your goal to remain unrevealed until the end of your study.
As with many things, online is not necessarily the silver bullet for efficiency. If you’d like to hear a second opinion or discuss the right approach for your specific project, don’t hesitate to get in touch with us!
As indicated, Mystery Shopping is not only about sending shoppers to a local point of sales, but there are other approaches worth considering. For example, Mystery calls and emails can help to evaluate the quality of support teams. Another option is doing mystery shopping in online shops to evaluate the whole customer experience from the first visit of the website to getting the order delivered and finally how possible returns are handled. The advantage of these approaches during data collection is clear. Mystery Shoppers are less likely to be uncovered, as they can better mask their identity with the help of conversation guides.
But even if you send shoppers to a local point of sales, you should provide them with a standardised interview guide or checklist on their mobile device. Someone checking the smartphone in a store has become so ubiquitous that mystery shoppers are unlikely to be uncovered because of using them while shopping. The opposite is true, online forms can give shoppers the opportunity to comprehensively document all observations in a standardised way and enhance the quality of their feedback. This data can also be used to validate the information and perform quality checks. You can, for example, check the geo-location to make sure the tests have been performed in the right place. The upload of images can give additional insights into the standardised feedback of the test buyers. In any case you should always include open text fields to allow test buyers to complement standardised feedback with personal observations.
It should have become clear that mystery shopping projects are quite complex and require a lot of knowledge and experience. Therefore, we strongly recommend to partner with an agency that specialises in Mystery Shopping in order to avoid possible pitfalls. An important criterion for selecting a quality provider is its membership in renowned industry associations, such as the MSPA (Mystery Shopping Professionals Association). Needless to say, we are also a member and stand for the highest quality standards.
All data collected during Mystery Shopping projects can be delivered in the typical ways, e.g. raw data, tables, charts or dashboards.
In any case, we need to make sure that all data is compliant with the legal requirements, i.e. that no employee or other customer is personally identifiable. Depending on the specific project, this may require an additional manual step to revise all data before sharing it with the contracting client.
Using a dashboard is an especially interesting option whenever data is collected online. Dashboards allow you to visualise the results right away and enable your client to monitor the progress in real time. This is particularly important if serious issues are detected during the test purchases (e.g. expired food in a store). In such cases, shop owners should be able to react in a timely manner.
As real-time reports may conflict with the legal requirements, you may consider a “dashboard light”-solution to monitor all progress in the field with preliminary data and only for selected variables and get a final overview once all data has been revised. As with most studies, this depends a lot on the specifications of a given project.
Interested in our Dashboard solutions?
Get the comprehensive data overview you need in any format you require. With Dashboards, you get an intuitive interface, optimally designed to visualise your KPI’s and make informed decisions with ease. See our dashboard solutions here.
We’d like to close our overview with an outlook on current developments in Mystery Shopping. Mystery Shopping is truly about customer centricity and, therefore, will remain an important research method in the future. However, we have also seen a growing importance of omni-channel customer journeys in recent years and this poses an interesting challenge. Webrooming or showrooming are just two of the many patterns we have to take into account when systematically assessing consumer behaviour. This doesn’t mean that the customer experience in local stores will become less important: While the quantity of contacts with stationary touchpoints may decrease over time, the quality of such experiences will probably become even more important. Either way, due to the increasing complexity of customer journeys, Mystery Shopping will have to take a broader perspective and conceptually integrate with other study formats (e.g. Usability / UX Tests, Focus Groups).
This underlines our point, that Mystery Shopping studies are quite complex and that no project resembles any other. Recruiting and training test buyers will become even more important as the complexity increases.
Dashboards can reduce the complexity of all available information and help your team to focus on the most relevant insights for them. Their general purpose is always to make someone’s life easier. Explore the advantages, and how to set up and maintain dashboards.
Setting up a tracking study can be difficult, because there are so many variables to get right, and pitfalls to avoid. We have created a guide to help you get started with your tracker.
At Norstat, we understand that effective market research is the cornerstone of informed decision-making for businesses. We meticulously gather, and curate data through suitable methods to ensure its accuracy and relevance. Through the following guide, we will explore what market research is, and how to do it effectively.
If you are looking for an experienced partner to power your study or if you just have a question, we would love to hear from you!
