What is a tracking study?

Tracking Studies help you to continuously monitor a research topic by repeatedly fielding the same questionnaire and comparing how your data points have changed over time.

The topics you may want to track as a business typically range from changes in your target group (e.g. demographics, values, attitudes, media consumption), the perception of your brand (e.g. image), the customer experience (e.g. frequency of purchase, customer satisfaction, customer retention) to the effectiveness of advertisement campaigns (e.g. brand awareness). In addition, tracking studies are very common in electoral polling, e.g. monitoring the preferences for political parties or the relevance and importance of certain topics for voters.

This being said, the mindset behind tracking studies has three major aspects: First, tracking studies have a 360° view on their subject and aim to draw the big picture. They do not necessarily focus on being able to explain all the details but rather on understanding the most important KPIs. This is why data from tracking studies is often integrated and contextualised with external data sources.

Second, tracking studies focus on developments rather than the status quo. Evaluating a specific issue in depth is less important for tracking studies than monitoring whether things are generally developing in the right direction. Tracking studies can also help to assess the impact of unforeseen events in the past, like a new competitor entering the market.

Third, tracking studies have a proactive mindset. They try to detect relevant changes early on to make sure you can take action before these issues become a real problem.

Setting up a tracking study

With this in mind, let’s explore the typical considerations and best practices, when setting up a new tracking study or changing an existing one. In an ideal world, you would try to avoid changing a running tracker, as every change in your method or questionnaire may affect your time series. Therefore, it is key to preserve a high comparability of all points in time. Also, as you will keep running your tracking study for a couple of months or years, design decisions during the setup may have a huge impact on the lifetime value of your tracking study as a whole. This is why it’s so extremely important to set up your tracking study correctly right at the beginning.

Don’t worry, if this requirement seems to be a bit daunting: this article will not only outline the most important considerations during the setup, but also give some guidance on how to change a tracking study later on, if needed. And, as always, if you have any further questions, don’t hesitate to get in touch with us. We are happy to share our experiences with you.

Designing the questionnaire

Tracking studies can cover almost any topic and survey any target group. This is why you will not find a general-purpose template for tracking study questionnaires. However, we’d like to share some thoughts and experiences with you that may help you write your individual questionnaire.

360° view: It’s easy to not see the forest for the trees. This may not only lead to a very extensive (and potentially irrelevant) questionnaire but also increases the risk of missing out on important aspects. Try to focus only on the main KPIs (e.g. brand awareness, brand image) and keep the first draft of your questionnaire as short and crisp as possible. You can then expand this questionnaire by adding relevant background variables that help to explain your KPIs (e.g. demographics, usage frequency, competitive set, media consumption).

Focus on Developments: Another thing you should know is that not every question is suitable for being tracked. If a question is not recurring in your organisation, it should probably not be included in your questionnaire. Also, try to assess the expected frequency of changes. If you don’t expect changes over time, it doesn’t make sense to monitor these questions. In contrast, if your changes are very volatile, you may end up with a lot of noise in your data without being able to explain any of these changes. Be aware that different questioning techniques can also have an impact on how stable or volatile your timeseries are. While, for example, opinion-polls are typically more stable, implicit questioning can reveal slight changes in the emotional attachment of a customer and create a much more differentiated picture. The bottom line here is that you should try to assess the typical time frames for all changes you’d like to monitor and keep them in mind for later, when planning the frequency of your sampling.

Proactive mindset: Think about how you plan to act upon changes in your tracking study. Your tracking study should align with the processes within your organisation. If you have regular quarterly meetings about a certain topic, you may not only want to have fresh data by that date, but also be able to explain possible changes and point you in the right direction about what should be done next. Connecting your tracking study to existing data in your company (e.g. revenues, ad spend) can help to contextualise your insights from the tracking.

Sampling

Sampling for tracking studies is always a case by case decision. However, the overall goal for all tracking studies is to keep your samples as consistent as possible throughout time. This goes especially for data quality: if you base your tracking study on a poor data source, the lifetime value of your tracking study will become inferior over time. Let’s look at the considerations that should be made when designing the sample of a tracking study.

Target group

The first thing to do is define your target group. It is good practice to not go too narrow in your definition. Instead, try to anticipate who may become part of your target group in the future, once your company and its products have evolved. In order to have monitored these developments later on, you should include all prospecting target groups into your sampling right from the beginning. This will also help you to measure to what degree your target group is growing or shrinking within the broader population.

A second point concerns operationalising the target group in the right way. This is especially important for B2B target groups, where a wrong target group definition can affect the validity and feasibility of your study. Let’s say, you may have (potential) buyers of promotional items in mind. If you define your target group by all people whose job position is “CMO” (because all CMOs can decide upon promotional items), you may miss out on companies who don’t have a CMO or all staff that can make certain decisions without asking their CMO. These groups of people still might be (potential) buyers and, therefore, should be part of your target group. Again, try to keep your target group definition as broad as possible, as poor operationalisation may also harm the validity and feasibility of your study.

Method of data collection

Once you have a first draft of your questionnaire and an idea of your target group, you can start thinking about the best method for conducting the interviews. For example, if your questionnaire requires the use of graphics (i.e. logos), online interviews might be the method of choice. The same is true if you’d like to automate your sampling by triggering survey invitations upon certain events in your CRM system (e.g. customer feedback after each purchase). On the other hand, some target groups suggest using telephone interviews, especially if they are not sufficiently available in online panels (i.e. so-called “nonliners”). In some cases, it may even make sense to recruit respondents via telephone to invite them to participate in online interviews. Selecting the right method for your individual study depends on your specific target group and your questionnaire.

Once you have a feeling for the methodological requirements, remember that data collection should always be regarded as a long-term decision in tracking studies. You have to make sure to find a consistent data source with a high quality that powers your time series. Plan ahead and try to assess if the sample source will be able to provide you with enough participants of your target group in the long run. If it isn’t, you may be forced to blend different data sources at some point in the future and jeopardise the consistency of your data as a consequence. In these cases, you might reconsider the method of choice or start with a blended sample right from the beginning to keep it consistently blended in the long-term.

Sampling frequency

Once you have selected a method of data collection, you should define the frequency of sampling. Sampling can be done on an ongoing basis, especially if you trigger the invitations by events in your CRM system as they occur. The more common way though is bundling all invitations and sending them in regular waves. That’s when your considerations while writing the questionnaire come into play, as well as the availability of a fresh sample.

Let’s start with the expected frequency of changes in your KPIs. If you expect frequent changes, you should measure more often. In contrast, if you expect your numbers to remain fairly stable over time, you can allow for longer periods between each wave of measurement. In either case, try to account for seasonal effects while defining the frequency of your sample. Very often, the usage of products and the image of brands vary a lot throughout the course of a year.

Another thing to consider is the availability of a sample. If you have an unlimited source of fresh respondents, you can run frequent waves without running out of participants. On the other hand, if you have a very narrow target group, the feasibility of frequent waves might be limited. In this case, you may either field your questionnaire less frequently or you successively combine the last waves in your analysis to obtain a stable basis for the analysis of your data (e.g. moving averages over the last two waves combined).

Sample size

Last, but not least, a few words on the recommended sample size per wave. The more breakdowns you’d like to analyse for different segments in your target group and the more subtle differences you like to spot, the more cases in your analysis you will need to obtain dependable results. However, the sample size is also related to the frequency. The more often you field a questionnaire, the more a (fresh) sample is necessary in a certain period.

As the topics sample size, frequency of sampling and method of data collection are so closely related, it might take some time to find the sweet spot for all these parameters in some cases. Take your time to go back and forth in your considerations and try to find the best possible option for your study.

Data delivery

The last thing to consider when setting up a tracking study is the most efficient way of data delivery. It is inherent to tracking studies to repeat the same steps in each wave. This is why especially tracking studies benefit from automating all repetitive tasks: automation is not only more cost efficient but makes all insights more consistent by removing the human factor as a possible source of errors.

Data can be delivered in all common formats: raw data, tables or chart reports. However, during the last years, dashboards have become the preferred choice for many buyers. Dashboards are not only able to work with live data, they also allow you to integrate different data sources (e.g. revenues from the CRM system, social media streams, Google Analytics) and provide a comprehensive view of a development. Last but not least, many dashboards have analytical capabilities that go far beyond merely visualising frequencies in colourful (and downloadable) charts. They can be seen as the one-stop shop for all relevant business insights. This is why they have become so relevant for tracking studies in recent years.

Revision of running tracking studies

Even though unnecessary changes of tracking studies should be avoided, some changes may become necessary over time. The business world is constantly evolving: New media channels become relevant and patterns of media usage change. The competitive landscape evolves with new players entering the market or new product categories are getting established. New topics become more relevant. An example from recent years is Corporate Social Responsibility. In addition, our research methods keep evolving. Innovative technologies have become more affordable, and the quality of many methods has improved. Some of the previously hard-to-reach target groups can be interviewed with online panels by now, for example. To keep it short: The golden rule of preserving high consistency should not serve as a justification for sticking to an outdated setup. This is when changes become necessary. But how should you do it?

The most common changes concern the questionnaire by adding new questions or items and removing irrelevant ones. Whenever these changes fundamentally concern the subject and logic of the questionnaire, we recommend performing parallel tests of the new version with the old version in order to quantify the impact of your changes. Another option is doing a separate ad-hoc study outside the tracking study to make sure none of the long-established time series are in any way affected by the changes.

Another typical change is switching from telephone interviews to online interviews, as this can be a more cost-effective solution in some cases. All interviews can be fully automated and do not rely on the availability of a personal interviewer anymore. The same recommendation as above is true in this case: try to do some parallel testing or conduct a separate Ad-Hoc study to benchmark the old time series against the new method. It is common to see shifts in the overall numbers when changing the method and these benchmarks help you to address the effect (and maybe weight your data accordingly to preserve comparability).

Last, but not least, as technology has become more affordable in recent years, try to automate your data reports by incorporating a dashboard. This will also allow you to integrate your tracking data with other data sources, like revenue numbers from your CRM system. The investment of setting up a dashboard will not only provide you with an added value in your analytics but amortise as the tracking study continues.