“Without data, you’re just another person with an opinion.” The famous quote from W. Edwards Deming, renowned American business theorist and consultant, is as true now as when he first said it.

Data is the cornerstone of almost any business decision, from what products to launch, to which markets to go after, to what advertising to run. But knowing the best way to get that data – and what to do with it once you have – remains as difficult as ever.

That’s where market research comes in. Data, intelligence and insights are the foundation of any good business, whether it is starting out, looking to grow or working on consolidating its position. Good market research can tell a company what consumers are thinking and feeling, what motivates them and what they buy. The keyword, though, is good.

Surveying current and prospective customers or running focus groups sounds like a simple job. The reality is it’s full of obstacles waiting to trip up those who don’t know what they’re doing. Surveys can be too long and the questions too confusing; people running focus groups can ask leading questions or accidentally influence the participants; or the research can be done with the wrong people or too few of them. And the old saying, “garbage in, garbage out” applies just as much in market research as it does in other areas. To get results, you need to know what you are doing.

The difference between quant and qual research

There are two types of market research, qualitative (qual) and quantitative (quant). The former involves in-depth interviews, focus groups, or observations to collect information that has a lot of detail and context. The latter is much broader, involving data in numerical form, usually obtained from surveys or tracking something such as sales or brand awareness.

“Qualitative research really helps you gain an in-depth understanding of what people need and want, what they like and don’t like – and why,” says Jane Ostler, executive vice president of global thought leadership at the research agency Kantar. “Quantitative research is done at a larger scale to get the data and the insights you need – and may be cut into subgroups.

“You can either do these things as dips – just a quick market intervention to find out what’s going on – or you can do it on more of a longitudinal basis, which is generally some kind of tracking programme where you’re asking people the same questions over time to understand the variances.”

Whether to quant or qual first depends on the scenario. For brand positioning, strategy or advertising research, Ostler suggests starting with a qual element to understand the deeper desires, needs and wants of people. You can then test the results with a larger group using quant research.

That is also a better strategy for businesses that have not done much market research. For businesses with more mature market research that are already running quant surveys, qual can be useful to understand an issue that data throws up.

“Quant can tell us what is happening, but it doesn’t necessarily tell us why,” says Andrew Tenzer, co-founder of the research and thought leadership consultancy Burst Your Bubble. “Which one you need depends on your objectives.”

Why a hypothesis is so important in market research

Before a business even starts to think whether it wants qual or quant, it must work out the reason for the research. Failing to do so is a common mistake, says Tenzer. “Often, businesses think they need research when they don’t,” he says. “Plus, research should be hypothesis-driven. People will often say something like: ‘We need some insight on this audience.’ But what is it they really need? What’s the hypothesis? Often people can become a bit unstuck on that.

“They must be hypothesis-led in terms of what they want to find out, and they must be willing to have their hypothesis challenged and their opinion changed. I can’t tell you the number of times stakeholders have found it hard to accept something in the data that goes against their view.”

Ross Farquhar, marketing director of the mochi ice cream Little Moons, recognises this. There have been “countless times” when he has been developing creative work and used research to evaluate it. He describes the horrible feeling of watching people “trash” work you think is good and might have spent months working on.

“That’s incredibly tough,” he says. “But you cannot be the ostrich that buries its head in the sand … You have to show that you are listening to the research, that you have heard the key messages and that you are not stuck on an idea come hell or high water.”

There are other tips to ensure that businesses get the best out of market research. In quant, Ostler warns against surveys that are too long, too convoluted or don’t ask the questions in a way that makes sense to people. Routing is important; no one wants to answer irrelevant questions.

“You have to think of concise ways of doing a survey to make it interesting and possibly even entertaining to motivate people to answer the questions,” she says. “And you should use tried and tested market research techniques, such as a one-to-five scale, and make sure it works on mobiles.”

Sample size is also important. Whether quant or qual, the participants need to be representative of the whole market. For a consumer-based survey exploring the attitudes of the whole UK, a sample of around 2,000 should work – although that will depend on how many subsets a business wants to divide the results into.

For more niche audiences, the sample size will depend on the addressable market. Tenzer uses the example of a law firm that wants to understand high-net-worth individuals better. There are not very many in the UK, so getting data from 100 could well be representative.

However, the more specific the audience, the harder it will be to reach – and the more it will cost. He says: “Businesses need to understand that research with hard-to-reach audiences, whoever they are, is not cheap. Unfortunately, if you work in a niche category and need to understand your audience, you’re going to have to pay.”

Research also shouldn’t be done among current customers only. Farquhar recalls how Little Moons used to gather customer feedback from, for example, its pop-up concessions, and treat that as insight. And while it was valuable, it told him nothing about how the company could expand beyond its current market.

“The customer feedback we got was really valuable,” he says, “but it told me next to nothing about how we’re going to grow the business over the next three, four or five years. The only way we were able to get to how we are going to grow the business is by understanding the market, who is buying products like Little Moons and which groups we think we can legitimately win with. It’s only by thinking about what research is going to get me closer to the market, rather than existing customers, that you sense the value coming in.”

That’s where many medium-sized businesses – as well as the market – are moving. It’s not just about understanding what consumers are thinking, feeling and doing now, but about what they might think, feel and do in the future.

Advances in technology, particularly artificial intelligence, are helping. Ostler says Kantar is using data science and AI to predict consumer reactions. For example, its LinkAI product can look at advertising and predict what people’s responses will be. “One way to use all this amazing data, once a business has enough traction with it, is to use it in a predictive capacity,” she says.

Good market research leads to good business decisions

In the end, good market research should help a business make good decisions. It takes considerable time and money to do well, so the results need to be worth it.

Sampling, as Tenzer has already pointed out, can be expensive, but so can bringing in the expertise needed to conduct the research. Many businesses think they can do this on their own, but Ostler, Tenzer and Farquhar caution against this for all but the most basic research. “The best thing to do is work with people who understand your business but aren’t invested in it,” says Tenzer.

At Virgin Media O2, a blend of in-house expertise and specialist agencies – from one-person consultancies to the bigger market researchers – are used. The company decides who to use on a case-by-case basis. It carries out smaller or more tactical work itself with a team trained in questionnaire design, moderation, sampling and execution.

Claire Rainey, its head of research and insight, wouldn’t suggest doing this in-house unless the team has the training. For larger pieces of work, such as pricing or brand tracking, it uses agencies. The cost means getting something meaningful out is paramount. Businesses should not be asking “nice-to-have questions” that yield little for them, believes Rainey.

Companies also need to ensure results are communicated and necessary changes made. “You cannot take the research and simply debrief via email to your stakeholder,” she adds. “It needs to be embedded. There is a big role here for facilitating change, relationship management and influencing for impact.”

For Farquhar, the impact is seen in the making of investment decisions. When he joined, he had to make the case for market research in a new business. He argued that if the company didn’t measure what it was doing and how well it was performing, it would have no idea how to make better decisions. He says it is now “hooked”.

“The great thing about insight in a medium-sized business is that when you finally persuade the business it is worth investing in, it can get hooked on it quite quickly,” he says. “There is something incredibly interesting and releasing and curious about hearing from genuine people, and how that influences your thinking around the business.

“The next stage is to use market research to identify where our source of growth is going to come from. We’ve gone through such rapid growth, but now we need to stimulate the next wave of that. Really understanding where the opportunities lie is the primary use of research.”

Market research, then, is pivotal not only to understanding your current customers and market but where you might need to go next. As Tenzer says: “Market research should be the foundation on which you build your business.”