The Perils of Skewed Sampling

October 2016

THERE ARE plenty of market research reports floating around the internet, but it pays to check their source and sampling before you jump to any conclusions.

Some organisations are providing superficial research and anecdotal data which purport to offer insights that aren’t really there. For example, a skewed survey sample size can lead to misleading results.

The complexity of the Australian Foodservice market is such that in-depth market research is not widely available from generalist sources. You need to go to a specialist like Food Industry Foresight for accurate insights and analysis. 

But with so many businesses eager for information, some Graphs With Peopleorganisations are providing superficial research and anecdotal data which purport to offer insights that aren’t really there. For example, a skewed survey sample size can lead to misleading results.

In market research terms, your sample size is the percentage size of subjects from the ‘universe’ you’re researching. When you think about it, it’s immediately clear that your sample size needs to be big enough to be truly representative, in statistical terms, of this universe that you’re trying to learn about.

There is a statistical science behind how this is worked out and it’s all done mathematically. Obviously a sample which is too small will skew your results, but it’s not just the size you need to be concerned about. You also need to ensure your sample constitutes a true representation of the universe you’re studying.

The only way you can make sure of that is by accurately identifying that universe before you begin your research. At FI Foresight we spend a lot of effort making sure we know how many restaurants, how many cafés, how many school canteens, etc there are nationwide and across each state. We need to understand how many market channels there are, how many outlets within each channel, and how these channels change over time. For example, in recent times we have seen big changes to how the institutional sector operates, with the widespread introduction of pre-packaged frozen meals produced by third party providers. Obviously, this information is essential in ensuring we’re able to create an accurate sample for our data, not only in terms of size but in the way we conduct our research.

There is a statistical science involved in researching and modelling the data, and there is also a definite art to the way the data is collected. The way questions are phrased, the use of language, the way interviews are structured, the length of time allocated for questionnaires – all these factors can affect the types of responses obtained.

Chef With ManagerRecently we have seen some foodservice market surveys which have been circulated and touted as offering valuable insights, but it’s obvious that their methodology is suspect.

One common error is to take your sample from a very small subset of the market – for example, professionals who frequent cafés in the Melbourne CBD – then extrapolate your results to suggest that they are representative of a much broader base of consumers. Obviously this is going to skew the results.

Your research might show that blueberry muffins are the number one takeout choice of one particular type of Melbourne CBD café customer to accompany their morning coffee, but you could hardly then claim that blueberry muffins are the most popular takeaway breakfast choice in Australia! But we have seen research which makes those sort of grandiose, unsubstantiated claims.

That kind of research is fine as long as you are setting out to obtain information about that tiny universe. But it really is no help for businesses seeking data on a much larger universe – for that, you need a proper representative sample.

Journalists in particular see spurious research and survey results like this on a regular basis, often delivered as part of a media release designed to push a particular line about a product or service’s popularity (and including the familiar line, “research shows that…”). Those who are well trained know better than to take them at face value.

The danger we see for businesses operating in foodservice is that if you base your product line decision-making around the kind of false premises that some of these types of research are touting, it can be a recipe for disaster. 

That’s why at FI Foresight we take the business of market research very seriously. We know that decisions both large and small are made based on the data we provide. So we take the time and effort to ensure we know our universe, we qualify every single one of our survey respondents to make sure they’re genuine foodservice operators – whereas an online survey conducted via your own website can allow anyone to respond and you have no way of knowing whether they are genuine!

Making sure your sample is the correct size, and that it contains an accurate representation of the market you want to study, is of paramount importance in getting trustworthy, accurate results. So next time you see some market research results in a magazine or on a website, ask yourself: how can I be sure this is accurate? Look for information on the sample size and the kind of methodology used. Often just by reading this you can see for yourself that the results are not to be trusted.

Contact us to discuss further.