How to Use Surveys to Get the Data You Need, When You Need It

What Market Research Really Means

Louis: Right, so market research, surveys, all of that, as I was just telling you before hitting the record button, I think there’s a lot of stigma around it, a lot of mis misconceptions, a lot of lies told about market research. But before we talk about those a bit, can you, if you had to, like, define it in a few words as. As fewer words as you can, what is market research to you?

Morgan Molnar: Ooh, yeah, well, so market research, you’re right, there’s a lot of stigma around it, but at its core, it’s gathering information from the outside, from the market to inform decision making and Actions that you take in your business. Simple as that. Gathering information. And it can be in a lot of different ways. Obviously primary, secondary research, qualitative, quantitative research. But at the end of the day, it’s just gathering information to help you make a better decision.

Primary vs Secondary and Qualitative vs Quantitative Research

Louis: So let’s then define briefly again because you’ve done such a good job for the first definition, primary versus secondary, and then coal versus quantum.

Morgan Molnar: Sure. Yeah. So when you’re looking at primary versus secondary research, primary is gathering that information yourself. So you’re going out there, you are collecting the raw data. Secondary research means you’re consuming data that someone else has already collected for you. And you may be repackaging it or using it in a different way, but you’re not the initial data gatherer of that research.

Louis: Okay, and then quantitative.

Morgan Molnar: Sure. Yeah. So qualitative versus quantitative quantity. The easiest way you think quantitative quantity, numbers. That’s the way I think about it. So quantitative research is really rooted in harder statistics and numbers. Qualitative can be a little bit more fluid. It can be whether it’s analyzing images or text or having an interview, it’s a little bit more unstructured in that way. And you know, there’s a lot of pros and cons to each and a lot of different things you can do with either qualitative or quantitative. But usually when you bring the two together, that’s when you can tell a really rich story where you’ve got the numbers, but then you’ve also got the why or the experience or the emotion behind the numbers.

Louis: Thanks. That makes a lot of sense. And yeah, it’s great to be able to define those key terms because I suspect we’re going to use them throughout this conversation. So I’ve been very lucky to have been able to speak to Seth Godin twice. Even I’m showing off here. But like, he’s one of my hero. He’s the. I think one of the first marketing book I ever read was from him. And anyway, in the second interview, I challenged him a bit when he was talking about the fact that he doesn’t really use anything but intuition to test stuff to launch products. He doesn’t seem to be very big fan of interviewing people, interviewing customers, sending surveys or anything like that. He rather show something to people, see how they react and then adjust. Right. And it’s difficult to disagree with someone like that who’s done an amazing job at making marketing more available for everyone. But it surprised me a bit because to me, markets research is a huge part of what I do. As a marketer, almost every time I launch something new, I start with customer interview surveys and whatnot. So the stigma that he has is the fact that people don’t know what they want so they can’t tell you. And basically market research is a bit skewed and all of that. So what do you say to that kind of comment? In general?

Challenging the Anti-Survey Sentiment

Morgan Molnar: That’s a very Steve Jobs mentality as well. I’ve heard that take and the way that I would push back or challenge is there was a small piece of what you said in there is where he likes to put something out there, have people react to it and then adjust. Market research is a lot of doing just that. But before you’ve done your major launch, so you can say have early prototypes, early pieces of marketing copy or messaging or creative, get it in front of people early, get feedback, iterate. And all of that can be done very quickly and rapidly in any sort of either product or campaign development process before you launch to the masses. And in doing so it’s that same kind of mentality like I want to make something, I want to build something on a prototype, get it and then get feedback. But if you’re doing that before you launch, you’re likely saving a ton of money. You are able to iterate faster. If you think about getting something out into the market, it has to be pretty baked. You know, in tech we talk about MVPs a lot, minimum viable products, but it has to work end to end. When you’re doing market research you can kind of chunk that up and do the same thing that Seth is talking about, but in pieces and before you launch and before you spend a lot of money. In that way it can help a business greatly, whether you’re talking about startups or even more established companies that are maybe launching new features or launching new products or expanding into different markets.

Louis: So that’s nice to test new stuff, you want to test creatives copy and whatnot. So I understand that in this instance you basically still look at behavior of people. They might pick an option over three and they might give you some feedback. So that’s close enough to a real life scenario where people will actually behave. But what about then cases that are a bit more less close to the prototype, more closer to actual primary research, gathering data around, I don’t know, people’s pain, challenges and what they want to do and whatnot for this type of scenario, like what do you think are the absolute, no nos, the absolute biggest mistake you see when people send, survey or not send, but design Surveys in the first place. So let’s talk. We’ll talk about surveys specifically. What do you think are the top three mistakes for people listening right now who want to send surveys?

Top Three Survey Design Mistakes

Morgan Molnar: It’s a good question. Yeah. And so you’ve kind of reined it into survey research. And I think that’s obviously a place that I know a lot about, having been at SurveyMonkey for five years. I’d say that when you’re starting out writing a survey and you’ve never done it before, you look at that blank canvas and it can be really intimidating. I write survey questions in the way that I talk. The way that I would ask you a question on this podcast is the way I like to write survey questions. I like to make it really colloquial and not use a lot of jargon. I think people get really formal when they write surveys, which can be good in some instances, but can also be a mistake, especially if you’re talking about really in depth topics or niche jargony word terminology that maybe is only relevant to you or people who are really intimately familiar with your industry. So that’s one thing. I mean, you need to write questions in ways that people on the receiving end will be able to understand them and respond appropriately. The other thing is, man, there’s so much bias you can introduce into a survey and you could probably do an entire podcast just on that. But the one that I always harp on as a fundamental best practice is avoiding biases in the question wording that might lead a respondent to some sort of basically showing up to the researcher the way the researcher. The way the thing they were. Researcher wants them to show up. So if you’re asking a question, for example, this happens a lot in political polling. If you’re asking, you know, a favorability question about a particular candidate or something of that nature, and you add in adjectives or you add in framing of a question that leads a respondent down a particular path. That is a big no. No. You’d like to have questions that are balanced, that are. That have equal opportunity for positive and negative responses. And that’s going to lead you to a much more unbiased piece of data and results that come from that. When I think about the answer options themselves, a lot of times people will make a mistake as far as the, I guess the golden rules of survey question writing, which is making sure that your answer options are mutually exclusive and collectively exhaustive. And what I mean by that, cause that’s kind of a mouthful, is mutually exclusive, means you just have all answer options never overlap. So when you’re talking about, let’s go with this very simple question. How many hours of TV do you watch in a given week? You don’t want to have answer options that are 1 to 3, 3 to 5, 5 to 7, because the threes and fives show up in multiple answer options. So that’s, that’s one thing you want to avoid. The next would be collectively exhaustive. So you want to make sure that your answer span the entire range of the possibilities of human behavior. So in that same question, you want to make sure you have an I don’t watch TV option. You want to make sure that your highest hour count has a and more at the end so that you’re basically capturing all possible responses that could be for that question. So those are probably the things I’d think about. It’s very granular, it’s very tactical into survey writing. So I think we could probably even step back a little bit and talk about, you know, some of the benefits of surveys if, if that makes sense.

Louis: Louis? Yes. So before I talk on benefits, just thing I want to talk about which is the, the fact that, that humans have incredibly complex, you know, decision making and, and the way, the way we do things, we re rationalize past decisions even though most of them are completely unconscious, based on our self stories and based on our what we want to achieve. It’s very difficult for people to externalize know wide about something, for example. And that’s something I keep thinking about when I design surveys or ask a question. It’s very difficult to take everything as face value and consider every single answer to be the truth. So I’m curious for you. How do you advise folks to ask the kind of questions that are there to understand why someone has bought something or why they took this decision? And even the no nos like the on the other side, the no gos questions such as would you buy something like this or are you interested in this stuff that are predicting future behavior. So how do you deal with this in general?

The Problem with Predicting Future Behavior

Morgan Molnar: Yeah, well, so you make a good point and not intention doesn’t always equal behavior. And that’s true with surveys, especially if you’re asking future forward questions like what are you going to do? Would you et cetera. The way that. So for example, in, in most concept testing surveys there is a golden question around purchase intent. And so you’ve just been exposed to a new idea, you’ve maybe answered some follow up questions. The question that a lot of folks use as the most closely tied to what will actually happen from a behavior standpoint is a purchase intent question. And usually the way that this is worded is how likely would you be to purchase this product, download this app, et cetera. And you have a nice range in scale, not from a likely to unlikely, but more of a absolute to not at all. And so that helps to frame it from like a positive negative, but more it’s extremely likely, very likely, somewhat likely, not so likely and not at all likely. So we’ve gotten away from the strongly to not to strongly for versus strongly against. That’s kind of been something that we’ve seen kind of more positivity, biases, skewing results in terms of how that maps to behavior. So that’s the question that we usually use. There have been some academic studies that tie that to behavior. It’s not going to be perfect. But what you can think about as you’re designing a survey like that is the use of benchmarks. And so you can ask this about your product, you can ask this about a competitive product, you can ask us about past products that you’ve launched where you know the in market performance and you can start to build a database of, of results and benchmarks so you can compare these things against each other. And so then what you’re doing is comparing like to like survey data to survey data versus survey data to market data which is not always going to be apples to apples in a case.

Louis: That’s interesting, right? That’s interesting because you would, so you would ask the same question, the same, like the exact same word, the exact same type of answer to like for let’s say you launch your first product, you pre launch your first product via this survey, just want to get data. Then you know the actual purchase that you got out of this and then so you have a benchmark and then you can do the second product. Okay, we have a bit more likelihood to purchase. So we are, if there is some sort of correlation then we expect that much. So, so then you, as you said, you compare like for like with the exact same question instead of just, you know, trying to reinvent the wheel or think of stuff that might or might not happen. How do you do it with competitions though? With competition data? What’s the deal?

Morgan Molnar: So for something like that, you can’t have your competitors sales data all the time. I mean there are some syndicated point of sale data providers that do offer syndicated data, but that’s really expensive. If you’re a startup, that’s not going to be something you are going to have at your fingertips. So what you can do is set up essentially the same kind of like an experiment, but using surveys. You set up the same survey, but instead of your own product concept, you use a competitor’s product concept. In that case, you, you set up the survey the same way, you target the same people. And then you can. Again, that’s, that’s comparing like to, like, responses. And you can set that up with, you know, competitors who are at the top of the pack, competitors who are middle or at the bottom of the pack. And, you know, you don’t have the exact sales data to tie that back to, but you have a sense generally of market share or the most popular thing in there in the market. And so then what you can do is, is do that comparison and just see, okay, how does my, my concept perform against things that are already out there in the market?

Concept Testing Against Competitors

Louis: So how would you. Let’s dive into that a bit because that’s, that’s quite interesting. I’ve never done it before, so I’m genuinely curious. What’s the. So let’s go through just a hypothetical here. Let’s say we are setting a new toothbrush that is fucking phenomenal because it’s actually the best toothbrush ever, actually removes the yellow on everyone’s teeth with just one use. And we compare that to the likes of like on top of my head, Colgate and all of those big brands, whatnot. We want to know whether this concept is going to work with toothbrushes that already exist. Obviously, we’re not going to design the server entirely together right now, but if we had to pick maybe the top two, three questions that you would ask to get an understanding of whether we can expect some sales and whatnot, how would you design this?

Morgan Molnar: Yeah, so the main structure of a market research survey is going to have some initial screening questions, perhaps to make sure you’re targeting the people that you need. Then you’re going to have the meat of your survey and then potentially some demographics at the end so you can filter, slice and dice your results. So for a study like this, where essentially what we’re doing here is a concept test, we’re trying to test whether our product idea is viable and ready to launch and whether there’ll be good reaction or reception in the market. And in this case, you’ve got your Colgate brush, your oral B brush, what you want to do when you’re picking the stimuli for a survey, for a concept test is essentially what that means is what is the idea that we’re presenting in front of the respondents to react to. It could be a text description, an image, a video, a gif, et cetera embedded in the survey that you’re then reacting to with follow up questions. The stimuli in this case you’ve got your idea, unclear how well baked that is. Let’s assume that you’ve developed it pretty far and you’ve got product images or maybe even what it might look like in the packaging or maybe even testing the price point. You want to make sure that the stimuli you’re testing are similar enough across what you’re, what you’re trying to get feedback on. So in this case, if I’ve got a product image in the packaging with a price point, I want to create that same look and feel and image with my competitors products. So maybe you go and pie your competitors to toothbrushes and you photograph them in the same way you photographed your own products packaging. You, you put the actual price point that you received in the store and you, and you create those stimuli to be pretty equal. You know, the only thing that we’re testing here is the difference in, you know, what the, the consumer is reading as far as the pricing and on the packaging and you know, the visual of the color and shape and style of the toothbrush. And then so what you’re doing in the survey is you’re first embedding that stimuli and then you’re asking follow up questions. The few questions that you know, for a product like a toothbrush that would be really interesting. I mean I mentioned purchase intent. That’s gonna be one of the main questions. You might ask something around overall appeal. How much do you like this product? How much would you be likely to buy this product? Is the price point too expensive too, you know, so cheap you think it might be poor quality? Things like that. And you know, there’s a, there’s a variety of different kinds of metrics or attributes you could ask about these different products. What I always like to do is keep this survey at a five point scale in terms of the answer options. So extremely, I kind of mentioned this before but like extremely very not or somewhat not so not at all. And then whatever your metric is in those answer options, that 5 point scale, if you keep that consistent across all of those follow up questions, makes it really easy to analyze your results because you’ve got everything in the same scale. You can compare the top score or the top two scores combined. It’s called a top two box. And what you’re able to then do once you have your results is compare those Scores across all those different attributes or metrics for the different products. How does your product then score against your top two competitors? Which by the way, if they’re very established, like a Colgate and an oral B, for example, you can bet that they’ve done some research on their products, that they’ve got some of their best stuff out on the shelves. And if your product is doing well against those, then, you know, you’re, you’re pretty, you’ve got a pretty solid amount of feedback or evidence that what you launch is going to work, are going to, you know, sell is really at the end of the day what you want.

Louis: Yeah. And because if you compare yourself with mature products that are very well known, you, you will also see that, you know, the more people are aware of something, the more they, they are liking it. And so the more the leading brand will get favorable ratings. That’s, that’s a proven fact as well.

Morgan Molnar: And what’s interesting there that you bring up is if your product scores well against something that’s established and in the, in the market, it’s overcoming some of that brand awareness and brand affinity, brand trust, et cetera, that’s been built up over decades for some of these brands. And so that’s a really good indicator.

Louis: Let’s backtrack a bit because I’m pretty sure people are asking themselves, okay, that’s all well and good, but where do you find the people and all of that? So before you answer that, the first thing I’m wondering when you design such a survey is do you ask the same person the same question on the three, or do you basically do a three way test where you have an audience of, let’s say 6,000 people and 2,000 will get your product, 2,000 will get competitor one, 2,000 will get competitor two.

Monadic vs Sequential Monadic Designs

Morgan Molnar: So you’re really hidden at the fundamentals of concept testing here. What you’re describing is the difference between a monadic design and a sequential monadic design. A monadic design is where a single respondent sees a single product or stimuli and asks that single set of follow up questions. A sequential monadic design means that they may see a series of products or stimuli and maybe a random order and then are asking the follow up questions on all of those. There are pros and cons to each. Monadic is usually touted as the more pure method, methodologically pure or theoretically pure design. Because what you’re doing is you’re getting a fresh set of eyes for each of the concepts. They’re not biased. But what by what they saw previously. And they won’t adjust their responses. They’re just reacting to the first concept that they see. What you want to make sure, if you are doing that is if you’ve got subsets of respondent groups who are answering questions about each of the different products that you’re testing you is to make sure that the balancing and the targeting of those respondents are as comparable as possible. Because then you don’t want to make sure there’s any, you know, one. One subset skews much more female, one more male. Then, you know, it’s really hard to compare the results across. For sequential monatic, really, the pro is cost in a lot of ways. Um, if you’re purchasing responses from, say, a panel like we have at SurveyMonkey, then what you’re. What you’re doing is, you know, for, for a monadic design, you’re basically having to pay for 3x the number of responses. If you’re testing three products, but with a sequential design, then you can randomize those concepts and pay for fewer responses. Now, I’d say when I recommend monadic design is when you’ve only got a few stimuli to test. When you’re saying, try to test maybe, I don’t know, 10 to 15 different concepts, which I have seen customers at Server Monkey are doing this all the time. I recommend some sort of combination of a sequential design where maybe respondents are seeing a subset of five random concepts that you’re testing. It just, it’s like a little bit more bang for your buck in that way.

Louis: Okay, that makes sense. And you touch on the other question I wanted to ask, which is the panel side of things. So I’ve done it multiple times in my career where you want to get data from people who might not be customers of your brand, and you just have like a customer profile in mind, a real customer Persona, or at least, you know, category entry points. So people buying specific category, like toothbrushes, and you want to reach out to those people outside of your brand customers to ask them some questions. So SurveyMonkey does that. Right. And I don’t want to go too much into the tools right now as of 2020, because if people listen to this episode, in five years, who knows what it’s going to be? But we can touch on it a bit. So panel is basically a way to buy responses from folks who answer them in their free time because they’re paid to do so. Is that correct? I mean, for SurveyMonkey side.

Understanding Survey Panels and Audience Targeting

Morgan Molnar: Yeah, it depends on the panel. And we can take the toothbrush example and continue that. So let’s say you are a toothbrush startup and you don’t have a lot of customers yet. That limits you as far as who you can reach out to. You probably got a lot of friends and family and social media followers have you that you can reach out to, but you might not have a broad, nationally representative group of people that you can get feedback from. So panels are a great way to do that and source and purchase the responses for your survey in a targeted way. So exactly who you need to reach most panels and there are a couple different iterations of this, but most panels are at the very root of them a collection of people who have signed up to complete tasks in order to receive an incentive. And that incentive varies. The tasks even vary. Some panels are not just survey panels. They could be other types of task based and then the incentives vary because you could have pure cash payments or you could have points towards redeeming gifts or gift cards or in server monkey’s case, our contribute panel actually donates to a charity of their choice for taking the survey. And so there’s a, there’s a variety of different incentive models and even that varies country to country.

Louis: My biggest question regarding panel is it’s something that baffles me because obviously I’ve never done, I never replied to survey, an exchange of incentives or whatnot, but when I’ve done it before, when I try to reach out to specific folks, like in the B2B sector for example, like quite targeted and whatnot, I’m always baffled at the number of answers you get from a very targeted generated like demographic thermographic data such as I only want people in marketing who have college diploma, graduated from college at the minimum, who earn whatever. I mean just, I’m just making that up. And I’m surprised that there are actually people fitting those criteria available to answer panels. And every time I question the validity of the panel, thinking that can’t be right, who has the time to answer those surveys? You know, if they have full time, if they are like fully employed, if they have 30 years of experience, who, you know, who would do that? So is that an unfunded like misconception that I have or like what’s the.

Morgan Molnar: It’s just the beauty of scale. So most panels have millions and millions of people in them. And to get at the targeting that you just mentioned, what panels do when you sign up is profile the people by asking them a series of questions. You know, how old are you? What’s you know, your age, you know, Your gender, where are you from, your zip code, your education, do you have children, et cetera. And so that’s how they get that information. They’re usually keeping that pretty up to date and continuing to reprofile folks. But it’s, it’s the, really the beauty of scale. So if you think about any, any funnel in marketing even think about like an email funnel where you’ve got a pool of people to send to, only so many people will open, only so many people will click on, etc. And then convert. So same thing is true with surveys. So we’ve got this big panel of people that we’ve recruited over years and there’s probably a variety of recruitment methods that will source people. I mean, you mentioned B2B. It’s interesting. Some of the panels that we work with at SurveyMonkey source people through rental car loyalty programs or things that, you know, tend to have more business folks in them, just those kinds of groups. And, and so what you’re doing to make sure that you’ve got that quality is, is continuing to profile people, continuing to even use screening questions in the survey to make sure you’re, you’re getting the right people. But you’d be surprised how much turnover and, and churn and then re recruitment there goes on in panels. So almost, I mean, I’d say, oh gosh, it would be hard for me to throw out a statistic right now, but there is a lot of turnovers, turnover in panels. And so what you’re, what you’re doing is getting a pretty fresh group of people. In some of the ones that we work with, panelists might take five or six surveys before they drop out of the panel. And so you’re having to re recruit.

Louis: But it’s like any category buyers, when you look at the real distribution of light buyers versus heavy buyers in any category of purchase, turns out that at the minimum, 40% of buyers are light buyers, meaning they buy once or less a year and that’s the minimum. And so what make what, what, what your answer made me think about is that a lot of folks might answer, as you said, 1, 2, 3 surveys and then churn. And so they might. And then if you have very, very few answer a ton of survey, they do that almost professionally. Yeah. So the lie by the long term chart.

Morgan Molnar: Yeah, exactly.

Louis: The long tail is really long. Okay, cool. Thanks for answering those. I want to touch on something else and I want to kind of try to reverse engineer how you’ve done it. You’ve done an actual survey full SurveyMonkey for a specific thing like a naming exercise. So I want to go through that with you and kind of try to see how we can advise folks who want to do something similar to do it your real experience. So tell me more about this project.

Real Case Study: SurveyMonkey’s Product Naming Research

Morgan Molnar: It’s going to be so meta because everything that we just talked about is relevant to this project in many ways. What we did this year at SurveyMonkey was launch a suite of solutions to automate the concept testing methodology and process. So everything I just described you can Basically do with SurveyMonkey by just clicking a few buttons and not having to write your survey or do any of your own analysis. It’s really great. It’s a suite of seven solutions for concept and creative testing. And I keep, you know, even on this interview I’ve been using the term concept testing a good amount. And so when we were setting out to name these solutions, we thought, I mean our initial instinct was that concept testing was going to be the right name for them. I wanted to make sure that we had, we put some rigor into naming these products. And you know, product naming is something that is very nuanced and you can go a lot of different directions at SurveyMonkey. A lot of our products have more practical or descriptive names. Some also have names that are much more of a brand that doesn’t really have any meaning. It’s just more, you know, the product name and brand name versus describing what it actually does. We for these, especially since they were under SurveyMonkey’s umbrella of market research solutions, we wanted to keep them pretty descriptive so that people understood what they were exploring and buying. When we went to go actually test the names, we used the methodology I described before. We used a monadic design. We introduced the product description. We then said, hey, we’re thinking about a few different name options. What do you think of this one? And then asked a couple follow up questions.

Louis: So what were the, the names?

Morgan Molnar: Yeah, so we tested a couple different things. So, so we were test. So for, let’s take the example of our, our product testing product. So this was. We have seven of them for testing ads, logos, messages, whatever. Let’s just use the product one for, as an example. So we tested things like concept testing, product testing, product concept test like things, things like different iterations of that. And then we also tested product analysis and concept analysis things that, so the test. It’s interesting that test word versus the analysis word as part of the product name was something we were interested in because we think of test or testing. It’s a verb. It’s, you know, part of the process of doing this work. But the analysis is what you actually get. It’s the data, it’s the results, it’s the insights, it’s the recommendation of what you’re going to do with that knowledge. And so we wanted to kind of challenge conventional thinking and the term we’ve been using internally as we developed the products and just see how that would work. And when we got the results back, it was very, very clear that folks reacted positively to the analysis. Part of the name in some of the iterations we tested by far in a way was the number one winner in all seven products. And granted we did that monadic methodology and so for it to pop up in every single product name set that we tested for all seven, it was a clear pattern. It really developed that mountain of evidence that we could go then back to the executive steering committee and say, hey, we didn’t expect this to win, but look at the like, the data does not lie and let’s figure out what is behind this. And we actually followed up with a couple interviews. We had been doing user testing, you know, every week pretty much throughout the product development process. And we put some of the names in front of folks after we’d gotten the results. And for people who were gravitating more towards the analysis name, we dug in, we asked why and kind of what I described earlier is what people were saying like really the value here is in what you’re getting in the analysis and the insights that it’s generating for you. So that’s what I’m buying. And so what we ended up doing is we named our products that way. So if you even go on the Survey 1 site, you’ll see Product Concepts Analysis, Logo Design analysis, Brand Name analysis, etc as the actual product names of those solutions that we launched earlier this year.

Louis: What was the key, the key question that you, that you asked for those, Was it a choose between one of those name?

Morgan Molnar: Yeah, well, so in we, well the way that we talked about this methodology earlier and actually it’s funny, I’m going to look up and pull up the actual results here because it’s fascinating to share, but the, the way that we were talking about the methodology earlier was, okay, you expose the respondent to a stimuli and you ask follow up questions. And that is what we did for a good part of this survey. So now we, we had to kind of do a bit of a hybrid methodology because we had seven products and four or five names that we were testing for. Each one. So for us to also be purchasing B2B sample on top of that, we had to do a bit of a hybrid methodology. And so in many of the cases we asked questions like I described earlier. So a five point scale around things like appeal, purchase intent, I’m pulling them up, it’s litting. But then the, the second half of the survey, we actually exposed people to the other names. So hey, we’re also considering names C, D and E. What now we are asking a couple different question questions like, like best or easiest to understand or innovative. And we actually pit them head to head. And so in that case you’re not taking a score. That’s an aggregate of say extremely and very, very likely. You’re actually just looking at the metric of okay, which name actually won out. And so in this case we asked questions like overall appeal, uniqueness, relevance, purchase intent. But then the head to head metrics were what do you like best, what’s sounds more innovative? What do you want to learn more about? Which one gives you the most confidence that it is a high quality product and which one’s more memorable. So those were the kinds of questions that we asked. But it was interesting to just see those results. And by far and away, the names that had analysis in them were the clear winners.

Louis: So the head to head was, let’s say copy testing versus copy analysis, product testing versus product analysis.

Morgan Molnar: It was always within the same product line. So one of the seven would be the survey we’d be asking about just solely. And so it’d be something like product testing versus product concept, analysis versus product design solution or something like that. So those were the kinds of things that we put head to head.

Louis: Okay. And your dog is also very interested in surveys, which I, she loves it. Like that’s, that’s why, you know, it’s the right pet for you.

Morgan Molnar: Yes, that’s my golden retriever, Penny.

Louis: I can’t see it, but yeah, I can imagine how surveys excite dogs. So the, how do you get, which audience did you reach out to? Did you, did you reach out to customers, existing customers, as well as people who are not customers? What was the audience?

Morgan Molnar: So we actually used SurveyMonkey Audience, which is our global panel, to source the respondents. And the way that I usually like targeting for B2B. I mean, like you mentioned, there’s the profiling that the panels do and how much can you trust that? I mean I work in this industry and I trust it a great deal, but I still know that depending on how often you are reprofiling folks, people change jobs, people move across the country. So you know, you always want to make sure that you’re getting the right group of people. And so in that case I usually pair profiling with screening and screening questions at the beginning of your survey to ask people the to provide the most up to date information so that you can target. And in the what I did for this one was I targeted on the profiling side, employed full time. And in I think I selected a couple different job functions, not just marketing or insights or there’s a couple other functions that are interesting for us. I mean we’ve got some product solutions so you know, product management functions or even startup founders, et cetera. And so I paired that full time employed plus a select few job functions with screening questions that a confirmed that profiling. So I had screening questions for employment and job function but I also had a question around involvement in market research at their company. And so you don’t have to ask do you do market research or are you a market researcher? You can get around that by asking about their level of involvement. So it can be, you know, you could be the sole decision maker and hold the budget or you could be an influencer or you could be involved involved or you may not be involved at all. And so we basically just screened out anyone who wasn’t involved at all in at least this survey that we did.

The Importance of Proper Screening Questions

Louis: And screening questions are very important. I think I’ve made a mistake in the past of it’s easy to have a screening question that says are you involved in market research, yes or no. Or this kind of very easy binary question. So you need to be careful not to be too obvious about what you’re screening for.

Morgan Molnar: It’s a really good call out and probably something I could have mentioned at the very top of this interview when you were asking about survey no no’s or big mistakes people make. I think screening questions are one that do require some thought. Yes or no. And part of it is because we also talked about how, you know, there’s that small group of panelists who are a little bit more professional survey takers and are kind of onto you as far as wanting to qualify for a survey to get that incentive. And so you want want to not trick people. That’s not what it is. But you do want to make sure you’re getting the most accurate information. And so there’s two main types of screening questions. One is a behavioral screening question. So you know, are you in the market to buy a toothbrush? Maybe you wouldn’t ask that one. But something where you’re asking about the behavior. So how often are you watching tv? Or it could be something around, you know, what type of pet do you have, if any? And you’re asking about cats and dogs and all these other kinds of animals. You know, kind of a question where you’ve got a multi select in your answer options, you know, pick list is a little bit better for screening than a yes or no question because most of the time you’re after the yes answer and so professional survey takers will be on to you there. The other type of screening question is an industry screener and I see this a lot of times in B2B research, but also B2C especially when you’re testing new things that are a little bit more sensitive. You don’t want it really getting out there and you don’t want a competitor taking your survey or someone who’s in the same industry. And so you might ask a whole list of industries, which of the following do you work in or do you have a family member who works in any of these? And you screen out for people who work in the same industry as you. I see this a lot. You know, major CPG companies that are doing research. So if you’ve got someone in your family who works in beauty care and your, your, you know, your survey is all about this new beauty product, you don’t want someone, someone reading your survey and getting on to your next innovations.

Louis: Yeah, okay. Yeah, so yeah, this is important, very, very important point on screen questions that I, that I found the hard way. We just have a few minutes and I wanted to talk to you about another topic that to me is very, very misunderstood. But I think you have the knowledge to explain it simply, it’s the sample size. Right. This is something quite counterintuitive when you think about it because when you reach a certain threshold for some for your sample size in your survey, adding respondents or even 10,000 more respondents won’t actually move the needle that much. You’re still going to have, I’m going to forget the fucking metric name. But you’re going to have a high confidence regardless. Right? So I’m absolutely butchering this for you. But you’re going to be able to explain it much better. So tell me more about this concept of sample size, what to watch out for.

Sample Size: It’s Not What You Think

Morgan Molnar: Yes, and server Monkey is a great sample size calculator because I don’t even know that I would be able to tell you the exact formula for calculating a sample size based on the confidence level and margin of error you’re comfortable with, but we’ve got a great one that you can use. But the way that I like to explain sample size is it’s a function of your population size. So say you are launching a, say it’s a kid’s toothbrush. And so you want to, you know, the people buying this kid’s toothbrush are going to be parents of children of a certain age. And so there’s probably some estimation of how many parents of a certain age are there in, say, the United States. And so that’s a pretty big number. But your sample size for that group of people probably doesn’t need to be as large as a sample of the entire US Population because what you’re looking at is for something that’s a lot more specific. It becomes much more clear when you say you’re targeting something like neurosurgeons, where there aren’t that many neurosurgeons in the US So in order to sample enough to have answers that will be representative of that population’s beliefs, you just don’t have to survey as many people. So sample size is a function of the population size. It’s a function of the margin of error you’re comfortable with. Most people have been exposed to margin of error when it comes to polling results in the news. So in most cases they’ll say, oh, plus or minus 3% around any particular metric. And what that means is we’ve sampled enough people that we’re confident that what, what the sample is telling us. So say it’s, I don’t know, a 50% of people are aware of the Colgate brand, that the actual percentage of people aware of the Colgate brand at the population level is within 3% of that. So anywhere from 47% to 53%. And the more people that you’re sampling, the, the tighter that margin of error is, so the more confident you are that your sample truly reflects the population. The rule of thumb I like to give people, if you are, say, let’s just use the general population as an example. If you survey 400 people, that’s usually about a 4 plus or minus 5% margin of error. If you’re serving a thousand people, that’s plus or minus 3%. That 3% is pretty industry standard as far as where you’d want to be to be able to publish research, get it picked up by journalists or the media. So that’s why you see, I mean, so many people will say, oh, you need a thousand people. A thousand people, that’s because that’s right where the margin of error shrinks down to plus or minus 3%. If you’re just doing a quick survey for a gut check or a fun, you know, just for fun survey to publish on social media, you probably need a couple hundred. You might not need that full thousand. And then the other thing to think about is, you know, how hard is it going to be to find those people. So if it’s a really niche B2B target, you probably don’t need the thousand either because the population size is lower. That. I don’t know if that was as simple as you hoped it would be, but that’s how I like to think about it.

Louis: No, thanks for that. And as you said, there’s plenty of online calculators, online, a sample size calculator, online ServerMonkey being probably the simplest, most straightforward. But just as a quick thing, just to describe what you described here, let’s say we have 10 million as a population size, a 3% margin of error. You need 1067 people, as you mentioned. But if you remove, if you just your population, population size is 1 million, it’s actually the same very, very close number 1066.

Morgan Molnar: If it has to do for folks who are like mathematicians out there, it has to do with the denominator getting closer and closer to one as the, as the population size keeps going where, like, at some point, you know, you’re going to hit this threshold where whether Your population is 100,000 or 100 million, the sample size you need is very similar.

Louis: Okay, well, thanks, Morgan, for going through all of this with me into that much detail. I’m going to ask you one last question before I let you go. What are the top three resources you’d recommend? Marketers or people listening to this podcast?

Top Resources for Data-Driven Marketers

Morgan Molnar: Okay, yeah, sure. And this is funny because you told me you were going to ask me this question and I had this book on my nightstand. But one thing that I think is incredibly important for marketers to be able to do, especially those who are using market research in their job, is to be able to then tell a story with that data. It’s helpful whether you’re an entry level marketer just trying to understand what’s working, what’s not, whether you’re pitching your next big campaign idea or it’s also, you know, eventually, as you move into leadership roles, helpful to communicate strategy in the boardroom, for example. And so a book that I love is called Data Story by Nancy Duarte. It’s a great, really great resource. And the other thing is kind of a hack. You can go on YouTube and she’s done a couple talks on this topic and you know, 10 minute videos, 30 minute videos that you can watch and get a lot of really great tips and tricks for telling stories with data from her. And on the book front, one thing that I’m loving right now is an app called blinkist and what it does is it takes business books, but not just business books and really mostly nonfiction and a lot of different categories and it kind of turns them into this Spark Note esque summary both in written or audio form. And so it’s almost like, you know, my, well, what would have been my commute to work, which is now my 30 minutes to walk my dog in the morning, is, you know, an audio podcast like format of, you know, short summaries of really great business books. And I’ve been able to plow through books much faster than I would have otherwise. And then the last I’d say is honestly, I mean, good old LinkedIn. Follow and connect with other marketers that are in your field or your home city. Start building relationships, start building a following and then you can kind of start to create this. What I’ve I’ve gone through a couple workshops of this, but a personal board of directors both inside and outside of your company, through the relationships that you build over time, over the course of your career to help you and guide you through decisions or job changes or you know, just even you can even ping people for feedback like you would some qualitative research. So LinkedIn is just an incredible resource for connecting with people and building relationships. And honestly Louis, that’s one of the places that I found and heard of you. So comes full circle.

Louis: Exactly. Yeah. Great recommendation. Blinkist is also something I’ve been starting to use recently. I like to do it by when I’ve read a book a long time ago or listened to it, I like to reread the summary to remind me of the essence of it. So yes, a great recommendation all around. I had never heard of the first book you mentioned, so I’m going to check it out. And again, Morgan, you’ve been a pleasure. Thanks so much for going into the details of survey making and design and I’ll talk to you soon then. And that’s it for another episode of Everyone Hates marketers dot com. Thank you so much for listening. I’m super, super grateful. I’d love for you to consider subscribing to my daily newsletter Monday to Friday called Stand the Out Daily. I send very short, hopefully interesting surprising, shocking, entertaining content to help you stand the out. It’s at everyone hatesmarketers.com you can subscribe for free and obviously unsubscribe whenever you want. I’m just gonna read a couple of emails that I got recently as a reply. Juma said, your content attacks the mind primarily, which is such a good thing because most of us are skilled at what we do, but we don’t have the courage to do it it our way. Mark, who just subscribed couple days before, said, this is my first issue of your newsletter. Love it. Glad I subscribed. Brianna Said, I just realized this morning that my email habit is now to 1. Came through the list 2. Select all unread industry email except yours. 3. Delete and don’t think twice. 4. Quickly scheme yours. Amy said, Also loving the new content is coming from you. It’s feels really lovely. Candle said, I like your writing a lot. It really resonate. There’s so much out there. It’s good to touch the authentic. And Chloe said, where is the I love this email button? Brilliant. I hope you subscribe. You’ll be joining more than 14,000 subscribers at this stage, which is crazy. It’s the size of a small stadium. Anyway, thank you so much. See you on the other side. It.