[BEST OF] How to Experiment Quickly & Find HUGE Wins (5 Steps)

Symptoms of Companies Not Experimenting Enough

Louis: So tell me, what are the symptoms, would you say, of a company that are not using rapid experimentation, who are not experimenting enough?

David Arnoux: The symptoms? That’s kind of a funny one. I think that, I mean, the ones we see is usually that they’re growing slower than the competition and then they have slower, let’s say velocity of experimentation. And there’s a very gut driven mindset within that organization. And funnily enough, when I started, when we started training, so like you said, I have an extensive sort of startup background, more in sort of SaaS products or E commerce. So very short term sales cycle and since we started this training company, this digital partner company, we found out that actually we thought it would be the larger organizations that would suffer from these symptoms and actually has nothing to do with the size of the organization. We find that there’s very small, nimble, post product market fit startups that have exactly the same problem as some of the largest corporates out there. And it still amazes me to this day with the amount of tools and possibilities that there are that people are still not sort of adopting this way of working in this process.

Louis: So it’s interesting because as you said, it’s not based on size, right? It’s not based on demographic. So is it a psychographic element, do you think? It’s highly emotional, the fact that certain companies go with this let’s make mistakes and let’s try new shit, while others are very afraid of that. What do you think is the common trait between companies who do not do this? What do you think is the common denominator between all of those?

David Arnoux: Yeah, I mean there’s a few. The first one is age. I think I hate to compare corporates with startups. I prefer to compare digital natives with a little bit older organizations. So I’m in Amsterdam at the moment. My neighbor is booking.com since day one they’ve had this process, this mindset of experimenting and on the other side I have Oracle, which since day one has not. I think another thing has really got to do with the business model. I think a lot of people talk about mindset and culture of organizations and what we found is that your mindset and your culture is very oftentimes dependent on the business model of your organization. So I worked a lot with E Commerce for example. So business model was E Commerce or with affiliates it is natural for those organizations to have to experiment. Whereas when you’re tapping into things, maybe more around mobility or around health services, around financial services, with a little bit longer sales cycle then you have less of this need to experiment. Since day one, say the age of the organization, the business model of the organization and then of course normal variables like what is the DNA of the co founders, what is the DNA of the senior management. Also has the company been incredibly lucky. I think that A lot of the companies that we train or that I’ve sort of worked with that have had pain in the past, it’s forced them to learn to experiment because nothing’s come easy. Whereas other ones, where everything’s worked since day one, have less of this culture of experimenting. I think I’m obsessed with two companies which are PayPal and Amazon. And it’s been beautiful to look at the history of PayPal where it was just a struggle from day one. And it’s forced them to learn to be humble, to look at the data, to pivot regularly and to experiment with different things.

Louis: Right, okay. So now I think we’ve nailed the profiles of people who are not using it or not doing properly, and the profiles of people are doing it quite well. Before we go into the practical step by step on how to set up your own process and making sure you experiment and get good wins, there’s something I want to ask you. Right. As you know, this podcast is really about marketing foundations, things that won’t change in the future and unlikely to change, focusing on the customer. So we are not going to go into details about this, but before setting up any process like experimentation and growth, what do you think are the foundations that those company must have before that, what are the things that they have to have in place?

The Foundations You Need Before Rapid Experimentation

David Arnoux: Oh, yeah, well, I mean the foundations are kind of obvious. The first one being product market fit or close to having product market fit, which means that there’s a real pain to solve. You have the solution to solve that pain. You know what the market is, you know who your target Personas are. I think that’s something that’s very, very rarely looked into clearly. So who are we trying to solve the pain for? What are the jobs to be done of your product to solve that pain? And I think maybe one that’s a little bit less obvious is just what we call the go to market strategy, having at least a strategy for how we’re going to address the customer. Is it short sales cycle, long sales cycles, Is this a discovery type product or is this more an intent based product?

Louis: And so we’re not going to touch on that in this interview. We’ve talked about it a lot in past episodes on like Persona jobs to be done, go to market. We’ve done it all. So folks, if you’re listening to this right now, just go back to previous episode if you want to have those foundation ready now, we’re going to assume in the next few minutes that you do have those foundations. So you do have a problem. We’re solving with a solution that works. You do have a way to address customers. You know your customer inside out, and you have those foundations. But before talking about this step by step, there’s one last kind of item I want to talk about. I feel like in the marketing world and business world, there is a lot of people struggling with their mental health issue, a lot of people struggling at work with the pace at which things are moving. And I do put some responsibilities to the mindset of certain companies and certain people who think that growth is the only thing that matters for a business instead of just their people and making sure that there is a balance. So what are your thoughts on this? Where do you stand in terms of growth? Is it growth at all costs? Is it growth for the sake of growth? What is it?

Growth for the Sake of Growth vs Purpose-Driven Growth

David Arnoux: Do you know that Edward Abbey quote? I think it’s growth for the sake of growth is the ideology of the cancer cell. And that one really sort of really resonates with me and makes sense. I don’t know. I think I haven’t seen so many problems with mental issues, to be honest, at least not recently. And I would recommend to anybody who sort of struggles with this for their employees or whatever to read the book. It’s a pretty recent book. It’s called Stealing Fire. I think it’s by Kotler. And they talk about this state of ecstasies or this state of flow. And I think that’s one of the best ways to fight this sort of mental breakdown as possible. With all of the notifications happen on a regular basis, trying to reach this state of flow through either things like meditation or things like knowing how to focus, working on deep work. But your original question was, is growth for the sake of growth healthy? No, definitely not. But I do think that it depends a little bit on the business model that you have and whether you’re in a company that has real vision, a real mission. What’s funny to see is a lot of companies reverse engineer their mission and their vision later on. They start out just wanting to grow for the sake of growth, being a little bit sleazy, a little bit sneaky in the early days because they can. And then two years in, when they’re a little bit larger organization with a bigger target, let’s say, then they start building in that mission and that vision. I don’t know if that’s a good answer to your question, but I do think there needs to be a reason for the growth.

Louis: Right. So you don’t grow for the sake of it. You have an ultimate motive. Something a bit bigger than the business itself that forces you to like, I want more people to be aware of this. I want more people to use the product because I believe in a world where this should be more popular.

David Arnoux: There’s extensive literature that researches the fact that successful founders tend to be driven by a real mission and a vision, not simply growing for the sake of growth. Except maybe traders. Except maybe on financial markets where it’s cash for the sake of cash. I mean, just read the Bonfire of Vanities or read any book about Liars Poker about the world of traders. It might be one of the only areas.

Louis: So would there be companies that you would say no to that you wouldn’t train?

David Arnoux: Oh, yeah, sure. We have a whole list of guidelines on companies that we are willing to do or not. For example, we’ve been contacted by the arms industry, that’s kind of an easy one. And refuse to do that one. Then it becomes really difficult to be ethically responsible because some of the worst companies don’t actually look like the worst companies. So it’s easy. Sometimes it’s simple to shout at a, I don’t know, a petrochemical company like an energy company. It’s really easy to criticize them. But then you’ve got maybe a retailer that’s wasting a lot or you’ve got a clothing manufacturer that’s burning a lot of excess of stock. So that’s a really difficult one. It’s easy to judge a company by a sort of, by its cover. We do do very strong analysis of the companies that we, that we train. We’re not perfect, but I mean, average age here is 25 and a half. Really purpose driven, really want to make the world a better place. So we basically have no choice.

Louis: You know, a few years ago I had a, an agency and one of the belief was, yeah, we wanted to work with good clients that we’d be proud of. So I remember we had the debate around should we work with pawn companies? Because we had a few porn companies reaching out. And I remember the debate about it like, yeah, but is it good or bad? And it just went, you go to this rabbit hole of actually, as you said, even on the surface it might look like a bad choice or a good choice. But when you start investigating, everyone is connected to everyone. It’s getting very difficult to understand whether a business is doing good or not when things are in the beginning.

David Arnoux: I could give you one clear example. So we gambling, gambling companies, gambling organizations, you know, on the face of it, you would say that it’s not good to work with a gambling organization. They’re addictive, they make people lose their money, etc. Etc. Then when you study gambling a little bit, it seems that there’s a correlation between people who like to gamble and people who are quite successful in life and people who like to gamble and people who tend to not default on their credit payments. I think there’s a company called, I think it’s HealthIQ. I can’t remember exactly what the company is, but basically it’s a lending service, it’s a financial service. And what they do is they correlate how likely you are to repay your loan based on how you use your mobile phone. So they’re trying to find your psychographic and your personality based on how you use your mobile phone. So you accept, share everything that’s happening on your mobile phone, sort of anonymized and encrypted and on the other hand they’re going to give you a certain interest rate and whether they’ll give you a loan or not. And it turns out that people who have one or multiple gambling apps on their phone are more likely to pay back their loans and less likely to default. So when you actually start looking at the social science behind it, etc, etc, is it so bad to actually train or to work with a gambling company, especially one that has sort of limits in place? In that case, I can’t train Candy Crush either and we can’t work with Fortnite or we can’t work with most video games or anything that might be addictive. A lot of e commerce companies, we have data from a lot of e commerce companies. Some of them have their customers coming six times a day to browse for over in total 45 or 50 minutes. Is that a form of addiction? So like you said, it’s definitely a rabbit hole. I think it’s a very difficult question to answer. But like any question, the first step is to ask yourself that actual question. And as long as you have a radar, an ethical radar, that’s already a good start, I would say.

Louis: Right, let’s go to the practical side of things. Thanks for answering this question. I know it’s a tricky topic, but I think you’ve shared a lot of interesting data. Need to dig into the gambling side a bit more as well. So let’s consider. We have a company, you might want to take a real example if you wish, or a few examples. We have a company that have reached product market fit. They understand who their person are and they have a way to reach out to those people. But they don’t have this way to constantly experiment, to go fast and try and all of that. So what is the first step? What do we do to go from this point to a point where we can rapidly test and win?

Changing Mindset: The Hot Dog Champion Story

David Arnoux: I think the first step is a little bit some of the first questions you asked. And really we focused a lot on the tactics and even the process for a very long time. And we forgot to focus a little bit on the mindset. What’s interesting to me is that experimentation is actually at the core of many, many things in our lives. It’s almost like since the beginning of time, random experiments have determined what survives, what dies, what adapts, what dominates. It’s what we call natural selection, survival of the fittest. It’s also feedback loops, right? The law of the jungle is feedback loops, and it takes billions of years. It’s more luck than controlled experiments. And I think humans, they began to discover this. It’s debatable, but the scientific method was developed by, discovered by Francis Bacon and also by Galileo in 17th century Europe. And that was the first time in our common history where we looked towards facts rather than just faith. It was increasingly recognized that facts the result of a series of trials and errors of tests and experiments. The scientific method was born. And we owe some of the world’s greatest invention to this, like antibiotics, silicon conductors, spaceships, all that stuff. Now what’s weird is if the fittest species have embraced this and if science has sort of understood this, why do an overwhelming majority of people in organizations still stick to faith and certainty when taking decisions, when placing bets on ideas? Why hasn’t everybody sort of embraced this trial and error mindset of controlled selection? And some companies have. So, I mean, company like Amazon. I try to study absolutely everything that Amazon does because if you read one of my favorite business books, it’s not a business book, but it’s just reading the letters to shareholders from Jeff Bezos to shareholders. And one recurring theme is that their success at Amazon is a function of how many experiments they run per year, per month, per week and per day. And that’s one question that I really like to ask founders or companies. How many experiments did you run this year? How many did you run this month? How many did you run this week or how many did you run today, for example? Of course, depending on how much traffic you have, how much resources you have, you can’t run as many. But how many are you running? Every company runs experiments. Some call them projects, some call them campaigns, some call them hiring. And this mindset of trying to experiment and do it rapidly is probably the first step. One story that I do like to tell, it’s a story that was in the book, what was the name of the book? Smart Cuts by Shane Snow. And it’s the story of this 1 meter, 76 Japanese man who happens to be like 62 kilograms and who became the world international champion of the Coney Island Hot Dog Eating Contest. And this guy was up against people who were like, genetically born to be the Coney Island Hot Dog Eating Contest champion. The world record was at 25 hot dogs in 12 minutes. Just imagine eating 25 hot dogs in 12 minutes. This guy, the first time he participated, he ate 32 hot dogs. He ate 52 hot dogs in the same amount of time. So he doubled the world record. Yet he’s this small Japanese man who doesn’t seem physically built. What did he do is actually everybody focuses on the tactics that he used to win. He experimented basically for six months in his apartment in Nagawa. How can I eat hot dogs faster than the competition? And he did like 70 experiments in total. Turns out that two of those experiments actually worked out. So he changed the paradigm of hot dog eating. The first thing he did is he separated the sausage from the bread. So it turns out if you separate the sausage from the bread, you can cram down sausages really fast down your throat rather than having this mushy substance of the sausage and the bread. And the second thing that he did was that he found a hack, like a legal hack, a compliance hack. You were allowed to use any liquid that you wanted during the contest, and everybody was drinking the water, the liquid. He started pre dilution digesting the bread by dunking it into the water so then he could just cram it down its throat as well. And he obliterated the world record. So it’s really about the mindset. Everyone else was following the same paradigm. They were all eating hot dogs in the same way. Whereas he came along and he’s like, I’m going to apply some lateral thinking. I’m going to experiment and maybe I’ll find a better, faster way of eating it. And if you’re in business nowadays, there’s so much competition and you better learn to eat hot dogs really fast or unfortunately. And I know we don’t want to create too much anxiety, but you are going to get beaten by a company that goes faster.

Louis: I believe you make me hungry. Jesus. Beautiful story. Now, I really like this story. It’s a really good example of how it Applies to everything, as you said. So the obvious next question then is, okay, that’s all well and good, we need to have this mindset. But how the fuck do we change our mindset if we don’t have this mindset? If our CEO is too scared of running one experiment a month, how do

How to Change Organizational Mindset with Use Cases

David Arnoux: we convince them to change 100%? I think that what’s important is there’s experimenters in every company. They’re just sometimes hidden or they’re not allowed to express themselves. And we need to try to change the mindset. So now we’re in the example where you know there’s pushback, right? The mindset isn’t there. I can promise you that. I don’t know, 20, 30, 40% of your company at least has the mindset. They’re just not allowed to do that. The culture doesn’t allow it. So here we’re talking about change management. We’re trying to change the mindset of the organization. There’s ways of doing that after a while, if it doesn’t work very bluntly, change company. The way to do it is usually to start with a use case. We like to look for the typical cliche, low hanging fruit, which basically means it’s something that is important on the roadmap. Very important on the roadmap, so you understand what the strategy of the company is. Usually it’s linked to retention of users or increasing revenue. Important on the roadmap. And the it’s quite easy to fix. And there what we do is we try to build a pilot case, try to build a first team. It could be one person team, maybe one and a half people. And then we try to hit those low hanging fruit to prove through data that by experimenting a little bit we were able to improve specific metrics faster than anybody else has or quite rapidly, let’s say. And it’s a lot easier said than done. It’s about identifying what those possibilities are. It’s about actually executing it yourself and getting a little bit of buy in from stakeholders. And to be totally honest, if you don’t have buy in from stakeholders, just do it under the radar. There’s that famous story at Facebook where Mark Zuckerberg was convinced that videos would never work, that they shouldn’t dive into videos. So they just did it under the radar. They didn’t tell him about it. And fast forward six months later and Facebook Videos was the second biggest video streaming platform in the world. So we’re looking for that sort of use case and then you use that use case, you Showcase it within your organization. If it still doesn’t work, then sorry, you’re in the wrong place.

Louis: All right. Okay. So I like this approach because we are starting slow. That’s one of the core user psychology facts, like the foot in the door principle. You start with something quite small and to add something quite big afterwards. But you don’t start with trying to say your company is shit, your mindset is shit. Let’s change everything right now. You start with something small that you can prove that has actually a business impact that directly relates to what people care about, and you move on from there. If you can’t do that, and I like your idea, if you can’t do that, do it under the radar. Be careful not to be fired, though, but do it in a way that, you know what? Fuck it, let’s try that. If it doesn’t work, it doesn’t work. If it works, it’s a good story to tell.

David Arnoux: I mean, what’s a job? A job is an experiment in your personal life as well, right? So maybe this is your experiment. See if this is the right place for you to be at.

Louis: Exactly. When you start thinking this way, everything is an experiment, isn’t it? So how do you then advise? We’ll go into what quick wins to pick and how do we select the right ideas. But how do you advise to present such use case once you have it? Let’s say in an ideal scenario, if you tested something, it’s working. How do you advise to present it? Especially to people who, you know, don’t have necessarily the right mindset might come from a place where they’re not super into it just yet. How do you present it?

Visual Storytelling to Present Experiments

David Arnoux: You mean, like physically, how do you present it?

Louis: Yeah, I mean, let’s say if you have the possibility to be in the same room or online or whatever. Like, how do you show that it works? Like, what’s the best way?

David Arnoux: Oh, we use a bunch of templates for this. It’s basically doing a little pitch of what the use case was. So basically we say this was the assumption. This was how I designed my experiments. This is the experiment that they executed. These were the metrics that we were focusing on, and these are the results that we had. And what we found is that visual storytelling is extremely important for this. So nobody wants to read a piece of paper. Let’s say it usually needs to be a little bit sort of rich, a rich story that’s told. One thing that works really well is to record you. Basically, I’m very practical. If you have the person in the room, you just pitch it on PowerPoint for like 10, 15 minutes. If you don’t, if it’s harder. If you want to spread this within your organization, we like to make small videos. You open QuickTime on your Mac, you put your PowerPoint or your keynote and then you present it. You do the audio on top and then you share that on Slack or via email. We also think that it’s very agile way of working, similar to Scrum, to have demos. So when there’s a team that’s working on experiments, they have their Kanban board of experiments that’s sort of easily visible for everybody and a bi weekly demo of the experiments that they’ve been working on. I would argue that you don’t even need to get wins yet you can actually motivate people to work in this way simply with learnings. Although I’m not a big fan of running experiments just for learnings, but already having learnings about the customer, about the product, about the product market fit, about the customer journey is already interesting. So we’ve learned four things about our customer journey in the past two weeks that can also be sort of interesting. And just a side Note, I think 90% of people do adhere to this, would adhere to this way of working. It does kind of make sense. It is kind of common sense. The scientific method does quite make sense. We don’t usually get that much pushback. I can talk later about what the main blockers are, where the biggest pushbacks come from. It has nothing. It’s not. So we don’t see it often.

Louis: Okay, so that’s really good. So it’s really about visual storytelling. It’s about telling a story from the perspective of a customer. And it’s not about sharing a story spreadsheet that is ugly as fuck, that no one can understand. Right. You tell a story and you focus on the user. Okay, so now let’s say we have the right mindset in the company, people are ready for it anyway, but we don’t have the process. So what is step two then? What do we do to start really firing experiments left, right and center?

The GROWS Process Step 1: Focus on One Metric That Matters

David Arnoux: Yeah. So there’s two ways to do it. There’s the cookie cutter approach and then there’s the fuck it shit approach. I would say if you’re not in a team, if you’re on your own, go with gut feeling, focus on one metric and try something out and tell yourself, can I try four things? Can I try four different experiments on the same metric in the next two weeks? Is that possible? So usually we say this and that applies to maybe like 1/4 of people because it’s kind of difficult. So then we have actually sort of taken the scientific methods all the way back to Francis Bacon and then broken it down. We have a video about this on YouTube. It’s quite simple. There’s like these five steps that you should follow. The first one is to focus on one metric at a time. So there’s a guy, I can’t remember who he is, he interviewed 1,500 successful entrepreneurs who had built at least 1 million dollar companies, semi successful entrepreneurs. And the one thing he found that they all had in common was to focus. They were able to focus because the scarcest resource that we have is time. So I would say the first part is to really identify what Alistair Kroll calls in his book Lean Analytics. What’s the one metric that really matters for me right now? So you look at your customer journey, you need to have. One thing I forgot to tell you earlier on when you asked me, what do you need to have in place? Some sort of data tracking, some sort of tracking of what your conversion funnel looks like. So let’s say we would look at your conversion funnel, your user funnel, your customer journey funnel. Where am I bleeding customers? Where am I bleeding users? What’s the real big pain points that I have right now? It usually is revenue, but is there one I can fix before revenue? So if nobody’s coming to my website, my product, my service, there’s no point in trying to experiment on revenue. I need to find ways to drive traffic to my website. We use Dave McClure’s pirate funnel. I think it’s our favorite sort of customer journey frame. So we look at six metrics. We look at awareness, which is driving people to my website, product or service. We look at acquisition. How many of these people don’t actually drop off when they visit my website? My product, my service. We look at activation. What percentage of people reach the wow, I understand the value of my product moment. We look at retention. How many people are actually coming back Daily, weekly, monthly, yearly. We look at referrals, what’s my viral coefficient? And finally revenue. How many people are actually paying for it? And you sort of try to paint a picture of this funnel once you’ve identified that funnel. So we try to identify what is the one metric I want to focus on right now, the omtm, the one metric that matters. And there’s usually like one or two metrics. Typically it’s either a retention metric if you’re a subscription product or it’s a conversion metric. If you’re a transactional product and if you’re a media product, if you’re a media entity, it’s usually something a bit vanity like number of page views or number of ad views. Focus on one metric. I think if we were to stop the podcast right now, there would already be like 50% of the value. Try to focus on one metric at a time once you’ve identified that metric.

Louis: So let me go back, let me go back because you said a lot of important stuff and I want to make sure we cover them. So you said, yeah. Something you forgot to say was you need some sort of tracking. Right. So I don’t want to go too much into the tools right now because I hope that this episode can still be relevant in five years or 10 years and those tools might have gone already. But basically what you’re saying here is you want, in an ideal world, you want to know how many people go through each step of this funnel and you want to know whether or not there’s a huge drop off in one particular stage. Right. At a big, At a high level. This is what you want to know, right?

David Arnoux: Yeah. Exactly where is my most expensive drop off? Exactly where am I the most? Yeah.

Louis: So your most expensive drop off. So it does not necessarily mean that it’s the place in the funnel where you have the highest drop off in terms of percentage.

David Arnoux: No, it’s really. So let’s use a few scenarios. It could be, if you’re running ad campaigns, that’s expensive. If people, it’s a very small number of people are converting on those ad campaigns. Campaigns. If you need to test for retention, that means that you need cohorts of users coming to your website or to your service on a weekly basis. So what’s killing you right now is that you’re not driving enough traffic to be able to measure retention and what your retention rate is like. So that would be the most expensive. If you’re a transactional website, if you’re selling something, you’re doing drop shipping, something like this. If you’re a consulting firm, well, let’s say drop shipping, you need to be converting more. You need to be increasing your conversion rates on your website. And if you’re like a consulting firm, it’s probably your deal size. So that’s your most expensive metric is, oh, our customer lifetime value. Our deal sizes are like 10k at the moment. How do we get them up to 100, 100, 150k?

Louis: And so the way the place in the funnel where you have the most expensive drop off and the one metric that matters are connected, right?

David Arnoux: Yes, 100%, I would say so. There’s this. Sean Ellis came up with the notion of the North Star metric. That’s the most important metric for your business. On the very long term, the one metric that matters is the most important metric for your next two weeks. Short term, next two weeks. What’s the one metric that you need to fix? Like you’re in pain right now and you need that band aid. You need to fix it right away and from experience.

Louis: And I suppose it’s the same for you. When you look at your funnel like this, there will be something screaming at you. It’s not going to be a point where you’re like, oh, there’s five things, I don’t know where to start. There’s one that is bigger than the other.

David Arnoux: Usually eight times out of ten you don’t even need a data analytics tool right now. If you’re not experimenting, you already know where it is. No one’s visiting my website. People are dropping off after one week. We’re not making enough money. They tend to be really simple. All of this stuff is actually quite simple. That’s why we stopped consulting on this stuff and we started training people on it because it’s actually quite simple.

Louis: Interesting. So before we go to the next step, what do you mean by this? So you’ve stopped consulting on it because it was too simple and you felt there was no value in the consulting? More value in training people to do it.

David Arnoux: Yeah, exactly. So the thing is consultants or agencies, we work with agencies and consultants but they try to sometimes complexify things in order to increase the price. There’s a saying like this, I can’t remember what it is, but it’s complexify to something if I can’t remember what it is. And before starting Growth Tribe, there was this one project where I trained three interns on my process analytics implementation, how to run experiments, what tools to use, etc, etc. To be totally honest, they did it better than I did after like a month and a half. And when I used to do consulting projects, six months later I would give them a call. How’s it going? Are you still growing? No, you left. Whereas where I trained people, those three interns inside the team called two months later. Hey, how’s it going? You’re still growing. Yeah, 100%. By the way, I’m head of growth now and we’re running experiments you want. And by the way, we’ve done this, this, this like, oh wow, I just learned something, something. And so we found that actually growth should be embedded within the DNA of the company. I still don’t understand that we internalize product development yet so many companies still outsource their growth and their distribution. They’re outsourcing their tone of voice and that’s crazy in a world where distribution is maybe your biggest competitive advantage. So in that sense, educating people inside the organization is more cost effective and you have longer term impact on the org, embedding that growth into sort of the. Yeah, into the DNA.

Louis: Interesting. And yeah, this is one of our biggest marketing pet peeves, right? This addiction to name and coin new terms and create new processes on top of processes to make yourself look smart instead of simplifying to the core. Because as you said, this thing, even though it’s hard to do, right, it’s not complex, it’s not difficult, it’s just simple. But it takes hard work, right?

David Arnoux: Yeah. And I think the difficulty grows with the maturity of the organization. The first steps are quite easy and your first experiments will be quite easy. But then you get more mature and you grow in maturity. Then your tool set will become more difficult, your tech stack will become more difficult. The types of experiments you’re running will become more difficult. The metrics you’re going after will become more difficult. We always advise people for their first experiments to tap into the top of the funnel, the tofu. That’s usually where you have the least blockers to experimentation. It’s usually where you have the most traffic, so you can get the biggest numbers if you need statistical significance, for example, it’s usually where you have the least stakeholders and it’s usually where you need the least support. The deeper you go into the product journey, the more that you’re stepping on people’s toes. Operations, sales, marketing, developers. If you’re working on a SaaS product subscription product and you need to make changes to features, then you need to get buy in from a very scarce research, which is the developers. So we always say usually the easiest one tends to be at the top of the funnel. Right.

Louis: So we have our one metric that matters now, what’s next?

Gathering Ideas: Best Data Sources for Experiments

David Arnoux: So we like it when people remember this. So we call this sort of the Grows loop G R O W S. And basically first you’re going to gather as many ideas as possible. So you know how like developers have a feature backlog. Grow’s team, they have an ideas backlog which is basically a dump of all the ideas. We like it to be a democratic idea backlog. We’ve actually developed a tool called the Grows tool where people can drop ideas whenever they want into that tool. And one thing that’s really interesting is we found that most of the best ideas that have worked or the most creative ones don’t happen during working hours. That’s an interesting fact. It’s basically a super fancy UI for people to drop ideas in there. And the ideas, usually the best ones, come from the best internal and external quantitative and qualitative data. This is one of the reasons why larger organizations that have business intelligence units, they tend to be better at coming up with successful experiments, whereas a smaller startup, a smaller organization who’s doing a lot of guessing, they can run more experiments, but they have less sort of successful ones. So the first part is to just do a brain dump of all of the ideas that you have based on data analytics, based on user testing, based on user intelligence, business intelligence, and just based on your gut feeling, for example. So we call that gathering as many ideas as possible.

Louis: Okay, so based on user testing, based on analytics, based on your gut feeling, from your experience, what is the best source? If you have to select one source of data and try to be as specific as you can, like what is the best source of data that you can rely on to truly come up with good ideas?

David Arnoux: Let me give you two. I’ll do two. For top of the funnel for campaigns, your competition just copy reverse engineer reversing engineer campaigns from competition. There’s a million tools out there that allow you to do this and they’re probably using a tactic that’s going to work for you. Really good one. For deeper in the funnel for things like retention metrics or revenue metrics. It’s actually just really hard, cold, raw data from your analytics software, usually by doing a little bit of correlation analysis. So there you want to have something nice set up like an amplitude or a mix panel or adjuster or whatever to be able to look at the data. That’s why also those experiments on retention tend to be a little bit more complicated than the ones on marketing campaign. And it’s the hard data, but also the qualitative data linked to that.

Louis: And what you mean by correlation analysis would be, for example, when this user used this tool and this tool together, they tend to retain more than when they don’t. For example.

David Arnoux: Yeah, I’ll give you a practical example. We were helping out a task management app and they found out that when we ran correlation analysis. So of course correlation is not causation. Just because two things are linked doesn’t mean that one caused the other. But we found out that people who had integrated their calendar into the app were more likely to be retained longer. And the calendar integration was lost somewhere really deep in the app. It wasn’t part of the onboarding. So then we did an A B test, causal inference test, where actually the onboarding was only three steps and the second step was integrating your calendar. And yes, in fact, when we run that test, we were able to. I can’t remember what the uplift was, but more people were actually retained on the platform once they integrated the calendar. No one would ever have thought of that feature as helping with the retention. The second one that we found was customizing the color of the left menu. It’s stupid, but being able to customize the color of the left menu actually meant that people felt like they owned part of it. It had less of an uplift, but it also had an uplift there.

Louis: What do you advise people who don’t necessarily have the resource to do that? What other tools should they use? So you said competitor, is there something else they can replace it with that is maybe not as good for sure, but maybe good enough?

David Arnoux: Yeah, sure. So you mean for deeper metrics that are deeper in the funnel?

Louis: Yeah. To have another source to rely on to generate new ideas.

David Arnoux: Yeah. So this is a lot of the stuff we teach as well. So this one’s pretty straightforward. You do the same thing, but with qualitative data. I call it Sunday Data Science. So basically what you’re going to do is you’re going to take three batches of customers. Ones that never use your product, they dropped off right away. Ones that are still using your product and ones that started using your product but stopped. Or people who didn’t buy versus people who almost bought and people who actually bought. And then you fight like crazy to try to have an interview with them or to talk to them. And then you find out what is the difference between the ones that stayed, the ones that didn’t stay. Now, if you don’t like talking to customers, of course you can use tools like hotjar. Hotjar user recordings are fantastic for this. They’re like. So I used to do this. I would watch between 100 to 1,000 user recordings at five times speed. Wait, is that a thousand? I’m exaggerating. It was more like 100. I would look at 100. And with hot. This is becoming sales pitch for hotjar. But I would look at 100 user recordings of people who were retained on our platform at 5 times speed. On Hotjar, and look at 100 user recordings of people who were almost onboarded and almost started using the product but dropped off. And my brain would do the correlation analysis, I would start to spot some patterns. This is the number one tactic that most people use. We’re too lazy to set up good analytics. And it works. I mean it just works.

Louis: I’m glad you mentioned this tool and I don’t mean hotjar, I mean your brain. Because as you said, your brain is actually pretty good at do correlation analysis. I mean it’s definitely fallible. There’s a lot of mistake there. You have a lot of pre assumptions you want to validate. But as you said, if you look at the behavior of people who didn’t convert versus the one who did convert or the ones who almost didn’t, you can start seeing patterns. Right. We’re pretty good at doing pattern analysis. So you see pattern differences and this is when you can start saying, ah, okay, those ones who didn’t convert, they didn’t understand that you could close this window while the ones who did. So you start seeing those stuff. Right. And that really should make your brain run and, and come up with shit ton of ideas as well.

David Arnoux: Yeah, and I think what’s beautiful with the digital world is you can have access to this in one or two days for a very low price. This used to take focus groups, which are bullshit anyway because people act differently when they’re under the microscope. If you look at bacteria in a normal state, bacteria doesn’t move. As soon as you shine a light on bacteria and look at it through the microscope, it starts moving. I think it’s called the Heidegger effect or something like this. Nowadays with all of these beautiful tools that are available, I’m able to launch a 5 second test on my website in a matter of 15 minutes. I can understand qualitative data based on hot draw user recordings in about a week for like 19 bucks. And I think it might even be more powerful than conducting customer interviews, because customer interviews, there’s always the problem of if I only interview 10 people, maybe it’s up to luck that those I can see a pattern in those 10. In those 10 people. We have access to so much data thanks to these tools. It’s exciting and there’s almost no excuse to not be using it. I think the excuse might be lack of knowing how to Google and a little bit of laziness maybe, or lack of drive.

Louis: All right, so at this stage we’ve got our ideas, right? So anyone can pitch in. And you said it’s a democratic process, meaning people vote.

Prioritizing Ideas with ICE and BRASS Frameworks

David Arnoux: Yeah, so that’s the next part that’s sort of ranking the ideas. So we use some ways to rank the ideas. You want to try to have as many brains as possible in the room. So typically a creative person, analytical person, and then sort of salesperson, a hacker, hustler, hipster, or a developer, a marketeer and maybe somebody who’s customer facing. Sales is customer facing, of course, but like customer success, you know, everybody always says you should talk to your customers, by the way. I find the best shortcut for doing that is to talk to sales or to talk to people. Customer support, for example, because they talk to the customer every single day. And so we take all of the ideas and then we rank those ideas based on one or two frames. If it’s a marketing strategy idea, we have a framework called the Brass framework. It’s a mix of gut feeling, Blink R is relevance. How relevant is this channel for me? And I won’t go into all of the details, but basically there’s two frameworks. A little bit of gut feeling, a little bit of how big of an impact will this idea have and how easy is it to implement this experiment? Because I’m fine for you to redo your whole onboarding or redo the whole navigation process on the website, but can you actually execute on that experiment? And we try to look for what’s called a low hanging fruit is something that has a high probability of succeeding based on your gut feeling that it will have a high impact on the metric that you’re trying to improve. And it’s just easy to pull off once you start to be a bit more mature. You can then filter these depending on do you want a quick win now or do you want a really impactful win? So I mean we are training, for example, a large pension fund at the moment and they’re actually running now 15 experiments every two weeks. They’ve reached a maturity where they were going for that high impact experiments because everything is sort of set up so we try to get together, rank those ideas and then what happens is organically you have always three, four ideas, five or six that really stand out. So we’re going to put those aside, say, okay, we’re going to do those. And then we have two Takeru Kobayashi ideas you like, really radical ones. You know, that’s the hot dog eater I was talking about before, like paradigm shifting, not so obvious ones where the probability is not so high. And then somebody usually puts a veto and says, I want to do this crazy Idea and we try to have 20% of experiments to be a bit wacky. Not based on data, just based on. I really want to carry out this experiment and I think one thing that’s really important is an experiment will never succeed if somebody doesn’t truly believe in it. If there’s not an experiment owner, because usually there’s like one or two strikes the first time you launch it. Somebody does need to really believe in it and be the leader of that idea.

Louis: Give me an example of a crazy experiment that was actually successful and no one really thought it would be that.

David Arnoux: No one actually thought it. No, I think saying no one thought it would be, would be maybe, maybe just a few.

Louis: It was a crazy enough idea to say shit like if it works. Well, if it doesn’t. Yeah, no surprise.

David Arnoux: I mean, let’s use a personal example. For us it was, for example, giving stuff for free that was usually extremely expensive and thinking that we would actually be able to convert people after it. So our products have a high value, so the price is sometimes quite high. And we thought, could we repackage this into something that’s a little bit smaller, a little bit more of a teaser, but that actually we don’t make people pay for. So, for example, a power session. Would it be possible then to actually convert these people even though they already had all of the value from the course? I think another. The one I gave before about the calendar. Nobody believed that the causation was true. Everybody thought it’s just correlated. People are integrating the calendar because they’re using the tool. They’re not using the tool because they’re integrating the calendar. And it turns out it was exactly the opposite. I do have to say that these radical ideas though, 90% of the time they actually don’t work. So don’t do only those. I think people should. Yeah. And that’s actually a problem. People do tend to focus on these radical paradigm shifting ideas, actually. The easy stuff, the really obvious stuff. It’s like Occam’s razor. The. The solution is usually quite simple. And you were talking about basics before. 9 out of 10 companies or teams are just not doing the basics yet. They’re not. So usually it’s the simplest stuff that actually ends up working. So I have one example, one more, if you want to, or we can.

Louis: Yeah, let’s go.

David Arnoux: This was for a consumer electronics company and they were launching this big new razor and somebody’s idea was. One of the ideas was the big Ogilvy firm came in and they said you should start A movement and this should be the copywriting, etc. And one of the interns in there that was in the room said, why don’t we try something crazy? Let’s just call it this is not a razor. And it was actually a razor. Let’s try the campaign this is not a razor. And there were seven people in that room out of eight didn’t believe that that would work. But we tried it anyway and it just exploded all of the other campaigns. So that one was pretty fun.

Louis: Nice. So to summarize what you said around prioritization, you basically have two frameworks that you use. So there’s the ICE framework, like the impact, confidence and effort that it takes. You can rank that pretty well. And the second one you mentioned is what the brass. Brass, you say?

David Arnoux: Yeah, yeah, just Google it. It’s called Brass framework B R A S S. So it’s blink. So this is only for traffic generation ideas blink. What’s my gut feeling? R relevant. Do I have product channel fit? If I’m GoPro on YouTube, fantastic. GoPro on LinkedIn, do I have product channel fit availability? Do I have the resources in house to do this? So if it’s videos, do I have video software? Do I have people who can actually make video? If I want to do content, do I actually have somebody who can make content? And then important one that we sometimes forget, it’s the S. So V A R S scalability, can I crank this up? SEO is sometimes hard to scale PR stunts, they’re hard to scale. Conferences, they’re hard to scale. So if we want something that really allows it, we have very, very strict lead generation targets for next quarter and we need something that’s easy to scale, then we will also maybe give a more important score to the scalability. Hopefully a very scalable channel works like a social channel or a search channel.

Louis: So as you said at the end of this exercise, we have a bunch of experiments we know we want to run. You mentioned six that you don’t have to stay to six, I suppose. But what do you do then? What’s next?

Designing Experiments in Detail with Two-Week Time Boxes

David Arnoux: Yeah, now you actually have to design the thing, design the experiment in very precise details. And for that we just use like an experiment card. And basically it’s. We believe that by doing this, by doing X, we can improve this metric. To verify that we are going to. And then you describe your experiment. So to verify that we are going to do this, this, this and this for a period of one week, we are right if we hit these numbers. So we are right if so, let’s say we’re doing, I don’t know, like a campaign. I have no idea. Let’s just say we’re doing a campaign. We believe that by launching this campaign, we’re going to be able to increase sales. To verify that, we will create 20 different assets for this campaign. We will launch them on this platform. This platform. From this platform, we will let the ads run for a period of one week. We will spend €100 per ad asset. And we are right. If we get at least this much increase on our one metric, that matters. So, for example, we sell 25% more than we usually do. That’s the hardest one. We are right. If putting a line in the sand, See if you’re right or not. I actually shouldn’t say this, but it’s also okay not to put a line in the sand. Just run your experiment, see what happens, happens, and see if it increases it. Yes or no. So that’s basically it. You design your experiment with excruciating detail, and then you reevaluate your E score once you’ve actually designed your experiment, because it turns out you forgot a lot of stuff when you were actually doing the E score. So then it’s good to have those five, six. That’s basically it. So you design your experiment with excruciating detail, as much detail as possible. We call it outlining experiments. We have a theory at GROSS Tribe that 90% of experiments should last two weeks maximum. From the thinking about it to the setting it up, to the letting it loose into the wild and analyzing the information. If it lasts more than two weeks, you need to break it down. Even if this is like a big strategic partnership, even if this is a big campaign you want to launch or a big new feature release. If you’re smart about how you design your experiment, you can actually test the riskiest assumptions of that big campaign or that big feature release with smaller experiments. I want to have a strategic partnership with Virgin Media. I have no idea. Well, okay, that’ll probably take six months to test. Okay, but let’s break it down. Would it be possible for me to already. Can I even find 10 contacts of the people I want? Okay, that’s one experiment. Probably takes me like, I don’t know, an hour. If I reach out to them, can I get two of them to contact me back? You know, that probably takes like a day or two. So you can always break these things down. You’re basically trying to hedge your bets. You’re trying to limit the risk as much as possible by placing as many small bets as possible.

Louis: Way to focus on only one week or only two weeks, especially when you get started.

David Arnoux: Yeah, so this is just. We did this, you know, Tim Urban’s blog, and he sort of looks like how he turns your life into boxes and how much time you have. We realized that, that there’s not that many two week periods in a year. If you include sick days, holidays, weekends, all that stuff, there’s not that many two week periods. If you want to run as many experiments as possible, you’re going to want to shorten it. You’re going to want to time box yourself as much as possible. So it’s just to increase the quantity of experiments. I used to be so focused on pushing people on the quality of experiments and then I realized that actually people already run quality experiments. No one’s running stupid experiments. Everybody talks about changing the color of the button on a landing page. Nobody does that. Nobody really does that. So now I really try to focus on the quantity of experiments. I trust you as a founder, as a head of growth, as a marketeer to run quality experiments. I trust you to understand your customers. I just want you to run as many as possible, as fast as possible.

Louis: Okay. I think we’ve covered the basics in quite a lot of detail. So thanks for going through that with me, man. Really appreciate it. I think people really got a lot of value out of it. I get quite a lot of emails from people who are either lost in marketing, they haven’t started, they want to use marketing, but they don’t know where to start, or they want to become marketers and they don’t know where to start. I’m curious from your perspective, you’ve worked with multiple companies, you founded multiple companies, you have yet another successful company under your belt. What would be your biggest advice for them to get started in the marketing world?

Career Advice for Aspiring Marketers

David Arnoux: Yeah, I think look at your personality type. So maybe look at the ocean model. Openness, conscientiousness, extroversion, agreeableness, neuroticism. And based on that you can find out what type of marketeer do you want to be? Do you want to be a number cruncher or do you want to be sort of consumer facing? Do you want consumer products or do you want B2B products? The best way to do this is always to find a marketeer. You look up to something that you really like and try to reverse engineer what process that person went through. If possible, even contact them and ask them what were your first steps? I mean, there’s three, four types of marketeers nowadays, right. There’s still sort of the brand marketeer which is very good at storytelling, very good at conceptualizing, very good at creativity. Then you have direct marketing which can basically be pushed into two. One is the pure hardcore performance marketeer, all about conversion optimization and short sales cycle. And then you’ve got the sort of data driven marketeer on longer sales cycles. And then maybe you have like the content marketeer that’s much more about generating content, building content. So I would say look at your personality and try to find somebody who has a similar personality, who you really look up to and then try to reverse engineer what they do and which skills they actually have. And the best way is just experiment. Luke, our head of digital, the way he got into marketing was he was going too slow in his previous company. He just started to play around with fake dropshipping just to sort of experiment with what he actually liked. And then he found what he actually liked there. So it’s experimenting at home, basically.

Louis: How do you so repeat for me the different, the letter, the process to find out your personality? What is it again?

David Arnoux: Oh, so this is one of the most researched. So it’s called the big five or the ocean model. I also think that this is very important for marketeers to understand. There’s a beautiful use case of behavioral psychology done by Sandra Matt. We are not bad people as marketeers. The research she did, it really contrasts to decades of research reporting that there was a weak relationship between buying products, consumption and happiness. It turns out that’s not completely true. Her findings suggest that you can increase happiness if you buy the right things based on your ocean personality profile. So what they did is they used more than 76,000 bank transactions and then they found that individuals who spend more on products that match their personality tend to actually be happier in life. I’ll give you an example. If you’re extremely high on conscientiousness, very long term thinking, and you’re kind of an introvert, you’re actually going to get life satisfaction from investing in accounting software or legal advice. If you’re very much of an extrovert, a little bit less conscientious, you’ll get a lot of happiness and life satisfaction, not happiness, life satisfaction from buying experiences, from going on holidays, from going to the movies, for example.

Louis: Well, okay, so quite a good answer. I hadn’t heard of this action process and I hadn’t heard of this research, so thanks so much. I’m pretty sure a lot of listeners would get value from that as well. What do you think marketers should learn today that will help them in the next 10 years? 20 years, 50 years?

What Marketers Should Learn for the Next 10-50 Years

David Arnoux: From a purely skills point of view, we write a lot about this stuff and I start to get lost in it. I think the data visualization is powerful. Ux also very powerful. Data science is definitely interesting. Developing intuitive understanding of data science. But like, in 10 years, what were the time frames?

Louis: Sorry, 10 years, 20 years, 50 years. Long time.

David Arnoux: Honestly, I have no clue. In 10 years, I would say. More importantly, I would say learning to focus and enter a state of flow. I think entering a state of flow is more important now than ever. We’re so distracted. Knowing how to do deep work has become a competitive advantage. So learning to read, learning to learn, your competitive advantage is going to become deep work. Not many people know how to do deep, meaningful work anymore because it’s like trying to get deep sleep. You can’t do deep sleep if you keep getting woken up by somebody patting you. If you can get into these deep work, deep flow, then you can actually build something that’s more powerful than other people are. So that would actually be an edge. Learning to sort of learn and learning to. Yeah, learning to learn. I would really recommend diving into this. There’s a book, I forget who it’s by. It’s called Deep Work. That’s interesting. And then there’s that book I recommend in the beginning, it’s called Stealing Fire as well. And Staying extremely Curious. And then on the hard skills, you’ll always need UX and you’ll always need something. Something with regards to the customer, whether that’s through data and data science and analytics or maybe sort of something more psychological. Behavioral psychology. At the end of the day, marketing is a lot of behavioral psychology. The person who invented PR was Edward Bernays and he basically invented propaganda.

Louis: I knew you’d find something good for this question. So thanks as well. And maybe on the back of that, as you said, you shared a lot of resources on this episode already, but perhaps you could come up with three others that you recommend our listeners today. So it could be anything from podcasts, books, conferences, anything you want.

David Arnoux: Oh, yeah, actually, I prepared this one. Yeah. So podcast, I would recommend the A16Z podcast. It’s the Andreessen Horowitz podcast. It’s amazing. It’s like having some of the best VCs in the world as your mentor. The Internet is a beautiful place. The second one is kind of random. It’s a subreddit called Lectures and it’s basically watching. I like to watch Random lectures on subjects that I don’t. Didn’t know I would be interested in. And it allows you to develop your lateral thinking. So it’s a subreddit called Lectures and you’ll find out about the history of mathematics or the mating call of some weird bird in South Africa. Really interesting stuff. And then of course there’s, I need to do a little shameless plug. There’s our YouTube channel where we share Growth Tribe YouTube channel. Just type Growth Tribe YouTube. And we share a lot of sort of these tools, these tactics. We have specifically one called Growth Insights where every month or every three weeks we share the latest tools, latest tactics and resources in a condensed, very fast sort of manner. So those would be like, yeah, the top three. I’m not going to recommend any more books because I know you guys aren’t going to read them anyway.

Louis: David, once again, thanks for your time. Like really, really insightful episode. I really appreciate especially the fact that you shared a lot of resources from different angles, different stories. I think people listening really got a lot of value and I genuinely mean it. So how can they connect with you and learn more from you?

David Arnoux: Yeah, so connect with me on LinkedIn. I’m really, really focused on LinkedIn at the moment. And you can just connect with us Growth Tribe on Instagram as well. We’re also hiring, so come and check out growth tribe IO. Come and check out the website. We have like 22 positions open at the moment, so if you’re smart, come over.

Louis: Nice one. Thank you so much.

David Arnoux: Thanks. This has been great.

Louis: That’s it for another episode of everyone hates marketers.com. and this is the moment where I tell you to subscribe to our email list. So before you leave and go to another podcast or listen to another episode, I don’t treat email lists the way people usually treat their email list. I really treat that as a one to one conversation. So I’m going to send you very short personal emails every two weeks. I would say I’ll inform you of guests in advance. I’ll share with you my numbers and how many listens we get and I’ll also ask you for your feedback in terms of the questions we can ask future guests. And perhaps I can also have you on the show someday. So don’t be afraid to subscribe. I’m not gonna spam you and you can always unsubscribe for sure if you wish. The second thing we need from you is your harsh and honest feedback. We know that this show is not perfect yet and we always can improve so you can send us your email@feedBACKYone hatesmarketers.com Good or bad? Please feel free to send me an email and the last thing I’d like from you is that if you did like the episode, please share it to your friends, your colleagues or whoever might like it. And also please review it on itunes or another service that you might use to listen to your podcast. Because if you leave us a five star review, it means that more people will be likely to listen and we can spread the word quicker. So thank you so much once again and Aubrey. And that’s it for another episode of everyone hates marketers.com thank you so much for listening. I’m super, super grateful. I’d love for you to consider subscribing to my daily newsletter Monday to Friday called Stand the Out. Daily. I send very short, hopefully interesting, surprising, shocking, entertaining content to help you Stand the Out. It’s at everyone hatesmarketers.com you can subscribe for free and obviously unsubscribe whenever you want. I’m just gonna read a couple of emails that I got recently as a reply. Juma said, your content attacks the mind primarily, which is such a good thing saying because most of us are skilled at what we do, but we don’t have the courage to do it our way. Mark, who just subscribed couple days before, said, this is my first issue of your newsletter. Love it. Glad I subscribed. Brianna Said, I just realized this morning that my email habit is now to 1. Skim through the list. 2. Select all unread industry email except yours. 3. Delete and don’t think twice. 4. Quickly skim yours. Amy said, also loving the new content is coming from you. It feels really lovely. Kendall said, I like your writing a lot. It really resonates. There’s so much bullshit out there. It’s good to touch the authentic. And Chloe said, where is the I love this email button? Brilliant. I hope you subscribe. You’ll be joining more than 14000 subscribers at this stage, which is crazy. It’s the size of a small stadium. Anyway, thank you so much. See you on the other side.