MBA Masterclass The Chicago Booth Approach to Marketing

Kara: Good morning, everybody, or afternoon or evening, depending on where you're joining us in the world. We'll start in a couple minutes. We just wanna let everybody have some time to join. So thank you.

Kara: JP, are you in Chicago?

Jean-Pierre Dubé: I am indeed. I'm in Winnetka, actually.

Kara: Great. If those joining us wanna let us know via the Q&A where you're joining us from that would be great.

Jean-Pierre: For a little bit of trivia for everyone, I'm just a few blocks away from the "Home Alone" house in the movie "Home Alone."

Kara: That's funny. Is it a tourist stop in your hood, I assume? In your neighborhood?

Jean-Pierre: Well, I will admit that when I have some of my friends coming, especially from Canada, they get a really big kick out of going and seeing the "Home Alone" house -- and also seeing the park where the little kid runs across and there's a skating rink in the movie, it's still there.

Kara: That's funny. Nice. Well we have people from India, Hong Kong, France.

Jean-Pierre: All right, yeah, we have some folks from Hong Kong. I must admit I'm missing my trips to Hong Kong with the pandemic. I'm looking forward to getting back to Hong Kong. I used to go once or twice a year; I'm supposed to teach again this fall there but it looks like that will be done by remote.

Kara: OK. Is that the plan? But hopefully soon enough. I've never had a chance go but it's high on my list.

Jean-Pierre: Oh, it's awesome. I love going there. And our new campus is really beautiful -- and still sufficiently close to central. The quick cab ride back into central can get back into the city.

Kara: Awesome. Costa Rica; Indiana -- that's where I happen to be -- North Carolina.

Jean-Pierre: I saw there.

I know I was literally just in Cartagena two weeks ago, so.

Jean-Pierre: Oh nice, OK.

Kara: It was wonderful.

Jean-Pierre: I was just having a conversation yesterday about potentially going out, just a little bit south of there, down to Chile. I wanna go skiing in August.

Kara: Oh my gosh! That sounds amazing.

Jean-Pierre: I know. So, South Korea; hey, that's awesome. Yesterday I was barbecuing kalbi on my barbecue outside. Excellent.

Kara: D.C. of course, Houston. Some of our core demographics here in the U.S.

Jean-Pierre: OK. This is a big group.

Kara: Yeah. Yep.

Jean-Pierre: People keep arriving.

Kara: I'll start in like 30 seconds that everyone can join. Japan. Great. New York. I am probably most excited to actually just go for a weekend in New York. That's something I'm itching to do now that we can do a little more traveling.

Jean-Pierre: Well, I admit that once I was vaccinated my first stop was to snowbird in Utah for some skiing.

Kara: Nice. I'm jealous. We went back and forth ...

Jean-Pierre: Patience was rewarded.

Kara: ... in the ski trip we didn't end up doing it. We talked about it a lot all winter, but...

Jean-Pierre: Well, my patience was rewarded. We got 48 inches of fresh powder over about two and a half days. Go Hoosiers!

Kara: I had to give a shout out for that. I happen to be a Hoosier by birth and by education, so...

Jean-Pierre: That's hilarious.

Kara: ... I appreciate that. All right, let's go ahead and get started in two seconds here. All right: Once again, welcome everybody. It's great to see such a variety of geographic areas represented. My name is Kara Northcutt. I am Senior Director of Employer Engagement and Admissions at Chicago Booth. On behalf of the evening, weekend, full-time and executive MBA admissions teams, thrilled to welcome you to our marketing masterclass with John-Pierre Dube -- JP, as we typically call him. And the class will be all about the Chicago Approach to marketing focused on pricing. Feel free to submit questions throughout using the Q&A. So what we'll do is JP will present for, you know, 45 minutes or so or up to an hour, then we'll do an open Q&A after that. So today you'll get to experience two key components to the Chicago Approach to business education. Our data-driven and evidence-based approach helps you learn how to ask the right questions, think more strategically and analytically, which prepares you and gives you the confidence to take on business challenges of today and tomorrow. Another key advantage of Booth's program is our supportive and collaborative community. When you join Booth, you're a part of this really collaborative and supportive group of students, alumni, administrators that are really there for you throughout the duration of your MBA, and then your life as an alumni, of course. And an integral part of that community is our faculty. The same faculty teach across all of our MBA programs; as you may have heard, JP even referenced his teachings in Hong Kong, which we do. And, you know, in the non-COVID times, he's one of those faculty that's teaching in London and Hong Kong, as well as Chicago. So we're really excited for you to get a little bit of that experience of the Chicago Approach today. JP is the Sigmund E. Edelstone Professor of Marketing at the University of Chicago Booth School of Business. He is also Director of the Kilts Center for Marketing at the Booth School and a research associate at the National Bureau of Economic Research and appointed academic trustee at the Marketing Science Institute. From 2008 to 2010, he was a research consultant for the Yahoo! Microeconomics Research Group. And he has been working as a research consultant with Amazon since 2018. JP is focused obviously on marketing and teaches both quantitative marketing and pricing strategies. And with that, it's my great pleasure to turn it over to JP. Thanks, everybody.

Jean-Pierre: All right, thanks so much for that introduction. Hi, everyone. So let's try and just make sure we understand what it is I'm gonna try and do with our time together this morning. I've tried to take some concepts from my pricing class but use them less to teach you about pricing and more to try and give you a flavor for what sort of values we espouse at Booth -- and more importantly how we tackle business problems. What is it that we do? And I guess in that regard, I'm trying to tell you a little bit about what our brand means, at least from the purposes of education, from the perspective of education. So I'm gonna talk about pricing, but keep that in mind. First, let me tell you a little bit about pricing and really why I think my pricing class has become really popular. I've been teaching it for about 15, oh gosh, 16 years now, I've been teaching pricing. But it's become an increasingly popular course. And I think, you know, who could say this better than Warren Buffett? You know, Warren Buffett has been quoted in a number of occasions saying that, "The single most important decision in evaluating a business is pricing power." And what does pricing power mean? Pricing power is the ability of a company due to the value it creates for its customers -- and some of that value reflecting the differentiation and uniqueness of its product or service -- it's their ability to raise their prices above unit costs to generate margin. And in addition to that ability to generate margin is to have a sufficient customer base who gets that value that they would be willing to buy at that margin. So it's about being able to sell it scale at a non-trivial margin. That may sound like a little bit aspirational but I'm gonna show you what that means in a moment. And of course, once you realize how important pricing power is to the value of a business -- this is now value to stakeholders -- it's always surprised me how ad hoc pricing decisions typically are in practice. That most companies use very heuristic and simplified rules to set prices. And again, there's nothing wrong with simplicity unless simplicity can be deceptively simple, and as a result of that simplicity, companies are leaving money on the table based purely on a lack of knowledge and a lack of skills to do better. So there's tons of evidence in academia; I've devoted most of my research career to studying marketing problems and pricing has been one of the most frequent themes in my research. I also work with companies. I'm gonna show you in a moment a collaboration I had with a company that was relatively recent. So I have a lot of experience both in the field, and the one thing I've learned -- especially through these collaborations with companies -- is how much value using the Chicago Booth Approach can bring to the field. So let move into that to try and explain, well, what is the Chicago Booth Approach and why does it add value? 'Cause his may sound very self-serving to you but I don't think this is per se about Booth as much about science. Because that's what the Chicago Booth Approach is. And in a moment, I'll show you how to apply this idea to setting a price. But Chicago Booth is actually a scientific approach and that really does differentiate your experience at Booth from our other schools. What that means is that instead of showing you hundreds, or maybe even thousands of case studies -- right? that is one approach to business education -- thousands of case studies. And the idea is that when you graduate, when you're in the field and you're found confronting a real world business problem, you try and remember from your memory: What was the case study that matches the facts most closely? And how did we solve that case study? Well, the reality is in today's era where there's so much evidence, data and skills, it's kind of like too simple -- and kind of too approximate -- to try and think that there's a case study that exactly solves the problem at hand. And instead what we do at Booth is we give you scientific tools to figure out a solution that makes sense in the context you're trying to make that decision. So let me now define what it means to use the Chicago Booth Approach. To solve a business problem, we start off with theory. So that could be economics, for example, and economics is a really great starting point for most business decisions, because most business decisions are geared towards objectively improving the value of the firm -- and economics is all about objective frameworks to think about how to measure stuff. But you also might immediately appeal to psychology because as helpful and useful as it is to write down objectively what one ought to be doing, we also know from decades -- or frankly, from more than a century of research on human decision making and behavior -- that individuals don't always behave objectively. Sometimes firms do things that are grossly illogical, such as having a management team that's more obsessed with building an empire than building long-term stakeholder value. Or consumers who are willing to literally blow up or destroy all the potential value they could get for their money simply because they're put off by a marketing theme or put off by some obscure notion of fairness. So marketing is just as important to use economics as psychology, sociology, culture, anthropology. It's about building scientific frameworks so that you can craft the simple and coherent theory of what you think are the issues to consider when you're making a decision. Once you've got the theory in place -- and by the way, sometimes there are competing theories, which means part of making the right decision is gonna be able to test between those theories -- the next step is the evidence. What kind of data are available? And depending on the quality and quantity of the data available that will in turn dictate what kind of quantitative methods are best suited to your problem. And contrary to the so-called big data revolution, not every decision problem -- and in fact, most decision problems -- don't actually require so-called big data and don't per se need artificial intelligence. I think that AI has become a bit of an oversell now. And before AI, it was the word "big" data. I'm almost starting to come to the conclusion that what we've always been arguing is that you need some analytics in your business, and that the vocabulary we use to brand analytics just keeps evolving so that people think it's fresh and new. But the reality is what I'm describing here on the slide is just an analytic framework. You start with a theory; you then go to the evidence; and then thirdly, it's about implementation and trying to determine why the decision that emerged from this is good for your business. What's the return on investment? And most importantly, this whole process is about making your decisions more accountable. Now, I teach in marketing and surprisingly, perhaps for some of you, one to my main audiences has been CFOs. In fact, next week, I'm gonna do a half-day course on marketing analytics exactly for CFOs, because I think a lot of CFOs are keen on trying to update and modernize the marketing divisions of their companies to make marketing spending more accountable. But in a nutshell here, the Chicago Booth Approach to marketing is gonna teach you how to use theory, how to apply models to analyze data in a ... excuse me, methodical and logical manner. And that's why we start out with the theory, even when there are not data. Even when there is no method for analyzing your data because the data are in too crude or rugged a form. Having completed step one is already in and of itself transformative. And I use the word transformative because in a nutshell, that is the Booth experience. We use a transformative approach to teaching because you come in, we teach your principles and that all transforms you into a scientific decision maker. So let's get to pricing, then. I'm gonna couch my discussion today on pricing in a collaboration I did a few years ago with a company called ZipRecruiter.com. If you watch CN -- excuse me, if you watch CNN, you've probably seen some of their ads. Although recently I've noticed they've reduced the weight on their ad campaigns. But in 2018 and '19, they had ads going on during, you know, prime hours at almost every other advertising pod. If you haven't heard of ZipRecruiter, it's basically a human resources platform. What they do is they manage a massive resume database where employees -- individuals, excuse me, people looking for work can post their jobs and create a profile. This is all free. Anybody can create an account and post a profile and upload a resume. The paid part, the cash part of their business, is coming from employers or companies that pay monthly subscriptions with ZipRecruiter to be able to query and access the database. And of course, the value-add is much more than just access to the database; it is in fact machine learning algorithms that allow companies to digitize what their needs are from an employee, and then the algorithm does a large-scale matching -- a classification, if you will -- to try and find the best matching employees, right? So it's about matching resumes to prospective employers and helping them automate and facilitate the recruiting process. So it's an online matching platform. They have high conversion. The quality of the matches is high. The company has been growing. They've been the fastest-growing HR company now for several years in a row. But my collaboration with them started in August of 2015. It actually started in July when I started having calls with the CEO and CO. And during those calls, one of the things I learned was that they were charging $99 per month. This was to their biggest segment of customers called starter businesses. Think of these as small businesses: most of them recent startups but some of these businesses a little bigger. Usually just a handful of employees. These are companies that have a lot of recruiting needs but not a lot of bandwidth to do physical -- to hire an HR director, let alone to hire a team of people to interview folks to do basic jobs for them. So, they were charging $99 a month and of course it didn't take very long in the conversation to ask the obvious question, "Why do you charge $99 a month?" And like most companies I engage with, ZipRecruiter didn't have a logical reason for 99. Instead, what they found was that $99 seemed to attract a lot of perspective employers; that they had a lot of growth; they were making a lot of money -- but as you all know, the fact that you're making money doesn't mean that there's no room for improvement because that would neglect the opportunity cost of potentially making more money. And this is where ZipRecruiter was a little different from a lot of startups. Even though they were being very successful on growing profitability, they were keenly aware that they weren't optimizing the front end of their business. Oh sure, they optimized their website and they've optimized that matching algorithm, but they weren't actually optimizing the way they created and generated revenues. And that, my friends, is a very common observation I have with tech companies. A lot of human capital in the backends of the business, supply chain when relevant. but especially in product development; software; engineering; humongous amounts of state-of-the-art tech; and of course, a team of incredibly impressive experts developing that tech. But not even close to that kind of human capital in the management of revenues. So what differentiated ZipRecruiter was seeking expert input on how to do this. Well, let's think about what goes into pricing. And this is something that is also frustrating for a lot of customers of my own when I'm engaging with companies as an expert or as a consultant: It's to discipline. Say, "Let's slow down, let's pump the brakes before we start diving into data methods, et cetera. Why don't we sort out the theory?" What is the objective that we're trying to achieve with the decision? You may think that's obvious but you'd be surprised how often companies say, "Well, isn't my objective just to maximize my market share?" Well, if that was true, pricing would be easy. You just give products away for free and you get maximum rev -- and you get maximum, excuse me, market share. Obviously the fiduciary duty of a company alone or its commitment to stakeholders would dictate that they need to do a little bit more than maximize their market share. So a reasonable objective for a company like ZipRecruiter -- and this was not my opinion, it was their opinion as well -- was to maximize profit profitability. So let's write down the equation for profitability. We're not writing down this equation to make things unnecessarily mathematical. It is only when we stare at this equation for a moment that we'll start to understand what we need to do with pricing and why it's gonna be harder than we might have thought. Well, let's look at this: What is profitability? It's a combination of our margin. This is the difference between the price and the cost, we incur from selling a product. This margin is how much margin or, if you will, profit we contribute -- right? -- from the sale of each unit of our product. Well in the case of ZipRecruiter, this would be the profit we get in a given month from each incremental subscriber that we can compel to register and pay. But of course, as we raise and lower the price that affects that margin: But it also affects the quantity of people who are willing to pay. It will affect how many subscribers we get. Which means on the one hand, high prices are pulling margin up and yet the same time it's gonna reduce the number of people we can objectively convert. So there's a bunch of reasons this equation is useful. First, it teaches us about the obvious price trade-off, but it also immediately tells us what we would need to know and what we would need to be able to measure if we wanted to quantitatively determine the best price possible. And the best price possible would be the price that actually maximizes profitability. So let's look at what data we would need to know. First, we would need to know marginal cost: literally the incremental cost associated with each unit that we could supply. I could now have a 90-minute discussion only on the topic of what is a marginal cost. Because I'm not gonna get into it this morning, but companies routinely flub and overload costs into this decision that are totally irrelevant to pricing. And the result is, even when they try to optimize they almost surely end up massively overpricing, OK? But for the time being I'm just gonna say this is the incremental cost of each unit you sell. Over here, this is demand. This is the hard part. This is the relationship between what you charge and what you sell. Now, obviously we have a problem at a company like ZipRecruiter because if they've only charged $99, then they actually have no idea how much more conversion they would get with lower prices; or conversely, how much -- excuse me, how much less conversion they would get if they were to raise their price. And like most companies, they found themselves very reluctant to raise their prices because they were very worried about the risks of lost subscribers. So let's look and see why you have to be careful before you let those risks keep you caught and keep you stuck at the same prices year over year. You'd be amazed how much inertia you observe in the way firms set a price. And often that's because of a perception of a risk without a proper understanding of how to manage that risk. So let's think about how this will pan out. You can see how we're using economics right now -- it's all through pictures, we don't need a lot of fancy math here -- but it's just guiding us and thinking about how to frame this pricing decision. So here's demand: This is theoretical, but imagine this is the demand curve, right? You can ignore this notation over here, I should have deleted that. I'm not gonna explain what that notation means. Just what this demand curve tells us, however, is how much we sell at a given price. If I charge a really high price -- let's call that price pA, right? -- I'm only gonna serve this part of my demand. The rest of this demand curve represents segments of consumers who wouldn't be willing to pay that much. These are people who don't perceive as much value. So charging a high price is great for margin but it may not be great for volume. So how much money do I make when I charge a high price? Literally, it's gonna be this price minus this cost. This height, if you will, is my unit margin, right? The margin on each unit. And the width of this thing is the quantity of people would be willing to pay. And now think about that for a second. The area of this blue rectangle exactly measures how much profit I generate when I charge that high price pA. Now imagine in your company there's somebody who, for totally ad hoc reasons, is advocating for lower prices. If we could just lower our prices, we will sell more. So we don't like the price pA; we think we should charge price pB. OK. Objectively speaking, however, I agree that as we lower the price to pB we will sell more, but we also get less margin on each unit you sell. Think about that for a second. All the people who would've bought at price pA will still be willing to buy the lower price pB, but we're not getting as much money from them. So whether or not it makes sense to charge a lower price pB, as opposed to a higher price pA, depends on whether we're getting enough incremental buyers to compensate for this loss of margin. And similarly, when you get a CEO or somebody at the helm who's an intrinsic empire builder, be careful. Because while they may end up taking decisions, and putting into place decisions, that will go after the mass market customer -- you'll sell more -- but you are leaving now money on the table. So when you start asking questions like, "What is the optimal price?" that optimization problem is basically a trade-off between how many customers you can serve and how much money you make at that price. So the quantity you sell always needs to be weighed against how much margin you're getting at that price. And the fact that your company has very high margins doesn't mean anything to me unto itself if you're not serving a lot of customers. And similarly, serving a ton of people and being the market share leader doesn't really tell me anything, if in the process of being the market share leader you're just not making a lot of money. We only need to turn to like 40 or 50 years of history in the auto industry to see how the market share leadership claim has often been associated with lack of profitability. It wasn't until after the Great Recession where discipline from the fact that the large U.S. auto firms had huge cash injections through stock acquisition by the U.S. taxpayer -- we had public supports and buyouts to help the companies through the crisis -- but they were forced to become disciplined. And by 2012, GM lost its market share leadership. It ceded dominance to Toyota. Toyota for the first time ever became the first non-American manufacturer to be the world global market share leader. What happened in that process? GM became profitable for the first time in decades and Toyota started being faced with losses. They started being faced with quality concerns; you know, all sorts of callbacks on their cars, right? Safety issues at which point market share leadership was ceded to none other than Volkswagen. And as we all know, Volkswagen then started to see its quality erode: falsification of emissions to try and maintain its leadership. As you can see in the auto industry, market share leadership has historically been associated with disappointing performance. So that's exactly speaking to the pricing problem at hand here. Simply pricing to maximize market share is not to be seeing the full picture on value. And I don't mean value to consumers now: I mean value to your stakeholders. So demand measurement is of course gonna be the big obstacle. And in my pricing class, we do spend several weeks going through methods and data to who measure and quantify demand -- not so that the firm can determine what its market share will be at its current price, but rather to figure out what would my market share have been hypothetically had I charged a different price. Because how can I possibly know if my current pricing is even close to optimal if I don't know how much more or less I would've sold had I simply charged a different price. So there are many methods; I'm not gonna go through them this morning. But an obvious method, and it's one that is often overlooked by companies, is simply why not run the price experiment? And that's something I've been doing a lot lately, is running price experiments with companies to help them learn demand and do a better job on their pricing. I'm about to now go back to ZipRecruiter and tell you how we ran an experiment with them. But I also wanna let you know, this is something I'm doing in real time. As we speak, I have a field experiment ongoing right now with Microsoft. We're testing prices on some of their hardware and looking for interdependence between selling certain kinds of products, and then as a result of that, selling other complimentary goods. We're doing razors-and-blades type experiments. And I'm also about to start doing some price experiments with Warby Parker on various kinds of eyewear products, right? So to say that this is an isolated example that I'm about to go through would be, to be, misleading you. I run price experiments routinely with companies but I'll tell you, most companies are reluctant to run a price experiment. I've always found this strange because as you're gonna see in a moment, it can give you high-quality data and it can give it to you really fast. So let's take a look at these price experiments. So what I'm gonna do now, remember, is run an experiment where the goal is to try and figure out whether or not $99 is optimal. Obviously this is more than just an A-B test because there isn't one other price that ZipRecruiter could test. There's a whole continuum of different prices they could have charged. Maybe they should be charging $20 a month. They'll get way more subscribers. They just won't make much money on each individual subscriber. Maybe should be charging more than $99. They may lose some conversion but maybe they'll get way more margin as a result. Bottom line is our method here is to run a B2B price experiment to measure demand at ZipRecruiter and figure out what their best price is. Now here's an, in a moment ... just my screen is not cooperating -- There we go. This is a visual now of the experiment and the results of that experiment. As you can see in the experiment, we tested 10 different prices. The yellow bar here is our control cell, if you will, that's the $99 cell. But the other cells represent 10 different prices we charged. This was a very ambitious experiment. Let me explain what these bars mean. During the month of August of 2015, we had almost 8,000 enterprises hit the paywall at ZipRecruiter for the first time. These are all new businesses, companies that had not previously had an account with ZipRecruiter or paid. So these are first-time buyers. There were 8,000 of them and when filled in your, when you signed up and you hit the paywall -- before you could see a price, right? This is in order to then pay at that price -- the amount that you were shown on the website was chosen at random from these 10 prices. So some businesses were told your account is going to cost, your subscription will cost you $19 per month. Some were chose were shown it was $99 and other businesses were told it would be as much as $399 a month. Imagine an experiment right now for Tropicana Orange Juice where some people walk into the store and they see the usual price -- it's about $5.99 for a 64-ounce container -- imagine some will happen to walk in during the high-price-sell period of the day and they're told it's gonna be $20 for that pack of juice. That's not so different from what this experiment is doing. Now, let's see what we learned from the experiment right out of the blocks, 'cause I've already got the so-called data right here in front of you. The height of each of these bars is indicating to you what percentage of the companies who were randomly assigned to this sell actually paid. So you can see here that when you charge people $19 a month, those businesses on average have a 36 percent chance of converting. What that means is 36 percent of the people who were quoted $19 a month actually paid; they were converted. At the opposite extreme when we charge $399 a month, right? You see way less conversion. It falls to about 11 percent. Now, unfortunately we don't have the ability to do interactive discussion right now 'cause at this point, I would ask you to stare at this picture and tell me what you think about ZipRecruiter's pricing. So I guess I'm gonna have to tell you what I think about ZipRecruiter's pricing when I look at this picture. Here's the first thing that pops to my mind when I look at this picture. OK, I mean you'll have to forgive me here. Ah, there we go. My screen is not behaving. Sure, ZipRecruiter makes money at $99 and sure, they get a relatively high conversion rate that looks like about 24 percent conversion rate or 23 percent conversion as you can see right here. But what I also observe is if you raise your price by a factor of four, you lose less than 50 percent of your conversions. Conversion only falls to about 11 percent. Think about that. I can increase the revenue per paying customer by a factor of four and meanwhile, the defraction of customers who would pay falls by approximately half -- not even half. Hopefully you can see that that mathematically means revenues have to be going up as you raise your prices. And not only, I mean, when you raise your prices a little, it means ZipRecruiter would make more money per month if it's raised its price by a factor of four. So imagine this was Tropicana Juice charging $4.99 right now, I would be telling them, "Folks, you should be charging $20 a pack." You imagine that Tropicana's brand management team would be panicked, say, but if we charge $20 a pack we're not gonna sell nearly as many packs of juice. I go, "That may be true." And so maybe there's some option value from having a high market share. But if our goal here was just to make money you will lose some sales, but you will more than make up for it on margin. Let's try and visualize how much money they make. I'm showing you now that same experimental design; I have the 10 bars for each of the 10 price cells. The yellow cell is indicating what ZipRecruiter's current decision was; they are charging $99 a month. The only thing that's different in this particular slide is that as you can see I've got little wisps; these heights of the bars here are no longer indicating the percentage of people who actually pay. Instead, it's the revenue per lead; this is the average revenue per customer. In other words, let's suppose there's approximately 800 people in each cell. You may be wondering: Wait a minute, how come I'm making about $7 per customer when I'm charging $19? And the answer was, remember, only 36 percent of them actually paid. Most people don't pay and you get zero. Which means on average when you charge $19, you get higher conversion, but your average revenue per paying -- excuse me, average revenue per lead is only about $7 or $8. Even when you charge $99 -- remember conversion was still about 24 percent -- you only make about $24 per customer here. Because again, only a small percentage of them are converted. Now, we saw that conversion does fall as you raise your prices, but you can see that even though conversion is falling, you're just charging enough more per paying customer to more than make up for this. In fact, somewhere in this vicinity is what we'll call the optimum. It turns out on this test grid -- right? -- on this test grid, the optimal price is in fact $399. Think about that for a second. It says that even though you're losing a lot of conversion, if our business objective -- and again, some of you may have be thinking, maybe I have a different objective, fine: State that objective first. But if the objective was indeed to maximize profits, which is what ZipRecruiter wanted to learn when we did this, right? Then the answer at least according to the experiment is $399 is the profit maximizing price, at least amongst those prices we tested. Obviously there's a lot of prices we didn't test but you can see at least on the test grid, this is the highest profit price. And moreover, it generates -- you ready for this? -- 60 percent higher revenue per customer. Why am I looking at revenue, by the way? Because this is a digitized business, there is no marginal cost. Marginal cost per customers is basically zero. Everything is automated. So revenue per customer is kind of the same thing as profit, variable profit per customer. Now, ZipRecruiter actually didn't end up charging $399. And this is where the economics can only to take you so far, and we now just need to use our managerial instincts, right? Their argument was, "Well, if we look carefully here, well, it's true that $399 is slightly more profitable." We've kind of been ignoring these whiskers, which has to do with statistical uncertainty. Obviously when you run an experiment, nothing is for sure. And these bars represent how much uncertainty there is around the point estimate, if you will. On average, I make $43 per customer but given the statistical uncertainty, there's always statistical error in a field experiment like this. I can't rule out that the profits may be as high as $53 or $54 per customer but they may be as low as only about $36 per customer. And now when I compare these three cells in the experiment, I fail to reject that they are all equally profitable. Well, that's actually not surprising because you would expect demand as we're optimizing to rise as you raise the price, and then eventually to start falling again. So probably what we're observing here is we're somewhere near the peak of the profit function, where it's getting flat. So ZipRecruiter stared at this and said, "For that reason, then we're not gonna charge $399 per month. We're actually are gonna charge $249 per month," which is, granted, much lower than $399. But again, this is where management took over and said, "Fine, but I really can't reject that I'm equally profitable. I'm still making more than 60 percent higher revenue per customer here. I should have added for ZipRecruiter back in 2015." This was a three-old startup, and this translated into almost a million and a half dollars incremental revenues per month, simply by changing the number that they printed on the screen, where it says price. And their view was, this is also less risky. I'm making approximately the same profit as $399. I'm gonna convert more customers. There's obviously some option value with having subscribers in your base and so this just seems safer in some sense. They are using risk management logic. It's not optimal risk management, but it's still approximately right thinking that, well, this is slightly less profitable than here. The fact that there's statistical uncertainty says, well, I'm feeling more comfortable charging at this point because I know that I'm definitely gonna get conversion here, whereas here I'm worried about low conversions. So you can see how we're blending science -- that was the experiment, the economic frame for this problem -- and then the optimization, and then managerial experience. Science doesn't replace the experience and judgment of the manager. On the opposite, what science does is helps the manager make the more informed decision. The experiment doesn't teach us 100 percent of what we know, want to know or need to know, but it sure as heck helped them, ZipRecruiter, see they were leaving a lot of money on the table. So they were under pricing by over 60 percent. Notice how this didn't require big data? It just required us to think logically about what information we needed. And it turned out a simple price experiment was more than enough for ZipRecruiter. Now, I'm not finished with this discussion 'cause I do wanna show you that: just because I didn't need AI, or big data, whatever the flavor of the month is when it comes to jargon; to say just because I didn't need really fancy algorithms for analytics to be useful for ZipRecruiter. 'Cause everything we did right now was the analytics. It just didn't need a lot of computer science. That doesn't mean that ZipRecruiter couldn't do even better. Let's think about this for a moment. This is maybe a better picture of describing the pricing problem we just solved a moment ago. This is demand. Let's imagine this is ZipRecruiter's demand curve and we did indeed see that ZipRecruiter faced the downward sloping demand curve when we looked at the results of that experiment a moment ago. And here's marginal cost. So this is approximately the same picture. We decided that $249 is what ZipRecruiter would charge. And we know then just from our earlier discussion that theoretically speaking, if they charge $249 and we have a good estimate of demand, then we would expect this red box here to represent the profits per month that ZipRecruiter would be generating when it optimized. But now let's think about what optimized meant. We were optimizing under the structure that prices would be uniformly charged: that everyone would be charged the same amount. Maybe we could do better. 'Cause after all, anybody who is up here on the demand curve, these are customers who would've been willing to pay more than $249. And anyone down here on the demand curve, these are people who would've been willing to buy and pay more than unit costs, but they weren't willing to pay $249 so we failed to convert them. Both of these are missed opportunities. This purple area here actually represents the monetary value of money left on the table because there's a surplus we regenerated for consumers by undercharging them. We did this 'cause with $249 was the optimal price, but only in a world where we charge everyone exactly the same amount. And similarly, this bigger purple area is all of the money we left on the table because we failed to serve all these people who weren't willing to pay $249. Now, remember lowering the uniform price would've made us less profitable. So the question mark then is: Is there something I could potentially know about these people before they hit the paywall? Is there some way I could classify these people and predict which customers are up here on the demand curve, and which customers are way down here on the demand curve, so that when they complete their registration -- and that takes them however much time it does for them to set it up, maybe a few minutes. And then I have 20 milliseconds between when they hit OK and then they advance to the paywall page, the paywall page loads for them. And in those 20 milliseconds, the question mark is, Can I take all the information I've learned during the registration page and use it to predict who's a high willingness- to-pay customer and who's a low willingness-to-pay customer? Because if I could, if I wanted to, I could then charge everyone what I predict would be their actual willingness to pay. And my profitability now would be the entire area under the demand curve above unit cost, right? This whole purple area. And depending on how expensive it was for me to go in and implement the human capital to manage this analytics initiative, and programmers to do the classification that I'm about to describe in more detail, and then just collecting and managing the data. So there's an IT component to being able to do all of this, right? If I needed to go to the CFO -- and let's suppose the CFO estimated that it was gonna cost $5 million for our company to implement the capabilities, to be able to do all of this -- well, I could show them my demand and analytics from that experiment I just ran a moment ago and say, "Look, we wouldn't capture this. We would capture also this." These two things, the sum of these two things, are the incremental profits from being able to price discriminate or target prices using our IT capabilities. And therefore those two dark shaded areas represent the return on investment. We already know that simply optimizing took ZipRecruiter a long way. Maybe the incremental benefits from being able to target prices based on what we think are our customers willingness to pay, might be worth the additional IT investment. So what does it mean when I say "classify people"? What it means is all the stuff I learned about somebody at the registration stage, we wanted to determine whether or not it's associated with their willingness to pay. So what do I learn from enterprises as they're signing up for an account at ZipRecruiter? I know their geographic location. So you're in one of the 50 states. You're also the -- ZipRecruiter, they also service Canada, so there's also a bunch of provinces to add into the mix here. So we've got approximately 60 or 65, I think in the end it's 65 geographic locations, right? I know what type of company you are. You'll tell me something about your industry vertical. I know how you compensate your employees; I know how much recruiting needs you have; what kind of benefits you offer employees; et cetera, et cetera. Why would a company voluntarily disclose all of this information? Because that's how ZipRecruiter's platform works. Do you need to tell me all of this so I can find the correct resumes for you. But what I now wanna do is to determine if all of this self-reported information by businesses also helps me predict who's a high willingness-to-pay payer at ZipRecruiter and who's a low willingness-to-pay payer. So in short, I'm trying to figure out: Can ZipRecruiter monetize these data about the customers who are about to be quoted a price? Now, this is not as easy as you might think. Why? Because even with just a short registration stage, each of those questions had a whole bunch of categories of answers -- which means overall, I suddenly end up with thousands of pieces of information. You might say, "Wait, how did I get to thousands?" Well, just on geography alone I already have 60 different variables that describe you. You're either in Illinois or you're not. You're either in, you know, New York state or you're not. You're either in Ontario, Canada, or you're not. That's 60 plus true-false questions. Then I have all the true-false questions about your business type. I have a whole bunch of quantitative information about how much recruiting needs you have. There are thousands of, potentially, things I can potentially learn about you. The problem is, what are my data? The data I have are the experiment I just showed you a second ago. Remember I had 8,000 businesses I've randomly tested. I randomly assign them prices. So I'm gonna use those data to now entrain an algorithm to price discriminate but I have a small problem. I have 8,000 observations in that experiment I just ran but I also have thousands of potentially useful variables that I can include -- and I really don't know which variables are useful or not. This is where machine learning can really be helpful. This is where AI can potentially be really, really useful. What I'm gonna do is use a variation on what's called a Lasso Regression. This is off the shelf; there's nothing very fancy about a lasso. But I'm gonna use this Lasso Regression to help me figure out which subset of these thousands of, potentially, variables are actually useful in predicting who will pay and who won't pay at a given price. In short, what I'm gonna do is take these thousands of potentially useful variables, and I'm gonna use Lasso Regression to whittle it down to approximately 20 or 30 that are actually relevant and useful in determining willingness to pay. I'm not gonna get into the details of this algorithm today. It's not even 9 a.m. Central Time, so this would be cruel and unusual punishment for the non-analytics folks in the room. But that's really not the purpose of today's discussion because most of our Booth graduates are not programming machine learning. They are leading teams of people who have those capabilities to help the organization make better decisions. What our Booth graduates need to understand is why these methodologies are useful to have a working understanding of how they can be implemented so that they can build and lead those teams. If you're interested, by the way, this is based on a research paper I have it's co-authored with one of my colleagues, Sanjog Misra. You can download the paper from my personal website if you'd like to read more of the technical specifications. But let's try and think about why this might be useful for ZipRecruiter. Let's imagine that only 30 off that list -- remember we started with thousands of variables that described customers -- what if only 30 of them are actually relevant? Does that mean that we didn't learn much? Absolutely not. Because even with 30 true-false variables about customers that are relevant, these are 30 true-false variables that are known to be relevant to willingness to pay in demand. I can already construct over 1 billion segments, which means I could easily implement an incredibly granular targeted pricing scheme. So this looks pretty useful to me. So let's test it. Here's the algorithm rhythm I train. So again, I'm using machine learning and algorithm in machine learning to try and help me figure out automatically -- not by hand but through an automated approach -- which variables to include. And now what I'm showing you in this picture is the distribution of the actual prices I would have charged in August of 2015 if instead of randomizing -- which is what we actually did -- instead, suppose we already had our machine learning algorithm in place and I actually targeted each customer in that original experiment; remember there was 8,000 people. Suppose I, instead of charging you a randomized price on that 10-cell grid, I instead used my ML algorithm to predict what you'd be willing to pay and I charge you that amount. This is the distribution. We capped it at 499; maybe we shouldn't have capped it. We would've ended up charging some people, a very small number of them, over a thousand dollars. But what we're looking at here is a histogram of all the prices I would've charged. And you can see here, there's a spike at $499 because we capped it. But what's really interesting is that you can see that over half of the people -- right? -- over half the people are charged way below the price that ZipRecruiter actually implemented. Right? So there's a whole range. Some people would've been charged as low as $119, some as much as $499. What this tells us is according to the machine learning algorithm, at least, at ZipRecruiter, there's a very wide array of companies and in terms of how much value they're perceived -- and therefore how much they're willing to pay. So here's what I predict. I predict that if ZipRecruiter kept charging $99, their average profit per lead would be $25. We already saw a moment ago in our experiment that if we raised it to $249, we could get that up over 60 percent to $40. I now predict with my machine learning algorithm that I could get this up an additional 10 percent: I could get it up to $44. So I predict that not only was there a huge jump in profitability from optimizing but in addition, there's an additional incremental jump in profits from using a more sophisticated pricing structure and targeting differential prices. Now, the problem is that's an algorithm. Maybe my algorithm's a piece of garbage, right? Maybe you'd like to know as my client whether I'd be willing to put my money where my mouth is, right? So what we did then in November -- this was three months after the first experiment -- we ran a second experiment with a whole new batch of customers. There's no overlap. These are a whole new batch of customers and we had over 5,000 of them. And instead of randomly assigning them to 10 cells, we randomly assigned them to one of three cells. In the control cell, we charge everyone $99. That was the price that ZipRecruiter was charging up until we had our meeting for the first time. There's the uniform pricing cell. This is where we charge $249. That was the price ZipRecruiter decided to charge people based on the first experiment. And then there's the targeting cell where we actually charge people whatever our ML algorithm predicts they should be willing to pay. So here's what we predicted. Here's what we learned from the second experiment. Stare at that for a moment. And you can see, as we predicted, our model actually worked extremely well. We got a 65 percent revenue improvement from optimizing the price. And here's the real kicker. We got an 83 percent revenue improvement from actually implementing targeted pricing. And when we count for statistical error, anyhow, we confirm that our algorithm seems to have worked very nicely, right? So let me just summarize here and then I'll start taking some questions from everyone. What I wanted to show you in this very short introduction to the Chicago Booth Approach is why it is that using the Chicago Booth Approach can change how you make a decision. When it comes to pricing, pricing should not be a gut instinct guessing game. The fact that you make money at your current prices shouldn't make you satisfied that you fully monetized your business opportunity. And you can see from the ZipRecruiter example how silly it is to say, just because we made profits last period means we therefore shouldn't try to do better. There could be a lot of money left on the table. And the first way to figure that out is by setting us, just thinking about, well, what should be our objectives? Theoretically speaking, what should our pricing objectives be? And we saw from that what our opportunities might be, right? When we stared at our pricing objectives, we realized, "Hmm, well, it's true that if we lower our prices, we might get more market share, but we also give up margin. And if we raise our prices we might lose market share, we would get more margin." And it also you might learn that you really have no idea just how much opportunity is out there -- which means the next step was maybe we should try and figure out what our demand is. Maybe a customer insight team that besides running these very qualitative surveys -- which by the way, can be extremely useful in and of themselves -- maybe we need a little more quantitative expertise in our customer insights team. And what we really wanna know is how to measure demands. And we looked at one methodology. But at Booth we'll also teach, we'll also take you through a bunch of other courses -- stats courses, economics courses, psychology courses and behavioral science courses -- to learn about what different things affect demand behavior. But also for you to get a toolkit on how to collect data; how to do statistical measurement; most importantly, how to build a body of evidence to support your opinions about what your objectives are and what the optimal decision is. And then finally I showed you that the Chicago Booth Approach is where AI might turn out to be useful. Instead of slapping a machine learning team at the problem -- which would've been massive overkill for just figuring out the optimal uniform price -- instead where we saw the opportunity from using AI was from automating the determination of prices in real time and letting the price determination be in part based on what a customer discloses at the registration stage of the business. And that allowed us to monetize, in addition to monetizing with the price, monetizing some of that customer information. So that brings me to the end of my discussion here. What I'm gonna do is unshare my screen. Kara, I don't know if you wanna unmute and facilitate specific questions, or if you want me just to go through the Q&A list. But you've probably been paying attention to this list, so you might be in a better position to curate for me...

Kara: You got it.

Jean-Pierre: ...what are the most common questions that came up.

Kara: Yeah, absolutely. And they are quite a few. So anyone can feel free to resubmit. And you know, JP, if you wanna glance feel free, but yeah. So the first question we'll start with, from Cindy: "Would you be able to get meaningful results for this kind of price experiment for low-volume, high-margin products? Or would this only be suitable for high volume," excuse me, "high volumes of identical products?"

Jean-Pierre: Well, let's make sure we understand how to interpret that question 'cause that question can mean a lot of different things. But let's suppose your question is literally, "Does it make sense for me to run a price experiment if I have a very small number of customers?" The answer almost surely is no, but it depends on how small you think small is, right? So one of the problems with running an experiment -- and this is something you can judge before or you commit to running the experiment -- is you need statistical power. If you Google that, you can find Wiki or something will define what I mean formally when I say statistical power. But our working definition for right now is, I need to be able to have enough precision in my data that my results aren't just noise. I kind of showed you that in the pictures; remember those whiskers and the bars? Those were the statistical uncertainty. And for a price experiment, yeah, if you only have 10 customers, you're gonna mostly have noise. There would be no point using an experiment. There are other methods you could use instead. At the same time, though, I would also argue you don't need 10 million customers in order to run an experiment to learn about pricing. You might need 10 million people if you're running an ad experiment where the effects are super, super small and very hard to detect. When I was at Yahoo!, we would run experiments where we'd have like sometimes five or six million people in the eight cells of our A-B advertising experiment. But when you're wiggling around prices, this is a much more blunt instrument, and the needs of your sample sizes are much less demanding. Now, Cindy, another interpretation of your question might have been, "Is it necessary to reevaluate your prices at all when you have a small number of customers?" Well, that depends on what your account managers have been doing with those customers. Most sales forces and even most experienced sales reps really don't understand pricing. What sales reps typically wanna do is maximize conversions, 'cause that's fast and easy. Managing of the margin is extremely difficult for them because they themselves probably don't understand what drives willingness to pay. In fact, what many organizations have done -- I've been doing this within a couple of instances -- is trying to take insights like I just showed you and figuring out how to build easy-to-use apps, sometimes these are mobile apps, for their sales reps, so that in real time they can look at the math and get recommendations. So literally I could manually say, I'm currently visiting account X -- I might even input some attributes of that account -- and the app will start giving them guidance on what what your analytics think should be approximately the prices you'd wanna charge. And that can help you waste a lot of time offering discounts to a customer who doesn't really need discounts and focus instead on other value-added services.

Kara: Great. Thank you. There are a couple of versions of this question. I'll give you the most basic and then you can, you know, interpret it and answer however you see fit. So very insightful. Thank you. This is from Sriram: "How would this work for B2B pricing models apart from B2C?" So there's a couple questions about, yeah, B2B versus B2C with this sort of model.

Jean-Pierre: Sure, well, first let's -- let's stop and digest that question for a minute. ZipRecruiter is a B2B company. That was B2B pricing. The customer here is not somebody sitting in their home buying a pack of juice. The customer here is an enterprise: it's an employee of an enterprise who's using a corporate credit card to pay for a month of enterprise services. And what are these enterprise services? They're human-resources related. This is recruiting. This is a digitized headhunting agency. In my books, at least, pricing at a headhunting agency would be considered B2B pricing. In fact, this is not B2C pricing. So I guess what we might, maybe, if there's somebody who wants to follow up on that and maybe, maybe Kara can pay attention to that, you'd have to clarify for me what it is you think when you're asked about B2B, that would be different from what we just did. But I just wanna reiterate that was a B2B price experiment.

Kara: Great, that's really helpful. If I see any others come through, I will review those... "Can you give a brief, give brief information on how you decided what variables are more important than others?"

Jean-Pierre: Well, that's the whole point, I didn't know. Even ZipRecruiter really didn't know. They know they service many different kinds of businesses but it's really hard to tell which kind of businesses are likely to perceive more value from a resume database than others. And that's why we use machine learning. There was a whole bunch of variables that we had that describe these companies; I just didn't know where to start. Sometimes you do know. Some times ex ante to doing, to doing the analytics, you already kind of know who your segments are. So for example, on some kinds of products -- now, this actually is more business to consumer -- you may know which sociodemographics describe your high willingness-to-pay segments and your low ones -- which means you would already know which sociodemographic variables to feed into your algorithm, right? But we didn't know it. ZipRecruiter didn't know either. Basically, our starting point was I have over a thousand things that might describe you, I don't know what's relevant. And that's why we use machine learning. Machine learning was going through, think about it, a thousand factorial regressions we could potentially run. I'm never gonna run those as a human being, right? Even an army of scientists wouldn't run a thousand factorial regressions. We're gonna instead use a computer science algorithm that will go through those a thousand factorial regressions and algorithmically determine which is the best fitting of those, of those regressions. And it takes moments: Literally in moments it comes back to me and says, here's the regression and here's the set of variables you wanna target on.

Kara: Great, thank you. This is a little long so bear with me. But Canav asked, "Would this approach be relevant if there was also a possibility of losing customers, in addition to the possibility of gaining, with decreasing price points" -- sorry, my cadence was strange there -- "at a lower price point may lower, then, perception of premium nature of the product service?" Sorry, if that wasn't very clear.

Jean-Pierre: Well, I think the question was largely about: Well, wait a minute, sure, you can acquire customers. And somehow, and then the question, let's think about why I'm not sure it would be any different for an existing customer. When we failed to acquire a customer, that's really not very different from we lost a customer. Had we charged less than $249, we would've had more paying customers. The point of what I was showing you was we would've also made less money. Not every customer is necessary. It is not necessarily the case that every paying customer you have is actually good for profitability. Because remember in a business that charges a uniform rate to everyone, those last few percentage points of market share can be really expensive. In order to achieve those additional percentage points of market share, you have to give everybody a really low price. So if I raise my price, I think that's per your question, yes, I'm gonna lose some existing customers but everyone I retain is now paying more. And this is a conversation that's often really hard to have with the company, but sometimes you should fire customers. Once you think through the conceptual logic of pricing, firing customers starts to make a lot more sense. And if you're really, really concerned about lost option value when you fire a customer, well then fine. Instead of giving them a price concession, raise your price, think about what other value-added services you could give them instead. Or perhaps maybe you're just failing in communicating the value of your offering to those customers. Maybe low prices is not the right way to deal with them. Raise your prices but do a better job marketing the benefits to them. But everyone will tell me, "That's easy to say, you're a professor. In the real world, you know, customers will threaten to walk away." Yeah, I get it. I've actually, believe it or not, run a couple businesses. And that includes running a restaurant. I for a long time owned a restaurant. It was a little side pet project. It was a very successful restaurant. It was a sushi restaurant, which was an izakaya pub with a chef who now has a new restaurant with a Michelin star, so this was a successful place. I can tell you, I know firsthand that you lose customers when you raise the prices. But that doesn't mean that you charge low prices just to keep everyone happy. Because you have a good product, you should monetize it.

Kara: Yeah. I appreciate that. OK, back to the B2B question -- and I should say business-to-business, I shouldn't use so much vernacular. "How would you do this in an in-person negotiation; i.e., if your sales style is not a paywall?

Jean-Pierre: Yeah, no problem. Well first of all, you may not be able to run an experiment as easily. And that would be a separate discussion, we're not gonna answer in real time right now, what are other methodologies you could use. This wasn't meant to be a statistics discussion today. But there are lots of other methodologies you can use, and in my pricing class, we go through them in the first four weeks of the course, right? So there are many ways. And of course, the very first examples I use in my class are manufacturing. Let's suppose I'm manufacturing an industrial product. It has very high value. There's no way I could run a field experiment to test prices. We have a different approach that we use in the course to think about how to determine the objective value that we create for a customer, and what should be their objective willingness to pay. But the key to that sales rep is, it's not the sales rep who's doing the analytics. The sales rep is just gonna sit down and negotiate, right? But before the sales rep sets foot in that client's office, you need to train them. And you may have trained them how to speak to the clientl you may have trained them how to dress; you may have trained them how to manage the account, to be professional, to be knowledgeable about the product. But you may not have trained them how to think about what prices mean and why lower prices for certain types of clients may not make any sense because the client's getting a lot of value. And so it's about training them about, well, how to determine the right price -- and most importantly, how to communicate that to the client. So for example, yes, I know you want a lower price. But for the record you might actually have some case studies with this sales rep: Here's how much value you get when you pay the price we're asking and you do business with us. And by the way, here's what we think is your next best competitor -- the one you're claiming or threaten to go to if we don't lower our price -- here's how much value you get from them. And what you need to explain to them -- and this is called selling . That's what selling is, and part of selling is selling a premium price. You need to train your agents to sell the premium price. You need to show them, even at that lower price, you're losing all these benefits. You actually get more net monetary value paying a price premium with us. So sure: Go to the custom competitor, they will give you a lower price. There's a reason for that. They have a demonstrably inferior product. And if you've already worked out why that is, then you can show it to them. By the way, you could also do the following. This is a really great case study recommendation for selling. You could have a simulation prepared for them, and you say: OK, let's imagine we manage, we match the competitor's price but we also only give you the benefits that you would get if you went to them. We can actually afford to give you the competitor's version of this product or service. We can even beat their price. But you're losing all this value in the process. Wouldn't you rather pay the price premium and get all of these incremental benefits? So my answer to your question is, learn how to train your sales reps because most sales reps do not know how to sell premium pricing.

Kara: Yeah. Makes me think about what I do in selling the Chicago Booth world to everybody. It's very relatable.

Jean-Pierre: Sometimes. It's funny. I've actually dealt with a lot of consulting businesses over the years, and it's interesting watching consulting firms doll out expert advice that they themselves don't apply when managing their own business. That's unfortunately a reality...

Kara: It is.

Jean-Pierre: ...when you're in the business of selling expertise, it's funny to watch experts not apply those same rules to their own business decisions.

Kara: Yeah, no, it's great. It's really helpful. And a little bit more on competition here, is a question about such. "What is the impact on this theory of competition? Like for someone like ZipRecruiter, there are some competitors, certainly. But that is many at the same reach. However, for something like Tropicana, where the competition is more obvious and right there next to it on the shelf, wouldn't we need to include competitors' pricing in our pricing decision?"

Jean-Pierre: Absolutely. First of all, I would strongly dispute what you said about ZipRecruiter. Back in 2015, they were a startup, right? And remember, the key players in this space are CareerBuilder. You have, now Google's entering this space. I mean, there's some really big, big people out there. You're right, that ZipRecruiter today, its valuation is in the billions. They grew really, really fast. And I suspect optimizing revenues was one of the many ingredients into that recipe for success. But I'd say there was definitely competition there. But you're right: At the time ZipRecruiter did this since most of their testing was at higher prices -- and that what they implemented was a higher price -- if anything, raising their prices if it would generate a competitive reaction, would actually be the opposite. It would be that the competitor would raise their prices, too. But ZipRecruiter would've definitely needed to be cautious if -- let's imagine the data said, cut your price to $20 a month. A massive price cut on an already rapidly growing business -- you're right -- might have created competitive response. So when I did some work for Kraft Heinz a couple of years ago, what we did instead was we used point-of-sale data. We went into retail stores -- well, we didn't do it -- but we took data from inside retail stores. I think Kraft Heinz has subscriptions with both IRI and Nielsen to get those point-of-sale data. And we actually use point-of-sale data to figure out how much demand they have based on the prices they've charged over the course of, let's say, a year in the store. And we definitely, as you suggest, we control for the competitor's price. There is a cross-price elasticity of demand that we think about before we jump onto what's the optimal price for a bottle of ketchup, or what's the optimal price for a pack of Philadelphia cream cheese.

Kara: A couple questions along this theme, too. So I'll read a brief one: "Is there a way to implement personalized pricing and offline retail?"

Jean-Pierre: I mean, offline retail and personalization? Well, sometimes it's gonna be a little more qualitative. Remember you do, especially in a relationship-based business. So if you're running a beauty salon or you're running a clothing store and you have a regular flow of repeat customers, part of what managing relationships means, is learning about what customers value and what they're willing to pay. And it may very well be that you're really good at predicting which clothing items your repeat customers like; you may know which kind of cosmetics and beauty products they like; but have you been tracking what they're willing to pay for of them? For example, if you're running a beauty salon there are gonna be certain very specialty in-salon products that are harder to find elsewhere. And you may find that if you recommend them, people are very insensitive to what you're charging. And part of that comes from the fact that they're not sitting there and Googling Ulta to figure out what they can buy it at ulta.com when they're sitting in the beauty salon. So things like shampoos and conditioners, especially specialty ones, these are things where there might be a very high willingness to pay. And you just sort of get a sense of who are the customers of mine who typically value my expert recommendations. And as long as you're giving them products that will really help them with their hair needs, for example, they may not be upset if you charge them premium prices because they're not just paying the premium price for the shampoo: They're paying for your expertise. And when they go to ulta.com, they're not getting that expertise -- at least not from the ulta.coms that don't have an in-store salon, right? You make get a sense here that I might have navigated the pricing in this space at some point in my career. But yeah, no, this is all... and that qualitative insight is useful. Yeah, maybe you can't write down a demand curve like ZipRecruiter did, but what I'm trying to force you to think about is how to write down that profit equation and to try and think about what is demand, and what do I know about the people who are probably at the high end of demand for these complimentary services I sell. By the way, we can get way more sophisticated on this as you start getting more and more clients coming into your beauty salon who are more than happy to plunk down $40 for very high-end, you know, shampoos that will help them avoid damage to their hairs. These are people getting coloring treatments, you know, people who might pay $300 for a keratin treatment, for example. You know, you might actually start gradually lowering the prices for your salon services. Now, you never wanna go too low 'cause you're supposed to be premium. And somebody already mentioned that there is some signaling, you know, there's a reason you wanna charge price. But maybe the profit-maximizing price for haircuts and hair treatments might be lower than you would've thought because the more people you bring in your salon, the more opportunities there are to sell 'em all of these complimentary goods: the things that you can recommend to them based on your expertise about their hair. That's razors and blades. But look how much progress we just made on pricing beauty salon products and I don't have a data point in front of me. It's about thinking. And I'm obviously improvising here, right? But even spitballing, I'm showing you that basic logic of an economic model -- while thinking about some of the psychology of what drives willingness to pay -- just brought me a step closer towards revamping how you might be pricing in-salon products.

Kara: Yeah, that's helpful. Really interesting. From Lynn Chen: "What are the key points of value selling in a commodity market where price is oftentimes an important, if not the most important, consideration for buyers?"

Jean-Pierre: Yeah, so when you're in a real commodities market -- like if you're selling residential electricity in the spot market, for example -- the models we have right now that we just talked about are basically useless. Why is that? Because demand and willingness to pay is evolving in real time. The supply itself may be evolving in real time. There may be periods of the day or types of weather events that cause you to hit capacity. Or there are other periods or events where there's just massive excessive capacity and just a lot of people who'd be willing to supply what is very undifferentiated. And in this situation we don't use what I've been describing: We instead use a market mechanism. In fact, we have a really good course on mechanism design, it's taught by Eric Budish. You might have heard of his name because he was very involved in the process of figuring out how to restructure automated transaction markets. He was, like, high-frequency trading markets and he was part of the team that led them to introduce -- well, maybe some of your aren't so happy about this -- those five millisecond delays in the execution of trades so that digitized large-scale investment, sorry, brokerage operations, couldn't front-run their own customers. But that's his whole area of expertise, is: Can I build a mechanism? So here's an example of a mechanism you'd all be familiar with. You can use an auction. What's an auction? An auction is essentially price discovery. When I know that if I instead figured out demand, by the time I learn demand for a given moment in time in a residential electricity market, it's irrelevant -- 'cause demand's already changed. So you'd use an auction, right? And an auction mechanism is all about discovering the right prices to charge. So commodities markets are not very well-suited to my class on pricing, but the truth is most of the markets that we encounter out there involve differentiated products and differentiated services. And what's really fun for me is when I get approached by a client who thinks they have a commodity. So two years ago, I worked with a door manufacturer, it was a big door manufacturer. They make these relatively low-cost doors that you buy for a house, right? So this would not be for a mansion, right, but they're making sort of mass-market doors for houses. And they kept talking about how it's commoditized until we started talking about, not the doors themselves, but the process of how somebody buys a door. They have an incredibly sophisticated e-commerce platform, so that even very unsophisticated contractors can easily pre-order it in a digitized way. They offer all sorts of customization benefits, et cetera, you can can potentially opt into. There are lots of things in the service of selling a contractor a door. And in the services they offer -- not just to contractors but to retailers like Lowe's, for example, who often resell to contractors -- they have all of these services that they use that take what might be a commoditized door, but make it very, very differentiated once you consider the process of buying it. So there are lots of things that take products that seem like commodities, and when you think about the process of buying them, suddenly make them incredibly differentiated products and services.

Kara: That's great. I like this question, 'cause it makes me think about kind our, at Booth, a multidisciplinary approach and how even within the marketing group, there's such a varied level of experience and expertise. So, just as a short question: "Can we explore the gap between 'willing to pay' and 'ability to pay' using pricing strategy and also, like, customer psychology?" So Booth, we also have consumer behavior courses and a lot of other ways you can tap into this, but would love your perspective on that.

Jean-Pierre: Absolutely. So yes, of course we have behavioral science classes taught by ... well, we have people who won Nobel Prizes in this domain, right? So we have -- yes, that's not an area that we're in short supply by any means. And actually that's a fun thing about Chicago because we tend to be thought of as this very old-school, hardcore microeconomics and capital markets. But the funny thing is the people who are the largest critics of the rational paradigm are also at Booth, right? So it's actually such a fun place. And by the way, I say it's fun for me as an academic: I just find it so fun to be here. And I've actually collaborated with some of these people. I've worked not with Dick Taylor, but I've with Chris Shi. I've had a lot of fun working with behavioral science folks. But coming back to the question: Willingness to pay and ability to pay are the same thing, right? They are at some point, they're the same thing. The only time they're not the same thing is when you are trying to intuit what someone should objectively be willing to pay. So I might actually have good reason to think, "Hey, if you buy this product, I know for a fact that once you implement it, I'm creating $30 of value for you." I mean, for example, using my software might save you $30 per usage because I save you time, or I help you avoid breakdowns, right? So I might be able to figure out objectively how much value I create for a customer. And I might be able to tabulate and say, "Hey, did you know that by the end of the year, if you use ..." You know, I did some stuff a few years ago for an enterprise software panel that would help manage public utilities. So it's basically a dashboard that you would implement that would monitor the entire water supply for a municipality. And it would predict and use AI to figure out where breakdowns might occur in real time, so you could fix them before they happen. And of course, you could tell a municipality based on your last four years of data, we know that had you had our software in place, we could have saved you $25 million, right? 'Cause you had some really big emergencies, right? And we're gonna charge you now, you know, $900,000 a year for a subscription to the basic version of our enterprise software. And the municipality is gonna say, "No way we're paying $900,000 a year." You have to work with them, right? Because you have to help them understand how quickly this product will pay off, right? And the municipality may actually come back and say, "We're really sorry, we just don't have a capital budget for this," right? "It's not that we don't appreciate this. We went back and tried to put this into our budget for next year and we just don't have the tax resources. In order to pay for this, there's gonna be other things that suffer and..." That's life. You can't service every customer. And it's really frustrating in the space I just described because you may be spending millions of dollars on each customer that you're trying to recruit. You may be sending PhDs in engineering who work on your sales team to go out and personally meet with these municipalities and spend six months learning about their needs to figure out how much value there is, only to be told after six months, "Sorry, we're not interested." So you've just wasted -- maybe hundreds of thousands of dollars, maybe millions of dollars -- working with that municipality without a conversion. Well that's life, there's risk, right? That doesn't mean, by the way, you now cut your price to a hundred thousand, right? You don't give up a whole chunk of margin just because you can recruit them. Because there may be another municipality you were simultaneously managing that would be willing to pay that amount. But again, coming back to ability to pay and willingness to pay, you're absolutely right. It may be objectively worth $900,000 or more than $900,000 but they may simply not have a budget. And that may be part of the answer to the question people had earlier about B2B, is that capital budgets can indeed be an obstacle and institutional buying can be a real obstacle. The person you're talking to in the organization doesn't understand the value: They just have to deal with a budget. A shipping manager doesn't care if you can ship things more quickly. But meanwhile, there's a lawyer in that company who, if you only could've talked to them, would've been delighted to know you could actually ship things twice as fast.

Kara: Yeah. That's really helpful. Thank you. "In your experience, have you come across any situations where sophisticated AI may have yielded wrong predictions or recommendations? Does this happen often? Can this be mitigated?"

Jean-Pierre: Oh yeah. AI sucks. I mean, people massively overestimate what AI can do for them. And part of the reason AI sucks is that we forget that AI still needs to be trained. And I know you, everyone's been reading Science and Nature or Scientific American and we're all excited about the fastly approaching theoretical singularity, where machines become smarter than humans. And maybe things will be different then. But in the meantime, a human being still needs to point the algorithm in the right direction. Now, where do things often go wrong? Where's the starting point of "go wrong"? Most companies hire, what, a data science team because the people they're hiring are willing to work cheap. So somebody with a physics degree, for example, it's very hard to get an academic position in physics, unless you are truly outstanding. So there are a lot of folks with physics PhDs who are extremely good at math, are extremely good at programming, and if you show them a data science problem you need solved, they can probably solve it for you. But what they are not trained to do is to figure out what that data science problem should be. And so these teams get built and they're built on, you know, it's a lot cheaper to hire a PhD in physics than a PhD in economics. You wanna hire somebody with good skills in economics, you're gonna have to pay probably double. But what the economists will be able to do, they may not even be as good at solving the math or writing software. But what they will know is they'll be able to craft the decision objective and they'll be able to point the AI in the right direction. So the answer to your question is, sometimes the AI itself works absolutely fine. The most common mistake is applying AI to solve the wrong problem because nobody in the firm could actually articulate what the question was they were trying to answer. But of course, besides that, yeah, AI often gives terrible predictions. There's a whole bunch of ongoing, really funny examples. And then when I use the word funny, I mean, scary funny, where people have tried to use AI, for example, to figure out what kind of content to put on -- like, this was the .. What was Microsoft, someone maybe can put it into the Q&A for us, but Microsoft's intelligent agent? It was gonna be like Siri on steroids. But at any rate, this thing was trained to have conversations with people and share information with them. And the course of a day started speaking, like, extremely, extremely right-leaning racist, right? Started sharing propaganda about white supremacy and whatever. I mean, this was actually one of the most famous failures of AI. The whole point was to create a digitized assistant that would learn what people want. And as it scoured the web, it saw lots of hate -- like a lot of hate and a lot of vitriol being published online -- and then learned very quickly this must be what people want. So it's really horrifying. So yeah, AI always needs a human being. If you want AI to get to work, you need a human being to point it in the right direction. Or another way of saying that is, even the AI needs a human team leader.

Kara: Yeah. I appreciate that. Cortana was the product...

Jean-Pierre: Thank you.

Kara: ...that's responsible, yeah. OK we'll do like one, maybe two more questions?

Jean-Pierre: Sure.

Kara: So: "How do we set a pricing strategy when sales are plummeting on account of macroeconomic environments that reduce demand and how do you prepare for a rebound?" It is quite relevant today.

Jean-Pierre: Absolutely. Well, you can imagine that, that this might have come up once or twice while I'm at Amazon. In fact, I was just involved with the, over the weekend -- this was in Europe; I was in a tribunal in front of the, in Geneva. This was all done by Zoom -- which I gotta tell you, by the way, testifying in front of a tribunal at 5 a.m., that's pretty heavy. I'm looking forward to getting on a plane again, hey, have a good night's sleep and to do any kind of work like this overseas with sun shining. At any rate, that was the basis of the case, was like, was all about how companies respond to or how they responded to the pandemic. And as we all know today, many companies massively overreacted: that part of our supply chain shortages right now are based on companies that literally went into hibernation mode: and hibernation mode meant, "Shut down everything." Not just turn off the lights, but literally stop investing. And, you know, eight months of zero CapEx when it's time to turn the lights on again, "Oh, we're reopening now?" You don't just turn on the lights and all of a sudden you're putting out world, you know, you're putting out 250,000 cars a year. You're not putting out, you know, whatever global demand is in tonnage for steel, right? It takes a long time to warm up the engines and not to mention reactivate your own supply chain, right? So -- but your question is all about forecasting, right? But you know, it's also about risk management. You know, we really don't know what's gonna happen in the next few months. There's things that could be good. Consumers are desperate to get out of their homes and spend money. So we know there's gonna be a surge in travel or, you know, again, subject to all of the caveats that our health keeps improving and that there isn't gonna be an unexpected surprise, like the ones we had last fall or even the spring. But, you know, yeah, we can see lots of ways in which demand is gonna increase, but we're also seeing a lot of inflation. Things are getting more expensive to supply. It's more expensive to serve people food, even in a restaurant or a hotel. It's more expensive to sell someone a car, right? It means there are fewer Uber drivers, which means transportation is getting more expensive, right? And all of these factors could curb some of these forecasts we have about otherwise intrinsically growing consumer demand for travel and leisure, right? For consumption. So yeah, you need a good macro forecasting model. And you know, I think the real answer to your question is nobody has a crystal ball, but there are methods you can use. I'm not gonna answer them here, but there are methods you can use to try and forecast demand. But here's another thing that I find really frustrating, is, that businesses do not store historic data. And as we all know, those who failed to study history are doomed to repeat it, right? Or in simpler terms: History is doomed to repeat itself, right? And this is something I've found very strange: Companies often delete their data after a year or two. I understand that sometimes it's because it's perceived as not having any value. Sometimes it's because there might be something in those data that they don't want someone to be able to come and study and analyze three months, three years later, or 10 years later. But the bottom line is our failure to track history makes us repeat mistakes over and over. And one of those is, how do you respond to a crisis? We learned a lot during the financial crisis. Do you wanna know what we learned? There's actually data and ... There are papers written about this. Companies that increased their advertising during the Great Recession, ended up coming out of the recession stronger than before they went into it and stronger than their competitors. And there's a lot of reasons for this. This isn't just an empirical factoid. This is logical, right? During a recession, everybody's cutting back their investment. They don't wanna spend on marketing. They don't wanna invest in new products. They don't wanna innovate, right? Because they're really worried that, you know, they're gonna be in a liquidity crisis. Meanwhile, your one competitor is scaling up on all this. They're opportunistic. They're realizing, yeah, this is a terrible time but it's also an opportunity because not only is nobody else doing this, right? -- which means my returns to differentiation are gonna be higher than usual. It also means I'm gonna emerge from the crisis with a much more solid base of customers. And that's exactly how we saw: Companies that launch new products in the recession, those new products were more likely to succeed long-term than those same company's new products before the recession. Now of course, there's lots of grains of salt we need to assign before we take these messages too literally and too globally. But the bottom line is, don't be quick to assume that a liquidity crisis, like last year, or like during the financial crisis -- or maybe a liquidity crisis later this year, you know -- if interest rates start going up and capital starts to become more expensive, a necessary means of tackling inflation concerns -- that doesn't necessarily mean you start scaling back. These can be opportunities if an equilibrium, right? All your competitors are gonna be scaling back. You might be saying to yourself, "Hmm. It may be tight to get money, but the ROI is so much higher now than it's ever been in the past that I would be an idiot, I mean, literally a business fool, not to capitalize on this moment.

Kara: Yeah, it's really interesting. I'll give you a last question, relatively easy. You've done a great job of really exemplifying the Chicago Approach. Of course, our analytical, data-driven value of working across disciplines, you know, talking across lines of business when making decisions and negotiating and such. But anything else you would share about why Booth, you feel like, is a great place to get your MBA? And then we'll close it out.

Jean-Pierre: Yeah, I have a really short answer for you. Yesterday, I was in court from 5 a.m. 'til noon. At 1 p.m., I had 10 former MBA students, they're all in their late 30s and early 40s, come over for a big brunch. We were outside drinking wine, barbecuing and went all the way until 11 o'clock last night. It's a great community. It's a great community. We have some really fantastic alumni. I have another bunch of students who just got married and I was very excited to go in Australia two weeks ago. And it's a great group. And what I really find remarkable is how the group stays connected. Our alumni clubs are all over the world and, you know, this sounds like propaganda. It's not propaganda. These students join the alumni clubs, not to just to remain engaged to Booth, but to talk to each other. And what's really cool -- and probably one of the things I find the most exciting, is I'm starting to get a little more involved personally in investing in startup businesses -- is that there is a Booth finance community. I think you'll find this, I should be fair and say you'll find this at other schools as well. But these alumni communities are really amazing and they, you know, if you're having a problem with your career, you can reach out to your alumni network and people will give each other career advice. If you're starting a new business, you'll find friends will be willing to give you feedback on your business plan. In fact, some of them may be willing to give you some angel money, right? They may put you in touch with their friends who are potential private equity investors in a company. You know, this just, it's a great network of really cool people. Whenever I tell our international offices I'm gonna be in city X, you know, I'll be in Amsterdam or I'm gonna be in Beijing or whatever, they will immediately come back and we do an event. Absolutely. And it is really easy on the fly to have a dinner set up in a nice restaurant where you have like 15 or maybe 30, depending on the topic or the setting, alumni who show up. And for some of you, you might be sitting at a table -- I did one in Paris a few years ago -- and you'll be sitting next to the European president of Microsoft, right? So it's a really great way to be in that network, to get to meet people who've been very successful in that work, and your booth credentials will give you access to them.

Kara: Great. That's a fantastic answer. I agree. And we are right on time, too. So I cannot thank you enough, JP, for the session today. It was incredibly insightful. And thank you everyone for joining us. Please stay in touch. I know many of you are maybe in the admissions process as I speak, so do not hesitate to reach out. We're here to help you and guide you through this entire process. And these are gonna be recorded. There are many other master classes that have been previously recorded and we have some coming up. So keep an eye on us, stay engaged with us and again, thanks and have a great day. Thank you so much, JP.

Jean-Pierre: My pleasure.

Kara: Take care.

Jean-Pierre: Take care, everyone.

Kara: Bye.

Jean-Pierre: Good luck with your admissions.