Chicago Booth Review Podcast Are AI Interviews Better?

Brian Jabarian: All the candidates when they start talking to an AI, they know that the AI is an AI. So the AI discloses its identity saying, "I'm an artificial..." Et cetera. And it's also disclosed at the job interview will be reviewed by human recruiters. That's super important for compliance.

Hal Weitzman: Have you had an AI job interview? Would you prefer to be interviewed by AI than by a human? And how would you expect AI recruiters to perform compared to humans? Welcome to The Chicago Booth Review Podcast, where we bring you ground-breaking academic research in a clear and straightforward way. I'm Hal Weitzman, and today I'm talking with Chicago Booth's Brian Jabarian about his research on AI interviews. Will they make human-to-human job interviews a thing of the past?

Brian Jabarian, welcome to The Chicago Booth Review Podcast.

Brian Jabarian: Thank you very much for having me, Hal.

Hal Weitzman: Now, listen, you are an economist at Booth's Center for Decision Research and also at the Center for AI. And we're here to talk to you about AI, specifically AI interviews. And the paper that you've written looking at using AI for job interviews, voice AI. What does voice AI mean and what are you trying to do with this research?

Brian Jabarian: Yeah. So in this research, basically we're trying to understand how voice AI, which is a combination of three technologies. One is a large language model, how to process and ask questions. The second technologies like speech-to-text, how to understand what you're saying in a spoken way, transforming this into text. And then the reverse, which is transforming text to speech such that the AI can answer back in a spoken way as well.

Hal Weitzman: Okay, so it's basically as a user, what I'm experiencing is like a phone call with a chat bot.

Brian Jabarian: Well, yes and no in the sense that the chat bots you have been used to over the phone are quite yes-no answers and there is no conversation. Here, the AI has a way to follow-up in a smarter way, ask you questions. In general, when you talk to an automat over phone in general so far...

Hal Weitzman: Oh, it's infuriating.

Brian Jabarian: Yeah, you get mad.

Hal Weitzman: And it never understands what you're... Particularly if you have an accent like me, it never understands what the hell you're trying to say.

Brian Jabarian: Exactly.

Hal Weitzman: Okay, so you are looking at this voice AI, and just so I understand, voice AI, it has all the intonation of regular speech? Because sometimes they can be quite flat.

Brian Jabarian: No, absolutely. It has a voice. It's a female voice here that we are studying because of compliance and it's just easier for people to talk to a female voice. It has all the intonation, all the wow aspects, and it takes time to think sometimes, not too much, which is why we can deploy it at very scale. You have microseconds like a human would think, not more, like we have a conversation now. So that's the beauty of the technology and it has all the accents better than mine, to basically answer the users.

Hal Weitzman: It speaks French better than you?

Brian Jabarian: Yeah. And actually, yes, you can decline it in all type of language, which is our next study in all different types of accents. So you can match the identity of a British person or someone in the Philippines or India. So it's really-

Hal Weitzman: Which is why this is scary, but that's another topic.

Brian Jabarian: Yeah.

Hal Weitzman: So what is the question? What is the thing that animates this research?

Brian Jabarian: So the question we are asking is, if we deploy voice AI or AI voice agents as job interviewers, how will they perform basically? So to answer this question, to explore this question, I've developed a partnership with a global recruiting firm, PSG Global Solutions, which was funded by one of the alumni actually, Vivek Panamanban. And with David Koch who is leading the AI effort there, we've been basically thinking about how David can deploy the AI technology, but also how I can bring this expertise with field experiments to evaluate the impact of this technology in the process. And the key risk for them was the changing radically their business operation, going from talking to a human over years and now having the candidate talking to an AI. So the first risk is like people don't showing up to the interview. So that's what the first risk that we actually tackled.

Hal Weitzman: Okay, will they show up?

Brian Jabarian: Absolutely. And this is why we have this choice branch. So I'm going into the detail of the design letter, but one way to know if your user don't want to speak to an AI, you give them the choice. So that's what we did.

Hal Weitzman: That makes sense, okay.

Brian Jabarian: And turnout's like 78% choice to talk to an AI.

Hal Weitzman: Wow. Okay, well let's get to it in a second. But I'm interested that you mentioned the Philippines, you conducted this with this firm in the Philippines. So why the Philippines?

Brian Jabarian: So I was looking for a firm, first of all, willing to meet my scientific standards, which took me years to find one. So very grateful to the firm there. But also I was looking for a firm which was going to deploy AI in high volume market. So everyone has been talking about AI as the next Einstein guy, which I don't deny, it's totally possible. But right now you have massive potential social benefit and economic returns by deploying AI in frontline jobs. So we are, PSG is basically integrated within a BPO leader, business process outsourcing firm called Teleperformance. And basically PSG is recruiting on behalf of Teleperformance clients, which are basically asking for customer service roles, and the field-

Hal Weitzman: Customer service roles?

Brian Jabarian: Absolutely.

Hal Weitzman: So what kind of thing?

Brian Jabarian: So for instance, in our study here, we have 43 different firms. So it can go from technology as working for Google or working for any type of Fortune 500, but also healthcare and this type of more legal oriented jobs which actually required a human down the line doing the decisions live. So you just don't call to say, Hey, my project doesn't work, what I should do? Sometimes you need for which type of doctor I should talk to. Here you need a human advising, the user calling. So the customer services role are wide and they're very diverse, but most of them are out processed to other countries. And one of the leaders in the world is the Philippines.

Hal Weitzman: The Philippines, okay.

Brian Jabarian: Exactly.

Hal Weitzman: So this is all the kind of the call center type jobs.

Brian Jabarian: Yeah, absolutely.

Hal Weitzman: All right, so tell us kind of your big headline finding.

Brian Jabarian: Yes, absolutely. If I may, before the headlines, the way we thought about how do we measure impact? So, so far the way we've been measuring impact in AI is asking questions, do you use AI? Don't you use AI? Et cetera. And it's kind of great, but survey has sometimes limitation. One of the best way to know if basically there is an impact or not attributed to the technology is randomized control trial or like a natural field experiment, where you randomize who is going to get assigned to an AI versus a human recruiter here.

So that was the big win, that now PSG is using that type of technique to deploy their own AI in the firm. So in our case here, candidates, were randomly assigned to getting a job interview with an AI voice agents or with a human recruiter, or as I was saying, giving them a choice, do you want to be interviewed by an AI or do you want to be interviewed by a human? So that's what the randomization. And before getting the results, we ask the professional recruiters in the firm and we say, do you think AI is going to do a good job in term of picking, leading to more job offer rate, leading to more job starters, leading to more higher retention rate over 30 days, et cetera, et cetera. And on all these questions, they all said absolutely no.

Hal Weitzman: And who's they?

Brian Jabarian: The recruiters, the professional recruiters. So we asked them-

Hal Weitzman: The professional recruiters, right.

Brian Jabarian: Absolutely. And no one believed that basically AI would be able to just do as good as themselves or even better. And that was the big surprise because what we found are AI leading to 12% more job offers, 18 more percent job starters and 17% more retention rates over 30 days basically.

Hal Weitzman: So the outcomes are a lot better.

Brian Jabarian: Absolutely.

Hal Weitzman: Okay. So just tell us briefly, walk us through how you learned this. What was the design of your experiment?

Brian Jabarian: So the way it works is, we are recruiting for a customer service job. So why do we have... And in high volume market, so we have 70,000 applicants. So the big first question is why we just don't screen CV and send a bunch of questions to candidate, and then we use AI as a way to analyze the data. Well, the key reason is that customer service role involve talking to someone. So you want to see how people will perform before giving them the job. So the interview is not just about answering question, it's actually about seeing how people behave in a conversation. It's trying to measure communication soft skills that you can't basically elicit otherwise. In addition to that type of skills, we have two more tests, which are quantitative. One is analytical thinking test after the interview. Well, because one of the key roles of the customer service is fixing problems, that people call you with a problem, you need to fix it.

And the other skill is making sure that people have great English skills. So we have an English test. So this is how the process is structured for making the recruitment. So the way it works, it's that when people are getting invited to the job interview, they're randomized to get the job interview with the AI, human or getting the choice, they get the interview, those who pass the interview then go to the next stage doing these two type of standardized test. And after that the human recruiter takes these two source of information and make the decision about getting a-

Hal Weitzman: So they don't get interviewed by a human at all, if you've been interviewed by an AI or you do? So if I'm part of the sample that's interviewed by the AI and I'm one of the 78% people who say, yes, I'm happy to do that, and I go to the interview-

Brian Jabarian: You are interviewing-

Hal Weitzman: Then I go onto the next step and then everything that's been collected in the test and the interview is forwarded to a-

Brian Jabarian: Yes.

Hal Weitzman: Real human who's going to make a decision.

Brian Jabarian: Yes.

Hal Weitzman: But do I actually get to speak to a real human at all in any next stage?

Brian Jabarian: No. If you have been-

Hal Weitzman: So the whole process is automated except for the actual final decision?

Brian Jabarian: Exactly. So even the scheduling part, which is why actually we have this massive bump within the people who have been randomly assigned to the choice branch. One of the attractiveness of AI is like it's always available. And we are talking about customer service role, so it's a very high turnover market, but also a very low conversion rate in the final. Why? Because people are constantly applying to jobs. So any minute saved here, to the point of the interview, is actually a big win for the firm and for the candidates. So the availability of AI is one of the key reason I believe leading to this type of choice to be interviewed by an AI.

Hal Weitzman: Okay. Now you said that you asked the expert that the recruiters, no surprise, they said the AI couldn't do the job that they've been doing. I mean, what would anyone say? But the results were surprised them, I guess, did it surprise you? What did you expect to find?

Brian Jabarian: Yeah, I was very surprised because so far we've been thinking in the academic community, but also in the technology sector of AI as some technology that provide information to humans. And here we are actually shifting the paradigm and saying, can AI actually collect information from humans, directly from talking to them? So it's a very complex social task. What we have here it's a complex coordination problem, talking, reacting. And so I was not expecting AI to be better than humans in some dimensions. So yeah, very surprised.

Hal Weitzman: The big question is why this is happening. Why do you think AI is better at interviews than people are?

Brian Jabarian: Yeah, so there is one key result I want to unpack because I think it's very important for the readers to know. So when we say that AI leads, AI led interviews leads to 70 more percent of retention rate, what we are seeing here is something that we need to unpack. It means two things. It means extensive margin, so it hire more people so they are more likely to also stay longer on the job. But it can also mean the intensive margin, which means that AI might be actually detecting all the kind of people who are more likely because of the characteristic to stay on the job. So when you condition that on people who accept the job, we actually find that people who have been interviewed by AI, condition accepting the job are 6% more likely to stay on the job after 30 days. And this is this part that tell us that actually AI is doing something better during the interview.

So what is actually doing better during the interview? So the way we have been pursuing this question is in the following way, we have been taking the transcripts and analyzing them through different type of natural language processing techniques, to understand what are the linguistic features that actually matter for a job decision made by the humans. And actually for some of those type of linguistic features that matter for the job decision in a positive way, as in predicting in a good way the job offer, AI is better at picking these type of signals. Which one? The interactivity. Interactivity during a conversation is a good predictor of you as a candidate getting a job offer down the road. Why? Well, because if you show basically interactivity, interactiveness during the interview, it means that you're active and engaged in the conversation. And that's a good thing for customer service job, basically.

Hal Weitzman: I see. But you're showing you're interactive with a bot, not with a real human.

Brian Jabarian: Yes, but who is seeing that you're active with a bot At the end of the day? It's a human who has access, a human recruiter with evaluating your job interview performance based on-

Hal Weitzman: I see.

Brian Jabarian: So it can look at the audio, it can listen to your audio you had with a bot.

Hal Weitzman: Yeah, I'm just wondering because you know are being interviewed by a bot, then you might behave differently than you would than if you were with a customer.

Brian Jabarian: Yes, for sure. So we have this map of different linguistic features like vocabulary, richness, etc. The one that AI is speaking is the one that matter most for the job decisions. But also there are some linguistic features that are negative predictors of a job offer, back channel cues, mm, ah, these kind of things which are not great for customer service, as said by the human recruiters. And actually the AI is good as shifting the job interview from going away from these type of linguistic features and prompting the humans to be more able to show the type of interactivity that matter for the job decision.

Hal Weitzman: Have you ever wondered what goes on inside a black hole, or why time only moves in one direction, or what is really so weird about quantum mechanics? You should listen to Why This Universe. On this podcast you'll hear about the strangest and most interesting ideas in physics, broken down by physicists Dan Hooper and Sharma Waxman. If you want to learn about our universe from the quantum to the cosmic, you won't want to miss Why This Universe, part of the University of Chicago podcast network.

Brian Jabarian, in the first half we talked about your research about AI interviews and how all the good things that can... Well, if you're not a recruiter I guess, all the good stuff that can flow from AI interviews. And you mentioned right at the end there before we went to a break about filler words, the ums and the uhs and how that's usually a bad sign for a recruiter, but that the AI was somehow... I want to know how, able to steer people away from that language towards more productive, positive type of language that recruiters do look for. But just explain and tell us why is that happening?

Brian Jabarian: Yeah, so the why and the dynamic of the conversation itself, we are exploring in another project with a team. But for now what I can say is that what we observe when we compare the two types of interviews, human-human versus human-AI interviews, what we observe is that first of all, the human recruiters tell us, look, we don't like people when they... I mean, they don't say from the data, that if people have too much of those filling words, it's not a good sign and we not make a job offer. And what we see is that during the human-AI interviews, those type of occurrences occur less. So in some way during the conversation, the AI and the humans together are able to shift away from the negative cues and going towards the one which are more positive for the human recruiters basically.

Hal Weitzman: So AI can help you reduce your filler words.

Brian Jabarian: Yes, in some dimensions. For instance, I think it's quite important to say that some of those cues that are still positively predicting a job offer, like vocabulary richness, are still better collected by human recruiters. But-

Hal Weitzman: Vocabulary richness?

Brian Jabarian: Yes.

Hal Weitzman: Okay.

Brian Jabarian: But interactivity-

Hal Weitzman: That's interesting, because I would have thought that an AI could pick up on that.

Brian Jabarian: Yes, but actually this is not for now what we see. I mean just FYI, this is a live paper, this is a working paper. We are just starting the analysis, so we might update the words, but in the future. But right now it's the worst case scenario where AI is great at shifting from the negative cues to more the positive one, basically. The other thing which is great during a conversation between a human and an AI is that... Well, the human recruiters are making the decisions. Just by the way, I want to stress that all the candidates, when they start talking to an AI, they know that the AI is an AI. So the AI discloses identity saying I'm an artificial, etc. And it's also disclosed that the job interview will be reviewed by human recruiter. That's super important for compliance.

Okay, let's back to what's happening good in the job interview. The human recruiters, they need information to make the decisions. So they need the job interviews to be comprehensive. Comprehensive meaning covering a list of topics that the firm has instructed the human recruiters and the AI recruiters to cover. Like, we live far away from the job, can you take it? One of the key things that they're trying to assess from the job interview as well, in addition to the communication skills is the attrition risk. Are you going to take the job? Are you going to stay the job? Because of the high turnover, anyone you can save 30 days is really amazing down the line. The AI constantly leads to more comprehensive job interviews, which themselves are a good positive predictor of getting a job offer.

Hal Weitzman: Okay, and comprehensive, you mean longer?

Brian Jabarian: Covering more topics. At least eight of the 14 topics, and actually you have 50 more percent of chance to covering 10 topics with AI compared to 25% of chance of covering topics in the human interviews. What does that mean?

Hal Weitzman: Is that just because they're constantly feeling rushed?

Brian Jabarian: No, who?

Hal Weitzman: The recruiter, the human recruiters.

Brian Jabarian: Well, I think it's not necessarily feeling rushed because the duration of the both interview are the most same, but there is more likely... That's why we want to explore this in a different project because it's already big enough here, but there is different framework of a conversation. Right now we are co-constructing the framework of the conversation. And in some of the job interviews, even if I were the candidate or you were the candidate, I will try to adapt myself more. The AI here actually you enter in some sense in its framework of the conversation. So it's good as saying, hey, by the way, I would like to come back to this question and it doesn't drift.

Hal Weitzman: Right, it's focused, I suppose.

Brian Jabarian: Yeah.

Hal Weitzman: It's not doing its email in the background while you're speaking to it.

Brian Jabarian: Or it's not letting you speak for potentially-

Hal Weitzman: These were phone conversations-

Brian Jabarian: Yes.

Hal Weitzman: We said that earlier. So it's an interesting dynamic, right? It's a different sort of dynamic. Because I was asking you about chemistry, a lot of interviews, I mean interviews are so artificial anyway, but one of the main things you get out an interview is chemistry and a sense of the person, their character, their warmth, their trustworthiness, whatever. That might be hard if you're just taking what they're saying and putting it down in text and feeding it to a bot.

Brian Jabarian: Yeah, absolutely. So that's also why I stress that five-year partnership, because we have several projects. I set up a team now of collaborators with me, so it's not alone. And in one of the other project basically we're trying to investigate this, what would be the optimal hiring system concerning the type of signal that you care about as a recruiter. If you care about the type of signal you mentioned here, well, maybe the optimal setting here would be for some part of the interview you talk to an AI and then you switch back to a human for the human values and the values of the company. Maybe to show a good face to the candidate you want first to talk to a human and then switch them for-

Hal Weitzman: Well, I'm also interested in this statistic, about 78% of people offered the chance chose to be interviewed by the AI. Because there's some other research about humans interacting with AI in terms of they're more likely to be persuaded by AI. AI can show them evidence. And I think one of the explanations there might be that it's sort of somehow objective, the fact that it's not embodied in a person.

Brian Jabarian: Interesting.

Hal Weitzman: Sort of makes people somehow more open to it. People are using a lot of AI therapy. They'll tell... Like you say, they're available 24 hours and you can do a five-minute therapy session instead of an hour. So the relationship that people have with a chat bot is different than the relationship we have with other people, which might mean it's less of a test. You screw up the AI interview, okay so who cares? And you can say pretty much anything. So I just wonder if part of the challenge here is that this is not actually replicating the environment you're going to be working in. That's what I was trying to get at earlier. So I'm wondering if AI is really a good test for whether you're a good phone service agent.

Brian Jabarian: Interesting. That's a very nice point actually. So I can say a few things about this. First of all, we don't observe any biases against women or by the human-led interviews or the decisions, and yet the reported gender discrimination is half for people who got job interview with an AI done with a human. So what you're saying about potentially, we don't have psychological reasons here, but at least for sure there is something happening in the minds-

Hal Weitzman: So we perceive AI that suggests to be more objective.

Brian Jabarian: Or at least less bias against gender. And women for instance, within the 78% when you decompose that, you see that 76% of men choose the bot, but actually it's 80% of women. And the difference between the two group is statistically significant, saying women are more likely to choose than men to be interviewed by an AI. That says a lot about how people perceive the job interview, how it can be threatening for minorities potentially. The other thing I want to say is that we were using a lot the word bot, but actually-

Hal Weitzman: Well, I am. I know you're not.

Brian Jabarian: I am not. Because the bot I was used to when I was calling firms for whatever, taking a train or whatever, it's really a bot. Here, if the AI doesn't tell you that it's an AI, you would forget that it's an AI. It's so good.

So there is actually more potential for AI to emulate what would be your actual job environment than if you were with a human recruiter. Because right now what we're thinking with David Koch for instance, is like rethinking, revamping basically their pipeline of recruiting. What if now, instead of asking question and having conversation, you could simulate what would be your job during the job interview. So now you have a pure signal collection, you want to see how Brian is good as a customer service, give him the job right now and see how it works. So you could emulate problems instead of having back and forth, something you can't really do with recruiters.

Like we are talking about human recruiters. We don't have a lot in our sample, 135, I mean I think 45 are covering 90% of the job interviews they're doing. So it's like a massive amount of effort for a human to repeat. So the likelihood of being able to personalize a job interview with a human is quite low compared to what you could do with an AI. As I was saying at the beginning, the accents, the voice, whatever, but also the way you are actually designing the interview itself.

Hal Weitzman: Okay. And presumably, like you said it could be convenient, you could interview anytime of day and night and as long as you want.

Brian Jabarian: Exactly. And as we were saying, the more a candidate talk, of course it has to be on the point and not drifting, but the more it provide relevant answers, the bigger the information source for the human recruiters to make this decision is bigger. So it helps us with making more informed decision. I want to stress something, in this high volume market, anything we can do to increase the standardization of the job decision making would be highly beneficial for this industry. Imagine you can shift to 1% on the right of the talent distribution, the type of people you recruit. We're talking about millions of job candidate per year for tele-performance, right? So 1% of improvement on the right side of the curve, it's amazing for everyone, for the first-

Hal Weitzman: No I understand the significance, but it does seem to be... I mean, is this limited to a certain type of job? We're not going to hire faculty at the University of Chicago using an AI interviewer.

Brian Jabarian: I mean, well, so... Absolutely right for now.

Hal Weitzman: Or whoever. I mean, you're not going to hire an artist in residence by having them interview with an AI.

Brian Jabarian: Well, we should not probably, we will not, I don't know. What I can tell you right now is that this study shows that it can be seen at the worst-case scenario and already doing better. For sure, the job interview here is structured and to some extent unstructured. So what I mean by structured means you have very clear expected answers. Do you live far away? Yes or no. Are you willing to go back to school or not? Yes or no. But you have 30% at least around of open-ended question, why do you want to get the job? So of course, the more you go up the job ladder and the more you go to more sophisticated conversations, and the less likely right now the AI would be good. But it's not saying that tomorrow it won't be that good.

Hal Weitzman: So we're all going to be interviewed by AI.

Brian Jabarian: I mean, I hope that when I go to-

Hal Weitzman: Then of course we're going to have our own agents that will just do the interviews with the AI.

Brian Jabarian: Yeah. So that's actually a threat that the company is taking very seriously about AI fraud, and there will be new type of AI detection. So it's a bit the funny thing where you have an AI agent potentially talking to an AI recruiter and then an AI system tracking whether you have an AI or not in the loop on which side. But this is something that is going to be very important for actually higher type of jobs like PG level or engineers, which are actually already being held by AI live. Like in my glasses, I could have an AI right now. I don't but-

Hal Weitzman: Scary prospect. Brian Jabarian, thank you very much for coming on the Chicago Booth Review podcast.

Brian Jabarian: Thanks a lot for having me.

Hal Weitzman: That's it for this episode of the Chicago Booth Review Podcast, part of the University of Chicago Podcast Network. For more research, analysis, and insights, visit our website at chicagobooth.edu/review. When you're there, sign up for our weekly newsletter so you never miss the latest in business-focused academic research.

This episode was produced by Josh Stunkel. If you enjoyed it, please subscribe, and please do leave us a five-star review. Until next time, I'm Hal Weitzman. Thanks for listening.