Can a chatbot be an effective search engine? We tried it out.

Artificial intelligence-powered chatbots have certainly made a splash. They can take law exams, write essays and engage in weird, existential and often creepy conversations with users.

But can they actually tell you something that’s useful when you use them to search the web?

That’s a key question as tech titans roll this technology into the tools we use every day.

Marketplace’s Meghan McCarty Carino spoke with Joanna Stern, technology columnist at The Wall Street Journal, about her experience with Bing’s chatbot. Stern had a short conversation with the chatbot, which has not yet been released to the public, to demonstrate how it functions.

The following is an edited transcript of her conversation with McCarty Carino.

Joanna Stern: I’ve got it up on my computer, so I’m ready to be the intermediary between you and the AI.

Meghan McCarty Carino: Wonderful. So I’m currently recording in my little pantry studio, where I have a lot of cans of San Marzano tomatoes, crushed tomatoes, that I bought, you know, by the caseload during lockdown. So my question is, what should I cook with this large amount of canned, crushed San Marzano tomatoes, with no red meat and that includes some green vegetables?

Stern: All right, I’ve typed it in. I have, “What should I cook with a case of San Marzano crushed tomatoes I have left over from lockdown? No red meat and including green vegetables.” So it takes a little bit of time. So it’s doing that search. But then on the other side, it’s taking all of that information it comes up with and it’s trying to digest that and put that into natural language, right? So it’s not just going to be the list of links, but it’s going to be a paragraph. And so here’s what it has back to you. Are you ready?

McCarty Carino: Sure.

Stern: This is a three- or four-paragraph response. “There are many recipes you can make with San Marzano crushed tomatoes. They are sweet, vibrant and have a rich tomato flavor.”

McCarty Carino: Accurate.

Stern: The AI thinks these are delicious. OK. “You can use them to make pasta sauces, soups, stews and more. One popular recipe is marinara sauce. You can make it with onion, garlic, butter, basil, oregano, sugar, salt and pepper. You can serve it with pasta or bread and add some cheese if you’d like. Another recipe is Mediterranean braised green beans with tomatoes. You can make them with fresh green beans, onion, garlic, olive oil, oregano, parsley and lemon juice. You can serve it as a side dish or maybe a main course with rice or bread. What kind of dish are you looking for?” And on my screen, I want to talk through two other things that are interesting to me that happened. First, there are citations next to some of the sentences.

McCarty Carino: Yeah, they basically look like footnotes.

Stern: Yeah, exactly. That’s where the information is coming from, or presumably where the information is coming from. I would say that most people should check at this point because there’s a lot of factual inaccuracies along the way. So that’s one thing that’s interesting and very different from ChatGPT, which is nice, it is going to try to send you to the original source of that information. Then what happens second, which is really no different than any search we’ve really come to know, is that I now have some shopping results for these crushed tomatoes, where it’s suggesting different places I can go buy these. So advertising is also really front and center here, depending on what those queries are. Then below that it is giving me some more prompts about what I might want to ask next.

McCarty Carino: All right. Well, I will take one of those suggestions and say, I’m looking for a stew recipe without garlic.

Stern: It’s come back with three different bullet points of different recipes, grilled New York steaks with San Marzano sauce.

McCarty Carino: That definitely has red meat.

Stern: That definitely has red meat.

McCarty Carino: They forgot about the red meat. So it’s not quite like talking to a person who remembers what you said two seconds ago.

Stern: No. No it did forget here.

McCarty Carino: So this AI is being integrated both into the Bing search engine, which you can sort of access on any website, as we’re doing, and also Microsoft’s web browser Edge, its successor to Internet Explorer. What is kind of the difference between the browser and the search engine versions?

Stern: In the browser, what often happens is, is you’ll be on a webpage and maybe you want to not read that entire webpage, you want to summarize it. Well, that’s something that this new Edge browser can let you do. You could copy the webpage. And what it does is then brings up a panel on the right side that lets you interact with the AI, whether you ask it, “Hey, summarize this webpage and paste some copy in” or “Can you write an email to my boss saying I’m not going to be in the office on Thursday, I have to go pick up my kid from school?” And it gives you options there to actually do that in different tones. And so I think that that utility where we are able to lean on this AI for some of the more arduous tasks in our life, where maybe it can save us some time, is really where this may head for many people. And Microsoft seems to really think of this as a productivity tool that can help us in our jobs. This is a first step for them. They’ve integrated it into the Edge browser, but they’re going to integrate it into Microsoft Office and all those different, the Words and the PowerPoints of the world, to help us and to help users lean on AI for more of that writing and more of that creation. But I think in terms of everyday search, the things that, like the example we went through with the recipe, I think that is a better type of search than hey, I’m looking for this specific piece of information. For instance, if I wanted to know information about the weather or the stock market, this is not the tool. Things where it can give you options as well, like that saves you the time from going to maybe 10 websites.

McCarty Carino: Big, big picture, sort of, do you feel like the hype about this is warranted? Is it too much? Is it not enough? Like what’s your kind of general takeaway about how big of a breakthrough this is?

Stern: This really feels like a moment where you’re kind of wowed by how a computer can do something. What’s obviously very different is how it interacts with us, right? There is a sense of conversation here that has not been present in the other technologies we’ve had. I think a lot of people are like, well, haven’t we had this with Alexa and Siri? And honestly, Siri and Alexa and Google, they’re really good for setting a timer, but they’re not going to tell you some complex thing. Overall, yeah, I think this is a big moment. I think it’s all going to come back to the guardrails. And that’s what I find very worrisome, is in our last decade of tech, we were already struggling with guardrails around social media and device usage and data usage. And now we have a whole other new form of interacting with computers, and I’m just not sure the guardrails are ready or the people who enforce the guardrails are ready.

Related links: More insight from Meghan McCarty Carino

Stern detailed her experience with Microsoft’s AI-powered search engine and browser in a column for The Wall Street Journal. She also wrote about why she considers Microsoft Edge a superior browser to Google Chrome.

If you’re interested in giving the new Bing a try, you can join a waitlist on its website here.