AI advice helps businesses only when paired with human judgment

Kim Byoung-pil The author is a professor of technology management at KAIST. A friend who is an executive at a small company recently told me something intriguing. His CEO asks AI about virtually every decision and follows its judgment. Whether setting the company's direction or hiring employees, he relies on AI. He posts AI-generated recommendations in the executives' group chat and tells them to plan accordingly. 'It feels like AI is running the company now,' my friend joked. Another friend who works in sales described a different change. He now sends product materials to clients the day before meeting them. Almost invariably, they run the material through AI beforehand. Because AI suggests questions to ask at meetings, clients tend to raise similar points. Preparing has become easier because he can anticipate them. The way he prepares materials has changed, too. Knowing clients will have AI summarize them, he carefully structures documents to produce favorable summaries. To succeed in sales now, he joked, you have to keep 'Mr. AI' happy. Perhaps reign of AI has already begun without our noticing. Philosophers have long worried about 'algocracy,' governance by algorithms. Traditionally, the concern was that those in power would use algorithms to control people's behavior and choices. What is happening now is different. Nobody forces us. We voluntarily turn to AI and delegate judgment to it. This is not control imposed from above but voluntary delegation from below. A recent case in China shows where that can lead. A farmer had long consulted AI about planting schedules and fertilizer. Its advice proved useful, building his trust. Then he sprayed an AI-recommended herbicide across acres of sesame fields. By the next day, not only the weeds but much of the sesame crop had withered. Because the AI had been right so often, he had gradually stopped verifying its advice. How can such failures be prevented? Should we use AI less, or simply guard against excessive dependence? Neither is enough. More important is the ability to distinguish which AI recommendations to accept and which to reject. An experiment by researchers at Harvard and the University of California, Berkeley illustrates the point. Researchers recruited small-business owners in Kenya, including restaurant, poultry farm and internet cafe operators, and gave half of them free access to a GPT-4-based business-advice chatbot. The goal was to determine whether AI improved business performance. The average results were disappointing. Overall, the group given AI access showed no statistically significant increase in sales or profits. But dividing participants by their previous performance revealed a striking difference. Businesses that had already been performing well saw sales and profits improve by more than 15 percent after using AI. Those that had been performing poorly, however, saw results deteriorate by roughly 10 percent. In this experiment, AI helped only those who were already good at running their businesses. The reason was even more interesting. Researchers examined whether the two groups used AI differently. Did the better business operators ask better questions and therefore receive better advice? Neither was true. The questions asked by both groups were similar, as was the advice AI provided. The difference was their ability to filter that advice. Owners whose businesses had previously performed well selected AI suggestions that fit their particular circumstances and acted on them. Poorer performers tended simply to follow AI's generic recommendations. They incurred additional costs without seeing corresponding benefits. In other words, even when people received similar advice from the same technology, outcomes diverged according to the human ability to decide what advice was actually worth following. The secret to using AI effectively, then, was not some special method of prompting it. What separated good outcomes from bad ones was human judgment: the capacity to evaluate, filter and selectively apply what AI suggested. That makes genuine knowledge of one's own work more important, not less, in the age of AI. As machines increasingly provide answers on our behalf, we may be tempted to conclude that there is less reason for us to learn. The opposite is true. The more answers AI gives us, the more important it becomes to possess the knowledge and judgment needed to tell a useful answer from a harmful one. That is precisely why, even as AI grows more capable, we cannot afford to stop learning. This article was originally written in Korean and translated by a bilingual reporter with the help of generative AI tools. It was then edited by a native English-speaking editor. All AI-assisted translations are reviewed and refined by our newsroom.