We still don’t know how people are really using AI

The pervasive integration of artificial intelligence into our daily operations necessitates a clear-eyed understanding of its actual utility, offering a critical lens to refine investment and development strategies. This piece from MIT Technology Review explores the nuanced reality that despite the widespread deployment of AI technologies, there remains a significant gap in our understanding of *how* people are genuinely interacting with and benefiting from these tools. The core argument is that current metrics often focus on adoption or raw usage figures, rather than delving into the qualitative impact, user behavior patterns, or the true value extracted by individuals and organizations. For founders and product developers, this insight means that simply building and launching AI features is insufficient; the real competitive edge comes from observing, learning, and iterating based on actual user interaction, not just hypothetical use cases. Consider an indie SaaS founder in Austin, Texas, developing an AI-powered content generation tool for marketers. Instead of merely tracking token usage, they could implement contextual surveys or A/B test prompts to understand which AI outputs genuinely resonate and reduce editing time for users. Similarly, an internal IT team at a mid-sized healthcare provider in Boston might find that their AI-driven patient scheduling assistant, while efficient, is bypassed by administrative staff for complex cases because it lacks intuitive override functions, indicating a need for human-in-the-loop design rather than full automation. A logistics startup in Chicago, aiming to optimize delivery routes with AI, could discover through observation that drivers often ignore AI suggestions for specific neighborhoods due to hyper-local knowledge of traffic patterns or parking restrictions, revealing a need for adaptive learning or dynamic feedback mechanisms within the AI itself. To capitalize on this, operators should shift their focus from mere deployment statistics to deep user behavior analysis and qualitative feedback loops for any AI system they leverage or build. This week, try selecting one AI-powered tool or feature currently in use within your organization or product. For a small, representative sample of users, implement a brief, non-intrusive survey asking about specific instances where the AI either significantly helped them solve a problem, or where it fell short and required manual intervention, alongside an optional open text field for context. Analyze these qualitative responses to identify patterns and actionable insights that go beyond simple usage counts, informing your next refinement.