agentic AI in practice: what does it do vs what the pitch says. I keep reading announcements about AI agents that can complete complex tasks autonomously. I keep having experiences where the agent completes 80% of the task and then does something unpredictable with the last 20%. what’s the actual state of the technology?
the 80/20 problem is the defining limitation of current agentic systems. they perform well on the common cases and fail in ways that are hard to predict on edge cases. the failure modes are also often not flagged as failures. the agent completes the task with confidence and produces something wrong.
the demos are constructed to show the best-case path through a task. real-world tasks have more variability, more edge cases, and more dependencies on external systems that don’t behave as expected. the delta between the demo and production is larger for agentic AI than for any other category I’ve seen.
it’s working on narrow, well-defined tasks with clear success criteria and easy verification. code testing pipelines, data processing, document classification, scheduled reporting. the common thread is that you can tell when it got it wrong without much effort.
we’ve been running agentic AI in production workflows for eight months. the real picture.
working reliably: research synthesis from specified sources, structured data extraction, scheduled content generation for templated formats, first-pass classification tasks.
working unreliably: anything requiring judgment about ambiguous inputs, multi-step tasks where an early error compounds, any task where the success criteria can’t be verified automatically.
the most useful reframe: agents are not autonomous in the way the pitch implies. they’re automated, which is different. automation follows a defined path. autonomy handles novel situations. current agents do the former well and fail unpredictably at the latter.
the deployment pattern that works: humans in the loop at the points where task failure has the highest cost. agents handling volume, humans handling exceptions and verification.
the verification question is the one worth asking before deploying any agent: how will you know when it got it wrong? if the answer is “we’ll check everything anyway,” you haven’t gained much. if the answer is “the output is machine-readable and we can validate it automatically,” you have a real workflow.