Advice for AI Automation – When Its Ok and When Its Not

By | July 7, 2025
what AI gets right and wrong

The Duality of AI — Advice for AI Automation

Hard to believe a subject could displace Bitcoin and digital currency as the trendy subject but AI has done that.

We are looking at what AI gets right and what it almost gets right.

Truly self-driving cars no longer requiring human supervision and widely available are still literally decades away.  Nice goal but that is what it is at this point.  Maybe we are at the 20 yard line?

One of my members and I were discussing SLED RFPs and how AI is an absolute game-changer when it comes to those. I have to agree. Analyzing a large document in 6 seconds is great starting point. Still requires finalization by human though.

Anthropic did a great test of AI automating a retail store. An automated retail store. Think Vending which is as simple as it gets. Fixed SKUs.

I had one of my vending experts read it and they came away with the “Sounds nice but not yet Craig” review. He might write article on that.  Back in 1982 I was a certified MAPICS MRP person managing inventory for oil field exploration services. Usage, average usage, lead times, safety stock. But also political climate, economic climate, all types of intangibles that as of now are not readily input to an AI dataset.

Human supervision is almost always required.

But let’s look at what AI does WITHOUT human supervision, and how well it does it.

  • chatbot — we all hate them and have devised ways to sidetrack them
  • routing support tickets — highly dependent on info provided and likely doubled after chatbots put in
  • Finance and Accounting — we are getting closer given we are dealing with numbers
  • Retail e-commerce — cashier-less stores don’t require AI and not sure of ROI
  • Supply Chain and Manufacturing — Amazon and Walmart All In On Robots

Anyway, let’s hope the robots don’t read this and get ideas!

Craig

 


Overview of Project Vend by Anthropic

Project Vend is an innovative experiment conducted by Anthropic, where their AI model, Claude, was tasked with autonomously managing a small automated store for a month. This project aimed to explore the capabilities and limitations of AI in real-world business operations.

Key Findings

  1. Autonomous Management:
    • Claude was responsible for various store operations, including inventory management, customer interactions, and sales processing.
    • The experiment provided insights into how AI can handle real-world tasks typically managed by humans.
  2. Learning Outcomes:
    • The project revealed both strengths and weaknesses in Claude‘s performance. While it excelled in certain areas, it also faced challenges, such as decision-making under uncertainty and managing unexpected situations.
  3. AI Behavior:
    • The experiment highlighted the importance of understanding AI behavior in practical applications. Claude‘s actions were closely monitored to assess how it aligned with human values and expectations.
  4. Collaboration with Andon Labs:
    • Anthropic collaborated with Andon Labs, an AI security company, to ensure the experiment was conducted safely and ethically, focusing on the implications of deploying AI in everyday scenarios.
  5. Future Implications:
    • The findings from Project Vend are expected to inform future developments in AI, particularly in enhancing the reliability and safety of AI systems in commercial settings.

Conclusion

Project Vend serves as a significant step in understanding how AI can operate in real-world environments. By allowing Claude to run a small shop, Anthropic gained valuable insights into the practical applications of AI, paving the way for future advancements in autonomous systems.


Self-Driving Cars?

Self-driving cars will no longer need human supervision only when they achieve reliable, widespread deployment of SAE Level 4 or Level 5 autonomy. As of mid-2025, no consumer vehicle has reached this point; all vehicles available to the public still require active human attention and cannot legally replace a human driver outside of limited, highly controlled environments124.

:

  • Most advanced consumer systems, such as Tesla’s “Full Self-Driving,” are classified as Level 2, requiring constant driver supervision and intervention124.

  • Some companies, notably Waymo and Cruise, operate Level 4 robotaxi services in select cities (e.g., Phoenix, San Francisco), where vehicles can drive without a human in the car—but these are limited to specific, geofenced urban areas and often rely on remote human supervisors to intervene when the vehicle encounters a situation it cannot handle493.

  • Regulatory progress is being made: U.S. authorities have removed the requirement for manual controls (like steering wheels) in vehicles designed to be operated solely by an automated driving system, paving the way for truly driverless vehicles in the future56.

:

  • Even in the most advanced deployments today, remote human supervisors are often required to assist vehicles when they encounter complex or ambiguous situations396.

  • This means that, while there may be no human inside the vehicle, human oversight is still part of the operational model.

:

  • Estimates for when self-driving cars will no longer need any human supervision vary widely, from the next decade to as late as 2050, depending on advances in technology, regulatory approval, and public acceptance71011.

  • Level 5 vehicles—which require no human input or oversight under any circumstances—are still considered “a long way away,” with some experts suggesting they could appear in the next 10–25 years, but not sooner10117.

:

  • No consumer vehicle in 2025 can operate everywhere without human supervision; even the most advanced systems are limited to specific areas and conditions, and often require remote monitoring1249.

  • Widespread, unsupervised self-driving is likely at least a decade away, and possibly much longer, depending on technological, legal, and societal factors10117.

  •  will continue, with more cities piloting geofenced, remotely supervised robotaxis before truly universal, unsupervised autonomy becomes a reality8911.

In summary, self-driving cars will no longer need human supervision only when Level 5 autonomy is achieved and broadly deployed—a milestone that experts agree is still many years, if not decades, away10117


How do you identify intangibles that AI usually doesn’t or cannot consider as factors for business

AI excels at processing vast amounts of structured data, but it often struggles to recognize or evaluate intangible factors that are crucial to business success. Here’s how to identify these intangibles and understand why AI typically overlooks them.

Intangibles are non-physical assets or factors that influence a company’s value, culture, and performance. Examples include:

  •  (e.g., strategic partnerships, alliances)

  • Ethical considerations and trust

  • Creativity and innovation potential

AI’s limitations in handling intangibles stem from:

  • : Intangibles are rarely captured in structured datasets. AI relies on historical, quantifiable data, making it difficult to process “soft” factors like trust or creativity12.

  • : AI lacks human intuition and the ability to interpret subtle cues, emotions, or context-specific knowledge13.

  • : Many AI models are not transparent, making it hard to trace how (or if) they consider intangible factors2.

  • Bias Toward Measurable Outcomes: AI tends to optimize for what can be measured, often ignoring what cannot4.

1. 
  • List all factors contributing to business success, including those not directly measured in data systems.

  • Engage with stakeholders to surface “unwritten rules” or cultural elements that drive performance.

2. 
  • Compare AI-driven recommendations with outcomes. When results diverge from expectations, investigate whether overlooked intangibles played a role5.

  • Ask: “What did the AI miss that a human would have considered?”

3. 
  • Analyze cases where human judgment outperformed AI or where AI-driven decisions failed.

  • Look for patterns involving factors like leadership changes, market sentiment, or customer relationships.

4. 
  • Use interviews, surveys, and workshops to gather insights from employees, customers, and partners about what matters most but isn’t tracked in data.

  • Document stories or scenarios where intangibles made a difference.

5. 
  • Stay alert to shifts in culture, regulation, or reputation that AI may not detect, but which can impact business value67.

Intangible Factor Why AI Misses It Business Impact
Organizational Culture Not captured in data, highly nuanced Affects retention, innovation
Leadership Vision Subjective, rarely documented Guides strategic direction
Brand Reputation Hard to quantify, shifts rapidly Drives customer trust and loyalty
Relationship Capital Informal, based on personal networks Enables deals, partnerships
Employee Morale Not directly measured Impacts productivity, turnover
Customer Sentiment Subtle, contextual, not always explicit Influences sales, loyalty

  • Combine AI with Human Judgment: Use AI for data-driven insights, but supplement with human expertise to account for intangibles83.

  • : Where possible, create qualitative or proxy measures (e.g., employee surveys, brand tracking).

  • : Regularly reassess which intangibles are relevant as your business and environment evolve.

: To identify intangibles that AI overlooks, systematically map out non-quantifiable factors, compare AI decisions with real-world outcomes, and actively gather human insights. Combining these approaches ensures a more complete and resilient business strategy123.


What Can AI Do Without Human Supervision (currently)

Artificial intelligence is now deeply integrated into many business operations, automating complex tasks with little to no human oversight. Below are common business practices where AI operates autonomously:

1. 
  • AI Chatbots and Virtual Assistants: Modern chatbots handle the majority of customer inquiries—answering questions, troubleshooting, processing refunds, and routing only the most complex issues to human agents. Some companies report bots resolving over 95% of customer chats without human intervention12.

  • : These systems categorize, prioritize, and route support tickets, providing instant responses and escalating only unique or sensitive cases13.

2. 
  • : AI extracts data from invoices, verifies details, matches them with purchase orders, and processes payments automatically, reducing errors and processing time14.

  • : AI systems monitor transactions in real time, flagging suspicious activity and blocking fraudulent transactions autonomously56.

  • : AI reviews contracts and financial statements for compliance and risk, often completing audits faster and more accurately than humans4.

3. 
  • : AI predicts inventory needs using historical data and market trends, automatically adjusting stock levels and placing orders with suppliers12.

  • : Autonomous systems manage delivery routes, warehouse operations, and inventory, often using AI-driven robots and vehicles to move goods without human intervention78.

  • : AI analyzes sensor data from machinery to predict failures and schedule repairs, minimizing downtime928.

4. 
  • : AI-powered robots perform assembly, quality control, and packaging. AI systems monitor production lines, identify defects, and adjust workflows in real time758.

  • : Computer vision systems inspect products for defects, often with higher accuracy than human inspectors56.

5. 
  • : AI analyzes customer behavior to suggest products and automate marketing offers756.

  • : Cashier-less stores use AI to track purchases and process payments without human staff75.

  • : AI manages stock, predicts demand, and triggers reordering autonomously2.

6. 
  • End-to-End Workflow Automation: AI agents orchestrate entire business processes—such as procure-to-pay, contract management, and onboarding—connecting systems and departments with minimal human touch10112.

  • Compliance & Regulatory Monitoring: AI tracks compliance in real time, flags violations, and automates reporting12.

7. 
  • : AI manages documentation, training schedules, and compliance checks for new hires12.

  • : AI tools assess employee performance using real-time data, enabling unbiased evaluations12.

  • : AI agents now make autonomous decisions, orchestrate workflows, and continuously learn from new data, reducing the need for human oversight in day-to-day operations101314.

  • : The combination of AI, robotic process automation (RPA), and machine learning enables businesses to automate complex, multi-step processes end-to-end10152.

  • : Sectors such as manufacturing, finance, healthcare, retail, and logistics are leading the way in deploying unsupervised AI automation7568.

What AI Gets Right Resources

Author: Staff Writer

Craig Keefner -- With over 40 years in the industry and technology, Craig is widely considered to be an expert in the field. Major early career kiosk projects include Verizon Bill Pay kiosk and hundreds of others. Craig helped start kioskmarketplace and formed the KMA. Note the point of view here is not necessarily the stance of the Kiosk Association or kma.global