Every software demo I see lately includes “AI-powered” something. AI-powered customer service. AI-powered analytics. AI-powered scheduling. AI-powered everything.

Some of it’s genuinely useful. Some of it’s ChatGPT with a thin wrapper and aggressive marketing. And for small businesses trying to evaluate what’s real and what’s worth paying for, it’s gotten incredibly confusing.

Here’s a practical framework for choosing AI vendors when you’re a small business with limited budget and no patience for snake oil.

What “AI-Powered” Actually Means (Or Doesn’t)

Let’s start by cutting through the jargon. When a vendor says their product is AI-powered, it might mean:

They’re using a foundation model API. The product calls GPT-4 or Claude via API to handle certain tasks. This is fine—lots of useful products do this. But it’s not proprietary AI, and you’re often paying a markup on what you could access directly for cheaper.

They’ve built custom models. Less common, but some vendors train their own models for specific tasks. This can provide real value if the model is good and genuinely specialised.

They’ve added basic automation or rules-based logic and called it AI. This is marketing BS. If-then rules and basic automation aren’t AI. They’re useful, but they’re not machine learning.

It’s vaporware. The AI features are promised but not actually built yet, or they’re in “beta” indefinitely. You’re paying for future potential that may never materialise.

Ask explicitly: “What AI model or technology is powering this feature?” If they dodge the question or give vague answers, be sceptical.

Questions to Ask Every AI Vendor

Here are the specific questions that cut through marketing and get to substance:

“What does the AI actually do?” Not in broad terms—specifically. “It uses AI to optimise scheduling” is too vague. “It analyses historical booking patterns to predict busy periods and suggests optimal staff allocation” is specific. Demand specificity.

“What happens when the AI gets it wrong?” Everything fails sometimes. How does the system handle mistakes? Can users override it? Is there a human-in-the-loop option? If the vendor claims their AI never makes mistakes, they’re lying.

“How is the AI trained and what data does it use?” If it’s trained on your data, how is that handled? If it’s a generic model, what’s it trained on? Does it improve over time based on your usage, or is it static?

“Can I see the AI’s confidence level or reasoning?” Black-box AI that just gives answers without explanation is risky. Better systems show confidence scores or explain their reasoning. This matters for trust and debugging.

“What happens to my data?” Is your data used to train their models? Is it shared with third parties? If they’re using API-based models from OpenAI, Google, etc., is your data going to those providers? Privacy matters, especially for customer or financial data.

“How much does the AI component cost separately?” Some vendors bundle AI features into expensive tiers to force upgrades. Understand what you’re paying specifically for AI versus other features.

“What’s the performance baseline?” How much better is the AI version than the non-AI version, or than doing it manually? If they claim “50% time savings,” ask to see the evidence. Real numbers from real customers, not theoretical maximums.

Red Flags to Watch For

These are warning signs that you’re dealing with AI hype rather than AI substance:

Vague claims about “machine learning” without specifics. If they can’t explain what it does concretely, it probably doesn’t do much.

No option to turn it off or override it. Trustworthy AI tools let you disable features or override recommendations. Forced AI suggests the vendor doesn’t trust it either.

Promises of full automation replacing human work entirely. AI augments human work well. It rarely replaces it entirely, especially in small business contexts. Be sceptical of “eliminate your customer service team” claims.

Pricing that jumps dramatically for AI features. If the base product is $50/month and the AI version is $500/month, you’re paying a huge premium. Make sure the value justifies it.

Reluctance to provide trials or demos of AI features specifically. If they’ll demo the core product but the AI parts are “not ready to show yet” or “requires custom setup,” that’s a bad sign.

References that don’t actually use the AI features. Ask to talk to customers using the AI capabilities specifically, not just users of the base product who haven’t enabled AI.

What Good AI Products Look Like for SMBs

In contrast, here are characteristics of AI products that tend to actually deliver value for small businesses:

They solve specific, narrow problems well. AI that does one thing really well beats AI that does everything poorly. Look for focused solutions.

They’re transparent about limitations. Honest vendors will tell you what their AI can’t do and when you should rely on human judgment instead.

They show, not tell. Good vendors demonstrate the AI working on your actual use case during evaluation, not just canned demos.

Pricing is reasonable relative to value. If the AI feature saves you 5 hours a week and costs $100/month, the ROI is clear. If it’s $1,000/month for unclear time savings, it’s questionable.

They improve over time with your data. AI that learns from your specific usage becomes more valuable over time. Static AI that never adapts has limited growth potential.

Integration is straightforward. If it requires massive implementation projects and custom development, it’s probably not right-sized for a small business.

Category-Specific Considerations

Different types of AI tools have different evaluation criteria:

Customer service AI (chatbots, email automation). Test how it handles realistic customer questions. Don’t just look at happy path scenarios—throw it curveballs and see how it fails. Can it escalate to humans gracefully?

Sales and marketing AI (lead scoring, content generation). Ask for evidence of performance—conversion rate improvements, lead quality metrics. And understand how it integrates with your CRM or email platform.

Workflow automation AI. Test whether it actually understands your processes or just applies generic templates. Custom workflows need custom AI, not one-size-fits-all.

Analytics and insights AI. Question whether the insights are genuinely novel or just repackaging what you could see in basic reports. AI-generated dashboards that tell you nothing new aren’t worth paying for.

Accounting and finance AI. Accuracy is critical. Understand error rates. Ask about human review requirements. AI that categorises transactions wrong can create tax and compliance problems.

The Build vs. Buy Decision

Sometimes the right answer isn’t buying off-the-shelf AI software—it’s building something custom for your specific needs.

For straightforward use cases (chatbots that answer FAQs, content generation for social posts, basic data analysis), you might be better off using foundation model APIs directly or working with a developer to build a simple custom solution.

Team400 and similar consultancies can build custom AI tooling tailored to your business for less than multi-year subscriptions to enterprise AI platforms. If your needs are specific and you want control, this can be the smarter path.

Off-the-shelf makes sense when your needs are standard and the vendor has solved your exact problem at scale. Custom makes sense when your workflows are unique or when you want to own the solution long-term.

Practical Evaluation Process

Here’s how to actually evaluate AI vendors systematically:

Define the problem clearly. Before looking at tools, write down exactly what you’re trying to solve. “We spend 10 hours a week manually categorising customer support tickets” is specific. “We want to improve customer service” is too vague.

Shortlist 3-4 vendors. Don’t evaluate 20 options—you’ll get overwhelmed. Pick a few that seem credible based on reviews, demos, and pricing.

Run real-world tests. Use your actual data and workflows during trials. Don’t settle for vendor-created test scenarios. You need to know how it performs on your specific edge cases.

Involve actual users in evaluation. The people who’ll use the tool daily should test it, not just the person making the purchasing decision. They’ll spot usability issues and functional gaps you might miss.

Calculate ROI conservatively. Assume the AI will deliver 50% of what the vendor claims. If it still pays for itself at that performance level, it’s probably a safe bet. If you need 100% of claimed performance to break even, it’s risky.

Check references thoroughly. Talk to at least two current customers in similar industries or business sizes. Ask specifically about the AI features—do they actually use them, do they work, what are the gotchas?

Negotiate before committing. AI features are often negotiable. Ask for discounts, longer trials, or performance guarantees. Vendors with confidence in their AI will offer trials and flexible terms.

After You Buy

Once you’ve chosen a vendor, maximise your chances of success:

Set realistic expectations with your team. AI won’t be perfect. It’ll need human oversight. Make sure people understand that going in.

Monitor performance actively. Track whether the AI is actually delivering the promised benefits. If it’s not, push the vendor for improvement or consider switching.

Provide feedback to improve the system. If the AI learns from usage, make sure you’re correcting its mistakes and reinforcing good outputs. Passive use won’t optimise the system.

Plan for the AI features to fail sometimes. Have backup processes. Don’t create single points of failure where AI breaking means your business stops.

The Bottom Line

AI is genuinely useful for small businesses in specific contexts. But not every AI tool is useful, and definitely not every “AI-powered” product is using real AI effectively.

Be sceptical. Ask hard questions. Demand specifics. Test thoroughly. Calculate ROI conservatively.

Don’t buy AI because you feel like you should be “doing AI.” Buy it because it solves a specific problem better or cheaper than alternatives.

And remember: the best AI tools are often the ones you barely notice. They quietly make something easier or faster without requiring you to become an AI expert.

Choose vendors who deliver that kind of practical value, not the ones with the flashiest AI marketing. Your business will thank you.