AI Agent Vendor Evaluation: 20 Questions to Ask First

Choosing an AI agent vendor feels like comparing phone plans written in another language. Every demo looks smooth, every price looks small, and every salesperson promises the system learns your business. The differences that matter show up later: where your data went, what the busy-month bill was, who fixes it at 9 pm, and whether you can leave.

This checklist gives you twenty questions in plain language, grouped so you can run three vendor calls in a week and score them on one sheet. It draws on what small business owners actually worry about. In a 2026 Small Business Majority survey of 222 owners, the top concerns about big-provider AI were accuracy and reliability at 73 percent, data privacy at 72 percent, IP protection at 63 percent, and security risks at 63 percent, all far ahead of cost at 39 percent. Ask in that order.

Key takeaways

• Evaluate three vendors on the same scoring sheet: your current provider, one trade specialist, and one cross-tool automation firm.
• Ask data questions first: storage, retention, model training, and who can see customer records.
• Price the busy month, not the starter tier, including usage, seats, and integration fees.
• Confirm approvals, logging, and who owns what before you sign, not after the pilot.
• Get exit terms in writing: exports, credential revocation, and what stops working if you cancel.
• Run a paid pilot with one measurable goal before committing to a contract.

How to run the evaluation in one week

Keep the process small or it will stall. Monday, write a one-paragraph job description for the agent: which task, which systems, what good looks like in numbers. Our guide to hiring an AI agent walks through that brief. Tuesday through Thursday, hold one 30-minute call per vendor using the same twenty questions below. Friday, score all three and pick a pilot winner.

Score each question from zero to two: zero means no answer or a bad answer, one means an acceptable answer with caveats, two means a clear, written, verifiable answer. A vendor can demo well and still score poorly if answers live only in the salesperson's memory. Ask for the security page, the data processing terms, and the price list as documents, and score what is written over what was said.

Bring one real scenario to every call. For example: a customer texts at 10 pm asking to reschedule, then disputes a charge the next day. Walk through it end to end: what the agent sees, what it may do alone, what waits for approval, where the log lands, and how you undo a mistake. Vendors that handle the scenario concretely usually handle production concretely too.

Data, privacy, and security questions

These come first because they are the hardest to fix later and the most common source of regret.

One, where is our data stored and for how long. Two, does our data train your models or anyone else's, and can we opt out in writing. Three, who inside your company can see our customer records, and is that access logged. Four, what happens to data when we cancel, including backups and retention windows. Five, which compliance posture do you support: data processing terms, access controls, and audit exports a small firm can actually use. The data privacy checklist expands each of these into checks you can verify.

Six, how are agent permissions scoped: dedicated logins, least privilege, and per-system read or write control. Seven, what do approvals look like: which actions can require a click, who configures them, and where the decision log lives. Eight, how do you handle prompt-injection and misuse attempts, such as a customer pasting instructions that tell the agent to ignore its rules. Listen for concrete controls, read-only defaults, allowlists, approval gates, monitoring, rather than assurances that the model is smart.

These questions reflect real owner sentiment. The Intuit QuickBooks 2026 AI Impact Report, based on more than 34,000 survey responses, found privacy and security the top barrier to AI adoption at 36 percent, followed by lack of knowledge at 28 percent and accuracy concerns at 26 percent. A vendor that educates you during evaluation is previewing the support you will get later.

Pricing, model routing, and commercial terms

Nine, what exactly is included in the quoted price: platform fee, usage, seats, integrations, onboarding. Ten, what does a busy month cost at twice our expected volume, with overages itemized. Eleven, which models or engines do you use, can they change without notice, and does a model change alter behavior or price. Twelve, are there pricing traps: per-minute rounding, per-seat minimums, charges per connected app, or fees for logs and exports.

Thirteen, what does support include: hours, channels, response targets, and who fixes a misbehaving agent at night. Fourteen, what is the contract term, what renews automatically, and what does the pilot cost with a defined exit. Goldman Sachs surveyed 1,256 small business owners in March 2026 and found 76 percent use AI but only 14 percent say it is fully embedded in core operations, with 73 percent wanting more training and resources. Training and onboarding support is not a bonus item; for most firms it is the difference between installed and adopted.

Ask for two reference customers near your size and volume, and ask those references what they paid last quarter. Headline prices describe the starter tier. Your decision should rest on the busy-month invoice.

Fit, support, and exit questions

Fifteen, show the same scenario in our tools: our CRM, our calendar, our phone setup. Generic demos hide integration gaps. Sixteen, who configures and maintains the agent month to month, and what do we do ourselves versus what requires a ticket. Seventeen, what do reporting and attribution look like: can we see bookings, response times, and outcomes per workflow without exporting to a spreadsheet. The marketing attribution service shows the level of visibility worth asking for.

Eighteen, who owns the prompts, conversation logs, and integration work if we leave. Nineteen, what is the exact offboarding sequence: credential revocation, schedule shutdown, data export, deletion confirmation. Twenty, what keeps working if we stop paying, and what breaks on day one. Agents hold live credentials to inboxes and calendars, so exit terms matter more here than for ordinary software. Get them in the order form, not just the sales deck.

Finally, ask about mistakes. Who is responsible when the agent sends the wrong message, and what does remediation look like: log access, correction workflow, customer communication help. A vendor with a practiced answer has handled incidents before, which is reassuring, not alarming.

Scoring the answers and picking a pilot

Add up the forty possible points. In practice, scores cluster: a vendor under 24 has gaps you will feel monthly, 24 to 32 is pilotable with written conditions, and above 32 with no zeros in the data section is a strong candidate. Weight the data and exit sections double if you handle health, legal, or financial information, since those trades face stricter duties around customer records.

Then run a paid pilot with one measurable goal over thirty days, for example every after-hours call answered with a booked or triaged outcome, or every new lead touched within five minutes. Define the goal, the tier of approvals from our approval workflows guide, and the stop criteria before the pilot starts. Review weekly against the numbers, not the novelty. If the pilot misses, your runner-up notes from the same scoring sheet make the second attempt fast.

One caution from the broader market: most firms are still early. About 7 in 10 small and mid businesses use AI regularly according to Intuit, yet only about 1 in 10 pay for dedicated AI tools. A careful evaluation now puts you ahead of the businesses improvising later, and a short pilot teaches you more than a quarter of demos.

FAQ

What is the most important question to ask an AI agent vendor?

Ask where your data goes, who can see it, and whether it trains their models. In a 2026 Small Business Majority survey, 72 percent of owners named data privacy a top concern about big-provider AI, ahead of cost. A vendor that cannot answer plainly in writing is not ready for your customer data.

How do I compare AI agent pricing fairly?

Separate the platform fee from usage charges, ask what a busy month costs, and check whether model or minute overages, extra seats, and integrations are billed separately. Ask for two reference customers near your volume and what they actually paid last quarter, not the starter-tier headline.

Should I ask whether I can leave the vendor?

Yes, before you sign. Ask who owns the conversation logs, prompts, and integrations, what export looks like, how long data is kept after cancellation, and whether automations keep running if you stop paying. Exit terms matter more for agents than for ordinary software because agents hold live credentials.

How many vendors should I evaluate?

Three is enough for most small businesses: your current phone or CRM provider if they offer agents, one specialist in your trade, and one automation firm that configures across tools. Score all three on the same sheet, run a paid pilot with the winner, and keep the runner-up notes in case the pilot fails.

Next step

Write the one-paragraph job description for your agent this week, then run three vendors through the twenty questions. If you want help scoring the answers or structuring the pilot, book a call or start with the free six-step AI automation plan.