Blog & Insights
What Consumer Complaints Can Teach Us About Better AI Design in Collections
A consumer tries to withdraw $300 from an ATM. The machine dispenses $40 but debits the full $300. He reports the problem and finds himself caught in an automated process that seems more prepared to question his claim than recognize what went wrong.
Another consumer, a 75-year-old retiree, disputes a $3,407 credit card charge for a biomedical product she says she never purchased. Documentation supporting the transaction includes an invoice with a curious detail: It’s invoice number 000001.
Consumer rights attorney Peter Barry shared both stories during a recent insideARM webinar about AI and the consumer experience. He argued that a person looking at the circumstances surrounding the credit card charge would immediately see reasons to ask more questions.
These stories illustrate a key point for organizations using or considering AI.
The technology doesn’t have to fail for the experience to fail
Think about that ATM dispute. The consumer used the dispute process, and the bank received the claim. The workflow did what it was configured to do.
And yet, from the consumer’s perspective, something was clearly broken. Peter described getting caught in an “AI loop” when he tried to resolve the discrepancy.
That observation is worth sitting with.
A workflow can follow its rules and still produce an experience that leaves someone feeling unheard. When that happens, the consumer’s behavior can give you valuable information.
- Are they repeatedly providing the same information?
- Trying multiple channels?
- Asking to speak with someone?
- Reopening an issue that the system considers resolved?
These points of friction are signals that the workflow may need another path.
AI design should account for those signals. What happens when the information coming in doesn’t fit the expected pattern? When should an interaction be flagged as an exception? When does repeating another automated step stop helping?
You don’t necessarily need a human to handle every interaction, but you do need a clear way to recognize when the interaction in front of you needs something different.
The edge case is part of the use case
The disputed $3,407 charge illustrates a related problem: A system can perfectly answer the question it was asked and still miss the bigger issue.
Leonard Yampolsky, CIO at NCB Management, used Peter’s example to distinguish between using AI as an assistant and simply delegating work to it. Routine, genuinely repeatable work may be well suited to greater automation. Outliers need to be identified and elevated for human judgment.
That changes how you think about an AI use case.
When you map the workflow, the straightforward path is only part of the design. You also need to think through what happens when:
- The data is inconsistent.
- A consumer disputes the information you have.
- The circumstances are unusual enough to require context or judgment.
- The technology can’t confidently determine the appropriate next step.
- The consumer keeps trying and failing to resolve the same issue.
Those situations aren’t distractions from the use case; they’re part of the use case.
Testing should not only confirm that AI handles the routine interaction correctly, but also explore the messy situations. Give the system contradictory information. Test unusual scenarios. See whether a dispute goes where it’s supposed to go. Find out what happens when a consumer needs to leave the automated path.
Responsible AI takes more than a well-designed use case.
Sometimes the signal is already there
One of the webinar’s most relevant examples for collections came from two calls Peter had reviewed. The consumer was disabled following an accident, had no income coming in, and was waiting for an injury plan to be processed. Yet the calls continued to focus on making payment arrangements.
Peter posed an interesting question: Could AI actually help recognize when that conversation has stopped being productive?
That flips the usual automation conversation on its head. Instead of asking how AI could automate more of the collection conversation, consider how it could help you recognize what’s happening within it.
A consumer may mention a job loss, a serious illness, a dispute, an identity issue, or a lack of income. They may repeat the same concern several times or make it clear that the options being offered simply won’t work.
What should happen next? The answer will depend on your policies and the situation, but AI can potentially help surface those signals. This can give you the information you need to decide whether the conversation needs to change direction, be escalated, or end.
Treat frustration as feedback
Of course, you can’t design for every scenario before implementation. That’s why the feedback loop after launch matters as much as the testing that comes before it.
Leonard emphasized the need to keep reevaluating system performance after go-live, including what the data trends and the consumer experience are telling you. Complaints are one source of that information; so are escalations, repeat contacts, abandoned interactions, opt-outs, and direct consumer feedback.
Peter offered a simple suggestion: Ask consumers about their experience. He pointed to the short surveys companies use after transactions and suggested collections organizations could similarly use feedback to identify interactions worth reviewing.
Then comes the important part: connect the feedback to the workflow. Where did the problem begin? What information did the system have at that point? Was there a signal it could have recognized? Was an escalation path available? Did different systems have the same information?
One complaint may describe one person’s frustrating day, but 10 complaints describing the same frustration start to look like a design signal.
Build the experience around the person using it
For Angie Erwin, VP of Risk and Compliance at Finvi, putting AI to best use starts with clarity about the problem you’re trying to solve and collaboration among the different people who understand the technology, risk, and consumer experience. Compliance, she argued, should be considered as the system is being built rather than added later in the process.
And ultimately, she brought the discussion back to the experience you’re trying to create.
That’s a useful bar for evaluating AI long after implementation.
When consumers complain, resist the temptation to look only at whether the technology technically did what it was designed to do. Instead, look at:
- What the consumer was trying to accomplish
- Where the experience became difficult
- Information that should have prompted a different response
- Patterns that can help you improve the next interaction
AI can provide you with powerful new ways to automate and scale consumer interactions. It can also give you new ways to learn from those interactions.
A complaint may be telling you exactly where to start.
Want to hear more from the conversation? Watch the full webinar replay →
Is your AI ready for the consumer experience?
Good AI design starts with asking the right questions. Use our Consumer-First AI Readiness Checklist to evaluate your use case, human escalation paths, monitoring, consumer feedback, and other considerations before putting AI into production.
