Did you know? 7 things brokers should know about AI and consistency

Did you know… most “AI-powered” broker tools can give two different answers to the same question?

Try it. Take a public AI chatbot, ask it a mortgage question, wait a minute, ask it again. You’ll often get two different answers. For drafting an email that’s harmless, but if you’re basing regulated advice around this then it’s a liability.

Did you know… general-purpose AI models are probabilistic by design?

When a large language model (LLM) produces an answer, it isn’t retrieving a fixed piece of information. It’s predicting the most likely next word based on statistical patterns. That prediction can shift every time the model is queried, even when nothing about the client has changed.

Did you know… variability in AI output can break your audit trail?

If the AI output your firm captured at the point of advice isn’t the same output the tool produces today, your audit trail isn’t reconstructable. The record exists, but it’s fuzzy. Fuzzy doesn’t belong in a regulated file review, and “the model gave a different answer” won’t be an acceptable defence.

Did you know… this creates a structural problem under Consumer Duty?

Consumer Duty expects two clients with identical circumstances to receive equivalent outcomes. A probabilistic system can’t guarantee that. If your AI tool flags a case as straightforward on Monday and high risk on Wednesday, the consistency Consumer Duty requires and the consistency the tool can deliver are not the same thing.

Did you know… AI outputs can shift even when you haven’t changed anything?

Underlying models get updated and retrained by their providers, often with little notice. The tool your firm signed off in January may behave differently by July. If you don’t know when the model behind your advice journey last changed, your compliance position has moved without you.

Did you know… there is a way to use AI that doesn’t carry this risk?

It’s called deterministic design. A deterministic system is one where the same input produces the same output, every time without any surprises. Where consistency is essential, and in regulated advice it is, AI should be built to behave deterministically.

Did you know… there are six questions every broker should put to their AI provider?

Get clear written answers on:

• Is your AI deterministic or probabilistic?
• If I run the same client case through your system twice, will I get the same output both times – and can you show me?
• How are AI outputs used in regulated advice reproducible months or years after the fact?
• What controls prevent output shift when the underlying model is updated or retrained?
• Can you produce written evidence of output consistency that would satisfy a compliance auditor?
• Where AI is used in a regulated advice journey, how does your system document the basis for each output?

A provider who can answer all six clearly is building for regulated use. A provider who can’t is asking your firm to carry the risk.

Did you know… Mortgage Brain handles this differently?

Because we build and run our own AI, we control how it behaves. Where consistency is required, we design systems that behave deterministically: the same client case run on Monday and on Wednesday produces the same answer. The output is reproducible on demand, the basis for each output is documented, and nothing about the model shifts under your feet between renewals, because in regulated advice, consistency is key.