Source: Pixlr

TODAY'S BRIEF

June 12, 2025

Hi there,

This week, the various ways that banks measure their returns on the huge investment in tech. Several execs tell us how to be model agnostic, following up on our piece on the “age of promiscuity” in the last Brief. Plus five stories to drive the AI conversation.

Reminder: Next Wednesday we’ll publish a special edition of The Brief with takeaways from the inaugural Evident AI Index for Insurance. Want more info about our new sector? Check it out here.

People mentioned in this edition: Lori Beer, Dave McKay, Zachery Anderson, David Weekly, EJ Achtner, Sofia Zanon di Valgiurata and others.

Plus these banks: JPMorganChase, Morgan Stanley, Truist, RBC, NatWest, Deutsche Bank, HSBC, CommBank, BNY and others.

The Brief is 2,217 words, a 7 minute read. Check out this Brief online.

If you were forwarded this newsletter, please subscribe here. Write us at thebrief@evidentinsights.com.

– Alexandra Mousavizadeh & Annabel Ayles

USE CASE CORNER

3 WAYS TO MEASURE AI RETURNS

Banking’s newest obsession is how to measure ROI on AI.

Calculating returns has been anything but uniform so far: Some cite a usage rate for an AI tool, others the promise of efficiency. Banks have their own time horizons for calculating financial gains and their own formulas for how much of a vendor contract should count towards AI spend. Whether annual returns should encompass every use case in production or just the ones launched that year differs from bank to bank.

Yet as banks get serious about deploying AI at scale (see: “Death of the Use Case,” The Brief, May 15), they need to show they can measure ROI consistently – it’s the only way they can prove their massive spend has been worth it and to compare themselves against their peers.

This week, we highlight use cases that illustrate three ways banks are measuring returns that could become kind of standard – with revenue uplift, cost avoidance or savings and time savings.

#1 CODE TIME SAVER

Use Case: DevGen.AI

Line of Business: IT security

Vendor: OpenAI

ROI Type: Time savings

Bank: Morgan Stanley

Why it’s interesting: Outdated programming languages like COBOL and Perl don’t mesh with modern AI models, and rewriting the code to make them AI-ready was eating up developer hours at Morgan Stanley. The bank built a tool to speed that infrastructure modernization up which, in turn, cuts down the time it takes to produce new AI tools.

How it works: The bank’s tool was built on top of OpenAI's models and works in a few ways: For smaller code sections, it can fully translate one coding language into another, and for bigger chunks of code, the tool will say plainly what that code does to make it simpler for developers to understand and rewrite it in an updated language.

ROI: Since it launched the tool in January, the bank says it’s saved 280,000 developer hours and reviewed 9 million lines of code.

#2 FEEDBACK SAVINGS LOOP

Use Case: Truist Client Pulse

Line of Business: Retail banking

Vendor: n/a

ROI Type: Cost avoidance

Bank: Truist

Why it’s interesting: Banks interact with customers in more ways than ever, and Truist built an AI tool to let it know how customers are feeling about their products to make them better.

How it works: The bank built an enterprise data lake – a centralized way to store data – that could pull in customer feedback from call centers, surveys, app reviews and customer complaints. A core team of 10 (with 80 other employees helping part time) then built a tool that analyzes each customer interaction and feeds it into a dashboard that lets bankers see customer sentiment in real time and filter by geography or for specific products. It took between 12 and 18 months to take the idea to production, the bank says.

ROI: Truist expects to save $35 million from reduced complaints and said that building the tool in-house saved $10 million compared to what it would have cost to get a third-party vendor.

#3 EFFICIENTLY USEFUL ADVICE

Use Case: Advisor Assist

Line of Business: Wealth management

Vendor: n/a

ROI Type: Revenue uplift

Bank: RBC

Why it’s interesting: The bank’s wealth managers are using AI to summarize meetings and research investments, freeing wealth advisors up to spend more time with existing clients and onboard more.

How it works: The bank’s AI team, Borealis, built a tool that scans emails, the bank’s CRM system and past notes to make recommendations to advisors about the best way to approach a meeting and frame new investment opportunities for a client. It also summarizes meetings as they happen so the bank’s advisors can quickly take action off the back of the calls.

By the numbers: Around a third of the AI-generated insights have resulted in clients bringing more money into the bank. Over time, the tool is “expected to make 80% of our staff as effective as the top 10%,” RBC’s CEO Dave McKay told investors.

Want to know more about the specific ways banks are rolling out AI? Check out our Use Case Tracker – the inventory of all the AI use cases announced by the world’s largest banks available to members.

FROM THE EVIDENT ROUNDTABLE

SPLIT ROAI DECISION

At the Evident AI Outcomes Roundtable on Tuesday, our panelists all predicted banks would share more about the ROI of AI in the coming year. The audience wasn’t convinced.

We asked the people who joined us for the session to predict how disclosures might evolve in the next 12 months. Their message was clear: Don’t hold your breath.

NOT GREAT EXPECTATIONS

The majority of those polled said banks would only discuss returns when they had big wins to share.

Source: Evident AI Outcomes Roundtable, n = 62 respondents

Want to watch our full, hourlong roundtable on AI outcomes? You can see the full panel and browse the highlights here.

After three years of benchmarking AI maturity in banking, we’re turning our attention to the insurance sector.

For the first time, insurers will be able to see how they compare to peers in terms of AI investment, deployment, and outcomes. Find out more.

TREND LINES (CONT'D)

EVAL-UTION

As we wrote last time in The Brief, we are entering an age of model agnosticism where businesses pick and choose between the latest LLMs getting rolled out by the big players (see: “Banks play model field,” The Brief, May 29). Execs at banks say the opportunity comes with challenges.

One of the biggest is keeping track of all the models coming out and seeing how they best fit in. The work required to evaluate models can strain overloaded tech teams and create bottlenecks for AI development, but it’s not stopping this shift.

At NatWest, for example, going agnostic means being “super-fast at getting new updates and models through governance,” Zachery Anderson, the firm’s chief data and analytics officer, wrote in to us. “Systemic eval is the name of the game to enable it.”

Banks can’t rely on the model makers to tell them what’s good. According to research out this week, Claude 3.7 Sonnet performed worse on financial tasks like detecting fraud or reading a balance sheet than the version of the model that preceded it.

As a result, banks are coming up with their own ways to assess each model to determine how well it’ll work with their tools. JPMorganChase developed a benchmarking test that rates models on their cybersecurity skills, like how often they can accurately flag phishing attempts in emails. Morgan Stanley built a benchmark that showed how well different models could summarize meetings, which the bank used to build its virtual assistant, AI@MS.

Bottom line: As model agnosticism (and the agentic AI it enables) spreads, these evaluations will only become more important. As EJ Achtner, head of applied AI at BMO put it on LinkedIn last week, “No evals, no scaled Product! No scaled Product, no strategic value creation. Simple!”

Have an idea you think we should feature in our Trend Lines section next time? Write us at thebrief@evidentinsights.com

IN THE NEWS

FIVE STORIES DRIVING AI CONVERSATION

Move over, agents, here come “digital employees.” BNY CEO Robin Vince told TIME that the bank recently made its first ever AI “hire.” This autonomous employee comes with its own email address, has a face (via an avatar) and can hop on video calls for what Vince describes as a research role. Does it spell the end for humans? Not yet. This digital employee still has a human manager it’s required to keep updated. Vince says the new tech will “free our people up to go do more things for clients, solve more problems and grow ourselves faster.”

Nvidia’s Jensen Huang made some Euro waves this week. The chipmaker partnered with the U.K.’s Financial Conduct Authority to create a safe environment for banks to experiment with AI products. Then it joined forces with Barclays and Microsoft to launch a London innovation hub. Finally, in Paris, Huang announced a partnership with Mistral to create a new platform for developing AI systems.

TD Bank unveiled a “fully internally built predictive foundation model” called TD AI Prism. The bank says the new model can predict customer needs more accurately because it “processes 100 times more data variables” and will be used to personalize the bank’s marketing this year. The bank says its cloud migration last year enabled the team to build the new model.

CommBank finished its migration to the AWS cloud – which began in July 2024 – ahead of schedule. The cloud push allows the bank to “unleash its full potential with AI and Agentic AI,” the bank’s data platforms lead Terri Sutherland said.

The fastest way to a payday is through AI, PwC’s 2025 Global AI Jobs Barometer shows. Jobs that required AI skills paid 56% more on average, and within financial services, that figure is 60%. The report found that roughly 5% of all finance jobs now require AI skills.

In our upcoming virtual roundtable, we’ll dive into how the insurance sector stacks up on AI maturity, examine which companies are surging ahead and explore how leaders are maximizing the value of their AI. Join the roundtable.

TALENT MATTERS

A FIRST AT NATWEST

NatWest hired Maja Pantic as the bank’s first chief AI researcher. Pantic spent four years at Meta as AI scientific research director and will join the bank to lead research efforts related to “accelerating state-of-the-art AI use cases,” the bank’s announcement said.

Commerzbank’s Oliver Dörler moved up to be the German bank’s chief data & AI officer. He previously served as head of big data & advanced analytics since the beginning of last year.

Eugene Gilerson was promoted to group chief data officer at Deutsche Bank. Gilerson, who is based in the U.S., joined the German bank from Citi and has served as U.S. chief data officer since last September.

Standard Chartered tapped Yusuf Demiral, formerly data analytics group head at HSBC's wealth and personal banking unit, to be the bank’s new global head of wealth & retail banking data, analytics & AI.

David Weekly left Capital One, where he served as VP and headed a 50-person innovation team known as “The Lab,” which prototyped the bank’s first in-house Gen AI applications.

JPMC hired Citi’s Suman D’Souza to be its new EMEA chief data officer. D’Souza reports to Teresa Heitsenrether, who leads the bank’s data and analytics teams.

Sofia Zanon di Valgiurata took over as head of AI enablement for HSBC’s corporate and institutional bank. She’s been with the bank since 2019.

Danske Bank UK is on the hunt for a new head of AI based out of Belfast.

Morgan Stanley’s Budapest office is hiring an associate and a VP for its Gen AI model risk management team.

NOTABLY QUOTABLE

“We focus on not how we embed AI into an existing workflow, but how could we potentially solve that problem differently. But the reality is in the short term, you will have a combination of both.”

– Lori Beer, CIO at JPMorganChase, on a Bloomberg podcast, May 29

CODA

EARTH TO SAM ALTMAN

Many billionaires want to go to space these days. After hearing what Sam Altman had to say last week, we suspect he is already living there.

At San Francisco’s Snowflake Summit, Altman said he was hearing that people’s whole jobs were now to “assign work to a bunch of agents, look at the quality, figure out how it fits together [and] give feedback.”

AI is certainly changing jobs – the growth of agentic systems may even eliminate lots of them if you believe Anthropic CEO Dario Amodei. But it’s laughable to claim that large amounts of work are being delegated to armies of AI agents today when most companies are lucky to piece together a ragtag militia.

Banks are no doubt working towards Altman’s vision: Leaders are investing in infrastructure to enable agentic AI (see: “Banks play model field,” The Brief, May 29) and even piloting small scale solutions for specific use cases (see: “Banks go agentic,” The Brief, April 17). But it will be years before managing teams of agents like junior employees becomes the norm.

Until then, new grads making their way to Wall Street can look up each morning on their way to the office and wave to Sam Altman in the clouds.

WHAT'S ON

Weds 18 June

Evident AI Insurance Index Launch, Everywhere

Weds 18 - Thurs 19 June

Banking Transformation Summit, London

Tues 24 June

The Evident AI Insurance Roundtable, Virtual

Tues 15 - Weds 16 July

Momentum AI, San Jose

THE BRIEF TEAM

Alex Inch | Data Scientist | alex@evidentinsights.com

Gabriel Perez Jaen | Research Manager | gabriel@evidentinsights.com

Matthew Kaminski | Senior Advisor | matt@evidentinsights.com
Kevin McAllister | Senior Editor | kevin@evidentinsights.com

Sam Meeson | AI Research Analyst | sam@evidentinsights.com