Loan fraud: the true cost of a DIY AI project

Blog
November 23, 2021
By Francois Saulnier

You have started thinking about launching an AI based anti-fraud program because you want to reduce residual fraud. You know that you are losing money every day!

Your initial thinking is that it should be easy enough with the data that you have and a few data scientists. However, you quickly begin to realize that the number of key stakeholders needed is much larger and collaboration much wider than you initially thought.

However, you quickly realize that the scope and collaboration needed to deliver this project is immense - requiring not only your data science team, but also time and resources from your risk team, compliance team, marketing team, and IT team.

In this post, I will describe the multidisciplinary team and timeline required for such a project. 

Business Case

As with any project, it begins with a kickoff meeting including the key stakeholders: Risk& Compliance (primary stakeholders), Marketing, IT and your data science team (depending on your org, they are either attached to the business team or the IT team).

The path to success for this kind of project involves the perfect alignment of stakeholders, and you need to ensure that they all understand the respective timelines, goals, and business constraints.

Next you will hold a series of meetings to frame the project. Everyone in the company is busy with various constraints, thus this phase of the project might end up lasting around 3 months. With half a dozen 2-hour long meetings involving 3-5people, emails exchanged, and of course some research on the topic is needed too - all just to evaluate the potential implications, estimated the ROI (return on investment), and finally the required budget for the project.

After this initial 3-month framing period, you invest 30 M-days (Man days) to build a preliminary business case for budget approval – all the while fraudsters are stealing your money and you still do not have proof of potential savings.

Finally, you present your budget so that you can launch the project - you have made some assumptions and split the project into 3 phases: validation with historical data, build, and the run phase.  

Phase 1: Validation with historical data

As you are experienced with IT projects, the validation with historical data includes a few workshops, meetings, and committees with your data science team, IT teams, risk& compliance team, and the marketing team.

Great, we’re moving, but all of them still need to learn the business complexity and constraints. In your mind, the main party involved in the project is your data science team.Can they do it in 3 months with just a part-time senior and full-time junior data scientist? Have you thought about the tools that they need? Do they need support from data engineers for the data preparation? Are your data scientists experts in fraud detection models?

By the end of this 3-monthlong validation period, you have invested 200 M-days, dealt with a few unexpected surprises, and you finally have the model designed for offline analysis. Let me do the math for you: you spent 230M-days at 450€/day. Add in the cost of the extra resources needed: servers, licenses(5k€/month) … You are playing poker and have just spent roughly 120 000 € just to stay in the game.

After6 months, you have finally reached a key milestone. You now know whether this will be a “go or no go”! Next up is the building phase where you need to actually industrialize the entire process.

Phase 2: Build

There are two options:you can either rely on a Data Science platform that you must accelerate yourself or you can build your own technology - but you still need to think about how you will connect your data stream and what the AI model lifecycle will be.

Let’s assume that you decide to build on top of an existing solution. Your team will spenda couple of weeks on this particular model – first on building, training, and automation,and then it could take 6 months for the data integration. All of which should alignwith the crazy company roadmap. At the end of this period, you end up  investing 500-man days including management committees…

Have you thought about running/maintaining your models? Your IT teams are used to SLAs, and software monitoring, but has anyone considered the model monitoring yet? This is key as the model needs to be monitored in order to be efficient. This requires specific knowledge and expertise to understand the impact of feature drift and deciding when the right time is to retrain the model.

After 9 months, you are ready to go live. Crazy how time flies, isn’t it ?

Not only did the fraudsters not stop during this period, but they actually improved, evolved, and changed their strategies - and you have now pushed a model that was maybe accurate6 months ago. Let me do the math for you again. Your total investment is now 730M-days and don’t forget the cost of the extra resources (servers, licenses, etc.), you have spent around 360,000€ and lost hundreds of thousands of euros to fraudsters in the meantime.

Phase 3: Run

Now that you are in the run phase of your project your costs should be lower, but you still need to maintain the whole application. In addition to your usual IT costs, you also need to consider the cost of model monitoring, model retraining, model updates, and data evolution - because fraudsters also adapt and unfortunately, they adapt even faster than you.

In this run phase, you need 25 M-days per month, add the extra resources needed on top of it, and you’re at about 180 000€ per year.

Takeaway

Building and running your own AI fraud model by yourself may seem like a good option, but at the same time, it is a massive risk. Louis Colombus (principal at Dassault system) mentioned in state of MLOps 2021, that an astounding 87% of ML projects never land in production.

It’s also critical to be aware that the cost of building your own model is not only comprised of the initial investment, but also the cost of running it, which is much higher than standard apps. Let alone the fact that, in this specific fraud context, you’re losing money every day while your system is not in place. It is a tradeoff between ownership, long-term strategy, as well as immediate ROI and efficiency.

At Bleckwen, we already know how to tackle these challenges. We offer a smooth path from idea to efficient fraud reduction, to live in production- all in under two months.  

Want to know more ?  Contact us

Envie d'essayer ?