Measure & track health of your Fraud Management function

If you don't track it, you won't crack it!

Q- Is our fraud management function operating efficiently ?

Q - How is our performance management framework ? Are we using effective metrics ?

Q - What are the KPIs to measure the performance of the overall FM operations and also the individuals ?

Well, I have faced questions similar to above during numerous customer meetings, conference calls and internal discussions within my organization.

I have also seen operators struggling and making 'not-so-effective' decisions based upon some handful KPIs which do not showcase accurate health of the fraud management operations.

It also breaks my heart sometimes to see CxOs trying to make sense out of some technical cum operational level KPIs which have been supplied to them. Do they really need them, always ?

So, what are the set of KPIs which can provide the much needed, audience specific insights on the ongoing fraud management operations ? KPIs, which collectively provide a fuller, much elaborate picture of the current state and when created a trend, can work as an effective tool in tuning the operational performance and maturity further ?

With whatever limited experience I have had till now being part of telecom fraud management, I have tried to come up with my own list of "Base KPIs" which I feel are important to capture in terms of measuring the overall health of fraud management operations and also can help derive 'KPIs.

What more ? If the 'historical' results against these KPIs are preserved, they may prove to be useful as an inputs to that 'let's-optimize' analytics programme which your top line management is so interested in.

For the same reason, with the KPI list, I am also providing the recommended trending cycle.

So, lets get to the KPIs.

I have distributed KPIs in two large buckets - Executive (CxO Level) and Fraud Operations (Lead & Manager Level)

Bucket A - Executive KPIs

Executives, considering they mainly deal with the numbers revolving around finance, I have defined the following 2 categories of KPIs for their use:

Category 1 - Fraud Loss & RoI: High Level

This category deals with one of the most exciting but controversial numbers out of fraud management like fraud loss and fraud loss avoided. I will not be gong into details on how to calculate the FL & FLA against a fraud case as part of this blog, but will definitely cover them sometime in future.

The KPIs which I made part of 'Fraud Loss & RoI' category are:

• Fraud losses encountered vs. number of fraud cases identified - daily, weekly & monthly trend
• Fraud loss avoided vs. number of fraud cases identified - weekly & monthly trend
• Fraud losses encountered vs. fraud loss avoided - weekly & monthly trend

Category 2 - Revenue & Bad Debt vs. Fraud

This category helps executives view the impact of fraud & fraud management operations on a much larger scale. These are the statistics which most of the industry studies concentrate on. It covers:

• Revenue stream revenue vs. revenue stream fraud loss - monthly trend
• Total bad debt vs. fraud loss encountered - monthly trend
• Total never pay vs. fraud loss encountered - monthly trend
• Total bad debt vs. Total never pay - monthly trend

One point to note here is that these KPIs depend on KPIs (Bad debt, Never Pay, Revenue etc.) available outside fraud management (Analytics Team, Credit Risk team etc.) purview and because of this dependency, it is better to set documented interfaces with teams responsible for the delivery of the required statistics which can help govern the periodicity & timelines of the data being delivered.

Bucket B - Fraud Operations KPIs

Managers and leads who are responsible to guide fraud operations on a day to day basis need to track the ground level KPIs and work with the more more complex numbers to make improvements in terms of individual performances, fraud run times, fraud coverage and operational efficiency & maturity.

KPIs to be tracked by managers and leads have the following categories of KPIs to deal with:

Category 1 - Fraud Loss & RoI: Low Level

While this category is similar to the 'Fraud Loss & RoI' category under Executive KPIs, the intent is to get to more granular level of identifying where the FL or FLA is concentrated:

• Fraud loss per fraud type - weekly & monthly trend
• Fraud loss avoided per fraud type - weekly & monthly trend

Category 2 - Efficiency

This category of KPIs are an indicator of fraud management function efficiency. Any improvement in the performance if a direct sign of improvement in fraud operations. But mind it, any large deviations (both +ve or -ve) in trend can also be an indicator of glaring issues and anomalies:

• Average Total Case Run time - weekly & monthly 
• Average Fraud Case Run time - weekly & monthly 
• Average Fraud Case Run time: business hours vs. non business hours - monthly 
• Average Fraud Case Run time per fraud type - weekly & monthly 
• Average Non Fraud Case Run time - weekly & monthly 
• Average Non Fraud Case Run time: business hours vs. non business hours - monthly 
• Average Fraud Loss per Case - weekly & monthly 
• Average Fraud Loss per Case per fraud type - weekly & monthly 
• Max. Fraud loss per case - weekly
• Max. Fraud loss per case per fraud type - weekly
• Total cases decisioned to fraud hit ratio (Count & %age) - weekly & monthly 
• Total cases decisioned to Non fraud ratio (Count & %age) - weekly & monthly 

Category 3 - Violations/Cases

These set of KPIs measure fraud function's performance in terms of identifying total cases generated and decisioned by the analysts. This helps identify any anomalies with respect to timely case closure:

• Case creation rate - hourly, daily, weekly and monthly
• Total cases generated vs. total cases decisioned (Fraud/Non Fraud) - daily
• Total open cases - Snapshot

Category 4 - Rules

Identifying and controlling the efficiency of fraud controls/rules is one of the most tedious task for a fraud manager or his SME/Lead. Rules are hard to interpret. They require a much larger theater to guess their actual performance. Hence the following KPIs use much larger data set than usual:

• Rule health & RoI: Covering Rule Name, Total Closed cases, case participation against total cases closed, Fraud cases, Fraud Participation against total cases declared as fraud, Fraud Hit Ratio, Fraud Loss Identified, Fraud Loss Avoided
• Detailed Rule trend: Periodic (Daily,Weekly, Monthly) trend of each rule covering Total cases generated, case participation against total cases generated, Total Closed cases, case Participation against total cases closed, Fraud cases, Fraud Participation against total cases declared as fraud, Fraud Ratio, Non Fraud Ratio, Open case Ratio, Total Risk Identified (Value of total cases generated), Fraud Loss/Risk Identified (Value of total fraud cases), Non Fraud Risk Identified (Value of total non fraud cases), Open Risk Identified (Value of total open cases), Fraud Loss Avoided, case Run Time, Fraud Run Time

Category 5 - Analyst Performance

Analysts are the souls of any fraud operations. Judging their performance is a complicated and sensitive matter. Data against the below provided KPIs must be collected on a much higher frequency (atleast daily) than any other KPIs:

• Individual analyst work rate - daily, weekly and monthly
• Team work rate - daily, weekly and monthly
• Individual analyst Non Fraud vs. Fraud - daily, weekly and monthly
• Individual analyst fraud hit %age - daily, weekly and monthly
• Analyst wise under investigation cases snapshot
• Individual Analyst Average Case Investigation Time - daily, weekly & monthly 
• Individual analyst Average Total Case Run time - daily, weekly & monthly 
• Individual analyst Average Fraud Case Run time - daily, weekly & monthly 
• Individual analyst Average Non Fraud Case Run time - daily, weekly & monthly 
• Individual Analyst Average case Investigation Time - daily, weekly & monthly 

Additional Trends

While KPIs provided above may provide satisfactory coverage in terms of fraud management function's performance metrics, there are certain trends which I feel are worth capturing and can come handy in times of quick decision making or to visualize certain 'not-so-critical' items, when the operations demand:

Trends 1 - Cases

• Fraud Case Count - daily, weekly and monthly 
• Non Fraud Case Count - daily, weekly and monthly 
• Business hours vs. Non business hours cases - monthly
• Business hours vs. Non business hours fraud cases - monthly

Trends 2 - Rules

• Average Latency per rule - Average time passed between first CDR participated and case generation
• Rulewise business hours vs. non business hours case generation - monthly
• Rulewise business hours vs. non business hours fraud case generation - monthly

Trends 3 - Subscriber

• Fraud loss per subscriber category - weekly & monthly 
• Subscriber Deviation: Status wise (Active, suspended, disconnected) subscriber count deviation between Billing & FMS - Monthly
• Subscriber Count: Total subscribers in each category within FMS - Monthly
• Subscriber category coverage: Number of rules monitoring the group/category - Monthly
• Subscriber category Wise Total case Generation - Monthly
• Subscriber category Wise Fraud case Generation - Monthly
• Subscriber category Wise Non Fraud case Generation - Monthly
• Subscriber categories not covered as part of any rules - Monthly
• Subscriber categories exclusion (not covered) against fraud type - Monthly

Trends 4 - Traffic

• Total Traffic per revenue stream according to record type (MOC, MTC etc.) and Service Type (Data, SMS, MMS, Voice) - daily, weekly and monthly 
• Total International Country Wise Outgoing & Incoming Traffic per service type - daily, weekly and monthly
• Total VPMN Wise Outroamer Outgoing & Incoming Traffic per service type - daily, weekly and monthly 
• Total Inroamer Outgoing & Incoming Traffic per service type & operator code - daily, weekly and monthly

Trends 5 - Reference Data

• Phone Numbers with no subscriber information - daily & weekly
• Cellsites (Geographic Position) not configured in FMS but being received as part of CDRs - monthly
• Rate plans not configured in FMS but being received as part of Subscriber Information - monthly

Trends 6 - TopX

• Top 5-10 fraud types - weekly & monthly 
• Top 5-10 high risk subscriber/account groups/categories - weekly & monthly 
• Top 10-20 high risk regions (geographic positions) - weekly & monthly 
• Top 5-10 high risk international destinations (countries) incoming - weekly & monthly 
• Top 5-10 high risk international destinations (countries) outgoing - weekly & monthly 
• Top 5-10 high risk national destinations (prefixes) - weekly & monthly 
• Top 10-20 high risk dealers/outlets/agents - weekly & monthly 
• Top 10-20 high risk accounts - weekly & monthly 
• Top 10-20 high risk accounts - weekly & monthly 
• Top 10-20 high risk rate plans/packages/bundles - weekly & monthly 
• Top 10-20 high risk IMEI series - weekly & monthly 
• Top 10-20 high risk IMSI/CCID series - weekly & monthly 
• Top 10-20 high risk Phone Number series - weekly & monthly 
• Top 10-20 highest value fraud incidents - monthly
• Top 10-20 longest run fraud incidents - monthly
• Top 10-20 highest traffic regions (geographic positions) for voice & sms - weekly & monthly  
• Top 5-10 performing rules w.r.t. fraud to non fraud ration - weekly & monthly
• Top 5-10 performing rules with total frauds identified - weekly & monthly
• Top 5-10  rules with highest fraud loss and fraud loss avoided identified - weekly & monthly
• Bottom 5-10 rules with highest case run times - weekly & monthly
• Bottom 5-10 rules with highest fraud run times - weekly & monthly
• Bottom 5-10 rules with highest false positives - weekly & monthly