{"id":7926,"date":"2023-10-20T15:53:59","date_gmt":"2023-10-20T23:53:59","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7926"},"modified":"2025-04-24T17:05:28","modified_gmt":"2025-04-24T17:05:28","slug":"how-to-monitor-a-fraud-detection-model-in-production","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/","title":{"rendered":"How to Monitor a Fraud Detection Model In Production"},"content":{"rendered":"\n

In this blog post we will leverage Comet\u2019s Model Production Monitoring<\/a> tool to monitor one of the most popular types of model deployed today: fraud detection. Not only will we monitor our for performance, but we will also inspect if our Model free from any bias!<\/p>\n\n\n\n

One of the best ways to improve a machine learning model\u2019s performance is to monitor it the moment it\u2019s deployed to production. Models are trained rigorously on a training dataset. However, in the real world, models can start to infer on data which is different from the data it was trained on. This phenomenon is denoted as \u201cmodel drift\u201d and is a signal to a Data Scientist that their model needs to be re-trained.<\/p>\n\n\n\n

Continuing to Serve a Drifted Model can have significant negative business and monetary impact. Machine Learning Models today are making recommendations, driving our cars, estimating loans, and detecting diseases. Given the growing responsibility of AI in our society, detecting Model Drift as soon as possible should be top of mind for every ML organization.<\/p>\n\n\n\n

No Labels No Problem<\/h2>\n\n\n\n

Having access to ground truth labels isn\u2019t a necessary requirement to start monitoring your models. In fact, most teams cite that it takes them weeks to get access labels for their Fraud Detection Models. In that amount of time it is possible your model would have already drifted. We\u2019ll first cover strategies some strategies that will help team\u2019s detect model drift as early as possible.<\/p>\n\n\n\n

Observe Model Output Distribution<\/h3>\n\n\n\n

Fraud Detection Models are binary classification models that output whether a transaction should be flagged as fraud or not. Like other Anomaly Detection use cases, the distribution of the model\u2019s output will skew heavily to \u201cNo fraud\u201d rather than \u201cFraud\u201d. Based on previous data, data scientists are aware of the \u201cnormal\u201d range of frauds (ie, fraud happens 3\u20135% of the time). If a model starts predicting frauds at a more frequent rate or conversely a significantly less rate, it\u2019s a sign for a ML team that their model might be starting to drift.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

Track Input Feature Drift<\/h3>\n\n\n\n

Data drift refers to the phenomenon where the statistical properties of a dataset change over time in a way that can negatively impact the performance of machine learning models or data analytics. By tracking data drift on input features, ML teams can pinpoint exactly where and why a model is failing.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

To calculate Input Feature Data Drift, teams need to align on which baseline distribution to calculate data drift from. One option is to use the the training dataset distribution as a reference. The other option is to a use prior input data your model saw in prediction at a specified time window.<\/p>\n\n\n\n

Regulate Model Fairness<\/h3>\n\n\n\n

Model Fairness refers to the ethical and equitable treatment of individuals and groups by an AI or machine learning model. A fair model means that its predictions, decisions, or outcomes are not biased or discriminatory against any particular demographic group or characteristic, such as race, gender, age, or other sensitive attributes.<\/p>\n\n\n\n

Engineers want to avoid scenarios in which their Model is disproportionally flagging fraud on a segment of their data. Practitioners quantify fairness by using a metric called disparate impact.<\/p>\n\n\n\n

Disparate Impact = (% Positive Outcomes for Sensitive Group) \/ (% Positive Outcomes for Reference Group)<\/code><\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

The accepted values for Disparate Impact ranges from 0.8\u20131.25.<\/p>\n\n\n\n

Calculate Accuracy of Fraud Detection Models<\/h2>\n\n\n\n

Once a team gets access to ground truth labels they can upload them to Comet\u2019s MPM and calculate Accuracy Metrics. Accuracy itself is often not the best metric to evaluate Fraud Detection Models. For example, let\u2019s say fraudulent transactions only occur 3% of the time. A model that\u2019s consistently predicts \u201cno fraud ever\u201d would be 97% accurate. Hence, metrics like precision, recall, and F1 score are often better metrics to pay attention to.<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

For instances where false negatives need to be minimized (the model predicts no fraud when there is fraud), teams should monitor Recall. If a team finds it unacceptable to mis-classify non-fraudulent transactions as fraud, then it makes more sense to pay attention to Precision. A Weighted F1 Score allows team factor both metrics.<\/p>\n\n\n\n

Try Comet  Today<\/h2>\n\n\n\n

Whether you want to send your predictions real-time or in batch, logging data to Comet MPM is extremely straight forward! Comet also provides the ability to track the exact code and training data  used to train a particular model. Having immediate access to the training lineage makes debugging failing models much more efficient! If you want to learn more about how you can bridge the gap between training and production today with Comet, fill out this form<\/a>!<\/p>\n","protected":false},"excerpt":{"rendered":"

In this blog post we will leverage Comet\u2019s Model Production Monitoring tool to monitor one of the most popular types of model deployed today: fraud detection. Not only will we monitor our for performance, but we will also inspect if our Model free from any bias! One of the best ways to improve a machine […]<\/p>\n","protected":false},"author":21,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[8,9],"tags":[],"coauthors":[134],"class_list":["post-7926","post","type-post","status-publish","format-standard","hentry","category-comet-community-hub","category-product"],"yoast_head":"\nHow to Monitor a Fraud Detection Model In Production - Comet<\/title>\n<meta name=\"description\" content=\"Monitor Fraud Detection Models in Production by tracking model out, input feature drift, fairness, and recall\/precision.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Monitor a Fraud Detection Model In Production\" \/>\n<meta property=\"og:description\" content=\"Monitor Fraud Detection Models in Production by tracking model out, input feature drift, fairness, and recall\/precision.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-20T23:53:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:05:28+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*henSs5PpvyAGe0RCfB2B3Q.png\" \/>\n<meta name=\"author\" content=\"Siddharth Mehta\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Siddharth Mehta\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How to Monitor a Fraud Detection Model In Production - Comet","description":"Monitor Fraud Detection Models in Production by tracking model out, input feature drift, fairness, and recall\/precision.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/","og_locale":"en_US","og_type":"article","og_title":"How to Monitor a Fraud Detection Model In Production","og_description":"Monitor Fraud Detection Models in Production by tracking model out, input feature drift, fairness, and recall\/precision.","og_url":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-10-20T23:53:59+00:00","article_modified_time":"2025-04-24T17:05:28+00:00","og_image":[{"url":"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*henSs5PpvyAGe0RCfB2B3Q.png","type":"","width":"","height":""}],"author":"Siddharth Mehta","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Siddharth Mehta","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/"},"author":{"name":"Siddharth Mehta","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/652eb7d782d18f295922f50ea3b9e54c"},"headline":"How to Monitor a Fraud Detection Model In Production","datePublished":"2023-10-20T23:53:59+00:00","dateModified":"2025-04-24T17:05:28+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/"},"wordCount":772,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/#primaryimage"},"thumbnailUrl":"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*henSs5PpvyAGe0RCfB2B3Q.png","articleSection":["Comet Community Hub","Product"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/","url":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/","name":"How to Monitor a Fraud Detection Model In Production - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/#primaryimage"},"thumbnailUrl":"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*henSs5PpvyAGe0RCfB2B3Q.png","datePublished":"2023-10-20T23:53:59+00:00","dateModified":"2025-04-24T17:05:28+00:00","description":"Monitor Fraud Detection Models in Production by tracking model out, input feature drift, fairness, and recall\/precision.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/#primaryimage","url":"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*henSs5PpvyAGe0RCfB2B3Q.png","contentUrl":"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*henSs5PpvyAGe0RCfB2B3Q.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-monitor-a-fraud-detection-model-in-production\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"How to Monitor a Fraud Detection Model In Production"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/652eb7d782d18f295922f50ea3b9e54c","name":"Siddharth Mehta","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/940c7280faea9e1b8b086c2ed7ec01db","url":"https:\/\/secure.gravatar.com\/avatar\/27a672e997fa7a66796e4be0503e0efeec6bd34daae185bb6de163227a5a0739?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/27a672e997fa7a66796e4be0503e0efeec6bd34daae185bb6de163227a5a0739?s=96&d=mm&r=g","caption":"Siddharth Mehta"},"description":"ML Growth Engineer @ Comet. Interested in Computer Vision, Robotics, and Reinforcement Learning","sameAs":["https:\/\/www.comet.com\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/siddharthmcomet-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7926","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7926"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7926\/revisions"}],"predecessor-version":[{"id":15495,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7926\/revisions\/15495"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7926"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7926"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7926"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7926"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}