The earliest recorded attempt of fraud can be found as far back as the year 300 BC in Greece.<\/p>\n\n\n\n
A Greek sea merchant named Hegestratos sought to insure his ship and cargo, so took out an insurance policy against them. At the time, the policy was known as a `bottomry` and worked on the basis that a merchant borrowed money to the value of his ship and cargo.<\/p>\n\n\n\n
As long as the ship arrived safe and sound at its destination with its cargo intact, then the loan was paid back with interest. If on safe delivery the loan was not repaid, the boat and its cargo were repossessed.<\/p>\n\n\n\n
Hegestratos planned to sink his empty boat, keep the loan, and sell the corn. The plan failed, and he drowned trying to escape his crew and passengers when they caught him in the act.<\/p>\n\n\n\n <\/p>\n\n\n\n Source<\/a><\/p>\n\n\n\n Some 2,000+ years later financial institutions are still fighting the same battle.<\/strong><\/p>\n<\/blockquote>\n\n\n\n Traditionally, financial institutions automatically flagged transactions as \u201chigh risk\u201d based on a set of clearly defined rules, and then either denied or manually reviewed flagged transactions.<\/p>\n\n\n\n While these rules-based systems are still an important part of the anti-fraud toolkit, they were never designed for modern internet businesses.<\/p>\n\n\n\n This new era of finance and banking has seen tremendous innovation from companies such as PayPal, Stripe, Square, and Venmo which have made it easy for any one to send and receive money right from their phones.<\/p>\n\n\n\n This has led to many financial institutions being flooded by a tsunami of data.<\/p>\n\n\n\n <\/p>\n\n\n\n Source<\/a><\/p>\n\n\n\n Machine learning algorithms capable of processing these ever increasingly large datasets have been an indispensable in helping financial institutions identify correlations between user behavior and the likelihood of fraudulent actions.<\/p>\n\n\n\n Data scientists have been successful in authenticating transactions using machine learning and predictive analytics.<\/p>\n\n\n\n Automated fraud screening systems powered by machine learning can swiftly and accurately detect a fraudulent transaction, mitigating the risk of that transaction going through and saving financial intuitions and their customers millions of dollars.<\/p>\n\n\n\n Imagine yourself as a data scientist at a company that builds products which enable millions of small e-commerce merchants and small, local business owners to conduct business online.<\/p>\n\n\n\n <\/p>\n\n\n\n Source<\/a><\/p>\n\n\n\n Unlike the brick-and-mortar counterparts, online merchants are held liable for fraudulent purchases.<\/p>\n\n\n\n When a cardholder\u2019s bank declares a transaction as fraudulent, the cardholder is fully reimbursed while the merchant is left footing the cost of the fraud.<\/p>\n\n\n\n This cost is always more than just the dollar amount of the transaction, because not only do they lose the value of the item sold \u2014 whatever it cost them to make it or purchase it themselves \u2014 they\u2019re also responsible for any fees resulting from the dispute.<\/p>\n\n\n\n These costs can add up and have a significant negative impact on these merchants livelihoods.<\/p>\n\n\n\n Your company wants to help merchants focus on their product and customer experiences \u2014 and not on fraud \u2014 so they\u2019re developing a suite of modern tools for fraud detection and prevention.<\/p>\n\n\n\n Central to which is a machine learning model which evaluates transactions for fraud risk and takes action appropriately.<\/p>\n\n\n\n Using the dataset provided, your task is to build a machine learning model which provides greatest protection against fraudulent transactions by correctly classifying and blocking fraudulent transactions.<\/p>\n\n\n\n Here\u2019s some high level details about the dataset we\u2019re working with:<\/p>\n\n\n\n And here are the features present in our data set:<\/p>\n\n\n\n <\/p>\n\n\n\n Note, this data is actually simulated data from the PaySim Kaggle Competition. You can visit the source by following this link:<\/p>\n\n\n\n In any binary classification problem \u2014 such as the one we\u2019ve got here \u2014 there are two types of misclassifications we can make: False positives and false negatives<\/p>\n And the main challenge here is: how do we tell if our model is any good?<\/strong><\/p>\n Let\u2019s explore this a bit deeper.<\/p>\n These are instances of fraud which are not identified or prevented before a dispute occurs, when your model says a transaction is not fraudulent but it really is.<\/p>\n These are legitimate transactions which are blocked by a fraud detection system, when your model says a transaction is fraudulent but it really isn\u2019t.<\/p>\n A tradeoff exists between false negatives and false positives.<\/p>\n Optimize for fewer false negatives, and you must be willing to tolerate a greater occurrence of false positives (and vice versa).<\/p>\n A false negative incurs the cost of goods sold and the fee for disputes. A false positive incurs the margin on the goods sold.<\/p>\n Businesses need to decide how to trade off the two.<\/p>\n When a business has small margins, the false negative is a more costly mistake to make than a false positive. In this case, we would want to be more robust and try to stop as many potentially fraudulent cases as possible. Even if it means more false positives.<\/strong><\/p>\n The opposites is true when margins are larger: false positives become more costly than false negatives.<\/p>\n The data scientist faces the problem of building a good machine learning model by engineering the right features and finding the best algorithm for their business, and the business problem of picking a policy for actioning a given machine learning model\u2019s outputs.<\/p>\n Suppose that our business prescribes policy to block a payment if the machine learning model assigns the transaction a probability of being fraudulent of at least 0.8 \u2014 in math terms P(fraud) >= 0.8.<\/p>\n This is because from the businesses point of view we want to be fairly confident before we block a transaction, because it can be a nuisance to merchants if they have too many non-fraudulent transactions flagged as fraudulent. This situation would cause a poor user experience and result in unhappy customers.<\/p>\n There are A LOT of metrics we can choose to evaluate our machine learning model.<\/p>\n A simple confusion matrix yields 14 metrics.<\/p>\n <\/p>\n Source<\/a><\/p>\n Let\u2019s examine three of these in closer detail: precision, recall, and false positive rate.<\/p>\n The precision of our model is the proportion of transactions our model flagged as fraudulent which are truly fraudulent.<\/p>\n Example: Suppose we had 100 transactions, and our model flags 60 of these as fraudulent.<\/p>\n And of those 60 transaction that were flagged as fraudulent, 40 are actually fraudulent.<\/p>\n Then the precision would be 40 \/ 60 = 0.66.<\/p>\n A higher precision implies a fewer number of false positives.<\/strong><\/p>\n The recall of our model (aka true positive rate) is the proportion of all truly fraudulent transactions which were flagged as fraudulent by our model.<\/p>\n Example: Suppose we had 100 transactions, and 50 of these are truly fraudulent.<\/p>\n And of these 50 truly fraudulent transactions, our model flags 40 of them as fraudulent.<\/p>\n Then the recall is 40 \/ 50 = 0.8.<\/p>\n A higher implies recall implies a fewer number of false negatives.<\/strong><\/p>\n <\/p>\n Source<\/a><\/p>\n The false positive rate of our model is the proportion of all legit (non-fraudulent) transactions which the model incorrectly flagged as fraudulent.<\/p>\n Example: Suppose we had 100 transactions, and 50 of these are transactions are legit.<\/p>\n And of these 50 legit transactions, our model flagged 20 of them as fraudulent.<\/p>\n Then the false positive rate is 20 \/ 50 = 0.4.<\/p>\n If we had a perfectly clairvoyant model, then 100% of of the transactions it classifies as fraud would actually be fraud.<\/p>\n This would imply a few things about the values of precision, recall, and false positive rate:<\/p>\n We don\u2019t often build clairvoyant models, so there is a tradeoff between precision and recall.<\/p>\n As we increase the probability threshold<\/strong> for blocking (the business policy we discussed earlier), precision will increase<\/strong>.<\/p>\n That\u2019s because the criterion for blocking a transaction is more strict, which implies a high level of \u201cconfidence\u201d for classifying the transaction as fraudulent. This, in turn, results in fewer false positives.<\/p>\n By the same logic, it follows that increasing the probability threshold will cause recall to decrease, implying fewer false negatives. That\u2019s because we now have fewer transactions match the high probability threshold.<\/p>\n Conversely, when we decrease the probability threshold the reverse happens: precision will decrease and recall will increase.<\/p>\n One way to understand and visualize this trade-off is through the ROC-curve, which plots recall (aka true positive rate) against the false positive rate<\/p>\n A perfect model will have an ROC curve that hug the top left of the graph \u2014 where recall is 1.0 and the false positive rate is 0.0.<\/p>\n A random model would have the true positive rate equal the false positive rate, and we\u2019d end up with a straight line.<\/p>\n We hope to build a model that\u2019s somewhere between a perfect model and a random guesser.<\/p>\n One way to capture the overall quality of the model is by computing the area under the curve ROC curve (ROC AUC).<\/p>\n <\/p>\n Source<\/a><\/p>\n When the classification problem becomes an imbalanced classification problem \u2014 the ROC AUC becomes misleading.<\/p>\n This imbalanced situations a small number of correct or incorrect predictions can result in a large change in the ROC Curve or ROC AUC score.<\/p>\n A common alternative is the precision-recall curve and area under curve.<\/p>\n A precision-recall curve (or PR Curve) is a plot of the precision (y-axis) and the recall (x-axis) for different probability thresholds.<\/p>\n <\/p>\n Source<\/a><\/p>\n A perfectly skilled, clairvoyant model would be depicted as a point at a coordinate of (1,1).<\/p>\n A random, no-skill classifier will be a horizontal line on the plot with a precision that is proportional to the number of fraudulent transactions in the dataset.<\/p>\n As a model becomes better and better the it will start to hug the top-right of the graph \u2014 where precision and recall are both maximized at 1.0.<\/p>\n Because the Precision-Recall curve puts more emphasis on the minority class, it proves to be an effective diagnostic for imbalanced binary classification models.<\/p>\n Just like with the ROC Curve, we can capture the overall quality of the model by computing the area under the precision-recall curve (PR AUC).<\/p>\n In our example, we\u2019ll use the area under the Precision-Recall curve as an evaluation metric.<\/strong><\/p><\/blockquote>\n Using the data provided by our stakeholders, we\u2019ll perform some basic data understanding, exploratory data analysis, engineer features, balance our dataset, spot check candidate models, choose the best candidates for further hyper-parameter tuning, and finally evaluate models to identify which one we\u2019ll add to our model registry.<\/p>\n Each of the notebooks linked below has all the details for each step in the methodology.<\/p>\n We\u2019ll be tracking our experiments using Comet throughout!<\/p>\n We\u2019ll use the following libraries in python to assist us with this project:<\/p>\n I firmly believe the best way to learn something is by trying it for yourself. Instead of just sharing code snippets, I thought it would be more fun for you to get hands on, so I\u2019ve shared runnable Colab Notebooks so you can more easily follow along.<\/p>\n <\/p>\n In this notebook we access the data provided to us by our stakeholders and log the raw data to <\/p>\n Profiling data using the Learn more about how <\/p>\n More in-depth analysis of the data and creating new features from raw data. We\u2019ll also log versions of our data as an Artifact to We\u2019ll spot-check a suite of classification algorithms and choose the best one for further hyper-parameter tuning.<\/p>\n <\/p>\n We\u2019ll use Comet to run experiments and select the best combination of hyper-parameters for these algorithms.<\/p>\n We\u2019ll then evaluate the performance on the test set and register it on the Model Registry in Comet<\/a>. This is an incredibly useful tool which allows us to store trained machine learning models, metadata about the data and training jobs used to create the model, and keep track of which versions are in production. This is a critical piece of the puzzle for establishing lineage<\/strong> for ML models.<\/p>\n<\/div>\n![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/10\/0_3fRCDEh-qB2g4rM9-300x241.jpg\")
<\/figure>\n\n\n\n\n
Machine learning and fraud<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/upload.wikimedia.org\/wikipedia\/commons\/thumb\/3\/3b\/Parigi_griffe.jpg\/1200px-Parigi_griffe.jpg\")
<\/figure>\n\n\n\nProblem statement<\/strong><\/h2>\n\n\n\n
<\/figure>\n\n\n\nData<\/strong><\/h2>\n\n\n\n
\n
CASH_OUT<\/code>\u00a0(n = 4,116) and\u00a0TRANSFER<\/code>\u00a0(n = 4,097) type transactions.\u00a0TRANSFER<\/code>\u00a0is where money is sent to a customer \/ fraudster and\u00a0CASH_OUT<\/code>\u00a0is where money is sent to a merchant who pays the customer \/ fraudster in cash. Speaking to stakeholders we identify that the modus operandi for fraudulent transactions: fraud is committed by first transferring out funds to another account which subsequently cashes it out.<\/li>\n\n\n\nnameDest<\/code>, which is the feature indicating recipient id of the transaction. Speaking to the stakeholders learn that the naming scheme for both\u00a0nameDest<\/code>\u00a0and\u00a0nameOrig<\/code>\u00a0columns is as follows: IDs beginning with\u00a0C<\/code>\u00a0indicate it is a customer account, and IDs beginning with\u00a0M<\/code>\u00a0indicate it is a merchant account.<\/li>\n<\/ul>\n\n\n\n<\/figure>\n\n\n\nThe challenge in building a model<\/strong><\/h2>\n
False negatives<\/strong><\/h3>\n
False positives<\/strong><\/h3>\n
The tradeoff<\/strong><\/h3>\n
So, which do you choose to optimize for?<\/strong><\/h3>\n
The business policy<\/strong><\/h2>\n
Evaluating the machine learning model<\/strong><\/h2>\n
<\/figure>Precision<\/strong><\/h2>\n
Recall<\/strong><\/h2>\n
<\/figure>False positive rate<\/strong><\/h2>\n
So you\u2019re probably wondering\u2026What are \u201cgood values\u201d for precision, recall, and false positive rate.<\/strong><\/h3>\n
\n
The impact of the probability threshold<\/strong><\/h2>\n
ROC and precision-recall curves<\/strong><\/h2>\n
ROC Curves<\/strong><\/h3>\n
<\/figure>Class imbalance changes everything<\/strong><\/h2>\n
Precision-recall curves<\/strong><\/h2>\n
<\/figure>Methodology<\/strong><\/h2>\n
Dependencies<\/strong><\/h2>\n
\n
Notebooks<\/strong><\/h2>\n
Fetch raw data<\/strong><\/h3>\n
<\/a><\/figure>Comet<\/code> as an Artifact.<\/p>\nData understanding<\/strong><\/h3>\n
<\/a><\/figure>sweetviz<\/code> library to get a feel of it and see what\u2019s going on in there. If you haven\u2019t seen sweetviz<\/code> in action, prepared to be impressed! It\u2019s an an open-source Python library that generates beautiful, high-density visualizations to kickstart EDA (Exploratory Data Analysis) with just two lines of code.<\/p>\nsweetviz<\/code> integrated with Comet here:<\/p>\n<\/a><\/figure>Exploratory data analysis and feature engineering<\/strong><\/h3>\n
<\/a><\/figure>Comet<\/code> to keep track of it.<\/p>\nSpot-checking algorithms<\/strong><\/h3>\n
<\/a><\/figure>Experimenting with hyper-parameters<\/strong><\/h3>\n
<\/a><\/figure>