{"id":4593,"date":"2022-11-10T17:47:45","date_gmt":"2022-11-11T01:47:45","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=4593"},"modified":"2025-04-24T17:16:40","modified_gmt":"2025-04-24T17:16:40","slug":"model-interpretability-part-3-local-model-agnostic-methods","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/","title":{"rendered":"Model Interpretability Part 3: Local Model Agnostic Methods"},"content":{"rendered":"\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/max\/700\/0*L_NPS-PjyS_9ru-i\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"has-text-align-center\">Source: <a class=\"au kj\" href=\"https:\/\/www.datarevenue.com\/en-blog\/interpretable-machine-learning\" target=\"_blank\" rel=\"noopener ugc nofollow\">datarevenue<\/a><\/p>\n\n\n\n<div class=\"ir is it iu iv\">\n<p id=\"b13f\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">If you haven\u2019t already had a read of the other parts in this series, check them out:<\/p>\n<ul class=\"\">\n<li id=\"484d\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><a class=\"au kj\" href=\"https:\/\/heartbeat.comet.ml\/model-interpretability-part-1-the-importance-and-approaches-f93239edcd21\" target=\"_blank\" rel=\"noopener ugc nofollow\">Model Interpretability Part 1: The Importance and Approaches<\/a><\/li>\n<li id=\"e509\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><a class=\"au kj\" href=\"https:\/\/heartbeat.comet.ml\/model-interpretability-part-2-global-model-agnostic-methods-ff762b3bcc8a\" target=\"_blank\" rel=\"noopener ugc nofollow\">Model Interpretability Part 2: Global Model Agnostic Methods<\/a><\/li>\n<\/ul>\n<p id=\"6290\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">To recap from Part 1:<\/p>\n<p id=\"f9bd\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Local Interpretability<\/strong>&nbsp;aims to capture individual predictions. It focuses on the specific understanding of a data point and be exploring the feature space around it. This allows us to understand the model\u2019s decisions, allowing for better interpretability.<\/p>\n<h2 id=\"a71a\" class=\"lw lx iy bm ly lz ma mb mc md me mf mg ku mh mi mj ky mk ml mm lc mn mo mp mq ga\" data-selectable-paragraph=\"\">Local Methods<\/h2>\n<p id=\"2d4d\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">Local interpretability cares little or not at all about the structure of the model, and is treated as a black-box model. Understanding the distribution of data and its feature space at a local level, rather than a global can give us a more accurate explanation.<\/p>\n<p id=\"2a63\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">In this article, I will be going through three different types of local model agnostic methods.<\/p>\n<h1 id=\"b175\" class=\"mw lx iy bm ly mx my mz mc na nb nc mg nd ne nf mj ng nh ni mm nj nk nl mp nm ga\" data-selectable-paragraph=\"\">Local Surrogate (LIME)<\/h1>\n<p id=\"9403\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">If you read Part 2 of the Model Interpretability series, you will remember Global Surrogate.&nbsp;<strong class=\"bm lv\">Global Surrogate<\/strong>&nbsp;is a type of interpretable modeling that is trained to approximate the predictions of a black-box model.<\/p>\n<p id=\"0ec1\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">However, Local Surrogate, also known as LIME, which stands for Local Interpretable Model-agnostic Explanations is different from Global Surrogate. Where Global aims to explain the whole model, local trains interpretable models to approximate the individual predictions.<\/p>\n<p id=\"4bf3\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The idea of LIME originates from a paper published in 2016:&nbsp;<a class=\"au kj\" href=\"https:\/\/arxiv.org\/pdf\/1602.04938.pdf\" target=\"_blank\" rel=\"noopener ugc nofollow\">\u201cWhy Should I Trust You?\u201d Explaining the Predictions of Any Classifier<\/a>, in which the authors perturb the original data points, feed these data points into the black-box model, and then make observations on the outputs.<\/p>\n<p id=\"f3f4\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The method then uses those new data points and weights them, using it as a function of their proximity to the original point. Each of the original data points can be explained with the new trained explained model.<\/p>\n<p id=\"e3c7\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The learned model can be used as a good approximation of the machine learning model predictions locally, calling this type of accuracy local fidelity.<\/p>\n<p id=\"f70e\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">This can be mathematically expressed as:<\/p>\n<ul class=\"\">\n<li id=\"cb2a\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">X measures how close the explanation is to the predictions of the original model<\/li>\n<li id=\"80de\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">L is the minimum loss, such as the mean squared error<\/li>\n<li id=\"20c9\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">f stands for the original model, for example, XGBoost model<\/li>\n<li id=\"8baf\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">g stands for the explanation model for instance x<\/li>\n<li id=\"a5cf\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">\u03c0x is the proximity measure used to define how large the neighborhood around instance x is that we consider for the explanation.<\/li>\n<li id=\"b1a1\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">\u03a9(g) is the complexity of the model<\/li>\n<li id=\"648f\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">G stands for the family of possible explanations<\/li>\n<\/ul>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<div class=\"ka kb do kc ce kd\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/700\/0*LSiASotRLvgL-3ts\" alt=\"\" width=\"700\" height=\"110\"><\/figure><div class=\"gl gm nn\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*LSiASotRLvgL-3ts 640w, https:\/\/miro.medium.com\/max\/720\/0*LSiASotRLvgL-3ts 720w, https:\/\/miro.medium.com\/max\/750\/0*LSiASotRLvgL-3ts 750w, https:\/\/miro.medium.com\/max\/786\/0*LSiASotRLvgL-3ts 786w, https:\/\/miro.medium.com\/max\/828\/0*LSiASotRLvgL-3ts 828w, https:\/\/miro.medium.com\/max\/1100\/0*LSiASotRLvgL-3ts 1100w, https:\/\/miro.medium.com\/max\/1400\/0*LSiASotRLvgL-3ts 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\"><\/picture><\/div>\n<div><\/div>\n<\/div>\n<\/figure>\n<h2 id=\"1d4b\" class=\"lw lx iy bm ly lz ma mb mc md me mf mg ku mh mi mj ky mk ml mm lc mn mo mp mq ga\" data-selectable-paragraph=\"\">An example:<\/h2>\n<p id=\"1a07\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">Let&#8217;s look at the bike sharing dataset, which can be downloaded from&nbsp;<a class=\"au kj\" href=\"http:\/\/archive.ics.uci.edu\/ml\/datasets\/Bike+Sharing+Dataset\" target=\"_blank\" rel=\"noopener ugc nofollow\">UCI Machine Learning Repository<\/a>. The dataset contains daily counts of rented bicycles from&nbsp;<a class=\"au kj\" href=\"https:\/\/www.capitalbikeshare.com\/\" target=\"_blank\" rel=\"noopener ugc nofollow\">Capital-Bikeshare<\/a>, a bicycle rental company in Washington D.C. It also includes data on weather and seasonal information, where the overall goal is to predict how many bikes will be rented depending on the weather and day.<\/p>\n<p id=\"4e2f\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">In this example, a random forest has been trained with 100 trees on the classification task, aiming to answer this question: Is there a particular day where the number of rental bikes is above the trend-free average, based on weather and calendar information?<\/p>\n<p id=\"dc78\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The results show that the warmer temperature and good weather features have a positive effect on the prediction. The x-axis is defined as \u2018effect,\u2019 this is the weight times by the actuarial feature value, calculating the feature effect.<\/p>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<div class=\"ka kb do kc ce kd\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/700\/0*0FD0oD2Jk0H8mQFn\" alt=\"\" width=\"700\" height=\"490\"><\/figure><div class=\"gl gm ns\" style=\"text-align: center;\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*0FD0oD2Jk0H8mQFn 640w, https:\/\/miro.medium.com\/max\/720\/0*0FD0oD2Jk0H8mQFn 720w, https:\/\/miro.medium.com\/max\/750\/0*0FD0oD2Jk0H8mQFn 750w, https:\/\/miro.medium.com\/max\/786\/0*0FD0oD2Jk0H8mQFn 786w, https:\/\/miro.medium.com\/max\/828\/0*0FD0oD2Jk0H8mQFn 828w, https:\/\/miro.medium.com\/max\/1100\/0*0FD0oD2Jk0H8mQFn 1100w, https:\/\/miro.medium.com\/max\/1400\/0*0FD0oD2Jk0H8mQFn 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\">Source: <\/picture><a class=\"au kj\" href=\"https:\/\/christophm.github.io\/interpretable-ml-book\/ice.html#examples-4\" target=\"_blank\" rel=\"noopener ugc nofollow\">christophm<\/a><\/div>\n<\/div>\n<\/figure>\n<p id=\"5abb\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Advantages:<\/strong><\/p>\n<ul class=\"\">\n<li id=\"7a13\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">The LIME method&nbsp;<strong class=\"bm lv\">works for tabular data, text, and images<\/strong>.<\/li>\n<li id=\"41f7\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">LIME is&nbsp;<strong class=\"bm lv\">easy to use<\/strong>&nbsp;and is implemented in Python, using the&nbsp;<a class=\"au kj\" href=\"https:\/\/github.com\/marcotcr\/lime\" target=\"_blank\" rel=\"noopener ugc nofollow\">lime<\/a>&nbsp;library and R, using the&nbsp;<a class=\"au kj\" href=\"https:\/\/cran.r-project.org\/web\/packages\/lime\/index.html\" target=\"_blank\" rel=\"noopener ugc nofollow\">lime package<\/a>&nbsp;and&nbsp;<a class=\"au kj\" href=\"https:\/\/cran.r-project.org\/web\/packages\/iml\/index.html\" target=\"_blank\" rel=\"noopener ugc nofollow\">iml package<\/a>.<\/li>\n<li id=\"a0ae\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">They make&nbsp;<strong class=\"bm lv\">human-friendly explanations<\/strong>, for example when using short trees, the results are short, easy to explain, and contrastive.<\/li>\n<li id=\"0cd3\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">The&nbsp;<strong class=\"bm lv\">fidelity measure<\/strong>, which measures how well the interpretable model approximates the black box predictions helps us to understand how effective and accurate the interpretable model is in explaining the black box predictions.<\/li>\n<\/ul>\n<p id=\"5126\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Disadvantages:<\/strong><\/p>\n<ul class=\"\">\n<li id=\"edd2\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">The current methodology of sample data points is by using Gaussian distribution, however, this&nbsp;<strong class=\"bm lv\">ignores the correlation or link between particular features<\/strong>. This is important and can produce unlikely data points, which are then used to learn local models.<\/li>\n<li id=\"be27\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Accuracy of the explanations<\/strong>. Repeating the sampling process can show how the explanations for each can come out different to the previous. Therefore, it is difficult to say that we can trust the accuracy of the explanation.<\/li>\n<li id=\"9b91\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">The method is still going through the&nbsp;<strong class=\"bm lv\">development phase<\/strong>, therefore there are many problems that need to be solved before it can be safely applied.<\/li>\n<\/ul>\n<\/div>\n\n\n\n<div class=\"o dx nt nu id nv\" role=\"separator\"><\/div>\n\n\n\n<div class=\"ir is it iu iv\">\n<blockquote class=\"oa\"><p id=\"7f74\" class=\"ob oc iy bm od oe of og oh oi oj lg cn\" data-selectable-paragraph=\"\">Real-time model analysis allows your team to track, monitor, and adjust models already in production.&nbsp;<a class=\"au kj\" href=\"https:\/\/www.comet.com\/site\/lessons-from-the-field-in-building-your-mlops-strategy\/\" target=\"_blank\" rel=\"noopener ugc nofollow\">Learn more lessons from the field with Comet experts.<\/a><\/p><\/blockquote>\n<\/div>\n\n\n\n<div class=\"o dx nt nu id nv\" role=\"separator\"><\/div>\n\n\n\n<div class=\"ir is it iu iv\">\n<h1 id=\"ae0c\" class=\"mw lx iy bm ly mx ok mz mc na ol nc mg nd om nf mj ng on ni mm nj oo nl mp nm ga\" data-selectable-paragraph=\"\">Individual Conditional Expectation (ICE)<\/h1>\n<p id=\"6f76\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">Our second local method is Individual Conditional Expectation, which is very similar to the Partial Dependence Plot (PDP). However, instead of plotting an average which PDP does, ICE displays one line per instance that shows how the instance\u2019s prediction changes as a feature changes.<\/p>\n<p id=\"57d0\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">PDP is a global method, as it focuses on the overall average, not on the specific instances. ICE is more intuitive than PDP due to its locality, exploring how each line represents the prediction for one instance if a feature varies. The overall aim of ICE is to explain what happens to a prediction in the model if the feature changes.<\/p>\n<p id=\"5a08\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">Another way to remember the difference between PDP and ICE is that PDP is the average of the lines of an ICE plot.<\/p>\n<p id=\"f5c6\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">ICE plot shows the dependence between the target function and a particular feature of interest, where it visualizes the dependence of the prediction on a feature for each sample; one line per sample. Only one feature of interest is supported for ICE plots.<\/p>\n<p id=\"3adc\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">An ICE plot has the ability to unravel the curve that PDP fails to do so. Each ICE curve shows the predictions of the feature value for an instance when the features have been varied. When presented in a single plot, we are able to see the relationships between the subsets of the instances and the differences in how the individual instances behave.<\/p>\n<h2 id=\"99ed\" class=\"lw lx iy bm ly lz ma mb mc md me mf mg ku mh mi mj ky mk ml mm lc mn mo mp mq ga\" data-selectable-paragraph=\"\">An example:<\/h2>\n<p id=\"f8f7\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">To keep this blog consistent, using the same dataset from the LIME example: The bike sharing dataset, from&nbsp;<a class=\"au kj\" href=\"http:\/\/archive.ics.uci.edu\/ml\/datasets\/Bike+Sharing+Dataset\" target=\"_blank\" rel=\"noopener ugc nofollow\">UCI Machine Learning Repository<\/a>. Using the underlying prediction model a random forest, which has been trained with 100 trees, aims to predict how many bikes will be rented depending on the weather and the day.<\/p>\n<p id=\"eafd\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The ICE plots shown below are the predicted bicycle rentals based on weather conditions. Looking at each feature, all the curves follow the same course, lacking to present any obvious interactions.<\/p>\n<p id=\"a723\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">Therefore, from this interpretation, we can explore using PDP to see if we can get a better explanation between the features and the predicted number of bicycles.<\/p>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<div class=\"ka kb do kc ce kd\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/700\/0*ZIFGXQYucacOtCrM\" alt=\"\" width=\"700\" height=\"404\"><\/figure><div class=\"gl gm op\" style=\"text-align: center;\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*ZIFGXQYucacOtCrM 640w, https:\/\/miro.medium.com\/max\/720\/0*ZIFGXQYucacOtCrM 720w, https:\/\/miro.medium.com\/max\/750\/0*ZIFGXQYucacOtCrM 750w, https:\/\/miro.medium.com\/max\/786\/0*ZIFGXQYucacOtCrM 786w, https:\/\/miro.medium.com\/max\/828\/0*ZIFGXQYucacOtCrM 828w, https:\/\/miro.medium.com\/max\/1100\/0*ZIFGXQYucacOtCrM 1100w, https:\/\/miro.medium.com\/max\/1400\/0*ZIFGXQYucacOtCrM 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\">Source: <\/picture><a class=\"au kj\" href=\"https:\/\/christophm.github.io\/interpretable-ml-book\/ice.html#examples-4\" target=\"_blank\" rel=\"noopener ugc nofollow\">christophm<\/a><\/div>\n<\/div>\n<\/figure>\n<p id=\"6281\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Advantages:<\/strong><\/p>\n<ul class=\"\">\n<li id=\"be21\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">ICE curves are able to&nbsp;<strong class=\"bm lv\">uncover heterogeneous relationships<\/strong>, unlike PDP.<\/li>\n<li id=\"33f0\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">ICE curves are&nbsp;<strong class=\"bm lv\">easier to understand<\/strong>, in comparison to PDP. Where one line represents the predictions for one instance.<\/li>\n<\/ul>\n<p id=\"000d\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Disadvantages:<\/strong><\/p>\n<ul class=\"\">\n<li id=\"0fe6\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">ICE curves&nbsp;<strong class=\"bm lv\">cannot display more than one feature<\/strong>. Anything more than one feature would need the drawing of several overlaying surfaces, which would make it difficult to interpret anything.<\/li>\n<li id=\"4fc9\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">If ICE curves are overlapping one another, they become&nbsp;<strong class=\"bm lv\">overcrowded<\/strong>; making it useless for model interpretability.<\/li>\n<li id=\"2de9\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">Just like PDP, if the feature which is of interest has correlations with other features; there is a possibility of&nbsp;<strong class=\"bm lv\">invalid data points<\/strong>, due to joint feature distribution.<\/li>\n<\/ul>\n<h1 id=\"50b9\" class=\"mw lx iy bm ly mx my mz mc na nb nc mg nd ne nf mj ng nh ni mm nj nk nl mp nm ga\" data-selectable-paragraph=\"\">Shapley Values<\/h1>\n<p id=\"feae\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">Shapley Values aims to explain with a machine learning model produces the outputs it does. Shapley value was named in honor of&nbsp;<a class=\"au kj\" href=\"https:\/\/en.wikipedia.org\/wiki\/Lloyd_Shapley\" target=\"_blank\" rel=\"noopener ugc nofollow\">Lloyd Shapley<\/a>, a concept that was borrowed from the cooperative game theory literature.<\/p>\n<p id=\"10b3\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">Shapley Values were originally used to fairly attribute a player\u2019s contribution to the end result of a game. For example, if there are a set of players which each collaborate to create some values, we can then measure the total outcome of the game. Shapley values represent the marginal contribution of each player to the end result. In a simpler example, it is the split of a bill between friends; it tells us how to distribute the \u201cpayout\u201d fairly among the features.<\/p>\n<p id=\"87b3\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The Shapley value of a feature is the contribution its value has to the payout, which is then weighted and summed over all the possible feature value combinations. This can be expressed as:<\/p>\n<ul class=\"\">\n<li id=\"4fcd\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">S refers to the subset of the features used in the model<\/li>\n<li id=\"b92b\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">p is the number of features<\/li>\n<li id=\"8ef6\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">x is the vector of feature values that will be explained<\/li>\n<li id=\"961b\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">valx(S) is the prediction for feature values in set S, which are marginalized over features that are not included in set S<\/li>\n<\/ul>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<div class=\"ka kb do kc ce kd\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/700\/0*XhnbqTAQfWRAvywl\" alt=\"\" width=\"700\" height=\"124\"><\/figure><div class=\"gl gm oq\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*XhnbqTAQfWRAvywl 640w, https:\/\/miro.medium.com\/max\/720\/0*XhnbqTAQfWRAvywl 720w, https:\/\/miro.medium.com\/max\/750\/0*XhnbqTAQfWRAvywl 750w, https:\/\/miro.medium.com\/max\/786\/0*XhnbqTAQfWRAvywl 786w, https:\/\/miro.medium.com\/max\/828\/0*XhnbqTAQfWRAvywl 828w, https:\/\/miro.medium.com\/max\/1100\/0*XhnbqTAQfWRAvywl 1100w, https:\/\/miro.medium.com\/max\/1400\/0*XhnbqTAQfWRAvywl 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\"><\/picture><\/div>\n<\/div>\n<\/figure>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<div class=\"ka kb do kc ce kd\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/700\/0*7c2dgBhS58CwjGqF\" alt=\"\" width=\"700\" height=\"128\"><\/figure><div class=\"gl gm or\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*7c2dgBhS58CwjGqF 640w, https:\/\/miro.medium.com\/max\/720\/0*7c2dgBhS58CwjGqF 720w, https:\/\/miro.medium.com\/max\/750\/0*7c2dgBhS58CwjGqF 750w, https:\/\/miro.medium.com\/max\/786\/0*7c2dgBhS58CwjGqF 786w, https:\/\/miro.medium.com\/max\/828\/0*7c2dgBhS58CwjGqF 828w, https:\/\/miro.medium.com\/max\/1100\/0*7c2dgBhS58CwjGqF 1100w, https:\/\/miro.medium.com\/max\/1400\/0*7c2dgBhS58CwjGqF 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\"><\/picture><\/div>\n<\/div>\n<\/figure>\n<p id=\"d15a\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The Shapley Value method satisfies the following properties: Efficiency, symmetry, dummy, and additivity. It is the only model which has these properties which work together to define a fair payout.<\/p>\n<p id=\"c750\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Advantages:<\/strong><\/p>\n<ul class=\"\">\n<li id=\"e6e5\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Efficiency.<\/strong>&nbsp;The prediction and the average predictions are fairly distributed among the feature values of the instance. Whereas, other methods such as LIME do not promise a fair distribution between the features.<\/li>\n<li id=\"f036\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Explanation.<\/strong>&nbsp;Shapley values are very popular due to their full explanation as it is based on theory and the distribution of the effects of the features is fair. It also allows the comparison of subset\/single data points to the predictions.<\/li>\n<\/ul>\n<p id=\"c60c\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Disadvantages:<\/strong><\/p>\n<ul class=\"\">\n<li id=\"92d3\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Computational Power.<\/strong>&nbsp;The Shapley value method is computationally expensive as there is a range of possible coalitions as well as the absence of features. This increases the computation of random instances increasing the variance.<\/li>\n<li id=\"88a9\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Features.<\/strong>&nbsp;Shapley value uses all features, which may not be the explanation that some are looking for. Some tasks require explanations using selective features, such as methods like LIME.<\/li>\n<li id=\"03a9\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">Shapley values&nbsp;<strong class=\"bm lv\">do not make predictions<\/strong>&nbsp;but return a value per feature. This means that you cannot make predictions using Shapley.<\/li>\n<li id=\"0feb\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\"><strong class=\"bm lv\">Correlated Features.<\/strong>&nbsp;Shapley value can be difficult due to the inclusion of unrealistic data instances due to the features having correlation, similar to permutation-based interpretation methods.<\/li>\n<\/ul>\n<h1 id=\"73a5\" class=\"mw lx iy bm ly mx my mz mc na nb nc mg nd ne nf mj ng nh ni mm nj nk nl mp nm ga\" data-selectable-paragraph=\"\">SHAP<\/h1>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<div class=\"ka kb do kc ce kd\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/700\/0*jmeHnq07O8gjCU8d\" alt=\"\" width=\"700\" height=\"384\"><\/figure><div class=\"gl gm os\" style=\"text-align: center;\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*jmeHnq07O8gjCU8d 640w, https:\/\/miro.medium.com\/max\/720\/0*jmeHnq07O8gjCU8d 720w, https:\/\/miro.medium.com\/max\/750\/0*jmeHnq07O8gjCU8d 750w, https:\/\/miro.medium.com\/max\/786\/0*jmeHnq07O8gjCU8d 786w, https:\/\/miro.medium.com\/max\/828\/0*jmeHnq07O8gjCU8d 828w, https:\/\/miro.medium.com\/max\/1100\/0*jmeHnq07O8gjCU8d 1100w, https:\/\/miro.medium.com\/max\/1400\/0*jmeHnq07O8gjCU8d 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\">Source: <\/picture><a class=\"au kj\" href=\"https:\/\/shap.readthedocs.io\/en\/latest\/index.html\" target=\"_blank\" rel=\"noopener ugc nofollow\">SHAP<\/a><\/div>\n<\/div>\n<\/figure>\n<p id=\"98f3\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">SHAP is based on the game theory Shapley Values to explain the output of any machine learning model. It differs from Shapley Values due to its kernel-based estimation approach. The aim of SHAP is to explain the prediction of an instance, which is done by computing the contribution of each feature to the prediction.<\/p>\n<p id=\"62ac\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">Shapley values distribute the predictions fairly among the features. Each player of the game can be considered as an individual feature or a group of features. SHAP combines both LIME and Shapley Values in one, and can be expressed as:<\/p>\n<ul class=\"\">\n<li id=\"14da\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">g refers to the explanation model<\/li>\n<li id=\"dfb5\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">Z\u2032 \u2208{0,1}M refers to the coalition vector<\/li>\n<li id=\"262b\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">M refers to the maximum coalition size<\/li>\n<li id=\"f864\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">\u03d5j\u2208R refers to the feature attribution for a feature j<\/li>\n<\/ul>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/382\/0*VlTL66pqLMerEzx9\" alt=\"\" width=\"382\" height=\"176\"><\/figure><div class=\"gl gm ot\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*VlTL66pqLMerEzx9 640w, https:\/\/miro.medium.com\/max\/720\/0*VlTL66pqLMerEzx9 720w, https:\/\/miro.medium.com\/max\/750\/0*VlTL66pqLMerEzx9 750w, https:\/\/miro.medium.com\/max\/786\/0*VlTL66pqLMerEzx9 786w, https:\/\/miro.medium.com\/max\/828\/0*VlTL66pqLMerEzx9 828w, https:\/\/miro.medium.com\/max\/1100\/0*VlTL66pqLMerEzx9 1100w, https:\/\/miro.medium.com\/max\/764\/0*VlTL66pqLMerEzx9 764w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 382px\" data-testid=\"og\"><\/picture><\/div>\n<\/figure>\n<h2 data-selectable-paragraph=\"\"><\/h2>\n<h2 id=\"72bb\" class=\"lw lx iy bm ly lz ma mb mc md me mf mg ku mh mi mj ky mk ml mm lc mn mo mp mq ga\" data-selectable-paragraph=\"\">An example:<\/h2>\n<p id=\"338e\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">Using the&nbsp;<a class=\"au kj\" href=\"https:\/\/archive.ics.uci.edu\/ml\/datasets\/Cervical+cancer+%28Risk+Factors%29\" target=\"_blank\" rel=\"noopener ugc nofollow\">cervical cancer dataset<\/a>&nbsp;which explores and indicates the risk factors of whether a woman will get cervical cancer. The below SHAP figures explain the features and their correlation to two women from the cervical cancer dataset:<\/p>\n<figure class=\"no np nq nr gx jz gl gm paragraph-image\">\n<div class=\"ka kb do kc ce kd\" tabindex=\"0\" role=\"button\">\n<figure><img loading=\"lazy\" decoding=\"async\" class=\"ce ke kf c aligncenter\" role=\"presentation\" src=\"https:\/\/miro.medium.com\/max\/700\/0*zr5gvK8U-ZSTTfYH\" alt=\"\" width=\"700\" height=\"386\"><\/figure><div class=\"gl gm ou\" style=\"text-align: center;\"><picture><source srcset=\"https:\/\/miro.medium.com\/max\/640\/0*zr5gvK8U-ZSTTfYH 640w, https:\/\/miro.medium.com\/max\/720\/0*zr5gvK8U-ZSTTfYH 720w, https:\/\/miro.medium.com\/max\/750\/0*zr5gvK8U-ZSTTfYH 750w, https:\/\/miro.medium.com\/max\/786\/0*zr5gvK8U-ZSTTfYH 786w, https:\/\/miro.medium.com\/max\/828\/0*zr5gvK8U-ZSTTfYH 828w, https:\/\/miro.medium.com\/max\/1100\/0*zr5gvK8U-ZSTTfYH 1100w, https:\/\/miro.medium.com\/max\/1400\/0*zr5gvK8U-ZSTTfYH 1400w\" sizes=\"(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px\" data-testid=\"og\">Source: <\/picture><a class=\"au kj\" href=\"https:\/\/christophm.github.io\/interpretable-ml-book\/shap.html#examples-5\" target=\"_blank\" rel=\"noopener ugc nofollow\">christophm<\/a><\/div>\n<\/div>\n<\/figure>\n<p id=\"fa0d\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">The baseline, which is the average predicted probability is 0.066. The first woman, which is the first SHAP plot has a low predicted risk of 0.06. The second woman, which is the second SHAP plot has a high predicted risk of 0.71.<\/p>\n<p id=\"012b\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">For the first woman, factors such as STD have balanced out the effects of age. For the second woman, factors such as age and the year of smoking have increased her predicted cancer risk.<\/p>\n<h1 id=\"a4d6\" class=\"mw lx iy bm ly mx my mz mc na nb nc mg nd ne nf mj ng nh ni mm nj nk nl mp nm ga\" data-selectable-paragraph=\"\">Conclusion:<\/h1>\n<p id=\"6e3e\" class=\"pw-post-body-paragraph kk kl iy bm b km mr ko kp kq ms ks kt ku mt kw kx ky mu la lb lc mv le lf lg ir ga\" data-selectable-paragraph=\"\">If you have kept up to date with this series of Model Interpretability, we have covered:<\/p>\n<ul class=\"\">\n<li id=\"81c6\" class=\"lh li iy bm b km kn kq kr ku lj ky lk lc ll lg lm ln lo lp ga\" data-selectable-paragraph=\"\">Model Interpretability Part 1: The Importance and Approaches<\/li>\n<li id=\"c2e1\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">Model Interpretability Part 2: Global Model Agnostic Methods<\/li>\n<li id=\"bd88\" class=\"lh li iy bm b km lq kq lr ku ls ky lt lc lu lg lm ln lo lp ga\" data-selectable-paragraph=\"\">And now Model Interpretability Part 3: Local Model Agnostic Methods<\/li>\n<\/ul>\n<p id=\"385c\" class=\"pw-post-body-paragraph kk kl iy bm b km kn ko kp kq kr ks kt ku kv kw kx ky kz la lb lc ld le lf lg ir ga\" data-selectable-paragraph=\"\">If you would like to know more about Model Interpretability, I would highly recommend reading&nbsp;<a class=\"au kj\" href=\"https:\/\/christophm.github.io\/interpretable-ml-book\/\" target=\"_blank\" rel=\"noopener ugc nofollow\">Interpretable Machine Learning<\/a>&nbsp;by Christoph Molnar. He has given me the guidance and better understanding to write this three-part series about Model Interpretability.<\/p>\n<\/div>\n\n\n\n<div class=\"o dx nt nu id nv\" role=\"separator\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Source: datarevenue If you haven\u2019t already had a read of the other parts in this series, check them out: Model Interpretability Part 1: The Importance and Approaches Model Interpretability Part 2: Global Model Agnostic Methods To recap from Part 1: Local Interpretability&nbsp;aims to capture individual predictions. It focuses on the specific understanding of a data [&hellip;]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[139],"class_list":["post-4593","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Model Interpretability Part 3: Local Model Agnostic Methods - Comet<\/title>\n<meta name=\"description\" content=\"Local Interpretability\u00a0aims to capture individual predictions. It focuses on the specific understanding of a data point and be exploring the feature space around it.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Model Interpretability Part 3: Local Model Agnostic Methods\" \/>\n<meta property=\"og:description\" content=\"Local Interpretability\u00a0aims to capture individual predictions. It focuses on the specific understanding of a data point and be exploring the feature space around it.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2022-11-11T01:47:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:16:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/max\/700\/0*L_NPS-PjyS_9ru-i\" \/>\n<meta name=\"author\" content=\"Nisha Arya Ahmed\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nisha Arya Ahmed\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Model Interpretability Part 3: Local Model Agnostic Methods - Comet","description":"Local Interpretability\u00a0aims to capture individual predictions. It focuses on the specific understanding of a data point and be exploring the feature space around it.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/","og_locale":"en_US","og_type":"article","og_title":"Model Interpretability Part 3: Local Model Agnostic Methods","og_description":"Local Interpretability\u00a0aims to capture individual predictions. It focuses on the specific understanding of a data point and be exploring the feature space around it.","og_url":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2022-11-11T01:47:45+00:00","article_modified_time":"2025-04-24T17:16:40+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/max\/700\/0*L_NPS-PjyS_9ru-i","type":"","width":"","height":""}],"author":"Nisha Arya Ahmed","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Nisha Arya Ahmed","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"Model Interpretability Part 3: Local Model Agnostic Methods","datePublished":"2022-11-11T01:47:45+00:00","dateModified":"2025-04-24T17:16:40+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/"},"wordCount":2006,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*L_NPS-PjyS_9ru-i","articleSection":["Machine Learning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/","url":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/","name":"Model Interpretability Part 3: Local Model Agnostic Methods - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*L_NPS-PjyS_9ru-i","datePublished":"2022-11-11T01:47:45+00:00","dateModified":"2025-04-24T17:16:40+00:00","description":"Local Interpretability\u00a0aims to capture individual predictions. It focuses on the specific understanding of a data point and be exploring the feature space around it.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/#primaryimage","url":"https:\/\/miro.medium.com\/max\/700\/0*L_NPS-PjyS_9ru-i","contentUrl":"https:\/\/miro.medium.com\/max\/700\/0*L_NPS-PjyS_9ru-i"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/model-interpretability-part-3-local-model-agnostic-methods\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Model Interpretability Part 3: Local Model Agnostic Methods"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4593","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=4593"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4593\/revisions"}],"predecessor-version":[{"id":15655,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4593\/revisions\/15655"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=4593"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=4593"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=4593"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=4593"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}