![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Pe6ibd8O4G225zvqGA1wNQ.jpeg\")
<\/figure>Introduction<\/strong><\/h2>\nHave you ever questioned how we can become so engrossed in watching videos on YouTube and how some of these posts end up on your Facebook or Instagram timeline? For those of us who are familiar with Netflix, we frequently receive recommendations for movies that are similar to the ones we are looking for or perhaps the ones we have just finished watching. This is made possible by an algorithm known as the “recommendation system.” In this tutorial, we\u2019ll talk more about the recommendation system, build a system for recommending movies, and then integrate it into Comet.<\/p>\n
Recommendation Systems<\/strong><\/h2>\nThere are 3 main types of Recommendation systems<\/p>\n
\n- Content-Based Recommendation system.<\/li>\n
- Popularity-Based Recommendation system.<\/li>\n
- Collaborative Recommendation system.<\/li>\n<\/ol>\n
Content-Based Recommendation system:<\/p>\n
It is a type of recommendation system that operates on a similar content principle. If a user is watching a movie, the system will look for other movies with similar content or in the same genre as the one the user is watching. When comparing similar content, various fundamental attributes are used to compute similarity.<\/p>\n
Popularity-Based Recommendation System:<\/strong><\/h2>\nIt is a type of recommendation system that operates based on popularity of anything that is currently popular. These systems examine the products or movies that are popular among users and directly recommend them.<\/p>\n
For instance, if a product is frequently purchased by the majority of people, the system will learn that it is the most popular, so for every new user who has just signed up, the system will recommend that product to that user as well, and the chances are that the new user will also purchase that.<\/p>\n
Collaborative Recommendation system<\/strong><\/h2>\nCollaborative recommendation, also called collaborative filtering, works by analyzing the preferences and data of numerous users. Collaborative filtering predicts a user\u2019s interests. This is accomplished by applying techniques involving cooperation among numerous agents, data sources, etc. to filter data for information or patterns. The underlying premise of collaborative filtering is that users A and B are likely to have similar tastes in products if they have similar tastes in one product.<\/p>\n
Comet<\/strong><\/h2>\nComet is a platform for experimentation that enables you to monitor your machine-learning experiments. Comet has another noteworthy feature: it allows us to conduct exploratory data analysis. We can accomplish our EDA objectives thanks to Comet\u2019s integration with well-known Python visualization frameworks. You can learn more about comet here.<\/a><\/p>\nThis lesson will teach us how to connect Comet with our recommendation system. We will carry out some EDA on our movie dataset to achieve this, and we will log the visualization onto the Comet experimentation website or platform. Let\u2019s begin without further ado.<\/p>\n
Prerequisites<\/strong><\/h2>\nYou may install the Comet library on your computer if you don\u2019t already have it there by using the following line at the command prompt.<\/p>\n
pip install comet_ml<\/span><\/pre>\n\u2014 or \u2014<\/p>\n
conda install -c comet_ml<\/span><\/pre>\nAbout the data<\/strong><\/h2>\nAll 45,000 films mentioned in the Full MovieLens Dataset are represented by these files\u2019 information. Movies that were released on or before July 2017 are included in the dataset. Cast, crew, narrative keywords, budget, revenue, posters, release dates, language breakdowns, production firms, nations, and TMDB vote counts and averages are just a few examples of the data points.<\/p>\n
Additionally, this collection includes files with 26 million user ratings for all 45,000 movies, collected from 270,000 individuals. A scale of 1 to 5 stars was used to rate each item on the GroupLens official website. You can get the data from Kaggle here.<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\nReal-time model analysis allows your team to track, monitor, and adjust models already in production. Learn more lessons from the field with Comet experts<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\nMovie Recommendation system<\/strong><\/h2>\nThe first step involves importing the dependencies.<\/p>\n
import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport difflib\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.metrics.pairwise import cosine_similarity<\/span><\/pre>\nWe then move to load our dataset into the pandas dataframe.<\/p>\n
movie_data = pd.read_csv('\/content\/jmovies.csv')<\/span><\/pre>\nWe then move to the next step, which is data preprocessing.<\/p>\n
movie_data.head()<\/span><\/pre>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*lMRIcZWegabMcQY2NrMDzw.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nmovie_data.info()<\/span><\/pre>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:531\/1*dWpOC_qpVd7ew6qN1HXxKQ.png\")
<\/figure><\/picture><\/div>\n<\/figure>\nmovie_data. isnull()<\/span><\/pre>\nThe movie_data. isnull() function returns false if the values in the dataset are not missing and return true if they are missing.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*3TJvmz5EO1YT1Hqm3pec7g.png\")
<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\nThe movie_data. isnull().head() function only gives us the details on the first 5 columns, but to know the exact column and how many values are missing, we use the movie_data. isnull().sum() function tells us which column has missing values and the number of missing values it has.<\/p>\n
movie_data. isnull().sum()<\/span><\/pre>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:352\/1*VZ8lBKwf-j9e338jBlhQsQ.png\")
<\/figure><\/picture><\/div>\n<\/figure>\nWe have a lot of elements in our dataset and not all can be used for our recommendation system, so we select only those that can be used. We also pick elements that will affect the accuracy of our recommendation system. Users are most likely to watch a movie based on the class of genres, the type of cast in the movie, and also the director in charge of the movie<\/p>\n
From our data preprocessing, we can see that we have a lot of missing values in our dataset, which will affect the result of our recommendation system. We need to fill it up with null strings, but before we do that, we need to perform some EDA.<\/p>\n
movie_data[['keywords','tagline','cast','director']].fillna('')<\/span><\/pre>\nWe then create a variable named “combined features” input all the selected features and print them. This is done so we can input it as a whole into our vectorizer, which converts the text to feature vectors.<\/p>\n
combined_features= movie_data['genres']+' '+movie_data['keywords'] \\\n+' '+movie_data['tagline']+' '+movie_data['cast'] \\\n+' '+movie_data['director']<\/span><\/pre>\nThe next step involves converting our text data to numerical data and this is done so it can be understood by the cosine similarity, which works well with numerical data we be using TfidfVectorizer to convert.<\/p>\n
vectorizer = TfidfVectorizer()\nfeature_vectors = vectorizer.fit_transform(combined_features)<\/span><\/pre>\nSo once the numerical data is ready, we then try and get the similarity scores using cosine similarity, and we then load our feature vector into the cosine similarity to find movies that are similar to each other. So what the cosine similarity does is iterate through the list and try to find similarities between the selected character and the remaining movies in the dataset, it does this for each of the movies.<\/p>\n
similarity = cosine_similarity(feature_vectors)<\/span>print(similarity)\nprint(similarity.shape)<\/span><\/pre>\nTo get our recommendation system working we need to create an input section for the user to input the name of their selected movie.<\/p>\n
movie_name = input(' Enter your favourite movie name : ')<\/span><\/pre>\nWe then need to create a list of the names of all movies in the dataset.<\/p>\n
list_of_all_titles = movie_data['title'].tolist()\nprint(list_of_all_titles)<\/span><\/pre>\nThe next step is finding a movie match for the movie name that the user inputs.<\/p>\n
find_close_match = difflib.get_close_matches(movie_name,\n list_of_all_titles)\nprint(find_close_match)<\/span><\/pre>\nThe above code gives us a list of related movies but we need to narrow it down to the exact movie the user inputs.<\/p>\n
close_match = find_close_match[0]\nprint(close_match)<\/span><\/pre>\nTo get the index of each movie we use the title of the movie to find the index<\/p>\n
index_of_the_movie = movie_data[movie_data.title == close_match]['index'].values[0]\nprint(index_of_the_movie)<\/span><\/pre>\nSo, using the index value, we will be getting a list of similar values. So when you run this code you\u2019ll notice the index of the movie.<\/p>\n
similarity_score = list(enumerate(similarity[index_of_the_movie]))\nprint(similarity_score)<\/span><\/pre>\nWe then print the length of the similarity score.<\/p>\n
len(similarity_score)<\/span><\/pre>\nWe then sort the movie based on the similarity score. So the same genres of movies are sorted together.<\/p>\n
sorted_similar_movies = sorted(similarity_score, key = lambda x:x[1], reverse = True)\nprint(sorted_similar_movies)<\/span><\/pre>\nThen we work on printing a list based on the similarity scores.<\/p>\n
print('Movies suggested for you : \\n')<\/span>i = 1<\/span>for movie in sorted_similar_movies:\n index = movie[0]\n title_from_index = movies_data[movies_data.index==index]['title'].values[0]\n if (i<30):\n print(i, '.',title_from_index)\n i+=1<\/span><\/pre>\nSo we can see that our recommendation system is up and running and the movie\u2019s recommendation works just fine. So the next step includes logging our visualization on the comet platform.<\/p>\n
So we need to visualize some of the characters in our dataset using matplotlib and seaborn.<\/p>\n
mask_df = movie_data[\"director\"].value_counts().head(10)\nfig1 = plt.figure(figsize=(12, 10))\nplt.bar(x = mask_df.index, height=mask_df, color=\"sienna\")\nplt.xlabel(\"director\")\nplt.ylabel(\"budget\")\nplt. title(\"Budget of each director per movie\")\nplt.xticks(rotation=45)<\/span><\/pre>\nSo the above code shows the budget of each director per movie and we can see how the budget for each movie varies per director.<\/p>\n
In the next step, we input the budget per movie, and we can see which of the movies has the highest budget.<\/p>\n
fig2 = plt.figure(figsize=(12, 10))\nsns.histplot(movie_data[\"budget\"], color=\"darkslategrey\", bins=50)\nplt.title(\"Budget of Movie\");<\/span><\/pre>\nThe next step will be to log the visualization onto the Comet platform once we have finished it. In this section, we\u2019ll use the Comet experiment library to build a brand-new project.<\/p>\n
You will require a Comet API key to continue with this phase. Click here to sign up for their platform if you haven\u2019t already. By clicking on your profile photo, selecting the settings icon, and then scrolling down, as demonstrated below, you can find your API key.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*P2DLupXzej9mAP4EGPNc2w.png\")
<\/figure><\/picture><\/div>\n<\/div>\nImage by Author<\/figcaption>\n<\/figure>\nfrom comet_ml import Experiment<\/span># Create an experiment with your API key<\/span>experiment = Experiment(\napi_key=\"xGZbEEC8bB2VnpxWFcWXd3BI3\",\nproject_name=\"Recommendation system\",\nworkspace=\"olujerry\",\n)\nexperiment.log_figure(figure_name=\"Matplotlib Viz\", figure=fig1)\nexperiment.log_figure(figure_name= \"Seaborn Viz\", figure=fig2)<\/span><\/pre>\nWe imported the Experiment<\/code> library from comet ml<\/strong><\/code> into the code above. We then instantiated it and assigned it to the experiment variable. Your API key, the name of your project, and the name of your workspace are some of the parameters needed by the Experiment library. The Comet platform will display the workspace name, such as:<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*cLzvP2vCU7HYWg6PK4dUzw.png\")
<\/figure><\/picture><\/div>\n<\/div>\nImage By Author<\/figcaption>\n<\/figure>\nexperiment.log_figure(figure_name=\"Matplotlib Viz\", figure=fig1)experiment.log_figure(figure_name= \"Seaborn Viz\", figure=fig2)experiment.end()<\/span><\/pre>\nTo log our visualization to the Comet Platform, use the two lines of code above. Seaborn Visualization and Matplotlib both use the.log figure()<\/code>methods. Then we give our visualization a name.<\/p>\nRecall that we previously gave our visualizations the names fig1 and fig2, which we then pass to the figure parameter.<\/p>\n
If you are using a notebook, such as Jupyter or Colab, you must type the experiment.end ()<\/code> after your test.<\/p>\nWe can now view our experiments on the project session on the website<\/a>.<\/p>\nConclusion<\/h2>\n
There are 3 main types of Recommendation systems<\/p>\n
- \n
- Content-Based Recommendation system.<\/li>\n
- Popularity-Based Recommendation system.<\/li>\n
- Collaborative Recommendation system.<\/li>\n<\/ol>\n
Content-Based Recommendation system:<\/p>\n
It is a type of recommendation system that operates on a similar content principle. If a user is watching a movie, the system will look for other movies with similar content or in the same genre as the one the user is watching. When comparing similar content, various fundamental attributes are used to compute similarity.<\/p>\n
Popularity-Based Recommendation System:<\/strong><\/h2>\n
It is a type of recommendation system that operates based on popularity of anything that is currently popular. These systems examine the products or movies that are popular among users and directly recommend them.<\/p>\n
For instance, if a product is frequently purchased by the majority of people, the system will learn that it is the most popular, so for every new user who has just signed up, the system will recommend that product to that user as well, and the chances are that the new user will also purchase that.<\/p>\n
Collaborative Recommendation system<\/strong><\/h2>\n
Collaborative recommendation, also called collaborative filtering, works by analyzing the preferences and data of numerous users. Collaborative filtering predicts a user\u2019s interests. This is accomplished by applying techniques involving cooperation among numerous agents, data sources, etc. to filter data for information or patterns. The underlying premise of collaborative filtering is that users A and B are likely to have similar tastes in products if they have similar tastes in one product.<\/p>\n
Comet<\/strong><\/h2>\n
Comet is a platform for experimentation that enables you to monitor your machine-learning experiments. Comet has another noteworthy feature: it allows us to conduct exploratory data analysis. We can accomplish our EDA objectives thanks to Comet\u2019s integration with well-known Python visualization frameworks. You can learn more about comet here.<\/a><\/p>\n
This lesson will teach us how to connect Comet with our recommendation system. We will carry out some EDA on our movie dataset to achieve this, and we will log the visualization onto the Comet experimentation website or platform. Let\u2019s begin without further ado.<\/p>\n
Prerequisites<\/strong><\/h2>\n
You may install the Comet library on your computer if you don\u2019t already have it there by using the following line at the command prompt.<\/p>\n
pip install comet_ml<\/span><\/pre>\n\u2014 or \u2014<\/p>\n
conda install -c comet_ml<\/span><\/pre>\nAbout the data<\/strong><\/h2>\n
All 45,000 films mentioned in the Full MovieLens Dataset are represented by these files\u2019 information. Movies that were released on or before July 2017 are included in the dataset. Cast, crew, narrative keywords, budget, revenue, posters, release dates, language breakdowns, production firms, nations, and TMDB vote counts and averages are just a few examples of the data points.<\/p>\n
Additionally, this collection includes files with 26 million user ratings for all 45,000 movies, collected from 270,000 individuals. A scale of 1 to 5 stars was used to rate each item on the GroupLens official website. You can get the data from Kaggle here.<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n\n\nReal-time model analysis allows your team to track, monitor, and adjust models already in production. Learn more lessons from the field with Comet experts<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n\n\nMovie Recommendation system<\/strong><\/h2>\n
The first step involves importing the dependencies.<\/p>\n
import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport difflib\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom sklearn.metrics.pairwise import cosine_similarity<\/span><\/pre>\nWe then move to load our dataset into the pandas dataframe.<\/p>\n
movie_data = pd.read_csv('\/content\/jmovies.csv')<\/span><\/pre>\nWe then move to the next step, which is data preprocessing.<\/p>\n
movie_data.head()<\/span><\/pre>\n\n \n<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\n movie_data.info()<\/span><\/pre>\n\n <\/figure><\/picture><\/div>\n<\/figure>\n movie_data. isnull()<\/span><\/pre>\nThe movie_data. isnull() function returns false if the values in the dataset are not missing and return true if they are missing.<\/p>\n
\n \n<\/figure><\/picture><\/div>\n<\/div>\n<\/figure>\n The movie_data. isnull().head() function only gives us the details on the first 5 columns, but to know the exact column and how many values are missing, we use the movie_data. isnull().sum() function tells us which column has missing values and the number of missing values it has.<\/p>\n
movie_data. isnull().sum()<\/span><\/pre>\n\n <\/figure><\/picture><\/div>\n<\/figure>\n We have a lot of elements in our dataset and not all can be used for our recommendation system, so we select only those that can be used. We also pick elements that will affect the accuracy of our recommendation system. Users are most likely to watch a movie based on the class of genres, the type of cast in the movie, and also the director in charge of the movie<\/p>\n
From our data preprocessing, we can see that we have a lot of missing values in our dataset, which will affect the result of our recommendation system. We need to fill it up with null strings, but before we do that, we need to perform some EDA.<\/p>\n
movie_data[['keywords','tagline','cast','director']].fillna('')<\/span><\/pre>\nWe then create a variable named “combined features” input all the selected features and print them. This is done so we can input it as a whole into our vectorizer, which converts the text to feature vectors.<\/p>\n
combined_features= movie_data['genres']+' '+movie_data['keywords'] \\\n+' '+movie_data['tagline']+' '+movie_data['cast'] \\\n+' '+movie_data['director']<\/span><\/pre>\nThe next step involves converting our text data to numerical data and this is done so it can be understood by the cosine similarity, which works well with numerical data we be using TfidfVectorizer to convert.<\/p>\n
vectorizer = TfidfVectorizer()\nfeature_vectors = vectorizer.fit_transform(combined_features)<\/span><\/pre>\nSo once the numerical data is ready, we then try and get the similarity scores using cosine similarity, and we then load our feature vector into the cosine similarity to find movies that are similar to each other. So what the cosine similarity does is iterate through the list and try to find similarities between the selected character and the remaining movies in the dataset, it does this for each of the movies.<\/p>\n
similarity = cosine_similarity(feature_vectors)<\/span>print(similarity)\nprint(similarity.shape)<\/span><\/pre>\nTo get our recommendation system working we need to create an input section for the user to input the name of their selected movie.<\/p>\n
movie_name = input(' Enter your favourite movie name : ')<\/span><\/pre>\nWe then need to create a list of the names of all movies in the dataset.<\/p>\n
list_of_all_titles = movie_data['title'].tolist()\nprint(list_of_all_titles)<\/span><\/pre>\nThe next step is finding a movie match for the movie name that the user inputs.<\/p>\n
find_close_match = difflib.get_close_matches(movie_name,\n list_of_all_titles)\nprint(find_close_match)<\/span><\/pre>\nThe above code gives us a list of related movies but we need to narrow it down to the exact movie the user inputs.<\/p>\n
close_match = find_close_match[0]\nprint(close_match)<\/span><\/pre>\nTo get the index of each movie we use the title of the movie to find the index<\/p>\n
index_of_the_movie = movie_data[movie_data.title == close_match]['index'].values[0]\nprint(index_of_the_movie)<\/span><\/pre>\nSo, using the index value, we will be getting a list of similar values. So when you run this code you\u2019ll notice the index of the movie.<\/p>\n
similarity_score = list(enumerate(similarity[index_of_the_movie]))\nprint(similarity_score)<\/span><\/pre>\nWe then print the length of the similarity score.<\/p>\n
len(similarity_score)<\/span><\/pre>\nWe then sort the movie based on the similarity score. So the same genres of movies are sorted together.<\/p>\n
sorted_similar_movies = sorted(similarity_score, key = lambda x:x[1], reverse = True)\nprint(sorted_similar_movies)<\/span><\/pre>\nThen we work on printing a list based on the similarity scores.<\/p>\n
print('Movies suggested for you : \\n')<\/span>i = 1<\/span>for movie in sorted_similar_movies:\n index = movie[0]\n title_from_index = movies_data[movies_data.index==index]['title'].values[0]\n if (i<30):\n print(i, '.',title_from_index)\n i+=1<\/span><\/pre>\nSo we can see that our recommendation system is up and running and the movie\u2019s recommendation works just fine. So the next step includes logging our visualization on the comet platform.<\/p>\n
So we need to visualize some of the characters in our dataset using matplotlib and seaborn.<\/p>\n
mask_df = movie_data[\"director\"].value_counts().head(10)\nfig1 = plt.figure(figsize=(12, 10))\nplt.bar(x = mask_df.index, height=mask_df, color=\"sienna\")\nplt.xlabel(\"director\")\nplt.ylabel(\"budget\")\nplt. title(\"Budget of each director per movie\")\nplt.xticks(rotation=45)<\/span><\/pre>\nSo the above code shows the budget of each director per movie and we can see how the budget for each movie varies per director.<\/p>\n
In the next step, we input the budget per movie, and we can see which of the movies has the highest budget.<\/p>\n
fig2 = plt.figure(figsize=(12, 10))\nsns.histplot(movie_data[\"budget\"], color=\"darkslategrey\", bins=50)\nplt.title(\"Budget of Movie\");<\/span><\/pre>\nThe next step will be to log the visualization onto the Comet platform once we have finished it. In this section, we\u2019ll use the Comet experiment library to build a brand-new project.<\/p>\n
You will require a Comet API key to continue with this phase. Click here to sign up for their platform if you haven\u2019t already. By clicking on your profile photo, selecting the settings icon, and then scrolling down, as demonstrated below, you can find your API key.<\/p>\n
\n \n<\/figure><\/picture><\/div>\n<\/div>\n Image by Author<\/figcaption>\n<\/figure>\n from comet_ml import Experiment<\/span># Create an experiment with your API key<\/span>experiment = Experiment(\napi_key=\"xGZbEEC8bB2VnpxWFcWXd3BI3\",\nproject_name=\"Recommendation system\",\nworkspace=\"olujerry\",\n)\nexperiment.log_figure(figure_name=\"Matplotlib Viz\", figure=fig1)\nexperiment.log_figure(figure_name= \"Seaborn Viz\", figure=fig2)<\/span><\/pre>\nWe imported the
Experiment<\/code> library fromcomet ml<\/strong><\/code> into the code above. We then instantiated it and assigned it to the experiment variable. Your API key, the name of your project, and the name of your workspace are some of the parameters needed by the Experiment library. The Comet platform will display the workspace name, such as:<\/p>\n\n \n<\/figure><\/picture><\/div>\n<\/div>\n Image By Author<\/figcaption>\n<\/figure>\n experiment.log_figure(figure_name=\"Matplotlib Viz\", figure=fig1)experiment.log_figure(figure_name= \"Seaborn Viz\", figure=fig2)experiment.end()<\/span><\/pre>\nTo log our visualization to the Comet Platform, use the two lines of code above. Seaborn Visualization and Matplotlib both use the
.log figure()<\/code>methods. Then we give our visualization a name.<\/p>\nRecall that we previously gave our visualizations the names fig1 and fig2, which we then pass to the figure parameter.<\/p>\n
If you are using a notebook, such as Jupyter or Colab, you must type the
experiment.end ()<\/code> after your test.<\/p>\nWe can now view our experiments on the project session on the website<\/a>.<\/p>\n
Conclusion<\/h2>\n