Book Recommendation Engine using KNN

Note

The hard part of this problem is to understand how to use KNN from sklearn. The concept of KNN is easy to understand (find nearest neighbors in multi dimension space). After reading several tutorials, I finally found the 2 most useful lines that help me use it correctly.

# pivot ratings into book features
df_book_features = df.pivot(
    index='isbn',
    columns='user',
    values='rating'
).fillna(0)

# convert dataframe of book features to scipy sparse matrix
# this part requires a lot of memory!
mat_book_features = csr_matrix(df_book_features.values)

Total time for this: 4h

Problem description

Content is copied from FCC Google Colab link:

https://colab.research.google.com/github/freeCodeCamp/boilerplate-book-recommendation-engine/blob/master/fcc_book_recommendation_knn.ipynb

In this challenge, you will create a book recommendation algorithm using K-Nearest Neighbors.

You will use the Book-Crossings dataset. This dataset contains 1.1 million ratings (scale of 1-10) of 270,000 books by 90,000 users.

After importing and cleaning the data, use NearestNeighbors from sklearn.neighbors to develop a model that shows books that are similar to a given book. The Nearest Neighbors algorithm measures distance to determine the “closeness” of instances.

Create a function named get_recommends that takes a book title (from the dataset) as an argument and returns a list of 5 similar books with their distances from the book argument.

This code:

get_recommends("The Queen of the Damned (Vampire Chronicles (Paperback))")

should return:

[
  'The Queen of the Damned (Vampire Chronicles (Paperback))',
  [
    ['Catch 22', 0.793983519077301],
    ['The Witching Hour (Lives of the Mayfair Witches)', 0.7448656558990479],
    ['Interview with the Vampire', 0.7345068454742432],
    ['The Tale of the Body Thief (Vampire Chronicles (Paperback))', 0.5376338362693787],
    ['The Vampire Lestat (Vampire Chronicles, Book II)', 0.5178412199020386]
  ]
]

Notice that the data returned from get_recommends() is a list. The first element in the list is the book title passed in to the function. The second element in the list is a list of five more lists. Each of the five lists contains a recommended book and the distance from the recommended book to the book passed in to the function.

If you graph the dataset (optional), you will notice that most books are not rated frequently. To ensure statistical significance, remove from the dataset users with less than 200 ratings and books with less than 100 ratings.

The first three cells import libraries you may need and the data to use. The final cell is for testing. Write all your code in between those cells.

Solutions

We gonna graph a few things, so let config matplotlib to have nicer and bigger output first

Let's take a look at the data first.

Let's try to make a scatter plot for the number of rating of each user

From above fraph, we can see the outlier who rarely rate the book.

I tried to plot the similar graph for the book, but the kernel keep failing. I guess it's because there's more unique books than users, so the server run our of memory (or time) while making the image. Let's keep it for now.

Now, we first remove books that has less than 100 review. We have to do it before remove the users who submit less than 200 review.

After remove unqualifed books, we can now remove unqualifed users

For more efficient calculation and less memory footprint, we need to transform the values of the dataframe into a scipy sparse matrix.

Create and fit the model with our book features

Now we can implement the function.

Use the cell below to test your function. The test_book_recommendation() function will inform you if you passed the challenge or need to keep trying.

Test