IMDB 5000 Movies
Background
How can we tell the greatness of a movie before it is released in cinema?
This question puzzled me for a long time since there is no universal way to claim the goodness of movies. Many people rely on critics to gauge the quality of a film, while others use their instincts. But it takes the time to obtain a reasonable amount of critics review after a movie is released. And human instinct sometimes is unreliable.
Question
- Given that thousands of movies were produced each year, is there a better way for us to tell the greatness of movie without relying on critics or our own instincts?
- Will the number of human faces in movie poster correlate with the movie rating?
Method
To answer this question, I scraped 5000+ movies from IMDB website using a Python library called "scrapy".
The scraping process took 2 hours to finish. In the end, I was able to obtain all needed 28 variables for 5043 movies and 4906 posters (998MB), spanning across 100 years in 66 countries. There are 2399 unique director names, and thousands of actors/actresses. Below are the 28 variables:
- "movie_title"
- "color"
- "num_critic_for_reviews"
- "movie_facebook_likes"
- "duration"
- "director_name"
- "director_facebook_likes"
- "actor_3_name"
- "actor_3_facebook_likes"
- "actor_2_name"
- "actor_2_facebook_likes"
- "actor_1_name"
- "actor_1_facebook_likes"
- "gross"
- "genres"
- "num_voted_users"
- "cast_total_facebook_likes"
- "facenumber_in_poster"
- "plot_keywords"
- "movie_imdb_link"
- "num_user_for_reviews"
- "language"
- "country"
- "content_rating"
- "budget"
- "title_year"
- "imdb_score"
- "aspect_ratio" To answer question 2, I applied the human face detection algorithm on all the posters using python library called dlib, and extracted the number of faces in posters.
탐색적 분석¶
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import datetime as dt
In [2]:
%matplotlib inline
- How people can know rating(like greatness of moives) before watching a moive
- without critics or our own instincts? It's the point of this Analysis
In [3]:
movies = pd.read_csv("data/movie_metadata.csv")
In [4]:
movies.head()
Out[4]:
In [5]:
movies.columns
Out[5]:
In [6]:
movies['content_rating'].value_counts() # content rating -> 상영등급.
Out[6]:
상영등급별 분포¶
In [7]:
sns.factorplot('content_rating',kind='count',data=movies, size=8)
plt.xticks(rotation=45)
Out[7]:
최대,최소 평점¶
In [8]:
movies['imdb_score'].max() # 10 is highest, maximun of rating is 9.5 in this db
Out[8]:
In [9]:
movies['imdb_score'].min() # 0 is lowest, minimun of rating is 1.6 in this db
Out[9]:
In [10]:
sns.factorplot(y='imdb_score',kind='box',data=movies)
Out[10]:
우선 영향도를 보기 위해서 평점을 반올림을 통해서 구분이 잘되도록 변환¶
In [11]:
df = movies.copy()
df['imdb_score'] = df['imdb_score'].apply(lambda x:int(round(x)))
In [12]:
df['imdb_score'].value_counts()
Out[12]:
In [13]:
sns.factorplot('imdb_score',kind='count',data=df, size=6)
Out[13]:
IMDB Score VS Country¶
In [14]:
tmp = df['language'].value_counts()
language_list = tmp[tmp>3].index.tolist()
In [16]:
sns.boxplot(y='imdb_score',x='language',data=df[df['language'].isin(language_list)])
plt.xticks(rotation=45)
Out[16]:
IMDB Score vs Movie Year¶
- 년도가 높아 질수록 영화의 평점 수는 갭도 커지면서, 아마 엔터 위주의 영화만 출시 되는 경향이 있는 듯하다.
- 년도가 높을 수록 Score는 낮다.
In [17]:
title_year_df = df[['title_year','imdb_score']]
title_year_df = title_year_df.dropna()
title_year_df['title_year'] = title_year_df['title_year'].astype(int)
In [18]:
sns.factorplot(y='imdb_score',x='title_year',data=title_year_df,kind='box', size=10)
plt.xticks(rotation=45)
Out[18]:
IMDB Score vs Facebook Popularity¶
- Number of facebook likes effect high rating of imdb_score
- Correlation : 0.24
In [19]:
sns.boxplot(x='imdb_score',y='movie_facebook_likes',data=df)
Out[19]:
In [20]:
df[['movie_facebook_likes','imdb_score']].corr()
Out[20]:
Correlation analysis With Continuous Varibles¶
In [21]:
columns = df.columns
discrete = []
continuous = []
for i in columns:
if df[i].dtype =='object':
discrete.append(i)
else:
continuous.append(i)
In [22]:
# sns.pairplot(df[continuous].dropna(),kind="reg")
In [23]:
df_corr = df[continuous].dropna().corr()
연속변수 상의 Correlation¶
- Number of Critic for reviews
- Duration
- Gross
- Number of User Reviews
- Number of voted Users 가 긍정적인 영향을 미쳤다.
In [24]:
fig, ax = plt.subplots(figsize=(10,10))
sns.heatmap(df_corr,
xticklabels=df_corr.columns.values,
yticklabels=df_corr.columns.values,
annot=True, linewidths=.5, ax=ax)
Out[24]:
In [25]:
df['facenumber_in_poster'].head()
Out[25]:
In [26]:
df2 = df.dropna().copy()
In [27]:
df2['facenumber_in_poster'] = df2['facenumber_in_poster'].astype(int)
In [28]:
sns.factorplot('facenumber_in_poster',kind='count',data=df2, size=6)
Out[28]:
In [29]:
realtionOfFN = df[['facenumber_in_poster','imdb_score']].dropna().corr()
In [30]:
realtionOfFN
Out[30]:
Histogram of IMDB scores(0~10)¶
In [31]:
sns.set(rc={"figure.figsize": (8, 6)});
sns.distplot(movies['imdb_score'])
Out[31]:
Delete columns don't need¶
In [32]:
del df['movie_imdb_link']
del df['color']
Relation with director_name and imdb_score¶
- A : A little relation with each other but... not a lot
number of directors¶
- 2399
In [33]:
len(df['director_name'].unique())
Out[33]:
In [34]:
tmp_X = pd.get_dummies(df['director_name'])
tmp_X['imdb_score'] = df['imdb_score']
In [35]:
df_corr = tmp_X.corr()
In [36]:
df_corr['imdb_score'][:5]
Out[36]:
In [37]:
corr = df_corr['imdb_score']
corr[corr>0.05]
Out[37]:
In [38]:
director_name_list = corr[corr>0.05].index.tolist()
In [39]:
director_name_list
Out[39]:
In [40]:
df[df['director_name']=='Steven Spielberg'][['imdb_score','movie_title']]
Out[40]:
In [41]:
df['director_name'] = df['director_name'].astype(object)
In [42]:
sns.boxplot(y='imdb_score',x='director_name',data=df[df['director_name'].isin(director_name_list)])
plt.xticks(rotation=45)
Out[42]:
3. Relation with Genres¶
- Bad Effect : horror & comedy have a bad effection to rating of moives
- Good Effect : Biography, Documentary, Drama, History, War have good factors for rating of movies
In [43]:
genre_list = df['genres'].str.split('|')
In [44]:
genre_list[:5]
Out[44]:
In [45]:
# set 으로 활용하면 되려나 (사용은 안함.)
genre = set()
for i in range(len(genre_list)):
genre |= set(genre_list[i])
In [46]:
genre
Out[46]:
worst Way to make difference length list to DataFrame¶
In [49]:
genre_df = pd.DataFrame()
for i in range(len(genre_list)):
genre_df = genre_df.append(pd.DataFrame(genre_list[i]).T)
In [50]:
genre_df.head()
Out[50]:
- Way 1.
In [51]:
genre_df = pd.DataFrame(genre_list.values.tolist(), index=genre_list.index)
In [52]:
genre_df.head()
Out[52]:
In [53]:
genre_df = pd.DataFrame(genre_list.values.tolist(), index=genre_list.index).replace({None:np.nan})
In [54]:
genre_df.head()
Out[54]:
- Way 2
In [55]:
genre_df = genre_list.apply(pd.Series)
In [56]:
genre_df.head()
Out[56]:
Making dummy data¶
- Count Varibles like dummy coding
- way 1
In [57]:
pd.get_dummies(genre_df.stack()).groupby(level=0).max().head()
Out[57]:
- Way 2
In [58]:
genre_df.stack().groupby(level=0).value_counts().unstack(fill_value=0).head() # int
Out[58]:
In [59]:
genre_df.stack().groupby(level=0).value_counts().unstack().fillna(0).head() # float
Out[59]:
- Way 3
In [60]:
genre_df.apply(pd.value_counts, 1).fillna(0).astype(int).head()
Out[60]:
In [61]:
df_genre = genre_df.stack().groupby(level=0).value_counts().unstack(fill_value=0)
df_genre['imdb_score'] = df['imdb_score']
In [62]:
df_genre.head()
Out[62]:
In [63]:
genre_corr = df_genre.corr()
genre_corr = genre_corr['imdb_score']
In [64]:
genre_corr[abs(genre_corr)>0.1]
Out[64]:
