Data Diversions

Jun 15

The Best of TV: Analyzing IMDB Data on 90,000 TV Series

Founded in 1990, IMDB is a treasure trove of data with “…2,882,569 titles (includes episodes) and 5,909,020 personalities in its database.” Using a simple scraper written in Ruby, I was able to grab data on the >90,000 TV shows in the IMDB database. This is what I found.

 
There has been significant growth in the number of new television series created each year:
image

 
Here is the same chart from 2000 forward:
image


It is important to note that our collective record-keeping has improved since IMDB’s launch in 1990, and with the rise of internet usage generally (and IMDB specifically) over the past decade. IMDB’s Google Trends chart

image

War and history series are the highest-rated genres on average. Among common genres, documentaries, animated series, and dramas are the highest-rated. Game shows and reality TV are rated significantly lower than other genres:
image

Certain genres have grown or faded significantly since 1950:
image

For example, westerns have almost completely disappeared:
image

The number of reality TV shows has had an astonishing rise starting in the late 90s, but has thankfully flattened out:
image

The prevalence of fantasy shows has been relatively steady over time, but has made a comeback in the past few years:
image

The overall quality of TV peaked in the 1960s, declined through 2004, and has since steadily improved (although I would take this chart with several grains of salt):
image

Here is the same chart from 1990 through today:

image

The top ten highest-rated television shows of all time are as follows:

1. Breaking Bad  (2008): 9.6
2. Game of Thrones  (2011): 9.5
3. Planet Earth  (2006): 9.5
4. Cosmos: A SpaceTime Odyssey  (2014): 9.5
5. The World at War  (1973): 9.5
6. The Wire  (2002): 9.4
7. True Detective  (2014): 9.4
8. Sherlock  (2010): 9.3
9. The Sopranos  (1999): 9.3
10. Cosmos  (1980): 9.3


The top TV series that have debuted since 2010 (note that I did *not* filter out shows with a small number of ratings):

1. Game of Thrones  (2011): 9.5
2. Cosmos: A SpaceTime Odyssey  (2014): 9.5
3. True Detective  (2014): 9.4
4. Sherlock  (2010): 9.3
5. Rick and Morty  (2013): 9.3
6. Africa  (2013): 9.3
7. Kardes Payi  (2014): 9.3
8. Leyla and Mecnun  (2011): 9.2
9. Fargo  (2014): 9.2
10. 24: Live Another Day  (2014): 9.2


Best of the 2000s:

1. Breaking Bad  (2008): 9.6
2. Planet Earth  (2006): 9.5
3. The Wire  (2002): 9.4
4. The Life of Mammals  (2002): 9.3
5. Al-Taghriba Al-Filistinia  (2004): 9.2
6. Arrested Development  (2003): 9.2
7. Firefly  (2002): 9.2
8. Life in the Undergrowth  (2005): 9.2
9. Life in Cold Blood  (2008): 9.2
10. Avatar: The Last Airbender  (2005): 9.1

Best of the 90s:

1. The Sopranos  (1999): 9.3
2. The Private Life of Plants  (1995): 9.2
3. The Trials of Life  (1990): 9.2
4. The Life of Birds  (1998): 9.1
5. Friends  (1994): 9.0
6. Twin Peaks  (1990): 9.0
7. Freaks and Geeks  (1999): 9.0
8. Batman: The Animated Series  (1992): 9.0
9. Cowboy Bebop  (1998): 9.0
10. Ice TV  (1996): 9.0

Best of the 80s:

1. Cosmos  (1980): 9.3
2. The Living Planet  (1984): 9.2
3. The Simpsons  (1989): 9.0
4. Seinfeld  (1989): 9.0
5. Dragon Ball Z  (1989): 8.9
6. Only Fools and Horses….  (1981): 8.9
7. Blackadder Goes Forth  (1989): 8.9
8. Fauji  (1988): 8.9
9. Solunski patrdii  (1985): 8.9
10. The Morning Program  (1987): 8.9

Best of the 70s:

1. The World at War  (1973): 9.5
2. Life on Earth  (1979): 9.2
3. I, Claudius  (1976): 9.1
4. La zulianita  (1977): 9.0
5. Doña Bárbara (1975): 9.0
6. Fawlty Towers  (1975): 8.9
7. Matador  (1978): 8.9
8. El chavo del ocho  (1972): 8.8
9. The Man and the City  (1971): 8.8
10. The Julie Andrews Hour  (1972): 8.8

Best of the 60s:

1. Monty Python’s Flying Circus  (1969): 9.0
2. Fujimaru of the Wind  (1964): 9.0
3. The Prisoner  (1967): 8.8
4. Nu, pogodi!  (1969): 8.8
5. Ferien in Lipizza  (1966): 8.8
6. The Great Adventure  (1963): 8.8
7. Alle Hunde lieben Theobald  (1969): 8.8
8. Hafenpolizei  (1963): 8.8
9. Landarzt Dr. Brock  (1967): 8.8
10. Die Gäste des Felix Hechinger (1964): 8.8

Best of the 50s:

1. Twilight Zone  (1959): 8.9
2. Walt Disney Presents: Annette  (1958): 8.8
3. Corky and White Shadow  (1956): 8.8
4. Mayor of the Town  (1954): 8.8
5. The Spade Cooley Show  (1958): 8.8
6. Alfred Hitchcock Presents  (1955): 8.7
7. Mr. Squiggle and Friends  (1959): 8.7
8. Studio 57  (1954): 8.7
9. J.P. Patches  (1958): 8.7
10. The Honeymooners  (1955): 8.6

Best of the 40s:

1. Mama  (1949): 8.5
2. Mary Kay and Johnny  (1947): 8.5
3. Kukla, Fran and Ollie  (1947): 8.2
4. The Howdy Doody Show  (1947): 8.1
5. Stump the Stars  (1947): 8.1
6. Fireside Theatre  (1949): 8.1
7. The Lone Ranger  (1949): 8.0
8. The Goldbergs  (1949): 8.0
9. Time for Beany  (1949): 8.0
10. The Aldrich Family  (1949): 8.0

May 03

Movie Scripts Ranked by Flesch-Kincaid Grade Level

Remember the Flesch Kincaid Grade Level? If you ever ran Microsoft Word’s spell check in in the 90’s, you probably do:

image

The Flesch and Flesch-Kincaid tests are “readability tests designed to indicate comprehension difficulty when reading a passage of contemporary academic English.” The scores incorporate two primary factors: the average number of words per sentence, and the average number of syllables per word. A higher Flesch-Kincaid Grade Level is supposed to reflect more difficult text, with longer sentences and more multi-syllabic words.

The Ruby gem Lingua makes it quite easy to perform readability analysis against text. And with more than 950 scripts available on the Internet Movie Script Database, I had the raw ingredients necessary to run a fun and entirely inane analysis of the Flesch-Kincaid scores of movies. Here are the results.

Movies with the lowest Flesch-Kincaid grade level (simpler):

  1. Glengarry Glen Gross: 0.11
  2. Win Win: 0.13
  3. Last Tango in Paris: 0.25
  4. Silver Linings Playbook: 0.45 [Oscar winner, Best Adapted Screenplay]
  5. Gran Torino: 0.54
  6. Next Friday: 0.60
  7. Hannibal: 0.64
  8. Sling Blade: 0.65
  9. Frozen: 0.66
  10. Boogie Nights: 0.68 [Oscar nominee, Best Original Screenplay]

image


Movies with the highest Flesch-Kincaid grade level (more complex):

  1. Fletch: 5.94
  2. The Adventures of Buckaroo Banzai Across the Eighth Dimension: 5.48
  3. The Ninth Gate: 5.07
  4. L’Avventura (The Adventure):  4.74
  5. Mulan: 4.31
  6. Omega Man: 4.18
  7. The Last Samurai: 4.17
  8. Gattaca: 3.99
  9. The Saint: 3.76
  10. American Psycho: 3.76

image


If you’re interested in seeing the full list, click here. This list includes two additional scores: the Flesch Reading Ease score, which uses slightly different weights (with a lower score indicating more readable text), and the Gunning fog index, which incorporates average sentence length and the density of words with three or more syllables.

A few caveats: first, this list is far from complete. In fact, there were ~100 scripts on the IMSDb that have been taken down (e.g., most of the Batman and all of the Harry Potter movies).

Second, the analysis is unfairly favorable to scripts with detailed stage directions, which tend to be more complex than dialogue. Here is the first passage from the “most sophisticated” movie, Fletch:

Seagulls squawk, and the waves pound, but we’re not talking about Malibu Colony, here. This is a fairly rundown beach area, catering to lower-echelon surfers, vagrants, and strung out druggies of all ages, several of whom stand or sit on their haunches by a dilapidated old hamburger stand. Over the stand is a faded sign: “FAT SAM’S HAMBURGERS”.

In any case, enjoy. In the immortal words of John McClane:

Yippe-ki-yea