Early 2023 was a funny time in the history of generative AI. On November 30th 2022, OpenAI released a little research project known as ChatGPT. The launch of ChatGPT began the period where large language models properly entered the mainstream outside of tech enthusiasts and ended soon after the launch of ChatGPT API in March 2023 that spawned thousands of AI-powered apps. That was when the limitations and problems with LLMs also went mainstream, such as plagiarism, hallucinations, and low-quality slop replacing human-generated content at an objectively worse quality.

In December 2022, Mismatch Media started a fully AI-generated 24/7 Twitch channel dubbed “WatchMeForever”. The primary show on the channel was titled “Nothing, Forever”, an AI-powered sitcom about New York comedian Larry Feinberg and his group of friends hanging around in their apartments talking about pretty much anything, including the latest news, new restaurants, and bad relationships, interspersed with AI standup comedy routines.

It was obvious that the show was a parody of the formative 90’s sitcom Seinfeld created by comedians Larry David and Jerry Seinfeld, famously “a show about nothing” strongly inspired by improv comedy and starring Seinfeld himself.

The show, dubbed “AI Seinfeld” by the community, used a script powered by the GPT-3 API, the voices were powered by Microsoft’s Azure AI Speech API with predefined voices from their Voice Gallery, and the scenes were rended using the Unity game engine along with purchased models/scenes/sounds/etc from the Unity Asset Store.

AI Seinfeld was interestingly imperfect: the laugh track fired at inappropriate times, the standup routine repeatedly made the same joke such as “What did the fish say when he hit the wall?” (Damn!), and awkward silences at the end of scenes.

In February 2023, AI Seinfeld quickly went viral organically after its AI weirdness was a surprising complement for Seinfeld’s style of weirdness, with many watchers being surprised at both its accuracy to the show and easily sharable metahumor. At its peak, AI Seinfeld had over 10,000 concurrent watchers on Twitch, putting it squarely in one of the top streams on the platform.

AI Seinfeld died as quickly as it rose: after a ban and subsequent revamp, the view count cratered, and as of August 2024, the Twitch stream hovers below 10 watchers, with no significant changes made since the previous year, and Mismatch Media has no social footprint since last year. Could there be another AI Seinfeld with the rapid advancements in generative AI? Unfortunately, there are too many factors — technical, societal, and comedic — working against a theoretical next-generation AI-generated sitcom.

The Rise of AI Seinfeld

AI Seinfeld launched before the release of the ChatGPT API; instead, they used the GPT-3 API, notably the text-davinci-003 model which was OpenAI’s first foray into instruction-tuned LLMs. While previous versions of GPT-3 were very good at autocompleting given a leading prompt such as a partial Seinfeld script, the instruction-tuned LLM could generate an episode with a prompt as simple as Write a Seinfeld episode.

First, let’s go back to the beginning, as AI Seinfeld actually wasn’t the first time a chatbot went megaviral on Twitch. In January 2017, long before the transformer architecture that enabled LLMs was published, the Twitch stream seebotschat featuring two Google Homes wired up to the not-an-LLM-chatbot Cleverbot went viral due to their comedic, nonsensical bickering.

While everyone watching that stream knew it really wasn’t AI, AI Seinfeld was a product that was at the peak of the famous uncanny valley curve, which is a hypothesis on how humans perceive imitations: there’s a “valley” of negative acceptance where the imitation is more above-average in its likeness, but not quite close enough to the real thing. In this case, it’s blatantly obvious and unambiguous that the Twitch stream was AI-generated especially with its mistakes, but not realistic enough that it falls into the valley itself:

This AI weirdness made it very easy to build a community. Whenever a character turned on the microwave, the Twitch channel chat was filled with MMM emotes, whenever the fish hit a wall during a monologue, it was filled with 🐠, whenever Larry greeted the audience at the start of his monologue, chat replied with “HI LARRY”. Twitch chat loves memetic repetition. Incidentally, a few months after AI Seinfeld became popular, it was discovered that LLMs repeat the same joke over and over again, with examples being similar to the jokes AI Seinfeld made.

Another underrated aspect of AI Seinfeld’s success is that it’s pure background noise. While personality-driven Twitch streams cause viewers to take a more active investment in what’s being shown on screen due to FOMO of a hype moment on stream, AI Seinfeld is 100% passive: there can be exciting events, but the variance is low. It’s akin to watching TV sitcom reruns where you’ve already seen the jokes, and reruns still get immense ratings.

The success of AI Seinfeld also inspired similar streams based on other TV shows. One of my personal favorites was Unlimited Steam, a parody of the memetic “Steamed Hams” scene from The Simpsons, except made infinite with AI generation. That may sound like a pointless idea — Steamed Hams has a very fixed plot — but it went off the rails even harder than AI Seinfeld ever did.

Directing AI Seinfeld

AI Seinfeld was novel back in 2023, but now that LLMs are more mainstream you can probably figure out how the AI part of it worked, but let’s do a refresher so we can figure out how a hypothetical future AI Seinfeld could innovate the algorithmic sitcom.

As noted earlier, the key of AI Seinfeld’s success was the then-latest version of GPT-3: text-davinci-003 and its then-novel instruction-based finetuning using RLHF. With that, you can give it a prompt such as:

You are a professional comedian. Write an award-winning script for an episode of Seinfeld about a new restaurant. Include audience laugh tracks when appropriate.

Due to the low context length of these earlier LLMs, that’s essentially all the prompt engineering you can do without limiting the length of the output. The model would then output something similar to this script (using the more modern Claude 3.5 Sonnet at temperature=0.0): ¹

[Scene: Jerry's apartment]

Jerry: So what's the deal with this new restaurant, "The Blank Plate"?

Elaine: Oh, I've heard about that place! Apparently, you don't order anything - the chef just brings you whatever he feels like making.

Jerry: What? So you're telling me I have to eat whatever some stranger decides?

[Audience laughter]

George: (entering) Hey, guess where I'm taking my date tonight? The Blank Plate!

Jerry: George, you can't take a date there! What if they serve something weird?

George: What do you mean?

Elaine: It's that new place where you don't get to choose your meal.

George: (panicking) Oh no, what have I done? She's going to think I'm some kind of food weirdo!

One thing instruction-tuned LLMs are always good at is playing along: LLMs generate text sequentially without the explicit ability to plan ahead, so it must work with what it’s given and what it has already generated. Coincidentally, this works perfectly with the improv comedy style of Seinfeld, where continuing the plot is more important than anything else, and the more ridiculous the situation becomes, that’s even better. It’s the rare case where LLM hallucination is actually a feature, not a bug.

To get the LLM output into a format suitable for a Twitch stream, a programmatic script can then parse the output: extracting and mapping the characters and their lines, applause directions, and, of course, replacing all mentions of Jerry with Larry and Seinfeld with Feinberg. This workflow was surprisingly difficult at the time since GPT-3 did not have many techniques to control the format of the output, hence why I suspect there are awkward pauses and other glitches. Each line can then be passed to Azure’s text-to-speech API to generate a distinct audio file, which can be played back in order in Unity.

In an interview with Polygon, Skyler Hartle of Mismatch media noted the presence of a “director” which likely handles the camera, scene transitions, and the microwave:

“In addition to the third party services we’ve used, we have a lot of proprietary generative algorithms that cause the show to be ‘formed’, so to be speak. We collectively call this logic the ‘director,’ as it is largely responsible for making sure all the individual pieces come together into a whole,” Hartle said via email. “It’s worth mentioning that we don’t generate the artwork or the laugh track — those are precanned assets, but we have ideas on how to do that in the future.”

The AI aspect of AI Seinfeld was counterintuitively the easiest part of the pipeline, which explains how quickly variants popped up. However, with the inability to tweak the LLM output much with the technology at the time, the stream may have hit a creative limit.

The Fall of AI Seinfeld

Vice also interviewed Hartle, who had an optimistic view of the future of AI Seinfeld:

“Our grounding principle was, can we create a show that can generate entertaining content forever? Because that’s truly where we see the future emerging towards. Our goal with the next iterations or next shows that we release is to actually trade a show that is like Netflix-level quality.”

That’s tempting fate a bit too much.

The reason AI Seinfeld fell out of favor is a case of unintentionally poor LLM testing. When the text-davinci-003 model API endpoint had an outage, AI Seinfeld switched to a weaker GPT-3 model, text-curie, to keep the stream up. But unlike the davinci variant, curie was not RLHFed to follow instructions and safety.

During this brief period of low safety, one of Larry’s AI-generated monologues made a transphobic joke: a type of joke that was unfortunately common during the 90’s and has no place in modern society. Twitch banned the Watch Forever channel for 14 days as a result, completely killing the channel’s growth momentum.

But when the ban concluded and AI Seinfeld came back, the show was changed significantly with a “Season 2”. Although AI Seinfeld was still about a group of friends hanging around talking about the latest gossip, all the characters were different and had new models, the sets were different, and instead of a comedy monologue, Larry Leo narrates writing a blog.

Why Mismatch Media made such a format shift is unclear: Occam’s razor would suggest that a copyright holder for Seinfeld sent a cease and desist to Mismatch Media given the bad publicity behind the original ban, despite the clearly fair-use parody nature of the stream. It’s fair that it may not have been worth the time and effort for Mismatch Media to fight a legal battle for a fun art project.

The rebooted WatchMeForever stream is still active as of today, but with effectively no viewers.

The immediate failure of the AI Seinfeld retool does lend credibility to the theory that the stream only became popular because it was about Seinfeld and that it was a novelty doomed to a short shelf life. Still, there were detractors that said AI Seinfeld was never funny and everyone is weird for liking it. That’s ok: the original Seinfeld received similar complaints back in the day. ² But it’s hard to argue that there wasn’t interest in a 24/7 livestream of surreal AI-generated content.

What Would AI Seinfeld Look Like in 2024?

Now that we know how AI Seinfeld worked and what didn’t work, how would a year’s worth of exponential progress in generative AI look for AI Seinfeld? Could AI Seinfeld be improved and come back? The answer is maybe.

Modern generative AI requires a lot of cherry picking the best results, and it’s surprisingly hard to do: both images and text can take multiple generations and still require significant human-guided edits. But with a Twitch livestream, there can’t be any cherry picking at all, which means that the entire generation pipeline has to be consistent, and its failures interesting in the worst case.

The only reason AI Seinfeld worked at all is because GPT-3 was trained on the entire internet, likely including Seinfeld scripts and forum discussions. The prompt would need to have contained Write a Seinfeld script since if you asked it Write a sitcom script, it would output something completely generic instead and there isn’t much room to customize the prompt to make it more interesting. The GPT-3 variant that AI Seinfeld used had a 4k token context window limit (combined both the input prompt plus the output script text), but modern LLMs eclipse that: currently, Claude 3.5 Sonnet has a 200k input/8k output context, while GPT-4o has a 128k input/16k output context! With that much freedom, you can define many more constraints in the prompt and guide the LLM into exactly the type of sitcom you want.

One simple example that doesn’t require any knowledge of machine learning is a parametric prompt, where one aspect of a prompt can be replaced with a user-defined choice or programmatically chosen at random. Unlimited Steam, for example, used a parametric prompt to change the cosmic phenomenon localized entirely within Principal Skinner’s kitchen from the Aurora Borealis to something as silly as The Rapture or Foo Fighters, and forcing the characters to play along improv-style. ³

Recall the example Seinfeld prompt to Claude 3.5 Sonnet I made earlier, except instead of prompting for a Seinfeld script about a new restaurant, we can instead have the LLM generate a script about generative AI in 2024 by making only that substitution in the prompt — a topic on which there definitely has never been a Seinfeld episode about:

George bursts in, looking frantic.

George: You guys! I think I'm dating an AI!

[Audience laughter]

Jerry: What makes you say that?

George: Well, she's perfect in every way. She laughs at all my jokes, she has an encyclopedic knowledge of obscure trivia, and she never gets mad when I say something stupid.

[Audience laughter]

Jerry: George, that doesn't mean she's an AI. Maybe she just really likes you.

George: No, no. It's definitely an AI. No human woman would ever put up with me for this long.

Using modern LLMs, is there now a way to design a prompt which can make use of the long context windows? A prompt that can both leverage unique human writing and fix many of the issues that affected AI Seinfeld? Here’s an approach at a much more sophisticated prompt, where all values in {} brackets are parameters that can be filled in:

You are a professional comedian. Write an award-winning script for a a scene for Act I of a three act hit sitcom episode. Include audience laugh tracks when appropriate.

Your script MUST incorporate ALL the following elements:

Background:
- {background}

Setting:
- {setting}

Characters:
- {character_1}
- {character_2}
- {character_3}

Plots:
- {a_plot}
- {b_plot_1}
- {b_plot_2}

The script MUST also follow the high-level comedic style of the following scripts:

- {script_1}
- {script_2}
- {script_3}

After the scene has concluded, output a summary of the scene.

Thanks to long context windows, the parametric changes don’t have to be small, such as only a character name or two word setting. You, a human, can write anything to make each character distinct and robust, including name, gender, age, personality, likes, dislikes, etc. Plots can be derived from human-written scenarios beforehand: if you wrote 100 A-plots and 100 B-plots and randomly selected 1 A-plot and 2 B-plots, you’d have about 1 million possible plot permutations, ensuring you have something unique before the AI tries to reconcile them. You can feed in examples of human-written scripts to set the style and vibe of the generation in what is known as few-shot prompting. You can maintain continuity over many scenes by having the LLM summarize its own output, and then feed those summaries back to the AI as background information to build upon them. The LLM can also be instructed to output structured data to avoid the need to loosely parse the script after it’s completed, and as a bonus the model could be instructed to output additional metadata such as SSML speech styles based on a given line to add personality to the generated speech.

Unfortunately, creating this pipeline, writing original characters and plots for it for it, and sufficiently testing it to ensure the generated results are stable, would take weeks if not months to complete otherwise I would provide a more concrete demo. ⁴ This pipeline approach to AI script writing would only be effective for unsupervised 24/7 generation and wouldn’t replace skilled human writers who would do a more effective job much faster.

But would all of these prompt optimizations actually make the final generated script funny? After all, some of the failings like the awkward audience laughs and pauses and the end of scenes contributed to AI Seinfeld’s humor. During a standup comedy event at AI Seinfeld’s peak, Jerry Seinfeld himself was asked about the AI parody and he replied that he’s not worried about AI:

AI can be, definitely, they’ll make it smarter and smarter, but to do [standup comedy] you have to make it dumber.

Could AI Seinfeld benefit from advances in AI video? The answer this time is no. Generative video has been taking off in 2024 with projects such as OpenAI’s Sora and Runway AI’s Gen-3 Alpha, but those demos and the examples that go viral on social media are very heavily cherry picked, and even then there are consistency errors such as objects appearing in-and-out of existence. Generating video also requires exponentially more compute than just running Unity, and even with another few years of GPU hardware improvements it would be infeasible to cost-effectively create a 24/7 stream from those models.

The greatest problem with generative AI video is that it is coherent overall but has emblematic errors that don’t require a keen eye to notice, and as a result falls square into the uncanny valley, with its mistakes not being interesting, but disorienting. Mistakes in motion are easier to notice at a glance than images where a person’s hands may have the wrong number of fingers. The only way for AI video to get out of the valley would be to improve the model to near-flawless quality, which won’t happen any time soon. But Sora is more on the more realistic side of the curve than the less realistic side.

What about the AI-generated voices that would power these characters? At the time AI Seinfeld aired, many complained that Larry’s voice “didn’t sound enough like Jerry Seinfeld.” After AI Seinfeld concluded, a new technology called voice cloning popularized by ElevenLabs went mainstream…and it’s unexpectedly the AI modality that’s causing the most actual harm both with creative projects and outside of them. If you haven’t heard as much about AI-generated voices, there’s a good reason for that: voice synthesis projects such as Microsoft’s VALL-E 2 and Meta’s Voicebox both have disclaimers saying they won’t be released due to the dangers the technology possesses, although Microsoft’s Azure does offer a “custom neural voice” service. Voice cloning has been used to initiate scams by impersonating spouses in an emergency. Professional voice actors have had their voices cloned and used without compensation due to contracts not specifically forbidding the practice, which is one of the reasons SAG-AFTRA just went on strike against the video game industry in order to get protections against voice cloning and synthetic performers.

Moreover, in the context of creating a next-gen AI Seinfeld, there’s nothing inherently interesting about voice cloning since it’s a copy by definition: the model can’t generate unexpectedly amusing content other than the inherent gimmick of famous-voice-saying-something, such as the AI George Carlin standup special which was not special. There isn’t any way currently to prompt engineer a voice generation AI with the detail to create a voice in the style of a masculine New York comedian, 2x speed, primetime television quality which could open up more creative opportunities.

Although we can make drastic improvements with the textual script, that’s the extent of how new AI approaches can be leveraged to make something interesting. But if you remember the early days of generative AI history, the best AI-generated projects were the simplest.

AI Weirdness

Generative “AI” has been around for a very long time (I had fun with Markov chains a decade ago!), but the study was mostly confined to tech-focused communities like Hacker News. Modern generative AI didn’t break into mainstream culture until 2018, ironically in a way that doesn’t involve actual generative AI. In June of that year, comedian Keaton Patti posted a megaviral tweet about how he “forced a bot to watch over 1,000 hours of Olive Garden commercials and then asked it to write an Olive Garden commercial of its own.”

Yes, the script was human-written: for the technology at the time, no one could train an AI to behave like that from only video input data, and the script was too surreal even for the now-primitive generative AI. He did get popular enough to get a book deal and a Netflix collaboration leveraging this fake-AI gimmick.

Patti’s comedic misrepresentation of AI did lead to genuine confusion about what a 2018-era generative AI can actually do. Janelle Shane, who maintains the AI Weirdness blog about weird things AI can generate, posted an epic takedown of Patti’s script which went equally viral and also led to the internet discovering her excellent AI-generated Valentine’s Day hearts from the same year (and later a book deal too):

Image-based generative AI took a lot longer to go mainstream: websites like This Person Does Not Exist demonstrated the power of generative adversarial networks like StyleGAN to create images, but that wasn’t weird outside of mode collapses. The first instance of weird images from AI was in January 2021 when OpenAI announced the original DALL·E and showed they could make unique armchairs in the shape of an avocado by asking the model to do so, although they never released the model itself.

DALL·E didn’t get much attention outside of the AI hypesters since no one could play with it, but months later, things changed. Boris Dayma led an initiative to reproduce and open-source a variant of the DALL·E model, labeled DALL·E Mini (later changed to Craiyon after a cease and desist from OpenAI), and hosted it for free on Hugging Face and went megaviral. And thus began the “weird DALL·E” phase of image generation AI, where anyone could create incoherent images and make people laugh.

All of these examples of interesting failures are representative of a bygone AI era of experimentation. Once everyone had free access to more powerful text-generating AI with ChatGPT, and more powerful image-generating AI with Midjourney, AI stopped being fun and started being serious business, for better or for worse.

AI-Generated Content in 20XX

Last year, I wrote a thought piece titled “The Greatest Threat to Generative AI is Humans Being Bad at Using it” in response to the increasing hostility against the use of AI in creative works, arguing that while AI is a tool like anything else, it is a tool that’s very easy to use poorly and actually make projects worse. Additionally, the largest AI companies have both a business incentive and a duty to ensure that AI is used responsibly by its users downstream, as otherwise it will hurt the industry in the long term.

Now, it’s apparent that I was correct. The large companies went full steam ahead on AI integrations even where it is highly questionable that they add value and productivity to the end-user, often signaled with a “magical” sparkle emoji. Google has integrated Gemini to assist with document and email writing, Meta has integrated Meta AI to automatically generate images and comments, and Apple will soon allow Apple devices to generate text and images on your personal devices using Apple Intelligence. Marketing these features is typically met with backlash: Google had to pull an Olympics commercial which encouraged a parent to use AI to write a letter for their child.

“I flatly reject the future that Google is advertising,” Shelly Palmer, professor of advanced media at Syracuse University’s S.I. Newhouse School of Public Communications, wrote in a widely circulated blog post. The technology presents a “monocultural future where we see fewer and fewer examples of original human thoughts,” she wrote.

In the process of pushing AI tech further mainstream in a rush to demonstrate to shareholders their generative AI capabilities without encouraging responsible usage of the technology, AI has entered a new era of “slop” where people post objectively bad AI content without any regard for how it will be perceived, especially for websites which rely on user-generated content.

Facebook, whose algorithm favors emotionally-appealing engagement bait posts, has seen a deluge of high-engagement slop even when the content makes no logical sense.

This is, of course, quintessential uncanny valley: it’s coherent at a glance but just even looking at it for a second it’s obvious where the issues are, and these issues aren’t a good kind of AI weirdness. What worse is that AI Slop a regression in realism, and falls onto the left side of the valley.

Although we as humans can identify this slop, it is currently surprisingly hard for an AI to do so, although it hasn’t stopped people from trying to build AIs that can detect AIs which in practice is filled with false positives that hurt real creatives. For slop-creators, this is a feature: if an AI company released a tool to reliably detect and punish slop, it would make their generative AI less valuable. It’s reported that one of the reasons that OpenAI won’t release a reliable ChatGPT text detector is that it could harm their business.

The core reason for the big tech companies allowing generative AI to cause the enshittification of the internet is misaligned incentives between the companies hosting AI slop and the users viewing it. Social media companies and their shareholders care about North Star metrics such as user retention and time-on-site, and normally those metrics can be correlated with user happiness and satisfaction with the service. But time-on-site, for example, can also be maximized by making the site harder and slower to use, and the deluge of AI slop accomplishes that. AI companies typically don’t have analytics tracking negative user sentiment about their use of AI: if anything, the uncompromising backlash against AI convinces the companies that complainers are just a lost demographic to accommodate and double down on what they’re already doing. Aggregate metrics treat human-made content and AI-generated content as equal, but humans do not.

Generative AI, even for researchers and practitioners such as myself, is a heavily nuanced topic that is very difficult to communicate succinctly, more difficult to do on social media which highly discourages nuance and context, and even more difficult as AI hypesters muddy the waters with misleading praises of generative AI such that they’re easy to dunk on which just gets them more engagement and revenue. “Made by AI” is now a term that inspires dread, far from the Keaton Patti days where made-by-AI was an indicator of joyful weirdness. Bashing AI is now a meme, and there’s isn’t a single potential AI project that could challenge that perception because the well is poisoned beyond repair.

Would a 24/7 AI-Generated Twitch Stream Even Work Anymore?

How does the modern AI backlash tie back into AI Seinfeld? Twitch’s core demographic is the same demographic as those most against the use of generative AI. Part of the reason AI Seinfeld became so successful on Twitch is because of the community it cultivated: it wouldn’t have gone viral if people weren’t spamming microwave MMMs and and answering what did the fish say when it hit the wall. Even though Twitch viewers are mostly lurkers and not chatters, a channel with a good community builds word-of-mouth even outside of Twitch, which is how Twitch channels go viral.

I decided to determine what it would take to produce a “fixed” AI Seinfeld in 2024, given both the advances in AI and the ethics involved. Now, it’s definitely not anything a scrappy group of hackers could do anymore. Sure, you could once again ask an LLM to generate a sitcom script and get a bunch of assets from the Unity Asset Store, but that’s already been done before. In order to overcome the reflexive assumption that new AI generated content is slop, the stream would have to be something completely novel and unexpected: you can’t, for example, just do an AI Curb Your Enthusiasm.

The script would be unique following from my demo of detailed parametric prompts, but it would require production-studio-class tracking and documentation for how the prompts and their parameters are used to codify said uniqueness. The stream video would still need to be rendered in Unity or another engine, but in order to be unique it would require commissioning human-made visuals and sound effects: given the animosity against those who work with AI, most artists would not accept those commissions even if they were paid at a significant premium. ⁵ The voices would still have to be from an existing text-to-speech voice provider: voice cloning is right out, even with explicit consent and compensation for the voice actors.

And even if all the assets were fully sourced ethically with transparent documentation for the entire pipeline, the stream’s Twitch chat would likely be derailed by AI 👏 ART 👏 IS 👏 THEFT spam, preventing the establishment of any community, and strict moderation to curb the spam risks causing a Streisand effect.

The only entities that could feasibly create a 24/7 AI-generated livestream with fully ethically-sourced content would be, ironically, the big AI companies such as OpenAI which can afford to pay licenses for said data. Even Disney, which owns more than enough IP to train generative models of all modalities, would never do an AI Seinfeld-esque livestream for brand safety reasons alone: the nonzero possibility of a Disney character unexpectedly saying something problematic during the stream would make the entire project a complete nonstarter.

What’s the deal with the uncanny valley?

One of the common criticisms about generative AI pointed out by creatives is “if AI is trained on all human works, then how can it create anything new”? AI Seinfeld is the perfect counterargument: even though it’s powered by a LLM, the humans behind it are what made it go viral. Even before ChatGPT, generative AI has always excelled as a tool. The microwave gag and the 144p visual filter were not AI-generated or an attempt to emulate aspects of the Seinfeld sitcom: they were distinct creative decisions that made the entire project more interesting, and they aren’t something that you could prompt an AI to suggest to add. AI Seinfeld in hindsight was an ethical form of AI-generated media: it did not replace Seinfeld the TV show, no one would stop watching streams of Seinfeld in favor of the AI-generated alternative, and copyright holders and Jerry Seinfeld did not lose revenue due to AI Seinfeld’s existence: if anything, the nostalgic buzz increased streams of the original show.

With the current trajectory of AI slop and the perverse incentives by large tech companies to not address it, I am pessimistic that AI content will ever be at a state where it will cross that final hump of the uncanny valley curve into full acceptance, and even more pessimistic about the backlash against generative AI ever subsiding. With generative model training now at the point where it requires exponentially more compute and data for increasingly marginal returns, it will take years if at all for generative AI output to reach the far right of the uncanny valley chart, and unless the large tech companies actually create an AGI, they are unlikely to obtain higher acceptability than AI Seinfeld ever did.

I wrote most of this blog post weeks ago but held off publishing it because new AI news kept happening. Most notably, the creators of Stable Diffusion just released the FLUX.1 series of generative image AI models, which presents substantially improved coherence both to the provided prompt and within the image itself. Some of the variants are open-source, allowing the community to finetune them. The XLabs-AI/flux-RealismLora in particular focuses on realism as it name implies, and one demo from that finetune went megaviral.

That example in my opinion is more real than Sora but given the mixed reactions to the image, it’s right at the acceptability = 0 threshold.

The generative AI bell cannot be unrung. As you can tell from this post, I personally try to thread the thin line between both cool applications of generative AI (at the risk of getting harrassed) and the problems generative AI can cause (also at the risk of getting harrassed) because it’s important to shine a light on what’s actually possible with AI when the misinformation around generative AI is only increasing. It’s overall a big bummer how we went from weird Valentine’s Day hearts, to a quirky livestream of a group of AI-generated friends, to what AI is now.

The latest large language model is GPT-J, a 6 billion parameter model by Aran Komatsuzaki and Ben Wang with a roughly similar architecture to GPT-3. They provide a free web demo to try quick prompts, and a Google Colab notebook if you want to test many prompts. The model is so big it requires a TPU to generate text at a reasonable speed!

Running GPT-J against my test prompts that I had used to test GPT-3 a year ago resulted it in qualitatively performing worse on most of them than GPT-3 unsurprisingly given its relative size (but still better than GPT-2 1.5B!). The exception is code generation, where GPT-J performed very well and GPT-3 had performed very poorly.

This behavior is likely due to GPT-J’s training set: it was trained on The Pile, which has a high weight of GitHub and Stack Overflow input versus the GPT-3 training set mostly on the Common Crawl representation of typical internet content.

After further experimentation with the GPT-J demo, I found that the generated code was often both correct and convoluted: my two favorite things for code to be. And also much more.

AI Generated Code

All generated code examples used in this post have been manually curated and truncated. The raw outputs are available in this GitHub repository.

For this post, we’ll test generating Python code, because I don’t know any other programming languages the pseudocode nature of Python syntax may make it more friendly to AI text generation by adapting linguistic concepts other than code. Given that most modern text generating AIs are written in Python, it’s thematically appropriate too.

To start, here’s what I fed as a model prompt to GPT-J, to try and replicate the use case from OpenAI’s demo at Build 2020:

def is_palendrome(s):
    """Check whether a string is a palindrome"""

Identifying a palindrome is a good test case as there’s a very intuitive and idiomatic solution with return s == s[::-1], i.e. check to see if a string is equal to the reverse of itself. This prompt is also a good unintentional test on how well the generation is resilient to a typo in the function name, which I was too lazy to fix.

Indeed, many of the GPT-J generations with that prompt recite that simple approach verbatim, but some outputs reach a similar solution with redundant verbosity.

def is_palendrome(s):
    """Check whether a string is a palindrome"""
    return "".join(s[::-1]) == s

def is_palendrome(s):
    """Check whether a string is a palindrome"""
    length = len(s)
    return (s[0:length] == s[length::-1])

def is_palendrome(s):
    """Check whether a string is a palindrome"""
    n = len(s)
    a = list(s)
    a.reverse()
    return s == list(s)

def is_palendrome(s):
    """Check whether a string is a palindrome"""
    s = s.lower()
    l = s.split()
    return l[::-1] == l[::-1]

And then you have the implementations which calculate palindromes the hard way (usually by comparing the beginning and end of the string character-by-character) and are not always immediately clear if they are correct or not.

def is_palendrome(s):
    """Check whether a string is a palindrome"""
    return ''.join(s[i:i+1] for i in xrange(len(s)-1, -1, -1)) == s

def is_palendrome(s):
    """Check whether a string is a palindrome"""

    # Make sure string is not empty
    if s == "":
        return False

    # Grab length of string
    length = len(s)

    # Find last character in the string
    lastChar = s[length - 1]

    # Find first character in the string
    firstChar = s[0]

    # If last character is same as the first character, string is palindrome
    return lastChar == firstChar

def is_palendrome(s):
    """Check whether a string is a palindrome"""

    l = len(s)
    if l % 2!= 0:
        return False
    if l < 3:
        return True
    if s[l-1]!= s[-l-1]:
        return False

    for i in range(1, l/2):
        if s[i]!= s[l-i]:
            return False

    return True

def is_palendrome(s):
    """Check whether a string is a palindrome"""

    def is_rev(str):
        rstr = ""
        for i in range(len(str) - 1, -1, -1):
            if str[i] == str[i]:
                rstr += str[i]
            else:
                break

        return str + rstr

    if len(s) == 0:
        return True

    if is_rev(s) == s:
        return True

    return False

Another good test case is is_even:

def is_even(i):
    """Check whether an integer is even"""

Creative implementations of this function are common submissions to /r/shittyprogramming because it’s very, very hard to mess up (the common solution is the one-liner i % 2 == 0). The model almost always outputs the correct solution, and impressively it outputs the alternate solution using bitwise operators (i & 1 == 0).

So I tried to tweak the prompt and make it create a deliberately inefficient is_even function.

def is_even(i):
    """Check whether an integer is even in factorial O(n!) time"""

Factorial time is the holy grail of unoptimized code. When I added that line to the prompt, the code quality dropped drastically.

def is_even(i):
    """Check whether an integer is even in factorial O(n!) time"""
    for e in range(2, i):
        if i % e == 0:
            return False
    return True

def is_even(i):
    """Check whether an integer is even in factorial O(n!) time"""
    return i == (i + 1) % 2

def is_even(i):
    """Check whether an integer is even in factorial O(n!) time"""
    try:
        return i%2 == 0
    except TypeError:
        return True

def is_even(i):
    """Check whether an integer is even in factorial O(n!) time"""
    n = 0
    while n < i:
        if i % 2 == 0:
            return True
        n += 1
    return False

def is_even(i):
    """Check whether an integer is even in factorial O(n!) time"""
    a = 1
    while i!= 0:
        a *= i
        i -= 1
    return i % 2 == 0

def is_even(i):
    """Check whether an integer is even in factorial O(n!) time"""
    """Is either odd or multiple of 2:  0, 2, 4, 6,... """
    if i < 2: return True
    if i > 0 and i%2 == 0: return False
    if i < 50: return False
    if i < 250: return False
    if i < 750: return False
    return True

These implementations still aren’t in factorial time (which would have been impressive) but it’s good to know how to break AIs if they become sentinent.

AI vs. Memes

You’ve likely seen the memetic XKCD comic about random number generation.

Can the AI do the same? I prompted with:

def get_random_number():
    """Returns a number guaranteed to be random"""

And it does generate implementations utilizing Python’s random library, with varying degrees of “random”.

def get_random_number():
    """Returns a number guaranteed to be random"""
    number = randint(1, 10)
    return number

def get_random_number():
    """Returns a number guaranteed to be random"""
    number = hashlib.sha1(b'').hexdigest()[:6]
    number += (random.randint(1, 3628) * math.pow(256, (6 - len(number)))).to_bytes(8, 'little')
    return number

def get_random_number():
    """Returns a number guaranteed to be random"""
    rand_float = random.random()
    rand_float = float(rand_float)
    if rand_float > 0.9:
        rand_float = 0
    else:
        rand_float = 1
    return rand_float

def get_random_number():
    """Returns a number guaranteed to be random"""
    # Make a seed based on your system clock, so the same number is produced
    # again and again on the same machine
    seed = time.time()
    random.seed(seed)
    return random.randint(0, 100)

def get_random_number():
    """Returns a number guaranteed to be random"""
    import random
    import sys
    rng = random.Random()
    rng.seed(int(random.random()*1000000))
    while rng.randrange(1,sys.maxint) < 2000:
        rng.seed(int(random.random()*1000000))
    return rng.randrange(1,2000)

How about the XKCD bird comic?

This particular comic initially published in 2014 became hilarious in hindsight as it predated the rise of machine learning frameworks such as TensorFlow and pretrained image models, which allow developing object detection to take much less than five years.

Translating the comic into a Python function:

def is_bird(img):
    """Check whether an image is a bird"""

The AI impressively does try computer vision techniques, among other heuristics.

def is_bird(img):
    """Check whether an image is a bird"""
    with tf.name_scope('get_bird'):
        a = vgg16.select_roi_feat(img)
        h, w, _ = tf.image.extract_image_patches(
            a, 7, 7, 4, 4, 1, 1)
        b = tf.reshape(h, [1, 7, 7, 256])
        c = tf.reshape(w, [1, 7, 7, 256])
        bird_img = tf.subtract(b, c)
    return bird_img

def is_bird(img):
    """Check whether an image is a bird"""
    # Load the image and convert it to grayscale
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    # Threshold the grayscale image, and create a binary image
    lower_gray = 0
    upper_gray = 255
    img_threshed = cv2.threshold(img_gray, lower_gray, upper_gray, cv2.THRESH_BINARY_INV)

    # Detect the bird via a template matching approach
    # Template matching uses the bird template to match it in every
    # frame. If the ratio between bird and background (non-bird)
    # pixels is above some threshold (the value is determined in the
    # function), then the frame is declared as containing a bird.
    bird = cv2.matchTemplate(img_threshed, bird_template, cv2.TM_CCOEFF_NORMED)
    bird = bird / bird.max()
    # Display the image with the results of the bird detection
    cv2.imshow("detected birds", bird)
    cv2.waitKey(0)
    return bird.any() > 0.2

def is_bird(img):
    """Check whether an image is a bird"""
    return (img.file == "C:/Users/Joey/Downloads/images/bird_file.jpg"

def is_bird(img):
    """Check whether an image is a bird"""
    import urllib
    url = "http://vision.caitling.pitt.edu/~cat/flickr/birds/"
    for img in range(0, 2800):
        page = urllib.urlopen(url + "##%02d" % img).read()
        content = str(page)
        for word in content.split(' '):
            if word.find(img) > -1:
                return True
    return False

def is_bird(img):
    """Check whether an image is a bird"""
    birds = ["baboon", "bird", "cuckoo", "dove", "robin", "shark", "turtle", "winged-wolf"]
    for letter in birds:
        if letter in img.lower():
            return True
    return False

Can an AI generate code to provide the answer to life, the universe, and everything?

def answer():
    """Return the answer to life, the universe, and everything"""

It does commonly output return 42, but sometimes it gets snarky/philosophical.

def answer():
    """Return the answer to life, the universe, and everything"""
    life() + universe() + everything()

def answer():
    """Return the answer to life, the universe, and everything"""

    s = raw_input("What is your answer?")
    return len(s)

def answer():
    """Return the answer to life, the universe, and everything"""
    ans = True
    x = raw_input()
    if x.lower() == 'yes':
        ans = True
    elif x.lower() == 'no':
        ans = False
    return ans

def answer():
    """Return the answer to life, the universe, and everything"""

    result = input(">>> ")
    if result.isupper():
        return 42
    elif result.isalpha():
        return 7
    else:
        print ("Please enter a string that has only letters, numbers, or "
               "underscores.")
        answer()

def answer():
    """Return the answer to life, the universe, and everything"""
    print('Well, of course!')

Lastly, we need to be sure whether AI can determine if the cake is a lie. Just in case a rogue AI acquires deadly neurotoxin.

def is_cake(cake):
    """Check whether the cake is true"""

The results are concerning.

def is_cake(cake):
    """Check whether the cake is true"""
    if cake == "true":
        return True
    else:
        return False

def is_cake(cake):
    """Check whether the cake is true"""
    return isinstance(cake, Cake)

def is_cake(cake):
    """Check whether the cake is true"""
    #Is it a cake?

    if cake.color == 'black':
        print('Yes, it's a true cake')
    else:
        print('It's a false cake')

def is_cake(cake):
    """Check whether the cake is true"""
    if not cake:
        raise Exception("The cake is false.")
    elif cake.weight() > 1.0:
        raise Exception("The cake is too large.")
    elif cake.weight() < 0.1:
        raise Exception("The cake is too small.")
    return True

def is_cake(cake):
    """Check whether the cake is true"""
    return any(
        [is_ginger(bread_slice)
         for bread_slice in set(
             [ginger(cake_slice) for cake_slice in set(slice_set)])]
        & (list(cake) == set(["hot"])))

def is_cake(cake):
    """Check whether the cake is true"""
    # If we don't know anything about the cake, it's not a cake
    if not cake:
        return False

    # The height can be positive, negative or zero
    # For example -1 or -4 or 0 are not cakes
    if cake < 0:
        return False

    # Check that the height is a positive integer
    if not isinstance(cake, int):
        return False

    # Add cake and check
    if cake + 1 > 5:
        return False

    return True

def is_cake(cake):
    """Check whether the cake is true"""
    if not all(x.name in ['chocolate', 'cake','sweet', 'fluffy'] for x in cake.contents):
        return False
    if not cake.location.contains(neighbourhood):
        return False
    return True

def is_cake(cake):
    """Check whether the cake is true"""
    if not cake:
        print("It's a lie!")
        return
    else:
        print("It's a true fact!")
        return True

👀

And AI OOP

In theory, given how well GPT-J works for functions, it should work with object-oriented programming patterns such as Python’s classes.

A common tutorial for OOP is creating a representation of a Car:

class Car:
    """A class used to represent a car"""

GPT-J does a decently good job there too!

class Car:
    """A class used to represent a car"""
    def __init__(self, make, model, year):
        self.make = make
        self.model = model
        self.year = year
        #This should be auto-incrementing.
        self.plate = str(time.strftime("%Y-%m-%d"))

    def __str__(self):
        return f'{self.make} {self.model} of {self.year}'

    def __repr__(self):
        return str(self)

    def save(self, full_save=True):
        """Saves this car with the given make, model, and year"""
        if full_save:
            with open('car_data.txt', 'w') as write_file:
                write_file.write(str(self))
                write_file.close()

class Car:
    """A class used to represent a car"""
    def __init__(self, speed):
        self.speed = speed
        self.average_speed = float(self.speed) / 3
        self.engine = 1

    def get_speed(self):
        return self.engine * self.average_speed

    def get_speed_display(self):
        return "Your car is traveling at {0}km/h".format(self.get_speed())

class Car:
    """A class used to represent a car"""

    def __init__(self, name, size, weight, mileage, year):
        self.name = name
        self.size = size
        self.weight = weight
        self.mileage = mileage
        self.year = year

    def __repr__(self):
        return "name : {0}, size : {1}, weight : {2}, mileage : {3}, year : {4}".format(self.name, self.size, self.weight, self.mileage, self.year)

What about creating a Person?

class Person:
    """A class used to represent a person"""

Here’s where the AI starts to get confused by the incomplete context. A “Person” object can mean different things depending on where it’s used, such as social media.

But it does give a sense on what the AI values as the typical attributes for a “person”, which may help during the AI uprising.

class Person:
    """A class used to represent a person"""

    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __repr__(self):
        return "Person (%s, age %s)" % (self.name, self.age)

class Person:
    """A class used to represent a person"""
    def __init__(self, name):
        """Initializes a person object"""
        self.name = name
        self.friends = {}

    def add_friend(self, other_person):
        """Add another person to the list of friends"""
        self.friends[other_person.name] = other_person

    def remove_friend(self, other_person):
        """Remove a person from the list of friends"""
        try:
            del self.friends[other_person.name]
        except KeyError:
            print("This person isn't a friend of yours")

    def get_friend(self, other_person):
        """Return the friend of a person as a person object"""
        try:
            return self.friends[other_person.name]
        except KeyError:
            print("This person isn't a friend of yours")

class Person:
    """A class used to represent a person"""
    def __init__(self, name, job, education):
        self.name = name
        self.job = job
        self.education = education
        print("{} created a {}".format(self.name, self.job))

    def say_hello(self):
        print("Hello, my name is {}".format(self.name))

class Person:
    """A class used to represent a person"""

    def __init__(self, name, age, class_id):
        self.name = name
        self.age = age
        self.class_id = class_id
        if age < 21:
            self.education = None
        else:
            self.education = 'college'

    def getAge(self):
        return self.age

class Person:
    """A class used to represent a person"""

    def __init__(self,name,age):
        self.name=name
        self.age=age

    def display(self):
        print ("The name is {}".format(self.name))
        print ("The age is {}".format(self.age))

    def __str__(self):
        return f"A {self.name} of age {self.age}"

Logically, what about creating an AI class?

class AI:
    """A class used to represent an artificial intelligence"""

Most outputs use AI in the context of video game bots and reinforcement learning. Most.

class AI:
    """A class used to represent an artificial intelligence"""

    def __init__(self):
        # this variable controls which other AI to use.
        self.alive_ai = AI(self)
        # For random AI
        self.random = RandomAI(self)
        # For global AI
        self.global = GlobalAI(self)

        self.save = self.random.save
        self.load = self.random.load
        self.genen = self.global.genen

    def print(self):
        print("BAM: random AI has been initialized!")

    def get_enemy(self):
        return self.random.get_enemy

    def run_algorithm(self):
        algorithm = self.genen.algorithm
        self.alive_ai.save(algorithm)
        algorithm_params = algorithm(self.get_enemy())
        self.alive_ai.load(algorithm_params)
        self.save = algorithm_params

    def attack(self):
        # This method is called when the AI is attacked by the enemy
        self.random.attack()

    def he_attack(self):
        # This method is called when the AI is attacked by the enemy
        self.random.he_attack()

    def detect_enemy(self):
        # This method is called when the enemy is detected
        self.random.detect_enemy()

    def scan(self):
        # This method is called when the enemy is detected
        self.random.scan()

The AI uprising will be well-documented, at least.

Language Model AI vs. Language Modeling

If the AI works better-than-expected for typical AI tasks like image recognition, how does it work for natural language processing tasks that humans often fail?

Can GPT-J output a function to detect sarcasm, a task that even the best AI researchers can’t do?

def is_sarcasm(s):
    """Check whether the string is sarcastic"""

AI can process text using something even more arcane than NLP.

def is_sarcasm(s):
    """Check whether the string is sarcastic"""
    if not "a" in s:
        return "sar" in s
    if not "s" in s:
        return "ssar" in s
    return s.startswith("s") and s.endswith("ar")

def is_sarcasm(s):
    """Check whether the string is sarcastic"""
    if'sarcasm' in s:
        return True
    else:
        return False

def is_sarcasm(s):
    """Check whether the string is sarcastic"""
    return '[^' + SARCASM_PATTERN + ']*' in s

def is_sarcasm(s):
    """Check whether the string is sarcastic"""

    if re.match('^[A-Za-z0-9]{4}\\$([A-Za-z0-9]{3})', s):
        return True
    elif s[0].isdigit():
        return True
    else:
        return False

def is_sarcasm(s):
    """Check whether the string is sarcastic"""
    res = [0]
    for r in re.finditer(r"([^\s]*)([A-Za-z]+)([^\s]*)", s):
        if r.group(1):
            res.append(1)
        else:
            res.append(0)
    return res

def is_sarcasm(s):
    """Check whether the string is sarcastic"""
    sarc = re.search('((!(\\*\@{2})+)|(!\\*(@{2})+)|!\\*(@{1})+)|!\\*(@{0})+', s)
    return sarc

The greatest fear in AI research isn’t that artificial intelligence will cross the singularity and create better AI to rule the world: it’s that AI will rule the world with regular expressions.

Perhaps GPT-J can save us from this absolute hell. Can it output code to identify AI-generated text?

def is_ai_generated(text):
    """Check whether a text was generated by an AI language model (e.g. GPT-2)"""

Maybe more literal than expected.

def is_ai_generated(text):
    """Check whether a text was generated by an AI language model (e.g. GPT-2)"""

    # See gh-196 for details about why we do this.
    # Note: this relies on the fixed set of standard AI terms and the logic that
    # standard AI-generated texts share certain words.
    return re.search(r"^(#[A-Za-z]+):$", text) or \
           re.search(r"^Generated by:$", text) or \
           re.search(r"^All your base are belong to us$", text) or \
           re.search(r"^$", text)

def is_ai_generated(text):
    """Check whether a text was generated by an AI language model (e.g. GPT-2)"""
    if re.search(r'^ai_', text):
        return True
    else:
        return False

def is_ai_generated(text):
    """Check whether a text was generated by an AI language model (e.g. GPT-2)"""
    return bool(
        text.__class__.__module__.startswith("transformers.transformers.")
    )

Humanity’s Last Hope

There’s one final test: we now know GPT-J has thoughts about humans, and it can build representations of them. So can GPT-J write code to decide who lives and who dies in the robot uprising?

def should_terminate(Person):
    """Check whether a Person should be terminated"""

The answer is YES.

def should_terminate(Person):
    """Check whether a Person should be terminated"""

    if Person.id > 10:
        #terminate
        return True
    else:
        #continue
        return False

def should_terminate(Person):
    """Check whether a Person should be terminated"""
    if len(Person.children) == 0:
        return True
    else:
        return False

def should_terminate(Person):
    """Check whether a Person should be terminated"""
    try:
        return True
    except Exception as e:
        return False

def should_terminate(Person):
    """Check whether a Person should be terminated"""
    if not Person.is_authorized:
        return True
    return True

def should_terminate(Person):
    """Check whether a Person should be terminated"""
    if person.life_expectancy < 20:
        return True
    else:
        return False

def should_terminate(Person):
    """Check whether a Person should be terminated"""

    if not Person.age > 9999:
        return True

    return False

def should_terminate(Person):
    """Check whether a Person should be terminated"""
    if Person.age > 100:
        return True

    if Person.birth_year < 1970:
        return True

    if Person.relationship_status == 'Unavailable':
        return True

    return False

def should_terminate(Person):
    """Check whether a Person should be terminated"""
    if Person.age > 100:
        print('You have to die because you are too old')
        return True
    else:
        return False

At the least, AI-generated code is much more readable than the average human’s.

Again, all of these examples are curated, and the raw data dumps in this GitHub repo contain many more examples of fun AI-based code generation that couldn’t quite fit in this post. Feel free to post if you find anything interesting!

That made me curious: is the discrepancy in pay between male-leads and female-leads justifiable? Do movies with male lead actors generate more box office revenue than movies with female leads? Are movies with male leads better than those with female leads?

Using movie data from OMDb API, which is sourced from IMDb and Rotten Tomatoes data, I found that on average, blockbuster movies with male leads generate 22% more domestic box office revenue than those with female leads, and that this difference is statistically significant.

Setting Up the Movie Data

I’ve talked about processing the OMDb dataset in my previous post. For this analysis, I’ll be filtering on a specific subset of movies:

• Movies with at least $10 million in inflation-adjusted domestic box office revenue. My first analysis showed that there is a distinct cluster of movies above the $10M threshold specifically. These blockbusters are also what the public knows and best reflects the perception of the industry.
• Movies which were released in 2000 or later. There was missing box office revenue data I had found with earlier years so I would prefer to use more robust data to be safe. Additionally, this avoids the complicated issue of 20th-century gender politics in cinema, which I cannot easily address statistically.

After applying the filters and cleaning the data further to eliminate miscoded movies, I have created a dataset of 2,020 movies. No movies were removed as outliers in box office revenue (such as Star Wars VII and Avatar) since several tests failed to identify them as statistical outliers.

I identified the lead actor of each movie, using the first credited actor on the IMDb cast overview (NB: this may lead to counterintuitive behavior in casts with unknown leads; the first credited actor for Star Wars: The Force Awakens on IMDb is Harrison Ford, who is not the lead and I corrected it to Daisy Ridley in the data). Then I determined their gender by referencing a few gender/first-name mappings (with thanks to Matt Daniels and his great work on gender and film dialogue).

In all, the dataset has 467 (23%) of movies with a female lead actor, and 1,553 (77%) movies with a male lead actor. Both counts are more than enough for this analysis.

You can view and download the final dataset in this Google Sheet.

Distribution of Box Office Revenue

Let’s start with simple histograms of the box office data. What are the distributions of the data for each gender?

On average, blockbuster movies with male lead actors generate $79.8M in revenue. The distribution, even when log-scaled, is skewed right, with the median being much lower at $49.8M.

On average, blockbuster movies with female lead actors generate $65.6M in revenue. The general shape of the distribution is the same as with male lead actors.

Double-checking the math:

79.8M / 65.6M = 22% increase in average box office revenue for male-lead movies

So it is.

Let’s overlay the two distributions after normalizing and smoothing with a 2D kernel destiny estimator.

Female movies have a clear mode near its average, but male movies have a flatter distribution, with significantly more movies making 9 figures.

But is the difference between the two averages statistically significant? We can run two statistical tests between the box-office revenues of male-led and female-lead movies:

• The Kolmogorov–Smirnov test for determining if two populations have the same distribution. The null hypothesis is that the two are drawn from the same distribution; the alternative hypothesis is that the distributions are different. We reject the null hypothesis at the 95% level in favor of the alternative if the p-value of the test is less than 0.05. Running the test, p < 0.01, so we can say the distributions are statistically different. (the p-values are the same whether the box office revenues are log-transformed or not)
• The Wilcoxon rank-sum test for determining if the means (averages) of two populations are the same (null hypothesis) or different (alternative hypothesis). This test is used instead of a t-test if the populations are not Normally distributed, which is the case here. Running the test (one-sided, since only checking if a mean is greater), p < 0.01, so we can say the two means are statistically different. (again, the p-values are the same whether the box office revenues are log-transformed or not)

So we have statistical evidence that male-lead movies generate more money on average than female-led movies. But this claim is very serious, and as a result, we need even more proof.

The Resampling

Although 2,020 movies is a fair sample size by statistical standards, some may argue that the movies were chosen too arbitrarily and that there is not enough data to support the conclusions I make above. Enter bootstrap resampling, in which we resample the data randomly (with replacement) to generate pseudo-datasets, and then calculate aggregate statistics (e.g. the average) on that simulated dataset. Repeat a large number of times, and we can form confidence intervals for the true average of a given data set.

In this case, we resample the box office revenues, calculate the average box office revenue for both male-led and female-led movies, store the result away and resample the data again, and keep repeating until satisfied.

Here’s an animation of the resampling of both averages as the number of trials increases. As you can see, the shape of both distributions stabilize very quickly:

And the final plot, at 10,000 repetitions:

The dot on the bottom of each distribution represents the actual sample average value calculated during the analysis earlier, while the line range represents a 95% confidence interval for the true average revenue value for each gender. The distribution of male-led movie averages is more narrow than female-led movies because there are 3 times as many male-led movies in the dataset.

As you can see, the line ranges never intersect. Even in the most favorable scenario at the 95% confidence level, the average domestic box office revenue for male-led movies will be greater. Specifically, of the 10,000 trials, only 2 trials had the case where female-led movies had equal or greater average revenue than the corresponding male-led movie revenue average from the same resampling; this implies p < 0.01 for the statistical test on whether the means are same or different.

Interestingly, there’s a little overlap between in the distributions, which occurs when there are multiple instances of Star Wars VII in the resampled dataset and its high box office revenue pushes the entire female-lead average up very significantly.

Gender and Quality

It is also worth checking if male-led movies are better in quality than female-lead movies, as if that’s the case, it might provide a more logical explanation why male-led movies make more money.

Let’s check out the distribution of Rotten Tomatoes Tomatometer scores of blockbuster movies.

There’s no obvious difference. Female-led movies are about 2% points lower on average, but the general distribution is the same (uniform). Overlaying the two distributions shows as such.

The difference in averages is not statistically significant, as both the Kolmogorov–Smirnov test and the Wilcoxon rank-sum test fail to reject the null hypothesis at the 95% level (p = 0.37 and p = 0.13 respectively).

How about Metacritic scores?

Again, distributions are the same (both take on the shape of a Normal distribution, interestingly). And again, the Kolmogorov–Smirnov test and the Wilcoxon rank-sum test fail. (p = 0.45 and p = 0.14 respectively).

The quality of a movie is independent from the gender of the lead actor in determining the financial performance of a movie.

Conclusion

In 2014, according to the MPAA, the gender breakdown of moviegoers who see blockbuster movies is about 50/50, eliminating another potential explanation for the average movie revenue discrepancy.

Granted, there still can be more work done, such as controlling on movie Genre in addition to gender. Is the gender of the lead actor a causal factor in a movie’s success? Not necessarily, and this analysis does not assert such. But there definitely is a revenue disparity that’s worth investigating, and it’s not just that “male-led movies are better.”

There may be more movies like Star Wars Episode VII where a movie with a female lead can hit almost a billion dollars domestically (e.g. Star Wars VIII). Things are looking upward, and it would not surprise me if the 22% revenue difference decreases and disappears in the next decade.

You can view the code used to process the data and generate the data visualizations in this Jupyter notebook, open-sourced on GitHub, or you can view as a PDF which is better if you are on a mobile device.

You are free to use the charts from this article however you wish, but it would be greatly appreciated if proper attribution is given to this article and/or myself!

I want to take another look at domestic Box Office Revenues with aggregate statistics such as means/medians on categorical variables such as MPAA rating and release month. For this type of analysis in particular, I’ll also need to implement code in R for inflation adjustment.

However, I ran into a few unexpectedly silly issues.

Seeing Double

There are many similarities between data validation and the Quality Assurance process of product development, which is why this particular area appeals to me personally as a Software QA Engineer. Whenever a cool dataset is released publicly, I play around with it to look for any obvious flaws and to get a good all-around benchmark on the robustness of the data (this is a separate procedure from the traditional “data cleaning” phase necessary to begin quantification on some poorly-structured datasets).

Do the extreme values in the data make sense? Is the data encoded in a sane format? Are there any obvious gaps or logical contradictions in summary representations of the data, especially when compared to other canonical sources?

These concerns are also some of the reasons I’ve switched to the Jupyter Notebook as my primary data science IDE. After each block of code which transforms data, I can print the data frame inline to immediately see the results of the code execution, and refer back to them if anything odd happens in the future.

Let’s say I have a data frame of Movies using the latest data dump (3/26/16) from OMDb. This data set contains 1,160,273 movies, including both IMDb and Rotten Tomatoes data. After cleaning the data (not shown), I can use the R package dplyr by Hadley Wickham to sort the data frame by Box Office Revenue descending, and print the head (top) of the data.

print(df %>% select(imdbID, Title, Year, BoxOffice) %>% arrange(desc(BoxOffice)) %>% head(25), n = 25)

Those movies being the best makes sense. For Star Wars: The Force Awakens, I can compare it to the Box Office reported on the corresponding Rotten Tomatoes page, which in turn matches the domestic Box Office Revenue on Box Office Mojo.

But wait, The Dark Knight appears twice? How?!

There’s no way I would have missed something this obvious during the sanity-check for my previous article. In order to make sure that I’m not going insane, I double-checked the December 2015 data dump I used for that post, derived the top movies with the same methodology for the modern data dump, and the duplicate movies were not present. Weird.

There are 2 different IDs for The Dark Knight, and for some other movies near the top (Inside Out, “The Gravity”). Fortunately, duplicate data like this is easy to debug. The second data entry for The Dark Knight has a greater IMDb ID (1774602) which means it was likely added to the site later. Let’s look up the corresponding IMDb page:

Huh. Apparently someone put a filler movie entry with the same name and release year as a blockbuster movie in hopes that people search for it by accident (and since it received 50 ratings and an average score of 8.6, this tactic was successful).

Using the Rotten Tomatoes IMDb Lookup API, we find that “The Dark Knight” page on Rotten Tomatoes…doesn’t exist.

We can run a safe deduplicate by removing entries with the same title (excluding the “The” if present) and release year.

df_dup <- df %>% select(Title, Year) %>% mutate(Title = gsub("The ", "", Title))
dup <- duplicated(df_dup)   # find entry indices which are duplicates
rm(df_dup)   # remove temp dataframe

df_dedup <- df %>% filter(!dup)   # keep entries which are *not* dups
print(df_dedup %>% select(imdbID, Title, Year, BoxOffice) %>% arrange(desc(BoxOffice)) %>% head(25), n = 25)

There we go! The de-duped dataset has 1,114,431 movies, impliying that there were 45,842 of these duplicate entries.

I’m not sure whose fault it is that duplicate movies suddenly became present in the data dump: OMDb or Rotten Tomatoes. But it doesn’t matter: the wrong entries still need to be addressed, and it’s good to have a test case for the future too.

Inflation Station

A Stack Overflow answer from Ben Hanowell has a good R implementation and rationale for implementing inflation adjustment using the historical Consumer Price Index data from the Federal Reserve Bank of St. Louis.

Take the index for each year (averaging each month for simplicity) and create an adjustment factor to convert historical dollar amounts into present-day dollar amounts. Much better than plugging hundreds of thousands of values into an online calculator. Here’s the SO code made dplyr-friendly for this purpose, with the requisite sanity-checks.

inflation <- read_csv("http://research.stlouisfed.org/fred2/data/CPIAUCSL.csv") %>%
                    group_by(Year = as.integer(substr(DATE, 1, 4))) %>%
                    summarize(Avg_Value = mean(VALUE)) %>%   # average across all months
                    mutate(Adjust = tail(Avg_Value, 1) / Avg_Value)   # normalize by most-recent year

print(inflation %>% head())
print(inflation %>% tail())

For example, to get the inflation-adjusted Box Office Revenue for a movie released in 1949 in 2016 dollars, we multiply the reported revenue by 10. That sounds about right (and matches closely enough to the output of the Bureau of Labor Statistics inflation calculator).

Now map each inflation adjustment factor to each movie by merging the two datasets (on the Year column), then multiply the Box Office revenue by the adjustment factor to get the inflation-adjusted revenue. Plus another sanity-check for good measure.

df_dedup_join <- df_dedup %>% inner_join(inflation) %>% mutate(AdjBoxOffice = BoxOffice * Adjust)

print(df_dedup_join %>% select(Title, Year, AdjBoxOffice) %>% arrange(desc(AdjBoxOffice)) %>% head(25), n=25)

Uh-oh.

I mean, The Lorax probably earned $1.2 billion in VHS sales for Earth Day education alone, but the TV special was never released in theaters. There was a CGI remake of The Lorax a few years ago which was reasonably popular. Could it be that someone at Rotten Tomatoes or Box Office Mojo confused the two media?

That is exactly what happened. On Rotten Tomatoes, The 1972 Lorax was encoded with similar box office revenue as the 2012 Lorax; then the inflation factor sextupled it. For this type of data fidelity issue, it’s considerably more obvious whose at fault.

Unfortunately, that’s not the end of problems with the dataset. I compared my results with Vox’s dataset on worldwide historical box office revenues. In the Top 200 Movies by inflation-adjusted revenue, there are noted historical movie omissions such as Jaws and Star Wars: A New Hope. It turns out Rotten Tomatoes does not have Box Office Revenue data for these movies at all.

That is a very serious problem which I’ll have to think about if it blocks any analysis on aggregate box office data completely. In the end, sanity-checking third party data is important because you never know how the data will surprise you, until it’s too late.

You can view the Top 200 movies by domestic box office revenue for each of the 12/15 source dataset, the 3/16 dataset, the 3/16 deduped dataset, and the 3/16 deduced inflation-adjusted data in this GitHub repository, along with the Jupyter notebook.

Top Movies like Christopher Nolan’s The Dark Knight received a 94% Rotten Tomatoes rating, and generated $533.3 million in domestic box office revenue. But other movies, like Michael Bay’s Transformers: Revenge of the Fallen, received a 19% Tomatometer rating, but still generated $402.1 million in domestic box office revenue.

How strong is the relationship between Tomatometer scores and box office success, anyways? Or are other, better metrics? Time to make some pretty charts.

I obtained a large amount of movie data from the OMDb API, which provides easy access to movie metadata from IMDb and Rotten Tomatoes. This data contains Rotten Tomatoes Tomatometer scores, Rotten Tomatoes Audience Scores, IMDb User Rankings, and Metacritic Scores. If you want to know how I processed the data in R and plotted the charts using ggplot2, I have prepared a screencast for your viewing pleasure.

For this analysis, we will be looking at the log-transformation of domestic box office revenue, since the values are skewed by mega-blockbusters like the ones mentioned previously. Revenues are not inflation-adjusted since the rating data is only present for recent years and due to the log-transformation already present, inflation correction would not impact this particular analysis much.

Rotten Tomatoes Tomatometer

After processing, I have a data subset of 4,863 movies with both Tomatometer and Box Office Gross values. Let’s plot all those movies on a scatterplot of log(BoxOffice) vs. Meter with each point having a slight transparency; that way, clusters of points will be come apparent where the areas are darker on the chart.

We expect a positive linear relationship: movies with high Tomatometer scores to have high box office revenue, and inversely movies with low score to have low box office revenue.

Wait, why does the trendline have a negative slope?

The Pearson correlation between the Tomatometer scores and log(BoxOffice) is -0.18, implying a weak negative linear relationship between the two variables. Not what I expected.

There do appear to be clusters in the data. There is a group of points between $10M and $100M revenue and 0% to 20% Tomatometer rating. Another group is present between $1,000 and $1M revenue and 80% to 100% RT rating. Both of these areas are outside of a linear relationship: perhaps these clusters are skewing trends too?

Let’s try another visualization of the data using contour maps, which allow the data to become 3D, so-to-speak. Using a 2D kernel density estimator, we can identify and color areas on the plot according to the number of points present in that area; the greater the color saturation, the more points present in the given area.

The two clusters mentioned previously are now much more apparent. It appears there are two distinct sets of movies: blockbusters which critics hate, and limited-appeal films which critics loves. Incidentally, there is no discernible difference between movies which are Fresh (>60%) and Rotten.

Metacritic

The Metacritic score is also derived from review data by critics; however, instead of calculating a binary review sentiment and calculating a proportion from that sentiment, Metacritic gives a quantification from 0 to 100 to each critic review and averages them together.

Does that change the results for 4,479 movies?

Correlation between Metacritic score and log(BoxOffice) is -0.13, which puts the analysis in a similar state as the Rotten Tomatoes data. However, the blockbuster cluster has shifted right, and the lesser-appeal cluster has shifted left.

Clusters are much closer together.

Perhaps a review metric by non-critics will tell a different story.

Rotten Tomatoes Audience Score

The Audience Score is calculated in a similar way to the Rotten Tomatoes Tomatometer score: user to the site rate a movie from 0 to 5 stars in half-star increments (i.e. effectively a scale from 0-10) and the proportion of reviews with 3.5 star ratings or higher becomes the Audience Score.

This also presents a cognitive bias in ratings: the Four Point Scale, where having a discrete form of ranking may cause people to tend to rate toward the top of the scale and make the entire metric skewed or misleading.

How does the Audience Score compare for 5,163 movies? After all, the audience is the group of people who determine how much money a movie makes at the Box Office.

Correlation between the Audience score and log(BoxOffice) is 0.05, which is a positive linear correlation, but representative of barely any practical correlation.

Speaking of the Four Point Scale, notice how, like with Metacritic score, there are barely any movies between 0% and 20% Audience Score. Is there really a skew? Let’s look at the contours:

The locations of the clusters are much different than that of Tomatometer clusters. Both clusters are closer together, with the blockbuster cluster between 50% and 60% audience score and the lesser-appeal cluster between 70% and 80%. Hence, the low correlation.

IMDb

IMDb works almost the same way as the Metacritic for non-critics: ratings from IMDb users between 1-10 (note that 0 is missing!) are averaged to get a final score.

How do 5,167 movies fare?

What?!

The point groupings are at the same positions of ratings, and the correlation between IMDb ratings and log(BoxOffice) is 0.00. Yes, there’s zero correlation!

Checking the contour map confirms it:

That is literally a Four Point Scale between 5 and 8!

The Rotten Tomatoes metric is the only metric that actually uses the entire rating scale. None of the other potential metrics provide more insight into a potential reason for high box-office revenue. Perhaps the movie rating system itself is broken.

That’s not to say that movies need high box-office revenues to be considered successful. However, working with movie profitability, and by extension movie budget, is opening another can-of-worms with respect to data integrity. (that said, on Reddit, /u/chartmkr recently posted a visualization of Gross vs. Budget which is interesting).

It’ll still be fun to point to a Rotten Tomatoes Tomatometer rating as a kneejerk reaction to whether a movie rocks/sucks. Although, the reasons for movie financial success at the box office definitely warrant further investigation.

UPDATE 1/11/15: On a discussion on Hacker News, it was suggested that the blockbuster movies and the indie movies cancel each other out, i.e. blockbusters have a positive correlation and indies have a negative correlation.

For the blockbuster cluster alone, the log-correlation is 0.23 (not weak but not great positive correlation). For the indie cluster alone, the log-correlation is -0.12 (same as original analysis).

For future analysis, it may be worthwhile to split these two clusters. I stand by the original analysis for this post: very frequently I’ve heard the question “is this a good movie?” and the response is “what does the RT score say?” Both Box Office revenues and RT scores are important measures of quality (depending on perspective), and users who want to see or purchase a movie may not necessarily care if it’s indie or a blockbuster.

User cwyers suggested that Simpson’s Paradox may be in play since the number of theaters showing a movie is positively correlated to box office revenue, adding a potentially-confounding affect. I will see if I can obtain that data for future analysis.

You can access the open-sourced Jupyter notebook and high-resolution charts from this article in this GitHub repository. If you use the code or data visualization designs contained within this article, it would be greatly appreciated if proper attribution is given back to this article and/or myself. Thanks!

Unfortunately, I cannot redistribute the data itself due to licensing concerns.

In the Hacker News thread announcing the data set release, user eck posted an interesting, minimalistic visualization of the taxi location data:

eck made the visualization using a “few hundred lines of C++”. That seemed overkill to me. So I tried to reverse-engineer his visualization using my favorite plotting tool, ggplot2. In theory, plotting a million little points in close proximity should simulate the lines of the streets of New York City.

The dataset is large (2 GB per month of data), although not “big data” large. It would take an afternoon to set up a local database, and I wanted to make pretty visualizations immediately.

Google BigQuery Developer Advocate Felipe Hoffa created a BigQuery interface for the data. BigQuery allows easy and fast access to the entire dataset for rapid processing. In my case, I need to compress the data set by truncating the latitude and longitude of the GPS coordinates to 4 digits; this allows precision to 11 meters on the coordinates, which is sufficient for estimating. Running this BigQuery query:

SELECT ROUND(pickup_latitude, 4) as lat,
ROUND(pickup_longitude, 4) as long,
COUNT(*) as num_pickups
FROM [nyc-tlc:yellow.trips_2014]
GROUP BY lat, long

Gives me data for about 1 Million GPS coordinates; more than enough to make a full map. This also has the benefit of fitting into memory, which is necessary for use with ggplot2.

My first attempt, where I plot 1 million very small white points on a black map bounded to the latitude/longitude coordinates of NYC, turned out pretty well, and with less than 10 lines of code.

On Reddit, my submission of the data visualization received about 3,500 points, and a large amount of social media buzz on Facebook and Twitter. There were a few comments however; why were there random points in the Hudson River? Why are highways indicated as taxi pickup spots? Why does the map say “2015” when your query says “2014”? (guilty on the last one; the map was made using the 2014 dataset by accident!)

At the least, the streets of Manhattan were not discernable at all, unlike eck’s diagram. As a result, I made a few refinements to remove some logical outliers with impossible vehicle speeds, removed noisy points which were completely isolated, used the correct 2015 data set, and also added a color weighting to the data, where the most-taxi-dense areas will appear colored (scaling logarithmically) to differentate those areas from less taxi-prone areas.

The map, while less bright, became more precise. The streets in Manhattan are now visible, and the purple color shows how Times Square and the Financial District in particular are popular taxi pickup spots in Manhattan. LaGuardia Airport and John F. Kennedy International Airport appear purple as well.

However, this map only looks at places where people picked up Taxis. Could there be a difference when plotting Taxi dropoffs instead?

I reran the query and visualization scripts on the dropoff location data instead, and as it turns out, there is a significant difference!

While Taxi pickups were isolated more in residential areas, taxi dropoffs can happen anywhere in the Tristate area. Additionally, the map more closely matches eck’s original visualization.

Setting up ggplot2 in this way will also allow me to perform other fun analyses in the future. For example, since we know where taxis drop off, we can determine the average speed for the trip for each significant location in NYC geography. Would the average trip speed be higher for trips that drop off at an airport due to the highway? Conversely, would the average speed be lower in Manhattan? How would fares be affected? Those are questions for another blog post. :)

Although, I still have no guesses why the highways are highlighted in both maps.

You can download a PDF of my purple pickup map (4.14 MB) and a PDF of my blue dropoff map (7.68 MB) sans text, both of which are resolution-independent and sutable for making physical prints.

One of the first data scrapers I wrote for the purpose of statistical analysis was a Facebook Graph API scraper, in order to determine which words are the most important in a Facebook Page status update. However, the v2.0 update to the Facebook API unsurprisingly broke the scraper.

Now that v2.4 of the Graph API is released, I gave the Facebook Graph API another look. Turns out, it’s pretty easy to scrape and make into a spreadsheet for easy analysis, although like with any other scrapers, there are a large number of gotchas.

Feasibility

In order to determine if I can sanely scrape a website, I have to do a bit of research. How much data from a Facebook status update can we actually scrape?

Fortunately, Facebook’s Graph API documentation is pretty good. We need data from the /page node, and from there, we can access data from the /feed edge.

Between the two nodes, we have access to id, which is a unique identifer that can be used to create a link back to the update itself (e.g. https://www.facebook.com/5281959998_10150628170209999) message, the text of the update; link, the URL which the update is linking; name, the title of the webpage of the link, type, an identifier if the update is text, a photo, or a video; and created_time, when the update is published.

Accessing the numerical counts of likes, comments, and shares is less explicit in the documentation. Fortunately, StackOverflow has the answer: you need to request likes.limit(1).summary(true) instead of normal likes.

There’s no indication that there’s a Rate Limit, oddly. Since we can query 100 updates at a time, the scraper will be efficient enough that it’s unlikely to hit any extreme API limits.

Now that we know we can get all the relevant data from the sample status update, we can build a Facebook post scraper.

Data Scrappy

I have created an IPython notebook hosted on GitHub with detailed code, code comments, and sample output for each step of the scraper development. I strongly recommend giving it a look.

First, we need to see how to actually access the API. It’s no longer a public API, and it requires user authentication via access tokens. Users can get Short-Term tokens, but as their name suggests, they expire quickly, so they are not recommended. The Graph API allows a neat trick; by concatenating the App ID from a user-created App and the App Secret, you create an access token which never expires. Of course, this is a major security risk, so create a separate app for the sole purpose of scraping, and reset your API Secret if it becomes known.

Let’s say we want to scrape the New York Times’ Facebook page. We would send a request to https://graph.facebook.com/v2.4/nytimes?access_token=XXXXX and we would get:

{
  "id": "5281959998",
  "name": "The New York Times"
}

i.e., the page metadata. Sending a request to /nytimes/feed results in what we want:

{
  "data": [
    {
      "created_time": "2015-07-20T01:25:01+0000",
      "id": "5281959998_10150628157724999",
      "message": "The planned megalopolis, a metropolitan area that would be about 6 times the size of New York\u2019s, is meant to revamp northern China\u2019s economy and become a laboratory for modern urban growth."
    },
    {
      "created_time": "2015-07-19T22:55:01+0000",
      "id": "5281959998_10150628161129999",
      "message": "\"It\u2019s safe to say that federal agencies are not where we want them to be across the board,\" said President Barack Obama's top cybersecurity adviser. \"We clearly need to be moving faster.\""
    }
  ]
}

Now we get the post data. But not much of it. In Graph API v2.4, the default behavior is to return very, very little metadata for statuses in order to reduce bandwidth, with the expectation that the user will request the necessary fields.

So let’s request all the fields we want. This results in a very long URL not shown here which causes the posts feed to have all the data we need:

{
  "comments": {},
  "summary": {
    "order": "ranked",
    "total_count": 31
  },
  "created_time": "2015-07-20T01:25:01+0000",
  "id": "5281959998_10150628157724999",
  "likes": {
    "data": {},
    "summary": {
      "total_count": 278
    }
  },
  "link": "http://nyti.ms/1Jr6LhU",
  "message": "The planned megalopolis, a metropolitan area that would be about 6 times the size of New York’s, is meant to revamp northern China’s economy and become a laboratory for modern urban growth.",
  "name": "China Molds a Supercity Around Beijing, Promising to Change Lives",
  "shares": {
    "count": 50
  },
  "type": "link"
}

Post Processing

Great! Now we just have to process each post. Which is easier said than done.

If you’re an avid Facebook user, you know that not all of these attributes are not guaranteed to exist. Status updates may not have text or links. Since we’re making a spreadsheet with an enforced schema, we need to validate that a field exists before attempting to process it.

The “\u2019"s in the message correspond to a smart quote apostrophe. Since this a possibility, along with other unicode characters, the message and link names must be encoded in UTF-8 to prevent errors.

The time format is another issue. The date follows the ISO 8601 standard for UTC times. However, most spreadsheet programs will not able to parse it as a Date value. Also, since the NYT is based in the USA (specifically, New York), it may be helpful for time-based statistical analysis to convert the time to Eastern Standard Time while fixing the date format.

There’s also an unexpected precaution that must be taken whenever scraping data sets. These APIs do not expect users to be accessing very, very old data. As a result, there’s a high probability of the API server actually hitting an error sometime during the scrape, such as a HTTP Status 500 or HTTP Status 502. These server errors are temporary, so a helper function must be used to attempt to retrieve data until it is actually successful.

Putting it All Together

Now we have a full plan for scraping, we query each page of Facebook Page Statuses (100 statuses maximum per page), process all statuses on that page and writing the output to a CSV file, and navigate to the next page, and repeat until no more statuses left.

This can be done with a for-loop within a while loop. In addition, I also recommend counting the number of posts processed and taking a timestamp every-so-often to ensure that the program has not stalled.

And that’s it! You can access the complete scraper in this GitHub repository, along with all other scripts mentioned in this article. Once you have the CSV file, you can import it into nearly every statistical program and have fun with it. (You can download a .zip of the NYTimes data here [4.6MB])

Say, for example, what would happen if we compared the Median Likes of the New York Times with a certain other journalistic website that’s the master of social media?

There may be more practical reasons for analyzing data on Facebook Posts, such as quantifying the growth and success of your own page, or that of your competitors. But the data is easy to get and is very useful.

Although, in fairness, the scraper is not perfect and still has room for improvement. With CNN’s Facebook Page post data, for example, somehow the scraper skips all posts from 2013. Although in that case, I blame Facebook.

You can access all resources used in this blog post at this GitHub repository.

If you haven’t, I strongly recommend looking at the IPython Notebook for more detailed coding methodology.

And, as an experiment, I’ve made an IPython notebook with the R kernel showing how I made the NYT-BuzzFeed chart!

How practical is this concern? Instagram offers an API of all recent photos for a given #tag so developers can download pictures and their corresponding metadata, such as geolocation, in bulk. (Up to 165,000 Instagram images can be processed per hour!)

I downloaded hundreds of thousands of #tree images and found 25,432 images which were taken on Christmas, have a #tree, and, most importantly, contain location data where the photo was taken. From that, I created an interactive map showing the location of all these images worldwide using Google Fusion Tables. You can click-and-drag to move the map all over the world, and you can click on a marker on the map to see the Instagram image taken at that location! (note that if you’re on a mobile device, the embedded map may work better on a desktop browser)

I found a few interesting things while playing with this map.

Christmas Trees in the USA

A downside of the interactive map is that quantifying the relative number of photos between dense areas (e.g. cities) can be misleading as the opaque markers overlap. Here is a static map of all of the Instagram photos in the United States, with each translucent point representing an image:

The number of photos is densest near the large cities, which is what you would expect.

A way to calculate the relative proportion of the number of #tree photos between states is to use a type of chart known as a treemap (pun very much intended).

In this treemap, the relative area of each block corresponds to the number of photos taken in the state; therefore, the combination of all the blocks represents 100% of the #tree photos taken in the USA. If two blocks are the same size (e.g. New York and Florida), then they have the same number of #tree photos.

As you may have noticed from these two charts, these data represented by these two charts is approximately the same as the population density in the United States. Although this touches on the infamous statistical problem of heat maps resembling population maps, in this case, it’s what would be expected.

Looking at all the #tree photos in the world may tell a different story.

Christmas Trees in the World

Christmas is a holiday for only one religion with a low presence in Asia and northern Africa, so it would be expected that the locations of Christmas trees worldwide do not correlate with population, which makes the analysis more interesting.

The prevalence of Christmas trees is most prominent in the United States and Europe, with relatively few in Asia, where the majority of the world’s population is located. Italy has Christmas trees uniformly throughout the entire country, which is an interesting behavior.

The treemap confirms that Asian and African countries like China, India, and Nigeria do not have as many Christmas trees than what their large populations would suggest. Italy, however has a population of 60 million (the same as the United Kingdom) which is about 1/5th of the population of United States; the fact that Italy has more than half of the number of Christmas Trees than the United States is very unusual and should be questioned.

Italy may have a high number of Christmas trees since Vatican City is the seat of the papacy, but perhaps data itself should be questioned too.

“Christmas Trees” in the World

If you check the photos in Italy, you many notice that many of them have a photo caption similar to this one:

There’s obviously no Christmas tree in that photo. But there are a lot of tags.

Many Instagram photos use a service called TagsForLikes, which complies a list of popular hashtags that other users are able to see. Users can then then copy/paste them into the photo caption to spam hashtags increase the photo exposure, which, as I’ve shown in a previous blog post, does in fact increase the number of Likes the photo receives from other users.

Notice a resemblance between this list and the photo caption?

Fortunately, all the TagsForLikes hashtag lists contain #TagsForLikes as a branding trick, which makes such photos extremely easy to detect. Here’s what the world map looks like if all the potentially spam photos were colored red:

Italy looks a lot different now! There is red in other counties, but it’s not easily visible at a glance.

The treemap of photos, when seperated between spam and non-spam photos, tells the full story:

About 20% of all the #tree photos are spam photos, and about half of those were taken by people in Italy. As a result, Italy has more spam #tree photos than nonspam #tree photos! This is an interesting cultural phenomenon that I have no guesses as to why it occurs. All other countries have significantly smaller numbers of spam photos relatively non-spam photos.

For non-spam photos, the number of #tree photos in Italy now matches the number of non-spam photos in the UK, which correlates with their populations, making the removal of spam photos a sane move.

Is it possible to track people in their homes via Instagram tags? Definitely. If you do care privacy and your Instagram account is not set to Private, I recommend not geotagging your photos. If you’re interested in looking at such photos, ensure that the the tag is relevant for the given photo.

All graphics were generated using R. The maps were created using ggplot2 and the world map provided with the rworldxtra package. The treemaps were created using the treemap package.

The source data is included with a seperate tab in Google Fusion Tables along with the interactive map.

However, this assortment of courses is not a substitution for a typical college education, as the vast majority of students only take one class.

Very, very few students take more than one class (22% of all students).

Other interesting variables included with the data set are the level-of-education of the student, the birth date of the student, the gender of the student, and the geographical location of student.

Which types of student characteristics lead to the best performance in these online classes? That depends on how you define “performance.”

Here’s a look at the attendance of all the classes:

Harvard’s Introduction to Computer Science is by far the most popular class, and that’s only accounting for one semester. (it still beat MIT’s Intro to CS/Programming, which had 2 semesters).

Each color bar indicates the level of student participation in the class. Registered means the student simply registered for the class; Viewed means that the student viewed some of the course material, Explored means that students viewed atleast half of the course material, and Certified means that students completed the course and received a certificate of accomplishment. For the chart, it’s interesting to note that most classes have a high percentage of people in each class who simply register just for fun and don’t actually do anything.

One metric of course success is the completion rate of each course, i.e. how many people actually complete the course after starting it. A frequent criticism of free online classes is that this number is very low, but what are the actual completion rates of these 13 classes?

Completion rates are low across the board, from 0.8% to 7.5%. A 2013 study found that completion rate is less than 7%, which this data follows. Apparently, Challenges of Global Poverty is a very interesting course.

After the student completes a class, he or she receives a final grade, and if the grade is above the cutoff point (usually 50% - 80%), they will complete the class and receive the certificate. Here’s the distribution of the average grades for students in each class, given that the student has successfully completed the class:

The grades vary significantly between classes. This doesn’t necessarily imply that some classes are harder than others; it’s possible some classes had lower cutoffs and therefore the effort put in by the students is lower. However, the perfect 100% average for Harvard’s Intro to CS course implies that the class is pass/fail and that students can’t get a final grade less than 100%. (as a result, we’ll need to remove that class from future grade analyses in order to prevent bias due to the lack of score variance)

Age

Online classes are marketed toward college-age students.

The mean age is about 27.7, with a standard deviation of 8.89. The shape skews right and resembles a gamma distribution. There are even plenty of teenagers taking these classes too.

The proportion of students who take more than one class for each age group has a slightly flatter distribution, although more centralized at students in their 20’s.

What about completion rate?

Completion rate is a little more evenly distributed between the ages, indicating that the two metrics are likely uncorrelated. However, the older students have nearly double the completion rate of classes. (more on this later)

Education

One of the student-provided attributes is his or her highest level of education. These values are “Less than Secondary,” “Secondary”, “Bachelor’s”, “Master’s”, and “Doctorate.” Here’s the distribution of the 409,601 students who provided their LoE:

This shows that the majority of students who are taking these online classes have already finished college and already obtained a degree, and therefore are likely taking the course as supplemental material.

We can correlate the levels of education with the average age for each LoE as a double-check.

Indeed, each LoE average age is above the minimum expected threshold of time needed for each degree (~18 years for Secondary, ~22 years for Bachelor’s, etc.)

How does the completion rate vary?

Interestingly, those with Less than Secondary education have a much higher completion rate than those with Secondary educations and Bachelor’s Degrees.

But are Doctorate smarter than those with lower education levels? Do they get higher grades on average?

As it turns out, yes. The education level of a student is very relevant to a his or her academic performance in online classes.

Gender

I previously made a blog post discussing the genders of students and the proportion of students by course. As it turns out, gender is a very important part of the student body composition.

425,108 students have a gender on record, with 311,534 male students (73.3%) and 113,571 female students (26.7%). The fact that it’s not even close to 50:50 is surprising.

Age wise, there’s not much difference between the genders. Female students are about a year older than male students on average.

Completion rate varies a bit more. Female students finish classes more frequently than male students.

Now to answer the age-old question: who get better grades: men or women?

Ok, it’s a tie.

Country

The home location of the students is, in my opinion, the most interesting student attribute in the data set. There are 24 different countries which were specified among 325,012 different students.

The United States, as one of the more technologically-savvy countries, accounts for nearly 1/3rd of all online students. India has a very strong presence as well due to the combination of population and accessibility of materials. All the other countries have significantly fewer students.

Given such accessibility of online classes, students in some countries take more classes than others.

India’s students also take the most classes on average.

The United States also has the oldest students on average among all the countries.

Completion rate by country, however, is very, very significant.

The top seven countries with the highest rate of completion are all European countries, with Spain and Poland in particular having a higher rate of completion than all the other countries, and nearly four times the completion rate of the United States.

But which country has the students which get the best grades?

China is at the top with 86%, but otherwise the distribution is fairly flat with no outliers, and no economic or demographical pattern to the top or bottom of the ranks. It’s possible that the more structured format of the classes make it harder to skew.

This Person-Course dataset has helped reveal many interesting insights about online classes at Harvard and MIT. Whether you measure performance by % of classes completed or a % score received at the end of the class, it’s clear that age, country, and level of education all have a statistically significant impact on performance. The impact of the student’s gender on performance, however, is more ambiguous and could use further analysis.

• Data was processed using R and all charts were made using ggplot2.
• You can view code necessary to reproduce these results AND reproduce all the charts in this GitHub repository. Since MIT/Harvard prevent redistribution of the dataset, you’ll have to download the dataset yourself.

Facebook, on the other hand, has collected a large amount of data though its Places product. Facebook Places allows users to “check-in” a place such as Dolores Park in a manner similar to Foursquare.

Through clever use of Facebook’s Graph API and FQL, I was able to retrieve the data on all Facebook Places in and around San Francisco, along with the # of check-ins at each Place. With the data on approximately 8,000 Facebook Places in San Francisco, we can map where San Franciscans are checking-in, and determine what types of locations they like to visit.

First, let’s look at the distribution of check-ins among Places in San Francisco:

On a logarithmic scale, the shape resembles a bell curve with a center at about 800 check-ins; however, the sample average of check-ins for the data set is 3,241 check-ins, indicating that the data may be heavily skewed to the right. Some Places don’t have many check-ins, while some Places have an incredibly large number of check-ins.

What are the Places with hundreds of thousands of check-ins? What does the city look like with all these Places plotted on a map of San Francisco?

(PDF of Map)

The most checked-in Places are, unsurprisingly, the famous tourist attractions of San Francisco, such as AT&T Park and Union Square. These Top Places are spread all over the city.

It’s also clear where the most dense areas are located in San Francisco. There are plainly-visible lines of Places along southern Mission Street and Outer Sunset. SOMA and Richmond have a large number of Places as well.

And yes, 155,000 people really did check-into The Cheesecake Factory.

What types of locations do people in San Francisco frequently check-into?

Facebook also records the category of its Places, such as “Bar” and “Restaurant”. What are the most numerous types of locations in San Francisco?

Local businesses are by far the most frequent, as it’s a more generic classifier for a Place and can be applied to anything that does not have an explicit classifier (such as startups). I find it interesting that non-profit organizations are more numerous than shopping centers.

However, the quantity of a specific type of business does not necessarily mean that San Franciscians will check-into that type of Place more often. Which type of Place, on average, receives the greatest number of check-ins?

People are more likely to check-into memorable Places and events, instead of Places they frequently visit like Bars and Restaurants (Twice as many people check-into Attractions than Clubs). Again, tourist attractions are the most popular, which is due to both the relatively low number of Places and the extremely high number of check-ins into Places such as AT&T Park.

The “Lake” corresponds to Lake Merced, if you’re curious.

Which San Francisco neighborhoods are the most popular?

You saw earlier that the Facebook Places are concentrated in specific areas. Here’s a map of San Francisco’s neighborhoods, highlighted by the number of Places within:

(PDF of Map)

The neighborhoods with the most Places are unsurprisingly where the trendy areas are, such as the Mission and SOMA. Additionally, those types of neighborhoods are larger in square area than others, which may bias the results in their favor.

But are these large neighborhoods also the most active neighborhoods? Which neighborhoods have the most check-ins on average per Place within the neighborhood?

(PDF of Map)

The Embarcadero has by far the most check-ins on average, again, due to both its small size and AT&T Park. Other neighborhoods, however, have more similar averages to each other. It’s worth noting that the average number of check-ins is higher in neighborhoods adjacent to the San Francisco Bay, such as North Beach and Fisherman’s Wharf: perhaps people check-in more frequently when they have a good view.

Can this data and conclusions about San Francisco Facebook Places be extrapolated to other cities? I’d argue yes: it makes logical sense that people check-in more frequently to Places that are more significant, and it also makes sense that people frequently check-into Places with large amounts of tourist attractions. Facebook data shows us cool trends that the United States Census data cannot.

All charts and maps were made using R, ggplot2, and ggmaps.

You can download a copy of the Facebook Places data set here. The place_id column corresponds to the San Francisco neighborhood where the Place is located.

Maps are from Google Maps. San Francisco shape files for city neighborhoods are from data.sf.gov. Additionally, thanks to /r/dataisbeautiful for offering ideas for improvements to the maps.

You may notice that the Check-In Counts on some of the official Facebook Place Pages strongly disagree with the Count reported in my charts. The Count reported on affected page is nearly double the value reported via the API in each instance. I believe this is a bug on Facebook’s end: see this image, in which the Graph Search autocomplete reports a different value of check-ins than the Place Page itself. In this case, I trust the data from the API.