What Happens When an RNN Makes Your Startup Commercial for You?

Image for post
Image for post
The infamous Dollar Shave Club video: https://youtu.be/ZUG9qYTJMsI

Hi. My name is Aaron and I’m an engineer at Josh.ai. With companies like Apple and Amazon also having a stake in the home automation market, we need to make sure that we really stand out to the crowd of potential buyers.

One way you can do that is by having a really cool commercial appear during one of the most watched events of the year. I decided to take a small detour in helping develop Josh.ai and instead I began moonlighting as a director. I wanted to contribute to Josh.ai in a creative way, and so I did what I do best. I made my computer do the work for me.

Introducing storyboards, a fully automated and free pipeline for generating storyboards (hah!) for videos that you might want to make. Storyboards is built by combining the ever so wonderful AI tools of Andrej Karpathy whose tools I’ve used in a previous blog post to great comedic effect.

Storyboards work by first taking a set of videos and converting the videos into a set of images. It does so by grabbing one frame per second from the video which, with enough videos, provides an ample amount of data to train on in the next step. The next step in the pipeline is to caption the images using Karpathy’s neuraltalk2 project. These captions are aggregated together and used as the training data to Karpathy’s char-rnn project. Since the first set of captions describe the sequential images found from the videos, the char-rnn step learns the patterns in the text that describes the images and when the char-rnn program is prompted to generate new text, it ends up generating an entirely new video!

Since I strung these projects together for us to make our own commercial, I downloaded the commercials of some of the most famous Kickstarter campaigns and used those as the training data for neuraltalk2. I figured that all of these projects were successful and they must have absolutely done so with a great commercial.

Below is the commercial that storyboards created. Read it and just picture, in your mind, all of this leading up to the wonderful unveil of Josh.ai being in your home to aid and comfort you. Unfortunately, my boss shot down the idea when I pitched it to him, but I think if I try hard enough, he might be willing to reconsider.

a person is holding a pair of scissors
a man sitting in front of a laptop computer
a man and a woman standing in front of a store
a person is holding a remote control in front of a tv
a woman walking down a path with a backpack
a woman is sitting on a bench with a cell phone
a man sitting at a table with a laptop
a man in a suit and tie standing in a room
a woman standing in front of a refrigerator
a man in a suit and tie standing in front of a mirror
a close up of a cell phone
a person holding a cell phone in their hand
a picture of a computer seyboard and a mouse
a white refrigerator screzn
a man in a suit and tie standing in front of a laptop
a person holding a cell phone in his hand
a person is holding a laptop in a room
a man sitting in front of a laptop computer

And if that won’t do it, maybe this one will?

a man and a woman are playing a video game
a picture of a bathroom with a sink and a mirror
a man and a woman standing next to each other
a man in a suit and tie standing in the snow
a man is standing in the snow with a snowboard
a person holding a remote control in their hand
a train car is sitting on a window sill looking atrel thicro
a person is on a skateboard on a ramp
a man standing in a room holding a remote
a picture of a person in a suit

Not bad? Not bad.

Image for post
Image for post

This post was written by Aaron at Josh.ai. Previously, Aaron worked at Northrop Grumman before joining the Josh team where he works on natural language programming (NLP) and artificial intelligence (AI). Aaron is a skilled YoYo expert, loves video games and music, has been programming since middle school and just turned 21.

Josh.ai is an artificial intelligence agent for your home. If you’re interested in learning more, visit us at https://josh.ai.

Like us on Facebook, follow us on Twitter.

Written by

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store