Automatic Rotoscoping…For Free

Damian Allen

5 years ago

We’re still a few years away from where machine learning AI can produce rotoscoping that rivals what human artists can achieve (though only a very few), but even today machine learning offers results that are more than adequate for color correction, camera tracking garbage mattes, or temp roto for slap comps while waiting on final outsourced roto. And with just a couple of lines of code you—the unskilled, non-programmer—can have that machine learning goodness at your command. What’s better, you can start to learn a little Python in the process.

Most beginner programming tutorials start with some lame text-based guessing game. I say skip to the meat. In this ‘Impossible Shot’ video you’ll get a good intro to programming in Python and come out the other side with an amazing rotoscoping tool that you can put into use in production today.

Half the battle is getting everything installed right, so I’ve made sure to clearly lay out all the details in the video. Below is a summary of what’s covered.

Step 1: Install Anaconda

Anaconda is an amazing installer package for Python that allows you to install third party software without having to search the web for it. It also makes virtual environments easy: virtual environments are development ‘sandboxes’ you can experiment in without messing up any other software installed on your computer. (Click here to download Anaconda for your platform)

Once you have Anaconda installed on your system, open the Anaconda prompt (or the regular Terminal on OS X) and execute the following commands to create and activate a custom virtual environment (which we’ll call ‘autoroto’—you can choose any name you like):

conda create --name autoroto
conda activate autoroto

To exit out of a virtual environment, just type:

conda deactivate

Step 2: Install the libraries we need

We’ll be working with code provided by learnopencv.com, an amazing website for learning about computer vision and machine learning. The great thing about modern programming is that you don’t need to fully understand how something works to use the code. You can treat things like “black boxes” and just reap the benefits.

To use the code, we need to install some of the third party libraries required for it to run. Make sure your virtual environment is activated (see above) and type:

conda install torchvision -c pytorch
conda install -c anaconda pillow

Step 3: Download the sample code

Click here to download the code that we modified from learnopencv.org. We’ll use this as the basis for the auto rotoscoping. Now create a Python file. We’ll be using Wind Personal IDE here, but you can use another if you prefer. An IDE is an ‘Integrated Development Environment,’ basically a word processor specifically designed for writing code. It’ll help error check your programs and give you helpful hints and auto-completion of code as you type. Go ahead and download and install Wing if you want to follow along as closely as possible.

Save an new, empty Python file to the same directory as the sample code you just downloaded (FindMattes.py). Call it ‘autoroto.py ‘. As long as it’s in the same directory as FindMattes.py, Python will have no problem linking to the FindMattes functions.

Step 4: Do an initial auto rotoscoping test

Let’s test out our rotoscoper to see if it actually works before sending a whole image sequence through. Just two lines of code is all it takes to create an automatic roto of an image. In autoroto.py add the following lines, then save the file:

import FindMattes.py as fm
fm.fm.createMatte("Path/to/your/source/image.png", "Path/to/resulting/matte_file.png", 256)

Replace the first part with the path to your test image (keep the quotes around the path. e.g. “C:/Users/yourusername/Documents/myImage.png” or “/Users/yourusername/Documents/myImage.png”), the second with the intended path and name for the resulting matte image (e.g. “C:/Users/yourusername/Documents/myImage_matte.png” or “/Users/yourusername/Documents/myImage_matte.png”), and the last number with the vertical resolution of the output image. My recommendation is to keep this at 256 for testing. Full res 2K and 4K images could take a long time to process on more modest machines.

Now run your program by typing into the command line prompt:

python "Path/to/your/program/autoroto.py"

When you look in the directory you should now find a matte image, automatically rotoscoping identified features of the shot, color-coded according to the 8 bit values listed in FindMattes.py.

Step 5: Ask the user for folders

So the thing works, but you want to be able to use for any old image sequence, right? Well, here’s the full code. I step through the entire thing in reasonable detail in the video, but if you’re desperate just to have the working version…type away.

import FindMattes as fm
import tkinter as tk
from tkinter import filedialog
from tkinter import simpledialog as sd
from os import listdir
# Get the file directory
root = tk.Tk()
root.withdraw()
directoryName = filedialog.askdirectory(parent=root, initialdir="/", title = 'Please select a directory')
matteHeight = sd.askinteger("Set the matte size", "Output matte vertical resolution (256 recommended for quick tests)?", parent=root,initialvalue=256)
listOfFiles = listdir(directoryName)
for currentFile in listOfFiles:
    sourceFile = directoryName + "/" + currentFile
    mainNameEnd = currentFile.find('.')
    nameForMatte = currentFile[:mainNameEnd] + "_matte" + currentFile[mainNameEnd:]
    fullPathMatteName = directoryName + "/" + nameForMatte
    fm.createMatte(sourceFile, fullPathMatteName, matteHeight)
    print("Just created: " + nameForMatte)

Step 6: Test the results in a compositor

After you’ve entered the above code in your autoroto.py file, save it, then run the code again from the command line:

python "Path/to/your/program/autoroto.py"

Then hop in to your favorite compositor, use a chroma key to isolate specific colors in the matte, then use it for whatever creative or utilitarian purpose you have in mind.