Every year Geoff Boyle, the founder of CML, pulls together one or more camera, lens, or lighting tests. Back in July 2019 he asked if I could help with the next such effort, so in February 2020 I found myself spending a week at the Netherlands Film Academy, working with an Italian cinematographer and a Dutch AC/DIT to shoot tests combining fifteen different lights, eight cameras, and seven color meters.

The point of these tests was to mix and match various LED fixtures with a selection of cinema cameras, to see how lights and cameras interacted to render color. Geoff explained his reasoning:

“After the presentation I did at IBC 2018 I saw that people didn’t realise how bad LEDs were with skin or how widely they differed.

“How did I choose the cameras and lights?

“Cameras, I tried to choose a representative set of high-end cameras from each manufacturer. The FX9 instead of the Venice so that we could do a side by side with the C500-2 at the same time. The Alexa studio because I like it and it was what we could get for free 😊, that was always a consideration. The Varicam LT instead of the Pure because that was available, and we could record raw to the Odyssey. The Fuji GFX100 was also included because I’ve been impressed with the pictures from other Fuji cameras and this is the top of the range [but we weren’t able to include it in the final results; details in the “part 3” link below —AJW].

Lights, I started with the manufacturers that I knew and liked and those who had shown work in the last year that intrigued me, Kino Flo particularly in this regard; also BB&S, Cineo, K5600, DMG/Rosco. Dedo was meant to be included in this category but miscommunication messed that up. The ARRI SkyPanel was included because it had performed so badly in the 2018 test, I wanted to double check. Of course, I also included lights that I’d either shot with or that intrigued me, Aladdin, Hive, ETC, and then the lights people asked for that I’d missed.

I posted on CML well in advance asking for suggestions [for both lights and cameras]. I got far too many to include. Fiilex wasn’t on the list of lights to be tested but they contacted me, and I told them the list was full but if they sent them, we’d try and squeeze them in. I’m glad they did!”

What we tested, and how

Geoff wound up getting seven different digital cameras, plus 35mm film as a baseline; and fourteen LED lights, plus tungsten as a baseline. We used the standard CML test scene: two (very patient) skin tone models and a selection of test charts:

We shot the scene with every combination of lights and cameras available. All lights were used at both 3200 K and 5600 K settings, which meant changing settings on the fourteen different LED fixtures and seven of the cameras; adding CTB gels to the tungsten lamps; and swapping between Kodak 5219 (tungsten) and 5207 (daylight) stocks in an ARRI 435. Overall, we shot 240 setups in four days. 

If you’re interested in more details, or you simply have masochistic tendencies, I’ve written up the process on CML in three parts: Part 1, Part 2, and Part 3.

Additionally, I took readings with seven different color meters — two spectrometers and five tristimulus meters; I describe the difference here — to see how well various meters tracked the color temperature and tint of the different lights. We used a Sekonic C-800 spectrometer as our reference meter, and I compared it to a Lighting Passport spectrometer, a Minolta Color Meter II, first- and second-generation Lumu Power meters, and Cine Meter II running on an iPhone SE and an iPhone XS Max (full disclosure: I wrote Cine Meter II). 

Our DP had a Sekonic C-700 spectrometer. We didn’t use it throughout the tests, but whenever she took a reading, it was was always within 100 K of the C-800 reading, and usually within 50 K.

I captured full datasets from both the C-800 and Lighting Passport, but only recorded CCT and Tint from the tristimulus meters, as that’s all the info tristimulus meters provide about color.

The results

You can see the results here. That page links to several sequences for each camera: 

• Separate 3200 K and 5600 K captures with all lights, adjusted for exposure only. These let you compare the basic color balances for each light at their nominal 3200 K and 5600 K presets.
• A “balanced” clip with a one-light grade to make the gray card truly gray. This sequence shows in dramatic detail how individual colors are affected by the combination of each light with each camera.
• Various ‘scopes, captured from Resolve (not yet posted for all cameras; check back in a couple of days if you don’t see what you’re looking for). 

There are also graphs of the various color meters’ responses to each light, color quality metrics from the two spectrometers, and detailed spectrometer data for each light.

I also have full captures of the Lighting Passport’s data, including CIE and spectrum plots, CRI histograms and radars, TM-30 vector and rf/rg graphs, MacAdam ellipses, and more. Geoff is getting the plots integrated into the individual lighting data pages, and you can also browse the data on Dropbox or download the data in a zipfile.

Pro tip: if you want to critically examine footage, either download the “zipped file of one frame per light” on each camera’s page, or use the Vimeo link for each clip and download the original, and view the images on a properly calibrated monitor. That way you avoid downscaling / recompression / color-space-conversion artifacts and the vagaries of phone / tablet / computer displays, which may exaggerate some differences and downplay others in unpredictable ways.

What does all this show us?

First, don’t use these tests the way drunks use lamp-posts: for support, rather than illumination. These aren’t “shoot-outs”; there aren’t winners and losers. What we’ve done to take a snapshot of a selection of high-end lights and how their color rendering interacts with the spectral sensitivities of a variety of cine cameras to render skintones and reference colors. We captured data, not conclusions. 

Having said that, I have some observations.

Nine years after AMPAS alerted us to the issues of solid state lighting technologies, we’re still dealing with surprising differences, even among high-end instruments. Pointing out the problem was relatively easy, but solving it is clearly a bit more difficult!

It’s interesting to bring up the clips for different cameras side-by-side, and see how the cameras respond differently to the same light. Using a color meter to choose or balance lights only gets you halfway there; the color response of your production camera is still the final word on whether or not you’ve got the lighting color right.

When it comes to measuring correlated color temperature (CCT), all the color meters did a pretty good job. Tint is a different matter (put aside the Lumu tint readings for now; the Lumu folks are investigating and I expect they’ll update their software with better tint calculations in the future. And the poor old Minolta just isn’t up to the tint-measurement task. That leaves Cine Meter II running on two different generations of iPhone camera). The spectrometers and the Cine Meter IIs usually agree somewhat, but tint readings show far more variability between meters than CCT readings do, and sometimes the spectrometers disagree strongly with the Cine Meter IIs.

I’ve been working through the source clips, bringing them into Resolve, white-balancing them, and comparing the direction and magnitude of the tint correction to the meters’ readings, and the spectrometers often — but not always — do a better job of predicting tint imbalances than the Cine Meter IIs do. It’s probably down to whether the spectral responses of the spectrometers or the iPhone cameras are a better match for a particular production camera in any given test.

Most if not all lights in our test allow you to adjust both CCT and tint in fine-grained steps. We shot at 3200 K and 5600 K presets, but as you can see those presets may render warmer or cooler, somewhat green or slightly magenta. Having both CCT and tint controls lets you tweak these lights to a more perfect color, whether it’s matching a camera preset or balancing one light to another.

Sadly we didn’t have time to explore these adjustments, and how they affected color quality: it’s possible that moving off the calibrated presets may degrade color quality strongly, slightly, or not at all. Hmm, I see a whole new set of tests that need doing…

Finally…

There’s a tendency to recoil in horror at the differences revealed in a side-by-side test like this, but in the real world these problems (a) are often less troublesome than a test makes them appear, and (b) are variations on the same sorts of issues we’ve dealt with all along, even using tungsten instruments. Productions serious about color rendition have always had to do lighting / fabric / makeup / camera tests, whether with differing film stocks or different digital cameras, to make sure the entire imaging chain reproduced colors in an accurate and/or pleasing manner. “Accurate” and “pleasing” are different things, too; it’s often more important that you make a pleasing picture than an accurate one. 

Consider the tungsten lamp gelled to daylight. It has a TLCI of 85 — the worst of any light in our tests, and its images don’t look great in comparison to the ungelled lamp’s. Yet, somehow, we’ve been doing that sort of thing for decades and getting away with it. To borrow from Douglas Adams: Don’t panic!

Photos

Disclosure: CML paid for my travel and lodging and provided a small stipend for my time.