About a year ago, interface designers Nathan Shedroff and Christopher Noessel published their book 'Make It So: Interaction Design Lessons from Science Fiction', on how sci-fi and real world interfaces influence each other. User interfaces and human-computer interaction is a vast topic. Mostly we're interested in creating graphics that visualize what might be possible, or perhaps prototyping actual interfaces, so it's nice to peek at what others imagine and who's involved, especially if they use After Effects and Creative Cloud.

Make It So covers elements of sci-fi interfaces mechanical controls, visual interfaces, volumetric projections, gestures, audio interfaces, brain interfaces, augmented reality, and anthropomorphic interfaces — as well as related human activities in communication, learning, medicine, and sex. As reflected by the book's title (a phrase used by Captain Picard), the Star Trek media empire has introduced the most important innovations in human computer interfaces to the general public (“hello computor“), like the sleek control panels and various mobile devices.

There's even more on the website, Make It So: IxD Lessons from Sci-Fi, including follow-up articles on specific movies, like articles on Prometheus listed in chronological order, and video of presentations (embedded below). In an O'Reilly chat follow-up they note that SFX glitz is not necessarily a problem:  

“Sci-fi influences audience’s expectations, and sci-fi interface design is like an elaborate brainstorm… Even when they get it wrong, they get some things right, and we can work out to find out what it is. In fact, if you’ll look for the examples of 'apologetics' in the book, you’ll see it’s where sci-fi interfaces break that we can get some of the biggest insight.”

Even generals and spooks are not immune to sci-fi UI, as noted recently by revelations that the NSA director modeled his war room after the Star Trek NG Enterprise bridge. The spy versus spy lair is dubbed the Information Dominance Center; here's an old report from ABC with shots in passing:

More famous now though are the interfaces designed for the movie Minority Report. Here's the designer, John Underkoffler: Pointing to the future of UI,

In the decade since, criticism has appeared, like How 'Minority Report' Trapped Us In A World Of Bad Interfaces or Frog's Mark Rolston: the 'Minority Report' interface is a 'terrible idea' (Rolston sees a future dominated by voice). Technology Review and others retort, Think Gestural Interactions Suck? Design Your Own, since creating your own gestural patterns could make these interactions easier to remember. But even Google seems to be plunging in, as it acquired Flutter to bring Kinect-like features to computers (including OS X).

See also Minority Report UI designer John Underkoffler talks about the future of gestures by Kevin Kelly.

Somewhat similar styles found in Microsoft's Productivity Future Vision (2011) also found criticism by Bret Victor in A Brief Rant on the Future of Interaction Design, for limitations not found in real humans bodies. As he points out elsewhere, right now, today, we can't see the thing, at all, that's going to be the most important 100 years from now.

But today, gestural interfaces are something that can be built, and What Sci-Fi Tells Interaction Designers About Gestural Interfaces by co-author Christopher Noessel, a summary of a chapter Make it So, explains Hollywood’s pidgin language for gesture:

“…we can look at the cause and effect of what is shown on screen and piece together a basic gestural vocabulary. Only seven gestures are common across properties in the survey… and a designer who deviates from them should do so only with good reason or else risk confusing the user.

1. Wave to Activate
2. Push to Move
3. Turn to Rotate
4. Swipe to Dismiss
5. Point or Touch to Select
6. Extend the Hand to Shoot
7. Pinch and Spread to Scale

Lesson: Deviate Cautiously From the Gestural Vocabulary.”

The book has many additional insights, some shared by co-author Nathan Shedroff in Make It So: Five Lessons in Interaction Design from Star Trek in UX Magazine.

As mentioned, there's even more on the website, Make It So: IxD Lessons from Sci-Fi, though there's too much coverage of the campy '70s flick Westworld. Here's a few presentation from the authors of the book and website.

TWAB 2012 – Chris Noessel (Star Wars UI not based on Stephen Few's advice; catch Padmé Amidala's skills after 7:20):

Make It So: MacWorld Presentation 2011 from Nathan Shedroff:

Chris Noessel: lessons of science fiction computer interfaces from Boing Boing:

(page 2)

Another good web resource is HUDs and GUIs, which stretches from The future of firefighting – C Thru Smoke Helmet HUD to IRON MAN 3 : HUD + GFX Process reel and Design of ‘The Avengers’ – Interface designs and HUDs.

Also on point is Pushing Pixels by Kirill Grouchnikov (a user interface engineer on the Android team), with articles like Breaking the barrier – conversation with Ash Thorp and “The craft of screen graphics and movie user interfaces” conversations with Shaun Yue,  Jayse Hansen, GMUNK, Joseph Chan, and more.

In a similar vein is Inventing Interactive by David Young, who sometimes interviews designers of movie interfaces, and features discussion of recent trends, as in Living Future Visions, which riffs off The Future (According to Corporations) by Robert Bolton of Idea Couture, which itself riffs off a Future Tuesday talk by Scott Smith (the Changeist). He sounds a bit like Neo, who once worked at MetaCortex, the uber-company that Apple, Google, Facebook and Amazon are all trying to turn into:

What Screens Want—Some thoughts on digital canvases by Frank Chimero was published November 2013. It discusses plasticity, UI, motion, and more. Back again to Bret Victor, who in A Brief Rant on the Future of Interaction Design, reminded us of behaviors and limitations not found in real humans bodies. As he points out elsewhere, “right now, today, we can't see the thing, at all, that's going to be the most important 100 years from now.“

Fast Company is also a great resource for treats, with articles like 16 Of The Year's Best Ideas In UI Design (2012) and 3 Principles For The Future Of Gaming, From A Google Game Designer, who is trying to jumpstart mobile-gaming on Android. Interesting but not strictly connected is Emotion and Motion: Using Movement Design to Shape User Experience presented at Google by Katherine Isbister:

Check out Helen Tran on emotional awareness in The Future of UI and the Dream of the ‘90s. Skipping ahead to the future, according to DesignWorkLife Wired has said that the movie Her will influence the future of UI design even more than Minority Report.

Fxguide often has great information on movie interfaces, for example Iron Man 3: more suits to play with by Ian Failes. And back in the day (2008), Dav Rauch from The Orphanage walked through Nuke work on the HUD sequence from Iron Man in two YouTube movies.

Backpedaling just a bit to avoid Microsoft Kinect (this time), check out Ivan Sutherland's Sketchpad. The author at one point commented that he didn't realize what he pioneered was hard:

• the first GUI program
• interactive graphics
• constraint-based layout
• pen-based input
• object-oriented programming

No Mimes Media published to relevant stories on Transmedia Colition: Why Hollywood Needs More Experience Designers and Why Hollywood Needs Experience Designers – Part II.

There's a few relevant more academic-type stuff floating freely, like the Creative Commons collection of articles and Human Computer Interaction in Science Fiction Movies, plus older ideas on anthropomorphism from Linnda Caporael and Reeves and Nass. And while Jakob Nielsen does make good points in Usability in the Movies — Top 10 Bloopers, his earlier site design had been nearly unreadable. For balance see, for example, In Defense of Eye Candy by Stephen P. Anderson: “We’ve all seen arguments in the design community that dismiss the role of beauty in visual interfaces… Lost in these discussions is an understanding of the powerful role aesthetics play in shaping how we come to know, feel, and respond.” Note also that animation is not just eye candy – but then, as Bruce Sterling noted, the people in these movies “are fighting Norse gods and space aliens.”

Recent fortune-telling comes from Wired in Why a New Golden Age for UI Design Is Around the Corner and from Hongkiat in 8 Next-Generation User Interface That Are (Almost) Here. CNET added to the fun of this year with The Next Interface: You, when a CES panel explored the future of interface, with keyboards and mice being replaced by voice, hands, and eyes:

While the tech for Editing Video in Mid-Air with the Leap Motion Controller (by Eric Escobar on PVC) with Adobe Premiere Pro isn't ready for primetime, maybe custom controls leveraging the Leap SDK could work with an After Effects plug-in. There are plenty of ideas. Editor's Keys is already customizing for Final Cut X (video). For recent news in this vein, see When Film Invades Your Brain: New Technologies Pushing the Cinematic Experience by Oakley Anderson-Moore, which expands on a Tribecafilm.com article from their Future of Film series with content from Researchers laugh in the face of flatness with new approaches to haptic displays (video) at Engadget.

The trend toward flat design hit big in 2013 and 2014. Even iTunes got more obscure. For background, see Long shadows: adding body to flat design in After Effects.

Getting closer to something of direct interest, there are number of After Effects and Creative Cloud users creating visions of futuristic user interfaces. Perhaps the most well-known is Mark Coleran, who's been involved in numerous feature films including The Island, The Bourne Ultimatum and Alien Vs Predator. Motionworks interviewed him Unplugged 9. Also check out Mark Coleran's Flow of Visual Design for Film, and The MossyBlog Interview with Mark Coleran. Here's his older reel, and The Reality of Fantasy from UX Week 2010:

NLE Ninja later discussed Luca Visual FX Hi-Tech Overlays, pre-rendered elements that might be handy for some. You can find AE-specific advice in A Heads-Up on Using Reticles by Chris & Trish Meyer, which uses Artbeats Reticles (reticle/graticule). See also, The Futuristic HUD is a useful older After Effects tutorial project from Videocopilot.

Others in recent press include Ash Thorp (podcast), David Lewandowski, Jayse Hansen, Navarro Parker, Bradley G Munkowitz (gmunk), Shaun Yue, and Joseph Chan. In addition, entities specializing in designing UI & display graphics have included: OOOii, Imaginary Forces, Useful Companies, Teknoel, zero one, Playback Technologies, West Media, Decca Digital, and Stargate Studios (more here). Here's a collage from the work of Prologue, as well as Cantina Creative on their use of After Effects (movies can be seen in Google Chrome, or on Windows). Plus, there's the Alien UIs Of Star Wars from Noel Rubin – Teknoel:

Articles discussing these people and sci-fi interfaces include :

• The craft of screen graphics and movie user interfaces – conversation with Shaun Yue at Pushing Pixels
• HUD and UI graphics in ‘Oblivion’ + titles here on PVC
• Jayse Hansen on creating UI for The Avengers, touch control, holograms, Galaga and Project Glass
• Meet The Man Behind The Fantastical Technology In The Avengers, Rise Of The Planet Of The Apes, 2012 And More [Q&A]
• ooo-ii: The Design of the Star Trek Movie Information Displays at Infothestics
• “Marvel’s The Avengers”: Taking the HUD to new heights, an Adobe interview with the HUD's creative director and screen designer
• Upping the HUD ante with Adobe After Effects, on the next-generation heads-up display effects for Marvel’s Iron Man 3, also from Adobe
• Later, both Ash Thorpe and Jayse Hansen posted on UI design (with reel) for Ender's Game. Jayse also posted some of the go-to scripts used in After Effects.
   

Adobe After Effects World Conference – Takeaways for UX Designers was posted by @UX_in_Motion, who attended the September 2013 gathering and liked Matt Silverman's presentation on User Experience. There's a bit more on motion and user experience in UX Motion Design, Part 1: Why? by Jen Rizzo, Motion Perfection by Ben Lee, and Motion & Animation: A New Mobile UX Design Material by Rachel Hinman. For examples in action, see Interface Animations and Transitions: where to get inspiration by Stepahnie Walter.

Creating The Post-Apocalyptic Sky World Of Oblivion, by Kevin Holmes and The Creators Project has a great introduction to the interface graphics of Oblivion. See especially GMUNK on “the grid” after 3 minutes:

Finally, here's some interesting but perhaps ancient info from Mark Coleran (2007):

“When I first started it, each artist would take a scene and be responsible for perhaps 10-20 screens in a movie. 10 years later you can add a 0 to those numbers at times. Anyone with any sense develops huge libraries of reference, large scratch pads of elements to re-use (illustrator artwork, photoshop files and after effects projcts and favorites) so that you can concentrate on the initial design and turn over the animation very fast. The time frames are shorter and the numbers vastly increased. I have absolute respect for the guys who do the TV stuff, they have it even harder. It can be easy to knock sub par work in this area as being unrealistic and sometimes cheesy, but the time frames are ridiculous at times. The complexity is generally nothing more than a visual trick. Rule of threes. Have at least three things animating on any screen and it looks busy and complicated.

For the most part we deal with the production designers and only get director involvement on 'hero' screens that contain important plot elements. The actual design and implementation… to be honest, there isn't that much focus for the most part. The only two directors I have ever dealt with who really did care was JJ Abrams on MI3 who was very focused on what he wanted design wise, and Alfonso Cuarón on Children of Men who took a lot of care and consideration about what was done so it matched and worked well with the particular look he was after.

The importance varies widely. Some people care and get involved a lot, other far less so. Some don't even look at them until the first day they see them on set for shooting. It can cause problems!”