MulticoreWare’s LipSync technology uses deep neural networks to autodetect audio/video sync errors by “watching” and “listening” to videos. According to MulticoreWare, NVIDIA GPU-accelerated models find and match instances of human faces and human speech in up to 2-3x realtime, enabling highly scalable quality control for file-based or streaming content. LipSync technology is being demonstrated at NAB2017 by MulticoreWare, developers of the x265 HEVC video encoder.
Typical causes of audio-video misalignment include transmission and transcode errors, incorrect video cuts, or incorrect framerate conversions. With an ever-increasing amount of video content, sources, transmissions, and transcodes, sync errors can occur more frequently. MulticoreWare developed LipSync to automatically detect synchronization errors to ensure content integrity at scales where manual verification is intractable or expensive.
According to its creator, LipSync combines the latest deep learning neural network techniques with statistical analysis to test videos without relying on digital fingerprinting or watermarking. Audio/video synchronization detection is performed by analyzing moving lips and faces and listening for human speech patterns, similar to how a human viewer would watch a video. Unlike a human viewer, LipSync can process file-based content at 2-3x real-time or analyze multiple video streams in real-time using NVIDIA GPU-accelerated servers.
“We are the first-to-market with a machine learning-based solution,” says Arun Ramanathan, VP and GM for Machine Learning at MulticoreWare. “This was made possible by combining our expertise in video processing, GPU computing and deep learning.”
NVIDIA GPUs enabled the development of LipSync into a realtime solution. “LipSync is an impressive example of how deep learning, accelerated by NVIDIA GPUs, solves major challenges in creating and distributing video content,” said Will Ramey, director of Developer Marketing at NVIDIA. “This innovative application addresses a pervasive problem for the entire industry.”
Telestream and MulticoreWare are partnering to make LipSync available to enterprise customers. Shawn Carnahan, CTO of Telestream said that, “Identifying audio-video sync errors has long been a challenge in our industry and Telestream is excited to offer an automated solution using deep learning technologies. Telestream is working closely with MulticoreWare to integrate LipSync into our products.”
Telestream recently expanded its video quality-control portfolio with the acquisition of VidCheck and IneoQuest.
MulticoreWare is currently demonstrating LipSync technology for new partners and licensees. Video quality control providers, broadcasters, and content distributors can integrate LipSync into their existing software or pipelines, or use it as a standalone product on-site or in the cloud. On-demand usage is supported on Amazon Web Services (AWS), Google Cloud Platform, and other GPU-accelerated cloud services running Windows or Linux. Licensing models include perpetual on-premise installations, integration licenses, and per-usage pricing.
For more information visit lipsync.multicorewareinc.com or MulticoreWare’s booth SU14002 at NAB 2017.
Upcoming articles, reviews, radio shows, books and seminars/webinars
Stand by for upcoming articles, reviews, and books. Sign up to my free mailing list by clicking here. Most of my current books are at books.AllanTepper.com, and my personal website is AllanTepper.com.
Si deseas suscribirte a mi lista en castellano, visita aquí. Si prefieres, puedes suscribirte a ambas listas (castellano e inglés).
Listen to his CapicúaFM show at CapicúaFM.com in iTunes or Stitcher.
FTC disclosure
No manufacturer is specifically paying Allan Tépper or TecnoTur LLC to write this article or the mentioned books. Some of the other manufacturers listed above have contracted Tépper and/or TecnoTur LLC to carry out consulting and/or translations/localizations/transcreations. Many of the manufacturers listed above have sent Allan Tépper review units. So far, none of the manufacturers listed above is/are sponsors of the TecnoTur programs, although they are welcome to do so, and some are, may be (or may have been) sponsors of ProVideo Coalition magazine. Some links to third parties listed in this article and/or on this web page may indirectly benefit TecnoTur LLC via affiliate programs. Allan Tépper’s opinions are his own.
Copyright and use of this article
The articles contained in the TecnoTur channel in ProVideo Coalition magazine are copyright Allan Tépper/TecnoTur LLC, except where otherwise attributed. Unauthorized use is prohibited without prior approval, except for short quotes which link back to this page, which are encouraged!