(Page 2 of 2 pages for this article  <  1 2)

Filed under: Business •Cameras •compression •Distribution •Hardware •Post Production •

HPA Tech Retreat 2010 - Day 3

SOA, Collaborative Networking, File-based Workflows, Sync & Timing…

SOA What? Panel Moderated by John Footen, NTC, with David Rosenberg, Azcar; David Carroll, Sony; Felix Froede, Ceiton; Joey Faust, NTC. SOA is “service oriented architecture”, a (mostly) standardized, web-based way to link together disparate systems to automate business processes.

Froede: SOA can reduce dozens of different protocols to a single standardized interface. Business process management needs a focus on dependencies, not just functions. For example, in DVD production, function/task groups include subtitles, memos, video processing, audio, etc., with many cross-functional dependencies.

The coordination of complex processed can be automated, letting folks focus on doing their individual functions, improving control, reducing manual handoffs, and better tracking. None of this is new; logistics / chemical / automotive industries have known this for a while.

Rosenberg: SOA business drivers: reduces complexity of existing systems, lower costs though use of common standards, integrate info from disparate sources, increase flexibility and agility, extend lifetime of legacy systems, reduce custom software. File-based workflows let you develop new, more efficient workflows. Corporate IT is very good at keeping email running, etc. but not so good at media-related workflows. Media-centric IT skills include knowledge of broadcast workflows, deep understanding of component software and web-based services, experience with XML / SOA / object-oriented systems.

Is SOA right for you? Start with a readiness assessment; do a process decomposition to identify shared services, with good documentation and understanding. Identify bottlenecks and points of failure. Know the capabilities of legacy systems. Accept the need for change; engage executives (there will be turf battles and pushback); know 5-year roadmap. Understand that SOA is a way of thinking, it’s not a product on a CD. It’s complex; it’s not for everyone (size matters). Start small. Success requires highly skilled SOA developers, which most media organizations don’t have—so you may need to hire consultants, making sure that they know the broadcast business as well as SOA.

Faust: SOA makes cross-device compatibility easier, increases application agility, multi-purposing, software as a service. But can we trust the vendors to do a good job? Is the “rewrap everything” model workable? Will there ever be a “big SOA” project that’s financially justified?

Example: two transcoding systems from different vendors. Both do the same thing, but have very different interfaces and message protocols. Re-wrap with an SOA layer to make compatible? Possibly, but it may be a lot of work.

Will we ever be able to design generic classes of media services? Yes, possibly, but probably not for everything we do. How do we deal with it: big web-services layer with a SOAP interface, or something else? Consider REST (representational state transfer): get, put, post, delete. Constrained number of operations. Simple to understand. “Fractal realization” in that same ops repeat at various levels. Moves standardization problems into the metadata problem: what to ship around.

Unilateral case for business SOA not likely in most organizations. Start smaller and incrementally; consider REST; the metadata problem may wind up eating the SOA problem.

SOA is inserting a layer of abstraction, exposing services with common interfaces. A change from tightly-coupled systems that require complete re-architecting if workflows change.

Carroll: file-based workflow hasn’t delivered on all the problems: it’s not really faster, cheaper, easier after all. Many different workflows (different workflow on every project), security issues, metadata problems. Media SOA is designed for media transactions: long-running processes. Resource management, dynamic resource allocation, queue and priority management. SOA controls file access and movement, handles security, is aware of hierarchical storage management. SOA workflow manages assets and tasks, automates non-creative work, shares resources more effectively, automatically repurpose content, speed throughput.

Example: “2012”: 30 million frames, 347 hours of content, 240 TB @ HD res, 1.44 PetaBytes at 4K res, all online 6-18 months. How to move data between different 4K islands? Can’t sneakernet tapes around, after all.

The holy grail for media SOA is a graphical workflow UI, so you can automate workflows with drag ‘n’ drop without writing a line of code.

Discussion: Cross-vendor media SOA tends to work using introspection (“what services do you offer?”) whereas IT SOA tends to develop extensions of SOA using WSDL. How does SOA differ from automation? Automation is sort of a blind sequential process; SOA is a more flexible business-rules-driven workflow, or an event-driven workflow. Problems with changing people’s workflows? A lot of turf protection; have to convince folks that the media assets and workflows belong to the organization, not to them personally.

Collaborative Networking Panel Moderated by Leon Silverman, Disney, with Peter Wilson, HDDC; Laurin Herr, Pacific Interface; Bertrand Darnault, SmartJog; Anthony Magliocco, Entertainment & Media Technology Marketing.

Silverman: In the ‘80s I discovered the modem, and all of a sudden my computer wasn’t just sitting by itself… in the mid-‘90s 1.54 Mbit/sec for $2500-$3000/month was a big deal… it’s become essential.

Wilson: London MUPPITS (Multiple User Post Production IT Services) project for collaborative networking, started in 2008 to delegate rendering and storage around town and out-of-town. Partners BBC, DTG, HDDC, IT Innovations, Molinare (post house), ODS (Texas oil company IT co), Pinewood Studios, Smoke & Mirrors (post house), SoHoNet (hard-drinking post industry high-speed networking infrastructure), Technology Strategy Board. SOA-based wide-area networking

Offers security, management, service control, collaboration, automation, and marketplace (since everyone is networked, it’s easier to share jobs). Core is GRIA (http://www.gria.org/), multiple data warehouses, a rule engine, resource tracking (for billing). Typical rule: “send proxy of last take to director’s iPod”. Based on SOA, allows productions to choose desired services wherever located as long as it’s on a high-speed net.

Two main thrusts: remote rendering (includes rendering estimator so you can actually suss out how long it’ll run and how much it’ll cost), and tapeless workflow (BBC very interested, using BBC Ingex; see http://ingex.sourceforge.net/). http://www.MUPPITS.org.uk Fundd research project ends in about six months but everyone expects the MUPPITS project to continue independently and invites interested parties to join up.

Herr: CineGrid is a university-driven non-profit organized in 2004, for folks who want to work (non-commercially) with media over high-speed networks. 1 Mbit/sec and 10 Mbit/sec over Global Lambda Integrated Facility, GLIF. Uidn MPEG-4 and JPEG2000 codecs. Worldwide demonstrations, like shooting DALSA in Prague, deBayering in San Diego, color-correction in Canada, all networked.

CineGrid Exchange: distributed media repository, by spring 2010 will have 256 TB with 10 GigE infrastructure, nodes in Toronto, Prague, Chicago, Hollywood, and Monterey. Collaborative 4K grading, sound mixing: studios synced within 1 audio sample at 48kHz, which required careful latency evaluation across the network and even within the mixing studios (acoustic delays).

Darnault: On connectivity: IP over satellite has great coverage, good for one-to many multicast. Leased lines: dedicated circuits, clear QoS, very predictable. Internet: cost effective if/when available, but no guarantees. Community networks (SohoNet, etc.) have high bandwidth, good community connectivity. Transfer apps: depends on the network and its problems. Need reliability, traffic management, flexibility, and proof of delivery. UDP is often better than TCP but sometimes (e.g., in Asia) TCP multi-connection is the best protocol. Multi-stream trunking is often the most cost-effective high-speed link; two DSL lines paired may be faster/cheaper than a single higher-speed line.

SmartJog software, tools, and automation: automated rules-based transfers, transcodes, burn-ins, and a web services API. Example: send transcoded media to 10 different recipients with a common tracking watermark (burned in at the sending server) plus a per-recipient watermark (done at the receiving server).

Magliocco: Private Cloud for Content Creation. EMTM design goals: use standard IP protocol, interoperates with common tools, but uses a private network. Multiple security levels, low cost flat rate billing, international connectivity. Players: AboveNet, SohoNet, etc. Gigabit as a minimum, 10 Gig preferred (or faster). Example: MediaXStream Studio Culver City, CGI crated in San Francisco, sound in Culver City, Sound in Santa Monica. Another: feature film with Scratch system in LA, SF interconnected. 40 Gig networking now available in some markets.

Discussion: Cost of MUPPITS? Multiple millions. What bridges these different networks? CineGrid members peer with each other. Different vendors need to interoperate. Ethernet’s spread from LANs to WANs makes interop easier. Still making phone calls to get people to patch nets together.

File-Based Workflow Panel Moderated by Howard Lukk, Pixar, with Brad Collar & Michael D. Smith, Warner Bros; Gary Morse, Fox; Bob Seidel, CBS; and Arjun Ramamurthy, Fox.

Collar: Digital End-To-End (DETE) eMasters and eDubs (Warner Bros has 500 different transcoded formats) with eMasters stored in a digital vault to strike dubs to whatever distributions are needed.  Standard-def over 15k eMasters. 22 min TV show delivered in 25 min (best case) to 8 hours (worst case). HD eMaster: 1920x1080, 4:2:2 10-bit 23.976fps, uncompressed audio 16/224bit 48/96kHz. Why not 4:4:4? Service masters are simply for dubbing from, not original sources. JPEG2000 in MXF wrappers.

Testing: what’s the bitrate, quality, and efficiency of eMastering? 100 Mbit/sec CBR works, but variable PSNR (peak signal-to-noise ratio); so tried VBR (peak limit 250 Mbit/sec), with wildly varying VBR but more consistent (and higher) PSNR.

What’s the impact of processing for eDub quality? Focus on Bu-ray testing, both created from eMaster, and from high-quality source material without eMastering. Tested at 15 Mbit/sec and 23 Mbit/sec average data rates. Overall, about 0.044 - 0.060 PSNR difference with VBR eMasters (negligible); 036-0.43 PSNR diff with CBR eMaster encoding: about eight times worse!

Why such a difference in VBR vs. CBR results? JPEG2000 flicker artifact: same frames but with small, localized difference, say a small graphic element that changes from a moon to a cloud, and from a simple to compress item to a difficult to compress item. Thus the small change will rob bits from the rest of the frame, so even the unchanged areas will show a degradation. This artifact isn’t visible in the eMaster, but it winds up leading to “bit waste” in motion-compensated codecs like h.264, so it degrades prediction—in effect, every block in the frame changes from that slight degradation in the eMaster.

Morse: Fox initiative to review entire ecosystem. Endless changes ahead! An explosion of formats… Videotape is losing its ability to keep pace with the 24/7 On-demand world; stuff needs to be turned too quickly for tape ingest to keep up, especially for international clients already 8 hours ahead of us. Distro game plan: Digital Delivery Initiative. Effective 1 June 2011 everything will be delivered as files. Faster, higher quality, better security, “green” (currently industry deliverables are 85% tape, 15% digital; with manufacturing, shipping, disposal, tape ain’t green!). “It’s a Mad, Mad, Mad, Mad, Multiformat World”: Files, files, and more files; small ones, big ones, metadata, other ones… Capturing HD, capture once at a high bitrate (looking at JPEG2000), store under Fox control, create deliverables from that single capture. Components: digital media processing facility, catalog management, digital vault, and work order management (old catalog on 23-year-old IBM AS/400 system!). Security: watermarking, fingerprinting, encrypted delivery. Final thoughts: JPEG2000, disaster recovery, file-based network deliverables.

Ramamurthy: file-based workflows orchestrated via a BPMS (Business Process Management System). Sine 2006 180,000 files created and delivered, sent out to a multitude of destinations. Why BPMS / SOA? Integration through watch folders results in redundant copies, it’s hard to manage and forecast, no standard dashboard control, but it’s dead simple to set up and deploy. With BPMS / SOA, an ESB (enterprise service bus) connects all elements and it all just works (more or less, grin). How to implement? Identify workflows, map process flow, understand roles, define monitoring , identify toolchest maturity and API maturity. Looking at even simple workflows reveals a lot of complexity, a lot of dependencies; do you want people to change roles or assume multiple roles? SOA requires web services, SOAP/XML over HTTP. But most of our devices have command-line interfaces, serial lines, custom APIs. Need WSDL, UDDI. Stated integration with Digital Rapids (encode/transcode), Baton (QC), Rhozet (transcode), custom delivery engine, custom asset management systems.

It’s just gone live; come back next year to find out how well it’s working.

Seidel: “Pitch Blue” realtime HDTV store-and-forward delivery system. In April 2009 three companies found a need for HD distribution: GDMX, Ascent Media, CBS Worldwide Distribution formed a company to distribute to 800 TV stations. Each station gets a free MPEG-2 TS (transport stream) recorder (TSR) fed from satellites. Fully automated, can “call home” if there are TX/RX errors and ask for a re-send. All video is 1920x1080 @ ~15 Mbit/sec, Dolby 5.1 or L/R mix. All boxes monitored every 10 minutes. Boxes have 3 TB of storage, 100 hrs of SD or 370 hrs of HD. Station can choose 1080i outputs or downconverted outputs. Three disks, plus a flash drive for rebooting if all hard disks get trashed. Box has FTP connectivity to other play-to-air servers if using box’s playback isn’t done.

Discussion: CBS et al. built their own TSR box because many of the third-party solutions had exorbitant costs. Fox used Oracle middleware for their ESB after fairly extensive tests; also, Fox folks already knew Oracle.

Peter Symes, SMPTE: SMPTE Action on Future Timing & Synchronization. Current references are about 30 years old, based on color black. Not easy to sync with audio, lock 50 Hz and 60 Hz sync together. Timecode is also 30 years old, designed for edge track recording, tops out at 30 fps, getting creaky: we need new timing and sync.

User requirements on TRL: Time Related Label. Master clock feeds a Common Sync Interface, which feeds “some sort of transport” like Ethernet. Includes clock data including time zones, etc. Over 100 interested parties from broadcast, telcos, IT. Data structure for labeling video frames; resolution up to 1000 fps, sub-division down to fields & audio sub-frames. Track 2:3 pulldown, rate indication, multiple “timecodes”, varispeed and slo-mo support, etc. 

User requirements on sync: value & economy; deterministic phasing, absolute time reference, external lock, Time-Of-Day info, leap second management, time zones. Need sub-nanosecond precision for subcarrier-based work (NTSC through a switcher) but want ability to extract/generate lower-precision sync at lower cost. Prefer to use common, non-dedicated transport (Ethernet?), have a 1 km distribution range, fast master/slave lockup and relock after disruption.

Three parts: Master unit with a CSI, a sync transport (e.g., IP network), and a slave with a CSI. Proposed format: a data packet with the CSI dataset, and a time reference when the dataset is valid (IEEE 1588).

What SMPTE needs is input for the contents of the time label, to make production’s job easier. Two TRL types proposed: Type 1 is a pure time stamp, nice for absolute timing; type 2 is start time plus media unit count (e.g., frame count), nice for counting frames. Technical Committee 33TS, also working groups. Questions remain: does CSI design meet user requirements? How to use IEEE 1588? And so on; if you have ideas, please participate in SMPTE’s process.

What Just Happened? - A Review of the Day by Jerry Pierce & Leon Silverman. Q: how many folks here have Blu-ray? Most. How many have used BD-Live? Perhaps half. More than one? Only a few.

Dynamic range is definitely controversial, often misstated, often misunderstood… One thing we’re seeing is a desire to defer grading changes as late as possible; want enough bit-depth to be carried through. In the ASC camera assessment test showed is that the chokepoint is the 10-bit DPX file: the limiting factor in end quality (!) as most cameras capture much more nowadays. ASC IIF may have some solutions… Social networking works: the Mo Henry wikipedia page was updated within one hour of her speech here… Demo room hits: Joe Kane’s Samsung projector (discussed in my final report after tomorrow’s sessions)… file-based transfer from ProRes into a DCP (easyDCP)… Panasonic’s 3D camera out of its glass case, making pictures.

Day 4, Friday, is the final day; my report will appear over the weekend.

More:

Tech Retreat Day 1.

Tech Retreat Day 2.

Tech Retreat Day 4.

Other Tech Retreat coverage.

Demo room demo descriptions.

16 CFR Part 255 Disclosure

I attended the HPA Tech Retreat on a press pass, which saved me the registration fee. I paid for my own transport, meals, and hotel. The past two years I paid full price for attending the Tech Retreat (it hadn’t occurred to me to ask for a press pass); I feel it was money well spent.

No material connection exists between myself and the Hollywood Post Alliance; aside from the press pass, HPA has not influenced me with any compensation to encourage favorable coverage.

(Page 2 of 2 pages for this article  <  1 2)

Name:

Email:

Location:

URL:

Smileys

Remember my personal information

Notify me of follow-up comments?

Submit the word you see below: