Using MatrixStore for Netflix Production Asset Compatibility
In 2018 Netflix purchased $12bn of content. It is predicted that the figure rose to $15bn in 2019 and that spending will hit $17.8bn in 2020. 85% of that spend is estimated to be on original content. This dwarfs the budgets of individual national broadcasters (e.g., the BBC has a total budget of £3.7m but only a portion of that is spent on new content).
It is, therefore, no surprise that Netflix has quickly become a technology leader on how production should be made not just for itself but also in setting standards that other organisations have also signed up to as both sensible and practical.
One question we get asked is “Can MatrixStore help us adhere to Netflix Data Management requirements?”
And, the relevant Netflix article reads like Object Matrix marketing!:
Production Assets: Data Management
The verification and backup of data, including original camera files (OCF) and audio, throughout a workflow is crucial. Data loss can mean losing significant investment both in planning and shooting. Our requirements are designed to help minimize costly incidents during production.
Below are Netflix’s critical Backup Rule requirements:
|Netflix Backup Rule Requirement||
|Hold at least 3 copies of all Original Camera Footage (OCF) and audio at all times and on 2 different types of media.||
MatrixStore holds 2 copies with RAID6 and the original camera media could also be held to keep the data on “another media type”.
Also, see below.
|Maintain at least 1 backup offsite||Replication can quickly make an offsite backup. Other options are to use MatrixStore’s hybrid capabilities to make a copy into a cloud location – this can also cover the requirement to hold 3 copies of OCF.|
|RAID must be connected to computer by a hardware RAID controller. Software based RAID should not be used||MatrixStore uses Hardware RAID. Other systems (especially other object storage systems) tend to use less reliable software RAID.|
MatrixStore of course goes even further with automated data recovery if, e.g., one location goes down as well as multiple other data authenticity and data security functions.
Netflix also, sensibly, requires checksum verification of content. A checksum can guarantee that the content hasn’t been accidentally or maliciously changed during storage or transmission. For instance, Netflix writes that:
Checksums are incredibly valuable for detecting the slightest change in a file, which can occur at any point during transfer or storage. For example, if a movie file is corrupted during an upload from a hard drive to a server, the checksum of the copied file would differ from the checksum of the original movie file — even if a single bit is off. This comparison is referred to as checksum verification.
Netflix supports both MD5 and xxHash64BE checksums, and you’ll be unsurprised to hear that the stronger of these, MD5, which is also cryptographic, is supported by MatrixStore at all levels of data storage and transmission.
Netflix Checksum Best Practice
|Generate checksums for original camera footage (OCF) and store as a manifest||This is a stage prior to storing in MatrixStore|
|Calculate checksum during transmission to MatrixStore and during storage||MatrixStore vaults can be set to perform this in MD5|
|Compare stored data to checksum in OCF manifest files||A simple script can be provided to perform this checking. The advantage with MatrixStore is that the MD5 is literally compared to the read back copies of data thus showing that the content hasn’t just been transferred and stored correctly but also reads back correctly at all MatrixStore storage locations.|
|Checksum manifest, such as MHL, accompanies the files through all transitions||MatrixStore can keep the (verified) MHL file with the content at all stages.|
On Set Media Re-purpose
Finally, Netflix recommends on set media re-purpose. Or as Netflix state:
Original Camera Media may only be reformatted after both:
- Visual QC of all footage has been performed against camera reports/script notes, with all footage accounted for and signed-off on.
- OCFs have been copied and reside in a minimum of three storage mediums (LTO and/or RAID1, 5, 6 or 10).
On set RAID1, 5, 6 or 10 may only be reformatted after both:
- Visual QC of all footage has been accounted for per camera reports and script notes and signed off by editorial.
|Netflix Best Practice||MatrixStore|
|Visual QC||This can be carried out through MatrixStore Vision browsing and content viewing. Of course, comparing to reports/scripts is a manual action until AI can take over!|
|OCFs have been copied or reside in a minimum of three storage mediums||As discussed, MatrixStore is providing two of those locations|
Netflix Folder Structure Compliance
Netflix also requires that production assets conform to a set of standards. This requires that a folder structure containing the following elements is created:
- Show Title”
- YYYYMMDD (Shoot Date)
- Camera Media
- Camera roll folders (e.g A001C001) copied into the Camera Media folder must be an exact representation of the camera card contents and contain valid checksums.
- Sound Media
- Sound roll folders (e.g S001) copied into the Sound Media folder must be an exact representation of the sound card contents and contain valid checksums.
DropSpot – an ingest tool for MatrixStore can help with this. By simply creating a metadata form in DropSpot you can set it to auto dump content into a pre-defined folder structure that turns the metadata captured (or entered by the user) into the fields that define that structure.
This can avoid commonly made mistakes such as an ingestor putting content into a who-knows-where folder as well as spelling mistakes. This method is similar to ones that we have used for many years, for instance with BBC’s Strictly Come Dancing ingest workflows.
We’d be more than happy to provide a demonstration!
The OM View
Netflix is in a great position to see what can and has gone wrong with the supply of productions across a broad range of its suppliers and has come up with a very sensible and practical set of guidelines.
However, the last thing post-production houses need is the headache of countless hours of making copies of data and managing an ever-growing range of requirements.
That’s where MatrixStore steps in.
Out of the box, it provides a deep level of content protection, data security, auditing and metadata control. Out of the box, it helps you to become Netflix compliant.
About Object Matrix
Object Matrix is the award winning software company that pioneered object storage and the modernisation of media archives. It exists to enable global collaboration, increase operational efficiencies and empower creativity through deployment of MatrixStore, the on-prem, hybrid and cloud storage platform. Their unified deployment approach ensures content spans on-prem and cloud storage whilst their focus on the media industry gives them a deep understanding of the challenges organisations face when protecting, processing and sharing video content. Customers include: BBC, Orange, France Televisions, BT, HBO, TV Globo, MSG-N and NBC Universal.