As media companies move toward cloud storage, ideas about storage and infrastructure strategy are evolving. Traditional storage tiers are becoming less defined, backup practices are changing. But bad habits are the hardest to break, and there are certain storage sins that we see month in, month out. The worst part? These mistakes aren’t just bad practice day to day – they could cost your facility thousands of pounds in the long term.
“The number one issue we see with storage is a lack of long-term strategy, especially in growing facilities,” says Technical Pre-Sales Consultant Phillip Boettcher, Jigsaw 24. “The vast majority of facilities we work with have always archived footage for their clients, and that’s definitely something most media companies expect a post house to do. But the fact is, now that so many people are shooting in higher resolutions or with higher shooting ratios, your primary storage is going to fill up fast. Strategizing how data moves through your facility needs to be something you do early and review often.”
Let’s take a look at some common mistakes…
1. Relying on a random array of disks
Many new facilities don’t have a formal archive process, instead dumping completed projects onto hard drives (or, more recently, into low-cost cloud storage) and leaving that as their archive. However, this system fails on almost every level, from data security (there are no access restrictions on a shelf full of drives) to immutability (it’s easy to accidentally overwrite data) to accessibility (manually searching drives or unstructured cloud volumes will push your retrieval times up to days rather than hours).
2. Relying on proprietary formats
The cloud storage marketplace is crowded, and when choosing a provider you need to bear two contrasting points in mind. Firstly, scattering your projects over multiple cloud volumes from multiple vendors will make them harder to retrieve; secondly, that leaning too heavily on a single vendor can lock you into their workflow for the long term, either by locking your data into a proprietary format, using proprietary erasure coding formats to build redundancy into their systems, or using proprietary egress and transfer technologies to inflate the cost of migrating your data to a point where it’s simply not viable.
3. Relying solely on your production storage
Production storage is expensive, which is why your live storage volumes should only ever be a temporary resting place for your media. However, we see many companies storing everything from live projects to deep archive on the same array and simply adding to it when they run out of space. This is a huge and unnecessary cost: projects you no longer access to can be moved to much more affordable archive formats, be that LTO, Glacier storage or long-term disk storage.
And because file sizes are constantly increasing, sticking to a tierless system like this is only going to get more expensive, with larger storage needing to be purchased more often. Plus, the more media you amass, the more expensive it will be to transfer to a different platform when you finally do adopt a futureproof archive strategy.
4. Relying on separate data silos
Another issue we see often is that of data siloing. Media that’s no longer in use is placed in offline storage, a cloud volume or an LTO and forgotten about, with creatives unaware that content exists and can be re-versioned, or unable to access it in a timely way because the archive media is so isolated from their day-to-day storage. We’ve also seen facilities move from one system to another and leave behind silos in each, resulting in isolated pockets of media that are difficult to access and therefore unable to generate any further revenue.
Investing in your infrastructure strategy
At the root of most of these failures is a lack of futureproofing. Having an effective, futureproof strategy for managing your storage will not only save you money, it can help you generate revenue.
“There is a huge difference between simply having a copy of your data somewhere and actually being able to find, retrieve, and reuse said data,” says Phillip. “If you tell clients you’re going to hold on to their data for five years, you should have a clear plan for where that data will be over that time period, how you’ll ensure its integrity and security, and how you’ll retrieve it when the time comes. Hand in hand with that, planning like this ensures your end result is an effective, comprehensive media library that helps your creatives remonetise your content, whether that’s by reversioning work for a client, or finding clips you have the right to reuse rather than shooting something new.”
While every facility is different, there are a few core principles you can stick to:
• Make sure you files and their metadata are stored in open formats and that you’re not going to be forced to follow rules that can make it difficult to move files in the long term.
• Avoid platforms that force vendor lock-in or make removing your data unnecessarily difficult.
• Avoid having all your copies of a given piece of media tied to a single provider. If you lose access to a given cloud volume, you should still be able to get your hands on a copy of the data.
• As a corollary to this, ensure you have an effective method for searching for and retrieving data, no matter which platform it is in.
The advantages of a hybrid or multi-cloud approach
“Taking a hybrid or multi-cloud approach is one of the best ways to approach you long-term storage and infrastructure strategy,” says Phill. “By combining on-premise storage with an active archive for projects you’re likely to revisit and a deep archive for media you’re not going to use, you can strike a balance between keeping your data accessible and managing your ongoing storage costs. If you know what files will be on your production storage for six months, on your near archive for two years and in deep archive for another two, your storage spend is going to be far more predictable than if you’re buying storage ad-hoc and potentially paying to egress huge amounts of data from one platform to another.
“Plus, if you have an on-premise backup of your media, you don’t need to egress that data from the cloud in order to switch platforms or locations: you just turn off the cloud storage you no longer need and use your on-premise copy to re-ingest your media and its metadata to a new platform.”
The challenge then becomes ensuring your end users and media managers can locate media across all of these locations.
“Asset management is a huge part of managing your storage effectively, and there are some very intelligent solutions out there,” says Phillip. “Object Matrix Vision is particularly good – it’s a web-based management application that brings together content from on-premise, near archive and deep archive storage into a user-friendly GUI, so you can see and share everything you have in a single pane of glass, whether it’s on-premise, in MatrixStore Cloud storage or being held in a third party you’ve integrated with your Object Matrix solution.
“And because Object Matrix have dedicated solutions for moving ageing media from online to archive, verifying the data as it goes, you’ll always have a clear retrieval path and can be confident about your data integrity.”
Ultimately, whatever platform(s) you choose, the key to monetising your assets and keeping your storage cost profile manageable is a robust system for ageing assets from one level of storage to another, and a clear understanding with your clients how that process will work, and how it will impact what you can retrieve and how long that will take.
“Taking a knee-jerk approach to storage is going to cost you more, raise your retrieval times, and possible land you in hot water with clients because you don’t have proper access control and data security around their assets,” says Phillip. “A hybrid approach gives you the control and capacity that you need without forcing you to find space and money for a huge on-premise array. It’s one of the best decisions you can make at the start of your facility’s infrastructure develops.”
About Jigsaw 24
Jigsaw24 Media is a specialist division of Jigsaw24 and provides services and technology solutions to the media and entertainment, education and corporate sectors. It’s the only UK-based company of its kind that has in-house system integration capabilities, one of only two Avid Elite partners in both video and audio in the UK and Avid’s global partner of the year. Jigsaw24 Media’s team of industry-recognised experts design, deliver, integrate and support end-to-end solutions for some of the nations’ biggest broadcasters and facilities, underpinned by partnerships with over 30 leading technology vendors including Avid, Adobe, AWS, Nutanix and EditShare. With headquarters in Nottingham, an office and demo space at the heart of London’s post-production community in Soho, and a nationwide support team, Jigsaw24 Media provides local services on a national scale. For more information visit media.jigsaw24.com.
About Object Matrix
Object Matrix is the award-winning software company that pioneered object storage and the modernisation of media archives. It exists to enable global collaboration, increase operational efficiencies and empower creativity through deployment of MatrixStore, the on-prem, hybrid and cloud storage platform. Their unified deployment approach ensures content spans on-prem and cloud storage whilst their focus on the media industry gives them a deep understanding of the challenges organisations face when protecting, processing and sharing video content. Customers include: NBC Universal, TV Globo, Warner Bros. Discovery, MSG, ATP and the BBC, to name but a few!