• 24 September 2018

Media Storage is Horrible

Media Storage is Horrible

Media Storage is Horrible

Media Storage is Horrible 1024 522 Object Matrix

Media Storage is Horrible

(Says the CEO of Object Matrix, the Leading Storage Company!)

It’s probably not something that you would expect me to say, but let’s face it, media storage has been horrible since the day dot. To see where I am coming from and to describe just a few of the bad practices we have picked up for media storage we need only to look at how we’ve stored media to date.

Media Archives Have Literally Burnt Up

Nitrate tape. It was the technology that nearly every major film was recorded on from 1880 through to 1952. For the filming through to archive, the process was simple. Film something, send it to editing and you already have your ready made archive – the original film. Put it on a shelf.

The problem? First of all: film on nitrate degrades in quality over time and only if kept at exactly the right conditions could it last for up to 25 years without quality loss. Secondly, you end up with shelves of tape. Once I asked a film institute how much storage they had, expecting them to say “25 Petabytes” or some such similar answer. Instead the answer came back “9 acres”. Yes. 9 acres of warehousing full of film. And even if the film is indexed, you can only begin to imagine the trouble with finding the right place in the right film in time to use that shot, let alone answering questions like “give me a selection of shots including a Rolls Royce Silver Phantom”.

And, thirdly, nitrate could and would auto-ignite. This has caused fires at MGM, Universal, the United States National Archives and Records Administration to name but a few. These fires have lost originals of Tom and Jerry, countless movies like Cleopatra through to 12.7 million feet of newsreel. Cultural heritage lost forever.

But that’s the past right? We obviously learned well from nitrate and acetate film disasters?

Roll on (no pun intended!) the 1970s and digicam, DVD,  through to USB keys. In fact, digicam variant cameras were made all the way up to 2016. Much better right? Well not really. The fact remained that our cultural heritage, your digital property and therefore potential to make money is still being put on to degrading, difficult to search and index, potentially combustible media.

In fact in some ways even worse. The media had a shorter lifespan and the plethora of standards and formats makes it, if anything, harder than before to use the media as an archive media.

LTO Based Solutions Are NOT the Answer

So instead the answer became “let’s stick it on tape”, which became, “let’s stick it into LTO”. After all LTO has a longer lifespan… right? Well of course, as we all know, LTO has gone through 8 different formats with a change in format every 2 years or so. And with LTO 8 machines unable to read even LTO 6, this has resulted in the need to migrate archives or see them go to dust once again. In over 100 years of media storage – it seems we have gone nowhere.

Disk Based Solutions Are NOT the Answer

But hang on… we have disk based RAID systems, right? This is very true. Disk based scalable storage is very good indeed at storing archives of storage, storage that is connected to the tools that can search and use that archive. But is it really the answer?

First off, companies have a big problem. In the past they bought a disk based raid system for storage, but it quickly became undersized and later became out of date. So the archive on the storage needed to be migrated, again. Secondly, as that company bought the next disk based solution, probably from another manufacturer, maintenance became a mare, and that shiney new (and expensive) platform from a few years ago became valueless over a few years.

Nature has the Answer (1) DNA

So, as the CEO of a storage company looking to these issues we looked to nature. DNA to be precise. Just look at how amazing it is:

  • 1.2g of DNA can store a Petabyte of data
  • It survives corruption
  • It survives war, fire, 1000’s of years
  • 99.9% of our DNA is the same, the other 0.1% is what differentiates us

OK – even though some people are literally looking at DNA for storage, I don’t want to stretch the analogy too far but the point here is that DNA: (1) has the principal of multiple copies (2) can migrate from one carrier to the next generation.

I fundamentally believe that over time the cost of hardware for storage will become insignificant, but it will be the management of that storage, and in particular in this context, the ability to change out the hardware with automatic management less migration of the data to the new hardware.

Nature has the Answer (2) Intelligence

Artificial intelligence is the most cliched term of 2018, so let me be more precise. If we had intelligence and time, we’d know that this piece of film is needed in the UK and that production needs to be worked on in a post-production house in the USA. We’d realise that having data in just one location puts it at great risk and that that film should never be changed from its original format. AI should be making those decisions for us.

And, if we had 1000 eyes and a million hours we’d tag and analyse all the video in our archive, or find a person or item being searched for. AI can do that.

But we can’t build intelligence into our storage if it is sat on a shelf on a tape. We need connected digital storage. We need the principles of DNA and intelligence.

The (Private/Hybrid/Public) Cloud in our Hands

After over 100 years losing, burning, corrupting media assets we are on the cusp of an media archiving revolution. Why? Not because of hardware technology changes per se but because of software and how we use hardware.

Critically, we now have the ability to allow media to migrate from hardware platform to hardware platform without the need for heavy manual maintenance (such as migrating LTOs). The same media management software can make critical decisions about how many copies of data to keep and where to keep them. This means that we can have media assets libraries that can truly grow and grow.

Secondly, for the first time ever we are beginning to see the ability of AI video analysis to extract information from movies, newsreel, sports coverage, etc. This is unlocking value in media and discovering new usages. Having a tape on a shelf is no longer an option.

And then let’s not forget the key stages that have allowed us to get here: data is now digital and assets can now be network connected and online.

Most modern solutions that are bringing these benefits are loosely called “cloud solutions”: be that public cloud like Amazon, or private (and hybrid) cloud like Object Matrix.

But we still have a long way to go for nirvana. Nirvana for me, means having 10’s of copies of media, storing all media in a connected manner and having all media in systems that self-evolve over time. Nirvana for me means having all media searchable, auto indexed, and auto managed so that media houses can focus on creating content and monetising assets rather than data managing them.

And, what we can all do today is to get our assets into systems that are at the start of that journey…

You can read Jonathan Morgans Article in TVB Europes September Issue 

About Object Matrix

Object Matrix is an award winning UK based software company that pioneered Digital Content Governance (DCG), object storage and the modernisation of digital video workflows. Our media focused private and hybrid cloud solutions are tightly integrated into file based and IP workflows and bring economic and operational benefit to all of our customers. Our flagship product, MatrixStore, is used by the world’s largest organisations that create and distribute video content, including NBC, TV Globo, MSG-N, the BBC & BT.