LTO Based Solutions Are NOT the Answer
So instead the answer became “let’s stick it on tape”, which became, “let’s stick it into LTO”. After all LTO has a longer lifespan… right? Well of course, as we all know, LTO has gone through 8 different formats with a change in format every 2 years or so. And with LTO 8 machines unable to read even LTO 6, this has resulted in the need to migrate archives or see them go to dust once again. In over 100 years of media storage – it seems we have gone nowhere.
Disk Based Solutions Are NOT the Answer
But hang on… we have disk based RAID systems, right? This is very true. Disk based scalable storage is very good indeed at storing archives of storage, storage that is connected to the tools that can search and use that archive. But is it really the answer?
First off, companies have a big problem. In the past they bought a disk based raid system for storage, but it quickly became undersized and later became out of date. So the archive on the storage needed to be migrated, again. Secondly, as that company bought the next disk based solution, probably from another manufacturer, maintenance became a mare, and that shiney new (and expensive) platform from a few years ago became valueless over a few years.
Nature has the Answer (1) DNA
So, as the CEO of a storage company looking to these issues we looked to nature. DNA to be precise. Just look at how amazing it is:
- 1.2g of DNA can store a Petabyte of data
- It survives corruption
- It survives war, fire, 1000’s of years
- 99.9% of our DNA is the same, the other 0.1% is what differentiates us
OK – even though some people are literally looking at DNA for storage, I don’t want to stretch the analogy too far but the point here is that DNA: (1) has the principal of multiple copies (2) can migrate from one carrier to the next generation.
I fundamentally believe that over time the cost of hardware for storage will become insignificant, but it will be the management of that storage, and in particular in this context, the ability to change out the hardware with automatic management less migration of the data to the new hardware.
Nature has the Answer (2) Intelligence
Artificial intelligence is the most cliched term of 2018, so let me be more precise. If we had intelligence and time, we’d know that this piece of film is needed in the UK and that production needs to be worked on in a post-production house in the USA. We’d realise that having data in just one location puts it at great risk and that that film should never be changed from its original format. AI should be making those decisions for us.
And, if we had 1000 eyes and a million hours we’d tag and analyse all the video in our archive, or find a person or item being searched for. AI can do that.
But we can’t build intelligence into our storage if it is sat on a shelf on a tape. We need connected digital storage. We need the principles of DNA and intelligence.
The (Private/Hybrid/Public) Cloud in our Hands
After over 100 years losing, burning, corrupting media assets we are on the cusp of an media archiving revolution. Why? Not because of hardware technology changes per se but because of software and how we use hardware.
Critically, we now have the ability to allow media to migrate from hardware platform to hardware platform without the need for heavy manual maintenance (such as migrating LTOs). The same media management software can make critical decisions about how many copies of data to keep and where to keep them. This means that we can have media assets libraries that can truly grow and grow.
Secondly, for the first time ever we are beginning to see the ability of AI video analysis to extract information from movies, newsreel, sports coverage, etc. This is unlocking value in media and discovering new usages. Having a tape on a shelf is no longer an option.
And then let’s not forget the key stages that have allowed us to get here: data is now digital and assets can now be network connected and online.
Most modern solutions that are bringing these benefits are loosely called “cloud solutions”: be that public cloud like Amazon, or private (and hybrid) cloud like Object Matrix.
But we still have a long way to go for nirvana. Nirvana for me, means having 10’s of copies of media, storing all media in a connected manner and having all media in systems that self-evolve over time. Nirvana for me means having all media searchable, auto indexed, and auto managed so that media houses can focus on creating content and monetising assets rather than data managing them.
And, what we can all do today is to get our assets into systems that are at the start of that journey…