Adding to our proven Apple Final Cut Server and CatDV workflows we’ve made some recent improvements to using MatrixStore as both a Pre-Production Archive (ingest location) and a Nearline Archive in Avid workflows.
The workflow shown in this blog post uses Avid AMA and Avid Unity together with MatrixStore at a post-production house here in the UK.
The Phase 1 setup below was fully tested during whilst the items in phase 2 were left to be tested at a later stage (although they are assumed to be working and will be tested as required).
Phase 1 Usage
- Footage and metadata were ingested from P2 cards and local disks
- Windows clients (Vista 64 bit, XP 32-bit)
- 3-node MatrixStore Nearline cluster accessed from clients on a Gigabit network
- Avid AMA clients on Windows Vista 64-bit machines
- DropSpot (MatrixStore Client data browser application on Windows and Mac)
- Avid Unity on an FC network
Phase 2 (not tested but to be tested when required)
- Sony SxS cards
- Avid AMA Mac clients
- Panasonic P2 Viewer updating to MatrixStore
- Avid AMA building sequences to MatrixStore
- LTO tape writers running off MatrixStore via MXFS
Commentary and Results
The MatrixStore cluster and clients were installed, physically connected to the network and were usable within 3 hours of arriving onsite, with the main delay being working out which clients to use for which stages of the workflow.
Ingest media from P2 or disk was imported via the DropSpot application using the following steps:
- Data folders to move the footage to were created in DropSpot.
- Data was imported via DropSpot to the MatrixStore.
- Multiple clients could Ingest in parallel.
During the Ingest of data, the metadata from the P2 cards was examined and added to the footage itself. This allows the footage to be searched for and also means that the metadata will be kept securely with the footage for as long the footage is kept in the MatrixStore Pre-Production/Nearline archive.
DropSpot runs on Mac, Windows, and Linux, and MatrixStore was also found to be useful as a shared storage location between clients.
The resulting data movement occurred as fast as the data could be read. In the case where P2 cards were used this was approximately 50MB/s to 60MB/s, and in a test where data was read from a fast direct attached storage device, data moved at the full network card interface speed available. Sony SxS cards are expected to ingest at a slower rate where they are using USB technology.
For Edition, Avid AMA clients were set-up on Windows Vista 64bit and XP 32 bit machines and used the MatrixStore “MXFS” virtual filesystem to view and update ingested data as if it were on an attached SAN device. This was achieved in two stages:
- The folders of Ingested data were scanned by selecting the Avid AMA import volume option. Scanning a folder takes around 20-30 seconds per P2 card whilst Avid software browsed and created stubs.
- Avid AMA was then to consolidate the data onto the Unity platform. Where consolidating didn’t transcode the data moved from the MatrixStore to Unity at full speed with the client CPU utilized seen to be approximately 10%. Where consolidation was involved transcoding the data moved at 25-30MB/s and the 8 cores of CPU were at 90%+ (ie the bandwidth was limited by the transcoding CPU speed).
Multiple clients can consolidate/send data to Unity in parallel.
One typical usage scenario of the MatrixStore Nearline used was that prior to ingesting of data, the user would open up one to three data streams in an Avid AMA client to view the content. So long as the client could cope with the streams (e.g., the graphics card could cope) this worked very well.
- The set-up of the Nearline archive and clients was straightforward and resulted in the Nearline archive being ready to be used on the day that it arrived, validating the Object Matrix “plug and play” analogy.
- The main bottle-necks in the workflow were the speed of the cards being read from and the time was taken to transcode data (the users were very happy to see their dual quad core clients being used to their capacity!).
- The new MatrixStore MXFS connection from Windows and Mac was straightforward to set-up and provided up to full bandwidth across the network.
- MatrixStore acted as a first class Nearline archive keeping data safe, secure and available throughout.
Future plans for Object Matrix include forthcoming direct support for Avid Interplay and further support for metadata extraction and handling, but if you’d like to know more about our reference sites and plans for future workflow support let us know!