As files get bigger and workflows complex, companies are mixing on-premises storage with cloud for have better elasticity and capacity
By Vinita Bhatia
Not many people know that James Cameron’s award-winning film, Avatar, required over 1 Petabyte (PB) of data (that is roughly estimated to be equivalent to 13.3 years of HD TV video). Of course, the nature of content, including the VFX, called for that that kind of storage. However, over the years, content creators in the media industry are increasingly struggling with storage woes. Even broadcasters are realising the difficulties of storing data for the content created for their platforms or procured from various sources, not just for contemporary period, but also for posterity when they will need to repurpose and reuse it.
This is one of the reasons why transition to software-defined storage technologies is gaining lot of attention in the the broadcast world, given the exponential increase in digital content creation. It helps them better manage their data across various segments as it encompasses traditional and newer types of workloads, and is optimised for interoperability across hardware and software solutions.
Alan Cheng, director, APac, channel sales (exclude China, Japan & FAE), Promise Technology explained, “Legacy storage infrastructures not only fail to meet business needs, but also unnecessarily increase IT costs. The broadcasting industry can benefit from a software-defined storage architecture, that can accommodate traditional IT infrastructure, while meeting the needs for media content growth, new applications and user requirements. In short, it provides benefits like smarter work method, enhanced storage consumption, agility with efficiency as well as provision storage on-demand.”
The Rise and Rise of Storage India’s media and entertainment (M&E) industry has been growing exponentially in the last few years. According to a FICCI-EY report, the country’s industry revenue grew by almost 13% in 2017 as compared to the previous year. A young and burgeoning middle class, increasing consumption of data on the go and decreasing data costs, drove this growth.
According to Khalid Wani, director, channel sales India, Western Digital, the M&E sector by nature of workflow can benefit from storage solutions at every stage of content development. “From capturing of content to rendering and creation to post production activities to delivery of the final content across various channels to finally archiving – every step can benefit from robust and high capacity storage solutions. Apart from this the industry is undergoing a shift to the digital means of movie making. With HD cameras and digital post-production technologies becoming a norm, we are now entering the era of an all-digital workflow,” he said.
As more filmmakers opt for HD or Ultra HD videos on digital cameras with higher resolutions, this results in the generation of increasing volume of digital content. Wani added, “Accessibility of data is another challenge; it is important to not just store this digital content but also to ensure that it is available at all points. All the endpoints need to be up and running, and even a single point of failure might prove it to be fatal in the M&E industry.”
Elaborating on this, Robin Melhuish, director, customer solutions, Wazee Digital said that what is unique to the M&E industry is the size of professional video files, which are typically several gigabytes per-minute. “This means that large video catalogs or even the raw files for a series are often hundreds of terabytes. Larger media companies will regularly have petabytes of finished programs. By comparison, in the IT world, gigabytes are still considered large files. This means that storage, networking, egress, and compute are often expensive constraints for M&E companies,” he noted.
UNCOMPLICATING THE COMPLEX
In today’s content-hungry world, as files are getting bigger, workflows too are becoming more complex with companies managing data centers around the globe.
They often deal with many vendors and now mix in cloud storage for various workflows. According to Jon Finegold, chief marketing officer of Signiant it is critical to have enough storage available without bearing the cost of too much over capacity, which is why many companies are mixing on-premises storage with cloud storage for elasticity during peaks.
The other key storage challenges that most content producers and providers face include security, accessibility and distribution as well as ensuring its management and operational efficiencies. The exponential demand for new and archive digital content means that organisations are struggling to meet these challenges whilst using incumbent legacy storage platforms. In many cases, content creators have been adding to existing offline technology or making short-term tactical investments, that offer no strategic value, just to cope.
Nick Pearce-Tomenius, sales and marketing director, Object Matrix has often seen content creators and providers who have bought multiple storage silos, which may well sit in the same facility but have absolutely no integration possibilities with each other. When editors, producers or archivists looking for a specific piece of content, or type of content, it is a really challenging and long-winded process trying to hunt down the correct item across multiple storage silos. This approach also means staff are wasting time managing storage silos or manually locating tapes for the library when they could be adding value to the organization, he pointed out.
Creating and delivering video content is more challenging than ever across every type of media operation. Jim Simon, senior director, marketing, APJ at Quantum felt that shooting ratios are way up, resolutions are higher, and the proliferation of production and delivery formats all combine to push the boundaries of workflow efficiency. “Collaboration between team members—no matter where they are located—is also essential. Post-production professionals require performance and scalability to power the modern media workflow and extract maximum value from their content,” he noted.
As consumers come to expect more choice, storage strategies, too, need to reflect this by offering ease of expansion and flexibility in storage-tiering. Andrew Ward, business development manager, Cinegy GmbH explained that more content can also mean more network bandwidth, so the selected storage configuration also needs to support technologies like LACP to ensure that the system can grow with demand. “Reliability is crucial; configuration of backup strategies, mirroring, failover and the like need to be undertaken with the goal of guaranteeing continuity of service under all possible failure scenarios. Finally, ensuring that the correct technology is implemented ensures that budgets can be correctly allocated. If LTO tape is adequate storage, then there is no sense in paying for SSDs,” he clarified.
In his opinion, the M&E industry can benefit from the movement away from tape. Since video and audio are just data, the industry has access to tools and techniques for storing, safeguarding and delivering data that has been in use for decades. Furthermore, enterprise storage systems from all major vendors offer everything necessary to meet modern media industry requirements. “As more options become mainstream (cloud storage, object storage) the increased number of choices available to the media industry will allow the creation of hybrid storage infrastructures that balance speed, volume and availability to suit all kinds of media,” Ward emphasised.
SAFE AND EASY
Given the amount of content being produced, the one thing that is on the top of any content creator and broadcaster’s mind is, what are the best storage options that are safe and easy, and will not force them to break the bank? Ideally, they should invest in a storage platform for their media workflows that takes into account the total economic benefit, rather than opting for inexpensive solutions that are unlikely to offer any long-term operational inefficiency.
Also the storage requirements of each domain within the industry vary. Wani explained how the M&E industry could be segmented into film entertainment, digital media, animation and VFX, online gaming, broadcasting, etc. “Each of these segments has storage requirements that will vary in terms of capacity as well as performance. Take, for instance, movies with animation and CGI-heavy content, which that can create several TBs of data per day. Here, the requirement is for fast, robust and reliable, storage infrastructure, primarily to enhance workflow and support editing large volumes of digital footage,” he surmised.
According to Pearce-Tomenius, object storage and digital content governance is by far the most suited to the modern media landscape. It offers the ability to impose stringent access controls and can be very tightly secured to ensure content remains safe. At the same time, content is easily discoverable for authorised users. Because it is scalable, users can buy the amount of storage they need and expand it at any time. In terms of financial efficiency, this of course makes much more sense as you only pay for the storage you need.
“The other option emerging is that of a hybrid cloud workflow, where some content is stored in on-premise storage, such as object storage, and other content is stored in the cloud. This type of setup enables easy sharing of assets across multiple geographic regions, whilst avoiding unnecessary egress fees by keeping some content on-premise,” he added.
Finegold claimed that many companies are using a tiered approach to storage, mixing a variety of on-premises storage with cloud object storage using different classes of storage for different use cases. “During a project, speed is often of the essence, therefore, more expensive, high-speed storage is used. But for archiving or disaster recovery, lower cost storage may be all that’s required,” he said.
LOOKING TO THE FUTURE
A law exists in M&E that irrespective how much storage one provisioned, it will immediately be filled as it is never enough. Melhuish commented that creatives would always want to keep shooting and keep everything. “Traditionally, this storage has been on premises: individual hard drives, NAS, and SAN sometimes off-lined to LTO tape for archive. This is expensive, requires a lot of capex and a frightening amount of valuable video is often left on prosumer hard drives, which are a terrible long-term storage solutions especially when they are just left on the shelf,” he said.
That is exciting is that these ‘on-premises solutions are increasingly being replaced by on-cloud solutions where the video is uploaded to data centers provided by Amazon, Google, IBM, etc. This helps the entire media supply chain to access it wherever they are making the process very lean and improving efficiency from upload to delivery.
Many content creators and broadcasters often seek their storage platforms to be future-proof. Ward believed that once commissioned, any storage system in use is a snapshot of the technology available at the time and will be technically obsolete within five years. That is the hard fact that drives growth in the storage industry.
“Major storage vendors make the majority of their revenue from existing customers expanding and upgrading. There is no such thing as future-proof storage – only a future-proof storage strategy. New technologies are continuously being introduced but usually carry a high initial premium, SSD’s being a good example. What is important here, is to continue using standard IT technology, because that will allow easy migration to newer platforms as they become necessary or desirable and continuously evaluate new storage technologies as they become available,” he advised.
Since data transfer speeds also matter, whether on cloud or on premise, Elvin Jasarevic, MD, APAC, DDP says that the best option for that is SSD caching, which offers the fastest speed which is most important for any storage system, without burning a hole in the pocket. “Of course, many would have petabytes of SSDs, but how many companies have the money to invest in it? Hence, Dynamic Drive Pool (DDP) offers a advanced method of SSD caching,” he stated. “While some storage solutions only offer one method of caching, based on the block level, DDP has developed a dynamic file based caching. With on-demand, pinned, and locked caching algorithms, DDP administrators can dynamically choose any of the caching methods to be assigned to any folder volume.”
He further explained that these features offer many performance options for live ingest and playout, transcoding and editing, and various grading and finishing workflows. “Where other solutions ‘move’ files in and out of cache, DDP’s caching is based on our new Dual Path Technology (DPT). It ‘copies’ data in and out of cache, so that when one is done using cache the files are deleted instead of having to wait for another data move. Data is stored online, giving users continuous access and after use, it is moved to Nearline Storage if files are required in near future. However, if unrequired for almost immediate work, it can be copied to Tape libraries, or pushed to cloud,” Jasarevic described.
There are quite a few options when it comes to storage technology for the M&E industry. Ultimately, companies have to decide for themselves which ones are cost-effective, reliable, easily operable and equally important, scalable in terms of capacity and performance. The key is to future-proof their investment in storage for years to come, so that they can focus on what they do best.