Results 1 to 5 of 5

Thread: do we need the speed?

  1. #1

    Default do we need the speed?

    I was about to buy a thunderbolt RAID as part of the storage for a zero budget film shoot. The shoot will be on a Black Magic Cinema Camera and when the SSD's in the camera fill the footage will be moved onto portable HHD. At the end of each day the portable HHD will be coped onto a RAID and then the HHD will be stored at another location as back up.
    They want to shoot in Raw Cinema DNG so it's about 500Gb for an hour at 15:1 that's about 12TB. From the RAID This then will be opened in DaVinci Resolve and a ProRes Proxy made. The proxy should be about 1TB for the lot and once the edit is done, XML out, conform in resolve and do the final grade.
    And the question is if I need the RAID to be thunderbolt, in fact do I need a RAID at all. The Proxy's will be ProRes LT and a USB3 or firewire drive can handle that I would of thought. I've never used Resolve so this is where it's all a grey area for me. If the files came from the portable HHD's to a USB3 RAID JBOD (it's all backed up at this point) were opened from here in Resolve, a LUT applied and then the proxy's exported to the same RAID, different disk. The edit done via USB3 and then XML out, conform in resolve and at this point it's small enough to move onto the local drive for the final grade and such.
    Is USB3 going to be enough to perform the steps here outlined? I hate to buy tech I don't really need. Or is this workflow just some kinda crazy talk?

  2. #2

    Default

    A lot of people get really hung up about raids and HDD speads etc. If you look at the bandwidth required to play ProRes 422 (LT) it's pretty small. In fact you 'could' edit Prores 422 (LT) of a USB2 drive if you really had to and weren't trying to get multiple streams running in parallel etc.

    USB3 speeds things up from USB2, but in the regular playback you'll see no real difference because neither one is being stretched by the requirements of ProRes 422 (LT). The main difference I see between the two technologies is the pause after you hit play before you actually start getting footage.

    To further put things in to perspective I have a LaCie 1TB drive that has both Thunderbolt and USB3. Sustained data from this drive runs at about 110MB/s over thunderbolt and around 98MB/s over USB3. The CPU overhead is also a little higher on USB3. ProRes 422 (LT) requires around is 36Mb at 1080/24p (I know you're shooting at 2.5K but bear with me). 36Mb is only 4.5MB/s and yet my one drive can sustain 98MB/s.

    Next, I have a USB3 dual drive dock that had 2 3TB drives plugged and configured as RAID0. Configured this way (as a single volume) the drives are sustaining read/write speeds in the range of 230MB/s. Cost? About $60 for the dock and then two bare internal HDDs.

    Now, compare that to my other RAIDs and it's slower, yes, but if you are editing footage that requires only 10.25MB/s and you have USB3 drives capable of delivering close to 10 times that….. well what do you think?

    Now, there ARE reasons to use RAIDs, including JBOD, but it really depends on what kind of production you are making and how many tracks/streams/cross fades etc you are planning on having. When we edit multi cam I need the raids for sure, unless I'm willing to go to PreoRes Proxy, which even then, allowing for some pauses as the play starts up, you could still edit using USB2.

    If you're on a budget, you probably don't need to buy that Thunderbolt RAID. If however you're looking for an excuse to buy it, go ahead, do it, you'll likely love it

    [Edit] Just to add one more thing, my Atmos Ninja 2 is recording ProRes 422 HQ to a 5400 rpm drive quite happily. Now think about that. You're running 422 (LT) not HQ.
    Last edited by David Partington; 11-01-2013 at 08:35 PM.

  3. #3

    Default

    Thanks for getting back I appreciate you taking the time. I'm a big fan of doing things on the bear minimum and I can edit an off line ProRes proxy of this film on my laptop for sure but the grey area for me is having this 12TB odd of DNG's that then needs to be read by Resolve for the grade. A single stream of DNGs is listed by the camera manufacture as needing 150MB/s read, and I guess it has to be a RAID5 for the redundancy. I'll need to open the DNGs coming in with Resolve to do a quick correction and then to make the ProRes proxy's and then at the end I'll need to conform the edit in the DNGs in Resolve and then do the final grade, I thought I'd need to be playing The DNG files at both points from the storage I use? So this storage has to be able to hold about 12TB, be able to have at least one drive fail without losing anything and to be able to play the cinema DNGs for color corection.
    I'm new, obviously, to this set up and workflow so I could be missing something really simple, maybe I don't understand the workflow properly, if I only had to store the DNGs and edit the proxy it'd be much easier but it's the grade that seems to be the bottle neck. But as I've said I've not used Resolve or DNGs before so it's all estimations and guess work from me here. On the subject of price and budgeting the Drobe 5D is listed at under $700 with 5 bays and configurable for RAID5 with duel disk redundancy. The diffence in price between this and a 5 bay USB3 RAID is not so much. Compairing pricing is difficult as I'm in the UK and the pricing of eletrical goods is different over here.

  4. #4

    Default

    Quote Originally Posted by yadnom View Post
    Thanks for getting back I appreciate you taking the time. I'm a big fan of doing things on the bear minimum and I can edit an off line ProRes proxy of this film on my laptop for sure but the grey area for me is having this 12TB odd of DNG's that then needs to be read by Resolve for the grade.
    The need for capacity and speed are sometimes two different things - so keep that in mind. While something may satisfy both requirements, it doesn't always have to.

    Quote Originally Posted by yadnom View Post
    A single stream of DNGs is listed by the camera manufacture as needing 150MB/s read
    This is for real time record / playback of the DNGs. Once you have them in another format (like ProRes) you won't need that speed, and if you're simply grading from the original files you also won't need that speed unless you 'need' real time (or better) grading. Most people don't, so what speed the DNGs run at when rendering the final film is simply a function of how fast your system is (including CPU / GPU), and if the final film takes twice as long to render as to play, does that really matter? It won't change the quality of the final film. If you're CPU/GPU can't keep up with Resolve for real time playback then all the HDD speed in the world won't help.

    Quote Originally Posted by yadnom View Post
    and I guess it has to be a RAID5 for the redundancy.
    While RAID5 is great, it's never a substitute for a real backup, so whether you have RAID5 online or not is totally up to you. Note that if a drive does fail, 12TB of data could easily take a couple of days, or longer, to rebuild and make safe once you replace the drive, so keep that in mind. You may be able to keep on editing, but the speed reduction could be substantial.

    Quote Originally Posted by yadnom View Post
    I'll need to open the DNGs coming in with Resolve to do a quick correction and then to make the ProRes proxy's and then at the end I'll need to conform the edit in the DNGs in Resolve and then do the final grade, I thought I'd need to be playing The DNG files at both points from the storage I use? So this storage has to be able to hold about 12TB, be able to have at least one drive fail without losing anything and to be able to play the cinema DNGs for color correction.
    I can't comment on how much footage you'll need online at any given time, since I'm not familiar with your project. But, take a step back and think outside the box. You're looking at as RAID that can hold 12TB, because that's what you think you'll need. What if it turned out to be 20TB or even 30TB? How would you cope / work around that? Could you possibly break the 12TB project up in the same way you'd break a 50TB project up? Maybe by chapter / scene etc? Do you really need all the DNGs online at the same time? If so, you may need to be looking at larger units, maybe the 8 drive RAID, because having 12TB of data, plus scratch files plus render files etc could mean you need a lot more than 12TB and even a 5 bay Drobo with 5x4TB drives in single drive protection isn't going to give you much more than that, and dual drive protection won't even come close.

    Quote Originally Posted by yadnom View Post
    I'm new, obviously, to this set up and workflow so I could be missing something really simple, maybe I don't understand the workflow properly, if I only had to store the DNGs and edit the proxy it'd be much easier but it's the grade that seems to be the bottle neck. But as I've said I've not used Resolve or DNGs before so it's all estimations and guess work from me here.
    My suggestions is that before you spend ANY money on RAID etc, get Resolve installed and learn it. Figure out your workflow BEFORE you invest any money because once you know what your workflow will be, your options and decisions should become much clearer.

    Quote Originally Posted by yadnom View Post
    On the subject of price and budgeting the Drobe 5D is listed at under $700 with 5 bays and configurable for RAID5 with duel disk redundancy.
    Drobo doesn't use regular RAID5, instead using a proprietary "Beyond RAID" algorithm. I've owned a Drobo 5 in the past and it did it's job well, although it was connected to my system using eSata and was relatively slow to use. Great as a backup tool but wasn't fast enough to edit multiple HD streams from without using Proxy. I do keep looking a the newer models (5n and 5S) but haven't spent the money yet.

    They do have a newer (thunderbolt) option that is allegedly much faster than the model I owned, check out Larry Jordan's review of it.

    Using dual redundancy is a great idea but you lose both capacity and speed.

    Quote Originally Posted by yadnom View Post
    The diffence in price between this and a 5 bay USB3 RAID is not so much. Compairing pricing is difficult as I'm in the UK and the pricing of eletrical goods is different over here.
    I'm in the UK also, so I know how much RAIDs etc cost around here. It can be painful My USB3 dual dock was actually £56 here. Add a couple of 4TB drives (from somewhere like Scan.co.uk) and you have an 8TB system, or 4TB mirrored for security. Again, that was the cheapest way I found of putting 8TB together. It also means it's easy to swap drives around when you need because it's a simple dock. Add two of these and you suddenly have 16TB online.

    Capacity vs Redundancy
    Note that a 5 drive Drobo will give you only 10.89TB in dual drive redundancy mode and 14.52TB in single redundancy mode, but it's still not quite as simple as that. Once Drobo (and frankly some other RAID devices) get to 90% full then there is a fairly massive write speed decrease. When I ran my Drobo with 5 (3TB) drives the speed reduction for writes once I hit 90% was incredible, to the point it was unusable (think < 5MB/s). With this in mind you should not rely on a Drobo giving you more than about 9TB (dual) / 13.9TB (single) of storage, and even then you're pushing it. Things 'will' slow down as it fills up.

    It you really need 12TB online as a raid then you're looking at buying 20TB or drives, plus another 20TB as offline backups (don't rely on a RAID or Drobo, they could die, maybe the PSU or the main controller takes all your data with them), so you still need a good backup strategy.

    Hopefully this has given you even more to think about

  5. #5

    Default

    This point you make about not needing to see the DNGs in real time is the one I think. I had my storage die and needed to replace it for work and so was looking at buying the storage solution for the film workflow before the comp, that's why the guess work. I have now solved that problem with the help of a friend and will be able to do as you suggest and first buy the machine to do the edit on, install Resolve as you say and test it all out.
    At 15:1 the Drobo 5D could do it, 5x4TB gives 15TB on there calculator and I figured under 12TB would be safe but I may be wrong. If they shoot more that 15:1 the solution is to throw the HDD back at them from the window. Good to here that your's treated you well though there's some horror storys on line but I think thats the way with any product if you look hard enough. Thanks again for taking the time I think your right and I should take a good look at it and see if it's needed, it's so easy to get swept away in the whole newer and faster gig but if your off line in a proxy there's no need and there's other things like a grading monitor which are hard to get around and need the money instead.

Similar Threads

  1. Need for Speed gone BAD!
    By gokhan in forum User Videos
    Replies: 0
    Last Post: 04-08-2011, 02:44 PM
  2. How to increase video speed but not music speed?
    By jippyflip in forum Sony Vegas video editing apps
    Replies: 7
    Last Post: 10-19-2009, 04:26 PM
  3. Replies: 8
    Last Post: 04-30-2007, 05:42 PM
  4. dvd speed is off
    By zeroxsk8 in forum Blu Ray, DVD, SVCD and VCD authoring
    Replies: 0
    Last Post: 01-31-2006, 08:21 AM
  5. Speed
    By ianusher in forum User Videos
    Replies: 5
    Last Post: 06-09-2005, 12:03 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •