dvd-hq.info Forum Index dvd-hq.info
DVD & video forums
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

SATA RAID for HD
Goto page Previous  1, 2, 3
 
Post new topic   Reply to topic    dvd-hq.info Forum Index -> Hardware
View previous topic :: View next topic  
Author Message
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Fri 4 Jun 2004, 17:18    Post subject: Reply with quote

Yes, I think that's probably best (they might even tweak the hardware, until then). But maybe you can ask them what sort of performance you'll get in PIO mode? On x86 hardware, I know it's ridiculously slow, but maybe it's not quite so bad on PPC.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Sat 5 Jun 2004, 16:02    Post subject: Reply with quote

Maybe we should be getting ready for this. Mr. Green

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
deproduction



Joined: 27 May 2004
Posts: 8
Location: Denver

PostPosted: Sat 5 Jun 2004, 22:35    Post subject: NY Times Salesman Reply with quote

RMN, your link only works if you're a registered member of the NYTimes. Cut and paste for us.
My card is just a couple miles away, but I can't get it until monday (damn FedEx). The dream-team we had assembled for Sunday's test will have to wait a week or more, and my rag-tag team in the office will have to hope the card works right off the bat.
We'll test it on monday night, then bring the EEs and programmers in to troubleshoot next weekend. Good news though, HighPoint told me that DMA is NOT disabled in the current driver, just that they had tested it in PIO only to keep things simple.
I didn't get any info from them on the speeds possible in PIO, but I've read it limits out at between 15 and 20 MB/s per drive... so a max transfer rate of 160MB/s for an 8-drive array (barely faster than a single SATA drive in DMA).
Needless to say, we won't even fuck with PIO.
Do any of you know how/where we can make that determination? is it written into the driver?
Tony
Back to top
View user's profile Send private message Visit poster's website
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Sun 6 Jun 2004, 5:10    Post subject: Re: NY Times Salesman Reply with quote

deproduction wrote:
RMN, your link only works if you're a registered member of the NYTimes. Cut and paste for us.


Just register with a dummy account (ex., create a Yahoo or Hotmail account and use it to register at all these sites). After that, you'll just log in automagically (as long as you're cookied), no need to remember usernames / passwords.

deproduction wrote:
I didn't get any info from them on the speeds possible in PIO, but I've read it limits out at between 15 and 20 MB/s per drive... so a max transfer rate of 160MB/s for an 8-drive array (barely faster than a single SATA drive in DMA).


Well, I don't know any SATA drive that can actually sustain more than 50 MB/s, so 160 is a bit more than that. Smile But I doubt you'll actually get 160 MB/s out of a PIO-mode array. In PIO every operation has to be done by the CPU, and it will only send a new command when the previous one has finished. In other words, the speed will not scale linearly with the number of drives.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
mdc1138



Joined: 26 May 2004
Posts: 17
Location: Austin, TX

PostPosted: Tue 8 Jun 2004, 22:39    Post subject: SATA RAID article, faster than 50 MB/sec SATA drives Reply with quote

Hey all -

I just posted up a huge long honkin' article on most everything I know about SATA RAIDs at

http://www.hdforindies.com/archivedarticles/2004_06_01_archived_article#108672778918379763

Also, somebody on here mentioned that they didn't know of any SATA drives capable of over 50 MB/sec. Two I know of:

IBM/Hitachi 7K250 drives - 60 MB/sec at head, 30 at tail

Western Digital Raptor 10K drives - only 74 GB, but about 72 MB/sec at the head, around 52 at the tail. Small, but FAST.
_________________
Mike Curtis
HD For Indies
http://www.hdforindies.com
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Wed 9 Jun 2004, 0:13    Post subject: Reply with quote

You can find data for a lot of drives at http://www.storagereview.com/.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
deproduction



Joined: 27 May 2004
Posts: 8
Location: Denver

PostPosted: Wed 9 Jun 2004, 1:40    Post subject: SATA Tests Reply with quote

Our 8-port HighPoint test on the G5 was a total failure. THere's buzz about an external 4-port card that would work just as well. You'd have to use two, but teh HighPoint card requires two slots too, since you have to run teh cables out somewhere.

Rob seems to be able to count on about 50MB/s per drive sustained from the Maxtors and the Hitachis. RMN seems to feel you need 12 or more drives for full resolution HD. I'm sure once the Highpoint starts working, or once I get my hands on two 4-port externals, 9 drives will do the trick, as long as we keep the Array from being over-stuffed. (8 external, 1 internal)

As soon as I have either solution working, the challenge turns to making RAID 5 a possibility on the G5. It may slow things down a little, but by Rob's calculations, we'll have a little headroom with 9 drives.

I found one product, the FWB RAID Toolkit, which enables RAID 5 on OSX, but it doesn't support SATA RAID (yet). I wrote in and spoke to them and Mark from FWB wrote back: "I will talk to Firmtek to see if there is a way we can work with them."

HighPoint also agreed to talk with FirmTek about getting their drivers working. We might be helping this little company quite a bit.

Tony


Last edited by deproduction on Wed 9 Jun 2004, 17:44; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website
mdc1138



Joined: 26 May 2004
Posts: 17
Location: Austin, TX

PostPosted: Wed 9 Jun 2004, 3:04    Post subject: RAID issues Reply with quote

Bummer about Highpoint, but the future may bring new things.

If the 4 port Firmtek can do 50 MB/sec per drive, that and the two internals should get us where we want to go. 6X50 is 300 MB/sec, 6x30=180 MB/sec.

12 or more drives for full res HD? How are you defining that? 4:4:4? Even that should only be 180 MB/sec at 1080i29.97...doable with 6 drives, even if not for ALL of the array...but easily 80+% of the array.

RAID 5 on G5 via software - hmm...FCP HD seems to be awfully busy during playback, there may not be a lot of headroom to spare for drive interaction.

FWB was on death's bed last I heard. If they can get software out great, but I'd be concerned about getting orphaned, such as when 10.4 comes out.

and Yo! Rob-ART! Share the luv on the Firmtek card info...

-mike
_________________
Mike Curtis
HD For Indies
http://www.hdforindies.com
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Wed 9 Jun 2004, 17:48    Post subject: Reply with quote

I seriously doubt the drives can sustain 50 MB/s in the inner sectors.

Dual-stream HD 4:4:4, 10 bits, 30 frames or 60 fields per second, requires about 230 MB/s (see maths in one of my messages above). Add a couple of sound channels, and you really need closer to 270 MB/s. I'm aiming for 300 MB/s to be on the safe side. If you add an alpha channel, you need a little bit over 330 MB/s.

For 4:4:4, 10 drives should do the trick (assuming a worst case scenario of 30 MB/s sustained - most modern SATAs can sustain 35 MB/s in the inner sectors). For 4:4:4:4, you should go for 12 drioves at least. This is with 7200 RPM SATA drives. With Raptors or SCSIs you can get away with about 25% less drives, but they're more expensive and smaller.

Regarding playback, unless you're doing any real-time effects, the CPUs should be virtually idle (assuming you have DMA bus mastering drivers for both the drive controllers and the video card). Since most people will want real-time effects (colour correction, etc.), though, it's not a very good idea to put the RAID-5 load on the CPUs.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
mdc1138



Joined: 26 May 2004
Posts: 17
Location: Austin, TX

PostPosted: Wed 9 Jun 2004, 17:56    Post subject: Regarding throughputs, etc. Reply with quote

I picked up my throughput numbers from

http://www.blackmagic-design.com/site/25support.htm

NOT including audio. They differ from yours, by more than just audio.

As for 50 MB/sec inner sectors, SCSI and Raptors CAN, most SATA CAN'T.

Yeah, definitely don't want software RAID-5 on CPU.
_________________
Mike Curtis
HD For Indies
http://www.hdforindies.com
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
mdc1138



Joined: 26 May 2004
Posts: 17
Location: Austin, TX

PostPosted: Wed 9 Jun 2004, 18:11    Post subject: one more tidbit on drive throughput Reply with quote

...while it would be great to have the desired 300 MB/sec across the entire capacity of the array, it isn't necessarily required, I'd argue.

The rule of thumb, so far based on my own tests (see my huge honkin' blog entry at

http://www.hdforindies.com/archivedarticles/2004_06_01_archived_article#108672778918379763

and scroll down to "HEY BOY! HOW FAST DO SHE RUN!" for the test numbers.

Woops, I digress - from my testing, it seems that drives hold a large proportion of their speed for a decent chunk of the drive, and then drop dramatically towards the end. It is NOT a linear drop in speed.

For that 4xBarracuda 160 array I set up, for example, it held 160 MB/sec for about 3/4 of the capacity of the array.

That would be good enough, using the numbers from the BlackMagic codec link, to do 1080i29.97 10 bit 4:2:2 with audio on this inexpensive setup.

Partition it at the point where you don't trust it beyond that, and use that space for DVCPRO HD or other offline, non-throughput critical storage.

4:4:4:4 would be great, awesome, fantastic...but for my purposes, Final Cut Pro HD only does 4:2:2 at the moment as I understand it,there is no codec and hardware at this moment to do better.

In the future, I will want to do 12-bit, 4:4:4, 2048x1080 for DI work....but Final Cut Pro can't handle that...yet.

RMN, does all these seem accurate and/or viable? If not, please tell me why. Not trying to pick a nerd fight, just figure it all out.

: )

-mike
_________________
Mike Curtis
HD For Indies
http://www.hdforindies.com
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Wed 9 Jun 2004, 21:59    Post subject: Re: Regarding throughputs, etc. Reply with quote

mdc1138 wrote:
I picked up my throughput numbers from http://www.blackmagic-design.com/site/25support.htm NOT including audio. They differ from yours, by more than just audio.


Those numbers are for 4:2:2, and very (very) optimistic.

I've talked with some people from Backmagic and they say that if you want to edit in 4:4:4 with no risk of dropped frames, your array needs to support 274 MB/s at least. I think that's a bit optimistic too, but their quoted value of 136 MB/s for 4:2:2 isn't just optimistic, it's unreal. The video alone is more than that (it's almost 150 MB/s).

In fact, the various numbers on that page don't add up. If 1080 / 29.97 / 8 bit needs 124 MB/s, there's no way you can switch to 10 bits and only use 136 MB/s. 124 / 8 x 10 = 155.

I've already posted the maths above (in the first page of the topic), it's pretty straightforward.

Quote:

As for 50 MB/sec inner sectors, SCSI and Raptors CAN, most SATA CAN'T.


The Raptors are 10K RPM drives. You said "50MB/s per drive sustained from the Maxtors and the Hitachis", which, as far as I know, are all 7200 RPM. Check Storage Review (link above) for their speed across the various sectors. I expect they'll be similar to the WDs and Seagates: about 58 MB/s on the outer sectors, and about 36 MB/s in the inner sectors.

Quote:

Final Cut Pro HD only does 4:2:2 at the moment as I understand it,there is no codec and hardware at this moment to do better. [...] In the future, I will want to do 12-bit, 4:4:4, 2048x1080 for DI work....but Final Cut Pro can't handle that...yet.


I expect FCP will handle whatever you throw at it, as long as you have the appropriate card and codec. Don't the new AJA cards support 12 bpc? I know Premiere doesn't have a clue what format it's using, it just handles the "abstract" editing. Everything related to the actual video is handled by the codec and plug-ins. I'd be surprised if FCP was any different.

Quote:

does all these seem accurate and/or viable?


Personally, I wouldn't trust a 4-drive array to be able to keep always above the necessary data rate (even for 4:2:2). But there's no harm in starting with 4 drives and then adding more if / when you need them.

RMN
~~~


Last edited by RMN on Thu 10 Jun 2004, 0:10; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail Visit poster's website
mdc1138



Joined: 26 May 2004
Posts: 17
Location: Austin, TX

PostPosted: Wed 9 Jun 2004, 23:19    Post subject: FCP 4:4:4 and stuff Reply with quote

I was just reading somewhere the other day that FCP only processes in 8 bits per channel, 10 bits if you're careful about what effects you're using. So it may be possible to edit in 12bit, but you wouldn't be able to DO anything other than straight cuts without codec support...which doesn't exist in a usable, real-time format yet. Microcosm is great, but only for offline.

Ugh. If you're right and I'm wrong about throughput, that throws me all off. I gotta generate some test files and see what I get. Drat.

New AJA and DeckLink cards support 12 bit on the hardware side, when I talked to Grant Petty (The Dude at Blackmagic) at NAB, he said they were working on but didn't have a 12 bit codec yet.

So step 1 will be hardware support (happening in next month), step 2 is a codec for same (soon but not yet), and step 3 will be native FCP HD support for processing beyond cuts (such as transitions, color corrections, blurs, etc.) which will be who knows when. Anytime from now to next NAB I'd presume. Probably in the fall if I had to guess.

Thanks for the info, I'm off to test & mess with it.

I'll have yet another huge SATA RAID thing to post on my blog in the next few days.

-mike
_________________
Mike Curtis
HD For Indies
http://www.hdforindies.com
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
RMN
Site Admin


Joined: 04 Feb 2003
Posts: 587
Location: Lisboa, Portugal

PostPosted: Thu 10 Jun 2004, 0:16    Post subject: Reply with quote

Personally, I don't see much point in 12-bpc for storage / output. For processing, I expect (well, hope, at least) we'll all be using floating-point HDR in a couple of years.

DirectX 9 requires that graphics cards have a floating-point programmable pipeline, and hopefully that will drive software's evolution towards HDR. Some things that are very hard for a CPU to do (even with SSE / Altivec) are a piece of cake for a GPU.

RMN
~~~
Back to top
View user's profile Send private message Send e-mail Visit poster's website
mdc1138



Joined: 26 May 2004
Posts: 17
Location: Austin, TX

PostPosted: Sun 11 Jul 2004, 20:00    Post subject: Hullo, where is everybody? Reply with quote

Anything new happen with the Highpoint card?

Anybody got a working 6+ drive RAID yet?

Read that Seriteks in slots 2 & 3 takes a performance hit - how big?

Gotta have the capture card in 4 if a AJA or BlackMagic card, otherwise it drops down to PCI speed if it is sharing the 2&3 bank with a Seritek (PCI card, not PCI-X).

So what's the latest from everyone? I've been posting thoughts on my blog.
_________________
Mike Curtis
HD For Indies
http://www.hdforindies.com
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic    dvd-hq.info Forum Index -> Hardware All times are GMT
Goto page Previous  1, 2, 3
Page 3 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum



Powered by phpBB © 2001, 2002 phpBB Group