AVR-Chat

On Thursday 05 August 2004 08:14, Kathy Quinlan wrote:
> Graham Davies wrote:
> > --- In AVR-Chat@yahoogroups.com, Kathy Quinlan <kat-yahoo@k...> wrote:
> >>I need a contact for a semi manufacturer
> >>who can manufacter 2 devices for me:
> >>#1 an 8 bit cpu that is FAST (think Gflops) (able to address 0.1TB)
> >>#2 8 bit DDR (I need 10TB for the above cpu's)

> > Kat, I would recommend TSMC for the CPU.  You will need their 0.13
> > micron process with copper interconnect. Even then, you'll need a
> > highly parallel design. Once you have the design complete, make sure
> > you have a million dollars left for masks and the first fab run.

[snip]

> I have to process a text file that can be up to 10Tb in size, the

File? Does that mean that it's stored somewhere?

How long is the string to which you want to match?

> data is only ascii, so I can not see the point for a larger sized
> CPU. Once a string is found, all it needs to do is hand the string
> up to the next CPU for serving.

Matching even a non-trivial string in that volume of "random" data
is always probable.

> This is not a joke Guys, I have someone who is paying me to design
> this system, and speed is essential, so it has to be held in Ram.

How does it get into RAM? A byte at a time? Is the string byte-sized
or are the characters larger? I think you'll find that for any
reasonable volume of data, that you'll be I/O-bound.

Matching a string in un-indexed data is a matter of comparing each
character starting from the first and repeating that until a match
is found, checking the next character for match, etc...

It doesn't appear to require much RAM at all; not unless you're
matching multiple strings in the same data. If multiple is many,
you should investigate indexes; hashing for example.

> I do not plan to have one CPU to search the whole 10Tb I was
> planing on using 100 parallel CPU's each looking after 100Gb of
> ram.

Divide and conquer is a good approach. 
Perhaps you're byte-ing off more than you can chew.

Using 1000 Opterons with 10GB each will cost significantly less...
you'll get a good volume discount from AMD as well. If you use the
quad-processor version, then you can reduce the number of
"computers" to 250; with 4 CPU and 64GB RAM each; they can talk to
each other at very high speed.

Comms between computers at 10Gbit at least, using off-the-shelf
hardware. expect a total power draw of about 100kW. At that comms
rate, you'll be able to process the 10Tbytes in under an hour with
250 parallel links of that speed ... but if you're peeling the data
off high-speed hard drives at 100Mbytes/second, it's 10 seconds for
a gigabyte or 10,000 seconds for a terrabyte. 

Software gets more complex as you go multi-processor. Dividing the
data set is not enough; you have to have overlapping data sets;
overlapping by at least the size of the matching string to ensure
that you find all possible matches...

Final words; talk to Cray or Fujitsu.

-- 
/"\ Bernd Felsche - Innovative Reckoning, Perth, Western Australia
\ /  ASCII ribbon campaign | I'm a .signature virus!
 X   against HTML mail     | Copy me into your ~/.signature
/ \  and postings          | to help me spread!