[svlug] Linux + MySQL w/ very large data sets

[Jeremy Zawodny]

On Wed, Jun 12, 2002 at 03:13:41PM -0700, Aaron T Porter wrote:
> On Wed, Jun 12, 2002 at 03:00:22PM -0700, Drew Bertola wrote:
> > Can anyone tell me what the best architecture / OS support is
> > suggested for handling data with possible table sizes in the 100s of
> > GB range?  This would be under MySQL (though I'd consider other DBs)
> > and, other than monthly updates, only SELECT transactions need be
> > considered.
> 
> 	You'll probably be really stressing mysql at that point. And
> even monthly updates sound a bit scary at that size. My largest
> mysql tables currently are ~35m rows/2gb on disk (data file, not
> counting indexes).  Queries run fine, but building the table takes a
> good 2 days on a dual xenon 500 w/2gb of ram.

How are you doing it?

I've built tables faster than that (and much larger) in less time.
Since you used the term "building", I assume you're bulk loading the
data?  If you delay index creation until the data is loaded, and do so
with a big key buffer, you'll get decent performance.

Jeremy
-- 
Jeremy D. Zawodny     |  Perl, Web, MySQL, Linux Magazine, Yahoo!
<Jeremy at Zawodny.com>  |  http://jeremy.zawodny.com/