[svlug] Linux + MySQL w/ very large data sets
Jeremy at Zawodny.com
Wed Jun 12 15:55:15 PDT 2002
On Wed, Jun 12, 2002 at 03:13:41PM -0700, Aaron T Porter wrote:
> On Wed, Jun 12, 2002 at 03:00:22PM -0700, Drew Bertola wrote:
> > Can anyone tell me what the best architecture / OS support is
> > suggested for handling data with possible table sizes in the 100s of
> > GB range? This would be under MySQL (though I'd consider other DBs)
> > and, other than monthly updates, only SELECT transactions need be
> > considered.
> You'll probably be really stressing mysql at that point. And
> even monthly updates sound a bit scary at that size. My largest
> mysql tables currently are ~35m rows/2gb on disk (data file, not
> counting indexes). Queries run fine, but building the table takes a
> good 2 days on a dual xenon 500 w/2gb of ram.
How are you doing it?
I've built tables faster than that (and much larger) in less time.
Since you used the term "building", I assume you're bulk loading the
data? If you delay index creation until the data is loaded, and do so
with a big key buffer, you'll get decent performance.
Jeremy D. Zawodny | Perl, Web, MySQL, Linux Magazine, Yahoo!
<Jeremy at Zawodny.com> | http://jeremy.zawodny.com/
More information about the svlug