[svlug] Mount Points

[Javilk]

     All this discussion on where different files should be put is
somewhat interesting. 

     What SHOULD one do, when one is trying to balance fragments of
scratch and other disk drives in a high performance situation? 

     I've been making /whatever/tmp, directories for drives with other
primary purposes, as I shuffle half gig or larger gobs of data around,
(some in scads of little files, and some in half gig files,) look for temp
sort spaces that are not on the same drive as the primary or destination
data, etc. (And three hours into a run, something fills up and blamo, the
rest is worthless... at 2am when I want the reports out by 6am, of
course!)

     Is there a reasonably standard way of doing this kind of stuff that
doesn't end up with huge path names?  Does a symbolic link generate yet
another head seek, or two? (Head seeks being the slowest part of the whole
process.)

    I've also been using /aux-n and /otherMachineName directories off
root, and /tmp/task to share task information between some of my machines.
What is the convention on such things? 

     I've kind of predicated some of this on the supposition that
pre-sorting data into smaller files, then merging them later is more
efficient than sorting a huge monolith, as well as provides enough
interruptions that it does not bog everything to death.  (Before using
Nice, I was occasionally losing PPP when I hit a big step, as the data
fetch tasks that I also have running, would fail.)

     Also, on another project, I built up a directory tree on an /aux
drive that ran out of inodes as I was mashing, sorting, and straining
data. The drive was only 60% full, but it was full when the system tried
to add yet another file to the tree.  I really didn't find anything that
explained enough for me to understand much about how to direct inode
generation.  A lot of these files are in the 2K to 5K range. 

     "If I knew what I was doing, I'd be doing something else."

     Or as a friend put it: "If we knew what we were doing, it wouldn't be
called research." 

     I've been thinking of distributing some of this across my LAN, and
have done it with some machines, but the 100BaseT tulip drivers kill my
machine on large file transfers, so I am stuck with 10BaseT.  Another
potential client and I were talking about this, and he mentioned that he
too, was having trouble with Linux tulip drivers, and that the tulip cards
he was buying OEM in 100 - 500 unit batches, were being changed as he got
them, even at the chip level. 

     "There's got to be an easier way of earning a living!" "This isn't
even a living!"  Of course, the best one I heard was "We'll make up for
our loss on each transaction by volume." 

     Are any of you working with very large files that you have to massage
a lot?  What are some of the issues you are finding, and how do you cope
with them?

- javilk at mall-net.com -----------------------------
-------- MS asks "Where do you want to go?" -------
------- Linux asks "What do you want to do?" ------
-- It is doers, not goers, who built this world! --
--------- Member: http://www.svlug.org/ -----------

--
echo "unsubscribe svlug" | mail majordomo at svlug.org
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ to unsubscribe
see http://www.svlug.org/mdstuff/lists.shtml for posting guidelines.