[svlug] How bigs my home

Tim Utschig tim at tetro.net
Tue Oct 27 13:37:09 PST 2009


On Thu, Oct 22, 2009 at 02:13:55AM -0400, james at linuxrebel.org wrote:
> du -sh will print out a very nice accounting of the size of a
> dir (used in a script of course) but on some of the larger ones
> say 20GB it can take 25-30 minutes to return.
...
> This is done over a NFS mount rather than directly on the NAS
> itself. Can't change the file system (NAS only does what it
> does) Can't tweak the file system (3rd party managed by people
> who only know how to say no).  Can't install software on the
> NAS itself.


At $EMPLOYER we have the same problem.  In our case the NAS are
NetApp filers.  We make heavy use of quotas, tree-based and
user-based.

On some volumes where users demand a disk space "free for all" we
still enable user-based quotas with no enforced limits so that we
can take advantage of the instantaneous quota report to see the
disk usage breakdown by user.

Despite this we still need to run 'du' or 'find' fairly often for
one reason or another.  For us this typically takes at least a
half-hour per million files.  The volumes we need to search
usually contain between 5 and 10 million files each.

If your NAS is NetApp and you have admin privileges...

Don Marti pointed out on this list back in July [1] that NetApp
has an API interface.  Example scripts are available in the
"Manage OnTAP(TM) SDK":

  http://communities.netapp.com/docs/DOC-1365

I didn't see any API for change notification, only on-demand
virus scanner functionality ("screening") provided by 'fpolicy'.

I've yet to find the time to figure out if implementing a 'du' or
'find' using the API would be any faster, but I've since found it
very handy in quota management scripts.

The API is typically used over HTTP/HTTPS, however I noticed that
there is an "ontapi" command available if you switch to "priv set
advanced".  Using this method, the scripts can rely on SSH public
key authentication.

A session listing file contents might start with something like
this and then recurse:

  filer> priv set advanced
  filer*> ontapi execute <file-list-directory-iter-start><path>/vol/foo</path></file-list-directory-iter-start>
  filer*> ontapi execute <file-list-directory-iter-next><tag>64810137913</tag><maximum>123</maximum></file-list-directory-iter-next>
  filer*> ontapi execute <file-list-directory-iter-end><tag>64810137913</tag></file-list-directory-iter-end>

Where the directory entry count 123 and the tag 64810137913 are
returned in the result of <file-list-directory-iter-start>.


-- 
   - Tim Utschig <tim at tetro.net>




More information about the svlug mailing list