[volunteers] State of the mailing lists, again

Rick Moen rick at linuxmafia.com
Thu Jul 7 22:40:10 PDT 2016
Previous message: [volunteers] State of the mailing lists, again
Next message: [volunteers] (forw) Re: [revival] exim4 dead on svlug, restarting after this dumped state
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Fixed it.  It was a real bear, too.  There were two separate corruption
problems in the membership list for svlug@ .

I noticed a couple of dozen messages shunted off to
/var/local/mailman/files/shunt

Some searching found:


  Kevin W. Gagel wrote:

  >I have about 60+ files in my shunt directory. This is the result of a
  >show_qfiles, what's up? What am I supposed to do and how do I fix it?

  Look in your mailman error log for the original shunt messages and
  associated errors. You then may able to decide which errors might have
  been transient and which were the result of things that have been (or
  can be) fixed. At this point, you may want to delete some of the files
  from the shunt queue because they are too old to be relevant or for
  any other reason. You can then run bin/unshunt to try to reprocess the
  remaining messages.


  1. The shunt directory is a place where messages go when there is an
  error with some functions that process them. In my case, it was because
  of an archiving error in version 2.1.3 that was fixed with an upgrade to
  2.1.4

  2. To fix it, there is a program called unshunt in the ~/mailman/bin
  directory. You simply run it from the command line; you don't need any
  special parameters:

  % path-to-mailman/bin/unshunt

  3. Mark Sapiro gave an excellent answer to this on the -users list. Here
  it is in brief:

  "If a message is shunted because SMTPDirect.py can't write the 'post'
  log, it has already been sent, but OutgoingRunner.py doesn't know this
  when it catches the exception and shunts the message. So those messages
  which were shunted because of ownership/permission issues on the 'post'
  log probably were duplicated when you ran unshunt."

  I also imagine the same logic applies for various other errors. In
  short, a message CAN both be delivered and end up in the shunt queue, so
  you need to carefully look at everything in the shunt queue before
  unshunting.

./../bin/show_qfiles 1415074451.8136821+46c1884fee3a7dd248e55a197d6e6469c1563a4b.pck | less
lists:/var/local/mailman/qfiles/shunt# rm 1415074451.8136821+46c1884fee3a7dd248e55a197d6e6469c1563a4b.pck       
rm: remove regular file `1415074451.8136821+46c1884fee3a7dd248e55a197d6e6469c1563a4b.pck'? y
lists:/var/local/mailman/qfiles/shunt#

lists:/var/local/mailman/qfiles/shunt#../../bin/show_qfiles 1426601026.126564+54cef6f47d2df4f601520701a3d30090ff48a64b.pck | less
lists:/var/local/mailman/qfiles/shunt# rm -f 1426601026.126564+54cef6f47d2df4f601520701a3d30090ff48a64b.pck       
lists:/var/local/mailman/qfiles/shunt# 

And so on, for other held spam in the shunt queue.  Leaving:

lists:/var/local/mailman/qfiles/shunt# ls -al
ls: unrecognized prefix: rs
ls: unparsable value for LS_COLORS environment variable
total 388
drwxrws---    2 mailman  mailman      4096 Jul  7 21:15 .
drwxrwsr-x   12 root     mailman      4096 Oct 17  2003 ..
-rw-rw----    1 mailman  mailman     15353 Jun  3 10:50 1464976194.932795+2b7ba330ae7fae56b4a2aba41e0348a67c52c834.pck
-rw-rw----    1 mailman  mailman     15813 Jun  3 14:25 1464989107.255311+d0bef7b2b37995d406903b9744511e317371df30.pck
-rw-rw----    1 mailman  mailman     16760 Jun  3 16:51 1464997832.849174+ac626890240c5cd5d19d21e78d51c0a3b5acb867.pck
-rw-rw----    1 mailman  mailman     18101 Jun  3 18:46 1465004756.3070819+324adc566cfb9f663ec5bf80dd9ca72b65b91a51.pck
-rw-rw----    1 mailman  mailman     17319 Jun  3 21:07 1465013202.91295+61d8e6dee577cf8b8b8df5fe33f4df202d2f4221.pck
-rw-rw----    1 mailman  mailman     15268 Jun  3 23:33 1465022000.2646821+3b1d5a531d03fdf41c3de43542d4751b9e6ae6b1.pck
-rw-rw----    1 mailman  mailman     14893 Jun  7 08:24 1465313066.6503839+0bd13a3104546cbe564da818297b5caa5eda1b5b.pck
-rw-rw----    1 mailman  mailman     14032 Jun  7 11:01 1465322484.321697+ac59f0bf522794b1709a3388322745e2c36dab7b.pck
-rw-rw----    1 mailman  mailman     20026 Jun  7 11:12 1465323117.7044799+57213804c6bac614e5d87973b32bc0cf86bbdd23.pck
-rw-rw----    1 mailman  mailman     15762 Jun  7 11:22 1465323732.890147+5d6de25123b9239165c4da11ef9cc23c7d90021d.pck
-rw-rw----    1 mailman  mailman     15847 Jun  7 14:32 1465335122.582237+acb337a69a54362824b8c3ebc18a652deee2c592.pck
-rw-rw----    1 mailman  mailman     14926 Jun  8 16:43 1465429382.2427559+fde4f97db2defcbefb78215132bec2c84c7f5308.pck
-rw-rw----    1 mailman  mailman     17358 Jun  9 01:00 1465459185.299103+7e73285385ae2c37c0de0c038098d8c4eb65bca3.pck
-rw-rw----    1 mailman  mailman     14520 Jun  9 07:17 1465481839.5757289+448c03add74f00df132d681598a50ea984b725fd.pck
-rw-rw----    1 mailman  mailman     14367 Jun 17 07:39 1466174324.933311+f90eea7ace9c74f23243eaacd6f36a34eaebdb99.pck
-rw-rw----    1 mailman  mailman     15489 Jul  1 16:12 1467414699.8297131+b8e9802b270c91d3a7e6890af8011969dba72acb.pck
-rw-rw----    1 mailman  mailman     16556 Jul  6 15:17 1467843366.1405561+418a51865a94e6e42e1263bae32094e0b6e795ce.pck
-rw-rw----    1 mailman  mailman     16512 Jul  6 15:17 1467843405.357537+8c4ba1562150d7d3ce1f3089324f897b01c49540.pck
-rw-rw----    1 mailman  mailman     15783 Jul  7 13:11 1467922231.3513949+a0a7c95072f75b3b7f95512da09aa0e7947f579a.pck
-rw-rw----    1 mailman  mailman     21754 Jul  7 16:54 1467935647.2861271+d1201df77da757896d75229e473c3bc8740eedac.pck
-rw-rw----    1 mailman  mailman     22299 Jul  7 17:09 1467936568.1888299+e4549e8e3cffdd49db68cb0d3d806d417e460888.pck
lists:/var/local/mailman/qfiles/shunt#

ists:/var/local/mailman/qfiles/shunt# ../../bin/unshunt .
lists:/var/local/mailman/qfiles/shunt# ls -al
ls: unrecognized prefix: rs
ls: unparsable value for LS_COLORS environment variable
total 388
drwxrws---    2 mailman  mailman      4096 Jul  7 21:34 .
drwxrwsr-x   12 root     mailman      4096 Oct 17  2003 ..
-rw-rw----    1 mailman  mailman     15353 Jul  7 21:34 1464976194.932795+eef52927a39d48d5e92b6386434f47ac568959e3.pck
-rw-rw----    1 mailman  mailman     15813 Jul  7 21:34 1464989107.255311+7ba913177cb19694915efa975ff093e91181e1b2.pck
-rw-rw----    1 mailman  mailman     16760 Jul  7 21:34 1464997832.849174+613aeacdbc028a4e7cb4414d0541a09dcd612a7c.pck
-rw-rw----    1 mailman  mailman     18101 Jul  7 21:34 1465004756.3070819+9bda57d69e22fe33304c9886378a34b45a499914.pck
-rw-rw----    1 mailman  mailman     17319 Jul  7 21:34 1465013202.91295+5049d1dc42382e6b8ec51b9c4eb1621f253d7352.pck
-rw-rw----    1 mailman  mailman     15268 Jul  7 21:34 1465022000.2646821+b3c3d7df92866b22279f9a3639c6a2aaed0c9a8e.pck
-rw-rw----    1 mailman  mailman     14893 Jul  7 21:34 1465313066.6503839+795536b716d7e71f8d4ea643f4e3ea89c6d57197.pck
-rw-rw----    1 mailman  mailman     14032 Jul  7 21:34 1465322484.321697+b5194a47ee915ea986a99755d633ab891b1f4099.pck
-rw-rw----    1 mailman  mailman     20026 Jul  7 21:34 1465323117.7044799+7bce9ddc94d72209383c2b9fca395995ff042f4b.pck
-rw-rw----    1 mailman  mailman     15762 Jul  7 21:34 1465323732.890147+0dc9e842ec17e1bc5c28bcc64b5c1987faea5831.pck
-rw-rw----    1 mailman  mailman     15847 Jul  7 21:34 1465335122.582237+f4fe79a8644d21fc322bf3d67ad792b13f1aa743.pck
-rw-rw----    1 mailman  mailman     14926 Jul  7 21:34 1465429382.2427559+8784238e85791af678de985d7e1126996c9d2d82.pck
-rw-rw----    1 mailman  mailman     17358 Jul  7 21:34 1465459185.299103+dcb1ab906668cb936cb95db5f7e00233ffda27f2.pck
-rw-rw----    1 mailman  mailman     14520 Jul  7 21:34 1465481839.5757289+887eb344d605985bd0bd3a009a8a039969d565c0.pck
-rw-rw----    1 mailman  mailman     14367 Jul  7 21:34 1466174324.933311+f23cebeb64fe533c5b39bd22674ab50b80bf49f9.pck
-rw-rw----    1 mailman  mailman     15489 Jul  7 21:34 1467414699.8297131+d31d8c5fdfb8ab61d054c1cf469e5602a2708511.pck
-rw-rw----    1 mailman  mailman     16556 Jul  7 21:34 1467843366.1405561+316057caad86c656e51455b291c926d84c1c07a1.pck
-rw-rw----    1 mailman  mailman     16512 Jul  7 21:34 1467843405.357537+16d31d4b84d50e902a59a297644d4dad4a35ebfe.pck
-rw-rw----    1 mailman  mailman     15783 Jul  7 21:34 1467922231.3513949+8862b031e08f57a5d460bf26e4278eafa6f51c0f.pck
-rw-rw----    1 mailman  mailman     21754 Jul  7 21:34 1467935647.2861271+4335b464a4f77990f831b3e9bfc4a3eb777f2190.pck
-rw-rw----    1 mailman  mailman     22299 Jul  7 21:34 1467936568.1888299+4f208e3b443b030c1adc6c9781cce096e5b730e7.pck
lists:/var/local/mailman/qfiles/shunt# cd -
/var/local/mailman/logs
lists:/var/local/mailman/logs# 

New errors in /var/local/mailman/logs/error are all like:

Jul 07 21:34:03 2016 (1455) SHUNTING: 1467935647.2861271+4335b464a4f77990f831b3e9bfc4a3eb777f2190
Jul 07 21:34:03 2016 (1455) Uncaught runner exception: ASCII encoding error: ordinal not in range(128)
Jul 07 21:34:03 2016 (1455) Traceback (most recent call last):
  File "/var/local/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
    self._onefile(msg, msgdata)
  File "/var/local/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/var/local/mailman/Mailman/Queue/OutgoingRunner.py", line 73, in _dispose
    self._func(mlist, msg, msgdata)
  File "/var/local/mailman/Mailman/Handlers/SMTPDirect.py", line 152, in process
    deliveryfunc(mlist, msg, msgdata, envsender, refused, conn)
  File "/var/local/mailman/Mailman/Handlers/SMTPDirect.py", line 285, in verpdeliver
    Decorate.process(mlist, msgcopy, msgdata)
  File "/var/local/mailman/Mailman/Handlers/Decorate.py", line 58, in process
    username = username.encode(Utils.GetCharSet(d['user_language']))
UnicodeError: ASCII encoding error: ordinal not in range(128)

Searching 'ASCII encoding error: ordinal not in range(128)':

https://mail.python.org/pipermail/mailman-users/2003-October/032058.html
https://www.mail-archive.com/mailman-users@python.org/msg35001.html

The latter link calls my attention to the line 'username = username.encode(Utils.GetCharSet(d['user_language']))'.


lists:/var/local/mailman/logs# cd lists:/var/local/mailman
lists:/var/local/mailman# bin/list_members -f svlug | less

produces output that includes:

Jose De Ponte <
jdeponte2000 at yahoo.es>
Abhijit Pandey <abhijit_pandey at mcafee.com>
abvathomas at gmail.com
"A.C. Man" <acman18 at yahoo.com>
Alberto Duenas <add at inicia.es>
Ajay Mulwani <ajaymulwani at gmail.com>
Akkana Peck <akkana at shallowsky.com>

First entry is obviously malformed.  Perhaps an embedded control character.

Indeed, dumping that command's output to a file and looking at it in vi shows:

Jose De Ponte <^M
jdeponte2000 at yahoo.es>

Created file 'mem' containing only that portion of the output.

lists:/var/local/mailman# bin/remove_members --file=mem svlug
No such member: Jose De Ponte <
No such member: jdeponte2000 at yahoo.es>
lists:/var/local/mailman# 

Well, drat.

Web admin interface shows that there's a member
'jdeponte2000 at yahoo.esdeponte2000@yahoo.es'.  Deleted using Web
interface.

lists:/var/local/mailman# bin/list_members -f svlug > mem && vi mem 
shows that the list still starts with these two lines:

^M
jdeponte2000 at yahoo.es

lists:/var/local/mailman# bin/withlist -l svlug
Loading list svlug (locked)
The variable `m' is the svlug MailList instance
>>> adr = '\r'
>>> m.removeMember(adr)
Traceback (most recent call last):
  File "<console>", line 1, in ?
  File "/var/local/mailman/Mailman/OldStyleMemberships.py", line 220, in removeMember
    self.__assertIsMember(member)
  File "/var/local/mailman/Mailman/OldStyleMemberships.py", line 113, in __assertIsMember
    raise Errors.NotAMemberError, member
NotAMemberError: 
>>> del m.members[adr] 
Traceback (most recent call last):
  File "<console>", line 1, in ?
KeyError: 
>>> 

I did a Ctrl-D to exit.

Unlocking (but not saving) list: svlug
Finalizing
lists:/var/local/mailman# 

lists:/var/local/mailman# bin/remove_members svlug '^M'
No such member: ^M
lists:/var/local/mailman# bin/remove_members svlug "^M"
No such member: ^M
lists:/var/local/mailman# 

https://wiki.list.org/DOC/How%20do%20I%20remove%20a%20user%20name%20or%20email%20address%20with%20an%20illegal%20character%20in%20it%20(or%20Mixed%20or%20Upper%20case%20domain)%3F 
has a suggestion.

lists:/var/local/mailman# bin/list_members -i svlug

jdeponte2000 at yahoo.es
lists:/var/local/mailman# 

Promising!  So:

lists:/var/local/mailman# bin/list_members -i svlug | bin/remove_members -f - svlug
No such member: jdeponte2000 at yahoo.es
lists:/var/local/mailman# 

Dammit.  Let's try a custom remove script written by Mailman expert Mark Shapiro.

lists:/var/local/mailman/bin# wget http://fog.ccsf.cc.ca.us/~msapiro/scripts/remove_bad_address.py
--22:14:42--  http://fog.ccsf.cc.ca.us/%7Emsapiro/scripts/remove_bad_address.py
           => `remove_bad_address.py'
Connecting to fog.ccsf.cc.ca.us:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]

    0K -> .

22:14:42 (1.40 MB/s) - `remove_bad_address.py' saved [1468]

lists:/var/local/mailman/bin# ./withlist -r remove_bad_address svlug
Importing remove_bad_address...
Running remove_bad_address.remove_bad_address()...
Loading list svlug (unlocked)
List: svlug, Invalid address: 
jdeponte2000 at yahoo.es
    removed.
Finalizing
lists:/var/local/mailman/bin# 

ists:/var/local/mailman/bin# ./unshunt ../qfiles/shunt/
lists:/var/local/mailman/bin# 

lists:/var/local/mailman/bin# ls -al ../qfiles/shunt/
ls: unrecognized prefix: rs
ls: unparsable value for LS_COLORS environment variable
total 388
drwxrws---    2 mailman  mailman      4096 Jul  7 22:16 .
drwxrwsr-x   12 root     mailman      4096 Oct 17  2003 ..
-rw-rw----    1 mailman  mailman     15353 Jul  7 22:16 1464976194.932795+3154fd1584c275925c0f932b2b97cb3368d400b1.pck
-rw-rw----    1 mailman  mailman     15813 Jul  7 22:16 1464989107.255311+697c02887727d6d6ca9ed66d870fdc855fd4e934.pck
-rw-rw----    1 mailman  mailman     16760 Jul  7 22:16 1464997832.849174+07ab2139e6d996c038b3ceb04f0c8f4af2d3ff90.pck
-rw-rw----    1 mailman  mailman     18101 Jul  7 22:16 1465004756.3070819+d3933d21b4f19548dff2902af7c3909f8cfb4495.pck
-rw-rw----    1 mailman  mailman     17319 Jul  7 22:16 1465013202.91295+8e1a073c6807a549b4daaf347f625f4327df778a.pck
-rw-rw----    1 mailman  mailman     15268 Jul  7 22:16 1465022000.2646821+580ebde340a0baf5627fc6aa6471b1b95fa25eb1.pck
-rw-rw----    1 mailman  mailman     14893 Jul  7 22:16 1465313066.6503839+48b848ecf3dcbcecafdcf14fd0d0f0e940c66386.pck
-rw-rw----    1 mailman  mailman     14032 Jul  7 22:16 1465322484.321697+56449a7565f090424330457a32cc196124a8894e.pck
-rw-rw----    1 mailman  mailman     20026 Jul  7 22:16 1465323117.7044799+aa7773049bbc40fb827a7dc3dc127752a02cc4b9.pck
-rw-rw----    1 mailman  mailman     15762 Jul  7 22:16 1465323732.890147+85d899d35510cc932c314db66fb47e7b0df5e398.pck
-rw-rw----    1 mailman  mailman     15847 Jul  7 22:16 1465335122.582237+670ae58163fd23393a2a7dbd879843fd0d34e4f0.pck
-rw-rw----    1 mailman  mailman     14926 Jul  7 22:16 1465429382.2427559+ad65d10ead9baca52b332efae05c7b79069be560.pck
-rw-rw----    1 mailman  mailman     17358 Jul  7 22:16 1465459185.299103+a12ebf9d943b093a1716ecf1003ed4f497da25ff.pck
-rw-rw----    1 mailman  mailman     14520 Jul  7 22:16 1465481839.5757289+e0090efacc4d13e5193b28bea3c78faa805adb24.pck
-rw-rw----    1 mailman  mailman     14367 Jul  7 22:16 1466174324.933311+a03ed59611e8cb02e833593ee25acd94e649d596.pck
-rw-rw----    1 mailman  mailman     15489 Jul  7 22:16 1467414699.8297131+bb17c952c1b6cfc671fd30973e8653b5433a2f6d.pck
-rw-rw----    1 mailman  mailman     16556 Jul  7 22:16 1467843366.1405561+866963e47208b67486b15c2b2b4fafca8f8f5552.pck
-rw-rw----    1 mailman  mailman     16512 Jul  7 22:16 1467843405.357537+501cc6c730d7771fb6c2e81fe7a52a0dc02991c2.pck
-rw-rw----    1 mailman  mailman     15783 Jul  7 22:16 1467922231.3513949+a897b705df871fbfe659d1a391cfe5f7ca59c740.pck
-rw-rw----    1 mailman  mailman     21754 Jul  7 22:16 1467935647.2861271+5a3eca567e96e1ec6fb4ea6900d7e0fb795fed6c.pck
-rw-rw----    1 mailman  mailman     22299 Jul  7 22:16 1467936568.1888299+c9d682b823b9d6f4cc69d8929a61622fc4002222.pck
lists:/var/local/mailman/bin# 

lists:/var/local/mailman/bin# cd ../logs
lists:/var/local/mailman/logs# less error 

Errors now look like this:

Jul 07 22:16:46 2016 (1455) SHUNTING: 1467935647.2861271+5a3eca567e96e1ec6fb4ea6900d7e0fb795fed6c
Jul 07 22:16:46 2016 (1455) Uncaught runner exception: ASCII encoding error: ordinal not in range(128)
Jul 07 22:16:46 2016 (1455) Traceback (most recent call last):
  File "/var/local/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
    self._onefile(msg, msgdata)
  File "/var/local/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/var/local/mailman/Mailman/Queue/OutgoingRunner.py", line 73, in _dispose
    self._func(mlist, msg, msgdata)
  File "/var/local/mailman/Mailman/Handlers/SMTPDirect.py", line 152, in process
    deliveryfunc(mlist, msg, msgdata, envsender, refused, conn)
  File "/var/local/mailman/Mailman/Handlers/SMTPDirect.py", line 285, in verpdeliver
    Decorate.process(mlist, msgcopy, msgdata)
  File "/var/local/mailman/Mailman/Handlers/Decorate.py", line 58, in process
    username = username.encode(Utils.GetCharSet(d['user_language']))
UnicodeError: ASCII encoding error: ordinal not in range(128)


lists:/var/local/mailman# bin/list_members -f svlug > mem
lists:/var/local/mailman# 

Let's look some more using vi.  Aha!  Found member:

Diego Fernando Carri?n <theczar at sdf.org>

lists:/var/local/mailman# bin/remove_members svlug theczar at sdf.org 
lists:/var/local/mailman# 

lists:/var/local/mailman# cd bin
lists:/var/local/mailman/bin# ./unshunt ../qfiles/shunt/
lists:/var/local/mailman/bin# ls -al ../qfiles/shunt/
ls: unrecognized prefix: rs
ls: unparsable value for LS_COLORS environment variable
total 8
drwxrws---    2 mailman  mailman      4096 Jul  7 22:28 .
drwxrwsr-x   12 root     mailman      4096 Oct 17  2003 ..
lists:/var/local/mailman/bin# 

Seems to have worked.  But I'd like to put Diego back, because it's not
his fault.

The gentleman's name is as per http://theczar.sdf.org/ , and I feel
really bad about needing to deprive him of his accent mark for now
(until we get a newer and better Mailman), but:

Created new file 'mem'.
lists:/var/local/mailman/bin# ./add_members -r mem svlug
Subscribed: Diego Fernando Carrion <theczar at sdf.org>
lists:/var/local/mailman/bin# 

Fixed.
(Removed 'mem'.)
Previous message: [volunteers] State of the mailing lists, again
Next message: [volunteers] (forw) Re: [revival] exim4 dead on svlug, restarting after this dumped state
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the volunteers mailing list