[volunteers] (forw) [web-team] Linode Support Ticket 16507235 - svlugmicah (svlugmicah) - ToS Violation - Spam

[Rick Moen]

Lisa Corsetti, note something to you specifically, below.  I am
CCing you even though you're still subscribed to volunteers@ .


Just received.  Report of outbound spam from our 64.62.190.98 IP
can be seen (as noted below) at
https://linode.abusehq.net/share/7niCyMIuTTpdyvYCfEXRO260_cXQt3fb0dC814CnYwZ9hJ85yx7qtdJkZ9XeriZMzA28MOcV9sL3nNVLMN3uJQ
.

Assuming this is true (and I am inclined to believe it), this
suggests security compromise, which among other things could 
lead to a huge bill from Linode if it goes on for long.

The security report _could_ be a false positive, but there are 
reasons to suspect it isn't.


I am doing final backup of /home, /usr/local, 
/root, /etc, /var/www/ , trying to be careful to err on the side
of inclusiveness.

$ rsync  -aHxv --exclude={"/bin/*","/boot/*","/dev/*","/lib*","lost+found/*","/mnt/*","/proc/*","/run/*","/sbin/*","selinux/*","/sys/*","/tmp/*","/usr/bin/*","/usr/doc/*","/usr/games/*","/usr/lib/*","/usr/sbin/*","/usr/share/*","/usr/src/*","/var/backups/*","/var/cache/*","/var/lib/*","/var/local/*","/var/log*","/var/mail/*","/var/opt/*","/var/spool/*","/var/tmp/*"} root at www.svlug.org:/  /mnt/svlug-backup

(That just now completed.)


along with:

1.  Capturing dpkg --get-selections "*" > /root/selections-$(date +%F)
2.  Capturing tar cvzf /root/etc-$(date +%F).tar.gz /etc
3.  Capturing output of ifconfig -a and route -n
4.  Capturing a ps auxw snapshot of production services, which are:

/sbin/init  (still Upstart)
upstart-udev-bridge
udevd
upstart-socket-bridge
dhclient3  (pseudo-DHCP client, required because Linode
sshd 
cron
syslog
/bin/dd bs=1 if=/proc/kmsg of=/var/run/klogd/kmsg
klogd 
nullmailer-send  (This is something I set up in 2011 so that
  root's administrative mail (basically, just occasional reports
  from the NSD namserver) gets forwarded to me.  There are no other outbound
  SMTP services we intend to run on this host.  nullmailer is
  restricted to sending mail _only_ to rick at linuxmafia.com .)
ntpd
getty 
nsd
lighttpd
php-cgi

5.  Just now did, to try to interim-block outbound SMTP:
root at gruyere:/etc # iptables -A OUTPUT -p tcp --dport 25 --j DROP

root at gruyere:/etc # iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere             tcp dpt:smtpa
root at gruyere:/etc #

However, if the system is kernel-compromised, this may not 
work or may not persist in working.




6.  Once I have the final backup underway (update: backup completed),
will enter a note to Linode's support ticket (update: done) with the
plan, which is:

7.  Shut down the virthost.  (Update: done.)  As a reminder, the Linode
virthhost houses two SVLUG public services.  It serves www.svlug.org's
Web pages, and it is DNS authoritative nameserver ns1.svlug.org .  It
does _NOT_ house SVLUG's mailing lists.

Update:  Shutting down the www.svlug.org / ns1.svlug.org virthost
right now.  (Update: done.)  This is done using the Linode Cloud Manager
at "login.linode.com" using the sensitive credentials shared among the
active Web Team.

8.  Notify sysadmins of domains for which ns1.svlug.org does auth
DNS that downtime has begun.  Those are:

secondary:
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa  (local-administrative)
balug.org
berkeleylug.com
bluedreamz.com
substancez.com
substancez.net
substancez.org
cherylmorris.com
sf-lug.org
sf-lug.net
sf-lug.com
sflug.org
sflug.net
saclug.org
sflug.com
brie.com

primary:
svlug.org
svlug.net
svlug.com

I will need to work with the secondary providers for the last three 
domains to (tempoarily) re-point to a new DNS master, probably my
ns1.linuxmafia.com .

The non-LUG domains you see secondary DNS for are for LUG old-timers
Duncan MacKinnon and Brian Lavender, and for my mother-in-law Cheryl
Morris.

The present svn-based repo (/var/svn/web) is sad, hilarious, and
inefective.  I hope to rearchitect it using git, and take seriously
offsite replication (which was never done, making svn only a tiny bit
useful.  svn was yet another Lisa Corsetti choice made without
consultation with anyone else, that I thought was dumb and never liked.




Some history:  SVLUG VP Micah Dowty got us this free-of-charge Linode
virthost in 2007.  (Great!)  Without consultation with everyone else
(bad!), he picked Ubuntu Server, and then said, 'Hey, over to you, Rick.
Maintain this."  I objected, saying this would be much, much worse for
maintenance than using Debian, and that I, the almost sole active
sysadmin, didn't like it.  Micah, who is a coder, not a sysadmin, told
me to suck it up.  He also claimed that all of SVLUG's operations
including the SMTP and mailing lists would easily work on the virthost,
which for the first few years had 80 MB RAM and 3GB disk.  I said Micah
was mistaken on that; he said I was wrong.  I was correct, which is why
lists.svlug.org remains to this day elsewhere (on prgmr.com ).

"Free-of-charge" later turned into "free of charge provided that your
monthly usage of bandwidth doesn't exceed a cap.  If it does, there
will be an invoice.  I have paid one of those, and don't wish to keep
doing so.

Ubuntu Server has borne out my prediction of sucking a lot for
administration.  E.g., the NSD nameserver package has repeatedly broken
badly during upgrades, because Ubuntu sometimes doesn't bother to 
provide a migration path from, e.g., 2.x to 3.x other than telling
the admin "Oh, by the way, your production nameserver just broke, and 
you're going to have to rewrite its configuration files before it will
work again."

Over the last few years, I've increasingly _not_ waded in and done
further upkeep, because of repeated experience with sudden emergency
situations such as described above, when I was just trying to update
packages.  In consequence, the system is not in a maintained state:

root at gruyere:/etc # cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.5 LTS"
root at gruyere:/etc # uname -r
5.11.9-x86-linode162
root at gruyere:/etc #

I've been long expecting a security meltdown, especially 
following:

In 2011, Lisa Corsetti decided, without consulting anyone (bad!) to
convert the many legacy .shtml (server-side include) pages to PHP5.
At the same time, she converted antiquated widgets such as the
"sponsor spinner" on the front page from buggy Perl to PHP5.  

I recognise that PHP solved both problems, and Lisa's work was 
tidy and effective, _but_ I told her at the time that PHP is
an infamous security nightmare, and I strongly urged that all
PHP-based features she'd just coded be changed from relying on
public-facing FastCGI-loaded PHP to local automation, e.g., 
generating static HTML via cron jobs or Makefiles.  I stressed 
that this was important.  Lisa's reaction was basically 
"Well, that would be nice to do some day", and elected to blow 
off my request.


So, here we are.  I have no doubt that the host _is_ security-comprised,
having expected this for years.  All my pleading to keep the host
minimal and follow sysadmin best practices got ignored, first 
by Micah and then by Lisa.  This day has been a long time coming.

OK, then.  I'm going to architect a new www.svlug.org and ns1.svlug.org
host on a new Linode virthost.  This may take some time.  If
anyone notices (and cares) that the non-mailing-list Web site 
is down, point to this posting and tell why.

In the future, assuming SVLUG has a future, can we please, please,
please not have volunteers going cowboy and making major decisions
without consulting anyone else -- and _especially_ not have 
coders making system administration decisions without consulting
sysadmins?


9.  I expect relevant discussion about rearchitecting/rebuild will
be on the private revival at linuxmafia.com mailing list.  SVLUG
volunteers who's like to be there and participate, who aren't 
already subscribed let me know.

I'll admit I'm a little irritated because I _warned_ about this,
and also, I really don't have time for, once again, dealing with
an SVLUG emergency when I haven't even done my own long-delayed 
site rebuild.


-- Rick Moen
   rick at linuxmafia.com
   (650) 283-7902 cellular



----- Forwarded message from support at linode.com -----

Date: Fri, 12 Nov 2021 04:54:02 -0500
From: support at linode.com
To: webmaster at svlug.org
Subject: [web-team] Linode Support Ticket 16507235 - svlugmicah (svlugmicah)
	- ToS Violation - Spam


Support Ticket 16507235 has been updated by Linode:

--------------------------------------------------
Hello,

We have received a report of Spam originating from your Linode. This is most likely the result of a system compromise. If we have not heard from you in 24 hours, we may need to place network restrictions on your Linode to prevent further abuse.

In order to consider this resolved we will require the following from you:

   * Information about why this email might have been marked as spam

   * Steps taken to prevent this activity from reoccurring

We're here to help provide guidance, but keep in mind that investigating this on your behalf is beyond the [1]scope of our support. Our Community Questions site can offer guidance in resolving this issue:

   * [2]Why are my emails getting marked as spam?

   * [3]I've noticed some suspicious activity on my Linode, what do I do?

If you need additional assistance, you can always create your own post on our Community Questions site to get help from the [4]Linode Community. If you determine that you are unable to resolve this issue yourself, we strongly suggest that you [5]rebuild your Linode.

Regards,
Linode Support Team

[1] https://www.linode.com/docs/platform/billing-and-support/support/#scope-of-support
[2] https://www.linode.com/community/questions/17685/why-are-my-emails-getting-marked-as-spam
[3] https://www.linode.com/community/questions/467/ive-noticed-some-suspicious-activity-on-my-linode-what-do-i-do
[4] https://www.linode.com/community/questions/
[5] https://www.linode.com/docs/troubleshooting/rescue-and-rebuild


Report Details:

https://linode.abusehq.net/share/Aon0te-75JAitkJTrNIn6g

Within the Network Abuse Report page, please click the "Show" button at the right of every event for more details.


--------------------------------------------------
Please use https://cloud.linode.com/support/tickets/16507235 to respond to this ticket.

Thank you,

Linode.com

_______________________________________________
web-team mailing list
web-team at lists.svlug.org
http://lists.svlug.org/lists/listinfo/web-team


----- End forwarded message -----