CRITICAL ALERT

[shentino]

Sorry for falling behind with maintenance, RL has been cruel with multiple deaths in the family lately

However I discovered that the server hosting gables had run out of disk space which I found out when an attempt to perform a routine update started spamming me with "no space left" errors

A quick df confirmed the issue

I did an emergency delete of all snapshot backups from 2020 to free up some space, and I'm rerunning the maintenance script now from scratch

But be warned we may want to consider rethinking our backup strategy with automatic or periodic purges of old files and/or upgrading the VM's storage

[shentino]

@ChristopherA @noahgibbs @saraht45

[noahgibbs]

Thank you for checking this! I'll try to have a look this evening, and update this thread when I do.

[shentino]

We're lucky the dgd instance didn't crash, running out of disk space to write snapshot information would have been a fatal error.

[saraht45]

Good catch @shentino

[shentino]

We're out of space again, I'm deleting more backups

[shentino]

All snapshot backups from 2021 have been deleted

[noahgibbs]

Oh, huh, looks like something is generating huge amounts of userdb.log entries (see: /var/log/userdb*.log). Looks like AuthD is recording login attempts that are all garbage.

[noahgibbs]

And it's just been happening for under a week, since the 2nd or so.

[noahgibbs]

Yeah, something's hitting that port over and over. It's also produced many GB of driver logfile in the port 6000 DGD directory. I'm gonna delete some of that since it's just DGD logging receipt of garbage over and over.

[noahgibbs]

I'll cut out some of the huge userdb logs as well. We still have plenty with garbage, but we don't need many GB worth of it.

[shentino]

Oh, huh, looks like something is generating huge amounts of userdb.log entries (see: /var/log/userdb*.log). Looks like AuthD is recording login attempts that are all garbage.

This is a side effect of exposing that port to the wild internet when previously it was firewalled behind skotos's private network.

Skotos's original infrastructure IIRC assumed that a central server would handle all "skotos account" logins

I've seen the same spam getting generated when I was working with @shannona

The garbage is coming from the internet.

I'd like that hole closed sooner rather than later if possible, it's a potential security problem if anyone manages to brute force anything

[noahgibbs]

It looks like it's coming in via the DGD server, not directly to the AuthD (good, that shouldn't be exposed to the internet.)

[shentino]

my mistake then but yes I remember exactly where this garbage is coming from, spambots on the internet knocking on the wrong door.

As it is I already have kotaka automatically siteban these IPs for 90 days

If the DGD server is forwarding them as is that could still be a security issue if they aren't being screened.

[noahgibbs]

I'm not sure how we'd fully close the hole, since we need something available that can accept an internet login. We should probably stop logging every failed login, though.

[shentino]

I'm not sure how we'd fully close the hole, since we need something available that can accept an internet login. We should probably stop logging every failed login, though.

I think it's still valuable to log that they're happening

BUT

I think we should definitely ratelimit them, maybe only log one failure every 5 minutes.

On the first one, raise a masking flag and set a callout for 300 seconds the expiry of which will drop the flag again.

You also helped me realize this is also a potential denial of service vector.

Definitely go ahead and stop the log spam, a hostile attacker may deliberately crapflood us for the sole purpose of hogging all our disk space with junk