I’ve been stressing out for some hours now, but I think I know what has happened, although there are still some things that’s not quite adding up, and was hoping someone could help me get to the bottom of it. The actual question is at the bottom.
First some background I’m self-hosting Nextcloud on a Linode, and was notified that the public out network traffic exceeded my set threshold. I first assumed that I’ve had a breach on my server, but could find no trace of someone logging in. The reason I now feel at least somewhat easier is:
- No sign of anyone ssh-ing in successfully before the time this happened from /var/logs/auth.log (I guess this is not hard to cover though…)
- ssh through root is disabled - they would have to know my username and my password, which should not be brute-forceable, and the way it’s stored in my password manager does not immediately allow linking the two (although, if my password manager is compromised I don’t know what to do). I have no other signs that this has been compromised, and I think my Nextcloud-server would be a weird place to start if they had access to it all.
- I have 2FA on my Linode account, so accessing root (which also has a different and not easily brute-forceable password) through LISH should also be difficult.
- The amount of traffic (based on the average network traffic Linode reported) amounts to several times the total data stored on the server. I would expect a malicious actor to grab everything once, and not spend more time than necessary to needlessly duplicate the data.
What I now think happened instead is that my desktop client has resynced everything several times over. The reasons I think this:
- The network activity started more or less when I opened my laptop this morning
- The desktop client was for some reason entered twice in the autostart, causing two version of the client to be started at the same time. This caused some conflicts today - when I noticed this and resolved these, I quit the second instance, and that is about the time the network activity stopped
- The same thing happened later today, which caused a spike in CPU-usage on the server, but did not trigger the same network traffic as the desktop client seems to have crashed quickly after.
The actual question However, the last piece of the puzzle that I can’t figure out that still has me somewhat nervous: the maximum outbound transfer speed greatly exceeds my download speed (about 4 times). From the graph, it seems as though it maintains this high speed, but it seems to maybe just log the maximum value every five minutes, so maybe these are just spikes? The reported average over the two hours this occurred more or less matches my maximum download speed however, although I don’t really think I can get that from where I am sitting on my WiFi.
Is this the glove that doesn’t fit?
It seems your assessment is correct. You’d be surprised at the speeds you can get on poor wifi when you don’t care about latency. The average speed marching up with your download is a dead giveaway too. The fact that maximum over 5 minutes exceeds it is a bit weird, but it could be explained by some networking equipment in the middle (probably at your ISP if I was to guess) terminating MTUs for whatever reason. A common one is misconfiguring various solutions for capping internet speeds to subscribers, where your local MTU will be set correctly but the outgoing ones will be set to the maximum speed of the link.
Oh boy, I got my units mixed up. I am used to reading bits per seconds as bps and bytes per seconds as B/s. However, the network activity on Linode is given in Mb/s. Now the numbers make a lot more sense, and the transfer speeds are well within the limits of my network and what I am used to seeing on my laptop on WiFi.
Thanks, I didn’t consider something like that. Would have wanted to see some more detailed graphs from Linode to see how long these max speeds were sustained, but I can’t seem to find it.