Anything exposed to the internet will get probed by malicious traffic looking for vulnerabilities. Best thing you can do is to lock down your server.

Here’s what I usually do:

• Install and configure fail2ban
• Configure SSH to only allow SSH keys
• Configure a firewall to only allow access to public services, if a service only needs to be accessible by you then whitelist your own IP. Alternatively install a VPN

A pentester here. Those bad looking requests are mostly random fuzzing by bots and sometimes from benign vulnerability scanners like Censys. If you keep your applications up date and credentials strong, there shouldn’t be much to worry about. Of course, you should review the risks and possible vulns of every web application and other services well before putting them up in the public. Search for general server hardening tips online if you’re unsure about your configuration hygiene.

An another question is, do you need to expose your services to the public? If they are purely private or for a small group of people, I’d recommend putting them behind a VPN. Wireguard is probably the easiest one to set up and so transparent you wouldn’t likely even notice it’s there while using it.

But if you really want to get rid of just those annoying requests, there’s really good tips already posted here.

Edit. Typos

Fail2ban and Nginx Proxy Manager. Here’s a tutorial on getting started with Fail2ban:

https://github.com/yes-youcan/bitwarden-fail2ban-libressl

I mean, it’s not a big deal to have crawlers and bots poking at our webserver if all you do is serving static pages (which is common for a blog).

Now if you run code on server side (eg using PHP or python), you’ll want to retrieve multiple known lists of bad actors to block them by default, and setup fail2ban to block those that went through. The most important thing however is to keep your server up to date at all times.

Nothing too fancy other than following the recommended security practices. And to be aware of and regularly monitor the potential security holes of the servers/services I have open.

Even though semi-related, and commonly frowned upon by admins, I have unattended upgrades on my servers and my most of my services are auto-updated. If an update breaks a service, I guess its an opportunity to earn some more stripes.

I only expose services on IPv6, for now that seems to work pretty well - very few scanners (I encounter only 1 or 2 per week, and they seem to connect to port 80/443 only).

I’ve been using crowdsec with swag for quite some time. I set it up with a discord notifier. It’s very interesting to see the types of exploits that are probed and from each country. Crowdsec blocks just like fail2ban and seems to do so in a more elegant fashion.

I map them every day.

• Turn off password login for SSH and only allow SSH keys
• Cloudflare tunnel
• Configure nginx to resolve the real IPs since it will now show a bunch of Cloudflare IPs. See discussion.
• Use Fail2ban or Crowdsec for additional security for anything that gets past Cloudflare and also monitor SSH logs.
• Only incoming port that needs to be open now is SSH. If your provider has a web UI console for your VPS you can also close the SSH port, but that’s a bit overkill.

sometimes I grab popcorn and “tail -f /var/log/secure”

I use fail2ban and add detection (for example I noticed that after I implemented it for ssh, they started using SMTP for brute force, so had to add that one as well.

I also have another rule that observes fail2ban log and adds repeated offenders to a long term black list.

Ignore them, as long as your firewall is set up properly.

Any service I have that is public facing is proxied through Cloudflare. I run a firewall on the host that only allows traffic from Cloudflare IPs. Those IPs are updated via a cron job that calls this script: https://github.com/Paul-Reed/cloudflare-ufw I also have a rule set up in Cloudflare that blocks traffic from other countries.

For WAF, I use modsecurity with nginx. It can be a little time consuming to set up and weed out false positives, but it works really well when you get it configured properly.

Some of my applications are set up with Cloudflare Access. I use this with Azure AD free tier and SAML, but it could be set up with self hosted solutions like authentik.

I use Caddy as a reverse proxy, but most of this should carry over to nginx. I used to use basic_auth at the proxy level, which worked fine(-ish) though it broke Kavita (because websockets don’t work with basic auth, go figure). I’ve since migrated to putting everything behind forward_auth/Authelia which is even more secure in some ways (2FA!) and even more painless, especially on my phone/tablet.

Sadly reverse proxy authentication doesn’t work with most apps (though it works with PWAs, even if they’re awkward about it sometimes), so I have an exception that allows Jellyfin through if it’s on a VPN/local network (I don’t have it installed on my phone anyway):

@notapp {
  not {
    header User-Agent *Jellyfin*
    remote_ip 192.160.0.0/24 192.168.1.0/24
  }
}
forward_auth @notapp authelia:9091 {
  uri /api/verify?rd=https://authelia.example
}

It’s nice being able to access everything from everywhere without needing to deal with VPNs on Android^ and not having to worry too much about security patching everything timely (just have to worry about Caddy + Authelia basically). Single sign on for those apps that support it is also a really nice touch.

^You can’t run multiple VPN tunnels at once without jailbreaking/rooting Android

These requests are probably made by search/indexing bots. My personal server gets a quite a lot of these, but they rarely use any bandwidth.
The easiest choice (probably disliked by more savvy users) is to just enable cloudflare on your server. It won’t block the requests, but will stop anything malicious.
With how advanced modern scraping techniques are there is so much you can do. I am not an expert, so take what I say with a grain of salt.