A Month of Bans: A Cybersecurity Review

November 7, 2024 - Reading time: 33 minutes

If you've ever maintained a public internet server, either as a hobby or as a job, you're no stranger to random probes and port-scans from evildoers looking for trouble. You've seen brute force attempts to log in to servers or email accounts, probes for vulnerable web services, SYN floods, spammers checking for open relays, and many other attacks. If your server is on the internet, it's a target. Many hobbyist and small business systems administrators use Fail2Ban to respond to these threats. Fail2Ban is an always-vigilant process that monitors system logs and responds to perceived attacks by temporarily blocking the attacker's IP address at the firewall level.

Fail2Ban has served me well, but I wanted to gain some additional insight into the origins and behaviors of the villains behind the attacks. During the month of October, I used a custom Fail2Ban script to collect geolocation and other data about each IP address that Fail2Ban identified as malicious on three hosts, including 2 web servers and a mail server. Here are the results.

More details about the Fail2Ban rules used for this article can be found at the very end.

1,698 Bans in 31 Days

Combining data from all three servers, Fail2Ban blocked a total of 1,698 attacks from 887 unique IP addresses in October, including 1,222 against a web server and 442 against the mail server. Only 34 attacks against SSH were blocked, but this low number is a little misleading -- see below for an explanation.

While most IP addresses were only banned once or twice, some were used over and over (and over), despite the ban length increasing with each successive ban. Many addresses were independently banned from multiple servers, with the "winner" coming in at 17 bans across all four servers. (Naming and shaming, this IP address was 79.110.62.40, an IP hosted by Emanual Hosting in the Netherlands; other IPs in the same /24 subnet accounted for an additional 16 bans.) 

So, all analyses in this article are based on unique IP addresses.

Geolocating the Evildoers

Geolocation data (and the maps created using the data) came from the free API and databases available from IPInfo.io. They claim very high geolocation accuracy based on public and private databases as well as technical methods like network latency triangulation.

Unique IP Addreses Banned

Bubble size represents number of unique IP addresses banned. Maps created at at https://ipinfo.io/tools/map

These maps give a more zoomed-in view by region. Click any map and shout "Enhance!" for an enlarged view.

Even if we trust the accuracy of the data, though, IP-based geolocation can only tell us approximately where the abusive machine is located, which is not necessarily the same location as the abuser himself. Most attacks I observed came from cloud-based virtual private servers or VPN providers and some may have come from compromised machines or botnets. So, geolocation based on IP addresses can be a bit of a red herring but, nonetheless, it can still provide some interesting insights.

Although Americans tend to believe that malicious activity comes only from abroad, in this investigation US-based IP addresses were the most common, followed by Singapore and Ireland. Again, this data only shows the estimated locations of the IP addresses involved in the attacks, which may be different from the attacker's actual location.

Rank	Country	IPs	Percent
1	United States of America	223	25.1%
2	Singapore	149	16.8%
3	Ireland	104	11.2%
4	Germany	75	8.5%
5	Netherlands	56	6.3%
6	United Kingdom	42	4.7%
7	China	40	4.5%
8	Japan	34	3.8%
9	Hong Kong	26	2.9%
10	Australia	25	2.8%

US-based IP addresses were among the most common origins of attacks against both the mail server and the two web servers, but after that the data begins to diverge. For example, IP addresses from Singapore and Ireland tended to attack web servers while addresses from the Netherlands and Hong Kong tended to attack the mail server.

The Dirty Dozen: Who Hosts the Villains?

This chart shows which companies own or host the most abusive IP addresses. All of these organizations except Cogent Communications and Censys, Inc. are hosting providers which offer cloud-based virtual private servers and/or VPN service.

There are several interesting takeaways form the organizational data.

• Microsoft and DigitalOcean were the top two hosts for abusive IPs, with 14.3% and 12.3%, respectively. This isn't too surprising as both hosting companies are well-known to be tolerant of bad actors on their services and deaf to abuse reports. DigitalOcean, in fact, has at times had its entire ASN listed in various DNS blackhole lists.
• Internet Utilities Europe and Asia, the bronze medal abuse host at 9.1%, is a peer and downstream of Datacamp, which shows up as #6 on the list. If we pool the two companies together, their combined attacks would tie Microsoft as #1. Taking a closer look at this company's banned IPs, it seems that attackers came mostly from a few of their customers: Logicweb,Inc.(173.239.211.0/24 and 173.239.216.0/24), IPXO.com (63.135.161.0/24), Bandito Networks (216.73.160.0/23 and 216.73.163.0/24), and panq.nl (45.8.19.0/24).
• Singapore's #2 position in the country list is almost entirely attributable to attacks originating from IP addresses owned by DigitalOcean and Datacamp.
• Likewise, Ireland's #3 position is almost entirely attributable to attacks originating from IP addresses owned by Microsoft.
• Censys, Inc. at #5 is actually a security company that port-scans public-facing servers on the internet and probes for exploits. While they are nominally not malicious, their bots do sometimes get aggressive enough with rapid-fire requests multiple times per day, to trigger a ban. Thus, their appearance on the list.

With the exception of DigitalOcean, most IP ranges had a preference for what type of server their attacks were directed at:

Types of Intrusions: What the Abusers Tried

Mail Server

Excluding the probes from Censys and similar companies (Shodan, Shadowserver, and Strechoid), there were two common forms of malicious activity:

Brute Force IMAP or POP3 Attacks

Villains attempting to brute force their way into email accounts on IMAP or POP3 was the most common and most malicious mail server attack, with 886 total attempts recorded. These bots would try to log in using common username/password combinations:

• Top 10 most commonly-tried IMAP/POP3 usernames, in order: noreply, postmaster, admin, test, sales, user, scan, info, root, backup
• Some other strange or cute username attempts included: spam, luck, abby, password, baby, 666666, 1q1a1z1x1s1w, and 1a2b3c4d
• And several attempted usernames were non-English versions of common account names: teste, prueba, soporte, contacto, contabilidad, administracion, compras, prova

Presumably, had any of these login attempts worked, they would have used the mail server to send phishing spam emails. The majority of these brute force attempts originated from hosts in Netherlands (KProHost LLC and Emanual Hosting) or Turkey (Railnet) and many of the same IP addresses were often re-used throughout the month. Interestingly, at least one of the attackers adapted to my Fail2Ban configuration and slowed down the attacks so that they could continue trying without getting banned.

Attempts to Relay Spam

Spammers attempted (and failed) to relay their spam through my mail server 57 times during the month. Some of the spammers were quite persistent, like this fellow (who I like to call "spameri") who tried valiantly almost every day:

Out: 220 [mailserver] ESMTP Postfix (Debian/GNU)
In:  ehlo WIN-4TTI4DH7SGH
Out: 250-[mailserver]
In:  Rset
Out: 250 2.0.0 Ok
In:  Mail from:<spameri@tiscali.it>
Out: 250 2.1.0 Ok
In:  RCPT to:<spameri@tiscali.it>
Out: 450 4.7.1 <WIN-4TTI4DH7SGH>: Helo command rejected: Host not found
In:  Quit
Out: 221 2.0.0 Bye

In addition, there were several bans after clients connected connected to the mail server and spoke either nonsense or a "foreign language." In some cases, for example, bots connected on port 25 and then started speaking HTTP:

warning: non-SMTP command from ec2-3-10-24-69.eu-west-2.compute.amazonaws.com[3.10.24.69]: GET /cgi-bin/login.cgi HTTP/1.1

Or had their spam scripts misconfigured:

NOQUEUE: reject: RCPT from unknown[193.143.1.40]: 503 5.5.0 <aabbb$(curl${IFS}http://1.1.1.1)@mail.domain.com>: Recipient address rejected: Improper use of SMTP command pipelining; from=<aaaa@mail.domain.com> to=<"aabbb$(curl${IFS}http://1.1.1.1)"@mail.domain.com> proto=ESMTP helo=<localhost>

Web Server

Web server bans outnumbered mail server bans almost 3-to-1. Some of the bans were attributable to malformed requests or bots looking for other services (VPN servers, DNS servers, or proxies, mostly) But the most common reason for these bans by far was bots probing for common exploitable web applications and platforms. These are usually just an annoyance, but some particularly aggressive bots can be a problem when they start sending hundreds of requests per second. (After all, they have hundreds of exploits to probe for and time is money!). Here I'll mention just a few of the most common.

Wordpress Probes

So. Many. Wordpress. Probes. I've never used Wordpress since I've always considered it unsafe, but after reviewing my web server's error logs all month, I'm seeing "wlwmanifest.xml" in my sleep. Although I didn't keep a running tally of which web apps were probed, I believe Wordpress probes account for more than half of the attacks on my web servers during the month.

Wordpress is an extremely popular web platform and it has a bonanza of plug-ins for everything from web page formatting to ecommerce to forums. Unfortunately, some of these plug-ins are not hardened against attacks. Some are exploitable and can allow hackers to take over a website or even gain root access to the server. Once the hackers are in, they can change the website to serve malvertising or redirect to a website of the hackers' choosing -- some might even silently redirect to fake online stores. Hardly a month goes by that I don't read about a new Wordpress-based exploit being used in the wild. These three alone are a sample from earlier this year:

• Almost a million WordPress websites at risk from this security flaw — here's what you need to know to keep your site safe (TechRadar, February 2024)
• Backdoor slipped into multiple WordPress plugins in ongoing supply-chain attack (Ars Technica, June 2024)
• WordPress Plugin Jetpack Patches Major Vulnerability Affecting 27 Million Sites (Hacker News, October 2024)

And sometimes the websites are compromised so stealthily that webmasters don't even notice. I know of one website for a major hospital in Southeast Asia that was compromised two years ago and, as far as I know, it still redirects to malvertising to this day.

It's little wonder that so many malicious bots are out there searching for exploitable Wordpress installations.

Github Probes

How secure is your .git directory (if you have one)? Apparently, for many, the answer is "not very secure." A recent article reported that malicious web bots have found thousands of world-readable private Github repositories on open web servers and that lists of these unsecured web directories are now for sale to villains. Many of these unintentionally open repositories contain private API tokens and other credentials.

This explains the hundreds of probes for ".git" and ".gitlab-ci.yml" on my web servers. (And, no, I do not have unsecured Github repository.)

Amazon Web Services Probes

Amazon's cloud servers are popular among enterprise users and hobbyists alike. As with Github configurations, villains like to scan for unsecured configuration files that can give them control over the servers and their data. It was revealed in August, 2024 that 110,000 Amazon-hosted domains were exploited in this way. The probes for Amazon configurations and credentials continues. During October, my web servers logged over 500 failed requests for ".aws" (usually as part of ".aws/config" or ".env" (the latter of which could refer to other services as well).

Other Probes

Two other popular attack targets with over 100 failed requests include phpunit and phpMyAdmin, both of which have had known-exploits in the past. Probes for both of these utilities originated almost exclusively from German or Asian IP addresses.

SSH

I mentioned at the top of this article that only 34 brute force attacks against SSH were blocked during the month. This seems low, but I attribute that to two reasons:

• Although SSH is running on each of the servers, it is on a non-standard five-digit port. While the SSH server can still be found through aggressive port scanning, it is more common for SSH-seeking bots to only probe the standard port 22.
• In addition to the 34 banned brute-force attacks, every few days a stray evildoer would show up (almost always from a Russian IP address hosted at Chang Way Technologies) and engage in the slowest brute force attack ever. The villain would make two attempts (or sometimes four attempts using two different IP addresses) against root, then go away and return every 90 minutes for two more attempts. These slowpoke attacks usually lasted for a few days at a time and they were far too low-effort to be noticed by Fail2Ban.

Fail2Ban Rules

I left the boring mechanics for the end of the article. For the month of this investigation, I wrote a custom Fail2Ban script to log geolocation and other data for each ban. The script also sent an email with details and logs for each ban for further review and analysis.

Sample Custom Fail2Ban Email Alert

I ran Fail2Ban on three publicly-accessible hosts, including 2 located in North America and 1 located in Asia. Two of the machines hosted web servers, one hosted a mail server (including SMTP, IMAP, and POP3), and each hosted SSH on a nonstandard port.

The Fail2Ban rules were fairly forgiving and only triggered an IP block when an intruder was particularly aggressive with brute force attacks, request floods, or errors. All bans were temporary and were lifted after a set period of time, with the length of the ban generally doubling with each successive offense from the same IP address.

SSH bans were triggered when several failed login attempts occurred from the same IP address in quick succession

Mail server bans were triggered after several failed SASL login attempts, after multiple attempts to relay spam, or after multiple "gibberish" connections, all in rapid succession. A single attempt to relay spam from a Spamhaus-blacklisted IP address also earned a temporary ban, but these bans were not included in this data.

Web server bans were triggered when a flood of requests triggered rate-limiting, when multiple requests included "disallowed" terms (probing for Wordpress, Github, PhpMyAdmin, etc.), when SSL failed multiple times, or after a series of rapid-fire bad requests.

Copyright 2024 Steve Derby for The Status Line (https://www.statusline.org/)

fail2bancybersecurity