Bots behaving badly

You and I stand on the shoulders of giants. I often forget that everything I can do today is the collective result of those who have come before me. If we were to step back and look at the hardware supply chain in a similar light, we’d see a vast network of suppliers, manufacturers, developers, and logistics companies sitting between us and our electronics.

Each link in this chain has its own organisational culture, beliefs, and goals. It never ceases to amaze me that we can turn cosmic compost into computer systems. Standing on these shoulders we can see further than ever, but the knees… well, they seem weak.

With respect to supply chains, I want to share an experience that is two parts cautionary tale and one part interesting anecdote…

The short story

With supply chains being as complex as they are, do not trust what comes out of the box – verify wherever it is – and practical – to do so.

Supply chains have ample attack surface and with the potential impacts being as real as they are, it is important to verify and revert hardware to a known good state before beginning to make use of it.

Experience has shown that just because the box looks unopened, and the thing in your hand still has that new thing smell, that alone doesn’t guarantee all is as it should be. That thing will have passed through many hands before reaching yours.

The long story

Network engineering is an art worthy of the Hugo Boss prize

Picture this…. You have a multi-month project on your hands, one which will see you convert the local network in every one of your organisation’s offices into an “internet only” style LAN. Goodbye SD-WANs, split routing, and all that other funky networking stuff. Your offices are going to be slick with just the basics – a local LAN for folks to connect their computer to and a redundant ISP connection for those computers to access the Internet.

You crack on with your plan, research hardware suppliers, listen to sales pitches, and place orders. The equipment is fairly standard stuff, but the brand of router you select – Mikrotik – is somewhat new to the NZ market. Being so new, you nor the local reseller have much experience configuring these devices, but the principles remain the same – it routes packets, supports VLANs, and includes a firewall that blocks undesirable network traffic.

The local reseller ships your equipment and you unbox your first ever Mikrotik RB4011 router. It is a highly capable device and you quickly learn the ins and outs of how to configure it.

• You delve into the network settings and configure VLANs to segregate the office LAN into smaller groups of computers, printers, and IoT devices.
• You tweak the WAN parameters to establish a connection to your chosen ISP.
• You establish a dedicated VLAN for the router’s own management interface, one that’s hidden away from the Internet and general staff. Only a handful of trusted computers will ever be able to connect to this interface.
• Finally, you adjust the firewall to allow each office computer to connect to the Internet using standard web protocols. All other traffic is blocked, including all inbound connections from the Internet.

You’re done! Your settings look good and initial testing confirms that everything performs beautifully. You export the configuration from this Mikrotik router, as this configuration will become the baseline for each office refurbishment thereafter. Before departing you run a full GRC Shields Up port scan against the router’s public IP address, and confirm that the firewall is doing its job of denying all unsolicited connections from the public Internet. Perfecto!

The project is going smoothly as your solution is elegant. You check in to see how your very first Mikrotik router deployment is going. CPU utilisation looks good, it’s barely ticking over. Bandwidth utilisation is also well within limits, but you notice an oddity – the real-time logs show that there are vastly more TCP connections on the WAN interface (facing the Internet), than on the LAN interface (the one staff would connect their computers to).

That’s odd you think, these two numbers should be roughly similar given we don’t allow inbound connections from the Internet. This piques your interest.

The penny begins to drop

You start by looking at those rules configured within the firewall, as you want to confirm a few things before getting ahead of yourself… huh, that’s odd… there is a firewall rule here that allows inbound connections from the Internet using TCP port 5045. You certainly didn’t add that rule, it doesn’t follow the structure or naming conventions that you had been following to date.

You review the configuration of the other routers that you had recently deployed and notice a similar configuration is present within them. What is going on? Did I accidentally program this rule and then copy it across in my baseline configuration? Surely I would have picked this up in the GRC port scan that I completed after commissioning each device!

You start to trawl through each menu option in the router’s management interface. Huh, the SOCKS4 proxy feature is enabled on TCP-504… and there is this odd-looking ‘scheduler rule’ programmed to run periodically.

Triage

It’s all starting to come together now – the large discrepancy in TCP connections between the LAN and WAN interfaces is a result of third-party computers connecting to your router via the Internet and using it as an anonymous proxy. That’s what the unauthorised firewall rule was for!

The logs confirm your suspicions as you see inbound connections from all over the world, each proxying their connection through your router to destinations like banking websites in Russia, Google’s SMTP Mail interface, and a handful of Facebook IP addresses.

The scheduler rules are for remote command and control of the Mikrotik router. Your router has been periodically calling “home” to a third-party server asking for its next commands. This is all starting to sound familiar… it has all the hallmark signs of the Mēris botnet.

Containment

Okay, you’re in containment mode now. Straight away you take an export of the Mikrotik’s configuration for full forensics at a later date. Your first order of the day is to disable the SOCKS proxy feature, remove the unauthorised firewall rules, and purge the scheduled task in each infected router. Later that night, when everyone has gone home for the day, you’ll factory reset each router and replace the full configuration for something that is known to be clean.

You also engage your forensic experts to understand how each router came to be this way. A thoroughly detailed investigation finds nothing conclusive - each of those computers that connected to (or could have connected to) the Mikrotik routers management interface was never infected. Equally, because you placed the management interface in a segregated VLAN, at no point in time was this particular interface ever exposed to the public Internet or to users on the local LAN. What happened then?

Root cause analysis

Your evidence is starting to paint a picture, one that says the infection was probably not a result of your actions. If this is true, then one alternative scenario starts to emerge – but it’s speculative as you have no concrete evidence.

You start to suspect that because of your, and the reseller’s lack of familiarity with the Mikrotik brand, it’s possible that one router was used for training their technical sales staff. You suspect that during this time a computer system, potentially infected by the Glupteba malware, was connected to the Mikrotik router and allowed it to become infected. This infection took the form of a scheduled task that would wait until the router itself had an active Internet connection.

This “test” Mikrotik router has then been carefully repackaged and sold to you. You then made a fair assumption that because the router is brand new, it must only contain the factory default settings. As such, when commissioning the device you’ve only looked at those menu options that you cared about. There are a large number of sub-menus that you never explored, and rightly so, what would be the point of looking through those sub-menus if you never intended to make use of the features within.

Once your Mikrotik router was installed and commissioned within the very first office, it now had an Internet connection. The malicious scheduled task then did the rest, but not before you exported the configuration for use as a baseline when setting up subsequent routers.

Some takeaways to think about

One – With supply chains being as complex as they are, don’t trust what comes out of the box. Everything should be verified (so far as reasonably practicable). For networking equipment, this could mean factory resetting the device before it’s used, or re-flashing it with the latest OEM firmware.

Two – I cannot understate the importance of monitoring. You cannot control that which you do not monitor in cyber security. Going back and periodically monitoring systems, or better yet, having other systems do that monitoring for you is an important step in understanding what’s happening in your environment, and more importantly, raising the flag when things aren’t ‘normal’.

Three – Segregate IT networks in a manner that makes sense given your business context. This limits the blast radius should things go wrong and improves observability. If they aren’t already, never expose management-type system interfaces to the public Internet (not that these were). Hide these interfaces behind a VPN if you need to remotely access them, and/or only expose them on the local LAN to a limited number of computer systems.

Whilst this incident did ultimately explain why users within those offices saw a prevalence of reCAPTCHA’s when accessing some websites, the incident had minimal impact and was more of an annoyance than anything else. But it hits home just how a little gap in one’s defences can ‘travel’ through a system to find its way into other places. Tricky stuff!