Network Security Horror Stories: Change Control

When you’re in IT Security for as long as I have been you’ll most likely have quite a few horror stories regarding firewall change management and some shockingly dumb moves. Here are three isolated issues that I’ve seen in my more than 10 years in the business (and am allowed to discuss externally): 

Server Hosting Foreign Movies

Working one night in an operation center I noticed a large spike in traffic to a particular IP address. After drilling down into the traffic pattern I noticed that it was FTP related. Knowing that this particular server wasn’t hosting an FTP site I became very suspicious. Shortly afterward during troubleshooting I noticed an even larger spike in traffic, this time egressing our network. At this point I became very suspicious as to what was going on and discovered that FTP was anonymously enabled on this server and was now hosting that year’s blockbuster movie.  Turns out that our client had enabled FTP for this server to receive a download and left the permissions wide open on their network. The service was stopped and the server needed to be cleaned after being compromised.

This particular issue could have been eliminated if proper change management would have been followed. While change management won’t catch everything having the approval of other engineers and management can help eliminate simple oversights like opening anonymous FTP to the world.

Fighting for the Mouse

Once while reviewing why a mass mailing kept failing for a client I noticed that the mouse was moving on the console of a server. I grabbed the mouse and attempted to gain control from whoever was poking around on the server. After struggling for a few seconds for the mouse the attacker opened notepad and we started chatting to each other through it. The attacker was using VNC and was heavily rooted in this server. We removed the Ethernet connection and notified the client of the issue. There happened to be many IIS vulnerabilities that the attacker was able to exploit and gain unauthorized access.

This again is a change management issue, but allowing VNC on your network is even more of a risk. Having the ability to know what remote protocols are in your in your network and eliminating them if they’re not secure is a standard. When a change control ticket comes in asking to open the firewall outbound for VNC it should be denied.

High Availability Goes Wrong

Many years ago I worked with an organization that had routers in a high-availability cluster for redundancy. During one of our busy times of the year a consultant made a change to the cluster regarding a project he was working on which brought the company down during a very busy time of the year. There were no change control tickets or review process put in and the change cost the company a great deal of revenue.

Change control, change control, change control. If there were a ticket put in for this incident it would have been caught, but even if that wasn’t there having the ability to monitor changes in your network is a way to catch rogue changes like these.

There are many other instances that are much more detailed and dangerous, but due to NDAs I’m not able to speak about them. The above network security horror stories are just the tip of the iceberg of what I’ve come across in my years as an information security professional. Attacks that were allowed to happen due to simple change management process gaps not being in place that cost each company and client I was working for a great deal of time, resources and money to remediate. One of the companies lost more than could be estimated with an outage and most likely lost brand recognition due to simple changes not approved.

Since these all were related to poor change management it’s important to get a change management program setup in your company, but even with such a program implemented it doesn’t mean people will adhere to it. Being able to monitor and audit changes is the only way to have a full life cycle of coverage when it comes to changes that could potentially leave your network vulnerable. What are some of the network security horror stories you can share? And what are the lessons learned?