Security or Operational Snafu – What Went Down at the NY Times?

Midday yesterday the security news coverage focused on the NY Times website outage that lasted roughly 2 hours. Initially, there were reports of a cyber-attack from China, more specifically a DDoS attack. The NY Times said it was an internal issue. Speculation was running wild and you can read an entertaining rundown of the coverage on Richi Jennings’ IT Blog watch. So, we’ll have to wait and see to learn more information, but from the reports it seems like it was either a DDoS attack or potentially a change management issue. Here are some quick tips to address both of these challenges:

DDoS

Our guest blogger, Matthew Pascucci wrote a three-part series on DDoS that examines attacker’s DDoS tools and methods, offers suggestions to improve your DDoS defense, and a DDoS Do’s and Don’ts list. Here’s a cliff notes version of the blog series that includes some things to consider:

• Review your network environment to see if you’re set up well to defend against a DDoS attack. Questions such as “Do you have an IPS with DDoS signatures enabled?” “Is your router/firewall configured with rate limiting?”, “Should you consider blocking certain countries on your edge?”, etc.
• Look at external capabilities to further improve your defense:
  • See what capabilities your ISP can offer and determine if they can extend your DDoS defense beyond your internal capabilities.
  • Check out scrubbing facilities which allow you to route traffic over to them either by DNS redirects or BGP changes.
• Establish a process for responding to a DDoS
  • This process should include formal instructions for each department that could be involved.
  • Have regular reviews of your DDoS response plan, based on a specific incident at your company or based on current events, and look for ways to optimize and improve it. Run “Red Team” drills that include your DDoS incident management team.
• DO NOT ASSUME that the DDoS is ALL that is happening in your network!

Change Management

Poor change management can have horrible consequences. In our State of Network Security 2013 Survey, we examined this issue and found that:

• More than 75% of organizations suffered a network or application outage due to an out-of-process change
• More than 80% of organizations suffered an outage, security breach or decreased network performance due to an application-related firewall rule change

We don’t know if the NY Times “internal issue” was a network configuration error, possibly brought on by a poorly managed change or something else, but if it’s not a cyber-attack and things were humming along before, something had to change – and that’s typically where things break down. Improving change management processes and aligning the key stakeholders across security, operations and the business can provide immediate dividends in terms of improved uptime and also enabling the organization to be more adept when making changes based on dynamic business needs.

It will be interesting to see if more information comes out regarding this NY Times outage and see if there are any other lessons learned.