When Procedures Fail

Posted on Updated on

During a recent response call, I had the perfect storm of failure. It started on Friday afternoon when we got a notice about a potential compromise on a low value system. Normally, on these systems we just hand the information off to the system admin, but we wanted to verify some information. I was armed with my trusted IR tools. Once on the system, the first thing I noticed was that the server was three major OS versions behind and running a non-X86 platform. Hmmm this is going to be interesting….

I ran my live IR script and had a massive failure on every command. I switched to my trusty version of F-response to collect MAC times on the system. F-response also failed to run on the server. Fortunately the system had GCC on it and I pulled down the source-code for mac-robber. I did not have access to another server with this OS version and architecture combination, so I had to compile the binary on server.Make failed! Thankfully, in the install.txt, it references if make does fail try make -simple.  That worked, the first good news and now I was able to collect file times on the server.

Making changes to systems you are responding to is not a deal breaker, but you need to make sure that you document everything that you do and include the reason why.The rest of the volatile information was gathered by running the built-in binaries on the server. While using this method has its issues, it’s best to collect volatile data any way you can before the system is taken offline.

Lessons Learned

  • No matter how well tested your documentation is, be ready for hiccups.
    • This procedure had been tested many times on incidents, but never had all our contingencies failed when trying to collect data.
  • It’s critical in these instances that you have experienced team members that understand the systems they are responding to.
  • Compiling source code on a server you are responding is never a great option, but you should include source-code of critical tools to your response toolkit in case you need it.

IR Procedure Documentation

Posted on

Creating documentation is not something most IT people like to do. But when building an IR team, you must have  well tested procedures that should become living documents. These documents should be designed in a way that they can be modular and cohesive. This post will discuss the basic layout of these procedures and then breakdown the key parts that should be included in each.

You should have one overall procedure that changes less often, but points to the proper areas for the appropriate information. The major categories of documentation should be based on OS, Applications and/or infrastructure.

The overall procedure

When developing your process, you need to talk with management and your legal department to determine what is the purpose of providing incident response. During these talks it should be decided what questions are you trying to answer during your investigation.

Common questions:

  • How did the attacker gain access to the system?
  • Did they access sensitive/protected data?
    • If you can not determine what was accessed, what do you do?
  • Were other systems accessed?

Once the focus of your process has been determined, you should start building your overall incident documentation. A great, but lengthy, guide is the NIST 800-61 which covers a lot of the information discussed below.

  • Contact information.
    • For all members of the IR team: executives, data trustees, legal council, human resources, finance, law enforcement and other CERTs.
  • Who has authority to make decisions for certain groups.
  • Roles and responsibilities for each group at each stage of the process. (Preparation, Identification, Containment, Eradication, Recovery and Lesson learned).
    • Infosec
    • Helpdesk
    • System Admins
    • Executive
    • Legal
    • Public Relations
  • Checklists of your overall process (Should include major milestones)
    • (Include Date/time and Person who completed task)
      • Entered into tracking database
      • Collected image
      • Determine priority of incident
      • Preformed forensics
  • Communication plan
    • Have a daily notification template for major incident updates for management.
      • New developments
      • What’s planned for tomorrow and who it is assigned to
      • History of incident time-line
    • Addition e-mail templates
      • Internal incident notification for system admins
      • Notification to 3rd parties
  • Setting criticality for an incident
  • Escalation process to get law enforcement evolved

OS Specific Procedures

You should have a procedure for each critical OS that is in your environment. Linux and Windows are the most common, but others people forget are:OS X, Solaris, AIX, and smart phones.

Application Specific Procedures (SQL, Web Server)

Infrastructure Analysis

  • List of available sources
    • How long they retain information
  • What data to collect?
    • IDS/IPS
    • Network Flow
    • DNS Queries
    • Authentication Infrastructure Logs (AD, NDS, LDAP,RADIUS)
    • VPN Logs
  • Initial analysis to verify incident occurred

Wrapping Up

As you begin to develop these processes, its best to do several test incidents with the documents before you use them during a real incident. Always have your least experienced handler try out the document first because many times you will omit things that are not clear to junior members of your team.

Its also a good idea to document contingencies if you were not able to complete critical processes. A great example is collecting evidence. If your normal process is not working, you need to have at least one additional way to be able to collect evidence. Your backup plan may be to use cryptcat to shoot the image across the network or attach a USB drive to the compromised system.

Scheduling a yearly team incident exercise or at least a yearly document review of your procedures. While some procedures maybe changed on a monthly basis others will not need modification during that year, but it forces you to look over them. If new  response techniques have been developed  during the year, this review is a good time to document them and add to your procedure.