When Procedures Fail

Posted on Updated on

During a recent response call, I had the perfect storm of failure. It started on Friday afternoon when we got a notice about a potential compromise on a low value system. Normally, on these systems we just hand the information off to the system admin, but we wanted to verify some information. I was armed with my trusted IR tools. Once on the system, the first thing I noticed was that the server was three major OS versions behind and running a non-X86 platform. Hmmm this is going to be interesting….

I ran my live IR script and had a massive failure on every command. I switched to my trusty version of F-response to collect MAC times on the system. F-response also failed to run on the server. Fortunately the system had GCC on it and I pulled down the source-code for mac-robber. I did not have access to another server with this OS version and architecture combination, so I had to compile the binary on server.Make failed! Thankfully, in the install.txt, it references if make does fail try make -simple.  That worked, the first good news and now I was able to collect file times on the server.

Making changes to systems you are responding to is not a deal breaker, but you need to make sure that you document everything that you do and include the reason why.The rest of the volatile information was gathered by running the built-in binaries on the server. While using this method has its issues, it’s best to collect volatile data any way you can before the system is taken offline.

Lessons Learned

  • No matter how well tested your documentation is, be ready for hiccups.
    • This procedure had been tested many times on incidents, but never had all our contingencies failed when trying to collect data.
  • It’s critical in these instances that you have experienced team members that understand the systems they are responding to.
  • Compiling source code on a server you are responding is never a great option, but you should include source-code of critical tools to your response toolkit in case you need it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s