During a recent response call, I had the perfect storm of failure. It started on Friday afternoon when we got a notice about a potential compromise on a low value system. Normally, on these systems we just hand the information off to the system admin, but we wanted to verify some information. I was armed with my trusted IR tools. Once on the system, the first thing I noticed was that the server was three major OS versions behind and running a non-X86 platform. Hmmm this is going to be interesting….
I ran my live IR script and had a massive failure on every command. I switched to my trusty version of F-response to collect MAC times on the system. F-response also failed to run on the server. Fortunately the system had GCC on it and I pulled down the source-code for mac-robber. I did not have access to another server with this OS version and architecture combination, so I had to compile the binary on server.Make failed! Thankfully, in the install.txt, it references if make does fail try make -simple. That worked, the first good news and now I was able to collect file times on the server.
Making changes to systems you are responding to is not a deal breaker, but you need to make sure that you document everything that you do and include the reason why.The rest of the volatile information was gathered by running the built-in binaries on the server. While using this method has its issues, it’s best to collect volatile data any way you can before the system is taken offline.
Lessons Learned
- No matter how well tested your documentation is, be ready for hiccups.
- This procedure had been tested many times on incidents, but never had all our contingencies failed when trying to collect data.
- It’s critical in these instances that you have experienced team members that understand the systems they are responding to.
- Compiling source code on a server you are responding is never a great option, but you should include source-code of critical tools to your response toolkit in case you need it.
