Linux IR

Linux Memory Dump with Rekall

Posted on

Over at I’m handler of the day! My latest post discusses dumping Linux memory. Please stop by and read the post.


Stormcenter Post

Posted on Updated on

My first post as a Handler is up. It talks about litecoin mining and backdoors.

Using DVWA to Test Hardening Apache Techniques

Posted on

The great thing about Linux is just how flexible it is. It always seems like there are at least 3 ways to accomplish anything on the system. I wanted to figure out what was the easiest to manager and most effective way to segment the Apache service.

Vulnerable System Setup

The test systems were Ubuntu 10.10 server and CentOS 5.5 that was installed using Vmware scripted install.  I had to allow file includes in the php.ini, which was disabled by default. The vulnerability I used to test the against was the remote file include on DVWA . The web proxy and dns proxy PHP pages were uploaded to the server simulating a file upload vulnerability.

Test Methodology

A baseline test for the default install was done for both Cent OS5.5 and Ubuntu 10.10. Then I began testing other hardening methods for the Ubuntu server: chroot, SElinux, and AppArmor. The tasks that were tested are common techniques attackers use: executing commands, creating files, reading critical system files, use the system as a proxy to get deeper into the network.

Ubuntu 10.10 Default Install

Using PHPshell and Laudnam to interact with the system I was able to complete all tasks. While I did have to weaken the server configuration by allowing remote file includes,  no other controls in place prevented exploitation of the system. No alerts were generate about the exploit and the only evidence of the exploit was the apache access.log.

Ubuntu Test Results with AppArmor Enabled

Setting up AppArmor

First you’ll need to create a profile for Apache.

Once your profile is created you will need to place it into enforcement mode

#aa-enforce /etc/apparmor.d/apache2.d/*


#/etc/init.d/apache2 restart

Check and make sure that the service is being protected


You should see something like:

5 processes are in enforce mode : 
/usr/sbin/apache2 (23208)

AppArmor was able to stop all the test except for web proxy and DNS lookup files. Based on this analysis, AppArmor would make it very difficult to gain a significant foot hold on the server.  The attacker could read files to the web directory of the server, but little else. Even though the PHP shell is using the native calls to access files, AppArmor still prevents this. A good example is that access to /bin/cat is being blocked, but if you use a PHP language to read a file within www directories it works. Any read access outside of www is blocked by the profile.



Running command ls (failed)

Feb 26 22:44:55 owaspbwa kernel: [ 1854.653947] type=1503 audit(1298778295.140:1501):  operation="exec" pid=3166 parent=3165 profile="/usr/lib/apache2/mpm-prefork/apache2" requested_mask="::x" denied_mask="::x" fsuid=33 ouid=0 name="/bin/ls"

Web proxy (Successful, but created error due to AppArmor not allowing ipv6)

Feb 27 20:07:41 owaspbwa kernel: [ 2741.598238] type=1503 audit(1298855261.286:1516):  operation="socket_create" pid=3136 parent=3128 profile="/usr/lib/apache2/mpm-prefork/apache2" family="inet6" sock_type="dgram" protocol=0


Ubuntu 10.10 with Apache chroot

If you  are not familiar with chroot, check out the wikipedia article,  it does a good job of breaking down the basic concept. In Apache versions 2.2.10 and later the chRootDir configuration is available and make setup much easier.

Setting up Chroot

Make the directory that will be considered the “Root” directory for the jail. All other folders will need to be created under this directory to mimic a Linux file system.

#mkdir /var/www-jail

Apache Setup

copy your web files into /var/www-jail/var/www

#cp /var/www /var/www-jail/var/www

Add the following to  /etc/apache2/Apache2.conf

chRootDir /var/www-jail

Mysql setup

Add the following to /etc/mysql.ini

socket = /var/www-jail/var/run/mysqld/mysqld.sock
#mkdir /var/www-jail/var/run/mysqld/
#chown mysql /var/www-jail/var/run/mysqld
Setup Proc file system
#mkdir /var/www-jail/proc
#mount -o bind /proc /var/www-jail/proc

If you want to make this perminate you will need to auto mount the chroot proc dir.

#vi /etc/fstab
proc-chroot     /var/www-jail/proc        proc    defaults        0       0

Restart the service

#/etc/ini.d/apache2 restart



Strace was ran on the Apache process to determine what CHroot was doing.

strace for cat /etc/passwd

read(4, 0x7ffffa62bb2f, 1) = -1 EAGAIN (Resource temporarily unavailable)


Ubuntu 10.10 with SElinux


To setup SElinux, you must first remove AppArmor as they are not compatible.

#apt-get remove apparmor
#apt-get install selinux
#vi /etc/selinux/config

You will need to reboot a couple of times while Ubuntu makes the required changes to the system.  I used the default SElinux policy called Ubuntu while testing.  This policy seems very open and this is likely due to interoperability. I started to test the more restrictive MLS policy, but Ubuntu would not boot with it installed. I hope to at some point get this working properly.



Same format as CentOS SELinux


CentOS5.5 Default Install


By default CentOS 5.5 has SElinux installed by default with a fairly restrictive policy.  As with Ubuntu I had to enable remove file include in the php.ini to get DVWA to work properly. To Setup DVWA on CentOS follow this guide.


Running command (ls)

type=AVC msg=audit(1300816647.563:797): avc:  denied  { getattr } for   pid=31355 comm="sh" path="/bin/ls" dev=dm-0 ino=3391307  scontext=root:system_r:httpd_t:s0  tcontext=system_u:object_r:ls_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1300816647.563:797): arch=c000003e syscall=4  success=no exit=-13 a0=d36adc0 a1=7fffab9eb1e0 a2=7fffab9eb1e0 a3=0  items=0 ppid=10993 pid=31355 auid=0 uid=48 gid=48 euid=48 suid=48  fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=3 comm="sh"  exe="/bin/bash" subj=root:system_r:httpd_t:s0 key=(null)

Spawn Remote shell attempt

type=AVC msg=audit(1300821679.252:907): avc:  denied  { name_connect }  for  pid=756 comm="httpd" dest=8888 scontext=root:system_r:httpd_t:s0  tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1300821679.252:907): arch=c000003e syscall=42  success=no exit=-13 a0=13 a1=2b7d393c8c38 a2=10 a3=0 items=0 ppid=751  pid=756 auid=0 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48  fsgid=48 tty=(none) ses=126 comm="httpd" exe="/usr/sbin/httpd"  subj=root:system_r:httpd_t:s0 key=(null)


  • CentOS has the best out of the box configuration to prevent these attacks.
  • AppArmor and chroot showed similar resistance in fending off the attacks,but  AppArmor creates logs for failed system access.
  • AppArmor is a little easier to setup for less experienced administrators as Ubuntu has profiles for some of it most popular applications.

What it doesn’t stop

While many of the attacks today are SQL injections and remote file includes, Apache isolation would not prevent these attacks. It could help keep the attacker contained, but  you should still harden the system using a standard like CIS  in addition to what is recommended here. If you are watching  your logs, then it may allow you to detect an attacker quickly and prevent them from having time to gain root access.

Apache logs Timeline Analysis

Posted on Updated on

I love Log2timeline and have used it ever since the SANS forensic blog Super Timeline talked about it. While this tool currently focuses on mostly windows forensics, we can use its technique and the methodology Rob Lee talks about  combining multiple body files for analysis.

Case Background

Lets say you have a Linux server running Apache and it gets compromised using  a vulnerability that allows an attacker to post content to the server.  The attacker then places  a PHP shell on the system. Most of these shells allow attackers to upload and edit files, run commands on the local system, and interact with the database.  This is a fairly common scenario and  I wanted to see if I could determine what commands the attacker ran by examining the Apache access logs and matching it up with files that were accessed on the system at the same time.


I created a file system MAC times using fls.

# fls -r -m / /dev/sda1

I grepped through the access.log and  pulled out the lines that contained POST to the PHP shell.

#grep POST access.log >/tmp/post-access.log

I wanted to combine the Apache  logs with file system times. To do this I needed to convert the Apache logs to the sluthkit body format. I wrote a quick bash script to get the file in the correct format. I have rewritten this into a Log2timeline plugin and submitted it to the project (Currently its in the Dev tree and testing is being done on it) .

Then I used my script to convert it into the correct format.

#log2timeline -f apache /var/log/apache/access.log > /tmp/post-access.log

Then combine the two body files.

#cat /tmp/post-access.log /tmp/mactimes.csv >combined-body.txt

Then run mactime to convert the file into a CSV file.

#mactime -d -z EST5EDT /tmp/combined-body.txt  >output.csv


Below is an example of the  output I ended up with.    In this instance the attacker ran the /usr/bin/locate command from the shell and accessed /etc/passwd.  If this system is dedicated web server with out many users accounts, you should be able to get a good mapping of what went on.

Mon May 01 2010 08:10:07,0,macb,0,0,0,0,[apache log]  "POST /web/files/shell.php HTTP/1.1" 200 13600

Mon May 01 2010 08:10:07,0,.a..,r/rr-xr-xr-x,0,0,129187,/usr/bin/locate
Mon May 01 2010 08:10:12,0,macb,0,0,0,0,[apache log]  "POST /web/files/shell.php HTTP/1.1" 200 13600

Mon May 01 2010 08:10:12,0,.a..,r/rr-xr-xr-x,0,0,129187,/etc/passwd