Secure the system

First make sure to secure the backup of the system you will investigate and isolate the system from the network to prevent further compromise or tampering.

You can trust no one anymore

As the system we are investigating might have been compromised, we cannot trust any of it binary nor script files. So to make sure we are using only good know binary we are going to mount a readonly media (like a linux distribution burned on a CD)

Next, we are going to modify the PATH and LD_LIBRARY_PATH system variable to point to our CD

user@compromised:~$ export PATH=/mnt/cdrom/bin:/mnt/cdrom/sbin
user@compromised:~$ export LD_LIBRARY_PATH=/mnt/cdrom/lib:/mnt/cdrom/lib64

Verify it have been done properly using which command:

which ps
/mnt/cdrom/bin/ps

Next come the harder part, find some clues in the filesystem which contain several thousands files.
There are many ways to do this, I'll just try to give you some here that hopefully, will get you started.

Check the logs

First thing first, look at your applications and system logs.
Before even openning them, one possible indicator is the size of the log file.

Ex: If your application log file is usually between 1 or 2mo a day and last wednesday it jumpt to 5mo, there might be something there.

You can also look at the number of time a certain word occure in you logs like:

• error
• deny
• rejected
• unauthorized access
• failed login or the other way around
• Etc, you get the idea

Here again, a jump into the number of occurance could indicate something (Ex: An attempt to exploit a vulnerability or someone scanning your system)

zgrep -i error /var/log/nginx/* -c
.
mysite_access.log:0
mysite_access.log.1:1
mysite_access.log.2.gz:0
mysite_access.log.3.gz:1495
mysite_access.log.4.gz:0
mysite_error.log:0
mysite_error.log.1:0
mysite_error.log.2.gz:14
mysite_error.log.3.gz:2
mysite_error.log.4.gz:2

In the above example, we can see a jump in the file mysite_access.log.3.gz with 1495 occurance of the word error, while we have none the other days. It could be a good idea to look at that file to see what happen.

Search for bad files

Next you can start by listing all files that have been created in the last x hours or days, and check if there is any executable into them. Look at there path too, it might give you an hint if the file is legit or not. Search for any file own by root, with the sticky bit set and writable by anyone and check the last time they have been modifyed.

By the way, if you have any of the last type I've just mention, this is very bad (legit or not), as it is a door wide open to priviledge escalation.

# find /home/user -cmin -60
# find /home/user -ctime -1 -type f -executable -print
# find / -type f -user root -perm /4000 -perm -o+w -exec ls -l {} \; 2>/dev/null
# find /etc/ -type f -mtime -2 -exec ls -l {} \;

If you find any suspicious file, genarate the sha256 and check it against the virustotal database.

sha256 /path/file
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /path/file

If the system you are investigating is a web server
If you find any suspicious file, writedown the time the file was created and check the system logs arround that time.

You can also use debsums to check if the actual MD5 sum of file on the system differ from the MD5 sums of installed Debian packages:

$ sudo debsums -e -s
debsums: changed file /etc/sudoers (from sudo package)

Check get's in and when

Use the last command to see who logged in and when

$ last
user     pts/1         10.13.46.43               Mon Mar 18 15:44    gone - no logout
reboot   system boot  6.5.0-25-generic Mon Mar 18 15:43   still running
user     pts/1         10.13.46.43               Mon Mar 18 08:42 - 15:33  (06:50)
tony     pts/1         10.13.126.13              Sun Mar 17 00:24 - 22:30  (10:29)

Use lastb command to see failed login, alternatively you can check the file /var/log/secure to see more details.

Use the w command to display who is logged on and what they are doing.

w
10:06:49 up 12 min,  2 users,  load average: 0.96, 0.65, 0.32
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
user pts/0    10.100.2.80      09:55    9:03   0.02s  0.02s -bash
tony pts/1    10.8.73.153      09:57    1.00s  0.04s  0.00s -bash

Check if there was any user created recently that you don't know about and have a loow at the sudoer file as well:

# cat /etc/passwd
# cat /etc/sudoer

In the output from the last command we can see the user tony have logged in at an unusual time (sunday at 00:24am).
In such a case it can be a good idea to check tony's .bash_history file.

# cat /home/tony/.bash_history

You can also have a look at the sensitive file /home/tony/.ssh/authorized_keys check if it has been modified and make sure that all public key inside it belong to know person:

stat /home/tony/.ssh/authorized_keys
cat /home/tony/.ssh/authorized_keys

Check for rootkit

I'll propose you 2 rootkits here, but there are more:

• chkrootkit

• rkhunter

  # chkrootkit
  # rkhunter -c -sk