Linux incident response quickly

[Asphyxia]

Firstly, what commands were run?

mkdir /evidence; cd /evidence
mkdir bash_history
find / -name .bash_history | xargs grep "" > bash_history/log.txt
mkdir ip_logs
utmpdump /var/log/wtmp > ip_logs/wtmp.txt
utmpdump /var/log/btmp > ip_logs/btmp.txt
utmpdump /var/run/utmp > ip_logs/utmp.txt
grep "su\|wget\|\.com\|\.ru\|\.de\|\.country\|\.stream\|\.download\|\.xin\|\.gdn\|\.racing\|\.jetzt\|\.win" bash_history/log.txt
#You can change what to grep for, maybe unzip, gcc, .py, python, php, etc
#Anything from scp (moving files to another server), to wget (downloading files) could be worth looking at
#An attacker may also use nc - this project is not to do everything for you, just to get the logs you need quick and simple on Linux
mkdir misc; cd misc
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "root" > all_root
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "change" > changes #Shoutout to xxxtentacion RIP sir, your music is awesome; fly high
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "delete\|remove" > deleted
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "listen" > listen #maybe they changed SSHD port or something dumb
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "add\[\| added \| new \|New" > added
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "for user root" > for_user_root
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "for root" > for_root
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "password changed" > password_changed
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "Invalid user\|invalid user" > invalid_user
cat /var/log/secure* /var/log/auth.log* 2>/dev/null | grep "Accepted password" > accepted_password
systemctl | grep "server\|Server" > servers #Once you find running servers btw, go check your /var/log against those !!!PROTIP!!! example httpd.service, go /var/log/httpd/ and then cat * | grep "virus.exe" lol

#Maybe a hacker decides to dump your database and accidentally leaves the file there, go find that
find / -type f -size +10M 2>/dev/null > large_files

#Maybe the hacker ran "touch lol" and forgot to clear that file out of your /var/www/html?
find / -type f -size -1c 2>/dev/null | grep -v "usr/\|sys/\|proc/\|run/\|spool/\|ossec/\|lib/" > small_files
#cat small_files | grep "/var/www/"

Curious, what all do you do if you get hacked?!

I know there are more logs to grab, want to know what you think is important? We can jointly come up with somewhat of a Linux INCIDENT RESPONSE log grabber how-to or whatever?!

 

[Asphyxia]

In this second posting, we are going to take a further dive into application logs/auditing.

Building atop the other script:

mkdir /evidence; cd /evidence
mkdir apps; cd apps
ps -aux | grep -v "root " > non_root
ps -aux | grep "root " > root

#You may want to crawl through server logs available, to check for IP addresses/counts.
systemctl | grep "server\|Server" | cut -d. -f1 > servers_installed

find /etc/apache2/ -name "*.conf" 2>/dev/null | xargs grep "Log" > web_apache_log_info
find /etc/httpd/ -name "*.conf" 2>/dev/null | xargs grep "Log" > web_httpd_log_info
#Want to speed up log checking? cat web* | grep "\"\|CustomLog\|\.log\|TransferLog"

Here is an example of what we want to see in the web server configuration:

/etc/httpd/conf/httpd.conf:    CustomLog "logs/access_log" combined
/etc/httpd/conf.d/ssl.conf:TransferLog logs/ssl_access_log
/etc/httpd/conf.d/ssl.conf:CustomLog logs/ssl_request_log \

if

grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" /var/log/httpd/*ccess_log > httpd_ips
sort httpd_ips | uniq -cd | sort -nr
...................

Now we know to check /var/log/httpd/ssl_access_log for httpd, while elsewhere we may be looking for access.log, for example.

I am working further on this, but for now I have a way to check whether httpd/apache. I also plan on detecting NGINX, perhaps automation of log aggregation to make incident response more automated.

systemctl | grep "server\|Server" | cut -d. -f1 > servers_installed

if [ `grep httpd servers_installed` == 'httpd' ]
then
echo "You are running httpd"

elif [ `grep apache servers_installed` == 'apache2' ]
then
echo "You are running apache"

fi

Attackers have very specific behavior, finding ways to counteract and gather intelligence (counter-intelligence) is quite exciting.

 

[Asphyxia]

I am reviving this somewhat, purely for the sake of highlighting some additional tools:

1. https://labs.withsecure.com/tools/cat-scale-linux-incident-response-collection
2. https://docs.velociraptor.app/
3. GRR Rapid Response
4. Osquery
5. Enterprise / paid: https://www.cadosecurity.com/
6. Log2Timeline / Plaso - also keep in mind running Plaso via Docker is probably easiest

   log2timeline.py output.plaso /var/log

Keep in mind, several more exist to accomplish the task at-hand depending on what that is.
I often find collaborative tools help speed investigations along likewise. This can include flowing Velociraptor collection results (or other artifacts) into a platform such as Elastic/Splunk/LogScale. Of which all 3 are possible from Velociraptor; linked.

From here, some type of incident management platform is ideal. This could be Incident.io, DFIR Iris, TheHive (although there was a decision with licensing that caused anguish among the wider cybersecurity community), and lastly this is more of a timeline tool - Timesketch.

So if you asked me what I know now, versus what I knew in 2019 I would say probably a lot more lol.

...

I am back for a handful of more mentions, when doing investigations such as Incident Response (IR) or a Compromise Assessment (CA), it's good to have IOCs, IOAs, and more ways to identify suspicious behavior.

• Historically this has been simple items such as find the IP addresses outside of the USA
  • Threat actors have become more stealthy over time, and more sophisticated threat actors may even utilize cloud infrastructure that blends in with the victim's environment.
  • Now threat actors have residential proxies more readily available to align with same countries of user's which are compromised. This makes it less likely to set off 'superman travel' styled detections.
• Then frequency analysis and outlier detections are usually hunted for. I have observed teams use anything from PowerGREP (imho noob $$$ tool for people that do not know how grep works), and feels too weak compared to Elastic with Kibana dashboards. Then you have engineer-minded teams that are proficient with Python or at least capable enough to utilize Pandas DataFrames and/or Databricks with Photon (and PySpark). It is truly funny to observe all 'the ways it can go,' and make your own personal preferences around what is efficient. It entirely is dependant on the investigator's knowledge, and what resources they have access to. Like, can you launch a server? Yes, but only with manager approval. Manager said no.
• There is of course OSINT which allows for aggregating all sorts of indicators into a platform like OpenCTI or MISP. Likewise, there are awesome detection rules typically in a format like SIGMA format available here. One company that entirely focuses on writing detections is Nextron Systems - their SIGMA feed is named Valhalla. Tooling such as Velociraptor has the capability of running other tools within itself for SIGMA hunting, one quite popular for analysis of logs is Yamato-Security's Hayabusa (this is getting more into Windows though). Another similar tool is called Chainsaw. While these tools are not perfect, they can be a decent starting point if there are little-to-no leads. Following the order of volatility is of utmost criticality.
• There are also STIX files typically passed around from various ISAC bodies such as Health-ISAC, FS-ISAC, and the list goes on.
• Last but definitely not least supplementing investigations with EDR tools like CrowdStrike, can rapidly expedite an investigation.
  • Following the OSI model backwards with a security mindset runs us into various security tools (or log sources) to check such as NGFW logs, suspicious DNS, or otherwise network activity.

To emphasize back on the main/primary topic of this thread:
Quick incident response on a Linux endpoint has pre-requisites. One is to already be ingesting your logs to a SIEM, or at a bare-minimum mirroring the logs. Another is to ideally have EDR for the mitigations & detections offered, or at least have capability to rapidly deploy tooling: EDR, Velociraptor, or CatScale.