Elkstack Docker

Asphyxia · Feb 5, 2020

Anyone go like wtf am I supposed to do? Elk, what.. that a deer, he got hop hop through snow.

He go moo? No, wtf noise he make. He go Docker..

Time to get legit, sorry.. I am f**k*ng tired and need more sleep lmao. But time to show how we proper do an ELK build.

1. Get Debian because Ubuntu sucks.
2. Install your Debian.
3. apt update
4. apt upgrade
5. apt install docker docker.io docker-compose git -y

Here is the ELK party time:

Code:

cd ~
git clone https://github.com/deviantony/docker-elk.git
cd ./docker-elk

Time to find their changeme bullshit:

Code:

grep -rnw './' -e 'changeme'

The above gives us all the locations changeme exists.

Let's read instructions fast:

The stack is pre-configured with the following **privileged** bootstrap user:

* user: *elastic*
* password: *changeme*

Although all stack components work out-of-the-box with this user, we strongly recommend using the unprivileged [built-in
users][builtin-users] instead for increased security.

1. Initialize passwords for built-in users

```console
$ docker-compose exec -T elasticsearch bin/elasticsearch-setup-passwords auto --batch
```

Passwords for all 6 built-in users will be randomly generated. Take note of them.

2. Unset the bootstrap password (_optional_)

Remove the `ELASTIC_PASSWORD` environment variable from the `elasticsearch` service inside the Compose file
(`docker-compose.yml`). It is only used to initialize the keystore during the initial startup of Elasticsearch.

3. Replace usernames and passwords in configuration files

Use the `kibana` user inside the Kibana configuration file (`kibana/config/kibana.yml`) and the `logstash_system` user
inside the Logstash configuration file (`logstash/config/logstash.yml`) in place of the existing `elastic` user.

Replace the password for the `elastic` user inside the Logstash pipeline file (`logstash/pipeline/logstash.conf`).

> :information_source: Do not use the `logstash_system` user inside the Logstash *pipeline* file, it does not have
> sufficient permissions to create indices. Follow the instructions at [Configuring Security in Logstash][ls-security]
> to create a user with suitable roles.

See also the [Configuration](#configuration) section below.

4. Restart Kibana and Logstash to apply changes

```console
$ docker-compose restart kibana logstash
```

> :information_source: Learn more about the security of the Elastic stack at [Tutorial: Getting started with
> security][sec-tutorial].

### Injecting data

Give Kibana about a minute to initialize, then access the Kibana web UI by hitting
[http://localhost:5601](http://localhost:5601) with a web browser and use the following default credentials to log in:

* user: *elastic*
* password: *\<your generated elastic password>*

Now that the stack is running, you can go ahead and inject some log entries. The shipped Logstash configuration allows
you to send content via TCP:

Let's see all the files we have to change "changeme" in:

Code:

k:~/docker-elk# grep -rnw './' -e 'changeme'
./extensions/apm-server/config/apm-server.yml:8:    password: changeme
./kibana/config/kibana.yml:13:elasticsearch.password: changeme
./logstash/pipeline/logstash.conf:13:           password => "changeme"
./logstash/config/logstash.yml:12:xpack.monitoring.elasticsearch.password: changeme
./README.md:133:* password: *changeme*
./.travis/run-tests.sh:61:poll_ready elasticsearch 'http://localhost:9200/' 'elastic:changeme'
./.travis/run-tests.sh:64:poll_ready kibana 'http://localhost:5601/api/status' 'kibana:changeme'
./.travis/run-tests.sh:75:      -u elastic:changeme \
./.travis/run-tests.sh:79:response="$(curl 'http://localhost:5601/api/saved_objects/_find?type=index-pattern' -u elastic:changeme)"
./.travis/run-tests.sh:91:curl -X POST 'http://localhost:9200/_refresh' -u elastic:changeme \
./.travis/run-tests.sh:95:response="$(curl 'http://localhost:9200/_count?q=message:dockerelk&pretty' -u elastic:changeme)"
./.travis/elasticsearch-setup-passwords.exp:5:set password "changeme"
./docker-stack.yml:15:      ELASTIC_PASSWORD: changeme
./docker-compose.yml:22:      ELASTIC_PASSWORD: changeme

Simplify this shit with cut.. when things get too hard just cut AHAHAHAHAH:

Code:

grep -rnw './' -e 'changeme' | cut -d ':' -f 1

We noticed some duplicates, fuck that.

Code:

grep -rnw './' -e 'changeme' | cut -d ':' -f 1 | sort | uniq -c

This shows us:

Code:

k:~/docker-elk# grep -rnw './' -e 'changeme' | cut -d ':' -f 1 | sort | uniq -c
      1 ./docker-compose.yml
      1 ./docker-stack.yml
      1 ./extensions/apm-server/config/apm-server.yml
      1 ./kibana/config/kibana.yml
      1 ./logstash/config/logstash.yml
      1 ./logstash/pipeline/logstash.conf
      1 ./README.md
      1 ./.travis/elasticsearch-setup-passwords.exp
      6 ./.travis/run-tests.sh

Not hard, time to edit all this sh** could use sed maybe.. 1sec? Hold beer pls

Sed can do this shtufff

Code:

sed -i 's/old-text/new-text/g'

So, let's apply to our shtttufff:

Code:

sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./docker-compose.yml
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./docker-stack.yml
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./extensions/apm-server/config/apm-server.yml
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./kibana/config/kibana.yml
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./logstash/config/logstash.yml
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./logstash/pipeline/logstash.conf
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./README.md
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./.travis/elasticsearch-setup-passwords.exp
sed -i 's/changeme/myPasswd12inchesUrsIsOnly6/g' ./.travis/run-tests.sh

That is an example, you could probably further expedite the searching and replacing using xargs for example. idgaf, figure it outtt if you do..

Already started? Nuke your docker shit - CAREFUL THIS MAKES AL LDOCKER SHIT GO BYE BYE:

Code:

docker container stop $(docker container ls -aq)
docker container rm $(docker container ls -aq)
docker image prune -a
docker system prune 
docker system prune --volumes

Noice mate, try to do your compose all over:

Code:

docker-compose up -d

Thing is.. Kibana is going to take its sweet ass time to boot up, so don't go mad if it takes time for the port to be ready.

Kibana server is not ready yet

But keep in mind even when you change the changeme password from this elk stack stuff, you are going to want to listen to the instructions above with generating users

hopefully you can have fun and use this neat tool.. WITHOUT default creds.

Asphyxia · Feb 5, 2020

SERIOUSLY GUYS KIBANA IS SLOWWWWW to launch it has been 6 mintues wtf come up m8

Asphyxia · Feb 5, 2020

Okay, figured it out and updated instructions. If you set this sh** up you have to then:

Code:

docker image prune -a

To blow all the stopped server images away, gotcha.. damn I am going to learn a lot of Docker soon. WOOOOOOOOO!

Asphyxia · Feb 5, 2020

Okay now I am just getting pissed off..

You have to tweak all this config sh** all by hand, every little piece.

Fine fine fine.. fuck.

https://www.elastic.co/guide/en/kibana/current/configuring-tls.html we have to now redo the whole thing all over again.

Code:

docker container stop $(docker container ls -aq)
docker container rm $(docker container ls -aq)
docker image prune -a
docker system prune
docker system prune --volumes

Go back to your docker-elk folder, like:

Code:

cd ~
cd docker-elk
find | grep "kibana"

Notice the config file, ./kibana/config/kibana.yml

I am going to try turning SSL on without a key.

Let's try to nano this thing!

Code:

nano ./kibana/config/kibana.yml

The bottom of my config now looks like:

Code:

elasticsearch.username: elastic
elasticsearch.password: myPasswd12inchesUrsIsOnly6
server.ssl.enabled: true

I am going to save this and try to get this Docker running...

Code:

docker-compose up -d

Wowww!! Trying to see how this goes, btw configuring HTTPS all across looks like more configs being applied throughout files. This is fine, but having a config generator or something that just takes care of all this shit would sure be nice. What is convenience? lol ;0

I get this is like free Splunk but it doesn't have to be like a coffee cup that automatically flips into your lap randomly..

xD

............. and this just totally breaks itself, fuckin' cool - won't even start that way. Must have cert, heh kek roll floor laff laff

Asphyxia · Feb 5, 2020

Jesus, this is a nightmare but I am learning fast. Soon going to be making a series of ELK Stack videos and installing this sh*t everywhere I can to learn the best ways to gather information and continuously monitor things with for example Metricbeat for real-time CPU monitoring, Winlogbeat, and a buncha other stuff.

This should be real damn cool..

Asphyxia · Feb 6, 2020

Okay I fucked up SSL yesterday, time to try again now that I have gotten an awesome guy on my team to generate me an SSL cert (wildcard).

Time to try this shit, here we go weeeee...

Kill all our ELK - copy each of these line by line:

Code:

docker container stop $(docker container ls -aq)
docker container rm $(docker container ls -aq)
docker image prune -a
docker system prune
docker system prune --volumes

Go find our docker-elk folder:

Code:

cd ~
cd docker-elk

According to https://www.elastic.co/guide/en/kibana/current/configuring-tls.html we have to go find our Kibana config.

Code:

find | grep "kibana"

Hey, look ./kibana/config/kibana.yml

I moved my .key, .crt, and .pem inside of /root/docker-elk and named them accordingly to the documentation to make me less of a robot.. I think.

Now if we type pwd.

Code:

/root/docker-elk

and if we ls | grep server:

Code:

server.crt
server.key

Hell yeah, life is good at this point.

Nano that config now, eh?

Code:

nano ./kibana/config/kibana.yml

Somewhere in this damn config file, through nano just paste this shit - at the bottom works.

Code:

server.ssl.enabled: true
server.ssl.key: /root/docker-elk/server.key
server.ssl.certificate: /root/docker-elk/server.crt

Then modify FROM:

Code:

elasticsearch.hosts: [ "http://elasticsearch:9200" ]

TO (replace siem.ciphers.pw with your hostname):

Code:

elasticsearch.hosts: [ "https://siem.ciphers.pw:9200" ]

...

I also have my cacert.pem file (your pem file):

Code:

elasticsearch.ssl.certificateAuthorities: /root/docker-elk/cacert.pem

Asphyxia · Feb 6, 2020

Okay, I am saying fuck this here.

This did not work, I am just going to HTTPS/SSL/TLS proxy the fucker through so it hits the server, then sends it inside of Docker itself. Jesssssuusss fuuuuuuuuuuuuuuuck me w00000000000j0000

Asphyxia · Feb 6, 2020

Okay, start to finish this fucker:

Code:

cd ~
git clone https://github.com/deviantony/docker-elk.git
cd ./docker-elk

Now let's change the default changeme password:

Code:

sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./docker-compose.yml
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./docker-stack.yml
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./extensions/apm-server/config/apm-server.yml
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./kibana/config/kibana.yml
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./logstash/config/logstash.yml
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./logstash/pipeline/logstash.conf
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./README.md
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./.travis/elasticsearch-setup-passwords.exp
sed -i 's/changeme/Pleaseb3Nice2Me6oclock/g' ./.travis/run-tests.sh

Launch the fucker:

Code:

docker-compose up -d

Troubleshooting:
If you have already installed some Docker bullshit and failed then first you have to blow away your stupid failed Docker imagines with default pass "changeme" on there, like this.. running one line at a time and smashing y for some.

Code:

docker container stop $(docker container ls -aq)
docker container rm $(docker container ls -aq)
docker image prune -a
docker system prune
docker system prune --volumes

Get your lovely Apache (or NGINX), I am using Apache for simplicity to create an HTTPS proxy of sorts for the underlying http:

Code:

apt update
apt upgrade -y
apt install apache2 -y

Now I am going to borrow slightly from here https://www.digitalocean.com/commun...signed-ssl-certificate-for-apache-in-debian-9 but also borrowing from here for the proxy config https://stackoverflow.com/a/42399212

I initially failed because I needed to enable a module (proxy) and changed my SSL certs into ~ (root homedir) inside ./ssl

Code:

root@whaleshark:~/docker-elk# sudo nano /etc/apache2/sites-available/default-ssl.conf
root@whaleshark:~/docker-elk# sudo apache2ctl configtest
AH00526: Syntax error on line 28 of /etc/apache2/sites-enabled/default-ssl.conf:
Invalid command 'ProxyRequests', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.
root@whaleshark:~/docker-elk# a2enmo^C
root@whaleshark:~/docker-elk# sudo a2enmod proxy
Enabling module proxy.
To activate the new configuration, you need to run:
  systemctl restart apache2
root@whaleshark:~/docker-elk# sudo apache2ctl configtest
AH00526: Syntax error on line 45 of /etc/apache2/sites-enabled/default-ssl.conf:
SSLCertificateFile: file '/root/docker-elk/server.crt' does not exist or is empty
Action 'configtest' failed.
The Apache error log may have more information.
root@whaleshark:~/docker-elk# sudo nano /etc/apache2/sites-available/default-ssl.conf
root@whaleshark:~/docker-elk# sudo apache2ctl configtest
Syntax OK
root@whaleshark:~/docker-elk#

I am going to show a working demo build:
sudo nano /etc/apache2/sites-available/default-ssl.conf

Code:

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
ServerName siem1.lol.com

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLCertificateFile      /root/ssl/server.crt
                SSLCertificateKeyFile /root/ssl/server.key




        ProxyPreserveHost       On

        ProxyPass               / http://127.0.0.1:9200/
        ProxyPassReverse        / http://127.0.0.1:9200/

        </VirtualHost>
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Obviously do systemctl restart apache2.service after.

Now there is one negative to this current setup.

Asphyxia · Feb 6, 2020

Okay, at first I was gonna say fuck it not working but turns out this is fully working as an HTTPS-to-http proxy.

So I think I am going to run things this way, either w/ NGINX or Apache, but for now I'm rolling Apache for simple configs and documentation.

Turns out you just gotta :9200 on this and then I am going to listen on maybe 444 to https to 5601 for Kibana, but to access Kibana's port (444) I am going to likely require some sort of an IP whitelisting process for obvious reasons. Don't want a "bad guy" getting into the logs.. eh.

Asphyxia · Feb 7, 2020

Proof this is working at the 443:

Winlogbeat configured as such:

Code:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://siem1.ciphers.pw:443"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "Pleaseb3Nice2Me6oclock"

Noice m8..

Now how about we go back to the http method and see wtf is going on here look like this:

Code:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["siem1.ciphers.pw:9200"]

  # Optional protocol and basic auth credentials.
  protocol: "http"
  username: "elastic"
  password: "Pleaseb3Nice2Me6oclock"

This obviously requires us to re-run the setup:

Code:

PS C:\Program Files\Winlogbeat> Start-Service winlogbeat
PS C:\Program Files\Winlogbeat> Stop-Service winlogbeat
PS C:\Program Files\Winlogbeat> .\winlogbeat.exe setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
PS C:\Program Files\Winlogbeat> Start-Service winlogbeat

Redoing this again.. logging in/off after that..

Bam.. now every single damn endpoint that can in any way monitor traffic across the network essentially can scrape out all your Winlogbeat stuff crossing the network. Kind of a dumbass setup imho.

Soooooo this is why I am working on setting up the https/ssl reverse proxy shit and so far, fairly decent success.

Just considering the idea of having a proxy using cookie proxy stuff - let me try that now.

Asphyxia · Feb 7, 2020

Finding where Apache specifies the 443 port shit:

Code:

cd /etc/apache2
grep -rnw './' -e '443'

Notice the "Listen"... let's go inside ports to listen to another port.

Fuckin' easy m8, save.

Now open that default-ssl sh*t.

Code:

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
ServerName siem1.ciphers.pw

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLCertificateFile      /root/ssl/server.crt
                SSLCertificateKeyFile /root/ssl/server.key




        ProxyPreserveHost       On

        ProxyPass               / http://siem1.ciphers.pw:9200/
        ProxyPassReverse        / http://siem1.ciphers.pw:9200/

        </VirtualHost>

        <VirtualHost _default_:444>
ServerName siem1.ciphers.pw

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLCertificateFile      /root/ssl/server.crt
                SSLCertificateKeyFile /root/ssl/server.key




        ProxyPreserveHost       On

        ProxyPass               / http://siem1.ciphers.pw:5601/
        ProxyPassReverse        / http://siem1.ciphers.pw:5601/

        </VirtualHost>

</IfModule>

Should return Syntax OK:

Code:

sudo apache2ctl configtest

So then run:

Code:

systemctl restart apache2.service

Now try to load your site ;] using the shtuff m8.

Ahhhhhhhh this broke the bitch and replies back:

Secure Connection Failed

An error occurred during a connection to whaleshark.grcc.edu:444. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG

The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.

I understand why though!

Inside of the :444 (other SSL port) config, you must simply add SSLEngine On. Simple as hell, hmm.. CAKE!

Again, restart your serv.

Code:

systemctl restart apache2.service

I finally got this all working properly, damn that was a biatch hahaha.. WOOHOO win win pizza partyyyy.

btw if you start messing up w/ cookies, it could be something w/ proxy passing cookie session stuff, umm I'm going to research:

reverse proxy with multiple sessions in same browser

My reverse proxy seems to be mostly working, though when I open sessions to different backend servers, within the same browser, I get this message from the browser (tested with Chrome and IE): Alr...

serverfault.com

and other similar stuff.

Asphyxia · Feb 7, 2020

MAKING video really fucking soon, to feel like a fucking bad ass and prouddddd man

this was bullshit to make happen lmao.

Asphyxia · Feb 11, 2020

When you decide to switch to basic licensing, they like to strip some nice features out:

Confirm Revert to Basic License
Some functionality will be lost if you replace your TRIAL license with a BASIC license. Review the list of features below.
Watcher will be disabled
Logstash will no longer poll for centrally-managed pipelines
Security will default to disabled (set xpack.security.enabled to enable security).
Beats will no longer be able to use centrally-managed configuration
Multi-cluster support is disabled for clusters with [BASIC] license. If you are running multiple clusters, users won't be able to access the clusters with [BASIC] licenses from within a single X-Pack Kibana instance. You will have to deploy a separate and dedicated X-pack Kibana instance for each [BASIC] cluster you wish to monitor.
Graph will be disabled
Machine learning will be disabled
JDBC and ODBC support will be disabled, but you can continue to use SQL CLI and REST endpoint

So.. roll your own ELK with alternatives:

Elastic Stack (formerly X-Pack) Functionality	Alternatives
Elasticsearch Security (formerly X-Pack Security)	SearchGuard Sematext Cloud or Enterprise
Elasticsearch Alerting (formerly X-Pack Alerting)	Elastalert Logagent Sentinl Sematext Cloud
Elasticsearch Monitoring (formerly X-Pack Monitoring)	Sematext Cloud Elasticsearch integration, Prometheus, Datadog, New Relic, etc.
Reporting (formerly X-Pack Reporting)	Skedler Sentinl Sematext Cloud
Graph (formerly X-Pack Graph)	Kibi Kbn_network DIY: Cytoscape.js, Visjs.org (open source)
Machine Learning (formerly X-Pack Machine Learning)	Knowi Sematext Cloud and Enterprise
Elasticsearch Support	Sematext production support for Elasticsearch and ELK Stack (Elasticsearch, Logstash, Kibana), from Elasticsearch 1.x and up!

Asphyxia · Feb 13, 2020

(invalid user)	6	—	—	16 minutes ago	65.31.127.80
(unknown user)	3	—	—	16 minutes ago	65.31.127.80
florida1	2	—	—	16 minutes ago	65.31.127.80
nagios	2	—	—	45 minutes ago	221.163.8.108
saharah	2	—	—	26 minutes ago	193.0.69.25
ubuntu	2	—	—	26 minutes ago	195.223.211.242
yydx	2	—	—	39 minutes ago	65.151.176.53

These logged users were trying to get into SSH on a test box.

Safe to say they can fuck off.

I have a video on setting up ELK Stack coming soon!

What we want is either:

Demo Installer (Linux/Mac)

Search Guard ships with a demo installer for quickly setting up a working configuration. Use it for PoCs or checking out our features.

docs.search-guard.com

or

Free - ReadonlyREST

Get ReadonlyREST PRO. Start your FREE Trial Today.

readonlyrest.com

Nice ideas

Elkstack Docker

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

reverse proxy with multiple sessions in same browser

Asphyxia

Owner

Asphyxia

Owner

Asphyxia

Owner

Demo Installer (Linux/Mac)

Free - ReadonlyREST