This is part two of a four-part series on how to set up your own home lab with ELK.Read part one.
You may have noticed that the URLs we’ve been using to test our running services are using http. This can be okay in a home lab environment; however, it can be extremely insecure for anyone who could be monitoring a given network—no bueno. Plus, we also lose out on some great features of the ELK stack—also, no bueno.
We’ve intentionally neglected some pieces of the setup specifically to make sure we’re not having to change anything unnecessarily in our stack for this reason and at this point in the article, we do not have a fully functioning stack as a result. So buckle up—we’re about to finish this thing out and make a fully functional SIEM and implement some security in transit for our data.
To start, we need some form of verifying identities here by using certificates. These certificates in essence will be no different than the certificates you see while browsing the web over HTTPS; however, instead of being “verified” by a trusted entity, they’ll be verified by we, ourselves, and us.
There’s no reason to make this more difficult, however, so let’s start by creating a “template” to use to automatically generate some certificates for us using some elasticsearch utilities.
1. Start by navigating to the /usr/share/elasticsearch/ directory.
2. Create a new file here called instances.yml using your favorite text editor.
sudo vim instances.yml
3. Paste the following entries in the new file. These will be used to associate each service name with the IP of our machine. IMPORTANT NOTE: If you did not install logstash in the previous steps, remove it from this file and the subsequent certificate steps to avoid any “file or directory not found” errors.
What we’re essentially setting up is our own little Public Key Infrastructure (PKI) this is similar to how certificates are validated when you’re browsing the internet as well!
If you’re unfamiliar with key exchanges, asymmetric encryption, and other related topics, don’t worry—they’re not required for following the steps for getting this stack set up. However, it is extremely helpful to understand what’s happening in these steps to help remove some of the “behind the scenes magic.” We will assume some base knowledge here.
When you browse to a site over https, that site has to prove who it is to your browser in order for your machine to accept it. Think like a police officer checking an ID. Does it look real? Is it expired? Does the photo match? You may insist, yes that is me—but does the officer take you at your word? How do they know you are who you say you are? They may check their records, make some calls, and scan your ID to make sure it’s valid. This verification is the role of the certificate authority it’s who our machine reaches out to to ensure that the certificate is valid. It can vouch for the certificate and say, yes this is valid because I signed it or I have record of who it belongs to.
You can imagine there’s a TON of power with that—but what if that certificate is invalid or the certificate authority is untrusted? You’ve likely run into this before—your browser alerts you and says “Careful, I don’t know who this is—enter at your own risk.”
We will make an “untrusted” or “self-signed” certificate authority using some of elasticsearch’s built in utilities.
1. Create a Certificate Authority bundle using the elasticsearch-certutil.
sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --pem
2. Now, we’ll want to unzip this bundle using the unzip utility we installed at the beginning of the article.
sudo unzip ./elastic-stack-ca.zip
Now we have a ca/ directory, a certificate file, and a matching key for our certificate authority. Next we’ll want to generate some certificates to sign with it using that instances.yml file we generated earlier!
3. We’ll use the same elasticsearch-certutil utility to generate these certificates:
Now that we’ve created our certificate authority and generated our certificate files, we need to take another commonly missed step to secure our ELK deployment: proper permissions.
You notice how in some directories we had to run sudo to gain administrator access to their contents? That’s because of how the Linux file permissions are configured by default. With root permissions we can access most everything in the file system without issue. So we should just give root permissions to our applications right? That way they can access everything they need without permissions issues.
Absolutely not. While running things with open permissions may make things easier on set up, it can be a nightmare when those permissions are used against you. You can check out most any CTF box and find a perfect example of this. By running as root, an adversary who compromises this application then has root privileges giving them free reign of the system.
So what do we do? We’ll create a user and assign them ownership of the directories they require.
When we installed elasticsearch, logstash, and kibana a user was created for each service.
1. Navigate to the /usr/share directory where we’ll take ownership with our elasticsearch and kibana users.
So let’s recap up to this point. We’ve gone through all of this work to set up our services and generate certificates to verify their identities. Now we need to tell those services where to find the proper files to use them to communicate and to do so over https.
1. Edit Kibana YAML /etc/kibana/kibana.yml
Copy the following and paste it to the bottom of the /etc/kibana/kibana.yml file using your favorite text editor.
Yay! A new error and a bundle of junk. This is another JSON payload that we can parse with jq for readability. Try it now and we’ll see that after our changes, we require credentials to connect to elasticsearch.
We have a working Application Program Interface (API) now! We’re still missing some credentials though—let’s go ahead and generate those next.
Section 4: Password Generation
To generate those passwords, we will need to use another elasticsearch utility elasticsearch-setup-passwords to create the randomized passwords for our user and service accounts. This will only flash on screen once though! So be sure to document them (perhaps in a secure note in a password manager?).
1. Run the elasticsearch-setup-passwords utility to generate those passwords:
sudo /usr/shareare/elasticsearch/bin/elasticsearch-setup-passwords auto
Changed password for user apm_system PASSWORD apm_system = <Password>
Changed password for user kibana_system PASSWORD kibana_system = <Password>
Changed password for user kibana PASSWORD kibana = <Password>
Changed password for user logstash_system PASSWORD logstash_system = <Password>
Changed password for user beats_system PASSWORD beats_system = <Password>
Changed password for user remote_monitoring_user PASSWORD remote_monitoring_user = <Password>
Changed password for user elastic PASSWORD elastic = <Password>
Now, let’s edit our /etc/kibana/kibana.yml file to add the kibana_system generated username to interact with elasticsearch.