Enable Cloudera AUTO-TLS through API

In this post let’s see how we can enable TLS in Cloudera Manager and CDH.

Requirement: Enable TLS encryption on existing Cloudera Manager and Cloudera cluster.

Prerequisite: On all hosts I have created directory where I will store all certificates

/mounts/edge/app/cloudera/security/pki/

Note: To configure TLS encryption manually, see Manually Configuring TLS/SSL Encryption for CDH Services and Manually Configuring TLS Encryption for Cloudera Manager. Starting with Cloudera Manager 6.2, you can enable auto-TLS on existing deployments. 

To complete this task I performed following tasks: I implemented this on 10 hosts including Cloudera Manager host.

1. Use the keytool utility to create a Java keystore {Perform this on all hosts}
2. Use the keytool utility to certificate signing request (CSR) {Perform this on all hosts}
3. Export the p12 file {perform this on all hosts}.
4. Use openssl command to export the private key into its own file {perform this on all hosts}.

Let’s discuss above steps in detail, and then we will go to the next action item.

Step-1: Create a Java keystore: We have to create keystore on all the hosts.

-bash-4.2$ keytool -genkeypair -alias $(hostname -f) -keyalg RSA 
-keystore /mounts/edge/app/cloudera/security/pki/$(hostname -f).jks 
-keysize 2048 -dname "CN=$(hostname -f),OU=IT,O=COM,L=Dur,ST=NC,C=US" -ext san=dns:$(hostname -f)
OUT PUT: 
Enter keystore password: 
Re-enter new password:
Enter key password for  
(RETURN if same as keystore password):
Re-enter new password:
-bash-4.2$ ls -ltr
-rw-rw-r-- 1 hdfs hdfs 2344 Feb 2 02:57 host1.jks

Our keystore host1.jks is ready.  Now let’s move to next step.

Step-2: Create certificate signing request (CSR): Execute below command to create CSR.

-bash-4.2$ keytool -certreq -alias $(hostname -f) 
-keystore /mounts/edge/app/cloudera/security/pki/$(hostname -f).jks 
-file /mounts/edge/app/cloudera/security/pki/$(hostname -f).csr -ext san=dns:$(hostname -f) 
-ext EKU=serverAuth,clientAuth
Enter keystore password:
-bash-4.2$ ls -ltr
-rw-rw-r-- 1 hdfs hdfs 2344 Feb 2 02:57 host1.jks
-rw-rw-r-- 1 hdfs hdfs 1225 Feb 2 02:58 host1.csr

Once host1.csr is ready, we can send this file to CA, they will generate certificate based on this CSR file. Let’s move to next step.

Step-3: Export p12: We need private key for authentication for that we have to first generate p12  file. Execute below command to generate key file.

-bash-4.2$ keytool -importkeystore -srckeystore /mounts/edge/app/cloudera/security/pki/$(hostname -f).jks 
-destkeystore /mounts/edge/app/cloudera/security/pki/$(hostname -f)-key.p12 -deststoretype PKCS12 -srcalias $(hostname -f)
Out Put: 
Importing keystore /mounts/edge/app/cloudera/security/pki/host1.jks to /mounts/edge/app/cloudera/security/pki/host-key.p12...
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
-bash-4.2$ ls -ltr
-rw-rw-r-- 1 hdfs hdfs 2344 Feb 2 02:57 host1.jks
-rw-rw-r-- 1 hdfs hdfs 1225 Feb 2 02:58 host1.csr
-rw-rw-r-- 1 hdfs hdfs 2779 Feb 2 03:01 host1-key.p12

Step-4: Use openssl command to export the private key into its own file: In this step we will extract key file.

-bash-4.2$ openssl pkcs12 -in /mounts/edge/app/cloudera/security/pki/$(hostname -f)-key.p12 
-nocerts -out /mounts/edge/app/cloudera/security/pki/$(hostname -f).key
Out Put:
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-bash-4.2$ ls -ltr
-rw-rw-r-- 1 hdfs hdfs 2344 Feb 2 02:57 host1.jks
-rw-rw-r-- 1 hdfs hdfs 1225 Feb 2 02:58 host1.csr
-rw-rw-r-- 1 hdfs hdfs 2779 Feb 2 03:01 host1-key.p12
-rw-rw-r-- 1 hdfs hdfs 2007 Feb 2 03:02 host1.key

We have to execute above steps on all hosts. Create one directory at Cloudera manager host where we will store all keys (copy from all hosts) and certificates (received from CA).

Step-5: Create directory and save certificates/keys:

Copy all key files (i.e. host1.key) from all host to Cloudera Manager host. In our case first we have to copy all files from all hosts to jump host and then from jump host to Cloudera Manager host.

We are storing all files (keys and CA certificates) at this location Example : /mounts/edge/app/cloudera/security/CMCA/
On the CM host create AutoTLS directory and change the ownership to the CM runtime user
Cloudera manager will create all certificates through AutoTLS script: /mounts/edge/app/cloudera/security/AutoTLS

Note:
1. Make sure to run “dos2unix host1.key” command on Cloudera manager host to avoid CTL-M error on files. In case we are moving files from window to Unix machine.
2. Make sure CM runtime user (cloudera-scm:cloudera-scm) owns all the files and directories.
3. If your issued certificate is in binary (DER) format, convert it to PEM format by just rename it.

Final directory should look like

[admin@host1 security]$ ls -ltr
total 4
drwxr-xr-x 5 cloudera-scm cloudera-scm 114 Mar 31 03:21 AutoTLS
drwxr-xr-x 3 cloudera-scm cloudera-scm 4096 Apr 5 21:20 CMCA

Step-6: Get root and intermediate certificate and place it with all CA certificates in same directory. It is very important to verify certificates received from CA. Execute below command to verify each certificate. Specially check the hosts name in each certificate. It should be FQDN.

Print Certificate command
$keytool -printcert -file <path of the file>/host1.pem

Step-7: Create truststore.pem file as root.pem+intermediate.pem = truststore.pem

$cat root.pem Intermediat.pem > truststore.pem

Step-8: Create one truststore.pwd file which contain password of trust store.

Step-9: Create one key.pwd file which contain the password(same password which we used to generate key files) of key files and place it in the same directory along with all keys. Final directory structure should look like as shown below:

[admin@host01 CMCA]$ ls -ltr
total 96
-rw-r--r-- 1 cloudera-scm cloudera-scm 2821 Feb 10 21:21 Intermediate.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2004 Feb 10 21:21 root.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 4825 Feb 10 21:29 truststore.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm   13 Feb 10 21:30 truststore.pwd
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:26 host01.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:26 host02.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:26 host03.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host04.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host05.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host06.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host07.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host08.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host09.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host10.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host11.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2642 Feb 12 14:27 host12.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2675 Apr  5 21:19 host13.devopsbaba.com.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2675 Apr  5 21:19 host14.devopsbaba.com.pem
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Apr  5 21:52 keys

[admin@host01 keys]$ ls -ltr
total 76
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host01.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host02.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host03.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host04.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host05.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host06.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host07.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host08.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host09.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host10.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host11.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Jan 31 04:56 host12.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 13 Feb 10 21:43 key.pwd
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Apr 5 21:51 host13.devopsbaba.com-key.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 2007 Apr 5 21:51 host14.devopsbaba.com-key.pem

Step-10: Check API version: On cloudera manager host execute following command 

-bash-4.2$ curl -u : http://host1.devosbaba.com:7180/api/version
Output: V33

Step-11:  We have used cloudera manager API /cm/commands/generateCmca .  As shown below:

curl -i -v -u : -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d \
'{ 
"location" : "/mounts/edge/app/cloudera/security/AutoTLS", 
"customCA" : true, 
"interpretAsFilenames" : true, 
"cmHostCert" : "/mounts/edge/app/cloudera/security/CMCA/host01.devopsbaba.com.pem", 
"cmHostKey" : "/mounts/edge/app/cloudera/security/CMCA/keys/host01.devopsbaba.com-key.pem", 
"caCert" : "/mounts/edge/app/cloudera/security/CMCA/truststore.pem", 
"keystorePasswd" : "/mounts/edge/app/cloudera/security/CMCA/keys/key.pwd", 
"truststorePasswd" : "/mounts/edge/app/cloudera/security/CMCA/truststore.pwd", 
"hostCerts" : [ { 
"hostname" : "host01.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host01.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host01.devopsbaba.com-key.pem" 
}, { 
"hostname" : "host02.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host02.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host02.devopsbaba.com-key.pem" 
}, { 
"hostname" : "host03.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host03.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host03.devopsbaba.com-key.pem" 
}, { 
"hostname" : "host04.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host04.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host04.devopsbaba.com-key.pem" 
}, { 
"hostname" : "host05.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host05.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host05.devopsbaba.com-key.pem" 
}, { 
"hostname" : "host06.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host06.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host06.devopsbaba.com-key.pem" 
}, {
"hostname" : "host07.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host07.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host07.devopsbaba.com-key.pem" 
}, {
"hostname" : "host08.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host08.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host08.devopsbaba.com-key.pem" 
}, {
"hostname" : "host09.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host09.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host09.devopsbaba.com-key.pem" 
}, {
"hostname" : "host10.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host10.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host10.devopsbaba.com-key.pem" 
}, {
"hostname" : "host11.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host11.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host11.devopsbaba.com-key.pem" 
}, {
"hostname" : "host12.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host12.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host12.devopsbaba.com-key.pem" 
}, {
"hostname" : "host13.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host13.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host13.devopsbaba.com-key.pem" 
}, {
"hostname" : "host14.devopsbaba.com", 
"certificate" : "/mounts/edge/app/cloudera/security/CMCA/host14.devopsbaba.com.pem", 
"key" : "/mounts/edge/app/cloudera/security/CMCA/keys/host14.devopsbaba.com-key.pem" 
} ],
"configureAllServices" : true, 
"sshPort" : 22, 
"userName" : "root",
"password" : "password" 
}' \
"http://host01.devopsbaba.com:7180/api/v33/cm/commands/generateCmca"

Execute above API command with root user.

Step-12: Restart following services

• Restart the CM server (systemctl restart cloudera-scm-server)
• Restart the Agent on each node (systemctl restart cloudera-scm-agent)
• Restart the Cloudera Management Services
• Restart CDH and KTS (if used).

Note**

Permission on directories and files
chown cloudera-scm:cloudera-scm /mounts/edge/app/cloudera/security/CMCA
chmod 755 /mounts/edge/app/cloudera/security/CMCA
chmod 755 /mounts/edge/app/cloudera/security/CMCA/keys
chmod 644 /mounts/edge/app/cloudera/security/CMCA/ all files under CMCA

Congratulation your cluster is TLS enabled.

Next topic: After enabling TLS we have faced few issue:

• Impala service did not start.

I will explain how I resolve this issue in my next post.