Solve Impala issue after enabling TLS in Cloudera

In my previous post I configured Auto-TLS in Cloudera Manager and CDH 6x. After enabling SSL, my development team faced this issue. Most of the developer use hue to connect impala service, but they were not able to access impala editor due to this error

"Certificate error with remote host: hostname 'host1.devopsbaba.com' doesn't match 'host6.devopsbaba.com"

Initially I thought it’s Impala HA/LB configuration issue so to check that I executed below commands on Hue server host:

$ ls -l /etc/hue/conf 
$ ls -l /etc/alternatives/ |grep hue

But when I executed above commands I got below output:

[admin@host01 ~]$ ls -l /etc/hue/conf 
ls: cannot access /etc/hue/conf: No such file or directory 
[admin@host01 ~]$ ls -l /etc/alternatives/ |grep hue 
lrwxrwxrwx 1 root root 75 Feb 25 03:53 hue-conf -> /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p3739.1842613/etc/hue/conf.empty 
[admin@host01 ~]$ ls -l /etc/hue/ 
total 0

I did not get any logs in /var/log/hue directory as it’s not updating because after upgrade 6x.x to 6.x.x. Cloudera did not create or recreate the alternatives for hue-conf to point to /opt/cloudera/parcels/CDH/etc/hue/conf.empty/.  To resolve the issue, do the following on EVERY Hue server:

#RHEL/Centos
/usr/sbin/alternatives --install /etc/hue/conf hue-conf /opt/cloudera/parcels/CDH/etc/hue/conf.empty 10

Once it is fixed, restart the Hue Servers, reproduced the issue, and then checked /var/log/hue/runcpserver.log; this time you can see new lines of logs. I did lots of troubleshooting, also checked hue.ini configuration and finally got it that there is some issue in haproxy load balancing setting for impala.

As I have implemented Auto TLS so it’s required to use Haproxy SSL terminate while configure load balancer. Here is the link for more detail about SSL terminate.

To fix this issue and to configure haproxy I have to provide all certificates dedicated to each host in haproxy impala load balancer section so that each host can communicate smoothly as shown below: This is a haproxy.cfg file, you can be check and modify it at /etc/haproxy/haproxy.cfg

# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main *:5000
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
use_backend static if url_static
default_backend app
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
balance roundrobin
server app01 127.0.0.1:5001 check
server app02 127.0.0.1:5002 check
server app03 127.0.0.1:5003 check
server app04 127.0.0.1:5004 check
# This is the setup for Impala. Impala client connect to load_balancer_host:25003.
# HAProxy will balance connections among the list of servers listed below.
# The list of Impalad is listening at port 21000 for beeswax (impala-shell) or original ODBC driver.
# For JDBC or ODBC version 2.x driver, use port 21050 instead of 21000.
frontend impala
bind *:25003 ssl crt /opt/cloudera/CMCA/cert-key01.pem
mode tcp
option tcplog
default_backend impala_backend
backend impala_backend
mode tcp
option tcplog
balance source
server app05 host04.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app06 host05.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app07 host06.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app08 host07.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app09 host08.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app10 host09.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app11 host10.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app12 host11.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app13 host12.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app16 host13.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app17 host16.devopsbaba.com:21000 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem

# Setup for Hue or other JDBC-enabled applications.
# In particular, Hue requires sticky sessions.
# The application connects to load_balancer_host:21051, and HAProxy balances
# connections to the associated hosts, where Impala listens for JDBC
# requests on port 21050.

frontend impalajdbc
bind *:21051 ssl crt /opt/cloudera/CMCA/cert-key01.pem
mode tcp
option tcplog
default_backend impalajdbc_backend

backend impalajdbc_backend
mode tcp
option tcplog
balance source
server app18 host04.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app19 host05.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app20 host06.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app21 host07.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app22 host08.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app23 host09.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app24 host10.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app25 host11.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app26 host12.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app27 host13.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem
server app28 host16.devopsbaba.com:21050 check ssl ca-file /opt/cloudera/AutoTLS/trust-store/cm-auto-global_cacerts.pem

#Hive High Availability Config#
listen stats :25002
balance
mode http
stats enable
stats auth username:password
# This is the setup for HS2. beeline client connect to load_balancer_host:10001.
# HAProxy will balance connections among the list of servers listed below.
listen hiveserver2 :10001
mode tcp
option tcplog
balance source
server hiveserver2_1 host01.devopsbaba.com:10000
server hiveserver2_2 host02.devopsbaba.com:10000
server hiveserver2_3 host03.devopsbaba.com:10000

I have used cert-key01.pem file in frontend impalajdbc.Please find below steps to generate this file:

1. Create unencrypted key: Execute below command on the unix host
#openssl rsa -in devopsbaba.com-key.pem -out unencrypted.key
2. Merge certificate file and key file
#cat devopsbaba.com.pem unencrypted.key > cert-key01.pem
Note: Here devopsbaba.com-key.pem is the key we have used for Cloudera Manager hosts 
and devopsbaba.com.pem is the Cloudera manager CA certificate. 

Update haproxy with the path of this file as shown above.

I have mark all changes in green in haproxy.cfg file. Once this is done, restart haproxy through systemctl restart haproxy and also restart impala service.

Now you can use impala editor in hue without any issue and you will not see any certificate error.