Rename Hive Database – Issue-resolution

  • Post author:

After re-naming Hive database we faced one issue. This issue we faced on existing database.

Issue Detail:  

-bash-4.2$ hdfs dfs -ls /user/hive/warehouse/demo.db
ls: Permission denied: user=demo, access=READ_EXECUTE, inode="/user/hive/warehouse/demo.db":hive:hive:drwxrwx--x

We check the following detail: Grant Role and provided access through sentry as shown below

GRANT ROLE demo_rw_role TO GROUP demo;
GRANT ALL ON DATABASE demo TO ROLE demo_rw_role;
GRANT ALL ON URI 'hdfs://nameservice1/user/hive/warehouse/demo.db' TO ROLE demo_rw_role;

Checked the user group and id association

-bash-4.2$ id demo
uid=30019(demo) gid=30019(demo) groups=30019(demo)

Checked HDFS ACL

-bash-4.2$ hdfs dfs -getfacl /user/hive/warehouse/demo.db
# file: /user/hive/warehouse/demo.db
# owner: hive
# group: hive
user::rwx
group::---
user:hive:rwx
group:hive:rwx
mask::rwx
other::--x

But Still we have below issue with user “demo”. User demo can’t access HDFS URI.

-bash-4.2$ hdfs dfs -ls /user/hive/warehouse/demo.db
ls: Permission denied: user=demo, access=READ_EXECUTE, inode="/user/hive/warehouse/demo.db":hive:hive:drwxrwx--x

Resolution:

We log in to mysql as our sentry database is mysql and did the following check. We inspected Sentry backend database and found that the location of  “demo” database was missing in AUTHZ_PATH.

Select * from AUTHZ_PATH where PATH_NAME like ‘%/user/hive/warehouse/demo.db%’

We inspected Hive metastore backend database and found that the database “demo” is missing from NOTIFICATION_LOG.

Select * from SENTRY_HMS_NOTIFICATION_ID;

We found that database demo was made before the Sentry is enabled. So, we deleted data from the following tables to create full snapshot after backing up the whole Sentry and Hive Metastore backend database:

AUTHZ_PATH
AUTHZ_PATH_SNAPSHOT_ID
AUTHZ_PATH_MAPPING
SENTRY_PATH_CHANGE
SENTRY_HMS_NOTIFICATION_ID
SENTRY_PERM_CHANGE

Before deleting data we performed following activities:

  • Took sentry DB backup
  • Stopped sentry service
  • Executed bellow commands on sentry db.
delete from AUTHZ_PATH
delete from AUTHZ_PATHS_SNAPSHOT_ID
delete from AUTHZ_PATHS_MAPPING
delete from SENTRY_PATH_CHANGE
delete from SENTRY_HMS_NOTIFICATION_ID
delete from SENTRY_PERM_CHANGE

Once data is deleted

  • Restarted Sentry
  • Rolling restart NameNodes.

After we created full snapshot and re-synchoronizing we confirmed that the ACL was synched for the directory /user/hive/warehouse/irhfmovmda.db with the following command:

hdfs dfs -getfacl /user/hive/warehouse/demo.db

Also checked Sentry backend database and found that the location of  “demo” database start appearing in AUTHZ_PATH.

mysql> select * from AUTHZ_PATH where PATH_NAME = 'user/hive/warehouse/demo.db';

FINDINGS:

The database “demo”  issue occurred was created before Sentry is enabled. In all the databases created after Sentry is enabled the issue didn’t occur. Creating full snapshot in Sentry resolved the problem. 

Tags: , , ,

Leave a Reply