Apache HDFS (Hadoop Distributed File System) adalah sistem file yang dirancang untuk menyimpan file besar, blok demi blok didistribusikan di antara node cluster komputasi.
Apache Hive adalah sistem manajemen database berdasarkan platform Hadoop.
Apache HBase adalah proyek DBMS NoSQL open source dari ekosistem Hadoop.
Apache KNOX, REST API dan gateway aplikasi untuk komponen ekosistem Apache Hadoop, menyediakan satu titik akses untuk semua koneksi HTTP ke cluster Apache Hadoop dan sistem Single Sign On (SSO) untuk layanan dan antarmuka pengguna komponen Apache Hadoop.
Apache Ranger adalah kerangka kerja untuk menyediakan, memantau, dan mengelola keamanan data ujung ke ujung di platform Hadoop
Terjemahan dari posting 2015 oleh Securing hdfs hive hbase knox ranger . Saya tidak dapat menemukan postingan yang lebih baik dan lebih baru.
pengantar
Apache Ranger memberikan pendekatan komprehensif untuk mengamankan cluster Hadoop. Ini menyediakan administrasi kebijakan keamanan terpusat untuk persyaratan keamanan perusahaan utama, termasuk otorisasi, akuntansi, dan perlindungan data.
Apache Ranger memperluas fungsionalitas dasar untuk penggunaan terkoordinasi di seluruh beban kerja Hadoop, termasuk batch, SQL interaktif, dan Hadoop waktu nyata.
Dalam tutorial ini, kita akan melihat penggunaan Apache Ranger untuk HDP 2.3 untuk mengamankan lingkungan Hadoop Anda. Kami akan membahas topik-topik berikut:
Mendukung otorisasi dan audit Knox
Kebijakan baris perintah di Hive
Kebijakan baris perintah di HBase
REST API untuk Manajer Kebijakan
Prasyarat
Hortonworks Sandbox, SSH:
Knox LDAP
Ambari http://localhost:8080/ ( — admin admin ) Knox .
Service Actions
Start
.
Knox :
Service Actions
Knox
Start Demo LDAP
.
- LDAP :
Knox
, Ranger http://localhost:6080/ -. — admin
, — admin
.
,
sudo service ranger-admin start
sandbox_knox Knox Ranger Administration Portal
, .
Knox, global knox allow
.
Sandbox for Guest
Ranger Admin .
Sandbox for Guest
SSHd ( Sandbox) CURL WebHDFS.
curl -k -u admin:admin-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
Ranger Policy Manager → Audit screen , () Knox.
CURL, guest
.
curl -k -u guest:guest-password 'https://127.0.0.1:8443/gateway/knox_sample/webhdfs/v1?op=LISTSTATUS'
{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16393,"group":"hadoop","length":0,"modificationTime":1439987528048,"owner":"yarn","pathSuffix":"app-logs","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":4,"fileId":16389,"group":"hdfs","length":0,"modificationTime":1439987809562,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17000,"group":"hdfs","length":0,"modificationTime":1439989173392,"owner":"hdfs","pathSuffix":"demo","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16398,"group":"hdfs","length":0,"modificationTime":1439987529660,"owner":"hdfs","pathSuffix":"hdp","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16394,"group":"hdfs","length":0,"modificationTime":1439987528532,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16396,"group":"hadoop","length":0,"modificationTime":1439987538099,"owner":"mapred","pathSuffix":"mr-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16954,"group":"hdfs","length":0,"modificationTime":1439988741413,"owner":"hdfs","pathSuffix":"ranger","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1440165443820,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":8,"fileId":16387,"group":"hdfs","length":0,"modificationTime":1439988397561,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}]}}
Ranger Policy Manager → Audit screen.
Ranger Knox , Knox, , Ranger.
Knox Ranger, (WebHDFS, WebHCAT . .) , / IP-
grant/revoke Hive
Ranger grant
/revoke
, Hue for Hive
. Ranger , , Hive .
Hive Ranger.
Grant
, Hive
. beeline,
beeline -u "jdbc:hive2://sandbox.hortonworks.com:10000/default" -n it1 -p it1-d org.apache.hive.jdbc.HiveDriver
GRANT
grant select, update on table xademo.customer_details to user network1;
:
Ranger Administration Portal → Audit
, it1
.
Ranger it1
. Ranger , .
beeline , .
GRANT select, update on table xademo.customer_details to user network1;
, , / Ranger Admin Portal → Policy Manager. , , .
?
Ranger GRANT/REVOKE
Hive
. Hive
(Hiveserver2
).
GRANT
REVOKE
.
grant/revoke HBase
Ranger grant/revoke
, Hbase
. Hive, Ranger Hbase .
, , HBase Ambari — http://127.0.0.1:8080 ( admin
).
, Service Actions
.
Grant Hbase.
HBase Global Allow
Ranger Administration Portal — .
HBase it1
su - it1 [it1@sandbox ~]$ hbase shell
grant, mktg1
, iemployee
.
hbase(main):001:0> grant 'mktg1', 'RWC', 'iemployee'
, :
Ranger Administration Portal→ Policy Manager , admin
it1
.
HBase.
hbase(main):006:0> grant 'mktg1', 'RWC', 'iemployee' 0 row(s) in 0.8670 seconds
HBase Ranger Policy Administration. iemployee
, .
, Ranger. HBase
hbase(main):007:0> revoke 'mktg1', 'iemployee' 0 row(s) in 0.4330 seconds
,
?
Ranger GRANT/REVOKE
Hbase . .
GRANT
REVOKE .
REST APIs
Ranger REST API. API , , .
REST API
CURL. API hadoopdev-testing-policy2
HDFS sandbox_hdfs
curl -i --header "Accept:application/json" -H "Content-Type: application/json" --user admin:admin -X POST http://127.0.0.1:6080/service/public/api/policy -d '{ "policyName":"hadoopdev-testing-policy2","resourceName":"/demo/data/test","description":"Testing policy for /demo/data/test","repositoryName":"sandbox_hdfs","repositoryType":"HDFS","permMapList":[{"userList":["mktg1"],"permList":["Read"]},{"groupList":["IT"],"permList":["Read"]}],"isEnabled":true,"isRecursive":true,"isAuditEnabled":true,"version":"0.1.0","replacePerm":false}'
hadoopdev-testing-policy2
.
URL- http://127.0.0.1:6080/index.html#!/hdfs/1/policy/26
.
CURL, API.
curl -i --user admin:admin -X GET http://127.0.0.1:6080/service/public/api/policy/26
?
REST API. API , REST API Ranger.
Semoga dalam tur Ranger yang memusingkan ini, Anda merasakan kesederhanaan dan kekuatan Ranger untuk manajemen keselamatan.