pengantar
Izinkan saya menjelaskan akordeon saya segera. Ya, internet penuh dengan manual. Ya, penuh dengan penelusuran berbasis giliran. Ya, Anda dapat mengatakan bahwa semuanya dikunyah. Tetapi secara khusus dalam kasus saya, seperti yang selalu terjadi, ada beberapa "tetapi":
Ada manual tentang cara mengatur bundel NiFi dan NiFi Registry dengan otentikasi dan otorisasi diaktifkan. Tapi ... sertifikat yang ditandatangani sendiri digunakan.
Ada manual terpisah tentang cara memasang sekrup pada sertifikat komersial untuk NiFi; masing-masing untuk NiFi Registry "kagbe is same". Tetapi otentikasi dan otorisasi bersama akan terjadi dengan penggunaan SSL dua arah ... dan kami memiliki LDAP ... dan kemudian Anda tidak akan dapat memastikan koherensi pasangan manis hanya dengan menggunakan direktori eksternal pada intuisi Anda.
Ada manual untuk menghubungkan dengan LDAP untuk NiFi dan NiFi Registry. Tidak ... seperti pada "tetapi" sebelumnya, pertanyaan muncul, bagaimana bertahan hanya dengan LDAP, karena kita masih memiliki NiFi Cli, tetapi dia tidak tahu bagaimana melakukan LDAP.
Dengan kata lain, semua manual memiliki sedikit nuansa: hanya mencakup skenario yang paling sederhana. Tidak ada dokumentasi untuk bundel kompleks. Selain itu, dalam proses menyiapkan bundel, saya menemui kesulitan yang hanya dihadapi beberapa kali di web borjuis dan semuanya tanpa jawaban, atau jawabannya tidak relevan.
Apa yang kita miliki dan apa yang ingin kita dapatkan
Kita punya:
Sertifikat SSL / TLS wildcard komersial;
Server direktori yang dapat melakukan LDAP (dalam kasus saya, Apache DS);
Server NiFi tempat pengembangan berlangsung;
Server NiFi di lingkungan grosir;
Server Registri NiFi.
Kami ingin:
Otentikasi, otorisasi dan HTTPS. Selain itu, NiFi tidak mengizinkan Anda untuk mengaktifkan mekanisme otentikasi bawaan tanpa HTTPS;
Manajemen pengguna terpusat pada server direktori;
Aliran versi dan proses-grup dengan cara yang mirip dengan gith;
.
:
Apache DS;
Java Keystore Java Truststore, HTTPS;
LDAP NiFi;
LDAP NiFi Registry;
;
NiFi NiFi Registry;
NiFi NiFi Cli.
wildcard : *.love.you;
FQDN Apache DS: auth.love.you;
Apache DS: uid=admin,ou=system; : secret;
Apache DS love.you: dc=love,dc=you;
Apache DS nifiAdmin nifiUser NiFi NiFi Registry;
Apache DS nifiAdmins nifiUsers NiFi NiFi Registry;
FQDN NiFi, : nifi-deve.love.you;
FQDN NiFi Registry: nifi-re.love.you;
FQDN () NiFi: nifi.love.you;
, NiFi: $NIFI_HOME;
, NiFi Registry: $NIFI_REGISTRY_HOME;
, NiFi Toolkit: $NIFI_TOOLKIT_HOME;
, : /love/conf/cert;
PKCS12, Java Keystore Java Truststore:
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
Apache DS
, , Apache DS . . Azure AD, " ", NiFi, AD' . Apache DS , , Apache.
10389 . Apache Directory Studio. , , .
New connection:
:
: uid=admin,ou=system secret. Check Authentication:
.
. - member objectclass inetOrgPerson. LDAP Browser' ou=schema Reload Entry:
cn=inetorgperson ou=objectClasses. m-oid=2.16.840.1.113730.3.2.2. Open With -> LDIF Entry Editor:
m-may: member Ctrl+S:
Open Configuration:
Advanced Partition configuration...
Add, ID Suffix. Ctrl+S:
: . . . :
. - LDIF . - import.ldif .
import.ldif
dn: ou=people,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: people
dn: ou=groups,dc=love,dc=you
objectclass: organizationalUnit
objectClass: extensibleObject
objectclass: top
ou: groups
dn: cn=nifiUsers,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiUsers
uniqueMember: cn=nifiUser,ou=people,dc=love,dc=you
dn: cn=nifiAdmins,ou=groups,dc=love,dc=you
objectClass: groupOfUniqueNames
objectClass: top
cn: nifiAdmins
uniqueMember: cn=nifiAdmin,ou=people,dc=love,dc=you
dn: cn=nifiUser,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiUser
description: A nifiUser user
sn: nifiUser
uid: nifiUser
mail: nifiUser@love.you
userpassword: password
member: cn=nifiUsers,ou=groups,dc=love,dc=you
dn: cn=nifiAdmin,ou=people,dc=love,dc=you
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: nifiAdmin
description: A nifiAdmin user
sn: nifiAdmin
uid: nifiAdmin
mail: nifiAdmin@love.you
userpassword: password
member: cn=nifiAdmins,ou=groups,dc=love,dc=you
dn: cn=*.love.you,ou=people,dc=love,dc=you
objectClass: person
objectClass: top
cn: *.love.you
sn: *.love.you
LDIF Import:
:
:
cn=*.love.you
, ou=people cn=*.love.you, objectClass: person:
, : , *.love.you. NiFi Registry.
Java Keystore Java Truststore
:
PEM-: fullchain.crt;
PEM-: private.key;
certificate authority (CA) PEM-: cacert.pem;
, . , .
. :
-----BEGIN CERTIFICATE----- ... ... ... -----END CERTIFICATE-----
.
, .
, , , ( ) . CA .
Java Keystore . PKCS12:
openssl pkcs12 -export -out nifi.p12 -inkey private.key -in fullchain.crt -name nifi-key
PKCS12 . .
Java Keystore PKCS12 :
keytool -importkeystore -srckeystore nifi.p12 -srcstoretype pkcs12 -srcalias nifi-key -destkeystore keystore.jks -deststoretype jks -destalias nifi-key
, . Java Keystore, .
Java Truststore:
keytool -importcert -alias nifi-cert -file cacert.pem -keystore truststore.jks
Java Truststore. . Keystore Truststore. . .
JKS NiFi, , NiFi NiFi Registry, /love/conf/cert/;
NiFi , , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi-deve.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi-deve.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
: nifi.web.http.host nifi.web.http.port. , , NiFi .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi , $NIFI_HOME/config/nifi.properties, .
nifi.properties
# Site to Site properties
nifi.remote.input.host=nifi.love.you
nifi.remote.input.secure=true
nifi.remote.input.socket.port=10443
# web properties #
nifi.web.http.host=
nifi.web.http.port=
nifi.web.https.host=nifi.love.you
nifi.web.https.port=9443
# security properties #
nifi.security.keystore=/love/conf/cert/keystore.jks
nifi.security.keystoreType=jks
nifi.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.truststore=/love/conf/cert/truststore.jks
nifi.security.truststoreType=jks
nifi.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider
# Identity Mapping Properties #
nifi.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
: . .
21 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
NiFi Registry, $NIFI_REGISTRY_HOME/config/nifi-registry.properties, .
nifi-registry.properties
# web properties #
nifi.registry.web.http.host=
nifi.registry.web.http.port=
nifi.registry.web.https.host=nifi-re.love.you
nifi.registry.web.https.port=8443
# security properties #
nifi.registry.security.keystore=/abc/conf/cert/keystore.jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.truststore=/abc/conf/cert/truststore.jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
nifi.registry.security.needClientAuth=false
nifi.registry.security.identity.provider=ldap-identity-provider
# Identity Mapping Properties #
nifi.registry.security.identity.mapping.pattern.dn=^CN=(.*?)$
nifi.registry.security.identity.mapping.value.dn=$1
nifi.registry.security.identity.mapping.transform.dn=NONE
: . .
16 - , .
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
LDAP NiFi
LDAP : $NIFI_HOME/conf/login-identity-providers.xml $NIFI_HOME/conf/authorizers.xml
login-identity-providers.xml . LDAP- . .
login-identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<loginIdentityProviders>
<provider>
<identifier>ldap-provider</identifier>
<class>org.apache.nifi.ldap.LdapProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</loginIdentityProviders>
, 27 29. 27 , , LDAP-. {0} - , User NiFi; uid - , , . , nifiUser.
, . ApacheDS.
29 - : (USE_USERNAME), DN (USE_DN). , , . DN .
authorizers.xml , NiFi "" . .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
<property name="Node Group"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
:
31 - objectClass ApacheDS, ;
35 - , ;
39 - objectClass, ;
50 - , ( identifier: <identifier>ldap-user-group-provider</identifier> <userGroupProvider>);
52 - , - "" .
NiFi nifiAdmin , ldif-.
LDAP NiFi Registry
NiFi Registry NiFi. $NIFI_REGISTRY_HOME/conf/identity-providers.xml .
identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<identityProviders>
<provider>
<identifier>ldap-identity-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Search Filter">(uid={0})</property>
<property name="Identity Strategy">USE_USERNAME</property>
<property name="Authentication Expiration">12 hours</property>
</provider>
</identityProviders>
$NIFI_REGISTRY_HOME/conf/authorizers.xml .
authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">SIMPLE</property>
<property name="Manager DN">uid=admin,ou=system</property>
<property name="Manager Password">secret</property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://auth.love.you:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">1 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=people,dc=love,dc=you</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute">member</property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,dc=love,dc=you</property>
<property name="Group Object Class">groupOfUniqueNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute"></property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
<property name="User Group Provider">ldap-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifiAdmin</property>
<property name="NiFi Group Name"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers>
NiFi Registry nifiAdmin , ldif-.
Identity Mapping Properties nifi.properties nifi-registry.properties.
, NiFi NiFi Registry.
, "Proxy user requests". , , . , " " " ", , . , , . , . . .
? , NiFi NiFi Registry , . NiFi Registry NiFi , NiFi Java Keystore , , NiFi Registry. , wildcard , CN=*.love.you.
!
, NiFi . , , CN . . , CN cn. NiFi .
. ApacheDS cn . . . : . NiFi , : nifiUser, nifiAdmin *.love.you. cn=....
, , , , .
, , ....mapping.pattern.dn capture group, ....mapping.value.dn . CN=*.love.you *.love.you, ApacheDS.
NiFi NiFi Registry
NiFi . , , NiFi . , .
Operate:
Policies , :
, *.love.you proxy user requests:
Controller Settings:
REGISTRY CLIENTS:
, + , , :
NiFi Registry, , :
USERS :
:
Bucket Test
NiFi. process group Test. process group Version -> Start version control:
:
SAVE process group :
process group .
NiFi NiFi Cli
, process group flow Registry, () NiFi.
NiFi Toolkit NiFi .
: $NIFI_TOOLKIT_HOME/config/reg.properties $NIFI_TOOLKIT_HOME/config/nifi-prod.properties:
NiFi Registry:
reg.properties
baseUrl=https://nifi-re.love.you:8443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
() NiFi:
nifi-prod.properties
baseUrl=https://nifi.love.you:9443
keystore=/love/conf/cert/keystore.jks
keystoreType=jks
keystorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
keyPasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
truststore=/love/conf/cert/truststore.jks
truststoreType=jks
truststorePasswd=sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
proxiedEntity=nifiAdmin
sF9ZCZvX#9rKHQY-Jv1PSx%IXpeGZ4@nwrO
, .
:
NiFi Registry ID , flow.
flow ID ID flow, () NiFi.
flow;
flow () NiFi.
$NIFI_TOOLKIT_HOME/bin :
:
./cli.sh registry list-buckets -p ../conf/reg.properties
:
# Name Id Description
- ---- ------------------------------------ -----------
1 Test 84744b22-0e2e-4342-b6ab-6658ff0e54ec (empty)
flow 84744b22-0e2e-4342-b6ab-6658ff0e54ec:
./cli.sh registry list-flows -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -p ../conf/reg.properties
:
# Name Id Description
- ------------------ ------------------------------------ ------------------
1 PG versioning test bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 PG versioning test
flow:
./cli.sh registry list-flow-versions -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -p ../conf/reg.properties
:
Ver Date Author Message
--- -------------------------- --------- ------------------
1 Thu, Apr 22 2021 09:28 UTC nifiAdmin PG versioning test
flow () NiFi:
./cli.sh nifi pg-import -b 84744b22-0e2e-4342-b6ab-6658ff0e54ec -f bc9a3ff4-2d9c-44b0-bd8e-d86b51d3bac6 -fv 1 -p ../conf/nifi-prod.properties
:
f90df705-0178-1000-04fe-45ba27e76f99
- ID process group () NiFi:
, HowTo . , , (, NiFi Registry), "TRACE" NiFi NiFi Registry . , , .
Sejauh menyangkut CI / CD, saya tidak menggambar seluruh rantai. Setiap orang memiliki pola dan skenario mereka sendiri, dan jalur pipa itu sendiri akan menjadi offtopic. Dan contoh di atas NiFi Cli, menurut saya, adalah titik awal yang baik untuk mengimplementasikan skema pengiriman Anda sendiri.
Terima kasih kepada semua orang yang membacanya dan saya harap teksnya bermanfaat.