Halo.
Pada akhir tahun lalu, GlowByte dan Gazprombank membuat laporan bersama yang besar di konferensi Big Data Days, yang didedikasikan untuk pembuatan gudang data analitik modern berdasarkan ekosistem Cloudera Hadoop. Dalam artikel tersebut, kami berbicara tentang pengalaman membangun sistem, kesulitan dan tantangan yang harus kami hadapi dan atasi untuk mencapai kesuksesan dalam proyek.
Hadoop . — « ?». . - , - , , , , , Hadoop.
— Cloudera , “” . .
“” — . -3 .
, 2017 “ ” .
, , data driven .
. , : , . . .
:
( , );
;
;
-;
;
Self-service ;
Data Science .
. :
-
-: CRM, Real Time Offer, Next Best Offer, ;
- as is ( Data Lake);
;
;
;
( );
;
;
.
;
;
SLA;
ELT ;
Enterprise (, SAP Business Objects, SAS);
.
, , open source , — \ .
Hadoop Cloudera Data Hub
.
Cloudera Data Hub.
1.
. ETL . “” . .
Hadoop 40- - t-1 t-15 batch , real-time . :
CRM;
;
;
;
Collection;
MDM;
;
;
BI
2. “ ”
, , , . . Disaster Recovery .
science , , - . . , . . .
, , .
, , K8S, GPU .
, , ETL, , Cloudera.
CDH 5.16.1. .
Data : CPU 2x22 Cores 768Gb RAM SAS HDD 12x4Tb. HPE DL380 Cloudera Enterprise Reference Architecture for Bare Metal Deployments. “”, - , ETL . . , “100500” , , “”.
, , .
Hadoop;
(ETL);
«- –> Hadoop» «Hadoop –> Hadoop»;
;
;
.
Hadoop 1.0 , java , , , « » « ». , , SQL.
, , – SQL SQL. . SQL- « , ».
«» SQL Hadoop. Impala . Impala Cloudera Hadoop .
Impala ?
Impala – , HDFS, MapReduce, TEZ SPARK.
Impala – .
Impala Parquet, (bloom , ), . Impala , MPP Teradata GreenPlum.
Impala , , ETL .
Hadoop YARN . .
SQL , , SQL , 3-4 .
Hadoop :
- Hue, Cloudera. , SQL Excel.
Cloudera, – Impala ETL , ad-hoc BI ? - Impala « » Hive. E , .
– ETL .
ETL :
;
;
job’ .
- , , Hadoop , . Hadoop - SQL. “ ” ( , ), Hadoop “ ”.
, . metadata driven E-L-T ETL , SQL . SQL . ETL , SQL. SAS Data Integration.
ETL metadata driven ELT. airflow!
;
lineage ETL , API;
.. job’ ETL .
CI/CD
SAS DI API .
– .
– Data Replicator. Hadoop.
;
;
.. , ( ), ..
, , . , SLA Hadoop.
Data Replicator’ - Hadoop DR . , - , API. ETL , API . , DR , , «» .
, Hadoop ( Hadoop ) , , kafka, flume, ETL tool.
Hadoop . , , ( Hive) ( Impala).
– , . 247 . .. \ , ( , ..). .
, HIVE 3 ACID , , Hive ( Map Reduce), ACID Impala Hadoop .
HDFS snapshot VIEW.
HDFS, , VIEW.
VIEW, , .
– VIEW HDFS , Hadoop. UNDO Oracle, retention .
, HDFS , DDL VIEW .. metastore. .. VIEW .
HDFS Snapshot .
DataReplictor’. , , ETL API. , ETL API VIEW.
, 247 . HDFS HDFS. , 25%.
– .
;
;
, ;
Hadoop cgroups;
Hadoop;
Hadoop, YARN Impala;
Impala – .
– ETL Cloudera.
. SQL , .
900 SQL . .
, . 1,5 2 . .
, , , . Hadoop , , , open source ( Apache Big Top) .
Cloudera :
Active Directory (AD) ;
AD Sentry;
Sentry Impala HDFS;
Target VIEW ;
;
SSL . .
Hadoop ( )
;
ETL;
Hadoop ;
, , .
– .
Hadoop ( ) – , . .
. , Hadoop, , , .
ad-hoc , , .
, :
;
;
;
;
;
;
MDM;
;
;
;
;
;
;
;
;
;
.
, 177 2350 -. snappy 20 ( 100 RAW).
2010 . , . , . , , . . , , .
- -, . 40 , 550 13200 .
, Hadoop. Cloudera Data Hub - , . , .
, metastore ( ).
Impala. “” . – ( , ETL, , ) , . sqoop export. Impala .
, , decommission , , .
. 36 500 .
Cloudera Data Impact 2020 Data For Enterprise AI.
, Hadoop Cloudera . - . “ ”. “ ” , .
“”, “”, “” . . , , . «» .
time to market , data driven .
. “” , t - 3-5 - . , , CRM. , , . . - !
Hadoop. Hadoop . SQL MPP, “” , “ ” .
Cloudera Data Platform 7.1. , CDP . , , , , Impala 3.4, parquet, Zstd . Atlas Cloudera Data Flow « ». Cloudera BI - Cloudera Data Visualization.
Hadoop:
Real-time Kudu (real-time , ). Kudu, Parquet, «» SQL Impala. - .
ODS
ODS Oracle Golden Gate , Hadoop «» «» .
property Hadoop;
Arango;
Arango;
( );
( , , );
,
-
, ;
, . - , “ ”.
K8S
, . , .
:
, .
, ().