azkaban 安裝

luqiuwen 發(fā)布于2019-08-15 12:21 / 633人閱讀

摘要：否則沒有配置環(huán)境會報錯。重新編譯打包就可以

azkaban

azkaban 是一個用Java開發(fā)的開源調(diào)度工具workflow,同類型的工具有用Python開發(fā)的 airflow 和 luigi.

build

在我的windows上無法build成功，我在CentOS上build成功了

git clone https://github.com/azkaban/azkaban.git
cd azkaban
./gradlew build installDist

run

我的需求比較簡單 solo server 就可以滿足要求

cd azkaban-solo-server/build/install/azkaban-solo-server
bin/azkaban-solo-start.sh
bin/azkaban-solo-shutdown.sh

config

使用MySQL數(shù)據(jù)庫，MySQL新建用戶

CREATE DATABASE azkaban;
CREATE USER "azkaban"@"%" IDENTIFIED BY "azkaban";
GRANT ALL PRIVILEGES ON azkaban.* to "azkaban"@"%";

修改配置文件

#設(shè)置為上海時間(東八區(qū))，否則會按美國時間執(zhí)行
default.timezone.id=Asia/Shanghai

#注釋掉默認的H2數(shù)據(jù)庫配置后，配置MySQL數(shù)據(jù)庫
database.type=mysql
mysql.port=3306
mysql.host=192.168.0.1
mysql.database=azkaban
mysql.user=username
mysql.password=password
mysql.numconnections=10


#配置告警郵件
[email protected]
mail.host=smtp.163.com
mail.user=mailUsername
mail.password=mailPassword

啟動的時候會自動創(chuàng)建數(shù)據(jù)庫

plugin

https://github.com/azkaban/az...

使用ant編譯，需要安裝ant

sudo yum install ant
sudo yum install ant-junit

sudo apt-get install ant
sudo apt-get install ant-optional

git clone https://github.com/azkaban/azkaban-plugins.git

# step 1
cd azkaban-plugins
cd plugins
cd hadoopsecuritymanager
ant package

# step 2
cd ..
cd crypto
ant package


# step 3
cd ..
cd jobtype
ant package

azkaban-plugins/dist/jobtype/packages/azkaban-jobtype-3.0.0.tar.gz
解壓到
azkaban/azkaban-solo-server/build/install/azkaban-solo-server/plugins/jobtypes

commonprivate.properties
common.properties

execute.as.user 設(shè)置為false，否者會報錯

# configs for jobtype security settings
execute.as.user=false
azkaban.native.lib=

我目前只想運行java job，不想使用hadoop，修改源代碼，可以不使用hadoop。否則沒有配置hadoop 環(huán)境會報錯。

--- a/plugins/jobtype/src/azkaban/jobtype/JavaJob.java
+++ b/plugins/jobtype/src/azkaban/jobtype/JavaJob.java
@@ -75,8 +75,8 @@ public class JavaJob extends JavaProcessJob {
     classPath.add(getSourcePathFromClass(JavaProcessJob.class));
     classPath.add(getSourcePathFromClass(SecurityUtils.class));
 
-    classPath.add(HadoopConfigurationInjector.getPath(getJobProps(),
-        getWorkingDirectory()));
+    //classPath.add(HadoopConfigurationInjector.getPath(getJobProps(),
+    //    getWorkingDirectory()));
 
     String loggerPath = getSourcePathFromClass(org.apache.log4j.Logger.class);
     if (!classPath.contains(loggerPath)) {
@@ -157,8 +157,8 @@ public class JavaJob extends JavaProcessJob {
 
   @Override
   public void run() throws Exception {
-    HadoopConfigurationInjector.prepareResourcesToInject(getJobProps(),
-        getWorkingDirectory());
+    //HadoopConfigurationInjector.prepareResourcesToInject(getJobProps(),
+    //    getWorkingDirectory());
     super.run();
   }
 }

--- a/plugins/jobtype/src/azkaban/jobtype/JavaJobRunnerMain.java
+++ b/plugins/jobtype/src/azkaban/jobtype/JavaJobRunnerMain.java
@@ -104,7 +104,7 @@ public class JavaJobRunnerMain {
       }
       _logger.info("Class name " + className);
 
-      HadoopConfigurationInjector.injectResources(new Props(null, props));
+      //HadoopConfigurationInjector.injectResources(new Props(null, props));
 
       // Create the object using proxy
       if (SecurityUtils.shouldProxy(props)) {

重新編譯打包就可以