hadoop-2.2.0编译-白红宇

hadoop-2.2.0编译

阅读量：7227 次

发布时间：2019-06-29

本文共 6083 字，大约阅读时间需要 20 分钟。

由于从官网下载的hadoop中依赖包 native hadoop library是基于32位版本的，在64位机器上安装，会出现以下错误：

2014-05-30 19:47:49,703 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]

2014-05-30 19:47:49,887 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/data2/dfs-data should be specified as a URI in configuration files. Please update hdfs configuration.

2014-05-30 19:47:49,888 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/data3/dfs-data should be specified as a URI in configuration files. Please update hdfs configuration.

2014-05-30 19:47:50,144 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2014-05-30 19:47:50,412 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties

2014-05-30 19:47:50,482 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).

2014-05-30 19:47:50,482 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started

2014-05-30 19:47:50,485 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: File descriptor passing is disabled because libhadoop cannot be loaded.

2014-05-30 19:47:50,486 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is nobida122

2014-05-30 19:47:50,512 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:50010

2014-05-30 19:47:50,516 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s

2014-05-30 19:47:50,517 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain

java.lang.RuntimeException: Although a UNIX domain socket path is configured as /home/data3/dn_socket, we cannot start a localDataXceiverServer because libhadoop cannot be loaded.

at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:520)

at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:492)

at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:663)

at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:259)

at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1727)

at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1642)

at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1665)

at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1837)

at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1858)

2014-05-30 19:47:50,520 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1

2014-05-30 19:47:50,523 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down DataNode at nobida122/10.60.1.122

需要重新编译hadoop(http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/NativeLibraries.html)。

编译hadoop-2.2.0（centos6.4上）：

1、首先安装依赖：

(1) jdk

(2) yum install g++ autoconf automake libtool cmake zlib1g-dev pkg-config

(3) protbuf-2.5（）：下载解压后，依次运行 ./configure --prefix=/usr/local make make install

2、然后利用命令mvn package -Pdist,native -DskipTests -Dtar 进行编译，

3、编译有可能出现以下错误：

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 4:34.649s

[INFO] Finished at: Tue Jun 03 10:18:25 CST 2014

[INFO] Final Memory: 39M/584M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile (default-testCompile) on project hadoop-auth: Compilation failure: Compilation failure:

[ERROR] /opt/linshuai/tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle

[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found

[ERROR] /tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle

[ERROR] class file for org.mortbay.component.LifeCycle not found

[ERROR] /tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[98,10] error: cannot find symbol

[ERROR] symbol: method start()

[ERROR] location: variable server of type Server

[ERROR] /tardir/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[104,12] error: cannot find symbol

[ERROR] -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :hadoop-auth

问题解决 https://issues.apache.org/jira/browse/HADOOP-10110：

Index: hadoop-common-project/hadoop-auth/pom.xml===================================================================--- hadoop-common-project/hadoop-auth/pom.xml	(revision 1543124)+++ hadoop-common-project/hadoop-auth/pom.xml	(working copy)@@ -54,6 +54,11 @@          
           
     
      org.mortbay.jetty
     +      
     
      jetty-util
     +      
     
      test
     +    
    +    
    
     +      
     
      org.mortbay.jetty
            
     
      jetty
            
     
      test