Hadoop 1.1.1 ubuntu 12.10下安装和配置和启动(单机模式)

环境

os: ubuntu 12.10
java: 1.6.0_31
hadoop: 1.1.1

1.下载
从官网上下载hadoop-1.1.1.tar.gz文件
2.安装
下载后解压到hadoop-1.1.1目录，我安装的目录是

/opt/development/hadoop/1.1.1/hadoop-1.1.1

3.配置
3.1.配置jdk
打开<hadoop_home>/conf/hadoop-env.sh文件，配置JAVA_HOME路径如下

export JAVA_HOME=/usr/lib/jvm/jdk

3.2.加入环境变量

gedit ~/.profile

export HADOOP_HOME_WARN_SUPPRESS=1
export HADOOP_HOME=/opt/development/hadoop/1.1.1/hadoop-1.1.1
export PATH=$PATH:$HADOOP_HOME/bin

改完最好重启一下，使变更生效
3.3.三种配置方式
下面开始介绍如何配置，配置过程是hadoop的难点，尤其是分三种模式，建议在初学的时候掌握单机模式就够了，等熟了以后再了解集群模式。单机模式也是本文所要重点介绍的内容。
3.3.1.单机模式
core-site.xml 文件配置

<configuration>
 <property>
   <name>hadoop.tmp.dir</name>
   <value>/opt/development/hadoop/home/hadoop.tmp.dir</value>
   <description>A base for other temporary directories.</description>
 </property>

 <property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
   <description>The name of the default file system.  A URI whose
   scheme and authority determine the FileSystem implementation.  The
   uri's scheme determines the config property (fs.SCHEME.impl) naming
   the FileSystem implementation class.  The uri's authority is used to
   determine the host, port, etc. for a filesystem.</description>
 </property>

 <property>
   <name>dfs.name.dir</name>
   <value>/opt/development/hadoop/home/dfs.name.dir</value>
   <description>The name of the default file system.  A URI whose
   scheme and authority determine the FileSystem implementation.  The
   uri's scheme determines the config property (fs.SCHEME.impl) naming
   the FileSystem implementation class.  The uri's authority is used to
   determine the host, port, etc. for a filesystem.</description>
 </property>
</configuration>

hdfs-site.xml 文件配置

<configuration>
 <property>
   <name>dfs.data.dir</name>
   <value>/opt/development/hadoop/home/data</value>
   <description>Determines where on the local filesystem an DFS data node
   should store its blocks.  If this is a comma-delimited
   list of directories, then data will be stored in all named
   directories, typically on different devices.
   Directories that do not exist are ignored.
   </description>
 </property>

 <property>
   <name>dfs.replication</name>
   <value>1</value>
   <description>Default block replication. 
   The actual number of replications can be specified when the file is created.
   The default is used if replication is not specified in create time.
   </description>
 </property>
</configuration>

mapred-site.xml 文件配置

<configuration>
 <property>
  <name>mapred.job.tracker</name>
  <value>localhost:9001</value>
 </property>
</configuration>

建几个目录

jim@jim-laptop:~$ mkdir /opt/development/hadoop/home/data
jim@jim-laptop:~$ mkdir /opt/development/hadoop/home/dfs.name.dir
jim@jim-laptop:~$ mkdir /opt/development/hadoop/home/hadoop.tmp.dir
jim@jim-laptop:~$

4.启动
4.1.格式化

jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ hadoop namenode -format
13/01/02 14:57:22 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = jim-laptop/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.1.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1411108; compiled by 'hortonfo' on Mon Nov 19 10:48:11 UTC 2012
************************************************************/
13/01/02 14:57:22 INFO util.GSet: VM type       = 64-bit
13/01/02 14:57:22 INFO util.GSet: 2% max memory = 17.77875 MB
13/01/02 14:57:22 INFO util.GSet: capacity      = 2^21 = 2097152 entries
13/01/02 14:57:22 INFO util.GSet: recommended=2097152, actual=2097152
13/01/02 14:57:22 INFO namenode.FSNamesystem: fsOwner=jim
13/01/02 14:57:22 INFO namenode.FSNamesystem: supergroup=supergroup
13/01/02 14:57:22 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/01/02 14:57:22 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/01/02 14:57:22 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/01/02 14:57:22 INFO namenode.NameNode: Caching file names occuring more than 10 times 
13/01/02 14:57:22 INFO common.Storage: Image file of size 109 saved in 0 seconds.
13/01/02 14:57:23 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/opt/development/hadoop/home/hadoop.tmp.dir/dfs/name/current/edits
13/01/02 14:57:23 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/opt/development/hadoop/home/hadoop.tmp.dir/dfs/name/current/edits
13/01/02 14:57:23 INFO common.Storage: Storage directory /opt/development/hadoop/home/hadoop.tmp.dir/dfs/name has been successfully formatted.
13/01/02 14:57:23 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at jim-laptop/127.0.1.1
************************************************************/
jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ 

4.2.启动和关闭
4.2.1.启动
启动是本文的另一个难点，倒不是因为启动过程有多复杂，而是启动后可能会出现某些问题，你需要根据日志来进行有针对性的分析和处理，这是难点。

jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ start-all.sh 
starting namenode, logging to /opt/development/hadoop/1.1.1/hadoop-1.1.1/libexec/../logs/hadoop-jim-namenode-jim-laptop.out
localhost: starting datanode, logging to /opt/development/hadoop/1.1.1/hadoop-1.1.1/libexec/../logs/hadoop-jim-datanode-jim-laptop.out
localhost: starting secondarynamenode, logging to /opt/development/hadoop/1.1.1/hadoop-1.1.1/libexec/../logs/hadoop-jim-secondarynamenode-jim-laptop.out
starting jobtracker, logging to /opt/development/hadoop/1.1.1/hadoop-1.1.1/libexec/../logs/hadoop-jim-jobtracker-jim-laptop.out
localhost: starting tasktracker, logging to /opt/development/hadoop/1.1.1/hadoop-1.1.1/libexec/../logs/hadoop-jim-tasktracker-jim-laptop.out
jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ 

使用jps命令可以查看正在运行的服务

jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ jps
21550 Jps
20735 DataNode
20494 NameNode
20982 SecondaryNameNode
21337 TaskTracker
21083 JobTracker
jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ 

可以看出，一共启动了5个服务，如果启动的服务缺少某项，那就是出问题了，需要分析日志确定问题的原因。

还没完，服务起来了，也还得分析一下日志，因为是第一次启动，还得看看日志里是否有什么错误或者异常提示。 
比如我就遇到过下面几个问题
4.2.1.1.几个遇到的错误
a)
ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /opt/development/hadoop/home/data: namenode namespaceID = 840381380; datanode namespaceID = 798560325 这是提示namenode和datanode名字控件id不一致，只要把配置文件中设置的hadoop.tmp.dir,fs.default.name,dfs.name.dir所指向的三个目录中的内容都清空，然后重新格式化就可以了
b)File /opt/development/hadoop/home/hadoop.tmp.dir/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
造成这个问题的原因有很多，问题描述的也很笼统，我重启了一下服务就解决了，可能是第一次启动需要做的事情太多，多个服务没协调好吧（猜的）

c)WARN Invalid directory in dfs.data.dir: Incorrect permission for /opt/development/hadoop/home/data, expected: rwxr-xr-x, while actual:rwxrwxr-x 这是由于data目录的权限设置不正确，修改一下权限就ok了 d)org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /user/jim/demo1. Name node is in safe mode. 当前NameNode处于安全模式。在服务刚刚启动时还未就绪时是禁止对hfs做任何修改的，这时就是安全模式，等NameNode就绪后会自动离开安全模式。多等一会就好了

jim@jim-laptop:/opt/development/hadoop/home$ chmod u-w data
jim@jim-laptop:/opt/development/hadoop/home$

修改后，关闭服务，并重启服务后 4.2.2.关闭
关闭就很简单了

jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ stop-all.sh 
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode
jim@jim-laptop:/opt/development/hadoop/1.1.1/hadoop-1.1.1$ 

4.2.3.免密码启动/关闭 需要生成ssh密钥