hadoop伪分布式搭建

0.准备

虚拟机:centos7 mini(已关闭防火墙,配置本机免密,映射)
java环境:jdk1.8
远程连接工具:xshell6 家庭/学校版
hadoop版本:hadoop2.8.5
操作用户:root

1.下载解压
所在目录: /opt/soft

[root@localhost soft]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz
[root@localhost soft]# tar -zxvf hadoop-2.8.5.tar.gz -C ../app/
[root@mini app]# mv hadoop-2.8.5/ hadoop2.8.5

2.环境变量
在 /etc/profile追加

#set hadoop environment
export HADOOP_HOME=/opt/app/hadoop2.8.5
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

配置hadoop.env.sh

[root@mini hadoop2.8.5]# vi etc/hadoop/hadoop-env.sh 
# The java implementation to use.
export JAVA_HOME=/opt/app/jdk1.8

配置core-site.xml

[root@mini hadoop2.8.5]# vi etc/hadoop/core-site.xml
<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://mini:9000</value>
</property>

<property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/app/hadoop2.8.5/data/tmp</value>
 </property>
</configuration>

配置hdfs-site.xml
在格式化(hadoop namenode -format)之前,要自己创建好/home/hadoop/data/datanode和/home/hadoop/data/namenode文件夹

[root@mini hadoop2.8.5]# vi etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

    <property>
        <name>dfs.name.dir</name>
        <value>file:///home/hadoop/data/namenode</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>file:///home/hadoop/data/datanode</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>mini:50090</value>
    </property>
</configuration>

配置yarn-site.xml
(7天删除日志)

[root@mini hadoop2.8.5]# vi etc/hadoop/yarn-site.xml
<property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
</property>
<property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
</property>
<property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>604800</value>
</property>

配置mapred-site.xml

[root@mini hadoop2.8.5]# vi etc/hadoop/mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop01:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop01:19888</value>
    </property>
</configuration>

格式化

[root@mini hadoop2.8.5]# hadoop namenode -format
在倒数七八行的样子出现
common.Storage: Storage directory /home/hadoop/data/namenode has been successfully formatted.
成功

启动

start-all.sh