hadoop 配置LZO压缩

hadoop 配置LZO压缩

1）先下载lzo的jar项目

https://github.com/twitter/hadoop-lzo/archive/master.zip

2）下载后的文件名是hadoop-lzo-master，它是一个zip格式的压缩包，先进行解压，然后用maven编译。生成hadoop-lzo-0.4.20.jar。

3）将编译好后的hadoop-lzo-0.4.20.jar 放入/opt/hadoop/share/hadoop/common/

4）同步hadoop-lzo-0.4.20.jar到hadoop002、hadoop003

5）core-site.xml增加配置支持LZO压缩

cd /opt/hadoop/etc/hadoop
vim core-site.xml

<property>
        <name>io.compression.codecs</name>
        <value>
        org.apache.hadoop.io.compress.GzipCodec,
        org.apache.hadoop.io.compress.DefaultCodec,
        org.apache.hadoop.io.compress.BZip2Codec,
        org.apache.hadoop.io.compress.SnappyCodec,
        com.hadoop.compression.lzo.LzoCodec,
        com.hadoop.compression.lzo.LzopCodec
        </value>
</property>

<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
</configuration>

6）同步core-site.xml到hadoop002,hadoop003

scp core-site.xml hadoop002:/opt/hadoop/etc/hadoop/
scp core-site.xml hadoop003:/opt/hadoop/etc/hadoop/

7) 启动hadoop集群

/opt/hadoop/sbin/start-all.sh

8) Web查看：http://hadoop001:50070