寒假学习进度-9spark streaming编程初级实践

1.安装Flume

下载地址：

http://www.apache.org/dyn/closer.lua/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz

（1）解压安装包

sudo tar -zxvf apache-flume-1.7.0-bin.tar.gz -C /usr/local 
cd /usr/local 
sudo mv ./apache-flume-1.7.0-bin ./flume 
sudo chown -R hadoop:hadoop ./flume 

（2）配置环境变量

sudo vim ~/.bashrc 
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64; 
export FLUME_HOME=/usr/local/flume 
export FLUME_CONF_DIR=$FLUME_HOME/conf 
export PATH=$PATH:$FLUME_HOME/bin 

修改 flume-env.sh 配置文件：

cd /usr/local/flume/conf 
sudo cp ./flume-env.sh.template ./flume-env.sh 
sudo vim ./flume-env.sh 
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64; 

（3）查看flume版本信息

cd /usr/local/flume 
./bin/flume-ng version 

2.使用 Avro 数据源测试 Flume 

3.使用 netcat 数据源测试 Flume 

4.使用 Flume 作为 Spark Streaming 数据源 

------------------------------------------- 
Time: 1488029430000 ms 
------------------------------------------- 
Received 0 flume events. #这里省略了其他屏幕信息 
------------------------------------------- 
Time: 1488029432000 ms 
------------------------------------------- 
Received 8 flume events. #这里省略了其他屏幕信息 
------------------------------------------- 
Time: 1488029434000 ms 
------------------------------------------- 
Received 21 flume events.