Spark环境搭建与测试

概述：本文主要介绍Spark环境搭建过程，并实现塞缪尔·厄尔曼《青春》的词频统计，如果要测试HDFS上的词频统计，需要提取搭建HDFS环境，并将要统计文本上传到HDFS中。

1、下载并解压Spark

https://archive.apache.org/dist/spark/spark-2.4.2/spark-2.4.2.tgz

2、编译方式参考官网

#pom.xml中添加https://repository.cloudera.com/artifactory/cloudera-repos/
#在解压后的spark根路径下执行 下述命令仅供参考
./dev/make-distribution.sh --name 2.6.0-cdh5.7.0 --pip --tgz -Pyarn
     -Phadoop-2.6 -Phive -Phive-thriftserver
#免编译 直接下载hadoop对应spark版本
https://archive.apache.org/dist/spark/spark-2.4.2/spark-2.4.2-bin-hadoop2.6.tgz

3、修改配置

将下载好的Spark压缩包解压好后需要做如下配置

#在spark conf路径下
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
#添加如下内容（Standalone模式）
export JAVA_HOME=/root/app/jdk1.8.0_201
export HADOOP_HOME=/root/app/hadoop-2.6.0-cdh5.7.0
SPARK_MASTER_HOST=hadoop
SPARK_WORKER_CORES=2
SPARK_WORKER_MEMORY=2g
SPARK_WORKER_INSTANCES=2
​
#sbin下启动
./start-all.sh
#查看日志
cat /root/app/spark-2.4.2-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop.out

4、Spark词频统计

val file=sc.textFile("file:///root/app/hadoop-2.6.0-cdh5.7.0/bin/Youth.txt")
val wordCounts = file.flatMap(line => line.split(",")).map(word => (word,1)).reduceByKey(_+_)
wordCounts.collect()
wordCounts.foreach(println)
​
#读取本地文件
sc.textFile("hdfs://hadoop:8020/input/wc/Youth.txt").flatMap(line => line.split(",")).map(word => (word,1)).reduceByKey(_+_).collect().foreach(println)
​
#读取HDFS文件
​
sc.textFile("hdfs://hadoop:8020/input/wc/Youth.txt").flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_).collect().foreach(println)

结果如下：