2021-03-18

Spark 单词计数

先启动Hadoop,确保9000端口能被访问
1.进入/usr/local/src目录解压,重命名为spark
tar -xvf spark-3.1.1-bin-hadoop3.2
ln -sv ./src/spark-3.1.1-bin-hadoop3.2 ./spark
2.进入/spark/conf目录
cd /usr/local/spark/conf
cp spark-env.sh.template spark-env.sh
vi spark-env.sh
export SPARK_MASTER_HOST=192.168.43.100
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=2g
export SPARK_MASTER_WEBUI_PORT=8888
export JAVA_HOME=/usr/local/java/jdk1.8.0_161
export SPARK_disT_CLAsspATH=$(/usr/local/src/hadoop-3.0.3/bin/hadoop classpath)

3.cp workers.template workers
vi workers(输入ip)
192.168.43.100

4.vi /etc/profile

export SPARK_HOME=/usr/local/src/spark-3.1.1-bin-hadoop3.2
export SPARK_CONF_DIR=$SPARK_HOME/conf
export PATH=$PATH:$SPARK_HOME/bin
source /etc/profile

4.验证
cd /usr/local/spark
./bin/run-example SparkPi ##如果输出信息中存在Pi is roughly 说明测试成功。

单词计数(本地模式)
cd /usr/local/spark/bin
./spark-shell
var textfile=sc.textFile(“file:///root/task2”);
var count = textfile.flatMap(line => line.split(" “)).map(word => (word,1)).reduceByKey(+)
count.collect()
退出spark shell
单词计数(HDFS模式)
var textfile=sc.textFile(“hdfs://192.168.43.100:9000/sunhao/task2”);
var count = textfile.flatMap(line => line.split(” ")).map(word => (word,1)).reduceByKey(+)
count.collect()
退出spark shell

6.启动spark
cd /usr/local/spark/sbin
./start-all.sh
关闭spark
cd /usr/local/spark/sbin
./stop-all.sh
7.hdfs常见命令
创建目录
hdfs dfs -mkdir /sunhao
上传文件
hdfs dfs -put /root/task2 /sunhao
查看信息
hdfs dfsadmin -report