Spark实现WordCount案例

RDD操作实现

• 1.文本文件：
• • 方式一：没用正则处理的
  • 方式二：添加正则表达式
  • 方式三：利用sortBy()进行排序
• Spark sql实现
• • 导入隐式转换
  • createOrReplaceTempView使用

1.文本文件：

Preface
“The Forsyte Saga” was the title originally destined for that part of it which is called “The Man of Property”; 
and to adopt it for the collected chronicles of the Forsyte family has indulged the Forsytean tenacity that is in all of us. 
The word Saga might be objected to on the ground that it connotes the heroic and that there is little heroism in these pages.
 But it is used with a suitable irony; and, after all, this long tale, though it may deal with folk in frock coats, furbelows 

方式一：没用正则处理的

val file = spark.sparkContext.textFile("file:///F:\\dataset\\The_Man_of_Property.txt")
    file.flatMap(line=>line.split(" ")).map((_,1)).reduceByKey(_+_).sortBy(_._2,ascending = false)
      .take(10).foreach(println(_))

方式二：添加正则表达式

val p = "[0-9a-zA-Z]+".r
    file.flatMap(line => line.split(" ")).map(x=>(p.findAllIn(x).mkString(""),1)).reduceByKey(_+_)
    .take(10).foreach(println)

结果：
(welshed,1)
(mattered,1)
(someone,10)
(disregarded,2)
(jowl,2)
(bone,1)
(House,6)

方式三：利用sortBy()进行排序

sortBy(_._2,ascending = false) 
默认升序
ascending = false 代表：降序

val p = "[0-9a-zA-Z]+".r
    file.flatMap(line => line.split(" ")).map(x=>(p.findAllIn(x).mkString(""),1)).reduceByKey(_+_)
    .sortBy(_._2,ascending = false)
    .take(10).foreach(println)
结果:
(the,5168)
(of,3425)
(to,2810)
(and,2686)
(a,2564)             

Spark sql实现

导入隐式转换

 import spark.implicits._
//hdfs上文件
    val file = spark.sparkContext.textFile("/data/The_Man_of_Property.txt")
    val p = "[0-9a-zA-Z]+".r
    val dataFrame = file.flatMap(line=>line.split(" ")).map(x=>(p.findAllIn(x).mkString(""),1)).reduceByKey(_+_).toDF("word","count")

    dataFrame.createOrReplaceTempView("Property")


    dataFrame.show(5)
+-----------+-----+                                                             
|       word|count|
+-----------+-----+
|    welshed|    1|
|   mattered|    1|
|    someone|   10|
|disregarded|    2|
|       jowl|    2|
+-----------+-----+
only showing top 5 rows

createOrReplaceTempView使用

createOrReplaceTempView：创建临时视图，此视图的生命周期与用于创建此数据集的[SparkSession]相关联。
dataFrame.createOrReplaceTempView("Property")
    //指定列名
scala> val res = spark.sql("select * from Property ")
21/01/12 20:25:20 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
res: org.apache.spark.sql.DataFrame = [word: string, count: int]

scala> res.show(5)
+-----------+-----+
|       word|count|
+-----------+-----+
|    welshed|    1|
|   mattered|    1|
|    someone|   10|
|disregarded|    2|
|       jowl|    2|
+-----------+-----+
only showing top 5 rows