2.安装Spark与Python练习

一、安装Spark

1. 检查基础环境hadoop,jdk
2. 配置文件
3. 环境变量
4. 试运行Python代码

 

 

 

 

 

二、Python编程练习：英文文本的词频统计

1. 准备文本文件
2. 读文件
3. 预处理：大小写，标点符号，停用词
4. 分词
5. 统计每个单词出现的次数
6. 按词频大小排序
7. 结果写文件

with open("Under the Red Dragon.txt", "r") as f:
    text=f.read()
    text = text.lower()
    for ch in '!@#$%^&*(_)-+=\\[]}{|;:\'\"`~,<.>?/':
        text=text.replace(ch," ")
    
    
words = text.split() # 以空格分割文本
stop_words = []
with open('stop_words.txt','r') as f: # 读取停用词文件
    for line in f:
        stop_words.append(line.strip('\n'))
afterwords=[]


for i in range(len(words)):
    z=1
    for j in range(len(stop_words)):
    
        if words[i]==stop_words[j]:
            continue
        else:
            if z==len(stop_words):
                afterwords.append(words[i])
                break
            z=z+1
            continue
            

counts = {}
for word in afterwords:
    counts[word] = counts.get(word,0) + 1
items = list(counts.items())
items.sort(key=lambda x:x[1],reverse=True)
# 输出前10
for i in range(10):
    word, count = items[i]
    print("{0:<10}{1:>5}".format(word, count))  # 打印前十个元素