Spark本地测试异常之 System memory 259522560 must be at least 471859200.

解决Spark本地测试异常之 System memory 259522560 must be at least 471859200

• 一、异常如下
• 二、抛出异常原因
• 三、解决办法

一、异常如下

java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
	at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:332)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
	at com.temptag.app.TagGenerator$.main(TagGenerator.scala:14)
	at com.temptag.app.TagGenerator.main(TagGenerator.scala)
19/04/08 20:31:19 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
	at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
	at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:332)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
	at com.temptag.app.TagGenerator$.main(TagGenerator.scala:14)
	at com.temptag.app.TagGenerator.main(TagGenerator.scala)

二、抛出异常原因

Spark是非常依赖内存的计算框架，在Win环境下使用local模式时，实际上是使用多线程的形式模拟集群进行计算，因而对于计算机的内存有一定要求，这是典型的因为计算机内存不足而抛出的异常。

三、解决办法

在设置SparkContext时，多设置一个如下参数，来指定分配给Spark的内存，这样问题就解决啦！

 val sparkConf = new SparkConf()
        .set("spark.testing.memory","2147480000")