20/03/25 10:28:07 WARN UserGroup@R_715_4045@ion: PriviledgedActionException as:root (auth:SIMPLE) cause:org.apache.spark.SparkException: Exception thrown in awaitResult Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.security.UserGroup@R_715_404[email protected](UserGroup@R_715_404[email protected]:1930) at org.apache.spark.deploy.SparkHadoopUtil.runAssparkUser(SparkHadoopUtil.scala:66) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:284) at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala) Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77) at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100) at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:202) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:67) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroup@R_715_404[email protected](UserGroup@R_715_404[email protected]:1917) ... 4 more Caused by: java.io.IOException: Failed to connect to localhost/127.0.0.1:41640 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197) at org.apache.spark.rpc.netty.OutBox$anon$1.call(OutBox.scala:191) at org.apache.spark.rpc.netty.OutBox$anon$1.call(OutBox.scala:187) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:41640 at sun.nio.ch.socketChannelImpl.checkConnect(Native Method) at sun.nio.ch.socketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:257) at io.netty.channel.nio.AbstractNioChannel$AbstractNIoUnsafe.finishConnect(AbstractNioChannel.java:291) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:640) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) ... 1 more LogType:stderr Log Upload Time:Wed Mar 25 10:31:22 +0800 2020 LogLength:63452 Log Contents: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yarn/nm/usercache/root/filecache/4996/__spark_libs__5125379819107399169.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 20/03/25 10:29:18 INFO SignalUtils: Registered signal handler for TERM 20/03/25 10:29:18 INFO SignalUtils: Registered signal handler for HUP 20/03/25 10:29:18 INFO SignalUtils: Registered signal handler for INT 20/03/25 10:29:19 INFO ApplicationMaster: Preparing Local resources 20/03/25 10:29:19 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1585020115190_0150_000002 20/03/25 10:29:19 INFO SecurityManager: Changing view acls to: yarn,root 20/03/25 10:29:19 INFO SecurityManager: Changing modify acls to: yarn,root 20/03/25 10:29:19 INFO SecurityManager: Changing view acls groups to: 20/03/25 10:29:19 INFO SecurityManager: Changing modify acls groups to: 20/03/25 10:29:19 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, root); groups with view permissions: Set(); users with modify permissions: Set(yarn, root); groups with modify permissions: Set() 20/03/25 10:29:19 INFO ApplicationMaster: Starting the user application in a separate Thread 20/03/25 10:29:19 INFO ApplicationMaster: Waiting for spark context initialization... 20/03/25 10:29:21 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered! 20/03/25 10:29:21 INFO RMProxy: Connecting to ResourceManager at df1/172.16.252.11:8030 20/03/25 10:29:28 WARN YarnAllocator: Container marked as Failed: container_1585020115190_0150_02_000003 on host: df3. Exit status: 1. Diagnostics: Exception from container-launch. Container id: container_1585020115190_0150_02_000003 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runcommand(Shell.java:604) at org.apache.hadoop.util.Shell.run(Shell.java:507) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:789) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Container exited with a non-zero exit code 1
问题回顾： 编写好程序，在本地idea远程访问测试环境进行测试， 一切正常。 提交程序到测试环境，使用spark local模式执行程序 ， 一切正常。 使用cluster 模式执行程序，报错报错报错。。。 思路： 因为在测试环境跑local模式一切正常， 所以首先考虑到是不是因为环境问题，但是别的程序可以正常运行。 所以应该不是环境问题。 然后就想着应该是代码出现了问题， 但是看代码愣是没看出来， 就只能使用笨办法，重新写了一个最简单的程序
.png)
没有任何配置参数，没有任何逻辑， 就是读hdfs 然后打印到控制台。
结果正常运行。
就想到可能是 spark的配置参数出问题，然后观察程序的参数配置发现
.png)
spark.driver.host 参数解释： driver监听的主机名或者IP地址。这用于和executors以及独立的master通信
部署完Spark后，分别使用yarn-cluster模式和yarn-client模式运行Spark自带的计算pi的示例。
Spark的一些配置文件除了一些基本属性外，均未做配置，结果运行的时候两种运行模式出现了不同的状况。yarn-cluster模式可以正常运行，yarn-client模式总是运行失败。查看ResourceManager、NodeManager端的日志，发现程序总是找不到ApplicationMaster，这就奇怪了！并且，客户端的Driver程序开启的端口，在NodeManager端访问被拒绝！非Spark的其他MR任务，能够正常执行。
检查客户端配置文件，发现原来在客户端的/etc/hosts文件中，客户端的一个IP对应了多个Host，Driver程序会默认去取最后对应的那个Host，比如是hostB，但是在NodeManager端是配置的其他Host，hostA，所以导致程序无法访问。为了不影响其他的程序使用客户端的Host列表，这里在Spark配置文件spark-defaults.conf中使用属性spark.driver.host来指定yarn-client模式运行中和Yarn通信的DriverHost，此时yarn-client模式可以正常运行。
上面配置完了之后，发现yarn-cluster模式又不能运行了！想想原因，肯定是上面那个配置参数搞的鬼，注释掉之后，yarn-cluster模式可以继续运行。原因是，yarn-cluster模式下，spark的入口函数是在客户端运行，但是Driver的其他功能是在ApplicationMaster中运行的，上面的那个配置相当于指定了ApplicationMaster的地址，实际上的ApplicationMaster在yarn-master模式下是由ResourceManager随机指定的。
注释掉即可。 耐心找bug。。。。
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 [email protected] 举报，一经查实，本站将立刻删除。
