微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

【Hadoop报错】The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is exc

问题描述:调度系统执行hive任务失败,一直执行失败,报错如下:
java.io.IOException: java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 Failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Caused by: java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 Failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

CONSOLE# Ended Job = job_1638255473937_0568 with exception 'java.io.IOException(java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 Failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused)

CONSOLE# Failed: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 Failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

这个信息还看不出具体的问题所在,查看服务器上所有的日志,也没有看出问题,最后查看yarn的日志,看出问题所在。
根据调度系统,获取到ApplicationId:application_1638255473937_0568 , 然后从hdfs上查看对应的日志信息。

查看yarn日志信息:
[hdfs@centos hadoop27]$ yarn logs -applicationId application_1638255473937_0568

关键报错信息:
org.apache.hadoop.ipc.remoteexception(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is exceeded: limit=1048576 items=1048576

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 [email protected] 举报,一经查实,本站将立刻删除。

相关推荐