AI导航网
微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦！

解决 NN 连接不上 JN 的问题

分类：Hadoop作者：编程之家原文地址

自动故障转移配置好以后，然后使用 start-dfs.sh 群起脚本启动 hdfs 集群，有可能会遇到 NameNode 起来一会后，进程自动关闭的问题。查看 NameNode 日志，报错信息如下：

2020-08-17 10:11:40,658 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 0 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:40,659 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 0 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:40,659 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 0 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:41,660 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 1 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:41,660 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 1 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:41,665 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 1 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:42,661 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 2 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:42,661 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 2 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:42,667 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 2 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:43,662 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 3 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:43,662 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 3 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:43,668 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 3 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:44,663 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 4 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:44,663 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 4 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:44,670 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 4 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:45,467 INFO 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 6001 
ms (timeout=20000 ms) for a response for selectStreamingInputStreams. No 
responses yet.
2020-08-17 10:11:45,664 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 5 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:45,664 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 5 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:45,672 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 5 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:46,469 INFO 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 7003 
ms (timeout=20000 ms) for a response for selectStreamingInputStreams. No responses yet.
2020-08-17 10:11:46,665 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 6 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:46,665 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 6 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:46,673 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 6 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:47,470 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 8004 
ms (timeout=20000 ms) for a response for selectStreamingInputStreams. No responses yet.
2020-08-17 10:11:47,666 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 7 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:47,667 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 7 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:47,674 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 7 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:48,471 INFO 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 9005 
ms (timeout=20000 ms) for a response for selectStreamingInputStreams. No responses yet.
2020-08-17 10:11:48,668 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 8 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:48,668 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 8 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:48,675 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 8 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:49,669 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop102/192.168.6.102:8485. Already tried 9 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:49,673 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop104/192.168.6.104:8485. Already tried 9 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:49,676 INFO org.apache.hadoop.ipc.Client: retrying connect 
to server: hadoop103/192.168.6.103:8485. Already tried 9 time(s); retry 
policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-08-17 10:11:49,678 WARN org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input 
streams from QJM to [192.168.6.102:8485, 192.168.6.103:8485, 
192.168.6.104:8485]. Skipping.org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many 
exceptions to achieve quorum size 2/3. 3 exceptions thrown:
192.168.6.103:8485: Call From hadoop102/192.168.6.102 to hadoop103:8485 
Failed on connection exception: java.net.ConnectException: 拒绝连接; For more 
details see: http://wiki.apache.org/hadoop/ConnectionRefused
192.168.6.102:8485: Call From hadoop102/192.168.6.102 to hadoop102:8485 
Failed on connection exception: java.net.ConnectException: 拒绝连接; For more 
details see: http://wiki.apache.org/hadoop/ConnectionRefused
192.168.6.104:8485: Call From hadoop102/192.168.6.102 to hadoop104:8485 
Failed on connection exception: java.net.ConnectException: 拒绝连接; For more 
details see: http://wiki.apache.org/hadoop/ConnectionRefused

查看报错日志，可分析出报错原因是因为 NameNode 连接不上 JournalNode，而利用 jps 命令查看到三台 JN 都已经正常启动，为什么 NN 还是无法正常连接到 JN 呢？这是因为 start-dfs.sh 群起脚本默认的启动顺序是先启动 NN，再启动 DN，然后再启动 JN，并且默认的 rpc 连接参数是重试次数为 10，每次重试的间隔是 1s，也就是说启动完 NN以后的 10s 中内，JN 还启动不起来，NN 就会报错了。

core-default.xml 里面有两个参数如下：

<!-- NN 连接 JN 重试次数，默认是 10 次 -->
<property>
  <name>ipc.client.connect.max.retries</name>
  <value>10</value>
</property>
<!-- 重试时间间隔，默认 1s -->
<property>
  <name>ipc.client.connect.retry.interval</name>
  <value>1000</value>
</property>

解决方案：遇到上述问题后，可以稍等片刻，等 JN 成功启动后，手动启动下三台NN：

[root@hadoop102 ~]$ hdfs --daemon start namenode
[root@hadoop103 ~]$ hdfs --daemon start namenode
[root@hadoop104 ~]$ hdfs --daemon start namenode

也可以在 core-site.xml 里面适当调大上面的两个参数：

<!-- NN 连接 JN 重试次数，默认是 10 次 -->
<property>
  <name>ipc.client.connect.max.retries</name>
  <value>20</value>
</property>
<!-- 重试时间间隔，默认 1s -->
<property>
  <name>ipc.client.connect.retry.interval</name>
  <value>5000</value>
</property>

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 [email protected] 举报，一经查实，本站将立刻删除。

相关推荐

hadoop day2-搭建
hadoop搭建准备工作三台虚拟机：master、node1、node2检查时间是否同步：date检查java的jdk是否被安装好：java-version修改主机名三台分别执行vim/etc/hostname并将内容指定为对应的主机名 关闭防火墙：systemctlst...
Hadoop 二十三
文件的更名和移动:    获取文件详细信息       遇到的问题：不能直接在web上上传文件。   权限问题：修改后即可正常创建  参考：https://blog.csdn.net/weixin_44575660/article/details/118687993
大数据Hadoop之——HDFS小文件问题与处理实战操作
目录一、背景1）小文件是如何产生的？2）文件块大小设置3）HDFS分块目的二、HDFS小文件问题处理方案1）HadoopArchive（HAR）2）Sequencefile3）CombineFileInputFormat4）开启JVM重用5）合并本地的小文件，上传到HDF...
大数据Hadoop之——Hadoop HDFS多目录磁盘扩展与数据平衡实战操作
目录一、概述二、HadoopDataNode多目录磁盘配置1）配置hdfs-site.xml2）配置详解1、dfs.datanode.data.dir2、dfs.datanode.fsdataset.volume.choosing.policy3、dfs.datanode.available-space-volume-choosing-polic...
平台搭建伪分布式
平台搭建（伪分布式）伪分布式搭建在VM中搭建std-master修改配置文件centos7-cl1.vmdkstd-master.vmx-将配置文件中vm的版本号改成自己电脑对应的vm版本修改客户端的操作系统为centos764位打开虚拟机修改虚拟机网络cd...
Harley浅谈HadoopHDFS
 一、HDFS概述 1.1、HDFS产出背景及定义 1.1.1、HDFS产生背景   随着数据量越来越大，在一个操作系统存不下所有的数据，那么就分配到更多的操作系统管理的磁盘中，但是不方便管理和维护，迫切需要一种系统来管...
Hadoop 十二
配置workers进入hadoop/etc/hadoop  编辑workers文件  然后分发给另外两个服务器     准备启动集群第一次需要初始化.  初始化完成后增加了data文件，  进入上面那个路径，就能看到当前服务器的版本号 ...
第六周总结8.13
这周我对ssm框架进行了更深一步的开发，加入了多用户，并对除登录外的请求进行了拦截，这样用户在未登录的时候是访问不到资源的。并且对hadoop进行了初步的学习，包括虚拟机的安装等等。下周会对hadoop进行更深一步的...
安装Hadoop2.10.1
前言通过在Hadoop1安装Hadoop,然后配置相应的配置文件，最后将Hadoop所有文件同步到其他Hadoop节点。一、集群规划#主机名‘master/hadoop1’‘slave01/hadoop2’‘slave02/hadoop3’#启动节点NamenodeNodemanagerNod...
集群崩溃处理
1.先杀死进程（先进入到hadoop版本文件里，我的是/opt/module/hadoop-3.1.3/）sbin/stop-dfs.sh2.删除每个集群上的data以及logsrm-rfdata/logs/3.格式化hdfsnamenode-format4.再启动sbin/sart-dfs.sh
    pythonJavaScriptjavaHTMLreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-bootreact-native