ERROR: hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink
While running a job once I got the following exception
10/12/10 21:09:05 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.1.73.148:50010
10/12/10 21:09:05 INFO hdfs.DFSClient: Abandoning block blk_3623545154924652323_87440
10/12/10 21:09:11 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.net.ConnectException: Connection refused
10/12/10 21:09:11 INFO hdfs.DFSClient: Abandoning block blk_-4726571439643867938_87441\
REASON
The error contains the IP address (10.1.73.148) of the tasktracker/datanode machine for which the exception is thrown. The exception is thrown because the datanode daemon is not running on that machine; you can check this by logging into this machine, lets use 10.1.73.148 in the example, and running command
ps -eaf | grep "DataNode" | grep -v "grep"
If no lines are returned then this means that datanode daemon is not running on 10.1.73.148.
What happened is that machine 10.1.73.148 contain a data block that is required for the job that you are trying to run. If this block is replicated on other machines and those machines are running datanode daemons then this is not a problem, Hadoop will get the data block from some other machine and continue the job but if for any reason the data block is not available on any other node then your job will fail.
RESOLUTION
Logon to 10.1.73.148 and run the following command
hadoop-daemon.sh start datanode
The above command should start the datanode daemon on 10.1.73.148. You can double check this my running command
ps -eaf | grep "DataNode" | grep -v "grep"
It should return 1 line
Thats it. Try running the job again. It should not throw exception anymore
分享到:
相关推荐
其中一个问题是报but there is no HDFS_NAMENODE_USER defined. Aborting operation. Starting datanodes 直接解决 在Hadoop安装目录下找到sbin文件夹 在里面修改四个文件 1、对于start-dfs.sh和stop-dfs.sh文件,...
它尝试通过在可能的情况下对stdlib os包进行os.FileInfo来使其成为惯用语言,并从中实现接口,包括os.FileInfo和os.PathError 。 这是实际的样子: client , _ := hdfs . New ( "namenode:8020" ) file , _ := ...
当从本地上传文件到HDFS中时报错 fs.FSInputChecker: Found checksum error: b[0, 69]=6d6f77656968616861686168616868616686168616861686861680a org.apache.hadoop.fs.ChecksumException: Checksum error: file:/...
host:java.net.UnknownHostException: centos-wang: centos-wang: unknown error 在执行hadoop namenode -format命令时,出现未知的主机名。 问题原因: 出现这种问题的原因是Hadoop在格式化HDFS的时候,通过...
Hadoop常见异常,以及hadoop配置,等资料
读取 Win 7 NTFS C盘部分内容: ëRNTFS A disk read error occurred BOOTMGR is missing
HADOOP是一个用于构建云平台的Apache开源项目。使用HADOOP框架有利于我们方便、快速的实现计算机集群。在HADOOP平台上,采用了HDFS(分布式文件系统)来实现超大文件的存储和容错,而使用了MapReduce的编程模式来...
12.Hadoop入门进阶课程_第12周_Flume介绍与安装.pdf i
编译hadoophadoop-3.2.2-src的源码
原因:每次namenode format会重新创建一个namenodeId,而dfs.data.dir参数配置的目录中包含的是上次format创建的id,和dfs.name.dir参数配置的目录中的id不一致。namenode format清空了namenode下的数据,但是没有
返回一个可能承诺的对象。 fastResolveAll(数组,成功,失败) array是可能承诺对象的数组。 success是一种功能。 failure是一种功能。 返回一个可能承诺的对象。 用法 const { fastResolve , fastResolveAll ...
PXF has improved its performance when reading a large number of files from HDFS or an object store. PXF bundles newer tomcat and jackson libraries. The PXF JDBC Connector now supports pushdown of OR...
修改 资料配置文件\kylin.properties 中HDFS的路径,然后上传到 Linux的 Kylin/conf文件夹中
Flume是一个分布式,可靠的的框架,它能从许多不同的数据源高效地收集、聚合和移动大量日志数据到一个集中的目的进行存储。Flume OG主要是0.9及以前的版本,Flume NG主要是1.X的版本。类似的框架还有Logstash。对应...
1)创建一个数据库,数据库在 HDFS 上的默认存储路径是/user/hive/warehouse/*.db。 hive (default)> create database db_hive; 2)避免要创建的数据库已经存在错误,增加 if not exists 判断。(标准写法) hive ...
已经测试过workcount例子程序! [hadoop@test Desktop]... IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=129 File Output Format Counters Bytes Written=40
一、问题 思路: 主从集群:结构相对简单,主与从协作 主:单点,数据一致好...1、HDFS-HA解决方案 2、NameNode 的主备切换实现 NameNode 主备切换主要由 ZKFailoverController、HealthMonitor 和 ActiveStandbyElect
有时会统计某个目录下有哪些文件,每个文件的sha256及文件大小等相关信息,这里用python3写了个脚本用来实现此功能,此脚本可跨平台,同时支持windows和linux,脚本(get_dir_file_info.py)内容如下: import os ...
包含hadoop工具包及eclipse插件 hadoopbin.zip为工具包,解压官方haddop包之后把hadoopbin解压...hadoop2.8.0 下测试,可以显示Map/Reduce project,但是在DFS Locations下总提示Error:No FileSystem For Scheme:hdfs
增强了hdfs插件,支持分区表推送,支持动态参数传递(例如时间实现自增式抽取)。具体使用方式可以参照example模块。目前该服务已经稳定服务某上市公司半年,累计总任务数100+ ,日推送数据过10亿。具体如何使用,...