一. 任务描述使用flinkx去跑HDFS到HIVE的任务时出现如下报错CannotObtainBlockLengthExceptioncom.dtstack.flinkx.throwable.FlinkxRuntimeException:cant get file size from hdfs,filehdfs://xxx/.data/540240453caeb6fe4b3f118410a05315_235_0 atcom.dtstack.flinkx.connector.hdfs.sink.BaseHdfsOutputFormat.getCurrentFileSize(BaseHdfsOutputFormat.java:169)~[flinkx-connector-hdfs-1.12-SNAPSHOT.jar:?]atcom.dtstack.flinkx.sink.format.BaseFileOutputFormat.checkCurrentFileSize(BaseFileOutputFormat.java:137)~[flinkx-core-1.12-SNAPSHOT.jar:?]atcom.dtstack.flinkx.sink.format.BaseFileOutputFormat.writeSingleRecordInternal(BaseFileOutputFormat.java:128)~[flinkx-core-1.12-SNAPSHOT.jar:?]atcom.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeSingleRecord(BaseRichOutputFormat.java:427)~[flinkx-core-1.12-SNAPSHOT.jar:?]atcom.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeRecord(BaseRichOutputFormat.java:244)~[flinkx-core-1.12-SNAPSHOT.jar:?]atcom.dtstack.flinkx.sink.format.BaseRichOutputFormat.writeRecord(BaseRichOutputFormat.java:84)~[flinkx-core-1.12-SNAPSHOT.jar:?]atcom.dtstack.flinkx.sink.DtOutputFormatSinkFunction.invoke(DtOutputFormatSinkFunction.java:117)~[flinkx-core-1.12-SNAPSHOT.jar:?]atorg.apache.flink.table.runtime.operators.sink.SinkOperator.processElement(SinkOperator.java:72)~[flink-table-blink_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)~[flink-dist_2.12-1.12.7.jar:1.12.7]atStreamExecCalc$12.processElement(UnknownSource)~[?:?]atorg.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411)~[flink-dist_2.12-1.12.7.jar:1.12.7]atcom.dtstack.flinkx.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:135)~[flinkx-core-1.12-SNAPSHOT.jar:?]atorg.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)~[flink-dist_2.12-1.12.7.jar:1.12.7]atorg.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:267)~[flink-dist_2.12-1.12.7.jar:1.12.7]Causedby:org.apache.hadoop.hdfs.CannotObtainBlockLengthException:Cannotobtain block lengthforLocatedBlock{BP-908532681-10.203.2.20-1593930010000:blk_2513177302_1108300914318;getBlockSize()0;corruptfalse;offset0;locs[DatanodeInfoWithStorage[10.203.70.78:9866,DS-9aa641e8-4ce8-436b-9eaa-fd3eb676185e,DISK],DatanodeInfoWithStorage[10.203.22.205:9866,DS-cb07688e-a349-46eb-8929-71d8fd6156dd,DISK],DatanodeInfoWithStorage[10.203.34.146:9866,DS-efa9e16d-405b-4a05-8e52-0e35d681c3d2,DISK]]}atorg.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:363)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:201)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:185)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DFSClient.openInternal(DFSClient.java:1048)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1011)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:359)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DistributedFileSystem$5.doCall(DistributedFileSystem.java:355)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)~[hadoop-common-3.2.1U20.jar:?]atorg.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:367)~[hadoop-hdfs-client-3.2.1U20.jar:?]atorg.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:164)~[hadoop-common-3.2.1U20.jar:?]atorg.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:285)~[hadoop-common-3.2.1U20.jar:?]atorg.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:500)~[hadoop-common-3.2.1U20.jar:?]atorg.apache.hadoop.fs.FileSystem.open(FileSystem.java:912)~[hadoop-common-3.2.1U20.jar:?]atcom.dtstack.flinkx.connector.hdfs.sink.BaseHdfsOutputFormat.getCurrentFileSize(BaseHdfsOutputFormat.java:166)~[flinkx-connector-hdfs-1.12-SNAPSHOT.jar:?]...24more看报错可以得到当时任务的一些消息文件没有了 还去获取文件文件还在写的时候去获取文件暂时怀疑flinkx的hdfs写操作会偶发的出现或者当达到什么机制时后会出现。二、google解决方案Usually when you see “Cannot obtain block length for LocatedBlock”, this means the file isstill in being-written state, i.e., it has not been closed yet, and the reader cannot successfully identify its current length by communicating with corresponding DataNodes. There are multiple possibilities here, .e.g., there may be temporary network connection issue between the reader and the DataNodes, or the original writing failed some while ago and the under-construction replicas are somehow missing.In general you run fsck command to get more information about the file. You can also trigger lease recovery for further debugging. Run command:hdfs debug recoverLease -path path-of-the-file -retries retrytimesThis command will ask the NameNode to try to recover the lease for the file, and based on the NameNode log you may track to detailed DataNodes to understand the states of the replicas. The command may successfully close the file if there are still healthy replicas. Otherwise we can get more internal details about the file/block state.当看到这个消息时Cannot obtain block length for LocatedBlock,说明文件仍然是写的状态即写还没有关闭client无法成功的和datanodes通讯获取到当前文件的长度。有以下几种可能的原因会出现这种问题可能是短暂的网络通讯问题导致client和dataNodes不能通讯 来获取到文件信息flink写某些文件时写失败了并且创建副本时副本因为一些原因丢失了。解决可以通过run fsck相关命令获取文件更多的信息。或者执行如上debug命令来请求namenode来恢复文件的租约并且基于namenode日志可以追踪到datanode的一些细节来理解副本的状态。如果文件有健康的副本此时会关闭文件。三、进一步逻辑探究1. hdfs的租约机制ing
