'How to understand a CosmosDb exception (InternalServerErrorException)?

I receive sporadic an errors from the Cosmos Db driver like the following with a large message which is not helpful for me.

  • How can I read this exception?
  • How can I find the failing command or code line?
  • What is the cause of the exception?
  • How can I fix this?
{"ClassName":"InternalServerErrorException","userAgent":"azsdk-java-cosmos/4.17.0 WindowsServer2019/10.0 JRE/11.0.15",
"statusCode":500,"resourceAddress":"rntbd://172.22.173.30:10251/apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/",
"error":"{\"Errors\":[\"An unknown error occurred while processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support\"]}",
"innerErrorMessage":"[\"An unknown error occurred while processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support\"]",
"causeInfo":null,
"responseHeaders":"{x-ms-current-replica-set-size=1, x-ms-last-state-change-utc=Wed, 27 Apr 2022 20:54:25.931 GMT, x-ms-request-duration-ms=12.363, x-ms-session-token=0:-1#6689, lsn=6689, x-ms-request-charge=2.67, x-ms-schemaversion=1.12, x-ms-transport-request-id=63217, x-ms-number-of-read-regions=0, x-ms-current-write-quorum=1, x-ms-cosmos-quorum-acked-llsn=6689, x-ms-quorum-acked-lsn=6689, x-ms-activity-id=73c9465a-c670-11ec-b802-a31d99fd4ad1, x-ms-xp-role=0, x-ms-global-Committed-lsn=-1, x-ms-cosmos-llsn=6689, x-ms-serviceversion= version=2.14.0.0}",
"cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.17.0 WindowsServer2019/10.0 JRE/11.0.15","requestLatencyInMs":79,"requestStartTimeUTC":"2022-04-27T21:24:42.285251300Z","requestEndTimeUTC":"2022-04-27T21:24:42.365236700Z","responseStatisticsList":[{"storeResult":{"storePhysicalAddress":"rntbd://172.22.173.30:10251/apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/","lsn":6689,"globalCommittedLsn":-1,"partitionKeyRangeId":"","isValid":true,"statusCode":500,"subStatusCode":0,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":1.0,"itemLSN":-1,"sessionToken":"-1#6689","backendLatencyInMs":12.363,
"exception":"[\"An unknown error occurred while processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support\"]",
"transportRequestTimeline":[{"eventName":"created","startTimeUTC":"2022-04-27T21:24:42.285251300Z","durationInMicroSec":0},{"eventName":"queued","startTimeUTC":"2022-04-27T21:24:42.285251300Z","durationInMicroSec":0},
{"eventName":"channelAcquisitionStarted","startTimeUTC":"2022-04-27T21:24:42.285251300Z","durationInMicroSec":0},
{"eventName":"pipelined","startTimeUTC":"2022-04-27T21:24:42.285251300Z","durationInMicroSec":990},
{"eventName":"transitTime","startTimeUTC":"2022-04-27T21:24:42.286242Z","durationInMicroSec":13014},
{"eventName":"received","startTimeUTC":"2022-04-27T21:24:42.299256400Z","durationInMicroSec":0},
{"eventName":"completed","startTimeUTC":"2022-04-27T21:24:42.299256400Z","durationInMicroSec":981}],
"rntbdRequestLengthInBytes":297,"rntbdResponseLengthInBytes":353,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":null,"channelTaskQueueSize":1,"pendingRequestsCount":1,"serviceEndpointStatistics":{"availableChannels":3,"acquiredChannels":0,"executorTaskQueueSize":0,"inflightRequests":1,"lastSuccessfulRequestTime":"2022-04-27T21:24:31.868Z","lastRequestTime":"2022-04-27T21:24:31.859Z","createdTime":"2022-04-27T21:16:17.690994800Z","isClosed":false}},"requestResponseTimeUTC":"2022-04-27T21:24:42.300237900Z","requestResourceType":"DocumentCollection","requestOperationType":"ReadFeed"},
{"storeResult":{"storePhysicalAddress":"rntbd://172.22.173.30:10251/apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/",
"lsn":6689,"globalCommittedLsn":-1,
"partitionKeyRangeId":null,"isValid":true,"statusCode":200,"subStatusCode":0,"isGone":false,"isNotFound":false,"isInvalidPartition":false,
"isThroughputControlRequestRateTooLarge":false,
"requestCharge":1.67,"itemLSN":-1,"sessionToken":"-1#6689","backendLatencyInMs":10.778,"exception":null,
"transportRequestTimeline":[{"eventName":"created","startTimeUTC":"2022-04-27T21:24:42.285251300Z","durationInMicroSec":0},
{"eventName":"queued","startTimeUTC":"2022-04-27T21:24:42.285251300Z","durationInMicroSec":0},
{"eventName":"channelAcquisitionStarted","startTimeUTC":"2022-04-27T21:24:42.285251300Z","durationInMicroSec":990},
{"eventName":"pipelined","startTimeUTC":"2022-04-27T21:24:42.286242Z","durationInMicroSec":0},
{"eventName":"transitTime","startTimeUTC":"2022-04-27T21:24:42.286242Z","durationInMicroSec":78994},
{"eventName":"received","startTimeUTC":"2022-04-27T21:24:42.365236700Z","durationInMicroSec":0},
{"eventName":"completed","startTimeUTC":"2022-04-27T21:24:42.365236700Z","durationInMicroSec":0}],
"rntbdRequestLengthInBytes":297,"rntbdResponseLengthInBytes":21521,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":21240,"channelTaskQueueSize":1,"pendingRequestsCount":0,"serviceEndpointStatistics":{"availableChannels":2,"acquiredChannels":1,"executorTaskQueueSize":0,"inflightRequests":2,"lastSuccessfulRequestTime":"2022-04-27T21:24:31.868Z","lastRequestTime":"2022-04-27T21:24:42.285Z","createdTime":"2022-04-27T21:16:17.690994800Z","isClosed":false}},"requestResponseTimeUTC":"2022-04-27T21:24:42.365236700Z","requestResourceType":"DocumentCollection","requestOperationType":"ReadFeed"}],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{},"regionsContacted":[],"retryContext":{"statusAndSubStatusCodes":null,"retryCount":0,"retryLatency":0},"metadataDiagnosticsContext":{"metadataDiagnosticList":null},"serializationDiagnosticsContext":{"serializationDiagnosticsList":null},"gatewayStatistics":null,"systemInformation":{"usedMemory":"116830 KB","availableMemory":"407458 KB","systemCpuLoad":"(2022-04-27T21:24:16.127472200Z 96,7%), (2022-04-27T21:24:21.160494800Z 99,9%), (2022-04-27T21:24:26.126473Z 98,8%), (2022-04-27T21:24:31.142447Z 99,7%), (2022-04-27T21:24:36.147247800Z 100,0%), (2022-04-27T21:24:41.132241600Z 100,0%)","availableProcessors":8},"clientCfgs":{"id":-5,"connectionMode":"DIRECT","numberOfClients":6,"connCfg":{"rntbd":"(cto:PT5S, rto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:false)","gw":"(cps:1000, rto:PT5S, icto:null, p:false)","other":"(ed: true, cs: false)"},"consistencyCfg":"(consistency: null, mm: true, prgns: [])"}}}
    at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.messageReceived(RntbdRequestManager.java:807)
    at com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdRequestManager.channelRead(RntbdRequestManager.java:181)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1368)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
javaexceptionazure-cosmosdb


Solution 1:[1]

5XX errors are always service errors that usually can only be resolved by a support engineer. Nearly always the information you are looking for is found right up at the top of the error as it is here.

"An unknown error occurred while processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support\"

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Mark Brown

Related Questions