'How to prevent CORBA.COMM_FAILURE after underlying (idle?) connection died with "java.io.IOException: Broken pipe" on Payara Client/Server?
We have issues with CORBA/IIOP where one Payara node fails to call a remote implementation (on a different Payara node) when the servers have been sitting idle overnight. The error that shows up then is a "CORBA.COMM_FAILURE" with an underlying "Caused by: java.io.IOException: Broken pipe". Subsequent calls seem to work after this "initial" error.
I would expect that CORBA detects a stalled connection and transparently recovers before the error propagates back up to the calling class.
The architecture is a central business-logic Payara app, that is called via CORBA/IIOP/RMI from an outside Payara that offers some REST services. The Payara versions we're using is 5.201. Very often the first (or first few) REST calls in the morning fail with this broken pipe error at the lowest level.
My suspicion is that somehow after a long idling time the IIOP connection between the servers is cut but the client is not correctly detecting this broken connection. Only when the next communication over this supposedly working server connection would be required it realizes that it's broken and does not recover internally but fail the whole call.
It was already tried to set the Payara System Property "fish.payara.SO_KEEPALIVE" to "true" hoping that it keeps those connections alive or allows for a transparent recovery. But apparently this does not work (at least not as excpected) and the errors continue to happen.
Any help or further pointers would be very appreciated.
Thanks very much!
2022-03-23 08:48:16,624 [http-thread-pool::http-listener-1(29)] ERROR be_rest_admin 10.236.202.193 VSO01 [com.company.rest.api.security.R6OperatorRealm] java.rmi.MarshalException: CORBA COMM_FAILURE 1330446339 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent vmcid: OMG minor code: 3 completed: Maybe
javax.ejb.EJBException: java.rmi.MarshalException: CORBA COMM_FAILURE 1330446339 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent vmcid: OMG minor code: 3 completed: Maybe
at com.company.base.mandator._MandatorManagement_Wrapper.retrieveMandator(com/company/base/mandator/_MandatorManagement_Wrapper.java) ~[mandator-management-interface-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at sun.reflect.GeneratedMethodAccessor402.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_191]
at com.company.base.servicelocator.ServiceLocator$ServiceProxy.invoke(ServiceLocator.java:302) ~[company-commons-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at com.sun.proxy.$Proxy456.retrieveMandator(Unknown Source) ~[na:na]
at com.company.api.base.mandator.MandatorManagementDelegate.retrieveMandator(MandatorManagementDelegate.java:93) ~[company-api-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at com.company.rest.api.security.R6OperatorRealm.verifyMandatorExists(R6OperatorRealm.java:81) [rest-api-core-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at com.company.rest.api.security.R6OperatorRealm.doGetAuthenticationInfo(R6OperatorRealm.java:60) [rest-api-core-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:245) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) [shiro-core-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [shiro-web-1.4.2.jar:1.4.2]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.4.2.jar:1.4.2]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343) [spring-web-3.2.14.RELEASE.jar:3.2.14.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260) [spring-web-3.2.14.RELEASE.jar:3.2.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253) [web-core.jar:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) [web-core.jar:na]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257) [web-core.jar:na]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757) [web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577) [web-core.jar:na]
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) [web-glue.jar:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158) [web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:371) [web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:238) [web-core.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) [kernel.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) [kernel.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [nucleus-grizzly-all.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1330446339 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent vmcid: OMG minor code: 3 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:259) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:201) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:140) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:226) ~[glassfish-corba-orb.jar:na]
at com.company.base.mandator.__MandatorManagement_Remote_DynamicStub.retrieveMandator(com/company/base/mandator/__MandatorManagement_Remote_DynamicStub.java) ~[mandator-management-interface-6.5.1.4-CUSTOMER-32-PAYARA.jar:na]
... 63 common frames omitted
Caused by: org.omg.CORBA.COMM_FAILURE: FINE: 00410003: Write error sent
at com.sun.proxy.$Proxy398.writeErrorSend(Unknown Source) ~[na:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.sendWithoutLock(ConnectionImpl.java:774) ~[na:na]
at com.sun.corba.ee.impl.encoding.BufferManagerWriteStream.sendFragment(BufferManagerWriteStream.java:145) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.BufferManagerWriteStream.overflow(BufferManagerWriteStream.java:85) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_2.grow(CDROutputStream_1_2.java:266) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_2.alignAndReserve(CDROutputStream_1_2.java:229) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.internalWriteOctetArray(CDROutputStream_1_0.java:454) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeString(CDROutputStream_1_0.java:407) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_string(CDROutputStream_1_0.java:384) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_codebase(CDROutputStream_1_0.java:1255) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeValueTag(CDROutputStream_1_0.java:1303) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeRMIIIOPValueType(CDROutputStream_1_0.java:682) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:821) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:762) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:834) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:565) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.write_value(CDROutputObject.java:506) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.writeObjectField(IIOPOutputStream.java:766) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputClassFields(IIOPOutputStream.java:837) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.defaultWriteObjectDelegate(IIOPOutputStream.java:236) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputObject(IIOPOutputStream.java:609) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.simpleWriteObject(IIOPOutputStream.java:187) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueInternal(ValueHandlerImpl.java:238) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueWithVersion(ValueHandlerImpl.java:219) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValue(ValueHandlerImpl.java:179) ~[na:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.callWriteValue(CDROutputStream_1_0.java:711) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeRMIIIOPValueType(CDROutputStream_1_0.java:696) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:821) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:834) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:565) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.write_value(CDROutputObject.java:506) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.writeObjectField(IIOPOutputStream.java:766) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputClassFields(IIOPOutputStream.java:837) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.defaultWriteObjectDelegate(IIOPOutputStream.java:236) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.outputObject(IIOPOutputStream.java:609) ~[na:na]
at com.sun.corba.ee.impl.io.IIOPOutputStream.simpleWriteObject(IIOPOutputStream.java:187) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueInternal(ValueHandlerImpl.java:238) ~[na:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.callWriteValue(CDROutputStream_1_0.java:711) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.writeRMIIIOPValueType(CDROutputStream_1_0.java:696) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:821) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:834) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputStream_1_0.write_value(CDROutputStream_1_0.java:565) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.write_value(CDROutputObject.java:506) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl$14.write(DynamicMethodMarshallerImpl.java:393) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl.writeArguments(DynamicMethodMarshallerImpl.java:465) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:186) ~[glassfish-corba-orb.jar:na]
... 66 common frames omitted
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_191]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_191]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_191]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_191]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_191]
at com.sun.corba.ee.impl.transport.NioBufferWriter.write(NioBufferWriter.java:23) ~[na:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.writeUsingNio(ConnectionImpl.java:507) ~[na:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.write(ConnectionImpl.java:479) ~[na:na]
at com.sun.corba.ee.impl.encoding.CDROutputObject.writeTo(CDROutputObject.java:225) ~[glassfish-corba-orb.jar:na]
at com.sun.corba.ee.impl.transport.ConnectionImpl.sendWithoutLock(ConnectionImpl.java:758) ~[na:na]
... 112 common frames omitted
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValueWithVersion(ValueHandlerImpl.java:219) ~[na:na]
at com.sun.corba.ee.impl.io.ValueHandlerImpl.writeValue(ValueHandlerImpl.java:179) ~[na:na]
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|