'flink webui logs failed and jobmanager always shutdown on kubernetes
deployed a flink with Standalone Kubernetes Application Cluster(Deployment Modes is Application),
flink version is 1.14.4, and run in kubernetes, standalone-applicationn mode
deploy steps and yaml use: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/standalone/kubernetes/#application-mode
problems:
- the flink is running in kubernetes, but the logs cannot view in the flink-webui, only through "kubectl logs" or "docker logs"
- after a moment jobmanager restart then shutdown , not sure also the logger problem
about logger problem, search some solutions but cannot work in my case or just only see taskmanager stdout and no jonmanager logs or stdout; and jobmanager shutdown no idea now
some logger error info from console:
ERROR StatusLogger Reconfiguration failed: No configuration found for '74a14482' at 'null' in 'null'
UnknownTaskExecutorException: No TaskExecutor registered under 10.244.0.85:6122-cbd9ad.
ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler [] - Failed to transfer file from TaskExecutor 10.244.0.85:6122-b815fd. java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT does not exist on the TaskExecutor.
Thread-20 ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@778d1062] unable to create manager for [] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7a64d6ab[pattern=.%i, append=false, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=10, useMax=true), advertiseURI=null, layout=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@778d1062] unable to create manager for [] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@7a64d6ab[pattern=.%i, append=false, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=104857600)]), strategy=DefaultRolloverStrategy(min=1, max=10, useMax=true), advertiseURI=null, layout=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n, filePermissions=null, fileOwner=null]]
globalEventExecutor-1-1 ERROR Null object returned for RollingFile in Appenders. globalEventExecutor-1-1 ERROR Unable to locate appender "RollingFileAppender" for logger config "root" Thread-20 ERROR Unable to create file java.io.IOException: No such file or directory, at java.io.UnixFileSystem.createFileExclusively(Native Method)
globalEventExecutor-1-1 ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
jobmanager shutdown log:
2022-04-20 01:41:09,138 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested. 2022-04-20 01:41:09,140 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Shutting StandaloneApplicationClusterEntryPoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally.. 2022-04-20 01:41:09,142 INFO org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting down rest endpoint. 2022-04-20 01:41:09,146 INFO org.apache.flink.runtime.blob.BlobServer [] - Stopped BLOB server at 0.0.0.0:6124 2022-04-20 01:41:09,184 globalEventExecutor-1-1 ERROR Unable to locate appender "RollingFileAppender" for logger config "root" 2022-04-20 01:41:09,357 Thread-48 ERROR Unable to locate appender "RollingFileAppender" for logger config "root" 2022-04-20 01:41:09,397 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon. 2022-04-20 01:41:09,397 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting down remote daemon. 2022-04-20 01:41:09,404 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports. 2022-04-20 01:41:09,406 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote daemon shut down; proceeding with flushing remote transports. 2022-04-20 01:41:09,453 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down. 2022-04-20 01:41:09,455 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting shut down.
thx all!!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
