'jstack: Target process not responding
I am running Ubuntu server edition and I wanted to take a thread dump of Tomcat.
So, I first tried to find out which PID tomcat uses:
$ jps -l
5809 sun.tools.jps.Jps
But it's not there?
So, I used top instead and found out the PID 5730.
Then I called jstack to get the thread dump:
$ sudo jstack -l 5730
5730: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
What's going on? :-(
I already tried to export CATALINA_TMPDIR as described in Jstack and Jstat stopped working with upgrade to JDK6u23 but that didn't change anything:
$ export CATALINA_TMPDIR=/tmp
$ sudo /etc/init.d/tomcat6 restart
* Stopping Tomcat servlet engine tomcat6
...done.
* Starting Tomcat servlet engine tomcat6
...done.
$ sudo jstack -l 5934 // new PID after restart
5934: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
Update:
I also tried sudo -u tomcat6 jstack -l -F 5730 > threaddumpexceptions2.txt but it only gives me tons of exceptions on the console.
Solution 1:[1]
I got it working by doing two things:
- Changed call to:
sudo -u tomcat6 jstack -J-d64 -m pid - Replaced OpenJDK with Sun's original sun-6-jdk and sun-6-jre packages
Explanation for part 1: I switched to 64-bit mode, used sudo and run the command as Tomcat user.
Note: Part 2 might not be necessary. For some users it seems like part 1 is enough. In fact, try to add just the sudo command first. It might already do the trick.
Solution 2:[2]
I think you need to run jstack as the same user that runs the Tomcat process. Note also that jps only returns processes for the current user. You would get the pid for the Tomcat process by running jps with sudo or as the Tomcat process user.
This bug report may also be useful: https://bugs.launchpad.net/ubuntu/+source/sun-java6/+bug/597098
Solution 3:[3]
@Valmar, I find the same topic post here. Unable to get thread dump? Any ideas why my app blocks?
It seems the workaround is sudo -u tomcat6 kill -3 <pid>.
Solution 4:[4]
Try to switch to process user and then use jstack:
sudo -u {process user} jstack > dump
Solution 5:[5]
This also worked for me:
sudo -u tomcat6 kill -3 pid
It looks like nothing happens but when you look in the logs the stacks are there. They look like exceptions if your not expecting them.
Solution 6:[6]
I find it useful to use something like 'ps -eo pid,user,command | grep java' to find the actual java command being used, then use the directory to find the matching jstack etc.
# ps -eo user,command | grep '[j]ava' | cut -d' ' -f1
someuser /usr/lib/jvm/java/bin/java
# /usr/lib/jvm/java/bin/java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
So its 64-bit, running as 'someuser'. su to that user and run run jstack etc. from that same directory. (ie. /usr/lib/jvm/java/bin/jstack
Useful when you're on a server with various different installations / implementations of Java.
Solution 7:[7]
I had same problem, but none of below solution worked for me:
jstack <pid>
jstack -J-d64 -m <pid>
sudo -u <user> jstack ...
I finally upgraded JDK from jdk1.6.0_24 to jdk1.7.0_67 and every things worked.
Solution 8:[8]
For Tomcat users having this issue, check your catalina.out log file.
I was having the same problems "22693: Unable to open socket file: target process not responding or HotSpot VM not loaded". I gave up and was trying to find anything about what happened before it locked up, but then there was the jstack output in the log file.
Solution 9:[9]
For those finding this answer six years after it was asked and they are on CentOS 7 note that SELinux will stop jmap from writing the heap dumps as, out of the box, it will deny writing to a socket unless the directory has the type tomcat_tmp_t.
In my case, I write my heap dumps under /usr/share/tomcat/.jmap. This directory is owned by my runtime user.
Therefore to allow jmap to write to this directory:
semanage fcontext -a -t tomcat_tmp_t "/usr/share/tomcat/.jmap(/.*)?"
restorecon /usr/share/tomcat -vR
Which allows us to then run, as our tomcat user:
jmap -dump:format=b,file=/usr/share/tomcat/.jmap/tomcat-`date +%s`.bin <pid>
Solution 10:[10]
My problem was, I ran the process by the terminal. Then for testing Its jstack, I paused the process by ctrl+z. The process was not able to respond. To solve this problem, I resumed the process by fg and checked its jstack by another terminal.
Solution 11:[11]
In my case, I was looking at the wrong PID.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow