I was trying to figure out how to know if the threads of a process have been deadlocked on Unix/Linux machine? Also, is there a command for knowing what stage (or st开发者_运维百科atus) a process is in? If you know of any tools, please suggest. Thank you.
Thanks to /proc/<pid>/syscall
, this is how I ended up implementing a quick and dirty processes futex(op=FUTEX_WAIT) scanner.
#!/bin/bash
#
# Find all processes that are executing a futex(2) call with op=FUTEX_WAIT
# In some cases this can be helpful in finding deadlock-ed processes.
#
test ! $UID -eq 0 && echo -e "WARNING: Not running as root, only processes for this user are being scanned\n" >&2;
pids=$(ps -u $UID -opid --no-headers)
for pid in $pids; do
cat /proc/$pid/syscall |
awk "{if (\$1 == 202 && \$3 == \"0x0\") {
print $pid
}}";
# $1 is the syscall, we compare to 202 which is the futex call
# See: /usr/include/asm/unistd.h
# $2 is the 1st param, $3 is the 2nd param, etc
# We compare the second param to 0x0 which is FUTEX_WAIT
# See: /usr/include/linux/futex.h
done
Try to use a tool that traces system calls, e.g strace
on Linux or tusc
on HP-UX. When a deadlock occured, you should see the process hang in the blocking call. That's not a positive proof, though. It might be a regular block. You then need to determine whether the block may be resolved some time or not. This requires knowledge of the resource the process is waiting for.
Example
On RHEL4 there exists a... peculiarity... that may cause ctime
to deadlock. Find an example program exhibiting this behaviour below:
#include <sys/time.h>
#include <time.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
volatile char *r;
void handler(int sig)
{
time_t t;
time(&t);
r = ctime(&t);
}
int main()
{
struct itimerval it;
struct sigaction sa;
time_t t;
int counter = 0;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = handler;
sigaction(SIGALRM, &sa, NULL);
it.it_value.tv_sec = 0;
it.it_value.tv_usec = 1000;
it.it_interval.tv_sec = 0;
it.it_interval.tv_usec = 1000;
setitimer(ITIMER_REAL, &it, NULL);
while(1) {
counter++;
time(&t);
r = ctime(&t);
printf("Loop %d\n",counter);
}
return 0;
}
This will typically enter a deadlock after a couple thousand iterations. Now, attach strace
like so
strace -s4096 -p<PID>
where PID
is the program's process id. You'll see the program hang in a call with FUTEX_WAIT
in the parameters. (I cannot quote the whole line as I currently don't have access to a RHEL4 machine, please excuse that).
UNIX guarantees that OS processes never gets in deadlock. No such guarantees are, however, made for user defined processes. As far as I know, there is no direct way of determining whether a process has been deadlocked.
That said, you could determine the process state by ps -o pid,uname,command,state,stime,time
. man ps
shows more detailed explanation on process state codes too.
精彩评论