Q) How to debug a multi threaded application?
NOTE: "ulimit -c unlimited" command to set unlimited size for large core dump generation.
NOTE: "g++ thread1.cpp -g -lpthread" command to compile thread1.cpp file with pthread library.
NOTE: "break funcion_name or filename:line", "run","continue","step","list","print","attach -c core appNmae","bt", "thread apply all bt", "info thread", "thread thread id","bt","quit".
NOTE: step executes a function as a single instruction and continue stepinto the function definition of function.
Command to debug a core file core:
attach the core file and application that has dump the code and take the bt.
"gdb -c core ./a.out"
Then it will show function causing the core dump was called by which thread by seeing start_thread(arg=0xb6bedb40).
Here 0xb6bedb40 is the thread id.
Then run the commands: "info thread" to see the culprit thread "0xb6bedb40"'s stack id.
In this case culprit thread is "0xb6bedb40",whose thread stack id is 3.
So either we can switch to third thread by command "thread 3", and take bt of the third thread only by command "bt" or
we can take stack of all the threads by "thread apply all bt". and use the stack of the thread 3 only to debug or
In this case line at thread1.cpp:37 caused the crashed by thread 3 so we can take bt of "thread 3"
Detailed steps:
-----------------
(gdb) bt //STEP:1
#0 0x08048b3c in odd (o=0x0) at thread1.cpp:37
#1 0xb76f3f70 in start_thread (arg=0xb6bedb40) at pthread_create.c:312
#2 0xb754170e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
(gdb) info thread //STEP:2
Id Target Id Frame
3 Thread 0xb73eeb40 (LWP 15708) 0xb7723424 in __kernel_vsyscall ()
2 Thread 0xb73f0700 (LWP 15707) 0xb7723424 in __kernel_vsyscall ()
* 1 Thread 0xb6bedb40 (LWP 15709) 0x08048b3c in odd (o=0x0) at thread1.cpp:37
(gdb) thread 3 //STEP:4
[Switching to thread 3 (Thread 0xb73eeb40 (LWP 15708))]
#0 0xb7723424 in __kernel_vsyscall ()
(gdb) bt //STEP:5
#0 0xb7723424 in __kernel_vsyscall ()
#1 0xb76f7d4b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187
#2 0x08048a3b in even (v=0x0) at thread1.cpp:17
#3 0xb76f3f70 in start_thread (arg=0xb73eeb40) at pthread_create.c:312
#4 0xb754170e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
(gdb) thread apply all bt //STEP:5
Thread 3 (Thread 0xb73eeb40 (LWP 15708)):
#0 0xb7723424 in __kernel_vsyscall ()
#1 0xb76f7d4b in pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_wait.S:187
#2 0x08048a3b in even (v=0x0) at thread1.cpp:17
#3 0xb76f3f70 in start_thread (arg=0xb73eeb40) at pthread_create.c:312
#4 0xb754170e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
Thread 2 (Thread 0xb73f0700 (LWP 15707)):
#0 0xb7723424 in __kernel_vsyscall ()
#1 0xb76f5178 in pthread_join (threadid=3074353984, thread_return=0x0) at pthread_join.c:92
#2 0x08048c51 in main () at thread1.cpp:54
Thread 1 (Thread 0xb6bedb40 (LWP 15709)):
#0 0x08048b3c in odd (o=0x0) at thread1.cpp:37
#1 0xb76f3f70 in start_thread (arg=0xb6bedb40) at pthread_create.c:312
#2 0xb754170e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
Q) How to skip 20 lines and show next 10 lines:
list +20,+10
if list shows lines upto 52 then list +20,+10
will displays 10 lines after the line from 72(52+20).
ie it display line 72 to 82
Q) Program using two threads to print odd and even. In each thread function I have knowngly introduced dived by zero error. Need to debug which thread is causing the divide by zero error first and dump core and optimize the code for that thread accordingly.
//Improved Example synchronized using mutex and conditional variable.
#include<iostream>
using namespace std;
#include<stdio.h>
int count=0;
int maxx=10;
pthread_mutex_t mutex;
pthread_cond_t cond;
void* even(void *v)
{
//cout<<"even thread"<<endl;
while(count<maxx)
{
pthread_mutex_lock(&mutex);//lock before checking
while((count%2)!=0)
{
pthread_cond_wait(&cond,&mutex);//wait until count becomes odd
//it releases the mutex and it waits till condition cond is signaled as complete and mutex is available.
}
if(count==6){printf("%d",count/0);}
cout<<"even "<<count++<<endl;
pthread_mutex_unlock(&mutex);//release
pthread_cond_signal(&cond);//signal to another thread
}
pthread_exit(0);//pthread_exit is called from the thread itself to terminate its execution (and return a result) early.
}
void* odd(void *o)
{
while(count<maxx)
{
pthread_mutex_lock(&mutex);//lock before checking
while((count%2)!=1)
{
pthread_cond_wait(&cond,&mutex);//wait until count becomes odd
}
if(count==5){printf("%d",count/0);}
cout<<"odd "<<count++<<endl;
pthread_mutex_unlock(&mutex);//release
pthread_cond_signal(&cond);//signal to another thread
}
pthread_exit(0);
}
int main()
{
pthread_t t1;
pthread_t t2;
pthread_mutex_init(&mutex, 0);
pthread_cond_init(&cond, 0);
pthread_create(&t1,NULL,&even,NULL);
pthread_create(&t2,NULL,&odd,NULL);
pthread_join(t1,0);
pthread_join(t2,0);
pthread_mutex_destroy(&mutex);
pthread_cond_destroy(&cond);
//pthread_join is called from another thread to wait for a thread to terminate and obtain its return value
cout<<endl;
return 0;
}
-----------------------------------------------
Q) Ho to debug multi-threading Applications with gdb debugger?
Compiling Multi-threaded Code using g++ with Debug Info
g++ -g --std=c++11 -pthread sample.cpp -o sample
Q)How to List all active threads?
“info threads”
Q)How to Check Stack trace of threads ?
In non multi threading applications there is only one thread i.e. main function. Therefore, to check the stacktrace we use command “bt”.
But in multi threading applications, as there are many threads and each threads has its own stack.
But “bt” command will display the stack trace of current active thread only. Now what if want to inspect stacktrace of all the threads at same point ?
To display the stack trace of all the threads use following command
(gdb) thread apply all bt
//output would be
Thread 3 (Thread 0xb74a8b40 (LWP 4335)):
#0 __GI___nptl_death_event () at events.c:31
#1 0xb7faf104 in start_thread (arg=0xb74a8b40) at pthread_create.c:363
#2 0xb7dfc70e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129
Thread 1 (Thread 0xb7cab700 (LWP 4330)):
#0 main () at threadOddEven.cpp:17
(gdb)
Q) How to Switch between threads while debugging?
(gdb) thread <Thread Number>
(gdb) thread 2
(gdb) thread 1
