Systemtap is a free software tool that helps with gathering all sorts of information on a linux system. It allows you to examine the kernel or user space processes without interrupting them.
You do not need to rebuild your program to add some debugging information. With the use of a simple script you can fully instrument the running kernel or your program to find and fix bugs.
General advantages are that you don't need to do anything special except install Systemtap and compile your programs without stripping debug information. And it's fast. Systemtap uses the scripts you write to create a kernel module that actually hooks into the kernel. This means that you can debug your program at full speed (depending on the system and size of the program of course).
Example: You've created a new plasmoid but somehow the performance is low. You've looked over the code but can't find the source of the problem. Now, there are a couple of programs that let you examine your plasmoid like valgrind and gdb.
Systemtap however gives you much more control and speed when instrumenting your program. With a simple script of a few lines you can profile your plasmoid and see which files it opens, how long it takes to use these files, how many times a certain function is called, how much memory is used inside a certain function, how long it takes to perform a certain task in a function etc... With all this you can zoom in on the possible problems and fix them.
At the moment Systemtap is a set of commandline tools. But it would be extremely nice to have some sort of a graphical userinterface like Apple's Instruments.
Systemtap can be compared to D-Trace on Unix systems.
Note that the userspace probing in Systemtap is still in development, so it doesn't work 100% correct at the moment.
todo: Add sections on how to easily install systemtap.
todo: How to use Systemtap, the basics.