I've just come across an interesting article about DTrace which is a tracing solution developed by SUN. I still need to dive into it deeper but it's key features are:
- zero overhead when it's turned off
- C based query language that allows you to query available probes
- it allows you to analyze the whole system at very low(kernel level operations) and/or high level (number of garbage collections)
- it gathers probes only when it's safe for the system
- it's built-in into Solaris 10 and will be part of the next operating system from Apple called Leopard
It's been awarded many times and it really looks great because it provides very precise picture of the system.