9. Applications

DTrace has the ability to follow the operation of applications from within the application source code, through system libraries, through system calls, and into the kernel. This visibility allows the root cause of issues (including performance issues) to be found and quantified, even if it is internal to a kernel device driver or something else outside the boundaries of the application code. Using DTrace, questions such as the following can be answered.

• What transactions are occurring? With what latency?

• What disk I/O is the application performing? What network I/O?

• Why is the application on-CPU?

As an example, the following one-liner frequency counts application stack traces when the Apache Web server (httpd) performs the read() system call:

# dtrace -n 'syscall::read:entry /execname == "httpd"/ { @[ustack()] = count(); }'
dtrace: description 'syscall::read:entry ' matched 1 probe
[...]

              libc.so.1`__read+0x7
              libapr-1.so.0.3.9`apr_socket_recv+0xb0
              libaprutil-1.so.0.3.9`socket_bucket_read+0x5b
              httpd`ap_core_input_filter+0x294
              mod_ssl.so`bio_filter_in_read+0xbc
              libcrypto.so.0.9.8`BIO_read+0xaf
              libssl.so.0.9.8`ssl3_get_record+0xb5
              libssl.so.0.9.8`ssl3_read_n+0x144
              libssl.so.0.9.8`ssl3_read_bytes+0x161
              libssl.so.0.9.8`ssl3_read_internal+0x66
              libssl.so.0.9.8`ssl3_read+0x16
              libssl.so.0.9.8`SSL_read+0x42
              mod_ssl.so`ssl_io_input_read+0xf0
              mod_ssl.so`ssl_io_filter_input+0xd0
              httpd`ap_rgetline_core+0x66
              httpd`ap_read_request+0x1d1
              httpd`ap_process_http_connection+0xe4
              httpd`ap_run_process_connection+0x28
              httpd`child_main+0x3d8
              httpd`make_child+0x86
              httpd`ap_mpm_run+0x410
              httpd`main+0x812
              httpd`_start+0x7d
               31

					  

The output has been truncated to show only the last stack trace. This stack trace was responsible for calling read() 31 times and shows the application code path through libssl (the Secure Sockets Layer library, because this was an HTTPS read). Each of the functions shown by the stack trace can be traced separately using DTrace, including function arguments, return value, and time.

The previous chapter focused on the programming languages of application software, particularly for developers who have access to the source code. This chapter focuses on application analysis for end users, regardless of language or layer in the software stack.