C H A P T E R 16 Debugging Debugging is part of the development process and the ongoing maintenance of a kernel level driver or extension. Therefore, having the skills and knowledge to debug the kernel effectively is an important part of a kernel engineer's job description. Although great care is taken during the development and quality assurance process, bugs are often unavoidable. This is partly because, once released, your driver is likely to run against hardware/software combinations that haven't been as well tested, if at all. For example, your driver may run on a faster or slower CPU than was tested initially, thus uncovering timing issues. Many regard kernel debugging as a black art, and with good reason. When an application crashes, it can be dumped into the debugger and it then is often possible to pinpoint the exact code line that caused the problem. In the kernel, things are not that easy; although debugging with the GNU Debugger (GDB) is possible, it requires some setup and often two computers. Furthermore, crashes in the kernel can often manifest themselves in completely unrelated parts of your extension, making it hard to prove if your driver was involved. You may be lucky enough to extract information or attach a debugger after a crash; however, the memory may be corrupted and the values of data structures or the call stack may not be trustworthy. If you are writing drivers for hardware devices, things may be even more complicated as malfunctioning devices (more common if using a prototype device) may also corrupt memory or cause the computer to lock up or crash. This sometimes makes it difficult to determine if it's a hardware or software problem. In this chapter, we will look at various techniques and strategies to help debug common problems. Common Types of Problems There are many reasons why the kernel may crash or why other problems may happen. However, they are usually variations on common errors and once you know what class of problem you are dealing with, it makes it a lot easier to start examining your code for problems. Let's have a broad look at some of the problems you may encounter during kernel development. · Race Conditions: A general class of bugs used to describe a problem where multiple threads of execution conflict with each other and the outcome depends on which thread gets there first. Race conditions are quite common and are often due to poor design or poor locking in multi-threaded environments. They can sometimes be tricky to reproduce and may be hiding a long time before discovery. Things go wrong when a particular sequence of events happen in a specific order, for example, because it is dependent on user input. 381