Working on Linux kernel modules offers a nice change of pace

I'm fortunate to be able to do most of the low level programming for my team at work, a type of programming I enjoy more than application level programming.  I got my start in software because I taught myself machine language programming while I was a field engineer 40+ years ago.  Working on computers with front panel switches and lights which made it simple to enter short machine code programs and to step through them was impossible for me to resist.  Having started at that level of programming made me a natural for doing systems programming once I made the switch to software.

The majority of the low level programming I do these days involves mapping device registers to allow me to use them to control various devices on our custom hardware.  Last week I was trying to fix a problem with a Linux kernel module (aka device driver) which behaved strangely when the system was under a heavy load as it often is.  We use an open source device driver supplied by the vendor of one of the chips on our device to transfer data via PCIe (a peripheral I/O bus used to connect peripheral devices to a system).  This device driver exposes a few of its driver variables via sysfs (a pseudo filesystem used to provide access to device information and controls to application programs).  I was using one of the sysfs variables in the application program to dynamically change the timeout control to improve data transfer performance.  The problem was when the system was under heavy loads, there was a noticeable lag between when the application program updated the sysfs variable and when the device driver actually saw the linked variable change.  With this delay, the data transfer speed was noticeably degraded.

I find this type of problem particularly interesting to try to solve.  It usually starts with trying to build a mental model of the hardware and software involved by studying the code and any documentation available.  Once the model is somewhat accurate, you can consider what modifications might affect the problem and testing out your theories one at a time.  I first tried a custom callback procedure for the module parameter in question but that still allowed the delay to occur.  Then I decided to try adding an IOCTL command to allow me to change the variable directly and was happy to discover that this approach actually worked.  With this problem solved, I'll be moving back to application problems and looking forward to my next opportunity to do true systems programming.