Computing setup, software focused

Windows

My company gives software/firmware developers a very capable Windows laptop.  It's got 32 cores, 64 GB of memory, and a 500 GB SSD to which I was able to add another 2 TB SSD.  There are a couple downsides to this machine -- it's kind of heavy and the corporate IT team has this laptop very locked down.  All software installed on it must be on the approved software list which I understand.  It makes it harder to use my corporate laptop as a good development environment though.  I compensate by using Linux machines for the actual development work and relegating the Windows machine to acting as a terminal plus using it for web browsing and corporate communications tasks such as email, Teams, and Slack.  I was able to install VMware which allows me to use the laptop to run Ubuntu Linux VMs which are used for occasional software builds and to host some FPGA development tools to program the firmware I develop onto the embedded hardware it targets.

• Vim - I've been using Vi/Vim as my editor of choice since I took a job which gave me a Sun workstation on my desk.  I've got friends who love various editors such as Emacs, Visual Studio, Notepad+, Atom, and a few others.  I've occasionally experimented with other editors but it's really hard to overcome the muscle memory which comes from using Vi for 35 years.  The benefit is I'm still using a few macros I created that long ago.  I prefer GVim, the GUI version of Vim for its ability to resize windows.
• Putty - I've got Putty profiles for all the Linux machines which I connect to regularly.  Each includes X11 forwarding which gives me the ability to launch X11 apps on any of the Linux hosts I access.  I know that Wayland is more modern but most of the Linux machines I'm accessing run old enough versions of Linux to make Wayland too much trouble to try to use.  Putty supports both SCP and serial connections, making it a handy way to connect to serial console ports.
• WinScp -  This is a great little network file transfer program which supports the SFTP and SCP protocols.  I often use one Linux host to build my software and another to install it so being able to easily transfer files is a necessity.
• VcXsrv -  Since so many of the tools I use require an X11 connection, I need to run a X11 server on Windows.  I actually petitioned our IT group to add this to the approved software list.  It can be a little buggy at times but getting free software approved is always much easier than asking for commercial software be added to the budget.
• TeraTerm -  Believe it or not, sometimes I need to transfer files to an embedded machine which only offers a serial connection.  TeraTerm supports ZMODEM transfers which makes exchanging binary files much easier in that case.
• Kdiff3 -  Capable free GUI file/directory comparison tool and one of the few allowed on my work laptop.
• Microsoft Outlook - Outlook seems to work adequately for basic email and scheduling needs.  It has a terrible search feature which makes it difficult for me to find past emails I need to reference.
• Microsoft Teams - I have never cared for Teams but it's a necessity to allow me to join meetings while working from home.
• Slack - Most of the time this is okay but it's very quirky.  It's a necessity to allow easy connections to people while I'm working from home. 
• Google Chrome - It's the browser chosen by our corporate IT team.  I guess it's a necessity since it's captured the majority of the browser market but I must admit that I've never really cared for its UI. 
• Windows Calculator -  Having a calculator capable of hexadecimal and binary operation is a necessity at times.

 Linux

All of the embedded devices I work with run some version of Linux.  Some actually have multiple copies running on different CPUs.

• Vim - Vim comes preinstalled on all the Linux machines I need to access.  I do often still install Gvim to make it easier to edit files over an X11 connection.
• Putty - I manually install Putty on Linux machines.  It makes it easy to connect to devices on the Linux machine which use a serial port.  It's also handy to do that via an X11 connection.
• GNU Screen - Screen makes it very easy to maintain a persistent connection to a machine which is necessary as some of the firmware builds take over an hour and network hiccups used to kill the build before it was done.  Screen allows me to reconnect after the network issue and see the build continuing as if no interruption had occurred.
• Exuberant Ctags - I've been using ctags to make navigation through unfamiliar code easier for 30+ years now.
• Ack - Ack allows me to do a recursive search for files which contain some string and to limit that search to source files or some other category such as Bitbake files.  I use the ack executable which is implemented as a single perl module which makes it trivially easy to install on Linux machines.  I know there are faster source search tools available but this one is easy to use and works well for me.
• Kdiff3 -  Capable free GUI file/directory comparison tool.
• xxd -  Sometimes I need to examine binary files and this tool is readily available on Linux machines.  It can also help transfer binary files in a pinch by converting a file to its hexadecimal ASCII equivalent, transferring it, and using xxd to convert it back to binary.
• bash - I rely on the bash shell along with awk, sed, grep, find, and other standard GNU Linux tools for the vast majority of my scripting and interactive use needs.  I rely heavily upon bash scripts to help automate much of my development process.

Apps for both Mac & iOS

Here are the apps I run on both the Mac and iOS.

• Music - Sometimes I like to have background music playing to help mask out distracting sounds while I'm working from home.  I have a few playlists which work well for this purpose.  I buy most of my music in MP3 form from Amazon or Bandcamp and import the MP3 files into the Music app.
• Apple Mail -  It's free on Apple devices and does a reasonably good job of keeping my mail accounts in sync between my Mac and my phone.
• Apple Numbers -  Even though this is a spreadsheet, it can also function as a lightweight database app.  I use it to keep track of audio books and e-books I own and which I've read.  I have quite a few of these lightweight databases.  I used to use a great database app called Bento made by FileMaker which had versions for Mac and iOS.  It was discontinued in 2013 and I haven't found a replacement I like better than Numbers although I do occasional search still. 
• Google Calendar - I like it because it seems to be the most flexible in scheduling recurring events.  For example, you can choose the 3rd Saturday of each month or the 21st day of each month.
• Microsoft To Do -  A reasonably good cross platform to do app.

Mac 

My old Intel Mac Mini is getting long in the tooth but still manages to support some of my work and all of my personal computing needs.  Here are some of the apps I run on it.

• ChronoSync -  I've been using ChronoSync for backups for quite a while now and I've very happy with it.
• Little Snitch - I insist on running a top notch firewall program and Little Snitch fills the bill nicely.
• GnuCash - Since I pay most bills electronically, GnuCash lets me keep track of my checking account easily.  I really like its ability to change the sorting from ascending to descending dates since that makes it much easier to compare with my bank's statements.
• Waterfox browser -  I've always liked the Firefox browser for how easily its UI can be customized and for the rich set up plugins which allow safer browsing.  Firefox lost me as a user when they started on their quest to shove AI in every aspect of the browser.  Waterfox is a reasonable fork of Firefox which does not include the unwanted AI features.  I use this for all my general purpose browsing.  I also use the Chrome, Safari, and Vivaldi browsers each for a special purpose.  It limits having to worry about whether cookies from financial or social media accounts are being seen by other websites.
• Vim - The command line version of Vim comes preinstalled on Mac machines.  I still install MacVim to make it easier to edit files and to be able resize windows.

iOS

I use the cheapest iPhone available whenever I need to replace my phone as my personal device.  I've got iOS software I depend upon to make life easier.

• iTunes Remote - This allows me to easily select the playlist from the Mac's Music app.  I used to also use it for pausing music when I had an incoming call but now I rely upon an Anavi Macropad 10 for that purpose which allows me to pause songs without fumbling to bring up an app on my phone.
• PCalc -  Having a calculator capable of hexadecimal and binary operation is a necessity at times and I'm not always near my Windows laptop.
• Overcast - This is my favorite podcast app.  It has the best UI of any iOS app I can think of.
• Audible - Listening to audio books keeps me sane when commuting to work or in warmer weather when I'm doing 3-4 hours of yard work. 
• Libby - App which makes it easy to borrow library books.  Mostly e-books but they also have audiobooks.
• Hoopla -  App which makes it easy to borrow library books.  This seems to have more audiobooks and graphic novels than Libby does.  They also let you borrow and stream digital movies and tv content.
• Two factor authentication apps -  I enable two factor authentication for any website which supports it and apps such as Authy, Google Authenticator, and others make that possible.

Computing setup, hardware focused

Since I work from home the majority of the time and my job involves writing software and firmware for multiple small embedded ARM based devices, my computing setup tends to be a bit more complicated than normal.  I'm also cheap so I've gotten almost everything on sale.

 

The picture above shows some of the equipment at home which makes it easier for me to switch between the many devices I need to use both for work and my personal computing needs.  The red surface everything rests upon is a grounded anti-static mat which is a necessity as I often need to use bare circuit boards for easy access to JTAG connectors.

• Philips 4k monitor.  I often have ssh sessions with anywhere between 2 and 5 Linux systems simultaneously.  That many terminal windows take up a lot of screen space so the fact that a 4k monitor has 4 times as many pixels as the HD monitor I'm stuck using at the office has made working so much easier.  I got this at a good price thanks to a Black Friday sale 3 years ago.
• IOGear KVM switch.  I usually switch between my work laptop, my Mac Mini which is mostly for home use, and a Raspberry Pi 5.  Since this is a 4 port KVM switch, that leaves an extra set of cables to connect to other devices as needed.  I had a cheaper KVM but the move to a 4k monitor forced me to upgrade to one which supported the higher resolution.
• Technical Pro rack mount power supply.  This allows me to easily power on or off any individual device in my setup as needed.
• Raspberry Pi 5.  This is a great development device as there are much better free development tools available for Linux than there are for Mac or Windows.  That provides me with a lot of flexibility in my development and debugging tasks.  It's my favorite computer to work with.
• Anavi Macropad 10.  This device comes with CircuitPython installed which makes it very easy to configure the key codes it can to send to my Mac.  I like this one because it's also got a rotary encoder (which can be used to easily change the volume) with a button on top to pause the music.  I often use my Mac Mini to play background music while I'm working.  Being able to pause music playback, skip to the next song in the playlist, rewind to the previous song, or adjust the volume on the Mac while my keyboard and monitor are switched to my work laptop via the KVM is very handy.  The situation which this keypad helps with most is when I get a call from someone at work and need to quickly pause the music in order to take the call.
• Canon PIXMA G6020 All-in-One Megatank Printer - It's affordable, reasonably fast at printing, doesn't use proprietary ink cartridges, and has copying and scanning capabilities as well.  We've had it a year and are still on the original ink bottles supplied with the printer.  I think we've been through about 6 reams of paper during that time, most of it double-sided.

 

The picture above is an EDID (Extended Display Identification Data) HDMI adapter.  It causes the computer which it's plugged into to continue thinking that the 4k monitor is still plugged into the computer when it's actually switched to another computer via the KVM switch.  The EDID HDMI adapter shown above is connected to my work laptop.  I've got another one plugged into my Mac Mini.  This prevents the windows on whichever machine from being rearranged when the monitor configuration changes via the KVM switch.  More expensive KVM switches sometimes include this capability without requiring an external adapter like this.

 

 

Whiteboard coding during interviews

I recently responded to a writer I follow on Mastodon and Bluesky who was looking for input for an article she was planning to write about interviews for embedded systems positions.  Since I enjoy her writing and I have loads of experience with embedded systems, I responded to her request.

The act of recalling various interview questions I had been asked or had posed to others made me remember some questions which I disliked intensely.  The type of question I dislike most is being asked to write code on a whiteboard to solve some problem.  I generally enjoy coding and the problems are often interesting but coding on a whiteboard is so different from writing code on computer using a good editor that I've grown to hate it.  I'd almost rather go back to the bad old days of using a keypunch to generate punched cards as source for my program.  When programming I like to start with an outline of the program and fill it in as ideas occur to me which is nearly impossible when stuck with a more linear coding environment such as whiteboard or a sheet of paper.

I've had two memorable experiences responding to requests to code on a whiteboard.  The first was the most absurd question I believe I have ever been asked in a job interview.  Apparently the company I was interviewing with had guidelines for technical interviews and the engineer asking the questions was hellbent on sticking to these guidelines.  He asked me to write an assembly language program for him.  I asked which assembly languages he was familiar with since I was comfortable with about 5 at the time.  Once I discovered that we had no assemblers in common, I pointed out that it made little sense for me to write a program in a language he wasn't familiar with but he insisted on me completing this task so he could check it off on his interview form.  In addition to it being ridiculous to write a program which cannot be evaluated, it's also frustrating trying to write a meaningful assembly language program on the tiny whiteboards available in cubicles since assembly programs tend to be much longer than those written in higher level languages.  Apparently I passed muster as I ended up getting the job.

The second whiteboard coding experience which came to mind ended far more positively.  I was asked to solve a problem in C but ended up needing to keep adding lines which is no problem on a computer but presents a huge obstacle when stuck using a whiteboard and marker.  I managed to come up with an incomplete and messy solution before they called time on me.  Once I got home from the interview, I was still bothered by my performance.  I was able to quickly code up the solution on my computer at home and emailed a working program to the VP of Engineering I had been interviewing with at the time.  He was so impressed that I had followed through with solving the problem that I was offered the job and ended up staying at that company for 9.5 years.  I still hated the initial request for whiteboard coding but was pleased that it resulted in me getting the job.

I'm hoping that the days of being asked to code on whiteboards are relegated to the past.  To tell the truth, since I'm at my last job before retirement. it won't have a huge impact on my life either way but I hate the thought of others being subjected to this absurd practice.

The benefits of a varied technical background

This week I helped a colleague solve a strange problem which he had encountered.  He was modifying some application level code which needed to read from and write to a device driver I had created to control two LEDs on our device.  Previously the application code had only written to the driver but the decision was made to control the two LEDs independently which required reading the old state from the driver prior to updating the LED state.  Fortunately I had included the ability to read the LED status since it improved my ability to debug the device driver.

The problem was caused by the need to interleave reads and writes to the device driver which under Linux gets treated as a file.  Unbeknownst to my colleague, any read or write to a file stream affects the file pointer which keeps track of the location within the file which will be accessed next.  A simple device driver has no need of the file pointer concept but since Linux treats devices as files, the standard library code which enables accesses to devices and files keeps track of the supposed file pointer even if it doesn't need to do so.  In a standard file access, I should have been able to do a fseek (file seek) to the current position between the read and write calls to fix this issue.  Unfortunately, since my device driver is very bare bones, I suspect there was extra call within the device driver needed to handle fseek calls.  I used a brute force fix of closing the device driver and re-opening it within the application code.

Somehow this makes me think of the common wisdom from early in my career which suggested one shouldn't change jobs too often lest one be labeled a "job hopper".  It turns out that job hopping has given me a very diverse background which has improved my chances of finding jobs.  Changing jobs more frequently has helped me escape from jobs where the work was boring or which placed me under managers which were difficult to deal with.

More machine language fun

When I first starting working as a Senior System Analyst at GEISCO (GE Information Systems) in the mid-1980s, they had us logging into mini and mainframe computers via terminals.  Several of the commands we had to use needed elevated privileges which required us to enter a password of the day.  In order to get this special password, they gave us a small script which retrieved this password and most people put a call to this script as part of their network login to automatically show the password of the day.  Being a curious sort, I wanted to know how the script to display the password worked.  Most people found it cryptic since it consisted of several groups of 12 digit numbers and none of the digits were larger than 7.  I knew this likely meant that these digits were octal numbers which require 3 bits each to represent.  Couple that with the fact that the groupings of numbers were 12 digits long told me that they represented 36 bit words.  Since I knew GE made heavy use of Honeywell mainframe computers at the time, I concluded that the script was some type of interpreted machine language program.  So I dug out my old Honeywell assembly language documentation and discovered that the script was a simple little program to issue a system call (MME - Master Mode Entry) and then print out the results.  To test my theory further, I modified the program to shift the characters of the master password so they would print out backwards.  It basically served to entertain me each time I logged in.  It's amazing the little challenges which I find amusing, huh?

While I was working at GE, a project was launched to upgrade the storage device on the CC (Central Concentrator) network node.  One of the tasks performed by the CC was to load software on the other, smaller network nodes and its original 2 MB device was deemed too small to handle network expansion.  Believe it or not, that 2 MB storage device was a magnetic drum from Vermont Research.  I had signed up for this project because the replacement storage device was originally specified as a 10 MB hard drive similar to those used on higher end PCs of that time.  I was anxious to get experience on these disk devices which were cutting edge technology at the time and writing a device driver from scratch sounded like fun.  Somehow Vermont Research found out about the project and submitted a lower bid for an upgrade to a 10 MB drum device.  So my dreams of writing a device driver became the much less interesting task of updating the old device driver to extend the addressing to accommodate the extra storage.  The only challenging part of the project was that the diagnostic program also needed to be updated and somehow the source code for the diagnostic had been lost.  So I was forced to read the punched card deck into the mainframe in order to print out the binary data the deck contained so I could disassemble it.  Then I had to figure out how to write a patch for the diagnostic program.  And finally, I had to figure out how to get the mainframe's card punch to reproduce the same punch card format used by the diagnostic.  For a few days the computer operators for the mainframe got used to me making multiple daily attempts to convert the binary file containing my patches into a format which could be punched in the same format as the diagnostic deck.  They told me that they hadn't seen anyone use the card punch in many years.  Each attempt required me to tweak my program to convert the diagnostic's binary data into a slightly different format.  It wasn't as much fun as I had hoped for but it did prove pretty challenging.

The joys of machine language programming

When I started my career as a field engineer for Honeywell mainframe computers in the late 1970s, I worked a lot of swing and midnight shifts.  While day shift was always pretty busy, the night shifts were often boring.  To entertain myself, I read the CPU manuals with the goal of being able to modify the diagnostic programs used to test the computers.  Occasionally it proved handy to load one of the diagnostics and then to patch them in memory to loop through operations which were failing.  This allowed using an oscilloscope to trace signals of interest though the 80 wire-wrap boards which made up the CPU.

Eventually writing these machine language programs became my favorite pastime on slow nights.  Part of the draw was the maintenance panel switches which made it easy to read and write memory locations.  There was a definite thrill to getting a program working and watching its progress via the flashing lights on the maintenance panel.

For those who aren't familiar with low level programming, machine language programming involves directly entering the binary encoded instructions into memory locations for later execution.  More people are familiar with assembly language programming which replaced the binary programming with mnemonic names for the instructions and any modifiers.  For example, a Honeywell mainframe had an instruction called LDA which loaded the A (or accumulator) register with some value.  In machine language programming, that LDA instruction had the opcode of octal 235.  Older mainframes often used octal encoding instead of the hexadecimal encoding which is more often used today.  The other convenience offered by using assembly language over machine language is that the assembler would calculate the addresses automatically rather than forcing you to manually calculate the address offsets by hand which was painful.

My second job was as a field engineer for DEC PDP-11 minicomputers.  These smaller machines were so much less complex than the mainframes that fixing the hardware wasn't much of a challenge.  The saving grace was the PDP-11 instruction set was simple enough to allow me to quickly come up to speed on its machine language.  When I was in Boston for training, I wrote a machine language program to determine which terminal connected to the PDP-11 had had data entered on its keyboard.  Apparently the way I approached programming was different than most people's because the instructors had trouble figuring out how my program worked.

Believe it or not, the ability to decipher machine language is still useful when I have to use gdb to debug a program.

Configuring Windows/Mac/Linux for embedded development

A few days ago Scott Hanselman posted an interesting question on Bluesky.  He asked how much stuff people needed to add to Windows to make it useful for day to day work.  He also asked a similar question of Mac users.

Admittedly, my use case differs from that of most people.  I do embedded firmware development.  For me, my company Windows laptop mostly acts as a way to connect with the Linux build machines and target machines I use.  It's really little more than a glorified terminal except for running Outlook, Office, and Slack.

Windows

Having made the switch to a Mac at home 24 years ago, I only use Windows at work now.  On any new Windows machine, I first install the following software.  It's all free software as most companies I've worked for make it so difficult to justify the purchase of commercial software, that it's not worth the effort.

• Gvim - I occasionally need to do some local editing on Windows and for that a graphical version of vi is an absolute necessity for me.  I've been using some version of vi for 35+ years and while I've had occasionally dalliances with other programming editors, I've always returned to vi.
• VcXsrv - Being able to launch graphical applications remotely makes my life much easier.  That means using an X11 server.  I know there's pressure to move to Wayland but it strikes me as more effort than it's worth at this point.  It's the same feeling I have when I hear someone suggest that I try writing a device driver in Rust.  I just want to get work done, not spend time blazing a trail.
• Putty - I need to connect via SSH or serial communications to a number of Linux machines (build servers, target systems, etc) and Putty is my hands down favorite way of accomplishing this.  I make sure to enable X11 forwarding on Putty SSH sessions because this allows me to launch GUI programs and have them display on my Windows laptop.
• WinSCP - This allows me to easily copy files back and forth between Linux machines and my Windows laptop.  It also enables easy remote editing of files which reduces the pain of editing a file on a remote machine over a slow Internet link.

Mac

When I first started using a Mac at home, I loved the development environment which the combination of Mac OS X, Xcode, and the Quartz X11 server provided.  It was the best development platform I had seen since my days last using a Sun workstation in 1996.  Over time and Apple's push to combine features of iOS and Mac OS, it's become much harder for me to set up a reasonable development environment on the Intel Mac Mini which serves as my desktop machine at home these days.  Since most of my embedded development is done for work, that's not a deal breaker.

• MacVim - As mentioned above in the Gvim section, I need to edit files locally on my Mac.  MacVim gives me a version tailored for use on Macs.
• Homebrew - Unfortunately, many of the tools I've come to rely upon are only available through an alternate install path.  Homebrew gives me access to a number of development tools not available through the Mac AppStore.
• XQuartz - This X11 server used to be available in the Xcode tools but now the best version seems to require being installed via Homebrew.
• Unfortunately I have not found a free GUI SCP application for Mac I like yet so I resort to using the standard Mac Terminal app and the command line scp tool.

 Linux

I use a Raspberry Pi 5 at home since Linux is orders of magnitude better at interfacing with a variety of small embedded machines than either Windows or Mac are.  I typically use a pared down Linux distribution because I don't need the typical blend of applications like Open Office.  I've been using Debian Bookwork with the Xfce desktop environment.  

It's easy to install X11 apps, Gvim, and Putty on Linux.  The IT group at work has our Windows laptops very locked down so installing new software such as the GUI software for a USB protocol analyzer sometimes requires getting it approved which can take a few days.  Mac has gotten harder to run third party application software as well, much like the iOS app store which is very locked down.  Development goes so much faster when I can install any software I need without facing roadblocks.

Linux is also good at doing compiles for the firmware and application software I create for the newest embedded ARM device at work which is also an ARM 64-bit processor.  It has better USB support too.  Windows often requires the installation of device drivers for various USB serial devices which can be hard to do when using a laptop with limited admin rights.

WordStar

I recently commented on a post on Mastodon about Wordstar by Tom Jennings (yes, the one associated with FidoNet).  In his post Tom extolled the virtues of Wordstar for what a good piece of software it was and I completely agree.  Not only was it good for its time but it compares quite favorably against modern software.  It needed to be configurable because software which ran on CP/M often required customization of the display and printer settings to match the hardware connected to the user's machine.  It was also quite robust.  I last used it around 1989 and I don't recall it ever crashing.  I cannot say the same about any modern word processors I use.  Finally, it was remarkably full featured for its time.  I recall being excited to discover that it had a column editing mode which at the time I had only seen on IBM's PE (Personal Editor).

I appreciate both Mastodon and Bluesky because they allow me to see what favorite authors, scientists, engineers, and artists are up to at the moment. 

 

A few of my favorite programming environments

In my 40+ year career in software/firmware engineering, only 3 of the computers I've used have had what I considered superior development environments.  This type of environment makes it easy to be more productive.

The first was the Sun 3/80 Workstation i used at work in 1990 while writing software for packet switching communications equipment.  It had a Motorola 68030 CPU which is still my favorite assembly language and offered good performance.  The Sun offered my first extended exposure to Unix and it was love at first sight.  Unix was mature by that time and the development tools were top notch.  The only possible drawback was it only had a monochrome display albeit a high resolution one.  I enjoyed the way it allowed me to define a default window layout so I could have 4 terminal windows in standard locations each time I logged on.

The second machine I thought offered a great development experience was the Apple PowerMac G4 I bought for home use in 2002.  It was purchased to replace a PC because I had grown tired of Windows XP plus I had picked up an iPod 10 GB music player which integrated much better with Mac machines than it did with Windows.  The PowerMac G4 ran OS X, a Unix like OS derived from NeXT's NeXTSTEP. It had a PowerPC G4 CPU which was my first exposure to the RISC architecture.  OS X came with the X11 (aka X Windows) window manager which was also used on the Sun workstation.  X11 enabled me to run many of the same tools I had grown used to using on the Sun.  At the time I was working for an optical networking startup.  Some of the boards in our networking equipment used PowerPC G3 and G4 CPUs so it was handy having one at home to experiment with.  The Mac also gave me an environment which was very close the the Sun I had started using 12 years earlier with the added benefit of supporting a color display.  I've come to love color syntax highlighting of source files.  I eventually made the switch to Intel Macs and then Apple's own CPUs.  Sadly, they also seem to be making Mac OS look more like the iOS used on their mobile devices at the cost of making it less useful as a development machine.

The third (and my current favorite) development environment I've considered exceptional is a Raspberry Pi 5 which runs a multicore 64-bit ARM CPU.  It still amazes me that I could built a reasonable development machine (8 GB memory, 500 GB SSD storage) for less than $200.  Since it runs Linux with X11, I'm once again able to run the same tools (or their successors) on this machine.  It integrates better with the small embedded ARM machines I write firmware for than does my work laptop running Windows 10 or my Mac Mini which runs the latest iteration of MacOS.  Sadly X11 is no longer easy to get running on the Mac.  I know some will think I need to migrate to Wayland but I rarely feel the need to use tools which aren't fully mature yet.

All of these machines have run Unix or Unix-derived OSes which has been my strong preference for 35 years now.  This allows me to edit source files using vi (now vim) which had a steep learning curve but rewards the initial effort by being fast and quite powerful.  Having ctags allows me to quickly jump from a reference in a source file to where the function or variable is defined.  I've used it for so long that it has become second nature to me.  I'm happy to be able to search my source code efficiently again using grep.  And I love being able to use a shell with a proper scripting language (currently bash but I've used sh, csh, and ksh in the past).  Best of all, I've still got good support for the C programming language which I view as a portable assembly language and which has been my favorite since I first learned it in 1985.

These aren't the only environments which can improve productivity.  I've worked with a few engineers who swear by Visual Studio or Eclipse and I've seen how both can help speed up common tasks.  However both have a sufficiently steep learning curve for me to ignore them.  I feel the same way about Emacs when friends point out its advantages over vim.  At this stage in my career, I'm not willing to make radical changes to a workflow I've been comfortable with for a long time at the cost of weeks to months of feeling less productive.

The non-traditional start of my software engineering career

Unlike most software engineers, I started my career as a field engineer responsible for maintaining those large mainframe computers like the ones you see in movies.  You know the type with the line of tape drives whose reels keep spinning back and forth.  I got that job by attending the last full time field engineering class taught entirely by live instructors at the Arlington campus of Control Data Institute in 1976.  It was a 6 month full time program which taught electronics repair, debugging, and machine language programming.  They delved into topics such as optimizing Boolean logic which we then implemented by wiring together small boards each of which contained a single Boolean gate made up of discrete components such as transistors.  You haven't lived until you've built an adder circuit this way.  One of the big attractions was Control Data consistently managed to find jobs for more than 90% of their graduates from each class.  For the class which followed mine, they started using an early version of a computerized training system called PLATO for part of the training and eventually switched entirely to computer-based instruction.

In 1977, I completed my course and got hired as a field engineer by Honeywell Information Systems with the responsibility for maintaining their Level 6000 and 6600 series mainframes.  Much of the time I worked on peripheral equipment such as disk drives, tape drives, and line printers.  It's a sad fact of life that machinery with moving parts breaks down a lot more often than purely electronic equipment does.  Fixing mechanical problems is a necessary part of the job for field engineers but it's not always interesting and it often means getting covered in printer ink or grease.

My favorite part of the job was the more difficult debugging required when a computer fails to boot or when one crashes.  When the computer itself had problems, I occasionally had to troubleshoot the CPU the way the field engineer in the picture below is doing. 

The CPU for Honeywell's 6000 series of mainframes contained about 80 logic boards, each of which measured about 12" square and contained over 100 integrated circuits.  For the most difficult problems, we sometimes had to put the boards on board extenders similar to the one in the picture above which provided access for easier debugging using an oscilloscope to trace signals from board to board within the CPU.  For me, the best part of those mainframe computers was the large maintenance panels they had.  From that panel I could stop the CPU to examine the contents of registers, single step the CPU to see how it behaved while executing a section of software, and insert breakpoints to automatically stop on certain conditions.  That was when I was infected by the love of programming.  I'd get 2+ hour windows for preventive maintenance and if I rushed through running the system diagnostics, I'd have time left over to enter machine language programs on those maintenance panels.  Seeing the lights on the panel blink in ways I expected was a thrill.  I spent 5 years working at Honeywell but ultimately left because I was frustrated by the obstacles they presented to employees who want to make the switch from hardware to software.

My second job was with a small company called Atex which used DEC PDP-11 minicomputers to run publishing software used by magazines and newspapers.  For a while I was the onsite field engineer at USA Today and helped install the 12 PDP-11 minicomputers they used when they launched.  Most of my work was maintaining the 200+ terminals used by newspaper staff.  The terminals were simple but company policies declared the ORU (optimal repairable unit) to be the logic board which time consuming to replace.  So I started saving the chips which commonly caused failures from all of the defective boards I sent back to be repaired and I learned which chip to replace for various symptoms.  A vertical line across the screen meant the horizontal deflection chip was likely bad.  That allowed me to reduce my time to repair 45 minutes equired to replace the logic board to about 5 minutes to pop a new chip into a socket.  The time savings left me ample time to write machine language programs I could try during preventive maintenance windows.

Since the PDP was a smaller computer, its maintenance panel was much simpler (see below) than the one used by mainframes.  The CPU's instruction set was much simpler as well.  Maintaining minicomputers proved much less challenging than working on mainframes was.  Their CPUs only had a couple boards and instead of doing chip level repairs, we just replaced the bad board.  Frankly, I found the job a bit boring.  After a mishap where a minicomputer fell on top of me (which is a story for another time), I decided to make the switch from computer hardware to software. 

What I discovered at my first software job was thanks to my unorthodox introduction to programming, I had a much better understanding of low level programming than most of my colleagues did.  I also appreciated that as a software engineer, I was able to write assembly language programs rather then entering instructions as machine language which is just a group of numbers.  Writing assembly language was much less labor intensive.  The instruction "LDA" (load the accumulator register) was easier to remember than the fact that a 235 was the CPU's "op code" for a LDA instruction.   I also didn't have to manually calculate the jump offsets between instructions.  Yes sir, assembly language programming was much easier than what I was used to doing.

At GE, which was my 4th job, I wrote code for Honeywell 4500 series computers.  These were process control systems with a 24 bit CPU which had been modified for use in GE's packet switched network.  The 4500 was yet another another machine with a front maintenance panel.  Most of the code we wrote was in assembly language but the assembler wasn't very sophisticated and it wouldn't give a proper warning when a jump location was too far away.  What could happen was the jump offset might get large enough that it would set the top bit of the offset field, changing the jump direction from forward in memory to backward which caused unpredictable behavior.  I could catch these errors pretty quickly because the assembly listings from our programs also showed the machine language instructions one one side.  In the machine language column the direction of the jump offset was obvious to people who understood the instruction set.  Eventually I was put on a project to upgrade the network nodes to more modern hardware.  I was sent off to be trained on the new machines where I discovered that my management had completely ignored the class prerequisites of knowledge of the C programming language, much to the instructor's chagrin.  So I was forced to learn C during a several week course where I was expected to already have expertise in C and was also learning about the equipment's API.  It was hard until I finally realized that C is really just a portable assembly language.  After that, C became my favorite programming language and remains a favorite to this day.

Eventually I joined Sprint International where I was responsible for developing software for the equipment used in their packet switched network.  It was at Spring that I became obsessed with Unix because I had a Sun 3/80 workstation on my desk.  To this day, the Sun remains my idea of the ideal development environment.  It was also there where I first used a Motorola 680x0 CPU which features my favorite instruction set of any CPU.  One thing I found frustrating was Sun's version of vi (a visual editor) had restrictions on the width of a window and the overall size of the files which could be edited.  My machine language skills allowed me to patch my workstation's copy of vi to expand those restrictions to values I found easier to live with.

The vast majority of my career has been spent writing system software (operating system, device drivers, bootloaders, etc) or firmware.  The line between system software and firmware is sometimes hard to detect.  It can range from simple control loops which monitor and control hardware to what it has become today, hardware specific drivers on any one of multiple embedded operating systems, up to and including Linux.  Regardless of the platform I'm working on, the ability to decode machine language remains a valuable tool for me.  Since my firmware jobs often involve custom hardware, knowing how to read schematics has proved an essential skill as well.

Making changes to your development setup easier

As I'm occasionally assigned new projects at work and as those projects progress, I find the need to update my development environment on a regular basis.  Having the ability to customize my setup allows me to be more productive.

To that end, I have defined a few bash aliases which help make this process easier for me.  These aliases are defined in my ~/.bash_aliases file.

#  config bash

alias   cfgb='gvim ~/.bash_aliases'

# config vim

alias   cfgv='gvim ~/.vimrc'

# source my bash aliases to pick up any new changes

alias   redot='source /home/rod/.bash_aliases'

You'll notice that I have an alias to make changes to my vim setup.  It makes sense to also have a vim macro to read any new configuration settings or vim macros.  This macro is defined in my ~/.vimrc file.

" source .vimrc ro pick up any new macros

map \.  :source /home/rod/.vimrc^M

Default apps for 2023

Lately, I've seen a lot of the "what apps am I using" type posts, apparently prompted by a podcast episode.  You can see a collection of what a number of people are using here.  I find these interesting as they're a good way to discover new apps which I hadn't been aware of previously.  These are the apps I use for my iPhone, Mac, and Raspberry Pi at home as well as the Linux and Windows machines I use at work.

• Mail Client: Apple Mail, Outlook at work (barely tolerable but dictated by the IT team)
  
• Notes: Apple Notes for shared notes, Editorial for notes I don't want on the cloud (such as those containing sensitive information such as birthdays)
  
• Chat: iMessage
• Camera: Apple Camera, Night Capture for evening sky photos
• Photo Management: manual sync into directories on my Mac
• Photo Editing: Preview on Mac, IrfanView on Windows
  
• Calendar: Google Calendar - I like it for its ability to do custom repeats and for ease of having more than 2 notification reminders
• Browser: On Macs I use Safari for lightweight browsing for its ease of sharing bookmarks with mobile devices, Firefox for general browsing because it has a richer collection of security plugins.  At work I use Chrome (barely tolerable but dictated by IT).
  
• Backup: ChronoSync
• Read It Later: Instapaper
• RSS: Inoreader (web and app)
• News: RSS feeds from news sites which lets me keep up with events without being distracted by things I'm not interested in such as sports
  
• Podcasts: Overcast (great performance and the best UI of any app on my phone, bar none)
• Books: Kindle, Audible, Libby, Hoopla
• Database: Tap Forms (good sync between mobile and Mac)
  
• Personal Finance: GNUcash
• Password Management: 1Password - with 600+ non-trivial passwords to remember, this is a must
• Music - Apple Music, Remote app to control desktop from my phone
• Editing (code and general) - MacVim on Mac, GVim on work machines (Windows and Linux)
• X Servers - XQuartz on Mac, VcXsrv on Windows
• SCP client - WinSCP on Windows, scp from the shell on Linux/Mac machines
• Terminal emulation - putty on Windows and Linux machines
  

An easier way to edit remote files

 

My main development machine at work is an Ubuntu desktop machine which I've got set up with cross compilers and a number of other development tools.  It's reasonably fast but because it runs Linux, it can only be connected to our lab network at work.  This used to mean having to ssh into the machine to edit files which on a fast network with X11 forwarding means I can use a GUI editor.  However I've been noticing a maddening amount of network lag between our lab network and the corporate network which means I end up waiting several seconds for things like cursor movement to take place.  That can break my train of thought.

Fortunately I discovered that WinScp has the ability to not only transfer files between systems but it can also allow me to edit files on those systems.  I launch WinScp, tell it to connect to my development machines, navigate the the appropriate directory, right click on the file I want to edit, and select launch external editor.  It can be configured to use your choice of editors on the Windows machine where WinScp is running.

Once you make the request, WinScp transfers the file to your local machine and launches your editor of choice.  Once you're done editing, it transfers the updated file back to the remote system.  The actual editing all takes place on your local Windows machine so it prevents network lag from causing a painfully slow editing session.

Showing differences between binary files

At work I'm having to port some firmware changes to a new board.  This requires I understand the boot process in excruciating detail.  The old board boots with 3 different files stored in flash memory (bootloader, bootloader environment, and Linux file system).  I have the files as well as several raw dumps of various areas in the flash memory chip made with the "dd" command and needed to know whether there was any magic done to identify the different partitions of the flash device.

So I needed to show differences between binary files.  I normally use meld for file comparisons but it doesn't handle binary files.  It's easy to write a simple program in C to show simple byte differences but I wanted something able to display context for any differences.

It turns out that this is very easy to do in bash thanks to the ability to easily combine tools.  This is made possible by the Unix Philosophy.  The following command does exactly what I was looking for.

meld <(xxd file1.bin) <(xxd file2.bin)

So I whipped up a small shell script to make this easier and I've got a new tool for future use.  As you can see from the image, the display would be helpful in figuring out the differences between files.






Exploding windows

I've encountered "exploding windows" on both Windows and Linux lately.  I try to resize a window and by clicking the window near the title bar, the window suddenly and unexpectedly expands to fill the entire vertical area of the monitor it inhabits.  Sometimes I'm able to do the simple resize operation which I wanted but randomly the operation results in an exploding window.  Perhaps the UI designer thought this was a neat feature but it causes me to curse each time I encounter it.  I'd love to figure out how to disable this behavior but each time I try to describe it in a Google search, I'm led down a rabbit hole of unrelated results.  It seems silly spending too much time researching something which while annoying only costs me annoyance and a few seconds to correct.

Developers need IT skills

Often software developers can benefit from having skills typically found in IT support engineers.  For example I occasionally bring small embedded systems home from work since it makes working from home more productive.  It's hard to physically reconfigure equipment I've left at the office and the reset button is tough to hit across a VPN.  

In this scenario I need to be able to access the small embedded system on our home LAN as well as my build desktop machine which stays at work.  Having a VPN client which supports split tunneling makes this possible.  It requires that there be no overlap in IP addresses between the two LANs.  It also requires that I change the static IP address on the equipment I bring home to match the subnet which my home router uses.

The diagram gives a rough idea of how the split tunneling works.

Getting paid to solve puzzles

 

The thing I enjoy most about my job developing firmware for embedded systems is that it's a lot like being paid to solve puzzles.  Many of the projects I work on are just as challenging as the old text based adventure games such as Infocom's Zork series.

The datasheets which contain information about how the chips in embedded hardware are supposed to function can be challenging to decipher.  Vendors do their best but it's not unusual for the datasheets to either be incomplete or to contain subtle inaccuracies.  A chip I developed a device driver for recently had an accurate datasheet but the device driver gave the wrong results because of an issue with the C compiler for the ARM processor.  It turns out gcc for ARM CPUs does not support signed character types by default which this chip required.  Fortunately gcc includes a compiler flag "-fsigned-char" which allows this strange behavior to be overridden.

My worst technical interview

About 38 years ago I interviewed for a software job at GEISCO (GE Information Systems).  This would be my second software position after making the switch from field engineering.  It was a systems software position, writing software to run on one of their network nodes.  At the time GEISCO supposedly ran the largest private data network in the world

One of the engineers I interviewed with felt obligated to test my software skills by asking me to write a short program on his whiteboard.   The problem was most systems software was written in assembly language at the time.  I knew 4 assembly languages which didn't overlap at all with the assembly languages he knew.  Upon learning that we didn't share a language, I asked if he still wanted me to write a program since there was no way he could determine the correctness of my program.  Being a good corporate drone, he insisted that I still needed to complete this step so he could mark it off on the interview form.  So I proceeded to write some code in GMAP, the assembly language used by Honeywell mainframes since that was the one I was most familiar with.  I guess I talked my way through it successfully because I ended up getting the job.  

My program wasn't very functional as I wrote system software and it was rare for me to need to use system calls.  I definitely had never written a system call which performed any I/O.  I believe I loaded an immediate value into a register, performed a left shift, saved it into a memory location, and then ended the program with a MME GEFINI system call which terminates the current program.

Even though I first encountered GMAP assembly language 45 years ago, I still remember the numeric value for the opcodes of a number of the instructions.  Perhaps some day I'll once again need to know that a 235 opcode is a LDA instruction (load accumulator) but I'm skeptical.  There's a lot of old knowledge I'd love to be able to purge to reclaim the memory space.

Remote Desktop replacement

At work I have 5 Linux machines (1 desktop and 4 tiny embedded machines) and a Windows laptop in my office which I need to use.  There are KVM switches which would allow to connect my monitor, keyboard, and mouse to the machines in that large a setup but they're expensive and the cabling would be a nightmare plus I'd be stuck using just one machine at a time.  I tried using Remote Desktop and VNC to access the desktop of another machine remotely but that's cumbersome and it's slow across a VPN.  To be fair, there are situations when Microsoft's Remote Desktop works wonderfully across a VPN.  At my last job I used to use the Mac version of Remote Desktop to access my Windows laptop at work.  Since a Windows machine was the destination, Remote Desktop did a great job of data compression plus it allowed the client machine to arrange windows differently than they were arranged on my Windows laptop at the office.  So my old 27" iMac which has nearly 4k resolution gave me a better viewing setup than I had at work.

Now I use X11 (aka X-Windows) for accessing all the Linux machines I need access to.  Making this work requires a couple pieces of software.

First off, I need an X Server on my Windows laptop.  That is used to display the windows created by the remote Linux machine.  I use VcXsrv, a version of the classic xorg X Server but recompiled under Visual C++.  This recompilation allows it better access to Windows resources and I find it works better than any other X Server I've tried such as xming or cygwin's xwin.  VcXsrv needs to be run on your Windows machine prior to attempting a connection to another machine.

The next thing required is an SSH client which is capable of X11 forwarding.  That allows any programs launched using the SSH client to send their displays back to the client machine.  I use PuTTY, which has been around a long time and which can handle serial and SSH connections.

Once you've run VcXsrv on your Windows machine and then used PuTTY to connect to a Linux machine (taking care to first set the configuration to allow X11 forwarding), launching a remote program and having it display on your client machine is as simple as typing the program's name.  As an example you can try xterm which will create an X11 terminal window.  Just be sure to launch it in the background by appending an ampersand so the program is placed in the background.  Otherwise, your input and output will remain tied up by the program being launched.  The command to launch xterm this way would be "xterm &".

GNU Screen can make SSH sessions persistent

 I do a fair number of Linux kernel builds at work.  Doing a Yocto build for our target machine takes about an hour and 40 minutes for a the build to complete and the root filesystem to be prepared.  If I do that from home, there's a chance that I'll get a hiccup on my VPN connection which might disrupt the build in progress.  Even at work, I ssh into my Linux build machine from a Windows laptop and use xterm sessions for most of my development tasks.  If I kick off a build before heading home, there's always a chance that Windows will decide to do an update which could also disrupt my build.

Fortunately GNU Screen can make your sessions persistent through disruptions due to network disconnections or other reasons.  Now I make sure to start a GNU Screen session before doing my build.  I start named screen sessions with a command such as "screen -S build".  That way, if my session is disconnected, I can rejoin it with the command "screen -r build".

The one thing I don't like about GNU Screen is its default escape key, Control-A, since that interferes with the default Emacs command line editing mode in Bash.  So I've created a ~/.screenrc configuration file with the following command to change the escape key to Control-T.

escape ^tt