More machine language fun

When I first starting working as a Senior System Analyst at GEISCO (GE Information Systems) in the mid-1980s, they had us logging into mini and mainframe computers via terminals.  Several of the commands we had to use needed elevated privileges which required us to enter a password of the day.  In order to get this special password, they gave us a small script which retrieved this password and most people put a call to this script as part of their network login to automatically show the password of the day.  Being a curious sort, I wanted to know how the script to display the password worked.  Most people found it cryptic since it consisted of several groups of 12 digit numbers and none of the digits were larger than 7.  I knew this likely meant that these digits were octal numbers which require 3 bits each to represent.  Couple that with the fact that the groupings of numbers were 12 digits long told me that they represented 36 bit words.  Since I knew GE made heavy use of Honeywell mainframe computers at the time, I concluded that the script was some type of interpreted machine language program.  So I dug out my old Honeywell assembly language documentation and discovered that the script was a simple little program to issue a system call (MME - Master Mode Entry) and then print out the results.  To test my theory further, I modified the program to shift the characters of the master password so they would print out backwards.  It basically served to entertain me each time I logged in.  It's amazing the little challenges which I find amusing, huh?

While I was working at GE, a project was launched to upgrade the storage device on the CC (Central Concentrator) network node.  One of the tasks performed by the CC was to load software on the other, smaller network nodes and its original 2 MB device was deemed too small to handle network expansion.  Believe it or not, that 2 MB storage device was a magnetic drum from Vermont Research.  I had signed up for this project because the replacement storage device was originally specified as a 10 MB hard drive similar to those used on higher end PCs of that time.  I was anxious to get experience on these disk devices which were cutting edge technology at the time and writing a device driver from scratch sounded like fun.  Somehow Vermont Research found out about the project and submitted a lower bid for an upgrade to a 10 MB drum device.  So my dreams of writing a device driver became the much less interesting task of updating the old device driver to extend the addressing to accommodate the extra storage.  The only challenging part of the project was that the diagnostic program also needed to be updated and somehow the source code for the diagnostic had been lost.  So I was forced to read the punched card deck into the mainframe in order to print out the binary data the deck contained so I could disassemble it.  Then I had to figure out how to write a patch for the diagnostic program.  And finally, I had to figure out how to get the mainframe's card punch to reproduce the same punch card format used by the diagnostic.  For a few days the computer operators for the mainframe got used to me making multiple daily attempts to convert the binary file containing my patches into a format which could be punched in the same format as the diagnostic deck.  They told me that they hadn't seen anyone use the card punch in many years.  Each attempt required me to tweak my program to convert the diagnostic's binary data into a slightly different format.  It wasn't as much fun as I had hoped for but it did prove pretty challenging.

Configuring Windows/Mac/Linux for embedded development

A few days ago Scott Hanselman posted an interesting question on Bluesky.  He asked how much stuff people needed to add to Windows to make it useful for day to day work.  He also asked a similar question of Mac users.

Admittedly, my use case differs from that of most people.  I do embedded firmware development.  For me, my company Windows laptop mostly acts as a way to connect with the Linux build machines and target machines I use.  It's really little more than a glorified terminal except for running Outlook, Office, and Slack.

Windows

Having made the switch to a Mac at home 24 years ago, I only use Windows at work now.  On any new Windows machine, I first install the following software.  It's all free software as most companies I've worked for make it so difficult to justify the purchase of commercial software, that it's not worth the effort.

• Gvim - I occasionally need to do some local editing on Windows and for that a graphical version of vi is an absolute necessity for me.  I've been using some version of vi for 35+ years and while I've had occasionally dalliances with other programming editors, I've always returned to vi.
• VcXsrv - Being able to launch graphical applications remotely makes my life much easier.  That means using an X11 server.  I know there's pressure to move to Wayland but it strikes me as more effort than it's worth at this point.  It's the same feeling I have when I hear someone suggest that I try writing a device driver in Rust.  I just want to get work done, not spend time blazing a trail.
• Putty - I need to connect via SSH or serial communications to a number of Linux machines (build servers, target systems, etc) and Putty is my hands down favorite way of accomplishing this.  I make sure to enable X11 forwarding on Putty SSH sessions because this allows me to launch GUI programs and have them display on my Windows laptop.
• WinSCP - This allows me to easily copy files back and forth between Linux machines and my Windows laptop.  It also enables easy remote editing of files which reduces the pain of editing a file on a remote machine over a slow Internet link.

Mac

When I first started using a Mac at home, I loved the development environment which the combination of Mac OS X, Xcode, and the Quartz X11 server provided.  It was the best development platform I had seen since my days last using a Sun workstation in 1996.  Over time and Apple's push to combine features of iOS and Mac OS, it's become much harder for me to set up a reasonable development environment on the Intel Mac Mini which serves as my desktop machine at home these days.  Since most of my embedded development is done for work, that's not a deal breaker.

• MacVim - As mentioned above in the Gvim section, I need to edit files locally on my Mac.  MacVim gives me a version tailored for use on Macs.
• Homebrew - Unfortunately, many of the tools I've come to rely upon are only available through an alternate install path.  Homebrew gives me access to a number of development tools not available through the Mac AppStore.
• XQuartz - This X11 server used to be available in the Xcode tools but now the best version seems to require being installed via Homebrew.
• Unfortunately I have not found a free GUI SCP application for Mac I like yet so I resort to using the standard Mac Terminal app and the command line scp tool.

 Linux

I use a Raspberry Pi 5 at home since Linux is orders of magnitude better at interfacing with a variety of small embedded machines than either Windows or Mac are.  I typically use a pared down Linux distribution because I don't need the typical blend of applications like Open Office.  I've been using Debian Bookwork with the Xfce desktop environment.  

It's easy to install X11 apps, Gvim, and Putty on Linux.  The IT group at work has our Windows laptops very locked down so installing new software such as the GUI software for a USB protocol analyzer sometimes requires getting it approved which can take a few days.  Mac has gotten harder to run third party application software as well, much like the iOS app store which is very locked down.  Development goes so much faster when I can install any software I need without facing roadblocks.

Linux is also good at doing compiles for the firmware and application software I create for the newest embedded ARM device at work which is also an ARM 64-bit processor.  It has better USB support too.  Windows often requires the installation of device drivers for various USB serial devices which can be hard to do when using a laptop with limited admin rights.

Experience versus enthusiasm

We live in a somewhat rural area which means we have a well as there's no municipal water supply available.  Last week we discovered that we had no water pressure.  Since our house is 24 years old and supposedly well pumps seem to have an average lifetime about 20 years, this was an inconvenience but not a huge surprise.

What I found interesting was observing what it took to get the problem resolved.  The plumbing company we called did an excellent job.  They had a young plumber out to diagnose our problem within 4 hours of us reporting the problem.  The plumber they sent was very nice and extremely diligent.  Since he dealt mostly with houses on the eastern and more suburban portion of our county, he wasn't familiar with wells.  However he was able to get advice on how to troubleshoot the problem from more experienced plumbers at his company and after 3 hours, he determined that our well pump had finally died. 

The next day he returned early with a more experienced plumber (one closer to my age) who was familiar with wells and rural water supply equipment.  The two of them worked hard to replace our well pump in very cold temperatures (15-20°F).  During the times they came into the house, I had a few chances to chat with the more experienced plumber and found him to be not only very knowledgeable but also a really nice guy.

It struck me after they had left that the older plumber and I have found ourselves in somewhat similar situations.  I'm one of the two oldest engineers on our team at work and I'm definitely the oldest who still works full time.  I work on things that the younger engineers don't have experience with such as firmware, device drivers, and operating systems.  From time to time, the need to deal with old technology such as a serial port crops up and I'm happy to do it because it brings back memories of a simpler time.  I also seem to get all the core dumps to analyze which I find to be challenging puzzles.  Who needs brain teasers like Wordle when I can spend hours solving a crash?

I guess the lesson to be learned it that it's useful to have engineers of varying degrees of experience on a team as learning from people who have been around some type technology longer is more efficient than younger techs having to learn everything on their own.

Sometimes it pays to be skeptical

I may have been born a skeptic.  I've been questioning things I was told for as long as I can remember.  I'm sure many of my teachers were happy to see me advance out of their classroom because of that.  In many situations that doesn't make you popular, however it can serve you well in an engineering career.

On occasion I've needed to be skeptical of things colleagues tell me.  Such misinformation was most prevalent when I was a field engineer (aka FE) 40+ years ago.  If you're not familiar with that title, it's basically a mechanic for computers.  In my first job in the computer industry, I worked on mainframes and minicomputers.  For part of that time I was a specialist which meant I got called in on difficult problems after other engineers had tried and failed to fix.  I started these visits by asking questions of the FEs onsite only to sometimes have them tell me that of course they had checked the things I was asking about.  I learned which engineers I could trust to admit they hadn't checked something which seemed a logical troubleshooting step.  The challenge with engineers I didn't know well or with those I knew were too proud to admit they had missed something was to suggest that we check something together which they had assured me they had done already without embarrassing them too much.

These days my skepticism allows me to discover the discrepancies inherent in technical documentation.  I don't recall ever seeing a chip datasheet which didn't have a few errors (or instances of wishful thinking on the part of the documentation team).  Accepting the idea that the documentation can be wrong allows one to move beyond seemingly impossible situations such as a device register which occasionally isn't as persistent as the manufacturer's docs suggest.  Software documentation is frequently more error prone than hardware documentation.  I don't think I've ever seen an API document without a few mistakes.

Comments in code is another area it's dangerous to trust blindly.  Engineers will often add extensive comments in code when a function is first created.  Subsequent revisions may not see those comments updated to reflect changes in logic.

That makes the world of engineering seem somewhat bleak.  How do we combat it?  For my part, I try to report errors I discover.  That doesn't always work.  I've reported errors in compilers my company has had to pay healthy amounts of money to license only to be told that the compiler is EOF (end of life) and that no errors would be addressed.  I couldn't even convince the vendor to add my discovery to the list of known bugs.  The thing which keeps me trying is occasionally someone at a vendor will be appreciative of having a bug reported.

Long hours

For years I've been subscribing to Jack Ganssle's excellent newsletter, The Embedded Muse. It's a must read for anyone working with embedded systems. It's also very good as a general resource for engineers who work with any type of hardware or software. If you're a software or hardware engineer, I urge you to take a look at a few back issues and then sign up for the free newsletter.

 The issue which arrived today contains an interesting discussion about the number of hours engineers are asked to work. I fall heavily into the camp which questions the wisdom of working more than 40 hours. Lest you think I'm a slacker, consider that I worked for 8 straight years at startup companies, and I probably averaged 55-60 hours per week during most of that period. I can speak from personal experience that the weeks with extremely long hours were not nearly as productive as those where I worked 40-45 hours.

Of particular significance are the stories where upper management gets upset when they find no one in the lab after hours. Whenever you find upper management in the lab after hours, that's also a result of poor management. Good managers will be able to communicate with line managers without checking up on them. Good upper managers will also be sufficiently removed from day to day operations that they realize they may not have the entire picture. Employees connected to lab equipment from home are awfully hard for anyone to detect. If you think that you can only achieve results by having a negative impact on the engineer's personal life, I can safely say I don't want to work for you and I suspect most experienced engineers would say the same.

 I would suggest that any project where management asks for a long term commitment of long hours is a direct result of inexperienced or poor management. Any manager with a reasonable amount of time in the industry knows that while you may get a delivery out with a short burst of concentrated effort but if you ask for it over a period of months (or more), you're going to get a product which requires more and quicker maintenance releases thanks to the mistakes even the best engineers make when they're overtired. You're also asking for a large turnaround in your engineering team unless you're able to offer some form of extreme compensation (large bonuses and/or stock options).

For young engineers, I would advise that you consider what you're getting out of any company which is asking for extreme time commitments. If you're getting some form of financial compensation or some form of experience which will prove extremely valuable in the future, I'd say go for it although I'd get it in writing if at all possible. Otherwise, I would suggest that you update your resume and start looking around for a company whose management might not be so out of touch with reality.