Memories of my days as a Field Engineer

I recently read the book Talking About Machines by Julian E. Orr.  I enjoyed the experience as it brought back memories of my first and second jobs in the computer industry which started nearly 50 years ago.  Both those jobs involved computer field engineering (first mainframe and later minicomputers).  Many of the book's reviews seem to focus on ideas in the book which I don't find particularly interesting.  I believe this is largely due to the fact that the author's primary focus is on the sociological aspects of the job such as the ad-hoc methods of skill building required to augment company training. The downside of this perspective is that it overlooks some of the technical reasons for the job being quite difficult such as insufficient training, poor documentation, and inadequate support avenues.  Many of the reviewers seem to lack the technical background required to fully appreciate just how challenging work as a field engineer can be.

The book showed me that Xerox suffered from the same problems which both of my first two employers had.  The focus of all three companies seemed to be to reduce support costs by any means possible.    The book devotes much attention to the means by which the field engineers become competent and manage to maintain their productivity.  It attributes this largely to impromptu communications during meals and other social gatherings.  It glosses over the fact that companies scrimp horribly on training and documentation.  They rely on personal dedication from relatively low paid engineers devoting nontrivial amounts of their personal time to improving and maintaining their skills.  Companies seem to expect these levels of self training despite offering poor compensation and inadequate opportunities for advancement.

My first job was as a Field Engineer for Honeywell starting in 1977.  To qualify for that position, I had completed a 6 month full time training program in hardware maintenance at the Control Data Institute facility in Arlington, VA.  My class which was completed in early 1977 was the last at that location to enjoy completely instructor led training.  Subsequent classes used Control Data's PLATO system.  I cannot imagine being trained to diagnose and repair electronic equipment via software alone since it's so different from actual computer diagnosis and repait.  To train us in hardware diagnosis and debugging, we were required to construct computer circuits such as adders by hand wiring small circuit boards containing discrete electronics.  Once we demonstrated the correct operation of the circuit we had constructed to the instructor, we had to leave the room while the instructor inserted a bug of some sort.  This exercise helped teach us how to debug logic problems.

That training proved to be excellent preparation for maintaining computer equipment.  What I hadn't anticipated was the wide variety of equipment I would be expected to repair with no training at my first job.  In my first few months, I learned to rely on computer operators leading me to the problem equipment and either pointing out or reproducing the behavior they thought to be incorrect because I sometimes couldn't identify the equipment on my own having never encountered it before.  For my first year or so, after being shown the equipment, I frequently had to retire to the field engineer's office to peruse the documentation to figure out how to proceed.  Fortunately, Honeywell had excellent documentation with most equipment manuals containing section which offered a detailed theory of operations.  Reading how the designers intended for equipment to work which was key in being able to effectively diagnose problems. Once I eventually transitioned to software engineering, I've often been forced to build mental models of hardware and software systems because detailed documentation seems to be largely a thing of the past.

The field engineering support contracts at my first job specified that customers offer a field engineering office where the documentation, parts, tools, and test equipment required to service computers and peripheral equipment could be stored.  In that respect, I had it easier than Xerox field engineers did as they were required to carry all of the documentation, tools, and an inventory of spare parts with them to be able service trouble calls on whichever type of copier they maintained.  I feel fortunate since with such a wide variety of equipment supported at my first job, the manuals alone would have taken up a sizeable portion of a van and adding spare parts and test equipment necessary to do the job would have made carrying everything with me as I visited customer sites untenable.  I'm a bit jealous that the Xerox engineers could focus on one or two models rather than the 5 different models of mainframe computers and 6 or 7 types of each of the following - magnetic disk drives, magnetic tape drives, line printers, punched card readers and punches, computer terminals (both hard copy and CRT varieties).  I'll leave out the paper tape readers/punches, magnetic drum storage, and core memory units to avoid dating myself.  My second job involved maintaining minicomputers which required far fewer parts and less documentation but which also made it so much less challenging as to be boring.  Often there was only a cabinet in some obscure corner of the computer room devoted to on-site documentation and parts storage. 

I spent 5 years at Honeywell and that experience provide me with debugging skills that I've relied upon during my lengthy career.  As mentioned in the book, there were few opportunities for advancement for field engineers.  On the technical side, one could aspire to be promoted to be a specialist which is essentially a more skilled technician often only called for the most difficult problems.  The other path was management which shared few skills with the technical path.  There were too few positions in both paths to reward anything more than a few lucky engineers.  That was one reason why I elected to make the jump to software engineer after 5.5 years as a field engineer although I did spend a year as a specialist after managing to qualify for a specialist training class in Honeywell's training facility in Phoenix, AZ by working very hard for 3 years.

My second job involved maintaining minicomputers which were so much simpler than the mainframe computers I had spent 5 years maintaining that I found myself bored while easily being the best debugger in the district.  I say this not to boast but because during the 7 weeks of training I was sent to in Boston, every engineer in my district had tried to fix a customer's tape drive which had initially had a simple data read problem.  By the time I got back for my mid-training break, I had to first remove the 8 or 9 problems my fellow engineers had installed in the tape drive trying unsuccessfully to fix it.  When I got to it, the fusible diodes in the power supply would melt their solder and drop to the bottom of the chassis each time it was powered on.  It was not terribly difficult being the best engineer in a district filled with such poor engineers.  It was no wonder that I was offered a job in training while in Boston because I wrote a machine language program that none of my instructors could figure out.  I didn't accept because the training instructors were not terribly competent and I didn't want to move to Boston.

Instead of a CPU made of 80 wire wrap boards, the DEC PDP-11 managed to fit the entire CPU on just a board or two.  That reduces the task of debugging a system problem from putting boards on board extenders and using an oscilloscope to trace signals from board to board while running some software keyed in from the maintenance panel switches, to being able to swap the entire CPU in a matter of minutes which field engineers at the time derisively referred to as shotgun debugging.  I got so bored that I took the job of installing and maintaining the 12 DEC PDP-11 minicomputers at USA Today when they launched in 1982.  Taking that position freed me from the need to travel to a variety of computer sites located within a 120 mile radius of my office in Rosslyn, VA but left me working an unpleasant 8 pm - 4 am shift at USA Today's headquarters which was also in Rosslyn.  Walking the two blocks back to the parking garage at my company's office each morning at 4 am was definitely a weird experience.  The only entertainment I had was seeing how vandals had rearranged the letters on the Chinese Cinema, which depending on the movie being shown sometimes read Chinese Enema.  The 4 or 5 months I spent on that weird shift made me feel isolated since I was heading to work about the time that most normal people were getting home from work.

Since the DEC PDP-11 minicomputers at USA Today were all new, most of the problems I dealt with each night were with the computer terminals used by the newspaper staff.  My company had determined that the most common FRU (field replaceable unit) for the CRT terminal devices consisted of a board which took me about 20 minutes to replace thanks to all the screws and cables which needed to be removed.  After my first couple weeks of working on terminals, I learned to associate symptoms with one of four chips which were the most common failures.  An example was the vertical deflection chip which gave the symptom of a single horizontal line across the CRT when it failed.  I could replace the chip in less than 5 minutes.  So I removed the chips from all of the terminal boards which I sent back for repair to build myself a stock of the chips which my company would not allow me to order as individual chips.  And I started a letter writing campaign to the design group trying to make the case for stocking the chips.  I never heard back from them before I left a few months later.

I occasionally get a bit nostalgic for my first field engineering job.  I don't miss working on peripherals such as line printers (which are incredibly messy) and card punches (which are mechanical monstrosities) at all.  On the other hand, a huge computer system (which would easily consume all floor space of most convenience stores) which fails to boot or which misbehaves when a particular piece of software is run is like an incredibly challenging puzzle.  Being paid to solve puzzles made my first job feel like play at times.  The beauty of having made the move to software is I get to debug problems every bit as challenging but I don't have to worry about ruining clothes because I got stuck working on a line printer.