Dredging the Farm

Mr White asked me, the other day, a strange question;  it had to do with a background app, running on his linux machine (a daemon, kids), that was unexpectedly terminating (winbindd, losing connection to the mother ship and freaking out), and his question was whether or not it could be set up in a loop to auto-restart.  Right now, when one of them on a problematic machine dies, we get a ticket (an hour of work) to go get details (20 min) and get onto the box (2-30 min) from somewhere, via some method, and restart the process (2 sec; one command) .  This is no good for us, since by the numbers we're managing between 2.33 and 3.33 times as many machines as we should be doing safely, even in the absence of TPS reports and bizarre pre-change oversight.

After a few minutes, I remembered.  "Is Binky2 still around," I asked. Sure, it was, yeah.  I told him to go onto the box, look at something I installed five years ago when I was in his group, and see what I did there to make another process (ntpd, freaking out on a time shift) continually restart itself as well.  What I wrote there, on a machine slated for destruction, may help him do that which he wants in this case as well.  We could save ourselves hours of work, and the company thousands of dollars they won't realize isn't gone.

A lot of our job is experience, and remembering past successes;  I'm at a bit of a disadvantage, here, but I occasionally pull a rabbit like this out of my hat.  It's very validating, and it's occasionally concerning.  And, very quickly, the effort we spend in past solutions is re-used for new problems or inspiration to shorten the time required to solve totally new problems.  If we are to harvest any of our old work, it's usually buried very deep; but the field is very, very productive when put to use.

I'm reminded of the peer at the office who, after 20 years of working in the bizarre government environment, attempted to kill himself, at work, within 5 months.  This guy had 20 years in the organization, living its funny routine and solving problems in its environment; the sweatshop almost managed to kill him within the first half-year.  What it did do is completely remove him as a resource for the rest of us, and 20 years of solutions are lost.  The most mystifying part of it all is the blasé attitude expressed by the company about this loss.