These are questions for wise men with skinny arms

Knowing something is possible does not make it easier, it just makes it more frustrating when it doesn’t work the first few times.

This even goes for things you already know how to do. Putting the pieces together is much more difficult than the design. It’s true that a good design can help the pieces fit together seamlessly, but ‘bad’ designs can have the simplest implementations. Then what makes the design ‘good’ isn’t the ease of implementation, it’s the reliability, impact, and extensibility of the completed system. It’s odd to think that knowing ‘what to do’ and ‘how to do it’ would leave such a gap for ‘getting it done’, but it definitely depends on the tools and environment.

E.g When writing a dirty hack of a connection script to keep a device on the wi-fi when it has to re-authenticate with a gateway every 24h, I knew how to emulate the authentication page request. I’ve used Python urllib before, and knew it’s stupid simple to send a FORM POST. So I wrote up the script and gave it to another engineer to test and install when he was doing on-site maintenance. It didn’t work on his machine for whatever reason, it wouldn’t even find the correct network adapter. Fine, so I change the script to monkey patch and force windows to bind the socket to the correct interface based on the expected gateway IP. Still doesn’t work because the machine has a second network adapter that provides a valid route to the authentication server IP. I don’t know why they didn’t use a valid internal IP for the auth server, but this wasn’t my network so I just had to deal. Ok, so I know disabling the other network adapter in Python isn’t straightforward, so I look to PowerShell. I decide to wrap the existing script with PowerShell so it’s easy to enable-disable adapters.

Quick note about PowerShell: While I’m no fan of it’s Perl-style syntax, it’s probably 1000x better than TakeCommand or other Windows shell environments because of it’s deep integration. I’d even argue that I like the philosophy better than Unix shells where ’everything is a file’ but you still need to know where the file it is on different systems and how to manipulate it. MS’s ’everything is a program we built for you’ isn’t as flexible but it’s much easier to use for simple things like this (Did you know you can programmatically run Windows troubleshooters with pre-set answers?!). This is probably why systemd is taking off as well.

So after version 5 of this script I go out to the site to debug it myself. It works the first time on my machine and on the desired system, What sounded simple because I knew all the pieces turned into almost 3 days of just looking things up to do it the way I expected. I knew all the pieces going into this with very few surprises, but the time it took to actually look up the parameters and write a 50 line script wasn’t reduced much by knowing all the steps in the process. About the only advantage to knowing all the steps was for time estimation, so I knew how many things I had left to implement before it worked as expected.

The lesson wasn’t in any way technical. I didn’t need to learn new tools, or new patterns, new systems, or more gotchas. I learned no amount of design knowledge would help when I just had to sit down and engineer it.

Another engineer suggested writing a macro to do the auth through the browser. I knew enough about browser automation (Thanks WebDriver!) to know that this probably wasn’t the best route to pursue. After I finished my POST version I tried a version with mechanize just to see if that would have been easier. It turns out the auth page had a self-signed expired cert so the amount of wrangling required would probably be much higher. I didn’t try any in browser scripts like GreaseMonkey, but I’d assume similar problems. So the design would have saved me time compared to doing it a much more difficult way, but I’d expect with just a bit of research that I’d find and do just what I’d planned without the design beforehand. I wonder if I was more of a front-end guy would the design have been reversed.

For the lesson of design vs implementation: I did a very similar thing when converting some external Python queries to manage database tables to internal triggers. I knew what needed to be done, and I had it all written out but it still took forever to look up the syntax to do particular operations. Luckily my knowledge of T-SQL was low enough that I learned a whole bunch of things about Views, Partitions, CTEs, and transaction semantics.

I always thought keeping good system design in mind would help even for small things, but now I’m seeing that most tasks don’t require any design skill at all to be effective. The breadth of design knowledge that I’ve been chasing isn’t an implementation skill, it’s a planning and management tool! The design is important for the long run, but day to day I find it much more difficult to apply effectively.