Tuesday, April 1, 2014

Worst practices are viral for the wrong reasons

This short post describes my hypothesis for why poor software engineering practices are more viral. Simply stated:

Worst practices spread more quickly because they require more attention

The following two sections outline specific manifestations of this problem and how they promote the dissemination of worst practices.

Management

Corollary 1: Teams with the greatest technical debt mentor the most employees.

Scenario 1: You manage a team that ships an important feature but with a poor execution. You can make a compelling case to management that your team needs more head count to continue supporting this feature. More employees reporting to you increases your chances of promotion and you enjoy rewarding opportunities to mentor new software engineers in your specific brand of programming.

Scenario 2: You manage a team that ships an important feature in the exact same amount of time but with an amazing execution. Your code requires little maintenance, so little in fact that your team is now obsolete. If you're lucky you are retasked to work on a new project; if you're unlucky you are fired because your skills are no longer relevant. Training others how to write software of excellent quality is out of the question either way.

In other words, software engineers that produce excellent code cannot easily pass on their wisdom to the next generation of programmers unless they go into academia.

Open source projects

Corollary 2: Poorly implemented libraries or programming languages generate more buzz.

Scenario 1: You write a useful library or programming language, but it is incredibly buggy and problematic. This generates several scathing blog posts criticizing your work, prompting others to clarify in response how to work around those issues. Don't forget to check Stack Overflow, which is full of people asking for help regarding how to use your project. Did you know that people measure popularity in terms of Stack Overflow questions?

Scenario 2: You release a useful library or programming language at the same time, except that through some miracle you get it correct on the first try. Version 1.0 is your first and last release. That means you can't advertise your library through announcement posts for subsequent releases and there are only so many ways that you can say "it works" before people accuse you of shameless advertising.

In other words, well-written open source projects can become victims of their own success, with fewer opportunities to market themselves.

Conclusion

Obviously the above examples are exaggerated hyperboles and reality is somewhere in the middle. There is no perfect software project or team and there is a limit to how much crap a company or user will tolerate.

However, I write this because I value simple and understandable software and I want to start a discussion on how software engineering can culturally reward correctness and reliability. We need to think of ways to encourage programming excellence to spread.

17 comments:

  1. How about teams with mixed background? A teammate writes error-prone code, and inevitably, the code has a bug. I take the bug, track the origin of the bug to the error-prone practice, rewrite it in a simpler way, and show my teammate how my approach prevents this entire category of bugs. True story. Now when that teammate encounters this category of bugs in another project, they spread the gospel! That part of the story hasn't came true yet.

    ReplyDelete
    Replies
    1. I think your last sentence answers your first sentence. :)

      More seriously, I agree with you. I just think people who advocate correctness just need to be a little more proactive.

      Delete
  2. Aren't you a bit young to that jaded! ;-) (and most likely accurate!)

    ReplyDelete
    Replies
    1. Ha! I still enjoy programming, don't worry. :)

      Delete
  3. 1 - I have seen this myself in over 37 years of programming.
    2 - Isn't this similar to the position faced by medical doctors and probably some other professions - i.e. a good doctor gets you well, then you don't need him any more. A bad doctor keeps you sick, and so he is needed more. Perhaps it is just a professional hazard?

    ReplyDelete
    Replies
    1. Yeah, I believe good programmers can and should make themselves obsolete. A friend of mine warned me away from going into consulting exactly for that reason: consultants have an even larger incentive to delay solving the problem.

      Delete
  4. In my experience, the software teams that work best are the ones with self-driven individuals who share the common goal of releasing high quality code. In this cases management is almost unnecessary, they can help with communication and shielding from bureaucracy, customers, etc.

    The problem in many companies being a manager is seen as the equivalent to being the "boss" and programmers are seen as disposable pawns that just sit there toying around with nerdy machines. These managers have to be proving continuously, often clumsily, who is in charge. I guess that's the basis for Dilbert's cartoons.

    Getting rid of these managers, is out of question most of the times, as they are frequently, somehow, involved in bringing the money for the project.

    No wonder why the most successful software companies are founded by engineers.

    ReplyDelete
  5. And of course there is that "let's pull it together quickly, and fix the sloppy code later" attitude. Though it seems to be a universal human trait, actually, not restricted to programming.

    A year ago I watched several lets-plays of Far Cry 3 (about 6 or 7): the game takes place on an island with many mountains and hills, so there are plenty of slopes you can't (but want to) climb. Or you can, but after jumping just in the right direction, from the right place. That's the first premise, the second premise is that there are many, many roads and footpaths, and they lead to literally any place of interest.

    And that leads to a somewhat hilarious observation: all seven lets-players CONSTANTLY found themselves trying to climb a particularly uncooperative elevation, and after a minute or so they succeed—only to find that there is a road up there that is coming from exactly the place they were trying to climb, so it would be quicker to just walk around by that road.

    And none of them seem to even notice that, mind you! I personally tried to play the game trying to stick to the roads as much as possible—and it worked ridiculously well, never had to leave them for anything including escaping from the chase. But that required a certain sort of self-discipline, to suppress "oh, I could shorten the path if I would climb that" urges.

    Given the game itself defines insanity as "repeating the same thing over and over and over and over, hoping that *this* time it will be different", I conclude that we're all mad. Or we wouldn't have come here, right?

    ReplyDelete
    Replies
    1. At the same time, that's a nifty feature of humanity sometimes: finding the clever "hack" that confers a huge advantage. In some ways, humans are a reflection of their environment: inelegant and circumstantial on small scales yet modular at larger scales.

      Delete
  6. Had similar thoughts for a while. You've summed it up pretty well.

    ReplyDelete
  7. I'm not sold on this idea. I think it's more because best practice are fundamentally non intuitive. The typical example is: 'adding more people to a late project will make it ship even later', or the whole testing thing (this takes time and effort to design with testing in mind and testing is time consuming yada yada).

    What you're stating is the corollary of this: worst practice are just 'what everyone does naturally'

    ReplyDelete
  8. “More employees reporting to you increases your chances of promotion”

    That is complete nonsense.

    “You can make a compelling case to management that your team needs more head count to continue supporting this feature.”

    Even that is borderline ridiculous.

    “Your code requires little maintenance, so little in fact that your team is now obsolete.”

    Wat? Have you ever actually been in such a team? (No you haven’t.)
    You completely and deliberately left out that they will have much more money as a result of the success and as a result of… let’s say it that way: A boss wants to keep his job too, so he finds a new project for them to make even more money and brag about it and hire /even more/ people!

    Also: As if anyone would deliberately let his boss know that he has too little to do… lol. If anything, given that they are skilled, they’d probably present some cool project of their own to their boss who would then use that to make more money and brag and get more employees.

    Google is the best example: Barely anyone there is really essential. By your sorry attempt at logic they’d all be fired.
    Except in actual reality that’s not what happens, *now is it*??

    Sorry, a couple of hand-crafted made-up scenarios are not a valid basis for an argument.

    "… unless they go into academia."

    Aaah, now I get where all this is coming from. It’s an inferiority complex and you need something to make you feel superior. So you argue about something you have no competence in, to… conveniently… conclude that your is better.

    Silly. There was nothing wrong with academia in the first place. There’s probably nothing wrong with you either. You just have the stigma of intelligent people, that we are so good at seeing all the ways we could be wrong, that mechanisms emerge to blind oneself for the sake of one’s own sanity.

    Everything is OK with you. You don’t need to put this distortion filter between you and reality. Reality is OK.

    ReplyDelete
  9. Ohloh for example illustrates the problem you're describing quite well on the open source front. I, for example, wrote a bunch of very basic libraries which provide their functions well and are tested. By design, they shouldn't take up a lot of memory, so they shouldn't be extended inside the same source base. So essentially, the code is stable and will not be changed unless I find a bug (which I haven't so far).

    On Ohloh, the libraries have a big warning sign in front of their names saying «This project appears to be abandoned, there were no commits in the last 6 years». Sure, because why would there be? The code does exactly what it should, and not more and not less. I use the code every day.

    Now that's just Ohloh, but many developers work that way as well. They look at the source tree and see «Eww, the last commit is from 2008! The last release tarball as well! Clearly, this project is dead.» It's not, it's just working.

    That's why there's a tendency for Open Source developers to reinvent the wheel over and over again, unless the project has feature creep, which makes it relevant forever by those metrics.

    ReplyDelete
    Replies
    1. There is an easy fix: deploy a file: days-without-bugs-reported.txt and have the counter inside increment by some autoupdater each day. This makes you have daily commits without actual work involved. Now you only need to reset the counter once a bug was found ;)

      Delete
    2. I never get the complaint about "this software is not maintained." So? Does it have any bugs? cat(1) hasn't changed much in 20 years. Should I suddenly stop using it?

      Knuth did this right with TeX. Stabilize the core and leave it alone.

      Delete
  10. * The software community has a great environment of knowledge sharing through user groups, conferences and such. Instead of going into academia, you could use this, give talks about doing things the right way. This should give you some regards, i.e. leverage that can be used to argue against bad practices. Would at least help with getting a new task after making oneself obsolete at the current one.

    * successful projects can on occasion be detected by the management. If the whole team then states that they 'well, just did the right thing', people might take note. Could be a tad hopeful, this, but there are companies around where this actually works. Probably depends on company size.

    * there'll always be a desire for more features. If you did things right (low coupling, open architecture etc), implementing these probably isn't hard, even if they make things stand on their head. This should make people want to have you around for the next feature, even if you're kind of obsolete in between.

    Just some random thoughts to try a more positive angle, though I share your opinion at least halfway through ;-) Would like to see a follow-up that puts framework hype into that picture, too...

    ReplyDelete
    Replies
    1. Yeah, I exaggerated just to make a point. My outlook is not that pessimistic. :)

      Delete