misterfoo.com

Tag: software

  • 90% done, 90% to go

    Building a software product is hard. It always feels like it takes longer than it should, or at least longer than you expected. Why is that? And will AI deliver on the industry’s promises to revolutionize software development? Or is it all hype and the developers will be left holding the bag?

    First, some thoughts on why it seems to take so long.

    A new software product

    Once upon a time, you decided to build a new software product. You knew it would be hard but worth doing. You interviewed a bunch of users, or maybe you put together some ideas you’ve had for years, and you got started. You built a beautiful and responsive user interface, wrote the backend services in C# (so much better than Java), stored the data in Cosmos (no SQL for you!), and tested it all. It was a lot of work. 

    There were times when you weren’t sure it would come together, but it did. It looked great! It went into private preview with a few friendly customers and they loved it. I mean, let’s be real: they wanted more features, because of course they did. But it covered most of what they needed.

    You sat back and reviewed it all, and you were clearly 90 percent done. And it felt good.

    Excitement meets reality

    The preview broadened and things got harder. The bigger customers needed better security features. The smaller customers wanted a “lite” version. The reporting functionality you were sure was overkill was totally required. And then it went GA – hurray! Turns out your choice of C# was solid, but a third of your potential customers had old Java plugins that they just could not abandon, so you had to build a C#-to-Java bridge.

    Cloud-hosted Cosmos also didn’t meet your enterprise customers’ cybersecurity requirements, so you had to learn how to deploy and operate it on-prem too. And those same customers ran their antivirus software at “maximum paranoia”, which broke the system just often enough to make people really mad, but not enough to reliably reproduce when you tried to fix it. You eventually convinced the customer of the right settings and they stopped yelling at you.

    Looking back on the first year of production use, you spent as much engineering effort fixing all these issues as you did building the initial product. Frustrating but hardly unusual. And surely now you were 90 percent done.

    The lifetime of a product

    Since that first year, you expected to spend most of your time building the next big thing, but the product has never stopped needing attention. Customers want new features, bugs crawl out and have to be squashed, and support tickets need answering. And don’t forget all the libraries that need to be patched and updated (and patched, and updated). So many libraries.

    If you’re lucky and the product lasts long enough, you will have to replatform it, AI-enable it, replatform it again, and convert it to Rust when C# gets retired. As long as it has customers you will have to support it, if you want to keep getting paid.

    By the time the product finally goes end-of-life, you will do at least as much work supporting it as you did in the first two stages, and maybe more. Now you can put your feet up and relax.

    The three 90 percents 

    These are the three 90 percents: building what you think you need, turning it into what you really need, and then keeping that thing going. The pattern shows up time after time, across scales. From huge projects to little helper scripts, when you think you have done 90 percent of the work, you are probably still closer to the start than to the finish.

    Back to the topic of expectations and software seeming to take too long. Everyone expects the first part to take a while; it takes time to build something! But then the second part surprises you, because when you started you didn’t know what you didn’t know. You had a plan and you followed it, but your plan had no hope of being complete because you were building something new. The best you can do is to plan that there will be a lot more work to do after you go live. And the third part shouldn’t come as a surprise to anyone, but somehow it gets overlooked in the planning as we rush on to the next thing. Even after things stabilize in production, you still have a lot more to do, unless you abandon the thing entirely.

    Side note: if you are part of a startup or otherwise doing something totally new, your second “90 percent” may be much bigger than the first, if your original idea doesn’t pan out and you have to iterate. The point here is that you will have to do at least another 90 percent after that first stage, and maybe more, and then you’re on to support and maintenance.

    How this changes with AI (in 2026)

    AI in 2026 is very good at coding, and the first 90 percent is substantially about coding, so AI can help a lot here. You can even vibe-code your way through this part without involving a developer, but if you’re building a product it’s not a good idea – more on this below. Much of the hype around AI as a software industry disruptor (“I built a whole product with a team of one!”) is specifically about the first 90 percent, whether the authors realize it or not.

    The second 90 percent is much less about coding, and much more about revision, judgment, experience, and troubleshooting. It’s about fixing all the ways your original design didn’t work out. AI in 2026 can help with this, but it can’t lead the effort. It can give you ideas to help troubleshoot that thorny performance issue, but it can’t do the work because that requires making experimental changes to production systems, and you didn’t give AI the keys to your production systems, right? It can build some of a C#-to-Java bridge for you, but it can’t tell you if that’s a good idea or something you will regret forever. You need experienced developers to lead the second 90 percent, drive the design, and do a lot of the work. But using AI here will help you go faster..

    The third 90 percent is a similar story. AI can help you answer support tickets, but it won’t get on the phone with an angry customer and talk through the technical issues until a solution presents itself. It can fix a bug, but for now at least, it will work with a narrow focus and miss the three related fixes that should have been made at the same time. It can help you build new features, but it won’t drive the refactoring that keeps those features working as a coherent whole. You need developers and architects for the third 90 percent too.

    Where AI poses a risk

    Unfortunately AI doesn’t just help software products, it also brings some serious risks. With respect to the three 90 percents, two of the biggest risks are in the connections between the three steps.

    Leaning too heavily on AI in the first 90 percent leaves you unprepared to lead the second and third 90 percent. If your initial release is mostly or fully vibe-coded, you won’t understand the code and will struggle to find and fix the problems that come up. You will have to rewrite whole sections of the code that worked fine in private preview but melted under load. Your customers will lose confidence in your product when you are unable to explain why the system failed. You will eventually figure things out, but your time savings from the first 90 percent will be hugely diminished, along with your confidence and reputation. Or you will limp your way through the second and third phases fully dependent on AI, until the wheels come off and it crashes.

    Over-reliance on AI in the first 90 percent makes the second and third parts harder in another way as well: lower quality in the first iteration. Your senior developers have lived the second and third 90 percent for a dozen products before this, and from that they have learned how to do a better job up-front. They have gained experience and developed judgment and taste. That experience doesn’t let you skip the second or third parts, but without applying it in the first 90 percent, the later parts will be much harder. AI coding models have some of this knowledge, but far less than your human team does. Some of it is specific to your environment and the model could never know it; much of it just isn’t covered in the training data and isn’t easily guessed from first principles. Either way, if you disregard this experience now you will have a lot more work to do later. So you want human developers closely involved in the first 90 percent, even if they are substantially augmented by AI assistants.

    Benefitting from AI and offsetting the risk

    From the perspective of the three 90 percents, AI offers big upsides and corresponding risks. One recommendation that seems clear from all this is to keep human developers closely involved in the first 90 percent. That includes creating the initial design, helping to build the code, evaluating the quality of AI-generated tests, and ensuring that the code is readable.

    One way this could look:

    1. Humans create the initial design, consulting with AI on specifics as needed.
    2. AI writes the first draft, including tests.
    3. Humans read and improve the code, with optional support from AI, but with real hands-on-keyboard work as well.
    4. Different humans and AI review the later drafts.
    5. Move to user testing, preview releases, and then to the second 90 percent.

    Step 1 ensures that the judgment and taste of experienced developers is driving the design. Steps 3 and 4 ensure that human developers really understand the code from step 2 and are prepared to lead the work of the second and third 90 percents.

    However you do it, find a way to keep your developers closely involved in the first 90 percent, if you intend to keep (and maybe sell) the thing you’re building. As noted above, they will be key to the second and third 90 percent as well, as before.

    Summary

    Wherever you are as a developer, team, or company, the three 90 percents almost certainly apply to some degree. Thinking about them explicitly helps you understand why work progresses the way it does, and why things take longer than you expected. AI is changing the software world at an incredible pace, but it doesn’t let you skip any of the three steps, and you over-rely on it at your own peril. If you do nothing else, be intentional about your use, and think about how it will affect your work next month and next year. And then use it thoughtfully and according to its strengths.

    Acknowledgments

    The concept of the three 90 percents originated with Thom Mayer and the Austin Digital business.

  • if software development was like bridge-building

    I was thinking recently about metaphors for software development. There are lots of metaphors, my favorite being the garden metaphor, but I’ve also heard software development compared to bridge-building. To me the comparison is wildly inaccurate, and for fun I decided to develop the analogy farther.

    Building a typical enterprise software tool as a replacement for the Golden Gate Bridge:

    • Before building starts, the designers debate for weeks over whether the bridge should be made of steel, plastic, or air gel. The virtues of air gel are touted with great smugness and the steel advocates are shown to be woefully behind the times. Nobody takes the plastic advocates seriously.
    • A month into construction, the width of the Golden Gate channel doubles, and the rock underneath changes from granite to crushed gravel. Construction is delayed for a month while the bridge is redesigned. Air gel is found to be inadequate and the builders switch back to steel.
    • Two months into construction, a freak electrical storm causes the bridge to entirely disappear. Construction resumes again after much wailing, finger-pointing, and an off-site backup bridge is created in Oakland.
    • A week later another freak electrical storm causes the bridge to disappear again, but it’s restored from the Oakland backup without mishap.
    • At 75% completion it’s discovered that the road bed will collapse whenever a Toyota drives on the bridge. The builders explain that Toyotas were never meant to cross bridges in the first place. Toyota drivers are directed to the Sausalito ferry.
    • On the first day of active service, a last-minute Beatles reunion concert is scheduled in Napa Valley, causing the entire population of San Francisco to mob the bridge all at once. The bridge disappears and reappears four hours later. Everyone misses the concert.
    • A year after the bridge is completed, the rock under the channel is upgraded from crushed gravel to decomposed granite. The bridge sits at a funny angle for six months until the foundations are rebuilt.

    See what I mean?

  • Coders at Work

    I just finished reading Coders at Work, by Peter Seibel, and it was awesome. The book is a collection of interviews with some of the most respected and well-known programmers of our time. These are Turing Award winners, inventors of languages and operating systems that have touched millions of people, and authors of the most widely known and respected computer science literature you can find.

    I kept a list of notes as I read through the book, things that resonated with my own years of experience in the field. There isn’t any particular theme here, but I wanted to capture these ideas before they got away from me.

    Working programs are a given

    Bernie Cosell, who helped develop Arpanet (predecessor of the Internet), made a great point about the responsibilities of a professional programmer:

    You don’t get credit because the program works. We’re going to the next level. Working programs are a given.

    What he’s saying is that when you work as a programmer, you job isn’t done just because the program works. You need to generate something of high quality that you and your coworkers will understand six months from now, and which can evolve as the customer’s requirements change. Professional musicians don’t just play the notes, race car drivers don’t just drive around the course, and serious programmers shouldn’t be content with an ugly program just because it works.

    Magic is dangerous

    Guy Steele had a great way to sum up why programming is hard:

    [Being] able to get a machine to do what you want is the closest thing we’ve got in technology to adolescent wish-fulfillment. And if you look at the fairy tales, people want to be able to just think in their minds about what they want, wave their hands, and it happens. And of course the fairy tales are full of cautionary tales where you forgot to cover the edge case and then something bad happens.

    For example, take the story of King Midas, who wished that everything he touched would turn to gold; when granted in its most literal sense, this wish became a terrible curse.

    Unfortunately for programmers, computers are the epitome of the fickle genie, interpreting us at our most literal and wreaking havoc when we don’t clearly specify what we want. Therefore, as programmers our job is to provide such excruciatingly detailed instructions that the computer can’t possibly misinterpret us and do the wrong thing. This can be one of the most frustrating things to communicate to non-programmers; it’s not magic, and translating a simple idea into code is far from mechanical.

    Microsoft’s tools are awesome

    Say what you want about Microsoft, but their developer tools are world-class. Most of the interviewees in this book come from the non-Microsoft parts of the programming world, and often bemoaned the lack of good quality debugging tools. In fact, it sounds like many of them do their debugging with little more than print statements. This is one of the oldest and most basic forms of debugging, and it has its place, but to paraphrase one of my favorite Despair posters, just because you’ve always done it that way doesn’t mean there isn’t a better way. A good interactive debugger is worth its weight in gold for a lot of problems, and I’m glad to be working with one of the best there is: Microsoft’s Visual Studio.

    Names matter. A lot.

    Programs are nominally a set of instructions for the computer to follow, but they also have to be understood by humans who come along later and want to understand or change things. A program whose code can’t be understood is in fact nearly useless, like a book written in a forgotten language.

    One of the most crucial elements of legibility is using good names for things. If you write a procedure that flushes program data to the hard drive, and you call it FlushDataToDisk, a reader can immediately understand the basic purpose of the function, and might be able to skip reading all of the gory details of how it works. On the other hand, if you call it WriteStuff, the name tells the reader nearly nothing, and forces her to read the whole thing to discern its purpose. The problem is that when you’re writing the code it’s easy to just use the first name that comes to mind, without considering the impact that choice may have a few months or years later.

    I’ve only recently started to take this issue so seriously, but given how many programmers in this book seem to feel the same way, I think I’m on the right track. I now regularly consult a dictionary and thesaurus as I write, and you should too.

  • a hofstadter moment

    (or, why platypuses don’t make good biologists)

    Douglas Hofstadter loves the idea of self-reference. As a cognitive scientist, author, composer, and teacher, his works commonly refer back to themselves or fold in on themselves in strange and delightful ways. His recent book I Am a Strange Loop is all about self-reference, and I can’t recommend it highly enough. A few years ago I had what I would call a Hofstadter Moment, a surprising and unexpected case of self-reference. It came as I was developing Foogle, a tool to index and search computer code.

    As a software developer, I work every day with a code base made up of many thousands of files, containing among them millions of lines of code. I often need to search this code, to find where and how different parts of the system are used. Having grown accustomed to Google and its ability to instantaneously search billions of Web pages, waiting five or ten minutes for Windows Search to find what I was looking for just didn’t cut it. So I decided to write a tool that could index our code every night, and let me perform fast and accurate searches of that index any time I needed. In homage to Google (my favorite search engine) and “foo” (the official nonsense-word of programmers everywhere), I decided to call the tool Foogle.

    Foogle’s job would be to scan through each code file, break it up into individual parts (‘tokens’, in programmer-speak), and add an entry for each part to the index, which could later be searched. This would be very much like indexing a book, where you would separate the text on each page into individual words, and then add the words of interest to the index. The big difference is that computer code isn’t nearly as easy to decipher as the written word – it tends to look like an explosion of letters and punctuation. This meant that the code for tokenizing a file was relatively tricky to write.

    Once I had a working prototype, I turned it loose to start indexing code. First I tried it on some simple, made-up code files, and things worked pretty well. Then I tried letting it index our entire “utilities” folder, which happened to contain the code for Foogle itself. Much to my surprise, when it hit the Foogle code, it broke – it couldn’t index itself. Not only that, but as I investigated further, I was amazed to find that the line of code that couldn’t be indexed was the very line of code that had the bug! 

    This line of code, when running as part of a program, was unable to cope with its own written representation. It’s a bit like the double-take you might experience when reading the following: “I like the color red” (psychologists call this the Stroop effect).

    As it turned out, fixing the problem was relatively easy, and before long I had a working program. Still, this was one of the most interesting bugs I’ve ever found, purely because of its strange, self-referential nature. It seemed rather like a platypus trying to be a biologist. With their furry bodies and duck-like bills, their laying of eggs and nursing of young, platypuses1 just don’t properly conform to any of the standard animal categories. Imagine a platypus as a biologist, on the day when it walks past a mirror and realizes that it can’t even categorize itself!

    1. You might say I’m wrong and that ‘platypi’ is the correct form. However, I’ve consulted Wikipedia, and I stand by my ‘platypuses’ (and my platypuses).

  • 17,000 heads are better than one

    StackOverflow.com opened for business about a month ago. It’s dedicated to asking and answering concrete questions about programming. It already has more than 17,000 registered users, who have asked more than 31,000 questions. More than 90% of these questions have at least one useful response.

    This is great news for programmers.

    You probably know that computers are stupid, as evidenced by the incredible frustration they often cause. Programmers are tasked with training these blazingly fast mechanical morons to do useful tasks, and even the buggiest program probably represents hundreds or thousands of hours of someone’s time. Everything your word processor or email program does, every button and graphic, somebody made that happen. And it’s not easy.

    I recently had to implement printing in a basic graphing application. Going into it, I told several people that I figured it should be pretty straightforward, maybe a day or two of work. I’ve been doing this for almost ten years, so I figured I ought to be able to sort it out. But unless you’re working on a system that’s designed around paper (e.g. Microsoft Word), printing is a pain. You have to completely reconsider how to display the data so that it works on the printed page. In the end it took at least a week of work to get a solid implementation. I probably only wrote a few hundred lines of code, which isn’t a lot. The problem was that I’ve never done it before, and neither had any of my coworkers.

    One of the most important tools of in any programmer’s toolbox is other programmers. What might take hours or days to figure out on your own can often be solved in minutes by someone who has done it before. Certainly it’s also satisfying to figure something out yourself, but I’m not always in the mood. Sometimes I just want an answer so that I can go home for the day. Often coworkers fill this role, since they’re just an office away. But what if nobody at the company has ever done printing before? Or what if you’re working solo?

    That’s where StackOverflow comes in. With thousands of programmers on call, even the most esoteric question can find an answer in short order. It’s often only a matter of minutes before a question gets its first answer. And if the first answer doesn’t work, the second one or the third one might. You might even find out that a bunch of other people have already tried what you’re doing and they all figured out that it wasn’t feasible. It’s sure nice to know that before

     you spend hours and hours trying to do it.

    This is great news for all of us, because it means less time banging our heads on the table and more time making software. So far I haven’t needed it for anything terribly important, but I sleep better knowing that it’s out there.