I learned to code about the time I learned to read (fairly late), so in the modern world I am not necessarily as good a coder as people who learned in high school or college–the world of 0.995 MHz 6502 ASM is not the world of 2GHz Dual Core X86 Ruby. The practices appropriate to coding on tiny, hardware-level machines don’t really jive in a world with operating systems and enough memory.
It comes down to trust, and if we may paraphrase one Francis Fukuyama out of context and into our context of databases and high-level languages: Trust reduces transaction costs. One must trust the database to be efficient with its queries, and not simply grab whole tables and perform one’s own queries. One must trust hash functions to be efficient and not code them up as needed. One must trust libraries, sorting algorithms, internal data structures, drivers, and especially parsers. One must also remember overheard, and not make database queries on inner loops.
There is more to optimize for than execution speed and low-memory consumption (or rather, just speed given that all memory will inevitably be used). The 4K demo contests preserve and transmit the old ways, and these are a valuable and venerable tradition. But execution speed is not the only criterion for which one optimizes. Perhaps this is why modern programming is sometimes called software engineering, as if one were engineering solutions to balance a number of criteria. Despite the number of promising open problems (like the complexity of computing Nash equilibria for various classes of games), day-to-day optimization tasks tend to balance a number of concerns, and when obvious ‘optimize for speed’ cases come up, often the best solution is to trust the code base. And by best we might mean readable, quickly coded, stable, reliable, easiest to maintain, and not just fastest during execution. I have even heard tell that premature optimization is known to be something one wishes to avoid.
One of the grand, not-quite-over-looked but definitely under-appreciated prospects for accelerating the growth of growth (second derivative, if you will) is the phenomenon of re-usable code through libraries. The modern world suffers from superstar effects and network externalities–you only need a few really good writers and everyone reads them, if an operating system is really bad everyone still has to use it because everyone else still uses it–but these dynamics of path dependence are not totally on the side of entrenchment and against progress (as trendy as it is to belittle those who believe in ‘progress’ rather than simply ‘adaptation’ I should say that having lived a little while I find the empirical evidence strongly on the side of progress, perhaps due to adaptive arms races but too evident nonetheless).
I suppose that for a modern developer this is patently obvious, but it may have been less so back in the day. On an old computer, you really couldn’t just write functions in LISP without thinking about the internal representation and the costs of certain operations. Alan Perlis was correct when he observed, “Lisp programmers know the value of everything and the cost of nothing.” The STL in C++ back in the day too was a little iffy, and things would break for reasons obscure to the non-expert. Hardware libraries for graphics required a lot of empirical testing to infer which functions were faster than others in what circumstances. Risk aversion was indicated, as the cost of writing things from scratch (part of the fun really) might be less than the cost of debugging something that doesn’t quite work as we might expect. What has changed? The Internet.
If we use a library, and something does not go quite as expected, we can plug the error message directly into Google and almost always someone else has had the same problem. Typically there is a solution or workaround, and the next version of the library will be fixed. Because technology changes so quickly, rather than mastering a field and perfecting a technique, we end up becoming adept at figuring out how to do what we don’t already know how to do because we’ve never done it. Well, if you like learning new things, this makes development a good day every day.
One does acquire a familiarity with those things which remain somewhat stable, namely the libraries we return to again and again, along with a handful of techniques that come up in any language and on any project, programming being only one aspect of the greater ’systems’ or ‘enginerring’ perspective of the modern day. Refactoring is one technique. It is admittedly a challenge to get good enough at programming that one can get the computer to do what one likes, but this is like playing the right notes on an instrument, only the beginning, a base literacy. Proper code must be eminently readable, maintainable, stable, provably correct (if possible), documented, sensibly organized, and so on. Coding is like writing an essay but far more demanding. Writing is rewriting, and coding is refactoring. But what has refactoring to do with libraries?
Most code we write is specialized and not particularly reusable; at least, this should be the case if we are reliably refactoring. Theoretically, libraries are like refactoring that has already been done by others across independent projects. If a library hosts a number of functions shared by such independent projects, chances are that our independent project may also share those functions. In a sense, library functions are selected for utmost reusability and flexibility, and the more people who use them, the more reliable they tend to be. So here is one network externality, a superstar effect, that is distinctly on the side of progress (which is not analogous to complexity, indeed refactoring is our great technique against explosive complexity).
All this is to say that in the modern world, I do believe it is safe and wise to ‘rely but verify’ when it comes to libraries.