The grown-up programmer's (meta)manifesto

Another year is almost over and years should bring wisdom. Most people make lists for things they want to do in the new year, here, let's write a manifesto for programmers.

If you've been following me for a bit here or in an even-less-coherent form over twitter, these principles won't come as a surprise, but outside's cold, I'm taking a few days (mostly) away from work in my hometown, perfect time to organize some thoughts on this blog.

Meta-Rule 1: The only dogma is that we shall have none.

I love the mathematical exactness of Computer Science, we sit at the foundation of mathematics (formal systems), reasoning (logic) with fascinating implications on philosophy as well. But that theoretical foundation has nothing to do with the actual job of programming computers.

In practice, ours is a huge industry made of companies, teams, people, products and problems so heterogeneous that I doubt anyone can claim to understand it globally.

It's rare to find truly "wrong" ideas or algorithms and technologies that have been superseded by newer systems that better them in all accounts. Most of our work deals in tradeoffs. Computers are binary, programming is not.

Ideas that are appropriate for a stable team of senior programmers are often not appropriate for a team that sees a rapid turnover or that wants to grow a significant amount of juniors. 

Methodologies that work for innovative, research-driven projects are probably not the same that should be employed for less creative industries that on the other hand need to reliably evolve large, stable code-bases.

And of course, different industries have completely different cultures, requirements, expectations, constraints.

You can use goto(s).

Corollary: Understand your self-defenses.

Cosmic horror. When overwhelmed with complexity our minds tend to raise shields. Most often these come in two varieties, depending on if you're in your fascination phase or in your grumpy one.

Hype: In the former case, we tend to mindlessly follow others, ideas spread through hype and are embraced even when they are not appropriate in a given context. Eventually, we get burned by these, most of us learn our lessons and technology goes from hype to well-understood tool. 

Hate: This is the latter case, where on the other hand we harden around what we already know, the trusty tools and ideas we can rely upon, and see new ideas in an overly negative way.

Most times, this makes us actually effective, because most new ideas have high failure rates (research and innovation is defined by the high possibility of failure). Always predicting that something new will fail is a good bet. But this efficiency and resilience to variance comes with the risk of being blind to the next big thing. 

When the next big thing becomes obvious to all, it's usually too late to embrace it (the innovator's dilemma).

Note how both behaviors have virtuous traits. Hype helps new ideas to spread, and it's probably even necessary as we biased towards not wanting to move from our comfort zone, thus if we relied on perfect rationality for good new ideas to take their place in our collective tool belt, we would probably fail. And hate, as I wrote already, does indeed work most of the time, and makes people more effective at the job they do.

We have to understand, at a meta-level, that collectively these waves of hype and hate are there for a reason. Human behavior exists for a reason, and it is mostly there to serve us.

But if we understand this, then we can also be grounded and present to ourselves and understand when these behaviors do not serve us well. We should try to learn everything and idolize nothing. Know our history without becoming bound to it. Be curious without fully embracing something just because everyone else seems to be doing the same...

Corollary: Be skeptical of theories without context.

Only math is absolute. Any other science deals with primitives that we haven't invented (a.k.a. reality), thus, we can only hope to create models (theories) that are useful (predictive) in specific contexts (under given assumptions). Absolute statements are almost always propaganda, over-simplifications. We tend to like them because they are, well, simpler, and simplicity is attractive in an overly complex world. But unfortunately, they also tend to be wrong.

If a theory doesn't identify the conditions under which it applies, and the conditions under which it does not, then chances are it's not a good one. This applies even more broadly, to innovations, papers, algorithms, ideas in general.

Meta-Rule 2: Our minds are (usually) the scarcest resource, we should optimize for them.

"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? — Brian Kernighan, The Elements of Programming Style

Again, it's unfortunate that we tend to take all these mantras as absolutes, we like to quote them, smart, famous people can't be wrong, right? 

"Premature optimization is the root of all evil" amIrite?

Stop taking shortcuts to enlightenment. We need to actually think. To me, the famous quotes above is not really a call to not write clever code - we definitely do need to be clever, even often and often early on.

But the perils both Knuth and Kernighan (among many others) were trying to communicate are the ones that come with our tendency as passionate engineers to think too much about the machine.

And yes, I say that as someone who has few skills, but one of these few is a decent knowledge of modern hardware and how to exploit it. See. I didn't say that we should not think about the machine, we should. But the peril is to become persuaded that squeezing cycles or in other ways making the machine happy is the pinnacle of programming. The key part of what I wrote is "caring too much".

People (and products) are harder than hardware. Probably have always been, but it's especially true today with a stable industry and the commoditization of programming, where most software has hundreds of people behind and it's unlikely for new clever algorithms and implementations to be the key of the success of a product.

Corollary: Be as simple as possible.

Or, do not overcomplicate. Do not overengineer. As simple as possible might very well mean not simple at all, but we should fight needless complexity as our worst enemy.

Complexity is also very tricky to define, and that's why someone's overengineering might be someone else beautifully minimal code. Say, for example, that I need to do some numerical optimization in a C++ program. I can probably write my own small optimizer in say a thousand lines of code, from scratch. Maybe Nelder-Mead or Differential Evolution.

Or I could link Eigen to give me basic vector math and reduce my code to a hundred lines. Or, I could link an external optimization library and pick a ready-made algorithm with a single function call. Heck, I could even do a call into a Mathematica kernel that can do some fancy auto-selection of a good algorithm to use.

What is "simpler"? If you are a C++ programmer, you'll probably say one of the first two options. What if I said I'm doing this in Python? In that case, I'm pretty sure we would choose the latter ones! Why? Is it just because the library/module support in Python is better? I doubt it.

Complexity is contextual. It's not about the lines of code we write or the lines of code we depend on. It's about a window of concepts we care about. Our brain is limited and looks smarter than it is mostly because it aggressively focuses on certain things while ignoring others. We're perpetually blind. 

I think that this translates to coding as well. Like it or not, we can work only within a limited window. C/C++ programmers tend to be system-level programmers, and these are tools used when we need to focus our window on performance and memory. In this context, oftentimes writing relatively small amounts of code from scratch, with no dependencies, is easier than depending on lots of unknown "magical" code, because of attention is on a given level of execution and exactly what code we execute usually matters.

On the other hand, if I'm working in Python or Mathematica I actually like to think I am blessed by not having to look at the lower levels of code execution. That is not where the window is, typically. Scripting is great to sketch and explore ideas, the minutiae of execution don't matter in this "prototyping" mode, they would actually be distractions. We don't really want to write Python code thinking how the interpreter is going to execute it.

Usually, every tool is good in a given context (tradeoffs), the context dictates what matters and what we can and should ignore, and when we talk about complexity we should think specifically of how conceptually simple the components we care about are.

This also yields another corollary. If we chose the right tool, a good default is to be idiomatic. It's usually wrong to try to "bend" a tool that is made for a given thing to go way out of its comfort zone. It usually ends up in not very well hidden complexity (because we need to interface two different worlds) and also a loss of familiarity (others can't tell how things work, it's not obvious from the local context).

While we shouldn't fear to go outside the familiar, we shouldn't do it if it's not necessary.

Corollary: Constraints are useful.

If it's true that our brain is limited to a given window, explicitly controlling said window by imposing constraints must be a useful tool. 

This shouldn't be shocking, programming is creativity, and creativity does thrive under constraints. The paradox of choice that Barry Schwartz describes applies very much to programming as well. Writer's block.

I don't want to write much about this because I intend to write much more, and dedicate a future post to constraints in programming. I think that other than taking control, designing explicitly the window at which we operate for a given program, they can be useful when imposed artificially as well.

Over-constraining can help us focus on what matters for a given task, especially for people (like me) who tend to be "completionists" in programming - not happy if we don't know we did the "best" possible implementation. Here, adding constraints is a useful hack as we can re-define "best" to narrow the scope of a task.

C++, it’s not you. It’s me.

How I learned not to worry and tolerate C++

If you follow the twitter-verse (ok, and you happen to be in the same small circle of grumpy gamedevs that forms my bubble) you might have noticed lately a rise of rage and sarcasm against C++ and the direction it's taking.

I don't want to post all the relevant bits, but the crux of the issue, for the lucky among you who don't do social media, is the growing disconnect between people woking on big, complex, performance-sensitive and often monolithic and legacy-ridden codebases that we find in game development, and the ideas of "modernity” of the C++ standard community.

Our use-case is perhaps peculiar. We maintain large codebases, with large teams, but we never did great at modularization. This is our fault, and I’m not sure why it happened. Maybe it’s a combination of factors, including certain platforms and compilers not working well with dynamic libraries, or performance concerns. But most likely, it’s also the product of a creative environment where experimentation is a necessity, planning is inherently hard and architecture work is often simply neglected due to other production pressures.

The (AAA) gamedev use-case

Whatever are the historical reasons, the reality is that we live in an environment where most often than not:

• We don’t care about the STL, not its performance improvements. We developed our own bespoke containers, both because we need very ad-hoc algorithms tuned to specific problem sizes, and because of design issues of the STL itself which made it impossible to use (e.g. its historical reluctance to play well with things like memory alignment).
• We do care about all declinations of “performance”. Not only the final, all-optimizations enabled, code generation, but also performance in debug builds, and compiler performance. Iteration times.
• We care about being able to reason about code. Simplicity, not counted as lines of code, but as the ability to clearly understand what a line of code does.   We often would prefer verbose, even in the eyes of some more arcane code, to unpredictable code, where to understand the relation between what we wrote and what happens requires a lot of context, of “global” information.

Given this scenario, it should be clear to see how most of the C++ additions since C++11 have gone in the “wrong” direction. They are typically not adding any expressive power for our use-cases, but instead, bring lots of complexity to an already almost impossible-to-master language. Complexity that also “trickles” down to tooling, compilers, compile times, debuggers and so on.

It’s hard to overstate how bad this chasm is growing, with some direction being truly infuriating, but let me just bring a concrete example. Take the modernization of STL containers, say r-value references, initializer lists and all the ecosystem around that. Clearly, a huge feature for C++, allowing even significant savings for certain uses of the STL. And what you would call “a zero cost abstraction” - if you don’t use it, you don’t see it. Everyone’s happy, right?

Quite the contrary! The prime example of something that is entirely useless for people who already know about the cost of constructors, temporaries and the like, designed their code thinking of how to layout and transform bits, instead of higher level concepts sketched in a UML diagram.

Useless but dangerous, as these concepts increased the complexity of the language exponentially, to the point that very few can really claim to understand all their nuances, and yet are still hard to entirely avoid in projects made by lots of people, and where you do not necessarily even have the control of all the code due to external libraries.

And when all this humongous effort is taken on one side, features that would truly help performance sensitive large “system” programming, are still completely ignored. C++ still doesn’t have a “restrict” keyword. Strict aliasing is incredibly troublesome and even getting worse with certain proposals. Threads and memory alignment were implemented first (and arguably better) in the good old C.

We still don’t have vector types, not to mention fancier features like the ability of “transposing” the memory layout of arrays of structures (into SOAs or AOSOA and so on). We don’t have anything to help tooling, or compile times, like standardized reflection, serialization or proper modules. And we keep adding metaprogramming (templates) features before tacking complexity and usability issues (e.g. concepts, that now are scheduled to be in C++20).

What’s C++ anyway?

I think part of the disconnect, and even anger in the community, is due to a misinterpretation of what C++ is and always has been.

There is this myth that C++ is a low-level, high-performance/system programming language. It’s false, and it has always been false.

Clearly, a language that didn’t bother until recently to implement threads, is not a language for high-performance anything. There is nothing in C++ that concerns with system programming either, that is the C part of it, really. And so on and so forth, the more you look the more evident that truth is. Even the ancient Fortran can say it’s more concerned with performance than C++. C++ is not ISPC, nor Cuda. It doesn’t come with algorithms and data structure for low-latency, constrained memory use-cases, neither does care about large-scale data cases. Even python can be seen as a better language for high-performance computation, due to its ability to quickly implement embedded-DSLs. Nowadays, a lot of high-performance code leverages specialized compilers that use are embedded in relatively high-level languages via reflection. None of that is possible in C++.

C++ is a “zero cost abstraction” language. That part is true. But “zero cost” is not about performance. In fact, the guarantee that something won’t cost you anything when it’s not used doesn’t really mean that it will be fast when you do use it! What it gives, instead, is peace-of-mind. Bjarne said, do you like C? And its ecosystem? Great! Use this one, I guarantee I won’t screw with C, and if you like anything else I added, you get to use it.

And don’t get me wrong. This is genius. And a fundamental lesson that so many other languages still today don’t understand. Making a nice language is the easy part. If you’re even just a tourist of computer languages, have been exposed with concepts from veterans like Lisp and ML and so on, it’s not that hard to come up with a perfectly pleasant little language. And in fact, we have so, so many of them today. The hard part is selling such language! And if you don’t have a community with a strong need that nobody but you serve, that means you have to persuade people to hop over from whatever what they’re currently using. That is almost impossible.

Bjarne understood that the ecosystem is what matters most and C++ succeeded by being a drop-in replacement for C. Unfortunately now C++ is so complex, that creating a new language that seamlessly can work and replace it is a tall order, but it would also be an incredibly powerful tool for adoption.

Why things changed and what can we do about it?

Bjarne got marketing right, he probably understood people more than languages, which is not an insult, to the contrary! People are all that matters.

But if this is the premise, it wouldn’t even be surprising that C++ is drifting away from systems programming, from what C programmers care about. Nowadays, these use-cases are a minority. 

It is not unreasonable, from that point of view, to now try to appease to the cool kids writing web stuff. People who might be using python, or java, or go and so on. Programmers who are accustomed to working fast, gluing together frameworks and libraries more than they write any bespoke algorithm and data structure. In fact, when you think about it, adding OO to C was just what was trendiest at the time. It was a hype and marketing based decision, not really a smart one, as we now maybe see more clearly as the OO hype dissipates.

But I don’t even think it’s necessarily a conscious design decision. This language is huge today. Its community its huge, and it decided to go the way of the design by committee. Should you never do that? In my circles, the answer is definitely no, design by committee is the death of technology. But let’s present a more positive way of looking at it.

We know that democracy has a cost, a huge cost. It’s definitely not the most efficient way of governing, nor it produces the most brilliant decisions. It can be in fact, incredibly dumb. This is not news, even in the Roman Republic there were provisions for senators to elect a dictator for a temporary period when strong leadership was deemed necessary. Of course, the risk is for a dictator to become a tyrant, as Romans learned. So, democracies trade efficiency for risk aversion, variance for mean.

And that is what C++ is today. It’s an old language, even if it wants to play cool, it fundamentally is in maintenance mode, listening to a lot of people with a lot of ideas and going in the direction of the majority, not of strong design decisions. Sometimes people argue that the solution would be to have more representation of certain use cases in the committee. 

Perhaps a bit would help, but for how much I do like politics, I really don’t care about language ones. If I could vote on something, I would vote to remove people from the committee, make it a smaller one, not a noisier and bigger one, even if the noise was to argue “in my favor”. I simply don’t think you can have a lot of compromise in technology without ending up with something mediocre.

So? So we live with C++. Not because we like it, but because we need the ecosystem. The compilers, the IDEs, the language extensions, the low-level intrinsics, the legacy code. We restrict our usage to mostly C, and grab whatever rare new feature happens to integrate decently in our workflows. Maybe one day some new language will come, we already use bits and pieces of other ones when needed.

And maybe someday someone will learn the real lesson Bjarne had to teach, and really kill it! There is actually no reason for example that a language could not compile its own syntax in its files, but also allow to include C++ headers. Yes, you’d have to suffer and pay the pain of integrating a C++ compiler in whatever you come up with, and yes, your language would most probably just be something that expands to C++. Exactly how C++ started on top of C. But it’s unsexy, boring work, so most people will write yet another C-ish thing with a bit of ML in it on top of LLVM and call it a day...

Addendum: "Alex" asked an interesting question in the comments - why don't we make our own. I tried to answer that as well if you have a look below! 👇 

Engineering for squishy bags of meat

The section you can skip

I'm Italian, and I grew up in a family of what you could call middle-class intellectuals. My parents came from proletarian families, and were the first generation being university educated, not working in the fields. 

A teacher and a doctor, embedded in the life of the city and involved in local politics; we used to often host people over for dinner. And I remember, at the time, being fairly annoyed at how people could be highly regarded for their knowledge and intelligence even if they were limited to the sole study of the humanities, without knowing even the basics of math, or scientific thinking.

Fast forward and today, as a computer science professional, society in my small bubble seems to have undergone a (non-violent) cultural revolution. Today "nerds" are considered "smart", the lack of social skills almost a badge of honor, and things seem to matter only when they relate to quantifiable numbers.

Not always, I have to say the professional world of smart computer scientists is not a stereotyped as your average Hollywood portrayal, yet too often we live in a world that is yet again too vertical: programmers, artists, managers, often snarking at the other categories lack of "skills".

"We wanted to build an elegant, robust, and beautiful product"

I've been wanting to write this post for a while now, I have various sketches in my notebook from almost a year ago, but recently I found this very well written post-mortem analysis of RethinkDB failure as a startup. I'm no database expert, but what I found particularly interesting is the section talking about the design principles behind their product.

You want to make a new database product; what is that you're going to focus on? For RethinkDB, the answer was three key factors: correctness, simplicity of interface and consistency.

Seems pretty reasonable, databases are the central infrastructure for most of today's applications, and who would like to build a billion-dollar product on something that doesn't guarantee correct operation? And definitely, today's startups need to move fast and are willing to adopt whatever new technology that helps to get results fast, so simplicity and consistency could give a competitive advantage.

Of course, it's no surprise as they were talking candidly about the reasons why the failed, that these were "the wrong metrics of goodness" (their words). What did people want? According to the article, the right metrics would have been: timely arrival, "palpable" speed (benchmarks, marketing), and a good use case. In other words, a product that:

1) Solved a problem (use case)

2) Now (timely arrival)

3) While making people happy (the importance of perceived performance)

This misjudgment led not only to the demise of the startup but also to a lot of frustration, depression, and anger, as what could be seen as inferior products made big impacts in the market.

Clash of worlds

The sin is, of course, the idea that an abstract notion of beauty matter at all. We write software to achieve given results, and these results are for all but the most theoretical of works, something that somehow has to make people happy. 

Everything else is just a mean towards that, a tool that can be used well or not but not a goal. The goal is always to sell to your market, to do something that people will... like. 

It requires understanding your customers and understanding that we are all people. It's the clash between "features" and "experiences".

When we think about features it's easy to end up in the measurement fallacy. We are doing X, and we are doing X measurably better for certain axes of measurement than another product, thus, we are better. Wait for adoption and success.

This is the peril of living in a verticle bubble of knowledge. People in tech think about tech, we work with it, we create it, it's easy and to a degree even inevitable to start technology for technology's sake.

But the truth is that user experience, workflows, and subjective experiences are about technology, and they are about research and computer science. 

They are just -harder- problems to solve, problems that most often have no good metric to optimize at all, sometimes because we don't know how to measure "soft" qualities, but many times because these qualities are truly hard to isolate. 

Better evolves through "blind" iteration and luck, and it's subject to taste, to culture, to society. Look at all the artifacts around you. Why is a musical instrument better than another one? Or a movie, game, car, pen, table, phone... 

Good design is humanistic in nature, we have to understand people, what they like, what they fear, how they use technology, how much they can adapt to change, what risks they evaluate. Objectively good is not good enough. Good design makes objective sense but overcomes also emotional, social, environmental obstacles.

Computer Science should be taught together with sociology, psychology, design, interactive arts.

Specialization

To be fair though it's not surprising that people who are deeply invested in a given field, don't fare well in others. The tension between vertical knowledge, specialization, and horizontal knowledge that allows us to be well-rounded, is ineluctable. 

We are more and more pushed towards specialization because our fields are so deep and complex, and even the most brilliant minds don't have infinite mental resources, so there are always tradeoffs to be made.

Whether you went to a university or not, chances are that to become a specialist you needed a deep, technical immersion in a given field. We isolate ourselves into a bubble, and to a degree, we need to.

The real problem begins when we start thinking that a given bubble is representative of the world, that because given qualities matter inside it, they actually do matter.

When that starts, then we start adopting the wrong metrics to judge the world, and we end up frustrated when the world doesn't respond to them. 

Bubbles are of course a very general problem, even a very scary one when you think that really our world is becoming more and more complex to the degree that's not easily explicable even by experts, it's not easy to encapsulate in a set of rules. Complexity can push us inwards, to simplistic explanations and measure of goodness that we can become very attached to, even if they are very narrow-sighted.

It doesn't help that we, as humans, seek confirmation more than truth.

Academic research

Take for example the world of academia and scientific research. Researchers are judged on novelty, what a paper really needs is a measure, then, given such measure one can assert that a given technique is objectively better in a given context. 

If something is better, but not in a way that is easily measured, it's not something that can be easily published. This is not per se a terrible problem, it becomes a problem when we start thinking that these measures are all that really matters. 

I work in computer graphics; if I make a new system that generates simplified 3d models, and all the artists that use it think it's revolutionary, it produces a much better output and it's easier to use than anything else, did I make something novel and noteworthy? Surely yes, that's, in fact, all that matters.

But if this notion of how much better it is can't be measured in some kind of metric, it's not something that can be considered a novel research in academic terms. This is the nature of applied research, but it also creates a disconnect between academia and the rest of the world: a bubble, that we have to be aware of. 

When we ignore the existence of the bubble, frustration arises. Industry professionals are frustrated when academic research doesn't end up "working" for their needs. And researchers are frustrated when they see the industry "lagging behind" the theoretical state of the art.

Conclusions

Is it all bleak? To a degree, yes, it is when you realize that the world is not easily reducible to comforting measures that we like. That biases are inevitable, that specialization is inevitable and bubbles are inevitable.

I say that "experience is a variance reduction technique", we become more predictable and entrenched, better, at doing something, but we pay a price to it.

What we can certainly do though is to at least -aware- of these mechanisms. Doubt ourselves. Know that what we think it matters, might not matter at all, and seek different point of views. Know that what we think is rational, most of the times is just a rationalization. Be aware of the emotional factors in our own choices.

What we can certainly do is to break -a bit- out of our vertical pits of knowledge, and be just curious enough, learn just enough to be able to interface with different people. That can be just the solution we need, we can't know all, and we can't really understand the world without any filter. We can though make sure we are able to talk and engage with people that are different, that invested their verticality in other fields.

In the end, personally, I don't consider my journey towards being truly "smart" complete or even really started. But at least I am a bit aware... I also have the gift of a spouse that beats me down showing me how bad I still am at many aspects of "smartness", and a strong enough ego to take that beating. Small steps.

Over-engineering (the root of all evil)

Definitions

Over-engineering: using prematurely for tools, abstractions or technical solutions, resulting in wasted effort and unnecessary complexity.

When is a technique used prematurely? When it doesn't solve a concrete, current problem. It is tempting to define good engineering in terms of simplicity or ease of development, but I think it's actually a slippery slope. Simplicity means different things to different people. 

One could see programming as compression (and indeed it is), but we have to realize that compression, or terseness, per se, is not a goal. The shortest program possible is most often not the simplest for people to work with, and the dangers of compression are evident to anybody that had to go through a scientific education: when I was in university the exams that were by far the most difficult came with the smallest textbooks...

Simplicity also means different things to different people. To someone in charge of low-level optimizations working with fewer abstractions can be easier than having to dive through many software layers. To a novice, a software written in a more idiomatic way for a given language might be much easier than something adapted to be domain specific.

Problems, on the other hand, measurable, concrete issues, are a better framework. 
They are still a soft, context-dependent and team-dependent metric, but trying to identify problems, solutions, and their costs brings design decision from an aesthetic (and often egocentric) realm to a concrete one.
Note: This doesn't mean that good code is not or shouldn't be "beautiful" or elegant, but these are not goals, they are just byproducts of solving certain problems the code might have.
Also, "measurable" does not mean we need precise numbers attached to our evaluations, in practice, most things can't be better than fuzzy guesses, and that's perfectly fine.

Costs and benefits

Costs are seldom discussed. If a technique, an abstraction, an engineering solution doesn't come with drawbacks, it's probably because either it's not doing much, or because we've not been looking hard enough. 

• Are we imposing a run-time cost? What about the debug build? Are we making it less usable?
• Are we increasing build-times, lowering iteration times? 
• A human one, in terms of complexity, obfuscation, ability to on-board new engineers?
• Are we making the debugging experience worse? Profiling?
• Do our tools support the design well? Are we messing up with our source control, static analyzer and so on?
• Are we decreasing code malleability? Introducing more coupling, dependencies? 
• Reducing the ability to reason locally about code? Making details that matter in our context hidden at call-site? Making semantics less explicit, or less coupled to a given syntax? Violating invariants and assumptions that our code-base generally employs?
• Does it work well with the team culture? With code reviews or automated testing or any other engineering practice of the team?

We have to be aware of the trade-offs to discuss an investment. But our tendency to showcase the benefits our ideas and hide the costs is a real issue in education, in research, in production. It's hardwired in the way we work. It's not (most often) even a matter of malice, it's simply the way we are trained to reason, we seek success and shy away from discussing failure.

I've seen countless time people going on stage, or writing articles and book, honestly trying to describe why given ideas are smart and can work while totally forgetting the pain they experience every day due to them.

Under-engineering

And that's why over-engineering truly is the root of all evil. Because it's vicious, it's insidious, and we're not trained at all to recognize it. 

It is possible to go out and buy technical books, maybe go to a university, and learn tens or hundreds of engineering techniques and best practices. On the other hand, there is almost nothing, other than experience and practice, that teaches restraint and actual problem solving.

We know what under-engineering is, we can recognize duplicated code, brittle, and unsafe code, badly structured code. We have terminology, we have methodologies. Testing, refactoring, coverage analysis...

In most of the cases, on the other hand, we are not trained to understand over-engineering at all.
Note: In fact over-engineering it's often more "pronounced" in good junior candidates, whose curiosity lead them to learn lots of programming techniques, but that have no experience in their pitfalls and can easily stray from concrete problem solving.

This means most of the times when over-engineering happens it tends to persist, we don't go back from big architectures and big abstractions to simpler systems, we tend to build on top of them. Somewhere along the road, we made a bad investment, with the wrong tradeoffs, but now we're committed to it.

Over-engineering tends to look much more reasonable, more innocent than under-engineering. It's not bad code. It's not ugly code. It's just premature and useless, we don't need it, we're paying a high price for it, but we like it. And we like technology, we like reading about it, keeping ourselves up-to-date, adopting the latest techniques and developments. And at a given point we might even start thinking that we did make the right investment, that the benefits are worth it, especially as we seldom have objective measures or our work and we can always find a rationalization of almost any choice.

I'd say that under-engineering leads to evident technical debt, while over-engineering creates hidden technical debt, which is much more dangerous. 

The key question is "why?". If the answer comes back to a concrete problem with a positive ROI, then you're probably doing it right. If it's some vague other quality like "sharing", "elegance", "simplicity", then it's probably wrong, as these are not end goals.
When in doubt, I find it's better to err on the side of under-engineering, as it tends to be more productive than the opposite, even if it is more reviled.

"Premature optimization is the root of all evil" - Hoare, popularized by Knuth.

I think over-engineering is a super-set of premature optimization. In the seventies, when this quote originated, that was the most common form of this more "fundamental" evil.
Ironically, this lesson has been in the decades so effective that nowadays it actually helps over-engineering, as most engineers read it incorrectly, thinking that in general performance is not a concern early on in a project.

Intermission: some examples

- Let's say we're working on a Windows game made in Visual Studio. Let's say that you are using a Visual Studio solution and it's done badly, it uses absolute paths and requires the source-code and maybe some libraries to be in a specific directory tree on the hard drive. Anybody can tell that's a bad design, and the author might be scorned for such an "unprofessional" choice, but in practice, the problems that it could cause are minimal and can be trivially fixed by any programmer.

On the other hand, let's say we started using, for no good reason, a more complex build system, maybe packages and dependencies based on a fancy new external build tool of the week.
The potential cost of such a choice is huge because chances are that now many of your programmers aren't very familiar with this system, it's bringing no measurable benefits but now you've obfuscated an important part of your pipeline. Yet, it's very unlikely that such decision will be derided.

- Sometimes issues are even subtler, because they involve non-obvious trade-offs. A fairly hard-coded system might be painful in terms of malleability, maybe doing changes in this subsystem requires every time editing lots of source files even for trivial operations.

We really don't like that, so we replace this system with a more generic, data-driven one which allows to do everything live, doesn't even require to recompile code anymore. But say that such system was fairly "cold", and the changes were actually infrequent. Suppose also that the new system takes a fair amount more code and now our entire build is slower. We ended up optimizing a workflow that was infrequent but on the down side we slowed down the daily routine of all our programmers on the team...

- Let's say you use a class where you could have used a simple function. Maybe you integrate a library, where you could have written a hundred lines of code. You use a templated container library where you could have used a standard array or ad-hoc solutions. You were careless and now your system is becoming more and more coupled at build-time due to type dependencies. 
It's maybe a bit slower in runtime than it could be or it makes more dynamic allocations than it should, or it's slow in debug builds, and it makes your build time longer while being quite obscure when you actually have to step in this library code.

This is a very concrete example, happens often yet chances are that none of this will be recognized as a design problem, and we often see complex tools built on top over-engineered designs to "help" solving their issues. So now you might use "unity builds" and distributed builds to try to remedy the build time issues. You might start using complex memory allocators and memory debuggers to track down what's causing fragmentation and so on and so forth. 

Over-engineering invites more over-engineering. There is this idea that a complex system can be made simpler by building more on top of it, which is not very realistic.

Specialization and constraints

I don't have a universal methodology for evaluating return on investment, once the costs and benefits of a given choice are understood. And I think there isn't in general one because this metric is very context sensitive. What I like to invite engineers to do is to think about the problem, be acutely aware of it.

One of the principles I think is useful as a guidance is that we operate with a finite working set: we can't pay attention to many things at the same time, we have to find constraints that help us achieve our objectives. In other words, our objectives guide how we should specialize our project.

For example, in my job I often deal with numerical algorithms, visualization, and data exploration. I might code very similar things in very different environments and very different styles depending on the need. If I'm exploring an idea, I might use Mathematica or Processing. 
In these environments, I really know little about the details of memory allocations and the subtleties of code optimization. And I don't -want- to know. Even just being aware of them would be a distraction, as I would naturally gravitate towards coding efficient algorithms instead of just solving the problem at hand.

Often times my Mathematica code actually leaks memory. I couldn't care less when running an exploratory task overnight a workstation with 92 gb of ram. The environment completely shields me from these concerns and this is perfect, it allows me to focus on what matters, in that context. I write some very high-level code, and somehow magic happens.

Sometimes I have to then port these experiments to production C++ code. In that environment, my goals are completely different. Performance is so important to us that I don't want any magic, I want anything that is even remotely expensive to be evident in the location where it happens. If there was some magic that worked decently fast most of the times, you can be sure that the problems it creates would be lost until there are so many locations where that happens that the entire product falls apart.

I don't believe that you can create systems that are extremely wide, where you have both extremely high-level concerns and extremely low-level ones, jack-of-all-trades. Constraints and specialization are key to software engineering (and not only), they allow us to focus on what matters, keeping the important concerns in our working set and to perform local reasoning on code.

All levels

Another aspect of over-engineering is that it doesn't just affect minute code design decisions or even just coding. In general, we have a tendency to do things without proper awareness, I think, of what problems solve for us and what problem they create. Instead, we're often guided either by a certain aesthetic or certain ideals of what's good.

Code sharing for example and de-duplication. Standards and libraries. There are certain things that sometimes we consider intrinsically good, even when we have a history of failures from which we should learn. 

For engineering, sharing in particular is something that comes with an incredible cost but that is almost always considered a virtue per se, even by teams which have experience actually paying the price in terms of integration costs, of productivity costs, of code-bloat and so on, it came to be just considered "natural".

"Don't reinvent the wheel" is very true and sound. But "the wheel" to me means "cold", infrastructural code that is not subject to iteration, that doesn't need specialization for a given project. 
Thinking that sharing and standardization is always a win is like thinking that throwing more people at a problem is always a win, or that making some code multithreaded is always a win, regardless of how much synchronization it requires and how much harder it makes the development process...

In a videogame company, for example, it's certainly silly to have ten different math libraries for ten different projects. But it might very well not be silly to have ten different renderers. Or even twenty for what matters, rendering is part of the creative process, it's part of what we want to specialize, to craft to a given art-direction, given project scope and so on.

People

Context doesn't matter only on a technical level, but also, or perhaps even more, on a human level. Software engineering is a soft science!

I've been persuaded of this having worked for a few different projects in a few different companies. Sometimes you see a company using the same strategy for similar projects, only to achieve very different results. Some other times on the other hands, similar results are obtained by different products in different companies by employing radically different, almost opposite strategies. Why is that?

Because people matter more than technology. And this is perhaps the thing that we, as software engineers, are trained the least to recognize. People matter more than technology.

A team of veterans does not work the same as a team that has or needs a lot of turnover. In the game industry, in some teams innovation is spearheaded by engineers, in some others, it's pushed by artists or technical artists.
A given company might want to focus all their investment in a few, very high profile products, where innovation and quality matters a lot. Another might operate by producing more products and trying to see what works, and in that realm maybe keeping costs down matters more.

Even the mantras of sharing and avoiding duplication are not absolute. In some cases, duplication actually allows for better results, e.g. having a separate environment for experimentation than final production. In some cases sharing stifles creativity, and has upkeep costs that overall are higher than the benefits.

It's impossible to talk about engineering without knowing costs, benefits, and context. There is almost never a universally good solution. Problems are specific and local.
Engineering is about solving concrete problems in a specific context, not jumping carelessly on the latest bandwagon.

Our industry I feel, still has lots to learns.

The following provides no answers, just doubts.

Technical debt, software rot, programming practices, sharing and reuse etcetera. Many words have been written on software engineering, just today I was reading a blog post which triggered this one.

Software does tend to become impervious to change and harder to understand as it ages and increases in complexity, that much is universally agreed on, and in general it's understood that malleability is one key measure of code and practices that improve or retain it are often sensible.

But when does code stop to be an asset and starts being a liability? For example when should we invest on a rewrite? 

Most people seem to be divided in camps on these topics, at least in my experience I’ve often seen arguments and entire teams even run on one conviction or another, either aggressively throwing away code to maintain quality or never allowing rewrites to capitalize on investments made in debugging, optimization and so on.

Smarter people might tell you that different decisions are adeguate for different situations. Not all code needs to be malleable, as we stratify certain layers become more complex but also require less change, new layers are the ones that we actively iterate upon and need more speed. 

Certainly this position has lots of merit, and it can be extended to the full production stack I’d say, including the tools we use, the operating systems we use and such things.

Such position just makes our evaluation more nuanced and reasonable, but it doesn’t really answer many questions. What is the acceptable level of stiffness in a given codebase? Is there a measure, who do we ask? It might be tempting just to look at the rate of change, where do we usually put more effort, but most of these things are exposed to a number of biases.

For example I usually tend to use certain systems and avoid others based on what makes my life easier when solving a problem. That doesn’t mean that I use the best systems for a given problem, that I wouldn’t like to try different solutions and that these wouldn’t be better for the end product. 

Simply though, as I know they would take more effort I might think they are not worth pursuing. An observer, looking at this workflow would infer that the systems I don’t use don’t need much flexibility, but on the contrary I might not be using them exactly because they are too inflexible.

In time, with experience, I’ve started to believe that all these questions are hard for a reason, they fundamentally involve people. 

As an engineer, or rather a scientist, one grows with the ideal of simple formula to explain complex phenomena, but people behaviour still seems to elude such simplifications.

Like cheap management books (are there any other?) you might get certain simple list of rules that do make a lot of sense, but are really just arbitrary rules that happened to work for someone (in the best case, very specific tools, worst just crap that seems reasonable enough but has no basis), they gain momentum until people realize they don’t really work that well and someone else comes up with a different, but equally arbitrary set of new rules and best practices.
Never they are backed by real, scientific data.

In reality your people matters more than any rule, the practices of a given successful team don’t transfer to other teams, often I’ve seen different teams making even similar products successfully, using radically different methodologies, and viceversa teams using the same methodologies in the same company managing to achieve radically different results.

Catering to a given team culture is fundamental, what works for a relatively small team of seniors won’t apply to a team for example with much higher turnover of junior engineers. 

Failure often comes from people who grew in given environments with given methodologies adapted to the culture of a certain team, and as that was successful once try to apply the same to other contexts where they are not appropriate.

In many ways it’s interesting, working with people encourages real immersion into an environment and reasoning, observing and experimenting what specific problems and specific solutions one can find, rather than trying to apply a rulebook. 

In some others I still believe it’s impossibile to shut that nagging feeling that we should be more scientific, that if medicine manages to work with best practices based on statistics so can any other field. I've never seen so far big attempts at making software development a science, deployed in a production environment. 

Maybe I'm wrong and there is an universal best way of working, for everyone. Maybe certain things that are considered universal today, really aren't. It wouldn't be surprising as these kinds of paradigm seem to happen in the history of other scientific fields.

Interestingly we often fill questionaries to gather subjective opinions about many things, from meeting to overall job satisfaction, but never (in my experience) on code we write or the way we make it, time spent where, bugs found where and so on...

I find amusing to observe how code and computer science is used to create marvels of technological progress, incredible products and tools that improve people’s lives, and that are scientifically designed to do so, yet often the way these are made is quite arbitrary, messy and unproductive.

And that also means that more often than not we use and appreciate certain tools we use to make our products but we can’t dare to think how they really work internally, or how they were made, because if we knew or focused on that, we would be quite horrified.

P.S.
Software science does exist, in many forms, and is almost as old as software development itself, we do have publications, studies, metrics and even certain tools. But still, in production, software development seems more art than science.