Learning from failures

Failures present valuable opportunities for learning only when there is no blaming or shaming of individuals. What may initially appear to be a foolish oversight often reveals systemic issues. A common cause is placing inexperienced personnel under unrealistic time pressures—neither of which can be resolved in the short term. This can lead decision-makers to treat complex systems as if they were linear or simplistic, ignoring interconnected factors and feedback loops [The Logic Of Failure]. People often focus on immediate outcomes without considering long-term or indirect effects, resulting in burnout, stress, and demotivation [Death March].

If failures were treated as insights that help uncover the mysteries of the physical world, they might even become occasions for celebration—because each failure reveals something new. They could be thoroughly analyzed and shared widely so that everyone benefits. Of course, this requires a reality-based culture of critical thinking, rather than a rush to find someone to blame—whether to feel good about ourselves or crush our rivals—until the next mishap. The road to most engineering catastrophes is paved with cover-ups of smaller mistakes.

The best examples of failure analysis come from the aviation industry, where even seemingly outrageous mistakes [Aeroflot Flight 593, Pakistan Airlines 8303] are traced back to systemic root causes like problematic hiring processes and insufficient training.

Music: Adelita (classical guitar)

Kanban vs. Scrum

Kanban is highly flexible with no fixed timeframes. In Scrum, work effort has to be estimated and organized into fixed-length iterations, known as sprints, typically lasting 2-4 weeks.

Kanban is better for projects with changing priorities, while Scrum is more suited for projects with defined requirements and deliverables. If continuous delivery is essential, Kanban might be the better choice. For projects benefiting from iterative development, Scrum is preferable.

For the type of work I am doing, I prefer using the Kanban method, incorporating daily stand-up meetings where each person speaks for no more than 5 minutes. This ensures that everyone stays focused and that efforts are concentrated on high-priority issues.

Outsourcing software development

When outsourcing parts of a software development project, you'll need to overcome several challenges:

Clearly Define the Problem and Verification: It's essential to have a detailed design and create prototypes for testing your ideas before writing a statement of work. The outsourcing partner mainly contributes in terms of usability and maintainability, i.e. they should convert the proof of concept that can barely be used by experts to a product that can easily be used by normal people.

Identify Reliable Partners: This is arguably the most challenging step. In the software world, many claim expertise – after all, it's just coding, right? However, the ideal scenario is knowing firsthand that the team or individuals you're considering are truly capable of handling the project. If you are not sure, starting with small scale (~1 week) discovery projects (free of charge) will be helpful.

Manage Budgetary Constraints: As costs rise, so does the bureaucracy and the number of required approvals. One effective way to control cost is to narrow the project's scope and build on top of open source. While it's imperative to obtain the source code from your partner, consider handling documentation, formal verification and maintenance internally.

By adhering to these guidelines, you maintain control over the project and its future maintenance. You also gain access to top-tier talent that might not otherwise join your team, all while potentially reducing costs compared to in-house development.

Becoming a 100X Engineer

To become a 100X engineer, always ponder the following questions before attempting to solve a problem:

1. Is this problem worth solving right now?
2. Would buying an existing solution be feasible?
3. Would outsourcing be feasible?
4. What is the simplest design to solve this problem? Simple is smooth and smooth is fast.

Music: The Wellerman (Sea Shanty) 

Expectations vs. reality

During my 26 years of engineering, I had my fair share of failed projects. These projects never failed due sabotage or lack of effort. In the aftermath, everybody, from engineers to the customer, was miserable. It makes no sense to blame the people in the project for not having tried hard enough, because they did their best, even burning out in the process. 

The root cause of defeat was always a gap between expectations and reality. Reality includes difficulty of the problem, available budget and quality of personnel. Not understanding reality resulted in short deadlines and low budgets, which in turn assured failure. If expectations were more in-line with reality, all that money and effort would not have gone to waste and a valuable end result could have been achieved, avoiding lots of unnecessary suffering. The only way out of this spiral is having competent, experienced managers who have a strong link to reality, and therefore can help align expectations with resources. Further reading.

Project priorities and role of top management

An important role of top management in a large corporation is to make sure that projects are managed according to contractual priorities. One way project managers deviate from priorities is by pursuing features not expressed in the contract with the argument that they are trying to make the product better. This approach would be fine if there were infinite time and resources. Since projects have to operate within hard constraints, a manager has to first satisfy the minimum requirements, otherwise the project will be late and engineers will be confused and demoralized.

In my earlier years, I thought that when managers are reminded of priorities, they would quickly change their way of doing business. To my dismay, I saw that this is sometimes not the case, with catastrophic results. When you say to anyone "a project should follow priorities", you will get a "yes" response 100% of the time. But implementation can vary widely. After living in fantasy land, the second most common reason of project failures are not technical or budget deficiencies but wasting time on low priority tasks while many high priority ones were still waiting for attention.

Top management should from time to time have informal meetings with people one or two levels down the project manager and ask them whether they think the project in managed according to priorities.

Fantasy and quality

Estimating how long a complex project will take is difficult, especially if it contains software development. Experienced engineers can estimate cost based on past projects. However, I am puzzled to see time and time again senior managers who come up with overly optimistic schedules. 

One reason might be to motivate employees to work harder. The novices will think that a short duration overtime will solve the crisis and they will be back to normal where in fact it will take months. You can fool people only once, and when they find out, you won't be able to motivate them again.

Another reason is to avoid conflict with upper management and after some time, starting to believe in your own optimisim, see Waltzing with Bears.

When management continues to live in fantasy land, after a couple of months people will be disillusioned, morale drops and the project takes longer with lower quality than would be the case at a normal pace. This type of mismanagement is only rational if the manager is expecting a change of position soon, meaning that he/she won't have to deal with the mess. 

Quality suffers because everyone thinks that this will be over quickly with a firm push, resorting to quick and dirty fixes, reasoning that the mess will be cleaned up soon. The longer the crisis mode drags on, the more technical debt is accumulated, which in the end becomes so large that no one is eager to refactor the system, fearing that the slightest change will break it. This leads to high cost maintenance and can cause catastrophic failures whose root cause can be impossible to find. Due to endless rework and troubleshooting, total cost of project increases dramatically.

Intellectual work like software development requires good concentration which in turn requires a rested, motivated individual. Long overtimes are like killing the goose that laid golden eggs.

My suggestions in a nutshell:

1. If you are a manager, avoid more than a couple of weeks per year overtime. Have a look at my reading list. Always look at past data, be aware that avoiding conflict with upper management might lower your stress in the short run but it has a high price down the road.
2. If you are one of the "managed" ones, politely point out the risks. But if you see that no one is interested in that, shut up and concentrate on improving the quality of your work and documentation so that when disaster strikes, you will be able to quickly demonstrate that your piece of work is not to blame. Read Death March.

Software product maturity levels

Software products have three maturity levels:

1. Product sometimes works.
2. Product usually works.
3. Product always works.

Transition from lower to higher maturity levels is achieved through experienced personnel, better development practices and tests.

Character and failure

You can judge the character of people or establishments by looking at how they handle failure. Do they look for someone to blame or do they analyze the steps leading to failure and try to implement a system that has a good chance of preventing such mishaps in the future? That system should NOT become a huge bureaucracy, it should be invisible and make work life more pleasant than before by reducing ambiguity related stress.

Technical Team Leader

A software technical team leader's main responsibility is to make the team better, because that will result in highest efficiency due to each member becoming a better version of themselves. Examples of duties of a techlead:

• Review/refactor design and documentation that the team produces. See Scrum sprints.
• Review code, optimize/refactor for comprehensibility, speed, memory and safety. This will keep you up to date with the code.
• Evaluate and select code analysis tools.
• Evaluate tutorials and point team members to them.
• Use stack instead of heap.
• Solve difficult technical challenges that are beyond the capabilities of a team member.
• Assist in solving difficult problems. Since you have been continuously reviewing design and code, you won't need extra acclimation time.
• Review issue definitions and refactor for comprehensibility.
• Review commits to repository (commits should be multiple times a day, not once a week)
• Write simulators/emulators for external hardware and software components
• Write initial versions of build scripts. Scripts will be maintained by juniors.
• Write initial versions of setup scripts that makes setting a development environment as simple as a single mouse click.  Scripts will be maintained by juniors.
• Find ways to decrease build times, create tutorials.
• Disable antivirus
• Use Linux instead of Windows.
• Precompiled headers
• Optimize IDE settings, create tutorials.
• Shortcuts
• Folder view
• Output directory = bin

Notice that the list does not contain product work because fulfilling the above would take a lot of time. Product development consists mostly of repetitive/boring work that a junior can handle with guidance of a senior. By freeing the techlead from those chores and making him a team mentor and solver of only hard problems, you improve productivity of everyone.

Remote work challenges

The global coronavirus pandemic has forced many institutions to think seriously about remote work. The first idea that comes to mind is make people work on laptops. If the work place had to use a local network isolated from the internet due to security concerns, the following challenges have to be faced:

• A lot of laptops have to be reconfigured by the IT department so that they cannot be connected to the internet. Software needed for daily work have to be installed. That's a lot, depending on the number of laptops and the number of software to be installed on each laptop.
• Many software products needed for development rely on local license servers. How are you going to make them work on laptops that are neither connected to your LAN nor the internet? You may have to obtain standalone licenses (extra cost) and install them on each laptop. In my case, I need licenses for two different products. May be existing license servers can be made to work on a virtual machine on the laptop.
• Complex projects require multiple people to work on the same software components. How will you prevent merge conflicts when the oubreak has calmed down and everybody wants to push their work to the local repository?
• Embedded software projects require multiple people and hardware working together. You might take your hardware home but you won't be able to get your colleagues and their hardware/software due to social isolation requirements to prevent the virus spread.
• If you have kids at home, how will you prevent them from constantly interrupting you? By having a schedule and clear visual indicators that you are working (like a sign on the door).

If your software/hardware were highly modular, if you had abstractions/simulators, you could take only the parts that are suitable to work at home. But true modularity is almost never the case. In my opinion, during the remote period it is not realistic to expect work to be done as usual. The main areas to focus are:

• Limited development with only your hardware and software (if standalone licenses can be purchased).
• Improving documentation of existing software/design. Documenting the system will show you clearly where the design is too complex and needs improvement.
• Self education. Example subjects for me: Blockchain, real time operating systems, embedded software development.
• Daily physical exercise.

02.08.2022: TopTal The Suddenly Remote Playbook

The myth of developer independent systems

Today a colleague of mine was asserting that the core reason of sloppy development is not having detailed quality procedures, i.e. bureaucratic systems independent of who is following them. I told him that we have more than enough development procedures, that any complex system is always dependent on the quality of people who implement them. If a project is sufficiently complicated and there is time pressure and market competition, you need high quality people who can evaluate risks realistically, tailor standards and cut corners to save time and effort. But they are also acutely aware of technical debt and clean up the quick and dirty solutions when time permits it.

You never have enough resources to develop your ideal system and always have to settle on a viable product. If your chief engineers are not wise enough, they will underestimate complexity, confuse priorities and take unreasonable risks. The problem is that they won't know that those risks are high. For them it will look like a brilliant course of action, until disaster strikes. If they are bad enough, they won't even be able to learn the correct lessons from failure.

Bad products are not the result of bad procedures but bad engineers and managers. The shortest path to success is to find and nurture good people who can develop quality products, write realistic procedures and make the team better.

Why do we have different electoral systems?

Why do we sometimes have top down and other times bottom up elections? Examples:

• Top down: During presidential elections people vote for the president and the parliament. After the election, the president appoints people to top jobs.
• Bottom up: During worker's union elections workers vote for their representatives. The representatives vote for the union president (bottom up).

Do these election types have any logic depending on their context or are they a result of historical accidents?

Software development maturity levels

Software development maturity levels are:

1. Modular design and implementation.
2. Nightly build script and pass/fail email.
3. Automated tests with tools like Macro Scheduler.
4. Unit tests.
5. Continuous integration.
6. Continuous deployment.

Transforming low-tech companies to high-tech

I have been involved in the digital transformation of low technology companies to high technology ones. The motivation for tranformation was competitive advantage, higher profit margins and diversification into new markets. 

Almost every organization seems to include digital transformation in its vision and strategy, but most struggle with executing digital transformation initiatives. There are myriad reasons: the challenges of introducing new technologies and providing the workforce with relevant skills, ensuring that the company’s culture and organizational structures are conducive to change, and anticipating correctly which processes need to change and how, to name a few.

Key areas that I had to oversee and lead were the following:

1. Have short and to the point meetings, no excessive story telling.
2. Focus on the right problems, i.e. good analysis, instead of wasting effort on poorly defined objectives.
3. Create high tech transformation road map document.
4. Instead of local server infrastructure, use cloud services (e.g. G Suite vs Office 365, GitHub) as much as possible. Prerequisite: Fast and reliable internet connection of all offices and factories.
5. Mentor managers and lead engineers. Teach the importance and best practices of good documentation, reviews and good tests in achieving quality. Form a culture that embraces trial and error and treats failures as improvement opportunities. Focus on minimum viable products. Have short design/development/test cycles via sprints.
6. Eliminate micromanagement, communicate business context.
7. Develop critical tech (algorithms, software, electronics, tools, simulators/emulators and HWIL test infrastructure) in-house.
8. Tailor certification standards to project scope because most standards are a mess.
9. Make sure that quality procedures are adequate and applicable in real world scenarios.
10. Improve hiring practives, form capable tech teams. Provide remote working opportunities to attract talent. Isolate them from mundane work to decrease turn-over rates.
11. Improve information dissemination through wikis and forums. To improve technical docs, when an engineer is assigned to a task that he is a novice of, he also has to update documantation related to that task.
12. Have methodologies in place to support remote work (daily scrum video conferences, sprints, Google Drive, GitHub etc.)
13. Improve internet presence (dynamic web site, videos) for better marketing and recruitment.

The most important characteristics of an entrepreneur

I have worked with different types of entrepreneurs over the years and came to the conclusion that the most important characteristics of an entrepreneur is to have high energy and low pride. By low pride I mean not being offended easily. Average intelligence is enough until you want to exceed the 100M$ mark. In terms of monetary value:

• High Energy + Low pride = 100M$
• Medium Energy + Medium pride = 10M$
• Low Energy + Low pride = You shouldn't be an entrepreneur
• High Energy + High pride = You shouldn't be an entrepreneur
• High Energy + Low pride + High intelligence = No upper limit to the wealth you can create

Team lead vs mentor

The difference between a team leader and mentor is that the team lead is responsible for meeting project deadlines while the mentor is concerned with personal development of mentees.

Performance evaluation

Success of complex projects depends more on effective teams than rockstar programmers. Therefore, performance evaluation should focus more on the team than the individual. An individual's ability to make his teammates better is the highest achievement, his individual contribution is less important. If he does not fit into the team, he should leave. This implies that team members of similar rank should have similar pay. Pay and raise criteria should be made public. This results in less gossip.

Scrum sprints

I propose the following sprints for software development:.

1. Implementation sprint (4 weeks): Bugs are fixed, features are implemented. Output is software ready for informal testing, i.e. it can at least be built on the test computer and does not crash immediately when run.
2. Design review & cleanup sprint (1 week): Review the highest priority and most problematic sections of software. Code is refactored according to design review results and clean code principles. Output is documentation of reviewed sections and cleaner code.
3. Test sprint (1 week): Team decides on test scope, i.e. definition of "Done". As a minimum software should be built successfully on a clean hardware. Whole team tests the software. These tests are informal and limited to the Done definition. Tests should demonstrate that software is capable of passing the most important/frequent use cases, i.e. software is still in a useable state. Developers only fix bugs during this sprint, no design update is allowed. Output is a tag/snapshot. When you have automated tests with good coverage, you might not need a manual test sprint, because your software will be automatically tested at least once a day.

After these sprints, do a post mortem (1 day) to assess the efficiency of design, implementation and test sprints. Repeat this process until software is at a user acceptable quality level.

Technical reviews and mentoring

Recently a novice engineer sent a technical document for review, for which I was one of the reviewers. When I opened the document, I saw that it was littered with problems typical of a novice: No proper introduction explaining why this document was created, not specifying assumptions, simplifications and constraints, endless rows of data instead of clear graphs (i.e. signal buried in lots of noise), no clear conclusion, missing references, what still needs to be done, i.e. future work. See Why I gave your paper a Strong Reject

I told him that I will not review it and forwarded the document to a less senior colleague. As a grizzled veteran, I find it untolerably boring to provide basic feedback because I have done it a thousand times before.

How can we make both the novice and the veteran happy:

• Prepare a checklist listing frequent mistakes. Novices prepare their reports with this checklist in mind. My previous blog posts that can be helpful:
• How to write a technical report
• Basic Software Design Report
• Working with a junior developer
• Minimal Software Design
• Modular design
• Stop worrying that your expertise isn't worth much
• How to use engineering standards
• My software development cycle
• Have two levels for review/mentoring: Seniors (more than 2 but less than 10 years of experience) and masters (more than 10 years of experience). A novice/senior submits his work (code/document/presentation) first to a senior for review. Only after the work has been updated with a senior's feedback is it sent to a master. This way, the master is protected from boredom/loosing his faith in the new generation and the novice/senior is protected from humiliation.