Of failures by Philippe Barbe 27 Oct 2021

If you are a data scientist with some experience, you lived through this: the work if not going well. One bug fixed reveals another one. What was seemingly well designed encounters unexpected data points which question the assumptions made.

Some team members are getting nervous. When is this model going to work?

As more issues are being uncovered, this project is spiraling down. We are failing. The team is worrying.

If you never failed, you never lived!

If you have failed the right amount, you are likely someone successful!

Failures are very revealing of people.

Some take it for what it is, a failure, and lock themselves into it. This not a problem if you fail at something you do not really want to do.

Others claim to celebrate them. They simply work on mundane tasks. Failures that cost lives are nothing to celebrate. Failures that put folks on the street with no job are nothing to celebrate either.

On the mundane side, I tried pole vaulting, miserably failed at it, decided it was not for me, and never did it again! I would not mind to do it again though, and would be successful; not because my physical aptitude improved, but because my goal would be less pole vaulting than challenging myself. By shifting the focus, there is no more failure.

The same goes in our professional lives where we may not have the option to drop something, but always have the option to see things differently.

To fail the right amount, we need to approach failure in a different way, as an opportunity not to repeat our mistakes. We should not seek failure, but we should seek a reasoned possibility of failing, that is of tackling problems that are hard enough that we are not sure we can solve them.

A failure is foremost the expression of an error in judgment. We thoughts something would work, yet it does not. We believed in assumptions that do not hold.

Understanding deeply why we failed is a way to find patterns that will make us more successful.

The data points which made an algorithm failing presents an opportunity to reflect on several topics, which I am describing for data science, but are similar in other areas: what are the exceptions we may expect? Should be implement some checks on the input? Do we understand the business logic well? Should the company change some workflow? Do we have the required knowledge? The required skills?

Answering any of these questions opens a path to progress.

Failure is not an option because we rightly chose not to fail. But failure is a possibility which arises if we chose to do something. See it as a dead end to your career, the end of it, and it is what it becomes. See it as an end to some of your ignorance, as an unfortunate chance to get better or to do something else, and here is a possibility to make progress, to evolve.

Maybe if you do not like to understand how things work, failing can be discouraging. However, if you seek understanding, failing may be a gift, albeit a very uncomfortable one sometimes. Read about earlier heart surgeries, and you will realize that risks imply failures, but that they are worth taking for the common good.

Failure relates to change. It forces a change. Perhaps this is why it is hard for some people. But if you embrace motion, movement, evolution, life, you accept failure.

It also tests your willingness. Failing is similar to getting in a ditch… or down a cliff. Getting out of the ditch is not too hard, but climbing a stiff tall cliff can be really difficult. Once you are at the bottom, what can you do? The first things is to realize that from now on, you cannot go further down. Hence you should be optimistic. Look up, resolve, climb!

The more you climbed, the better you became at it. Hence, you are willing to take calculated risks, knowing that you have the resources to likely get you out of the hole. This is key to success.

As a former researcher, I am used to failures. I like to say that for over 20 years I failed most day. The nature of research is to try to solve problems that others could not solve. Everyone who tried failed. Yet, you have some hope and confidence that you can succeed. But because the problem is hard, you will fail to solve it almost every day. Eventually you make progress. However progress is not success, hence you are still failing. The difference between a problem almost solved and a problem solved lies in the adverb ‘almost’ which implies that it is not solved! And suddenly, sometimes in less than a second, you see the path. That less than a second is the only time in which you succeed.

This is something we tend to ignore: success and failure relates to our perception of time. We fail at solving a problem until the problem is solved. We have not done a task, we failed at doing it, until it is completed. Thus, a success is a single point in time, while a failure is a time interval. Therefore, we may as well get used to fail, for this is what we do most of the time, and without failure, there is no success. Failure is the norm, success the exception. What saves us is that most of our failures do not matter that much and we accept them. We may as well acknowledge that fact and keep moving.