The Art of Recursive Learning

Have you ever wondered how you learn?  Contrary to formal education, learning is not rote memorization.  You can’t learn to swim by reading a book, you need to actually do it.  The process of learning is through a feedback loop.  You do something, you see what happens, you adjust what you did until what you want to happen happens.  As you get better, this loop occurs faster and it feels easier.  This is when things “slow down” or you’re “in the zone”.  

Deliberate practice is the best way to learn.  It takes advantage of this flywheel effect, but there are essential requirements that are rare in real life.  You need a coach, formulated pedagogy, and you need a "kind" domain.  

(h/t Cedric Chin)

Psychologist Robin Hogarth defined kind domains as domains where “patterns repeat over and over, and feedback is extremely accurate and usually very rapid.”  An easy example to understand is golf.  There are also clear “rules and within defined boundaries, a consequence is quickly apparent, and similar challenges occur repeatedly.”  Within a few seconds, you can see if the ball is in the fairway or in the trees.  These characteristics make feedback loops incredibly effective in kind domains.  

In contrast, Hogarth defines “wicked” domains as the opposite.  Patterns may not exist or may not be obvious.  An example is business strategy.  Even in retrospect, it’s hard to attribute company success to certain initiatives or even figure out how much is luck.  Feedback is often delayed, inaccurate, or both in wicked domains.  Feedback loops don’t work in wicked domains for these reasons.

Is it possible to get the effects of deliberate practice without the preconditions?  In other words, how do we learn in domains where feedback is slow, noisy, and unreliable?

There’s a skill in reframing wicked domains so that the feedback is faster and clearer.  Beginners can’t recognize feedback loops.  All domains are wicked to them.  They don’t know which factor is causing the result.  For example, even though golf is a kind domain, a beginner won’t know what is causing his shot to land in the trees.  Is it his foot placement?  His wrists?  The beginner won’t know what to pay attention to.  The speed and quality of the signal depends not only on the environment but also on the quality of the receiver.  There's a skill in finding and redefining feedback loops.

Learning becomes a superpower when you realize that learning is the skill of improving feedback loops in order to gain intuition about a domain.  It’s the art of filtering out the unimportant feedback and making use of the important feedback.  Learning is transforming wicked domains into kind domains.

Many things in life occur in wicked domains. Relationships, writing, most job functions.  In fact, any topic or area past a certain level of expertise will be in a wicked domain.  Through the examples of Brian Lui, Naturalistic Decision Making (NDM), and Amazon, we can see examples of techniques to tame wicked domains.

Spray and Pray

Brian Lui was an equity analyst at a value fund. Investing is a classic example of a wicked domain.  The price of each stock can fluctuate without connection to the underlying valuation.  It could take months or years for your investment thesis to come true and you won't know if the price changed for that reason or for another.

Brian's solution, which defines his style of investing, is to slow down feedback loops.  Usually, faster and more frequent feedback loops are desired because each loop is a learning experience.  But feedback loops in wicked domains can be counterproductive.  Due to the noise, you can get stuck in a local maxima - a new technique may give you better results in the long term but your results may take a step back in the short term while you learn and incorporate it.  

So how does he slow down feedback loops? Brian creates, in parallel, different theories or models of how an investment will work, each with different metrics or causations.  Over time many of these will be proven false.  Even if there’s multiple surviving explanations, he’s cut down on different possibilities.  The noisy data of wicked domains is good for this method because falsification needs less examples than to prove something true.  To be confident of something as true you need multiple instances of strong evidence.  To prove something is wrong, you only need one counterexample.  

Brian’s had a successful track record in investing, but his method requires a large amount of experience in the area.  It's an example of tacit knowledge.  Tacit knowledge is knowledge that can't be explicitly transferred.  You can read this description of his method, but you won't be able to immediately put it to practice.  A beginner wouldn’t have the background knowledge or creativity to generate a wide enough range of models that would likely include the most actionable one, nor be able to confidently eliminate models.  A beginner wouldn’t have the equanimity to have a longer time horizon.  A beginner wouldn’t have the resources or political capital to hold the course until the strategy worked.

Is there a way to learn his tacit knowledge?

Experts as a Model

Traditionally, tacit knowledge has been taught through an apprenticeship or deliberate practice.  The first takes years of mimicry until the skills become internalized and the second takes years and requires a well-developed pedagogical method.  What if you have access to an expert, but are unable to extract the tacit knowledge?  The field of Naturalistic Decision Making developed a technique called ACTA to solve this problem.

ACTA is a process to extract and convert knowledge into training programs, and it involves interviews and simulations to understand the difficulties, errors, cues and strategies of a challenging domain.  You can apply this method to anything, for example, a tennis serve.

To return a tennis serve, you must identify the type of serve - spin, location, speed.  While you can identify a serve easily after the fact, in play, you must identify quickly enough to give yourself time to get into position and return the serve.  It takes about half a second for the ball to reach you and human reaction time is about half of that.  That means you have a quarter of a second to recognize the type of serve and position yourself accordingly.  Professional players have an intuition that allows them to return serves.  Could this be taught?

Through interviewing players, researchers quickly learned that players couldn't articulate their technique:

The experts can’t necessarily see what they’re seeing — they almost put it in the ‘ESP category’, like with the firefighters in (Gary) Klein’s story (in Sources of Power). And if you press them enough, they’ll start making stuff up. The way that experts do, because they want to give you an answer. So you really need to start ascertaining where that is.

So the researchers experimented. From the interviews, they knew that when the serve is identified is important.  But players couldn’t accurately describe when or how they identified a serve.  They devised an experiment.  The researchers showed videos of servers frame by frame and asked players to identify the type of serve.  Novice players needed to see 50 milliseconds after contact to identify the server, while expert players could identify the serve if the video cut off 50 milliseconds before contact!  

From this data, the researchers created a training program.  They started showing videos that showed one less frame than a novice player needed.  When the novice player could identify in that time frame, they removed another frame and continued until the novice players only needed as much time as an expert.

The researchers also wanted to know what the expert players were looking at.  They blacked out different variables in the video (racquet, lower body, etc) to see which were being used.  Novices were looking at racquets while experts were looking at lower bodies.

ACTA demonstrates that experts can be in a kind domain while a novice is in a wicked domain for the same skill.  What’s kind for experts is wicked for novices.  If you can extract knowledge from an expert and teach a novice what to look for with training techniques, the domain becomes kinder for the novice.  The novice will know what specific areas to focus on, will know what to look for and how to react.  The novice can now use feedback loops while previously they were floundering.

But what happens when there are no experts in a domain? What if you are pushing past the frontier of the experts?  How can you learn new knowledge?

Defining Systems

Business strategy is a wicked domain.  It can take months or years to see if a business strategy works.  There are innumerable variables, and it can be impossible to know which one was the actual driving factor.  The strategy may have been right, customer preferences could have changed, competitors may have made a wrong choice, or you may never understand what actually happened (blind luck!).

Most dominant companies slowly lose ground to competitors before being completely disrupted.  Some manage to pivot to new markets or strategies.  There’s very few companies that have remained dominant in the original market.  There are even fewer that have pivoted to becoming the market leader in another business line.  Amazon may be the only company that has done both, and they also invented wholly new product lines and businesses.  

Amazon started as an online bookstore and not only expanded its offerings, but has the largest market share in online retail by far.  They became market leaders in shipping, payments and logistics, and smart home devices.  Amazon has created new business categories in cloud infrastructure, and fulfillment.  While Amazon has had failures (Fire Phone), they realized it quickly and cut their losses, recognizing sunk costs.  What sets Amazon apart?

Amazon navigates the business ecosystem faster and clearer than their competitors.  They do this by deliberately creating and refining feedback loops.  Amazon formalizes the process of correlating the source of action with what actually happens by defining the controllable input and output metrics of every task.  For example if they want to measure output, what is the best way to do so?  Is it sales?  If so, what variables are the biggest drivers of sales?  Is it the number of page views?  Percentage of page view of products in stock?  There's innumerable variables that could have an effect.  Amazon constantly reviews how accurate these metrics are and how related they are to the desired outcome.  They simultaneously evaluate the output metric.  Are sales the correct way to measure output?

https://twitter.com/david_perell/status/1397565559236173832

(h/t David Perell)

This is institutionalized in a methodology called DMAIC (Define, Measure, Analyze, Improve, Control).  Without getting into the weeds, DMAIC recursively evaluates controllable input metrics, output metrics, and the explanatory power between them.  Amazon has set up a feedback loop to evaluate and improve feedback loops!  While in general, business is noisy with no standard time intervals, Amazon has created a system of constant evaluation within a consistent framework.  Through systems, Amazon has pacified the wild business world into a kinder domain where they can use feedback loops to learn and dominate.

The Art of Recursive Learning

Learning in wicked domains is hard, but it’s possible.  The trick is to make those domains kinder by figuring out the cause and effect between specific events and building feedback loops.  While feedback loops are commonly thought to not work in wicked domains, by redefining the constraints of the domain, wicked domains can become kinder.

Put simply, in order to learn you need to identify both specific actions and outcomes and the relationship between them.  When you learn, what you're learning is the effect of any action that you do.  If you understand that this is the underlying technique of learning, you'll be able to learn more effectively.  We’ve seen three examples of taming wicked domains:

• Use background knowledge to generate many parallel models, and over-time, see which ones succeed and fail (Brian Lui)

• Use expert models to practice analytically, and use simulations to gradually increase the difficulty (ACTA)

• Use a system that recursively redefines itself.  Set intervals, check in every X weeks and refine your model (Amazon)

The common theme is that your results need to inform your next action.  The key is not “doing something better,” the key is “how do I know if I did something right?”

Learning isn’t memorizing, it’s recursive.