Really really enjoyed this essay Chris! You are capturing the experience of transitioning from corporate to creator world so well. This idea of the dance between exploring and exploiting is one I’ve thought about often but haven’t been able to articulate. The bee metaphor resonates. Your journey resonates.
So many gems I wanted to quote, but this was the kicker:
“Creating builds on itself like a snowball. I went from zero creative output, to writing a weekly newsletter, to teaching people how to take action by starting newsletters. No longer was I drifting like a leaf. Instead, I had agency. My value isn’t arbitrarily and callously determined in some executive’s office; my value is determined by the quality of my ideas and effectiveness of my actions. Everyone should know this about themselves!”
Side note. It’s funny how context changes what you’re known for. I can see how you’re the “newsletter guy” but I feel like because of WOP, you’re the “editor” guy to so many people. All that to say I think it adds up to the point you’re making. The definition is always evolving.
Thanks! I've had an internal revulsion at niching for the longest time. I actually had an internal debate over being a jack-of-all-trades when I was younger. It all started making sense when I started following Daniel and hearing about Small Bets.
First of all, “Niches Get Stitches” made me laugh so hard when I saw it in my inbox. 😂
And so awesome to get to read this essay! I’ve been looking forward to it since you told me about it.
I love this: “My field was barren of flowers, yet I remained. Why did the bees’ natural inclination to explore the risky unknown elude me? If bees had made my mistake, their hive would have perished long ago.”
What an interesting point about being in the wrong phase leading a person to be inauthentic. Great read, nice job!!
Thanks for writing this! Resonates with my current feelings of limbo – when I started to write publicly, the Exploit side felt gross to me (touched on it here: https://kzhai.substack.com/p/018-i-literally-dont-know-why-im). However, in avoiding the Exploit side, I swung hard onto the Explore end of the pendulum. As a result, I've been feeling restless after 50+ days into a 100-day writing challenge. Overall, a great synthesis of why I've been feeling torn about how to direct my time and energy!
I was also feeling restless about a week ago but today I actually felt re-energized. I'm treating the whole thing as a "dot collecting" exercise. See the common ideas that come up over the 100 mini-essays and maybe go deeper into the ones that interest me the most.
What I like about the "Do 100 Things" is that is forces action. And it forces you to mutate the project until it's something that you can do 100 times without going crazy. I think the constraint of 100 iterations is a forcing function for you to do an explore/exploit cycle.
Like many comments already said, "Niches Get Stitches" is a great title 👍
I find the argument made by Joel Lehman and Kenneth O. Stanley in Why Greatness Cannot Be Planned: The Myth of the Objective (Why Greatness Cannot henceforth) to be directionally similar as yours and goes into deeper details.
Their book draws examples from multiple fields, art history, inventions history, AI, maze-solving, evolution, etc.
It's interesting that they said "It’s useful to think of achievement as a process of discovery. We can think of painting a masterpiece as essentially discovering it within the set of all possible images."
So at your essay, you contrast exploit vs explore. Within explore, Stanley et al point out you can have another kind of dichotomy: objective-based search vs novelty-based search
## Objective-based search vs novelty-based search
Here's a snippet on comparing objective-based search vs novelty-based search for solving mazes
"In further experiments in the maze domain, we generated hundreds of random mazes of different difficulties and applied both novelty search and objective-based search to solving them. The trend that resulted from the data was that as mazes grow increasingly complicated, both novelty search and objective-based search can fail to solve them. However, objective-based search’s abilities taper off much faster. In other words, the reach of novelty search is greater but not unlimited. This result raises a deep question: What can be done then for the most complicated problems, what approach is left that can consistently solve them?"
There are more interesting points raised by Stanley et al. But, I realize as I was writing this comment, the comment got longer and longer. Maybe I should just do a simple book review instead.
Would love the book review. I'm familiar with Stanley's ideas from a couple of podcasts. I'm especially intrigued of the parallels of his "connecting-the-dots" idea with John Vervaeke's Relevance Realization (h/t Tom Morgan).
What are the definitions of novelty and objective-based search?
Am writing the book review (more like highlights and some commentary) now. Will release on substack. Not sure when.
My cmd+F yields nothing abt "connecting the dots" using the key word "connect" on the actual book.
I noticed something while doing the review
## Explore-Exploit seems to lean towards the abstract and towards comparable payoffs/results
I have re-read the first few chapters while doing the highlights and commentary, and I realized that thinking in terms of explore vs exploit can accidentally trap me in abstract, generalized thinking patterns
Explore vs exploit seems to be the answer to the implicit question of "What general strategy should I choose to maximize my payoff?"
## Why Greatness seems to lean towards concrete results
Whereas Why Greatness seems to steer towards answering a *class* of *concrete* questions such as "what job should I get to become a millionaire?" "what piece of art should i work on to win an award?" etc
When framed as which specific concrete solution to pursue that's awesome (hence the word Great in the title Why Greatness), the book automatically collapses the class of these concrete questions into pure search.
As in imagine a room full of possible jobs that can help you become a millionaire. Or a room full of possible art pieces that you can work on to win an award. And so on.
These "rooms" are what the book calls "search space". And the implicit assumption in the book seems to be that the search space is finite. There's a right one (or at least a best one) amongst the search space of possibilities.
If you continue to hold onto the explore-exploit vocabulary, it _seems_ that Why Greatness is saying, the answer is explore. Because explore is a synonym for search.
I will recommend not to think like that because I have said the framing is completely different (abstract vs concrete).
Also, the book differentiates between novelty search and objective-based search. To cut a story short, neither are perfect, but the former is more likely to lead to _great finds_.
An objective-search is a bit like trying to find your keys at your house. There's a clear mental image of what your keys look like.
Novelty search is different. You have no idea of what exactly you're searching for and trust that you know it when you see it. in the meantime, any stepping stones or intermediate milestones are chosen based on how interesting they seem to you.
If you were asked to define exactly what interesting means, it's ineffable, highly contextual, and highly fluid -- changing from moment to moment depending on circumstances and what you do find.
The final difference between the two is that objective-based search (which also give rise to things like KPI, etc) works better for less complex things such as finding keys, but novelty search works better for more complex things.
Ultimately, the biggest tradeoff the book is asking you to make is if you're looking for something great, you must be willing to give up able to dictate what that specific great solution looks like.
As scientists, the authors are intellectually honest. They acknowledge even novelty search cannot guarantee results.
"Maybe there isn’t really a satisfying answer to this question. The idea that there’s a single cure-all method for success that always gets you where you want to go is so enticing that it absorbs a huge amount of human passion and energy. It’s like the empty quests of past explorers for the fountain of youth. But it just might be that we’ve been looking at the whole issue from the wrong perspective. Maybe you simply can’t always get what you want when you want it. Perhaps there is no magic bullet that can always reach each and every objective that can be imagined. Ultimately, there may be futility at the heart of search, as the next chapter will explore. But even if there is no magic bullet, that doesn’t stop us from finding things that are interesting anyway. Countless treasures are buried along the path to nowhere in particular. We can still dig them out and enjoy them even if we can’t control what they are or when we find them. That’s the real lesson of the interesting and the novel. But to see it most clearly, we need to appreciate the futility that lurks behind all methods of discovery, so that we can finally liberate ourselves from the fantasy of the elusive magic bullet and embrace the reality of the much more powerful treasure hunter."
Right from the jump, 10/10 title!
Really really enjoyed this essay Chris! You are capturing the experience of transitioning from corporate to creator world so well. This idea of the dance between exploring and exploiting is one I’ve thought about often but haven’t been able to articulate. The bee metaphor resonates. Your journey resonates.
So many gems I wanted to quote, but this was the kicker:
“Creating builds on itself like a snowball. I went from zero creative output, to writing a weekly newsletter, to teaching people how to take action by starting newsletters. No longer was I drifting like a leaf. Instead, I had agency. My value isn’t arbitrarily and callously determined in some executive’s office; my value is determined by the quality of my ideas and effectiveness of my actions. Everyone should know this about themselves!”
Side note. It’s funny how context changes what you’re known for. I can see how you’re the “newsletter guy” but I feel like because of WOP, you’re the “editor” guy to so many people. All that to say I think it adds up to the point you’re making. The definition is always evolving.
Great essay!
Thanks! I've had an internal revulsion at niching for the longest time. I actually had an internal debate over being a jack-of-all-trades when I was younger. It all started making sense when I started following Daniel and hearing about Small Bets.
my favorite way to annoy chris is to call him the _____ guy (replace ____ with whatever thing he was doing that week hee hee)
ha.
I'm the "reply guy" now
Chris,
First of all, “Niches Get Stitches” made me laugh so hard when I saw it in my inbox. 😂
And so awesome to get to read this essay! I’ve been looking forward to it since you told me about it.
I love this: “My field was barren of flowers, yet I remained. Why did the bees’ natural inclination to explore the risky unknown elude me? If bees had made my mistake, their hive would have perished long ago.”
What an interesting point about being in the wrong phase leading a person to be inauthentic. Great read, nice job!!
I'm glad you like it! Feel like I was hyping you up to much!
Thanks for writing this! Resonates with my current feelings of limbo – when I started to write publicly, the Exploit side felt gross to me (touched on it here: https://kzhai.substack.com/p/018-i-literally-dont-know-why-im). However, in avoiding the Exploit side, I swung hard onto the Explore end of the pendulum. As a result, I've been feeling restless after 50+ days into a 100-day writing challenge. Overall, a great synthesis of why I've been feeling torn about how to direct my time and energy!
You're doing Jibran's thing too!
I was also feeling restless about a week ago but today I actually felt re-energized. I'm treating the whole thing as a "dot collecting" exercise. See the common ideas that come up over the 100 mini-essays and maybe go deeper into the ones that interest me the most.
What I like about the "Do 100 Things" is that is forces action. And it forces you to mutate the project until it's something that you can do 100 times without going crazy. I think the constraint of 100 iterations is a forcing function for you to do an explore/exploit cycle.
For sure! I have ideas for a next iteration but I have a few drafts that I want to publish first because I think they'll be helpful for some people!
new shiny dime here!
Like many comments already said, "Niches Get Stitches" is a great title 👍
I find the argument made by Joel Lehman and Kenneth O. Stanley in Why Greatness Cannot Be Planned: The Myth of the Objective (Why Greatness Cannot henceforth) to be directionally similar as yours and goes into deeper details.
Their book draws examples from multiple fields, art history, inventions history, AI, maze-solving, evolution, etc.
It's interesting that they said "It’s useful to think of achievement as a process of discovery. We can think of painting a masterpiece as essentially discovering it within the set of all possible images."
So at your essay, you contrast exploit vs explore. Within explore, Stanley et al point out you can have another kind of dichotomy: objective-based search vs novelty-based search
## Objective-based search vs novelty-based search
Here's a snippet on comparing objective-based search vs novelty-based search for solving mazes
"In further experiments in the maze domain, we generated hundreds of random mazes of different difficulties and applied both novelty search and objective-based search to solving them. The trend that resulted from the data was that as mazes grow increasingly complicated, both novelty search and objective-based search can fail to solve them. However, objective-based search’s abilities taper off much faster. In other words, the reach of novelty search is greater but not unlimited. This result raises a deep question: What can be done then for the most complicated problems, what approach is left that can consistently solve them?"
There are more interesting points raised by Stanley et al. But, I realize as I was writing this comment, the comment got longer and longer. Maybe I should just do a simple book review instead.
Would love the book review. I'm familiar with Stanley's ideas from a couple of podcasts. I'm especially intrigued of the parallels of his "connecting-the-dots" idea with John Vervaeke's Relevance Realization (h/t Tom Morgan).
What are the definitions of novelty and objective-based search?
Am writing the book review (more like highlights and some commentary) now. Will release on substack. Not sure when.
My cmd+F yields nothing abt "connecting the dots" using the key word "connect" on the actual book.
I noticed something while doing the review
## Explore-Exploit seems to lean towards the abstract and towards comparable payoffs/results
I have re-read the first few chapters while doing the highlights and commentary, and I realized that thinking in terms of explore vs exploit can accidentally trap me in abstract, generalized thinking patterns
Explore vs exploit seems to be the answer to the implicit question of "What general strategy should I choose to maximize my payoff?"
Also when framed as explore-exploit as per your example at https://towardsdatascience.com/the-exploration-exploitation-dilemma-f5622fbe1e82 you're forced to be highly narrow on what the payoff looks like in order to meaningfully compare explore-vs-exploit tradeoffs.
In life the difficult choices are not apples to apples comparisons. As what Ruth Chang says, hard choices are those that cannot be easily compared against each other https://www.ted.com/talks/ruth_chang_how_to_make_hard_choices?language=en
## Why Greatness seems to lean towards concrete results
Whereas Why Greatness seems to steer towards answering a *class* of *concrete* questions such as "what job should I get to become a millionaire?" "what piece of art should i work on to win an award?" etc
When framed as which specific concrete solution to pursue that's awesome (hence the word Great in the title Why Greatness), the book automatically collapses the class of these concrete questions into pure search.
As in imagine a room full of possible jobs that can help you become a millionaire. Or a room full of possible art pieces that you can work on to win an award. And so on.
These "rooms" are what the book calls "search space". And the implicit assumption in the book seems to be that the search space is finite. There's a right one (or at least a best one) amongst the search space of possibilities.
If you continue to hold onto the explore-exploit vocabulary, it _seems_ that Why Greatness is saying, the answer is explore. Because explore is a synonym for search.
I will recommend not to think like that because I have said the framing is completely different (abstract vs concrete).
Also, the book differentiates between novelty search and objective-based search. To cut a story short, neither are perfect, but the former is more likely to lead to _great finds_.
An objective-search is a bit like trying to find your keys at your house. There's a clear mental image of what your keys look like.
Novelty search is different. You have no idea of what exactly you're searching for and trust that you know it when you see it. in the meantime, any stepping stones or intermediate milestones are chosen based on how interesting they seem to you.
If you were asked to define exactly what interesting means, it's ineffable, highly contextual, and highly fluid -- changing from moment to moment depending on circumstances and what you do find.
The final difference between the two is that objective-based search (which also give rise to things like KPI, etc) works better for less complex things such as finding keys, but novelty search works better for more complex things.
Ultimately, the biggest tradeoff the book is asking you to make is if you're looking for something great, you must be willing to give up able to dictate what that specific great solution looks like.
As scientists, the authors are intellectually honest. They acknowledge even novelty search cannot guarantee results.
"Maybe there isn’t really a satisfying answer to this question. The idea that there’s a single cure-all method for success that always gets you where you want to go is so enticing that it absorbs a huge amount of human passion and energy. It’s like the empty quests of past explorers for the fountain of youth. But it just might be that we’ve been looking at the whole issue from the wrong perspective. Maybe you simply can’t always get what you want when you want it. Perhaps there is no magic bullet that can always reach each and every objective that can be imagined. Ultimately, there may be futility at the heart of search, as the next chapter will explore. But even if there is no magic bullet, that doesn’t stop us from finding things that are interesting anyway. Countless treasures are buried along the path to nowhere in particular. We can still dig them out and enjoy them even if we can’t control what they are or when we find them. That’s the real lesson of the interesting and the novel. But to see it most clearly, we need to appreciate the futility that lurks behind all methods of discovery, so that we can finally liberate ourselves from the fantasy of the elusive magic bullet and embrace the reality of the much more powerful treasure hunter."
I heard about his "connecting the dots" idea from his podcast on Invest Like the Best
He actually doesn't use that term, but he talks about it at 14:07 (search in the transcript for the timestamp).
https://www.joincolossus.com/episodes/88083127/stanley-greatness-without-goals?tab=transcript
I signed up to take a look. At min 14 the one about picbreeder is indeed in the book
in trying to summarize from the book, I was forced to re-read the book and the more I re-read, the more nuances I found. hahaha