RSS

What is a Black Swan

02 Sep

A.  What is a Black Swan?

Medieval Europeans had seen only white swans. In fact, any impossible event was in those days termed a ‘black swan’. Therefore, when the first settlers reached Australia , they were shocked to find flocks of black swans all over the place! The sighting of the first black swan might have been an interesting surprise for ornithologists and others concerned with the color of birds, but that is not where the significance of the story lies. It exposes the severe limitation of the empirical learning methods and therefore the fragility of our knowledge based on instances rather on generalities.

The term Black Swan, was employed by generations of Europeans as a metaphor to signify an absurdity or to denote something that did not exist. It however took one unexpected sighting of a Black Swan to invalidate the belief of the generations that all swans in the world were white and that a Black Swan was an absurdity.

David Hume (1711-1776), the English philosopher, said: the observation of even a million white swans does not justify the statement “all swans are white.” There is no way to know that somewhere out there a black swan is not hiding, disproving the rule and nullifying our “knowledge” of swans. John Stuart Mill (1806-1873), another English philosopher, used the black swan as an example to illustrate the problem of induction in the philosophy of science according to which no amount of empirical evidence in support of a proposition can ever prove conclusively that it is true.

Later , Karl Popper (1902-1994), Austrian and English philosopher used the black swan narrative to discuss falsification and said “No matter how many times the results of experiments agree with some theory, you can never be sure that the next time the result will not contradict the theory “. Bertrand Russell (1872-1970) and Stephen Hawkins the celebrated English thinkers both believed “the general principles of science . . . are believed because mankind has found innumerable instances of their truth and no instances of their falsehood. But this affords no evidence for their truth in the future, unless the inductive principle is assumed”. In other words, one single observation can invalidate a general statement derived from millennia of confirmatory sightings of millions of white swans. All you need is one single (ugly) black bird.

A “Black Swan” is therefore anything that seems to us, based on our limited experience, to be impossible The Black principle thus implied:

a) Absence of proof does not necessarily mean proof of absence ;
b) A perceived impossibility can actually occur.;
c) Do not confuse improbability with impossibility. and
d) Expect an unexpected to happen.

*********

B.  Impact of the highly improbable

Nassim Nicholas Taleb, in his recent book The Black Swan: The Impact of the Highly Improbable (Penguin/Allen Lane-2007) expands on the idea of Black Swan and spins a web of ideas around randomness, uncertainty, our understanding of history, financial markets, illusion of “expert advice”, precaution- fire fighting, our perception of “heroes” and a number of other issues. The central idea of this book concerns our blindness to randomness, particularly the large deviations. It talks about skepticism, wild randomness and the power of stories. It is about how not to be a sucker to the unexpected and the unknown. It is in essence a Book of Ideas.

Nassim Nicholas Taleb has a more specific definition of the Black Swan event —for him, a “black swan” is an event that meets three conditions:

a) It was unpredictable;
b) It carried an extreme impact; and
c) After the event, we made it appear more predictable than it actually was, by concocting explanations.

A “Black Swan” is therefore anything that seems to us, based on our limited experience, to be impossible and which are only retrospectively predictable. Black swans can be positive or negative, a blockbuster book or a stock market crash. He cites the instances of 9/11, WW I and other events and explores how “black swans” throughout history have influenced civilizations, religions, and governments.

For instance, the astonishing success of Google was a black swan; so was the Harry Potter.

Black swans may occur almost anywhere, say from the rise of religions to events in our own personal lives. In short, black swans are some of the most important and influential events in the history of the human race. It started accelerating during the industrial revolution, as the world started getting more complicated, while ordinary events, the ones we study, discuss, and try to predict from reading the newspapers, have become increasingly inconsequential.

Life is the cumulative effect of a handful of significant shocks. A small number of Black Swans explain almost everything in our world, from the success of ideas and religions, to the dynamics of historical events, to elements of our own personal lives. Go through the following exercise.

Look into your own existence. Count the significant events, the technological changes and the inventions that have taken place in our environment since you were born; and compare them to what was there before their advent. How many of them came on a schedule? Look into your own personal life, to your choice of profession, say, or meeting your mate, the betrayals you faced, your sudden enrichment or impoverishment. How often did these things occur according to plan?

Extending this logic, many of the discoveries that have had tremendous impact on our living were accidents in the sense that they were discovered while searching for something else. Because of hindsight bias, histories of scientific discoveries are written with straightforward story lines telling that someone set out to do something and succeeded and that it was all about intention and design. However, in truth, “most of what people were looking for, they did not find. Most of what they found they were not looking for.” For instance, Viagra was devised to treat heart disease and high blood pressure. ; Lasers at first had no application but were useful as a form of radar; and the Internet was conceived as a military network

What makes the black swans particularly dangerous is that most of the times they are unexpected. He explains this by saying: “Consider the turkey that is fed every day. Every single feeding will firm up the bird’s belief that it is the general rule of life to be fed every day by friendly members of the human race “looking out for its best interests”. On the afternoon of the Wednesday before Thanksgiving, something unexpected will happen to the turkey. It will incur a revision of belief.”

C. Why Black Swans happen?

Why do Black Swans happen? It is because, Taleb says, we do tend to confuse improbability with impossibility. This inbuilt tendency makes us over-confident about predicting the future; and over-confidence can lead to disaster. Further, we fool ourselves into thinking we know more than we actually do. We restrict our thinking to the irrelevant and inconsequential, while large events continue to surprise us and shape our world.

The problem of induction tells us that we cannot really learn from our experiences. It makes knowledge very problematic, if not impossible. Yet, we have too much faith that past events will repeat and we can take care of them. We therefore concentrate on things we already know and repeatedly fail to take into consideration what we do not know. Black Swan logic makes what you do not know far more relevant than what you do know. Unfortunately, we do not know how much we do not know.

Largely, says Taleb, human beings are blind to the impact of randomness and fail to appreciate the asymmetry in our perception of events. In addition, we confidently predict the future in any number of ways. Unfortunately, we are nearly always wrong, though later, when things have happened, we tend to remember only the times when we were right. We attribute our successes to our skills, and our failures to external events outside our control, namely to randomness

Comforting simplifications make us over-confident. Uncertainty makes us curious and cautious. We often hear only what we love to hear. For example, when a doctor says, “there is no evidence of cancer” we often misinterpret that as “there is evidence of no cancer”; obviously, there is a huge difference between the two, but the latter is impossible to state with 100% certainty.

Another related human impediment is that we tend to learn the precise, not the general. “We’re suckers for a narrative,” he says.” When a story is attached to an event, it seems more probable than it actually is, causing us to err on the conservative side for negative events and on the risky side for positive ones.”

Not all Black Swans are negative, and not all are sudden. From one perspective, all modern history is one huge, slowly unfolding Black Swan. This is reflected in the ever-changing stories we live in. Stories of the earlier generations used to be more stable, their children saw the same world they did. Now the stories are as short-lived as a summer blockbuster. Still we believe in them, in the next as sincerely as in the last.

Most of the really big events in our world are rare and unpredictable, and thus trying to extract generalizable stories to explain them may be emotionally satisfying, but it is practically useless. It results in what Taleb calls the “retrospective distortion”. September 11th is one such example and stock market crashes are another. As he puts it, “History does not crawl, it jumps.”

Taleb insists that our brain is “the wrong user’s manual” for the complex unpredictability of the world we are now living in. He says, “human nature is not programmed for black swans”; “our inferential machinery is not made for a complicated environment”; our “statistical intuitions have not evolved” for our current habitat; we are “not well adapted to the present, post-alphabet, intensely informational, and statistically complex environment.”

I am not convinced of this, if Taleb were to be entirely correct how could the humankind survive and multiply successfully all these ages. In addition, I think Taleb’s argument does not give due credit to the human ingenuity and unconscious, which can handle multiplicity just well.

D. Experts and “Empty Suits”

Taleb is harsh on Experts dispensing advice and handing down oracle like predictions. He is particularly harsh on Economists when it comes to prediction of market events; and on Historians for the manner of their explaining the past events.

As in the good old medieval days, “experts” are many times empty heads with empty (and expensive) suits. Certain professionals, while believing they are experts, are in fact not. Based on their empirical record, they do not know more about their subject matter than the general population, but they are much better at narrating — or, worse, at smoking you with complicated mathematical models. They are also more likely to wear a tie.

Taleb tends to regard economists as not just the blind leading the blind, but leading them perilously close to the cliff’s edge. “Our cumulative prediction errors for political and economic events are so monstrous,” says Taleb, ‘that every time I look at the empirical record I have to pinch myself to verify that I am not dreaming”.

For instance, he states, in 1970, the US government’s official view was that, by 1980, the price of foreign crude oil might well decline, and, in any case, would not show a substantial increase. In fact, oil prices went up tenfold by 1980.The inability to predict outliers implies the inability to predict the course of history, given the share of these events in the dynamics of the economics .

Today our paid experts wear knowledge on their sleeve while they sleepwalk into the future of unpredictability. Today, all knowledge is available to us, or so it seems. We even teach our kids to go to Google or Wikipedia, but, unlike a library, the browser hides the trillions of bits of information we can never know. The blogosphere is, at its worst, a Tower of Babel . Our confidence grows, but the complexity of our environment has expanded exponentially.

Later in that same chapter, Taleb writes : It is often said, “Wise is he who can see things coming.” Perhaps the wise one is the one who knows that he cannot see things far away.

As regards History and its lessons, Taleb writes eloquently .

What did people learn from the 9/11 episode? Did they learn that some events, owing to their dynamics, stand largely outside the realm of the predictable? No. Did they learn the built-in defect of conventional wisdom? No. What did they figure out? They learned precise rules for avoiding Islamic proto -terrorists and tall buildings.

Taleb writes, and I agree, that USA took the wrong lessons from the September 11 terrorist attacks. It learned how to avoid another 9/11 like attack. It learned to fear the specific causes of that attack. What USA should have learned was to trust the stories less but to pay heed to the causes that prompted the attack. We all learned that 20th century history culminated in Osama bin Laden. What we should have learned was that history does not march, it jumps.

Good historians know that, but the bad ones tell better stories; and their good sounding false stories provide a canvas for the news media productions. In the hands of journalists, pundits and media experts, history becomes a mythology – a collection of moral lessons for our time. In reality, history is wildly random. While it happens, it is shocking, insane and painful. The stories come later, to heal the trauma, and make-believe a sane world.

Taleb writes, “as I formulated my ideas on the perception of random events, I developed the governing impression that our minds are wonderful explanation machines, capable of making sense out of almost anything, capable of mounting explanations for all manner of phenomena, and generally incapable of accepting the idea of unpredictability. These events were unexplainable, but intelligent people thought they were capable of providing convincing explanations for them – after the fact. Further, the more intelligent the person, the better sounding the explanation .”

To summarize: the extreme, the unknown, and the very improbable (improbable according our current knowledge) dominate our world and all the while, we spend our time engaged in small talk, focusing on the known, and the repeated. This implies the need to use the extreme event as a starting point and not treat it as an exception to be pushed under the rug. Further, in spite of our progress and the growth in knowledge, or perhaps because of such progress and growth, the future will be increasingly less predictable, while both human nature and social science seem to conspire to hide the idea from us.

E. How do we deal with Black Swan

You cannot predict a Black Swan. If you did succeed in doing so, it is then no longer a Black swan. Black Swans being unpredictable, we need to adjust to their existence (rather than naïvely try to predict them). There are ways to manage that uncertainty to our advantage, but it requires an understanding of basic psychology and to look for generalities. We can do so many things if we focus on anti knowledge, or what we do not know.

Taleb admits that the Black Swan too is a story. It is, however, a better story. We can only understand the world through stories, there is no way to change that. What we need are good stories to fight the bad ones. Stories that prepare us for the unpredictable encourage us to doubt what we know. As Taleb writes, you cannot avoid crossing the street, but you can try not to do it blindfolded. You can try not to be a sucker .

Taleb’s book is also full of advice about how to benefit from the unpredictable nature of the world: “focus makes you a sucker; it translates into prediction problems”; learn to “avoid ‘tunneling'” (“the neglect of sources of uncertainty outside the plan itself”); “train yourself to spot the difference between the sensational and the empirical”; and remember that “we are not natural skeptics,” that “not believing” requires an “expenditure of mental effort.”

At the same time, there is a generous dose of skepticism in Taleb’s work. . It is the view of one who tries to keep his mind in a state of suspended judgment, neither agreeing nor disagreeing, staying open to contrary facts and viewpoints. A Skeptic believes in reality, but not in human reasoning. A Skeptic laughs at certainty and feels most comfortable as a Devil’s Advocate. The true Skeptic is, however, an unattainable ideal. Therefore, we must do our best with what we have. Taleb’s advice is to pick your battles. Skepticism is hard work, so reserve it for the beliefs and choices that really matter. Be practical.

Taleb seems to suggest:

*Since we cannot control unpredictable events we should accept uncertainty and seek to maximize our exposure to Serendipity, as by putting ourselves in the way of new ideas.

*Since there is such danger in accepting conclusions based on too little information simply because they confirm our beliefs, we should try to remain aware in the present of what we are doing, paying attention to what actually happens and refraining as far as possible from imposing theories on our experience.

*We should recognize our poor record as a species in predicting the future, that we are much better at doing than knowing. Some things are more predictable than others are; we are safe enough in expecting tomorrow’s sunrise to plan on breakfast. We can start noticing which situations are most susceptible to black swans, and when we encounter them, remember how little we truly know so our ignorance does not lead us around by the nose

F. New terms coined by Taleb

Taleb has coined a few terms which I suspect may gain currency just as the hybrid terms E-mail, blog etc. I picked up a couple of them:

Extremistan : A province where a single event can have a huge impact. This is where chaos reigns, the wholly unexpected happens, power laws and fractal geometry apply and the events here are beyond the range of predictability.

Mediocristan : A province where a single observation does not affect the aggregate. A place ruled by mild forms of randomness .This falls within the area where the events are predictable to a certain extent.

We assume the entire world is “Mediocristan”, whereas in reality large bands of it are “Extremistan”. The problem is there is no clear demarcation of the areas of Extremistan and Mediocristan.

One can never tell whether one is in the relatively safe territory of Mediocristan or if one has wandered into the lawless regions of Extremistan. This means the events in either case are not predictable.

Taleb’s hypothesis is that as a result of globalization and the speed of electronic communications, the world is becoming more like Extremistan and less like Mediocristan.

Platonicity: This stands for our tendency to mistake the map for the country, the finger for the moon. It makes us think that we understand more than we actually do.

The other terms are the “lucid fallacy” or “uncertainty of the nerd” (basing studies of chance on games and dice); and GIF (the “Great Intellectual Fraud” that is the bell curve- where the data falls between the mean and standard deviation).

G .Prevention and cure

The wisdom of the ages preaches that prevention is better than cure; but rarely do we treat it as a better option.

“Everybody knows that you need more prevention than treatment, but few reward acts of prevention. We glorify those who left their names in history books at the expense of those contributors about whom our books are silent. We are not just a superficial race (this may be curable to some extent); we are a very unfair one

In other words, we glorify the heart surgeon who performs a bypass surgery but scarcely take notice of the dietician or a physiotherapist who strives to keep his charge healthy and fit. We make a hero of a general who wins war but neglect a diplomat who strives to maintain peace and prevent a war.

H. Flaws in the book

The book suffers from a few flaws, unfortunately. I wish Taleb used the services and skills of a professional editor. Some ideas and phrases are highly repetitive. It gets too technical at places.

Taleb carries on a personal battle with certain economists and the French in general through repeated jabs and diatribes. It looks rather unsavory. I wish he avoided bringing in his personal prejudices into a work of this nature.

There is a lot of needless sarcasm and bad humor directed against certain trades or professions like Dentists, bakers etc. that he labels as boring and non-scalable careers.

He carries the unpredictability principle to the extreme, saying that we can predict almost nothing about the future.

Many of Taleb’s ideas accord with those that are repeatedly explored, often advocated, on Serendip: “doubt everything,” “fight against dogma,” “shed the idea of full predictability,” and know that “you can benefit from it.” Most strikingly: “maximize the serendipity around you.”

http://serendip.brynmawr.edu/index.html

I am not convinced of Taleb’s argument about our brain being “the wrong user’s manual” etc. He does not give credit to the human ingenuity and the unconscious, which can handle multiplicity very well.

If you accept Taleb’s position, we are left with an uncomfortable question: How do we function in a world where accurate prediction is rarely possible, where history is not a reliable guide to the future and where the most important events cannot be anticipated?

Taleb suggest some answers. I am not convinced by most of them

***********

Despite the flaws, the main ideas of the book are important and merit reading and great consideration. It is not surprising The Black Swan: The Impact of the Highly Improbable, is it self a Black Swan.

Source:

http://www.fooledbyrandomness.com/

http://arlenegoldbard.com/2007/05/09/my-new-crush/

 
7 Comments

Posted by on September 2, 2012 in Books

 

Tags: ,

7 responses to “What is a Black Swan

  1. sreenivasaraos

    March 21, 2015 at 12:43 pm

    srinivasarao

    extremely fascinating. will have to read more to discuss with you. does he use statistics explicitly anywhere? where does a statement like ‘ sun may rise in the west tomorrow’ fit in ? one can only say that int the history of mankind (n number of days of record keeping), it has risen in the east.

    ..
    also i found your comment on jivaka and massage very interesting. i did look up some references. incidentally i have put them all in a new blog called vignettes of thailand. pl see it when you have time.

    regards

    Palahali

     
    • sreenivasaraos

      March 21, 2015 at 12:45 pm

      dear palahali,

      i am glad you read the post and found it fascinating.

      there are plenty of mathematical/statistical and economic/financial aspects to taleb’s ideas; the black swan does get very technical at times.

      not willing to fall in the same trap, i avoided using technical terms in the blog. even then, most will find it boring.

      a.

      taleb’s earlier book was on randomness. his new book appears to be an extension of that. black swan event is a statistical outlier, a data point that falls outside the bell curve. the arguments in the book are built around bell-curve and the power-law. the hypothesis of the black swan rests on that the most important events may occur so rarely that existing data points can never truly assure us that the future will not look very different from the present. we never see them coming, but we have no trouble concocting post hoc explanations for why they should have been obvious.

      we can be fairly certain that we will never meet anyone 14-feet tall, but it is entirely possible that, over time, we will hear of a man twice as rich as bill gates or witness a market crash twice as devastating as that of october 1987.

      the problem, is that most of the time we are in the land of the power law and do not know it.our strategies for managing risk are likely to fail at the worst possible time because they are generally (and mistakenly) based on bell-curve assumptions.

      taleb is fascinated by the rare but pivotal events that characterize life in the power-law world. he calls them black swans, after the philosopher karl popper’s observation that only a single black swan is required to falsify the theory that “all swans are white” even when there are thousands of white swans in evidence. the ideas of mediocristan and extremistan are built around this.

      if you accept taleb’s position, we are left with an uncomfortable question: how do we function in a world where accurate prediction is rarely possible, where history is not a reliable guide to the future and where the most important events cannot be anticipated?

      taleb suggest some answers. i am not convinced by most of them.

      i f you are into financial, capital markets, casino, economics etc. you will find the book interesting. past performance does not guarantee future returns is the loud and clear message to the market hawks.

      b.

      the sunrise you mentioned is an argument put forth by karl popper to illustrate the philosophical problem of induction. he said: just because the sun has risen every day for as long as anyone can remember, does not mean that there is any rational reason to believe it will rise tomorrow. there is no rational way to prove that a pattern will continue in the future just because it has in the past. popper states that while there is no way to prove that the sun will come up, we can theorize that it will. if it does not come up, then it will be disproved, but since at this moment in time it seems to be consistent with our theory, the theory is not disproved. this is “the absence of proof does not imply proof of absence” principle.

      there is a similar illustration of a man facing a lion.

      Bertrand Russell and hawkins’ position i briefly mentioned is also on these lines.

      i have not discussed, in detail, any of these because the blog was getting lengthier and i was trying to focus on taleb’s book.

      thanks for asking.

      please keep talking.

      regards

      ps: i am glad the links and jivaka were of some use.

       
  2. sreenivasaraos

    March 21, 2015 at 12:46 pm

    dear rao sir

    a very good, educative and fascinating blog.

    absence of proof does not necessarily mean proof of absence

    this essentially summarizes the theory of black swan.

    wisdom would be in acknowledging the fact the extreme and the improbable-not yet corroborated by empirical evidence-can also exist. keeping an open mind, rather than being dogmatic, is the key for survival of mankind. differentiating between knowledge and wisdom is important. mere knowledge, without the wisdom to guide in the proper utilization of said knowledge, would be futile. a true scientific temper would be an ideal blend of humility (in acknowledging the unknown) and initiative (in unravelling the unknown) . and a true scientific temper is what is needed today.

    regards

    Melody Queen

     
  3. sreenivasaraos

    March 21, 2015 at 12:49 pm

    shri rao,

    this was a very nice review of the book. throughout the reading, i kept getting many doubts in my mind but you summarised most of them in your personal observations in the end.

    this is something i learnt specially here:

    “taleb is harsh on experts dispensing advice and handing down oracle like predictions. he is particularly harsh on economists when it comes to prediction of market events; and on historians for the manner of their explaining the past events…

    taleb tends to regard economists as not just the blind leading the blind, but leading them perilously close to the cliff’s edge.”

    to one that is pathetic at economics, this sounds correct too…please see my latest blog “swadeshi economics” and the comments that follow it to know where i stand.

    my take (in my words, for my satisfaction) is this:

    every event has a set of factors (fixed or variable) working behind it. however, it is not possible to determine all the factors all the times. it is possible with a lot of experience, knowledge and wisdom to do it to a great extent, yet sometimes it might not happen. to be prepared for the unexpected is a correct attitude, yet to always expect things to go awry is like saying, “i am anyway going to fail in maths, so let me not study at all.”

    like palahali says, “what about expecting the sun to rise in the west?”

    taleb doesn’t seem to have said anything new, but looks like he has packaged it well – a ‘good’ black swan for him, may be?

    Riverine

     
  4. sreenivasaraos

    March 21, 2015 at 12:52 pm

    an interesting synopsis…much to chew on…slowly…i am currenly reading the randomness book. sometime back, i remember seeing a comment (by you? ) in plus ultra’s blog..on heroes and black swans and noted the name taleb…

    and very strangely – 2-3 weeks later, my friend’s son, who was going abroad, gifted away some books, as too heavy to lug, and she gave me taleb’s randomness one and a travel book..(which i passed on to my dotter)…in taleb’s book, there seems to be some focus on ‘what we do not know, that we do not know’ …some parts made me think of ‘est ‘- those erhard seminars jargons…as i said, for me it is suitable for bit by bit reading.

    btw, i liked your tongue in cheek comment on black swan comment!

    Anjala

     
  5. cheap fifa 17 points

    October 27, 2016 at 9:27 am

    Good Web page, Stick to the wonderful job. Regards!.

     
    • sreenivasaraos

      October 27, 2016 at 9:31 am

      Thanks for the visit and for the appreciation

      Regards

       

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

 
%d bloggers like this: