Wednesday, May 30, 2012

Hedersdoktor at LiU

Here is me last week at the Linköping University commencement ceremony where I received an honorary doctorate. I had a great time and was deeply honored to receive the award. The commencement ceremony was a special occasion.

I look forward to continued collaborations with all of my friends and colleagues at Linköpings universitet. I'd especially like to get started a student exchange program, as there is a lot of complementary work going on between CU and LiU.

Thanks LiU!!

Tuesday, May 29, 2012

Hot Hands and Guranteed Winners

In a 2009 paper I laid out an argument that explored what happens when "the guaranteed winner scam meets the hot hand fallacy" (PDF). It went as follows, drawing upon two dynamics:
The first of these dynamics is what might be called the ‘guaranteed winner scam’. It works like this: select 65,536 people and tell them that you have developed a methodology that allows for 100 per cent accurate prediction of the winner of next weekend’s big football game. You split the group of 65,536 into equal halves and send one half a guaranteed prediction of victory for one team, and the other half a guaranteed win on the other team. You have ensured that your prediction will be viewed as correct by 32,768 people. Each week you can proceed in this fashion. By the time eight weeks have gone by there will be 256 people anxiously waiting for your next week’s selection because you have demonstrated remarkable predictive capabilities, having provided them with eight perfect picks. Presumably they will now be ready to pay a handsome price for the predictions you offer in week nine.
The second,
. . . is the ‘hot hand fallacy’ which was coined to describe how people misinterpret random sequences, based on how they view the tendency of basketball players to be ‘streak shooters’ or have the ‘hot hand’ (Gilovich et al., 1985). The ‘hot hand fallacy’ holds that the probability in a random process of a ‘hit’ (i.e. a made basket or a successful hurricane landfall forecast) is higher after a ‘hit’ than the baseline probability.9 In other words, people often see patterns in random signals that they then use, incorrectly, to ascribe information about the future.
In the paper I used the dynamics to explain why there is not likely to be convergence on the skill of hurricane landfall forecasts anytime soon. The existence of (essentially) an infinite number of models of hurricane landfalls coupled with the certainty that unfolding experience will closely approximate a subset of available models creates a context ripe for seeing spurious relationships and chasing randomness. However, the basic argument has much more general applicability.

A new paper is just out by Nattavudh Powdthavee and Yohanes E. Riyanto from the Institute for the Syudy of Labor in Bonn, Germany provides some empirical support for this argument. The paper --titled, "Why Do People Pay for Useless Advice? Implications of Gambler’s and Hot-Hand Fallacies in False-Expert Setting --  looks "experimentally whether people can be induced to believe in a non-existent expert, and subsequently pay for what can only be described as transparently useless advice about future chance events."

In the study they authors operationalized the dynamics of the "the guaranteed winner scam meets the hot hand fallacy" using coin flips, while going to great lengths to ensure that the participants were aware that the coin being flipped was fair (i.e., the flips were random), even going so far as to have the participants furnish the coin.

They found that upon receiving an accurate "prediction" of the subsequent coin flip, many participants were willing to abandon any assumption of randomness and pay for a prediction of the next toss:
On average, the probability of buying the prediction in Round 2 for people who received a correct prediction in Round 1 was 5 percentage points higher than those who previously received an incorrect prediction in Round 1 (P=0.046). The effect is monotonic and well-defined; probabilities of buying were 15 percentage points (P=0.000), 19 percentage points (P=0.000), and 28 percentage points (P=0.000) higher in Rounds 3, 4, and 5 . . .
 The authors identify two interesting results:
The first is that observations of a short streak of successful predictions of a truly random event are sufficient to generate a significant belief in the hot hand of an agent; the perception which also did not revert even in the final round of coin flip. By contrast, the equally unlikely streak of incorrect predictions also generated a relatively weak belief in the existent of an “unlucky” agent whose luck was perceived to be likely to revert before the game finishes; evidence which was reflected in an increase in the subject’s propensity to buy in the final round of coin flip.
The study also looked at whether characteristics of the participants might be related to their behavior, finding: "there is no statistical evidence that some people are systematically more (or less) susceptible to the measured effects."

What does this study mean for how we thing about science in decision making?

While the authors focus on "false" experts, the findings have much broader relevance in the context of "true" experts. The simple reason for this is that the distribution of legitimate scientific findings about many complex subjects covers an enormous range of possible outcomes. Not all of these outcomes can simultaneously be correct -- whether they are looking at the past, at causality or offering projections of the future.

In the example that use from my paper cited above, I explain how a single scientific paper on hurricane landfalls provides 20 scientifically legitimate predictions of how many hurricanes would hit the US over the subsequent 5 years:
Consider, for example, Jewson et al. (2009) which presents a suite of 20 different models that lead to predictions of 2007–2012 landfall activity to be from more than 8 per cent below the 1900–2006 mean to 43 per cent above that mean, with 18 values falling in between. Over the next five years it is virtually certain that one or more of these models will have provided a prediction that will be more accurate than the long-term historical baseline (i.e. will be skilful). A broader review of the literature beyond this one paper would show an even wider range of predictions. The user of these predictions has no way of knowing whether the skill was the result of true predictive skill or just chance, given a very wide range of available predictions. And because the scientific community is constantly introducing new methods of prediction the ‘guaranteed winner scam’ can go on forever with little hope for certainty.8
Such models are of far more than academic interest -- they guide hundreds of billions of dollars in investment and financial decisions related to insurance and resinsurance. What if such decisions rest on an intellectual house of cards? How would we know?

The general issue is that a bigger problem than discerning legitimate from illegitimate expertise is figuring out how to use all of the legitimate expertise at our disposal. The dynamics of the "guaranteed winner scam meets the hot hand fallacy" also presents a challenge for experts themselves in interpreting the results of research in light of evolving experience. As experts are people too, they will be subject to the same incentives in and obstacles to interpreting information as were found by Powdthavee and Riyanto.
The dominant strategies in political discourse used to deal with this situation of too much legitimate science are to argue that there is one true perspective (the argument from consensus) or that experts can be judged according to their non-expert characteristics (argument by association). My experiences over the past decade or so related to extreme events and climate change provides a great example how such strategies play out in practice, among both experts and activists.

As we have learned, neither strategy is actually a good substitute for evaluating knowledge claims and understanding that uncertainty and ignorance are often irreducible, and decisions must be made accordingly.

Saturday, May 26, 2012

Some Items Not Blogged Last Week


I've been away this past week so I had little time to blog a few blog-worthy items that crossed my desk. Below is a quick round up of some of the most interesting ones.

Above is a music video featuring some local (to where I am now) talent. Normal service returns after the holiday and we see how well I perform in the Bolder Boulder with jet lag;-)
Coming next week on this blog ... an extended series of posts on economics, innovation, technology. Stay tuned!

Wednesday, May 23, 2012

UK GM Wheat War: Not Really About Science

In the UK there is a battle brewing over a scientific trial involving genetically modified wheat. Last weekend a protester attempted to vandalize the trial, and a larger civil action is expected on May 27.  The ongoing battle, and its close cousin in the climate wars, tell us something about what can happen to science when it becomes the central battleground over politics and technology. Unfortunately, the scientific community itself has contributed to such tactics.

Plant scientists at Rothamsted Research, a complex of buildings and fields in Hertfordshire, UK, that prides itself on being the longest-running agricultural research station in the world, have spent years preparing for their latest experiment — which will attempt to prove the usefulness of a genetically modified (GM) wheat that emits an aphid alarm pheromone, potentially reducing aphid infestation.

Yet instead of looking forward to watching their crop grow, the Rothamsted scientists are nervously counting the days until 27 May, when protesters against GM crops have promised to turn up in force and destroy the experimental plots.

The protest group, it must be acknowledged, has a great name — Take the Flour Back. And it no doubt believes that it has the sympathy of the public. The reputation of GM crops and food in Britain, and in much of mainland Europe, has yet to recover from the battering it took in the late 1990s. In Germany, the routine destruction of crops by protesters has meant that scientists there simply don't bother to conduct GM experiments any more.

The Rothamsted scientists have also attempted to win over the public, with a media campaign that explains what they are trying to do and why. After the protesters announced their plans to “decontaminate” the research site, the scientists tried to engage with their opponents, and pleaded with them to “reconsider before it is too late, and before years of work to which we have devoted our lives are destroyed forever”. The researchers say that in this case they are the true environmentalists. The modified crop, if it works, would lower the demand for environmentally damaging insecticides.
It would be a mistake to conclude that the protesters are in some way anti-science or fearful that the genetically modified crops might fail to work as advertised (though surely some protesters do have these views). Their main concern is that the crops will perform exactly as advertised, and lead to further gains in agricultural productivity.

It is not science that they fear, but the implications of scientific advances for economic and political outcomes. The organization leading the UK protests calls itself Take the Flour Back, and clearly explains its rationale as follows:
Our current political system chooses to deal with world hunger through the model of “food security”, arguing that there is not enough food to go around and that we need techno-fixes to solve this. This approach ignores the fact that there is a global food surplus – many people just can’t afford to buy food. This problem is being amplified by land grabs- communities that used to grow food for themselves are being forced out of their ancestral homes, often by corporations expanding cash crop production.

The industrial food system throws away (in the journey from farms to traders, food processors and supermarkets), between a third and a half of all the food that it produces – enough to feed the world’s hungry six times over. (2)

Free trade policies imposed by the International Monetary Fund make it much harder for governments to protect small and family farmers from big multinationals. With the expansion of free-market capitalism, agricultural systems in many countries in the global south have become focused on producing cash crops for export to rich western nations. At the same time, their markets have been opened to food imports, including imports from US and EU companies at less than the cost of production. US farmers benefit from billions of dollars in subsidies which make up as much as 40% of US net farm income. This means they can afford to export their crops at well below production cost. (3) This is ruining the livelihoods of small farmers in the global south.
This is not the statement of a group concerned primarily with the potential unanticipated risks of GM crops to the environment or people, but rather, it is the manifesto of a group concerned that GM crops will perform exactly as intended.

Like many issues where science and politics intersect, those opposed to the productivity gains made possible by agricultural innovation have sought to use science as a basis for realizing political ends. A primary strategy in such efforts is typically to argue that the science compels a particular political outcome.  In the case of GM crops, opponents of the technology (mainly in Europe) have argued that the techniques are unproven or risky. However, such tactics have not succeeded. So the next step beyond waging a political battle over science is now direct action against the technology of concern.

This situation is of course in many respects parallel to the climate debate. Efforts to compel emissions reductions through invocations that science compels certain political outcomes have borne little fruit, so some activists have taken it upon themselves to directly attack the technologies at the focus of their concern.

One difference between the climate wars and the GM wars is that some prominent scientists are participating in the direct action against technology (such as James Hansen and IPCC contributor Marc Jaccard). Another important difference is that in the case of GM crops, it is research itself being targeted, and the scientific community objects.

One argument invoked by scientists in support of GM technology is that the world needs more food. But the world needs more energy too. In condoning direct attacks on energy technologies, the scientific community may have opened the door to tactics that it does not much like when they are applied closer to home.

Monday, May 21, 2012

Beyond Manna from Heaven

Writing at The Breakthrough Journal blog, Ted Nordhaus and Michael Shellenberger argue that conventional economics is not up to the task of offering sound policy advice for the 21st century.  They write
In the 70 years that have passed since Joseph Schumpeter coined the term "creative destruction," economists have struggled awkwardly with how to think about growth and innovation. Born of the low-growth agricultural economies of 18th Century Europe, the dismal science to this day remains focused on the question of how to most efficiently distribute scarce resources, not on how to create new ones -- this despite two centuries of rapid economic growth driven by disruptive technologies, from the steam engine to electricity to the Internet.

There are some important, if qualified, exceptions. Sixty years ago, Nobelist Robert Solow and colleagues calculated that more than 80 percent of long-term growth derives from technological change. But neither Solow nor most other economists offered much explanation beyond that. Technological change was, in the words of one of Solow's contemporaries, "manna from heaven."
Where does that "manna from heaven" originate? In pricing incentives, of course, derived from economic theory. But once you take a closer look at both practice and the theoretical origins, you find that economics explains far less than we've been led to believe.

Nordhaus and Shellenberger revisit the climate issue to illustrate how far conventional economics has led us astray. They provide an overview of a debate that they engaged in with an economist from the Environmental Defense Fund, Gernot Wagner, who argues against evidence and common sense that by creating the right pricing incentives, drastic emissions reductions goals can be met in the near term:
 "[W]e can achieve US emissions reduction goals for 2020 and possibly even 2030 through deployment of existing technologies. . . Price goes up, demand goes down. Economists typically call it the 'law of demand'--one of the very few laws we've got."
The theory is sound, its application is not -- a point that readers of this blog and The Climate Fix will well understand.

The good news is that many are beginning to move beyond the precepts of economic theory and take a look at the simple mathematics of the real world. For instance,  Ulrich Hoffmann, an economist at the UN Conference on Trade and Development, has done the math, which is illustrated in the figure below showing how much the world would need to decarbonize its economic activity in order to stabilize carbon dioxide at 450 ppm.


Based on these straightforward mathematics he concludes (Hoffmann has a more in-depth analysis here in PDF):
The arithmetic of economic and population growth, efficiency limits related to the rebound effect, as well as systemic issues call into question the hopes of decoupling economic growth from GHG growth. One should not deceive oneself into believing that such an evolutionary (and often reductionist) approach would be sufficient to cope with the complexities of climate change. “Green growth” proponents need to scrutinise the historical macro- (not micro-) economic evidence, in particular the arithmetic of economic and population growth, as well as the significant influence of the rebound effect.
Such messages are not well-received by conventional economists. In their exchange, Wagner explains to Nordhaus and Shellenberger that economic theory trumps real world evidence, and this means that debate over such issues is not necessary:
The main points from climate economics are no longer up for debate: carbon is a pollutant; we need make polluters pay, either through a cap or a price. Virtually all economists agree--from Holtz-Eakin, Laffer and Mankiw on one side to Stiglitz, Sachs, and Krugman on the other.

Once again, this one is not up for debate. You can argue that politically we can't get there, so we need to do other things in the short term, but it's not up for debate whether this is the economically correct solution.
Like most debates on climate this one ends predictably, with Wagner appealing to the motives of Shellenberger and Nordhaus:
[Y]our entire operation seems to be geared toward propagating contrarian-sounding views that once in a while get you some attention and get picked up by an editor somewhere, but otherwise are just that: contrarian for the sake of wanting to be different from the pack
Snore. But the larger point here is that there are articles of faith in the discipline of economics which are viewed as taboo to challenge, even when they fail to represent themselves in practice with the simplicity and elegance of theory.

However, a closer look at economic theory finds a much shakier foundation than is represented within the discipline. Writing at Slate, Konstantin Kakaes has a great piece that sums up how economics went astray when it comes to innovation:
Robert Solow, winner of the 1987 Nobel Memorial Prize in Economic Sciences, is famous for, in the recent words of a high-ranking State Department official, “showing that technological innovation was responsible for over 80 percent of economic growth in the United States between 1909 and 1949.”. . Typically, technical or technological progress isn’t explicitly defined by those invoking Solow, but people take it to mean new gadgets.

However, Solow meant something much broader. On the first page of “Technical Change and the Aggregate Production Function,” the second of his two major papers, he wrote: “I am using the phrase ‘technical change’ as a shorthand expression for any kind of shift in the production function. Thus slowdowns, speedups, improvements in the education of the labor force, and all sorts of things will appear as ‘technical change.’ ” But his willfully inclusive definition tends to be forgotten.

Solow was constructing a simple mathematical model of how economic growth takes place. On one side was output. On the other side was capital and labor. Classical economists going back to Adam Smith and David Ricardo had defined the “production function”—how much stuff you got out of the economy—in terms of capital and labor (as well as land). Solow’s point was that other factors besides capital, labor, and land were important. But he knew his limitations: He wasn’t clear on what those factors were. This is why he defined “technical change” as any kind of shift (the italics are his) in the production function. He wasn’t proving that technology was important, as economists in recent years have taken to saying he did. All Solow was saying is that the sources of economic growth are poorly understood.
Instead of "technology" Solow was really talking about "innovation." That innovation need not be understood because it was "manna from heaven" is characteristic of many arguments from conventional economists and is particularly endemic in the climate debate. From such a perspective, of course anyone asking about where innovation comes from -- other than from the magic of the invisible hand -- must either be ignorant or malign.

But as Hoffmann's essay explains, once you actually do the math of energy innovation in the context of real-world social and political forces, you see that understanding processes of innovation requires more than simply understanding the "law of demand."

Kakaes contines:
The cautionary tale of Solow is emblematic of how economists get science and technology wrong. One economist creates a highly idealized mathematical model. The model’s creator is, as Solow was, honest about its limitations. But it quickly gets passed through the mill and acquires authority by means of citation. A few years after Solow’s paper came out, Kenneth Arrow, another Nobel Prize winner, would write that Solow proved the “overwhelming importance [of technological change in economic growth] relative to capital formation.” It’s a sort of idea laundering: Solow said that we don’t know where growth in economic output comes from and, for want of a better alternative, termed that missing information “technical change.” But his admission of ignorance morphed into a supposed proof that new technologies drive economic growth.
Remarkably, in the 21st century, our policy debates reflect the fact that we do not have a good idea where innovation comes from, how it is directed and how we prepare for its inevitable downsides. Too often conventional economics presents an an obstacle to debating and discussing this topic.

As Nordhaus and Shellenberger conclude,
Over the next century, global energy demand will double, and perhaps triple. But even were energy consumption to stay flat, significantly reducing emissions from today's levels will require the creation of disruptive new technologies. It's a task for which a doctrine focused on the efficient allocation of scarce resources could hardly be more ill-suited.
Read the three essays discussed here in full at The Breakthrough Journal blog, at Bridges Trade BioRes review, and at Slate.

Friday, September 11, 2009

Sudoku in Coders At work

Once upon a time, I wrote this blog entry. Peter Norvig references it (indirectly) in his interview in Peter Seibel's great new book "Coders At Work".


"Seibel: Though your job now doesn’t entail a lot of programming you still write programs for the essays on your web site. When you’re writing these little programs, how do you approach it?


Norvig: I think one of the most important things is being able to keep everything in your head at once. If you can do that you have a much better chance of being successful. That makes a small program easier. For a bigger program, you need extra tools to be able to handle that. It’s also important to know what you’re doing. When I wrote my Sudoku Solver, some bloggers commented on that.

They said, “Look at the contrast—here’s Norvig’s Sudoku thing and then there’s this other guy, whose name I’ve forgotten, one of these test-driven design gurus. He starts off and he says, “Well, I’m going to do Sudoku and I’m going to have this class and first thing I’m going to do is write a bunch of tests.” But then he
never got anywhere. He had five different blog posts and in each one he wrote a little bit more and wrote lots of tests but he never got anything working because he didn’t know how to solve the problem. I actually knew—from AI—that, well, there’s this field of constraint propagation—I know how that works. There’s this field of recursive search—I know how that works. And I could see, right from the start, you put these two together, and you could solve this Sudoku thing. He didn’t know that so he was sort of blundering in the dark even though all his code “worked” because he had all these test cases.

Then bloggers were arguing back and forth about what this means. I don’t think it means much of anything—I think test-driven design is great. I do that a lot more than I used to do. But you can test all you want and if you don’t know how to approach the problem, you’re not going to get a solution.

Seibel: So then the question is, how should he have known that? Should he have gone and gotten a PhD and specialized in artificial intelligence? You can’t know every algorithm. These days you have Google, but finding the right approach to a problem is a little different than finding a web framework.

Norvig: How do you know what you don’t know?
Seibel: Exactly.

Norvig: So I guess it’s two parts. One is to recognize that maybe there is a known solution to this. You could say, “Well, nobody could possibly know how to do this, so just exploring randomly is as good as everything else.” That’s one possibility. The other possibility is, “Well, probably somebody does know how to do this. I just don’t know what the words are for it, so I
have to discover those.” I guess that’s partly just intuition and saying, “It seems like the kind of thing that should be in the body of knowledge from AI.” And then you have to figure out, how do I find it? And probably he could’ve done a search on Sudoku and found it that way. Maybe he thought that was cheating. I don’t know

.
Seibel: So let’s say that is cheating—say you were the first person ever to try and solve Sudoku. The techniques that you ended up using would still have been out there waiting to be applied.

Norvig: Let’s say I wanted to solve some problem in biology. I wouldn’t know what the best algorithms were for doing gene sequencing or whatever. But I’d have a pretty good idea that there were such algorithms. Then I could start looking around. At another level, some of these things are pretty fundamental—if you don’t know what dynamic programming is, then you’re at a severe disadvantage. It’s going to come up time and time
again. If you don’t know this idea of search in general—that you can make a choice and backtrack when you don’t need it. These are all ideas from the ’60s. It was only a few years into programming that people discovered these things. It seems like that’s the type of thing that everyone should know. Some things that were discovered last year, not everybody should know.


Seibel: So should programmers go back and read all the old papers?
Norvig: No, because there are lots of false starts and lots of mergers where two different fields develop completely different technology and terminology, and then they discover they were really doing the same thing. I think you’d rather have a story from the modern point of view rather than have to follow all the steps. But you should have them. I don’t know what the best books are for that since I picked it up the hard way, piecemeal."


Dr. Norvig is a much nicer (and wiser) person than I am. I (completely) agree with him that bloggers' blather (including mine) doesn't mean much and TDD is great as one (vs the only) tool in one's arsenal.

That said, I still think many so called "agile" "gurus" can't program for nuts, as Jeffries so amply demonstrates. It is hypocrisy of the first order that people who don't know how to program well advise the rest of us how to. The "agile" movement brimmeth over with such pseudo programmer snake oil salesmen.

"These are all ideas from the ’60s. It was only a few years into programming that people discovered these things. It seems like that’s the type of thing that everyone should know."

"Should know" is correct. It is also true that most "developers" wouldn't know Dynamic Programming if it showed up at the front door and slapped them silly.

One reason for this is that most "enterprise" programming uses essentially two data structures, the List and the Hashtable and most language implementations provide default implementations of these. And a lot of "programmers" do this all their lives and think they are cool because they drank the latest/greatest shiny methodology Kool Aid. And these folks get the "gurus" they deserve!

"Seibel: What about the idea of using tests to drive design?

Norvig: I see tests more as a way of correcting errors rather than as a way of design. This extreme approach of saying, “Well, the first thing you do is write a test that says I get the right answer at the end,” and then you run it and see that it fails, and then you say, “What do I need next?”—that doesn’t seem like the right way to design something to me."


heh! heh!

Anyway, it is great that something I wrote got reflected, however indirectly, in this great book.


Coders at Work is a great book with legendary programmers talking in great detail about many aspects of programming. It is very interesting how Peter Seibel (the author) has captured each "voice" so perfectly. Every programmer should read this book, *especially* if you are stuck in a sucky job writing leasing systems (or accounting systems or whatever).

"What power would Hell have if those imprisoned there were not able to dream of Heaven?" as the Dream King says in Neil Gaiman's "The Sandman", but even at the cost of granting Hell some power, do dream of Heaven, say I, for that way lies eventual redemption.

This book provides a glimpse of what programming looks like for people who are already "in heaven".

"Coders At Work" is Awesome

An extract from Chapter 1

"Seibel: What about books? Are there particular computer-science or programming books that everyone should read?

Zawinski: I actually haven’t read very many of those. The one I
always recommend is Structure and Interpretation of Computer Programs, which a lot of people are afraid of because it’s Lispy, but I think does a really good job of teaching programming without teaching a language. I think a lot of introductory-level stuff focuses on syntax and I definitely saw that in the classes I had in high school and in the intro classes at Carnegie-Mellon during my brief time there.This is not teaching people to program; this is teaching people where the semicolon goes. That seems like the kind of thing that’s going to scare people away from it more than anything, because that’s not the interesting part. Not even to someone who knows what they’re doing.


There was another book—what was it called?—about debugging, written by someone from Microsoft. It was about how to use asserts effectively. I remember thinking that was a really good book, not because I learned anything from it, but because it was the book you wish your idiot coworker had read.


Then there was another book that everybody thought was the greatest thing ever in that same period—Design Patterns—which I just thought was crap. It was just like, programming via cut and paste. Rather than thinking through your task you looked through the recipe
book and found something that maybe, kinda, sorta felt like it, and then just aped it. That’s not programming; that’s a coloring book. But a lot of people seemed to love it. Then in meetings they’d be tossing around all this terminology they got out of that book. Like, the inverse, reverse, double-back-flip pattern—whatever. Oh, you mean a loop? OK."


heh heh!

This book is really good and I am sitting here and reading this instead of working (and I have a LOT of work to do). I'll do a full review later, but if you are any kind of programmer just go buy it already.

@Peter Seibel Great Work.