Trust, but Verify

Trust, but Verify (AMS-2):

(This article is about climate science, but you have to make it through the beginning.)

In the previous blog I wrote about the talk at the American Meteorological Society (AMS) Annual Meeting by Jim Rogers the CEO of of Duke Energy. I stated that often when I talk positively about the roles of corporations in addressing climate change, mistrust is one of the first things I hear about. (I am perpetually naïve.)

The United States is a culture of business and markets. There is a shared belief that markets provide a way to exchange goods and services that defines a “right” value for the exchange. Our economy is based on markets, which are believed to be resilient, or, perhaps, more resilient than government-controlled economies.

In a piece that I wrote with Gabriel Thoumi we talked about carbon markets, and we described some of the basic attributes of a market -- A market is a system that allows for exchange of quantities of expected risk and expected return, using various forms of capital within a regulated, contractually determined time frame. A well functioning market provides liquidity, assurance of completion, and transparency. Liquidity is the ability for market participants to convert an asset into capital in a timely fashion. Transparency refers to the ability for market information to be accurate, timely, and available to all participants. Finally, all markets must have assurance of completion, which is the guarantee that the intended transaction can be carried out, lessening systematic market risk. Non-systematic risk may occur if a counterparty defaults before the completion of a transaction, but this is not the responsibility of the market. An environmental market must also have a way to verify that the required environmental impact is achieved.

There is in this definition of a market the idea of regulation, and that regulation represents rules of behavior. It is obvious that at least a subset of those rules of behavior arise because of the need to generate trust. That trust translates into attributes like “expected risk and expected return.” Transparency is often listed as a market attribute, because transparency allows participants and observers to see into the market, to know the rules, and to have a chance of knowing how to play. The existence of formal practices to assure trust suggests that in their absence the market can’t be trusted. There is a natural tendency for mistrust when we are exchanging money, and most of us look to manage our money in some sort of regulated environment.

These ideas of transparency, regulations that support checks and balances, risk and return have come up again and again in every thing that I do. Turning this to science and the scientific investigation of climate, the notion of transparency in climate science came to the front with the release of private email exchanges that were either stolen or leaked from the Climate Research Unit.

In talking with my colleagues at both the Copenhagen meeting and the AMS meeting, they say that if there is anything of true substance to come out of the released emails it is a need to shore up transparency into our investigations and to assure the robustness of the peer review process. (see Judy Curry’s post at climateaudit.org) Yesterday, I was talking with a student at Michigan about observations of carbon dioxide and how those observations might provide a data set that is used by people interested in making climate policy. We reached a conclusion that there was a type of validation that was needed for the climate scientist, and that there was a whole different set of attributes that would be needed to make the data set usable for the policymaker. For example, can you imagine the United States “believing” the observations taken by a satellite from another country in a regulatory environment? Or framing it more positively, what sort of process and participation is needed by all parties to assure that the knowledge in the data set is accurate and accepted?

It is reasonable to assert that scientific investigation is held to a different standard than the average stock market. This is because scientific investigation in its ideal is directed at measurable, quantified results that can be validated by objective methods. This is a simple, yet high, standard. If you look in more detail at the “scientific process,” then there are some other attributes worth noting. There is the general notion that the results that a scientist gets have to be reproducible. Not only does it have to be reproducible, but it has to be reproducible by independent investigators. Then there is the value that the results have to be able to pass peer review.

Comparing scientific investigation with the market, independent reproduction of results and peer reviews are values that lead to practices that regulate the practice of science. They provide a formal method of checks and balances. These are practices that have arisen because of many reasons, including that scientists are self-interested humans that are all, I repeat all, influenced by their perceptions of a problem. They are motivated by being right. Peer review strives to provide an objective check, and it follows from years of experience that scientists have a perspective that they project onto their work.

I do not know a publishing scientist who has not had an issue with peer review. One of my most influential papers took more than two years of reviews and revisions before it was published? What are issues? Sometimes there is the prejudice of an editor. Sometimes there is pure competitiveness to be first or right. Sometimes it is a simple matter of presentation and style. Plus, peer review is potentially a very painful experience. You work, you place an idea out there, and people are supposed to challenge that idea, beat it up, and find its weaknesses. I have seen many young scientists shrink away, even when they have very good ideas.

So when you submit a paper, you pick a journal, maybe pick the editor, you list those who you think are prejudiced, and those who are your friends and collaborators are eliminated. It’s a messy process, which in my experience, works far more often than it fails.

It is true that it is difficult to introduce radical new ideas, and it is true that it is hard to challenge a massive body of knowledge.

There are so many places to go …. (have you noticed that this is now an “article,” not a “blog?”) OK … back to the email messages from the Climate Research Unit. In these emails you see discussions of finding favorable journals and discussions of the motivations of competing scientists. But in the end, is there evidence of a massive failure of peer review? Didn’t the points of view of scientists get objectified as they moved to actual publication? Didn’t the papers of those who were challenging the body of knowledge not only make it to journals, but also make it into the IPCC reports? Isn’t this evidence that the regulations in the science “market” worked rather than it did not work?

A paper that I frequently reference is by the law professor Daniel Farber. At some level he reviewed the process of review, and especially the process that is used to generate the IPCC reports. It was his conclusion that the process of review of the scientific investigation of climate change raised the conclusions to a higher level than normal. Why would this be true? Ultimately, the IPCC process is by most any standard transparent; it is known and published. The reports are reviewed, re-reviewed, and subjected to open review. There is controversy amongst scientists that this leads to muted and conservative conclusions; it is a process that reduces extremes, some of which are correct conclusions (muting), and some of which are incorrect conclusions(improving accuracy). This reflects the principle of reproducibility. Within the body of knowledge, from across the world, virtually every result in the the IPCC reports has been independently verified. By virtue of a long history of research, ideas have been visited and revisited.

I have been trying to imagine a more open field of research. Off the top of my head I go, for no particular reason - what if I wanted to probe the details and nuances of diabetes research. Here, I fall into proprietary drugs and papers influenced by industrial funding. Plus, I have been knowledgeable of medical research where I was told not to reveal the tests I had seen to the next group down the hall because they all thought they were on the path to the Nobel Prize. I and my students have tried to break into new fields and our work has not been sent out for review because we are not already expert.

This is not meant to contend that our process, our transparency could not be better. In fact, it is an obligation for us to improve access to data, documentation of validation, and objectiveness of review. This is an ethical obligation as well as an obligation that arises because of the consequences that follow from research of the Earth’s climate. We are faced with the problem that I suggested to my student above. We live in a world where our research does not only meet the standards of the scientist, but also that must be accessible and verifiable to, for example, policy makers, resource managers, and market regulators. The IPCC process is clearly developed with this in mind, but the requirements become more stringent as the consequences become more important. And the cultures that determine transparency, regulations that support checks and balances, risk and return are different for different communities.

In my mind, this blog was motivated by recent conversations with a couple of engineers and a taxi driver. I was told by the engineers that climate scientists needed to make sure the whole message got out, and that there was access to the literature so that anyone could check. That the climategate emails reminded them of groups in their organizations that presented data to support the desires of their group. If it is your goal to check for yourself, the information is there. It is a daunting task to address in your spare time, but it is far easier to access than any body of knowledge that I am aware of. Start with the IPCC report and work backwards. Look at those thousands of pages and follow the threads back to the original references. Data from models and observations is, more often than not, available and free. Look for those items that you think climate scientists ignore or dismiss, and if you find that they are truly ignored and dismissed then we want to know. Write to your favorite blogger.

By the way, the taxi driver told me it was pretty obvious that we were consuming the world in an unsustainable way. He invited me over for ribs, and I hope he calls me some day.

r

And here is

Faceted Search of Blogs at climateknowledge.org





Figure 1: A ton of carbon dioxide in Copenhagen.