More on causality and the national debt

As I pointed out in this post and this one, growth in the national debt can only be fairly evaluated by assigning causality to the different contributors to that debt.  Simple-minded comparisons of debt when a president came into office with current day debt will be misleading if previous occupants of the office implemented policies that continue into the next administration.

I recently discovered that the New York Times and the Washington Post have conducted analysis of the contributors to debt based on the policies of the Bush and Obama administrations, and the bottom line is about the same in each case.  President Bush’s policies increased the debt by about $5 trillion (T) from 2001 to 2009, while President Obama’s policies increased the debt by $1 to 1.4T.  By attributing debt to specific policies, these two analyses exclude debt attributable to previous administrations, so the comparison is a consistent one.

These analyses don’t seem to include explicit treatment of the effect of the Great Recession on costs and revenues, which is something worth exploring (for one such analysis, see this graph via Paul Krugman).  They also aren’t explicit about how they treat inflation and the time value of money, both of which make money spent in earlier years more valuable than money spent in later years.  Someone evaluating these numbers would need to understand how those two effects were treated to use the data in other comparisons.  In any case, correcting for those two effects would tend to make President Bush’s relative contribution to debt even larger if they are not currently included in these comparisons.

Energy harvesting in the news

The world is starting to pay greater attention to energy harvesting, through which ultra-low power sensors and controls can be powered by ambient energy flows (like light, heat, motion, or stray radio and TV signals).  This week, Electronics Weekly reported on a study that estimated the current market for energy harvesting at $19M/year, and projected that it will grow roughly tenfold by 2017.  Putting aside the difficulty of projecting the future for economic and social systems, it’s clear that people are waking up to the potential for energy harvesting. Thus far it’s mostly been a niche application, the most widely used example of which is tire pressure sensors in cars (they use the motion of the wheel to power themselves).

There has also been more interest recently in biomedical applications.  Proteus Digital Health has an ingestible sensor that has no battery.  Instead, it has a cathode and anode, and uses your stomach juices as the electrolyte.  It goes inside a pill, and when the pill dissolves in your stomach it sends a tiny signal to a patch on your skin, which relays the signal to your cell phone or other mobile device, recording accurately when you took your medicine.  This is what Proteus calls “partial energy harvesting”, since the energy extracted is really embedded in the anode/cathode pair, and the electrolyte simple enables us to tap that energy for as long as the electrodes last.

Nature Biotechnology published an article recently on a device that can extract power from a biologic battery found in the inner ears of certain animals, including humans.

Mercier, Patrick P., Andrew C. Lysaght, Saurav Bandyopadhyay, Anantha P. Chandrakasan, and Konstantina M. Stankovic. 2012. “Energy extraction from the biologic battery in the inner ear."  Nat Biotech. Advance online publication, 11/08/online. [http://dx.doi.org/10.1038/nbt.2394]

The Wall Street Journal reported on the Nature Biotechnology paper as well as some other examples of energy harvesting in medicine, where the concept seems to be taking off.

It’s important to remember that energy harvesting is still in its infancy, and that it’s competing against single use batteries that are also improving over time.  Once an electronic device has achieved very low power (averaging micro watts or nano watts) then it’s relatively easy to attach it to a single use battery and achieve battery lifetimes in years (or even a decade or two).  For many applications, that’s more than sufficient, so the cost of energy harvesting needs to be compared to that for a single use lithium or lithium thionyl chloride battery, and in many cases the battery will come out ahead. That won’t always be true, but it’s often true now.

A real-world example of how wrong "likely voter" screens can be

As if in reply to my missive about hazards in political polling, Politico has a great story today about just how far astray likely voter screens can lead even seasoned political professionals:

For Republicans, one of the worst parts of the GOP’s 2012 trouncing was that they didn’t see it coming.

Top party strategists and officials always knew there was a chance that President Barack Obama would get reelected, or that Republicans wouldn’t gain control of the Senate. But down to the final days of the national campaign, few anticipated the severe setbacks that Republicans experienced on Nov. 6.

The reason: Across the party’s campaigns, committees and super PACs, internal polling gave an overly optimistic read on the electorate. The Romney campaign entered the last week of the election convinced that Colorado, Florida and Virginia were all but won, that the race in Ohio was neck and neck and that the Republican nominee had a legitimate shot in Pennsylvania.

In other words, the likely voter screen the Republican pollsters applied to figure out who would actually vote were grossly inaccurate.  And this conclusion is confirmed by Democratic pollsters and the Obama Campaign:

Democrats had argued for months before the election that Republican polling was screening out voters who would ultimately turn up to support Obama. In fact, Obama advisers said, if you applied a tighter likely voter screen to Democratic polling — counting only the very likeliest voters as part of the electorate — you could come up with results similar to what the GOP was looking at.

Keep in mind the twofold purposes of political polling next time you see polling results.  The first is to take an accurate snapshot of the electorate’s opinions on a certain date, but the second is to predict the results on election day.  The first goal isn’t easy to achieve, but the second one is even harder (because predicting behavior of human behavior is difficult in all circumstances, impossible in many).   Be much more skeptical of likely voter polls and focus more on polling averages than on a single poll, because you’re much more likely to have an accurate picture that way.  You should also read “polling postmortems” (like the one just published by Nate Silver at 538) to understand how each pollster stacked up against actual results.

This last conclusion applies to all kinds of forecasts, which is why I’m a strong advocate of retrospective comparisons of forecasting results to actual events (see for example Koomey, Jonathan G., Paul Craig, Ashok Gadgil, and David Lorenzetti. 2003. “Improving long-range energy modeling:  A plea for historical retrospectives.”  The Energy Journal (also LBNL-52448).  vol. 24, no. 4. October. pp. 75-92.   Email me for a copy.  Also check out this short post on a retrospective for a 1981 climate forecast).

My ARM Tech Con keynote ("Why ultra-low power computing will change everything") is now posted

I had great fun this past Wednesday (October 31st, 2012) talking at ARM’s Tech Con event, which is probably the world’s largest gathering of technologists devoted to low power innovation.  Compared to my Authors@Google talk, I’ve added some additional examples of ultra-low-power computing and communications and really boiled the talk down to its essential messages. Check it out!:

If political polls are driving you crazy, read this

As the election approaches, I’ve been musing over the nature of political polling.  There are many folks who make a living reporting on poll results, and a few who actually do solid analysis using such polls (with Nate Silver at 538 being the most prominent and sophisticated example).  Unfortunately, there are problems inherent in the enterprise of measuring public opinion that make it impossible to say with certainty what the outcome will be (at least for a close presidential election like this one promises to be).

There are two goals of a poll:  1) to create a “snapshot” of public opinion during the period over which the poll was conducted, and 2) to predict who will win the election.  It’s important to distinguish these two goals.

Taking a snapshot of public opinion seems straightforward, but it’s getting harder to do so, as different parts of the public change their preferences about answering calls from strangers (in part aided by technologies like call-waiting, which are becoming more widespread).   People are increasingly shifting to not having landline phones, and that may also introduce biases into the results.

The spread of polling results is largely the result of differences in how factors such as these are treated by the pollsters (and there are always buried assumptions and judgment calls in such analyses).  So it’s not at all clear that the snapshot of public preferences is accurate, even for the pollsters who are most sophisticated and use human interviewers and careful statistical methods.  Analysts try to adjust for these variations by taking averages of polling results, but such methods only work when there is no systematic bias affecting the results (as an aside, the “margin of error” that is commonly reported for polls is a simple statistical measure based on the number of respondents and does not reflect the kinds of structural biases I describe above).

I want to turn now to the second goal, which has received comparatively little attention (it’s the one that prompted me to write this post in the first place).  Doing a prediction of what will happen is fraught with problems, unless you are dealing with a physical system like planets orbiting the sun, and it is in attempting to do predictions that I think most pollsters get into real trouble.

The methods used to convert samples of “registered voters” to samples of “likely voters” are where the snapshot formally becomes a prediction.  In that conversion the pollster needs to decide who will submit a valid ballot by Tuesday November 6th.  While it is possible to make educated guesses based on historical data, each election is different.  Will an energized Republican base offset increased enthusiasm in the hispanic community?  Will Hurricane Sandy make voting difficult in some states?  Will undecided voters break for the incumbent or the challenger?  Will efforts to require photo ID reduce turnout of certain voter groups, and if so, how much?  Will someone be able to manipulate the voting results?  None of these things will be known with certainty until after November 6th, and history may give no guidance at all.

So I’m convinced that polling simply can’t tell us with certainty who will win the election, at least when it’s close.  According to Nate Silver, the state level polls suggest an electoral advantage for President Obama (with about 75% probability today), but what the polls can’t say is what will actually happen on November 6th, and the margins in many states are small enough that the election could go either way.

And that’s where we, the people come in.  Our choices are what determine the future. In an election where polls are close, your vote really does count.  What we decide to do will be what makes the difference on Tuesday. So don’t get hung up on contradictory polling results, just go out and vote!

“The best way to predict the future is to invent it.”  –Alan Kay

Addendum, October 30, 2012:  The statistician Andrew Gelman wrote a nice piece in The New York Times that analyzes what “too close to call” means in the context of this election.  Highly recommended reading.  It addresses the ostensible contradiction that President Obama has a 75% chance of winning but that the election could go either way based on unpredictable factors.

Why we need to stop coal exports and keep coal in the ground

Many observers have been heartened by the increase in natural gas production, which has contributed to significant significant declines in US greenhouse gas emissions.   It is not actually the most important factor reducing emissions in the first half of 2012, as my friends at CO2 Scorecard and I showed earlier this year.  And the methane emissions from fracking haven’t been measured very accurately, so there may be increased warming from methane that significantly offsets the reductions in other fossil fuels from the use of natural gas.

Another important issue addressed in the research note from CO2 Scorecard is the reduced price of natural gas resulting from fracking, which increases use of natural gas not just in the electricity sector (where gas displaces coal) but also in buildings and industrial sectors, and those increases offset emissions savings in the electricity sector.

Today there’s a story in the Guardian that shows another interesting (and troubling) price-related effect of natural gas fracking:  as US coal use has declined, an increase in coal exports from the US has reduced global prices of coal.  That price decrease makes it harder for countries with modest natural gas reserves to reduce use of coal-fired electricity, as the Guardian story demonstrates.  The pressure is particularly intense in developing countries, which are often more price sensitive than developed countries.

This story makes a compelling case for reducing and ultimately stopping exports of US coal, in order to keep global coal prices higher than they otherwise would be.  In part, US subsidies to the coal industry are subsidizing exports and reducing world electricity prices, and that’s just perverse, but even without subsidies, we need to slow and soon stop coal exports.

We need to either develop ways to sequester carbon from coal burning or keep the coal in the ground.  Since the first option is being tested but is nowhere near implementation on a large scale, our only current option is not burning the coal.  So that means not approving additional coal export terminals, diligently enforcing existing environmental regulations, and eliminating subsidies for coal mining.  It makes absolutely no sense to export coal that we don’t burn in the US, because no matter where it is burned, it will contribute to warming just the same.

US coal in decline: New Brattle Group report on coal-fired power plant retirements

On October 1, 2012, the Brattle Group published an update to its 2010 numbers on coal-fired power plant retirements, and it “finds that 59,000 to 77,000 MW of coal plant capacity are likely to retire over the next five years, which is approximately 25,000 MW more than previously estimated”.

The news release for the study states

Since December 2010 when the prior estimates of potential coal plant retirements were released, both natural gas prices and the projected demand for power have decreased, and environmental rules have been finalized with less restrictive compliance requirements and deadlines than previously foreseen. These shifts in market and regulatory conditions have resulted in an acceleration in announced coal plant retirements. As of July 2012, about 30,000 MW of coal plants (roughly 10% of total U.S. coal capacity) had announced plans to retire by 2016.

The updated study takes into account the most recent market conditions and the shifting regulatory outlook facing coal plants. To reflect the remaining regulatory uncertainty, the authors developed both “strict” and “lenient” regulatory scenarios for required environmental control technology. About 59,000 MW will likely retire under lenient rules versus 77,000 MW under strict regulations. Final regulatory requirements are still unresolved, but the authors suspect they will be akin to the lenient scenario. The study highlights that retirement projections are even more sensitive to future market conditions than to regulations, particularly natural gas prices. Likely coal plant retirements drop to between 21,000 and 35,000 MW if natural gas prices increase by just $1.00/MMBtu relative to April 2012 forward prices. Similarly, projected coal plant retirements would increase to between 115,000 and 141,000 MW if natural gas prices were to decrease by $1.00/MMBtu.“

Coal will continue to decline in importance in the US in the medium term, and it isn’t principally because of environmental regulations.  Of course, regulations are getting tighter, but cheap natural gas is the main culprit.  In addition, few new coal plants are likely to be built to replace the retiring plants.  Instead, natural gas and wind plants will likely pick up most of the slack.

My Authors@Google talk on computing trends, now posted

I had great fun at Google on September 12th, 2012 talking about the implications of computing efficiency trends for mobile sensors, controls, and computing more generally.  This version contains my latest thinking and is my most polished version to date.  You can watch it below (or by clicking here).

My friend Luiz Barroso introduced me, and he was gracious enough to highlight my latest book, Cold Cash, Cool Climate:  Science-based Advice for Ecological Entrepreneurs.

For background on computing trends, see my Technology Review article, and the supporting academic article:    Koomey, Jonathan G., Stephen Berard, Marla Sanchez, and Henry Wong. 2011. “Implications of Historical Trends in The Electrical Efficiency of Computing."  IEEE Annals of the History of Computing.  vol. 33, no. 3. July-September. pp. 46-54.

I'm on KCRW today, talking about the NYT story and its implications

KCRW in Santa Monica, CA had a discussion show this morning (“To the Point”) about electricity used by data centers, prompted by the NY Times article by Jim Glanz.  Jim led off the show in discussions with the host (Warren Olney), and then I, Andrew Blum of Wired, and Andy Lawrence of 451 Group/Uptime Institute added context and commentary.  It was a useful discussion, and the interviewer asked good questions.  You can listen in here.

Giga Om on the NYT data center articles

Katie Fehrenbacher over at GigaOm did a service for those of us interested in data centers by compiling some of the issues with the recent New York Times article.  She summarizes her conclusions (with which I agree) here:

I feel the same way about the NYT’s series that I do about Greenpeace’s dirty cloud reports. Yeah, they got a few things wrong, but the overall thesis is right, and can be used to make the Internet industry even more conscientious about their carbon emissions and energy footprint.

There are still a few Internet leaders who haven’t publicly embraced energy efficiency and greener technologies for data centers. For example, Amazon and its web services haven’t really stepped up to touting energy efficiency and clean power technologies so far, despite its prominent role in the industry. Though, they have made some strides.

Additionally while the largest and leading Internet companies have widely adopted energy efficiency practices, businesses running their own IT services haven’t adopted these technologies. That’s one of the biggest problems with the article, that the reporter is lumping together businesses’ in house IT server practices, with the webscale cloud giants. But clearly there’s still a lot more work to be done when it comes to the Internet an its massive power consumption.

I applaud Jim Glanz of the NYT for shining a light on the need for greater energy efficiency in data centers, but feel strongly that tackling the problem will require critical insights that someone just reading that article would not pick up.  Let’s hope this is the beginning of a deeper conversation about these issues.

The NYT article on Power, Pollution, and the Internet: My initial comments

Jim Glanz, writing in the New York Times this past Sunday, described existing inefficiencies in Internet infrastructure, but omitted important context that can help interested readers really understand the problem.  The article, in which I’m quoted, is Glanz, James. 2012. “Power, Pollution, and the Internet.” New York Times.  New York, NY.  September 23. p. A1.   A related “Room for Debate” section (in which I have an article) went online on Monday September 24, 2012.

The article conflates different types of data centers, and in the process creates a misleading impression for readers who are not familiar with this industry.   I like to divide the industry into four kinds of data centers:  public cloud computing providers (like Amazon, Google, Facebook, and Microsoft), scientific computing centers (like those at national laboratories and universities), co-location facilities (which house servers owned by other companies), and what I call “in-house” data centers (which are facilities owned and operated by companies whose primary business is not computing).  The fourth category is by far the dominant one in terms of floor area and total electricity use, and almost all the issues raised in the article apply most clearly to facilities in that category.

Each category of data centers has very different characteristics and constraints.   The scientific computing category is in a class by itself, because it runs computing jobs than can be queued up and these facilities thus do not need to respond to changes in demand.  The other three categories must respond in real time, which requires some slack in the system in case of unanticipated changes in demand.  That’s why quoting the 96.4% utilization of LBNL’s supercomputer in July 2012 (as the article does) says nothing about possibilities for increased utilization in the vast majority of data centers.

The public cloud providers are much more efficient than the “in-house” and collocation facilities.  One implication of the NYT article (as expressed, for example, by quotes from Hank Seader and Randall Victora) is that we’ll be using the computing resources one way or another, and it doesn’t matter where these are housed.  This conclusion is incorrect.  The low utilization numbers cited in the NYT article generally apply to the “in-house” and collocation facilities, not to the cloud providers (who have many more and different kinds of users, so utilization is generally much higher).  The infrastructure efficiencies in cloud computing facilities are higher as well.  For example, the Power Utilization (or Usage) Effectiveness in typical “in-house” data centers is between 1.8 and 1.9, while for cloud facilities it is closer to 1.1 (that means for every 1 kWh used in IT equipment, only 0.1 kWh is used for cooling, fans, pumps, power distribution, and other infrastructure).  So it really matters whether IT resources exist in cloud computing data centers or in standard “in-house facilities”, and the problems identified in the article mainly matter in the “in-house” facilities.

There are good reasons why cloud providers are more efficient, including economies of scale, diversity and aggregation of users, flexibility of operations, and ease of sidestepping organizational constraints.  There is also an underlying driver for greater efficiency that is critically important–the cloud providers have fixed the internal institutional problems that lead to separate budgets for the IT and facilities departments (split incentives) and dispersed responsibility for data center design, construction, and operations.  The vast majority of “in-house” and collocation facilities have not fixed these problems, so efficiency is not high (or not even) on the priority list.  And it’s institutional and not technical failures (the lack of proper cost allocation, management responsibility, and inventory tracking) that results in a large number of “comatose” servers, for example.

The problem is that the people who run the data centers for “in-house” and collocated facilities have little influence on these institutional issues.  It’s the people at the C-level in the corporation (CEO, CFO, CIO) who need to make these changes happen, and thus far there’s been little movement there in most companies.  That’s the biggest challenge, and it’s one I wish the article had highlighted.  Once these problems are fixed, big changes in efficiency follow rapidly and continue apace (they become part of the business culture and drive continuous improvements).

The article also ignores the value of the services being produced by data centers, which is the key reason why so many data centers have been built in the first place.  The value is so much higher than the costs that the inefficiencies in the “in-house” facilities are tolerated as long as reliability is maintained.

The article and the associated “Room for Debate” section seem to imply that it is consumers’ and companies’ demand for instantly available information that is at fault for the industry’s obsession with “uptime”, but the demand for information can be met in many ways, and the issue is how the industry chooses to satisfy the demand for information, and not the nature of the demand for information itself.  There are ways to deliver information with comparable levels of “uptime” but much lower costs and energy use (as the cloud computing providers have demonstrated), and we need to figure out ways for such innovations to be adopted in all “in-house” data centers.

Another (less important) issue I have with the article is that it uses the word “cloud” in its colloquial sense–i.e., anything on the other side of the users wall is “the cloud”.  In this context, however, it is more important to distinguish “cloud computing” from the other types of data centers I list above, because cloud data centers are designed and operated quite differently from those other types.  That’s the distinction that matters for understanding this issue, and the use of the colloquial term “cloud” just confuses people.

If you’ve already read the NYT article, I urge you to examine it again after reading this blog post.  Distinguishing between different types of facilities should yield crucial insight into why these inefficiencies exist and what we can do about them.  I’m interested to hear your thoughts.

New review of "Cold Cash, Cool Climate"

Writing in Environmental Research Web yesterday, Evan Mills of Lawrence Berkeley National Laboratory reviewed Cold Cash, Cool Climate:  Science-based Advice for Ecological Entrepreneurs.

Here are the first couple of paragraphs:

Entrepreneurs and investors alike will profit from Jonathan Koomey’s new book on how to cool the climate while garnering some cold cash. Starting with a well-reasoned case for urgent action to slash greenhouse-gas emissions, Koomey dispenses tips for innovators who can help turn the tide. While targeted at the business community, students, policymakers and even the general public will find this compelling book an easy read full of actionable suggestions. Koomey's blog summarizes the arguments.

Few of today’s climate and energy analysts have the skill or take the time to communicate their insights accurately to non-specialist audiences. Koomey – a seasoned energy and environmental researcher – effectively positions this book between the “hardcore technical” and “readable but imprecise popular”. He combines methods from multiple disciplines and boils an enormous literature down to its essential messages.

The review concludes:

The book ends on a highly optimistic reminder that the future is still ours to choose. The energy and economic pathways in front of us have never before been so divergent. Koomey’s book will help us choose wisely, and laugh all the way to the bank.

Read more…

A 2002 talk on climate change by the late Stephen H. Schneider, now posted online

The folks at PARC have posted online a 2002 talk on climate change by the late Stephen H. Schneider.  For those who didn’t know Steve, he was a genius unlike any other, and a very interesting speaker.  I heartily recommend that you check it out.

Is cheap natural gas really the cause of record low US carbon emissions?

I teamed up with the folks at CO2 scorecard to analyze the causes of Q1 2012’s record low carbon emissions for the US, and you can read the full research note here.  The summary findings follow below:

In this research note we show that the mild winter of 2012 was the biggest factor in slashing first quarter’s CO2 emissions in the US to the lowest level in twenty years. Demand for natural gas and electricity for space heating in residential and commercial sectors took a major dip as the number of heating degree days plummeted in the first three months of 2012. As a result, the warm winter alone accounts for 43% of the total quarterly CO2 reductions.

Replacement of coal generation by natural gas cut another 21%. Three additional factors—decline in end-use electricity consumption, reduced consumption of petroleum products, and increased generation of wind power together contributed 35% to the total CO2 reductions. However, gasoline was essentially unchanged from the first quarter of 2011.

We discuss policy implications of these findings.

Link: http://www.co2scorecard.org/link/Index/257,12

Here’s the key graph, which tells the story nicely:

Short piece from Congressional Budget Office documenting causes for the increase in the deficit over the past 11 years

This two-pager gives the CBO’s latest estimates of what caused the increase in deficits in the past 11 years (for a bit of high-level discussion of the results, go here).  For those interested in causality (i.e., which actions are responsible for changes in the deficit and national debt) this is a great place to dig into the data.  Also see Ezra Klein’s recent article as well as my previous related posts from January 31, 2012 and August 24, 2011.

Bottom line: Before blaming a president for increases in the debt and deficit, you need to first understand what caused those increases and who is responsible for the decisions that led to that result.  Just saying “the debt was X trillion when the president took office and Y trillion today” is at best misleading.  The only accurate way to analyze the issue is to assess whose decisions contributed to the deficit and the associated increase in debt.