Climate Progress points to a Yale University study on adoption of photovoltaics (PVs) in residences, in which the time lag between installations falls as the number of installations increases. The authors call this a “peer” effect, in which a greater concentration of PV panels makes it even more likely that neighbors will also install PVs.
This is a specific example of what in the economics literature is called “increasing returns to scale”. There are many different forms of this effect, including economies of scale, network externalities, learning by doing, and zero marginal costs for reproducing information (by using information technology). For those interested in carbon mitigation opportunities, this effect is critical, but it is omitted by assumption from virtually all computable general equilibrium models, because including it would result in path dependence and multiple possible end-points for a given starting point. The real world is full of such effects, and that’s one reason why conventional economic assessments of the costs of reducing carbon emissions almost invariably overestimate the costs of taking action. I will have a lot more to say about increasing returns in upcoming posts.
The NY Times.com article yesterday on the deluge of data from DNA sequencing raised a a couple of interesting issues for me.
Here’s one important item I noticed:
“The cost of sequencing a human genome — all three billion bases of DNA in a set of human chromosomes — plunged to $10,500 last July from $8.9 million in July 2007, according to the National Human Genome Research Institute.
That is a decline by a factor of more than 800 over four years. By contrast, computing costs would have dropped by perhaps a factor of four in that time span.”
This example highlights an important point: the cost to perform computations is driven by more than Moore’s law. It’s also a function of our cleverness in designing efficient algorithms and characterizing problems in the most effective ways, and that kind of cleverness can lead to much more rapid improvements in our ability to do useful computations than just the trends in raw computing horsepower would indicate.
Now on to my second point. The big constraint in DNA research is fast becoming our ability to make sense of the voluminous data being generated by the new sequencing machines, and that takes human thinking, it’s not just a computational task. Just as in many other areas that are likely to see an explosion in data generation (caused by the revolution in ultra low power mobile information technology) there will be big opportunities for those who can combine careful critical thinking with information technology to sort through vast piles of data and help people generate actionable information. This is also one of the conclusions of the recently released ebook by Brynjolfsson and McAffee titled “Race Against the Machine”, which I highly recommend.
The Canadian Broadcasting Company just posted my interview for their “Spark” radio show, which is “an ongoing conversation about technology and culture, hosted by Nora Young”. It talks about our work on trends in the energy efficiency of computing over the past six decades, which I wrote about here and here.
Unfortunately, the tagline from the announcer at the end incorrectly indicates that I called the long term trends (doubling of energy efficiency every year and a half) “Koomey’s law”, instead of noting that it was MIT’s Technology Review that popularized the term in their recent article. And the first person to use the term publicly was Max Henrion of Lumina Systems at a talk he gave at the Uptime Institute Symposium in 2010. I’ve asked the producer to correct that in the web version. Ah well..
This interview followed a news piece reporting on why many companies are considering building data centers in Colorado Springs. The interviewer asked some good questions and the discussion illuminates some important aspects of data centers and electricity use. It’s a good non-technical introduction to these issues. Listen to it here.
“A New and Improved Moore’s Law: Under "Koomey’s law,” it’s efficiency, not power, that doubles every year and a half or so“
"Researchers have, for the first time, shown that the energy efficiency of computers doubles roughly every 18 months.
The conclusion, backed up by six decades of data, mirrors Moore’s law, the observation from Intel founder Gordon Moore that computer processing power doubles about every 18 months. But the power-consumption trend might have even greater relevance than Moore’s law as battery-powered devices—phones, tablets, and sensors—proliferate.”
The reference for the actual article is below.
Koomey, Jonathan G., Stephen Berard, Marla Sanchez, and Henry Wong. 2011. “Implications of Historical Trends in The Electrical Efficiency of Computing." IEEE Annals of the History of Computing. vol. 33, no. 3. July-September. pp. 46-54. <http://doi.ieeecomputersociety.org/10.1109/MAHC.2010.28>
Subscription is required but I can send a pre-pub version if you email me.
This article describes long-term trends in the electrical efficiency of computation that enabled the creation of laptops and other mobile computing devices. If these trends continue (and we have every reason to believe that they will) they presage continued rapid improvements in battery powered computers, sensors, and controls.
The electrical efficiency of computation (measured in computations per kilowatt-hour, or kWh) grew about as fast as performance for desktop computers starting in 1975, doubling every 1.5 years, a pace of change comparable to that from 1946 to the present. Computations per kWh grew even more rapidly during the vacuum tube computing era and during the transition from tubes to transistors but more slowly during the era of discrete transistors. In 1985, Richard Feynman identified a factor of one hundred billion (1011) possible theoretical improvement in the electricity used per computation. Since that time computations per kWh have increased by less than five orders of magnitude, leaving significant headroom for continued improvements.
My interview on Intel’s Chip Chat was just posted. It’s about 7 minutes long, and we talked about my recent study on electricity used by data centers. I think it’s a good introduction to the topic, so please forward to interested colleagues as you see fit.
This week Google announced more details on their efficiency, electricity use, and carbon emissions. This is a big deal because it will give much of the rest of the industry efficiency targets to which they can aspire. Most of the other cloud computing companies have figured out clever tricks to improve their efficiency, but it’s the “in-house” data centers (the ones owned and operated by companies whose primary business is not computing) who have the most to learn from these announcements (and from the announcement of the Open Compute Project by Facebook awhile back).
Google’s announcement is also important because it puts real data on how much electricity is actually used for a search or the download of a Youtube video. Some of you may recall the little dustup about whether a Google search uses as much as boiling a pot of tea (it doesn’t, as Evan Mills and I documented here). But the biggest story is not about direct electricity use, it’s about the efficiency improvements in other energy uses enabled by the electricity used by data centers and other information technology equipment. As I describe in my recent report, the world’s data centers use roughly 1.3% of global electricity use, but they help us use the other 98.7% of that electricity as well as most of the rest of the other energy use a whole lot more efficiently.
Google’s announcements also confirm the points I made in my post on why cloud computing is more efficient, and the large savings from cloud computing estimated by WSP Environment and Energy when analyzing salesforce.com’s operations. Cloud computing will continue to pressure “in-house” data center operations because costs in the cloud are so much lower, a result driven significantly by much greater energy efficiency and equipment utilization.
There have been some helpful summaries exploring these announcements. The article at Data Center Dynamics was particularly interesting because it gives detail on the techniques Google uses to achieve high efficiency. Katie Fehrenbacher at GigaOm also did a nice job in one of her articlesof putting the announcements in a larger context, as well as giving some details in another article about the relative efficiency of Gmail compared to “in-house” email hosting.
Vivek Kundra, formerly the Chief Information Officer in the Obama Administration, argued today in a NYT op-ed that the economic benefits of cloud computing for government agencies will encourage more and more of them to adopt this innovation instead of running their own IT facilities (except in special cases). As I explained here, these economic benefits are driven in large part by the greater energy efficiency of cloud computing facilities, and they are large enough to encourage people to work out the non-trivial security, legal, and other issues with shifting computing to the cloud.
When people think of Lawrence Berkeley National Laboratory, where I worked for more than two decades, they often think of huge supercomputers and really smart computer scientists, which are two hallmarks of that institution. LBNL has no shortage of people who know computing, but even that pinnacle of computing innovation decided in the last few years to shift its email, calendar, and other routine computing services to the cloud. That to me says that even very technically sophisticated institutions have good reasons for shifting some of their computing services to the cloud, and those reasons will only become more numerous and compelling as the years progress.
“The debt was $10.626 trillion on the day Mr. Obama took office. The latest calculation from Treasury shows the debt has now hit $14.639 trillion.
It’s the most rapid increase in the debt under any U.S. president.
The national debt increased $4.9 trillion during the eight-year presidency of George W. Bush. The debt now is rising at a pace to surpass that amount during Mr. Obama’s four-year term.”
For technical reasons the real number as calculated by CBS news should have been just the debt owed by the US government to creditors, excluding the debt owed to itself for social security, and that’s about $3.7T.
But that’s not the big problem in these claims.
First, as TPM points out, the relevant metric is the size of the debt relative to the size of the GDP, not the absolute nominal amount of debt.
Second, there’s the question of causality. All statisticians know that assigning causality is often difficult, but that’s really the crux of the matter. If you can assign causality accurately you’ve got the gold standard of proof. And in that regard this report is deeply flawed.
The report implies that Obama’s policies are to blame for this increase in debt, and certain of his policies contributed, but they are small compared to the bigger structural problems he inherited. For example, one can fairly assign Obama responsibility for the stimulus package in 2009, the bailout of the auto companies, the end of year 2010 tax deal (which extended the Bush tax cuts), and the costs of war in Afghanistan in 2009 and 2010 (which he argued in support of in the campaign, and which he continued and expanded).
But Obama inherited an economy in freefall, and it turned out to be significantly worse than what the Congressional Budget Office initially projected in early 2009. He also inherited the Iraq war, which he strenuously argued against and it almost certainly wouldn’t have happened had he been president (he wound it down as quickly as he could, but those things take time to do responsibly). He inherited TARP (which has largely been paid back) and the AIG bailout (which still will cost us). He inherited Medicare Part D, which was not paid for. He also inherited the Bush tax cuts, which were slated to expire at the end of 2010 (and also weren’t paid for). Surely President Obama shouldn’t be assigned responsibility for the deficits induced by the tax cuts (except post 2010, when he extended the cuts in a deal with Republicans, but even then, he got some things in return for that deal that need to be weighed against those costs). I also suspect that some of this increase in debt is related to the honest accounting of the costs of the Iraq and Afghan wars, which were treated by the previous administration as “off budget” (perhaps someone with more knowledge of the budget process can illuminate us about that).
And remember that when Obama came into office in January 2009 the government was operating under budgets approved by the previous congress and administration, so it’s not reasonable to argue that the first month or two of his administration Obama is responsible for the debt. He’s also clearly not responsible for the unemployment figures in February 2009, which were the worst in 25 years. At some point (probably starting in May or June of 2009) he can reasonably be assigned responsibility but it’s not reasonable to argue that events in the first couple of months of his term were clearly caused by his policies, because they hadn’t been implemented yet. This was the same argument Ronald Reagan made when he came into office–it was correct then and it is correct now, as any fair minded observer must admit.
So whatever your political persuasion might be, there is a reasonable and fair minded way to look at this issue, and by any measure, the idea that President Obama’s policies added to the debt in an unprecedented way is simply false (for a text and graphical presentation of that result go to this Wikipedia entry on the national debt and a graph from the NYT based on Congressional Budget Office data). Those who imply otherwise are arguing against the laws of arithmetic, and they just embarrass themselves by doing so. One can reasonably argue against some of the President’s policies but disputing basic math and confusing elementary assignation of causality is something I’d expect from middle school students, not our elected representatives (but of course, I’m an optimist).
Finally, there is a separate question of what the debt is being used to do. Not all debt is created equal. Debt that is used to finance investments (like improved infrastructure, research and development (R&D), or education) is very different than that used to fund consumption (like wars or consumer spending). More debt to fund infrastructure that will yield returns to the economy in faster shipping or travel times is clearly worth doing when the private sector won’t fund these things. Same for educating the populace to do more skilled jobs or funding basic R&D. The justification for such investments is well established in economics, because they yield “public goods”, and the private sector will underinvest in those (because no one actor can exclude others from benefitting from those investments).
Distinguishing between the costs and benefits of debt used to do different things may be too much to ask (alas), but at least let’s agree that the laws of arithmetic are valid and that facts are facts. My granddad used to say that if his employees sent him budgets and 2 + 2 didn’t equal 4, he’d send them back until things added up, and that’s what we should do with this CBS report and the people trying to make political hay with it.
Addendum October 11, 2011: I just saw this nice chart summarizing the contribution of the various Bush administration policies for the national debt, which reinforces the arguments I made above.
Addendum October 21, 2011: Talking Points Memo posted a wonderful chart of the sources of US government revenue over the past 60 years, which I repost below. It shows significant declines in corporate and excise taxes, and an increase in payroll taxes. The explanation of the graph is here.
This is an outrage to data lovers everywhere: The Statistical Abstract of the US (as well as other compendia of statistics created by the census department) is on the chopping block and will be eliminated after the 2012 edition is published. Robert J. Samuelson, economics columnist for the Washington Post, writes about this travesty here. This is just ridiculous. The Stat. Ab. is an essential data book for US researchers, and it can’t be replaced by other sources. Write to Congress, folks!
A NY Times article appeared today on the effect of the new lighting efficiency regulations and it describes the complexity facing consumers as new technologies enter the market. What this article first made me realize was that the advent of the new efficiency standards is a real world example of the “Porter hypothesis” in action. Michael Porter at Harvard Business school has postulated that strict environmental (or energy efficiency) regulations can under certain conditions lead to increased innovation. This hypothesis is contrary to what I call the simpleminded Econ 101 view of the world, in which regulations are invariably less efficient than pricing mechanisms in inducing innovation. In any case, these particular regulations have clearly induced innovation in an industry that has been famously slow to change. Having old depreciated plants churning out incandescent bulbs is extremely profitable, so there was little incentive for the lighting companies to innovate except in certain niche markets driven by other efficiency policies (like utility efficiency programs promoting efficient lighting in California).
I’ll say more about the high level implications of such regulations in another post, but I wanted to describe our own personal experience with new lighting technologies for our new house. There are a lot of ceiling cans in the house, and the previous owner had used CFLs in most of them. The decorative parts of the cans were original equipment, about 12 years old, so they looked dingy and needed to be replaced. Our contractor said that would cost about $20 per can. He brought us a new LED downlight and said it cost $50 and could quickly be placed into every can. They use 11W and give off as much light as a 60W incandescent, but our perception was that they were much brighter than a typical bulb because they were directional. They come on very rapidly and can be used on dimmers with no problem (for some reason they don’t come on quite as quickly when dimmed, but are about as fast as incandescents in turning on when at full brightness). We tried the light in our old house to see if the color rendition was much different than an incandescent bulb, and we couldn’t see any difference. My wife even gave them her seal of approval (she hates CFLs and would never have allowed them except in a few fixtures). Finally, the long life of the LEDs (35,000 hours, or 35 years at 3 hours/day of use) was really important to us because we have high ceilings in the new house, and in a house with 48 ceiling cans that’s a lot of trips up and down a ladder to replace burned out incandescents. I had no interest in wasting my time doing that, so the LEDs were just the ticket. And we would have had to spend $20 anyway to replace the decorative parts of the can, so the economics were a lot better than in some other applications.
Here’s a link to the latest version of the lights we chose (updated July 2013). Definitely stick with major manufacturers for new technologies like LEDs. We had one out of 48 LEDs go bad on us, but that one was promptly returned for an exchange. We were told that the newer LEDs have a square LED active surface, whereas most of the older ones use individual high intensity LEDs that are shaped like the ones on your stereo (but are of course much more luminous). Go for the square LED active surface.
Finally, a note to people who are upset by the changes induced by the new regulations: you’ll still get to buy incandescents, they’ll just be lots more efficient. Eventually LEDs will sweep all the other technologies away, and I’d expect that to happen in 5 to 10 years, but we’ll see. The main lesson of rapid adoption of efficiency technologies over time is that they need to be better than what they replace to gain wide acceptance. That’s why the good light of LEDs combined with the much longer lifetime and lower heat output makes them such a powerful competitor to the standard bulbs (as well as to CFLs). And with most electronic devices, prices pretty much always go down, and we’re nowhere near the theoretical limits of efficiency for lighting, so more progress lies ahead. Exciting times!
I just released my new study on data center electricity use in 2010. I did the research as an exclusive for the New York Times, and John Markoff at the Times wrote an article on it that will appear in the print paper August 1, 2011. You can download the new report here.
Key findings:
• Assuming that the midpoint between the Upper and Lower bound cases accurately reflects the history, electricity used by data centers worldwide increased by about 56% from 2005 to 2010 instead of doubling (as it did from 2000 to 2005), while in the US it increased by about 36% instead of doubling.
• Electricity used in global data centers in 2010 likely accounted for between 1.1% and 1.5% of total electricity use, respectively. For the US that number was between 1.7 and 2.2%.
• Electricity used in US data centers in 2010 was significantly lower than predicted by the EPA’s 2007 report to Congress on data centers. That result reflected this study’s reduced electricity growth rates compared to earlier estimates, which were driven mainly by a lower server installed base than was earlier predicted rather than the efficiency improvements anticipated in the report to Congress.
• While Google is a high profile user of computer servers, less than 1% of electricity used by data centers worldwide was attributable to that company’s data center operations. To my knowledge, this is the first time that Google has revealed specific details about their total data center electricity use (they gave me an upper bound, not an exact number, but something is better than nothing!).
In summary, the rapid rates of growth in data center electricity use that prevailed from 2000 to 2005 slowed significantly from 2005 to 2010, yielding total electricity use by data centers in 2010 of about 1.3% of all electricity use for the world, and 2% of all electricity use for the US.
The new study is a follow-onto my 2008 article:
Koomey, Jonathan. 2008. “Worldwide electricity used in data centers." Environmental Research Letters. vol. 3, no. 034008. September 23. <http://stacks.iop.org/1748-9326/3/034008>
Suggested citation: Koomey, Jonathan. 2011. Growth in Data center electricity use 2005 to 2010. Oakland, CA: Analytics Press. August 1. <http://www.analyticspress.com/datacenters.html>
I can think of four reasons why cloud computing is (with few exceptions) significantly more energy efficient than using in-house data centers:
1) Economies of scale: It’s cheaper for bigger cloud computing folks to make efficiency improvements because they can spread the costs over a larger server base and can afford to have more dedicated folks focused on efficiency improvements. For example, there are usually significant fixed costs of implementing simple techniques to improve Power Usage Effectiveness (PUE), like the costs of doing an equipment inventory and assessment of data center airflow (same for implementing institutional changes like charging users per kW instead of per square foot of floor area). Whenever there are costs that are substantially fixed (i.e. only weakly related to the size of the facility), bigger operations have an advantage because they can spread the costs over more transactions, equipment, or floor area. There’s also a substantial advantage to having “in house” expertise devoted to efficiency, instead of having staff split between different jobs–technology changes so rapidly that it’s hard for people not devoted to efficiency to keep up as well as those that are.
2) Diversity and aggregation: More users, more diverse users, and more users in different places means computing loads are spread over the day, allowing for increased equipment utilization. Typical in house data centers have server utilizations of 5-15% and sometimes much less, whereas cloud facilities for major vendors are more in the 30-40% range.
3) Flexibility: Cloud installations use virtualization and other techniques to separate the software from the characteristics of physical servers (some call this “abstraction of physical from virtual layers”). This sounds like a great thing for software and total costs, but why is it an energy issue? Using this technique means that you can redesign servers to optimize them and drop certain energy costly features. For example, if software can route around physical servers that die, you no longer need to have two power supplies in each server–the death of any one particular server doesn’t matter to the delivery of IT services. In essence, this technique redefines the concept of reliability from one that is based on the reliability of a particular piece of hardware to one that is based on the reliability of the delivery of the IT services of interest, and this is a much more sensible approach.
4) Ease of sidestepping organizational issues instead of having to address them head on (which is hard and slow): For example, the problem of IT driving server purchases but facilities paying the electric bill is still a big issue for most in-house facilities, but it has largely been solved for the cloud providers (they generally have one data center budget and clear responsibilities assigned to one person with decision making authority). Economies of scale are more powerful in this scenario, because you’ve gotten rid of the impediments to taking action and can allow those economies to work their magic. Finally, it’s much easier and cheaper for people stuck with the in-house organizations to use a credit card to buy cloud services instead of waiting around for their internal IT organization to get its act together.
These big energy advantages will over time translate into more and more pressure for companies to adopt cloud services, because the economic advantages (driven by the energy advantages) are so large. And it’s not just energy costs, it’s the capital cost of all the supporting equipment, which in a standard in-house facility can be $25,000/kW and (together with the energy costs) add up to half or more of the total costs of the facility (for details see Koomey, Jonathan G., Christian Belady, Michael Patterson, Anthony Santos, and Klaus-Dieter Lange. 2009. Assessing trends over time in performance, costs, and energy use for servers. Oakland, CA: Analytics Press. August 17.)
Of course, there are still issues to work out. For example, people haven’t really ironed out the complexities about liability for cloud outages. And there will always be providers who will want to have their own in-house facilities for security reasons (like big financial institutions). But even in that case, the benefits of a virtualized cloud infrastructure can be brought to the in-house facilities. You won’t get the same diversity but the other benefits of cloud will still be powerful. I’ve also heard of companies creating “private clouds” for use by other companies that pay in to use them on a “members-only” basis, thus dealing with the diversity and security issues. So things are evolving rapidly, but the economic benefits are so large that we’ll see a whole lot more cloud computing in coming years.
Gary Gutting just posted a well-reasoned article about how to think about the expert consensus in the climate debate. He begins by noting that logical claims based on authority only have standing if there is a recognized means of determining who is an expert and who isn’t (see also Chapter 14 in Turning Numbers into Knowledge).
For climate, the experts can easily be identified:
“All creditable parties to this debate recognize a group of experts designated as ‘climate scientists,’ whom they cite in either support or opposition to their claims about global warming. In contrast to enterprises such as astrology or homeopathy, there is no serious objection to the very project of climate science. The only questions are about the conclusions this project supports about global warming.”
His concluding paragraph sums up the implications for his line of argument:
“…once we have accepted the authority of a particular scientific discipline, we cannot consistently reject its conclusions. To adapt Schopenhauer’s famous remark about causality, science is not a taxi-cab that we can get in and out of whenever we like. Once we board the train of climate science, there is no alternative to taking it wherever it may go.”
Another way to make this argument when confronted by a climate denier is to ask them “do you feel qualified to dispute the latest developments in quantum physics or thermodynamics? If not, what makes you think you are qualified to debate the latest climate science?” For someone who is actually qualified to make judgments about the fields to which you refer you can ask them about how likely it is that someone from another field could accurately critique his/her field even if that person is qualified in another scientific field. The answer in all but the rarest of cases is “not bloody likely”.
Another angle on this line of argument is to use the peer reviewed literature compiled on Skeptical Science. Virtually every argument that the deniers make is analyzed and debunked there, so when someone pesters me with their denialism I say “there’s an app for that!” and pull out Skeptical Science. When I can quickly pull up the critique of their claim and explain its implications it often has an impact.
The New York Times reports that the World Bank has released huge amounts of its previously proprietary data to the general public. The Bank talks about it in a news release here and on its data page here.
The key data are in the form of indicators of development and economic activity. For a list of the most popular indicators, go here. You’ll find data on GDP, finance, business activities, population, infrastructure, environmental insults, energy use, and much more.
For people I call “ecological entrepreneurs” (i.e. those who want to make the environment better while making a profit in the bargain) the World Bank database is truly a treasure trove. If you are considering a venture of this type (particularly one that involves commerce in countries outside the US), I heartily recommend digging into these data.
Of course, as with all data, you’ll need to check it for both internal consistency and accuracy, but having more data is always better, and having good data is better still.
