Inside Climate did a nice job summarizing the past 70 years of climate denial by fossil fuel interests, including links to key historical references. Read it here. They also made this visual, which is helpful.
Awhile back we stopped at Duarte’s Tavern in Pescadero, CA, which was founded roughly a century ago (I think the building predates the restaurant). They still had a couple of non-functional gas lamps on the wall, with wicks and everything. I took a couple of photos, which I show below. They look a lot like Coleman lanterns (roughly the same tech).
PS. We highly recommend Duarte’s for pie, grilled fish, and their amazing crab melts, among other things. If in the neighborhood, be sure to stop by.
Yesterday (May 31, 2018) I talked at the Foothill College Solutions conference about climate solutions, summarizing what I’ve learned in three decades of studying how to solve the climate problem. Here’s the main message:
We need to reduce emissions as much as possible, as fast as possible, starting as soon as possible. Everything else is noise.
I focus on the need for urgent action and the tools that give me hope that we can ultimately solve this problem. You can download a PDF of the talk here.
December 8, 2017. Beginner’s guide series on cryptoassets. Medium. This is a list of guides by the author, including beginner’s guides to: Ethereum, Monero, Litecoin, Ethereum tokens, 0x, Tezos, Bitcoin Cash, Decred, Zcash, and IOTA.
Because of recent interest in the electricity use of Bitcoin mining, I asked my colleague Zach Schmidt to create a reasonably comprehensive list of articles and reports on this topic. As we update this list we’ll create new blog posts so that the latest version is usually at the top of the heap on Koomey.com.
Please email me if you see a new article that seems especially well written and authoritative and we’ll add it to the list. There’s precious little credible information on this topic nowadays, but I hope that will change soon. My own summary of the key issues is still in process, but hopefully this list of articles will be of use in the meantime.
The most credible academic estimates are at the bottom of this post. Not surprisingly, the news articles are by far the majority of writing on this topic.
“It was in that moment that I realized that if our children look back to how we failed them, it will not have been for lack of scientific understanding or even technological prowess; it will have been due, fundamentally, to cowardice. A profound cowardice among those who actually do have a choice in this matter, a cowardice that confuses arrogance with intelligence, pettiness with importance, and, most fatally, comfort with necessity. “
The biggest current problem with data centers nowadays is poor capacity utilization (Shehabi et al. 2016). The most widely-cited example is that of comatose servers, in which 20-30% of servers in typical enterprise data centers are using electricity but doing nothing useful (Koomey and Taylor 2015). Beyond comatose servers, many more servers are used less than 5% of the year (Kaplan et al. 2008). A related problem is that many servers are far too powerful for the jobs they do, with more memory, processors, and hard drives than they can possibly use productively (I call this “over-provisioning”).
Until recently, this problem of poor capital utilization in the data center has been treated as an unavoidable cost of doing business (Koomey 2014). Most enterprise facilities (which I define as data centers run by companies whose primary business is not computing) suffer from these problems, largely because of institutional failures. For example, in enterprise data centers, the information technology department usually buys the computers but the facilities department buys the cooling and power distribution equipment and pays the electric bills. The business unit demanding the computers doesn’t care about the details, they just want to roll out their newest project. Those departments all have separate budgets and separate bosses, and rarely, if ever, optimize for the good of the company. In many cases, these companies can’t even tell you how many servers they have, never mind what their utilization is (Koomey 2016).
The big cloud computing providers have done a much better job than the enterprise data centers, with far fewer comatose servers, higher server utilization, and better matching of server capabilities to the loads they serve. These providers also track their equipment inventories much more carefully than do traditional enterprise facilities.
Data centers have become increasingly important to the US economy, and many more companies are confronting the massive waste of capital inherent in traditional ways of organizing and operating data centers. An increasing number are moving workloads to cloud computing providers, instead of building or expanding “in-house” data center facilities (Shehabi et al. 2016). As more consider this option, it’s important to understand the steps needed to make such a move in a cost-effective way.
Because of their economies of scale and other advantages, cloud providers can generally deliver computing services at lower costs than is possible in enterprise facilities, but because of poor capacity utilization in existing facilities, just mapping existing servers onto a cloud provider’s infrastructure will not necessarily lead to cost savings.
Consider Figure 1, which shows data from TSO Logic for data center installations of varying sizes, moving from a base case (a traditional data center) to a cloud computing facility, mapping those server “instances” on a one-for-one basis, without changing anything about them.
Server instances can be virtual or physical. In these case these are virtual instances, in which many instances run on a smaller number of physical machines. There are more than 100,000 virtual server instances in this data set.
Each red square represents a single installation (in one case, a single dot stands for five installations). The X-axis shows the number of instances in each installation, ranging from hundreds to almost 35,000. On the Y-axis, I’ve plotted the savings in annualized total data center costs when existing instances are mapped one-for-one onto the cloud, exactly replicating their utilization, the amount of memory, and the number of processors for each instance. These costs include cooling infrastructure capital, electricity, space, amortization, operating system licenses, and maintenance expenses.
Figure 1: Number of instances in each of 15 installations versus % savings to map them one-for-one into the cloud
The most important finding from this exercise is that a simple mapping of server instances to the cloud does not guarantee cost savings. In fact, only one third of installations show cost savings when doing such a mapping, with two thirds showing an increase in costs (i.e., negative savings).
The outlier in Figure 1 is the installation with about 2,000 instances and a negative savings (increase in cost) of more than 200% when workloads are mapped one-for-one onto the cloud. This case results from heavily over-provisioned instances combined with very low utilization for the vast majority of them.
Now consider Figure 2, which shows the some of the same data as in Figure 1, but plotted in a slightly different way. On the X-axis, I’ve plotted the percentage reduction in the number of instances in moving from the base case to either the “one-to-one mapping” or optimized cases. In the first case, there is no change in the number of instances, so all of those red squares are plotted at 0% on the X-axis, with the savings in annual costs exactly matching those shown in Figure 1. In the second case, I’ve plotted blue triangles for the optimized mapping onto cloud resources, which in almost every case results in elimination of large fractions of the instances existing in the base case.
Figure 2: % annual cost savings to map 15 installations one-for-one into the cloud, compared to optimizing them and eliminating unused instances before transferring to the cloud
Interestingly, the optimized case shows that every one of the 15 installations show significant cost savings compared to the base case, which comes about by eliminating comatose servers, consolidating low utilization servers, and “right sizing” server instances to use the correct amounts of memory and processor power. In all but one example, the optimized case results in the elimination of 60 to 90% of server instances.
For the single example where only about 10% of instances are eliminated in the optimized case (the blue triangle that is far to the left of all the other triangles), cost savings were achieved by “right sizing” the server instances, with only a small reduction in the number of server instances. For this example, optimization turns a 9% cost increase in the “one-to-one mapping” case into about 36% savings, which is mostly due to right sizing of those instances.
It is important to note that the optimized case does not reflect re-architecting software design to better utilize modern computing resources, which can take months to years. This means the optimized case truly reflects the low hanging fruit that can be harvested quickly. Modernizing software design can yield substantial additional benefits.
In addition, the shift to cloud can result in more than cost savings. Modern tools for workloads run in the cloud can speed time to market and increase the rate of innovation (Schuetz et al. 2013). These benefits can swamp the direct cost savings, but are not tallied here.
Summary
Many organizations can benefit by shifting their computing workloads to the cloud, but a smart strategy is to optimize those workloads first before making the switch. The data reviewed here show that such optimization can yield substantial benefits (saving more than one third of annual data center costs) and make it most likely that the shift to cloud will be a successful one.
Gartner estimates that $170B/year is spent globally by business on “data center systems”.[1] If the shift to cloud computing can reduce these costs by even 25%, that means savings of about $40 billion flowing directly to companies’ bottom lines every year, which are big savings by any measure.
This report on TSO Logic’s data was conducted by Jonathan Koomey on his personal time. TSO Logic contributed the data (under NDA) and helped explain the data and calculations but did not contribute financially to the creation of this blog post.
[1]Gartner Forecast Alert: IT Spending, Worldwide, 3Q17 Update, 29 September 2017 [https://www.gartner.com/technology/research/it-spending-forecast/]
Koomey, Jonathan, and Patrick Flynn. 2014. “How to run data center operations like a well oiled machine.” In DCD Focus. September/October 2014. pp. 81. [http://goo.gl/7sJHZb]
Masanet, Eric R., Richard E. Brown, Arman Shehabi, Jonathan G. Koomey, and Bruce Nordman. 2011. "Estimating the Energy Use and Efficiency Potential of U.S. Data Centers." Proceedings of the IEEE. vol. 99, no. 8. August.
Shehabi, Arman, Sarah Josephine Smith, Dale A. Sartor, Richard E. Brown, Magnus Herrlin, Jonathan G. Koomey, Eric R. Masanet, Nathaniel Horner, Inês Lima Azevedo, and William Lintner. 2016. United States Data Center Energy Usage Report. Berkeley, CA: Lawrence Berkeley National Laboratory. LBNL-1005775. June. [https://eta.lbl.gov/publications/united-states-data-center-energy]
I first wrote this book to train young analysts I was hiring when I was a Staff Scientist and Group Leader at Lawrence Berkeley National Laboratory. I found that even graduates of the best engineering schools still had a lot to learn about the art of analysis when they came to work for me. This book teaches them those skills in a readable and entertaining way, and I’ve used it to train hundreds of students and professionals since the first edition came out in 2001.
The book is composed of 39 short chapters, each covering a topic I think is important to beginning analysts. There are lots of quotations, examples, graphics, figures, and some of my favorite cartoons, so you’ll have a few chuckles as you read along. College students and advanced high school students should have no problem learning from it–the text is well edited and has been honed over the years.
For this edition, I’ve updated many examples, tightened up the text, and made sure the material related to the Internet was fully up to date.
Please email me if you have questions or comments.
Episode 52 of Chris Nelder’s Energy Transition show was posted September 20, 2017, and it features yours truly talking frankly about nuclear power economics, the role of storage, 100% renewable energy scenarios, and limits of economic models, among other things. I’m pleased to say we achieved a “geek rating” of 8.
Here’s Chris’s description of the show:
It’s the two-year anniversary of the Energy Transition Show, so we thought we’d take a break from the deep dives and just have a little fun skiing around on the surface for a change. Dr. Jonathan Koomey returns to the show for a freewheeling discussion about some of the interesting questions and debates swirling around the energy transition today, and hopefully help us glue together many of the themes that have emerged from our first 51 shows.
How do you go about an energy revolution? Is 100% renewables the right goal? How much seasonal storage will a high-renewables grid need? What will it cost? Is there a future for nuclear power? Or CCS? What should get the credit for declining U.S. emissions? How do we model the best pathways to a future of clean and sustainable energy? Can the IPCC modeling framework be fixed? What kind of carbon mitigation pathways should we be projecting? And how should we communicate the important messages on climate and energy transition? We tackle all these questions in one big omnibus episode.
The Energy Transition show is a paid subscriber podcast (so not a freebie), but if you are really interested in digging into issues about making a rapid transition to a zero emissions world, you owe it to yourself to subscribe.
Our previous work on trends in the efficiency of computing showed that computations per kWh at peak output doubled every 1.6 years from the mid 1940s to around the year 2000, then slowed to a doubling time of 2.6 years after 2000 (Koomey et al. 2011, Koomey and Naffziger 2016). These analyses examined discrete computing devices, and showed the effect (mainly) of progress in hardware.
The slowing in growth of peak output efficiency after 2000 was the result of the end of the voltage reductions inherent in Dennard scaling, which the chip manufacturers used to keep power use down as clock rates increased (Bohr 2007, Dennard et al. 1974) until about that time. When voltages couldn’t be lowered any more, they turned to other tricks (like multiple cores) but they still couldn’t continue improving performance and efficiency at the historical rate, because of the underlying physics.
Unlike that for computing devices, the literature on the electricity intensity and efficiency of network data flows has been rife with inconsistent comparisons, unjustified assumptions, and a general lack of transparency. Our attempt to remedy these failings was just published in the Journal of Industrial Ecology in August 2017 (Aslan et al. 2017). The focus is on the electricity intensity of data transfers over the core network and the access networks (like DSL and cable).
Here’s the summary of the article:
In order to understand the electricity use of Internet services, it is important to have accurate estimates for the average electricity intensity of transmitting data through the Internet (measured as kilowatt-hours per gigabyte [kWh/GB]). This study identifies representative estimates for the average electricity intensity of fixed-line Internet transmission networks over time and suggests criteria for making accurate estimates in the future. Differences in system boundary, assumptions used, and year to which the data apply significantly affect such estimates. Surprisingly, methodology used is not a major source of error, as has been suggested in the past. This article derives criteria to identify accurate estimates over time and provides a new estimate of 0.06 kWh/GB for 2015. By retroactively applying our criteria to existing studies, we were able to determine that the electricity intensity of data transmission (core and fixed-line access networks) has decreased by half approximately every 2 years since 2000 (for developed countries), a rate of change comparable to that found in the efficiency of computing more generally.
The rate of improvement is actually faster than in computing devices, but this result shouldn’t be surprising, because the aggregate rates of improvement in data transfer speeds and total data transferred are dependent on progress in both hardware and software. Koomey and Naffziger (2016) and Koomey (2015) showed that other metrics for efficiency can improve more rapidly than peak output efficiency if the right tools are brought to bear on those problems.
Email me if you’d like a copy of the new article, or any of the others listed below.
Bohr, Mark. 2007. “A 30 Year Retrospective on Dennard’s MOSFET Scaling Paper.” IEEE SSCS Newsletter. vol. 12, no. 1. Winter. pp. 11-13.
Dennard, Robert H., Fritz H. Gaensslen, Hwa-Nien Yu, V. Leo Rideout, Ernest Bassous, and Andre R. Leblanc. 1974. “Design of Ion-Implanted MOSFET’s with Very Small Physical Dimensions.” IEEE Journal of Solid State Circuits. vol. SC-9, no. 5. October. pp. 256-268.
Koomey, Jonathan. 2015. “A primer on the energy efficiency of computing.” In Physics of Sustainable Energy III: Using Energy Efficiently and Producing it Renewably (Proceedings from a Conference Held March 8-9, 2014 in Berkeley, CA). Edited by R. H. Knapp Jr., B. G. Levi and D. M. Kammen. Melville, NY: American Institute of Physics (AIP Proceedings). pp. 82-89.
One of the most surprising things about the data center industry is how cavalier it is about the number of servers sitting around using electricity but doing nothing. We call such servers “comatose”, or more colorfully, “zombies”.
In 2015 we did our first study of this issue using granular analysis on a small data sample (4000 servers) for a six month period in 2014, using data from TSO Logic. Now we’re back with a sample four times bigger, covering six months in 2015, and with additional detail on the characteristics of virtual machines.
My colleague at Anthesis, Jon Taylor (with whom I conducted the study) wrote up a nice summary of the work here. You can also download the study at that link.
Here are a few key paragraphs:
Two years on the data set from which the original findings were drawn has grown from 4,000 physical servers to more than 16,000 physical servers and additional information on 32,000 virtual machines (VM) running on hypervisors. The new findings show improvements, as well as an alarming wake-up call.
On the upside: when an enterprise acted to remove physical zombie servers when presented with evidence of the problem’s magnitude, they were able to reduce the amount from 30 percent to eight percent in just one year. On the downside: new data show that some 30 percent of VMs are zombies, demonstrating that the same discovery, measurement, and management challenges that apply to physical servers also apply to VMs.
The study confirms that the issue is still not being adequately addressed. New data indicates that one quarter to one third of data center investments are tied up with zombie servers, both physical and virtual. Virtualization without improved measurement technologies and altered institutional practices is not a panacea. Without visibility into the scale of these wasted resources the problem will continue to challenge the data center industry.
Here’s a key graph from the report:
There are some complexities in comparing the new data with the older data, because one facility in the 2014 sample decided not to allow its data to be used for the 2015 sample. The remaining facilities in the 2014 sample, when shown evidence that one third of their servers were comatose, took action and moved from more than 30% comatose to 8% comatose in just one year.
We corrected for these changes in an attempt to estimate the percent of comatose servers for enterprises that haven’t dealt with the problem, and the result is an estimate that about one quarter of servers in such companies are comatose (see the middle bar of the figure above).
Surprisingly, the percentage of virtual machines that were comatose was about 30% (see the right most bar above), indicating that the same management failures that lead to high percentages of comatose servers also afflict virtual machines. Virtualization without institutional changes is not a panacea!
One new issue raised in the latest report is important but often overlooked. Zombie servers are likely to not have been updated with the latest security patches, so they present a potent risk to the safety of the data center. Find them and remove them as soon as you can!
The US EIA Monthly Energy Review (Table 7.2a) includes data on net generation for the US (utilities plus independent generators), and from those data we can say something interesting about what was driving declines in coal generation from 2015 to 2016. The conventional wisdom is that cheap natural gas is the main driver of this decline, but that’s not true, at least in the 2015 to 2016 time frame.
I downloaded the data from the EIA web site, and converted the net generation numbers to billion kWh (equivalent to terawatt-hours) in my excel workbook. I then calculated the change from 2015 to 2016 and combined pumped storage hydroelectricity with conventional hydro.
The net result is shown in the following figure:
The graph shows that natural gas was responsible for offsetting about 40% of the total decline in coal and petroleum generation, but that wind plus solar displaced about the same amount. So natural gas is an important part of the story, but the other alternatives (including nuclear and hydro) are in the aggregate more important. With the recent exponential increases in installed capacity of solar and wind, they are destined to become much more important in short order.
It is not clear to me whether the growth in rooftop solar generation is included in these numbers. I know that EIA has been working to get better data on that sector, but sometimes the data gears churn slowly. If you can answer that question, please contact me!
Another important factor not accounted for here is that net generation and electricity consumption in the US has been flat since 2007 (Hirsh and Koomey 2015), which indicates decoupling between electricity demand and GDP. In the years since the mid 1990s this decoupling has been pronounced, while from 1973 to the mid 1990s electricity consumption grew in lockstep with GDP.
GDP has grown about 2%/year on average since 2010, and if we apply that 2% to net generation in 2015 we can estimate that electricity demand is about 80 B kWh lower than it would have been without that decoupling. That shift in demand (which is a function of both efficiency and structural change) is bigger than any of the other single contributors to changes in generation in the figure. For more discussion of these issues, see Hirsh and Koomey 2015.
