Google data centers

Google Data Centers are the large data center facilities Google uses to provide their services, which combine large drives, computer nodes organized in aisles of racks, internal and external networking, environmental controls (mainly cooling and dehumidification), and operations software (especially as concerns load balancing and fault tolerance).

Former Google data center in Eemshaven, Netherlands

There is no official data on how many servers are in Google data centers, but Gartner estimated in a July 2016 report that Google at the time had 2.5 million servers. This number is changing as the company expands capacity and refreshes its hardware.[1]

Locations
Google data center in The Dalles, Oregon

The locations of Google's various data centers by continent are as follows:[2]

North America:

  1. Berkeley County, South Carolina (33°03′50.8″N 80°02′36.1″W) — since 2007, expanded in 2013, 150 employees
  2. Council Bluffs, Iowa (41°13′17.7″N 95°51′49.92″W) — announced 2007, first phase completed 2009, expanded 2013 and 2014, 130 employees
  3. Douglas County, Georgia (33°44′59.04″N 84°35′5.33″W) — since 2003, 350 employees
  4. Bridgeport, Jackson County, Alabama (34°54′48.4″N 85°44′53.1″W)[3][4]
  5. Lenoir, North Carolina (35°53′54.78″N 81°32′50.58″W) — announced 2007, completed 2009, over 110 employees
  6. Montgomery County, Tennessee (36°37′37.7″N 87°15′27.7″W) — announced 2015
  7. Mayes County, Oklahoma at MidAmerica Industrial Park (36°14′28.1″N 95°19′48.22″W) — announced 2007, expanded 2012, over 400 employees[5]
  8. The Dalles, Oregon (45°37′57.04″N 121°12′8.16″W) — since 2006, 80 full-time employees
  9. Henderson, Nevada [6]
  10. Reno, Nevada — announced in 2018 : 1,210 acres of land bought in 2017 in the Tahoe Reno Industrial Center;[7] project approved by the state of Nevada in November 2018[8][9]

South America:

  1. Quilicura, Chile (33°21′30.5″S 70°41′50.4″W) — announced 2012, online since 2015, up to 20 employees expected. A million investment plan to increase capacity at Quilicura was announced in 2018.[10]
  2. Cerrillos, Chile - announced for 2020[11]
  3. Colonia Nicolich, Uruguay - announced 2019[12][13][14]

Europe:

  1. Saint-Ghislain, Belgium (50°28′09.6″N 3°51′55.7″E) — announced 2007, completed 2010, 12 employees
  2. Hamina, Finland (60°32′11.68″N 27°7′1.21″E) — announced 2009, first phase completed 2011, expanded 2012, 90 employees
  3. Dublin, Ireland (53°19′12.39″N 6°26′31.43″W) — announced 2011, completed 2012, 150 employees[15]
  4. Eemshaven, Netherlands (53.425659°N 6.8593522°E / 53.425659; 6.8593522) — announced 2014, completed 2016, 200 employees
  5. Agriport, Netherlands - announced 2019[16]
  6. Fredericia, Denmark — announced 2018, planned completion in 2021[17]
  7. Zürich, Switzerland - announced in 2018, completed 2019[18]
  8. Warsaw, Poland - announced in 2019, planned completion in 2021[19]


Asia:

  1. Jurong West, Singapore (1°21′04.8″N 103°42′35.2″E) — announced 2011, completed 2013
  2. Changhua County, Taiwan (24°08′18.6″N 120°25′32.6″E) — announced 2011, completed 2013, 60 employees
  3. Mumbai, India — announced 2017, completed 2019[20]
  4. Tainan City, Taiwan — announced September 2019[21][22][23]
  5. Yunlin County, Taiwan — announced September 2020[24]

Hardware

Original hardware
Google's first production server rack, circa 1998

The original hardware (circa 1998) that was used by Google when it was located at Stanford University included:[25]

  • Sun Microsystems Ultra II with dual 200 MHz processors, and 256 MB of RAM. This was the main machine for the original Backrub system.
  • 2 × 300 MHz dual Pentium II servers donated by Intel, they included 512 MB of RAM and 10 × 9 GB hard drives between the two. It was on these that the main search ran.
  • F50 IBM RS/6000 donated by IBM, included 4 processors, 512 MB of memory and 8 × 9 GB hard disk drives.
  • Two additional boxes included 3 × 9 GB hard drives and 6 x 4 GB hard disk drives respectively (the original storage for Backrub). These were attached to the Sun Ultra II.
  • SDD disk expansion box with another 8 × 9 GB hard disk drives donated by IBM.
  • Homemade disk box which contained 10 × 9 GB SCSI hard disk drives.

Production hardware

As of 2014, Google has used a heavily customized version of Debian (GNU/Linux). They migrated from a Red Hat-based system incrementally in 2013.[26]

The customization goal is to purchase CPU generations that offer the best performance per dollar, not absolute performance. How this is measured is unclear, but it is likely to incorporate running costs of the entire server, and CPU power consumption could be a significant factor.[27] Servers as of 2009–2010 consisted of custom-made open-top systems containing two processors (each with several cores[28]), a considerable amount of RAM spread over 8 DIMM slots housing double-height DIMMs, and at least two SATA hard disk drives connected through a non-standard ATX-sized power supply unit.[29] The servers were open top so more servers could fit into a rack. According to CNET and a book by John Hennessy, each server had a novel 12-volt battery to reduce costs and improve power efficiency.[28][30]

According to Google, their global data center operation electrical power ranges between 500 and 681 megawatts.[31][32] The combined processing power of these servers might have reached from 20 to 100 petaflops in 2008.[33]

Network topology

Details of the Google worldwide private networks are not publicly available, but Google publications[34][35] make references to the "Atlas Top 10" report that ranks Google as the third largest ISP behind Level 3.[36]

In order to run such a large network, with direct connections to as many ISPs as possible at the lowest possible cost, Google has a very open peering policy.[37]

From this site, we can see that the Google network can be accessed from 67 public exchange points and 69 different locations across the world. As of May 2012, Google had 882 Gbit/s of public connectivity (not counting private peering agreements that Google has with the largest ISPs). This public network is used to distribute content to Google users as well as to crawl the internet to build its search indexes. The private side of the network is a secret, but a recent disclosure from Google[38] indicate that they use custom built high-radix switch-routers (with a capacity of 128 × 10 Gigabit Ethernet port) for the wide area network. Running no less than two routers per datacenter (for redundancy) we can conclude that the Google network scales in the terabit per second range (with two fully loaded routers the bi-sectional bandwidth amount to 1,280 Gbit/s).

These custom switch-routers are connected to DWDM devices to interconnect data centers and point of presences (PoP) via dark fibre.

From a datacenter view, the network starts at the rack level, where 19-inch racks are custom-made and contain 40 to 80 servers (20 to 40 1U servers on either side, while new servers are 2U rackmount systems.[39] Each rack has an Ethernet switch). Servers are connected via a 1 Gbit/s Ethernet link to the top of rack switch (TOR). TOR switches are then connected to a gigabit cluster switch using multiple gigabit or ten gigabit uplinks.[40] The cluster switches themselves are interconnected and form the datacenter interconnect fabric (most likely using a dragonfly design rather than a classic butterfly or flattened butterfly layout[41]).

From an operation standpoint, when a client computer attempts to connect to Google, several DNS servers resolve www.google.com into multiple IP addresses via Round Robin policy. Furthermore, this acts as the first level of load balancing and directs the client to different Google clusters. A Google cluster has thousands of servers, and once the client has connected to the server additional load balancing is done to send the queries to the least loaded web server. This makes Google one of the largest and most complex content delivery networks.[42]

Google has numerous data centers scattered around the world. At least 12 significant Google data center installations are located in the United States. The largest known centers are located in The Dalles, Oregon; Atlanta, Georgia; Reston, Virginia; Lenoir, North Carolina; and Moncks Corner, South Carolina.[43] In Europe, the largest known centers are in Eemshaven and Groningen in the Netherlands and Mons, Belgium.[43] Google's Oceania Data Center is claimed to be located in Sydney, Australia.[44]

Project 02

One of the largest Google data centers is located in the town of The Dalles, Oregon, on the Columbia River, approximately 80 miles (129 km) from Portland. Codenamed "Project 02", the million[45] complex was built in 2006 and is approximately the size of two American football fields, with cooling towers four stories high.[46] The site was chosen to take advantage of inexpensive hydroelectric power, and to tap into the region's large surplus of fiber optic cable, a remnant of the dot-com boom. A blueprint of the site appeared in 2008.[47]

Summa papermill

In February 2009, Stora Enso announced that they had sold the Summa paper mill in Hamina, Finland to Google for 40 million Euros.[48][49] Google invested 200 million euros on the site to build a data center and announced additional 150 million euro investment in 2012.[50][51] Google chose this location due to the availability and proximity of renewable energy sources.[52]

Modular container data centers

In 2005,[53] Google was researching a containerized modular data center. Google filed a patent application for this technology in 2003.[54]

Floating data centers
See also: Google barges

In 2013, the press revealed the existence of Google's floating data centers along the coasts of the states of California (Treasure Island's Building 3) and Maine. The development project was maintained under tight secrecy. The data centers are 250 feet long, 72 feet wide, 16 feet deep. The patent for an in-ocean data center cooling technology was bought by Google in 2009[55][56] (along with a wave-powered ship-based data center patent in 2008[57][58]). Shortly thereafter, Google declared that the two massive and secretly-built infrastructures were merely "interactive learning centers, [...] a space where people can learn about new technology."[59]

Google halted work on the barges in late 2013 and began selling off the barges in 2014.[60][61]

Software

Most of the software stack that Google uses on their servers was developed in-house.[62] According to a well-known Google employee, C++, Java, Python and (more recently) Go are favored over other programming languages.[63] For example, the back end of Gmail is written in Java and the back end of Google Search is written in C++.[64] Google has acknowledged that Python has played an important role from the beginning, and that it continues to do so as the system grows and evolves.[65]

The software that runs the Google infrastructure includes:[66]

Google has developed several abstractions which it uses for storing most of its data:[74]

  • Protocol Buffers – "Google's lingua franca for data",[75] a binary serialization format which is widely used within the company.
  • SSTable (Sorted Strings Table) – a persistent, ordered, immutable map from keys to values, where both keys and values are arbitrary byte strings. It is also used as one of the building blocks of Bigtable.[76]
  • RecordIO – a sequence of variable sized records.[74][77][78]

Software development practices

Most operations are read-only. When an update is required, queries are redirected to other servers, so as to simplify consistency issues. Queries are divided into sub-queries, where those sub-queries may be sent to different ducts in parallel, thus reducing the latency time.[39]

To lessen the effects of unavoidable hardware failure, software is designed to be fault tolerant. Thus, when a system goes down, data is still available on other servers, which increases reliability.

Search infrastructure

Index

Like most search engines, Google indexes documents by building a data structure known as inverted index. Such an index obtains a list of documents by a query word. The index is very large due to the number of documents stored in the servers.[42]

The index is partitioned by document IDs into many pieces called shards. Each shard is replicated onto multiple servers. Initially, the index was being served from hard disk drives, as is done in traditional information retrieval (IR) systems. Google dealt with the increasing query volume by increasing number of replicas of each shard and thus increasing number of servers. Soon they found that they had enough servers to keep a copy of the whole index in main memory (although with low replication or no replication at all), and in early 2001 Google switched to an in-memory index system. This switch "radically changed many design parameters" of their search system, and allowed for a significant increase in throughput and a large decrease in latency of queries.[79]

In June 2010, Google rolled out a next-generation indexing and serving system called "Caffeine" which can continuously crawl and update the search index. Previously, Google updated its search index in batches using a series of MapReduce jobs. The index was separated into several layers, some of which were updated faster than the others, and the main layer wouldn't be updated for as long as two weeks. With Caffeine, the entire index is updated incrementally on a continuous basis. Later Google revealed a distributed data processing system called "Percolator"[80] which is said to be the basis of Caffeine indexing system.[72][81]

Server types

Google's server infrastructure is divided into several types, each assigned to a different purpose:[39][42][82][83][84]

  • Web servers coordinate the execution of queries sent by users, then format the result into an HTML page. The execution consists of sending queries to index servers, merging the results, computing their rank, retrieving a summary for each hit (using the document server), asking for suggestions from the spelling servers, and finally getting a list of advertisements from the ad server.
  • Data-gathering servers are permanently dedicated to spidering the Web. Google's web crawler is known as GoogleBot. They update the index and document databases and apply Google's algorithms to assign ranks to pages.
  • Each index server contains a set of index shards. They return a list of document IDs ("docid"), such that documents corresponding to a certain docid contain the query word. These servers need less disk space, but suffer the greatest CPU workload.
  • Document servers store documents. Each document is stored on dozens of document servers. When performing a search, a document server returns a summary for the document based on query words. They can also fetch the complete document when asked. These servers need more disk space.
  • Ad servers manage advertisements offered by services like AdWords and AdSense.
  • Spelling servers make suggestions about the spelling of queries.

Security

In October 2013, The Washington Post reported that the U.S. National Security Agency intercepted communications between Google's data centers, as part of a program named MUSCULAR.[85][86] This wiretapping was made possible because, at the time, Google did not encrypt data passed inside its own network.[87] This was rectified when Google began encrypting data sent between data centers in 2013.[88]

Environmental impact

Google's most efficient data center runs at 35 °C (95 °F) using only fresh air cooling, requiring no electrically powered air conditioning.[89]

In December 2016, Google announced that—starting in 2017—it would purchase enough renewable energy to match 100% of the energy usage of its data centers and offices. The commitment will make Google "the world's largest corporate buyer of renewable power, with commitments reaching 2.6 gigawatts (2,600 megawatts) of wind and solar energy".[90][91][92]

References
  1. "How Many Servers Does Google Have?". Data Center Knowledge. Retrieved 20 September 2018.
  2. "Google data centers, locations". Google. Retrieved 21 July 2014.
  3. "Jackson County, Alabama". Google.
  4. Us, Contact; Directory, Staff; Notification, Local Project. "Google kicks off construction on M Alabama data center". Made in Alabama. Retrieved 2019-08-19.
  5. Dawn-Hiscox, Tanwen (February 20, 2018). "Google to spend m on Pryor data center expansion". Data Centre Dynamics. Archived from the original on April 23, 2019. Retrieved April 23, 2019.
  6. "Henderson, Nevada – Data Centers – Google". Google. Retrieved 26 October 2020.
  7. Tanwen Dawn-Hiscox (18 April 2017). "Google is planning a massive data center in Nevada". Datacenterdynamics.com. Retrieved 8 December 2018.
  8. Jason Hidalgo (16 November 2018). "Nevada approves Google's M data center near Las Vegas, M in tax incentives". Rgj.com. Retrieved 8 December 2018.
  9. Jason Hidalgo (16 September 2020). "Google to invest $600 million in data center near Reno, gets tax break". Reno Gazette Journal. Retrieved 26 October 2020. “With our new data center in Storey County and our expanded investment in our Henderson site, Google will have two facilities in Nevada, bringing our total investment to over $1.88 billion.”
  10. "Google ha decido de invertir millones de dólares en su centro de datos en Chile". Newtechmag.net (in Spanish). 28 September 2018. Retrieved 8 December 2018.
  11. "Google instalará un nuevo data center en Chile". diarioeldia.cl (in Spanish).
  12. Observador, El. "Google instalará un centro de datos en Canelones". El Observador. Retrieved 2020-08-20.
  13. ICNDiario. "El gigante Google confirma que instalará su centro de datos en Uruguay | ICNDiario" (in Spanish). Retrieved 2020-08-20.
  14. "Google confirmó la instalación de su centro de datos en el Parque de las Ciencias". Montevideo Portal (in Spanish). Retrieved 2020-08-20.
  15. "Dublin, Ireland – Data Centers – Google". www.google.com. Retrieved 2019-04-02.
  16. "Google to Spend $1.1 Billion on New Data Centers in Netherlands".
  17. Sverdlik, Yevgeniy (November 20, 2018). "Google to Build M Data Center in Denmark". Data Center Knowledge. Archived from the original on April 16, 2019. Retrieved April 23, 2019.
  18. <<Cite web|url=https://www.datacenterknowledge.com/google-alphabet/google-building-cloud-data-centers-close-swiss-banks%7Ctitle=Google Building Cloud Data Centers Close to Swiss Banks||
  19. "Google to Build Cloud Data Centers in Poland".
  20. Stiver, Dave (1 November 2017). "GCP arrives in India with launch of Mumbai region". Google Cloud Blog. Retrieved 2019-07-30.
  21. "Google purchases land for new data center in Tainan". Taipei Times. 2019-09-12. Retrieved 2019-12-20.
  22. "Google to set up data center in Tainan". Focus Taiwan. 2019-09-11. Retrieved 2019-12-20.
  23. "Google to set up second data center in Taiwan". Taiwan News. 2019-09-11. Retrieved 2019-12-20.
  24. "Google confirms plans to build 3rd data center in Taiwan". Taiwan News. 2020-09-03. Retrieved 2020-09-03.
  25. ""Google Stanford Hardware"". Archived from the original on February 9, 1999. Retrieved 2017-03-23.CS1 maint: bot: original URL status unknown (link). Stanford University (provided by Internet Archive). Retrieved on July 10, 2006.
  26. Merlin, Marc (2013). "Case Study: Live upgrading many thousand of servers from an ancient Red Hat distribution to a 10 year newer Debian based one" (PDF). Linux Foundation. Retrieved 2017-06-09.
  27. Tawfik Jelassi; Albrecht Enders (2004). "Case study 16 — Google". Strategies for E-business. Pearson Education. p. 424. ISBN 978-0-273-68840-2.
  28. Computer Architecture, Fifth Edition: A Quantitative Approach, ISBN 978-0123838728; Chapter Six; 6.7 "A Google Warehouse-Scale Computer" page 471 "Designing motherboards that only need a single 12-volt supply so that the UPS function could be supplied by standard batteries associated with each server"
  29. Google's secret power supplies on YouTube
  30. Google on-server 12V UPS, 1 April 2009.
  31. Google Green infographics
  32. "Analytics Press Growth in data center electricity use 2005 to 2010". Archived from the original on 2012-01-11. Retrieved 2012-05-22.
  33. Google Surpasses Supercomputer Community, Unnoticed?, May 20, 2008.
  34. "Fiber Optic Communication Technologies: What's Needed for Datacenter Network Operations", Research, Google
  35. Lam, Cedric F. (2010), FTTH look ahead — technologies & architectures (PDF), Google, p. 4
  36. James Pearn. How many servers does Google have?. plus.google.com.
  37. "kumara ASN15169", Peering DB
  38. "Urs Holzle", Speakers, Open Network Summit, archived from the original on 2012-05-10, retrieved 2012-05-22
  39. Web Search for a Planet: The Google Cluster Architecture (Luiz André Barroso, Jeffrey Dean, Urs Hölzle)
  40. Warehouse size computers
  41. Denis Abt High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities
  42. Fiach Reid (2004). "Case Study: The Google search engine". Network Programming in .NET. Digital Press. pp. 251–253. ISBN 978-1-55558-315-6.
  43. Rich Miller (March 27, 2008). "Google Data Center FAQ". Data Center Knowledge. Archived from the original on March 13, 2009. Retrieved March 15, 2009.
  44. Brett Winterford (March 5, 2010). "Found: Google Australia's secret data network". ITNews. Retrieved 2010-03-20.
  45. Google "The Dalles, Oregon Data Center" Retrieved on January 3, 2011.
  46. Markoff, John; Hansell, Saul. "Hiding in Plain Sight, Google Seeks More Power." New York Times. June 14, 2006. Retrieved on October 15, 2008.
  47. Strand, Ginger. "Google Data Center" Harper's Magazine. March 2008. Retrieved on October 15, 2008. Archived August 30, 2012, at the Wayback Machine
  48. "Stora Enso divests Summa Mill premises in Finland for million". Stora Enso. 2009-02-12. Archived from the original on 2009-04-13. Retrieved 2009-12-02.
  49. "Stooora yllätys: Google ostaa Summan tehtaan". Kauppalehti (in Finnish). Helsinki. 2009-02-12. Archived from the original on 2009-02-14. Retrieved 2009-02-12.
  50. "Google investoi 200 miljoonaa euroa Haminaan". Taloussanomat (in Finnish). Helsinki. 2009-02-04. Retrieved 2009-03-15.
  51. "Hamina, Finland". Google. Retrieved 2018-04-23.
  52. Finland – First Choice for Siting Your Cloud Computing Data Center. Archived 2013-07-06 at the Wayback Machine Accessed 4 August 2010.
  53. Metz, Cade (10 April 2009). "Google streams data center pods to world+dog". The Register.
  54. "United States Patent: 7278273". Patft.uspto.gov. Retrieved 2012-02-17.
  55. Rory Carroll (30 October 2013). "Google's worst-kept secret: floating data centers off US coasts". Theguardian.com. Retrieved 8 December 2018.
  56. Rich Miller (29 April 2009). "Google Gets Patent for Data Center Barges". Datacenterknowledge.com. Retrieved 8 December 2018.
  57. Martin Lamonica (8 September 2008). "Google files patent for wave-powered floating data center". Cnet.com. Retrieved 8 December 2018.
  58. "Google's ship based datacenter patent application surfaces". Datacenterdynamics.com. 7 September 2008. Retrieved 8 December 2018.
  59. "Google barge mystery solved: they're for 'interactive learning centers'". Theguardian.com. 6 November 2013. Retrieved 8 December 2018.
  60. Brandon Bailey (2014-08-01). "Google confirms selling a mystery barge". San Jose Mercury News. Retrieved 2015-04-07.
  61. Chris Morran (2014-11-07). "What Happened To Those Google Barges?". Consumerist. Retrieved 2017-01-15.
  62. Mark Levene (2005). An Introduction to Search Engines and Web Navigation. Pearson Education. p. 73. ISBN 978-0-321-30677-7.
  63. "Python Status Update". Artima. 2006-01-10. Retrieved 2012-02-17.
  64. "Warning". Panela. Blog-city. Archived from the original on December 28, 2011. Retrieved 2012-02-17.
  65. "Quotes about Python". Python. Retrieved 2012-02-17.
  66. "Google Architecture". High Scalability. 2008-11-22. Retrieved 2012-02-17.
  67. Fikes, Andrew (July 29, 2010), "Storage Architecture and Challenges", TechTalk (PDF), Google
  68. "Colossus: Successor to the Google File System (GFS)". SysTutorials. 2012-11-29. Retrieved 2016-05-10.
  69. Dean, Jeffrey 'Jeff' (2009), "Design, Lessons and Advice from Building Large Distributed Systems", Ladis (keynote talk presentation), Cornell
  70. Shute, Jeffrey 'Jeff'; Oancea, Mircea; Ellner, Stephan; Handy, Benjamin 'Ben'; Rollins, Eric; Samwel, Bart; Vingralek, Radek; Whipkey, Chad; Chen, Xin; Jegerlehner, Beat; Littlefield, Kyle; Tong, Phoenix (2012), "F1 — the Fault-Tolerant Distributed RDBMS Supporting Google's Ad Business", Research (presentation), Sigmod: Google
  71. "Anna Patterson – CrunchBase Profile". Crunchbase.com. Retrieved 2012-02-17.
  72. The Register. Google Caffeine jolts worldwide search machine
  73. "Google official release note". Google.com. Retrieved 2013-09-28.
  74. "Google Developing Caffeine Storage System | TechWeekEurope UK". Eweekeurope.co.uk. 2009-08-18. Archived from the original on 2011-11-15. Retrieved 2012-02-17.
  75. "Developer Guide – Protocol Buffers – Google Code". Code.google.com. Retrieved 2012-02-17.
  77. windley on June 24, 2008 1:10 PM (2008-06-24). "Phil Windley's Technometria | Velocity 08: Storage at Scale". Windley.com. Retrieved 2012-02-17.
  78. "Message limit – Protocol Buffers | Google Groups". Groups.google.com. Retrieved 2012-02-17.
  79. "Jeff Dean's keynote at WSDM 2009" (PDF). Retrieved 2012-02-17.
  80. Daniel Peng, Frank Dabek. (2010). Large-scale Incremental Processing Using Distributed Transactions and Notifications. Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation.
  81. The Register. Google Percolator – global search jolt sans MapReduce comedown
  82. Chandler Evans (2008). "Google Platform". Future of Google Earth. Madison Publishing Company. p. 299. ISBN 978-1-4196-8903-1.
  83. Chris Sherman (2005). "How Google Works". Google Power. McGraw-Hill Professional. pp. 10–11. ISBN 978-0-07-225787-8.
  84. Michael Miller (2007). "How Google Works". Googlepedia. Pearson Technology Group. pp. 17–18. ISBN 978-0-7897-3639-0.
  85. Gellman, Barton; Soltani, Ashkan (October 30, 2013). "NSA infiltrates links to Yahoo, Google data centers worldwide, Snowden documents say". The Washington Post. Retrieved November 1, 2013.
  86. Savage, Charlie; Miller, Claire Cain; Perlroth, Nicole (October 30, 2013). "N.S.A. Said to Tap Google and Yahoo Abroad". The New York Times. Retrieved March 9, 2017.
  87. Gallagher, Sean (October 31, 2013). "How the NSA's MUSCULAR tapped Google's and Yahoo's private networks". Ars Technica. Condé Nast. Retrieved March 9, 2017.
  88. Miller, Claire Cain (October 31, 2013). "Angry Over U.S. Surveillance, Tech Giants Bolster Defenses". The New York Times. Retrieved March 9, 2017.
  89. Humphries, Matthew (March 27, 2012). "Google's most efficient data center runs at 95 degrees". geek.com. Archived from the original on June 13, 2016. Retrieved June 13, 2016.
  90. Hölzle, Urs (December 6, 2016). "We're set to reach 100% renewable energy — and it's just the beginning". The Keyword Google Blog. Retrieved December 8, 2016.
  91. Statt, Nick (December 6, 2016). "Google just notched a big victory in the fight against climate change". The Verge. Vox Media. Retrieved December 8, 2016.
  92. Etherington, Darrell (December 7, 2016). "Google says it will hit 100% renewable energy by 2017". TechCrunch. AOL. Retrieved December 8, 2016.

Further reading
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.