How Google's Core Data Solutions Team Uses Go - The Go Programming Language

Common problems companies solve with Go
Stories about how and why companies use Go
How Go can help keep you secure by default
The official Go language specification
A complete introduction to building software with Go
Reference documentation for Go's standard library
Learn what's new in each Go release
Tips for writing clear, performant, and idiomatic Go code
Videos from prior events
Meet other local Go developers
Learn and network with Go developers from around the world
The Go project's official blog.
Get help and stay informed from Go
Google’s mission is “to organize the world’s information and make it universally accessible and useful.” One of the teams responsible for organizing that information is Google’s Core Data Solutions team. The team, among other things, maintains services to index web pages across the globe. These web indexing services help support products like Google Search by keeping search results updated and comprehensive, and they’re written in Go.
In 2015, to keep up with Google’s scale, our team needed to rewrite our indexing stack from a single monolithic binary written in C++ to multiple components in a microservices architecture. We decided to rewrite many indexing services in Go, which we now use to power the majority of our architecture.
Go’s built-in concurrency is a natural fit because engineers on the team are encouraged to use concurrency and parallel algorithms.
When choosing a language, our team found that several of Go’s features made it particularly suitable. For instance, Go’s built-in concurrency is a natural fit because engineers on the team are encouraged to use concurrency and parallel algorithms. Engineers have also found that “Go code is more natural,” allowing them to spend their time focusing on business logic and analysis rather than on managing memory and optimizing performance.
Writing code is much simpler when writing in Go, as it helps lessen cognitive burden during the development process. For example, when working with C++, sophisticated IDEs might, “show that the source code has no compile error when there actually is one” whereas “in Go, [the code] will always compile when [the IDE] says the code has no compile error,” said MinJae Hwang, a software engineer on the Core Data Solutions team. Reducing small friction points along the development process, such as shortening the cycle of fixing compile errors, helped our team ship faster during the original rewrite, and has helped keep our maintenance costs low.
“When I’m in C++ and I want to use more packages, I have to write pieces such as headers. When I’m writing in Go, built-in tools allow me to use packages more easily. My development velocity is much faster,” Hwang also shared.
With simple language syntax and support of Go tools, several members of our team find it much easier to write in Go code. We’ve also found that Go does a really good job of static type checking and that certain Go fundamentals, such as the godoc command, have helped the team build a more disciplined culture around writing documentation.
…Google’s web indexing was re-architected within a year. More impressively, most developers on the team were rewriting in Go while also learning it.
Working on a product used so heavily around the world is no small task and our team’s decision to use Go wasn’t a simple one, but doing so helped us move faster. As a result, Google’s web indexing was re-architected within a year. More impressively, most developers on the team were rewriting in Go while also learning it.
In addition to the Core Data Solutions team, engineering teams across Google have adopted Go in their development process. Read about how the Chrome and Firebase Hosting teams use Go to build fast, reliable, and efficient software at scale.
Google is a technology company whose mission is to organize the world’s information and make it universally accessible and useful.
In this case study, Google’s Core Data Solutions team shares their journey with Go, including their decision to rewrite web indexing services in Go, taking advantage of Go’s built-in concurrency, and observing how Go helps to improve the development process.