How Go Allows You to Build a Scalable System

Published in

FAUN — Developer Community 🐾

11 min readJun 2, 2019

At Opera, despite being known as a web browser company, we have been experimenting and developing a portfolio of different products spanning from compression technology to AI news engine. The newest kid in the block is a mobile-first payment platform, which has recently overtaken the second spot in Nigeria as one of the largest mobile money companies in spite of being in the competition for a year. I believe Go has contributed to a large part of our success, enabling us to cope with the hockey-stick growth in user base and transaction volume.

There are commonly misconceptions, or rather incomplete views, about scalability of software systems. Not only is a highly-scalable system able to cope with high demand of user queries, it also needs to be maintainable by a fast growing team of people with different technical backgrounds. As a matter of fact, a majority of our team are novices to Go, yet are able to quickly get up to speed and drive the project forward.

There are several reasons why Go was chosen (one of them be that our director of engineering explicitly requested so), but they are distilled down to:

Simplicity
Performance (compilation/execution)
Tools/Libraries/Community

Simplicity

Go is a simplistic language, that is what everyone, despite distantly knowing Go philosophy, would first figure when getting in touch with the language. No matter from what paradigm of programming you come from, Go is a language which you could master within weeks. Coming from a background of Java, C# and C++ myself, I did not really miss the variety of features they provide. On the contrary, I feel much more freedom within the carefully restricted set of things which Go allows me to do.

I’m writing this article on an airplane back from Fosdem 2019, an open source conference held in Brussels, and what I remember the most about the Go talks was one particular slide where the speaker explained the new features added to Go since last year. The slide reads:

“This is intentionally left blank.”

Brilliant! Not only was I delighted because of the humour, but I also appreciate that Go actually sticks to its philosophy of simplicity and no redundancy. “Less is more” after all. Why do we need all the synthetic sugar added just so a couple of lines of code could be saved? For example, Go does not give you the fundamental ternary operator which could easily be substituted by a simple if-else. The following examples are written in C#, only the first is equivalent to Go:

Example 1:

if a != nil {
  x = a
} else {
  x = b
}

Example 2:

x := a != nil ? a : b

Example 3:

x := a??b

The 1st example is easiest to understand and is applicable in most programming languages.

The 2nd example is more concise but will become hard to read when there are multiple nested values.

The 3rd example is clearly synthetic sugar for one narrow scenario. It’s not apparent what it does if you are not familiar with the syntax. If you ever learn a second language, you’d feel the struggle speaking to native speakers as they don’t always use expressions in text books.

Note simplicity does not equal conciseness. You could write a brilliant one-liner which nobody could comprehend such as the following example of “Conway’s game of life” implementation in APL:

life←{↑1 ⍵∨.∧3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}

Or you could write an averagely smart piece of code which everyone in the team could understand and maintain. Software systems have grown too large that hardly anything is built by a single person. As the team grows and members change, the code is not an asset but liability. Brilliant code is code that could be understood by an average developer in the team.

I could rant on about various features, or the lack thereof, but among all, I’d love to discuss the following:

Implicit interface implementation
Missing of inheritance
Circular dependency prohibition

Implicit interface implementation

Before deep-diving into each of the points, I would like to take a step back and enumerate on programming fundamentals and principles. Programming fundamentals are all about abstraction/encapsulation and dependency, in other terms, “who knows what”. Then come programming principles where the fundamentals are elaborated on how we should design abstraction and dependency effectively. Here are some of the principles which I believe are the most significant:

High cohesion/Low coupling
Favour composition over inheritance
SOLID (Single responsibility, Open/Closed, Liskov substitution, Interface segregation, Dependency inversion)

I love this quote when it comes to explaining dependency inversion: “Would you ever solder a lamp directly to the electrical wiring in a wall?” Of course we wouldn’t, that’d be silly right? The modern world relies on specialisation if different areas. There’s no single company which could build an entire airplane. Components are developed all over the world and shipped to a central place for assembling. In order for this to happen, components must be built according to some standards, or interfaces, the same with software systems. Let’s take a look at the following example:

Figure 1 demonstrates a strict dependency of object A on object B. It implies that a complete implementation of B must be provided at the time A is constructed, and that A “knows” about B specifically. The problem comes when you want to replace B with C, then you either need to fix B totally or change all references to C. If you are lucky, you won’t need to replace B ever in your life, but another problem comes when you would like to write a component test for object A, then you still need to construct B every time A is tested. If there’s a test failure, you won’t know if the failure is inherent to A or if it’s a bug from implementation of B.

Dependency inversion solves this problem by introducing an interface in the same package where A is located and this interface is implemented by object B. Figure 2 shows us that the dependency now has been inverted as the arrow now comes from B to A. Go decided to take a step further by eliminating all together the dependency between A and B thanks to implicit interface implementation. In languages such as Java and C# you need to declare if a class implements an interface, in a way, creating a strict dependency between an interface and its implementers. Go does not think such relationship should be classified as dependency simply because an implementer does not need its interface to function. Object A does need whatever implementing the interface A to work (dependency), but B simply works without knowing about what contract it conforms to.

Missing of inheritance

This might be one of the boldest design decisions the authors of Go has made, also one of the most brilliant, not supporting the fundamental building block of object-oriented even though Go is a multi-paradigm languague itself. We need to understand that inheritance is a dependency. A subclass needs a parent class to function because the parent class partially dictates how the subclass should behave. In practice, inheritance is also a headache once the level of inheritance spirals out of hand. Take Android’s MultiAutoCompleteTextView for example, it’s the 5th level from android.view.View.

java.lang.Object
↳ android.view.View
↳ android.widget.TextView
↳ android.widget.EditText
↳ android.widget.AutoCompleteTextView
↳ android.widget.MultiAutoCompleteTextView

It’s virtually impossible to get a snapshot of its implementation without jumping back and forth among the inheritance chain with different overriding methods. Complication leads to less readability and maintainability. To further demonstrate the complication of inheritance, let’s take a classic example:

Imagine you are to implement 2 classes “Square” and “Rectangle”, it’s intuitive to let “Square” extends “Rectangle”, right? Not so fast… Check out the code below:

public class Rectangle {
  void SetSize(int x, int y) {}
}public class Square extends Rectangle {
  void SetSize(int x, int y) {}
}

If Square extends the method SetSize, would it set 2 sides to x or y? This is an outright violation of Liskov substitution as a Rectangle could not be substituted by any Square objects in the system.

Anyhow, the real question is what inheritance tries to solve? Two things: extensibility and substitution. We certain would like to extend a class with more functionality without duplicating code, yet conforming with single responsibility and open/closed principles. Composition and interface come at your disposal. Want to extend a class? Use composition. Want to substitute a class? Use interface.

Circular dependency prohibition

Actually with static linking during compilation, circular dependency is impossible because if A depends on B, B has to be built first. Languages have tried to overcome this “shortcoming” with different approaches such as forward declaration in C++. However, circular dependency signifies something fundamentally wrong in system design and this is easily encountered by inexperienced programmers. Disallowing that forces developers to rethink their design improves code quality. You are free to believe in the interconnectedness of everything in the universe but spaghetti code is not something we are in favour for.

On simplicity topic, Rob Pike, one of the Go authors, had a great talk back in 2015, where he explained how complicated it was to build simplicity into Go, including garbage collection and concurrency. I would strongly recommend it.

Performance

One of the biggest selling point of Go is performance. If you come to go, most of the times you’re on for performance. Performance is achieved thanks to various characteristics of Go as iterated below:

Go is built into binary so it operates directly with hardwares as opposed to Java which is built into byte code which runs in JVM, introducing an extra layer of overhead.

Efficient/Effective garbage collection: even though Go tries to do as much as possible concurrently in the background to collect unused objects, there’s still an embarrassing stop-the-world pause which is inherent to the mechanics of Go GC, where the whole system has to pause for a short while to wait for GC to finish collecting and sweeping garbages. The go team has tirelessly worked to improve this. By 2005 the gap was brought down to 10ms and as of now sub millisecond, virtually non-existent.

Values are compact, for example: var x int32 = 2018

4 bytes in Go
24 bytes in Python
4 bytes in Java but if used in map, 16 for x32, 24 for x64 JVM

It might sound irrelevant as modern day memory storage is no longer a problem. However, if you look at the graph below, there’s a widening gap between CPU clock speed and memory bus speed. As Dave claims in his blogpost, “the difference between the two is effectively how much time the CPU spends waiting for memory.”

https://dave.cheney.net/wp-content/uploads/2014/06/Gocon-2014-10.jpg

There are more reasons why Go is inherently performant as listed in the blogpost but of course we can’t ignore the simplicity of parallelism enabled by lightweight go routines, straightforward orchestration mechanism with “select”, mutexes and communication across go routines with channels. There’s no magic in go, you get what you ask for and that transparency enables you to build a system around correctness instead of scalability. Bill Kenedy, a go speaker, has coined a term “optimise for correctness” in his similarly named presentation. With go doing all the hard-lifting work on performance, developer’s responsibility is to build a functioning system without caring much about scalability. The payment system we are building has hit several system limits as we grow but most of the times, bottlenecks reside somewhere else, from file descriptor to database.

I admit that I was bit exaggerating saying we don’t care about scalability, we actually do and have performed lots of optimisation a long the way and Go has provided us with an awesome default tooling for benchmarking, race detecting and profiling, thus, we never have to guess the bottleneck and could correctly estimate and optimise even before it could have become a problem.

Profiling

The first step toward optimisation is detecting bottleneck, hence, profilers should always be a friend of performance savvy. In Go, it’s built-in the default test tooling which could easily be abled with a flag.

$ go test -cpuprofile cpu.prof -memprofile mem.prof

The generated result could be displayed by the default pprof, or better, by the one from Google.

$ go tool pprof cpu.prof
or
$ go get github.com/google/pprof
$ pprof -http=:8080 cpu.prof

The image below shows an example of the profiling result where critical path is highlighted in red, that’s where most of your optimisation will happen.

https://matoski.com/article/golang-profiling-flamegraphs/

Benchmarking

After detection comes optimisation and benchmarking the solution. Writing and running a benchmark is not any difference from writing and running a unit test:

$ go test -bench=.
func BenchmarkFoo(b *testing.B) {
  for i := 0; i < b.N; i++ {
    // perform the operation we’re analyzing
  }
}

By default, the benchmark will run as many operations as possible within 1 second and print out the result similarly as below:

goos: darwin
goarch: amd64
pkg: opera/transaction/engine
BenchmarkRunner_Run__0100_runners_00_listeners-4 10 4776570082 ns/op
PASS

Go-ing forward

Building a scalable system is not rocket science, it boils down to selection of fundamental archetypes and Go has an excellent foundation for enabling that to happen. Not only is the language itself inherently performant, it also strengthens collaboration, readability and maintenance thanks to its simplicity and the carefully calibrated features which align with programming principles.

Go however is not all rainbows and unicorns, improvement still needs to be done to steamroll the messiness of error handling for instance. Generics is also one of the most wanted feature in Go, which has recently showed up in Go 2.0 proposal. To be honest, I have not found myself nostalgic about generics at all since moving on to Go. I believe Go will still stick to its philosophy and maintain the principles which help it stand out from the rest as we are go-ing forward.

Also published on https://techproject.io/