Golang's pprof, generating and reading profiles — Part 1

Profiles are really important tools when it comes to collecting metrics and measuring resources consumption. For that purpose, Go has a tool called pprof that allow us to collect data from our Go applications, such as memory and CPU usage, allocations and others.

We can achieve that using two methods: Generating files or reading from an http web server. This article is a Part 1 and just the generating files will be covered. Part 2 will cover the usage of the http web server.

I’ll be very straight forward with the examples, so do not bother about the code refactoring.

Note: Also, I'm aware that pprof has an interactive way of visualizing profiles, but I don't like the idea that for doing so I have to import the "testing" package and add it has a dependency to the function I want to test.

Prerequisites:

Just Go (obviously) and Graphviz.

Graphviz is a program that Go’s pprof use to generate our files in the formats we want.

You can easily install Graphviz following the instructions on the official website https://graphviz.gitlab.io/download/ .

Generating files

In order to generate our files we need to import the os and the runtime/pprof packages to our project. The first one will be used to generate a stream that will be our file, and the second will be the one responsible to write the data to that file. The other packages are just for the purpose of the code.

package main
import (
    "fmt"
    "os"
    "runtime/pprof"
    "sync"
    "time"
)
type Storage struct {
    sync.RWMutex
    HashMap map[string]string
}
var storage = Storage{
    HashMap: make(map[string]string),
}
func runApp(numOfTimes int, c chan<- struct{}) {
    id := fmt.Sprintf("%s", time.Now().Format("20060102150405"))
    
    for t := 1; t <= numOfTimes; t++ {
        storage.Lock()
        storage.HashMap[id] = id
        storage.Unlock()
    }
    c <- struct{}{}
}
func main() {
    fmt.Println("Starting...")
    cpuProfile, _ := os.Create("cpuprofile")
    memProfile, _ := os.Create("memprofile")
    pprof.StartCPUProfile(cpuProfile)
    fmt.Println("Started")
    c1 := make(chan struct{})
    c2 := make(chan struct{})
    c3 := make(chan struct{})
    go runApp(50000, c1)
    go runApp(50000, c2)
    go runApp(50000, c3)
    <-c1
    <-c2
    <-c3
    pprof.StopCPUProfile()
    pprof.WriteHeapProfile(memProfile)
    fmt.Println("Finished!!!")
}

Understanding the code

The cpuProfile, _ := os.Create("cpuprofile") line is creating a stream that will be used to generate the file named cpuprofile (an absolute path can also be used). The same logic is used in the following line for the file memprofile.

After that, we start to collect the profile data with the line pprof.StartCPUProfile(cpuProfile) , but we don't need to do that for the memory, because Go's runtime already have that data.

Then we call our runApp() function that is nothing but a source of data for us.

Finally, we finish the data collecting with the last two lines (just ignore the Println call). At this point, both files should have been created at the same location as the binary of our code (or at the path that you passed).

Reading the profiles

The files are generated in binary format, so, unless you are The Chosen One, you won’t be able to read anything from them, that’s why Go supposes that you are not Neo and provides a Go tool called pprof. pprof is used to read the profile binary files that we generated previously in our code.

Using pprof

Using pprof is very straight forward, we simply need to call it passing the following arguments: the format which we want to generate a readable profile, the binary of our code generated with go build or go run (in this case, the file will be in the bin folder of the GOPATH) and the generated binary profile. Example:

go tool pprof -pdf ~GOPATH/src/go-your-code/binary-file cpuprofile > cpuprofile.pdf

The > cpuprofile.pdf is us to telling pprof to generate a readable pdf file from the binary profile that we are reading. Remember, you are not Neo.

After running that command, a pdf file with the name cpuprofile.pdf must have been created and its content is probably similar to this:

The memory profile looks like this:

The profiles show metrics about the usage of resources during the app life cycle and its own chain of calls.

Go's pprof also accepts other formats, such as web (which will generate a HTML file), png and graph. I'll talk about them in Part 2.

Also, other types of profiles can be generated, such as Tracing Profile that can be used to analyze latency throughout the lifecycle of a chain of calls, Allocs Profile (to analyze allocations), Threadcreate Profile (for created threads), Block Profiles (blocking on synchronization) and others. I will also talk about them in Part 2.

That's it folks. Be ready for Part 2. Thank you!

— — — — — — — — — — — — — — —

Any suggestion, please, contact me:

wanderson.olivs@gmail.com
LinkedIn: https://www.linkedin.com/in/wandsilva/
Instagram: @wandsilva_